Buffering Dilemma

How do I know if I’m buffering too much? Are we allowed to buffer sometimes? Every time I sit down to watch TV or even listen to music while driving I feel guilty. Is buffering about making bad decisions in the moment rather than planning to watch TV, relax, etc ahead of time.?