When you have a fairly demanding day-time job you are left with let's say 4 hours daily for everything else in your life (family, friends, career, spirituality, sciences as somebody else mentioned). Now, if you spend 2 hours daily on chess (50% of your leisure time) I'm sure that pretty soon you'll realize that you've neglected other things which you consider more important.
The problem with chess is that it sucks a lot of time for small improvements and without those improvements you wouldn't do it anyway.
Personally, I'm trying to cut down on my time spent on chess to better reflect its priority on my list.
I went a good 15 years without playing at all for some of these reasons plus work. I started playing again when the kids were in their 20's. Maybe if I'd played a couple hours a week during that time I'd be a better player now, but I don't regret it. Maybe the key is to identify your priorities like you have.
. Chess is something you can take pride in but I don't pride in watching TV. Chess is something to talk about when all else fails. Usually when you enjoy an activity it is a waste of time but chess is a purely intellectual game unlike gambling or other board games ( which do require thinking but has to do with luck) 
I think we are discovering a new religion.
No fun, only work!
That's a very old religion.