You could use your argument to say that watching TV is a sport, because you could be tired from hours of watching TV.
Chess doesn't involve physical exertion, and isn't a sport.
Watching TV isn't a sport though. 1. Not competitive and 2.NO skills required.
Even if chess isn't a sport it's definately LIKE a sport - we can all agree there.
It is true that chess torunaments have frequently been known to take EXTREME physical tolls on players. From the highest ranks to the lowest.