I have actually never said that anything involving physical activity is a sport, I have however stated that using the literal definition of physical activity you can not disqualify chess from being a sport (by definition).
"It means using your body in some way to influence something or perform an act."
Very well I will accept this definition of physical activity, now will you kindly name an action that your body can perform which does not fit this description? Playing chess is certainly "using your body in some way to influence something or to perform an act".
Your last point has absolutely nothing to do with anything by the way.
You are the one saying that all activities are physical and the definition you use does NOT use that as it's premise. Simple automatic body actions like breathing or your heart beating are not included in the popular definition of physical activity. Otherwise why is the word physical needed to describe certain activities?
You don't even need to move the chess pieces to play, you could just tell your moves to another and he can move for you. Eliminating all physical activity on your part. If you say speaking is one then I give up. You're just taking the broadest view possible for physical activity and that is not the popular definition, only one you acknowledge.
Alright now my question is to you.
What level of physical activity is required before you decide that a 'game' is a 'sport'?
Does shooting count as a sport to you?

I see sport as something which involves the body moving or being active ... and in a way Chess does sort of do that ... you move your hands and it keeps your mind active ... but I would say it was more a game than a sport in comparison to football, rugby, hockey, snooker, golf etc?