As an analogy: flipping a coin is also a game, but it's not a sport.
As to the last question: does it matter how many people take it seriously? Soccer, for instance, lacks the popular support here in the States, compared to the rest of the world - it's not taken seriously here. Does this in any way impacts your enjoyment of soccer? (Assuming you like soccer...)
I've never heard chess referred to as a sport, but this is the definition of what a sport is:
"An activity involving physical exertion and skill in which an individual or team competes against another or others for entertainment."
I don't really mind either way - I'm just curious as to what other people think about this. Maybe chess would be taken more seriously if it was referred to as a sport?