I think the best "comeback" to someone saying that chess is nerdy or making fun of you for playing chess - apart from ignoring them - just tell them to grow up. Then one day further down the track when they realise that they were being idiots you can say "i told you so" and if you run into them in the future and they still are immature brats then you can say "I thought I told you to grow up"
Hows that for thinking a few moves ahead?
I'm not a fan of absolutes (not even the vodka)
If all a person did was play and think about chess then yes they go into the spectrum of geek/nerd in the same way if all a sportsplayer does is live and breathe their sport than they are a jock. The thing is though, that there are very few people in this world that go to that extreme. And even when they are in that extreme you will probably find that they actually do other activities as well that go counter to the semantics of nerd/jock.
For example Bobby Fischer boxed (or so I've read)... boxing and nerd are hardly mutually exclusive but at the same time not really something you would expect.