Why wait until 2020? There was a World Mindsports Olympiad in Beijing in 2008, and they promised to have one in London as well.
The only problem is that no one really watched, and I'm pretty sure it lost money, and they group that put it on hasn't updated their web site in a couple of years.
It turns out that most people do want to watch athletic young people put forth great effort as they perform great physical feats and push their bodies to the limit. However, the appeal of watching great minds in a battle of concentration and calculation is.....not quite as interesting.
The World Mindsports Olympiad did get a brief segment on the Stephen Colbert show, which means it did not go completely unnoticed. While any publicity is better than no publicity, being featured on The Colbert Report does not exactly indicate that your activity has achieved popular acclaim. Indeed, it might suggest the opposite. Despite the association that IMSA tried to establish with the Olympics, Chess somehow failed to draw the sort of audience that might make the Olympic Committee consider changing its rules, which currently specify that only athletic competitions can be included in the games.
ETA: It seems that there is at least some effort to keep this sort of competition alive. There will be an International Mind Games in Beijing in 2011. Chess will have to share the spotlight once again with Go, Xiangqi, Draughts (Checkers), and Bridge.
http://www.sportaccord.com/en/multi-sports-games/index.php?idIndex=35&idContent=658
Short games , best of three and let's have a chess OLYMPIC CHAMPION !