You'll end up knowing the colors automatically after playing for a while.
I didn't think about it until I saw a video with Daniel Rensch
http://www.chess.com/video/player/achieving-full-board-awareness
I have been playing chess for 5 years in a club and never thought about that a1 to h8 is black.
...or you could just open your eyes and look...
However, on a more serious note, I'd be interested to learn how blindfold chess players visualize the board, and what sort of system they use.
I cannot claim to have trained or to be even halfway decent at blindfold, but blindfold abilities "just happen" to come to you when you play regular chess. The "system" is most probably that you do not remember single pieces or square but some set of pieces defending each other or stuff like that.
Now, the guys who play >10 simultaneous games might have a mental trick... But playing one blindfold game without losing the count of the pieces is not too hard. Of course you will miss three-move deep tactics, but you won't hang a piece.