If you wait too long, you won't have the chance.
Castling Early

in that case , castle later !
You mention your concerns about your opponent's moves, but castling is also part of your own development.
There is no correct answer to your question.

decide position wise. a lot of times I look at my opponent and ask, "is his king safe?" Its often a good thing to allow your opponent to commit himself first. I look at what attacks I can defend against while winning an advantage. I believe if you have a strong initiative, and his king is in danger, that the initiative is worth more. Just keep in mind that if the game opens up you want to have a safe king. :)
Castle early in open games where the center isn't locked or can be opened quickly, wait to castle if the center is locked as usually attacks go to the flanks, where an early castled. King provides an incentive and possible advantage to attack that flank.

Castling is very situational. Sometimes castling early is a mistake. Sometimes it's a mistake to delay castling.
My most recently completed rated game was a pertinent example... as Black in a Sicilian Najdorf, I castled on move 27 (!) and at the time I was a bit concerned that I might be castling too early.
http://www.chess.com/forum/view/game-showcase/a-rook-sac-followed-by-a-quiet-move
I know that castling early and getting your king to safety is a basic opening principle, but I was wondering if it is better to wait until your opponent starts an attack and then castling. I'm worried that castling early allows your opponent to attack the castled side, whereas castling later forces them to shift all their pieces across the board.