When do I castle?

to get your king to safety is the usual goal, along with getting your rooks together and in good spots. Castling is a mistake when you miss a better move or if you 'castle into trouble' which is hard to see at first, but if all his pieces point at your left corner, don't go left...

if you and your opponent castle on the same side the game should be a bit more tight. if you castle long, move your king over and protect the hanging pawn.
the purposes of castling are twofold. trying to get the king to safety and out of the center, and also trying to get the rook into play. if you only have to prioritize one it would be the safety of the king.
now if castling gets your king straight into an enemy attack then it shouldn't be done since you didn't fulfill the purpose of getting the king to safety.
another case would be if there is a move leading to a win of material or mate for you on this turn and you castle instead, then you might have fulfilled two purposes of castling but winning material or checkmating have a higher priority than getting the king to safety or activating the rook, so here castling is still wrong.
a related idea is if the opponent is threatening a move on his next turn to win material or checkmate you and on your turn now you castle. let's say it still fulfills the two purposes of castling, but not losing material and not getting checkmated have a higher priority still than getting the king to safety or activating the rook.
so first make sure there are no tactics that win material or checkmate for you or the opponent, and then check if your king will indeed be safe on the side where you intend to castle, or if you need the rook in the corner, and only then decide to castle.
i know this is lengthy but i hope it makes some sense and helps.
The blanket "it's good to castle early" is an over-simplification. You want to leave the option of castling open as long as possible, because it gives your king mobility which it will lose otherwise. Most of the time, getting all your pieces off the back rank is a higher priority.
The reasons for this are twofold. First, if you still have knights and/or bishops stuck on the back rank, then your rooks don't connect, which diminishes the advantage of castling. Second, until your minor pieces are bearing down on the center, there will be very little action on the center files, and therefore very little for your rooks to do there.
Most importantly, it is important to judge when you are castling into trouble. If your opponent has a bunch of pieces bearing down on your kingside, and/or all your own pieces are stuck over on the queenside, it's usually a bad idea to castle that way. In general, it's usually advantageous to castle after your opponent does, so you can survey the situation more clearly. If both of you castle the same way, then pawn storms become far less likely, but if you castle in opposite directions, each will have two rooks to support such pawns.
Lastly, if castling will thwart a plan your opponent has in mind, it's best to wait until the last moment to do it. Always let your opponent waste as many moves as possible on a plan that won't work before you spring the proper defense.

Thanks for the great insights, everyone! I will try to be more observant over the whole board before I castle. Hopefully that will see me castle on more appropriate/advantageous moves.

Also note that computer analysis is not always practical in these cases. Computers are virtually flawless at calculation and so it has no fear not castling or pushing pawns in front of it's king. But for a human player, especially lower rated, castling is a fine answer unless you hang a peice or lose a tactical exchange as a direct result.