Castling too early can lead to a loss of initiative and allow the opponent to know where to attack but you shouldn't delay castling only for the reason that your opponent hasn't done so yet.