Latvian Gambit or Caro-Cann
That's an odd pair of defences to 1. e4 to be choosing between. The Caro-Kann has more literature available, will teach you a lot more about strategy, and is sound. The Latvian has a few very detailed books and a lot of tommy-rot talked about it. Highly tactical in most lines, will yield some pretty wins and some crushing losses. Plus, of course, the player of the Latvian must also learn something against 1. e4, e5 and then: a) 2. Bc4 and b) 2. Nc3 and c) 2. f4 and d) 2. d4 none of which lend themselves to quite such aggressive play for Black. So learning the Latvian will not guarantee you a tactical slugfest. All depends what you're looking to achieve.
Strange question. Latvian is an unsound gambit. Caro-Kann is solid, positional, and can lead into an endgame. Like Latvian versus Elephant Gambit or versus Alehkine's or Sicilian. Caro-Kann versus Petroff's, French, or Pirc.
That's quite the mismatch in defenses, Latvian vs Caro. You would probably improve faster by choosing a classical 1.e4 e5 defense like the Petrov; that being said you should choose the defense that you are attracted to--if its chaotic swordfights choose the Latvian, if it's forcing your opponent to prove he can storm your castle walls, choose the Caro.