I myself have played the Caro-Kann for a very long time and have even got several bits of help to improve theory with it. I have a good friend who plays the French defense and in our countless games I agree that it can be tricky for white to defend d4, however it does become second nature after playing against it so many times.
Using the Caro-Kann has won many games against players that could often beat me. I tend to play it because people who are not versed in it struggle against it, and people who are versed in it, can still be disadvantaged if they play into theory. It is also nice in the Caro to get the light squared bishop out so it does not become a problem in the French. Regardless both opening are great opening and can carry people up to the ratings of 1500 or higher and still improve their positional play with experience in unique positions. However the Caro-Kann does have at least 6 important lines any serious Caro player needs to learn; Fantasy, Panov, Advanced, Exchange, Two Knights, and Classical. So it can be overwhelming for some players unexperienced in learning openings.
The Caro to me is preferable over the French since it can be more dangerous for white whereas in the French white can sort of improv after playing c3 for a couple moves.
Finally decided to try out the Caro-Kann at this week's chess club, and got 2 relatively easy wins against otherwise decent players. Didn't change my mind about the opening itself - it is slower and less dynamic than the French Defense - but I do think I understand its appeal to the amateur player a bit better.
Following Petrosian's example (Petrosian did get excellent results with it), I treated the C-K as a delayed French Defense, and the extra tempo afforded to white gives the opponent another turn to overextend - which both of my opponents this week did. Of course, this is still hoping for the opponent to blunder (as Lasker taught us, hope is not a strategy), but a lot of seemingly innocuous pawn moves do turn out to be serious mistakes.
Still prefer the French Defense (which forces white to defend a weakness on d4), but I must admit it's a nice change-of-pace to have in the arsenal. Hypermodern positional rules still apply, and since I have read My System, this is the kind of opening I can understand intuitively.