The fianchetto is a good way to go about it, but when I played the Caro-Kann I also felt that I got poor positions. You could try to play aggressively with c5, e5, etc. Overall, I do feel that the Fantasy is really hard to play against. Small part of the reason I switched to the Sicilian
How to counter the Caro-Kann (Fantasy Variation)?

Qb6 is an interesting sideline that morozevich likes. It’s also a recommendation of IM Christof Silecki in KIS black. Otherwise I would just stick to e6 and Bb4.
E5 might be a good side weapon recommended by GothamChess as a surprise weapon

As one that played the Fantasy against the Caro when I was an e4 player, I can also confirm the most problematic was the 3...Qb6 line.

I like ...dxe followed by ...e5.
I've added some arrows to show the basic threats/ideas for each side.
Once this position is reached, it's more or less equal. Both players just ... play chess from here on out.

I like ...dxe followed by ...e5.
I've added some arrows to show the basic threats/ideas for each side.
Once this position is reached, it's more or less equal. Both players just ... play chess from here on out.
As I recall, this is the mainline, correct? If I still played the Caro-Kann as Black, this would be my line against the Fantasy.

This is quite dangerous and probably too open for a caro kann player
It felt uncomfortable the first few times I played it, but after that (and after analyzing it a bit), it started to feel comfortable.
Now I see the Fantasy and enjoy going into this. It has a different feel from other Caro-Kann variations, which can be a nice change of pace.
Sam Shankland has a 23-minute video explaining all the main ideas of playing this variation, from the Black side: https://www.chess.com/lessons/the-complete-caro/the-complete-caro-the-fantasy-variation

As I recall, this is the mainline, correct? If I still played the Caro-Kann as Black, this would be my line against the Fantasy.
It's one of the mainlines, yes.
... e6 is actually the top mainline move, though.
That can lead to a transposition into the French, however, which not all Caro-Kann players may like:

As I recall, this is the mainline, correct? If I still played the Caro-Kann as Black, this would be my line against the Fantasy.
It's one of the mainlines, yes.
... e6 is actually the top mainline move, though.
That can lead to a transposition into the French, however, which not all Caro-Kann players may like:
I would not like to transpose into an inferior opening, thank you

I would not like to transpose into an inferior opening, thank you
The French is not inferior to the Caro-Kann.
To the Sicilian, perhaps.

I would not like to transpose into an inferior opening, thank you
The French is not inferior to the Caro-Kann.
To the Sicilian, perhaps.
I’m tired of arguing so I’ll just stay quiet

I would not like to transpose into an inferior opening, thank you
The French is not inferior to the Caro-Kann.
To the Sicilian, perhaps.
There’s not a single shred of respect for the French on this site, is there? Just you, I, that one NM, and ThrillerFan.

I would not like to transpose into an inferior opening, thank you
The French is not inferior to the Caro-Kann.
To the Sicilian, perhaps.
There’s not a single shred of respect for the French on this site, is there? Just you, I, that one NM, and ThrillerFan.
Hey, I love the French! I play the Horwitz Defense against d4 just for the chance to transpose into the French

I would not like to transpose into an inferior opening, thank you
The French is not inferior to the Caro-Kann.
To the Sicilian, perhaps.
There’s not a single shred of respect for the French on this site, is there? Just you, I, that one NM, and ThrillerFan.
for good reason
Usually I try to counter the Fantasy Variation with the fianchettoed Dark square bishop, however I have quite poor results with it. Also in case of a closed position, I feel Black gets suffocated and the extra pressure created due to limited mobility of the light square bishop doesn't help. Is there a better way to counter the fantasy variation or is it a inevitable loss (most of the times) for black in this variation?