yes, it's a weak move. 3...Bf5 develops a piece, the alternative 3...c5!? puts W ceter under pressure. What does 3...b5 obtain? W can play all his normal moves on the kingside and if you stick to your queenside expansion plan you will soon gift W a deadly advantage in develoment. You will be soon or later forced to play the usual moves, but with a big tempo handicap.
Of course if you score well with it you can continue with your pet variation, but as you get stronger you will find that your opponents will become capable of refuting your idea.
On the caro-kann defence when the passed pawn variation of e4 c6 d4 d5 e5...noone ever responds with b5...i know everyone plays bishop f5 which is totally unsound and cramped...after b5 white has too realize that blacks attack is queenside and could respon with pawn a4 which i get very rare i always get pawn a3 which i then play pawn a5 completeing my queenside pawn rush...i useually put my bishop on a6 too hold down the pawn while i push for pawn c5 as in the french defence...i have had much success with this defence from the caro-kann and i dont know why it isnt on any sites or databased...for the medium rated player this is a good opening because most people expect Bf5 and a center game with black having a cramped position.