Why is holding on to gambit pawn a problem here?
What should be Black's response?

Why is holding on to gambit pawn a problem here?
Because you can't, and trying to against somebody who knows what he's doing (e.g., an opponent who answers your ...b5 with a4, etc.) leads to wrecking your own queenside while simultaneously helping your opponent.
You're now saddled with an isolated pawn and no piece activity to compensate, together with a lousy bishop, while your opponent has a rock solid pawn structure, a lead in development (because make no mistake, you've developed your opponent's queenside rook for him), a space advantage, and a clear plan to follow since you've given him his space advantage on the same side where you've helped him open lines of attack for his heavy artillery.
You should NOT try to keep that pawn! The way white handled the position is the typical way how you can play against it. Black is just very bad.
However, we are now talking about the slav defense. And in that opening you really have the threat of playing b5 since c6 makes it stronger, but again not for holding the pawn.
1)d4-d5 2)c4-c6 3)Nf3-dxc4 4)e3-b4 5)a4-Ba6 6)axb5 Bxb5 7) b3
I was playing with black in this game and lost it. What should have been my response after b3....I played e6 8)bxc4 - Ba6. Was there a better move i missed? Were any of black'smove above sub optimal? If yes, why?