The usual sequence to punish attempts to hold the pawn with b5 is undermining with a4, which often allows tactical tricks. However, if that doesn't win immediately you can further undermine with b3, in which case you often end up with nice open files an center control, while black has decimated their own queenside:
Queen's gambit. What if black protects his pawn?
Ok, so remove the defender with a4. Seems obvious now. Thanks.
Next time I play white, I'll do Q gambit, and pounce with the a4!

If you play e4 instead of e3 or Nf3, he can hold the pawn, but it leads to a theoretically bad position for Black.

4. Nf3 is standard QGA theory. After ...Bg4 you play d5 followed by Bf4.
Reasons why I never played the e4 QGA. moves like d5 when black can still undermine easily with c6 and e6 rub me the wrong way.

Doesnt 6.b3 force black to give the pawn back? At least I dont see a way for black to hold it, but im just running through in my head
e.g
6..cxb3 7.Bxb5+ followed by Qxb3.
6..Be6 7.bxc4 bxc4 8.Bxc4 Bxc4 9.Qa4+ (Picking up the bishop)
6..Ba6 7.bxc4 bxc4 8.Rxa6 Nxa6 9.Qa4+ (picking up the knight, and probably the pawn later)
6..Qc7 7.bxc4 bxc4 8.Qa4+

4. Nf3 is standard QGA theory. After ...Bg4 you play d5 followed by Bf4.
Reasons why I never played the e4 QGA. moves like d5 when black can still undermine easily with c6 and e6 rub me the wrong way.
e4 is a little more fun and adventurous, although I can understand other d4 players not liking that. Carlsen plays e3. I play the e4 lines and generally find that Black doesn't handle it well. You just have to learn Nc6, Nf6 and e5 as responses.

I kinda hate the QGA, i usually play play 2..e6, but for some strange reason (night shift most probable) i played 2... dxc4 (online chess) and i'm in a real mess. I can't show you the game cause is still in progress :( .
The funny thing is i've never used an online database for my online games (i just have a couple of books), but i don't have a book for the QGA, so after this guy made his move, i looked throu a database and wow...
there was a move that was 100% winning (at least from stats) so i played it, when rechecked the database i suddenly realized the move was 100% winning for white... sigh.

Gambit is from the word Gamble 4 a reason....
Gambit is not from gamble, it's from italian word "gambetto", meaning tripping.

Doesnt 6.b3 force black to give the pawn back? At least I dont see a way for black to hold it, but im just running through in my head
e.g
6..cxb3 7.Bxb5+ followed by Qxb3.
6..Be6 7.bxc4 bxc4 8.Bxc4 Bxc4 9.Qa4+ (Picking up the bishop)
6..Ba6 7.bxc4 bxc4 8.Rxa6 Nxa6 9.Qa4+ (picking up the knight, and probably the pawn later)
6..Qc7 7.bxc4 bxc4 8.Qa4+
Yeah you're completely correct, my comment regarding playing without a pawn was in reference to the 6.Nc3 move, sorry I wasn't more clear haha.

There's a 2300 at my club that says amateurs should play QGA because White generally doesn't know how to handle it.

Gambit is from the word Gamble 4 a reason....
Gambit is not from gamble, it's from italian word "gambetto", meaning tripping.
No, it's from Marvel Comics.

Doesnt 6.b3 force black to give the pawn back? At least I dont see a way for black to hold it, but im just running through in my head
e.g
6..cxb3 7.Bxb5+ followed by Qxb3.
6..Be6 7.bxc4 bxc4 8.Bxc4 Bxc4 9.Qa4+ (Picking up the bishop)
6..Ba6 7.bxc4 bxc4 8.Rxa6 Nxa6 9.Qa4+ (picking up the knight, and probably the pawn later)
6..Qc7 7.bxc4 bxc4 8.Qa4+
6...e5
That move does create issues that can be tricky to solve. I usually play 6.Nc3 anyways if that line ever happens(usually when Im playing other bad bullet players like myself lol).

There's a 2300 at my club that says amateurs should play QGA because White generally doesn't know how to handle it.
thing is, i find that amateurs also dont know how to handle the black side. A big part of the QGA is being rather aggresive with center pawn breaks and amateurs tend to be timid about the center in my experience.
I think his point was that most d4 players play QG, so prepare the QGA and you have an edge.
I know when black takes Queen's gambit, it's fairly easy to recapture the lost pawn. However, what do you do if black defends his pawn?