I too play the Grob in periods, though I prefer to play it as Black (Borg gambit 1. e4 g5; 2. d4 Bg7!??). I love that you play b4 after having already played g4. Beautiful! Compare this game:
Grob Win
Not a bad Borg! I played more after that first one, so here. I'm not as aggressive as you, but I do find a certain interesting flavor that unorthodox openings like the Borg and Polish offer. I am sick with a nasty cold right now though, so there's no stuff of legend here (shudder gasp).
In both these games you immediately defend your g-pawn when your opponent moves his d-pawn. I don't think you need to do that before he/she protects the b-pawn.
After 1. g4 e5 2. Bg2 d6 (2nd game above) he can't play ...Bxg4 because you have Bb2. Or do you not like this type of position? Black will get some counterplay on the light squares after ...Nd7 followed by Bxa8 Qxa8, f3.
Well, it appears I'm drifting into one of my unorthodox periods as far as chess is concerned. Played this Grob blitz game earlier today.