No need to be confused. After Black's 4th move, what has he got? (1) A knight on the rim. (2) A misplaced queen on the back rank. (3) No challenge to your center.
Some things to consider: (1) Just keep building up your center with moves like c4, Nc3, O-O, Re1, Be3. (2) Where is Black going to put that knight on the rim? If ...Nb4 he gets chased off with ...a3. If ...Nc7 or ...Nc5, he's going to have to advance his c-pawn first, via ...c5, so expect that, and maybe advance and block the "Hungarian Diagonal" with d5 if he does that.
I thought 7. e5 was premature. Wait to see what he's going to do in the center, such as ...c5 or ...d6 before moving the same man twice in the opening before getting fully developed in the center with the moves I mentioned earlier. All those pawn moves you made to attack his queen's knight merely ended up creating a hole for him to outpost that bad knight.
17. Qb3 was unwise queen placement since of course he's just going to chase off that queen with ...Rb8, which he did. Eventually your position fell apart with a shattered pawn structure, uncoordinated pieces, space behind your pawns for his queen to invade, etc. I think you should have just hung onto the center you had and should have kept developing normally without all those premature pawn advances.
P.S.--48. f6+ might have given you a nice counterattack via discovered check.
See the opening? How should white punish black for this nonsense? I wasn't able to do anything other than play reasonable moves that led to an even game (I laughably blunder the game away)