Based on what I am seeing your biggest error isn't so much that you took so many of those exchanges, it's that you let Black make the first move. As the game opener you need to be on the offensive and if anyone is offering exchanges it should be you. He managed to push the game because you were too timid early in the game and fell for easy gratification.
Fast forwarding to 20 where you two were even you continued to let your opponent guide your movement. At that point your next move should have been to move your knight to D4 instead of moving a pawn in order to attack his side of the board affecting his ranks. Given, his next logical move is knight to E4 for check but you stay in control by moving Ke3. He has three options: Back off to a neutral position, attack a pawn, or stick around. Because this is his last attack piece he will back off to Nc5 or Nf5. Now you march your pawns and build an attack.
This is an OTB game I played recently. The time control was 90 minutes with a 15 second increment. I developed a nice centralised position out of the opening, however I was unable to gain an advantage or create real problems for my opponent. He offered piece exchanges which I felt that I couldn't decline without wasting time retreating or giving up central squares. If anyone would like to offer feedback or suggestions for where I could have improved that would be greatly appreciated.