How could i have avoided this?

I think your problems started around move 13, when you began retreating.

13. exd5 exd5 14. Nxe5 Re8 15. f4 or something like that looks much better to me.

N on d1, bad isdea, Rg1, even worse, what was the rook doing there? try to put out the rooks on open files, and finally, avoiding the fork, well I guess you overlooked it, could have placed the Q on e1, not the best choice though, keep in mind that both black knights were allowed to jump around unchecked while your own knights were misplaced. 

My rooks were in good position to start a king side pawn attack (where his king was).  Rooks are good (mainly) on open files OR files in which you would like to advance pawns. 

My knights temporarily where withdrawn in order to reposition them.  i think Qe1 would have been decent.

if your pieces are on the same colour, you could be forked by a knight-your king, queen and rook were on dark squares. havin said this, you shouldnt avoid putting your pieces on the same colour just because they can be forked, instead you just ask yourself:can he fork me? (obviously you didnt do this)if the answer is no, then theres no problem

also your knight retreats on moves 13 and 14 were silly and pointless. if the N was still on c3 this wouldnt happen since the knights control more squres on the 3rd rank than the 1st

Once you retrieved you Knights, all was lost...

They weren't there to defend you. Why move them back? They aren't useful there. You should always look to place them centered, as black did. They are more versatile in the central position. If your intention was to put them back so that they can hold your castled king, then you shouldn't have castled in the first place.

prophylaxis with

16.f3

would have avoided the problem. Then probably c3,g4, h4 and so on.-

(But 17.Qe1! instead of 17 Qg3?? would have done it also. Then 18. f3 and so on)

***

In position with different-side castling it is important, to keep the balance between attack and defense