Complicated game
To answer "when did things go wrong?"
Moves 12 and 16 weren't bad, but I think you were starting to lose the idea of the game.
Moves 17 and 20 were wastes of time but didn't directly damage your position. After these moves white is better but not winning.
Move 21 dammaged your position (or helps white... same thing ;). White gets his attack and black falls apart.
Specifically:
12...h5 This is a standard reply but I think I would have preferred 12...h6. As black I want to play f5 and eventually e4 but when I play h5 first g5 becomes weak and there's a lot of air around my king (the king is a bit more vulnerable).
16...Kh7 Up side, you have the option of Rh8 for some defense. Down side the king is on the same file as the h1 rook and white definitely will try an attack.
17...a6 This was a wasted move. Black would love to have his a7 pawn removed because it would immediately activate his rook against white's king. But the pawn wasn't free anyway, on Bxa7 you had b6 to trap the bishop.
20...c5 this move was more of a waste than a6 was. White is in the middle of an attack so your moves should either defend or create some kind of counter pressure. Instead moves like Rh8, Kg8, or Qd7 are fine.
21...Bxh5 This was bad (much too brave!). You open the h file and activate white's rook into the attack on your king. You had to play g5.
24...Bh5 keeps material even but white is still attacking. (RxB and Qf5 looks like a winning attack).
Oops, I stopped looking before move 30 because I assumed white eventually mated you with that attack. I see he let you back into the game! :)
35...b4 is wrong. You're losing so you want as many parts of the board open as possible! :) Think of it like pressure in a bottle. Right now all the pressure is on your king. If you poke a few holes in the bottle (open the center or queen side) then it can sometimes equalize the pressure.
When a player locks the other parts of the board (especially the center) it's a good sign for a successful attack. So again as the defender you want to open things up!
38...Ke7 was a mistake. The attack white immediately gets isn't entirely obvious, but more importantly the thinking behind it was a mistake. Besides Rh3 winning a pawn, you shouldn't be worried about his king coming to d6 with all the heavy pieces (queens and rooks) on the board! Like swimming with sharks, if his king did try to come to d6 it would likely be checkmated quickly.
After 40.Ra1 you're losing again and this time the attack works for white.
50...Ra3+ was a clever move :)
When a player locks the other parts of the board (especially the center) it's a good sign for a successful attack. So again as the defender you want to open things up!
Thanks for all the explanations, but specially thanks for this advice, as I thought that it was the defender who wanted to lock up the board to avoid being attacked in several zones at the same time.