I don't know why, but I can almost never win against the king's gambit as black (of all the games I have played). It's probably because I don't study it enough, but the (considered) normal opening sequence attacks me psychologically, preventing me from not fearing it. 

I'm never playing 1. e4 e5 ever again!

im where you are at study wise, but I dont fear it, its just annoys me.

Easiest, I think, is to go into the Falkbeer or the Cunningham. But the main line, 3...g5, is good too.  It too much fun, even if you lose, to avoid.

i always take the pawn ;)

That rarely helped me.

beware the 3. Bc4 line in the accepted

3...d5 against either the Knight's or the Bishop's gambit give black the easiest kinds of games to play, IMO.

Neither will be a non-stop thrillfest, but if the KG "scares" you, that's probably the way to go.

You know what's funny. I came into this forum because I was going to answer it with exactly what Vulpes Vulpes said. Recently I've tried different strategies for the sake of doing it, but I always take the pawn ;)

Ok. Thanks for the tips!

Accept it, go 3...d5, that's where the pawn belongs against the KG (ignore Fischer's advice hehe). From there you should get good activity, which is crucial - defending doesn't mean turtling, and a decent amount of space, and it's just a chess game with Black a pawn up although a weak one. Make sure also to find a safe spot for the king - as normal really. I find it helps against the KG to remember that it's a perfectly playable opening, not something that Black can knock out straight away - but the 3...d5 thing knocks out a central pawn and gives Black some comfort.

Oh, and after 4.exd5, it's Nf6, not Qxd5!

or just go with 2...Bc5 and decline the gambit.  I play a lot of KG's and find this move quite anoying.

King's Gambit offers lots of good choices for the second player...

But they tax me psychologically.

Maybe because you already feel that by accepting the pawn you should protect your own? Don't do that. It's okay to be even in material while developing your pieces normally instead of trying to hold on. Openings are just a part of the game. Middle games and end games are where games are won. It's beatable, or at least drawish in most cases. Still, I would recommend you look up some lines on the opening. Different variations, so that you kind of get the gist of what your pieces should be doing, and why.