Why is en passant only possible in the next move?

Title says everything.

Why is en passant only possible in the next move?

Would there be any downsides if it wasnt that way?

For clarification:

it would break the game, pushing pawns to the 5th rank would be a huge advantage

If you think of ability to move pawns forward to squares as two consecutive one-square moves it makes more sense.  Why is the ability to make a capture on the first square limited to just pawns?  This I don't know.

How would it break the game?

Look at the history of the rule. It used to be that pawns could only move one square, even on the initial move. When they gained the ability to move two on the first move, the en passant rule is an obvious implication from it.

In the old rules, you could only capture the pawn on the square while it was there. If it moved after that to the 5th rank, the option to take would be gone. With the two-space move, it wouldn't make sense to allow a capture on any other move other than right after and still follow the way the previous rules worked on capturing with pawns.

That is the clarification? Why would you ever be able to capture that pawn when the capturing pawn was never in position to make that capture at the time the black pawn was on f6?

Time travelling pawns?

I dont know if I understood you right, but if the pawn didnt move any after, it would be possible to always capture it, not only the next move.

The rule means 'in passing'. If it's not on the next move, it's no longer passing.

sry, my clarfication was bad :D i tried to make it better.

@scott: you would still capture while passing

Mr Steel is right: the e.p. rule is the natural result of the attempt to introduce double pawn steps without changing the strategical nature of the game.

If capturing e.p. was allowed on later moves, the game would change drastically.

in the position you've given, black is in dire straights if that rule was in effect

if white can manage to play Nf3, d4, black cannot develop his knight because Bg5 followed by exf6 will wreck his structure

in this new position, even if white cannot play Nxe5 he is close to winning because he can pressure e6 and always threaten to play dxe6

How long does it take to pass?

It is likely that the double-push/e.p.-rule emerged as an "I make a second step before my turn, but instead of complaining about it, you can then do it too", just to speed up the opening a bit. (Did you know there still exist Chess variants where the concept 'turn' does not exist before the first capture is made, and before that the players can rearrange their pieces at their own speed, ignoring what the opponent does?)

The e.p. rule then came from the fact that people sometimes did want to oject when the opponent made the second step before his turn, rather than retaliate in kind, typically because they wanted to take the Pawn after its first step, so it could never make the second one. So this "second step before your turn" was subject to a veto, and it would make no sense to use that veto unless you really wanted to capture. So this was later formalized to the e.p. rule as we now know it.

It would not make much sense to say: "hey, I can capture to e6 now, and some moves ago you had a piece there, so please put it back there so that I can capture it. (And never mind all those other moves we did in between.)"

Correct. If the pawn is still on the square after it had moved once (one square on its initial move), you could capture as long as it doesn't move to the next square. The idea behind en passant is that for that one turn you can treat it as if it had only moved one square, aftwards, you don't have that.

I wonder why there is no x-ray-ability for bishops. Let's say a bishop can move through other pieces and then kill one thats behind but still on the same line.

This is not a good explanation... The name just describes the rule. The rule wasn't modeled after the words "en passant."

Initially, pawns could only move one square on the first move. There was a problem with this: it took too long to get anything going in the game. 

To address this, a new rule was adopted: Allow pawns to move two squares, but only on their very first move. This fixed the problem, but created a new one: Pawns tended to get all locked up on the fifth and sixth ranks, making progress difficult for both sides. 

To address the new problem, another new rule was adopted: en passant. This rule was created specifically to address the second problem, and was never intended to be a general rule to allow captures on the fifth rank. Extending it to a general rule available at any move would unnecessarily complicate the game. If nothing else, it would force you to remember the history of each pawn for potentially many, many moves. And who wants to do that?

Well, if they had chosen a different rule back then, I'm sure they also would have chosen a different name.

When a pikeman wishes to trips another pikeman as he runs past, there is only one opportunity.

I'm sure the OP is reasoning about the rule, not the name.