It's probably cliche by now but the modern/exchange are the only two that seem dangerous to me. But they aren't aggressive at all.
Alekhine Defence (Chase variation): Practicality for white?

When white is claiming advantage in space, but not immediately attacking. The position of black can become difficult to handle, because black lacks space.
A setup after
1. e4 Nf6 2. e5 Nd5 3. d4 d6 4. Bc4 Nb6 5. Bb3 e6 6. Nf3. The position of white is flexible and black has less space. White has not overstretched his position, hence the space advantage can be for a long time. Moves I consider here are Be7, Nc6 and d5.

People always assumes a lot of things about why you are playing a certain defence. The thruth is that everybody (including alekhine players) is just trying to survive the opening with a good position. 2.d3 is absolutely welcome by alekhine players, since black gets fairly easy play (which doesn't mean that you can't score very well with it if you're a good player :)).
4.Bc4 is maybe a bit underestimated but still nothing special for white (Giri recentrly struggled to draw as black against a lower rated player, but it's just because he tryed too hard to unbalance the game from a level position). 4.Nf3 is the real headache. 4...dxe5 is safe but quite passive, and the more entertaining tryes (4...Bg4, 4...g6,4...Nc6!?) are sort of under a cloud.

4.Nf3 has a lot of theory behind it though and given that I'll be having to bone up on the Dragon and the KID, I doubt I'll have too much time to devote to preparing for less common defences to my 1.e4

I use 3.Nc3 Nxc3 4.bc with the idea of meeting 4...d5 with 5.f4 and never playing d4. This leads to a Winawer pawn structure. 4...d6 can be met by 5.f4 and then d4 with a big ol space advantage. Either way White gets a structure where the pawn on e5 gives you attacking chances.
2.Nc3 is the easiest way out though. I think a Vienna or four knights will not be what theory heads are after and after 2...d5 there are a few ways to go that frankly are nothing like the usual Alekhine positions anyway.
Thanks for your post Loek, I figured Alekhine players would not be afraid of 2.d3 as white is effectively bailing out and saying "ok, I don't want to get into this shit. Let's just play chess."
Can I ask, what DO you fear most after 1...Nf6?