How come so many people on my level play the Philidor Defense? It always seems to allow me to develop faster and attack his kingside quick.

There is nothing wrong with Philidor's Defence, it is slightly passive but completely playable.

Fear

I'm going to jump in with this:

I firmly believe that a good portion of philidor players at the lower elo levels play it because they do not know how to hold on to the e5 properly after 2... Nc6 3. Bb5

That doesn't the philidor is a bad opening by any means; I just have a feeling many weaker players play it for wrong reasons haha. 

Everytime I played 1.e4 e5 2.Nf3 Nc6 3.Nc3 Nf6 4.Bb5 I got 4...d6...

But you'd think they know of 3...a6 4.Bxc6 dxc6 and 5.Nxe5 is bad because of ...Qd4.

After 1 e4 e5  2 Nf3 there are three logical alternatives for Black.  Defending the e5 pawn with Nc6 or d6 are both very natural and logical moves.  Centuries of master practice have shown that ...Nc6, developing and keeping options open for both the d-pawn and the Bf8, is more flexible, especially because of the ability to defend the Ruy Lopez Bb5 threat to win the e5 pawn. 

Many beginning players aren't aware of this, it's not something immediately obvious.  They learn it when they are taught it by a coach or book.

So ...d6 gets a lot of play, and it is also a logical move, even though we know it doesn't offer Black as many options and counterchances because of those centuries of play.

The third option is counterattacking e4 with ...Nf6, the Petrov or Russian Game.

Defending e5 by ...Qe7 or ...Bd6 is not so logical, as both moves block other development and generally serve to get in their own pieces' way.  Naturally, ...f6 is just bad because it weakens the Kingside horribly.

No some don't, and that is precisely why I believe the philidor defense is as popular as it at the lower elo levels.

I think it's funny when I go into a ruy lopez as black and someone plays the exchange variation after a6 and then they play Nxe5.  After I play Qd4 the time on their clocks widdles away as they figure out what the hell just happened.  Then they do the craziest stuff like play Nxf7.  Anyways.  I love playing against the philidor defense, it's my favorite opening to play against because it is so passive it leaves white an easy game.

To avoid the Ruy/Scotch which someone who plays 2. Nf3 is likely to be looking for.

Just wait somebody makes mistake like this (trap):

Everytime I've countered the Philidor in the past 2 or 3 months on this site and/or ICC with 3. d4 Black plays 3...exd4.

No, not your fault, LOL.  On 3...exd4 I've been playing 4. Qxd4.  Perhaps you won't think that is as good as 4. Nxd4, but against the 2...d6 Sicilian I also play 3. d4 cxd4  4. Qxd4 so it gives me a system of sorts, where I have similar issues to contend with.  For instance, if 4...Nc6  5. Bb5 Bd7, do I play 6. Bxc6 or move my Queen?  Or, what to do if Black plays 4...a6 (against Philidor's I play 5. Bf4 Nc6  6. Qd2, in the Sicilian line I play 5. h3 Nc6  6. Qe3)

Thanks for the advice (in the Sicilian line lately I've been playing Qd3 after ...Nc6 instead of Bxc6).

In the Philidor line after 4...a6 I recall getting a decent game with 5. Bf4 Nc6  6. Qd2 but I don't think the quality of my opponent's play was so great.  Also possible is the innovation 5. Qa4+ when of course 5...b5 is not possible due to 6. Bxb5.  I mentioned this on chesspub.com a few years ago and that's where the Bf4/Qd2 line got suggested to me instead.

Any further insight you'd be willing to add would be much appreciated