Maybe because under time pressure, you can usually just safely shuffle a Bishop (but Knights can't lose tempi), so it forces your opponent to find something concrete quickly?
In Blitz and Bullet Chess End Game, Fischer says the Bishop trumps the Knight, How so?

Actually, many experts believe the Knight is worth more in Bullet.
I've heard this too. The reason isn't point value based - it is the simply that Bishops are easier to shuffle under time pressure. Plus Knights offer "tricky" forks and such more likely to be unseen by the opponent with low time.

The bishop is overall better than the knight, at least that's what Kasparov says. I wouldn't know why Fischer made emphasis on blitz and bullet on this.

There are plenty of examples in books of a bishop beating a knight in an endgame. It's so common that it's usually more instructive to talk about when a knight can beat a bishop, because that takes special circumstances.
Bishops are better able to give zugzwang as mentioned (they can also lose a tempo which knights can't do) but as to your idea that they only influence one color, I guess you're suggesting the opponent could just move everything to the other color.
Some fortresses involve that, but if you, for example, abandon the light squares, then your opponent can attack any square that's near a light square... which means all the dark squares (in an endgame this means a king would infiltrate).
Here's a famous endgame where you see that happen.
Bobby Fischer believed that the Bishop was worth 3.25 points and the knight was worth 3. However, in the end game, a single bishop can only protect or attack one color of squares, while the knight can protect and attack both. I believe the bishop and the knights value must be based on the pawn structure and the presence of the king. If any one can convince that the bishop is worth more than the knight, I would like to hear your case.