My guess would be for the same reasons B/N/R/Q/orKxe5 are designated. In all cases, it clearly shows exactly what took. It also is consistent, so perhaps less likely to be recorded incorrectly.
People also play blindfold, along with people being able to mentally follow a line. Xe5 - now one would need to recall the position. When dxe5- adding the file would be of help recalling.
Just my thoughts.
Edited to add - It may simply because all other pieces are described and to maintain uniformity, so are the pawns.
The file of departure is always indicated in the algebraic notation when a pawn makes a capture even if it isn't ambiguous, that is, even if there aren't two pawns that could have made the capture? For example, why not xe5 instead of dxe5?