[scandinavian defence]i don't want to take black's d-pawn. i want to just move my e-pawn forwards.

It is bad because you allow Black the best of both worlds.
In the Caro-Kann, Black gets the Bishop outside the pawn chain before playing ...e6, but then it takes him 2 moves to play ...c5, move 1 c7-c6 and later c6-c5.
In the French, Black's Bishop is hemmed in behind the pawns. However, in return, Black gets a quicker attack by playing c7-c5 in one go.
Declining taking on d5 and advancing to e5 gives Black the best of both worlds. With no pressure on the d5-square, no early e6 or c6 is necessary. Black can bring his Bishop out AND play c7-c5 in one go, giving Black the best of both worlds of the Caro and the French. Advantage Black.

Taking the pawn is the best move.
But pushing the pawn forward might be a reasonable try for lower-level players, the idea being that Black won't be prepared and will have to come up with their own moves. Try a couple games where you push, and see how you like it. You can play anything you want.

the scandinavian advance... it's why i want to quit it and start doing rousseau/luccini/calabrese & schliemann/jaenish instead. reversed king's gambits (NOT latvians!) are king's gambit enough for me