do you know you can win many games of chess by scholars mate for white

Even then, many lines, such as literally any knight move (except Na6 but who plays Na6) counters scholars mate.

All fun and games until you get an opponent that isn't completely braindead. Have fun getting slaughtered when your queen is bullied around the board via basic development. 🤷🏻‍♂️

Its pretty trivial to defend against this in your face attack.

The main line here is probably:

And white falls apart, since they lose a minor piece.

As a complete noob, I always went E5 against E4, after losing tons of games to tricks on the weak F7 pawn, I changed to Scandanavian and won a lot more, then Caro-Kann and did ok with that, now I actually can defend and punish the scholars mate as shown above so it has me rethinking my entire plan. Not sure if I should play E5 and battle there, or continue with Scandanavian and Caro-Kann.

Yes

The "Nelson-bot" is a good one to play as Black if you want to practice defending against this sort of thing. The responses in post 4 above is the most straight forward, and works well. Nelson-bot plays these early Queen attacks almost exclusively, although sometimes taking the Bishop out to c5 instead of c4, but that's a different line that's not going for Scholar's Mate. It's very easy to take advantage of White's overly aggressive initial play by simple opening principles while White violates almost all of them (gets their Queen out too early; moves the same piece multiple times, doesn't develop their minor pieces, doesn't fight for the centre, etc).

I had, it kinda doesn't work anymore. It's easy to defend as the others have said.

Both are decent responses, and if you're having success with them then I wouldn't drop them. Learning openings isn't just learning "they do this, I do that, they do this, I do that..." type of rote memorization, but rather pausing at each move and asking yourself "Why do I do this now?" Understanding why those series of moves are the "line" requires understanding what each response is trying to do. For example, in the Scandi, after 1. e4 d5 2. ed Qxe5 the reason White usually develops 3. Nc3 is, of course, to gain a tempo on the Black Queen, and that is why Black plays (generally) 3. ... Qa5, to move away from the threat on their Queen. It's not that Black "wants" their Queen on a5 per se, rather, they have to move it because it is under attack and a5 is one of the options Black can choose that they can work with.
But if White doesn't play 3. Nc3, and chooses a different line like 3. Nf3 .... Black should realize that they shouldn't just play Qa5 anyway because now their Queen isn't under attack. Understanding that 3. ... Qa5 is a response to the threat on the Queen is important because if one just memorizes moves, then one may simply play it anyway thinking "Oh, White made a mistake but I know I go Qa5 so I'm still good!" And at that point Black starts wondering how they are going to take advantage using their "properly placed Queen because they followed the memorized moves". Anyway, I know that's a very simplistic example, but it illustrates the importance of understanding why each move is being made, and what each move is trying to accomplish.
Also, as you play an opening more and more, you reach similar positions and will gain a better understanding of what sort of plans work well, and what sort of plans don't work so well. Are you going to attack on the Kingside? The Queenside? Down the middle? Are you better if you trade down into an end-game or should you try and keep more pieces on the board? Is there some outpost square that is just fantastic for a Knight (if one of yours, then maybe your plan should be to put one there - if for your opponent, maybe you should plan to try and prevent that, or at least be prepared to trade it off should it end up there). Sometimes the best thing to do with an "opening" is to just look at the position at "the end of the line", and try and work out what you should be doing next? If you can't see anything to work with, then maybe that is not a good choice of opening/defense for you. Remember, even if you can memorize theory to 20 moves deep, all you would be doing to heading to a position where you're still lost as to what to do!
As for going back to playing 1. e4 e5 ... I don't think that's a bad idea. You indicated you know where you were losing games (tricks on the weak f7 pawn), so you know what to look for (threats on your weak f7 pawn). Go through some of those losses, locate when the "trick" occurred, then go back a few moves and try and work out how you could have thwarted it, and look for the moves that indicated your opponent was trying to set it up. As you analyze those games, you will improve both your ability to spot when your opponent is trying to set you up, and you will gain better understanding of how Black defends that weak point in their position. And just like learning how to defend against the "Wayward Queen Attack" that Nelson-bot plays, learning how to deal with other attacks on your f7 pawn will help your chess.
Anyway, just my thoughts, that may or may not be of use to you.