Few things. Openings are the only point in the game where you can reliably play at 3600 strength. That should in itself be telling to how important it can be. It's true that if you had to choose between a gambit you know very well the berlin ruy, you'd have much better chances in a gambit than a berlin since the berlin can be complex and nuanced to play. That being said, at some point your opponents are going to punish your play if you're playing suboptimal gambits. If you're in the market for trying to improve, perfecting your theory in an opening that will net you a worse position in most cases is definitely not conducive to your long term chess and will just cause you to have to relearn solid openings at a later date. Lastly, this is the most telling thing I noticed you said. "Opening seems negligible considering chess is mostly dictated by how well the middlegame/endgame is played". This right here is the crux of the topic. Openings are not just the first few moves, but a roadmap for how the rest of the game should be played and dictates pawn structure, piece placement, attack/defense motives etc. If you have a losing opening, you're more than likely to suffer in the middle game and consequently the endgame. It's best to give yourself the best fighting chance from the very beginning and play like a 3600 strength player for as long as possible and hopefully that sets you up for success for the rest of the game. This is not to say you should always copy the engine's top move, but rather add your own flair picking between *good* moves (ie if two moves are +.35 and +.28, rather than just going with the highest eval move, add in your own interpretation.) Two moves may be equal in eval but lopsided in ticky-ness, hope this helps.
Does opening choice really matter?
It's good to know how to maneuver the opening in order to get a familiar position on the board, where you don't immediately feel lost. That said, something Magnus said in his recent JRE appearance stuck out to me. He said nowadays he just uses a randomizer to pick which opening he wants to play on the day, and whatever happens, he just accepts whatever position he gets from the opening. I think that's a great way to diversify one's repetoire and to gain experience and appreciation for different types of positions, and not just get the same-old positions every game.
Generally I agree with the OP - just looking at the raw data, even at the highest level, sidelines like the Vienna are still outperforming main lines like the italian and ruy lopez. And when you watch Nakamura or Carlsen play very often they're diverging from theory early on. i.e. Magnus has an a3 move he's played recently. Nakamura has played alot of b3.
There's alot of variety in styles and approaches to the game, even at the highest level you still get conflicting opinions. I really don't think there's a "correct" style.
There are certain lines that are better than others though, obviously. If a gambit has a pretty straightforward response you're only going to make it so far until people know it. But most underplayed sidelines are just rejected due to having slightly worse engine eval, in some cases a few centipawns difference. Those are trends which began with the engine era and nowdays I'd say there's been movement in the opposite direction... it's understood the mainlines are played out, and due to widespread use of databases w/ online game statistics people have seen how the sidelines are performing better. Chess streamers such as Hikaru or Danya have also popularized the ideas of avoiding theory, they talk about it frequently.
But opening positions have many different attributes, commonality is just one. You have to consider them all. The evaluation is also partly subjective, it'll depend on what your goals for chess are. Not everyone aspires to be GM one day - not everyone has time to do that.
The other thing is... I don't think it's ideal to forgo the process of experimentation with the opening. Generally I'd say you should narrow down your options to a couple you like, but then you're gonna have to play the lines and experiment before you know which ones you prefer.
I've heard a lot about how being prepared for openings is crucial, but I've also heard some people recommend against "bad" openings because the engine doesn't like it. I always thought that opening preparation was the most important aspect, not actual choice of opening (excluding grandmaster level play). I've found lots of success and ease with "bad" openings, and don't see why I should switch considering the downside of knowing a "bad" opening to a "good" opening seems negligible considering chess is mostly dictated by how well the middlegame/endgame is played.
Well, it both matters and doesn't matter.
It doesn't matter in the sense that for the majority of players, openings aren't going to make or break your game. I can push every pawn up one square in the opening and win against most people, because I am stronger than most people, and not because my opening won me the game.
But it also does matter in the sense that some people might enjoy playing a certain way more, which indirectly does help you play better. For example, I enjoy playing hypermodern and provocatively. Thus, I enjoyed playing openings like the Alekhine's defense and I did very well with it. Meanwhile, if I was forced to respond to 1.d4 with 1...d5, I would have quit chess. And you can't improve or win at chess if you quit the game.
Of course, openings have varying degrees of soundness, and the higher your rating, the more likely it is that your opponents can punish unsoundness. A 1200 will probably get jumpscared by the elephant gambit and lose in 20 moves, but a 2200 will probably know the strongest response to it, and will get a strong position and then from there, beat his opponent in 20 moves.
Depends upon your level, really.
Level below 1000: not really. Its good to know some openings but you dont need to learn them in depth because your opponents wont know them either plus you wont be able to punish bad openings that well just yet.
Level 1000-1800: definitely helpful to do.
Level 1800+: you need to get serious about openings or a weakness in this area will seriously impede any further progress because your opponents WILL punish bad openings.
I've heard a lot about how being prepared for openings is crucial, but I've also heard some people recommend against "bad" openings because the engine doesn't like it. I always thought that opening preparation was the most important aspect, not actual choice of opening (excluding grandmaster level play). I've found lots of success and ease with "bad" openings, and don't see why I should switch considering the downside of knowing a "bad" opening to a "good" opening seems negligible considering chess is mostly dictated by how well the middlegame/endgame is played.