Look at the center, what plans are there? What is the opponent trying to do? There are two situations when you need to stop the opponent's plan:
1.If it's faster than yours and gets something real out of it.
1a.It's okay to allow the opponent to go along with their plan if you're setting up an ambush, just make sure to blunder check first (LeMoir's Deadly Chess Tactician book addresses this). However, it isn't always appropriate and sometimes you just have to stop what they're doing.
2.You don't know what to do otherwise.
Pawn structure is an imbalance and a broken structure like doubled pawns is bad, right? Not always since piece activity compensates weaknesses and even doubled isolated pawns can restrict an opponent's activity (especially central pawns). I can think of two examples off the top of my head (a game from Dvoretsky's School of Chess Excellence 3: Strategy and an Anand-Nakamura game I studied both had these theme)
Piece activity also compensates weaknesses. Basically the imbalances that should be prioritized are those you can immediately convert into a better position gradually transposing into a better ending. This is where technique comes into play, instead of accumulating advantages you're trying to convert them.
Does anybody know any patterns to prioritise the silman imbalances to... what I mean is I can notice the imbalances like it‘s second nature but how do you learn which imbalances to drop and which to keep?