"He who fears the isolated queen's pawn should give up chess." -- Tarrasch
Maybe a bit harsh... he was trying to prove his Tarrasch defense was sound, contrary to current ideas at the time which held all isolated pawns as weak, but it's a fun quote to throw in here.
I used to feel the same way though... like I was playing against the clock (giving my opponent some kind of endgame odds). But especially in non-tournament games don't be afraid to experiment and see if you can make use of this space / active play. Maybe attacking chess just isn't your thing, in which case you can choose different opening variations to play. Or as pelik said you can try to just exchange it off every time.
does that mean that you're not a good player?
Or at least, not a well-rounded player?
I personally love to play AGAINST the IQP and I would never play with it. I just feel like it's being a pawn down. I don't see where is white's activity to compensate for it, unless black makes the mistake of not developing his lsB outside his pawn chain.
The main point is that I don't like to play a game where I'm fighting "against the clock" in the sense that every move that brings the endgame closer will be against me, and I can't even trade freely.
So, does that mean that I'm not a good chess player because I can't use effectively piece activity?
Or is it legit to just prefer a long term asset (positional weakness of the isolated pawn) versus dynamic compensation that will just wither away if not used?