First you have to understand the point of that fianchettoed bishop. Once you do and then if you see that it cannot do its thematic job, and you can't find other great uses for it, then maybe such a trade is reasonable. As you say, case by case at that point.
When is fianchettoed bishop worth trading
As with all questions like this, it depends on your unique position. However, you should very rarely trade a fianchettoed bishop unless it makes tactical sense to (you win a piece).
The reason why is its position on the board. Knights are most powerful on the center of the board because they have the most possible moves available to them (rather than in the corners where they have only 2 moves). They are also more powerful in closed games (since they can navigate more easily around pawns).
Bishops, on the other hand are generally more powerful in the corners, or on the long diagnols. This allows them the most control of the board. They are also made more powerful by an open game, where no other pieces inhibit their movement.
Generally, in classical lines where both players attempt to take the center, knights are stronger early on because the game will be closed. But later in the game, bishops will come to life once the center is resolved.
Another reason giving up a fianchettoed king-side bishop is usually a poor idea is that it creates a weakness on your black squares.
Keep in mind that all of these statements are generalities. I assure you that a little searching will yield numerous grandmaster games in which the opposite of what I have said has held true.
Try posting a specific game in which you believe you should trade your bishop and see what analysis people offer.
As I play mostly pirc/modern defense as black, I often find myself in the situation where I could trade my g7 fianchettoed bishop for the c3 knight. If white's d pawn is gone (so trading would create isolated doubled pawns), is it worth creating a defensive nightmare to wreck an opponent's pawn structure? I realize this is a case by case problem, but any ideas are appreciated.