I doubt Fischer feared losing. He seemed supremely confident in his chess abilities.
Was there a chance of Karpov winning the match? Of course. Karpov's play fit every definition of "exceptional".
But Fischer likely didn't think so. He appeared to consider everyone on earth beneath him, as far as chess ability went.
The way I see it, Fischer's refusal to defend his title was meant as a middle finger to the world - depriving us all of another chance to witness his brilliance at the board.
"Not going to meet my demands?" he appeared to say, "Then here are the consequences: you don't get to see me play anymore. Deal with that."
This seemed to be, from his perspective, the most severe punishment he could ever deliver.
Fischer had said that he would be a very active World Champion, he would play lots of tournaments
Sure, but knowing how that played out, we can take that as an insight into his values. He despised the idea that the Soviets were insulated from having to face the best challenger (himself) and he admired the idea of a competitor who wins against all opposition.