Fischer wasn't afraid of Karpov in particular. He just had too much of his ego and self-image invested in his chess successes to take the risk of losing the championship to anyone. It was easier for him to quit playing and keep thinking of himself as the greatest player ever.
It's pretty obvious from Fischer's behavior that he was always supremely confident in his chess playing, but less confident that the world, the Soviets, etc. wouldn't find some way to cheat him out of his just due. That paranoia, once he became world champ, allowed him to make even more unreasonable demands to assure himself he could not be cheated, and in his mind, much like Carlsen now, he was the best player by a country mile (and he was at the time), and so had no reason to compromise in any way.
The '92 match proved him correct in that regard, as the prize fund still dwarfed the FIDE WCC even after his 20 year hiatus, a fact that FIDE would grind their teeth about since they had missed out on 20 years of similar prize funds.
Fischer was not a nice human being, but the anti-Fischer (i.e. anti-American) sentiments regarding his chess play and whether he would have beaten Karpov are just sour grapes. Fischer and Karpov both said Fischer would have won in '75 (and Kasparov said so, too, for good measure), so...what does the kibitzing of amateurs amount to? Not much.
Fischer wasn't afraid of Karpov in particular. He just had too much of his ego and self-image invested in his chess successes to take the risk of losing the championship to anyone. It was easier for him to quit playing and keep thinking of himself as the greatest player ever.