He became world champion against the soviets, who were organizing against him..
Fischer said; I'm not scared of Karpov, I've got a higher rating, better tournament record, etc
Karpov said; he probably wasn't scared of me, but himself
My guess is Karpov was someone who could beat him, and was a big challenge for Fischer after he had already taken on a lot.
...I personally don't think Fischer was afraid of anyone...I believe he perceived himself akin to "Superman"...unbeatable, indestructable...and all that stuff...he clearly had an "air" about him...and seemed to feel that everyone else was beneath him...a larger than life persona...those with this sort of personality (or disorder) don't fear anyone, no matter the odds...
...could he have beaten Karpov at the time? Sure, why not? Could Karpov defeat Fischer? Sure, why not?...the cold hard truth of the matter is that Fischer refused to play the likes of Karpov and Kasparov for whatever reason, and has left the chess world forever thinking "what if?"...to forever fight amongst ourselves in verbal combat, about a great player who walked away from the 64 squares...