Almost everyone agrees that, atleast in their first match, Fischer would have had the advantage. Kasparov, Kramnik, and even Karpov have all said that the match in 1974 would have been favorable to Fischer, but that there's a good chance by 1978 that Karpov could win.
It's interesting to note though, however, that Fischer didn't stop with Chess altogether after 1972 after winning the title. He was still pretty obsessed. The same person that created his "Big Red Book" of Spassky also created one for him of Karpov to study. And if you check one of the issues of Chess Life in 1975 Larry Evan's column, Bobby Fischer is analyzing Karpov's game and telling Larry that Karpov probably should have lost the game against Pritchett. Here it is:
I'm pretty sure it was a forfeit win.