"All you people that think Bobby was afraid to play have absolutely no clue what you're talking about. Tracking turned off!"
I think Fischer quitting chess in 1972 did have a bit to do with fear of not being able to live up to his own myth. It had been very difficult to make him play already the last years before he won the title. An interesting article on Fischer with some quotes from Denker and others on the subject:
"The truth was that Bobby Fischer was running scared. "Bobby was always afraid of losing," Arnold Denker says. "I don't know why, but he was. The fear was in him. He said that if he played Karpov, he was going to insist on a long match. After not playing for three years, he was very concerned about how good he would be." Shelby Lyman echoes that assessment. "Hating to lose, and having the myth destroyed," he says, "was a big part of him not playing"
http://www.theatlantic.com/magazine/archive/2002/12/bobby-fischer-s-pathetic-endgame/302634/
I'd have a similar question - Why don't we ask the God? But maybe you are right, and Houdini would be able to say definitely "yes" or "no"