They don't play it because of 4...d5!. And white's position isn't necessarily bad, it's just that there are so many better ways of attacking 1...e5.
After 4...d5 white's attacking chances are gone (it is more likely that black will be the one who gets an attack). I would guess that if black were forced to accept the Gambit on c3, then the Goring would be much more popular.
I have recently been studying the scotch game and I tried to learn the Göring gambit. I kind of like it but I read that it isn't really used in higher ratings and I wondered why. Someone wrote on Reddit that the reason is because it simplifies the game too early in the game. Is that the case? Are there any other reasons?