I think way more gambit are stuff that can be very dangerous. people tend to pretend because they aren't seen at top level that they've all been refuted. Even if some have been refuted, many just falter out if opponent knows best lines. however if you play otb you learn pretty quickly that a master who plays any such line is going to make your life hell, because they know it better. that is all it amounts to in these lines, the person who knows more will win. In contrast there is less margin of error when you gambit, so they are less played because most chess players are a bit risk adverse. But don't let that make you think an opening like smith mora or any other you can just play without knowing what you're doing.
Your Scariest Gambits
The chess engines would probably agree that all gambits should be accepted, but since I play humans I'll always play the King's Gambit if my opponent responds 1...e5.

The success of the Tal Gambit (1.e4 c5 2.f4 d5! 3.exd5 Nf6!) led to the decline of 2.f4. This is a mainstream example where a gambit option is considered to be the objectively strongest approach.

The Queen's Gambit is one I can think of where if a person doesn't like the Slav Defense that it's better to play QGD.

QG was what I had in mind while posting, but accepting doesn't put Black in a lost position (this is what I'm thinking about)
Are there any opening gambits that should be outright declined?
What suspicious moves do you dread to see from your opponent?
Do you have a pet gambit that you'll always play if the opportunity presents itself?