<OP> You're right. It's really unpleasant. I hope and believe that most people wouldn't act like that - and that at least some of them would grant the draw.
It also depends how the position was BEFORE your mouse slip, and what's the relation between the ratings.
If you had a bad position anyway, or you had much less time or your clock (in blitz), or if you're lower-rated by about 100 points or more - I would never grant the draw.
best answer
Why should you get a draw? It was your error. If you make a fatal mouse slip just resign, tell your opponent how you screwed up, and ask if you can start another game.