When to give up? Whenever you would feel happy about your life without any Chess in it.
When not to give up? Whenever you would miss Chess significantly and benefit a lot from participating in it.
It's up to you to decide. Don't play Chess to be the champion, accept that natural talent is a barrier, that some nerds (not an insult) have the time to spend relentlessly studying the game while you have other life priorities potentially and so many other factors.
We all have our specialties and shortfalls in life. Chess may simply not be (or may well be) your calling, don't force it either way.
I'd say you should give up immediately, before you get even more involved in this thing we call "chess," when in reality it translates to "mess," bc chess is a mess.