Would we take Adam and Eve to be relatively our height? If true, then how can a very micro number of mutations that we see in 1 generation give rise to tall and short features in humans? It's also to be noted that Seth and Cain had very few mutations because they were the children of Adam.
Ah! I found an idea. What if the mark of Cain has to do with regulating the heights. Maybe Adam was the giant. And Cain was cursed to be short. God set spiritual laws that they should not impregnate. But they violated it and did it anyway and this gave rise to giants.
But I see a flaw in my reasoning. It presupposes that Adam was indeed tall.
It may be that a mutation suddenly caused Adam's offspring to be giant tall and the other offspring to be dwarfed to our own height.
Also, in Genesis 6, while it says "giants", there no reason to think it means anything other than very tall people. There was a man in Illinois who was almost 9 feet tall: https://en.wikipedia.org/wiki/Robert_Wadlow. He could be described as a giant. Goliath was only a few inches taller than that.
Perhaps the righteous line of Seth ("sons of God") or the wicked line of Cain or of Adam's other children ("daughters of men") included some very tall people. Anyone over 7-8 feet tall would be legitimately described as a giant. There's no reason to take the word "giant" as a reference to a mythical creature 30 feet tall.