or do I need to change up my memory setup? 

Everytime I play for an hour or two in "live chess" i end up haveing to close firefox and reopen it dumb it's memory usage.  Each time it hangs my resource manager shows the firefox process at around 1.6 gig memory usuage.  I guess my comps saving every move from each game.  How can cap its max memory usage?

Older versions of Firefox have known memory issues.  Make sure you are using firefox 3.0.9.  Go to Help\About Firefox to confirm your version. 

Do you have the seek graph up when you are playing?

I have firefox 3.0.9 and it happens on both my comps.  One is x32  xp and the other is running on x64 vista.  3 and 4 gigs of memory respectively.  I changed up my virtual memory usage on both comps over the weekend.  I do keep the "Room-Main Hall" form open and usually on the "seek graph" tab when playing.  

There is an excellent small free utility that optimises Firefox. It's called Finetune and Its options include preventing memory leaks. :

http://www.blogsdna.com/383/fine-tune-your-firefox-3-with-freeware-firetune-v12.htm

That's the reason I stopped using firefox.  Lately I use safari and chrome - you might want to check them out, opera is very good too. ymmv