As a matter of fact, I am reading that very book right now. I don't care about the history much, but that particular book is great as an intruduction to modular arithmetic and elliptical equations.
Wonderful popular science books

Hmm.. Well the modular arithmetic you can learn from that book is what you learn in 10 minutes of an actual class on it at university.. =) So maybe you should get a hold on a course book instead... Also there are a lot of books, that go more into the actual proof, but are still quasi-popular.. try searching for Fermat's Last Theorem on amazon or similar..

I would like to throw out a couple that I read as an undergraduate(I don't know all of the authors offhand, I will look for them later), these books all assume little formal background and should be fine for an enthusiast without a Math degree:
Music of the Primes
Journey through Genius (This book is actually used as the textbook for the history of Math class here at the University)
e: The story of a number (Great book on both e and a lot of information of Euler)
A History of Pi
Calculus: Its history and conceptual development (I think this is the title, it is a dover book)
One that I have not read but have flipped though and plan to read at some point:
The Poincare Conjecture
A book that I am currently reading is:
Is God a Mathematician (StrangeQuark, I get a sense that you would like this one- most libraries seem to have it)

Hmm.. Well the modular arithmetic you can learn from that book is what you learn in 10 minutes of an actual class on it at university.. =) So maybe you should get a hold on a course book instead... Also there are a lot of books, that go more into the actual proof, but are still quasi-popular.. try searching for Fermat's Last Theorem on amazon or similar..
Strange Quark, as a follow up to Summum_Malum's comment, you might like to get a copy of "Number Theory" by George Andrews. It is a dover publication so it is cheap, but George Andrews is not a small name author (In fact, he is currently the president of the American Mathematical Society). As I recall, the book does not require an exteme amount of Abstract Algebra.
I was thinking that this group would be a opportune place to have this sort of forum, since the group itself is named after a book that mediates science in a popular manor.
I for one would like to recommend a book called Fermat's Last Theorem by Simon Singh.
A book I first read while in high school, it gives a fascinating recount of the history of the world's most famous math-problem. But it goes beyond this and stretches it's history back to the old Greeks, such as Pythagoras and Euclid and other important thinkers of the Hellinistic period. It also provides an insight into the (often troubled) men and women that have had an impact on the problem, either by attempting to solve it or made contributions that eventually were employed in the solution.
A tale of suicide, murder and great mathematicians crumbling under the weight of what made them great - their genius - makes it a page-turner without comparison - the first time I read it, I did so in one day!