I know that feeling. I once read a book when in High School called "Wind from the Carolina" by Robert Wilder. It is a big book that follows a family that made a fortune in So. Carolina via slaves and cotton. It starts just before the revolutionary war. The patriarch of the family is upset by the stirrings of revolution and packs up everything, and I mean everything included the southern mansion, dismantled brick by brick, and moved to Great Exuma Island in the Bahamas. Rebuilds on the land grant from England and tries to make it all work from there. Anyway it goes for over 3 generations and follow the lives of the family, the ups and downs. From going broke to finally changing gears and getting their wealth back and so on.
It made me want to go to Great Exuma, Bahama's and find the ruins of the old mansion even though I knew that the whole book was historical fiction. I finally went on a cruise to the Bahama's, but just to Nassau where I was finally able to click my brain into acknowledging that the specific location was located only in the pages of the book. I have worn out one copy of the book and acquired another on Ebay about 20 years ago, but have not seen it available anywhere else since then.