We do when we can. It's so neat to see and "feel the soul" of the different resorts. I especially liked visiting the resorts this past year to see the holiday decorations. We worked it into our schedule so it was a good way to escape the parks and relax/wander the resorts.
For example, we left HS, took a bus to Boardwalk and headed to dinner at Epcot. On the way back to HS (for the Osbourne lights) we went through the Yacht and Beach Club.
On another day we were at the MK, took the monorail over to the Contemporary (had a few drinks), then to Polynesian (apps and drinks) and finished at GF, before Mickey's Merry Christmas Party.
I do wish I could have seen the Wilderness as well as some of the other resorts. This trip we stayed at POFQ and loved it. The mix of Jazz and holiday was really stunning.
When we stayed at the Wilderness Lodge years ago it seemed like a lot of people would take a boat over, mosey around and then chill out in the rockers. It's a great place to explore and unwind.