We did Cape May this past summer. Booked a very late breakfast - I think we may have been the last table seated. Boy, did that work out great for us! We had TONS of character time and the food was still good and fresh. DD declared the eggs and bacon the best she had ever had. (not sure mom and dad would go that far).
Since it is large and only has 3 characters, off-peak times would be the best. For us, though, it was perfect after doing EPCOT morning EMH - 3 hours in the park and we were ready for a good breakfast and relaxing character time.
BUT, if you are already doing Tusker House and Chef Mickey, I would go with Crystal Palace for some variety in character interaction. Tusker, Chef Mickey, and Cape May are all members of the fab 5, while Crystal Palace gets you Pooh and friends.