Yes!  That is my experience and that's why I go solo without hubby to WDW.  I have a great time, and I never try to force the same vacation on him, if he doesn't like to go there.  Different strokes for different folks.
@BuddyThomas , another idea is to do a sort of split-stay vacation.  What if you guys went for say, 3 days, and then went to another location (let's say, the Florida Keys for example here) for another 3-4 days?  Maybe that might work--best of both worlds for both of you?  Just a thought . . .