We've been going to WDW as a family for 7 years straight --- always for 8 days 7 nights.
We started taking 2 days of those 7 visiting Uni (when they opened WWoHP). We still stay on WDW property and are back for dinner and for a night in one of the WDW parks.
There is a lot to see at Universal and a lot of what they have there (rides, theming etc) is RIGHT on par with WDW.
I'm not going to lie (with our 8 day stint) I do feel like I'm missing out on our WDW vacation but only because I've gotten used to have 8 full days to "do our thing" at the World.
So.....with all that being said - If I had it my way?? We'd take a 10 day vacation and spend 8 at WDW and 2 at Universal.
For what it's worth - I'll give you my opinon: You're not crazy....well at least not for this

I get WHY you feel that way....
But I would DEFINITELY suggest taking at least a day and check out Uni and what it has to offer.
Thats true... we really did enjoy Universal and Seaworld, yet my Disney pride just took over! LOL (Now... if I could get DH to extend a few days, JUST might visit Universal!)
I'll bet you won't be disappointed!