I'm not saying a big chunk of Disney folks are also going to Universal but I think a high percentage of people coming from out of state that are going to Universal are also going to Disney for part of that stay.
Based on annual attendance difference between the two resorts, it's easy for both of those two things I just said to be true at the same time.
Also, to counter your self-admitted anecdotal evidence, I'll provide some of my own admittedly worthless anecdotal evedence not based on my circle of friends but from experience as a local in Universal parks: As a top level AP holder at Universal which includes Express Pass after 4, I see the crowd levels regularly and the impacts that Express Pass has on the regular lines both when it's open to the small percentage who pay for it and for the people who get it for "free" at Universal's deluxe resorts.
Most days at most attractions, it's a drop in the bucket until 4 when riffraff like me are allowed to use it so no, while their hotel presence has increased and their most discounted resorts don't have that perk, I don't think a majority of people visiting Universal parks are staying on Universal property, just like most people visiting WDW are not staying on Disney property, either. (and if you doubt that, ask yourself how all those hotels on I-4 between the two manage to stay in business - it's not off Sea World and the Ripley's Museum)
While I'm sure there are people staying at Universal for better accommodation at lower rates who also take advantage of that express pass perk for their days at Universal parks, I think you'd be hard pressed to find many that are spending more than three days at Universal for the parks there (today). Really, if they have the all-day express pass, a lot may not do more than two and the idea that there are a large percentage of people flying or driving in from out of state to spend a long weekend at Universal and leave doesn't feel like it lines up very well with the way either time off in this country or the financials for that kind of travel work out for most people.
Tacking on a visit at either location while in town for a convention is another story, of course but again, to recap: I doubt most people coming to Orlando for Disney are also planning to visit Universal while they're here. Otherwise, attendance at Universal parks would be through the roof but I absolutely believe a decent chunk of people who visit Universal also visit WDW. I'm not saying 100% but the number is probably closer to 50% than you're imagining since you'd need to be talking to people who weren't necessarily also Disney-first in mindset to get that impression.*
*I bet with international travel, the percentages are a good deal higher, too.