I think it depends on your age more than anything. I cannot imagine that when I was young it ever would have mattered to me at all. But then again, back then, I wasn't a fan.
I have been divorced now for 11 years (not Disney related) and my interests are much more limited. One of the big things for me is WDW. I have dated a few times since my divorce and been in a couple of relationships since then, but they all ended and they usually were connected to there lack of desire to be involved with Disney. There, of course, were other reasons as well, but when your interests are basically sacrificed because of an outside agent stifling them, it's hard to feel positive about being together.
Now that I am older then dirt, finding anyone interested in a relationship is difficult anyway, but since a mad, passionate love life isn't really going to happen at my age, there has to be some other bonding method.
When I meet someone within a brief period of time I will bring up the subject. I have gotten a variety of responses, the two most interesting was one person that just rolled her eyes at me
and the other that actually asked me.."How old did you say you were?
Needless to say, those two never got to stage two!
So that was a long way of answering the question which is Yes, it definitely can be a deal breaker for me.