I wouldn't. If they can afford to fly, I'd simply say "Sorry, but that won't work for us. But, we'd love to meet you there or fly down together." You'll be exhausted when you arrive and tired of being around your family. Flying will get you there faster and you'll be ready for the next day and not dreading a few hours with your kinfolk. If they want to do a family road trip, maybe you could do something closer to home.
If the trip goes through and everyone meets at WDW, I would suggest that no one be expected to stay with the group at all times. If someone or a couple/family wants to go their separate ways or to another park, that should be fine. No one should have to stand in line to meet a character or ride Little Mermaid if they don't want to. And, be sure to discuss any meetup times and give everyone a schedule before leaving so no one can complain they 'didn't know.'
Families I know who've gone on large family trips to DL (flying there, not driving...although the times they drove places together have given us many entertaining tales and made us thankful we weren't there) always split up and had strict ground rules for everyone.