Taking kids to higher end restaurants is a judgement call for parents. At 5, my son was very well behaved in a nice restaurant, so we didn't worry about it and booked a few nicer restaurants. All went well except one. After a long, hot, tiring day he had a meltdown right after we'd ordered. We ended up tag-team eating our meals, taking turns taking him out of the restaurant while the other ate a bit. Once we'd finished our entrees, he'd calmed down and our server brought him a special treat for dessert and was effusive thanking us for not just keeping him at the table the whole time. I'm pretty sure everyone in our vicinity was really grateful too.
It's ultimately up to the parents, who know how their kids are, to decide whether a younger whipper-snapper is going to be able to cope for a little while in that atmosphere. I've experienced a few meals where screaming kids running around the restaurant ruined the meal for the rest of the guests. There's not much you can do about what other parents do...