If the dent in your bank account takes your attention away from the more important things, like spending time with loved ones and generally enjoying your trip...then you shouldn't do it.
Shookie's analysis of the hotels (value vs mod vs deluxe) is spot on, I think. I can't say it any better. I normally stay at moderates, last year I did a value resort because I spent a few nights over at a Universal Resort Hotel, and they are pricey.
I noticed the difference between moderates and values, and while my stay at the Pop/AoA complex (Yeah, I think of them as a complex, not two separate resorts...comes from staying there, I ate at AoA all the time. Their food court is awesome!)...ahem sorry
While my stay at the Pop/AoA complex was amazingly fun, I missed the quieter feel of CBR. This year I'm trying PoFQ for our long stay. But, I splurged for a night at Contemporary right before our cruise. For one reason...Contemporary was where my parents stayed with me back in the early 80s on my first trip, and I'd like to have a night staying there as a guest with my daughter. That's it.
Cost wise, the Deluxe resorts are just way to much. While I could afford them, I'd find myself penny pinching on other parts of the budget, like Dining Plan and shopping. I'd rather not have those concerns when at the Parks. I like to show up, know that I have a deep shopping kitty that I don't have to penny pinch with (though, I can't afford stuff like in the Crystal shop, 20k for a Crystal Cinderella Castle...wha!?), I can afford a few upscale gifts (a skagan watch, for example) and other toys and souvaniers without worrying about it as I'm shopping, and only doing loose mental accounting.
So, for me, Mods are the way to go. But, that's me. Every family and even every trip is different.