Having just come back from our trip, it really depended on if we had done the ride before, if fast pass was available, and how much we really wanted to ride it.
For example, we used fast pass for Soarin' but managed to get it first thing in the park. But TSMM was out of passes by the time we got there, and since we've heard good things, we waited 60 minutes (though the sign said 45). We had a blast and would have riden it again, but the sign was up to 65 minutes when we left and looked longer than that. We decided that was too much to wait and moved on.
As much as we like Peter Pan, we decided not to get into the line if it wasn't listed at less than 25 minutes and appears to actually be that length.
I also think this relates to what else you can do while you are waiting. Even though TSMM was long, the que is neat and has the talking Mr. Potato Head. That occupied us for a while, so the wait felt shorter than the time we spent in line. Not so short that we wanted to ride again, but shorter than the 45 minute wait for Peter Pan.
Bring on more interactive ques!
I should count myself lucky that my favorite ride is Haunted Mansion, and we never waited more than 3 minutes to ride it. I should also thank my family for riding it more than any other ride... how much is too much really...