I like them both. Despite sharing some similar show scenes and style, they're completely different ride experiences. Phantom Manor is the more frightening and disturbing of the two. With realistic skeletons and a backstory involving the death of a father, and what i believe is the hints of the father coming back to life as a demonic phantom to kill off his son in law and eternally punish his own daughter for her disobedience. That's the feeling i got about the ride anyways. I know the phantom is sort of a mysterious entity, but when you understand the back story, it makes me really believe that the phantom is actually the dead father who sold his soul to punish his family. The entire thing is rather disturbing, with the phantom hanging Melanie's fiance in the stretching room, to him laughing at her mockingly when he fails to show up for the wedding ball, to the skeletons in the caves, and to Melanie herself growing old (still wearing her wedding dress) and eventually dying (again, heavily hinting that the phantom killed her as he is holding the shovel next to her grave). It's quite a disturbing experience. I feel it is great in its own way. Definitely worth riding. Before our Haunted Mansion got an overhaul in '07, i feel Phantom Manor was the superior experience.
Ours is more funny. The most disturbing scene is probably the one in the stretching room (either that, or the unsettling change of Master Gracey's portrait as his young face degenerates slowly into a skull). Otherwise, a lot of it is funny and not meant to be disturbing. I wouldn't say either ride is necessarily better, they're both equal IMO but totally different types of experiences.
Watch Martin's tribute of Phantom Manor sometime. One of his videos has it with the original Vincent Price narration (before they changed it to French only). He voices the ghost host (the Phantom) and is very awesome in it. Hard to beat the original ghost host, but Vincent Price is legendary too.