I lived there for about half my youth (though I was never truly fluent, at least I wouldn't consider myself such)...my sister was born there.
I love the country, and miss it...
The hardest part for a "non speaker", I'd imagine, is the gutteral sounds. The alphabet, if you learn that with the proper german phonetics, is EXACTLY how most things are pronounced (remember, they have a few more letters than our alphabet does...)
I actually have a German friend. The thing is he hasn't been to Germany since he was 5, though he has a dual citizenship. His mom said he used to be fluent in German, but he doesn't remember any of it. Actually, he knows some swear words from when his mom got mad at him.