Walt was a man ahead of his times, but still lived in them. There are many, many things that are now Politically Incorrect which were socially acceptable 50 years ago, and were NOT said in a consciously mean way. Heck, there were things said 20 years ago, by people who do NOT hate or dislike any groups, that would bring a conversation to a screeching halt today.
I was just forwarded a joke that I first heard a couple of decades ago. Only the ethnic designation has changed. What was a specific race has been changed to Redneck. I'm sure that the person who sent the joke doesn't really hate poor, Southern, white, mostly rural people. They're just the group to ridicule these days.
There were figures of speech that were derogatory to certain groups, but we (yes, me too) uttered them without thinking about the true meaning.
"He gypped me." "Portugese Blue" "Ghetto Blaster" "Dutch Treat" "Indian Giver" The list goes on; and the majority of people who used these and similar terms would have been embarassed if they had been made aware of how insulting they are.
Walt was no different. He was a product of a different culture, and it's unfair to look back and judge him, or anyone, using today's standards of political correctness. We just didn't know any better! We were all living in the world as it was, but many of us WERE working to change it.