His biggest "problem" is that he hasn't lived his life as a figure in front of a camera. His career wasn't made by shaking hands and smiling for photo ops. He tiptoes the line of understanding what people want or need to hear with mostly actual science, and he's not very polished doing it. The media would have slaughtered any attempts to have an NIH spokesman out there instead of a real, hands dirty epidemiologist. Most people can sift through what his "real" messages are, or what changes in those messages actually mean. Unfortunately, the world we're in now of instant meme creation and incessant social media headline garbage has attempted to hurt his credibility.
Imagine what the world would have done to him during the Reagan years and the early HIV days if Facebook and the like were around then. I was in middle school and couldn't keep up with how quickly we learned new things, but the experts figured it out relatively quickly and at least got ahead of the spread.