It's not about the word hacker. It's about what the word represents and that go so much beyond the /b/tards.

Keeping your analogy: there's a lot of gay people who think the "fabulousity" of the gay parade may hurt their cause in getting au pair with civil rights. Whether this is true or not, I think it's a valid argument to be discussed.

Same goes for hacker: those who attribute hackerism only to black hats, vandals and script kiddies may hurt the values true hackers stand for.