I mean, they probably do - so if 9/10 people said the word is "apple" and I said it's "banana" then it will accept apple. But at the same time I can't imagine I was the only one doing this.

Sure, but unless you're coordinating with other adversaries it shouldn't be hard for them to filter out the deviant responses.