> there's just not enough people around to form groups of people that are different.

That's certainly part of it, although I think the bigger factor is that people who are different just leave. Small towns are conformist because of survivorship bias.