Because all possible UTF-8 characters give you passphrases that are hard to write down, hard to transcribe from paper onto keyboard or vice/versa, hard to repeat aloud, and hard to recognize visually.

The tradeoff for using a smaller character set is longer passphrases (for shorter character sets) versus "less humane" passphrases, for a given level of target entropy.