Based on the relatively small number of billionaires/rich people in the world, that's a compromise I'm willing to make. If the rich want to house the poor they can do so through a charitable foundation.
Furthermore, as has been pointed out already, we could make it so that only properties that sit within a predetermined land mass size would be exempt from tax.
Why go by size and not by value? Someone having a huge estate in the middle of nowhere is not a problem, vs occupying the same amount of space (or even a much smaller, but still large one) in the city centre is.
A large estate in the middle of nowhere is still potentially wasteful. For example, could that land not be used for farming or wildlife? How many bathrooms does one person need?
Furthermore, as has been pointed out already, we could make it so that only properties that sit within a predetermined land mass size would be exempt from tax.