What you've said is true, but it should be noted that not only did Xerox never file a patent suit - nobody filed a software patent suit for at least a decade afterwards. Software patents were at that time unknown. If Xerox had wanted to file a software patent suit, a good lawyer would have disabused them of that notion instantly by pointing out that that wasn't how patents worked then.
So the fact that Xerox didn't file a patent suit doesn't say much, I don't think. That doesn't mean that Xerox wasn't in the wrong or that Apple wasn't in the right; it just means that patent lawsuits really weren't the way people went about these things then.
Conversely, if Xerox had software patents, they would have likely licensed them to Apple anyway.
It's probably likely that not only did Xerox not imagine that software patents were a thing, they probably didn't even imagine that you could copyright the "look and feel" of software.
So the fact that Xerox didn't file a patent suit doesn't say much, I don't think. That doesn't mean that Xerox wasn't in the wrong or that Apple wasn't in the right; it just means that patent lawsuits really weren't the way people went about these things then.