Idk, I personally did it for a HPC cluster two years ago. 2x100GbE + OmniPath was a sensible way to reduce cost, especially as the cluster was very light on GPU power and mostly focussed on CPU-bound jobs.
Last I heard everyone there is still very happy with what we built.
Was that before Intel spun it out? I can see people being willing to do that build if Intel was seemingly on board. Today things are pretty different.
Modern HPC mostly means GPU compute and tons of data shuffling, but point taken. CPU bound jobs aren't going to stress the i/o, so you probably could have done 100GE for less. I'm curious what you did for storage but I'm guessing with CPU-bound that is again much less of an issue.
