This is all technically impressive but was it all technically necessary? Was infiniband really just not good enough? All this R&D for a custom protocol and custom NICs seems to just be a massive flex of Tesla's engineering muscle.
Infiniband suppliers charge crazy prices due to having little competition. It might actually be cheaper for them to design their own than to pay the Infiniband tax.
And supply chain independence. I've heard that some GPU clouds are delayed because their Infiniband hardware was delayed due to the Israel–Hamas war. Optimally you probably want to avoid critical hardware that's being manufactured in a high risk of disruption zone.
It's not only about being "good enough" but also about reliability and maintenance, new protocols and hardware may take time to mature while other solutions are already there. Ah, wait, Tesla doesn't care about those kind of things too much...
