I suggest you take a deeper look at what's happening.
The Fed announced they will transition away from QE yesterday. When they did that 5 years ago, a lot of things happened and they completely gave up on the idea.
It's not that you need to turn on some extra library backends and rebuild, it's that the abstractions themselves are fundamentally at odds with hitting peak performance on many things so you have to rewrite your code.
Individual image processing operations are often very low arithmetic intensity. If you don't combine them into much larger subroutines—which are necessarily less generic and orthogonal—you spend all your time waiting on memory between every little op.
To make "next to nothing" concrete: MIT EECS PhD students are currently paid about $4700/mo. This is substantially less than they'd make in industry, but it's around the US median personal income across all working-age adults, and well above the average 24 year-old. They frequently make a substantial extra at summer internships, putting them well above US median in the years when they do.
Also: it is school, not just a job. They are developing deep expertise and specialized skills. As a result, among other things, their earning potential tends to be significantly higher coming out of the PhD than out of undergrad.
You're looking at students at a top tier university in a field that pays extremely well. The numbers are going to be at the high end for what a grad student can make. A quick search for PhD salaries suggests that $20-35k/year is more common.
The median wage number you cited is also for the total population. According to this graph the median wage for college graduates is around $7k/mo. I'm fortunate to make very good money but I'd still notice a $2k/mo pay cut.
That's MIT. At a state university my friends were making in the ballpark of ~30k.
And yes, that is "next to nothing" compared to the salaries they make now after quitting and just finding work. And their outlooks are in significantly better shape, whereas one friend was highly depressed before.
People can also develop "deep expertise and specialized skills" through their work, and network via conferences, generally paid by their employer. Well, if they can find a job as a junior nowadays.
The 1TW is the rated peak power output. It's essentially the same in space. The thing that changes is the average fraction of this sustained over time (due to day/night/seasons/atmosphere, or the lack of all of the above).
It's still the same 1TW theoretical peak in space, it's just that you can actually use close to that full capacity all the time, whereas on earth you'd need to over-provision substantially and add storage, so 1TW of panels can only drive perhaps a few hundred GW of average load.
The PyTorch version is using the GPU (with Metal Performance Shaders); this C version is currently using (in the docs I saw) a single CPU core, with AMX (via Apple Accelerate BLAS) but not yet with OpenMP for parallelism. It’s not slow because LLM code is bad, but because it’s not running on the same hardware. That said, it’s also not as fast as it is because of the LLM—all the critical code is in kernel libraries it calls (the same as for PyTorch).
Why are GPUs necessarily higher latency than TPUs? Both require roughly the same arithmetic intensity and use the same memory technology at roughly the same bandwidth.
And our LLMs still have latencies well into the human perceptible range. If there's any necessary, architectural difference in latency between TPU and GPU, I'm fairly sure it would be far below that.
My understanding is that TPUs do not use memory in the same way. GPUs need to do significantly more store/fetch operations from HBM, where TPUs pipeline data through systolic arrays far more. From what I've heard this generally improves latency and also reduces the overhead of supporting large context windows.
It is not only not that much more complex, it is often less complex.
Higher-level services like PaaS (Heroku and above) genuinely do abstract a number of details. But EC2 is just renting pseudo-bare computers—they save no complexity, and they add more by being diskless and requiring networked storage (EBS). The main thing they give you is the ability to spin up arbitrarily many more identical instances at a moment’s notice (usually, at least theoretically, though the amount of the time that you actually hit unavailability or shadow quotas is surprisingly high).
