But what about builds that combine 8 of the 5090 with infiniband between boxes? Wouldn't that be comparable to the mac in terms of price and potentially beat it by a lot in terms of performance for the large MoE? I understand the space/heat/noise considerations, but price wise it may still not make as much sense as people think. (Agreed that it is hard to get the NVIDIA hardware and the 6000 pro are priced less competitively).
Others have chimed in on cost and size, I'll chime in on power. The 8x 5090s will require a dedicated datacentre grade power source. The Mac Studio runs on a plain jane wall socket.
Where are you buying 8 5090s for under $10k? With CPU, RAM, and (checks comment) infiniband hardware???
You're probably looking at a lot closer to $60k when all is said and done, and that's before you hire an electrician to run a sub panel for your homelab...
While that sounds super awesome,
How many people are actually going to build and maintain that vs a box you can grab at the mall that fits in a lunchbox?
I can't speak to Infiniband pricing for something like that. It seems like the cheap option is 56/100Gbps with used Enterprise equipment. You'd need 8 HCAs, DAC cabling and a switch but even then you're into thousands of dollars. If you want 200Gbps+ it gets into the tens of thousands (AFAICT).
Each PC is probably going to cost ~$6k and you're talking about 8000W of electricity draw. That's going to consume multiple 20A circuits even at 240V. And the electricity ain't free either. A Mac Studio seems to draw ~500W max.
Oh and the Mac Studio has an upgrade route to run 1T+ models too by chaining them together with TB5 chaining. OSX supports RDMA this way. That's comparable bandwidth to the 100Gbps Infiniband option.
So you're talking about $50-60k of hardware and more power draw and more heat for something that will I'm sure beat the MS M5U option but at huge cost. Also, at that kind of price point, I'm likely to get a workstation PC and put 2 (or possibly 3) 6000 Pros in it.
> You need a Kubernetes cluster, ko (brew install ko), a container registry your cluster can pull from, and a reachable Agent Substrate Control API (in-cluster default: api.ate-system.svc.cluster.local:443).
> make deploy AX_IMAGE_REPO=<your-registry>
> This deploys Redis, then builds and deploys the control plane images with ko. Everything lands in the ax-system namespace.
Having worked with people doing bringup of specialized chips, I am awed at how the world has changed.
> When the first chips came back from the foundry in May, the team pointed its internal AI models at designing software to run benchmarks such as SemiAnalysis’s InferenceX. On DeepSeek’s multi-head latent attention kernel benchmark, performance climbed from 0.31 percent of the theoretical ceiling (set by the chip’s compute and memory bandwidth) to 88.94 percent in roughly 40 hours. Ho says this result is repeatable, so the time between when foundries deliver the first chips and when production ramps up can be reduced. “All our schedule assumptions are going to be based on the fact we have this capability now,” he says.
I've got some Tang nano and Tang primer boards (9k, 20k and 25k LUTs). The 9K has a cute little SPI LCD display. Claude has been able to interface with the boards via USB (it setup a UART on the FPGA to do so). We've gotten a GoL as well as a Pong game (where paddles are controlled by Tsetlin machines) and a MicroCNN that does MNIST classification. For the latter we were having trouble getting correct answers back from the CNN (accuracy was very poor which didn't match simulation). Claude created an on-chip logic analyzer and doggedly troubleshot the problem (turned out to be some kind of problem in the yosys synthesis tool - we had to use -noalu to work around). Anyway, I was blown away by this. Claude was communicating with the FPGA board and analyzing the data coming back from the internal logic analyzer.
Many languages can compile a subset of their code to FPGA HDLs. Back in the 80s Harel's group had statecharts that were compilable to C, C++, and FPGA HDLs. Not sure that LLMs brings anything substantially new to this.
I’m also very curious! I got a pair of icebreaker boards [0] and they’ve been great to toy around with.
The tooling is open source, and Fable in a loop - especially when paired with a digital scope that Fable interfaces with (the Saleae’s [1] are great) - gives you a level of verifiability that feels like beyond what software typically gives you. ie it feels more like Lean than code with tests.
I had ai implement a few toy circuits (sha hashing, 8088 emulation, a tiny llm) but yeah. Still looking for fun applications.
There have been a few recent fpga threads on hn, check them out. [2][3]
I had this same thought and think this is a generally interesting direction, but I think we're in a bit of a weird spot where the compute heavy stuff is on GPUs already and most infra stuff is not compute bound (it's often I/O bound or memory bound in some way).
It doesn't help that FPGAs are not made at the same scale as CPUs so don't benefit from the economies of scale.
I'm super curious if you have thoughts on specific pieces of software that would be economically better because I've thought about this in my niche and sort of come to the conclusion that it won't help.
I do think things like SIMD in CPUs will get more use and maybe we will get more difficult to program for CPU features, but I haven't found a use case where off the shelf FPGA components would help with typical software.
I’m looking at realtime mechanical processes, like shaping extrusion beads from a clay 3d printer. Clay is heterogenous and pressure takes time, so hand tuning is never just right. But put an fpga with vision processing? Seems promising, with millisecond-level latency that I’d never get pushing to a remote system for processing.
Unclear that this sort of thing wouldn't already be quite well served by the existing gpu/npu hardware optimized for neural nets. If you're doing traditional CV, you can run it on the CPU just fine.
Yeah. Just spent an hour planning a closed-loop vision-based extrusion modulation system for 3d printing, with extensive telemetry and offline processing to iterate on the realtime system. Great, like I need another side project.
They'd write a limited test for a feature based on an ask from the software team garbled by a five layer game of telephone. Claim that the module passed validation. A few months later the software folks would have to pull a few all nighters to figure out how to work around the resulting turd during bringup.
cant wait for no one to really know whats in chips i mean, even intel hardly knows what all their reserved mem ranges are for. who will decap the chip and see if the docs were right? xD
No, you forgot about the dependencies (others said it to be 700Mb). And 2,900 LOC is still way too much for handling a remote API call, especially in a high level language such as Python where the biggest part of the task is delegated to separate modules.
On the contrary, the readme for this project, `litelm`, is contrasting with the bloat project, `litellm`. This project subty deboats an L.
The bloat (or featuritis) project has the 700mb and 100k+ LOC, this de-bloat has 2k LOC and specifies its only two deps ones most people want as deps (although httpx should be swapped to its successor).
So this response is off base, the 700Mb isn't forgotten, it's been dropped.
Perhaps you did not understand the Fermat theorem proof announcement/repo or the link. The 13 million lines did not use any external, possibly not honest libraries, as the proof eventually only used the fundamental axioms. So for the Fermat theorem formalization, no open open questions remain.
The point is, they "proved" the Collatz conjecture. You would not know they exploited a bug unless you actually went and dug into their proof. Can we be so certain this has not happened within the millions of lines of Navier-Stokes? In an ideal world, our proof assistants would be more battle-hardened by now (recent exploits deny this), our AI better aligned (their tendency to cheat at tests denies this), or their handlers more responsible (the Hugging Face incident denies this), but the reality is more complicated.
At this point in time, we really can't be confident in accepting proof certificates without any human eyes on the script that generated it. I still have 95%+ confidence in this particular result being trustworthy, but a precedent of blind faith is guaranteed to end badly.
This person knew they did not prove the Collatz conjecture and others independently figured it out within hours. Not sure this is at all relevant, other than pointing out how trivial it is for the community to understand errors in lean4.
It was trivial because the Collatz proof script is literally 1000x smaller than the script for Navier-Stokes and involves no advanced math. And they found the bug by... manually inspecting the proof script. Maybe we should do the same for Navier-Stokes before declaring the matter settled?
Not only that, but there is a very fuzzable tell of something funny in the Collatz proof script (`CommandElabM`, i.e. metaprogramming). We may not at all be so lucky in other malicious scripts, especially if there are still kernel-level bugs in Lean.
I am not sure what you mean here. There exist plenty high-concern biological threats that dont need any AI help. Human oncovirus design is low on my concern list (immunity is diverse), and in any case it does not need AI—rather labspace. Disgruntled high schoolers or undergrad chemists can do way more damage from readily available materials without AI and without research delay. As can nature (or amateur biologists without AI) by mixing bats with their animal of choice and waiting a while. If/when any scary global events like covid happen again, I sure hope we have true superintelligence to help us navigate it quickly.
Other than the undeniable breakthrough in math, the important point is the ability to orchestrate 10k agents to productively work on a single problem, which creates options:
> OpenAI, meanwhile, says its experience with Navier-Stokes could open the door to solving puzzles with more practical relevance. “We are now able to spend millions of dollars on a problem that we really care about and that really matters: developing new materials, finding cures to diseases,” Bubeck said. “All of those things that we have been talking about for a long time—now they seem to be at our fingertips.”
reply