Strong agree on it all being a legal minefield / new grass.
> But was almost certainly part of its training data, which complicates things
On this point specifically, my read of the Anthropic lawsuit was one of the precedents was that if it trains on something but does not regurgitate it, its fair use? Might help the argument that it was clean-room but ¯\_(ツ)_/¯
Is your (1) description of clean room implementation, and (2) description of what was done, actually correct?
(1): my understanding was that a party _with access to copyrighted material_ made the functional spec, which was communicated to a party without access [1]. Under my understanding, theres no requirement for the authors of the functional spec to be 'clean'.
(2) Afaict, they limited the AI to access of just the functional spec and audited that it did not see the original source.
Edit: Not sure if sharing the 'test suite' matters, probably something for the courts in the unlikely event this ever gets there.
[1] Following the definition of clean room re implementation as it relates to US precedent, ie that described in the wikipedia page.
One thing I will add: while AI is getting really good at _doing_ the software building bits, I haven't yet seen it well integrated into the decision-making and political structure of organizations. Right now, it seems best in the hands of a high-agency individual empowered and able to make changes or 'ship' something, with them acting as the bridge.
This of course, is not a technical challenge, but I would expect the change in structure of organizations to make this more efficient to be slower than the pace of improvement we've seen over the last few years.
On (1):
Might be useful to separate investment flows from the rest of US's economic activity.
AI investment is propping up capital flows, the GDP statistic, and responsible for most of the gains on SPX, but its still a small fraction of the economy.
> Does Waymo have the same object permanence and trajectory prediction (combined) to that of a human?
On this note specifically ive actually been impressed, ie when driving down Oak st in SF (fast road, tightly parked cars) I've often observed it slow if someone on a scooter on the sidewalk turns to look toward oncoming traffic (as if to start riding), or to slow passing parked box trucks (which block vision of potential pedestrians)
Its not deterministic. Any individual floating point mul/add is deterministic, but in a GPU these are all happening in parallel and the accumulation is in the order they happen to complete.
When you add A then B then C, you get a different answer than C then A then B, because floating point, approximation error, subnormals etc.
It can be made deterministic. It's not trivial and can slow it down a bit (not much) but there are environment variables you can set to make your GPU computations bitwise reproducible. I have done this in training models with Pytorch.
Some of what OP is saying generalizes to the concept of being "too early" - if you are early, your engineering / innovation spend is used to discover that at-the-time reasonable ideas don't work, or don't work with the current appetite, whereas later entrants can skip this exploration and start with a simple copycat.
My (business-school) partner reminds me that first movers are seldom winners.
That perfectly surmised my experience. I've been "too early" far too frequently.
Before ElevenLabs, I built an AI TTS website that got 6.5 million monthly users at peak [1]. PewDiePie and various musicians were using it. It didn't have zero shot or fine tuning, so it got wiped out pretty easily when ElevenLabs arrived.
Before Image-to-Video models got good and popular, I built a ridiculous 3D nonlinear video editor [2] for crazy people that might want to use mocap gear and timelines to control AI animation. You couldn't control the starting frame, which sucked, but you could control the precise animation minus hallucination artifacts. Luma Labs Dream Machine came out just a few weeks after our launch and utterly wiped the floor with our entire approach.
I was late to build an aggregator, but I'm a filmmaker and I'm stubborn and passionate. I'm now trying to undercut the website aggregators with a fair source desktop "bring your own keys" system [3]. Hopefully I'm "just in time" for these systems to become desktop, with spatially controllable blocking, and with "world model" integration (nobody else has that yet). It's also Rust and when I port the UX to Bevy, it's gonna sing.
What convinces me is this: I live in SF and have friends at various top labs, and even ignoring architecture improvements the common theme is this: any time researchers have spent time to improve understanding on some specific part of a domain (whether via SFT or RL or whatever), its always worked. Not superhuman, but measurable, repeatable improvements. In the words of sutskever, "these models.. they just wanna learn".
Inb4 all natural trends are sigmoidal or whatever, but so far, the trend is roughly linear, and we havent seen seen a trace of a plateau.
Theres the common argument that "Ghipiti 3 vs 4 was a much bigger step change" but its not if you consider the progression from much before, i.e. BERT and such, then it looks fairly linear /w a side of noise (fries).
> But was almost certainly part of its training data, which complicates things
On this point specifically, my read of the Anthropic lawsuit was one of the precedents was that if it trains on something but does not regurgitate it, its fair use? Might help the argument that it was clean-room but ¯\_(ツ)_/¯
reply