My thinking here is that Waymo has logged hundreds of millions of miles at this point (and even more in sim), and there are a lot of nines in their safety record. So even the rare edge cases should have come up. You make a good point that the deep water scenario may be not only rare, but also under-represented in the training data. On the other hand, you'd think they would have thought to explicitly train for it; it's not that obscure of a possibility.
Ya, this does strike me as pretty high on the list of the thing I'd intentionally seek out if I was doing this job, along with object-in-the-road, icy road, tornado/hurricane, wildfire, and perhaps hail.
Writing a simulator requires understanding the game well enough to spec the simulator. Acquiring said understanding - within the action limits of the benchmark – seems like the heart of the challenge, so this doesn't strike me as "cheating" at all.
In essentially all cases where a Waymo and a human-driven car have collided, the human driver has clearly been at fault. This seems definitive and not susceptible to cherry picking.
That could just be, and seems to be in some cases at least, because Waymo doesn't behave like a human would, and people gets tripped up.
I don't doubt Waymos are very safe, but I always irk at these comparisons. Majority of human accidents are due to gross negligence and/or driving under some influence or serious fatigue. A system incapable of alcohol etc. is better than that? Well that is a substantially lower bar than you can possibly imagine. Add to that that all systems have constraints on how and where they are able to go. Combined even Tesla can be made to look good.
Depending on the context and question it might still be the question to pose. But people often make the leap to assume that a typical Waymo is x better than a typical human driver which is an entirely different question entirely.
Waymo is for sure one of the (if not the only) good players out there though, gives me some hope.
> could just be, and seems to be in some cases at least, because Waymo doesn't behave like a human would, and people gets tripped up
Driving conventions vary wildly across states and even within them. And foreign drivers are a thing. A human who gets tripped up by a Waymo acting unusually will also get confused by someone getting used to no turns on right in Manhattan, driving on the right side of the road if coming in from the Commonwealth or adapting from California's protected left turners can turn into any lane, not just the leftmost. They'll also get confused by children and pets, who aren't bound by social custom, and deer, who aren't bound by physics.
... no? That children, animals etc. acts differently everyone knows. But what about a self-driving car that looks the same as every other car?
Anyway, it was Waymos own findings when they started out. They got into more accidents, none of which where their own fault, than expected and realized that they had to make it behave more like a human to not confuse human drivers.
> what about a self-driving car that looks the same as every other car?
My Aussie friends, when they visit America, don't put up a sign in their window saying they're Aussie and will occasionally try to turn left at a red.
> They got into more accidents, none of which where their own fault, than expected and realized that they had to make it behave more like a human to not confuse human drivers
Sure. I'm still filing this in the nothingburger file. If anything, it screams that we have a lot of people on the road who should not be.
I think the other way to think about it is that Waymo is probably in the 99th percentile for not being distracted and 99th percentile for reaction time, always, just from a pure sensor and computation standpoint.
Water on the moon is limited and difficult to collect, it wouldn't make sense to use it for industrial purposes. It's a very challenging thermal environment (baking during the day, freezing at night). But perhaps worst of all, every month there's a 14-day period with no solar power. Overall seems worse than low-earth orbit.
+1 to this. It's also visually confusing, the gate looks like it's covering two cells.
Great game! Feature request: add a button that shows my submitted solution. I'd like to be able to compare it with the optimal solution (so it'd be nice if a single tap could toggle between my submission and the optimal).
LLMs are what they are, calling them "AGI" won't make them any more useful or exciting than they are, it's just going to devalue the term "AGI" which has revolutionary, disease-curing, humanity-saving connotations. What are you looking for us to say exactly?
1. We aren't even close to AGI and it's unclear that we'll ever get there, but it would change the course of humanity in a significant way if we ever do.
2. Wow we've reached AGI but now I'm realizing that AGI is lame, we need a new term for the humanity-saving sales pitch that we were promised!
I think getting out of the binary is good for the long run. We have something which is artificial, intelligent, and general in scope. We're there. Is it perfect? No. Is it even good? Sometimes! Do airplanes flap their wings? Also no, but they do a lot of stuff nonetheless.
That's where we disagree, I do not consider a system that isn't capable of learning, improving, or reasoning to be generally intelligent. My most basic criteria for "AGI" is a system that can absorb and integrate new knowledge through repetition and experience in real time, just like a human would.
Further, their statements, knowledge, and "beliefs" should be reasonably self-consistent. That's where I'm usually told that humans aren't self-consistent either, which is true! But if I ever met a human that was as inconsistent as LLMs usually are, I'd recommend that they get checked for brain damage.
Of course the value of LLMs isn't binary, they're useful tools in many ways, but the sales pitch was always AGI == human-like, and not AGI == human-sounding, and that's quite clearly not where we are right now.
Yeah, this is in 'flies like a plane, not like a bird' territory. But I think it's closer than you think.
The systems do learn and have improved rapidly over the last year. Humans have two learning modes - short-term in-context learning, and then longer-term learning that occurs with practice and across sleep cycles. In particular, humans tend to suck at new tasks until they've gotten in some practice and then slept on it (unless the new task is a minor deviation from a task they are already familiar with).
This is true for LLM's as well. They have some ability to adapt to the context of the current conversation, but don't perform model weight updates at this stage. Weight updates happen over a longer period, as pre-training and fine-tuning data are updated. That longer-phase training is where we get the integration of new knowledge through repetition.
In terms of reasoning, what we've got now is somewhere between a small child and a math prodigy, apparently, depending how much cash you're willing to burn on the results. But a small child is still a human.
I presume you're referring to the recent METR study. One aspect of the study population, which seems like an important causal factor in the results, is that they were working in large, mature codebases with specific standards for code style, which libraries to use, etc. LLMs are much better at producing "generic" results than matching a very specific and idiosyncratic set of requirements. The study involved the latter (specific) situation; helping people learn mainstream material seems more like the former (generic) situation.
(Qualifications: I was a reviewer on the METR study.)
For people who prefer reading to watching videos, I wrote a detailed account of my process for solving one of last year's IMO problems, along with thoughts on how this relates to AI:
My thinking here is that Waymo has logged hundreds of millions of miles at this point (and even more in sim), and there are a lot of nines in their safety record. So even the rare edge cases should have come up. You make a good point that the deep water scenario may be not only rare, but also under-represented in the training data. On the other hand, you'd think they would have thought to explicitly train for it; it's not that obscure of a possibility.