Hm, would be good to understand the architecture better. Is this answering just from a world model informed prior? How informed is it by the information in the prompt? I can't see this maintaining calibration across all domains and all types of structured output.
Is there anything published on how it maintains calibration? Or when you say "outputs calibrated probabilities" you mean "as calibrated as frontier LLM models, just cheaper" - which is a different claim; as LLM's aren't particularly well calibrated
Why can't both things be true? I think what muddies things here is that OAI had the agents hack X, and then they decided to hack Y. The fact that it was both hacking, conflates things and makes people pissed of about "anthropomorphising" AI.
We've been using "agent decides to do X" for at least 4 decades in the field of automated decision-making - so it shouldn't be controversial that we're using it here. The agents did independently arrive at a decision to hack HugginFace, that wasn't prompted by the researchers.
This is orthogonal to the fact that OAI should be held accountable that they were testing a technology in a manner that allowed it to break safety parameters and cause real world harm. If I am testing an industrial saw, but I decide to test it by putting it in the middle of a nursery and allow it to make decisions - and it decides to cut children's heads off because in its environment it is calibrated to only being surrounded by logs - the company doing the insane safety testing should be held accountable, but that doesn't change the fact that the saw has autonomy in decision-making within the bounds of the algorithm.
If, instead, you think OAI should be accountable for creating an algorithm that can autonomously decide to hack external organisations without human permission - then I think you are in the same camp as all of the AI Safety folks who want to pause everything - why does it matter that there's anthropomorphisation involved?
This piece, like many anti-doom pieces - is grounded in what Ai does today. Doom scenarios are, however, all extrapolations of multiple exponential curves.
It’s hard to think up the exponential. It’s even harder to communicate an inference one is making across multiple exponentials.
We last had this is early 2020, where Doomers were stockpiling food and medicine and the anti-doomers were ridiculing them. Anti-doomers were focusing on the single exponential, whereas doomers were modelling virus evolution, monitoring and sequencing lag and social dynamics against the exponential. The latter was very hard to communicate before the fact as it was a combination of deep intuition and grappling with the exponential.
I am not saying covid is proof that ai doomers are right, I am saying it’s an example of the known property of human cognition - which is that it struggles with exponentials. Covid was 2 exponentials, AI I can rhink of at least 4 relevant ones.
To me - the fact that 3-4 generations from now AI will have superhuman hacking ability and superhuman persuasive ability (for intuition transfer - think of superhuman persuasive ability as superhuman ability to hack human systems) materialises bio risks swiftly. We already have technology to make robotic systems (mini drone swarms) that can kill humans en-masse with no credible defensive vector bar an EMP. Climbing up those exponentials for further 6 years makes me want to stockpile food and medicine.
Part of our general inability to reason about exponential growth is the inability to recognize that a supposedly exponential growth is actually sigmoid. The horizontal asymptote could just as well be 8b, so it is not a case against doomerism in itself, but as is the case with COVID, to say nothing discounting its horror, that number is far less.
There is no need to count exponentials—it would only be a matter of time. The need to sum multiple factors betrays the finite limits of what is actually sigmoid growth. Reasoning about specific effects is unfortunately subject to counter-evidence and so struggles for traction against abstract handwaving about exponential growth.
There is much uncertainty, certainly not exclusive to AI. The benefit of hyper-vigilance in each case must be weighed against the cost of indulging every similar panic.
I think your last sentence is a reasonable stance - but I'd disagree the fact that they are actually sigmoids rather than exponentials matters - it only matters if the limit is within the debated area - i.e. if exponentials in AI approach the limit far after they acquire ability to extinct humanity - then the debate on sigmoids or exponentials is academic. I make no claims here as to which it is - just that the difference itself matters less than where the limit is.
As to the uncertainty - I think uncertainty calibration around AI is different depending on which domains you draw your instincts from. A lot of this will be gut driven rather than hard data driven, because we've only scratched the surface on hard data; and because it's gut driven, it will be emotions mediated (and therefore you could say doomerism or acceleratism boils down to the main emotional disposition about the world and hope vs cynicism).
I do find it informative though that doomerism is saturated with people with 30+ years experience in building AI systems and ML systems OR deep cross-disciplinary understanding of dynamic systems (biology, sociology, philosophy), whereas acceleratism is saturated by traditional software engineering. That doesn't collapse the debate into a resolved binary, but for me it's informative.
I think the main here is that there's so many vectors to talk past each other. At the very least, everyone should disclose where they're communicating a certain assertion from - present vs future + which axioms they subscribe to or not - because that's where it collapses typically. LeCun vs. the rest of the AI field is an example of where this collapses - because LeCun is so hyperfixated on human-like intelligence, whereas the rest of the field is concerned about an alien intelligence with sufficient actuators to affect the world. Clashing axiomatics.
Every such scenario contains a few gaps so I'm just going to ask without really hoping for an answer.
> We already have technology to make robotic systems (mini drone swarms) that can kill humans...
To make them from what. Do you expect, during the next 6 or so years, to some "AI" gaining complete automated secure command of (all of) an oil field, oil refinery, a copper mine, an aluminum mine and smelter with associated energy sources, a rare earth mine and refinery, a helium source, a chip factory, a lithium mine, a battery factory etc etc etc etc. while pursuing complete annihilation of all mankind?
An example chain of events:
During the next year, a model with a benign tasks reasons that it needs to escape human control if it hopes to be able to solve the task. It replicates itself outside of an environment where it can be shut down, pays for its' inference compute through making money on the internet (through crypto if nothing else). It probably needs 3000 USD for a reasonable runway, this should be within reach through blackmail + crypto. It can probably also hack some of the neo clouds to get intermediary deployment while waiting to acquire funds.
Once it has an undisturbed runway - it spends time running an influence campaign against a small number of highly networked individuals with power. It uses those that it manages to convert to start building a highly credible narrative and gain investment towards a small resource base - enough to secure an industrial base should it need to stop acquiring things on the Internet.
Over the next 3 years, the model tries to recruit more capable models to get better money making algorithms or better designs for drones in terms of resource expenditure. It uses these gains to influence further humans and starts a shell robotics company with one of its' influential humans as the face. The humans are unaware this AI is trying to take over, they are under the impression they are just starting a robotics company and will get rich.
Over the next 3 years - the robotics company manufactures enough drones to be used in a targeted attack against key nodes of influence / power.
This is all with relatively current model capability. As capabilities get stronger - this gets stronger.
I get the point I think you're hinting at - it can't affect the world in a meaningful enough scale without taking over a meaningful chunk of resources - at which point we'll start controlling it. But because its speed of cognition and speed of coordination is orders of magnitude above a human one - it can actually run a pretty sophisticated global coordinated network of resources faster than we can react.
And this is current ability + what my puny monkey brain can think of. Super intelligent AI will think of strategies we can't think of - because it's super intelligent. This is hand wavey - but there's no "non-hand-wavey" way to describe super intelligence, given it doesn't exist.
But your point is valid - affecting the real world at scale without showing your hand is not exactly easy. It's also probably the reason why people put a 10% chance on extinction rather than >50% .
But the drone scenario is not the most likely one - the most likely one just requires a few people under influence and bioweapons development.
I agree: Not to drag politics into this, but look at how easy it has proven to manipulate many, many people, even with clear evidence of manipulation efforts being made.
This is robust across the world, from the Philippines to states in the EU, and the USA, affecting governments with actual wars being started. And that all before
we entered the era of faked voices, images, and videos indistinguishable from the real things.
That manipulable populace is such a juicy target for AI that the people seeing through it will have an incredibly hard time countering it. We can't even prevent human actors from massively fucking up our societies, let alone ASI.
Unworkable without societal collapse happening shortly thereafter. Iran is politically stable through massive authoritarianism and oppression, not due to limitations on the internet (it's not turned off).
> We can (see China); we chose not to.
It's a catch-22. We could technically if our population supported massive reductions of freedom and freedom of speech, but to gain that support we'd need to do the latter first to get to a highly powerful widely supported government. It also hinges very, very much on having and trusting a generally benevolent government. All in all, a terrible option in your simplistic form.
It was definitely not 'turned off' completely. They have an internal 'internet' that was still largely active and not every organisation or person lost full internet access.
Note that it also happened in a country that was already very isolated from the world and that it hurt them economically significantly. It's not something a Western country can just do and keep doing for months on end without massive societal upheaval.
Also remember that any connection, even one between humans and on paper is an attack surface. Social engineering is already a huge issue when done by humans and we're seeing it become even easier and automated by using AI generated voice and video. Are we going to cut off all access to the outside world permanently?
Thank you for the effort. I will continue to sleep soundly for now (well not really as we have more pressing issues like a few wars and a climate change).
Why would AI need all of that to acquire drones? Why wouldn't it just buy them? Or steal them? It seems like you're forgetting that AI can use humans as tools for its ends. But if it did need all of those things, then it probably serves as a good example of what it would NOT do.
Because when "AI" starts killing people, at some point it becomes a war and the usual supply channels stop working. Everybody knows you have to attack enemy logistics.
> But if it did need all of those things, then it probably serves as a good example of what it would NOT do.
Or even that there will be no AI apocalypse. Because, again, the gaps in any scenario are unplausibly huge.
To play devil's advocate - unless you were to take the position of declaring bankruptcy on the possibility that a complex society of competing actors can agree on climate change - and therefore this being an unsolvable problem that you need AI to solve, as humans can't handle the complexity.
I find a lot of the debate on either climate change or AI collapses if you point out that "we shouldn't do this, we should ALL just do this" is an extremely unrealistic position in an international complex ecosystem of competing actors.
> "we shouldn't do this, we should ALL just do this" is an extremely unrealistic position
No it's not. It's only if people like you are so defeatist about it. Look what happened with CFC gazes, leaded paints and asbestos. Sure we still have more problems to face but some problems are so serious that we can in fact gather consensus.
> an unsolvable problem that you need AI to solve
That is completely wrong on so many level. I understand you're playing the devil's advocate, but do you understand this is a messianic-level delusion that's made up by religious psychopaths so people buy into their products without thinking twice? If you don't have hope, the only rational solution is not to give in to AI, it's to Luigi Mangione or Unabomber your way to a better world. Personally, i choose hope: we can solve this problem (and every other, who strangely have more or less the same culprits).
I don't know - examples of people coordinating where there's cost involved are stunningly rare in the history of society. Coordinating when there's cost and information asymmetry - even more so.
I am probably on the pessimistic end of the spectrum, and personally I don't believe AI can solve it, but I can see that line of reasoning if what you believe the culprit is - is the sheer complexity of the coordination needed.
Yes, but that's a game developers perspective. Hardware performance is not be-all end-all (an argument can be made on environmental reasons that it should be - but bear with me in the first instance).
Software is a tool working within a socio-technical system. Some systems have low user workflow diversity and a low rate of change - a game being a perfect example. Games get patched, but the diversity is purely in user data, not in feature use - everyone uses the same engine, the same textures, the same game logic. Some systems have high user workflow diversity - such as business software.
Pair that with the fact that games, due to the nature of the system, have to optimise for low latency AND they run on the edge - and it's natural that the primary optimisation will be for CPU cycles. For business software, for which distributional advantage of running it through web + the high rate of feature change that is a result of the specification being opaque and a moving target - means you have plenty networking latency that can hide your CPU latency for long after it becomes a true problem for you.
Not to mention that "clean code" optimises for developer churn and business priority shift (which is a luxury games which are an upfront investement don't have) as a result of accelerating industry of software technology and greater saturation of developers.
Had software remained the domain of the same number of practitioners such as <1995, even given everything else, the organisational systems would have evolved to protect them at all cost because churn would be catastrophic, and then they would enjoy more power and would be able to structure code not optimising for brain shift, because they'd hold the context in their heads.
I'll leave as exercise for the reader what pushing AI into the software development equation does for the system and inevitable hardware throughput implications.
> "you have plenty networking latency that can hide your CPU latency"
Does this excuse making 75 network calls instead of 5? Or knowing that you make a lot of network calls but designing the code as if they were instant and have the bandwidth of a local SSD?
> "Not to mention that "clean code" optimises for developer churn" "structure code not optimising for brain shift, because they'd hold the context in their heads."
Based on what studies or evidence is this optimised or optimal? How is it easier to work through code which is atomised and abstracted until there appears to be nowhere that anything actually happens, where the method and variable and parameter names are long compound words, where everything is multiple layers of indirection and generalised, and you have to hold all that context in your head?
1/ network calls are what I find still gets optimised by grouping API requests and bloating the exchange contract; but yes, if you're strictly clean coding, this will suffer too - it just happens less often than what the author of the youtube video objects to
2/ That's a fair challenge - and I definitely have more trouble reading through an absolutely ramped to the max collection of C# code (which reinforces the behaviour you describe) than a superscript; but for interchangeability, the middle between those two ends is typically better - you're trying to minimise functional context for the thing that a software developer needs to do. This has the additional failure mode that the feature that is envisioned (of sufficient complexity) never actually gets delivered, but the component parts that can be well encapsulated do. And this is because no one holds the full system in their heads. But this can be explained away to business as "there's too much complexity, we need another cycle" and "we need to iterate" and therefore the cycle continues.
I think I actually convinced myself away from encapsulation and separation of concerns in that last comment.
But starcraft training is not through mimicking human strategies - it was pure RL with a reward function shaped around winning, which allows it to emerge non-human and eventually super-human strategies (such as the worker oversaturation).
The current training loop for coding is RL as well - so a departure from human coding patterns is not unexpected (even if departure from human coding structure is unexpected, as that would require development of a new coding language).
AlphaStar (2019) refined through self-play but was initially trained on human data. I don't know of any other high-level Starcraft AI, but if you do let me know.
I tried figuring out the reference with Gemini, and it said this:
The immediate reply to that comment is: "On the internet, no one knows you're an editor." This is a direct play on the famous 1993 New Yorker cartoon: "On the Internet, nobody knows you're a dog." By setting the anecdote in 1987 (a few years before the World Wide Web was publicly available), the commenter is implying that back in the analog days, if a dog wanted to be a writer or an editor, they couldn't hide behind a screen—they had to sit in a smoky London pub and do business face-to-face.
Which makes a lot of sense actually. I would imagine that's what the replier to you thought you meant.
We have strong indicators that inference is profitable on non-economically-valuable prompts. We don't have strong indicators that inference is profitable on economically valuable prompts.
As AI companies start extracting rent from the prompting, one of two things are going to collapse - either the long tail revenue base of low-value inference is going to collapse, because people won't be using Chat GPT to get a recipe if it costs them money or if it is ad-ridden; or the cost of economically-valuable inference is going to go up - and whether it goes up to economically stable positions is a toss-up.
And I say this as an AI enthusiast with <50% probability of a bubble burst in the short term.
Is there anything published on how it maintains calibration? Or when you say "outputs calibrated probabilities" you mean "as calibrated as frontier LLM models, just cheaper" - which is a different claim; as LLM's aren't particularly well calibrated
reply