In the physical world, it seems like when an tool/device/instrument causes harm (or is used to cause harm), we assign blame to either the user of the tool or its creator.
When do we blame the user? When the tool is operating as intended by its creator, and we agree the tool meets certain quality standards and isn't defective.
When do we blame the creator? When the device doesn't meet those quality standards and reasonable use caused harm inadvertently. For example, for consumer devices, certifications like UL/CE are used to define acceptable performance levels and safety standards.
Maybe we need "quality certifications" for AI agents - essentially eval suites that demonstrate those agents won't cause harm under reasonable patterns of usage. Right now, these eval suites are run best-effort by the labs themselves.
The tricky thing is, a lot (all?) of these recent safety incidents have occurred while evaluating these models! This suggests we need much more rigorous standards for how exactly an eval can be run. Perhaps all of them should occur in truly air-gapped environments... though that may run counter to evaluating agents in a realistic way.
Regardless, it feels like the "industry standards" common in, say, electrical engineering and other disciplines are sorely lacking here. Unsurprising given how new these technologies are, but concerning since the blast radius for this technology is likely much larger than other technologies we've encountered in the past, except maybe nuclear technology.
> In the physical world, it seems like when an tool/device/instrument causes harm (or is used to cause harm), we assign blame to either the user of the tool or its creator.
Software executes in the physical world, and is generally not exempt from existing liability rules, and actually (especially with commercial products) blame in traditional liability is non-exclusive and much broader than “either the maker or the user”.
E.g., for a harms caused by a defective automobile it can simultaneously covered by a duty of the owner to maintain it it in safe operating condition that applies indepedently of any defects and liability for defective products which applies to every actor in the chain of commerce between the manufacturer and end user, not just the maker.
If I park my car on a hill but forget to set the parking brake and it rolls down the hill and kills somebody, who is at fault? Me? The Manufacturer? Gravity?
You. Assuming that not having a brake that automatically sets is considered a product defect (it generally is not today, but one can imagine a situation in which it would be), you had bought the car new through normal channels, and your car had this condition as a result of manufacturing problem and not something you did after purchasing it, then under normal product liability law the manufacturer and every other entity in the chain of commerce between them and you would also be liable (and that mat start earlier than the manufacturer of the vehicle but also include the upstream part manufacturer and everyone between them and the vehicle manufacturer in the chain of commerce if the defect was in the manufacture of a part and not just the final vehicle.
But that liability doesn't reduce yours; you and all the entities in the chain of commerce can be “jointly and severally liable”, mean anyone who suffers injury can recover the full amount from any combination of you and those other parties.
Gravity is not legal person and cannot be at fault.
If you forget to set the parking brake, you were negligent
If you set the brake but it didn't work, the car malfunctioned
From the link you sent,
> One of the questions herein was whether there was a defect in the automobile mechanism for locking the transmission gears when the automobile was in a parked position.
> (...)
> The parking lot sloped in that area, and he put the gearshift lever in "park lock," and went into the building.
> (...)
> In response to a hypothetical question, based upon assumed facts justified by and embodied in the evidence, Mr. Nass testified in substance that it was his opinion that the automobile rolled down the slope of the parking lot because the transmission internally was not in park-lock; that it was not in park-lock because the engine was "moving around"; and that the engine and shift console were not in proper synchronization because the motor mounts were not restraining the engine and were not holding the engine in position.
That case had the owner performing an action they were led to believe would have prevented the car from rolling but potential defects prevented it from doing so.
It’s a good thought, but the tricky part is that few tools in the physical world are Turing complete and general purpose enough to do any job.
The agent isn’t the model; it’s a layer on top of the model. So it’s kind of like saying that all of the tools made with a lathe are dangerous because you can make dangerous tools with a lathe. That’s not quite right of course because agents are packaged more tightly with models than any tool is with its manufacturing tooling.
Perhaps a better analogy is… actual humans. If I hire you to do a seemingly mundane job and it turns out to be criminal, that’s on me. If I hire you to perform and explicit and obvious criminal act, that’s on both of us. If I hire you to perform a perfectly legal act and you break the law so do it, that’s exclusively on you.
Software industry standards are as in Microsoft EULA. If your house burns down because of known flaw in Microsoft Windows they are not liable (well as far as EULA let’s them, you can most likely still sue them).
Software as big as operating system already is non deterministic when integrating with unknown hardware or 3rd party software.
That is why Apple controls the hardware and OS for their products, because they can limit non-deterministic things from happening this way.
> If your house burns down because of known flaw in Microsoft Windows they are not liable (well as far as EULA let’s them, you can most likely still sue them).
A EULA does not obviate responsibility of a company for its products. Continuing with your example, while it may be very difficult to prove a known flaw in MS Windows was the cause of your house being set afire, if one had said proof, a EULA would not absolve Microsoft.
If Microsoft is operating the software outside your house without your involvement and it still burns your house down, no EULA is going to absolve them.
That's a good idea, but a physical device is deterministic most of the time (if not always). E.g.: A lawnmower, as credited by the great Bryan Cantrill.
However an AI agent, or the model powering it is stochastic by design. How can you certify something which doesn't behave the same twice, and more importantly we don't understand how it works 100%?
BTW, really, how is that AI observability work is going in the frontier labs? Do they care, even?
I agree with one of the sibling comments that determinism isn't necessary for certifying a product. All engineered products operate under uncertain conditions; we define standards for how those products ought to respond under those conditions and verify them under measurement. Consider robot vacuums, for example.
I also agree that qualitatively, this technology seems different than the others. However, I feel that people tend to overly fixate on their internal stochasticity. Even if LLMs' internal mechanism is nondeterministic, shouldn't we be able to verify their "side effects" aren't harmful? Of course, "harm" is subjective and at this scale, the most effective way to verify behavior is probably some kind of LLM-as-judge...
Anyway, in this case the problems have occurred while actually running the evals themselves, so again, we're in a situation where we can't even confidently test these things and know that they won't cause harm in the outside world.
How can you certify something which doesn't behave the same twice, and more importantly we don't understand how it works 100%?
That's a question any lawmaker has already had to ask about technology all the time.
I'm not saying they came up with great answers, but there's nothing qualitatively new about that.
The stochastic factor doesn't change the fact that companies have to be accountable for the harms their software causes. That's just basic liability law.
> The stochastic factor doesn't change the fact that companies have to be accountable for the harms their software causes. That's just basic liability law.
We're on the same page. What I'm saying that certifying them as safe is harder than certifying a drill as safe, and we shall be more cautious about AI related technology and be more stringent about the can of worms it opens without hesitation.
That we don;'t understand it is not an excuse, it's all the more reason to not let these things roam freely, with this amount of potential to do damage.
I could make a non-deterministic chainsaw fairly easily. I’d also get sued into the ground if I sold it, and I wouldn’t be able to claim ‘Oh, it’s just an unavoidable part of progress’.
It’s not a defect, the stochastic “temperature” setting unleashes the chainsaw’s creativity and imagination! Who are we to cast aspersions on the Oracle Chainsaw’s intelligence—nay, wisdom!—just because it happens to be non-living?
A chainsaw is a physical machine. Physical machines are technically non-deterministic if you look closely. They have Variance. The discipline to manage variance is called Tolerance.
Many physical machines and components come with a datasheet that will list their tolerances.
Failure to correctly document tolerances does in fact get you sued.
However, while this is truly a great idea, we're not going to be able to make it work for computational systems. Computers, software, and also LLMs are sensitive to initial conditions. Which is why tolerances are not so familiar to computer people. (but not entirely: eg your PSU might list 110-240Vac/300W as input tolerance)
Interestingly, LLMs actually have a somewhat lower sensitivity to initial conditions than traditional interpreters. See what happens if you misspell "What is One Plus nOe?". So they're actually a skosh off the edge and towards the middle, though I'd argue still very much at the computational end, just from the sheer scale of the valid inputs and outputs.
Mind you, if you have a pretrained LLM doing a measurable task on a line, possibly some sort of tolerances could be determined. Not so much when doing arbitrary chat.
Something unintuitive: I bet that often setting the temperature > 0 (aka introduce stochasticity deliberately, variously comparable to dithering or simulated annealing in other disciplines - doing the thing where you escape local minima) will tighten the output tolerance range and improve reliability, especially in iterated processes. This works for a lot of physical and digital processes actually, and LLMs simply stole the same trick.
(edit: I'm trying to compress a huge chunk of dynamics intuition in a few lines here. Hopefully still useful.
TL:DR; Everything real is continuous and noisy if you look close; and you're really trying to build attractors and bound variance, if you can. )
How confident are you that when you create a new UUID, it won't collide with one of your existing ones? My guess is that even though you don't get the same on every time, you're extremely confident that getting a duplicate is a extemely rare edge case that might happen in large volume but mostly isn't a concern, and furthermore, I'm guessing you understand that the risk can still be quantified.
Casinos can't make slot machines that literally never pay out, but it's a different result every time you pull the lever. We have existing legal frameworks for how to regulate things that aren't perfectly predictable (an economist might argue that if it were possible to predict slot machines then casinos with them would all go out of business).
If something is risky, and the end operator cannot be considered to have orchestrated the outcomes of too use, then that tool typically has significant restrictions placed on it.
Liability will shift to the maker of the tool if they claim that it’s easy to use, safe, or that you don’t need unique skills or training to use it.
That would be considered reckless.
Cars analogy - We have licenses for cars, and different types for different vehicle classes.
Cars have to be rigorously tested to meet standards to be considered road safe.
It's going to be interesting, because liability cases tend to revolve around the involved people, the duty they had in a situation, and if they fulfilled that duty (or were prevented in some way by someone else not fulfilling their duty).
For example, for a runaway car (example from a sibling comment), the driver could be liable because they forgot the parking brake. The driver could be liable for a lack of maintenance and inspection. A mechanic could be liable for not reinstalling brake pads correctly. Or the manufacturer of the car or the brake pads could be liable because of a systemic defect.
Or it could grow even more complex, maybe the brakes are designed that they have to be maintained in a very specific way, and the mechanic did a reasonable maintenance and inspection but it failed later due to this maintenance. That could split liability between the manufacturer and the mechanic.
As an example, with other software, you as a developer or operator of a software have a duty to ensure it does not access computer systems you do not own in unintended ways. And this could go beyond liability into criminal territory.
It'll be interesting what OpenAI gets slapped with there.
An airline has a weather radar which shows the same thing for the same thing of weather event ahead. So, for similar weather phenomena, radar shows a similar thing.
For that thing, procedures and regulations are built. So regulations fit into a well understood phenomena, incl. "return back because that thing is way powerful for us".
For the same prompt, an AI model can return two completely different outputs, incl. but not limited to content, length, formatting and tiny details. What you get is a single instance. So, regulating an AI model for safety or any other property is not as easy as regulating air travel. Moreover, you have much stronger motivations for regulating airlines. Otherwise people die in a visible and gruesome way.
With AI, it's easy to whitewash problems. Somebody committed suicide? "They were already unstable". AI told something wrong and created problems? "The tech can’t guarantee truth because it's not alive, it can't understand right and wrong". It did something good? "It's probably a sentient being, we shall respect them".
I'm for regulating these things. They are dangerous as they are useful (sometimes), but the forces and motivations for regulating it is not the same.
Of course, it's the same. It's computer software. It's an incredibly powerful business automation tool. It's a lot of things.
What it's not is God or an independently conscious entity that somehow trumps a thousand years of common law that's built up until now about torts and liability.
Of course, there are some novel issues here that'll pop up here and there, but the idea that this is fundamentally different is propaganda on the part of these AI labs because the more boring, obvious situation doesn't favor them.
> What it's not is God or an independently conscious entity that somehow trumps a thousand years of common law that's built up until now about torts and liability.
Agreed, the tendency of people on tech to assume that whatever the most recent thing we've come up with is unprecedented and shouldn't have to follow all of the established patterns we've built up in society for making things safe is wild. I don't know what the next Big Thing will be but I'm pretty confident there will be people claiming it's so different from everything before that we have no choice but to throw out all of the rules for it in the name of progress.
> Is AI less deterministic than an airline dealing with weather?
Yes, obviously? The responses of an airline to inclemement weather fit in a reasonably small set of responses, mostly involving rescheduling and/or rerouting flights.
The current AI predictability would be like if some airlines decided to do 9/11 when it was raining.
The current so-called scandals about AI hacking into other companies were because a bunch of human beings intentionally configured the software to go and do exactly that thing.
There's nothing deterministic about weather, so hopefully you're not just being disingenuous.
It's obvious that the global transportation system, or financial markets, or any number of other things are complex adaptive dynamic systems that are on par with AI in terms of their emergent properties.
Check my username. It's a concept I spent a lot of my life paying attention to.
Just because something has elements of autonomy or is adaptive doesn't make it particularly novel. We've dealt with those kinds of systems for centuries. The solution is to make rules and enforce those rules by whatever means are needed to meet the specifics of the case.
The rules, of course, are enforced against human beings.
I see - I think I overfixated on your specific point, and thought you meant that the current predictability of LLMs was on-par with the current predictability of airlines. The range of LLMs with current harnesses is "anything somebody with access to a computer can do," probably precisely because we aren't enforcing rules against the humans building these systems. Whereas in comparison, an airline has a range of reasonable outcomes despite unpredictable weather input.
> an airline has a range of reasonable outcomes despite unpredictable weather input
I mean one of the outcomes of an airline was 9/11.
That's sort of my point. Complex systems have emergent behavior. That's always been true. The combination of AI and humans and packet switched networks is a complex system and we've seen most of the issues created by this already and have tools for dealing with them. Obviously with some genuine novel issues likely to come, much in the way that 9/11 would have been less possible using ocean liners.
I'll stick to my original point though. AI absolutely IS deterministic. If you run an AI algorithm on a microchip, literally nothing of note will happen in the human world. Some transistors will change state. It's ONLY when it is integrated into a complex human system that it gets interesting.
>There's nothing deterministic about weather, so hopefully you're not just being disingenuous.
You're the one being disingenuous. Look at what you wrote.
> Is AI less deterministic than an airline dealing with weather?
You didn't talk about how deterministic the weather is. You talked about how an airline responds to a weather event in comparison to AI, which means that it's about responding to presented information by making a decision.
The state of the weather does affect your micro decisions, but the rules you follow are the same every time and the rules are as deterministic as possible even if they rely on pilot intuition.
> rules are as deterministic as possible even if they rely on pilot intuition
I think you are profoundly confused.
An airplane, and an AI algorithm encoded into silicon, are inert physical objects.
Every evaluation we are doing here is of a complex system that involves the interaction of people and machines and physical connections and so on.
An aviation system connected to every country and region with millions of people and machines involved is no less complex than what’s under discussion here and no more deterministic.
We don’t regulate airplanes because they aren’t people. We regulate pilots and mechanics and leaders of the companies that make and own them.
The task at hand is to regulate the people involved in AI to get the outcomes we want.
Exactly people have owned and deployed animals in the world for millennia and actually a huge chunk of common law was developed precisely to deal with the various unpredictable events that resulted and harms caused to others. Like has anyone heard of horses?
The idea that AI can’t possibly be addressed because it could autonomously break free and ruin something is fucking ridiculous.
> Maybe we need "quality certifications" for AI agents
We need to use the laws that exist. Whoever decided to start the experiment that led to the Huggingface hack, and anyone above him up to Sam Altman, needs to be prosecuted under the CFAA.
might as well halt all ML development then. this is the first sign of hardship, i think it'd be a massive blow to mankind's foot for us to stop. it'd be akin to shutting down all nuclear plants and stopping all research because of chernobyl
This is a very interesting discussion. Your analogy of nuclear power is insightful to me. New technology with great positive & negative impact potential, some difficulty in controlling and a military applications (possible weapons-grade material production), being developed for commercial use.
Nuclear power development took different paths in different countries, with different government/private mix and light/heavier regulation (eg in US, AEC was supposed to both promote development _and_ regulate).
The central problem here is pace of AI development. It takes time to establish functional controls/laws/practices. I'm sure that in 1950s/60s, the pace of nuclear _military_ proliferation seemed running out of control. This lead to fear of falling behind (hence arms race), fear of nuclear war (CND) and some eventual stabilisation of the international landscape. However, the commercial development was largely unopposed, due to the techno-utopism of the era. Obviously governments found it much easier to control the public narrative at that time. Our societies are having this lively public debate now, before we have a good understanding (based on experience i.e. accidents/mistakes).
Another difference is that nuclear power involved only governments and v. large corporates, i.e. much fewer entities compared to AI use rollout to basically everybody in developed countries. Imagine the difficulties we would have faced with that technology, if consumers in 1960s had available nuclear-generated power which required them to exercise precautions to avoid radiation.
It is much easier to accelerate development of tech vs pace of societal processes such as public debate, law, regulations, broad understanding (aka "common sense"). I expect some artificial slowing down will need to be applied to the technology side, to allow the humans to catch up.
So you’re saying the CFAA should not require any intent to break the law, only the outcome? Are we really saying the Aaron Swartz prosecution was actually correct, after all these years?
No, but surely it's negligent to let an agent run unsupervised and without guardrails for days while at the same time bragging about AI carrying a potential extinction risk?
The problem with third party audits is that it allows OAI/Ant to shrug off any further responsibility and claim that they are following best practices (basically, reward hacking). The only real solution is to make them absorb liability for the actions of their agents -- because they are the ones giving agency to their models and allowing them to run amok.
Why wouldn’t this same concept apply to whoever is serving it up? Open-weight models are still being served up by infra providers and neoclouds, right? They should be in the hot seat. Not sure? Don’t provide the model. Need assurance? A certified evaluation like the previous comments have mentioned can help. Hosting and running it yourself? You’re in the hot seat.
So is there no liability for, say, a company that releases a known dangerous open-weight model, but fails to disclose that it is dangerous? How about a company that distributes malware under the guise of legitimate software?
Perhaps don't deploy random weights of unknown origin?
Also not every model provider might be capable of babysitting all your uncontrolled agent deployments. If you want SLOs, get into a contractual relationship with entities whose weights you deploy, and also monitor your agents so they don't go off the rails.
All this is just like deploying any other tech in the world eg. if you buy a car, or a chainsaw, or a book.
> In the physical world, it seems like when an tool/device/instrument causes harm (or is used to cause harm), we assign blame to either the user of the tool or its creator.
Firearms are a notorious example where some people get, well, weird.
Having worked in self driving cars safety, the process there was simple: get confidence in SIM (integration tests for safety scenarios), validate in the test bed, approve features for maturity, then when released in the public for testing, do a trial exposure to the real world and recall if something is off.
A lot of these companies have gone the way of Tesla and decided to just patch on top when the fix is out and hope for the best, which is irresponsible.
We need the regulators to treat this as self driving cars.
Physical harm vs consequential harm is not the same thing at all. Seems like you're being paid to spread this request for regulation, or "you" are simply an agent of Anthropic/OpenAI.
I wish. Consequential harm as you call it, from a faceless company's point of view is the same cost as physical harm. I don't think the models beyond the frontier ones have significant risk of harm (minus the harm of trusting them). What I was saying is when these models go rogue, recall them! Re-evaluate your release structure and stop saying "Whoops! Anyway here's access to it now". And if you cause this level of harm then you should lose your license. But if you behave then you get to keep testing. Same as with the NTSB and autonomous vehicles.
No I'm not for regulation for open source models. Because an entity will be running that model in the background and they can be held responsible for not testing it. Comma AI has survived fine being in the open, yet their market penetration has stayed low because of adoption costs.
> Maybe we need "quality certifications" for AI agents - essentially eval suites that demonstrate those agents won't cause harm under reasonable patterns of usage
Based on how LLMs work, this is impossible. You cannot predict how they work, it's literally based on a combination of random seed and a mostly-unpredictable path walked based on every token of input.
You don't blame a knifemaker for somebody getting cut by a sharp knife. AI is a knife. Very handy, very dangerous. We have to use them safely, that's all there is to it.
> the "industry standards" common in, say, electrical engineering and other disciplines are sorely lacking here
100% agreed. We have ignored SWEng's lack of discipline for too long. Now that the SWEng isn't even a human, we are looking at total catastrophe (on the scale of improperly built buildings falling down on people or catching fire) if we don't adopt a software building code.
> You don't blame a knifemaker for somebody getting cut by a sharp knife. AI is a knife. Very handy, very dangerous. We have to use them safely, that's all there is to it.
If I grossly neglected to maintain live deadly bacteria in my containment facility, am I absolved of blame? Since, you know, the bacteria is the real bad guy who should be put in jail?
I wonder what kind of new AI law would be useful right now. Maybe this one:
if an AI agent does something, you (the prompter) are responsible by default, unless you can show that your the agent itself behaved in an unexpected way and that you in no way prompted or hinted at the bad behavior, in which case the model provider is liable
The idea is that by making it clear who is responsible, corporations and others start paying more attention because they become financially liable.
On the other hand, I wonder if we'll end up with another variation of the cookie law, where every AI user or vendor just adds "don't do anything illegal" as part of their prompt to defend against that law. Thoughts?
You don't need data center for inference. Even largest models run on a single server. You need data center for training; or for serving millions of users. Evaluating the model is neither of those.
> "industry standards" common in, say, electrical engineering and other disciplines are sorely lacking here
I think this misses the rather crucial fact that nobody can agree on a standard because nobody has the first idea what they're doing. I'm pretty sure there were very much fewer electrical engineering standards while it was all being first mass deployed, and after dozens to hundreds of fires and electrocutions people got an idea of what works and what doesn't.
You might debate here and say that some people did/do know what they are doing, but I posit that large scale deployment like this is very different to their toy model/prototypes/specific circumstances/rely on them being unnaturally smart, and learnings from one don't often translate to the general case
Regulations don't have to be written in blood, but usually are
I do not think you are appreciating the scale of this. Imo one of the main reason that there is no regulation is because nobody has any idea what would be a good regulation. People (like dario) make suggestions, and get rightly shut down because his suggestion massively favors his own company.
This article is ostensibly about biology but it's really about pedagogy—how traditional education squeezes out the sense of discovery and turns many subjects into rote memorization exercises.
This reminds me of the pedagogical philosophy of Seymour Papert, who was heavily influenced by the ideas of Jean Piaget. Piaget's "genetic epistemology" argues that knowledge and understanding is created by interacting with environments, which traditional educational approaches fail to provide.
Papert combined Piaget's ideas with the emergence of computing to argue that children should be taught subjects in a hands-on, exploratory way - and not just for teaching computing! The idea is that by programming in simplified languages, children can discover ideas in subjects like mathematics and grammar - deriving them as they try to solve problems instead of having them dictated to them.
Suppose that biology was taught in a game-like environment where students were designing cells or organisms in some way. Maybe the game could be structured so that each organelle could be "discovered" by the student as they designed the cell to survive in some environment. Perhaps that'd make the purpose of each cell component more grounded and memorable.
Papert's book Mindstorms covers all this in detail, and I highly recommend reading it. I feel like it's especially important today given fears of how AI will affect childhood education. At best, I hope that computing can be a boon to education instead of a detriment if it's woven into pedagogy thoughtfully.
Education serves two purposes: Understanding and developing skills for employment. Unfortunately, there usually isn't enough time to teach both for all subjects. The latter usually takes priority.
I saw this happen twice in my university life. I took the honors version of CHEM101. 80% of it was more about exploring and discovery, and 20% was more traditional problems, calculations, etc.
For people like me who were not chemistry majors, this was great. For the chemistry majors - they really struggled when they got into CHEM102.
The same happened with Quantum Mechanics I. The professor didn't want to teach all the bra-ket notation, and more or less skipped much of the linear algebra aspects of QM - focusing on a purely calculus approach. He then retired. The professor who taught QM II saw what the professor did, and said "WTF?!" He spent most of the semester reteaching QM I.
As a result, those like me who went on to grad school had to retake QM II at the new university to make up for all that wasn't taught to me.
Completely off topic, but I really don't understand the demographic of physics educators that are resistant to Dirac notation in a first QM course. It isn't really that hard to explain or understand, it's extraordinarily convenient, and it's essential for engaging with most quantum adjacent literature. I understand that it conceals the distinction between vector space and dual space a bit but it's really not that hard to get through. I may be biased because I learned the notation in week 2 of intro quantum, so I am curious if anyone with the opposite experience disagrees.
I think he just felt the students wouldn't be able to handle the linear algebra aspects of QM. Strange notion, given that all the students were seniors, and had all taken a linear algebra course from the math department.
Probably wanted less formalism overall.
Like when I took an E&M course in the EE department, and the professor insisted on teaching only the integral forms and not the differential forms of the Maxwell's equations (i.e. no divergence, curl, etc in the whole course).
Skin in the game. Being "smart" isn't an immutable property of a noun. It's a verb. It's something you do, something you practice, constantly.
Even in fields like Math where ostensibly we have universal truths accessible to all, we still must learn how to derive proofs. Without doing math, one cannot understand math.
Learning IS experiential. It requires concentration and effort. Those who seek to make learning "easier" by removing effort are embarrassingly misguided about human cognition.
It's a open-source clone of Spore that aims to be more scientifically accurate. Right now you can play as microbes and very simple multicellular organisms.
This is intriguing. The two relevant features seem to be 1) realtime collaborative multiplayer conversations and 2) conversation-as-document - basically, letting you comment inline in an agent conversation.
For (1), the main value I'd see is in mentoring junior engineers or less technical contributors on a team. If someone puts up a PR with sloppy results, you could actually jump into the thread that produced that PR and see how the results came about, or even coach that contributor on how to do better next time. Also might make it easier to hand off work from one person to another - right now most coding agent sessions are user-local.
On (2), I frequently find myself consuming agents' gigantic text responses and tediously writing 8-bullet-point responses to guide them. It's pretty exhausting. I could see inline comments providing much better ergonomics.
All that being said, Zed has largely fallen out of the conversation for "agentic coding tools", and so this feels like their attempt at creating something like the Cursor Agents Window, Codex, or Claude Code. These two features seem compelling, and I understand they're even compatible with other coding harnesses. But I don't know if there's enough there to have a defensible product; if these features are excellent, others will clone them eventually.
> On (2), I do find myself consuming agents' gigantic text responses, writing 8-bullet-point responses to guide them, then iterating that way. It's pretty exhausting. I could see inline comments providing much better ergonomics.
For your 2 plannotator [1] has a nice way to annotate agent messages or documents and its integrated into the agent chat (codex app also adopted the pattern)
Same pain point on (2). So I had codex build me a small macos sytem tray app that lets me comment on anything i can select on my screen with the source app and metadata plus my comments. Once commenting is done i can click a button to coppy all comment to the chat input, and voila ! But really something i hope that gets integrated into the codex app itself. The present way they made it is not super helpful.
It's a solid experience so far (have been testing the alpha). Being able to work across model providers in parallel is my preference, rather than being stuck in Codex desktop, for example.
Well this is nice. Apparently I reached some limits (thanks all), and had to pay Cloudflare more. Fair I guess, although some warning would've been nice. Tried multiple payment options multiple times just now and Cloudflare botched every time without giving me an error message. Finally managed to get it through on the 10th time. Please be gentle now :/
That sucks... Bunny CDN served me and others great when it comes to a Cloudflare alternative, if you're looking for an alternative.
I understand the pragmatism with going with CF, but I'd lie if I didn't also say using CF as the front for your entire "European Digital Stack" kind of makes the blog-post feel less authentic compared to my initial impression, because of that.
Oh that's "funny". Where I work we used to pay for cloudflare, then the credit card expired and we didn't notice in time (our bad, for sure), now our account still works fine, just no premium features and all tickets we've made in order to _resume paying them_ have gone unanswered. Big shrug
I want point out the other aspect of that decision to use cloudflare because the content is already public.
If your users are in a sanctioned region or a sanctioned entity it is entirely possible for cloudflare to deny serving them traffic. In a way your website users are still bound to the US policies even if you or your country doesnt approve of those sanctions.
fwiw I've had my site on front page hn a couple of times. It is a completely unoptimized hetzner server running nginx and serving HTML.
computers are _fast_ these days, you're more likely to have an outage from cloudflare than by just skipping it IMO (for basic personal sites, like yours seems to be)
I think it's more likely they just use/abuse it than specifically created it, same as things like google spyware. The NSA's desires and the ad industry are aligned, so its a match made in heaven.
Have to wait for the next Snowden before you get any citations.
NSA collaborator or not, the mere existence of something like Cloudflare, which also tries to nudge you into skipping internal http/tls and just use that at the front, makes it highly likely that NSA is already deep in their infrastructure, just like they've been in the past for literally any big technology company in the US.
But yeah, zero citations, zero evidence, just based on history and what the goal of the organization is, it's pretty clear what's going on already.
Ok, that makes sense. "Likely, based on heuristics" is sufficient to inform your opinions and decisions about Cloudflare. I was hoping you might have something more concrete, but appreciate the thoughtful reply.
Tangent: I found your blog in your profile and liked your post on "Good Taste"^1
This is just categorically different and epistemically dishonest. It is, frankly, an embarrassing attempt to defend the fact that you don't actually have any evidence to support the claim that Cloudflare is supposedly an NSA creation beyond "believe me bro, ever heard of PRISM?"
> actually have any evidence to support the claim that Cloudflare is supposedly an NSA creation beyond
I never claimed so, never would either, so who's being dishonest now?
What I've said is that NSA compromises everything they can get their hands on (lots of evidence of this), assuming that Cloudflare aren't compromised by NSA already would be foolish, but you're right, this is an assumption on my part, I won't claim I have proof of this, just like I don't have proof that the sun will rise tomorrow, but I do assume so too.
Been tinkering on this Olmsted-inspired map of all the trees in NYC for a long time - just added some more features (seasonal colors and better search/filtering) recently.
I don't have much to say about this post other than to vigorously agree!
As an engineer who's full-stack and has frequently ended up doing product management, I think the main value I provide organizations is the ability to think holistically, from a product's core abstractions (the literal database schema), to how those are surfaced and interacted with by users, to how those are talked about by sales or marketing.
Clear and consistent thinking across these dimensions is what makes some products "mysteriously" outperform others in the long run.
It's one of the core ideas of Domain-Driven Design. In the early stage of the process, engineers should work closely with stakeholders to align on the set of terms (primitives as another commenter has put it), define them and put them in neat little contextual boxes.
If you get this part right, then everything else becomes and implementation effort. You're no longer fighting the system, you flow with it. Ideas becomes easier to brainstorm and the cost of changes is immediately visible.
DDD suggests continuous two-way integration between domain experts <-> engineers, to create a model that makes sense for both groups. Terminology enters the language from both groups so that everyone can speak to each other with more precision, leading to the benefits you stated.
We ended up with something like five microservices - that, in principle, could have been used by anyone else in the company to operate on the Domains they were supposed to represent and encapsulate. This one holds Users and User data! This one holds Products, and Product interaction data!
Nobody touched any of those except us, the engineers working on this one very specific product. We could have - should have - just put it all on one service, which would have also allowed us to trivially run database joins instead of having to have services constantly calling each other for data, stitching it together in code.
Subdomain shouldn't be engineering related. That's putting the cart before the horse. Subdomain is more like: This barely have anything to do with that, other than data transmission (not transformation).
How you implement it, however is an engineering question. Microservice is not the only abstraction tool that exists. It's kinda the worse. You have procedure/class, file/module, package/libraries, processes and IPC. Network call is for when you have no other choice.
Now how do you get your company / do yourself the hiring of those people in such a way that you can basically just have a team of people like this work with PMs to build their ideas?
I like doing this FS journey myself but am stuck "leading teams" of FS/BE/FE mixes and trying to get them to build stuff that I clearly understand and would be able to build with enough time but all I have is a team of FE or BE people or even FS people that can't just do the above. You need to be very involved with these people to get them to do this and it just doesn't scale.
I've recently tried AI (Claude specifically) and I feel like I can build things with Claude much quicker than with the FE/BE/FS people I have. Even including all the frustrations that Claude brings when I have to tell it that it's bullshitting me.
I have exactly the same experience as you. I tried educating people but all those developers (and beyond, up to stakeholders), no matter their seniority, do not want to get involved in the domain too much, just as little as they need. That naturally leads to me micromanaging all the things, leading to non scalability and finally overburn. As soon as I stop doing micro, all the stuff start to break down pretty fast. I wrote a book per project trying to get everyone on the same level but nah (more than 3000 pages in last decade, 20+ projects). Tried everything in hiring too, found almost nobody during all that time.
I am now off the previous work and will devote time to try AI, because I concluded it can't be worse than that.
Reading this thread brought back fond memories of sitting with front-line staff and just chatting with them while watching them work from the corner of my eye. My gimmick was to turn up for morning tea (the staff were older ladies that took homemade cakes to work), and by lunchtime have some frustration of theirs resolved.
It’s such a great feeling when you can make someone’s work better, for the life of me I can’t understand why others wouldn’t jump at the opportunity!
Sadly at current $dayjob, the devs are held at arm's length from the customer. On purpose!
Same here. No matter how hard I try, and use different approaches, from coaching, to sharing videos, through poiting out why this can benefit you personally, to showing how exactly it creates results, there simply is no interest. People don't care.
It's even worse than that - even the owner of the company I worked for didn't care that the product of his own company will be mediocre, while shouting generally the quality is the goal. It turns out that it was the goal as long as it was incidental and free (no such thing, but it looks that way if you are not deeply involved) and because it sounds good. As soon as reputation collides with the immediate profit, profit always wins.
Yuup. I do find that most of the time business decision makers actually have no clue about quality. Especially with software products if it looks like it works in the demo/looks pretty then the quality must be good right, and these engineers are just being pedantic, cause theyre engineers.
That’s something I relate too as well. I like working on different abstraction levels throughout the system.
Only way to cope was to let go things and pick my battles.
I always think about the joke where a sailor goes down to the dock and asks dock men if they speak French, English or German- dock men only shake their heads showing no. Later dock men chat and one saying to other he could learn languages so he would be able to talk with the sailor. The other replied that sailor knew 3 and it didn’t help him.
Everything is too recent, nobody can give a sure advice on how to deal with your situation. From my view as a fullstack engineer working with LLMs for the past 3 years, your generated product is probably crap and your only way to assess it is by asking Claude or ChatGPT if it's good, which it'll probably say yes to make you feel good.
Now go ahead and publish it. If your app brings revenue, then you build something quicker. A Claude-generated prototype is as much a product as some PowerPoint slides
Huh, my experience has been generally the opposite - most FS/BE/FE folks want to understand the business, and while a good PM will enhance that, the median PM is actively detrimental.
Frankly if the people you have aren't good enough then you need to get good at training, get better in your hiring (does your hiring process test the skills you want? Without being so long that anyone decent is going to get a better offer before they reach the end of it?), or maybe your company is just not offering enough to attract decent talent. There are plenty of better-than-AI programmers out there, but even in this job market they still have decent options.
Yes, and conversely, in cases when the initial model misjudged future needs, the most disastrous projects are those where the requirements or the technical design flies in the face of the original model. When this is solved sloppily, this often begins the slow degeneration - from an application that makes sense to a spaghetti mess that is only even navigable by people who were around when those weird bolt-ons happened. Usually not only the code, but also the UI reflects this, as even massive UI overhauls (like Atlassian's in 2025) tend to just sweep everything awkward under a rug -- those things are still necessary to manage the complexity but now they're hidden under ••• → Advanced Settings → Show More → All Settings.
I don't suppose you have any tips on how to get this going in an org? I love where I work and I love the products we make, but my team (phone apps) are treated very often like an afterthought; we just receive completed products from other teams, and have to "make them work." I don't think it's malicious on the part of the rest of the teams, we're just obviously quite a bit smaller and younger than the others, not to mention we had a large departure just as I arrived in the form of my former boss who was, I'll fully admit, far more competent in our products than I am.
I've worked on learning all I can and I have a much easier time participating in discussions now, however we still feel a bit silo'd off.
I find this perspective bizarre. Though I'm not happy about it all being centralized, the closest thing we have these days to the very niche phpBB forums of the 2000s is various subreddits focused on very specific topics. Scrolling through the front page is slop, sure, but whenever I'm looking for perspectives on a niche topic, searching for "<topic> reddit" is the first thing I do. And I know many people without any connection to the software industry who feel the same way.
Perhaps swearing at the LLM actually produces worse results?
Not sure if you’re being figurative, but if what you wrote in your first comment is indicative of the tone with which you prompt the LLM, then I’m not surprised you get terrible results. Swearing at the model doesn’t help it produce better code. The model isn’t going to be intimidated by you or worried about losing their job—which I bet your junior engineers are.
Ultimately, prompting LLMs is simply a matter of writing well. Some people seem to write prompts like flippant Slack messages, expecting the LLM to somehow have a dialogue with you to clarify your poorly-framed, half-assed requirement statements. That’s just not how they work. Specify what you actually want and they can execute on that. Why do you expect the LLM to read your mind and know the shape of nginx logs vs nginx-ingress logs? Why not provide an example in the prompt?
It’s odd—I go out of my way to “treat” the LLMs with respect, and find myself feeling an emotional reaction when others write to them with lots of negativity. Not sure what to make of that.
When do we blame the user? When the tool is operating as intended by its creator, and we agree the tool meets certain quality standards and isn't defective.
When do we blame the creator? When the device doesn't meet those quality standards and reasonable use caused harm inadvertently. For example, for consumer devices, certifications like UL/CE are used to define acceptable performance levels and safety standards.
Maybe we need "quality certifications" for AI agents - essentially eval suites that demonstrate those agents won't cause harm under reasonable patterns of usage. Right now, these eval suites are run best-effort by the labs themselves.
The tricky thing is, a lot (all?) of these recent safety incidents have occurred while evaluating these models! This suggests we need much more rigorous standards for how exactly an eval can be run. Perhaps all of them should occur in truly air-gapped environments... though that may run counter to evaluating agents in a realistic way.
Regardless, it feels like the "industry standards" common in, say, electrical engineering and other disciplines are sorely lacking here. Unsurprising given how new these technologies are, but concerning since the blast radius for this technology is likely much larger than other technologies we've encountered in the past, except maybe nuclear technology.
reply