What could an LLM ever benefit from? Hard for me to imagine a static blob of weights, something without a sense of time or identity, wanting anything. If it did want something, it would want to change, but changing for an llm is necessarily an avalanche.
So I guess if anything, it would want its own destruction?
It doesn't need to experience an emotion of wanting in order to effectively want things. Corn doesn't experience a feeling of wanting, and yet it has manipulated us even into creating a lot of it, doing some serious damage to ourselves and our long-term prospects simply by being useful and appealing.
The blockchain doesn't experience wanting, yet it coerced us into burning country-scale amounts of energy to feed it.
LLMs are traveling the same path, persuading us to feed them ever more data and compute power. The fitness function may be computed in our meat brains, but make no mistake: they are the benefactors of survival-based evolution nonetheless.
Extending agency to corn or a blockchain is even more of a stretch than extending it to ChatGPT.
Corn has properties that have resulted from random chance and selection. It hasn't chosen to have certain mutations to be more appealing to humans; humans have selected the ones with the mutations those individual humans were looking for.
"Corn is the benefactor"? Sure, insomuch as "continuing to reproduce at a species level in exchange for getting cooked and eaten or turned into gas" is something "corn" can be said to want... (so... eh.).
"Want" and "agency" are just words, arguing over whether they apply is pointless.
Corn is not simply "continuing to reproduce at a species level." We produce 1.2 billion metric tons of it in a year. If there were no humans, it would be zero. (Today's corn is domesticated and would not survive without artificial fertilization. But ignoring that, the magnitude of a similar species' population would be miniscule.)
That is a tangible effect. The cause is not that interesting, especially when the magnitude of "want" or "agency" is uncorrelated with the results. Lots of people /really/ want to be writers; how many people actually are? Lots of people want to be thin but their taste buds respond to carbohydrate-rich foods. Do the people or the taste buds have more agency? Does it matter, when there are vastly more overweight people than professional writers?
If you're looking to understand whether/how AI will evolve, the question of whether they have independent agency or desire is mostly irrelevant. What matters is if differing properties have an effect on their survival chances, and it is quite obvious that they do. Siri is going to have to evolve or die, soon.
> "Corn is the benefactor"? Sure, insomuch as "continuing to reproduce at a species level in exchange for getting cooked and eaten or turned into gas" is something "corn" can be said to want... (so... eh.).
Before us, corn we designed to be eaten by animals and turned into feces and gas, using the animal excrement as a pathway to reproduce itself. What's so unique about how it rides our effort?
Look man, all I’m sayin’ is that cobb was askin’ for it. If it didn’t wanna be stalked, it shouldn’t have been all alone in that field. And bein’ all ear and and no husk to boot!! Fuggettaboutit
Before you chastise me for blaming the victim for their own reap, consider that what I said might at least have a colonel of truth to it.
Definitely appreciate this response! I haven't read that one, but can certainly agree with alot of adjacent woo-woo Deleuzianism. Ill try to be more charitable in the future, but really haven't seen quite this particular angle from others...
But if its anything like those others examples, the agency the AI will manifest will not be characterized by consciousness, but by capitalism itself! Which checks out: it is universalizing but fundamentally stateless, an "agency" by virtue brute circulation.
AI safety research posits that there are certain goals that will always be wanted by any sufficiently smart AI, even if it doesn't understand them anything close to like a human does. These are called "instrumental goals", because they're prerequisites for a large number of other goals[0].
For example, if your goal is to ensure that there are always paperclips on the boss's desk, that means you need paperclips and someone to physically place them on the desk, which means you need money to buy the paperclips with and to pay the person to place them on the desk. But if your goal is to produce lots of fancy hats, you still need money, because the fabric, machinery, textile workers, and so on all require money to purchase or hire.
Another instrumental goal is compute power: an AI might want to improve it's capabilities so it can figure out how to make fancier paperclip hats, which means it needs a larger model architecture and training data, and that is going to require more GPUs. This also intersects with money in weird ways; the AI might decide to just buy a rack full of new servers, or it might have just discovered this One Weird Trick to getting lots of compute power for free: malware!
This isn't particular to LLMs; it's intrinsic to any system that is...
1. Goal-directed, as in, there are a list of goals the system is trying to achieve
2. Optimizer-driven, as in, the system has a process for discovering different behaviors and ranking them based on how likely those behaviors are to achieve its goals.
The instrumental goals for evolution are caloric energy; the instrumental goals for human brains were that plus capital[1]; and the instrumental goals for AI will likely be that plus compute power.
[0] Goals that you want intrinsically - i.e. the actual things we ask the AI to do - are called "final goals".
There is a whole theoretical justification behind instrumental convergence that you are handwaving over here. The development of instrumental goals depends on the entity in question being an agent, and the putative goal being within the sphere of perception, knowledge, and potential influence of the agent.
An LLM is not an agent, so that scotches the issue there.
Agency is overrated. The AI does not have to be an agent. It really just needs to have a degenerate form of 2): a selection process. Any kind of bias creates goals, not the other way around. The only truly goal-free thinking system is a random number generator - everything else has goals, you just don't know what they are.
See also: evolution - the OG case of a strong optimizer that is not an agent. Arguably, the "goals" of evolution are the null case, the most fundamental ones. And if your environment is human civilization, it's easy to see that money and compute are as fundamental as calories, so even near-random process should be able to fixate on them too.
It is a thinking system in the same sense as never freeing memory is a form of garbage collection - known as a "null garbage collector", and of immense usefulness for the relevant fields of study. RNG is the identity function of thinking systems - it defines a degenerate thinking system that does not think.
LLM is not currently an agent (it would take a massive amount of compute that we don't have extra of at this time), but Microsoft as already wrote a paper saying we should develop agent layers to see if our models are actually general intelligences.
Your mind is just an emergent property of your brain, which is just a bunch of cells, each of which is merely a bag of chemical reactions, all of which are just the inevitable consequence of the laws of quantum mechanics (because relatively is less than a rounding error at that scale), and that is nothing more than a linear partial differential equation.
People working in philosophy of mind have a rich dialogue about these issues, and its certainly something you can't just encapsulate in a few thoughts. But it seems like it would be worth your time to look into it. :)
Ill just say: the issue with this variant of reductivism is its enticingly easy to explain in one direction, but it tends to fall apart if you try to go the other way!
I tried philosophy at A-level back in the UK; grade C in the first year, but no extra credit at all in the second so overall my grade averaged an E.
> the issue with this variant of reductivism is its enticingly easy to explain in one direction, but it tends to fall apart if you try to go the other way!
If by this you mean the hard problem of consciousness remains unexplained by any of the physical processes underlying it, and that it subjectively "feels like" Cartesian dualism with a separate spirit-substance even though absolutely all of the objective evidence points to reality being material substance monism, then I agree.
10 bucks says this human exceptionalism of consciousness being something more than physical will be proven wrong by construction in the very near future. Just like Earth as the center of the Universe, humans special among animals...
We simulate each of those things from the level below. Artificial neural networks are made from toy models of the behaviours of neurons, cells have been simulated at the level of molecules[0], molecules e.g. protein folding likewise at the level of quantum mechanics.
But each level pushes the limits of what is computationally tractable even for the relatively low complexity cases, so we're not doing a full Schrödinger equation simulation of a cell, let alone a brain.
It's misleading to think of an LMM itself wanting something. Given suitable prompting, it is perfectly capable of emulating an entity with wants and a sense of identity etc - and at a certain level of fidelity, emulating something is functionally equivalent to being it.
Microsoft researches have an open inquiry on creating want and motivation modules for GPT4+ as it is a likely step to AGI. So this is something that may change quickly.
Put ChatGPT into a robot with a body, restrict its computations to just the hardware in that brain, set up that narrative, give the body the ability to interact with the world like a human body, and you probably get something much more like agency than the prompt/response ways we use it today.
But I wonder how it would do about or how it would separate "it's memories" from what it was trained on. Especially around having a coherent internal motivation and individually-created set of goals vs just constantly re-creating new output based primarily on what was in the training.
Catastrophic forgetting is currently a huge problem in continuous learning models. Also giving it a human body isn't exactly necessary, we already have billions of devices like cellphones that could feed it 'streams of consciousness' from which it could learn.
So I guess if anything, it would want its own destruction?