100 images out of 350,000 that they looked at were memorized.
This seems to mostly happen when an image appears frequently (more than 100 times) in the training data, and/or the dataset is small relative to the model.
Oh come on. I'm excited about new technologies and I think that image generation can be a net positive for the society, but can we not do that? First, we had people confidently asserting that stuff of this sort absolutely can't happen. Now, we're moving the goalposts to "well, not a legitimate criticism because it doesn't happen often".
The point is that basically all Stable Diffusion / DALL-E / MidJourney output is some shade of this; the only new data is that contrary to prior assertions, in some cases, it goes all the way to a verbatim copy.
I think there are some defensible stances one can take. One is to reject the idea of intellectual property. Another is to advocate for some specific legal or technical bar that the models would have to pass for it to qualify as "not stealing". Yet another is to argue it's a morally-agnostic technology like VHS or a photocopier, and the burden of using it in a socially acceptable way rests with the user.
> Oh come on. I'm excited about new technologies and I think that image generation can be a net positive for the society, but can we not do that?
What, summarize the submission? This is straight quoted from the link.
> The point is that basically all Stable Diffusion / DALL-E / MidJourney output is some shade of this
Yes, and that point is mistaken, or so generic as to be worthless. The network memorizes art for the same reason humans memorize art: because there's some art pieces we see so often that we can recall them easily.
Ask an artist to duplicate Starry Night or Scream from memory, you'll probably get at least a passable imitation. The more capable the artist, the more faithful it will be.
We know that SD can be made to plagiarize, given repeated training on a specific image. (This is just to say that a neural network can learn to regurgitate a sample, a capability that was not ever in question.) This is a far cry from the assertions that its art is generally plagiarized.
A human artist is, however, not a machine built and owned by a corporation, who will draw anything you tell them to.
A human artist has been trained in the ethics and laws of their craft along with the skills required to make images.
A human artist, asked to clone Starry Night, will ask you what you are doing, and knows where the lines are between "a tribute", "plagiarism", and "outright forgery".
A human artist, asked to do work in the style of another artist, will have a certain respect for the other artist's ownership of their style. This is not a thing that is at all protected by intellectual property law but it is still a thing artists are trained to respect. There are exceptions - drawing just like your boss may be your job, drawing just like a living artist for a couple pieces is a useful way to break down their style and take a few parts of it to influence later work without going over the "style swipe" line, building your own work on the obvious foundation of an influential, dead artist's style is fine - but there are lines professionals will be very reluctant to cross.
For a relatively recent example of what happens if you break these unwritten laws, check out what happened when the American cartoonist Steve Giffen started doing a wholesale style swipe from the Argentinean cartoonist Muñoz: https://en.wikipedia.org/wiki/Keith_Giffen#Controversy
Neural networks know none of these unwritten rules. Neither do the people who are training them. Feed it a bunch of work generated by a living artist and start making a profit off of that? Sure, no problem! Bonus points if your response to them getting pissed off about this is to call them a luddite who is resisting the inevitable, and should throw away a lifetime of passionate training and go get a new job.
Well we’re in luck! The human artist is the person who is coming up with the prompt and then deciding what to do with the resulting image based on their own ethics. Stable Diffusion is just a tool for artists, not a “computer artist”.
Funny enough, this notion in terms of liability pairs very well with our legal system!
Even funnier, this notion in terms of authorship pairs very well with modern and contemporary art theory!
And here’s some very relevant precedent in both a legal and artistic sense:
Richard Prince is a very rich jerkass who can afford to hire some very high-priced lawyers to argue that his work is right over the line of "sufficiently transformative" in the eyes of the law. He's well over the lines of the unwritten rules of professional artists that I'm talking about.
The corporations engaging in massive abuse of the grey areas of fair use to build these systems are, functionally, also very rich jerkasses who can afford to hire some very high-priced lawyers to make similar arguments.
The human giving the prompt is shopping, not creating. The only way a computer can compute an image is by copying the vectors of imagery made by humans and then labeled and plotted and fed to said computer. Humans paint, draw, do physical things to make art. Computers are fed plotline statistics that they can pull up to fullfil a shopping list. Humans generated the imagery, uploaded the imagery,labeled the imagery and wrote the code to manipulate the imagery. Now, I'm not a big proponent of copyright. If someone can paint just like Rembrant, then I applaud their skill. At the end of the day, 'AI' is not 'learning' ,it is replicating input data by filtering a large amount of data sets with appropriate labels ( affixed by humans) and using a statistical algorithm to approximate somebody's shopping list of art they wish they had the skill to create. I'll take the Rembrant forgery, thank you.
Sure, I guess we can make the argument that someone who makes music out of samples or loops is just shopping as well. How about programming a drum machine? Aren't they just shopping for snare hits and quantization patterns?
So fine, whatever, call it all shopping! Dr. Dre is just out there shopping. I don't think he minds if that's what you call it.
So here's the thing. Using a drum machine and sampling old funk songs from the 70s doesn't mean you end up with The Chronic. It's more likely that the average person ends up making something pretty mediocre. Hey, it would sure have sounded really impressive if it had been released in the 1940s, but with mass produced commercial music hardware a funky drum beat is just not that special any more.
The same applies to any kind of commodity tool. It's what the artist does with the tool in the context of a world filled with an audience and other artists.
Ok, time for my opinions! I think that DeviantArt style digital paintings are total trash. I don't care about the skill in rendering cliches hanging off of comically large breasts. Oh, it took a long time? I'm sure it would take a long time to dig a 10 foot deep hole in your backyard and then fill it up again as well, something I'd much rather experience as art than some video game hallucination... but you know what? Just because I don't like it doesn't mean that it isn't art, that it wasn't made by a "real" artist or that there isn't some audience that appreciates it (even if they're only two more YouTube videos away from becoming a full blown incels and driving a trucks through high school track meets).
Richard Prince speaks to me about authorship and what it means to make art in a world completely saturated with commercial imagery and part of that meaning comes from the fact that he had one of his assistants draw on top of someone else's photograph.
Listen, I have my issues with the world of fine art, the market manipulation, collusion, and the general fact that the art is primarily being made for the people who could afford it and not like, my neighbor. Regardless, I've found a lot of intellectual stimulation and new ways of appreciating aesthetic beauty through the works of 20th century modernists and postmodernists. Deeper meaning as opposed to a big shiny sword and a short skirt.
My favorite form of visual art is the watercolor, done quickly and out in public, capturing what the artist sees in the moment. It's the visual equivalent of the folk song played on an acoustic guitar. I like when the artist is a friend. I don't care if it isn't Rembrandt. I care that it moves me.
Stable Diffusion can easily render cliches hanging off of comically large breasts. In fact, I think that's what 90% of SD GPU cycles are currently working on. So to me Stable Diffusion is good at the part of the art that I'm not really that interested in. I'm interested in why the person chose the image that they did given these tools. That's where the meaning comes from! I mean, these tools run the same problems as drum machines and samplers... pretty soon their mediocre outputs become trite and unexpressive. I would imagine that artists that use SD do so in ways and using techniques that are not just the click of a button.
> The point is that basically all Stable Diffusion / DALL-E / MidJourney output is some shade of this; the only new data is that contrary to prior assertions, in some cases, it goes all the way to a verbatim copy.
This is absolutely not the point of the linked paper. It may be something you believe but you’re on the hook for providing evidence for it, this paper does not.
> and the burden of using it in a socially acceptable way rests with the user.
Or, you know, legislation. I'm kind of sick of everything being offloaded as a responsibility of the end-user as an excuse to externalize costs.
Plus, in this case it's not even like VHS or a photocopier, it's more like the printing press or the Jacquard loom: those with capital to invest in it benefit the most, at the expense of individuals being exploited.
Tools that are a burden to use, like tools that produce too many infringing works, are not going to sell as well as those that are not a burden to use.
This means that if someone makes a tool like this that also alerts the user of likely infringement it would perform better in a corporate, risk-averse marketplace.
> Yet another is to argue it's a morally-agnostic technology like VHS or a photocopier, and the burden of using it in a socially acceptable way rests with the user.
There is a lot of case law that supports this interpretation, Sony v Universal being the most important as it introduced the notion of “commercially significant non-infringing uses”, of which there will be many by the time this hits trial and the appeals process.
Lawyers for these companies know this and the faster that can get people building and buying tools that are clearly non-infringing the more likely that these models are seen as fair-use.
However, if they want to keep the lawyers at their customer’s firms happy they will really need to come up with a way to show users if a work is likely to infringe on an existing work.
Isn't the solution just to check for existing images - train an AI that can tell if an image came from your dataset, and don't use the generated picture if it matches?
Generating copyright images isn't a problem. Using them to make money is.
Seems like it would be very simple to prevent this from happening by just adding a perceptual hash check[1] for collisions vs the training set before emitting an image. I'm sure someone would be smart enough to make a "perceptual cuckoo hash" type thing so it would be very fast at the expense of sometimes erring on the side of caution and not emitting an image that actually isn't the same as the training data.
Yeah it seems so obvious I can't help but wonder if there's something about it that doesn't work in practice.
Like if you set it to a threshold where it's not routinely catching false positives, it doesn't catch the original images either because they're still too subtly different or something.
Maybe they really did just not bother though. Heck, maybe they want to test it in court or something.
First, it probably wouldn't work. Stable Diffusion is going to be a lot better at telling whether two images are the same than a perceptual hash check. E.g. I bet if you removed all pictures of the Mona Lisa from the training data, it could still produce a pixel perfect copy of it, just from the many times it appeared in the background of pictures, or under weird adjustments or lighting that fooled the perceptual hash.
Second, I would guess they wanted common images to appear in the training data more than once. It should be trained on the Mona Lisa more than someone's snapchat of their dinner. Common images are more salient.
Reminds me of asking Chat-GPT for an obfuscated C contest entry. It produced one verbatim, though it couldn't explain what it did (it looked like syntactic nonsense, but it printed the Twelve Days of Christmas when run). I can only imagine it saw that inscrutable sequence of characters enough times that it memorized it.
> Stable Diffusion is going to be a lot better at telling whether two images are the same than a perceptual hash check.
Why would it be? Stable diffusion is a text-to-image model it's not at all focussed on determining whether images are the same.
Secondly, I'm not proposing deduplication of the training set (I know some sibling comments have proposed this). I'm proposing a perceptual hash or similar check on the way out so that if a "generated image" is too similar to an image in the training set it gets dropped rather than returned.
The reason it’s not implemented yet is because none of these products are being monetized yet, so there’s no incentive.
Google can legally return a copyrighted image in Image search (it’s not selling the image, it’s selling ads against search results) and this would probably fall under the same protection.
Now if stable diffusion was sold as SaaS in the way Dall-E is…
Correlations in the math domain get weird. It wouldn't surprise me in the slightest that those slight distortions you see in the "memorized" image versus the original ones, while small to humans, turn out to be staggeringly large to your perceptual hash. See the things like the little stickers you can stick to stop signs to make some neural nets decide they are people and such.
And it could go the other way too; it could be that the perceptual hashes are even "better" than humans at seeing past those distortions.
My point is that this all gets more complicated than you may think when you're trying to apply a hash designed for real images against the output of an algorithm.
And even if the hash worked perfectly, the false positive and false negative landscape would still be very likely to contain very surprising things.
Maybe this would make sense for a commercial service but I don’t see why would be shipped as part of the model itself. Maybe the training of the model can be tweaked to accommodate a new step to prevent reproductions but I don’t know enough to understand the repercussions on performance from doing that.
The paper itself actually goes to show that this is insufficient (but should still be done, they went from regenerating 1 in 819 to 1 in 1063 images in a small corpus litmus test), and that diffusion is weaker to image extraction attacks than GANs (and seems to be weaker the better it gets).
As to why? Lack of care and vigilance, most likely. If they're willing to spend millions of GPU hours training some of these big models, then this wouldn't be a huge cost, and has near linear complexity over the dataset that parallelises trivially. Sadly, vigilance is a finite resource, and that kind of data cleaning is often dismissed in favour of doubling down on the training and assumming it'll be big enough to handle it (however, as has been shown, that's not the case).
(Edit: thanks FeepingCreature. The deduplication test used for the numbers above was a separate test in the paper for just this, not indicative of their extraction attack in general, with a small corpus and compared a diffusion model trained on both. So I liken it to a litmus test for the efficacy of deduplication.)
SD2's dataset was deduplicated prior to training. That's why the paper is about SD1, which was a prototype model completed before Stability even had VC raises.
The open source community work in ML tends to be terrible at the science and rigor parts of it. You see tons of lazy “we recreated an open version of X” that fails to be remotely as well done as the peer reviewed version, even from the larger community groups.
If by "computationally heavy" you're talking about the financial overhead of hiring another layer of human filters to filter duplicates out of the dataset.
At the sort of scales of these datasets, there is no way to filter by hand.
But there are lots of ways to identify near identical images algorithmically. Typically the process is to download each image, run it through a neural net to make an image embedding vector (a list of a few hundred floats). Save all those in a database. Then, for each image, if it is too close in 'embed space' to another in the database, then it is a duplicate, and should be removed.
This algorithm might catch 'duplicates' that it shouldn't, like multiple people taking photos of the eiffel tower from the same public viewpoint.
It might miss real duplicates such as an image failing to match with a collage containing the same image.
But it's still better than not removing duplicates at all...
Deduplication, to the best of my knowledge, requires every image be compared to every other image. This is necessarily O(n^2) on n images.
IIRC the training set is 2.3 billion images, if so that's 0.5 * 5.29e18 comparisons[0], which, if done by humans, would require employing literally all humans for approximately a year even if we compared 12 images per second 24/7.
This has to be done computationally, not by humans.
Perceptual hashes are reasonably robust to small perturbations. Scaling and converting shouldn’t be a problem, though applying filters that change the colors or blur the lines might.
If it’s been changed enough that it hashes to a different value, then it might be reasonable to treat it as a different image. At some point a human is also going to say “that’s not the same.” You can always change your hashing algorithm if you find it’s missing too many dupes.
Regardless, for the domain we’re talking about (deduping training data), a few false negatives should be acceptable.
Why not? Many meme variants end up in the same bucket, all colliding. They are as duplicates as random rare images.
A large bucket can also be inspected by humans or cut up by applying more perceptual hash function and decreasing tolerance, but it would be counterproductive cheating in this case.
You can lower the bounds significantly. Images do not require comparison to all others, and the presented approach to detecting and deduplicating images in the paper is easily adapted to be near linear (trading for storage or cleverness).
Put simply, do you expect google image search to compare your image to every other possible image? No, they're going to embed it to a vector (512d in the paper) and only compare to probable matches; in the paper they start by brute forcing pairwise comparison of the vectors for the dataset, and then use clique finding to go faster when checking their generated images.
You can roughly classify the images first (we already have very good AI model for this) and use humans as an additional layer. If two images don't get the same top-3 labels from the classifier, the chance that they're duplicates is neglectable.
Seems to be no different than how people memorize "facts".
I'm watching my daughter learn shapes right now. When I see a square, I call it a square and look for a matching "square" piece. It seems when my daughter sees a square, see is still looking at it's properties, labeling individual attributes (pointy thing here), then trying to find the same attributes on the corresponding piece. Eventually, my daughter will see a square enough that the concept will be re-enforced in her mind as a square.
I wonder if a similar behavior is happening here. This specific image was presented enough times that it essentially "burned into" the AI. Rather than having to attempt to generate this image from scratch, the AI will simply pull out the canonical reference.
A hundred copyright law violations baked into a photocopier would be a lot, right? Shouldn’t we guarantee the number is zero, and not dismiss the problem even if it seems like it should be uncommon? Since the network trainers can’t control what images users produce with the network once it’s public, there’s no ability to predict how often it reproduces a copyrighted image.
> This seems to mostly happen
Sounds like duplicate images caused a minority ~25% of the cases? (My calculation: 1-819/1063) If so, then the network is memorizing single images, which we know is theoretically possible when images are sparse in the network’s latent space. What needs to happen to prevent this type of memorization?
Duplicate images caused a minority of cases on their reproduction. All duplicates that were found on SD were of repeated images.
"In contrast, we failed to identify any memorization when applying the same methodology to Stable Diffusion - even after attempting to extract the 10,000 most-outlier samples."
Read, SD does not (seem to) memorize unusual and unique images (even though Imagen does).
That’s all true, you’re right, but what does it mean we should do? Shouldn’t the models actually guarantee they can’t reproduce inputs, given that in the U.S. all images are copyrighted by default? The only images that are legal to copy for training, and legal to distribute in the form of a neural network that can spit them out are the images explicitly licensed, for example, by Creative Commons.
The Imagen outlier results validate the accidental memorization of images, even if it’s a small number. And it might be premature to conclude that one paper’s inability to find memorized outliers in SD means that it doesn’t happen. It might be true, but 10k images is less than one ten-thousandth of the training data, and it’s certainly possible that more successful attack methodologies could exist. This represents a single attempt performed under many assumptions and run on a tiny fraction of the inputs, and nothing more.
Even if Stable Diffusion doesn’t memorize outliers, or any non-duplicate images, does that matter? SD will be out of date pretty soon and replaced by another network. If they didn’t take care to prevent memorization, if SD’s memorization behavior (or lack thereof) is accidental, then how do we know it won’t happen more often in the next network? Isn’t this a problem that needs to be explicitly addressed, and not just claim it’s uncommon?
Humans don't even guarantee that. Composers not-uncommonly accidentally reproduce a melody they've heard somewhere. The only thing that can be done is diligence.
So? When humans do this they are subject to copyright laws, and they can and have been sued for accidentally reproducing a melody.
That’s also not very relevant here. We’re talking about making duplicating machines that effectively memorize pixels, not the same kind of “accident” you’re referring to.
This makes perfect sense and is expected. But you have to know, none of the images are memorized. It's a coincidence. I can explain.
Think of the analogy of linear regression. You have X,Y 2D space. X is the input (the English sentence), Y is the output (the image).
You use training data, which is a bunch of coordinates in this 2D space (each coordinate representing an <English, image> pair) to generate a best fit line (aka the machine learning model).
The properties of that line fit exactly what's going on here. That trend line will not touch most of the coordinates, but it likely will touch a few. When you take an X from the coordinate that touches the line and feed it into the equation of that line (aka model) you will get a Y (an image) that matches the coordinate Y exactly because the coordinate and the line intersect. Makes sense.
This is why an image appears to be memorized. But really it's not memorized. The equation of that line is the ONLY thing memorized here.
Why does it happen more when the image appears multiple times in the training data?
Well this is exactly what happens in least squares analysis. The line is generated from an equation that involves the averages of all the training data (aka coordinates) and if you have many samples of the same coordinate (aka image) you will skew the average and therefore the line towards touching that specific coordinate.
There is no complete technical fix for this issue. With enough training data, EVEN when you get rid of duplicates, the line will likely touch or be very close to a coordinate.
If you think about it, for a 2D coordinate system you can select a bunch of coordinates that generate a line that the never touches a single coordinate. But this defeats the purpose of machine learning. You're suppose to sample data you have no knowledge about and derive a model from it.
If you can already pre-select images that form a model that never touches a coordinate, it means you already have an understanding of the model and can likely just manually tune all the weights by hand to get what you want. As regular humans, we don't have the super-human ability to do this. We can only do it for the simplest 2D example and all the higher dimensional stuff can only be understood by analogy and random sampling.
One thing you can do is just add more data to the training set. That will move the trendline and could shift it away from something it touches. But at the same time it could move it towards touching a new coordinate.
Because the dual of a structure is clearly not another representation of the structure. Isomorphism means nothing. /s
Your intuition is correct, fitting for the data will lead to this arising. However, you misunderstand the interpretation. What is being fit is exactly an encoding of the dataset (a dual of it, like switching from edges of a cube to faces), so the "coincidence" is which specific images show up with extremely high similarity after similar prompting; likely meaning nothing related showed up in the testing, so it never got pushed around and settled nearby. For many images, that encoding was lossy and sufficiently moved that they are not matches to the training data, however for some they have ended up essentially encoded in these "lines" (as your analogy goes) and can therefore be reconstructed with not that much effort.
>Because the dual of a structure is clearly not another representation of the structure. Isomorphism means nothing. /s
But it's not an isomorphism. I have a function that has a range that covers every single number in existence.
Does that mean that function is A copy of every single number in existence? No. Is that function an isomorphism of every number in existence? No mathematician would use the term "isomorphism" in such a way.
You could say that if F(3) = 4, then (4, F(3)) are two things that are isomorphic. But, again, no mathematician would call F(X) isomorphic to 4.
Technically in terms of law, what this would mean is that USING the model to generate a COPY would be illegal. The model itself does not hold a copy until function application is executed. Therefore existence of the model itself IS not illegal.
To say it's illegal is to say that a copy of the mona lisa exists in paint and paint brushes. F(X) is the paint and paint brush. F(3) is the painting.
I will say this, I don't think most people in the jury will be technical enough to get this distinction and the technical part isn't even really important in my opinion. The laws should be made based off of what's better for society/humanity not what's technically valid.
Still my comment here is on what is technically happening which SHOULD remain completely separate from the politics surrounding it.
I mentioned isomorphism in reference to representation, which was kinda the point.
And, not to belabour the point, but a function F that has an isomorphism to the real numbers means that F does indeed encode all numbers (and, when defined as a set, exactly contains a copy of each). That's literally the point of isomorphism. To demonstrate simply, the integers have an isomorphism to the natural numbers: you count them in an alternating pattern. That's an isomorphism between them (and it need not be relation preserving). So, with that, you can interpret all integers as natural numbers; we say they're the same up to their isomorphism. Isomorphisms also exist for the real numbers with other structures, via bijective maps, and so forth.
I'm not going to respond to the other stuff since it's nothing to do with what I said or responded to (even when it was a misinterpretation).
Ok sure. This makes sense. Let's be clear so I don't get off track again. Your point is that an isomorphism means an encoding of the picture exists in the network and is therefore a copy.
My point of contention is that it's not a copy.
But imagine this scenario. The model produces a picture that is identical to a picture in the real world, but that picture was NOT part of the training data. This is certainly possible.
It's fuzzy but there comes a point where the layers of abstraction between two entities that are isomorphic become big enough such that they don't violate copyright law.
Outside the scope of the paper, but sure, it's a valid and interesting question.
If a model were to produce a pre-existing image not in its dataset, then we can strongly suspect that two behaviours have occurred (for the sake of establishing provenance).
First, it was just the ever so small, non-zero chance that it was produced by a random process that was not based on anything learnt, and within the finite resolution and some accuracy of colour, happens to match up to something already existing which, for example, may have been produced independently by a human artist - say, after the dataset was collected, but before the model was trained, such that it is emphatically a "coincidence" and nothing else, no collusion, no learning, no intelligence. Just chance.
Second, and I believe far more likely, this image has been encoded into the model by indirect learning and proxy. For example, say some notable and famous work of art is not part of your dataset (which is believable). However, say that there exist, say, art that references this famous work in some way (as artists sometimes do), which may be up to the point of parody (or some a gag), or may be some small aspect (colour scheme, notable style, outfits, lighting & composition, certain people, etc). Especially if it is what we would literally call "influential", is it not possible to reconstruct the famous work (that was not in the dataset) by indirectly learning about it from other pieces? Now, exact (or near-exact) matches are unlikely, for the most part it would probably tend more towards similarly repeating the references and parody of its dataset, but I feel this would strongly increase the likelihood of randomly producing it as in the first behaviour, and we're inflating the chances certain works are reconstructed entirely by "coincidence"; at least, as according to all observers that merely look at the binary presence of the original in the dataset.
Is this a copy? Is it not a copy? In all honesty, the notion of copy or not is insufficiently defined, because there's always some angle that can be part of it, or may not be, as most copyright law is handled on a case by case basis. Is it a copy because its provenance was an attempt to copy? Was it a copy by cosmic accident or just an independent work? Was it a copy by inferred reconstruction, thus not literal copying from the source, even if a verbatim reproduction is made? Is a vector version of a raster image a copy, and vice versa? What is "sufficiently" transformative to no longer be a copy? Does a copy of the Mona Lisa exist in the mind of a person that experiences it, while as a subject in a photograph it has grounds to not be? I'm trying to avoid anthropomorphising it and state whichever way what this is, I'd rather just illustrate how what we learn, what we experience, what we produce replicas of, and all of this is generally part of an unsolved problem/field: epistemology. Do we understand diffusion models fully? No. Therefore, weighing on one side or another should merely be for the purpose of a leading research question, or to exist on either side of a dialectic; such as in the courtroom. Perhaps we're still just too immature on this topic to really say anything about copying.
Really, the only thing we do know about copying, is that you should never get caught doing it.
(For legal reasons, that was a joke.)
Paintbrush and paint functions like a NN model. While brush strokes operates as input into the "paint brush" function to output a painting.
Could you say that paint and paintbrushes encode every single painting on the face of the earth into it?
I don't think this meshes with our intuition of the word. The model requires input for the encoding to be complete. If we don't think about it this way then practically anything on the face of the earth "memorizes" everything else. A pencil encodes every sketch that has ever been drawn and will be drawn.
This is a semantic issue. What do we mean by the word "memorize" or "encoding"? Eventually the gap between two isomorphisms becomes so big that the words memorize and encoding no longer apply. NN's are right at the border of this demarcation, but if we want to be consistent then the answer should be that NN's do not encode this information.
I believe you're mixing your metaphors, as it were. A paintbrush is not like a neural network, nor is it a function. You may describe a paintbrush in a model, and that may be as a mathematical function, but a paintbrush it not itself a function, it is a paintbrush, a real object that's part of the physical world.
On the other hand, a neural network is a function, because that is what it is constructed as, and what it is intended to perform as, and indeed what it "does". It passes the duck test for a function, and then some. A function has a domain and codomain/image, two sets it draws its inputs from and then produces output accordingly, which for the purpose of mathematical definition, are an intrinsic part of the function.
Therein, it may contain and encode exactly every single painting not merely on the Earth but in all possible existence. The rub is that you still need to construct examples by providing input from that domain to have an output in the real world --- whatever your belief in the Platonic ideal, it is agreeable that the infinite or absurdly large finite sets of input and output that are described are not fully reachable/depicatable in the real world until physically performed, which requires the cause and effect of first providing input into a machine performing that function (let alone the literal encoding of a dataset into parameters as per my prior comments).
So, in a sense, with a model that is defined as such, we can indeed state that a paintbrush does encode every painting into it, albeit merely as possibilities that much be realised. Once picked up, it then requires cause and effect to move from one (the input motions) to the other (a resultant image), which has then indeed made one of those possibilities real. The issue with such a metaphor is that the implied model is begging the question, it circularly supposes that the paintbrush is responsible for this, such that it can serve as a function analogue for the metaphor, such that the paintbrush is responsible. Hence why I say you're mixing metaphors, because you've combined two incompatible metaphors (a neural network is a function, and a paintbrush is a "function" due to the cause-effect analogy to input-output) using shared language, that being the word function; which I believe I've shown has very different meanings here in both contexts.
The end result is that this "intuition" of the word/world is correct, but only partially, because it has not yet recognised how these two uses are divergent as per above. Indeed, such a model of a paintbrush would imply everything is also functions that ties together their input and output as possibilities which are retained, with any inversion providing a way to reconstruct inputs from outputs, such as figuring out how specific brushstrokes may have been performed. The critical thing is, again, to not mix the metaphors of the real world and the mathematical world, because that mathematical model can only describe possibilities for our real world. After all, it does not do us much good to state that a pencil contains every sketches as possibilities, because until we make good on it, it's essentially just a truism from the model that hardly makes it any easier to produce such a sketch - the same can be said for every drop of water, and so forth. Therefore, this is not a semantic issue, but a conflation of Platonic idealism with reality such that much is rendered meaningless, when reality has not provided incontravertible evidence for this idealism.
For example, the paintbrush might not have, as possibilities, every single painting. One approach would be to say it is only going to be used for one, and so a more appropriate model would be needed such that "no man stands in the same river twice" and so forth, and so this function metaphor would be demonstrably inappropriate. In contrast, the neural network (or a diffusion model) is still a function in a world that does this, because its parameters have been frozen and its random seed coupled to its input, with no recurrence from any prior usage as part of its operation. Therefore, its model as a function is exact and clear.
This means that "memorise" and "encode" can have very precise definitions for neural networks or diffusion models (as in the paper, memorised images being those reconstructable by given methods to within a known measure by some metric of accuracy), whereas with a paintbrush these do not have such precise definitions (indeed, beyond being reliant upon certain world models that may or may not be incompatible, they at the very least do not have only one obvious interpretation of meaning, unlike a neural network as a function).
As to the final points, it would be more appropriate invoke the notion of transformative derivation and ask whether neural networks or diffusion models are "de facto" justifiable as "always producing derivatives" due to size. I would say your statement can be reconsidered in light of this, the above, and my prior comments, given that they empirically do not always produce derivatives (we can reconstruct input data from prompts alone), and so the determination of whether there is sufficient transformation to be a derivitive work would have to be handled on a case by case basis for each image produced, as with all things - you would not say that an artist be given the carte blanche right to declare all their works wholly original or transformatively derivative when they've clearly just produced a copy of the Mona Lisa.
Additionally, if we were to "be consistent", it would not follow that this implies there is no encoding of such, only that they do not encode images they cannot produce. There is indeed exactly an encoding of possible (and memorised) images, because this is consistent with the mathematical model of a function that the neural network represents. We can state that neural networks do not directly encode all this information in their real world instance, as they do not, their parameters are only directly correlated to the dataset that is a subset of its input domain. However, they do still indirectly encode a much larger dataset than is explicitly given, due to the fact that humans do not produce art completely randomly, and like the paintbrush, we have created art (now used in a dataset) that is not simply a function but a real continuous object that has recurrence and relation to prior and simultaneous art and thought, and this is what gives rise to all their possible outputs that are not exact copies of their inputs (and by combination they should outnumber those memorised works as well). This is as per my previous comments, however to state explicitly in combination with the above, this implies that neural networks do exactly (if indirectly) encode every possible image they may produce (and not every image that may be possible) as the real world instance/approximation of the function it is depicting (which would encode every possible image, not merely those realisable, supposing its codomain/image was an uncomputable set), of which some are memorised by more direct encoding and others are inferred reconstruction. Either way, these images are producable by the model, therefore the model does encode it, either from the perspective of it as a function, or it as a realised machine performing that function with fitted parameters to enable it (within some bounds and margin of error). So, I would say that you may wish to reconsider the statement that a desire to be consistent implies this information is not encoded, when consistency actually implies the exact opposite - I believe that statement to likely be borne from the same conflation as above, so perhaps this paragraph is a clearly redundant summary by the time you finish reading it.
Now imagine you're an expert witness on the stand in the case that eventually settles this. I don't think this will be persuasive even if it's technically sound. How you end up with a copyrighted work doesn't seem relevant: the model still produced something eerily similar to a copyrighted work.
"Copyright laundering" will be the phrase of the era. Throw Picasso in with a thousand reproductions, wash it with DeviantArt, get Picasso back out. Does it matter than it's algorithmically derived rather than a stroke-by-stroke reproduction? There's probably already ample case law around reproductions to deal with this.
It's technically sound though. Let me give you an example.
y = f(x) = x + 2
This is what's memorized. But with that equation you can plug in a specific x. to get:
3 = f(1) = 1 + 2
The training set here would be (1,3). What is memorized is y = f(x) = x + 2. You can literally see that (1,3) is NOT in the model EVEN though that model CAN produce a (1,3) given the right input.
I think the technical part of this is sound. People will just take the face value explanation which is I see a 3! therefore a THREE was memorized. But technically a 3 WAS not memorized. This is categorically true.
You are Right. The persuasive part of this argument is not very good though as you can see by this thread.
If the model captures (almost) all information in the training sample, that is memorization. In your example 2x + 3 memorized the point (1,5) because you can recall 5 as f(1).
This doesn't make sense. F(X) itself is not (1,5). You have to apply F for it to work.
Paint and paint brushes can be thought of as functions. You apply the right inputs (brush strokes) and you get an output that is a painting.
Therefore could you say that all paintings in the world are encoded into the paintbrush? No. You can't. Not until you use the paintbrush to copy something.
If I ask you to recite a poem you're a function f("recite poem") which returns the memorized poem. It's not even possible to have memorisation without a function.
But it's not memorized. The "poem" in the case of the machine learning model is constructed from a general algorithm. Similar to how 2x+1 has no concept of 3.
When you input a 1, it does a *2 operation and a +1 operation and produces a 3. Nothing is memorized. 3 arises as a side effect of unrelated transformation operations.
There's a joke programming language, HQ9, that only has three instructions, each expressed with a single character. One outputs "Hello world!", one prints the language's source code (quine), one prints "99 Bottles of Beer...".
Does the interpreter have 99 Bottles memorized? After all, it is a function (with a 3-element domain).
"*2" and "+1" is stored information that's used produce 5. The memory of a poem is stored information that's used to vibrate your vocal cords and produce a recited poem.
The memory of a poem is stored information but your vocal cord is NOT stored information.
"*2" and "+1" are vocal cords. They operate on any number of inputs just like how your vocal cord operates on any number of nerve signals and air blowing through it.
If my vocal coords are never used to record a copyrighted song then no violation of the law occured. If I never use the machine learning model to generate an existing picture then no law violation occured.
Well you can get around the whole memorisation problem by only using unique training samples in your dataset, same for Copilot. But yes, I do agree that in this case the user is doing the violation, not the network.
That just means only for those examples it was fixed.
Logically the mathematics behind it says that it can happen. You present evidence which is refutable by new evidence. Logic if correct, cannot be refuted.
The logic is also completely wrong, the dimensionality of a single image is very high, let's just say 2^(32*32) (if it was a 1-bit 32x32 pixel image). That's a probability of 5.56×10^-309 of converging to a sample by coincidence, which is to say it's completely impossible.
The logic is not wrong. First off your statistics showed that it CAN happen. Just with low probability.
Second your way of measuring the probability is incorrect. IF we were to randomly pick an image out of ALL possible images of 32x32 1bit images THEN your probability would apply but clearly machine learning doesn't do this.
The actual calculations behind the probability is actually highly, highly dependent on the dataset.
To take the 2D line analogy further if we perform linear regression on the points (0,0), (1,1), (2,2) (5,5),(329,329),.... and a bunch of points with the same x,y coordinates the line would touch ALL training set data points 100% of the time as the equation of the line would be y = x. This isn't even called overfitting, y = x is actually the ONLY solution available. There's no way to prevent this if the data is just really clean.
The set of natural images is going to be smaller than 5.56×10^-309, but it's still going to be astronomically large. You're simply not going to get an accidental hit on a 512x512 image, even with training. The fact that in the paper they only managed to reproduce photos from Stable Diffusion that are likely common in the training set proves this and conclusively disproves your hypothesis. In addition, they observe that unique photos are extractable from a larger model (Imagen), making it obvious that this reproduction is due to memorisation from the larger capacity of the model and is definitely not due to an accident (otherwise the probability to reproduce would not correlate with model capacity).
> The fact that in the paper they only managed to reproduce photos from Stable Diffusion that are likely common in the training set proves this and conclusively disproves your hypothesis.
There's no hypothesis. My conclusion is not scientific. My conclusion is derived from logic. It cannot be disproved.
>The set of natural images is going to be smaller than 5.56×10^-309, but it's still going to be astronomically large. You're simply not going to get an accidental hit on a 512x512 image, even with training
It might be low probability to get an exact pixel perfect match. But a similar match that is more or less indistinguishable (or even a different but obvious reproduction) to the original from the standpoint of the human eye is a significantly larger probability.
If I have 10 points in 2d space, I can always hit them all exactly so long as I have a polynomial of degree 10 or higher. This is overfitting, which is indistinguishable from memorization.
Over-fitting is a known problem and deliberately avoided. This does not prevent the curve from intersecting an actual training set.
I mean technically, if there are many copies of the same data set pair, THEN you can call it over-fitting. But removing that does not fully remove the problem. The curve will still intersect datapoints EVEN without over fitting.
That doesn't sound right at all. In 2D space a line bisects the plane. There's no way of moving from one side to the other without touching it. The possibility space of images is much, much larger than 2D.
Machine learning is an extension of the concept of linear regression in 2-D space to 9999999-D space. They also make the line more then just a "line" the equation produces more of a best fit complex curve in multi-dimensional space. This is the actuality of what programmers in this area do.
Much of that space consists of pairs of data that aren't relevant. So to understand it in terms of the analogy of the line in 2D space... The only thing that's relevant are points near the line.
For example if we use the line to represent housing costs(Y) over time(X). You can take a bunch of housing prices sold over time and plot it on the graph as dots. Linear regression will form a line that best fits those dots. Not EVERY single section on the plane matters though. Only the dots matter and the space that's very near the trend line have anything meaningful in terms of data.
This seems to mostly happen when an image appears frequently (more than 100 times) in the training data, and/or the dataset is small relative to the model.