Seems like it would be very simple to prevent this from happening by just adding a perceptual hash check[1] for collisions vs the training set before emitting an image. I'm sure someone would be smart enough to make a "perceptual cuckoo hash" type thing so it would be very fast at the expense of sometimes erring on the side of caution and not emitting an image that actually isn't the same as the training data.
Yeah it seems so obvious I can't help but wonder if there's something about it that doesn't work in practice.
Like if you set it to a threshold where it's not routinely catching false positives, it doesn't catch the original images either because they're still too subtly different or something.
Maybe they really did just not bother though. Heck, maybe they want to test it in court or something.
First, it probably wouldn't work. Stable Diffusion is going to be a lot better at telling whether two images are the same than a perceptual hash check. E.g. I bet if you removed all pictures of the Mona Lisa from the training data, it could still produce a pixel perfect copy of it, just from the many times it appeared in the background of pictures, or under weird adjustments or lighting that fooled the perceptual hash.
Second, I would guess they wanted common images to appear in the training data more than once. It should be trained on the Mona Lisa more than someone's snapchat of their dinner. Common images are more salient.
Reminds me of asking Chat-GPT for an obfuscated C contest entry. It produced one verbatim, though it couldn't explain what it did (it looked like syntactic nonsense, but it printed the Twelve Days of Christmas when run). I can only imagine it saw that inscrutable sequence of characters enough times that it memorized it.
> Stable Diffusion is going to be a lot better at telling whether two images are the same than a perceptual hash check.
Why would it be? Stable diffusion is a text-to-image model it's not at all focussed on determining whether images are the same.
Secondly, I'm not proposing deduplication of the training set (I know some sibling comments have proposed this). I'm proposing a perceptual hash or similar check on the way out so that if a "generated image" is too similar to an image in the training set it gets dropped rather than returned.
The reason it’s not implemented yet is because none of these products are being monetized yet, so there’s no incentive.
Google can legally return a copyrighted image in Image search (it’s not selling the image, it’s selling ads against search results) and this would probably fall under the same protection.
Now if stable diffusion was sold as SaaS in the way Dall-E is…
Correlations in the math domain get weird. It wouldn't surprise me in the slightest that those slight distortions you see in the "memorized" image versus the original ones, while small to humans, turn out to be staggeringly large to your perceptual hash. See the things like the little stickers you can stick to stop signs to make some neural nets decide they are people and such.
And it could go the other way too; it could be that the perceptual hashes are even "better" than humans at seeing past those distortions.
My point is that this all gets more complicated than you may think when you're trying to apply a hash designed for real images against the output of an algorithm.
And even if the hash worked perfectly, the false positive and false negative landscape would still be very likely to contain very surprising things.
Maybe this would make sense for a commercial service but I don’t see why would be shipped as part of the model itself. Maybe the training of the model can be tweaked to accommodate a new step to prevent reproductions but I don’t know enough to understand the repercussions on performance from doing that.
I don't know why they didn't do this tbh.
[1] https://en.wikipedia.org/wiki/Perceptual_hashing