1) something like systemd starts a process in a cgroup, to limit memory usage
2) it disables the kernel OOM killer, and expects to
receive notification itself and kills the offending process
3) systemd crashes? This it can no longer process the
notifications, so you have a child process which is at
the memory limit but not killed.
This is a great example of why an init system should be as simple as possible and not die under any circumstance :) Why would your init system die? If you something like daemontools, they are designed to not even allocate memory after startup, because that could fail.
Was she talking about systemd? Which process in systemd does the killing? Is it systemd-nspawn or something? (I'm not using systemd now.)
systemd is the most notable process manager that has cgroups support, and the article is talking about process managers that "fail".
It doesn't matter if it's not the PID 1 that fails. The article is saying: if the process that is supposed to receive OOM notification and kill the process fails, you will observe symptoms that are very hard to diagnose.
PID 1 dying is of course really bad, but if other processes in the init system die, your system can still be hosed, as pointed out here.
I don't think the standard Linux kernel supports user-space OOM handling. That's a patch set that Google has been pushing for a few years, but that hasn't been accepted yet (as far as I know). So I'm guessing she's talking about Google's server infrastructure.
My understanding is that the program running in the cgroup is disabling the OOM killer and doing it itself, so only whatever is inside of the cgroup is frozen (Presumably not systemd). Most programs have no need to disable the OOM killer anyway so it's unlikely they'd run into this problem anyway (They'd just get killed off instead of hanging things).
I don't think that's how that works. A process inside a cgroup has consumed all of the cgroup's RAM. Outside that cgroup you run `ps aux`, which will list all processes, including those inside the container, and this will hang forever. Unless, I am misunderstanding it, and each cgroup gets its own /proc filesystem.
This is only an issue when you disable OOM killer. If your OOM killer replacement dies, then you have a problem, and you shouldn't have disabled it in the first place.
The kernel isn't going to protect you from turning off critical bits.
Yes, I read the article. My point is that a container that ran out of RAM and is running a buggy process manager affects the entire system, not just the container.
Was she talking about systemd? Which process in systemd does the killing? Is it systemd-nspawn or something? (I'm not using systemd now.)