This sounds quite similar to https://bors.tech. If the authors are here, did you see this, and can you compare and contrast it with what you built? https://graydon2.dreamwidth.org/1597.html also has a good overview of the problem and the original bors.
Yes we have seen this before! The main difference is that throughput is extremely important for us, which we would not get worth Bors. Also, compatibility of multiple simultaneously merging PRs is the case that we are optimizing for, vs. compatibility with current master.
If you don't mind me asking, How long does a CI run take for you? How do you manage running CI with so many merges?
Our CI takes ~8 hours of machine/VM time, which is about 35 minutes of wall time with our current testing cluster (including non-distributed parts like building). We skip certain long tests during the day, so that brings wall time down to ~13 minutes. But we also test 2-3 branches with decent churn. So even if we're only doing post-merge CI based on the current state of master, we're still getting 5+ commits fairly often.
I want to get to a world where CI is run before and after each merge with master, on every commit (or push/pull), but it seems like it would take so much more resources and infrastructure than we currently have.
