Hmm. I've asked the authoritative DNS team to explain what's happening here. I'll let HN know when I get an authoritative answer. It's been a few years since I looked at the code and a whole bunch of people keep changing it :-)
My suspicion is that this is to do with the fact that we want to keep affinity between the client IP and a backend server (which OP mentions in their blog). And the question is "do you break that affinity if the backend server goes down?" But I'll reply to my own comment when I know more.
Looks like this has nothing to do with session affinity. I was wrong. Apparently, this is a difference between our paid and free plans. Getting the details, and finding out why there's a difference, and will post.
What’s somewhat complicated here is its apples and oranges. Cloudflare offers DNS and a proxy service. The OP is using both. The comparisons are merely DNS services. I wasn’t clear on X whether OP was getting confused that the IP we return via DNS (which points to our proxy) doesn’t change, or if they were concerned that behind the proxy we’re not routing correctly. I think after reading this the answer is the latter. Confident we always will route optimally as it’s in our interest and our customers’. But why we’re not failing over on failure is interesting. That looks like, as John said, a difference between free and paid plans that if it made sense at some point doesn’t obviously today. Will figure out what’s up and get fixed.
My suspicion is that this is to do with the fact that we want to keep affinity between the client IP and a backend server (which OP mentions in their blog). And the question is "do you break that affinity if the backend server goes down?" But I'll reply to my own comment when I know more.