If you think that my opinion is not worth listening to, i.e. that I am wrong, would you mind elaborating why? There is a real opportunity to sway my opinion here, because I am not unsure. I could just be crazy. I don't know anymore. But, generally, I don't think that you necessarily have to have multiple back-end instances, and that if you have multiple back-end instances, that you will necessarily have race conditions. Am I wrong in this?
Well, your opinion doesn't consider many real world factors. (It's also worth noting that you toned down your response to me with hedging language... "generally", "necessarily")
No matter how good a particular server it, it isn't immune to power outages, fiber cuts, fires, basic hardware failure or even just downtime for basic updates (os, application deployment, etc).
As soon as there is any sort of cost with downtime (direct or indirect monetary cost, or reputational cost), basic engineering rigor requires that you use redundancy to handle such failures, and that means spending money (in the form of vendors or engineering) on it. If money and time is being spent to create an application, and there is a reasonable assumption that downtime to that app will have costs to it, one way to amortize the cost of retrofitting redundancy into an app is to start with it.
Having multiple instances of the backend also allows for other cost saving measures:
* N instances of smaller sized server may be cheaper than 1 instance of really good server
* Multiple instances of backend allows for update deployments while the app is live, indirectly driving other cost savings (no overtime or pager pay, happier employess... "they don't give us snacks but we never have to work late", lower costs associated with downtime in due to botched deployments)
* Its cheaper to hire engineers that follow this de facto standard pattern than to sit down and pave new ground with other tools, and using that pattern they will achieve the desired result in terms of reliability and uptime.
* It allows for scaling if your app's traffic is seasonal, meaning you don't need to spend as much on resources as you scale (note this is the first time scaling has been brought up, and it's as a minor point).
Does every app have these concerns? Of course not. Do a very large number of apps have these concerns, or have an expected value calculation around these concerns that says the smart money is on planning for them? Yes.
In the context of a discussion of hardware being so good that a single bit of hardware can handle any load you're likely to throw at it - declaring multiple instances to be pointless is an analysis that hasn't considered any of the factors i brought up, in terms of a real cost analysis. Particularly if the complaint doesn't even introduce the concept of uptime as irrelevant to the app.
So generally is it "necessary" to have multiple instances of backend? i don't know, generally necessary is a very narrow scope. Is it bad to dismiss multiple backends as probably not needed - yes, its equally foolish to immediately requiring them. Proper engineering rigor requires considering not just the technical, but the financial realities of an application.
About race conditions... will multiple backends necessarily end up with them? No in the same way that lottery ticket won't necessarily be a loser. But multiple backends aren't required for race conditions... a single backend will presumably have concurrency within each instance too. And where there is concurrency, there almost certainly will be race conditions, I've rarely seen software that doesn't have them at some point or another. Maybe TigerBeetle doesn't and never will, but that is a unicorn team working on a very narrowly defined bit of software that is merely a component of other systems, working under conditions that are extremely expensive to reproduce for most engineering projects. The general case is that you will write, deploy and be frustrated by race conditions since the cost-benefit analysis doesn't call for absolute perfection... I know I have, odds are you've read about the consequences of it on this site at some point.
The point of all of this is that engineering rigor goes beyond technical rigor. It includes understanding the tradeoffs in terms of budget, uptime, technical decision making, speed of iteration and so on.
I did consider some factors, including the ones you mentioned, when drafting my original message. I just didn't include my considerations into my message. I prefer addressing factors on-demand rather than attempting to immediately address every possible factor, especially for topics such as this one where the number of possible factors is virtually unlimited.
My point is that, by default ("generally"), there is no need to run multiple instances. But I admit that sometimes it can be justified, such as if you (truly) need zero-downtime deployment or N+1 redundancy (which is questionable, because, as senko pointed out, services usually go down for reasons that aren't a power outage or a fire). My complaint wasn't really about this anyway, sorry for not being clear. I mentioned that my company runs multiple services "for absolutely no fucking reason," but I did not state their intention, which is not zero-downtime deployment or redundancy, but rather "scalability," and I find this pointless because a single $5 VPS could easily handle many times the amount of traffic their IO-bound app receives. The end result is that they now must always think about the implications of running multiple instances, and failure to do that properly creates race conditions and other kinds of obscure bugs.
On the other hand, if you only run a single instance, reasoning about the system becomes much easier, and even though that does not eliminate the possibility of a race condition, it becomes much harder to create one. I also like the architectural benefits of a single instance system, such as being able to keep transient state in memory worry-free.
