Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

I thought that in order to create a distributed locking system, you need to be able to reliably fence unreachable nodes. "Network Partitions" sound a lot like "Split Brain" to me. I am more familiar with traditional clustering solutions such as Pacemaker/Corosync and the GFS2 DLM than RabbitMQ, so perhaps I am missing something here?


That's more of a software design consequence than a fundamental limitation.

It's common to build fault-tolerant systems by designing a master-slave architecture, and then bolting a failover mechanism on top so that exactly one node is the master at any time. That approach suffers from exactly the problem you describe: if you can't contact a node, there's no foolproof way to ensure it isn't acting as a master, so you risk split-brain unless you can remotely fence it.

Consensus algorithms like Paxos/Raft don't suffer from this problem; in order to make steady progress, there should be only one master, but they still operate consistently if that assumption is violated in the short term. So no fencing is needed.

A lot of people seem to have a fundamental misunderstanding that using Paxos for leader election is enough to make a system consistent, and it's emphatically not. For instance, I hope this isn't still true in current versions, but for a long time HBase was vulnerable to losing committed data because it misused Zookeeper and allowed multiple regionservers to simultaneously "own" a given table region. (I found about that issue when it happened to me in production, so now my default assumption is that all "distributed locking" systems are broken until proven otherwise.)




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: