This is perhaps only tangentially related...
but I'm trying to do something to support sharding of websocket connections based on a user_id....
I want the connections to be distributed to separate instances of a WS server based on modulo-based or consistent hashing algorithm... above that in the stack we have a message broker or queue partitioned using the same sharding algorithm for the number of websocket servers running. This queue/message broker sends publishes messages to each web socket service which in-turn pushes the message to the client. This is how I want to handle horizontally scalable web socket connections...
The question I have is the best way to do this and the best way to migrate connections. From one service to another when/if the replication factor of web socket servers changes (we add/remove servers)...
One way I would think would be to send a disconnect message to the websocket service when this event happens and it would disconnect it's clients and then have a random delay for when re-connection happens... so as to avoid trying to reinitialize all of those connections at once. This seems sub-optimal as with consistent-hashing many of the connections will stay on the same server, and terminating those connections would be unnecessary.
What is the best way to migrate an open TCP from one server to another? Or if the connection must be closed... how can we minimize the number of connections that are to be closed.
BTW: I know I can broadcasts to all instances of the websocket server, and then unless that server holds the connection it is no-op... if the server does have a connection then send the message back... This also seems it would have a limit to how far this scaling-strategy would grow before running into limitations.
Linux has TCP connection repair, which can be used to transparently migrate an active TCP session between two Linux boxes. The TCP traffic will be paused for the duration of the migration, but the other endpoint will not notice as long as the migration is complete before TCP timeouts kick in.
I am one of the co-founders of https://ably.com. We have built a Serverless websockets platform that is near infinitely scalable and designed to handle the complexity you’re describing such as cluster resizing, connection failures, automatic sharding and reallocations using hash rings across our global edge service etc. I’d be really interested to hear why you wouldn’t consider offloading this to a service like ours which powers the likes of HubSpot, Expedia, Spotify etc. Feel free to contact me directly @mattheworiordan if you’d prefer not to comment here. I’m really keen to hear different perspectives on what we’re doing right and where we can improve!
Can you just store the shard id for each user in a database? Each server could check if it is authorative for each incoming connection, and redirect if not.
That way, you can migrate users one by one (sending a 'redirectUserConnections(userId)' to the original server, and even pick a shard that is geographically close to the user. (If you care about race conditions during migrations, you need to be really careful though.)
race conditions during migrations are the biggest concern.... You do not want a client with a connection to server they will never receive a message on.
Could be anything (redis pub/sub with a channel for each shard, sqs with separate queues for each shard, separate partitions in kafka, etc)...
haven't committed to using one particular technology. queues that support retries could be useful in the event that a disconnect happens messages wouldn't be necessarily lost. Something custom built on top of Redis could accomplish the same perhaps.
Yes... but this seems to be then a timing issue to me... what happens if the replication factor changes again while they are reconnecting to the wrong server... then they will miss the second 'reconnect' message... and will stay connected to this server where it will never receive messages... perhaps an edge-case scenario.
I recommend storing the server topology somewhere that's super highly available, like Cloudflare. Then having the client choose from that. If their connection drops (use heartbeats), retry a different one you have and refresh the server topology in parallel. If the second connection fails, retry again with the new topology. This is convergent and avoids putting load on your database to connect, so you don't suffer from thundering herd problems.
I want the connections to be distributed to separate instances of a WS server based on modulo-based or consistent hashing algorithm... above that in the stack we have a message broker or queue partitioned using the same sharding algorithm for the number of websocket servers running. This queue/message broker sends publishes messages to each web socket service which in-turn pushes the message to the client. This is how I want to handle horizontally scalable web socket connections...
The question I have is the best way to do this and the best way to migrate connections. From one service to another when/if the replication factor of web socket servers changes (we add/remove servers)...
One way I would think would be to send a disconnect message to the websocket service when this event happens and it would disconnect it's clients and then have a random delay for when re-connection happens... so as to avoid trying to reinitialize all of those connections at once. This seems sub-optimal as with consistent-hashing many of the connections will stay on the same server, and terminating those connections would be unnecessary.
What is the best way to migrate an open TCP from one server to another? Or if the connection must be closed... how can we minimize the number of connections that are to be closed.
BTW: I know I can broadcasts to all instances of the websocket server, and then unless that server holds the connection it is no-op... if the server does have a connection then send the message back... This also seems it would have a limit to how far this scaling-strategy would grow before running into limitations.