Can’t use cloud stuff on-prem and also if your clients have a server room of their own. Same for homelab.
Also it’s nice not to shift the pets attitude from servers to clusters and instead treat everything as cattle - provided you have backups of persistent data and the config versioned in a Git repo and there’s maybe some Ansible in the mix, being able to recreate an environment in the case of a fuckup is nice and also helps against bit rot.
Disclaimer: I actually prefer Docker Swarm/Compose over K8s due to simplicity (which matches my deployments and scale), but in the cases where I had to use a variety of K8s, going for K3s was pretty okay.
If you peel off all the layers in Docker Swarm and K8s, technically it has the same level of complexity. In k8s there are a lot of concepts. I would argue you have the same network, storage, and compute complexities as an operator.
The cluster isn't that hard to recreate if things go south. Everything is in YAML configs already. And since I'm managed, it's just a few clicks in DigitalOcean to create a new cluster. And actually, I think I can create clusters through their CLI too, so if I did want to automate it, it's already ready to go. So I'd say I'm cattle-ready, but too cheap to pay for more cattle.
I nearly went Docker Compose/Swarm by accident when I was just getting started. I knew I wanted to dockerize my app but then couldn't figure out how to get it into prod. Then I found out people don't actually use Docker Compose for prod it seems and eventually stumbled into Kubernetes. It took a few weeks to wrap my head around, but I'm happy with it now.
Once you have a nice set up, I'd say it's pretty simple to maintain. DevSpace is fantastic for development, and then for deployment I just wrote a little script which builds my images, updates the kustomize with new images and applies the manifests. Pretty simple.
Glad to hear you prefer Docker Swarm/Compose, I feel it’s a solid tool for when you need some of the more complex multi server deployments, and helps you avoid all the costs and complexities around K8s. Plus, running docker compose in development and docker swarm in production feels like the closest you can get to having your development environment match your production environment.
> Plus, running docker compose in development and docker swarm in production feels like the closest you can get to having your development environment match your production environment.
I run Minikube in Podman for dev. And then I use kustomize to customize dev, staging and prod environments. The environments are 99% the same, they just have different env vars and memory limits.
I was managing k8s cluster for a CDN, we had small clusters in each of our ~120 datacenters around the world, plus a number of larger clusters in our redundant back offices.
We couldn't use managed clusters because these were running on our own hardware, and they needed to run on the same infrastructure as the CDN itself.
The point of the local clusters was for workloads that needed to run in each data center, and then multiple clusters in the back office for both compliance and operational reasons.
I would not have used a tool like this, though. We used Rancher to manage our clusters.
Well, for hobby purposes I terraformed a completely free cluster on OCI today. It's a "toy" of course, but k3s + OCI "Always free" tier includes Oracle Heatwave mysql which k3s can use in place of etcd. I'm deploying 2x (2 CPU / 12GB mem) VMs to run workloads. Also get an internal load balancer so nodes can find each other an external network load balancer for Traefik. It's quite usable.
I implemented a system that included the OP functionality (plus a whole lot more.) It was for on-premise deployment at customers. It can also be used to spin up stand-alone instances of our system in the cloud, for development, testing, etc. While you could, in theory, do many deployments on a single k8s cluster, there are some benefits to the automatic isolation you get from deploying on a standalone VM.
I'm doing many deployments in my single k8s cluster. I just put them each in a different namespace.
The only piece that's maybe a little dicey is the single load balancer/gateway. If there's a hiccup in that, then everything goes down.
But I've only blown up my cluster once in like 8 years or something, that's not too bad. It was a learning experience :-)
What other kinds of isolation do you want? I can see maybe a separate staging environment if you want to test gnarly things like that ahead of rolling them out to prod. And I guess maybe they can eat eachother's resources if you don't have request limits nor auto-scaling enabled.
But I'm cheap and managing more clusters sounds like a pain. Then I'd have to deal with more kubectl credentials and what not too.
What you’re saying makes it sound like you’re a one-person operation, or somewhere close to that scale. That obviously doesn’t have same requirements as much larger organizations.
I ran a job last night which provisioned a cluster with 4TB of RAM and nearly 1000 vCPUs. It ran for 20 minutes, ingested about 800 GB of data from nearly an million files, and was then deleted. To do that on a single cluster that’s also used for serving production requests would be unnecessarily complex and risky. Our production system has users in every timezone using the system 24x7. At the very least you’d have to provision separate node pools anyway, but why would you bother to do that?
It's applied big brain memetics. k8s turned pet servers into cattle. People then do the next step and want to treat their clusters as cattle as well. Also it has a bit of the "can it run DOOM" vibe to treat whole k8s clusters like this.
I have one. And it's managed. I don't think there's significant cost savings to going unmanaged, but maybe. Even so, why would I need a ton of them?