CharmingGroot

Production Server Setup Notes, Ep. 2 - Why We Chose k3s

infra3 min한국어

Also available in Korean: 운영계 세팅 회고 ep2 - k3s를 고른 이유

Requirements

Our services had to run on a single customer server.
The system is split into microservices, and all of them run on that one server.

  • Web frontend
  • API server
  • Service workers
  • PostgreSQL (backed up with pgBackRest) and Redis
  • Scheduled jobs such as DB backups (CronJob)
  • Observability: logs (Loki), traces (Tempo), metrics and dashboards (Prometheus, Grafana), collector (OpenTelemetry Collector)

It was one server for now, but we had to keep high availability in mind for later.
So we needed a foundation that could orchestrate many services and scale out from a single server.

Why k3s

With microservices, each service has to be deployed on its own, restarted when it dies, and reachable by the services that call it.
In Kubernetes you declare the desired state, and controllers keep reconciling the actual state toward it.

  • Self-healing: restarts failed containers, replaces containers that fail their liveness probe, and sends no traffic until readiness passes.
  • Automated rollouts and rollbacks: rolling updates deploy without downtime and roll back to the previous version when something goes wrong.
  • Service discovery and load balancing: other services are called by Service Name. The Service Name stays the same when Pods change, and traffic is spread across the Pods.
  • Secret and configuration management: configuration and secrets are kept out of the image as ConfigMaps and Secrets.
  • Batch execution: scheduled jobs such as DB backups run as CronJobs.
  • Horizontal scaling: scale out by raising the replica count.

Docker Compose can also run several containers together.
But it is a single-host tool, so growing to more servers means moving to a multi-host orchestrator such as Kubernetes or Docker Swarm anyway.
With scaling in view, starting on Kubernetes was the right call.

k3s instead of kubeadm

Installing Kubernetes with kubeadm gives you only the control plane.
The container runtime, Pod network, DNS, Ingress controller, load balancer, and volume provisioner each have to be chosen and installed separately.
Installing and maintaining all of that one by one on a single server was a burden.

kubeadm installs only the control plane; you pick and run the rest Container runtime Pod network (CNI) Cluster DNS Ingress controller Load balancer Volume provisioner k3s bundled in a single binary containerd Flannel CoreDNS Traefik ServiceLB local-path SQLite
kubeadm leaves everything outside the control plane for you to choose and attach. k3s packs components for the same roles into a single binary.

k3s puts all of that into a single binary.
Its default datastore is SQLite, so there is no separate etcd to manage.

On minimum specs alone the two are similar. The kubeadm docs list 2 GB of memory per machine and 2 CPUs for control plane machines; the k3s docs list 2 CPUs and 2 GB for a server node. The difference is the number of components you install and watch separately.

High availability and scaling

k3s stays the same when you add servers.
Restarting a server that runs on SQLite with --cluster-init switches it to embedded etcd.
Adding two more server nodes to make three gives a highly available control plane. etcd needs a majority, so server nodes come in odd numbers.
When more nodes are needed to share the workloads, add agent nodes.

Single node server node control plane + workloads add server nodes High availability server node 1 control plane server node 2 control plane server node 3 control plane server nodes in odd numbers (3 or more) agent node 1 workloads agent node 2 workloads
An odd number of server nodes, three or more, makes the control plane highly available; agent nodes are added when more workload capacity is needed.

k3s is standard Kubernetes, so if we ever have to move to another distribution, the manifests and Helm charts come along as they are.

The sales angle

It also sells well, so to be honest we did not think very hard about the alternatives.

References