Key Features and Improvements ✨​

Core Storage Evolution: etcd 3.6 & SQLite Upgrades 💾​

We have officially upgraded our bundled etcd to the v3.6 series. While this brings large performance improvements and memory optimizations upstream, it introduces a critically important caveat for administrators running High Availability (HA) clusters: etcd v3.6 changes the data directory structure, which prevents a clean downgrade to v3.5 once your cluster has updated. If you ever need to rollback a v1.36 cluster to v1.35, you must follow specific manual snapshot restoration and node cleaning procedures. Be sure to review our dedicated migration guide before upgrading production environments.

Revolutionizing Builds with the Nix Snapshotter ❄️​

In an exciting move for reproducibility and declarative environments, K3s now includes native support for the Nix snapshotter plugin. This integration allows users to leverage Nix store paths directly as container layers, opening the door for blazingly fast image provisioning and native NixOS container deployments within K3s clusters.

Streamlined Runtimes: Default Imports to Containerd Base Templates 🏗️​

Customizing your container runtime configuration just got much easier. K3s now supports importing containerd configuration drop-ins. This architectural change allows administrators to modularly add configuration snippets that add to or replace configuration from the base file without having to extend or replace the entire default configuration template.

Dynamic Security: Secrets Encryption for Existing Clusters 🔐​

Security is never static, and neither should your cluster configuration be. Historically, enabling at-rest Secrets Encryption required setting it up during the initial cluster bootstrapping phase. With v1.36, you can now enable secrets encryption on existing clusters without needing to recreate them, making it vastly easier to harden legacy deployments. You can also switch between encryption providers - both aescbc and secretbox are natively supported.

AI & Hardware Acceleration: Containerd 2.2 with NRI Support 🚀​

We have bumped our embedded container runtime to Containerd 2.2, which introduces out-of-the-box support for the Node Resource Interface (NRI). This is a game-changer for clusters running edge AI and heavy GPU workloads, enabling cleaner integration with the NVIDIA Container Toolkit and runtime-level device management.

Language & API Stability: Go 1.25 & Gateway API CRDs 🤝​

K3s continues to pull in the latest performance and toolchain enhancements by building on Go 1.25. Additionally, we have adjusted our ingress defaults: if you choose to disable or remove the built-in Traefik ingress controller, K3s will now keep the Gateway API CRDs intact instead of cleaning them up. This ensures your custom ingress infrastructure won't break during adjustments, paving the way for bundled Gateway API CRDs coming natively in v1.37.

Bug Fixes and Notable Changes 🛠️​

As always, we’ve ironed out numerous bugs under the hood. Here are the most impactful fixes included in this release:

• Resilience & etcd Cluster Hardening This release eliminates critical race conditions during cluster expansion to guarantee that slow-responding nodes register reliably during initialization. It also patches edge-case bootstrap data bugs, allowing the cluster to recover quorum and reconcile smoothly even after sudden multi-node power failures. Finally, the process for transferring voting rights during dynamic member upgrades has been refined, successfully eliminating transient API timeouts.

• Core Networking & Logic Fixes To improve storage reliability, we resolved a subtle network synchronization issue that previously caused metric deadlocks during simultaneous multi-volume attachments. Additionally, internal certificate rotation tracking has been significantly hardened to prevent nodes from entering continuous reload loops if the system time is modified. K3s also resolves private dual-stack registration anomalies, eliminating unexpected node join delays in mixed IPv4/IPv6 topologies.

Version Bumps for Key Components 🚀​

Special Thanks to Our Contributors 🙏​

We are incredibly grateful to our community members who contributed key improvements, rigorous testing, and code during this cycle. A massive thank you to:

@Ada, @CharlieTonnesla, @DT1mote, @FabianoFidêncio, @JonHermansen, @MichaelMoll, @RahulRai, @RicardoNoriega, @TomRisse, and @zijiren

Your passion and hard work are what push K3s forward!

Join our Adopters list 💎​

If K3s is making your life easier, the best way to say "thanks" is to add your company to our official Adopters list. It’s a tiny gesture that carries a lot of weight for the project's health and visibility within the CNCF ecosystem. We are currently working hard to get our 'status' inside the CNCF to progress and showing a large list of Adopters would help tremendously.

The task is easy: create a PR that adds your name in https://github.com/k3s-io/k3s/blob/main/ADOPTERS.md.

Thanks a lot!

From satellites to smart farming, K3s is proving that Kubernetes can run anywhere — and this year’s KubeCon is packed with sessions that highlight exactly that.

K3s at KubeCon Europe 2026

Whether you're a long-time user or just discovering K3s, there are plenty of opportunities to learn, connect, and contribute during the event.

Lightning Talk​

K3s will be featured in a Lightning Talk, delivering a fast-paced update on the project and its latest innovations.

Project Lightning Talk – K3s
📅 Monday, March 23, 2026
🕒 14:22 – 14:27 CET
📍 Elicium
🎤 Speaker: Manuel Buil, Maintainer, SUSE

View session details

Expect a concise overview of how K3s continues to evolve and support modern cloud-native workloads at the edge and beyond.

🚨 ContribFest: K3s Session​

Want to get hands-on with K3s and contribute to the project? Don’t miss the K3s ContribFest Session!

K3s ContribFest Session
📅 Wednesday, March 25, 2026
🕒 14:15 – 15:30 CET
📍 Room G107
🎤 Manuel Buil (SUSE) & Orlin Vasilev (SAP)

Join the session

This interactive session is perfect for:

• New contributors looking to get started
• Developers interested in K3s internals
• Users who want to give back to the community
• Anyone curious about how open source collaboration works in practice

K3s in the Wild: Must-See Talks​

K3s continues to shine in real-world use cases across edge computing, AI, and distributed systems. Here are some exciting sessions where K3s plays a role:

🚀 Edge Computing in Space​

Bringing Cloud Native PaaS to Space: Onboard Edge Computing for Satellites
Speakers: Adèle Karam-Hankache & Sergiu Weisz
Watch session

🚜 Kubernetes on Tractors​

Cloud Native at the Farm Edge: Running Kubernetes and AI on Tractors
Speakers: Mauro Morales & Jordan Karapanagiotis
Watch session

🏰 Multi-Tenancy & Isolation​

Three Shades of Isolation: A Multi-Tenancy Fortress
Speakers: Braulio Dumba & Paolo Dettori
Watch session

⚙️ Platform Engineering Evolution​

Platform Engineering 2.0: Just Enough Kubernetes and AI-Native DevOps
Speaker: Shweta Vohra
Watch session

These talks showcase how K3s is enabling innovation in environments where traditional Kubernetes would be too heavy or complex.

☁️ Open & Accessible Cloud​

Krafting the Cloud: Building a Free, Open, and Accessible Cloud
Speaker: Alex Bissessur
Watch session

Panel Discussion​

K3s is also represented in broader CNCF conversations around security and project sustainability.

Strengthening CNCF Projects: Impact of Security Self-Assessments
🎤 Featuring Orlin Vasilev and other industry experts

View panel details

This panel dives into how CNCF projects, including K3s, are improving security practices and building trust across the ecosystem.

Find K3s at the Project Pavilion​

Stop by the K3s kiosk at booth P-1A during the following times:

Wednesday: 10:00 – 13:30
Thursday: 10:00 – 12:00

Come chat with contributors, ask questions, and learn more about K3s in real-world use cases!

Don’t forget to explore the Project Pavilion, where CNCF projects come together to showcase their work and connect with the community.

Learn more:
https://events.linuxfoundation.org/kubecon-cloudnativecon-europe/features-add-ons/project-engagement/

While K3s may appear across multiple booths and partner ecosystems, this is the perfect place to:

• Meet contributors and maintainers
• Discover integrations and real-world use cases
• Ask questions about running K3s in production
• Explore ways to get involved

Join the K3s Community​

K3s has grown into one of the most impactful Kubernetes distributions thanks to its vibrant and welcoming community. Whether you're running clusters at the edge, in the cloud, or somewhere entirely unexpected — there’s a place for you in the K3s ecosystem.

KubeCon is the perfect opportunity to:

• Meet fellow K3s users and contributors
• Share your use cases and experiences
• Learn from real-world deployments
• Start contributing to the project

See You in Amsterdam!​

We’re excited to connect with the global cloud-native community and showcase how K3s continues to push the boundaries of Kubernetes.

If you're passionate about edge computing, lightweight Kubernetes, and open source collaboration, don’t miss the K3s sessions at KubeCon Europe 2026.

See you in Amsterdam!

For more details and to register for KubeCon + CloudNativeCon Europe 2026, visit the official event page.

Orlin Vasilev
K3s Community Manager

Our focus this release was on stability. We know that our users value a platform that remains consistent and reliable year over year. From major security milestones to core language upgrades, here is what’s new in the world of K3s.

Key Features and Improvements ✨​

Modular Executor Interface & CNI Startup 🏗️​

One of the most significant architectural improvements in this release is the refactoring of the Executor interface. By making CNI startup a first-class part of this interface, we have removed reliance on "CLI flag hacks" and direct dependencies on networking providers like Flannel or Kube-router. This decoupling makes K3s more modular and significantly easier to integrate into diverse platforms and distributions.

Embedded Kine Metrics 📊​

Visibility into your database layer just got a lot better. We’ve enabled embedded metrics for Kine, allowing you to monitor the performance of your external database (like PostgreSQL or MySQL) directly through the K3s metrics endpoint. This is essential for debugging latency at the storage layer before it impacts your workloads.

Node Password Secrets 🔐​

Security and automation take a step forward with the evolution of the Node Password Secrets. Instead of using the generic opaque type, we use a custom type. This allows the node password secret controller to only watch node password secrets instead of all secrets, which improves resource utilization and increases responsiveness in larger clusters. Node password secrets are also now automatically cleaned up if node registration fails, instead of leaving orphaned secrets.

Security Self-Assessment 🛡️​

As part of our commitment to excellence, we have completed a comprehensive Security Self-Assessment. This deep dive into our architecture and processes ensures that K3s continues to meet the highest standards for production environments, providing a clear roadmap for how we protect your data. link

Language Evolution: Go 1.25 🚀​

K3s is now built with Go 1.25. This brings the latest performance optimizations and security patches to the K3s binary, ensuring that our "default" remains the most efficient way to run Kubernetes.

Inclusive Naming Migration 🤝​

Words matter. We have officially transitioned our primary branch naming from master to main and updated internal references to align with inclusive naming standards. This ensures our codebase remains welcoming to all developers who wish to contribute.

You can find more information on what v1.35 brought upstream in this cool blog post

Bug Fixes and Notable Changes 🛠️​

As always, there were a lot of bug fixes and here, are the most impactful:

• Networking & Dual-Stack Resilience: We've pushed several fixes to stabilize complex networking setups. This includes corrected IPv6 handling for LoadBalancer addresses, resolving fatal errors in Network Policies (NetPol) when node IPs change, and hardening our Tailscale integration to handle pre-existing configurations more gracefully.
• Reliable HA Cluster Management: Resolved critical edge-case bugs related to etcd member promotion and bootstrap data reconciliation. This ensures that when etcd members join or leave, or are restarted, the quorum remains stable and the promotion process is seamless.
• CI Pipeline Migration: We successfully migrated the K3s Pull Request and Release CI pipelines from Drone to GitHub Actions, increasing transparency and ensuring the continuity of our build process for years to come.

Version Bumps for Key Components 🚀​

Special Thanks to Our Contributors 🙏​

We are incredibly grateful to our community members who contributed key improvements during this cycle. A massive thank you to:

@systemj, @farazkhawaja, @AshiqN, @rorosen, @xelus22, @jvassev

Join our Adopters list 💎​

If K3s is making your life easier, the best way to say "thanks" is to add your company to our official Adopters list. It’s a tiny gesture that carries a lot of weight for the project's health and visibility within the CNCF ecosystem. We are currently working hard to get our 'status' inside the CNCF to progress and showing a large list of Adopters would help tremendously.

The task is easy: create a PR that adds your name in https://github.com/k3s-io/k3s/blob/main/ADOPTERS.md.

Thanks a lot!

Online & Offline Strategies: The Power of Local Import 📦​

K3s provides mechanisms for ensuring large images are available quickly, that address two common scenarios:

• Online Clusters: To avoid slow image pulls from an external registry when a pod starts, K3s can pre-pull images from a manifest file.
• Offline (Air-Gapped) Clusters: Where no external registry is available, K3s can import images directly from local tarball archives.

1. Pre-Pulling Images via a Manifest File (Online) In scenarios with internet connectivity, the goal is to initiate image pulls as early and efficiently as possible. K3s can be instructed to sequentially pull a set of images into the embedded containerd store during startup or while K3s is running. This is ideal for ensuring base images are ready the moment the cluster starts or the moment the application is deployed. However, if this process is done before the cluster is started, K3s won't successfully start until all images have been pulled, which could make K3s fail to start if it takes more than 15 minutes. If you suspect this is happening to you, you'd better do the pre-pulling while K3s is running.

Users can trigger a pull of images into the containerd image store by placing a simple text file containing the image names, one per line, in the /var/lib/rancher/k3s/agent/images directory. As we have just explained, this can be done before K3s starts or while K3s is running. For example, you can execute the following in one of the nodes:

mkdir -p /var/lib/rancher/k3s/agent/images && echo docker.io/pytorch/pytorch:2.9.0-cuda12.6-cudnn9-runtime > /var/lib/rancher/k3s/agent/images/pytorch.txt

In the previous command, we have created the images directory on the node and dropped a file names pytorch.txt that contains the image: docker.io/pytorch/pytorch:2.9.0-cuda12.6-cudnn9-runtime.

The K3s process will then pull these images via the CRI API. You should see the following two logs:

# When the k3s controller detects the file
level=info msg="Pulling images from /var/lib/rancher/k3s/agent/images/example.txt"
level=info msg="Pulling image docker.io/pytorch/pytorch:2.9.0-cuda12.6-cudnn9-runtime"

# When the import is ready. It specifies how much time it took in ms:
level=info msg="Imported docker.io/pytorch/pytorch:2.9.0-cuda12.6-cudnn9-runtime"
level=info msg="Imported images from /var/lib/rancher/k3s/agent/images/example.txt in 6m1.178972902s"

2. Importing Images from Tarballs (Offline & Ultra-Fast)

For the absolute fastest startup—critical or when being in an air-gapped environment, the images should be available locally as tarballs. K3s will load these images directly into the containerd image store, bypassing any network traffic entirely.

Place the image tarballs (created using docker save or ctr save) in the same /var/lib/rancher/k3s/agent/images directory. K3s will decompress the tarball, extract the image layers, and load them.

For example, I have created an image tarball with all the images required to deploy the popular microservices-demo with the name microservices-demo.tar.gz.

# Example: Save the image and place the tarball
mkdir -p /var/lib/rancher/k3s/agent/images
cp microservices-demo.tar.gz /var/lib/rancher/k3s/agent/images/

The K3s process will load those images and you should see the following two logs:

level=info msg="Importing images from /var/lib/rancher/k3s/agent/images/microservices-demo.tar.gz"
level=info msg="Imported images from /var/lib/rancher/k3s/agent/images/microservices-demo.tar.gz in 1m39.8610592s

You can verify the successfully imported images at any time using the bundled client: k3s ctr images list

Optimizing booting times with tarballs​

By default, image archives are imported every time K3s starts to ensure consistency. However, this delay can be significant when dealing with many large archives, for example, microservices-demo.tar.gz took 1m39s to import. To alleviate this, K3s offers a feature to only import tarballs that have changed since they were last processed. To enable this feature, create an empty .cache.json file in the images directory:

touch /var/lib/rancher/k3s/agent/images/.cache.json

The cache file will store archive metadata (size and modification time). Subsequent restarts of K3s will check this file and skip the import process for any large tarballs that haven't changed, dramatically speeding up cluster boot time. Therefore, to check that this is working, check .cache.json is not empty and, after restarting, that the two log lines do not appear anymore.

Note that the caching mechanism needs to be enabled carefully. If an image was removed or pruned since last startup, take manual action to reimport the image. Check our docs for more information.

Embedded Registry Mirror 🕸️​

K3s offers an in-cluster container image registry mirror by embedding Spegel. Its primary use case is to accelerate image pulling and reduce external network dependency in Kubernetes clusters by allowing nodes to pull cached image content directly from other nodes whenever possible, instead of requiring each node to reach out to a central registry. To enable this feature, server nodes must be started with the --embedded-registry flag, or with embedded-registry: true in the configuration file. When enabled, every node in your cluster instantly becomes a stateless, local image mirror listening on port 6443. Nodes share a constantly updated list of available images over a peer-to-peer network on port 5001.

# Enable the embedded registry mirror
embedded-registry: true
# To enable metrics that can help with the embedded registry mirror
supervisor-metrics: true 

And then, on all nodes, you must add a registries.yaml where we specified what registries to allow a node to both push and pull images with other nodes. If a registry is enabled for mirroring on some nodes, but not on others, only the nodes with the registry enabled will exchange images. For example:

mirrors:
  docker.io:
  registry.k8s.io:

If everything boots up correctly, you should see in the logs:

level=info msg="Starting distributed registry mirror at https://10.11.0.11:6443/v2 for registries [docker.io registry.k8s.io]"
level=info msg="Starting distributed registry P2P node at 10.11.0.11:5001"

And you should be able to see metrics of Spegel by querying the supervisor metrics server:

kubectl get --server https://10.11.0.11:6443 --raw /metrics  | grep spegel

For more information check the docs

Bonus: eStargz images ⚡​

A different solution to speed up the creation of pods is by using a special image format called eStargz. This enables lazy pulling, which means that the application can start almost instantly while the rest of the image is pulled in the background. This strategy requires both the image to be specifically built in the eStargz format and the K3s agent to be configured to use the stargz snapshotter: --snapshotter=estargz flag, or with snapshotter: estargz in the configuration file.

This is currently an experimental feature in K3s and we have more information in the advanced section of our docs. We would love to hear your feedback if you are using it.

Conclusion 🏁​

K3s provides robust, flexible tools to tackle slow image pulls, a problem magnified by today's multi-gigabyte cloud-native and AI images. By leveraging pre-pulling manifest strategies, tarball loading or optimizing image distribution with the embedded Spegel registry mirror, you can shift slow network operations into quick local operations. These mechanisms ensure your resource-constrained and air-gapped clusters achieve rapid, predictable startup times, delivering a consistently better user experience.

Containerd and Runc​

First we need to talk briefly about how containerd works with runc. Containerd is a long running daemon that is responsible for:

• Image Management: Pulls and stores images from registries.
• Container Management: Manages the lifecycle of containers (create, start, stop, delete).
• Snapshot Management: Uses snapshotters to manage the filesystem layers for containers.
• Runtime Management: Delegates the creation of containers to OCI-compatible runtimes like runc.

When you create a pod in Kubernetes, kubelet uses the CRI plugin implemented in containerd to request a pod sandbox (RunPodSandbox) and then container creation (CreateContainer). Containerd then calls a shim process that acts as a middleman between containerd and the OCI runtime (for example runc). This shim process allows the container to keep running even if the containerd daemon crashes or restarts.

The shim generates the OCI runtime bundle (config.json and rootfs path) and then executes the runc binary. Runc reads the config.json, sets up the container’s namespaces and cgroups, and then launches the container process.

Runc is the component that directly interfaces with the Linux kernel — configuring cgroups, namespaces, seccomp, capabilities, and mounts. After runc finishes creating the container, it exits, leaving the shim to manage the lifecycle and I/O of the container.

Sysbox Runtime​

Sysbox is an open-source, next-generation container runtime created by Nestybox. Unlike traditional runtimes (such as runc), Sysbox is designed to let you run "system containers". It primarily leverages Linux user namespaces and other features to provide containers that behave more like lightweight virtual machines.

This means you can run workloads like Docker, Systemd, containerd, or even K3s inside your pods — all without requiring privileged mode.

In short, Sysbox bridges the gap between application containers and virtual machines, enabling use cases like running Kubernetes-in-Kubernetes (K8s-in-K8s), CI/CD pipelines that need full OS-like environments, or development sandboxes with VM-level isolation but container speed.

Sysbox-runc Containerd Integration​

After investigating this issue, I was able to locate the root cause, as explained in this PR. Containerd was failing to run a specific subcommand for sysbox-runc called features, which led to the following error:

level=debug msg="failed to introspect features of runtime \"sysbox-runc\"" error="failed to unmarshal Features (*anypb.Any): type with url : not found"

Because of this, containerd instructed sysbox-runc to run containers without user namespaces, causing container creation to fail. The fix for this bug was recently merged in the sysbox-runc repo, enabling containerd to work with sysbox-runc.

Running Sysbox-runc With K3S

In order to run sysbox-runc with K3s you need to have a running K3s cluster, and then you can proceed to install the latest version of sysbox, However, since the fix for containerd support hasn't yet been integrated to sysbox main repo only in sysbox-runc, we need to build the binaries from source to get the latest updates.

1. Install docker in your system.

2. Clone the repo and prepare the code

git clone --recursive https://github.com/nestybox/sysbox.git

cd sysbox/sysbox-runc

git pull origin main

cd ..

make IMAGE_BASE_DISTRO=ubuntu IMAGE_BASE_RELEASE=jammy sysbox-static

You can then copy the binaries built to /usr/bin or if you are building on the same machine that you will run containerd you can just run:

make install

3. Run sysbox binary

sysbox

4. Create sysbox runc runtime class

apiVersion: node.k8s.io/v1
handler: sysbox-runc
kind: RuntimeClass
metadata:
  name: sysbox-runc

5. Add sysbox runc to containerd configuration, you can do that by creating /var/lib/rancher/k3s/agent/etc/containerd/config.toml.tmpl:

[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.sysbox-runc]
  runtime_type = "io.containerd.runc.v2"

[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.sysbox-runc.options]
  SystemdCgroup = false
  BinaryName="/usr/bin/sysbox-runc"

6. Finally you can create pod running with the runtime class for sysbox-runc and hostUsers: false:

apiVersion: v1
kind: Pod
metadata:
  name: ubuntu
spec:
  runtimeClassName: sysbox-runc
  hostUsers: false
  containers:
  - name: ubuntu2204
    image: ubuntu:22.04
    command: ["sleep", "40000000000"]
  restartPolicy: Never

Conclusion​

Sysbox brings a powerful capability to Kubernetes: the ability to run system-level workloads inside containers with strong isolation, without requiring privileged mode. When combined with K3s, this opens the door to new use cases such as:

• Running Kubernetes-in-Kubernetes clusters for virtual clusters (k3k).
• Creating secure developer sandboxes that behave like lightweight VMs.
• Running system daemons or nested container engines inside pods.

While Sysbox is officially supported with CRI-O today, the recent fixes in sysbox-runc allow it to run on containerd as well — making it possible to integrate with K3s. The integration is still evolving, but it shows how the container ecosystem is moving beyond traditional app containers toward more flexible "system containers."

If you’re experimenting with K3s and want to explore system workloads inside pods, Sysbox provides a compelling way to do so while maintaining Kubernetes-native workflows

It's been a busy and productive few months, and we're excited to share some of the amazing progress we've made. From new features that make your lives easier to important bug fixes and foundational improvements, our team and community have been hard at work.

Key Features and Improvements ✨​

Here’s a look at some of the most impactful features and updates we've rolled out:

Automatic Certificate Renewal Window Increase 📅​

We've made certificate management more user-friendly by increasing the automatic certificate renewal window from 90 to 120 days. This means if you perform quarterly upgrades, your certificates will be renewed more frequently, preventing them from expiring and making your clusters unusable. For more information check our docs

Optional Airgap Image Tarball Imports 💨​

For those who use K3s in an airgap environment, you now have the option to skip importing all image tarballs. You can select to import only images that have changed since they were last imported, even across restarts. This can significantly speed up the startup process when deploying a large number of images, as the kubelet can start sooner. Check our docs for more information

Enhanced Certificate Check Output ✅​

We've improved the output of our certificate checks to provide clearer information about certificate usage and expiration dates. The output can also now be viewed in different formats. Our docs show an example of the new output format.

Kube-scheduler and Kube-controller-manager Certificate Management 🔐​

We are now generating and managing certificates for kube-scheduler and kube-controller-manager, and they can be rotated using our existing certificate rotation tool.

Retention Flag for S3 Stored Snapshots 💾​

We've added a new retention flag for snapshots stored in an S3 bucket. This allows you to keep snapshots in S3 for longer periods while maintaining a smaller number of local snapshots. This is a great feature for balancing long-term disaster recovery with local storage efficiency. Check out our docs for more information about how to use it.

Official Governance Model 🤝​

We're proud to announce that we now have an official governance model in place! This will help bring more clarity to our project and hopefully encourage more developers to join our fantastic community. You can read it here.

Bug Fixes and Other Notable Changes 🛠️​

We've also been busy tackling a number of bugs, tech debt items and making other important improvements under the hood:

Numerous bug fixes, including those for secrets encryption timeouts and race conditions, DNS fallbacks, various authorization and authentication handling issues and replacing go-bindata with the go native embed package.

Improvements to our test and build infrastructure, e.g. migrating of K3s release artifacts to GitHub Actions (GHA).

Version bumps for key components 🚀​

Apart from Kubernetes v1.34.x, we bumped versions for several key components. Here is the list with the latest versions:

Special Thanks to Our Contributors 🙏​

We want to give a special shout-out to the incredible contributors who are not part of our core maintainers list. Your work is invaluable to the project's success. Thank you to: @ErikJiang, @eggplants, @muicoder, @eugercek, @yulken, @l2dy, @OrlinVasilev

We look forward to an even more productive future with all of you!

This blog post delves into the heart of K3s's efficiency: its initialization process. We'll embark on a journey through the steps that enable K3s to materialize a fully functional Kubernetes cluster so quickly. By examining K3s’s own logs, we'll unravel the meaning behind each step, providing you with a practical understanding of how K3s achieves its remarkable speed. This exploration not only illuminates the inner workings of K3s but also equips you with the knowledge to troubleshoot your deployments.

The Embedded Powerhouse ⚙️⚡​

K3s leverages go-bindata to embed essential Linux userspace binaries and manifests directly into its executable. This eliminates external dependencies and streamlines the deployment process. Within the K3s binary, you'll find core components like runc and containerd, along with the k3s-root tarball (e.g. k3s-root-amd64.tar). This tarball contains all the userspace binaries necessary for K3s to function, reducing the reliance on the host OS. If you would like all the K3s embedded binaries to take preference over the host OS binaries, you should use the --prefer-bundled-bin flag.

The embedded binaries are always deployed in the same directory: /var/lib/rancher/k3s/data. If you inspect this folder, you will notice it contains at least three subdirectories: cni, current and a long string of characters (or SHA). That long string of characters is generated when building K3s and it is the result of a sha256sum operation made on the tarball with the embedded binaries. As these change in each release, you will see a different string of characters for each release. In fact, after an upgrade, there will be two directories with a long string of characters as their name.

current is just a symlink to the SHA directory and cni includes different cni plugins that are also symlinks to the cni binary in the SHA directory. This is because we are building all cni plugins in just one binary using multi-exec tooling. This is again a way to be more efficient and less resource consuming. If your current K3s deployment underwent an upgrade process, you will see one extra directory called previous, which is another symlink to the previous SHA directory. For clarification this example:

$> ls -ahltr /var/lib/rancher/k3s/data/
total 24K
-rw------- 1 root root    0 Mar 19 06:22 .lock
drwxr-xr-x 4 root root 4.0K Mar 19 06:22 ..
drwxr-xr-x 4 root root 4.0K Mar 19 06:28 82142f5157c67effc219aeefe0bc03e0460fc62b9fbae9e901270c86b5635d53
lrwxrwxrwx 1 root root   90 Mar 19 06:28 previous -> /var/lib/rancher/k3s/data/82142f5157c67effc219aeefe0bc03e0460fc62b9fbae9e901270c86b5635d53
drwxr-xr-x 4 root root 4.0K Mar 19 06:30 b13851fe661ab93938fc9a881cdce529da8c6b9b310b2440ef01a860f8b9c3a9
lrwxrwxrwx 1 root root   90 Mar 19 06:30 current -> /var/lib/rancher/k3s/data/b13851fe661ab93938fc9a881cdce529da8c6b9b310b2440ef01a860f8b9c3a9
drwxr-xr-x 2 root root 4.0K Mar 19 06:30 cni
drwxr-xr-x 4 root root 4.0K Mar 19 06:40 .

Additionally, K3s includes embedded Helm charts and manifests for deploying critical services such as CoreDNS, Traefik, and local storage. These embedded charts, formatted as yaml files, can be found in the control plane nodes in the directory: /var/lib/rancher/k3s/server/manifests.

For 67MB our K3s binary includes a lot of stuff!

The Boot Sequence, Step-by-Step 👣​

Now that we established how K3s is carrying its embedded tools, we can explore the boot up sequence. Let us look at the typical logs you can find in journalctl when deploying a control-plane or K3s server instance. It all starts with:

Starting Lightweight Kubernetes...

And then:

/usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service

This part is checking for a network manager utility which must be disabled as described in the docs. It configures some parts of the network stack, specifically the routing tables, which conflict with Kubernetes networking, and that is why we verify if it was correctly disabled.

The next message should look familiar

Acquiring lock file /var/lib/rancher/k3s/data/.lock
Preparing data dir /var/lib/rancher/k3s/data/f8e9b5e7d85085972f4a9ddfd539d4dcf887be2e380a55f415c93cac5516dad5

When this message is shown, the directory where K3s deploys the embedded binaries has already been created. At this point, K3s will extract the binaries. We use the lock to avoid concurrent modifications, preventing K3s embedded commands like kubectl or ctr from executing and disturbing the K3s initialization.

The next block of logs point at the K3s version and the datastore. In this case, I am using the default datastore which means kine with sqlite. For more information on the different datastores available check this link

Starting k3s v1.32.2+k3s1
Configuring sqlite3 database connection pooling: maxIdleConns=2, maxOpenConns=0, connMaxLifetime=0s
Configuring database table schema and indexes, this may take a moment...
Database tables and indexes are up to date
Kine available at unix://kine.sock

Once the datastore is available k3s locks the bootstrap key. This step is useful for HA mode and this this key is just a placeholder so that other control-plane nodes do not start generating new CA certs. As we are not using HA mode in this example, this is not relevant.

Bootstrap key locked for initial create

K3s then generates all the TLS certificates required for the internal communications:

generated self-signed CA certificate
certificate CN=system:admin,O=system:masters signed by CN=k3s-client-ca@1742309831
certificate CN=system:k3s-supervisor,O=system:masters signed by CN=k3s-client-ca@1742309831
certificate CN=system:kube-controller-manager signed by CN=k3s-client-ca@1742309831
certificate CN=system:kube-scheduler signed by CN=k3s-client-ca@1742309831
certificate CN=system:apiserver,O=system:masters signed by CN=k3s-client-ca@1742309831
certificate CN=k3s-cloud-controller-manager signed by CN=k3s-client-ca@1742309831
generated self-signed CA certificate CN=k3s-server-ca@1742309831
certificate CN=kube-apiserver signed by CN=k3s-server-ca@1742309831
generated self-signed CA certificate CN=k3s-request-header-ca@1742309831
certificate CN=system:auth-proxy signed by CN=k3s-request-header-ca@1742309831
generated self-signed CA certificate CN=etcd-server-ca@1742309831
certificate CN=etcd-client signed by CN=etcd-server-ca@1742309831
generated self-signed CA certificate CN=etcd-peer-ca@1742309831
certificate CN=etcd-peer signed by CN=etcd-peer-ca@1742309831
certificate CN=etcd-server signed by CN=etcd-server-ca@1742309831
certificate CN=k3s,O=k3s signed by CN=k3s-server-ca@1742309831

And then saves the bootstrap data in the bootstrap key. Again, this is not relevant for this example:

Saving cluster bootstrap data to datastore

After that, K3s starts the different Kubernetes components. These components are all run within the k3s process as goroutines, which are lightweight, concurrent functions in Go, allowing for efficient resource usage. This is another design decision taken to reduce boot time and reduce resource consumption.

Running kube-apiserver
Running kube-scheduler
Running kube-controller-manager

Manifests for packaged components are extracted to /var/lib/rancher/k3s/server/manifests/. When all the different Kubernetes components are running and K3s initialization is ready, the deploy controller begins watching this directory and applies all the manifests. This is how components like CoreDNS or Traefik eventually get installed.

And that’s it, in a short period of time, you end up with a fully deployed and running Kubernetes distribution. 🎉

Conclusion 🏁​

This exploration has hopefully demystified some of the initial steps that enable K3s to materialize a fully functional Kubernetes cluster. By examining the logs, we've shed some light on the meaning behind each step, providing you with a deeper understanding of how K3s deploys in such a fast manner. We hope you find this knowledge useful to troubleshoot or at least to understand a bit deeper how K3s works.

Baseline HA Cluster 💻🖥️💻​

Whenever we get a question around HA, this is the cluster configuration I start with. It provides a solid foundation when deploying beyond a single server.

Our cluster will have:

• 4 nodes or VMs:
  • 1 load balancer
  • 3 servers
• A k3s-upgrade plan that will automatically update the cluster to the latest patch version of a given minor.

Cluster Setup 🌐🔧​

I'm using vagrant to provision 4 Ubuntu 24.04 VMs for this setup, all on a flat network. Setup of nodes is left as an exercise for the reader 😅.

My nodes are configured with the following names and IPs:

Load Balancer​

I'm using haproxy as it supports later expansion to multiple LB nodes (via keepalived).

SSH into the load balancer and install haproxy:

sudo apt install haproxy

The haproxy config is simple, just forward traffic to the servers:

#/etc/haproxy/haproxy.cfg
frontend k3s
    bind *:6443
    mode tcp
    default_backend k3s

backend k3s
    mode tcp
    option tcp-check
    balance roundrobin
    server server-0 10.10.10.50:6443 check
    server server-1 10.10.10.51:6443 check
    server server-2 10.10.10.52:6443 check

Restart haproxy to apply the config:

systemctl restart haproxy

Install K3s on first server​

On the first server, install K3s with embedded etcd and a known token:

curl -sfL https://get.k3s.io | INSTALL_K3S_CHANNEL=v1.31 sh -s - \
--cluster-init --token k3sblog --tls-san 10.10.10.100

We pass the --tls-san flag adds the load balancer IP as a Subject Alternative Name (SAN) for the certificate.

Join the other servers​

On the other servers, join the cluster via the load balancer:

curl -sfL https://get.k3s.io | INSTALL_K3S_CHANNEL=v1.31 sh -s - \
--server https://10.10.10.100:6443 --token k3sblog

Grab the kubeconfig​

Now that the cluster is up, we can grab the kubeconfig from the first server:

scp server-0:/etc/rancher/k3s/k3s.yaml k3s.yaml

Modify it to access the cluster via the load balancer:

sed -i 's/127.0.0.1/10.10.10.100/' k3s.yaml

No we can manage the cluster from our local machine:

export KUBECONFIG=$(pwd)/k3s.yaml
kubectl get nodes

Upgrade Plan 🏗️📝📐​

The plan I'm using will keep k3s updated to the latest patch version of the channel we give. In this case I'm using the v1.31 channel, the same channel used above. Kubernetes v1.31.4 just released at time of writing this post, so with this plan we have stable upgrades handled for the next 10-12 months (depending on how many patch releases this minor gets).

Install the system-upgrade-controller​

The upgrade plan is managed by the system-upgrade-controller. Install it:

kubectl apply -f https://github.com/rancher/system-upgrade-controller/releases/latest/download/system-upgrade-controller.yaml
kubectl apply -f https://github.com/rancher/system-upgrade-controller/releases/latest/download/crd.yaml

Create the upgrade plan​

#server-plan.yaml
apiVersion: upgrade.cattle.io/v1
kind: Plan
metadata:
  name: server-plan
  namespace: system-upgrade
spec:
  concurrency: 1
  cordon: true
  nodeSelector:
    matchExpressions:
    - key: node-role.kubernetes.io/control-plane
      operator: In
      values:
      - "true"
  serviceAccountName: system-upgrade
  upgrade:
    image: rancher/k3s-upgrade
  channel: https://update.k3s.io/v1-release/channels/v1.31

kubectl apply -f server-plan.yaml

See the automated upgrade docs for more details.

Conclusion 🚀​

We now have a high-availability cluster, accessible via a single IP. Upgrades are handled for the next year. This is a great starting point to:

• Add agent nodes to expand our workload capacity
• Add another load-balancer for additional redundancy

We will explore aspects of K3s, Kubernetes, and other related topics. These long form posts will be written by the K3s team and help illuminate aspects of the project that are not easily covered in the documentation.

Stay tuned for more posts in the future.