Fully automated canary deployments in Kubernetes

Blue/Green deployments are very powerful when it comes to easy rollbacks, but they are not the only approach for updating your Kubernetes application.

The small subset of live traffic to the new version acts as an early warning for potential problems that might be present in the new code. As our confidence increases, more canaries are created and more users are now connecting to the updated version. In the end, all live traffic goes to canaries, and thus the canary version becomes the new “production version”.

The big advantage of using canaries is that deployment issues can be detected very early while they still affect only a small subset of all application users. If something goes wrong with a canary, the production version is still present and all traffic can simply be reverted to it.

While a canary is active, you can use it for additional verification (for example running smoke tests) to further increase your confidence on the stability of each new version.

Unlike Blue/green deployments, Canary releases are based on the following assumptions:

If you cannot guarantee these two points, then blue/green deployments are a much better approach for safe deployments.

In a Kubernetes cluster without Istio, the number of canary pods is directly affecting the traffic they get at any given point in time.

So if for example you need your canary to get at 10% traffic, you need at least 9 production pods. With Istio there is no such restriction. The number of pods serving the canary version and the traffic they get is unrelated. All possible combinations that you might think of are valid. Here are some examples of what you can achieve with Istio:

This is why we recommend using Istio. Istio has several other interesting capabilities such as rate limiting, circuit breakers, A/B testing etc.

At Codefresh we have several customers that wanted to use Canary deployments in their pipelines but chose to wait until Istio reached 1.0 version before actually using it in production.

Even though we fully recommend Istio for doing canary deployments, we also developed a Codefresh plugin (i.e. a Docker image) that allows you to take advantage of canary deployments even on plain Kubernetes clusters (without Istio installed).

The canary deployments steps expect the following assumptions:

These requirements allow each canary deployment to finish into a state that allows the next one to run in a similar manner.

When you run a deployment in Codefresh, the pipeline step will print messages with its progress:

First, the Canary plugin will read the Kubernetes services and extract the “version” metadata label to find out which version is running “in production”. Then it will read the respective deployment and find the Docker image currently getting live traffic. It will also read the number of current replicas for that deployment.

Then it will create a second deployment using the new Docker image tag. This second deployment uses the same labels as the first one, so the existing service will serve BOTH deployments at the same time. A single pod for the new version will be deployed. This pod will instantly get live traffic according to the total number of pods. For example, if you have in production 3 pods and the new version pod is created, it will instantly get 25% of the traffic (1 canary, 3 production version).

Once the first pod is created, the script is running in a loop where each iteration does the following:

This goes on until only canaries get live traffic. The previous deployment is destroyed and the new one is marked as “production” in the service.

If at any point there are problems with canaries (or restarts), all canary instances are destroyed and all live traffic goes back to the production version.

You can see all this happening in real-time either using direct kubectl commands or looking at the Codefresh Kubernetes dashboard. While canaries are active you will see two docker image versions in the Images column:

Kostis Kapelonis is a software engineer/technical-writer dual class character. He lives and breathes automation, good testing practices and stress-free deployments.

Related posts:

As you might be aware of, sweating is good way to burn some calories and detox your body. But if you are injured, then how do you sweat? Or when you are unable to exercise? I really like to sweat in…

Another week on the books in this amazing journey, what started with a few lines of codes shared between 2 friends for a Hackaton, is slowly turning into something bigger than we all could have…

Talisman Wallet is one wallet for Polkadot & Ethereum that works with Ledger. It’s a safe & secure wallet trusted by many users to manage, stake, and send smoothly different crypto assets. ○ Press…