Skip to content
CaseCloud and operations

Zero downtime: architecture for critical applications

How to deploy without downtime using Kubernetes, GitOps and Argo CD: rolling updates, graceful shutdown and database migrations at a financial services firm.

Stian BrendlienCEO at PXL
Published
Updated
Reading time
7 min
several deploys a day, up from once a month
Daily
from commit to production, down from 5 days
45 min
uptime in the first year
99.99%
night work, down from 12 hours
0 hours

In short

  • Zero downtime takes more than Kubernetes: rolling updates with maxUnavailable 0, graceful shutdown, correct health probes and GitOps with Argo CD.
  • The expand-contract pattern lets you change a database schema without downtime, because old and new code stay compatible throughout the rollout.
  • For the financial services client, deployments went from once a month to several a day, with 99.99% uptime in the first year.

Summary

How do you ship new code every day without users noticing? This case study shows the architecture it takes to avoid downtime in business-critical systems. The client is a financial services company that asked not to be named.

We go into the technical detail: Kubernetes configuration, database migrations and what the application itself has to do. The principles are the same whether your stack is PHP, Python, Go or Node.js.

Background: the fear of deploying

Where they started

The situation will be familiar to many: a large monolith that had grown unwieldy, and deploys that were manual, slow and risky.

Maintenance windows. Every deploy required planned downtime, usually late in the evening. Users expected the services to be available around the clock, so developers had to deploy at night.

Fear of deploying. Every deploy was big and risky, so the team put it off as long as possible. Changes piled up, merge conflicts multiplied, and each new deploy became riskier still.

Clumsy scaling. During traffic peaks the whole monolith had to scale up, even when only a small part of the system was under pressure.

The goals

The company set three requirements. The first two are DORA metrics for software delivery (DORA):

Requirement Goal
Deployment frequency From monthly to daily
Change fail rate Under 1%
True zero downtime No interrupted sessions and no 5xx errors

Strategy: hybrid CI/CD with GitLab and GitHub

The client has internal code that must stay closed, and also public SDKs and integrations. That's why we chose a hybrid setup.

GitLab for internal code

Internally the client runs GitLab Self-Managed, with code, CI pipelines, a container registry and security scanning (SAST and DAST) in one place.

Through the GitLab agent for Kubernetes, the platform team decides which projects may use the cluster. CI jobs authenticate with their job token, so no cluster credentials are stored in CI variables (GitLab).

GitHub for public code

Public SDKs and partner integrations live on GitHub, the standard for open source. GitHub Actions runs the public tests and publishes packages to npm, PyPI and Packagist.

Synchronisation

GitLab is the source of truth. Code is mirrored automatically to GitHub, so developers work in one place even though the code lives in two.

GitOps with Argo CD

Zero downtime starts with removing manual sources of error. When we moved the client to Kubernetes, we built the platform on GitOps from day one. Nobody changes the cluster by hand with kubectl apply; everything goes through Git.

Argo CD keeps the cluster in step with Git

Argo CD is a continuous delivery tool for Kubernetes. It continuously compares what's running in the cluster with what's defined in Git, and corrects any drift (Argo CD).

Pull instead of push. The CI server doesn't push changes into the cluster, which would require admin access to production. Argo CD pulls the changes from a separate manifest repo instead.

Security. The CI system never has direct access to production, only write access to the manifest repo. That removes a large attack surface.

From code to production

  1. Build (CI): A developer pushes code. The pipeline runs the tests, builds the Docker image and scans for vulnerabilities.
  2. Update (CD): If the build succeeds, the CI job updates the version in the manifest repo.
  3. Sync: Argo CD detects the change, works out the difference and rolls out the new version in a controlled way in Kubernetes.
A rollout of api-gateway from commit to four new pods.

How to deploy without downtime

With the setup below, new versions can be rolled out in the middle of the working day without losing a single request.

Rolling updates

By default, Kubernetes uses a rolling update with maxUnavailable: 25%, so up to a quarter of the pods can be down at once during a deploy (Kubernetes). A critical application can't afford that, so we set maxUnavailable to 0:

Rolling updatereplicas: 4maxSurge: 25%maxUnavailable: 0
Ready pods4 of 4 · never below 4

Starting point: four pods run v2.0 and take all traffic.

Step 1 of 5
Step through the rollout.
deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: api-server
spec:
replicas: 4
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 25%
maxUnavailable: 0
template:
spec:
containers:
- name: api
image: registry/api:v2.1.0
ports:
- containerPort: 8080
● Rolling update in KubernetesYAMLUTF-8LF

maxUnavailable: 0 is the key. The number of available pods never drops below replicas. Kubernetes starts a new pod and waits until it's Ready before stopping an old one.

maxSurge: 25% lets the cluster use a little extra capacity for a short while, so the new pods come up quickly.

Graceful shutdown: how to avoid 502 Bad Gateway

The most common mistake when moving to Kubernetes is ignoring the application's lifecycle. When a pod is shut down, two things happen at the same time:

  1. Kubernetes marks the pod as terminating in EndpointSlices, so the load balancers should stop sending it traffic.
  2. Kubernetes sends SIGTERM to the container to stop the process.

The problem: Many servers exit as soon as they receive SIGTERM (Nginx treats it as a fast shutdown), often before kube-proxy, the ingress or the cloud load balancer has stopped routing to the pod. Requests then land on a pod that has just stopped, and the user sees "502 Bad Gateway". AWS describes this as a race condition and recommends a short wait in a preStop hook (AWS, LearnKube).

graceful-shutdown.yaml
spec:
terminationGracePeriodSeconds: 30
containers:
- name: api
lifecycle:
preStop:
sleep:
seconds: 15
● preStop hook for graceful shutdownYAMLUTF-8LF

When Kubernetes shuts the pod down, it is marked as terminating while the preStop hook runs. The hook does nothing but wait 15 seconds, and SIGTERM is only sent once it has finished.

The application keeps answering traffic throughout, but receives less and less as the load balancers update. The deadline in terminationGracePeriodSeconds covers the hook and the shutdown together, so with 30 seconds the app has 15 seconds to stop after SIGTERM. The built-in sleep action is stable from Kubernetes 1.34 and on by default from 1.30. On older clusters, use exec with sleep 15.

The hook must finish before SIGTERM is sent, and the deadline covers the whole shutdown (Kubernetes, Kubernetes 1.34).

Health probes: liveness and readiness

The liveness probe only checks that the process is alive. Keep it simple and don't let it check the database. If the database goes down, the probe fails in every pod at once, and Kubernetes restarts them in a loop that fixes nothing.

The readiness probe checks that the pod can take traffic, for example that the database responds and the cache is warm. If it fails, the pod is taken out of load balancing but keeps running.

The startup probe is for applications that take a while to start. Until it succeeds, Kubernetes holds off the other probes, so a slow start isn't mistaken for a dead process (Kubernetes).

probes.yaml
spec:
containers:
- name: api
startupProbe:
httpGet:
path: /health/live
port: 8080
periodSeconds: 2
failureThreshold: 30
livenessProbe:
httpGet:
path: /health/live
port: 8080
periodSeconds: 10
readinessProbe:
httpGet:
path: /health/ready
port: 8080
periodSeconds: 5
failureThreshold: 3
● Liveness and readiness probesYAMLUTF-8LF

The liveness probe is deliberately simple and only checks that the process responds. The readiness probe checks that the application can actually take requests, including that the database responds.

The startup probe gives the application up to 60 seconds, 30 attempts two seconds apart, before Kubernetes gives up and restarts the container.

The database: the biggest challenge

Code is replaced on every deploy, but the data stays. During the rollout, old and new versions run side by side against the same schema. How do you change the schema without locking tables or making the old version fail?

The answer is the expand-contract pattern, also called parallel change (Danilo Sato on Martin Fowler's site, 2014).

Phase 1: expand

Should the address column become billing_address? We add the new column, keep the old one and roll out the code. Now both columns exist.

Phase 2: migrate

The application writes to both columns but reads from the new one. A background job copies the existing data across.

Phase 3: contract

Once we're sure every pod runs new code that uses billing_address, we drop the old column in a final migration.

Database schemacustomers
idnameaddressbilling_address
1Ola NordmannStorgata 1, 0155 Oslo—
2Kari HansenBryggen 3, 5003 Bergen—
3Per OlsenTorget 8, 7011 Trondheim—

A new column is added next to the old one. Old code notices nothing, and no table is locked.

Code running
v1 · oldRunning
reads addresswrites address
v2 · newRolling out
reads addresswrites both

Both versions have to work against the schema in every phase, because old and new pods run side by side during the rollout.

Pick a phase. The example renames the address column to billing_address.
expand-contract-migration.php
// Migration 1: expand with a new column
Schema::table('customers', function (Blueprint $table) {
$table->string('billing_address')->nullable();
});
// The model writes to both columns during the transition
class Customer extends Model
{
protected function address(): Attribute
{
return Attribute::make(
get: fn (?string $value, array $attributes) => $attributes['billing_address'] ?? $value,
set: fn (string $value) => ['address' => $value, 'billing_address' => $value],
);
}
}
// Migration 2: contract and drop the old column
Schema::table('customers', function (Blueprint $table) {
$table->dropColumn('address');
});
● Expand-contract in LaravelPHPUTF-8LF

First we add the new column. Then the model writes to both columns and reads from the new one, with the old one as a fallback. Finally we drop the old column.

It takes discipline, but the database is never out of step with any of the versions running.

The application: ready for Kubernetes

Whatever the language, the application has to follow the twelve-factor principles for zero downtime to work.

Configuration and environment variables

In a cluster where pods come and go, you can't rely on .env files on disk. All configuration must come in as environment variables from Kubernetes ConfigMaps and Secrets.

  • Node.js, Python and Go read straight from the environment (process.env and os.environ).
  • Laravel reads the environment variables when php artisan config:cache runs. Run it when the container starts, not when the image is built, or the build environment's values end up in production. Once the configuration is cached, .env is no longer loaded (Laravel).

Queues and serialisation

Queued jobs (RabbitMQ, Redis or Kafka) are a trap many teams overlook. When a new version is rolled out, the queue may still hold jobs serialised with the old code structure. If a worker running new code picks up an old job, the job can fail as soon as it's deserialised.

The fix is versioned queues, or jobs that are always backwards compatible, much like expand-contract for the database. For breaking changes, drain the queue before you deploy. A Laravel worker that receives SIGTERM finishes its current job before exiting (Laravel), so set --timeout lower than terminationGracePeriodSeconds.

Ready for the cloud12-factor4 of 4 met
  • Configuration in the environment

    Never hardcode credentials or values that vary between environments. Everything comes in via ConfigMaps and Secrets.

    ConfigMap · Secret
  • Stateless processes

    Every pod must be able to stop and be replaced at any time. No local state on disk.

    0 volumes
  • Port binding

    The application serves on its own port and does not depend on an external web server.

    :8080
  • Fast start, clean stop

    Starts quickly, shuts down in a controlled way and handles SIGTERM correctly.

    SIGTERM · 30 s

Results and business value

Deployments went from once a month to several a day, and developers now ship small changes continuously. Time from commit to production fell from 5 days to 45 minutes. Uptime was 99.99% in the first year, even through major rewrites.

For comparison, the 2024 DORA report is the last one to group teams into performance tiers. Elite teams deploy on demand, often several times a day. They have lead times under one day, a 5% change fail rate and recover in under an hour. 19% of teams were at that level (DORA, 2024). The client is well within it, and the target of under 1% change fail rate is stricter than the elite level.

The biggest change, though, was cultural. The fear of deploying disappeared, and Tuesday evenings went from on-call duty to time off.

Key takeaways

You don't get zero downtime for free just by choosing Kubernetes. It takes a deliberate architecture with reliable CI/CD pipelines, infrastructure defined in Git (GitOps) and a good understanding of the application's lifecycle.

For companies that have to be available around the clock, a release process like this is a competitive advantage in itself.

We help companies improve how they release software. We've set up fault-tolerant environments for applications written in everything from PHP and Python to Go and Node.js, from containers and CI/CD to running and monitoring Kubernetes.

Sources and further reading

Frequently asked questions

What is zero downtime deployment?

Zero downtime deployment (ZDD) means rolling out new versions while the application stays available throughout. With techniques such as rolling updates, blue-green deployment or canary releases, new versions go live without users noticing interruptions or errors.

Why am I getting 502 Bad Gateway during deployment?

It usually happens because Kubernetes stops the pod before the load balancer has stopped sending it traffic. The fix is a preStop hook that waits a little, often 15 seconds, and an application that shuts down cleanly on SIGTERM. Remember that terminationGracePeriodSeconds has to cover both the wait and the shutdown.

What's the difference between liveness and readiness probes?

The liveness probe only checks that the process is alive, and it shouldn't check the database. If it fails, Kubernetes restarts the pod. The readiness probe checks that the pod can take traffic, for example that the database responds. If it fails, the pod is taken out of load balancing but keeps running.

How do I migrate a database without downtime?

Use expand-contract: add new columns or tables without removing the old ones, have the code write to both and read from the new, move the existing data, and drop the old columns once every pod runs the new code. Old and new code then stay compatible throughout the rollout.

What is GitOps, and why should I use it?

GitOps means Git is the source of truth for infrastructure and configuration. Tools such as Argo CD continuously make sure the cluster matches what's in Git. You get full traceability and easy rollback, and the CI system needs no access to production.

What does it cost to implement zero downtime deployment?

It depends on your current infrastructure and how complex the application is. You need a Kubernetes cluster, CI/CD tooling such as GitLab or GitHub, and Argo CD. The biggest job is often adapting the application to the twelve-factor principles and training the team. The return comes as less downtime, fewer failed deployments and faster delivery.

Written by

Stian Brendlien

CEO at PXL

Stian has built and run web platforms for more than 18 years, including 13 years as CTO at Anbudstorget and five as contracted tech lead at GlobalConnect. He is a full-stack developer and the one you talk to about CI/CD, Kubernetes and architecture.

Want to deploy without downtime?

We set up Kubernetes, CI/CD and GitOps so you can deploy several times a day without users noticing. Get in touch for a no-obligation chat.

All articles