Zero downtime: architecture for critical applications
How to deploy without downtime using Kubernetes, GitOps and Argo CD: rolling updates, graceful shutdown and database migrations at a financial services firm.
- Published
- Updated
- Reading time
- 7 min
api-gatewayProductionOverviewPodsLogargocd · syncedHealthy- several deploys a day, up from once a month
- Daily
- from commit to production, down from 5 days
- 45 min
- uptime in the first year
- 99.99%
- night work, down from 12 hours
- 0 hours
In short
- Zero downtime takes more than Kubernetes: rolling updates with maxUnavailable 0, graceful shutdown, correct health probes and GitOps with Argo CD.
- The expand-contract pattern lets you change a database schema without downtime, because old and new code stay compatible throughout the rollout.
- For the financial services client, deployments went from once a month to several a day, with 99.99% uptime in the first year.
Summary
How do you ship new code every day without users noticing? This case study shows the architecture it takes to avoid downtime in business-critical systems. The client is a financial services company that asked not to be named.
We go into the technical detail: Kubernetes configuration, database migrations and what the application itself has to do. The principles are the same whether your stack is PHP, Python, Go or Node.js.
Background: the fear of deploying
Where they started
The situation will be familiar to many: a large monolith that had grown unwieldy, and deploys that were manual, slow and risky.
Maintenance windows. Every deploy required planned downtime, usually late in the evening. Users expected the services to be available around the clock, so developers had to deploy at night.
Fear of deploying. Every deploy was big and risky, so the team put it off as long as possible. Changes piled up, merge conflicts multiplied, and each new deploy became riskier still.
Clumsy scaling. During traffic peaks the whole monolith had to scale up, even when only a small part of the system was under pressure.
The goals
The company set three requirements. The first two are DORA metrics for software delivery (DORA):
| Requirement | Goal |
|---|---|
| Deployment frequency | From monthly to daily |
| Change fail rate | Under 1% |
| True zero downtime | No interrupted sessions and no 5xx errors |
Strategy: hybrid CI/CD with GitLab and GitHub
The client has internal code that must stay closed, and also public SDKs and integrations. That's why we chose a hybrid setup.
GitLab for internal code
Internally the client runs GitLab Self-Managed, with code, CI pipelines, a container registry and security scanning (SAST and DAST) in one place.
Through the GitLab agent for Kubernetes, the platform team decides which projects may use the cluster. CI jobs authenticate with their job token, so no cluster credentials are stored in CI variables (GitLab).
GitHub for public code
Public SDKs and partner integrations live on GitHub, the standard for open source. GitHub Actions runs the public tests and publishes packages to npm, PyPI and Packagist.
Synchronisation
GitLab is the source of truth. Code is mirrored automatically to GitHub, so developers work in one place even though the code lives in two.
GitOps with Argo CD
Zero downtime starts with removing manual sources of error. When we moved the client to Kubernetes, we built the platform on GitOps from day one. Nobody changes the cluster by hand with kubectl apply; everything goes through Git.
Argo CD keeps the cluster in step with Git
Argo CD is a continuous delivery tool for Kubernetes. It continuously compares what's running in the cluster with what's defined in Git, and corrects any drift (Argo CD).
Pull instead of push. The CI server doesn't push changes into the cluster, which would require admin access to production. Argo CD pulls the changes from a separate manifest repo instead.
Security. The CI system never has direct access to production, only write access to the manifest repo. That removes a large attack surface.
From code to production
- Build (CI): A developer pushes code. The pipeline runs the tests, builds the Docker image and scans for vulnerabilities.
- Update (CD): If the build succeeds, the CI job updates the version in the manifest repo.
- Sync: Argo CD detects the change, works out the difference and rolls out the new version in a controlled way in Kubernetes.
- 1GitLab CIBuild
- Tests312/312
- Docker image142 MB
- SAST/DAST0 critical
- 2Manifest repoUpdate versioncommitted by CI, no access to the cluster- image: api:v3.4.1+ image: api:v3.4.2
- 3Argo CDSync
- OutOfSync detected14:32:38
- Sync started14:32:41
- Healthy14:32:49
- 4KubernetesRolling updatev3.4.2v3.4.2v3.4.2v3.4.24 of 4 pods ready, 0 dropped requests
How to deploy without downtime
With the setup below, new versions can be rolled out in the middle of the working day without losing a single request.
Rolling updates
By default, Kubernetes uses a rolling update with maxUnavailable: 25%, so up to a quarter of the pods can be down at once during a deploy (Kubernetes). A critical application can't afford that, so we set maxUnavailable to 0:
Starting point: four pods run v2.0 and take all traffic.
Step 1 of 5apiVersion: apps/v1kind: Deploymentmetadata: name: api-serverspec: replicas: 4 strategy: type: RollingUpdate rollingUpdate: maxSurge: 25% maxUnavailable: 0 template: spec: containers: - name: api image: registry/api:v2.1.0 ports: - containerPort: 8080maxUnavailable: 0 is the key. The number of available pods never drops below replicas. Kubernetes starts a new pod and waits until it's Ready before stopping an old one.
maxSurge: 25% lets the cluster use a little extra capacity for a short while, so the new pods come up quickly.
Graceful shutdown: how to avoid 502 Bad Gateway
The most common mistake when moving to Kubernetes is ignoring the application's lifecycle. When a pod is shut down, two things happen at the same time:
- Kubernetes marks the pod as terminating in EndpointSlices, so the load balancers should stop sending it traffic.
- Kubernetes sends SIGTERM to the container to stop the process.
The problem: Many servers exit as soon as they receive SIGTERM (Nginx treats it as a fast shutdown), often before kube-proxy, the ingress or the cloud load balancer has stopped routing to the pod. Requests then land on a pod that has just stopped, and the user sees "502 Bad Gateway". AWS describes this as a race condition and recommends a short wait in a preStop hook (AWS, LearnKube).
spec: terminationGracePeriodSeconds: 30 containers: - name: api lifecycle: preStop: sleep: seconds: 15When Kubernetes shuts the pod down, it is marked as terminating while the preStop hook runs. The hook does nothing but wait 15 seconds, and SIGTERM is only sent once it has finished.
The application keeps answering traffic throughout, but receives less and less as the load balancers update. The deadline in terminationGracePeriodSeconds covers the hook and the shutdown together, so with 30 seconds the app has 15 seconds to stop after SIGTERM. The built-in sleep action is stable from Kubernetes 1.34 and on by default from 1.30. On older clusters, use exec with sleep 15.
The hook must finish before SIGTERM is sent, and the deadline covers the whole shutdown (Kubernetes, Kubernetes 1.34).
Health probes: liveness and readiness
The liveness probe only checks that the process is alive. Keep it simple and don't let it check the database. If the database goes down, the probe fails in every pod at once, and Kubernetes restarts them in a loop that fixes nothing.
The readiness probe checks that the pod can take traffic, for example that the database responds and the cache is warm. If it fails, the pod is taken out of load balancing but keeps running.
The startup probe is for applications that take a while to start. Until it succeeds, Kubernetes holds off the other probes, so a slow start isn't mistaken for a dead process (Kubernetes).
spec: containers: - name: api startupProbe: httpGet: path: /health/live port: 8080 periodSeconds: 2 failureThreshold: 30 livenessProbe: httpGet: path: /health/live port: 8080 periodSeconds: 10 readinessProbe: httpGet: path: /health/ready port: 8080 periodSeconds: 5 failureThreshold: 3The liveness probe is deliberately simple and only checks that the process responds. The readiness probe checks that the application can actually take requests, including that the database responds.
The startup probe gives the application up to 60 seconds, 30 attempts two seconds apart, before Kubernetes gives up and restarts the container.
The database: the biggest challenge
Code is replaced on every deploy, but the data stays. During the rollout, old and new versions run side by side against the same schema. How do you change the schema without locking tables or making the old version fail?
The answer is the expand-contract pattern, also called parallel change (Danilo Sato on Martin Fowler's site, 2014).
Phase 1: expand
Should the address column become billing_address? We add the new column, keep the old one and roll out the code. Now both columns exist.
Phase 2: migrate
The application writes to both columns but reads from the new one. A background job copies the existing data across.
Phase 3: contract
Once we're sure every pod runs new code that uses billing_address, we drop the old column in a final migration.
| id | name | address | billing_address |
|---|---|---|---|
| 1 | Ola Nordmann | Storgata 1, 0155 Oslo | — |
| 2 | Kari Hansen | Bryggen 3, 5003 Bergen | — |
| 3 | Per Olsen | Torget 8, 7011 Trondheim | — |
A new column is added next to the old one. Old code notices nothing, and no table is locked.
Both versions have to work against the schema in every phase, because old and new pods run side by side during the rollout.
// Migration 1: expand with a new columnSchema::table('customers', function (Blueprint $table) { $table->string('billing_address')->nullable();}); // The model writes to both columns during the transitionclass Customer extends Model{ protected function address(): Attribute { return Attribute::make( get: fn (?string $value, array $attributes) => $attributes['billing_address'] ?? $value, set: fn (string $value) => ['address' => $value, 'billing_address' => $value], ); }} // Migration 2: contract and drop the old columnSchema::table('customers', function (Blueprint $table) { $table->dropColumn('address');});First we add the new column. Then the model writes to both columns and reads from the new one, with the old one as a fallback. Finally we drop the old column.
It takes discipline, but the database is never out of step with any of the versions running.
The application: ready for Kubernetes
Whatever the language, the application has to follow the twelve-factor principles for zero downtime to work.
Configuration and environment variables
In a cluster where pods come and go, you can't rely on .env files on disk. All configuration must come in as environment variables from Kubernetes ConfigMaps and Secrets.
- Node.js, Python and Go read straight from the environment (
process.envandos.environ). - Laravel reads the environment variables when
php artisan config:cacheruns. Run it when the container starts, not when the image is built, or the build environment's values end up in production. Once the configuration is cached,.envis no longer loaded (Laravel).
Queues and serialisation
Queued jobs (RabbitMQ, Redis or Kafka) are a trap many teams overlook. When a new version is rolled out, the queue may still hold jobs serialised with the old code structure. If a worker running new code picks up an old job, the job can fail as soon as it's deserialised.
The fix is versioned queues, or jobs that are always backwards compatible, much like expand-contract for the database. For breaking changes, drain the queue before you deploy. A Laravel worker that receives SIGTERM finishes its current job before exiting (Laravel), so set --timeout lower than terminationGracePeriodSeconds.
- Configuration in the environmentConfigMap · Secret
Never hardcode credentials or values that vary between environments. Everything comes in via ConfigMaps and Secrets.
- Stateless processes0 volumes
Every pod must be able to stop and be replaced at any time. No local state on disk.
- Port binding:8080
The application serves on its own port and does not depend on an external web server.
- Fast start, clean stopSIGTERM · 30 s
Starts quickly, shuts down in a controlled way and handles SIGTERM correctly.
Results and business value
The fear of deploying disappeared. Tuesday evenings went from on-call duty to time off.
Deployments went from once a month to several a day, and developers now ship small changes continuously. Time from commit to production fell from 5 days to 45 minutes. Uptime was 99.99% in the first year, even through major rewrites.
For comparison, the 2024 DORA report is the last one to group teams into performance tiers. Elite teams deploy on demand, often several times a day. They have lead times under one day, a 5% change fail rate and recover in under an hour. 19% of teams were at that level (DORA, 2024). The client is well within it, and the target of under 1% change fail rate is stricter than the elite level.
The biggest change, though, was cultural. The fear of deploying disappeared, and Tuesday evenings went from on-call duty to time off.
Key takeaways
You don't get zero downtime for free just by choosing Kubernetes. It takes a deliberate architecture with reliable CI/CD pipelines, infrastructure defined in Git (GitOps) and a good understanding of the application's lifecycle.
For companies that have to be available around the clock, a release process like this is a competitive advantage in itself.
We help companies improve how they release software. We've set up fault-tolerant environments for applications written in everything from PHP and Python to Go and Node.js, from containers and CI/CD to running and monitoring Kubernetes.
Sources and further reading
- Deployments, pod lifecycle and container lifecycle hooks, Kubernetes documentation
- Argo CD documentation
- Parallel change, Martin Fowler
- DORA's software delivery metrics and Accelerate State of DevOps 2024, DORA
- Load balancing, AWS EKS Best Practices, and Graceful shutdown in Kubernetes, LearnKube
- The Twelve-Factor App
- Accessors and mutators, Laravel documentation


