From 2-hour manual releases to 15-minute deploys

The situation

I was a founding engineer on a loan origination platform for regional banks, used to process more than $700M in loans a year. Each bank had around 100 people working in the system every day. I was also the primary on-call engineer for production, so when something went wrong, I got the call.

The problem

Every release was a manual deployment that took about two hours. Each one tied up an engineer and risked downtime for banks that were in the system all day. Releases that cost that much get put off and batched together, and a bigger release is a riskier one.

What I did

• Moved the platform to GitOps: the desired state of every environment lives in Git, and ArgoCD keeps Azure Kubernetes Service in line with it. A release became a reviewed, merged change instead of a person running steps by hand.
• Defined the infrastructure in Terraform, so environments were built the same way every time and every change was reviewed.
• Added canary releases and autoscaling behind an Istio service mesh, so a new version took a slice of traffic first and could be rolled back before most users saw it.
• Set up observability: centralized logs, host and service monitoring with Zabbix, and alerting, so problems showed up before a bank called. I also added Snyk dependency scanning to the build pipeline.

The result

Releases went from a two-hour manual process to fifteen minutes with zero downtime, and the platform held 99.99% uptime. Shipping a fix stopped being an event that needed a maintenance window and an engineer's afternoon.

Need something like this? See the Pipeline build package.