A newly retrained model looked better in every offline test. Then it shipped to everyone at once, and something the tests didn’t catch showed up in production. Blue-green vs canary deployment exists specifically to avoid finding that out the expensive way.
Blue-green: an instant, clean switch
Blue-green deployment runs two full environments in parallel, the current live version and the new one, fully staged and ready. Traffic switches from one to the other all at once. If something’s wrong, switching back is just as instant, since the old environment is still sitting there, ready. The tradeoff is that a problem, if one exists, affects all traffic the moment the switch happens.
Canary: a gradual, limited exposure
Canary deployment routes a small percentage of traffic to the new version first, watches how it performs, and gradually increases that percentage if things look healthy. A problem shows up on a small slice of traffic before it ever reaches everyone, which limits the blast radius of a bad model release considerably compared to switching all at once.
Why offline evaluation alone isn’t enough
The overfitting detection and baseline comparison practices covered elsewhere both happen before deployment, on historical data. Neither one catches a problem that only appears once a model meets live traffic patterns that didn’t exist in the training or test set. Blue-green and canary deployment are the layer that catches what offline evaluation structurally can’t.
A real example: where a version comparison already exists
The support ticket triage platform tracks every model iteration in MLflow, comparing a new version against its predecessor before trusting it with real tickets. That version tracking is exactly the foundation a canary rollout needs: a way to know precisely which version is serving which slice of traffic, and to roll back cleanly to a specific prior version if the new one underperforms once live.
Which to actually choose
- Blue-green suits situations where the new version has already been thoroughly validated and a fast, clean rollback matters more than gradual exposure.
- Canary suits situations with more uncertainty about how the model will behave on live traffic, where limiting exposure to a small percentage first is worth the added rollout complexity.
- Canary generally requires more infrastructure to route and monitor partial traffic split by version, which is a real cost blue-green avoids.
A quick checklist
- How confident are you that offline evaluation actually reflects live traffic patterns?
- Do you have infrastructure to route a percentage of traffic to a specific model version, or only an all-or-nothing switch?
- If something goes wrong after deployment, how quickly could you detect it and roll back?
- Is the added complexity of canary rollout worth it for this specific model’s risk profile?
FAQ
Is canary deployment always safer than blue-green?
It limits exposure to a smaller slice of traffic first, which generally reduces risk, but it also adds infrastructure complexity that blue-green avoids.
Do I need either of these for a small portfolio project?
Not strictly, but understanding the concept demonstrates awareness of a real production concern that offline model evaluation alone doesn’t address.
What metric should trigger a canary rollback?
Whatever the model’s core success metric is in production, error rate, latency, or a business outcome, monitored specifically on the canary slice compared to the stable version.
