A model that scored well at launch can quietly get worse every month afterward, without a single line of its code changing. Model drift in machine learning is the reason, and most portfolio projects never mention it because they’re evaluated once and never revisited.
What model drift in machine learning actually is
Model drift in machine learning happens when the real-world relationship between inputs and outcomes shifts after a model is trained, so the patterns it learned stop matching reality. The model itself hasn’t changed — the world it’s predicting has moved on without it.
Two different kinds of drift
- Data drift: the distribution of incoming inputs changes — new customer segments, a new product category, seasonal shifts — even if the underlying relationship between features and outcome stays the same.
- Concept drift: the relationship itself changes — the same input now genuinely predicts a different outcome than it used to, which is the more dangerous and harder-to-catch form of model drift in machine learning.
A real example: why version tracking exists
The support ticket triage platform logs every model iteration in MLflow specifically so a retrained model can be compared against its predecessor before replacing it — that comparison is exactly how model drift in machine learning gets caught early: if a fresh retrain no longer beats the previous version by the margin it used to, something in the underlying pattern has likely shifted.
A real example: seasonal drift in forecasting
The bike-share demand forecasting project predicts hourly demand from weather and seasonal patterns — exactly the kind of model where drift is expected by design, not a failure. A model trained only on summer data will degrade predictably once winter patterns arrive, which is a textbook case of data drift rather than a bug to be fixed once and forgotten.
How to actually detect model drift
- Monitor prediction distributions over time. A sudden shift in what the model is predicting, even without new ground truth yet, is an early warning sign.
- Track live performance against ground truth as it arrives, not just the original test-set score — a model’s real-world accuracy is the only number that actually matters after deployment.
- Compare input feature distributions between training data and current production data — a growing mismatch signals data drift before it necessarily shows up in output quality.
- Set an explicit retraining trigger — a performance threshold or a fixed schedule — rather than retraining reactively only after something visibly breaks.
When retraining isn’t the right answer
Not every drop in performance means retrain immediately. Sometimes it means the pipeline itself broke — a feature stopped populating correctly, an upstream data source changed format. Retraining on broken input data doesn’t fix model drift in machine learning; it just teaches the model to be wrong in a new way. Confirming the input data is still healthy comes before deciding to retrain.
FAQ
How often should a model be retrained?
There’s no universal answer — it depends on how fast the underlying patterns change. A model over stable physical processes might go months without needing retraining; one over fast-changing consumer behavior might need it weekly.
Can model drift happen even if accuracy looks stable?
Yes — aggregate accuracy can hide drift that’s concentrated in a specific subgroup or class. Monitoring per-class or per-segment performance catches drift that an overall metric can mask.
Is model drift the same as data leakage?
No — data leakage inflates a score during evaluation, before deployment. Model drift is a real degradation that happens after deployment, as the world changes.
