A model that passed every test before deployment can still quietly fail six months later, without a single line of code changing. That’s what model monitoring in production exists to catch — problems that only show up after the world keeps moving and the model doesn’t.
Model monitoring in production: why testing once isn’t enough
Offline evaluation answers one question: did the model work on the data it was tested against, at that moment. Model monitoring in production answers a different one: is the model still working now, on the data actually arriving today — which can drift away from the training distribution in ways no pre-deployment test could have caught.
Data drift: the input distribution changes
The real-world inputs a deployed model sees can shift gradually — new customer behavior, a changed product catalog, a shifted user base. A model trained on last year’s patterns doesn’t automatically know the world moved, and its predictions can degrade quietly rather than failing loudly. Model monitoring in production means tracking the statistical properties of incoming data over time, not just watching for outright errors.
Concept drift: the relationship itself changes
Sometimes the inputs look similar, but the relationship between inputs and outcomes shifts — what predicted “high demand” last season might not this season. This is harder to catch than data drift because the input distribution can look perfectly normal while the model’s underlying assumptions have quietly stopped holding.
A real example: why deployment behind an API matters for monitoring
The bike-share demand forecasting project is served through a containerized FastAPI endpoint — a structure that makes monitoring straightforward, since every prediction request passes through one place where inputs and outputs can be logged and compared against expectations over time. A model only ever run manually in a notebook has no equivalent checkpoint to monitor at all.
A real example: why version tracking supports monitoring
The support ticket triage platform logs every model iteration in MLflow, which does double duty for monitoring: when a live model’s performance looks off, having a tracked history of prior versions makes it possible to compare current behavior against a known-good baseline, not just guess whether something changed.
What to actually track
- Prediction distribution over time — a sudden shift in the range or frequency of predicted classes/values is often the first visible sign something upstream changed.
- Input feature statistics — comparing incoming feature distributions against the training distribution surfaces data drift before it shows up in outcomes.
- Downstream outcome feedback, when available — the closest thing to ground truth for whether the model is still accurate, not just still running.
- Latency and error rates — not model quality directly, but a serving layer failing silently is just as damaging as a model that’s drifted.
A quick checklist
- Does your deployment log predictions somewhere you can actually review later, or does each request disappear once served?
- Do you have a way to compare current input data against the distribution the model was trained on?
- If performance degrades, can you tell whether it’s data drift, concept drift, or a pipeline bug — or would you just see “it got worse”?
- Is there a tracked history of prior model versions to compare current behavior against?
FAQ
How often should a deployed model be re-evaluated?
Depends on how fast the underlying data changes — some domains need weekly checks, others can go months. The right cadence comes from watching how quickly drift shows up in your specific data, not a fixed rule.
Is model monitoring in production only necessary for large-scale systems?
No — even a small deployed model benefits from basic prediction logging, since “is it still working” is a question every deployed model eventually needs answered.
What’s the simplest way to start monitoring a model with no existing infrastructure?
Logging every prediction request and its inputs to a simple database or file is enough to start — sophisticated drift-detection tooling can come later, once there’s actual data to analyze.
