Not every model needs to answer instantly. Batch vs real-time inference is a decision that gets made too late in a lot of projects, usually after a serving architecture has already been built around an assumption nobody checked.
What separates the two
Batch inference scores a large set of inputs on a schedule, hourly, nightly, whatever fits, and writes the results somewhere for later use. Real-time inference answers a single request the moment it arrives, usually through an API. The difference isn’t really about the model. The same trained model can often serve either pattern. It’s about when the prediction is actually needed.
When batch is the right call
If nobody is waiting on an individual prediction the instant it’s generated, batch is usually simpler, cheaper, and easier to monitor. A dashboard that refreshes overnight doesn’t need a live-serving API sitting behind it. The retail analytics warehouse and Olist analytics engineering project both fit this pattern. Their dbt models run on a build schedule, and the resulting tables feed a Streamlit dashboard that’s read whenever someone opens it, not the instant new data arrives.
When real-time is actually necessary
Real-time inference earns its added complexity when a decision genuinely can’t wait. The support ticket triage platform needs to classify a ticket the moment it’s submitted, since routing it later defeats the point of triage entirely. The bike-share demand forecasting project is served through a containerized FastAPI endpoint for the same reason, an operator planning rebalancing routes needs a current forecast, not yesterday’s batch run.
The cost difference is real
Real-time serving means keeping infrastructure running and responsive at all times, with its own monitoring, latency requirements, and failure modes. Batch inference can fail and simply rerun on the next scheduled cycle, with far less operational overhead. Choosing real-time when batch would have worked adds ongoing cost and complexity for no actual benefit.
A middle ground: micro-batching
Some problems don’t need true real-time but can’t wait a full day either. Micro-batching, scoring in small batches every few minutes, splits the difference. It’s worth considering before defaulting straight to a fully real-time API when the actual requirement is closer to “fairly fresh” than “instant.”
A quick checklist
- Does a human or downstream system need this specific prediction the moment it’s generated, or would a scheduled refresh be just as useful?
- Have you estimated the operational cost of keeping a real-time endpoint running versus a scheduled batch job?
- Would micro-batching meet the actual freshness requirement without full real-time complexity?
- Is the current architecture matching an actual requirement, or an assumption made early and never revisited?
FAQ
Can the same model be used for both batch and real-time inference?
Yes, generally. The model itself doesn’t usually change, only how and when it’s invoked.
Is real-time inference always more expensive than batch?
Usually, since it requires infrastructure to stay available continuously, though the specific cost depends heavily on request volume and the serving setup.
How do I know if my use case actually needs real-time inference?
Ask whether a delay of an hour, or even a few minutes, would meaningfully change the outcome for whoever’s using the prediction. If not, batch is usually the simpler and cheaper choice.
