A random 80/20 train/test split works for most machine learning problems. For time series, it quietly cheats — and the accuracy number it produces is often meaningless.
Time series cross-validation: why order matters
A random split can put a Tuesday from three weeks from now in the training set and a Monday from last week in the test set. The model ends up training on the future to predict the past. In production, you will never have next month’s data available to predict today — so a validation setup that allows it is testing something that can’t happen in reality.
What proper time series cross-validation looks like
The fix is simple in concept: training data must always come before test data in time. Two common approaches:
- Expanding window. Train on everything up to a point, test on the next chunk, then expand the training window forward and repeat. Every fold respects chronological order.
- Rolling window. Keep the training window a fixed size and slide it forward, rather than always expanding — useful when older data is less relevant to current patterns.
Both are meaningfully different from k-fold cross-validation’s random shuffling, and libraries like scikit-learn’s TimeSeriesSplit implement them directly.
Where this actually matters: a real example
The bike-share demand forecasting project uses honest time-series splits specifically so its reported hourly accuracy reflects real forecasting performance, rather than a number inflated by letting the model peek at patterns from dates after the ones it’s supposedly predicting. SHAP is then used on top of that honestly-validated model to explain which factors drive each hour’s demand — an explanation is only trustworthy if the model behind it was evaluated correctly in the first place.
The tell that a time series result is inflated
A time series model that performs suspiciously well compared to naive baselines — like “tomorrow will look like today” — is worth double-checking for a leaked random split before trusting the number. If a model can’t beat a naive lag-based baseline once evaluated correctly, that’s real information, not a failure to hide.
A quick checklist
- Does your train/test split respect chronological order, or was it shuffled randomly?
- Would your validation setup ever be possible in production, where future data genuinely isn’t available yet?
- Have you compared your model against a naive baseline (yesterday’s value, last week’s average) evaluated the same honest way?
- If you’re using k-fold cross-validation on time series data, have you replaced it with TimeSeriesSplit or an equivalent?
FAQ
Can I ever use a random split for time series data?
Only if you’re confident there’s no temporal dependency in what you’re predicting — rare for real forecasting problems. When in doubt, respect chronological order.
How many folds should time series cross-validation use?
Enough to cover meaningfully different time periods — 3 to 5 expanding-window folds is common, more if you have enough historical data and want tighter confidence in the result.
Does time series cross-validation apply to non-forecasting time-indexed data?
Yes — any dataset where order carries information (sensor logs, user session sequences) benefits from the same chronological-split discipline, not just explicit forecasting tasks.
