A single train/test split works fine when you have a hundred thousand rows. With a few hundred, that same split can hand you a lucky or unlucky test set almost at random, and the reported score barely means anything.
The problem with a single split on small data
Split a small dataset once, and the specific rows that land in the test set matter a lot. A different random seed can shift the reported accuracy by several points, not because the model changed, but because the test set happened to be easier or harder that time. Cross-validation for small datasets exists to average that luck away.
How k-fold cross-validation helps
Instead of one split, k-fold cross-validation splits the data into k parts, trains on k-1 of them, tests on the remaining one, and repeats until every part has served as the test set once. The final score is an average across all k runs. That average is far more stable than any single split, which is exactly what small datasets need most.
A real example: benchmarking against literature instead of a shaky split
The fact-check triage NLP project benchmarks its results against published literature on the LIAR dataset. That comparison matters more on a dataset this size than it would on a massive one, since a single split’s noise could otherwise make a mediocre model look strong, or a solid one look weak, purely by chance.
Why more folds isn’t automatically better
Leave-one-out cross-validation, the extreme case where k equals the number of rows, uses almost all the data for training on every fold. That sounds appealing for small datasets, but it’s computationally expensive and can produce a high-variance estimate of its own. Five or ten folds is the usual practical range, balancing stability against compute cost.
Stratification matters even more when data is scarce
With a small dataset, an unlucky fold can end up with almost none of a minority class by chance. Stratified k-fold cross-validation keeps the class balance roughly consistent across every fold, which prevents a fold from accidentally testing on a class distribution that looks nothing like the training data.
A quick checklist
- Is your dataset small enough that a single split’s luck could meaningfully change the reported score?
- Are you using stratified folds if your classes are imbalanced, even on a small dataset?
- Have you reported the variance across folds, not just the average, so a reader can judge how stable the result actually is?
- Is there a published benchmark on the same or a similar dataset you can compare against, the way the LIAR-based project does?
FAQ
How small does a dataset need to be before cross-validation matters more?
There’s no hard cutoff, but once a single test set drops below a few hundred examples, its score starts becoming noticeably noisy from split to split.
Is 5-fold or 10-fold better for small datasets?
10-fold uses more data for training on each fold, which can help when data is scarce, at the cost of more compute. 5-fold is a reasonable default when compute time matters more.
Does cross-validation replace the need for a separate test set?
Not entirely. A common pattern is cross-validation during model development and tuning, with one final untouched test set reserved for a last, honest check.
