Fraud is maybe 1% of transactions. Network attacks are a sliver of network traffic. Defective units are a small fraction of a production line. Almost every interesting classification problem is imbalanced — and almost every beginner tutorial pretends it isn’t.
Why accuracy lies to you first
If 99% of a dataset is the negative class, a model that predicts “negative” every single time scores 99% accuracy while catching nothing. This isn’t a hypothetical edge case — it’s the default failure mode of imbalanced classification, and it’s exactly what a model quietly learns to do when accuracy is the only thing being optimized and rewarded.
The fix isn’t a better model. It’s measuring the right thing in the first place.
Metrics that actually mean something
Three numbers matter more than accuracy on imbalanced data:
- Precision — of everything the model flagged as positive, how much was actually positive? Low precision means false alarms.
- Recall — of everything that was actually positive, how much did the model catch? Low recall means missed cases.
- Per-class breakdown — not one aggregate number, but precision and recall for every class separately, especially the rare ones.
A network intrusion detection classifier is a clean example of why this matters: on the NSL-KDD benchmark, both a binary (attack vs. normal) and a 5-way (attack-type) version are reported with precision and recall broken down per class, rather than a single headline accuracy figure. That breakdown is what shows whether the model is missing the rare attack types that actually matter — information a single accuracy number would hide completely.
Techniques that actually help
Resampling. Oversample the minority class (duplicate or synthetically generate examples, e.g. with SMOTE), undersample the majority class, or both. This changes what the model sees during training so it can’t just ignore the rare class.
Class weights. Most libraries let you penalize mistakes on the minority class more heavily during training, without touching the dataset itself. This is often the simplest first thing to try.
Threshold tuning. A classifier’s default 0.5 probability cutoff is arbitrary. Moving the decision threshold up or down trades precision for recall directly, and picking the right tradeoff depends entirely on what a false positive costs versus what a false negative costs in your specific problem.
Anomaly-detection framing. When the positive class is extremely rare, it can be more effective to model what “normal” looks like and flag deviations, rather than trying to directly classify a handful of positive examples.
What a false positive costs vs. what a false negative costs
The right tradeoff between precision and recall isn’t a modeling question — it’s a business question, and it’s different in every domain. A missed fraud case costs money directly. A missed defective unit on a production line reaches a customer. A missed political misstatement doesn’t get investigated. In every case, the cost of a false negative has to be weighed against the cost of a false positive — an analyst chasing a false alarm, a good transaction blocked, a healthy unit scrapped.
The visual quality inspection project makes this tradeoff concrete: a missed defect reaches a customer, while a false alarm wastes an inspector’s time on a good unit. Neither error is free, and treating them as equally costly is itself a modeling mistake on an imbalanced problem like defect detection.
Benchmark against something, not against nothing
A number with nothing to compare it to isn’t evidence of anything. The fact-check triage NLP project, trained on the real LIAR dataset of labeled political statements, benchmarks its results directly against published literature on the same dataset — which is what turns “this seems to work” into a checkable claim. On a new or unfamiliar dataset, even a simple majority-class baseline is worth reporting, because it’s the number your model actually has to beat.
A quick checklist
- Have you checked the class balance before trusting any accuracy number?
- Are you reporting precision and recall per class, not just one aggregate metric?
- Have you tried class weighting or resampling, and compared the result to the untouched baseline?
- Have you picked a decision threshold deliberately, based on what a false positive and false negative actually cost?
- Do you have something to benchmark against — a baseline, prior literature, or a simpler model?
FAQ
Is SMOTE always the right choice for imbalanced data?
No. SMOTE can introduce unrealistic synthetic examples, especially in high-dimensional or noisy data. Class weighting is often a safer first attempt, with resampling as a second step if that isn’t enough.
What’s a good baseline accuracy to compare against on imbalanced data?
The accuracy of always predicting the majority class — if your dataset is 95% negative, that baseline is 95% accuracy with zero recall on the positive class. Any model has to meaningfully beat that on recall, not just match it on accuracy.
Should I optimize for precision or recall?
Neither by default — it depends on which error costs more in your specific problem. Missed fraud and missed disease diagnoses usually favor recall; spam filters and content moderation often favor precision to avoid false alarms.
