Your Model Scored 95% Accuracy. Is That Actually Good?

Why a high score can hide a useless model, and how to check against a simple baseline

Quick answer

A high accuracy score can hide a model that misses every important case.

Compare any model against a simple baseline, such as always predicting the most common answer.

Choose your metric based on the cost of mistakes, using precision and recall.

A model scores 95%. Your manager is happy. The team wants to ship it.

But before you celebrate, ask one simple question: better than what?

A high score means very little on its own. It only means something when you compare it with a simple alternative, and when the metric matches the real goal. This post shows you how to do both, using a small example you can work through in your head.

Why read this? Because a high accuracy score is one of the easiest ways to fool yourself, and interviewers love to test whether you notice.

Try this first

Imagine you have 100 messages. 95 are routine and 5 are urgent. You build a classifier that says "routine" for every single message.

Before reading on, answer these three questions:

What is its accuracy?
How many urgent messages does it catch?
Would you use it to detect urgent messages?

Write your answers down. Then keep reading and check them.

Check your thinking

Here's the answer:

Accuracy: 95 correct out of 100, so 95%.
Urgent messages caught: zero.
Would you use it? No. It looks great on paper, but it does the one job you need it for not at all.

This is the trap. The model is right most of the time because most messages are routine. It never does the thing you actually care about.

A simple rule like "always predict the most common answer" is called a baseline. It's a useful reference point, not a clever solution. If your fancy model can't beat it, something is wrong.

Follow the evaluation

When you judge a model, follow these steps in order:

Task objective
      ↓
Simple baseline
      ↓
Candidate model
      ↓
Relevant metrics and error inspection
      ↓
Decision based on consequences

Notice the order. You define the goal first, then the baseline, then the model. Starting with a model and hunting for a good number afterwards is how people end up with misleading results.

A second model

Now let's look at a different classifier. Here are its results, with "urgent" as the case we care about:

	Predicted urgent	Predicted routine
Actually urgent	4	1
Actually routine	6	89

Work out three things:

How many predictions were correct?
What share of the truly urgent messages did it catch?
Of the messages it flagged as urgent, what share really were urgent?

Give yourself a minute before scrolling on.

The worked answer

Accuracy: (4 + 89) out of 100, so 93%.

Recall: 4 out of 5 urgent messages caught, so 80%.

Precision: 4 out of 10 flagged messages were truly urgent, so 40%.

Look at what this means. This model's accuracy is lower than the always-routine baseline's 95%. Yet it catches four urgent messages where the baseline catches none.

It also creates six false alarms. Whether that trade-off is worth it depends on your situation. How many urgent messages would be costly to miss? How many false alarms can your reviewers handle? There's no universal answer, and that's the point.

The two metrics, in plain English

Precision asks: when the model raises a flag, how often is it right?

Recall asks: of all the real urgent cases, how many did the model find?

Neither one tells the whole story on its own. A model can have high recall by flagging almost everything, and it can have high precision by flagging very little. You need to look at both, and at the actual errors.

If you want a refresher on how these metrics fit into model evaluation more broadly, our free machine learning practice sets cover them step by step. The good score can hide data leakage post shows another way a strong score can mislead you.

An interview follow-up

Interviewers often ask: "Which metric would you optimise?"

Don't reach straight for the one you learned first. Start with the consequences:

If missing an urgent case is costly, recall usually matters most.
If false alarms waste your team's time or annoy users, precision matters more.
If both matter, look at how the two trade off, and pick a threshold that fits the business need.

A good answer shows you think about the cost of mistakes, not just the number.

A useful tool for baselines

Scikit-learn has a tool called DummyClassifier that makes simple baselines easy. One of its strategies predicts the most frequent class, which is exactly the trap in this post.

Using it is a quick way to check whether your added complexity earns its place. If your model can't beat the dummy, go back to the drawing board.

A quick checklist

Before you trust a score, ask yourself:

What does a simple baseline score on this same data?
Which mistake is more costly, a missed case or a false alarm?
Which metric reflects that cost?
Have I looked at the actual errors, not just the summary number?

A note on the example

The numbers here are made up to teach the reasoning. Five urgent messages are far too few to show how a real model will behave. Use this to practise thinking, not as proof that a model is ready to deploy.

Keep learning

Want to practise these ideas until they feel natural?

Free machine learning practice sets: https://roy-s-ai-lab.vercel.app/topics/machine-learning. Each answer comes with an explanation.
Overfitting vs underfitting, explained simply: https://roy-s-ai-lab.vercel.app/blog/overfitting-vs-underfitting-interview
Full guides and scenario packs: https://roy-s-ai-lab.vercel.app/products

Try explaining the 95% example to a friend in one minute. If you can say why the score looks good but the model is useless, you've got it.

Sources

Scikit-learn, "Metrics and scoring": https://scikit-learn.org/stable/modules/model_evaluation.html
Scikit-learn, "DummyClassifier": https://scikit-learn.org/stable/modules/generated/sklearn.dummy.DummyClassifier.html

The message counts, worked comparison and checklist are original teaching examples.

Ritesh Rai