A Good Score Can Hide Data Leakage

Quick answer

A high evaluation score can come from data leakage, not real skill.

Check that preprocessing learned only from the training data.

Check that every feature exists at the moment you would make the prediction.

For AI learners, career switchers and interview candidates. Suggested practice time: 10 to 15 minutes.

Your goal

By the end, you should be able to ask two important questions:

"Did preprocessing learn from the test data?"

"Would this information exist when the prediction is made?"

Those checks can reveal why an evaluation looks better than it should.

1. The core picture

Data leakage happens when model building uses information that would not be available at prediction time.

For example:

Learning preprocessing parameters from the test data.
Using a feature recorded only after the event being predicted.

The problem is not simply that the model has useful information. It is that the evaluation gives the model information it would not have in the situation it is meant to handle.

2. Your first task: choose the sequence

You need to standardise numeric features before training a model. Which sequence avoids the preprocessing leakage shown here?

Sequence A

All rows
   ↓
Fit the scaler
   ↓
Split into training and test data
   ↓
Train and evaluate

Sequence B

Split into training and test data
   ↓
Fit the scaler on training data only
   ↓
Transform training and test data
using the training-fitted scaler
   ↓
Train and evaluate

Before reading further, explain your choice in one sentence.

3. Check your thinking

Sequence B avoids this particular leakage.

A fitted scaler learns parameters from data. For example, StandardScaler learns feature means and scales during fitting. If you fit it on the entire dataset, the test portion influences those parameters.

Instead:

Fit preprocessing on training data.
Apply that fitted preprocessing to the test data.
Do not refit it on the test data.

The test data is still transformed. It just does not help fit the transformation.

4. See it in code

Illustrative scikit-learn example:

from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

model = make_pipeline(
    StandardScaler(),
    LogisticRegression(max_iter=1000)
)

model.fit(X_train, y_train)
predictions = model.predict(X_test)

Assumptions:

X_train and X_test were already split appropriately.
The features are suitable numeric inputs.
The data has already been checked for other leakage sources.

The scaler is fitted during training. Prediction applies the learned transformation to X_test without fitting the scaler again.

This code illustrates the pattern. It was not executed or benchmarked for this guide.

5. Your second task: the suspicious feature

You want to predict: "Will this customer cancel next month?"

You make the prediction on 1 October. Your dataset contains:

Account age.
Usage during September.
Support tickets created before 1 October.
Cancellation confirmation timestamp.

Which feature needs particular investigation, and why?

Do not decide only by the column name. Ask when the value becomes available.

6. Check your thinking

The cancellation confirmation timestamp is suspicious. If it is populated only after the cancellation being predicted, it reveals information from the future. You would not have that value when making the intended prediction.

1 October                         Later
Prediction made ─────────────────→ Cancellation confirmed
      │
      └── Use information available
          at this point

A feature existing in today's database does not prove that it existed at the historical prediction time. The other features also need a timing check. Their descriptions alone do not establish that they are safe.

7. A pipeline helps, but it is not magic

A pipeline keeps preprocessing and model fitting together. It does not automatically detect:

A feature created after the outcome.
Duplicate records crossing the split.
Inappropriate separation of related customers or entities.
A split that does not match the intended prediction setting.

You still need to understand how the data was produced.

8. Interview follow-up

"What changes when you use cross-validation?"

Preprocessing should be fitted separately within each training fold. It should not be fitted on the full dataset before cross-validation.

Using a pipeline inside cross-validation helps keep the transformation and model fitting within the appropriate folds.

A useful way to say it: "I would include the learned preprocessing inside the pipeline so each fold fits it using only that fold's training portion."

That addresses preprocessing leakage. It does not establish that the features or split strategy are otherwise appropriate.

9. Prove you understood it

Consider this sequence:

Fit scaler on all training rows
        ↓
Cross-validate the model
using the already-scaled rows

Is there still a concern? Pause before reading the answer.

Explained answer: yes. Although the held-out final test set may be untouched, the scaler used information from rows that later act as validation data inside cross-validation. Put the scaler and model together inside the cross-validation process.

The question is always: "Which data was allowed to influence fitting at this stage?"

10. Explain it aloud

Try answering without looking:

"Before trusting the score, I would check feature availability, preprocessing and the split strategy. Learned preprocessing should be fitted only on the appropriate training portion. I would also check that each feature exists when the intended prediction is made."

Use your own words. The reasoning matters more than memorising this sentence.

11. Try it on your own project

Choose three features and complete:

Feature	When does it become available?	Available at prediction time?
Feature 1	Your answer	Your answer
Feature 2	Your answer	Your answer
Feature 3	Your answer	Your answer

If you cannot establish the timing, investigate before treating the feature as valid.

Keep this limitation in mind

These examples teach a few common leakage patterns. They do not cover every way an evaluation can become misleading.

A pipeline is a useful implementation tool, not a certificate that the entire experiment is leakage-free.

Go one layer deeper

Scikit-learn: common pitfalls and recommended practices. Explains data leakage, fitting preprocessing on training data and avoiding test-data influence. https://scikit-learn.org/stable/common_pitfalls.html

Scikit-learn: pipelines and composite estimators. Explains how pipelines help prevent preprocessing leakage during cross-validation. https://scikit-learn.org/stable/modules/compose.html

Evidence note: the technical explanation follows scikit-learn's guidance. The customer scenario, diagrams and questions are original teaching examples. The code is illustrative and was not run for this guide.

Preparing for your next AI interview?

Strong preparation includes the fundamentals behind your results, not only the latest frameworks.

At Roy's AI Lab, I create practical assessments, scenario-based interview Q&A and structured preparation resources. Explore the guides at https://roy-s-ai-lab.vercel.app/products. Check each description for availability and pricing.

Created by Ritesh Rai, Roy's AI Lab. Understand it. Try it. Explain it.