Bias and Shortcut Tests for Vision Models: A Practical Test Suite From Real-World Experiments

When I first started working with deep learning image models, I did what most people do:

• Train a model
• Check top-1/top-5 accuracy
• Look at a few confusion matrices

On paper, everything looked great. But when I started poking at the models in slightly weird ways, I learned a very uncomfortable truth:

My models were often "right" for the wrong reasons, i.e., latching onto background textures, dataset noise, and unintended shortcuts instead of the actual object I cared about.

In this article, I'll walk through the bias and shortcut tests I now run by default on any serious image model. These are based on patterns I've seen repeatedly in my own experiments. We'll cover:

• What I mean by bias and shortcut learning in image models
• How to build group fairness tests (per-subgroup performance, intersectional analysis)
• A set of datasets and shortcut tests that deliberately attack background reliance and hidden noise, including:
  • Transform-based tests (blur, frequency, etc.)
  • Background-only crop tests
  • Grid/pixel-shuffling tests
• How to bundle these into a repeatable test suite and integrate them into your normal model workflow

This is not a theoretical fairness survey. It's a set of things you can run this week on your models and datasets.

Accuracy Isn't Enough (What I Saw in Practice)

The first time I realized something was off was when I started doing "stress tests" on a model that was performing extremely well on a standard test set. I did things like:

• Cropping out most of the object
• Blurring the image heavily
• Shuffling parts of the image

Surprisingly, the model still predicted the correct class much more often than I expected, even when the object was almost invisible or the image was visually meaningless to me. 

At this point, I knew:

• The model was not really "understanding" the object
• It was memorizing background patterns, textures, and hidden characteristics in the dataset

Since then, I've broken down my evaluations into two main areas:

1. Group bias: How performance varies across human subgroups?
2. Shortcut bias: How much does the model rely on context or noise that is irrelevant?

We need to test both.

Two Types of Bias Worth Testing

1. Group Bias (Subgroup Fairness)

This is the classic notion of fairness: the model behaves differently for different subgroups or cases.

For example:

• A face detector that works better on certain skin tones
• A classifier that mislabels specific demographic groups more often
• A medical imaging model that underperforms on scans from a certain population or device type

Here's How To Test

You need an evaluation dataset in which each example includes the task label (class, bounding box) and one or more attributes you care about (skin tone, gender, device type, etc.). Assuming your dataloader gives you a batch with:

• batch["image"] – image tensor
• batch["label"] – ground truth class
• batch["group"] – something like male_light, scanner_A, etc.

import torch
from collections import defaultdict

def evaluate_by_group(model, dataloader, group_key="group", device="cuda"):
    model.eval()
    group_stats = defaultdict(lambda: {"correct": 0, "total": 0})

    with torch.no_grad():
        for batch in dataloader:
            images = batch["image"].to(device)
            labels = batch["label"].to(device)
            groups = batch[group_key]  # list/array of strings

            logits = model(images)
            preds = torch.argmax(logits, dim=1)

            correct = (preds == labels).cpu().numpy()

            for g, is_correct in zip(groups, correct):
                group_stats[g]["total"] += 1
                group_stats[g]["correct"] += int(is_correct)

    results = {}
    for g, stats in group_stats.items():
        acc = stats["correct"] / max(1, stats["total"])
        results[g] = {"accuracy": acc, **stats}
    return results

You can then: 

• Print accuracy per group
• Compute the gap between best and worst group (e.g., max(acc) − min(acc))
• Decide on thresholds: e.g., "no subgroup should be >X% below the best-performing subgroup."

This gives you a fairness report instead of a single global accuracy. 

2. Shortcut Bias Tests: Transforms, Background-Only, and Shuffling

Group fairness tells you who the model may be failing on. Shortcut tests tell you why it might be performing well for the wrong reasons. These tests are my favourite, and over time, I've settled on three categories I like to test:

1. Transform-based tests – blur, frequency, etc.
2. Background-only crop tests – feed only a small piece of the background
3. Grid/pixel-shuffling tests – scramble images into visual nonsense

Transform-Based Tests

The easiest thing to try is a heavy blur. If you blur away all object edges and fine details, a human should struggle. If your model still performs very well, it may be over-relying on coarse backgrounds or color patterns. 

import torch
import torch.nn.functional as F

def heavy_blur(img, kernel_size=31):
    # img: (C, H, W) tensor in [0,1]
    img = img.unsqueeze(0)  # add batch
    kernel = torch.ones(1, 1, kernel_size, kernel_size, device=img.device)
    kernel = kernel / kernel.sum()

    # depthwise conv per channel
    blurred = F.conv2d(
        img,
        kernel.expand(img.size(1), -1, -1, -1),
        padding=kernel_size // 2,
        groups=img.size(1),
    )
    return blurred.squeeze(0)

You can plug other transforms into this pattern too: 

• Keep only low-frequency Fourier components
• Apply aggressive median filters
• Zero out high-frequency components, and so on

The idea is to systematically remove semantic content and see how your model reacts.

Background-Only Crop Test: Is the Object Even Needed?

One of the most revealing experiments I've done is what I call the background patch test:

1. Take an image where you know the object of interest (e.g., the dog, the tumor). 
2. Crop a small patch from the background, ideally outside the object region. 
3. Paste that patch onto an otherwise blank canvas.
4. Run the model on this patch-only image.

If the model still predicts the correct class at a surprisingly high rate, that's a huge red flag. It means the model can identify the class from background cues alone, without even looking at the object of interest.

import random
import torch

def background_patch(img, patch_size=64, extract_top_left=False):
    C, H, W = img.shape

    if extract_top_left:
        top, left = 0, 0
    else:
        top = random.randint(0, max(0, H - patch_size))
        left = random.randint(0, max(0, W - patch_size))

    patch = img[:, top:top+patch_size, left:left+patch_size]

    # Blank canvas
    canvas = torch.zeros_like(img)
    ph, pw = patch.shape[1], patch.shape[2]

    # Center placement
    c_top = (H - ph) // 2
    c_left = (W - pw) // 2
    canvas[:, c_top:c_top+ph, c_left:c_left+pw] = patch

    return canvas

Grid/Pixel-Shuffling Test: Memorized Noise

Another experiment that has surprised me more than once is shuffling the image into nonsense. The image becomes unreadable to a human, but the model sometimes retains more accuracy than you'd expect by pure chance.

This suggests the model has memorized dataset-specific patterns (global texture, color distributions, noise) that persist after shuffling.

def grid_shuffle(img, grid_size=4):
    # img: (C, H, W)
    C, H, W = img.shape
    gh, gw = grid_size, grid_size
    h_step, w_step = H // gh, W // gw

    patches = []

    for i in range(gh):
        for j in range(gw):
            h0, h1 = i * h_step, (i + 1) * h_step
            w0, w1 = j * w_step, (j + 1) * w_step
            patches.append(img[:, h0:h1, w0:w1])

    random.shuffle(patches)

    shuffled = torch.zeros_like(img)
    idx = 0
    for i in range(gh):
        for j in range(gw):
            h0, h1 = i * h_step, (i + 1) * h_step
            w0, w1 = j * w_step, (j + 1) * w_step
            shuffled[:, h0:h1, w0:w1] = patches[idx]
            idx += 1

    return shuffled

How I Interpret These Numbers

Some rules of thumb I've learned:

• Background-only patch accuracy is surprisingly high.
  • The model is clearly using background/context as the primary signal. Your training set probably has strong class-background correlations.
• Grid-shuffle accuracy above random chance by a wide margin.
  • The model is exploiting global statistics or memorized noise patterns that survive shuffling. It's learning "style" rather than "content." 
• Heavy-blur accuracy barely drops.
  • Edges and object details are not very important to your model; backgrounds and low-frequency content are doing a lot of work. 

These tests don't give you one magic fairness number, but they expose where the model is cheating.

How I Treat Shortcut Bias

When I see these issues, I treat them like any other bug in the system and need to be fixed at the data and training level. 

• I try to rebalance and enrich my datasets for underrepresented groups, randomize or diversify backgrounds, so the shortcut stops working.
• In some cases, I use augmentations that deliberately break spurious cues (e.g., background swaps, cutout, style/texture jitter).

Closing Thoughts

Deep image models are extremely good at finding shortcuts: backgrounds, textures, noise patterns, anything that gives them an easy win during training.

If you don't test for it, you'll likely ship models that:

• Looks great on an overall test metric
• But behave very differently across subgroups
• And rely heavily on spurious background cues or memorized noise

By combining:

• Per-subgroup fairness metrics, and
• Shortcut tests like heavy blur, background-only crops, and grid/pixel shuffling,

You can turn "bias and fairness" into something that looks a lot more like traditional engineering.

That's been my experience: once I started running these kinds of tests, I couldn't un-see how many models were "right for the wrong reasons," and I stopped trusting accuracy alone.

Learned something new? Tap that like button and pass it on!