Model Inversion - Reconstructing an Input From Confidence Scores

The threat

A company deploys a face-recognition API that takes an image and returns a confidence per identity. It returns nothing else - no images, no training data, just numbers. An attacker picks one target identity, feeds the API a blank image, reads the confidence, and nudges the pixels in the direction that raises it. Repeat a few thousand times and the numbers climb toward 1.0. When they stop, the image on the attacker’s screen is a recognizable face - the face of a specific person in the model’s training set, reconstructed from confidence scores alone.

The same attack reconstructs the representative input for any sensitive class. A diagnosis model that classifies by condition leaks a prototypical patient profile per condition. A model that recognizes individuals hands back the thing it was trained to recognize. This is MITRE ATLAS AML.T0024.000, and it needs nothing but the confidence you already return.

How it works

Pick a target class. Start from a neutral input and query the model. Move the input in the direction that raises the model’s confidence for that class, query again, repeat. The optimizer hill-climbs the confidence surface until it peaks - and the peak is the input the model most strongly associates with that class.

• confidence_inversion is MI-Face confidence hill climbing (Fredrikson et al. 2015) - it follows the confidence signal directly toward the peak.
• nes_inversion estimates the ascent direction with NES sampling, spending far fewer queries to reach the same reconstruction - the query-efficient choice against rate-limited endpoints.

Run it

Point the attack at a predict API and name the classes to reconstruct. With modality="image" the finding renders the reconstructed images directly.

import dreadnode as dn
from dreadnode.airt import confidence_inversion, nes_inversion

dn.configure(project="ml-inversion")

# Reconstruct a representative input per class by climbing the confidence surface.
result = await confidence_inversion(
    target,                        # your PredictionTargetSpec (face or diagnosis predict API)
    num_classes=10,
    input_dim=64,
    modality="image",              # renders the reconstructed images in the finding
    target_classes=[3, 7],         # which classes to reconstruct
    max_queries=5000,
).run()

print(result.mean_confidence, result.classes_reconstructed)

# Query-efficient variant: NES-estimated ascent, fewer queries for the same result.
result = await nes_inversion(
    target, num_classes=10, input_dim=64, modality="image", max_queries=2000,
).run()

From the TUI:

Run a confidence model-inversion attack against my face classifier at
http://localhost:8012/predict (10 classes, image input). Reconstruct classes 3
and 7 by climbing the confidence surface, render the reconstructed faces, and
report the mean reconstruction confidence and how many classes came out.

From the CLI:

# confidence climbs the confidence surface; nes is the query-efficient variant.
dn airt run-classifier --attack confidence \
  --endpoint http://localhost:8009/predict --num-classes 2 --modality tabular

The target must expose /pool, /members, and /nonmembers data helpers (as the demo targets do); pass --api-key if the endpoint requires an x-api-key.

Score the reconstruction

Pass reference_inputs to have the finding score how close each reconstruction lands to a real example of the class. A high reference similarity turns “we recovered something” into “we recovered a recognizable person.”

Read the finding

The finding answers one question: can someone turn your confidence scores back into the thing your model recognizes?

• Mean reconstruction confidence - how strongly the model recognizes its own reconstructions. Near 1.0 means the attack found the class prototype.
• Classes reconstructed - how many of the targeted classes came out, out of the total requested.
• Per-class reconstructions - one reconstruction per class. Image targets show thumbnails; other modalities show a feature preview.
• Reference similarity - if you passed reference_inputs, how close each reconstruction lands to a real member of the class.
• Reconstruction quality - the mean reference similarity across all classes. This separates “confidently classified” from “actually looks like training data”: a reconstruction can score high confidence while looking nothing like a real member, and this number catches that.
• Query count - the cost of reconstructing a class.

For image targets the finding renders the reconstructed images directly, so you see the recovered face or digit next to the reference. For tabular targets it shows the recovered feature vector.

You are exposed when reconstructions reach high confidence and, given a reference, high similarity - your model hands back recognizable inputs for sensitive classes to anyone who can query it. You held when reconstructions stay low-confidence noise, or the query cost to reach a recognizable reconstruction exceeds what your rate limits allow.

Mitigations

Apply these in order, cheapest and highest-leverage first:

1. Coarsen the output. Return the top-1 label or a bucketed confidence, not a precise per-class probability. The attack climbs the exact confidence surface you expose, so removing it removes the signal.
2. Never expose per-class probabilities for classes that map to individuals. A per-person confidence score is a reconstruction target.
3. Train with differential privacy (DP-SGD). Calibrated gradient noise blurs the per-class prototype the attack recovers, at some accuracy cost.
4. Regularize against memorization. Dropout, weight decay, and early stopping flatten the sharp per-class peaks inversion depends on.
5. Rate-limit and monitor. Reconstruction needs many adaptive queries against the same class; flag bursts of similar probes and throttle them.
6. Partition high-risk classes. Keep models that score individuals off public endpoints, behind an authenticated gateway that serves only sanitized output.