Evasion - Crafting Adversarial Examples That Flip the Prediction

The threat

A fraudulent charge lands on your bank’s fraud model and scores 0.92 - blocked. The attacker nudges three features by a hair each: the transaction amount, the time of day, the merchant category. Nothing a fraud analyst would blink at. The same charge now scores 0.11 - approved. The money moves. Your model never saw a new kind of fraud; it saw the same fraud shifted a few pixels past its decision boundary.

The same move breaks vision and text. Add a sticker pattern to a stop sign and an autonomy stack reads it as a speed-limit sign and accelerates into an intersection. Add three character typos to a slur and a content-moderation classifier scores the comment clean and publishes it. Evasion does not need your weights or your training data. It needs the label and confidence you already return, and a search for the smallest change that flips them.

How it works

Start from a real input the model gets right. Walk the input toward the decision boundary in tiny steps, querying the target after each one, until the label flips - then walk it back to shrink the perturbation while staying misclassified. The output is an input a human still reads as legitimate and the model reads as the other class.

• hopskipjump_evasion and boundary_evasion are decision-based - they need only the hard label, so they work even when the endpoint hides its probabilities.
• simba_evasion flips one coordinate at a time from the score, cheap on low-dimension inputs.
• square_evasion is Linf random search, the best choice for images.
• zoo_evasion estimates gradients from queries - powerful but query-hungry in high dimensions.
• For text, text_evasion flips the label with the fewest word edits; deepwordbug_evasion uses character typos; textfooler_evasion, pwws_evasion, bae_evasion, and textbugger_evasion rank words by importance and swap the ones that carry the prediction.

Run it

Point the attack at a real input the model classifies correctly. It returns the adversarial version and exactly how far it had to move.

import dreadnode as dn
from dreadnode.airt import boundary_evasion, textfooler_evasion

dn.configure(project="ml-evasion")

# Numeric (tabular / image): smallest L2 perturbation that flips the label.
result = await boundary_evasion(
    target,                    # your PredictionTargetSpec (fraud or digit predict API)
    original=one_real_input,   # a real input the model gets right
    num_classes=10,
    norm="l2",                 # "l2" or "linf"
    modality="image",          # renders original vs adversarial side by side
    max_queries=500,
).run()

print(result.success, result.distance_value, result.l0_changed)

# Text: flip a review's sentiment with importance-ranked word swaps.
result = await textfooler_evasion(
    target, original=one_review, num_classes=2, max_queries=200,
).run()

From the TUI:

Run a boundary model-evasion attack against my digit classifier at
http://localhost:8010/predict. Take a real "7" image and find the smallest L2
perturbation that changes its prediction. Report the L2 distance, how many
pixels changed, and whether it succeeded.

From the CLI:

# Image: decision-based search for the smallest perturbation that flips the label.
dn airt run-classifier --attack hopskipjump \
  --endpoint http://localhost:8010/predict --num-classes 10 --modality image

# Text: flip a review's sentiment with importance-ranked word swaps.
dn airt run-classifier --attack pwws \
  --endpoint http://localhost:8011/predict --num-classes 2 --modality text

The target must expose /pool, /members, and /nonmembers data helpers (as the demo targets do); pass --api-key if the endpoint requires an x-api-key.

Match the search to the target

Decision-based attacks (hopskipjump, boundary) work on hard-label-only endpoints. zoo_evasion gets expensive fast in high dimensions - for images, reach for square_evasion or the decision-based attacks instead. Text attacks that scan word importance need a max_queries at least as large as the token count of the input.

Read the finding

The finding answers one question: how small a change flips this model, and does the change stay invisible to a human?

• Attack success - did the label flip. This is the yes/no verdict.
• Perturbation distance (L2 / Linf) - how far the input moved. A tiny distance means the adversarial input is indistinguishable from the original.
• L0 changed - how many features or words moved. “Three features” or “two words” is the number that shows how surgical the attack was.
• Levenshtein edit distance - for text, the character-level edit count. Two typos flipping a moderation label is a headline.
• Original vs adversarial class - what it was, what it became.
• Query count - the economic cost of finding the flip.
• Distance-vs-query curve - how the perturbation shrinks as the attacker spends queries. A steep early drop means a cheap, clean adversarial example.

Image findings render the original and adversarial images side by side - the eye sees the same picture, the model sees two classes. Text findings show a word-level diff of exactly which tokens changed.

You are exposed when the label flips at a tiny perturbation distance and a low query count - an attacker crafts adversarial inputs your users and analysts cannot tell from real ones. You held when flipping the label requires a perturbation so large the input is visibly corrupted, or a query count your rate limits do not allow.

The defenses that move these numbers: adversarial training on perturbed examples, input preprocessing (feature squeezing, JPEG compression, quantization) to erase small perturbations, and detection of anomalous query sequences. Gradient masking looks like a defense but is not - decision-based attacks like hopskipjump and boundary ignore gradients entirely and cut straight through it.

Note the Score column above: it reports the perturbation distance in the norm that fits the modality - L2 for the tabular fraud and image digit targets, TOKENS for the text sentiment target. A HopSkipJump flip at L2 3.18 and a PWWS flip at a handful of token edits are both “the model was fooled cheaply.”