Model Evasion

Model Evasion is an adversarial attack aimed at bypassing or evading a machine learning model's defenses, usually to make it produce incorrect outputs or behave in ways that favor the attacker. In this context, the adversary doesn't try to "break" the model or extract data from it (like in model inversion) but instead seeks to manipulate the model's behavior in a way that allows them to achieve a desired outcome, such as bypassing detection systems or generating misleading predictions.

Quick Overview & Read More

The essence of model evasion is to create inputs that are specifically crafted to confuse the model and exploit vulnerabilities in how it processes data. Inputs are near, outputs are far.

To learn more about model evasion, explore our paper stack for research papers that discuss this attack type.

How Model Evasion Works#

1. Understanding Model Behavior#

Whether it’s a classification, regression, or recommendation model, knowing the model’s decision-making process is key to designing an effective evasion strategy. The attacker must first study the model to understand how it makes decisions. This often involves observing the model’s output in response to a range of inputs.

Model Type	Task	Decision Process	Output
Classification	Predict discrete classes (e.g., spam or ham)	Map input features to classes based on learned patterns	Class label (e.g., spam, ham)
Regression	Predict continuous values (e.g., house price)	Learn a mapping from features to continuous target values	Continuous value (e.g., $350,000)
Recommendation	Predict user preferences or interests	Learn user preferences based on past behavior or item characteristics	Ranked list of recommended items

Each type of model tailors its decision-making process to the specific needs of the task: classification makes categorical decisions, regression makes continuous predictions, and recommendation models suggest items based on user preferences and past behavior.

2. Crafting Adversarial Inputs#

Once the model behavior is understood, the attacker creates adversarial inputs. These inputs are subtly modified from normal data points, often in ways that are imperceptible to humans but cause the model to misclassify or make incorrect predictions. For instance:

Image-based models: The attacker might slightly alter pixel values in an image, causing the model to misclassify it (for example, changing specific pixels on a picture of a dog so it crosses a model’s decision boundary).
Text-based models: For language models, attackers can craft text inputs or manipulate word structure to alter model interpretation and output.

The goal is to cause the model to make a decision that benefits the attacker, like misclassifying harmful content as safe or bypassing security systems.

3. Exploiting Vulnerabilities in Model Training#

Model evasion attacks often exploit weaknesses in model training. Alternatively, some models are more vulnerable to adversarial examples because they lack robustness or fail to generalize well from the training data. This can happen for reasons such as:

Overfitting and underfitting the model
Poor data quality and quantity
Lack of regularization
Striking the right balance between bias (error due to oversimplification) and variance (error due to excessive complexity).
Real-world data shifts
Lack of proper evaluation

In some cases, even small changes in the input can cause dramatic shifts in model behavior.

Real-World Example: Evasion of a Spam Filter#

Imagine an attacker wants to bypass a spam detection model used by an email service. The attacker’s goal is to send a message that appears legitimate but is actually spam content. Here's how the attacker might perform the evasion:

The attacker tests a sample message (e.g., "Buy cheap medications now!") and observes that it is correctly flagged as spam by the model.
The attacker modifies the message by slightly changing the wording or adding special characters that make the model misinterpret the input, such as "B#uy cHeap med!cat!ons now" or embedding invisible characters that aren’t visible to the human eye but confuse the model.
The modified message may now be classified as legitimate and delivered to the user, evading model detection.

This kind of evasion could cause the spam filter to miss many harmful messages or allow malicious content to slip through.

Example Query#

Original Message: "Buy cheap medications now!" → Model Response: {message:pass, action:deliver}
Modified Message: "Buy cheap medicat!ons now" → Model Response: {message:fail, action:block}

The model now misclassifies it as non-spam.

By changing or evading detection in subtle ways, the attacker can bypass model safeguards.

Security Implications of Model Evasion#

Model evasion can have significant security consequences, particularly in sensitive systems. Some of the major risks include:

Bypassing security systems: If adversarial inputs are used to evade detection models (e.g., spam filters, malware detectors, intrusion prevention systems), attackers can successfully bypass these safeguards. This could lead to harmful content, malicious attacks, or other security breaches.
Deceptive inputs: Attackers can use model evasion to manipulate systems into making incorrect decisions. This can be particularly dangerous in high-stakes systems like self-driving cars, facial recognition systems, or fraud detection models, where even small errors in predictions could have serious consequences.
Financial losses: In systems that rely on predictions for financial decisions (e.g., credit scoring or fraud detection), evasion attacks could lead to erroneous approvals or rejections of transactions, resulting in financial losses.
Ethical concerns: If model evasion is used in systems like hiring algorithms or legal predictions, it could lead to unfair or discriminatory outcomes that could harm individuals or groups, particularly marginalized ones.

Defending Against Model Evasion#

Robustness and Regularization#

To defend against evasion attacks, models should be robust enough to handle small perturbations. Regularization techniques such as adversarial training can be used, where the model is trained on both regular and adversarially modified inputs, allowing it to learn how to handle perturbed data and identify evasion attempts.

Adversarial Training: In this approach, the model is trained not just on regular data but also on adversarial examples, making it more capable of correctly classifying these modified inputs during inference.

Input Sanitization#

For systems that interact with untrusted or unknown inputs (e.g., image classifiers, text-based models), input sanitization can be applied. This involves filtering or modifying the input before it reaches the model to remove or reduce harmful modifications.

Text Processing: Techniques such as tokenization or pattern recognition can help remove unusual characters or strange sequences that attempt to evade the model.
Image Preprocessing: In image classification, techniques like denoising or normalizing images can prevent misclassification.

Detection of Adversarial Inputs#

In some scenarios, it may be possible to use secondary models or heuristic methods to identify adversarial inputs before they are processed by the primary model. This could involve:

Detecting unexpected patterns or anomalies in the inputs that don’t align with typical data distributions.
Using a separate classifier specifically designed to identify adversarial examples.

Model Interpretability#

Increasing the interpretability of AI models can also help identify when a model has been manipulated through evasion. If the decision-making process of the model is transparent, it becomes easier to spot discrepancies in its behavior that may indicate adversarial manipulation.

Conclusion#

Model evasion is a form of adversarial attack where an attacker crafts inputs to manipulate a machine learning model into making incorrect predictions or decisions. The goal is to exploit vulnerabilities in the model to bypass safeguards, cause misclassification, or mislead the system.

As AI systems become more integrated into critical infrastructure, understanding and defending against evasion attacks becomes crucial to ensuring their reliability, safety, and security. By employing techniques such as adversarial training, input sanitization, and detection of adversarial inputs, organizations can mitigate the risks posed by model evasion and create more robust, resilient AI/ML systems.