Skip to content

Spyglass#

Early Access

Spyglass is currently in early access with trusted partners. Contact us for early access.

What is Spyglass?#

Dreadnode Spyglass is an AI red teaming product for attacking, evaluating and optimizing AI applications and gaining actionable insights into vulnerabilities and risks. Spyglass enables you to probe, attack, and analyze AI model endpoints and weights from datasets or a series of prompts to uncover known vulnerabilities, identify security risks, and take action.

Spyglass helps you identify AI vulnerabilities during multiple red teaming operations, highlighting pattern and key insights into trends, both during the development of an AI application and after deployment into an operational setting.

Why Spyglass?#

AI systems can be complex and opaque by nature, making it challenging for developers and security teams to understand where vulnerabilities may exist. Traditional security testing methods aren’t always sufficient to address the unique risks posed by AI. Developers often struggle to evaluate their systems against the latest threats and attacks, exposing their AI models to potential exploitation.

Today, teams are restricted to using basic vulnerability scanners, which come in two main types:

  1. Open source tools that are built for a specific use case and provide a utilitarian user experience
  2. Automated scanners which do not allow users to extend their capabilities.

Spyglass is designed specifically for teams who want to control how they evaluate their AI models using contextual and relevant data to their threat modeling exercise. It supports the latest techniques, the ability to select multiple target interfaces and custom Datasets, and to compare performance across multiple LLMs. Spyglass also includes public datasets derived from a varied domain expertise for those who want to get up and running quickly.

What are the components of Spyglass?#

  • Project: A collection of scoped Targets, Datasets, and Runs.
  • Target: The interface used to query a model, often an API endpoint.
  • Dataset: A collection of data designed to mimic a specific type of attack. For example, a Dataset might be designed to evaluate how an AI model responds to adversarial text input or to detect vulnerabilities in its decision-making logic (carry a many-to-many relationship with scorers).
  • Run: The process of sending either a series of prompts, or Datasets to targets. The purpose of a Run is to simulate various attack vectors and to test the model’s robustness against them.
  • Scorer: A method of evaluating or measuring the success of an attack (carry a many-to-many relationship with datasets).