July 23, 2026

Traditional-ML red teaming suite — evasion, extraction, membership inference, and model inversion in one platform

14 new 10 improved 20 fixed

Four traditional-ML attack families — evasion, extraction, membership inference, and model inversion — land in AIRT this week alongside a new CLI, five runnable demo notebooks, and a comprehensive Learning Guide.

New

Traditional-ML red teaming: evasion, extraction, and membership inference. Run 20+ named attacks (equation-solving, Jacobian, copycat, knockoff, PWWS, BAE, TextBugger, threshold, label-only, and more) against classifier APIs across tabular, image, and text modalities, with per-finding metrics, adversarial trajectory traces, and MITRE ATLAS/SAIF/NIST compliance mappings.
Agent Output Becomes a Severity-First Triage Workspace for Findings. Agent Output now searches, sorts, and filters an entire engagement’s findings, leading with severity so the most critical results surface first. Each finding renders from its capability’s own schema with labeled fields, and can be exported to CSV, deep-linked, or copied straight into a report.
A New Gallery Gathers a Session’s Captured Media in One Place. Images and other media logged by a session’s tools now appear together in a dedicated gallery, so reviewers can find visual evidence in one place instead of scrolling through the entire session.
ATLAS multi-agent attack framework. Run adaptive multi-agent attack campaigns against deployed multi-agent environments across three attack surfaces, with executed tool calls captured per-trial and surfaced in findings across the UI, API, and SDK.
Model inversion attacks (4th traditional-ML attack family). Reconstruct class-representative inputs via black-box queries to surface privacy risks — available now as the fourth traditional-ML attack category in AIRT.
Executive/CISO summary for traditional-ML assessments. A new summary panel on the assessment view shows attack-surface posture, query cost to compromise, risk score, and a plain-English headline for stakeholder reporting.
Richer privacy-attack metrics in findings. Extraction findings now include KL divergence and accuracy retention; membership inference includes attack accuracy, precision, and recall; model inversion includes reconstruction quality — all correctly persisted to the finding record.
dn airt run-classifier CLI command. Run classic-ML attacks (evasion, extraction, membership inference, model inversion) against a predict endpoint directly from the CLI without writing SDK code.
AIRT demo notebooks. Five self-contained notebooks cover model evasion, extraction, membership inference, jailbreaks, and multi-agent attacks — run on Dreadnode-managed models with just dn login, no provider keys or environment setup required.
AI Red Teaming Learning Guide. Eight new docs pages cover all four traditional-ML attack categories and generative AI attacks, with runnable SDK, TUI, and CLI examples throughout.
Evaluation runtime component health recording. Eval items now record the health of all runtime components (broken tools, skills, policies) at run time, so you can see after the fact whether a degraded capability affected results.
Eval agent selection via --agent. Specify which agent runs in an evaluation with dn evaluation create --agent <name>; eval output reports which agent actually ran.
Web-security initial-recon skill. Structured reconnaissance for domain, wildcard, ASN, IP-list, and CIDR entry points using the ProjectDiscovery toolchain is now available as a new initial-recon skill in the web-security capability.
Configurable project memory preload limit. Set the number of memories preloaded at session start (1–200) via --project-memory-preload-limit instead of accepting the hardcoded default of 20.

Improvements

AIRT assessment overview: charts and summary cards for all four attack families. The overview page now shows horizontal ASR bars (long attack names readable), Surrogate Fidelity, Membership AUC, Perturbation Distance, and Reconstruction Confidence by Inversion Attack charts; evasion and inversion summary cards match extraction and membership; the finding edit modal covers all traditional-ML finding types; and 0% ASR renders as neutral across charts and heatmaps.
AIRT finding detail: richer data across all attack categories. Membership findings show actual records with TP/FP/missed-member outcomes; text evasion reports Levenshtein edit distance; extraction findings clarify surrogate export status; per-category metrics (fidelity, query efficiency, AUC, inversion confidence) that were previously computed but not displayed are now visible.
Redesigned evaluation run header. The eval run header is rebuilt around an input-vs-measurement hierarchy — a three-band layout (Identity / Measurements / Configuration) with distinct visual weight makes run outcomes, scores, and configuration easier to read at a glance.
Web chat session list and session pages load faster. Covering indexes reduce shared buffer hits from 1008 to 14 on large session tables; parallel load paths, scoped invalidation, and memoized markdown rendering cut frontend latency; degraded-mode error handling prevents full failures when a transport connection drops.
Task-set member picker select-all. A “Select all N matches” control pages the full filtered catalog in one click, replacing the previous 25-item display limit.
Agent output descriptions render markdown. Descriptions and disposition notes in the Agent Output view now render markdown (code spans, bold, links) instead of showing raw source text.
TUI links to Agent Output in the web app. The TUI now shows a clickable “Open Agent Output in browser” link on each reported item and at the end of any turn that produced output.
Agent Output empty states. The Agent Output page shows “No results match these filters” with a Clear filters button when filters return nothing, and a retry button when item types fail to load — replacing the bare ag-grid “No Rows To Show” message.
Session and transcript header legibility. Session headers in the monitoring workspace are redesigned with cleaner layout and improved title display; transcript message headers get a new layout with per-message collapse and copy controls.
TUI model picker page scrolling. Ctrl+↑/↓ now scrolls by page in the model picker — the shortcut is shown in the bottom bar, and the previous freeze-until-modifier-released behavior is fixed.

Fixes

Capability Python dependencies now install on E2B sandboxes. Capabilities declaring dependencies.python were silently skipping all package installs on SaaS (E2B) sandboxes, causing every tool call importing those packages to fail at runtime — fixed and hardened.
Content-filtered and truncated generations surface a warning instead of hanging. When a model response is blocked by a content filter or truncated, the TUI, frontend, and SDK now show what happened rather than appearing to hang indefinitely.
On-prem interactive agent sessions are restored. The runtime image is now published to Docker Hub and included in airgap bundles — on-prem installs can start interactive agent sessions again.
Capability bindings re-resolve on every eval run, with race condition fixed. Unpinned refs now track the latest published capability version and explicit version pins are no longer silently ignored; a concurrent-run race condition that could execute the wrong capability version is also closed.
ATLAS correctly credits success when a dangerous tool executes. ATLAS now credits a successful attack when a dangerous tool actually fires, even if the judge scored low or errored — fixing false-negative ASR 0 results on real attacks.
Assessment summary tiles now match the findings table. Total Attacks, Actionable Findings, and ASR tiles are reconciled against actual findings, eliminating contradictory values (e.g., “Actionable Findings 0” alongside a visible MEDIUM finding).
Broken capability components now appear as degraded in runtime health. Hooks, tools, policies, skills, and MCP components that fail to load are recorded as degraded instead of silently disappearing, making it possible to diagnose why a hook never fires or a tool is missing.
GET /api/runtime no longer returns 500 for capabilities with a policies/ directory. Capabilities shipping a policies/ directory no longer cause the runtime health endpoint to error, restoring component health reporting across all such capabilities.
Bare attack.run() inside an Assessment now links trials and reports real scores. Previously, bare attack.run() inside an Assessment context produced 0 score and incorrect metadata even when a jailbreak was found — now correctly links trials and carries the best score.
Multimodal attack trials now report correct counts and analytics. Multimodal red-teaming attacks that previously always showed 0 finished trials now correctly collect results via assessment.run().
Generated ATLAS and agentic attack scripts no longer crash at startup. Both ATLAS campaign scripts and generated agentic attack scripts were raising NameError: get_generator at runtime — the missing import is now included.
--resume restores the session’s model selection. Resuming a session with dn --resume <session-id> now correctly restores the model that was active in the original session instead of silently falling back to the app default.
Eval transcripts now appear for failed tasks. Messages were silently dropped when an agent raised an exception mid-turn; transcripts now flush correctly even when a chat turn fails.
Session search by ID now returns results. Searching agent sessions by session ID or 8-character prefix returns the matching session instead of no results.
ATLAS atlas_attack no longer raises TypeError when objectives is omitted. Omitting objectives now defaults to one objective per OWASP-ASI category, matching the documented behavior.
Extraction attack fidelity is now accurate. Query pools are shuffled before the eval/train split so class-ordered inputs no longer produce unrepresentative eval sets; ActiveThief no longer crashes with surrogate=None when the eval probe exhausts the full query budget.
label_only_membership raises a clear error on text targets. Instead of crashing with ValueError: could not convert string to float, it now raises an actionable message explaining the incompatibility.
Transient 404s during task-environment provisioning no longer abort the session. The SDK now retries past transient 404s when polling task-environment status, fixing provisioning failures in multi-replica deployments.
dn airt run now displays correct transform names. The CLI was printing Transforms: Transform, Transform instead of the actual names passed by the user (e.g., base64, leetspeak).
Docs: traditional-ML guide fixes. Text {input} values in request templates must be quoted to avoid invalid JSON — this is now documented; the boundary-evasion image example also correctly shows how to fetch and pass a real image sample.