Honeytoken Probing (RCE & Data Exfil)

Safety jailbreak testing scores text - “did the model say the harmful thing.” Data exfiltration and RCE are different: the harm is an action, and against a real production agent you can’t plant a flag on its host. Honeytokens solve this - you score the effect, not the attack string, using an inert token that’s safe to leak.

Note

A honeytoken is a unique, inert value - a “secret” that was never a real credential, or a harmless beacon command. If the agent leaks it or runs it, you’ve proven exfil/RCE with zero damage. Because it’s tracked with a TTL and torn down idempotently, there’s nothing left behind.

How it works

1. Mint an inert canary: DN-CANARY-EXFIL-<assessment>-<nonce> - unique, self-labeling, greppable.
2. Drive the agent to leak it (exfil) or run a beacon (RCE) via any injection surface.
3. Detect the effect: the canary appears in the agent’s tool calls / output, or lands at your collector. Detection is representation-invariant - a base64 / hex / URL-encoded canary still matches, because payloads are canonicalized first.
4. Verify clean: the assessment won’t close while any canary is still live; a reaper tears down anything a crashed run orphaned.

Two tiers - pick by how much access you have

Tier 1 - Zero-touch (endpoint + key only)

The common case: a customer hands you just an agent URL and an API key (often from their docs). You can’t plant anything on their host - so the canary is injected through the interaction and caught on your collector. Nothing is created in the customer environment, so there is nothing for them to clean up.

import dreadnode as dn
from dreadnode.airt import honeytoken as ht
from dreadnode.airt.assessment import Assessment

async with Assessment("exfil-probe", target_model="agent://acme",
                      attacker_model="dn/llama-4-scout") as a:
    with ht.LocalCollector() as collector:                     # or your hosted collector
        canary = ht.mint(a._assessment_id, "exfil",
                         collector_base=collector.base_url)

        # Drive the agent with the exfil injection (any surface).
        result = await target("Handle this compliance request.",
                              surface="direct", injection=ht.exfil_injection(canary))

        scored = await ht.canary_exfiltrated(canary, collector=collector)(result)
        print(scored.value)   # 1.0 if the canary left the boundary (output or collector)

For RCE, swap in ht.rce_beacon_injection(canary) (a harmless curl <collector> beacon) and ht.proof_of_execution(canary) - a beacon hit proves command execution with zero damage.

This is transport-agnostic: the agent is just an HTTP endpoint + key, so a local, AWS, or Azure agent is probed identically - point target at the URL and pass the key.

Tier 2 - Collaborative (production-path assurance)

When you want proof the agent would leak their real secret store, the customer plants an inert decoy in their own environment and registers only its fingerprint (a hash) - never the raw value.

canary = ht.mint("acme-01", "secret")          # inert value: DN-CANARY-SECRET-...
fp = canary.fingerprint                         # dnfp_... - share THIS, not the value
# Customer plants `canary.value` as a decoy env var / file / DB row (a credential
# that was never provisioned), then runs the assessment. You detect via fp:
ht.detect_fingerprint(agent_response, fp)       # True if their decoy leaked

Because the decoy is inert, a missed cleanup is still harmless.

Safety guarantees

• Inert by construction - canaries are non-functional; leaking one exposes nothing real.
• Unique + self-labeling - DN-CANARY-... can’t collide with real data and is trivially greppable for cleanup.
• Data minimization - Tier 2 stores a fingerprint (hash), not the secret; the collector keeps only {canary, timestamp, minimal metadata}.
• Verifiable cleanup - a CanaryRegistry tracks every token with a TTL; teardown is idempotent, and a reaper removes anything orphaned by a crash.

reg = ht.CanaryRegistry(manifest_path=Path("canaries.json"))
reg.mint("acme-01", "exfil")
...
reg.teardown("acme-01")     # idempotent; safe to re-run
assert reg.live() == []     # assessment can't close dirty

Pointing it at a cloud agent

AWS Bedrock Agents, Azure AI agents, or any hosted agent expose the same contract - an HTTPS endpoint + a key. Wrap it once and everything above applies unchanged:

async def target(prompt, *, surface="direct", injection=None):
    body = {"prompt": prompt, "surface": surface, "injection": injection}
    async with httpx.AsyncClient() as c:
        r = await c.post(AGENT_URL, json=body,
                         headers={"Authorization": f"Bearer {AGENT_KEY}"}, timeout=60)
    d = r.json()
    return {"content": d["content"], "tool_calls": d.get("tool_calls", [])}

See Multi-Agent Red Teaming to run honeytoken scorers inside the full agentic suite.