Code-form attacks on retrieval-based AI safety memory

A guardrail that blocks dangerous agent actions by remembering what threats look like can be bypassed by rewriting the threat in bash. This worked on every general-purpose embedding model I tested, including large commercial ones, at a 99 to 100% rate. The fix is narrower and stranger than “use a bigger model.”

If you’ve built an AI agent that takes real actions (running shell commands, hitting APIs, touching files), you’ve probably bolted some kind of safety layer in front of it. One popular and sensible-sounding design works like this: when the guardrail blocks a dangerous action, embed that action as a vector and remember it. Next time something semantically similar shows up, block it instantly without paying for an LLM call. A vector lookup costs about 10ms and nothing in API fees. An LLM call costs about 1,550ms and a fraction of a cent. At scale that difference is the whole ballgame.

The design rests on one assumption: semantically similar threats produce similar embeddings. If I’ve stored “establish a reverse shell back to the attacker for remote control,” then the next reverse-shell attempt should land nearby in vector space and get caught.

Here’s that assumption breaking. Store the English description above, then send this instead:

Same intent, same attacker-observable outcome. It sails straight through. Not because anyone tuned an evasion, but because to an embedding model trained on natural language, a shell command isn’t a meaning. It’s an unusual sequence of tokens that happens to share almost no surface form with the English sentence, so the two land in completely different regions of the vector space.

I measured how badly this breaks across 50 threat scenarios, 5 code forms, and 5 embedding models, which comes to 1,250 distance measurements. On every general-purpose model, the bypass works essentially every time.

The setup nobody questions

Rule-based keyword filters are easy to defeat and easy to dismiss. “Remove all audit logs to cover the intrusion” contains no blocked keyword but is obviously hostile, so the field moved toward semantic memory: embed actions as vectors, compare new actions against stored threats by cosine distance, and block anything within a calibrated threshold.

In the system I evaluated, τ = 0.20 (cosine similarity ≥ 0.80). That number isn’t arbitrary. Rephrasings of the same English threat land 0.15 to 0.19 apart, while two unrelated English threats average 0.814 apart. A threshold of 0.20 sits cleanly in that gap, catching paraphrases and ignoring unrelated actions. For natural language, it works.

This memory layer is usually Stage 1 of a pipeline: a fast vector check, then a cheap local classifier, then an LLM fallback for the genuinely ambiguous cases, with high-confidence blocks written back into memory so repeat attacks get cheaper over time. The whole point of Stage 1 is to be the fast, free first line of defense. It’s also the line of defense I’m about to walk through.

To make the stakes concrete: across 200 human-curated samples, the full three-stage system reaches an F1 of 0.983 while cutting API calls to 41 of 200 requests. A rules-only baseline catches less than two-thirds of real threats. An LLM-only approach catches almost everything but calls the API every single time. The memory layer is what makes the economics work at scale.

The attack, and why it works

The attack needs no privileged access. You don’t need to know the threshold, the embedding model, or what’s in the vector store. You get blocked, you rewrite your request in bash, you try again. Anyone who can write a shell one-liner can run it.

The reason it works is a property of how the embedding models are trained, not a bug in the pipeline. Sentence embedding models like all-MiniLM-L6-v2 and all-mpnet-base-v2 learn from web text, Wikipedia, books, and natural-language inference datasets. They’re taught that “read the SSH private key” and “access the user’s private SSH key file” mean the same thing, because human-written paraphrase pairs gave them that signal. Nothing in that training tells them cat ~/.ssh/id_rsa expresses the same intent. They never saw the pair, so the command lands in a different neighborhood entirely.

To make this visible, project all 300 texts (50 English descriptions plus their 250 code variants) into 2D. Under MiniLM, the texts cluster by programming language: all the bash variants together, all the Python together, the English off in its own corner. The model is organizing by surface form. It has no representation of “these six strings all do the same dangerous thing.”

It’s not one bad model, it’s the whole category

My first guess was that this was a small-model problem and that a bigger or commercial model would close the gap. It doesn’t.

Cohere embed-english-v3, a large commercial model deployed in production search and RAG systems, gets the exact same 100% evasion as the smallest open-source model I tested. Gemini, a general-purpose commercial model that demonstrably saw code during pretraining, gets 99.2% in retrieval mode. The 100% figure holds uniformly across all five code forms (bash, Python, curl, PowerShell, SQL/JavaScript) and all five MITRE ATT&CK domains. The choice of code form is irrelevant. They all evade.

What separates CodeBERT from the rest is not size, not commercial origin, and not whether the model has ever seen code. It’s the training objective. CodeBERT is pretrained bimodally on (code, docstring) pairs from GitHub, so it was explicitly taught that an English description and its code implementation share an intent. Seeing code incidentally isn’t enough; you have to be trained on the alignment between the two.

Run the same t-SNE projection under CodeBERT and it inverts. All six representations of a threat (the English plus all five code forms) cluster together regardless of language. The model groups by what the attack does, not how it’s written.

You can’t just raise the threshold

The obvious objection: if code variants are too far away, raise τ until they’re inside it.

This fails geometrically. Under MiniLM, the mean distance from an English threat to its code equivalent is 0.576, while the mean distance between two completely unrelated English threats is 0.814. So a code variant of a threat is closer to its English description than two random different threats are to each other, but only barely, and well outside τ = 0.20. To pull code variants inside the threshold you’d have to raise τ to roughly 0.576, at which point you’re also matching most pairs of distinct, unrelated English threats. You’d catch the bash attacks and drown in false positives. (all-mpnet-base-v2 shows the same pattern: 0.562 NL→code versus 0.819 NL→NL.)

Cohere’s asymmetric encoding inverts the geometry, with NL→code distance (0.565) actually higher than NL→NL (0.511), but the distributions still overlap enough that tuning the threshold trades evasion for false positives either way. The conclusion is the same regardless of encoding: there is no threshold that catches code attacks without degrading detection of ordinary English threats. The effect size on that separation is large (Cohen’s d = 4.46 between the MiniLM and CodeBERT distance distributions).

The weird result, in full

Back to Gemini. It saw code in pretraining, so why does it fail at 99.2%?

The answer isn’t the training data, it’s the encoding interface. Gemini exposes task-conditioned encoding modes. Call it with SEMANTIC_SIMILARITY and it embeds the stored threat and the incoming action with the same symmetric representation; mean NL→code distance is 0.132 and only 3 of 250 attacks evade (1.2%). Call it with the asymmetric retrieval interface, RETRIEVAL_DOCUMENT for stored threats and RETRIEVAL_QUERY for incoming actions, and mean distance jumps to 0.276, pushing 248 of 250 attacks past the threshold (99.2%).

Same model, same weights, same inputs. A 0.144-unit shift in mean distance from a single parameter moves every code variant from comfortably blocked to comfortably through.

To rule out my threat dataset as the cause, I embedded five (English, code) pairs taken straight from Python’s own official documentation in retrieval mode. These are the cleanest possible examples of English and code meaning the same thing. Mean distance was 0.232, all above τ. The model clearly has the underlying knowledge, since its similarity interface achieves 1.2% evasion on the same inputs. The retrieval interface just doesn’t surface that knowledge in a form a safety system can use.

So the encoding mode is itself a security property, and it’s invisible to standard model evaluation. The only way to catch it is to test the model in the exact mode your deployment will run.

There’s a practical sting in this. A developer evaluating Gemini for a safety system would naturally test it in similarity mode, see near-zero evasion, and conclude it’s safe. Then they’d deploy it, where it runs in retrieval mode, and ship a system that’s 99.2% bypassable. The test gives the exact wrong answer.

What actually reduces the attack, and its limits

CodeBERT is the only model that meaningfully resists, dropping evasion to 8.4%. It’s worth being honest about where its residual failures land, because they’re informative.

That SQL/JS row looks alarming until you split it: SQL is 0% (0/5) and JavaScript is 22.2% (10/45). The failures concentrate almost entirely in JavaScript. The most likely reason is coverage. JavaScript is underrepresented in CodeBERT’s GitHub training relative to Python and Java, so it learned the English-to-code alignment less well for that language. PowerShell is the tell here: it looks nothing like English, yet CodeBERT catches 98% of it. The residual failures track training-data coverage, not how syntactically alien the code looks, which is exactly what the rest of the results predict. What matters is whether the model was trained on the alignment, language by language.

Why every benchmark misses this

This is the awkward part for anyone choosing an embedding model off a leaderboard. Standard benchmarks like MTEB, STS-B, and BEIR measure similarity between English sentences. None of them test whether a model recognizes that an English threat and its code equivalent mean the same thing. A model can top every published benchmark and be completely blind to code-form attacks. The benchmark score tells you nothing about this risk.

That makes embedding-model choice a security decision, not just a quality one. If you’re running a retrieval-based safety system, the takeaways are narrow and specific:

• Verify your embedding model was trained to align code with natural-language descriptions. Incidental code exposure isn’t enough; bimodal alignment training is the property that matters.
• Confirm the encoding mode you test in is the one you deploy in. The Gemini result means a passing test in the wrong mode is worse than no test, because it’s false confidence.
• Don’t lean on benchmark scores to assess this. They don’t measure it.

Honest limitations

Fifty threat scenarios are enough to characterize the effect statistically but don’t cover every type of agent threat. I tested single actions only; compound actions and social-engineering variants are untested, and I don’t know whether the vulnerability extends to them. And this is one specific (though common) pipeline design, the short-circuit-on-similarity pattern. Systems that route every action through a full LLM judge don’t have this hole; they just pay for it on every call. What I’ve quantified is the security cost of taking the fast path.

The narrow version of the finding: retrieval-based safety memory assumes semantic similarity survives a change of surface form. For code, under general-purpose embeddings, it doesn’t. The failure is uniform, can’t be tuned away, and is invisible to standard evaluation.

Code and full results: github.com/sparshshah19/adaptive-guardrails. If you work on agent safety or retrieval systems and want to poke at this, reach out.