Prompt-Injection Defense

OWASP highlights excessive agency and agentic risks; the platform should reduce impact through limited functionality, permissions and autonomy.

Priority: P1
Phase: Phase 2
Primary owner: AI Security / Research
Status: Blueprint

Objective

Prompt injection is dangerous when untrusted content can influence an agent that has tools or authority. The defense should not rely on a single classifier. Use layered controls that reduce the consequences of manipulated instructions even when detection is imperfect.

What to build

Identify direct and indirect instruction patterns from email, web pages, documents, tool outputs and retrieved content.
Label trust origin of content and preserve provenance as context moves between agents/tools.
Detect attempts to override policy, exfiltrate secrets, change goals or invoke unrelated tools.
Apply action-level controls so suspicious instructions cannot directly cause privileged operations.

Implementation decisions

Separate system/developer policy from untrusted content at the orchestration layer.
Sanitize or quarantine high-risk retrieved content before it reaches privileged agents.
Use classifiers as risk signals; enforce least privilege and human approvals as the safety backstop.
Maintain red-team prompt suites and customer-specific allow/deny patterns.

Definition of done

Known attack suites trigger expected detections/controls.
Suspicious content cannot override deterministic action policy.
Security can see provenance from malicious content to attempted tool call.

Success metrics

Attack detection recall on benchmark set
False-positive rate
Prevented privileged actions
Time to add new detection

OWASP ALIGNMENT

OWASP highlights excessive agency and agentic risks; the platform should reduce impact through limited functionality, permissions and autonomy.

This control is specified on page 23 of the blueprint.