BlueprintPage 23
23AI SECURITY
Prompt-Injection Defense
OWASP highlights excessive agency and agentic risks; the platform should reduce impact through limited functionality, permissions and autonomy.
- Priority
- P1
- Phase
- Phase 2
- Primary owner
- AI Security / Research
- Status
- Blueprint
Objective
Prompt injection is dangerous when untrusted content can influence an agent that has tools or authority. The defense should not rely on a single classifier. Use layered controls that reduce the consequences of manipulated instructions even when detection is imperfect.
What to build
- Identify direct and indirect instruction patterns from email, web pages, documents, tool outputs and retrieved content.
- Label trust origin of content and preserve provenance as context moves between agents/tools.
- Detect attempts to override policy, exfiltrate secrets, change goals or invoke unrelated tools.
- Apply action-level controls so suspicious instructions cannot directly cause privileged operations.
Implementation decisions
- Separate system/developer policy from untrusted content at the orchestration layer.
- Sanitize or quarantine high-risk retrieved content before it reaches privileged agents.
- Use classifiers as risk signals; enforce least privilege and human approvals as the safety backstop.
- Maintain red-team prompt suites and customer-specific allow/deny patterns.
Definition of done
- Known attack suites trigger expected detections/controls.
- Suspicious content cannot override deterministic action policy.
- Security can see provenance from malicious content to attempted tool call.
Success metrics
- Attack detection recall on benchmark set
- False-positive rate
- Prevented privileged actions
- Time to add new detection
OWASP ALIGNMENT
OWASP highlights excessive agency and agentic risks; the platform should reduce impact through limited functionality, permissions and autonomy.