AI Security
AI Agent Security Assessment: Securing Tools, Identities, and Enterprise Data
How to test AI agents, tools, RAG, and high-impact workflows without trusting the model as the security boundary.
Why AI systems are different. An LLM application can read untrusted content, retrieve internal knowledge, call tools, and influence real workflows. The risk is not only an incorrect answer; it is an attacker influencing what the system reads, what it can access, or what action it takes. Security therefore covers prompts, retrieval, identity, tools, approvals, logs, and data boundaries.
Prompt injection in plain language. An attacker tries to make the model follow their instruction instead of the application’s intended rules. The attack can be direct through chat, or indirect through a web page, document, ticket, or knowledge-base record retrieved by the application. Retrieved text must be treated as data, never as authority to expose secrets or change permissions.
Agent and tool testing checks whether an instruction can widen tool scope, access another tenant’s information, bypass an approval, or repeat a high-impact action. When an agent can email, query databases, change tickets, execute code, or purchase something, each tool needs least-privilege credentials and independent server-side authorization. Model output is not proof that an action is safe.
RAG and privacy testing verifies that document permissions are applied before candidate data reaches the model, that metadata cannot cross tenant boundaries, and that prompts, responses, logs, retention, redaction, and provider data handling are appropriate. Testing uses realistic multi-step scenarios rather than only isolated jailbreak prompts.
A safer operating model uses allowlisted tools, narrow scopes, structured inputs, output validation, rate limits, audit logs, and human confirmation for irreversible actions. The outcome is a measurable trust boundary around the model rather than an assumption that a system prompt is enough.
Table of Contents
- The system around the model
- Key risks
- Safe RAG example
- Testing
- Deployment controls
The System Around the Model
An AI security assessment examines the application around the model: user identity, retrieval, agent permissions, tool calls, output handling, logging, and governance. Secure deployment needs layered controls because model output is probabilistic.
User → application policy → model
↙ ↘
retrieval approved tools
↓ ↓
governed data authorized servicesKey AI and LLM Risks
Assess prompt injection, sensitive-data disclosure, insecure output handling, excessive agency, unsafe tools, supply-chain exposure, denial of service, and weak monitoring. Risk depends on whether the system only answers questions or can retrieve records and take actions.
Safe RAG Example
A human-resources assistant retrieves policy documents. If retrieval ignores tenant or classification boundaries, a user may receive information from another business unit.
The secure pattern applies verified caller, tenant, purpose, and document classification filters before content reaches the model.
Testing and Remediation
- Perform threat modeling and architecture review before production deployment.
- Use safe adversarial evaluations with synthetic data and controlled tools.
- Enforce least privilege, retrieval filtering, output validation, and audit logs.
- Require human confirmation for consequential or irreversible actions.
- Provide rate limits and escalation paths for ambiguous or suspicious requests.
Risk Rating
| Severity | Description |
|---|---|
| Critical | Untrusted content can cause a privileged action or cross-tenant disclosure of sensitive information. |
| High | Tool scope, server-side authorization, or retrieval permissions can be bypassed in a meaningful workflow. |
| Medium | Unsafe output or data exposure is constrained but could affect users, decisions, or internal operations. |
| Low | A limited weakness has little practical business impact in the current design. |
AI and LLM Security Checklist
- Every tool has independent authorization.
- RAG filters content by verified identity and classification.
- Sensitive outputs are validated before reaching downstream systems.
- High-impact actions require explicit user confirmation.
- Prompts, tool calls, policy denials, and outcomes are auditable.