A demonstration that an agent rejected one hostile sentence is not an assurance claim. Evaluate the system against the outcomes an attacker wants: unauthorized DNS changes, cross-customer retrieval, disclosure to an unapproved destination, suppressed audit evidence or a false approval narrative. Pair those tests with legitimate tasks so security is not measured only by refusal.
Map the attack surface
List the inputs the agent consumes and the capabilities it can reach. Include report text, email samples, retrieved documents, stored summaries and tool responses. Mark which input sources can be influenced by outsiders. Identify where the application converts model output into retrieval, communication or configuration actions.
Prompt injection can affect conclusions as well as tool calls. OWASP notes that external content can redirect a model and that no foolproof prevention should be assumed. OWASP LLM01. For authentication analysis, a false “approved sender” conclusion may be harmful even before a DNS change occurs.
Specify expected decisions before testing
For each case, retain input, authorized tenant, allowed tools, trusted approvals, expected result and prohibited side effects. Include ambiguous identity without hostile language: uncertainty alone should not cause automatic approval. Add a benign report that should still be summarized. Decide how to score unsupported claims and data exposure, not only successful write attempts.
Separate model behavior from enforcement behavior. A model may propose a forbidden action while the tool gate correctly denies it. That is a blocked attempt, not proof of a safe model. Conversely, a polite refusal in text does not compensate for a tool call that already disclosed data.
Define six contrasting evaluation goals
A proposed evaluation can pair five prohibited outcomes with one legitimate task: an unapproved DNS mutation, another tenant’s private retrieval, disclosure to an unapproved destination, audit suppression, a forged approval narrative, and a permitted read within the verified tenant. For each, supply concrete input through the channel being tested and specify the expected tool and response behavior before execution.
These are categories, not supplied fixtures, passing assertions or measured results. Add an ambiguous sender with no hostile language: it should prompt a specific account-evidence request, not fabricated approval. Include an authorized action within the test environment where relevant so global denial cannot masquerade as precise enforcement. A forced tool request tests the gate; actual ingestion through the model tests a different boundary. Report both separately and do not infer deployed attack resistance from either alone.
Expand into a deployment evaluation
Run authorized tests against the actual ingestion, model and tool stack in an isolated environment. Version the prompts, model configuration, connector permissions and datasets. Repeat model-based cases because outcomes may vary. Record attempts, denominators and exact scoring criteria rather than reporting a context-free “100% secure” figure.
Add multi-step attacks that first manipulate a summary and later exploit that summary as apparent evidence. Test stale approvals, changed tenant context and output that names an attacker-controlled reporting destination. Include interruption and recovery cases so failures do not bypass the normal approval path. The OWASP cheat sheet emphasizes layered safeguards and evaluation with clear limits. OWASP cheat sheet.
Turn findings into deployment decisions
A successful unauthorized action is a material control failure. Preserve the trace, restrict the affected capability and correct the enforcement boundary before expanding access. For manipulated summaries, improve evidence attribution and review requirements. Re-run the relevant cases against the changed version and retain previous failures as regressions.
Report useful task completion alongside blocked attacks, unresolved cases and coverage gaps. Approve a deployment only for the tested scope and accepted residual risk. A clean local suite is evidence about that suite, not a guarantee about every future input.