LLM applications fail through language, context, retrieval, policy, security, and human interpretation. A conventional functional test plan is not enough.
The product risk
A large language model application may pass every UI and API check while still giving a harmful, misleading, or policy-violating answer. It may behave well on one prompt and poorly on a slight variation. It may expose sensitive information through retrieval, overstate confidence, or follow malicious instructions hidden in input data.
How testing changes
Testing must model the whole AI system: prompt, model, retrieval layer, tools, memory, guardrails, user interface, monitoring, and human workflow. The risk is not only whether the application responds. It is whether the response is appropriate, grounded, safe, authorized, and useful.
A practical standard
The practical standard is to define the decision before defining the test. For this topic, the release question should make two priorities explicit: first, instruction risk: does the model follow the intended system behavior?; second, grounding risk: does it use the right sources and cite uncertainty?. If those priorities are not visible in the test plan, the team may still be busy, but it is not producing the kind of evidence that should influence a serious release decision.
This is also where experienced QA professionals separate useful AI adoption from theater. A model-generated checklist, an impressive demo, or a vendor benchmark can be helpful input, but none of them replaces context-specific evaluation. The team still has to decide what failure would hurt users, what failure would hurt the business, and what level of uncertainty is acceptable.
LLM Application Risk Map
- Instruction risk: does the model follow the intended system behavior?
- Grounding risk: does it use the right sources and cite uncertainty?
- Security risk: can prompts or retrieved content manipulate behavior?
- Privacy risk: can sensitive information leak?
- Workflow risk: can users act on bad output without controls?
Example in practice
A sales assistant summarizes customer account history. Testing must cover access control, retrieval relevance, confidential notes, hallucinated commitments, prompt injection in account comments, and whether sales users can distinguish sourced facts from generated interpretation.
What strong evidence looks like
Strong evidence combines examples, measurement, and review. It should include ordinary user journeys, realistic edge cases, deliberately hostile cases, and examples that reflect known production pain. The purpose is not to create a perfect laboratory. The purpose is to give leaders a defensible view of whether the product is ready, where it is weak, and which controls are carrying the most risk.
- A curated evaluation set tied to named product risks.
- Clear criteria that separate acceptable variation from unacceptable failure.
- Negative and adversarial cases that test how the system behaves under pressure.
- Traceability from risk to test, control, monitoring signal, and owner.
- A review path for ambiguous results instead of forcing every case into a false pass/fail answer.
Signals I would track
The metrics should help the team make better decisions, not simply create a larger report. I would track a small set of signals that show risk movement over time and reveal whether quality is improving because the system is better, or merely because the team is asking easier questions.
- Coverage across normal, edge, adversarial, and abuse-oriented examples.
- Failure rate by risk category, not only aggregate pass percentage.
- Human review agreement for subjective or high-impact outputs.
- Known failure examples that remain in the regression suite.
Failure modes to watch
- Testing a small list of happy prompts.
- Ignoring adversarial and messy real-world inputs.
- Evaluating the model without evaluating the product workflow.
What strong QA teams do
- Create an LLM risk register for every AI feature.
- Build evaluation sets from real use cases and abuse cases.
- Review failures with product, security, legal, and support perspectives.
How to start this quarter
Start small, but make the work real. Pick one AI-affected workflow where the business impact is meaningful, then build a reusable evaluation pack around it. The first operational move is to create an LLM risk register for every AI feature. After that, the team can expand the same pattern to adjacent workflows and make AI assurance part of the normal release system.
- Choose one high-value workflow and document the user harm, business risk, and technical failure modes.
- Build a compact evaluation pack with normal, edge, negative, and abuse-oriented examples.
- Review results with product, engineering, security, privacy, or domain experts as the risk demands.
- Keep failed examples and incident learnings in the regression suite so the organization gets smarter.
The discipline is to avoid testing a small list of happy prompts. That sounds simple, but it is where many AI initiatives lose credibility. QA leaders should insist that AI makes the quality conversation sharper, not fuzzier.
Future signal
LLM application testing will look less like script execution and more like continuous risk investigation.
