Software Quality Engineering Journal: September 2026

AI for Test Maintenance and Flaky Test Triage

AI can help diagnose flaky tests and maintenance hotspots, but teams still need engineering discipline to remove the causes.

The opportunity

Flaky tests often involve timing, data dependencies, environment instability, asynchronous behavior, external services, or poor assertions. AI can cluster failures, summarize logs, compare screenshots, and suggest likely causes. That speeds triage, but it does not fix weak test design by itself.

How to use AI well

The best use of AI is to reduce diagnosis time and reveal patterns. It should help answer whether failures share a dependency, data condition, recent code change, browser behavior, or infrastructure issue. The output should lead to durable fixes, not repeated reruns.

A practical standard

The practical standard is to define the decision before defining the test. For this topic, the release question should make two priorities explicit: first, failure history by test, component, environment, and time; second, recent code, dependency, data, or infrastructure changes. If those priorities are not visible in the test plan, the team may still be busy, but it is not producing the kind of evidence that should influence a serious release decision.

This is also where experienced QA professionals separate useful AI adoption from theater. A model-generated checklist, an impressive demo, or a vendor benchmark can be helpful input, but none of them replaces context-specific evaluation. The team still has to decide what failure would hurt users, what failure would hurt the business, and what level of uncertainty is acceptable.

Flaky Test Triage Signals

Failure history by test, component, environment, and time.
Recent code, dependency, data, or infrastructure changes.
Screenshots, traces, logs, and network calls.
Timing patterns and retry behavior.
Ownership and fix history.

Example in practice

An AI assistant clusters 40 UI failures and finds they all involve the same delayed event publication. QA uses that insight to move key checks to an API layer and add deterministic waiting around the remaining UI journey.

What strong evidence looks like

Strong evidence combines examples, measurement, and review. It should include ordinary user journeys, realistic edge cases, deliberately hostile cases, and examples that reflect known production pain. The purpose is not to create a perfect laboratory. The purpose is to give leaders a defensible view of whether the product is ready, where it is weak, and which controls are carrying the most risk.

A curated evaluation set tied to named product risks.
Clear criteria that separate acceptable variation from unacceptable failure.
Negative and adversarial cases that test how the system behaves under pressure.
Traceability from risk to test, control, monitoring signal, and owner.
A review path for ambiguous results instead of forcing every case into a false pass/fail answer.

Signals I would track

The metrics should help the team make better decisions, not simply create a larger report. I would track a small set of signals that show risk movement over time and reveal whether quality is improving because the system is better, or merely because the team is asking easier questions.

Human acceptance rate of AI-generated test assets after review.
Defects found from AI-assisted exploration versus conventional activity.
False confidence signals, including generated tests rejected as weak.
Time saved without reducing risk coverage or review quality.

Where teams get misled

Using AI to justify reruns instead of fixes.
Treating symptoms as root causes.
Ignoring test design debt because triage got faster.

How to govern the practice

Track flaky-test diagnosis time and recurrence.
Require root-cause categories for repeated failures.
Use AI summaries to prioritize test architecture improvements.

How to start this quarter

Start small, but make the work real. Pick one AI-affected workflow where the business impact is meaningful, then build a reusable evaluation pack around it. The first operational move is to track flaky-test diagnosis time and recurrence. After that, the team can expand the same pattern to adjacent workflows and make AI assurance part of the normal release system.

Choose one high-value workflow and document the user harm, business risk, and technical failure modes.
Build a compact evaluation pack with normal, edge, negative, and abuse-oriented examples.
Review results with product, engineering, security, privacy, or domain experts as the risk demands.
Keep failed examples and incident learnings in the regression suite so the organization gets smarter.

The discipline is to avoid using AI to justify reruns instead of fixes. That sounds simple, but it is where many AI initiatives lose credibility. QA leaders should insist that AI makes the quality conversation sharper, not fuzzier.

Future signal

AI will make noisy suites easier to analyze. Mature teams will use that visibility to make suites quieter.

Sources worth reading

Using LLMs for Exploratory Testing

LLMs can strengthen exploratory testing when they help testers ask broader questions, but exploration remains a human learning activity.

The opportunity

Exploratory testing depends on curiosity, observation, product understanding, and adaptation. AI can support that work by suggesting charters, personas, data variations, failure modes, and questions for unfamiliar domains. It cannot observe the product with human accountability or decide which surprise matters.

How to use AI well

A strong workflow uses AI before, during, and after sessions. Before the session, generate charters and risks. During the session, ask for new angles when observations emerge. After the session, summarize notes and identify follow-up tests. The tester remains responsible for judgment.

A practical standard

The practical standard is to define the decision before defining the test. For this topic, the release question should make two priorities explicit: first, generate charters from requirements and architecture notes; second, ask for personas, edge data, and abuse cases. If those priorities are not visible in the test plan, the team may still be busy, but it is not producing the kind of evidence that should influence a serious release decision.

AI-Assisted Exploratory Workflow

Generate charters from requirements and architecture notes.
Ask for personas, edge data, and abuse cases.
Use observations to prompt for adjacent risks.
Summarize session notes into findings and open questions.
Convert repeated discoveries into regression or monitoring candidates.

Example in practice

A tester exploring a claims workflow asks an LLM for edge cases around partial documentation. After finding one confusing status transition, the tester prompts for related state risks and discovers cancellation and resubmission gaps.

What strong evidence looks like

A curated evaluation set tied to named product risks.
Clear criteria that separate acceptable variation from unacceptable failure.
Negative and adversarial cases that test how the system behaves under pressure.
Traceability from risk to test, control, monitoring signal, and owner.
A review path for ambiguous results instead of forcing every case into a false pass/fail answer.

Signals I would track

Human acceptance rate of AI-generated test assets after review.
Defects found from AI-assisted exploration versus conventional activity.
False confidence signals, including generated tests rejected as weak.
Time saved without reducing risk coverage or review quality.

Where teams get misled

Letting AI define the whole exploration mission.
Ignoring what the tester observes because the charter looked complete.
Using generic personas that do not match real users.

How to govern the practice

Provide examples of strong AI-assisted charters.
Keep exploratory notes grounded in observed behavior.
Coach testers to challenge AI suggestions openly.

How to start this quarter

Start small, but make the work real. Pick one AI-affected workflow where the business impact is meaningful, then build a reusable evaluation pack around it. The first operational move is to provide examples of strong AI-assisted charters. After that, the team can expand the same pattern to adjacent workflows and make AI assurance part of the normal release system.

Choose one high-value workflow and document the user harm, business risk, and technical failure modes.
Build a compact evaluation pack with normal, edge, negative, and abuse-oriented examples.
Review results with product, engineering, security, privacy, or domain experts as the risk demands.
Keep failed examples and incident learnings in the regression suite so the organization gets smarter.

The discipline is to avoid letting AI define the whole exploration mission. That sounds simple, but it is where many AI initiatives lose credibility. QA leaders should insist that AI makes the quality conversation sharper, not fuzzier.

Future signal

Exploratory testing will become more powerful when AI helps generate angles, but the craft will still depend on human attention.

Sources worth reading

AI-Generated Test Cases: Where They Help and Where They Mislead

AI-generated test cases are useful for breadth and speed, but dangerous when teams mistake generated volume for risk coverage.

The opportunity

Given a requirement or user story, a model can produce many test cases quickly. This is helpful for brainstorming, onboarding, and identifying obvious gaps. But generated tests often reflect generic patterns. They may miss domain-specific failure modes, architecture constraints, historical defects, and business impact.

How to use AI well

The right workflow is generation followed by professional curation. Testers should ask: which generated cases map to real risk, which are duplicates, which are shallow, which are missing, and which should be automated, explored, or discarded?

A practical standard

The practical standard is to define the decision before defining the test. For this topic, the release question should make two priorities explicit: first, risk relevance: does this case protect a meaningful failure mode?; second, specificity: does it use real domain rules and data conditions?. If those priorities are not visible in the test plan, the team may still be busy, but it is not producing the kind of evidence that should influence a serious release decision.

Review Criteria for Generated Tests

Risk relevance: does this case protect a meaningful failure mode?
Specificity: does it use real domain rules and data conditions?
Evidence value: would the result change a decision?
Level fit: should this be unit, API, integration, UI, or exploratory?
Maintainability: will this test survive normal product change?

Example in practice

A model generates password tests around length and special characters. A tester adds account lockout, breached passwords, reset-token reuse, session invalidation, audit logging, and support impersonation because those risks are specific to the product.

What strong evidence looks like

A curated evaluation set tied to named product risks.
Clear criteria that separate acceptable variation from unacceptable failure.
Negative and adversarial cases that test how the system behaves under pressure.
Traceability from risk to test, control, monitoring signal, and owner.
A review path for ambiguous results instead of forcing every case into a false pass/fail answer.

Signals I would track

Human acceptance rate of AI-generated test assets after review.
Defects found from AI-assisted exploration versus conventional activity.
False confidence signals, including generated tests rejected as weak.
Time saved without reducing risk coverage or review quality.

Where teams get misled

Accepting generated tests as complete.
Automating generated cases without pruning.
Failing to include defect history in prompts.

How to govern the practice

Define review rules for AI-generated test assets.
Track how often generated tests find new risk.
Reward useful evidence, not artifact volume.

How to start this quarter

Start small, but make the work real. Pick one AI-affected workflow where the business impact is meaningful, then build a reusable evaluation pack around it. The first operational move is to define review rules for AI-generated test assets. After that, the team can expand the same pattern to adjacent workflows and make AI assurance part of the normal release system.

Choose one high-value workflow and document the user harm, business risk, and technical failure modes.
Build a compact evaluation pack with normal, edge, negative, and abuse-oriented examples.
Review results with product, engineering, security, privacy, or domain experts as the risk demands.
Keep failed examples and incident learnings in the regression suite so the organization gets smarter.

The discipline is to avoid accepting generated tests as complete. That sounds simple, but it is where many AI initiatives lose credibility. QA leaders should insist that AI makes the quality conversation sharper, not fuzzier.

Future signal

AI will make test-case creation cheaper. That means curation becomes more important, not less.

Sources worth reading

Regression Testing When the Model Changes Under You

AI model upgrades can change product behavior even when application code is untouched. Regression strategy must account for model, prompt, and data changes.

The product risk

Traditional regression testing is triggered by code changes. AI systems can regress when the model version changes, a prompt is edited, retrieval content is reindexed, moderation policy changes, or a provider updates behavior. The product may look the same while responses shift materially.

How testing changes

Regression suites for AI products should include stable evaluation sets, output comparison, severe-failure checks, safety and policy tests, latency and cost checks, and human review for high-impact changes. Teams need baselines and tolerances rather than exact string matching.

A practical standard

The practical standard is to define the decision before defining the test. For this topic, the release question should make two priorities explicit: first, model version or provider behavior change; second, prompt, system instruction, or guardrail change. If those priorities are not visible in the test plan, the team may still be busy, but it is not producing the kind of evidence that should influence a serious release decision.

AI Regression Triggers

Model version or provider behavior change.
Prompt, system instruction, or guardrail change.
Retrieval corpus, embedding, or ranking change.
Tool permission or integration change.
Policy, safety, or compliance threshold change.

Example in practice

A model upgrade improves fluency but starts answering policy questions with more confidence and fewer caveats. The regression suite catches a rise in unsupported claims even though user satisfaction samples initially look better.

What strong evidence looks like

A curated evaluation set tied to named product risks.
Clear criteria that separate acceptable variation from unacceptable failure.
Negative and adversarial cases that test how the system behaves under pressure.
Traceability from risk to test, control, monitoring signal, and owner.
A review path for ambiguous results instead of forcing every case into a false pass/fail answer.

Signals I would track

Coverage across normal, edge, adversarial, and abuse-oriented examples.
Failure rate by risk category, not only aggregate pass percentage.
Human review agreement for subjective or high-impact outputs.
Known failure examples that remain in the regression suite.

Failure modes to watch

Testing only after application-code changes.
Using exact output matching for variable responses.
Ignoring cost and latency regressions from model changes.

What strong QA teams do

Version AI dependencies like release-critical components.
Create model-change release gates.
Keep a rollback plan for AI behavior regressions.

How to start this quarter

Start small, but make the work real. Pick one AI-affected workflow where the business impact is meaningful, then build a reusable evaluation pack around it. The first operational move is to version AI dependencies like release-critical components. After that, the team can expand the same pattern to adjacent workflows and make AI assurance part of the normal release system.

Choose one high-value workflow and document the user harm, business risk, and technical failure modes.
Build a compact evaluation pack with normal, edge, negative, and abuse-oriented examples.
Review results with product, engineering, security, privacy, or domain experts as the risk demands.
Keep failed examples and incident learnings in the regression suite so the organization gets smarter.

The discipline is to avoid testing only after application-code changes. That sounds simple, but it is where many AI initiatives lose credibility. QA leaders should insist that AI makes the quality conversation sharper, not fuzzier.

Future signal

AI regression testing will become continuous because AI products will evolve through data, prompts, models, and controls, not only code.

Sources worth reading

Privacy Testing for AI Features

AI features create privacy risks through prompts, logs, training data, retrieval, memory, generated output, and third-party model calls.

The product risk

A conventional privacy test may verify masking, consent, retention, or access control. AI features add more paths. Users may paste sensitive data into prompts. Retrieved documents may contain confidential records. Model outputs may expose private context. Logs may store conversations longer than intended.

How testing changes

Privacy testing for AI must follow data across the full lifecycle: collection, prompt construction, model call, retrieval, storage, logging, evaluation, monitoring, and deletion. QA needs to work closely with security, privacy, legal, and platform teams.

A practical standard

The practical standard is to define the decision before defining the test. For this topic, the release question should make two priorities explicit: first, what sensitive data can enter prompts or retrieved context?; second, where are prompts, outputs, and tool calls logged?. If those priorities are not visible in the test plan, the team may still be busy, but it is not producing the kind of evidence that should influence a serious release decision.

AI Privacy Test Questions

What sensitive data can enter prompts or retrieved context?
Where are prompts, outputs, and tool calls logged?
Can one user retrieve another user's data?
Can generated output reveal hidden context?
Can data be deleted or excluded from future use where required?

Example in practice

A banking assistant uses customer transaction data. QA tests cross-account isolation, prompt logs, support access, generated summaries, and whether the assistant can be tricked into revealing data outside the user's authorization.

What strong evidence looks like

A curated evaluation set tied to named product risks.
Clear criteria that separate acceptable variation from unacceptable failure.
Negative and adversarial cases that test how the system behaves under pressure.
Traceability from risk to test, control, monitoring signal, and owner.
A review path for ambiguous results instead of forcing every case into a false pass/fail answer.

Signals I would track

Coverage across normal, edge, adversarial, and abuse-oriented examples.
Failure rate by risk category, not only aggregate pass percentage.
Human review agreement for subjective or high-impact outputs.
Known failure examples that remain in the regression suite.

Failure modes to watch

Assuming the model provider handles all privacy risk.
Ignoring logs and evaluation datasets.
Testing data access without testing generated disclosure.

What strong QA teams do

Add AI data-flow review to release readiness.
Use synthetic or anonymized evaluation data when possible.
Treat privacy failures as product quality failures.

How to start this quarter

Start small, but make the work real. Pick one AI-affected workflow where the business impact is meaningful, then build a reusable evaluation pack around it. The first operational move is to add AI data-flow review to release readiness. After that, the team can expand the same pattern to adjacent workflows and make AI assurance part of the normal release system.

Choose one high-value workflow and document the user harm, business risk, and technical failure modes.
Build a compact evaluation pack with normal, edge, negative, and abuse-oriented examples.
Review results with product, engineering, security, privacy, or domain experts as the risk demands.
Keep failed examples and incident learnings in the regression suite so the organization gets smarter.

The discipline is to avoid assuming the model provider handles all privacy risk. That sounds simple, but it is where many AI initiatives lose credibility. QA leaders should insist that AI makes the quality conversation sharper, not fuzzier.

Future signal

Privacy-aware QA will become a differentiator for AI products because trust can be lost through one convincing but unauthorized answer.