Agile Does Not Guarantee Quality: What QA Must Add

Agile can improve feedback speed, but it does not automatically improve quality. A team can run daily standups, sprint planning, reviews, and retrospectives while still producing fragile software.

Quality comes from engineering discipline, product clarity, technical practices, and honest risk communication. Agile creates opportunities for those practices; it does not replace them.

The common misconception

Some teams assume that frequent delivery means frequent validation. In reality, frequent delivery without strong testing, automation, code review, observability, and product discipline simply creates faster opportunities to release defects.

Short iterations expose problems sooner, but only if the team is looking for the right problems.

What QA must add

• Risk analysis during backlog refinement.
• Concrete examples that turn vague stories into testable expectations.
• A balanced automation strategy across unit, API, integration, and UI levels.
• Exploratory testing for ambiguity and workflow risk.
• Clear release evidence and residual-risk communication.

The leadership point

A QA leader should not ask whether the team is Agile in name. They should ask whether feedback is fast, evidence is meaningful, quality ownership is shared, and retrospectives lead to better engineering behavior.

Agile is a delivery framework. Quality is an engineering capability.

How to apply this inside an Agile team

The practical move is to bring this thinking into refinement and sprint planning. Before implementation starts, ask what risk the story carries, which examples clarify the expected behavior, and what evidence will be needed before the work can be considered releasable.

Agile quality improves when testers influence the conversation early. If QA only reacts after development is finished, the team may be using Agile ceremonies while still operating with a late-inspection quality model.

Testing Must Be Business Driven: Aligning QA Effort With Risk and Value

Testing should be driven by business risk, not by habit, checklist volume, or the easiest tests to execute. A defect in a rarely used report and a defect in a payment workflow do not carry the same consequence.

This is where QA becomes strategic. The team must understand what the business needs to protect and then design testing around that reality.

Business context changes the test strategy

For a banking workflow, data integrity, auditability, authorization, and transaction correctness may be critical. For a consumer app, usability, performance, compatibility, and conversion may dominate. For an internal operations tool, reliability and workflow efficiency may matter most.

The same generic regression plan cannot serve every context equally well.

Questions QA should ask

• Which user journeys create revenue, trust, safety, compliance, or operational impact?
• Which failures would create the most expensive recovery effort?
• Which risks are most likely given the change, architecture, and defect history?
• Which evidence would actually change the release decision?

From testing activity to decision support

Business-driven testing does not mean ignoring technical quality. It means connecting technical evidence to business consequence. Performance, security, accessibility, data quality, and maintainability all become stronger arguments when framed in terms of user and business impact.

Testing earns leadership attention when it explains risk in the language of value.

How this shows up in QA leadership

A QA leader can use this idea to improve the quality conversation in a team. Instead of asking only whether testing is complete, ask what risk has been reduced, what evidence supports that claim, and what decision the team is now better able to make.

That is the difference between QA as activity tracking and QA as technical leadership. The strongest quality professionals make uncertainty visible in a way that helps people act.

Testing vs field observed defects

Myers put forth the counter-intuitive principle in software testing which states that the more defects found during formal testing the more that remained to be found later.

There seems to be a positive co-relation between the rate of defects found during formal testing with the rate of defects reported from the field. Higher rate of defects reported during a formal testing exercise usually means that there has either been a higher rate of error injection during the development process or that a new and more effective approach to testing has been followed. It could also be the case that a lot of additional and extra-ordinary test effort was expended resulting in the higher rate of defects being found.

A popular analogy to describe the relationship between defect rates during formal testing and field trials is to consider the overall defect rate as an iceberg. The tip that is visible is likened to the defects found during testing and the submerged portion as the latent field defect rate. The overall size of this iceberg is determined by the level of error injection during development. Formal testing normally happens once the code is developed and integrated by which time the “iceberg” is already formed. The larger the tip that is visible, the larger would be the entire iceberg.

This does not mean that we just get into a mode of acceptance about the latent defects that would be revealed during field usage. We can take steps to reduce the extent of the latent defects and bring up more of the iceberg above water. It must be stressed that managing of quality of the development process is important and can contribute towards reducing the rate of error injection. Prevention is definitely better than trying to determine and fix defects (in the process probably introducing other defects). Even with robust processes, some amount of error injection cannot be ruled out and this is where practices such as good design & code reviews and inspections are needed. Additionally, unit and integration tests by developers prior to checking in code into the repository should help reduce the number of defects that are left lurking around. The testing team must also continually enhance their tests, improve coverage and analyse defect rates & trends across releases to make sure that testing is doing its best to find as many issues as it can.

Defect Effort: Why Fix Cost Depends on When and How You Find Problems

The effort associated with a defect is not only the cost of changing code. It includes diagnosis, retesting, communication, planning disruption, release delay, support impact, and sometimes customer recovery.

That is why defect prevention and early detection matter. A small ambiguity in requirements can become a large effort later if it spreads into design, code, tests, documentation, and customer behavior.

Not all defects cost the same

A typo caught during review may cost minutes. A data corruption issue found after production release can require emergency fixes, data repair, customer communication, audit review, and loss of trust. Counting both as one defect hides the real economics.

Severity, detectability, affected users, recovery effort, and timing all influence the true cost.

What QA should measure

• Where defects are introduced and where they are detected.
• How long diagnosis takes after discovery.
• How much retesting and release coordination each defect creates.
• Which defect classes repeat across releases.
• Which upstream decisions would prevent recurrence.

The better conversation

Instead of celebrating a high defect count, ask what the defects reveal about the engineering system. Are requirements weak? Is test data poor? Are interfaces unstable? Are logs insufficient? Are automation signals too late?

Defect effort is a feedback signal. Use it to improve the system, not only to report activity.

How to use this in defect reviews

A practical way to use this idea is during defect triage or retrospectives. Pick a recent defect and separate the visible failure from the underlying cause. Then ask what would have prevented it, detected it earlier, or made it easier to diagnose.

That conversation turns defect handling into engineering improvement. It also helps QA move beyond counting defects and toward explaining what defect patterns reveal about requirements, design, data, automation, and team communication.

The strongest defect reviews end with an action the team can actually take. That might be a clearer acceptance example, a new API-level check, better logging, improved test data, a design-review prompt, or a change to release criteria. Without that action, defect analysis becomes commentary rather than improvement.

Privacy of test data

Privacy of data used in testing is something that organizations must consider. It is not uncommon to observe organizations using a copy of their production data to facilitate testing of their applications. This usage automatically exposes private data to internal constituents such as testers, administrators of the database, developers and others who have access to the data. Organizations tend to assume that since the test data and its associated environment reside within the organization's firewall, this data would be safe. In addition, the focus on securing test environments is often not high on the priority list. However, the fact remains that employees now have access to private data which include items such as credit card information, financial data, ssns, etc. Providing such access violates privacy regulations, enables data theft and misuse by internal staff and even exposes the data to external hacking. Given the levels of security surrounding a test environment, all that hackers need to do is to break into the corporate network and help themselves to the data mine resident in the test databases.

The reasoning for use of production data in testing is to perform real-life and comprehensive testing of the application. While this may be true, organizations cannot ignore the risks involved in simply using a copy of production data as-is on the test databases. Couple of techniques that may be followed to mitigate the risks would be – to generate test data and to mask sensitive data.

Generating test data eliminates the need to use copies of production data. Organizations may choose to use a mix of production (non-sensitive) data along with generated (sensitive fields such as card numbers, etc) data. Test data generation is not as simple as it sounds. Difficulties in generating data that represent the various possible real life use cases is not an easy task. The greater the complexity of the application being tested, the greater is the difficulty in generating suitable test data.

Masking of production data is another technique that may be used to maintain data privacy. Masking is also known as scrubbing or sanitization of data. Sensitive data is masked using various algorithms so that private data remains hidden from view. Several vendors offer data masking solutions. The advantage of masking data is that testing can happen with real data. However, data masking for larger and complex applications requires considerable effort and expense to implement.

The pesticide paradox

An interesting analogy comparing Software Testing with the use of pesticides in farming was presented by Beizer in his book on Software Testing techniques. He called it the pesticide paradox.

Repetitive use of the same pesticide mix to eliminate insects during farming will over time lead to the insects developing resistance to the pesticide thereby rendering the mix ineffective. A similar phenomenon may be seen while testing software. As testers keep repeating the same set of tests over an over again, the software being tested develops immunity to these tests and fewer defects show up. As you execute the same set of repetitive tests over an over again, your software eventually builds up resistance resulting in nothing new being revealed by the tests.

Further, every method you use to prevent or find bugs leaves a residue of subtler bugs against which those methods are ineffectual. In order to overcome the pesticide paradox, testers must regularly develop newer tests exercising the various parts of the system and their inter-connections to find additional defects. Also, testers cannot forever rely on existing test techniques or methods and must be on the look out to continually improve upon existing methods to make testing more effective.

***

Liked this entry? Join my community of professional testers to receive fresh updates by email. Use this link to add your email address to the community. Rest assured, I will neither spam nor share your email address with anyone else. Your email id will remain confidential. Subscriptions are handled by Google's FeedBurner service.

Defects Are Useful When They Improve the System

A defect is not good news, but it can be useful information. It reveals a gap in understanding, design, implementation, test coverage, data, environment, communication, or release control.

The value of a defect depends on what the team does after finding it.

Beyond fixing the individual issue

Fixing the immediate bug is necessary, but it is not the whole lesson. A mature team asks why the defect was possible, why it was not caught earlier, and whether similar defects could exist elsewhere.

That mindset turns defect management into engineering learning.

Questions worth asking

• Was the requirement ambiguous or missing?
• Was the design difficult to test or observe?
• Did automation cover the wrong level?
• Was the test data unrealistic?
• Did the defect expose a pattern rather than a one-off mistake?

The leadership behavior

Blame makes defects less useful because people hide risk. Good QA leadership creates enough safety for honest analysis while still expecting professional discipline.

The goal is not a culture that celebrates defects. The goal is a culture that learns from them faster than competitors do.

How to use this in defect reviews

A practical way to use this idea is during defect triage or retrospectives. Pick a recent defect and separate the visible failure from the underlying cause. Then ask what would have prevented it, detected it earlier, or made it easier to diagnose.

That conversation turns defect handling into engineering improvement. It also helps QA move beyond counting defects and toward explaining what defect patterns reveal about requirements, design, data, automation, and team communication.

The strongest defect reviews end with an action the team can actually take. That might be a clearer acceptance example, a new API-level check, better logging, improved test data, a design-review prompt, or a change to release criteria. Without that action, defect analysis becomes commentary rather than improvement.

Wide band Delphi (WBD)

Wide band Delphi (WBD) is a structured estimation technique involving an expert group. There is a lot of literature around the details of implementing this technique. In brief, this technique involves getting a group of “experts” to make estimates, discuss their assumptions and arrive at a consensus estimate. The estimates made by a group of experts with their varied perspectives are expected to be better than that made by any single individual who may not have the breadth or depth of understanding about the various activities involved.

In this technique, the team of experts begins by analysing the scope / specification of the work being estimated, brainstorms assumptions and creates a work-breakdown structure (WBS). Members of the team then make estimates individually for items in the WBS and note any further changes to the WBS and assumptions. The team then meets together to arrive at a consensus on the estimates. The meeting is facilitated by a moderator who charts the estimates without revealing the estimators and guides the group towards understanding the range of estimates, clarifying any assumptions, revising estimates, in a cyclical process until a consensus is reached.

While implementing the WBD technique it is important to assemble the appropriate team to generate estimates. It is a good idea to involve representatives from different functions who have a stake in the product so they together can agree upon the estimates and feel a sense of ownership of the plan. The technique is useful for new projects or projects where there are multiple factors and uncertainty. WBD helps refine and develop the WBS as well as clarify assumptions around estimates. The technique however, does take time and requires multiple experts to come together and make estimates.

Software Complexity: Why Complex Systems Need Smarter Test Strategy

Software complexity changes the way testing must be designed. A simple feature can often be verified with straightforward examples. A complex system requires thinking about dependencies, data states, timing, configuration, permissions, and operational behavior.

The danger is pretending that more test cases automatically control complexity. Often they only create more activity.

Where complexity hides

Complexity appears in branching business rules, integrations, asynchronous processing, legacy dependencies, feature flags, data migrations, distributed systems, and exception handling. It also appears in human workflows where users make unexpected but reasonable choices.

The more paths a system has, the more important it becomes to choose tests by risk rather than attempting exhaustive coverage.

How testing should respond

• Use models, diagrams, and examples to understand the system before designing tests.
• Prioritize high-impact paths and high-change areas.
• Test boundaries, states, permissions, failures, and recovery paths.
• Use automation where repeatable signal is valuable.
• Use exploratory testing where ambiguity and interaction risk are high.

The senior QA perspective

Complex systems require test strategy, not test volume. The best testers simplify the problem enough to reason about it without ignoring the risks that matter.

Complexity is not an excuse for weak evidence. It is a reason to be more deliberate.

How to use this in practice

A useful way to apply this topic is to take one active feature or release and map the concept to real risk. Identify what could fail, who would be affected, what evidence already exists, and what evidence is still missing.

The point is to turn software complexity: why complex systems need smarter test strategy from a definition into a working habit. Good QA practice changes how teams review requirements, choose tests, interpret failures, and explain release confidence.

Cyclomatic Complexity

One of the more popular complexity measures is McCabe's Cyclomatic Complexity (CC). The theory behind CC is simple: CC is a measure of the number of control flows within a module. A module is defined as a set of executable code that has an entrance and an exit. Control flow helps determine the number of paths through the module. The greater the number of paths through the module, the greater is the module's complexity.

The cyclomatic number for a module is equivalent to the number of linearly independent paths through the module and can be used to determine the minimum number of distinct tests that must be executed to test every executable statement at least once.

CC measurements may be performed ..

1. by counting the nodes (correspond to the corners) and edges (correspond to the bodies of the arrows) of the module graph
CC = # of edges - # of nodes + 2
2. by counting the number of binary decision points.
CC = # of binary decisions + 1

After we calculate the CC number for a module, what do we do with it and what does the CC number mean ? Stated simply, a higher CC signifies greater complexity of the module and corresponds to greater difficulty to test and maintain the module. Rules have been put forth on interpreting CC numbers. One such rule indicates that a CC > 20 signifies a high degree of complexity and risk of code being prone to defects. There are also rules that try to predict the probability of introducing regressions or inserting defects while trying to fix another defect, using the CC number. Here too, the higher CC corresponds to a greater probability of introducing new / additional defects while trying to make fixes to other defects. CC is helpful in trying to gain an insight into the difficulty to maintain and test code.

The following are extensions of Cyclomatic Complexity.

• CCD (Cyclomatic Complexity Density) is used to predict maintenance productivity and is derived by dividing CC by LOC (Lines of Code). Higher CCD corresponds to lower maintenance productivity.

• ECC (Essential Cyclomatic Complexity) measures the cyclomatic complexity after the structured constructs (such as if, while, case, sequence) are removed.