Testing is not where quality lives anymore

The next evolution of QA will belong to the teams that can turn operational data into engineering confidence, not the teams running the largest regression suites. Most organisations haven’t realised that yet. 

Quality is moving from pre-release gatekeeping and defect counting to continuous operational intelligence. Testing is now just one signal amongst many. Here is why that matters, and what it means for anyone leading QA today.

The pre-production illusion

For a long time, QA operated on a fairly simple assumption that if we test enough before release, we create confidence in production. That belief defined almost everything – from automation strategies and release gates to coverage models and even how QA teams measured success.

The problem is, modern systems no longer behave predictably enough for that model to hold.

Most applications today are no longer self-contained products with clearly understood execution paths. They are distributed ecosystems made up of APIs, event streams, cloud infrastructure, AI services, third-party integrations, feature flags, edge compute, and asynchronous communication.

The system your customers experience is no longer the same system you test in pre-production. And I think QA is still underestimating how significant that shift really is.

The real signals emerge post-deployment

What changed my perspective wasn’t AI, it was observability. But we need to stop looking at observability as just operational dashboards, and start treating it as a primary source of quality intelligence.

That distinction matters because most organisations still separate testing from operational insight – QA owns validation, SRE owns reliability, platform teams own telemetry, and operations owns incidents. But modern software systems don’t fail neatly inside those boundaries anymore.

The most critical quality signals now emerge after deployment:

• Latency degradation
• Retry storms
• Dependency failures
• Regional instability
• Customer abandonment patterns
• Degraded AI model responses
• Traffic anomalies
• Intermittent service interactions

Traditional testing rarely catches these behaviours because they only emerge under real-world, production conditions.

What the data shows

One engineering organisation I worked with analysed their observability data against escaped defects over a twelve month period. The findings were uncomfortable:

This wasn’t due to insufficient testing. The issue was that testing effort had become completely disconnected from operational reality. Large sections of their regression suite had almost no measurable relationship to production stability.

So, they rebuilt their strategy around observability signals instead of test volume. Counterintuitively, this resulted in fewer tests, but a much more resilient product.

Observability-driven quality in practice

When observability becomes a decision engine for quality, telemetry data becomes the starting point, rather than an afterthought.

Instead of guessing where risk lies, teams ask: 

• Which services degrade most frequently? 
• Which deployments correlate with incidents? 
• Which APIs produce recurring escalation patterns?
• Which dependencies create instability under load?

Telemetry flows in, engineering decisions flow out. Production traces dictate regression priorities, customer behaviour influences risk models, runtime telemetry exposes architectural weaknesses long before a customer reports a bug, and testing becomes adaptive and continuous.

The AI opportunity

Right now, most conversations around AI in testing are still centred on generating test scripts faster, and I think that misses the bigger opportunity entirely.

Generating more tests is no longer a hard problem. Interpreting system behavior at scale is.

Modern engineering teams generate an overwhelming amount of telemetry: logs, traces, metrics, user behaviour, and infrastructure signals. Humans simply cannot correlate that volume of information across complex distributed systems. AI changes that by helping engineering teams identify the patterns they would otherwise miss (e.g., a deployment pattern linked to latency degradation, or a flaky dependency causing customer abandonment).

QA’s strategic renaissance

For years, QA has been positioned as a delivery function focused on execution efficiency. Observability-first engineering changes that value proposition entirely.

The future QA leader will need to understand system behaviour, telemetry pipelines, runtime risk, AI observability, customer impact analytics, production reliability patterns, and platform engineering principles.

Confidence in modern systems comes from understanding how they behave under real conditions – continuously, at scale. The industry hasn’t fully caught up to this yet, but the teams that do will define what quality engineering looks like for the next decade.