The Tests Passed. But Could We Debug It in Production?

I have been thinking about something we don't talk about enough in testing.

We spend a lot of time proving that software works.

But how much time do we spend proving that, when it doesn't work, we can actually understand why?

There is a difference.

A feature can pass its unit tests.

Its API tests can pass.

The automated regression can be green.

Performance checks can look fine.

The deployment can complete successfully.

And then something strange happens in production.

At that point, a completely different test of software quality begins.

Can we actually see what went wrong?

For me, observability is becoming much more connected to Quality Engineering than we sometimes acknowledge.

Especially now that more code is being generated faster, including with AI.

Passing the test is only part of the story

Consider a simple workflow.

Customer submits request
        ↓
API receives request
        ↓
Business rule executes
        ↓
Database updated
        ↓
Downstream service called
        ↓
Response returned

Our automated test might validate:

Expected response: 200 OK
Actual response:   200 OK

PASS

Great.

But production systems are rarely that simple.

What happens when the downstream service takes 15 seconds instead of 200 milliseconds?

What happens when the database update succeeds but the next operation fails?

What happens when the same message is processed twice?

What happens when the customer tells support:

"I submitted it, got confirmation, but nothing happened."

Now somebody needs to investigate.

And suddenly 200 OK is not very helpful.

This is where another type of quality becomes visible

When something goes wrong, we normally start looking for evidence.

Logs.

Events.

Correlation IDs.

Traces.

Metrics.

Error messages.

Audit records.

Database state.

External service responses.

And sometimes we discover that the application technically has logging, but the logs don't actually tell us anything useful.

Something like this:

INFO  Request received
INFO  Processing started
ERROR Operation failed
INFO  Request completed

Thanks.

We now know it failed.

We already knew that.

The useful questions are still unanswered.

Which operation failed?

Why?

For which transaction?

What was the state before it failed?

Was anything partially completed?

Can it safely be retried?

That is why I think we need to make a distinction between having logs and having useful observability.

They are not the same thing.

AI-generated code makes this even more interesting

AI coding tools are becoming very good at producing functional code quickly.

Give an agent a requirement and it may create the implementation, validation, tests and error handling in a surprisingly short time.

That is useful.

But there is an easy trap.

We ask the AI:

Build this feature.

Then we check:

Does it work?

Maybe we should also be asking:

If this fails at 2 AM,
will somebody understand what happened?

That is a very different requirement.

Functional correctness tells us whether the software produced the expected output.

Operational quality tells us whether we can understand its behaviour when reality becomes messy.

And reality always becomes messy eventually.

Testing should include failure visibility

When designing tests, we naturally think about failures.

Invalid input.

Timeouts.

Unavailable services.

Database errors.

Permission problems.

Unexpected responses.

But sometimes we only validate what the user sees.

For example:

Service unavailable
        ↓
Application handles exception
        ↓
User receives friendly message
        ↓
TEST PASS

From a user experience point of view, that may be correct.

But I would also want to know:

Was the failure logged correctly?

Can we identify the affected transaction?

Do we know which dependency failed?

Can operations distinguish this from another error?

Did an alert fire if the impact was significant?

Can somebody trace the request across services?

Those are also quality questions.

Maybe not every application needs sophisticated distributed tracing or huge observability platforms.

That is not my point.

The point is much simpler:

When important software fails, it should leave enough evidence for somebody to understand what happened.

The worst bugs are sometimes the ones we cannot reproduce

Most testers know this conversation.

"It happened once in production."

Okay.

Can we reproduce it?

No.

Logs?

Nothing useful.

Exact user journey?

Not sure.

Data state at the time?

Unknown.

Downstream response?

Not captured.

Correlation ID?

Don't have one.

And then comes everybody's favourite defect status:

Unable to reproduce.

That doesn't necessarily mean there wasn't a bug.

Sometimes it means the system did not leave enough evidence behind.

That is an important difference.

If a customer experiences a genuine failure but engineering cannot investigate it because the evidence does not exist, I would consider that a quality gap.

Maybe not a functional defect.

But definitely a quality problem.

Observability should start before production

This is where I think QA can contribute much earlier.

Instead of waiting until a production incident to discover what information we wish we had captured, we can ask those questions during design and testing.

For an important workflow:

What would we need to investigate if this failed?

That one question can lead to surprisingly useful discussions.

For example:

Critical transaction
        ↓
What could fail?
        ↓
What evidence would we need?
        ↓
What should be logged?
        ↓
What should be measured?
        ↓
What should trigger an alert?

Now observability becomes part of testability.

And I have always believed that good testability makes good testing much easier.

A green dashboard can still hide weak signals

There is another thing worth thinking about.

Engineering teams have become very good at creating dashboards.

Green ticks everywhere.

Build              ✓
Unit Tests         ✓
API Tests          ✓
Regression         ✓
Security Scan      ✓
Performance        ✓

READY

There is nothing wrong with that.

I like green dashboards as much as anyone working in QA.

But green tells us what we decided to measure.

It cannot tell us whether we chose the right signals.

The same applies in production.

A dashboard might show:

CPU             Normal
Memory          Normal
API             Available
Error Rate      Normal

Meanwhile customers are unable to complete an important business process.

Technically healthy.

Functionally broken.

That is why business-level observability becomes important as well.

Instead of only asking:

Is the API responding?

sometimes we should also know:

Are customers successfully completing the transaction?

Those two questions can produce very different answers.

QA needs to connect testing with production reality

I don't think QA should suddenly become SRE.

And I don't think every tester needs to become an observability engineer.

But the boundaries between these areas are becoming more interesting.

Testing before release gives us controlled evidence.

Production gives us evidence from real behaviour.

Strong Quality Engineering should probably learn from both.

Before release:

What do we believe will happen?

After release:

What is actually happening?

And when those two answers differ, we should be able to investigate why.

That feedback loop can improve requirements, automated tests, monitoring and future risk assessment.

AI can help here as well

There is a lot of potential for AI in this area.

Imagine giving AI access to:

Application logs
+
Distributed traces
+
Deployment changes
+
Test results
+
Known incidents
+
Metrics

and asking:

What changed?

What failed?

What is correlated?

What should we investigate first?

That could reduce investigation time dramatically.

But there is an important catch.

AI cannot analyse evidence that the system never produced.

If useful information was never logged, traced or measured, even the smartest model cannot magically reconstruct everything that happened.

Garbage in, garbage out still applies.

Perhaps in this case:

No evidence in, confident guess out.

And that is something I would be careful with.

"Works on my machine" is not a quality strategy

Software behaves differently in real environments.

Different data.

Different load.

Different integrations.

Different timing.

Different user behaviour.

Different failure conditions.

That is why I increasingly think quality has three important stages:

Can we prove it works?
        ↓
Can we detect when it doesn't?
        ↓
Can we understand why?

Testing helps answer the first question.

Good observability helps enormously with the second and third.

Together they give us something much more valuable than a green test result.

They give us evidence.

My view

As AI helps engineering teams create software faster, I think QA needs to keep widening the definition of what we mean by quality.

It cannot only be:

Does the feature work?

We also need to consider:

Can we understand its behaviour?

Can we identify failures quickly?

Can we trace important transactions?

Can we distinguish a technical success from a business failure?

Can we investigate something we did not anticipate during testing?

Because some of the hardest production problems will always be the scenarios nobody thought to automate.

And when that happens, the quality of the evidence left behind becomes extremely important.

So next time every test is green, there may be one more useful question to ask:

If this breaks in production tomorrow, will we actually know why?

For me, that is not just an operations question.

It is a Quality Engineering question too.