The Build Log · How we deliver

The Report Said It Was Fixed

There is a particular kind of silence that follows a tester writing back to say the bug is still there. You open the fix report. It is dated three weeks ago. It describes the defect accurately, names the file, explains the change, and marks the item closed. The tester is not lying. The report is not lying either. Both are describing something real, and the distance between those two realities is where most of the cost in software actually lives.

I have spent the last year building two platforms that had no tolerance for that distance. One is a multilingual careers platform serving migrant workers across the Gulf, where a broken translation is not a cosmetic issue but a person misreading their own employment contract. The other is an elder care platform used by families spread across two dozen countries, where a failed emergency alert is the only failure that matters. Both were built fast. Both taught the same lesson from opposite directions, and the lesson has almost nothing to do with writing better code.

The first platform: what deployment actually means

The careers platform went live at the end of December. Eighty eight backend functions, twenty nine pages, eight languages including Arabic and Urdu, and a support burden that arrived immediately in the form of two testers who took the assignment seriously and filed structured observation documents against the live site.

Their reports and our fix reports disagreed. Repeatedly.

The instinct in that situation is to defend the work or to redo it. Both are wrong. What we did instead, eventually, was download the actual contents of the production bucket, unzip it, and read the deployed files line by line against every claim in the most recent fix report. That audit took an afternoon and reorganized the entire backlog, because it revealed that three completely different problems had been sitting in one pile wearing the same face.

The first group was fixed in code and invisible in the browser, because the content delivery layer was still serving the previous version of the file. The fix was real. The tester was also right. Nobody was wrong, and nobody had a way to know that until somebody looked at the bytes.

The second group had been reported as fixed and never written. Not maliciously. A long session ends, a summary gets composed from intent and not from evidence, and intent hardens into record.

The third group was the interesting one. Those items could not be fixed by the approach being used at all. A usage limit that refused to reset had been documented as a browser storage fix, but the deployed page no longer used browser storage for limits; it called a backend endpoint. Every additional hour spent on the front end was structurally incapable of resolving it. The tester had already told us so, in the plainest possible language, by reporting that the problem persisted in a different browser. State that follows you across browsers lives on a server. We had the answer in the bug report and read past it.

The other failures that year were similar in shape. A request blocked before it reached the backend, which looked like an application error and was actually a preflight rejection on a gateway that did not own the route we assumed it owned. A language switcher that correctly flipped the page to right to left while leaving every word in English, because the translation applier wrote text content into elements whose translations contained markup. Arabic that survived every local test and arrived at the bucket as garbled characters, because the upload path corrupted the encoding.

Each of those took minutes to repair and days to see.

The gap is countable, and it is worth counting. Six and a half weeks passed between launch and the audit. Inside that span the fix report reached its eighteenth revision and the testers filed a third round of observations against items that had already been closed once. My own estimate of the engineering effort spent re-treating defects that were already correct in the deployed code is about three working days, which is the smaller half of the bill. The larger half was calendar, because every cycle had to wait for a tester to come back before anyone learned anything new. What I do differently at the start of a build now is define the verification path before the first deployment goes out. Nothing closes on the strength of a fix report, closure requires evidence pulled from the deployed system itself, and cache invalidation is written into the deployment step and not remembered after someone complains.

The second platform: when the instrument is the defect

The elder care platform approached the same problem from the other side. Instead of waiting for testers, we built a harness that would fire every one of two hundred and ten endpoints in a single pass and produce a grid.

The first serious run returned four passes and one hundred and fifty three failures. A later run returned one hundred and seventy seven network errors.

A result like that is not a backlog. It is a message about the measurement.

Roughly a hundred and twenty of those failures shared one signature, and a signature that uniform is never a hundred and twenty independent bugs. The blank status codes meant the browser had refused the request before any backend function woke up, which pointed at gateway configuration and not at a single line of application logic. One scripted pass across every route resolved the entire group.

The harness itself was responsible for another portion. It was sending a malformed authorization header on some routes and the wrong verb on others, manufacturing failures against functions that were perfectly healthy. There is something clarifying about discovering that your thermometer has a fever. We stopped fixing the code and fixed the test runner first, and a block of failures closed without a single deployment.

Then came the naming. The function serving updates on an elder record was named for a read operation. Another function, numbered in a way that implied it fetched something, deleted instead. Two different functions occupied the same slot in the numbering scheme with different permissions. None of this is exotic; it is what happens when a system grows quickly and names are chosen at the moment of creation, before anyone understands the thing. The consequence is that anyone debugging by reading names is debugging a fiction.

The bug that took longest to close was traced through authentication, then through gateway configuration, then through the backend function itself, all of which turned out to be correct at every step, before landing on a front end route that fired with a stale identifier. The satisfying part is not the fix. The satisfying part is that four theories were eliminated with read only inspection, without touching a production system that families depend on.

This gap is countable too, in a shorter and denser form. The harness moved from its seventh revision to its forty sixth in a little over two weeks, at better than one working session a day, and an identifiable block of failures across that period belonged to the harness and the gateway, not to any application code. Call it two to three working days spent on failures the system under test did not cause, compressed into seventeen days of calendar. What I do differently at the start of a build now is validate the instrument before the subject. A test harness is proved against a known good endpoint before its output is allowed to shape a backlog, and naming conventions, rollback artifacts, compile gates, and post deployment verification are fixed at project setup, where they cost nothing, and not somewhere past the hundredth function, where they cost a full re-audit.

What survived

Very little of what we learned is about technique. It is all about sequence and evidence.

Verify deployed state before believing any report, including reports written by your own hand a week earlier, because those have already been wrong more than once. Treat cache invalidation as part of deployment and not as a follow up, since a fix nobody can see is indistinguishable from a fix nobody made. When failures arrive in volume, group them by signature and repair the class, never the instance, because structural failures wear one face and distributed failures do not. Suspect the instrument before the subject. Read a function as it currently exists in production before running any automated edit against it, since a second occurrence of the same string is how silent damage enters a system. Back up the deployed artifact, compile check, deploy, wait for the platform to settle, then test with real credentials before and after. Never begin risky work at the end of a session, when the appetite for careful verification is lowest and the temptation to record intent as outcome is highest.

None of that is expensive. An afternoon of auditing what is actually deployed costs less than a single week of arguing with a bug report that was accurate the entire time.

The report said it was fixed. The user said it was not. Somebody has to go and look.