Three times infrastructure, twice a bug

In a single day, four things went wrong with my trading system. A failed morning run, two failed builds, and a bug that only showed itself when a sale went through. Four red flags.

But two of those four were not a bug. They were infrastructure hiccups, noise that had nothing to do with the code. And recognising that difference, without falling for it, turned out to be the most important skill of the day.

What the noise was

The first hiccup: the connection between my server and the place where my code lives dropped during a build. The error message looked dramatic, a whole stack of red lines. But the core was at the top: the code could not even be fetched. The build never got to building.

The second hiccup: during another build a download dropped halfway. A broken connection to the place software packages come from. Another stack of red lines, another thing that looks scary, and again nothing to do with my code.

Both solved the same way: try again. No code change, no diagnosis, just one more time. The second attempt succeeded in both cases, because the hiccup had passed.

How you see the difference

The treacherous thing about a red build is that it always looks alarming, whether it is a real fault or a network hiccup. The trick is in where the fault lands, not in how loudly it shouts.

A few signals that help. Does it fail while fetching the code, or while building it? A fault before the build is almost never your code. Does the error message contain words like "connection broken" or "no access"? That points to infrastructure, not to a logical error. And the strongest check: would the same code work elsewhere? When eleven of my twelve services built fine with the exact same code, the one that failed was obviously not a code problem. A real fault in the code would have hit all twelve.

That last one is the sharpest filter. If the same thing works somewhere else, it is not down to what you just changed.

Why this matters

The temptation with a red flag is to immediately start digging into your own work. What did I break? But if it is noise, you lose an hour searching for an error that is not there. And worse, sometimes you then go and "repair" something that was not broken at all, and only then do you really break it.

The other side is just as dangerous. If you wave away every red flag as noise, you miss the real bug. Two of the four things that day were real, and they deserved full attention. The art is not to always stay calm or always dig, but to ask the right question each time: is this mine, or is this the world around it?

What this taught me

A system that has to run with money does not run in a vacuum. It hangs on servers, connections, external services, and those sometimes just hiccup. Part of building reliably is learning which failures to take seriously and which to simply retry.

The signal that helped me most here was not technical. It was the habit of first asking, with every fault, where it comes from, before I assume I did something wrong. That one question saves hours, and keeps you from repairing things that are not broken.

Not every red flag is your fault. But you do have to look each time to know that.