Verify over assume

A change to the database returned the message "Success. No rows returned." Sounds good. The problem: nothing had actually changed. The message was true and misleading at the same time.

When a script tried to use that change a little later, it failed immediately. The function that should have been there according to the message did not exist. The reassurance had been false, and only because the next step tripped over it right away did it come to light.

The message that says nothing

"Success. No rows returned" with this kind of change only means there was no error. It says nothing about whether the intended effect actually occurred. An empty command gives the same message as a successful one. No error is not the same as done.

That is a subtle but important distinction. We are inclined to read "no error message" as "it went well". But the absence of an error is not proof of success. It is only the absence of one specific kind of failure.

The lesson was simple and cost almost nothing: after such a change, do not trust the message, but separately check whether the intended result is really there. One extra check, a few seconds, and you know for sure instead of hoping. Since that morning it has become a fixed step: make the change, then separately confirm it is really there before you build anything on top of it.

Assumptions that cost money

This touches on a broader principle, and it became clear in a second way that same day. The system has a separate step that checks the exit rules every morning, the moments when positions should be sold to limit a loss or secure a gain. That step runs separately from the rest.

The assumption was: if that step is set up, it also runs. But what if it quietly does not run? No error message, no alarm, just a morning where the check is skipped. A missed exit check is the most expensive fault there is in this system, because then a position stays open that should have closed.

The same principle as with the database message: do not assume something runs, but make sure you know if it does not happen.

A watchdog that stands outside the system

The solution is a mechanism that works the opposite way to what you would expect. Instead of the system raising an alarm when something goes wrong, an external service expects a sign of life every morning. A short signal that says: I ran, all is well. If that signal fails to arrive, the external service raises the alarm.

The beauty is in that reversal. An alarm that sits inside your system fails along with your system. If the whole machine is down, the alarm is down too. But a watchdog that stands outside your system and expects a sign of life notices exactly the silence. No signal is the alarm.

That is why it catches everything at once. Whether the connection to the broker drops, a step fails, or the whole thing does not run, in all those cases the morning signal fails to arrive and the bell rings. You do not need to anticipate in advance in what way it can go wrong. You only need to know that in the morning a sign of life is supposed to come.

What this taught me

Both stories, the false message and the silent check, come down to the same thing. A system that is going to run with money must not lean on assumptions about what happens. It has to prove that it works, or it has to warn you when it does not.

"No error message" is not proof. "It is set up" is not proof. The only proof is that you checked it separately, or that there is an independent watchdog that notices when it goes silent.

Do not trust the silence. Make the silence itself the alarm.