Marathon day five: the column that held up everything

Thursday evening May 14, around 22:00. I'm still at my laptop. My first real production cron was supposed to fire the next morning at 06:30 NL, pulling data, processing scores, and submitting orders to IBKR. On paper everything was tested. Tests green. Render services deployed. A paper account ready.

And yet something was off.

Every manual test on a test date kept returning 0 BUYs. No errors, no warnings. Just zero. For a system built to generate 5 to 15 signals daily that is not a small detail.

The cause I missed

It was in a column called score_30d_ago. A field in my scores_daily table that stores each ticker's composite score from exactly 30 trading days ago. The system needs that field for one of the BUY rules from the design doc: score must be trending up over 30 days. Without that field, no BUYs.

The problem was that the column contained NULL for every ticker. Nothing was filling it.

That was strange. I was sure there was a script that calculated percentile ranks and included score_30d_ago. Or so I thought. When I finally dug in, the script did exist, but it was missing one essential step. The 30d-lookup itself had never been implemented. Tests didn't catch it because tests used mock data where the value was simply pre-filled.

First fix, not enough

Friday morning May 15, 03:00 NL. I wrote a Cursor prompt for the first fix. A migration to clean up the column, plus the real lookup logic in the script. Three hours later it was done and tested. Commit pushed. CI green. Render auto-deployed.

First real production cron 07:00 NL. Output: 14 BUYs. It worked.

Except it didn't fully work. An hour later I noticed that score_30d_ago wasn't filled for all tickers. Only for a subset. The pagination in the script stopped at a short Supabase page, while it should have continued until there was truly no data left.

Two bugs in one day, in the same feature. Or really: one bug hiding behind another.

Second fix

Friday morning 08:30. I was now 14 hours at my laptop. The marathon had started before I noticed. Cursor wrote a second prompt for the pagination fix. Tests covering this specific scenario. Commit pushed.

Friday 08:45 NL: first real production cron with this fix in container. 14 orders submitted to IBKR. On IBKR Portal I could see they were actually placed.

Then on Friday evening came the question whether the orders had actually filled. And Saturday a full day of debugging to find out why the dashboard showed 0 positions while IBKR Portal showed fourteen. But that story is for another time.

What I really learned

What hit me most about this whole saga wasn't the bug itself. Bugs happen. It was that I didn't realize two things:

The first: I thought tests protected me from this kind of error. But tests using mock data only test the happy path. A NULL value in production isn't mock data. That needed manual verification by pulling real rows from Supabase, not by getting tests green.

The second: after 12 hours I'm no longer capable of making good decisions. That sounds obvious but in the moment it feels different. You think: just this last thing, then it's done. But the second fix I pushed in the morning was only possible because I wasn't making any risk-relevant changes anymore. Pure pagination fix, no risk to live trading state.

It would have been different if the fix had been risk-related. I shouldn't have done it then. My AI assistant should also have said "Erik, you've been at this for 22 hours. This is no time for production SQL." It didn't, that's something to sharpen.

The take-away

A system doing live trading doesn't become reliable by being perfect. It becomes reliable by quickly recognizing when something is wrong, and by discipline not to make risk-relevant changes under fatigue.

Score_30d_ago wasn't the end of the marathon. It was just the first in a series of launch blockers I came to know during a weekend. And each launch blocker is an opportunity to make the system stricter, simpler, or better protected for June 22. Five weeks until live with €10,000.