AdPanel: the post that tried to publish 3,045 times

AdPanel is the system a content studio runs its clients' social media on. Each client gets their own workspace: a publishing calendar the team schedules into, a share link the client opens with no account to approve a post or send it back with a note, Instagram publishing on a schedule, comment-to-DM automations, and Stripe billing underneath it.

I am the only engineer on it. I built the client workspace model and its permissions, the calendar and the scheduler that publishes from it, the client approval flow and the notifications around it, per-client timezones, the Instagram and Shopify integrations, the Stripe subscription plumbing, and the product's own analytics. The team is a founder, an operations lead, an account manager and a creative lead, none of whom read logs.

Stack: Next.js 15 with the App Router and TypeScript, Supabase and Postgres, Clerk for auth and organisations, Stripe for billing, Vercel for hosting and cron, the Instagram Graph API for publishing, PostHog for product analytics, Tailwind for the interface.

Nobody reported this. There was no bug report, no angry client, no error on screen. Two months into the product I went looking through what it had been recording about itself, and found one post that had been retried 3,045 times since 3 September. About 271 attempts a day. One every five and a half minutes. A second client account had joined the same loop the day before and was already at 501.

For scale: at the time this was a studio running two live client accounts, publishing 27 posts in September and 9 the month before. A real post takes one attempt. So one broken post had generated, in eleven days, more publish attempts than the studio's entire legitimate output for the next nine years at that rate.

The cause was four lines of code in the wrong order. When a post comes due on an account that has not finished connecting Instagram, the publisher stops before it gets as far as marking the post failed. The post stays scheduled, so the next run picks it up again. And the next.

The contrast is what made it obvious. A post that fails for any other reason gets marked failed and retried once. Only this one path never stopped, because the check that fails it sits a few lines above the code that records the failure.

The calendar showed the post as scheduled. The client, looking at their own approved content, also saw it as scheduled. It was never going to publish and nobody was going to find out.

Put in the only terms that matter: a studio that charges a monthly retainer to run a brand's social media had a client whose account sat silent for eleven days, and did not know. Not "an error rate went up". A paying client received nothing, and the people responsible for noticing had a screen telling them everything was scheduled.

And the condition that triggered it, a client who has not finished connecting their Instagram, is the normal state of every account during onboarding. Not an edge case. The default one.

The publisher now marks the post failed with a message saying what to fix, before it gives up.

While I was in there I found a second way for a post to disappear: it is marked publishing just before Instagram is called, and moved on afterwards. If the function dies in between, neither happens, and the cron only ever picks up posts marked scheduled. One post had been sitting in that state since 3 September. The cron now fails anything that has been publishing for more than fifteen minutes.

It deliberately does not retry either of them. Instagram may already have published the post before we lost the thread, and publishing twice to a client's account is worse than not publishing at all. So it fails loudly and a person decides.

The same pass found the cron was starting up to twenty posts inside a sixty second budget, so the eighth post inherited whatever the first seven had left behind. That budget is now five minutes, and it stops starting posts it has no time to finish.

Live at midday on 14 September. Against a loop that had fired every five and a half minutes for eleven days, the last retry was at 10:55 and there has been nothing since.

While the loop was running, the product had fired 3,568 failure events between 3 and 16 September. A daily automated check was already pulling that number every single morning, and it had been read by nobody for eleven days.

So the bug was not really the four lines. The bug was that a system could be failing continuously, in writing, in a place someone had set up specifically to watch it, and still nobody knew.

A check that runs on a schedule and emails one person, only when something is actually wrong. It watches two things: a post whose scheduled hour has passed and never went out, which is what a stopped publisher looks like from outside, and any post that failed. Both mean a client sees nothing on their account and nobody knows.

A post still waiting is reported on every run, because it is still broken. A failure is reported once. If the check cannot read the database it says so rather than going quiet, because silence from a broken check is exactly what had just cost us eleven days.

Then I tested it against a real failing post instead of trusting it. The first scheduled run found the problem, composed the alert correctly, and could not send it, because the address it sends to had been added after the running deployment was built. Nothing was lost only because the check writes the alert to the log when it cannot send. The second run, after a redeploy, delivered the email.

That failure was the most useful thing that happened that day. A check nobody has watched fail is not a check yet.

For the studio: if a client's post does not go out, somebody now knows the same day instead of finding out when the client asks. That is the difference between a service you can sell a retainer for and one you hope nobody audits.

• The retry loop is gone, verified against the product's own record of attempts rather than against a screen.
• Posts can no longer disappear into publishing and stay there.
• A failing post now reaches a person within hours instead of never.
• The alert has been proven end to end, including its own failure path.

Every background job I write has to answer one question before I consider it done: if this stops working, who finds out, and how long does it take them. If the answer is nobody, the job is not finished, whatever it does when it works.