The Reconciliation Job: A SaaS Pattern Founders Should Know
A reconciliation job is a scheduled background process that compares two sources of truth, finds discrepancies, and either fixes them automatically or surfaces them for manual review. In SaaS the most common version compares local subscription state against the billing provider. But the pattern appears everywhere there are two systems that need to agree. It is the safety net under distributed state.
What you actually need to know
- Webhooks fail. Networks partition. Processes crash. The reconciliation job is what catches the state drift that results.
- The pattern is always the same: compare two sources of truth, categorize discrepancies, correct or escalate.
- Start in report only mode. Ship automatic correction only for discrepancy types you fully understand.
- Idempotency in the correction logic is not optional. The job will run multiple times on the same records.
- Teams that skip the reconciliation job discover they needed it during their first billing incident.
Approach to state synchronization
Reliability
Complexity
When it breaks
Webhooks only
Medium
Low
Network failures, handler bugs, retries
Polling only
Low
Medium
High latency, rate limits, missed events
Webhooks plus reconciliation job
High
Medium
Rare; discrepancies caught on next job run
Event sourcing with replay
High
High
Complex to build and operate
Manual sync on customer complaint
Very low
Minimal
Constantly
The core argument
Distributed systems drift. That is not a flaw in your architecture. It is physics. Two systems that communicate over a network will, at some point, disagree about the state of the world. The webhook that should have updated your subscription table arrived during a deploy. The retry that should have fired did not. The batch job that syncs accounts from your billing provider timed out halfway through. The state is wrong and nobody knows yet.
The reconciliation job is the pattern that catches this. It runs on a schedule, reads both sides of the agreement, finds the gap, and does something useful about it. For most SaaS products the most important instance is the billing reconciliation: compare your local subscription records against what the billing provider says is true. But the same pattern applies to any pair of systems that need to agree.
What I find surprising is how many SaaS products do not have this. They have webhook handlers. They have retry logic. They believe the system is consistent because it usually is. The first time a billing cycle runs while the webhook handler is broken, they find out the hard way. A week of subscription state corruption is a very expensive way to learn about reconciliation.
The job is also a monitoring tool. A reconciliation report that finds zero discrepancies is information. It tells you the real time path is working. A report that finds a hundred discrepancies tells you something upstream is broken. Teams that treat the reconciliation report as a daily dashboard catch upstream failures faster than teams that wait for customer complaints.
The structure of a reconciliation job
Phase one: data collection
Fetch the local records. Fetch the external records. Align them by a shared key, usually a billing provider customer ID or subscription ID. Identify the set that exists in both systems, the set that exists only locally, and the set that exists only externally.
Phase two: comparison
For each matched record, compare the fields that should agree. Subscription status. Plan tier. Renewal date. Trial expiry. Each field comparison produces a result: match, mismatch, or missing. Aggregate the results into discrepancy categories.
Phase three: correction or escalation
For each discrepancy category, a policy. Correct automatically if the right answer is unambiguous and the correction is low risk. Escalate to a review queue if the right answer requires human judgment or the correction could have customer visible consequences. Log everything.
The correction logic must be idempotent. Running it twice on the same record produces the same state, not a doubled change.
What it requires
Component
Engineering time
Notes
Basic billing reconciliation (report only)
Three to five days
Compare local vs provider, surface discrepancies
Automatic correction for clear cases
Three to four days
Idempotent fixes for unambiguous drift
Review queue for ambiguous cases
Two to three days
Operations interface for human review
Reconciliation report and alerting
Two days
Daily report, alert on discrepancy threshold
Extension to second system (inventory, partners)
Two to three days per pair
Pattern is reusable
What the reconciliation system must have
- A defined scope: which systems, which fields, which frequency.
- A dry run mode that reports without correcting. Start here.
- Idempotent correction logic for every automatic fix case.
- A review queue for discrepancies that require human judgment.
- A reconciliation report per run with counts for checked, matched, automatically corrected, and escalated.
- Alerting when the discrepancy rate exceeds a threshold.
- A run log that shows start time, end time, and outcome for every execution.
- Retry handling for the job itself, with a dead letter path for persistent failures.
Expert opinion
The reconciliation job is one of those things where the absence is invisible right up to the moment it is not. The teams that build it early have a boring, quiet billing history. The teams that skip it have an incident on a Sunday afternoon because a webhook silently failed three weeks ago and three hundred subscriptions are now in the wrong state.
Yashveer Singh, founder of Yashveer Labs
How this played out on a real project
A subscription SaaS client came to me after a billing incident. Their Stripe webhook handler had a bug that was silently failing for a specific payment method type. The bug had been present for eleven days. In that time, 42 accounts had renewed but were still marked as past due in the application. Support had been receiving confused tickets. Nobody had connected the pattern because each ticket looked like an isolated problem.
We spent two days writing a billing reconciliation script in report only mode. It found the 42 accounts immediately. We corrected them manually and fixed the webhook handler. Then we spent another week turning the script into a proper scheduled job with a nightly report and an alert when discrepancy counts crossed a threshold. The job has run nightly for eighteen months since. It has surfaced two more upstream issues before customers noticed.
The second issue it caught was a Stripe API change that shifted the format of a trial end date field. The reconciliation report showed trial expiry dates drifting on new subscriptions. We found and fixed the parsing bug within six hours of the first report. Without the job, we would have found it when trial users could not access the product after their trial ended. For the underlying delivery reliability pattern, see the outbox pattern a saas reliability cheat code and background job queues the architecture decision founders skip.
Common mistakes teams make
- Relying entirely on webhooks. The real time path will fail at some point.
- Shipping automatic correction before running in report only mode. You will not understand the discrepancy categories until you have seen them in production.
- Correction logic that is not idempotent. Two runs of the job create double corrections.
- No reconciliation report. The job runs, finds or does not find discrepancies, and you have no idea which.
- Running the job too infrequently. A nightly job on an active product still allows a day of drift.
- No alerting on discrepancy count. The report goes to a log nobody reads.
- Extending automatic correction to ambiguous cases too quickly. Human review exists for a reason.
- Building the reconciliation job as a one off script. It needs to run reliably for years.
A two week plan
- Days one to three. Identify the highest risk pair of systems. For most SaaS, that is local subscription state versus the billing provider.
- Days four to six. Write the comparison logic in report only mode. Fetch both sides, align by ID, compare the fields that should agree.
- Days seven to nine. Run the report against production in read only mode. Review the output. Categorize the discrepancy types you see.
- Day ten. Wire the job to the background queue infrastructure. Set a nightly schedule.
- Days eleven and twelve. Add automatic correction for the clearest, lowest risk discrepancy category. Keep the rest in the review queue.
- Days thirteen and fourteen. Add alerting. Ship the reconciliation report to a shared channel the team reads daily.
For the broader reliability context, webhooks the reliable pattern that most companies get wrong covers the real time path that the reconciliation job backs up.
FAQ
Frequently asked
- What does a reconciliation job actually do?
- When do I need a reconciliation job?
- What is the difference between a reconciliation job and just handling webhooks correctly?
- How do I prevent the reconciliation job from creating duplicate corrections?
- What is the reconciliation report?
- How often should the reconciliation job run?
- Can the reconciliation job break things?
Author
The reason my name is on this page
My name is on this page because I wrote what is on this page. Yashveer Singh. Full stack developer. Founder of Yashveer Labs. The portfolio is on the homepage. The projects are live. The code is real. The work is provable. If you have read this far, you already know whether the voice matches the standard you are looking for. The next move is yours.