The Replication Lag Problem: How to Detect and Defend
Replication lag is the delay between a write being committed on the primary database and that write becoming visible on a replica. In most systems it stays under a second and nobody notices. When it grows to seconds or minutes, features that read from replicas show stale data in ways that look like bugs. I treat replication lag as an observable metric with alerting, not a background condition to ignore.
What you actually need to know
- Replication lag is normal in asynchronous replication. Unmonitored replication lag is a reliability gap.
- The failure mode is invisible to developers until a user reports seeing data that does not match what they just saved.
- The two most common causes are bulk writes on the primary and long running transactions on the replica.
- Read after write consistency is the application layer defense. Without it, a pattern of writing then immediately reading will sometimes show stale data.
- In my experience, teams that have never measured their replica lag are surprised by how often it exceeds one second under normal load.
Scenario
Lag behavior
Risk
Recommended defense
Normal transactional load
Under 100 milliseconds
Low
Monitor, alert at 2 seconds
Bulk insert or migration on primary
Spikes to seconds or minutes
High
Route reads to primary during migration
Long running transaction on replica
Blocks replay, lag grows
High
Time out long transactions on replica
Network partition between primary and replica
Lag grows until reconnection
High
Alert at 10 seconds, failover procedure ready
Write heavy spike
Lag grows during spike, recovers after
Medium
Alert at threshold, reduce replica read routing
The core argument
Adding a read replica is straightforward. The managed database services handle the setup in a few clicks. What the setup wizard does not tell you is that you have now introduced a consistency gap into your application that needs to be measured, managed, and communicated to the engineering team.
Replication in Postgres is asynchronous by default. The primary commits a write and returns success to the application. At some point later, that write is applied to the replica. The gap is usually milliseconds. Under load, during migrations, or when something goes wrong on the replica, the gap grows. If your application reads from the replica after a write and the write has not replicated yet, the user sees old data.
This is not a bug in replication. It is the documented behavior. The mistake is building an application that routes reads to replicas without any mechanism to handle the lag, and then being surprised when users report that their changes disappear and come back a second later.
The fix has two parts. First, measure the lag continuously and alert when it exceeds your application's tolerance. Second, implement read after write consistency for any operation where a user writes data and immediately reads it back. Neither is difficult. Both require deliberate engineering.
How to detect lag
Primary side measurement
In Postgres, the pg_stat_replication view on the primary shows the current write, flush, and replay lag for each replica.
SELECT application_name, write_lag, flush_lag, replay_lag FROM pg_stat_replication;
This is the most direct measurement. Replay lag is the number you care about. It is the gap between the last WAL record the primary sent and the last one the replica applied.
Replica side measurement
On the replica, pg_last_xact_replay_timestamp() returns the timestamp of the last transaction replayed. Comparing it to now() gives the lag in wall clock time.
SELECT now() - pg_last_xact_replay_timestamp() AS replication_lag;
This is the metric to export to your monitoring system. Export it every ten seconds. Alert at your defined threshold.
Managed service metrics
AWS RDS and Aurora, Google Cloud SQL, and Supabase all surface replica lag as a managed metric. Connect it to CloudWatch or your observability stack and set alerts there. The managed metric is slightly less granular than the Postgres view but requires no custom instrumentation.
How to defend against lag in the application
Read after write consistency
After a user writes data, route their next read for that resource to the primary for a short window. Two common approaches:
Session based routing. After a write, set a short lived flag in the user's session. While the flag is set, the application sends reads to the primary. The flag expires after a few seconds or after the lag metric drops below the threshold.
Replication watermark. Record the primary's current LSN (log sequence number) after a write. Before routing a read to the replica, compare the LSN to the replica's last applied LSN. If the replica is behind, route to the primary instead.
The session based approach is simpler. The watermark approach is more precise. For most SaaS products, session based routing is sufficient.
Lag aware routing
Route reads by workload type, not randomly. Transactional reads that follow a recent write go to the primary. Analytical reads, dashboard aggregations, and background report queries go to the replica. This separation reduces the number of code paths that need read after write consistency, because only the transactional reads carry the risk.
What it actually costs to ignore vs. defend
Approach
Engineering time
Ongoing risk
User experience impact
No monitoring, no defense
Zero
High
Occasional unexplained stale data
Monitoring only
Half a day
Medium
Known when lag spikes, no automatic protection
Session based read after write
1 to 2 days
Low
Consistent reads after write
Watermark based routing
3 to 5 days
Very low
Precise, minimum primary load
Synchronous replication
Configuration change
Near zero
No lag, but every write is slower
What to look for in a defended replica setup
- Replication lag exported as a metric with a named alert threshold.
- A documented read and write split policy that every engineer on the team can find.
- Read after write consistency implemented for any flow where a user writes and then reads.
- Timeouts on long running queries set on the replica to prevent blocking replay.
- A runbook for lag spikes that covers when to route all reads to primary and how to confirm the replica has caught up.
- Replica promotion tested in staging, not discovered for the first time during a primary failure.
Expert opinion
Replication lag is one of those problems that feels theoretical until it is not. A team ships the read replica, routes some queries to it, and ships the feature. Six months later a support ticket arrives from a user who saves a setting and sees the old value. The engineer cannot reproduce it in local testing because local has no replica. The root cause is a two second lag spike during a deployment. The fix is half a day of work. The damage to user trust takes longer to repair than the code does.
Yashveer Singh, founder of Yashveer Labs
How this played out on a real project
A client running a B2B project management tool had added a read replica to handle their growing dashboard query load. Three months later the support team started receiving tickets from users who reported that creating a new project did not show up in the project list. The list showed the state from before the create. Refreshing after a few seconds fixed it.
The root cause was a bulk data migration a teammate had run during business hours, which created a replication lag spike of twelve seconds. During that window, users who created projects were being routed to the replica for the list read, and the replica had not yet applied their write.
We added replica lag monitoring with a two second alert, implemented session based read after write routing for the project create flow, and added a policy that bulk operations on the primary require a maintenance window. The user reports stopped immediately. The lag monitoring has since fired four times, all during deployments that ran database migrations. Each time we temporarily increased routing to the primary and the lag closed within a minute of the migration completing.
For the broader read strategy context, see the read heavy workload: strategies that move the needle. For how consistency guarantees interact with replica behavior, see ACID vs BASE: when each belongs in your architecture.
Common mistakes
- Adding a read replica without monitoring its lag. You will discover the lag problem from a user support ticket.
- Routing all reads to the replica including reads that immediately follow writes. This is the most common source of the "my changes disappeared" bug.
- Running heavy migrations on the primary during business hours without temporarily reducing replica routing.
- No alert on lag at all. The replica can fall minutes behind before anyone notices.
- Not testing replica promotion before you need it. The first time you promote under pressure is not the time to learn the procedure.
- Using the replica for sessions or authentication reads. Those are pairs of writes and reads by nature and belong on the primary.
- Setting no timeout on long running queries on the replica. A slow analytical query on the replica can block WAL replay and cause lag to grow while the query runs.
A 30 day plan
- Week one. Add replication lag monitoring. Export the lag metric from the replica every ten seconds. Set a warning alert at two seconds and a critical alert at thirty seconds.
- Week two. Audit your application's read routing. Document every code path that routes reads to the replica. Identify any that follow a user write.
- Week three. Implement session based read after write for any flow where a user writes and then reads. Deploy and verify that the stale data bug class is gone.
- Week four. Write a runbook for lag spikes. Define the threshold at which all reads route to primary, how to confirm the replica has caught up, and who is responsible for making the call.
For the write side of database reliability, see zero downtime database migrations: a step by step guide. For the broader reliability patterns around background jobs and event delivery, see why your SaaS should have a job queue from day one.
FAQ
Frequently asked
- How do you measure replication lag in PostgreSQL and what is an acceptable threshold?
- What causes replication lag to spike on a database that has been stable for months?
- How do you prevent a user from reading their own stale write when the app uses a read replica?
- What happens to a read replica under a heavy migration on the primary database?
- Can you use a read replica as a failover target for high availability?
- What is the difference between replication lag and write amplification?
- How should you alert on replication lag in a production SaaS system?
Author
About the author and why it matters
Yashveer Singh wrote this. I run Yashveer Labs out of New Delhi. The work I take on tends to come from founders who have been burned by an agency, a freelancer, or their own ambition. I do not promise miracles. I promise that the system will be online, the code will be readable, and the next engineer who touches it will not curse me. That is rarer than it should be.