The Black Friday That Took Down the Startup

What you actually need to know

• Database connection pool exhaustion is the most common cause of outages driven by a traffic spike. Fix it before the spike.
• Load tests run at steady state do not reveal spike behavior. Run tests at 3x your peak with a period of ramping up.
• Latency percentiles are your early warning system. An alert on p95 at 2x baseline gives you response time before the outage.
• Scaling vertically first is faster than scaling horizontally. Have a vertical scaling runbook ready.
• The postmortem must be written within 24 hours and acted on within two weeks or it will not be acted on.

Failure Mode	Detection	Mitigation	Prevention
DB connection pool exhaustion	High p95 latency	Scale vertically, add PgBouncer	Connection pooler in production
Cache miss storm	CPU spike on DB	Warm cache before spike	Cache warming runbook
Queue backup	Job lag metric	Scale workers	Automatic scaling for queue workers
Server OOM	Instance crash	Restart, scale up	Memory profiling, load testing

The core argument

The spike that takes down a startup almost always comes on the day that matters most. A press feature. The product launch. Black Friday. The spike is predictable. The failure is not, because the team never ran the system under spike conditions in a controlled environment to see what would break first.

The autopsy is consistent across the outages I have seen. The database connection pool hit its limit. Requests started queuing for database connections. Response times climbed from 200ms to 2 seconds to 30 seconds. Users refreshed. The load increased. The queue grew faster than it drained. The system stopped responding. The founder was on the phone with AWS support while their biggest opportunity of the year was going down.

The failure is almost never in the code. The logic is correct. The feature works in staging. The failure is in the configuration and the capacity assumptions. A connection pool sized for 100 concurrent users fails under 1000. A server sized for ordinary, steady load runs out of memory under a ramp. A database that handles 200 queries per second at p50 hits 8 seconds at p99 when the query mix changes under load.

These failures are findable before they happen. A proper load test with a realistic traffic ramp reveals the connection pool limit, the memory ceiling, and the cache miss behavior. The load test is the most underinvested hour in every startup's engineering calendar.

What the load test should cover

A load test that is useful for spike preparation runs in three phases. First, the ramp up, where traffic increases linearly from zero to 3x peak over 15 minutes. Then the steady state, where traffic holds at 3x peak for 30 minutes. Finally, the ramp down, where traffic decreases back to zero.

During the test, watch four metrics. Database connection pool utilization: if it hits 80 percent during steady state, the pool is too small. p95 response time: should stay under 2x baseline during steady state. Memory utilization on the application servers: should not exceed 85 percent. Queue depth for background jobs: should not grow continuously during steady state.

When any of these metrics breach their threshold during the load test, that is the finding. The load test result is not pass/fail. It is a list of the things that will break first, in order, so you can fix them before the spike arrives.

Common mistakes startups make before a traffic spike

1. Not running a load test at all. Hoping is not a capacity plan.
2. Running a load test at 1.5x instead of 3x. Spikes are not gentle. The test should be more stressful than the expected spike.
3. Running the load test once in staging but not on infrastructure equivalent to production. Staging and production often have different database configurations, different connection pool settings, and different memory allocations.
4. Not having a scaling runbook. When the spike arrives, there is no time to figure out how to scale. The runbook should be written and practiced before the day.
5. Not alerting on latency percentiles. By the time errors appear, the outage is already in progress. The p95 alert catches it earlier.

Where to start: a three step spike preparation plan

Step 1: Run a load test this week. Use k6, Locust, or Artillery. Write a scenario that represents your most common user flows. Ramp from zero to 3x your current peak traffic over 15 minutes. Hold for 30 minutes. Watch the four metrics. Write down what broke first.

Step 2: Fix the first failure mode. Whatever hit its limit first in the load test is the thing most likely to cause an outage under a real spike. If it was the database connection pool, add PgBouncer or increase the pool size. If it was server memory, size the instances correctly or look for the memory leak. Fix the first failure mode before the next load test.

Step 3: Write the scaling runbook. Define the steps to scale vertically (bigger instances), to scale horizontally (more instances), and to enable circuit breakers or graceful degradation if the load is unmanageable. The runbook should be runnable by any engineer on the team without the original author present.

FAQ

Frequently asked

• How do I prepare for a traffic spike if I have never had one?
• What is the most common single cause of startup outages during traffic spikes?
• Should I scale horizontally or vertically to handle a spike?
• How do I detect that a traffic spike is happening before the system goes down?
• What is the most important thing to do after a traffic spike outage?

Author

Why this work lands with me

I am Yashveer Singh. Founder of Yashveer Labs. I take this kind of project because I have done enough of them to know what kills them. The version of me that writes a post like this is the same one who builds the system afterward. There is no handoff to a junior, no agency middleman, no surprise scope. That is the bet I am making on my own brand.

Start the conversation See the work DM on Instagram