This is one of the most common calls we get: "Our API is fine most of the day, then traffic spikes and everything times out." Below is an example scenario, a composite of patterns we see repeatedly, showing how we would debug it and what usually fixes it.
The setup
A B2B SaaS runs a Node.js API on four containers behind a load balancer, backed by a managed Postgres instance. At a normal 20 requests per second, the main GET /api/projects endpoint answers in about 120 ms. Every Monday morning, when customers log in at once, traffic climbs to around 200 requests per second, p95 latency jumps past 8 seconds, and the load balancer starts returning 504s.
The team's first move was to double the containers. It got worse.
That detail is the biggest clue. If more app servers make things slower, the bottleneck is something they all share.
Step 1: Measure the right thing
Average latency looked acceptable because most requests were still fast. The pain was in the tail. We added three views:
- p50, p95 and p99 latency per endpoint, not averages.
- Connection pool metrics: active, idle, and, most importantly, time spent waiting for a connection.
- Database load by query, using
pg_stat_statements.
Then we reproduced the Monday spike with a k6 load test against staging, using an arrival-rate executor that ramped from 10 to 250 requests per second.
Step 2: Find the queue
Every latency collapse under load is a queue forming somewhere. Under the load test, the picture was clear:
- Each container had a pool of 20 database connections; all 20 were busy, and pool wait time was the largest part of request latency.
- Database CPU sat at 60%. Busy, but not saturated. The database was not the bottleneck; access to it was.
- Doubling containers had doubled connections to 160, pushing the database into contention and making each query slower.
So the question became: why were connections held so long?
This is plain queueing. Once requests arrive faster than connections free up, the wait grows with every new arrival, which is why latency does not rise gently but goes nearly vertical near saturation. Counterintuitively, a smaller pool is often faster. HikariCP's pool-sizing guide starts from roughly (CPU cores × 2) + disk spindles for the database server, far below what most teams configure.
Step 3: The four root causes
1. An N+1 query inside the hot endpoint
GET /api/projects loaded 50 projects, then fetched each project's owner and latest activity separately: over 100 queries per request. Each held a pooled connection for its round trip. Fixing the N+1 with two batched queries cut connection time per request dramatically.
2. A third-party call made while holding a connection
The endpoint opened a transaction, read from the database, called a billing provider's API to check plan limits, then wrote an audit row. During the Monday spike that provider slowed down, and every request held a database connection while waiting on an external API. We moved the plan check before the transaction and cached plan limits for 60 seconds.
3. No timeouts anywhere
Neither the HTTP client nor the database driver had a timeout, so slow calls waited for as long as it took. We set explicit limits: 2 seconds for the billing API with a fallback to cached limits, a 5-second statement timeout on the database role, and a pool acquisition timeout so requests fail fast with a clear error instead of hanging.
Timeouts alone can backfire, because clients retry. The mobile app retried failed requests three times with no delay, so every timeout added load exactly when the system was weakest. We switched to exponential backoff with jitter, capped retries, and made the write endpoints safe to retry with idempotency keys. Finally, the API now sheds load with a fast 503 once its internal queue passes a limit. Rejecting some requests quickly beats timing out all of them slowly.
4. Row lock contention on a counter
Every request incremented a last_seen counter on the same accounts row. Under load, requests from the same account queued on that row lock. We moved it to a batched background update every few seconds.
Step 4: Right-size, then scale
Only after reducing work per request did we revisit capacity. Rather than 20 connections per container going straight to Postgres, we put PgBouncer in transaction mode in front of the database, with a smaller pool on the database side. Our PgBouncer guide covers the settings. Containers went back to four.
The outcome (illustrative)
In a scenario like this, re-running the same load test after the fixes would typically show:
| Metric at 200 req/s | Before | After |
|---|---|---|
p95 latency, /api/projects | 8+ s, with 504s | about 250 ms |
| Queries per request | 100+ | 3 |
| Pool wait time | most of the request | near zero |
| App containers | 8 (after panic scaling) | 4 |
The exact numbers vary from system to system. The pattern does not: reduce work per request and remove waits from the critical path before adding hardware.
A checklist you can run this week
- Track p95/p99 per endpoint, plus pool wait time.
- Load test your top five endpoints at a fixed arrival rate, with realistic data, until they break.
- Check the slowest endpoints for N+1 queries.
- Never hold a database connection or transaction while calling an external service.
- Put timeouts on every network call: database, HTTP, cache, queue.
- Retry with jittered backoff and a cap, and only retry idempotent operations.
- Look for hot rows that every request updates.
- Pool database connections centrally, and shed load before queues grow unbounded.
- Scale out only once per-request work is lean.
If the same slowdown is hitting your storefront or marketing site, our plain-language guide to a website that crashes during a traffic spike covers the first hour and the preparation.