# Case study: an RDS outage caused by polling

## The problem

MSA Grid is safety-critical: safety managers rely on it to see gas-detection alarms from workers in hazardous environments. A Datadog monitor fired on rising API errors, and the cause turned out to be the RDS/PostgreSQL cluster behind the API, not the API itself. There was no traffic spike from new users. Every open client polled one endpoint every 2 seconds, and that endpoint's query was cheap to write but expensive to run. Multiplied by the poll rate across every customer, it saturated the database.

## What I did

• -Symptom: a Datadog monitor alerted on API errors; correlating it with RDS metrics showed the database, not the Lambdas, was the bottleneck.
• -Diagnosis: RDS Performance Insights ranked the top SQL by load, which pointed to a single query behind the polling endpoint; EXPLAIN ANALYZE showed it doing a sequential scan.
• -Why it hurt: each client polled every 2 seconds, so a modest per-query cost was multiplied by poll frequency and the number of customers into sustained load.
• -Fix: added a composite index matching the query's WHERE clause, and moved data that had been embedded in a larger table into its own table so the hot path reads far less.
• -Outcome: mitigated within hours; cluster CPU dropped from ~95% to ~30% and the API errors cleared.
• -Prevention: added Datadog monitors and alarms on the cluster's CPU, introduced index review in pull requests, and proposed replacing polling with WebSockets.
• -Takeaway: on polling-heavy APIs, per-request query cost multiplies by poll frequency, so it needs to be designed and monitored from day one.