📢 Community Announcement: Root-Cause Fixes and Performance Improvements Following the steemit.com 504 Incident

Dear Steem community,

On September 18, steemit.com experienced a significant intermittent 504 outage. I completed a full end-to-end investigation and took the opportunity to land a comprehensive set of permanent fixes and capacity improvements. All of these fixes are now live in production and the platform is running stable. This post shares the technical details and the relevant code with the community.

Incident Recap

Starting around 04:00 UTC on September 18, steemit.com began intermittently returning 504 errors. The investigation traced the failure chain to:

1. The frontend application (condenser) performs server-side rendering (SSR), which waits on blockchain data served by hivemind;
2. hivemind's database suffered a slow-query avalanche — an index on the hive_follows table (~120 million rows) was in an INVALID state, causing critical queries to degrade into full table scans (single-query cost of 1,514,525);
3. The API gateway layer (jussi) cancels requests to hivemind after a 3-second timeout, but the database connections remained occupied while new requests kept arriving — connection pool saturation turned the slowdown into a self-amplifying storm.

The Four Permanent Fixes

1. Database index repair (hivemind)

We repaired the invalid ("zombie") index and added a symmetric partial index for the get_following ignore branch, bringing the database schema to v30. The migration ran in production using CONCURRENTLY (completing in ~150 seconds with no user-visible impact). After verifying v30 was in effect, slow-query volume returned to baseline and the multi-second latency plateau seen during the incident has not reappeared.

• PR: steemit/hivemind#398 (merged, merge commit 6fd0e6b)

2. Circuit breaker in the API gateway (jussi)

We added a per-upstream circuit breaker to jussi: when an upstream (such as hivemind) keeps timing out or failing, subsequent requests fail fast instead of piling up, preventing the storm amplification caused by timeout-cancellation plus lingering connection occupancy. The breaker parameters are configurable via environment variables, the /health endpoint exposes the circuit state of every upstream, and Prometheus metrics are aggregated per host. This version is deployed to production jussi.

• PR: steemit/jussi#266 (merged, squash commit dc1983d)

3. Frontend deployment reliability (openresty)

The production frontend runs openresty as a sidecar on each EC2 instance. Previously, nginx performed a hard DNS lookup of the application container hostname at startup — if the application container had not yet completed its DNS registration, nginx would crash immediately ([emerg] host not found in upstream), sending the instance into a crash loop. We changed this to:

• a variable-based nginx upstream, resolved lazily at request time through Docker's internal resolver (127.0.0.11), so startup no longer depends on the application's DNS entry;
• a health check on the application container, with openresty starting only after the application is actually ready to serve.

The fix was verified in both dev and production: nginx crash logs dropped to zero after deployment, and the 50% 5xx rate observed during the incident fell to 0.1%.

4. Anonymous page caching (openresty + condenser)

To address the anonymous traffic that made up the bulk of the timed-out requests, we implemented the standard two-layer "application + cache" approach: condenser now serves publicly cacheable responses for anonymous post-page requests (zero Set-Cookie), the openresty layer caches those pages (5-minute TTL, coalesced concurrent fetches to the origin, and stale-on-error as a fallback), and any request carrying a session cookie always bypasses the cache and hits the application directly.

This feature is deployed to production condenser, and the caching behavior passed end-to-end verification (MISS→HIT, per-language isolation, cookie bypass — all as expected).

• condenser PR: steemit/condenser#4032

Additional Capacity and Protection Measures

• hivemind production Auto Scaling MinSize raised from 2 to 4 (incident mitigation), with the frontend environment later scaled to 6 instances;
• per-IP rate limiting in openresty (20 r/s + burst 100, using the real client IP restored from CF-Connecting-IP), validated with controlled tests to confirm correct behavior.

Status

All four permanent fixes are complete, and steemit.com and api.steemit.com are running stable. Thank you for your patience during the incident.