A
Sep 15, 2026A third of our releases went out through the emergency path
A deploy at eleven on a Tuesday night broke search for about forty minutes. It had skipped the...
Sep 15, 20262 min read0 reactions1 comments
Back to articles
Tag archive
#sergeyshinder
E
Sep 15, 2026Every new replica took twenty more database connections
A marketing email went out at ten on a Thursday and traffic roughly tripled over four minutes....
Sep 15, 20262 min read0 reactions0 comments
N
Sep 15, 2026Nobody could tell me when the latency had started
The orders API had a p95 of nine hundred milliseconds. It is supposed to be around two hundred and...
Sep 15, 20262 min read0 reactions0 comments
T
Sep 15, 2026Their sandbox accepted everything their production refused
Six weeks of integration work with a new settlement provider, a sandbox we could hit as often as we...
Sep 15, 20262 min read0 reactions0 comments
T
Sep 15, 2026The apply was green and nothing in the environment could get out
Our disaster recovery environment is built from the same modules as production, in a second region,...
Sep 15, 20262 min read0 reactions0 comments
A
Sep 15, 2026A temporary file on one pod evicted every other pod on the node
At 03:40 on a Wednesday, seven pods belonging to four unrelated services were killed on the same...
Sep 15, 20262 min read0 reactions0 comments
T
Sep 15, 2026The secret was masked until we base64 encoded it
An engineer scrolling a failed build on a Monday afternoon found our registry deploy token sitting in...
Sep 15, 20262 min read0 reactions0 comments
W
Sep 13, 2026We read the partner's file while they were still writing it
At 14:07 on a Tuesday, four thousand three hundred products went out of stock on our storefront....
Sep 13, 20262 min read0 reactions0 comments
O
Sep 13, 2026Our dashboards were green because the request never reached us
The first report arrived by phone: the application loads a blank page. By the time I had opened our...
Sep 13, 20262 min read0 reactions0 comments
O
Sep 13, 2026One instance was bad at its job and nothing was designed to notice
Checkout errors sat between three and eight percent for most of a Thursday. Not enough to page,...
Sep 13, 20262 min read0 reactions0 comments
O
Sep 13, 2026Our own retention policy deleted the version we needed to roll back to
Friday, ten past four. A release went out, error rates climbed inside two minutes, and we made the...
Sep 13, 20262 min read0 reactions1 comments
S
Sep 12, 2026Somebody silenced that alert in March and it stayed silenced
A broker node ran out of disk on a Thursday morning and stopped accepting writes. We found out from a...
Sep 12, 20262 min read0 reactions0 comments