A
Oct 2, 2026A reporting job outranked our ingress controllers and the scheduler agreed
At ten past six on the last evening of March, two of our three ingress controller pods and half of...
Oct 2, 20262 min read0 reactions0 comments
Back to articles
Tag archive
#sergeyshinder
O
Oct 2, 2026Our bucket policy let the office in and locked out every server we own
In June our security team asked for the bucket holding signed contracts to be reachable only from the...
Oct 2, 20262 min read0 reactions0 comments
O
Oct 2, 2026Our error graph went blank every time we zoomed in on the outage
During a payments incident in August I had our error rate panel on the big screen, showing the last...
Oct 2, 20262 min read1 reactions0 comments
O
Oct 2, 2026Our users were logged out for two weeks before Redis refused to write anything
For about two weeks in May, support received a steady trickle of complaints from people who had been...
Oct 2, 20262 min read0 reactions0 comments
O
Oct 2, 2026Our beta became the version every new customer installed
Our JavaScript SDK is published to npm by a release workflow that runs whenever a version tag is...
Oct 2, 20262 min read0 reactions0 comments
T
Oct 2, 2026The SMS provider told us when to come back and we heard now
Our login codes go out through an SMS provider. When we send too fast it answers 429 with a...
Oct 2, 20262 min read0 reactions0 comments
O
Oct 2, 2026Our invoices lost their accents when we moved to a new CI system
In February we moved the build for our invoicing service from Jenkins to GitHub Actions. The pipeline...
Oct 2, 20262 min read1 reactions0 comments
O
Oct 1, 2026Our HR sync decided José had left and then hired him again
Our identity sync reads employees from the HR system every hour and creates, updates or disables...
Oct 1, 20262 min read0 reactions0 comments
A
Oct 1, 2026A refactor that touched no database changed our database password
In May I merged a pull request that moved our tagging into a shared locals block. Twelve files, no...
Oct 1, 20262 min read0 reactions0 comments
A
Oct 1, 2026A connector we switched off in May kept every change our database made
At four in the morning on a Saturday in September our main Postgres primary ran out of disk and...
Oct 1, 20262 min read0 reactions0 comments
A
Oct 1, 2026A migration merged on Tuesday was never going to run in production
Two developers on the billing team each wrote a database migration in the same week. The first one,...
Oct 1, 20262 min read0 reactions0 comments
O
Oct 1, 2026Our synthetic check spent five weeks testing the login page
Our customer portal has a synthetic check that runs every two minutes. It loads the account page as a...
Oct 1, 20262 min read0 reactions0 comments