Rollback criteria that survive a stressful night
Criteria written in calm daylight often fail at 02:00. We sketch language that ties rollback decisions to observable reliability signals.
Vague advice like “roll back if things look bad” forces on-call engineers to invent a bar while users wait. Better criteria name the signals: a sustained rise in checkout errors past a stated threshold, failed health checks on a canary cohort, or a data migration that cannot be forward-fixed within an agreed window.
Pair each criterion with a named decision owner and a time box. If the owner cannot be reached, the default is reverse the release — not debate philosophy. Deployment frequency goals should never override those defaults; shipping more often only works when reverse paths stay boring and rehearsed.