Rotating Secrets Without Downtime

The naive approach — update the secret in the secrets manager, restart every process — creates a window where processes that haven't restarted yet are using a credential that's about to become invalid, and if the rotation and the restart aren't perfectly coordinated, some fraction of instances start failing auth mid-deploy. The safer pattern is dual-validity: for credentials the receiving system controls (a database, an internal service), configure it to accept both the old and new credential simultaneously for a bounded overlap window. Roll the new secret out to all instances first while the old one is still valid, confirm every instance has picked it up (via a health check or a version marker the process reports), and only then revoke the old credential. For third-party APIs where you don't control acceptance of multiple valid keys, the overlap has to happen on the provider's side — most support issuing a second key before revoking the first for exactly this reason, and if a provider doesn't, that's worth flagging as a real operational risk, not just an inconvenience. Whatever the mechanism, rotation needs to be routine enough to rehearse — a secret you've never actually rotated in practice, only planned to rotate, tends to reveal surprises (a hardcoded reference, a cache that doesn't expire, a process that reads the secret once at boot and never again) at the worst possible time: during an actual suspected compromise.