blog

Notes from the field

Real incidents and postmortems from close to thirty years of building and running production systems: what actually broke, why it took longer to fix than it should have, and what changed afterward. Also the parts of the job that never show up in an architecture diagram, like earning the trust of a team that has no reason to give it to you.

// opinions are my own, not my employer's, past or present.

Environmental Scores Don't Only Go DownEnvironmental scoring usually turns a scary base score into a smaller number, but it can just as easily push a low, boring finding higher than it started. Here's a real example, and the open source tool I built to make it visible.
The CIA Triad Is Why Severity Means AnythingConfidentiality, integrity, and availability aren't security trivia, they're the three questions every vulnerability score is actually answering. Here's what each one means and why integrity is the one most people underrate.
That CVSS 10 Might Actually Be a ZeroA vulnerability scanner reports a 10.0 and everyone panics. Here's what that number actually measures, why it almost never describes your environment, and how the CVSS calculator can tell you the real story.
Minutes From Commit to a Live ClusterA CI/CD pipeline that spins up a real, disposable Kubernetes cluster inside a GitLab runner for every commit, deploys it with the same Helm charts as production, and tears it down when it's done. How it worked, and the one thing it demanded from developers in return.
The Key That Never Touched the DiskA customer needed remote machines to netboot an encrypted disk with no copy of the key anywhere on the image or the hardware. Here's how Kea, Tang, Clevis, and LUKS made that possible.
Earning the Server RoomI inherited an IT department nobody trusted and a server room buried under years of junk. Here's how cleaning it up, one shift at a time, turned into the best compliment I've ever gotten.
The Race Condition That Ate a KeyA customer split their data across two encryption keys, a race condition silently overwrote the first one, and nobody noticed until a restore failed. How we got it back using the ext4 journal instead of luck.
Capacity Isn't Throughput: An IOPS StoryA data center migration, a state-of-the-art storage array, and a production database that needed ten times the IOPS it was built for. Why disk count and capacity planning aren't the same problem as throughput planning.
The Backup Nobody TestedA production outage, a rack of offsite tapes, and the moment we discovered every one of them was unreadable. Why having a backup and having a restore are not the same thing.
Hello, WorldWhy I'm starting this blog and what I plan to write about.