loading
loading
Here’s an awkward question for any codebase that’s been around a few years. Do your tests describe what the system does? Or what somebody, at some point, thought it should do? Not the same thing. Not even close, usually. Most regression tests get written right alongside a feature, by people writing down their intentions. This function returns that. This endpoint rejects bad input. Perfectly good tests! But give a system a few years in production and it picks up a whole extra layer of behavior that nobody planned and nobody wrote down. Quirks. Weird edge cases. A default value someone picked in 2019 for reasons that are now completely lost. And I’d bet money some of your users, or some other team’s service, quietly depends on every one of them. That’s where a lot of the really nasty regressions live. And funnily enough, there’s a kind of test built for exactly that gap. Almost nobody writes it. Hyrum’s Law and the behavior nobody documented You’ve probably heard of Hyrum’s Law, named after Hyrum Wright at Google. The gist: once enough people use your API, every observable behavior of your system ends up being depended on by somebody. Not just the documented contract. All of it. The order results come back in. The exact wording of an error. How long a call takes. Whether an empty field shows up as null or just... doesn’t show up. If you’ve ever maintained something popular, you’ve lived this. You fix a thing that was obviously wrong, ship it, feel good about yourself, and within the hour there’s a bug report from someone whose integration was built on the wrong thing. To them, you just broke their stuff. And are they wrong? Not really. Regular regression tests can’t help much here. They only check what somebody thought to check. Everything accidental slips right through. What characterization tests actually are Michael Feathers put a name on this back in Working Effectively with Legacy Code. He called them characterization tests, and the idea is almost cheeky. Before you touch code you don’t fully understand, you write tests that capture what it does right now. Not what it should do. What it actually does. Run the code, see what comes out, pin that in a test. Looks like a bug? Pin it anyway, maybe leave yourself a note. Correctness isn’t the goal here. The goal is a safety net that yells the moment behavior shifts. It’s kind of a mental flip, honestly. A normal test says “here’s what I want, code, please live up to it.” A characterization test says “here’s what’s real, and if you change it, you’d better have a reason.” Why almost nobody writes them So if they’re that useful, why doesn’t everyone have a pile of them? Mostly human reasons. They’re a slog to write. Characterizing one gnarly function means throwing a bunch of inputs at it and carefully noting every output. Characterizing a whole service? Now you’re doing that across every endpoint, every odd case, every conversation it has with some downstream system. Who’s got time for that? They also just feel wrong. Writing a test that asserts behavior you’re pretty sure is a bug goes against every instinct a developer’s got. It takes a real shift to see that pinning the bug is the whole point. You’re not saying it’s fine. You’re making sure that if it ever changes, someone decided to change it. Golden masters and snapshot tests People have been chipping away at the slog for years. Golden master testing, which some folks call approval testing, grabs a big chunk of output once, saves it as the blessed version, and checks every future run against it. Frontend snapshot tests run on the same instinct. Both make characterization way cheaper, since you’re capturing output all at once instead of asserting on it one field at a time. Where they really shine is stuff you can call directly. A function. A report generator. A rendered component. Services are trickier, though. A service’s behavior isn’t just what it returns. It’s how it reacts to real requests, and everything it does along the way. Which queries it runs. Which other services it calls. Which messages it fires off. Characterizing a whole service with real traffic And this is where it gets fun if you run services in production. The best record of how your service actually behaves isn’t sitting in a test file anywhere. It’s in the traffic. Think about it. Every real request and response is basically a characterization test that hasn’t been written yet. Capture what comes into a service and what goes back out, and you’ve pinned how it behaves for real inputs, including all the strange ones nobody would ever dream up by hand. Capture what it sends to its own dependencies too, and now you’ve got its side effects pinned as well. Recording down at the network level makes this doable without touching the application code, and it catches database and queue traffic right along with HTTP. Replay those recordings later, with dependencies answered from that same capture, and you’ve got a characterization suite that behaves the same way every run. Same inputs in, same downstream behavior. If a response changes, or the service suddenly makes an extra database call, it shows up in the diff. That last bit doesn’t get enough credit. Some of the most expensive regressions keep the response exactly the same while quietly changing what happens underneath. A new query that’s harmless in testing and brutal at scale, say. A suite built on real traffic catches those too. Living with characterization tests They fail differently than normal tests, and that takes a bit of getting used to. A red characterization test doesn’t mean “bug.” It means “behavior changed.” And the very next question is always, did we mean that? Sometimes yes. You fixed the quirk on purpose, you warned the folks relying on it, and the new behavior becomes the new baseline. Sometimes no, and congrats, you just caught a regression no intent-based test was ever going to notice. Either way you’re better off than finding out from a user. Teams that review these diffs in pull requests tend to pick up a really healthy habit: behavior changes stop being accidents and start being decisions. The diff turns into a little conversation. Did we actually want this? Noise needs some attention, though. Timestamps and generated IDs change every run, and they’ll drown out the real signal if you let them. Ignore those specific fields, and nothing broader, and the suite stays honest. Where this pays off most A handful of situations keep coming up where these tests really earn their keep. Refactoring legacy code is the classic one, the case Feathers was writing about in the first place. Pin the behavior, then move things around without the cold sweat. Framework and library upgrades are another. A new major version of your web framework or ORM can nudge behavior in tiny ways intent-based tests never look at, like how something serializes or what order results come back in. Rewrites might be the best case of all. When a team ports a service to a new language or a new architecture, the old service’s recorded traffic basically becomes the spec for the new one. Fire the same requests at both. Any difference is either a deliberate improvement or a regression. No more arguing about what the old system was “supposed” to do. And then there’s AI-assisted changes, which are only getting more common. When an assistant drafts a big refactor and updates the tests in the same go, a baseline captured beforehand is one of the very few checks that doesn’t share the same assumptions as the change itself. Conclusion Here’s how I’ve come to see it. Every long-lived codebase has two specs. One lives in its tests. The other lives in its behavior. Most regression testing only protects the first, and for any system with real users, the second one is often the one that bites. Want to try it? Pick a service you’re a little scared to change. Before your next refactor, capture what it really does, run your change against that baseline, and look hard at every difference. I’d bet you find at least one behavior you didn’t know you had. Much nicer to meet it in a diff than in a bug report.
document fetched 2026-10-24 11:26 · canonical https://stackoverflow.blog/2026/10/22/the-regression-tests-nobody-writes-capturing-how-software-actually-behaves