The Importance Of Failure
Why we shouldn't try to prevent people from making mistakes
"If you're not breaking the build, you're not doing your job."
I remember messing up quite a bit my first job out of college. I lost count of how many times my code broke things and I was feeling really bad about it. Strangely, my team was happy with my performance. That quote is what one of the senior engineers told me (which was originally told to him when he was a junior engineer).
If you look at any engineering team, you'll probably find that the engineers doing the most, and the engineers that everyone else respects the most, are likely also the ones who are breaking quite a few things. It is also likely that that reason is they are tackling the hard things and hard things are... hard. This is less of an endorsement of the "move fast and break things" mantra. It is more of a "you can't move at all if you're afraid of breaking things".
I think back at hardest things I've ever worked on as well as the problems I've seen others work on and I realize that these were not problems you could just sit and think your way through. They all had a large number of unknowns. There's only one way to bring unknowns to light: try something. You wade into whatever mess of code or blackbox system there is and you start seeing what happens when you change a variable. You may have an educated guess of what could happen, but only by trying it will you know for sure. Then you collect the data, analyze how that made sense, and you change another variable.
Changing those variables can be scary. The hardest problems tend to occur only on production systems and are not reproducible in dev environments. Changing any variable could have catastrophic and non-recoverable consequences. There's good reason to be worried about changing anything, especially if you're worried about being blamed for anything that goes wrong.
There are consequences for doing nothing as well however. A friend of mine once worked on a split brain issue with a distributed file storage system. Anything he did could have caused a loss of user data. If no one did anything, we'd lose that user data over time anyway. In one scenario, he could have been at fault for breaking things. In the other, we all would have been at fault for doing nothing (though for some reason, inaction is less frowned upon than action in most organizations).
The aftermath of this work is also an interesting thing to look at. There's usually a post mortem and some documentation written up. Knowledge is "transferred" on the how and why something happened. Prevention tips are discussed. What's missed is the minute by minute decision making done by the engineer(s) who worked on the issue. For every possible solution discussed in a post mortem, there are likely dozens if not hundreds of angles that were looked at, investigated, and dismissed. All those failed variable changes tried by the engineer are always overlooked.
Why does that matter if none of those ended up being the problem? How is a possible solution, that ended up being wrong, relevant at all?
The work in coming up with those solutions and investigating them help build an engineer's intuition. My team once investigated an issue that only occured on production. I immediately said "it's probably a race condition with X". I then went to a slew of meetings while my team investigated. They thought I rushed the decision and spent an entire day looking into it, only to confirm my initial reaction.
I knew what the problem was in seconds because I had spent many years working on similar issues. I had built an intuition about them. I didn't begrudge my team spending time to investigate a whole host of other possibilties because they needed to spend time building their intuition. An intuition that only comes from trying things and seeing them fail.
The predominant engineering culture would say that we should document as much as possible or set up systems so that future engineers don't make the same mistakes we made. I used to be a part of that camp. Now I think that mentality robs younger engineers of the experience required for them to be great engineers.