The Problem With Software Requirements
The goal is not actually to build software that meets all requirements
"If we complete all of our sprint goals in every sprint, we'll have a ton of sales and everyone profits!"
- No company ever
A company we technically compete with went under recently. What I found extremely interesting was a claim by one of their engineering managers that they completed every sprint and the fault was with other parts of the business. I'm not going to attempt to put the blame back on engineering because I honestly don't know what happened internally. However, people tend to put too much attention on completing sprints and using that as a measure of success.
Why would anyone do this in the first place? Software development is hard to track. Projects can go on for months and no one wants to be blind on how a project is going. The most obvious thing to use as a substitute is the total complexity of tasks completed (many people use story points here). That allows people to see if a project is going well or not.
The problem with this is that the true measure of success for any project isn't the number of tasks completed. It is how well the project solves the problems of its intended users.
Throughout my time in college and the first few years of my career, I was bought into the notion that changing requirements were an unfortunate reality of software development. It was something we hope doesn't happen, but need to be prepared for if it does. If people are doing their jobs right, requirements shouldn't need to change. As a developer silo-ed on the programming side of things, I could happily live with this notion that other people just need to do their jobs right.
Spend enough time trying to create requirements yourself and you realize something: a lot of assumptions go into making the requirements for any piece of software. Any one of those assumptions being wrong could be the difference between people loving your software or not caring about it. Software requirements are not deterministic like math formulas. You can't put in a certain set of inputs into your software requirements and expect to get a certain reaction from users. Software requirements function more like hypotheses. You make some assumptions and create an expectation of a result, but are prepared for the experiment not to have that expected result.
That's not how most software development is done though. When requirements are created, they are assumed to be correct. Users not reacting well to the final product is treated as a surprise. Months of development time is put into the project without actually testing that assumption. Maybe focus groups were used, but that rarely maps well to the actual user base. This happens in both companies that practice waterfall and places that break up their waterfall into sprints.
I think one of the bigger things compounding this is the engineering culture I was apart of early in my career and that still persists today. Many engineers want solidified requirements before they touch code. Code is put on a pedastal as the most important and most expensive thing. The result of any coding activity should be a final product. That forces everyone to treat requirements as truth rather than an educated guess.
This is where the question comes up of what is a software engineer's job? Is it just to write code given a set of requirements? Or is it to solve problems, of which writing code is one means to an end?
I've been in both positions and I find that software tends to be built much quicker when the job is to solve problems. There's less miscommunication. There's less overhead in managing that communication. The quality of that software is also higher despite being built quicker.
If that sounds crazy, consider this: if users think something is a bug when it was actually built exactly to spec, is it actually a bug? We could argue semantics and definitions, but the end result is the same. Users expected X and the software gave them Y. That makes the software less useful.
I'm not arguing that we should never write software requirements. That would be chaos for any system of reasonable complexity. What I am suggesting is that we should stop treating software requirements as the goal. The goal is discovering value for users. That makes software requirements a lot more malleable than many organizations treat them as.