Success Criteria vs. Project Analysis Data
Software Quality Assurance people love measuring things. Test coverage, schedule slip, hours slip, velocity, number of reviews, the existence of reviews, number of defects, the average time it took to resolve a defect, function points, cyclomatic complexity, and of course the famous KLOC. If you got all excited reading these last couple of lines, you must be in the quality assurance business (or you are a really enlightened project manager).
Measuring various aspects of what you do is an integral part of improving skills, practices and the development process. Most types of data mentioned above (and of course many others I’ve failed to mention) provide valuable information. The question is for what purpose this information is gathered and how it is used.
The Challenge of Defining Success Criteria
Here’s the problem. We are constantly talking about failed projects and successful projects. However, most of the time both of these terms remain vague. Defining the success criteria for a project is not easy. There are so many stakeholders involved, each of them with a slightly more complex view on the project than the binary response we are seeking.
Still, project managers (and process people) really want to know if a project was successful or not. The easiest solution seems to be finding a set of objective measurements that will be used to rate the project. Functional defects per KLOC is one such classic metric for evaluating a project. This is where all these potentially useful metrics are being misused and often create a distorted picture.
Success Criteria: Look For The Interested Parties
None of the metrics mentioned above (or any combination of them) can be used to evaluate a project. They all describe various attributes of the project and the product. None of them relates to the purpose of the product.
We create software products for a purpose: to create value for our customers. Wait a minute, something is missing here… Oh yes, and to create value to our business! These two goals are complementary, not identical.
Naturally, we have to serve our customers and make them satisfied. But we also have a business to run. Our customers might not care about our future maintenance costs, but we should. Therefore, creating a product that only meets our customers’ expectations, but fails to meet our business needs, is not really desirable.
Both these goals are external to the project we are running. They relate to the world outside the project. They define the interface between the project and the two major stakeholders: the customer and the company we work for. The balance between these two forces should define the success criteria for a project.
It’s pretty clear that neither one of these stakeholders will talk directly about any of the metrics above. You won’t hear customers declare that they expect a 90% test coverage, or a defect rate of four defects per KLOC. Similarly, the company you work for does not directly care about cyclomatic complexity or function points. It cares about reducing development costs and creating a product with long-term value.
Of course, all these things are related to one another. Defect rate can affect customer’s satisfaction, and code complexity will certainly affect future development costs. But, these numbers are not the goal of the project. They don’t define what a successful project is. A project might have a low defect rate and perfectly clear code, and still create no value to the customer and to your business.
I won’t pretend to have a uniform definition of success criteria in software projects, but I know what you should do to define one: go talk to your customer; go talk to your project sponsor; get to know your management professional vision. This knowledge will help you create a clear vision of what your project is expected to achieve. Success should be measured according to this expectation.
Project Attributes Should Be Used For Project Analysis
We began this discussion with a common list of metrics. I then claimed they are all valuable. They are all valuable for analyzing the project – not for defining its success or failure.
It is meaningless to say that a project succeeds if every iteration is completed on time or if less then four functional defects were found per KLOC. However, this data can be extremely useful for analyzing the project.
When a project fails to meet its success criteria, you should ask yourself what were the reasons for that. You can then use your insights to avoid similar problems in the next project. Similarly, when your project is a great success, you should find out what made it such a good project so you can recreate the success next time.
The data collected during the project will not provide you all the answers to these questions. It will provide you, however, some important insights for the analysis. Sometimes, the numbers will raise additional questions, but that’s how analysis is done.
Let’s say you are trying to understand why the customer was not satisfied with the product. You discover a high defect rate for a certain feature. This is a valuable piece of information, but it raises additional questions: why did you have such a high defect rate? Was it because your team didn’t have enough time to implement the feature? Was it because the requirements were ambiguous? And what was the reason for that? Such a line of thought is necessary in order to learn and improve.
***
The data collected during a project can help us learn a lot about how to improve our skills and practices.
Measure whatever you can during the project without interfering with the work of the development team. But don’t be tempted to set arbitrary success criteria based on these metrics. Only the major stakeholders (the interested parties) can define what would be considered a success for a certain project.
