The Quality Of Software Development Cannot Be Measured

I often read about the cool sides of agile and lean and other process frameworks or methodologies: you can measure where you are and how good you are doing what you want to do.

Concepts like Velocity or Lead Time help you predict what you will probably be able to achieve in the near future. But this is only the easy part: pure arithmetics.

Stop dealing with the easy, focus the difficult.

Did I say easy? – Yes, and I mean it. The above concepts (velocity, lead time) are easy as soon as you are able to measure them. You need a few requirements like: definition of done, being able to track tasks, estimations, discipline. Given you have all the requirements you defined, deducting the velocity (or lead time) is straightforward, if not to say trivial.

So please stop talking or writing of these concepts as some great discovery! – We all know that one of the hard parts is to fulfill those requirements.

If you are in a situation where the requirements can easily be fulfilled, chapeau! – You are one of the lucky few whose environment is in a great condition.

Now, what is the hard part? – It’s the quality.

Quality is hard.

How can you know if your quality is right?

Can you give me a sound definition of what quality means for you?

Can you even give me a definition of quality that you share with a few others?

Can you give me a definition that you can share with your team, with your organization?

There should be a consensus on these definitions that is supported by everyone who is concerned with the outcomes. And then you have the even greater dilemma: How can you prove that a certain result matches your definition of quality?

If you measure quality it becomes – by definition – a quantity!

Quality measures are only indicators.

Measures always face the problem that – once defined – people try to match the measure, not the intention behind it. Wouldn’t it be great to work together with a shared intention of quality instead of simply fulfilling metrics and measures?

If you have few/many bugs this can mean you have good/bad software. It need not be so.

If you are able to build new features quickly/slowly this can mean that the inner quality of your product is ok/not ok.

Metrics and measures are useful as indicators. I find them much more useful as indicators for bad quality than for good quality. Never be too satisfied when the indicators show everything is fine, but really be worried if they indicate a potential problem. – But this is just based on my experience. I’ve just never experienced a project with bad indicators not having serious problems sooner or later, but I’ve seen the opposite. This goes along the lines of watermelon reporting, which you may have experienced, too.

Quality is not objective.

I remember when Apple released their first iPod shuffle. A German consumer review organization (Stiftung Warentest) downgraded the device because it didn’t have a display. It simply didn’t fit their quality criteria. Nowadays they downgrade Apple devices because you need proprietary software to fill them with music. This can be a turn-off for some people but I guess it is not for many of those who own one.

As you can see by these examples – there are many more, just think about it – it is all about meeting expectations: The expectations of the people that define the measures. If these expectations are unmet, they regard the quality as being poor and vice versa.

And in software development? – Do we really have “hard” measures for software quality? – Do we even agree on very trivial topics? – Just ask a few developers how many LOCs per method are ok and you’ll probably end up in a joyful discussion. – We don’t even have a standard definition of LOCs!

Who can define measures for software quality? Insiders of the project rather than outsiders? There is always a trade-off, but I would always prefer the trustful outsider over the insider. Especially in legacy products it’s been the insiders that have produced the software as it now is. Especially if the system is a mess right now, would you trust them to give you a suitable advice? They see more resistance than outsiders do and they are part of the system, which means they are part of the resistance.

Back up quality by simple measures.

The struggle we are in should be the struggle for quality. Establish few but critical measures and teach your mates the intention behind them. In the domains I usually crawl around these may include:

  • Method length (combined with e.g. cyclomatic complexity)
  • Class size (number of methods)
  • Module dependencies (especially: no circular dependencies)

Why those? – They give an early indication of when you are in trouble to lose control of your code. But if you only strive for superficially fulfilling them through technical changes, it will result in a shattered system and the result on quality will be minimal. Finally, trust your experience or a friendly stranger with more experience than you have right now to judge if the code feels alright. If you can’t trust your feelings, start working on improving your intuition. And please don’t contaminate your intuition with knowledge. Intuition means you are able to judge the quality of a given piece of software before you can tell why. The explanation of your judgement is based on knowledge.

So please don’t mix quality and quantity. If you really want quality, stop treating it as numbers!

Does this sound obvious? – Then take your time and think about it.

6 thoughts on “The Quality Of Software Development Cannot Be Measured

  1. Is quality really difficult? At least for the sake of the argument let me say: No, quality is dead simple.

    It´s simple. And it´s ubiquitous. We can´t avoid talking about quality all the time. The very reason for this is – as you said -, quality is subjective. It´s only in the eye of the beholder – who is the recipient of an artifact or service.

    This gives rise to three questions, I´d say:

    -Who´s the recipient?
    -What are her quality criteria?
    -Can the quality criteria be objectified? (I.e. can they be described in a manner so somebody else than recipient can check if they are met?)

    My feeling is, one or more of these questions haven´t been answered in many discussions about quality. Or the discussing parties have different answers in the back of their heads.

    But once at least the first question has been answered, quality can be measured exactly: just present the recipient the requested artifact/service and ask for feedback. This can be a single score from 0 to 10 or from “:-)”, “:-|”, “:-(“. Or it can be a dozent scores for different aspects of the artifact/service. Or it can be a lengthy description of its strengths/weaknesses.

    In any case the quality can be measured by asking the recipient – if she herself is clear about what´s her quality criteria 🙂

    And maybe that´s the most difficult part: As a requestor of some artifact/service do you have made a conscious decision as to how to assess the quality of the requested artifact/service?

    My guess is, most requestors haven´t done their homework in this regard. Nevertheless, had they done it, quality would be easy to measure.


  2. Ralf, thanks for your formulation. Your formulation helps clarify my intentions. The real hard problem is the judgement of the recipient. And often the recipient is not an individual…

    Is quality judgement a people problem?

  3. Is quality judgement a people problem? Hm… No, I dont think it’s a particular people problem. Even individuals have a hard time becoming conscious about what they want (or what they need).

    If there’s a group requesting sth, this problem is just amplified.

    Maybe we should use “to judge” instead of “to measure” with regard to quality. Measurement is just some form of judgement. Measurement implies quantification – judgement does not.

    It just that measurarbility makes it easier to anticipate the recipient’s judgement.

    • Thomas, thanks for pointing me to this blogpost. In fact we (I and colleagues) observed similar facts: There is a strong relationship between method/class length and complexity metrics.

      I guess I will do more observations and research according to these aspects.

  4. A very good discussion picking up on the above points is from Gerald M. Weinberg “How Software Is Built” and his books on software quality.
    A succinct narrative by an old master.

    His definition of software quality: “Quality is meeting some person’s requirements.”
    For different people the same software product has different quality.

    Though the kindle versions of his books are really unformatted, the wisdom is still in.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s