Can software quality be assured?

Reading time: 6 minutes

Author:

Ger Cloudt is global R&D software manager at Bosch and author of the book “What is software quality?”.

If you ask people what’s meant by software quality assurance (SQA), most likely you’ll get one, or maybe both, of the following two answers. SQA is the definition and execution of and compliance with processes defining how to do your work. The second most likely response would be, it’s about sufficient testing to reveal defects and resolve them. Ger Cloudt takes a closer look at both of these answers in the context of software quality.

Many people believe that high levels of quality can be achieved by having the proper work processes in place. Processes that define what activities should be performed, how we should perform them and in which order. Each process step has to serve a purpose. One of the steps could be to perform a code review. The purpose is twofold: by reviewing, errors might be detected early so that the cost of resolution is low, and it’s a way of knowledge transfer among team members.

For sure, proper processes and ways of working will contribute to the level of quality. However, I consider the activities performed by craftsmen to be even more important. Having an incompetent engineer conduct the code review lets you checkmark the process step as done, thus complying with the process definition, but it won’t contribute to better quality.

To put it differently: a process is a tool that helps you apply your skills. If the skill isn’t available, a process doesn’t help you. However, if the skill is available, a process can help to apply this skill.

Defining purpose

In the nineties of the previous century, process improvement models like CMM (Capability Maturity Model) and Aspice (Automotive Software Performance Improvement and Capability Determination) were popular frameworks to measure the capability of a software team to develop high-quality software. Aspice is still being used in automotive and even required. Both frameworks define several levels of maturity.

Targets are set by management to achieve a certain capability level in either CMM or Aspice. Assessments are performed by trained and certified assessors to determine compliance against the chosen framework. By this setting of a target level by management, the focus of the team will shift from creating high-quality software to successfully passing the assessment. The British economist Charles Goodhart formulated a law stating that a measure ceases to be a good measure when it becomes a target. And this is exactly what happens in many cases concerning process compliance and quality assurance.

On top of setting a target to be achieved, assessors use extensive checklists to verify whether defined process activities are actually performed. No wonder engineers experience quality assurance as a bureaucracy to fulfill compliancy.

Rather than setting a target level in CMM or Aspice, management should define a purpose to be achieved. Assessors should inquire about this purpose. Instead of asking whether a process activity has been performed, they should ask whether the intended purpose has been achieved. An audit shouldn’t be a tool to show that a certain target or level has been reached; it should be used to measure which purposes have been realized, which have not and what could help to achieve them.

Lazy engineers

Many people believe that quality assurance is about testing. Test, test, test to reveal as many defects as possible in the software so that they can be resolved. Build short feedback loops using test automation so that defects are uncovered as soon as possible. Detect as many defects as possible using unit testing because it’s so much easier, and thus cheaper, to solve defects detected by unit testing than when the same defect is detected by a system test.

Creating short feedback loops, eg by applying test-driven development (TDD) and test automation, definitely is a benefit and a necessity. However, there’s a pitfall as well. Having extensive automated testing in place might result in ‘lazy’ engineers, who put their trust in testing as a safety net without understanding whether the approved code indeed is good. No process can prevent bad code, no test can’t prevent bad code, but still, code quality, being bad or good, is an important quality attribute.

Software quality is so much more than processes and process compliance. It’s so much more than mere testing. Yet, when people talk about software quality, in many cases, it’s about processes and testing.

1+3 SQM

In my book, “What is software quality?”, I define the 1+3 Software Quality Model (SQM) as a quality model consisting of four quality types: product quality, design quality, code quality and organizational quality. In doing so, I try to convey a holistic view on software quality. All quality types should be considered when looking at software quality.

Closest to what, I think, people would recognize as software quality is product quality. This is the quality as perceived by the user of the software. It’s an external quality type. Typically, it’s about “the software does what it’s expected to do”, “the software is easy and intuitive to use” and “the software contains no bugs”.

Design quality is an internal quality type – it’s not visible as such. It’s about how well principles like separation of concerns are used in the design and how well the design is supporting the requested functionality and non-functionality of the software. But it’s also about the gap between the intended design as modeled and its implementation in the actual code.

Code quality is an internal quality type as well. Very simply stated, high-quality code is clean code. It works and is readable and understandable by other human beings.

Organizational quality isn’t part of the software itself. It’s the ability of an organization to develop high-quality software. The first thing that comes to mind when I think about organizational quality is the craftsmanship and professionality of the developers. But processes and culture are equally important, as are infrastructure and tooling. Organizational quality is the enabling quality type and without a high level of organizational quality, you won’t achieve high quality levels in the other three quality types. This effectively makes it the most important quality type of the 1+3 SQM.

Holistic view

When considering software quality in your development, take the holistic view and address all four quality types. Think about your business drivers. Based on these, you can set up your software quality strategy.

As an example, in case of a long-term development, you should adapt your strategy to achieve high levels of design and code quality by mitigating technical debt, simply because low levels of these internal quality types slow down and might even obstruct your development. High levels of design and code quality are also required if your team isn’t stable and many different developers will be working on your product. Mitigating the learning curve of new developers requires, for example, clean code and a small gap between the intended and implemented design.

For product quality, the required level is determined by the kind of product you develop. If related to safety, like in automotive, a very stable and mature product is required. The quality strategy needs to achieve this high level of product quality, for example, through strict analysis and extensive testing. But the level of design and code quality will influence the stability of your software as well.

Any quality strategy needs to be enabled by the organizational quality. Which level of craftsmanship do you need? Which level of education? Do you need certified developers? Which tooling is needed? Which processes should be in place and how strict should they be?

Insufficient on its own

Per the abbreviation, Aspice is providing software performance improvement and capability determination. I’d position it in organizational quality, but it’s insufficient on its own. Organizational quality is about much more than mere processes. A myriad of other aspects, including culture, infrastructure, tooling, collaboration, professionalism, mindset and education, should be considered as well.

Testing is a necessity because it’s an illusion to think that we can produce error-free software. I’d position it in product quality, but again, mere testing is insufficient. Product quality is more than that; it’s about building the right product, which starts with requirements engineering. Design quality and code quality need to be considered as well to achieve the desired level of software quality.