Often when we talk about quality of software,we define it in terms of number of testcases,bugs,requirement coverage but this numbers in itself are not enough to incrementaly measure the quality of the application as well as quality of testing.The quality goals are owned by QA but defined by entire team and should be as important artifact as a test plan.
What are Quality Goals?
It is collection of individual metrics that helps to quantify quality of application as well as quality of testing.
Why define and measure Quality Goals?
Agile process gives QA as well as management to change or prioritize tasks at end of each iteration.The quality goal metrics accomplishes two things
- Establish protocol between product management and engineering what are the expectations from the product.
- Empower QA to have product management/development prioritize between new feature development vs fixing old bugs
How to define Quality Goals
The answer to above question will largely rely on processes in your individual institution,but I will take the lowest common denominator which I believe should be present in any QA infrastructure
The first part is to define ways to measure if the testing is sufficient.If we are not doing the testing right,we lose the right to say that quality of application is sub-par.
Quality Goals for Testing
- Written Testcase /Requirements ratio:This number should be greater than 1.Ofcourse this assumes that requirements are complete and sufficient.So for this discussion let’s assume that all the requirements that are known to entire team should qualify for this metrics.
- Written Testcases/Executed testcases ratio:Again the ideal goal is this number should be 1,but remember quality goals are determined by prd management,developers,QA as a team,so you can estimate lower.
- Bugs yet to be verified
Quality Goals for Application
- Non-executed testcases:The number to achieve is 0.
- High severity Open Bugs/Total Bugs
- Medium severity Open Bugs/Total Bugs
- Low severity Open Bugs/Total Bugs
- Untargeted Bugs
The above metrics will vary depending on your own process and outputs that QA efforts produce.You can define it in terms of iteration,release milestones or combination of both.E.g it can look like following.The following figure shows
quality goals for measuring quality of testing
|
|
Written Testcase /Requirements ratio |
Written testcase/Executed testcases |
|
|
|
Iteration |
1 |
0.8 |
|
|
|
Soft Code Freeze |
1 |
0.9 |
|
|
|
Hard Code Freeze |
1 |
1 |
|
|
|
Release |
1 |
1 |
|
|
What to do with this numbers ?
The idea is to measure this against actual numbers and depending on that take corrective action to either achieve the goal or lower the goal(Ofcourse in practice,management will never like it if you say that we would be shipping with 50% of intended quality).The more often we measure this,we have the option to do something about quality of application.
Please add more metrics and ideas about how to quantify testing and application quality.
Next time a brief talk about integration testing.
PS:Just got a chance to view a presentation by Selenium author and WebDriver author.Glad to know that the Selenium 2.0 will merge the two.It is a huge step towards correcting some of the shortcomings of each tool.Presentation can be found here.Video is here
