The Analysis coverage charts can be extremely useful and help answer questions like: "How much coverage is each of my tests adding?" and "How much testing is enough?". Learn how to use the Coverage Matrix and Coverage Graph to better understand specific interaction coverage after every scenario.
Coverage Matrix
Click on the “Analysis” button to use this feature
Understanding the Matrix
No, you’re not Neo, but this is like having superpowers.
Each square of the matrix chart represents a pair.
How to read the chart:
- Read the values listed across the top
- Read the values listed down the left side
- Find an intersection of two values (one from top and one from side):
- If it is green: the pair is already covered.
- If it is red: the pair is not covered yet given the scenario you selected via slider.
- If it is black: the pair is marked as invalid.
Slide your way to precise coverage!
The slider at the top allows you to visualize which pairs are covered with each test. By default, the slider will “settle” at the position that achieves around 80% coverage.
Communicate your testing coverage precisely
With the Matrix, communicating your testing coverage has been made even easier! Imagine a situation where you have created some tests and your manager comes by and says, “Development is running behind, so we’ll only have time to execute 2/3s of your tests before Go-Live. What will be missed?” You can easily respond with a slide like the one below:
Can I see 3-way and higher interactions?
At this time, the Matrix only supports demonstrating the pairs that are covered with each test.
Special Note for Color-Blind Users
Pressing “C” on your keyboard changes the color scheme of the Matrix Chart. The following 3 options are available.
Option 1: Default
Option 2: Blue-Orange
The Blue squares represent pairs that have been covered while the Orange squares represent pairs that have not been covered yet.
Option 3: Purple-Orange
The Purple squares represent pairs that have been covered while the Orange squares represent pairs that have not been covered yet.
Dealing with Large Numbers of Test Cases
As seen in the above picture, large numbers of test cases can appear to be overwhelming in the Matrix. DesignWise can help make these overwhelming instances more manageable with Zoom and Click-Drag features. If I want to key in on the Java console: Show console and Show sandbox warning banner combination: Select, I can hover my mouse roughly over where I think that combination will appear in the Matrix and use my mouse wheel to zoom in.
Like this:
As you can see, the square that accounts for the “Java console: Show console” and “Show sandbox warning banner combination: Select” is able to be examined easily after zooming. By hovering my mouse over that square, I can see that this particular pair of values is covered within the current 11 test cases and this pair is first featured in Test Case 2. It is also possible to click and drag the Coverage Matrix to check out different squares at your desired zoom level.
By clicking and dragging and then hovering over the square for the combination of Shortcut Creation: Prompt user and Show sandbox warning banner combination: Select, I can see that this pair of Parameter Values would not appear until Test case 26. With this information in mind, I can then weigh my testing options. Do I need to test for this 2-way interaction? Is it worth generating 26 test cases? Am I better off writing a requirement that involves these two parameter values etc.?
The Coverage Matrix can be invaluable in the decision-making process.
Coverage Graph
Click on the "Analysis" link to use this feature.
Let’s walk through a small example to understand what the valuable information in the chart means.
The number of Parameters and Values you entered in the "Parameters" screen will determine how many total possible pairs of Values there are in your test model. A simple example with 8 Values makes this table:
Given these inputs, your model will have exactly 24 possible combinations of pairs of values, as shown below:
The first test case (Large / Heavy / Purple / Hexagon) will test for 6 of the 24 possible pairs.
So after the first test case, the Coverage Graph (and Matrix) will show that 25% of the total possible pairs that could be tested in this simple example have actually been tested at this point. So far so good.
The second scenario (Small / Light / Purple / Circle) will test another 6 pairs. Importantly, none of these 6 pairs of Values have been tested yet. In our first two tests, we will have tested a total of 12 pairs of Values.
So after 2 test cases, the chart shows that 50% of the possible pairs (e.g., 12 tested out of 24 possible) have been tested.
Why do coverage charts start off with a steep trajectory (with lots of added coverage per test) only to flatten out towards the end (with only a little added coverage per test)? Analyzing test number 3 shows us why:
There is no possible way to select values in such a way that we could test for 6 new pairs of values as we did in each of the first two tests. The best we can do is test 5 new pairs of values and 1 previously tested pair. In this 3rd test, "Large and Hexagon" had already been tested in the first test.
After test 3, we have now tested for 17 pairs of 24 total possible pairs. The coverage chart shows 70.8% (vs. 75% had we been able to squeeze in 6 new pairs into test 3).
What is up with the final two test cases? We were able to achieve 25% coverage of pairs in test 1 and test 2. Why do test 5 and test 6 only achieve a measly 4.2% increase each?
The final two scenarios each only add a tiny amount of coverage because we managed to test all but two pairs of Values in the first four test cases and because it will require at least two additional test cases to test those final two remaining pairs.
The only pair tested for the first time in scenario 5 is "Small and Hexagon". The only new pair tested in scenario 6 is "Large and Circle". That is only one-sixth as many new pairs in each test as compared to either of the first two tests.
The likelihood of finding a new defect in test number 5 or 6 is much lower than finding a new defect in test case 1 or 2.
The key difference between the Matrix and the Graph
- Matrix provides both the aggregate (total % of interactions) and the granular (specific pairs) insight for each test case, but only applies to 2-way generation methods.
- Graph provides only the aggregate (total % of interactions) insight but applies to any n-way generation methods (including mixed-strength).
A few important points to consider when analyzing coverage information
1) First, when used correctly (and thoughtfully), the information can be extremely useful. It gives you a quick method for objectively assessing "how much extra testing coverage am I achieving with each new test?" and "how much testing is enough?". Many testing teams have a rule-of-thumb. For example, to stop executing the DesignWise-generated tests after they have achieved 80% coverage because they can clearly see diminishing marginal returns to further testing after that point.
2) The second thing to keep in mind is caution. As George Box says, "All models are wrong. Some models are useful." It would be a mistake to look at the graph, see that "100% coverage" has been achieved after the final DesignWise-generated test, and conclude that the tests cover everything that should be tested.
An "Analysis" chart generated by DesignWise, like all software testing coverage reports, is an imperfect model of what should be covered (which is itself based on an imperfect model of the System Under Test). There could be significant aspects of the System Under Test that were not entered into the "Parameters" screen.
It is important to remember that one or more of those excluded aspects (hardware configuration, software configuration or plug-in, the order in which actions are executed, whether a user is navigating with mouse or a keyboard, whether or not "submit" buttons are clicked multiple times quickly, etc.) could potentially cause a defect that might not be identified in your current set of tests.