Tell your quality team that checklists are not required for reviews and you are likely to get beaten up. Severely. Most people cannot look beyond checklists when it comes to review and review evidence.
Checklists are no doubt the most powerful tool know to mankind to compensate for human memory and (lack of) attention. But are they useful everywhere?
Consider this checklist point: Are the inputs correct and as per requirements? Straight forward, right? Afterall RBT demands this checklist point. But wait! Be honest and tell me if you would check this point for every test case in the document, say there are 20 test cases to be verified in the test cases and procedures document. Ideally, it should be ...
Test case 1 - checked Yes
Test case 2 - checked Yes
Test case 3 - checked No
...
Test case 20 - checked Yes
Have you ever seen a filled checklist that remotely resembles this. Normally, there would be one point and that is supposed to cover all the test cases (or whatever you are reviewing). I can understand if there is one checklist per test case, but one checklist for 20 test cases, not effective.
This is a trivial point, you would say. We do not need to look at the checklist for things that are basic. Checkling for correct inputs is one such. Then why have a checklist point? Besides, there are likely to be 15 more checklist points that you need to verify. Remember ... human memory and attention ...
That brings be to the second point. To be effective, the checklists need to be the right size. Not too big not too small. To be effective the checklists need to comprehensive. To be effective the checklists need to evolve incorporating lessons learnt from previous project. Hmmm... sort of contradicting, don't you agree? Before long, we have a 2-page checklist built on lessons learnt. Taking the example of 20 test cases, how many reviewers have you seen verifying every checklist point on the 2-page checklist for each of the 20 test cases? The reviewers depend on memory for that. But we started with the assumption that checklists are memory compensators. Catch 22!
Let's see what DO-178B says about checklists. Refer 11.3.c(1). "Review methods, including checklists or other aids." Let's repeat it together: "or other aids" So there is life beyond checklists? Now what could DO-178B possibly mean by other aids? I am tempted to convert this into a quiz, but no, let me carry on. Have you ever considered 'Template' as an aid? And no, templates are not just a listing of contents in a pre-formatted document. Well designed templates are not only powerful aid to review but also to development. A thoughtfully developed template will have a choice of actual contents embedded within "<" and ">" to pick from. This restricts the imagination of the developer to the choices available, but guarantees compliance. I can think of the software plans developed and reviewed using good templates. all the reviewer has to ensure that the template has been followed (besides, checking if the plans make sense with respect to the project, of course, but we don't need template for that).
Hmmm.. what are the other aids that I can think of? What about tools? I would love to have a tool that scans my test scripts to get rid of silly errors. The tool can be used to parse the test script and tell me if things are ok, by and large. Of course, depending on what is being checked, the tool needs to be qualified.
I wish DO-178C would give examples instead of leaving it entirely to our imagination: like this, Review methods, including checklists or other aids, such as, templates, parsing tools, ..., etc.
What other aids do you use besides checklists for review?
Recent Comments