Review Checklists

Tell your quality team that checklists are not required for reviews and you are likely to get beaten up. Severely. Most people cannot look beyond checklists when it comes to review and review evidence.

Checklists are no doubt the most powerful tool know to mankind to compensate for human memory and (lack of) attention. But are they useful everywhere?

Consider this checklist point: Are the inputs correct and as per requirements? Straight forward, right? Afterall RBT demands this checklist point. But wait! Be honest and tell me if you would check this point for every test case in the document, say there are 20 test cases to be verified in the test cases and procedures document. Ideally, it should be ...

Test case 1 - checked Yes

Test case 2 - checked Yes

Test case 3 - checked No

...

Test case 20 - checked Yes

Have you ever seen a filled checklist that remotely resembles this. Normally, there would be one point and that is supposed to cover all the test cases (or whatever you are reviewing). I can understand if there is one checklist per test case, but one checklist for 20 test cases, not effective.

This is a trivial point, you would say. We do not need to look at the checklist for things that are basic. Checkling for correct inputs is one such. Then why have a checklist point? Besides, there are likely to be 15 more checklist points that you need to verify. Remember ... human memory and attention ...

That brings be to the second point. To be effective, the checklists need to be the right size. Not too big not too small. To be effective the checklists need to comprehensive. To be effective the checklists need to evolve incorporating lessons learnt from previous project. Hmmm... sort of contradicting, don't you agree? Before long, we have a 2-page checklist built on lessons learnt. Taking the example of 20 test cases, how many reviewers have you seen verifying every checklist point on the 2-page checklist for each of the 20 test cases?  The reviewers depend on memory for that. But we started with the assumption that checklists are memory compensators. Catch 22!

Let's see what DO-178B says about checklists. Refer 11.3.c(1). "Review methods, including checklists or other aids." Let's repeat it together: "or other aids" So there is life beyond checklists? Now what could DO-178B possibly mean by other aids? I am tempted to convert this into a quiz, but no, let me carry on. Have you ever considered 'Template' as an aid? And no, templates are not just a listing of contents in a pre-formatted document. Well designed templates are not only powerful aid to review but also to development. A thoughtfully developed template will have a choice of actual contents embedded within "<" and ">" to pick from. This restricts the imagination of the developer to the choices available, but guarantees compliance. I can think of the software plans developed and reviewed using good templates. all the reviewer has to ensure that the template has been followed (besides, checking if the plans make sense with respect to the project, of course, but we don't need template for that).

Hmmm.. what are the other aids that I can think of? What about tools? I would love to have a tool that scans my test scripts to get rid of silly errors. The tool can be used to parse the test script and tell me if things are ok, by and large. Of course, depending on what is being checked, the tool needs to be qualified.

I wish DO-178C would give examples instead of leaving it entirely to our imagination: like this, Review methods, including checklists or other aids, such as, templates, parsing tools, ..., etc.

What other aids do you use besides checklists for review?

 

Comments

[Best Dissertation Writing Services]

Keep it up guys keep helping each others

[Amitabh]

@Pradeep, if you read the subsequent sub-paragraphs of the DO-178B para 6.4.3, you will notice that the HW-SW Integration Tests (HSIT) should be carried out within the target computer environment. For the other tests no specific environment is specified. Meaning, you could carry out SIT and/or UT in a host environment or target environment as long as you correctly and completely verify all the functionality (or trade the ones involving harware to HSIT).
Also note that Objective A-6#5 is the only objective that is specific to HSIT.
Hence, the exception. I think.

[Pradeep]

Hi Amitabh
There is phrase mentioned at sec 6.4.3 of DO-178B "With the exception of hardware/software integration testing, these methods do not prescribe a specific test environment or strategy " here why HSIT is an exception .Can u pls clarify.

@Amitabh : Thank you very much for the prompt responses i am getting from you and many others.

[Amitabh]

@Pradeep, there is no preference in case of MCDC testing. DO-178B specifies that it is an objective of verification at Level A. You need to do it for Level A. Of course, there is nothing stopping you from verifying MCDC coverage at lowere levels.
If you have achieved 100% MCDC at LLT it is sufficient. You do not have to do so for SIT. In fact trying to aheve 100% MCDC for SIT is extremely taxing, and is best avoided. In some cases, it may not be possible at all.

[Pradeep]

Hi Amitabh

Pls clarify the query regarding MCDC
In which level it is preferred to perform MCDC ?
We are achieving MCDC at LLT , and SIT levels .Is that 100 % MCDC coverage is sufficient at LLT level ?

[Dissertation topics]

Its great to see that people are sharing quite profitable information with each other and now we can move our selves to a new era.

[Lizard Labs]

I have been visiting various blogs for my research work. I have found your blog to be quite useful. Keep updating your blog with valuable information... Regards