Aerospace Software

Interpreting Object Code Verification

The best way to interpret paragraph 6.4.4.2.b of the DO-178B is by using an "AND" logic.

Just to be clear what I am talking about, I quote the document: "The structural coverage analysis may be performed on the Source Code, unless the software level is A and the compiler generates object code that is not directly traceable to Source Code statements."

My interpretation:

The following 3 conditions apply:

(A) Structural Coverage analysis has to be performed on Object Code,

(B) The software level is A, and

(C) Object Code cannot be directly traceable to the Source Code

A == B AND C

Meaning,

We have to provide 100% Structural Coverage on the Object Code

If

The Software Level is A

AND

The Object Code does not have a direct correspondence to the Source Code

Thus,

If the Software Level is B, C, D or E, we can do Unit Level using any Qualified Testing Tool available in the market, since the condition (B) is False. Please note that  (i) most OTS tools instrument the source code, and (ii) you have to qualify the tool for the environment you are testing in.

If the Software Level is A, but we can prove that Object Code has a one to one correspondence with the Source Code, then condition (C) is False. This happens when you have an interpreter which trranslates each line of your source code to a set of object code. (Proprietary languages have this feasture). In this case again, go ahead and use OTS tools without any problem.

However, since compiler use is more wide-spread, condition (C) is never False. It is therefore easier to just go ahead and carry out an Object Code Verification whenever the software level is A.

A word of caution: it is fine to take credit from a previous project, as long as the the environment is identical AND the constructs of the language being used is identical.

Object Code Verification is applicable even when someone claims that s/he has set the compiler option to no-optimisation. After all the idea of verification is to doubt every claim. 

CAST - Independence of Test Case & Procedures Developer and Executor

In order to meet independence objectives A6-3 (Level A and B), A6-4 (Level A), and A7-1 (Level A), CAST-26 takes the following position:

"Test Execution: The person responsible for executing the tests should not be the same person who developed the requirements or code being verified by the tests, nor the developer of the test cases and procedures being executed."

I can understand that the test executor should not be the person who wrote the requirements or the code, but cannot really understand why the developer of test cases and procedures cannot execute the test cases. There are occassions when the development of cases and procedures is separated from test execution to ensure that the requirements of gate audits or SOI audits. But I have never come across any software plan or DER insisting on having an independent executor. Is this recommendation an overkill or is there a deeper thought that went into this? I wonder.

Data and Control Coupling

The most confusing aspect of DO-178B the verification of Data Coupling and Control Coupling. In fact, a study of the clarification in DO248B and CAST Paper 19 leaves you even more confused. Is it a feature of good design or code review or verification testing. Clarification indicates that it is a combination of all three. That is no good. No wonder "CAST believes that additional guidance in the analyses of data coupling and control coupling will be required in the future."

This is one aspect that I would like to see resolved in DO-178C. What engineers look for actionable items; not some vague philosophical statements.

Review Checklists

Tell your quality team that checklists are not required for reviews and you are likely to get beaten up. Severely. Most people cannot look beyond checklists when it comes to review and review evidence.

Checklists are no doubt the most powerful tool know to mankind to compensate for human memory and (lack of) attention. But are they useful everywhere?

Consider this checklist point: Are the inputs correct and as per requirements? Straight forward, right? Afterall RBT demands this checklist point. But wait! Be honest and tell me if you would check this point for every test case in the document, say there are 20 test cases to be verified in the test cases and procedures document. Ideally, it should be ...

Test case 1 - checked Yes

Test case 2 - checked Yes

Test case 3 - checked No

...

Test case 20 - checked Yes

Have you ever seen a filled checklist that remotely resembles this. Normally, there would be one point and that is supposed to cover all the test cases (or whatever you are reviewing). I can understand if there is one checklist per test case, but one checklist for 20 test cases, not effective.

This is a trivial point, you would say. We do not need to look at the checklist for things that are basic. Checkling for correct inputs is one such. Then why have a checklist point? Besides, there are likely to be 15 more checklist points that you need to verify. Remember ... human memory and attention ...

That brings be to the second point. To be effective, the checklists need to be the right size. Not too big not too small. To be effective the checklists need to comprehensive. To be effective the checklists need to evolve incorporating lessons learnt from previous project. Hmmm... sort of contradicting, don't you agree? Before long, we have a 2-page checklist built on lessons learnt. Taking the example of 20 test cases, how many reviewers have you seen verifying every checklist point on the 2-page checklist for each of the 20 test cases?  The reviewers depend on memory for that. But we started with the assumption that checklists are memory compensators. Catch 22!

Let's see what DO-178B says about checklists. Refer 11.3.c(1). "Review methods, including checklists or other aids." Let's repeat it together: "or other aids" So there is life beyond checklists? Now what could DO-178B possibly mean by other aids? I am tempted to convert this into a quiz, but no, let me carry on. Have you ever considered 'Template' as an aid? And no, templates are not just a listing of contents in a pre-formatted document. Well designed templates are not only powerful aid to review but also to development. A thoughtfully developed template will have a choice of actual contents embedded within "<" and ">" to pick from. This restricts the imagination of the developer to the choices available, but guarantees compliance. I can think of the software plans developed and reviewed using good templates. all the reviewer has to ensure that the template has been followed (besides, checking if the plans make sense with respect to the project, of course, but we don't need template for that).

Hmmm.. what are the other aids that I can think of? What about tools? I would love to have a tool that scans my test scripts to get rid of silly errors. The tool can be used to parse the test script and tell me if things are ok, by and large. Of course, depending on what is being checked, the tool needs to be qualified.

I wish DO-178C would give examples instead of leaving it entirely to our imagination: like this, Review methods, including checklists or other aids, such as, templates, parsing tools, ..., etc.

What other aids do you use besides checklists for review?

 

Eliminating Classes of Errors

As per DO-178B para 11.3.h (Reverification Guidelines): "The reverification should ensure that previously reported errors or classes of errors have been eliminated."

It is the use of "classes of errors" that I am interested in. A regression analysis / testing can only ensure removal of reported errors; not classes of error. The method that can be used to eliminate classes of error - that I know of - involves Root Cause Analysis (RCA) - carry out an RCA on all errors and determine if they can be categorised. Once that is done, plan for corrective and preventive actions.

Now, RCA and subsequent Preventive/Corrective actions fall pat into organisation-wide quality system processes. I, therefore, do not understand how the creators of DO-178B have included elimination of classes of error under Reverification Guidelines in Software Verification Plan.

Need to checkup if ther eare any CAST papers discussing this aspect in more details.

Regression Analysis

Unless I have missed it completely - which is quite possible and I shall keep looking - DO-178B does not deal with Regression Analysis in any great detail.

That the software life cycle is iterative is acknowledged by addressing it under Software Life Cycle Definition and Transition Criteria Between Processes. Also as per DO-178B, the Software Verification Plan (SVP) should include "Reverification guidelines"; see para 11.3.  But consider this: All other issues that the SVP needs to address and listed in para 11.3, such as, Verification methods, Verification envoironment, Transition criteria, Partitioning considerations, Compiler assumptions, etc. are also addressed elsewhere in DO-178B. But not Reverification guidelines.

I would like to see output of Regression Analysis clealy defined and recommended to be included as Software Verification Results with a CC2 control.

Ditching Software-Software Integration tests

Many DO-178B projects avoid performing SW-SW Integration Tests, claiming that Low-Level Tests and Hardware/Software Integration Tests are sufficient.

Do you agree?

If yes, why? In not, why not?

Effective Verification - an Important Quality Metric

When you perform verification, you assume that the input is correct; and that any problem can only be in the item being verified. The rational is simple: the input to the next verification stage is the output of the previous verification stage and so all defects and inconsistencies should have been resolved.

That is an ideal condition.

However, most of the time you can see a domino effect. A Problem Report raised, say, during an Unit Test, will often, on anaylsis, identify a problem in, the Software Requirements in an SRD - not just in the low-level requirement in the SDD.

Now why does that happen? Often such ripple-back effect can be ascribed to reverse engineering. Since the code is working and all other documents are generated around the code, the drive to scrunity supporting documents is little lax. "The code is working, isn't it?"

But I have seen this happening in ab-initio projects too. And the blame lies squarely on the persons involved - the developer and the reviewer. For whatever reason. Perhaps, it is lack of pride in workmanship. Or the attitude that if there is an error it will be trapped down the line. Or whatever. Only it is too late in the day when such errors surface leading too delays and costs.

And since this is such a serious issue, I think the ratio of (number of Problem Reports cascading up the chain) to (number of PRs raised) is an important quality metric.  A high value points to an ineffective verification process and definitely needs a Root Cause Analysis to figure out why.

Compiler Optimization and Level A Software

All those who wonder why a DO-178B level A software should be tested on the object code (refer paragraph 6.4.4.2.b) should read The Embedded Muse Newsletter #189. It has not yet found its way to the "back issue section" (obviously!), so I quote what one Mr. Geoff Patch has to say about compiler optimisation in this issue ...

For example in 1998 I was using a compiler from a well known DSP manufacturer and my application was doing one of those "That's impossible, it  simply can't be happening!" things. As a last  resort I started reading the compiler generated assembler, and Lo and Behold, the compiler had decided that an entire function, including accesses to volatile variables, wasn't necessary. It simply hadn't generated the code to call the function. Reducing the optimisation level fixed that.

I hope this scares you enough.

There are a few automated tool that gives you structural coverage on the object code. But the standard technique remains the following:

a) Set compiler to "no optimization"

b) Carry out a manual source code to assembly code comparison to ensure that the compiler has not introduced code (or deleted them) that cannot be traced back to the source code. (And no, you do not have to do this for the entire 50,000 lines of the software). 

By the way, I strongly recommend subscribing to Jack Ganssle's Embedded Muse. It is free and has interesting information.

To see what you have missed so far, check out the Embedded Muse back issues.

Host Testing versus Target Testing

I always thought the distinction between host testing and target testing should be clear to all. Apparently not. But before that, one thing should be clear: DO-178B does not demand that low-level testing be done on target testing. Testing of the code on target processor in a high fidelity target environment is highly desirable; however, it may not always be possible to carry out such testing at unit level.

Coming back to the topic of host testing versus target testing: I think this site explains it best. Click on the link and scroll down. You will find a neat little diagram that explains every aspect of Host and Target testing.

Caution: Since the website belongs to LDRA, the diagram describes the testing from LDRA Testbed perspective. If you are uncomfortable with LDRA (it is a good tool though), you may replace it in your mind with any other similar tool to get an understanding.

I wish the term Host/Host testing and Host/Target testing, as described in this LDRA site, is made standard so that there is no ambiguity when such tests are being discussed.