Aerospace Software

Eliminating Classes of Errors

As per DO-178B para 11.3.h (Reverification Guidelines): "The reverification should ensure that previously reported errors or classes of errors have been eliminated."

It is the use of "classes of errors" that I am interested in. A regression analysis / testing can only ensure removal of reported errors; not classes of error. The method that can be used to eliminate classes of error - that I know of - involves Root Cause Analysis (RCA) - carry out an RCA on all errors and determine if they can be categorised. Once that is done, plan for corrective and preventive actions.

Now, RCA and subsequent Preventive/Corrective actions fall pat into organisation-wide quality system processes. I, therefore, do not understand how the creators of DO-178B have included elimination of classes of error under Reverification Guidelines in Software Verification Plan.

Need to checkup if ther eare any CAST papers discussing this aspect in more details.

Rationale for Safety-Related Design Decisions

The DO-178B para 11.10 indicates what Design Description should include. One of these is: "Rationale for those design decisions that are traceable to safety-related system requirements."

Unfortunately, DO-178B does not have very good cross-referencing. Otherwise, it would have been very easy to trace this requirement to the DO-178B para 2.1.2 titled, 'Information Flow from Software Processes to Systems Processes'. It states, "Traceability between system requirements and software design data is important to the safety assessment process."

Given this importance, it is necessary to describe how a design decision affects the safety requirement it traces to, ideally as a rationale below the requirement. That way any subsequent modification to that particular design decision may be analyzed for its impact on safety requirement.

By the way, that is also the reason why derived requirements should have a rationale. Since, these are, by definition, not traced to a high-level requirement, a rationale is an input to the safety assessment process of the impact,. if any, of such derived requirements on system safety.

I had the fortune of briefly interacting with a Safety Assessment expert a few years ago. This person insisted on identifying Systems Requirements that are safety related by unique set of tags - different from the tagging scheme for other systems requirements. This helped him (and the software developers, i.e., us) keep a track of lower-level requirements traced to the system safety requirements. I thought that was good practice. Unfortunately not many Systems Document identify Safety Requirements. Hopefully all readers of this blog will demand differently identified tag for system safety requirements and it will become an industry standard.

Regression Analysis

Unless I have missed it completely - which is quite possible and I shall keep looking - DO-178B does not deal with Regression Analysis in any great detail.

That the software life cycle is iterative is acknowledged by addressing it under Software Life Cycle Definition and Transition Criteria Between Processes. Also as per DO-178B, the Software Verification Plan (SVP) should include "Reverification guidelines"; see para 11.3.  But consider this: All other issues that the SVP needs to address and listed in para 11.3, such as, Verification methods, Verification envoironment, Transition criteria, Partitioning considerations, Compiler assumptions, etc. are also addressed elsewhere in DO-178B. But not Reverification guidelines.

I would like to see output of Regression Analysis clealy defined and recommended to be included as Software Verification Results with a CC2 control.

Cost and Schedule Aspects of DO-178B

DO-178B takes cost very seriously. Obviously! Why else whould we have 5 SW Levels? Afterall it is a consensus opinion of people from the commercial domain.

I often wonder: if DO-178B was evolved by military, perhaps there would be only one level. No? Or it might look like Mil-Std-2167 and its subsequent versions (I must confess I do not have much experience on this; and though I had worked once upon a time on a military project - I do not remember anything of this standard).

Anyways, here is an article, Adopting Legacy Systems for DO-178B Certification by Paul Hicks of Avista, that looks at DO-178B from cost and schedule point of view. It appears in the August 2006 issue of the magazine CrossTalk.

Why is Tool Qualification an Additional Consideration?

DO-178B includes tool qualification as "additional consideration". Clearly there is a problem.

Once it is deteremined that a development tool or the verification tool requires qualification, the tool qualification should be done prior to its use. And since this determination is done at the planning stage, this activity should be moved from section 12 of the DO-178B (dealing with Additional Consideration) to Chapter 11 (Software Life Cycle Data).

Tool Qualification data should be made an integral part of Software Life Cycle data.

Ditching Software-Software Integration tests

Many DO-178B projects avoid performing SW-SW Integration Tests, claiming that Low-Level Tests and Hardware/Software Integration Tests are sufficient.

Do you agree?

If yes, why? In not, why not?

Requirements Traceability

Not sure if I have mentioned this before in one of my posts, but it is definitely worth mentioning once more.

The right time to develop a traceability between higher-level requirements and lower-level requirements is while developing the lower-level requirements; not after the entire exercise is over.

Most errors in traceability creep in because traceability is treated as a post-facto activity. 

Good Software Requirements

Give me a requirements document (Systems, Software or Design) and I will tell you if it has been written by committee. This is epecially true for requirements that are written in text form (as opposed to those written in pictorial form)

Good requirements should read like a story. As you move from one requirement to the next, a picture should form in your mind. There should be a flow and it should make sense.

Poorly written requirements are all over the place and the best way to check that is to do a review of the traceability document. If you need to scroll up and down a page of the traced document, or jump to and fro from one page to another, be extra careful - there is a disaster in waiting. 

Effective Verification - an Important Quality Metric

When you perform verification, you assume that the input is correct; and that any problem can only be in the item being verified. The rational is simple: the input to the next verification stage is the output of the previous verification stage and so all defects and inconsistencies should have been resolved.

That is an ideal condition.

However, most of the time you can see a domino effect. A Problem Report raised, say, during an Unit Test, will often, on anaylsis, identify a problem in, the Software Requirements in an SRD - not just in the low-level requirement in the SDD.

Now why does that happen? Often such ripple-back effect can be ascribed to reverse engineering. Since the code is working and all other documents are generated around the code, the drive to scrunity supporting documents is little lax. "The code is working, isn't it?"

But I have seen this happening in ab-initio projects too. And the blame lies squarely on the persons involved - the developer and the reviewer. For whatever reason. Perhaps, it is lack of pride in workmanship. Or the attitude that if there is an error it will be trapped down the line. Or whatever. Only it is too late in the day when such errors surface leading too delays and costs.

And since this is such a serious issue, I think the ratio of (number of Problem Reports cascading up the chain) to (number of PRs raised) is an important quality metric.  A high value points to an ineffective verification process and definitely needs a Root Cause Analysis to figure out why.

Answers to DO-178B Quiz #5

True or False:

1) User-Modifiable Software is defined as the software that includes optional functions which may be selected by software programmed options rather than by hardware connector pins.

FALSE. The definition I had given is actually for 'Option-selectable software'. The definition of User-Modifiable Software is rather trivia: "... that part of the software that is intended to be changed by the user. ... Example include a single memory bit used to select one of the two equipment options ..."

2) User-Modifiable Software is completely driven by systems requirement.

TRUE.

3) User-Modifiable Software is not permitted for Level A software.

FALSE. All levels of software may have this.

4)  User-Modifiable Software is not included in the software product baseline.

TRUE. Quoting from DO-178B, "User-modifiable software is not included in the software product baseline, except for its associated protection and boundary components."

5) Modifications to User-Modifiable Software can be made without affecting the configuration identification of the software product baseline.

TRUE. Continuing the above quote, "Therefore, modifications may be made to user-modifiable software without affecting the configuration identification of the software product baseline."

6) Details of User-Modifiable Software should be included in 'Software Life Cycle Data' of the PSAC.

FALSE. User-modifiable data should be included in 'Additional considerations' of the PSAC.