example, software embedded in a ventilator usually requires other mechanicalelectrical
subsystems in order to function. It is difficult to write sophisticated simulations
to trick the ventilator software into thinking that all of the other subsystems of the
ventilator are present. Therefore, a software test system might resemble a completesystem.
It is up to the responsible parties to ensure that the degree of overlap is made
known. More importantly, it is vital that everything is tested at some point in the
product development.
Test Execution and Reporting
Test Plan
For software validation, a test plan is written prior to executing a test pass. The format follows
that defined in ANSI/IEEE Std 1012. The key aspects of the test plan are the description
of the software changes, and the tests that will be run to prove that those software
changes are implemented correctly. The test plan is written whenever a full validation or
a regression test is run.
Test-Configuration Form
Because it is very important to be able to repeat a test, it is necessary to document the
configuration of the equipment that is used to run a test against. This documentation is
written on a test configuration form. The contents of these forms can vary. However, the
primary fields are date, item, and item serial number, and there is a place for the test
engineer to sign his name. The test configuration form becomes part of the testdocumentation
suite.
Executing Manual Protocol
Executing manual protocol for a formal test involves printing out the test procedures
and going through the steps in the procedures. Because these printouts are a part
of the official documentation, the test engineer must sign and date the documents. If
the tester finds problems during the test, he will document the issue on the test
procedures.
It is important that the handwritten notes on the paper procedures be addressed.
The notes on the procedures could be problems found or procedure deviations. All
of these markings must have closure. This closure is formally documented in the
procedures response document. The procedures response document is covered later in
this chapter.
Executing Automated Protocol
Typically, executing automated protocol amounts to setting up a batch job to submit a
series of tests to be run. A test engineer fills out a test configuration form to have a piece
of paper to start off the test. The batch job is then run overnight, and the results are analyzed
the following day.
Experience has shown that a test in the batch occasionally can fail. This failure usually
has been attributable either to timing issues or to the target-device feature having changed
but the test not having changed. When this happens, it is usually a practical matter simply
to correct the test script and to rerun it. However, in instances where correcting the
test script would consume a large amount of time, it is sometimes more efficient to run
that test manually to complete the test pass.
Test Results
Once test results have been generated, they must be analyzed. The results from
the automated tests are reviewed by running searches for keywords that indicate
whether the test had problems. Additional analysis is performed by reviewing samples
of the automated test results. This ensures that the tests executed as expected. Other
analysis might need to occur, such as when a large amount of data are generated during
the test, after which data reduction and analysis must be performed to assess correctness.
Test results must be managed along with the other test documentation. The results from
the manual test procedures must be included with the other paper documentation and
stored in the formalized location. The electronic results files also must be put under configuration
control. This is done with a configuration-management tool or a process of
backing up the files to controlled media.
Test Reports
Software Validation Test Report
The key items in this report are a summary of what was tested, how it was tested, any
problems that were encountered, and a recommendation for release.
Requirements to Test Cross Reference
This document covers the ways in which the requirements were allocated to tests.
Procedures Response Document
As mentioned earlier, all handwritten marks on the manual test procedures must be
reviewed and provided a closure. The results of this closure are documented in the procedures
response document. Typically, each mark-up in a test procedure is a software
problem, a procedure error, a procedure deviation, or a comment.