Verifying compliance with essential criteria is relatively easy. Evaluators simply need to
check with manufacturer specifications and to determine whether the devices being considered
are specified to meet the essential criteria. If they do not, they can be immediately
ruled out of the evaluation process. However, verifying the more specific performance and
safety, ease of use, or other criteria is not that easy. In order to get it right and to avoid
inappropriate or more seriously ineffective, overly expensive, or unsafe choices, a comprehensive
effort is required. In some cases, physical testing is required. In other cases,
qualitative assessments and comparative analyses must be used. With life cycle analysis,
all costs must be identified, and projected volumes must be carefully estimated. The data
then must be entered into analytical tools such as life cycle cost calculators to obtain
results for comparison.
The person who does all of the research, testing, and analysis depends on the technology
being evaluated and the hospital’s previous history with the technology being considered.
Older, more established, or simple technology might not require substantial
hands-on analysis by the hospital. Consultation with an outside organization like ECRI
for its experience with, and purchase advice for, the technology might be sufficient.
Newer or more sophisticated (especially highly customized) technology like PACS or
hospital information systems likely will need a dedicated team, including outside experts
and nonbiased analyses, to determine how effectively the systems comply with the criteria,
especially those that address the evaluating hospital’s unique circumstances or customization
needs.
It is important to include potential users on any assessment that occurs at the hospital.
Users can provide valuable insights on any assessment of performance, device usability,
or other characteristics and will be much more willing to embrace new technology that is
eventually acquired if they believe that their opinion was sought and headed in the selection
process. On many occasions, ECRI has observed technology being thrust upon clinical
staff without its input. These situations create ill will between equipment selectors
and users and often result in poor utilization of the new technology.
When ECRI conducts a comparative evaluation of a medical device, it uses detailed test
methods and presents these and its findings in its monthly publication, Health Devices.
Hospital equipment evaluators should look to this publication for guidance on ways to
perform their analyses. In many cases, ECRI will have tested and presented results on the
specific products being considered by the hospital. In this case, all that the hospital must
do is to verify that ECRI’s findings apply to the hospitals’ unique circumstances. ECRI
technical staff also can be used by the hospital evaluators for advice on ways to perform
analysis and testing, which testing is most important to consider, and whether testing by
an independent body is warranted for a particular device evaluation.
Dealing with the Findings
No technology is perfect. Some devices or systems might work well but are somewhat difficult
to use. Another might work well yet might have a safety problem that is correctable
with routine user intervention. Another might be the best performer, might be the easiest
to use, and might possess the most desirable feature, yet it might be significantly more
costly than the alternatives. These are the types of findings that equipment evaluators are
likely to be confronted with at the end of the evaluation process. The evaluator’s role is
to assess the importance of each finding, then apply a weight or some other type of judgment
scale to the finding, and tally the results to come to a final conclusion. The challenge
for the evaluators is to determine the appropriate weight for each finding. And, unfortunately,
one weighting system cannot be used for every class of device. For example,
image quality is a critical factor when assessing digital radiography system. However, it
is important, but not critical when assessing a physiologic monitoring system.
Equipment evaluators must look at the individual features of each evaluated technology
and determine the features that are most important. Findings relating to the more important
features can be assigned greater weight in the analysis. Evaluators then need to look
at individual results in every category to determine whether any finding could affect the
safety or overall effectiveness of the equipment significantly. Depending on the nature of
the problem, a finding relating to an infrequently used feature could end up receiving a
high weight in the device’s final analysis. As an example, consider an infusion device
evaluation where the device was found to work well, except for over a range of infrequently
used flow settings. The accuracy at tho