Radiation Portal Monitor Tests Scored in Secret DHS-Sponsored Review

Statement

Radiation Portal Monitor Tests Scored in Secret DHS-Sponsored Review

Tests used by the Department of Homeland Security to certify new radiation screening machines were flawed, according to a Department of Homeland Security (DHS)-sponsored review. Members of the Committee on Energy and Commerce called for independent testing of the new generation of radiation detectors, which are expected to cost $1.2 billion to deploy.

"We should not spend a single penny to install these machines at our ports and borders until valid testing is done to demonstrate that these costly new machines work significantly better than the existing radiation detectors under real world conditions," said Rep. John D. Dingell, Chairman of the Committee on Energy and Commerce. "I urge Homeland Security to follow one of the key recommendations in its own report, which calls for an independent group to run tests on new technology to overcome internal organizational conflicts of interest."

"I am disappointed to learn that DHS ordered this critical review to be stamped "Official Use Only" without any apparent consideration for the public's need to know about the problems identified," said Rep. Bart Stupak, Chairman of the Subcommittee on Oversight and Investigations. "The public deserves to see an unvarnished assessment of how DHS conducted its tests and I will be requesting that they release a public version of this report. It is unclear what DHS gains by hiding the details from the public, except for the embarrassment of having its testing program criticized for the third time in five months."

The certification tests examined a new generation of radiation screening machines—called Advanced Spectroscopic Portals (ASPs). The review that found the tests to be flawed was prepared by the Homeland Security Institute (HSI), a DHS-sponsored and federally funded research and development center. The review was labeled "Official Use Only" under DHS direction.

The 178-page review by HSI found:

"The overall testing approach was not designed to measure the range of ASP system performance; notably, minimum detectable amounts were not measured. Under this approach, an improved ability to detect or identify test objects cannot be readily translated into an improved ability to detect and identify actual objects that might be smuggled, and the operational significance of any such improvement cannot be readily assessed."

"Even after collecting all available test results, it was difficult to form conclusions about operational effectiveness."

"Very little testing has been done to evaluate ASP reliability in an operating environment."

"The limitations found by DNDO's [Domestic Nuclear Detection Office's] test approach, analysis methods, and scoring schemes makes it almost impossible to evaluate the performance of the ASP."

The February 20, 2008 review supports the earlier findings of a September 18, 2007 review conducted by the Government Accountability Office (GAO). GAO concluded that DNDO's tests "were not designed to test the limitations of the ASP's detection capabilities", "did not use a sufficient amount of the type of materials that would mask or hide dangerous sources," did not use procedures that would allow an apples to apples comparison with existing technology, and did not replicate the real world conditions that would be present in a terrorist smuggling effort.

Following hearings by the Committee on Energy and Commerce on September 18, 2007, the Committee urged DHS to suspend full scale deployment because DHS did not understand the risks of a false negative reading, which could allow nuclear materials to slip through undetected.

The HSI review equivocated on a key GAO finding that DNDO "used biased test methods" by giving key information about the radiological materials to the vendors of the ASPs in advance of the tests. According to GAO, this allowed the vendors to calibrate their machines and boost performance. GAO noted, "It is highly unlikely that such favorable circumstances would present themselves under real world conditions."

While HSI found that the DHS "test procedures were not as rigorous as they might have been, due to the use of threat objects as calibration sources," they claimed that "system configurations were locked…and test results had not been modified to benefit from the reduced set of possible outcomes." However, GAO never contended that the test results had been modified by unlocking the machines.

The HSI review did note that the DHS's organizational structure contributed to a flawed testing regimen. It stated, "A generally accepted principle in government and industry is that the sponsoring or developing agency should not be the testing authority because they have an inherent conflict of interest." Accordingly, HSI recommends "An independent operational test and evaluation process and organization to ensure that testing measures the operational performance and reliability of new systems."

This supports GAO's earlier recommendation that "If additional testing is needed, the Secretary should appoint an independent group within DHS, not aligned with the ASP acquisition process, to conduct objective, comprehensive, and transparent testing that realistically demonstrates the capabilities and limitations of the ASP system."

In comments published as part of the HSI review, DHS largely disagreed with the technical findings, except for one finding that DNDO's earlier tests had not been biased. DNDO is planning its next round of tests at the Nevada Test Site, but has not established an independent testing organization as recommended by both GAO and HSI.

Source