JASON CAUTIONS ON PREDICTING TERRORIST EVENTS

JASON CAUTIONS ON PREDICTING TERRORIST EVENTS

SECRECY NEWS
from the FAS Project on Government Secrecy
Volume 2009, Issue No. 88
November 4, 2009

Secrecy News Blog: http://www.fas.org/blog/secrecy/


Attempts to predict the occurrence or the likelihood of extreme acts of terrorist violence on the scale of 9/11 should be discouraged because the available data are too sparse to permit the reliable modeling of such "rare events," according to a new report to the Pentagon (pdf) from the JASON defense advisory panel.

In a nutshell, "it is simply not possible to validate (evaluate) predictive models of rare events that have not occurred, and unvalidated models cannot be relied upon."

On the other hand, the JASONs said, it may be possible and useful to assume that rare events are correlated with more frequent, observable events which can be reliably modeled. If one assumes that "rare events events occur on a continuum with more frequent events," then the latter can be used to help predict the former.

In this way, the JASONs calculated that the probability of another 9/11-scale event in the world could be about 7% in the next ten years. But for reasons they went on to enumerate, the underlying assumption of continuity between rare and frequent events is not demonstrably correct.

"Much of the work on [anticipating] rare terrorist events seems to take for granted that 'the truth is out there' and we can discover it in a sufficiently timely fashion with the right mixture of motivational assessment, social network analysis, capability measures, etc." This may not be true, they indicated.

The JASONs offered suggestions for improving the modeling process, and they stressed the need for "good, large datasets of [terrorist] events and incident data" that currently do not exist or are not widely available. It is "surprisingly hard to obtain primary datasets" even on "straightforward" questions of terrorist event frequency and magnitude.

They cautioned that the complexity of the problem and the presumed urgency of the threat have "led some to advocate the suspension of normal standards of scientific hypothesis testing, in order to press [predictive] models quickly into operational service." But "while appreciating the urgency, JASON believes such advice to be misguided.... Experience in the development of many other scientific fields shows the importance of adhering to rigorous scientific standards, so that small successes are tested, communicated, critically examined, reproduced, and built upon."

"Although patient husbandry of a long-term research program may fall short of addressing the immediate operational needs, JASON believes it is the best way forward for success in the long term." A copy of the new JASON report was obtained by Secrecy News. See "Rare Events," October 2009.