Modern Alchemy

Modern Alchemy, Freudian Slips, Quick-Fixes and Suchlike

If you are taking it for granted that the quest for a magic formula capable of transforming severe lack of knowledge / information into substantial knowledge was abandoned with the Enlightenment, I have news for you!

Apparently, against all scientific odds, Info-Gap scholars were successful in imputing likelihood and chanceto results generated by a non-probabilistic model that is completely devoid of any notion of likelihood!

Recall that Info-Gap decision theory prides itself on being non-probabilistic and likelihood-free. Yet, Info-gap scholars -- the Father of Info-Gap included -- now claim that Info-Gap's robustness model is capable of identifying decisions that are most likely to satisfy a given performance requirement.

Consider for instance the following quotes (emphasis is mine):

Information-gap (henceforth termed 'info-gap') theory was invented to assist decision-making when there are substantial knowledge gaps and when probabilistic models of uncertainty are unreliable (Ben-Haim 2006). In general terms, info-gap theory seeks decisions that are most likely to achieve a minimally acceptable (satisfactory) outcome in the face of uncertainty, termed robust satisficing. It provides a platform for comprehensive sensitivity analysis relevant to a decision.

Burgman, Wintle, Thompson, Moilanen, Runge, and Ben-Haim (2008, p. 8).
Reconciling uncertain costs and benefits in Bayes nets for invasive species management
ACERA Endorsed Core Material: Final Report, Project 0601 - 0611, July 2008.
(PDF file, Downloaded on March 21, 2009)

However, if they are uncertain about this model and wish to minimize the chance of unacceptably large costs, they can calculate the robust–optimal number of surveys with eqn 5.

Tracy M. Rout, Colin J. Thompson, and Michael A. McCarthy (2009, p. 785)
Robust decisions for declaring eradication of invasive species
Journal of Applied Ecology 46, 782–786.

This is a major scientific breakthrough!

For, until now we have been warned repeatedly by Info-Gap scholars that no likelihood must be attributed to results generated by Info-Gap decision models. Indeed, we have been advised that this would be deceptive and even dangerous (emphasis is mine):

However, unlike in a probabilistic analysis, r has no connotation of likelihood. We have no rigorous basis for evaluating how likely failure may be; we simply lack the information, and to make a judgment would be deceptive and could be dangerous. There may definitely be a likelihood of failure associated with any given radial tolerance. However, the available information does not allow one to assess this likelihood with any reasonable accuracy.

Ben-Haim (1994, p. 152)
Convex models of uncertainty: applications and implications
Erkenntnis, 4, 139-156.

This point is also made crystal clear in the second edition of the Info-Gap book (emphasis is mine):

In info-gap set models of uncertainty we concentrate on cluster-thinking rather than on recurrence or likelihood. Given a particular quantum of information, we ask: what is the cloud of possibilities consistent with this information? How does this cloud shrink, expand and shift as our information changes? What is the gap between what is known and what could be known. We have no recurrence information, and we can make no heuristic or lexical judgments of likelihood.

Ben-Haim (2006, p. 18)
Info-Gap Decision Theory: Decisions Under Severe uncertainty
Academic Press.

So the question is: have Info-gap scholars managed to accomplish a major feat in the area of decision-making under severe uncertainty?

Of course the answer is that this new claims (Burgman et al. 2008, Rout et al. 2009) are not due to a breakthrough in decision-making under severe uncertainty, but rather to a serious misrepresentation of Info-gap’s robustness model, culminating in a thoroughly incorrect representation of the results.

My view on these episode -- based as it is on numerous discussions with Info-Gap scholars over the past five years -- is that these new claims are simply -- but not surprisingly -- ... Freudian slips.

The point is that -- see my FAQs about Info-Gap -- without imputing some sort of "likelihood" to Info-Gap's decision model, Info-Gap decision theory is, and cannot escape being, a voodoo decision theory.

So, all that these Freudian slips manage to do is to extend the already existing error -- an alternative that some Info-Gap scholars seem to prefer to an admission to a mistake.

It is interesting to note, though, that some Info-Gap scholars have taken note of my criticism of Info-Gap's robustness analysis to the effect that they now introduce an assumption that explicitly imputes "likelihood" to Info-Gap's uncertainty model. For instance, consider this (emphasis is mine):

An assumption remains that values of u become increasingly unlikely as they diverge from û.

Hall, J. and Harvey, H. (2009, p. 2)
Decision making under severe uncertainty for flood risk management: a case study of info-gap robustness analysis.
Eighth International Conference on Hydroinformatics
(January 12-16, 2009, Concepcion, Chile)
(PDF file)

Although this attempt at a quick-fix fails to fix the problem (see FAQ # 78), it does attest to a recognition that without such an assumption, conducting an analysis of the kind prescribed by Info-Gap's robustness model is utterly senseless.

One can only wonder then: how long will it take other Info-Gap scholars such as Burgman et al. (2008) and Rout et al. (2009) to reach this unavoidable conclusion?

Only time will tell (March 21, 2009).

Postscript:

The good news is that I am extremely pleased that, apparently for the first time, an official Government commissioned report takes notice of my criticism of Info-Gap decision theory.

This is long overdue!

Sadly, it is not an Aussie report!

As they say,
You can't be a prophet in your own land!
What a pity!

What a waste!

I hope that AU government agencies that sponsor info-gap projects will soon follow suit and re-examine this voodoo decision theory.

This is long overdue.

More important: I hope that senior academics promoting this theory in Australia and elsewhere will reconsider their position, especially insofar as supervising PhD students on this subject.

This is long overdue!

In any case, the following paragraph is a quote from page 75 of the 2009 Department for Environment Food and Rural Affairs (DEFRA) report:

More recently, Info-Gap approaches that purport to be non-probabilistic in nature developed by Ben-Haim (2006) have been applied to flood risk management by Hall and Harvey (2009). Sniedovich (2007) is critical of such approaches as they adopt a single description of the future and assume alternative futures become increasingly unlikely as they diverge from this initial description. The method therefore assumes that the most likely future system state is known a priori. Given that the system state is subject to severe uncertainty, an approach that relies on this assumption as its basis appears paradoxical, and this is strongly questioned by Sniedovich (2007).

Mervyn Bramley, Ben Gouldby, Anthony Hurford, Jaap-Jeroen Flikweert
Marta Roca Collell, Paul Sayers, Jonathan Simm, Michael Wallis
Delivering Benefits Through Evidence
PAMS (Performance-based Asset Management System)
Phase 2 Outcome Summary Report (PDF File)
Project: SC040018/R1
Environment Agency -- December 2009
Department for Environment Food and Rural Affairs
UK

The diplomatic language of the report cannot veil the obvious fact: Info-Gap decision theory is a voodoo theory!

But, .... if the flaws are so obvious, how is it that senior academics continue to promote this paradoxical, voodoo theory?

Hence, ... my campaign is still on!