Overview
One need hardly point out that the mathematical modeling of socio-political-economic systems and processes is an extremely difficult task. As many academics/analysts/consultants would no doubt testify, modeling even small-scale real-world problems with prima facie well-defined attributes, goals, and so on, can present such difficulties that formulating a precise mathematical model that faithfully captures all the problem's nuances may defeat even the most experienced of analysts/modelers.
Yet... given Bueno de Mesquita claims (For example, his Feb Feb 2009 Ted.TV talk), it turns out that not only does he possess a remarkable ability to model socio-political/economic systems, the method that he has come up with
can predict complex negotiations or situations involving coercion, that is in essence everything that has to do with politics, much of what has to do with business, but sorry, if you're looking to speculate in the stock market, I don't predict stock markets -- OK, it's not going up any time really soon. But I'm not engaged in doing that. I'm not engaged in predicting random number generators, I actually get phone calls from people who want to know what lottery numbers are going to win. I don't have a clue.
One thing is certain. Bueno de Mesquita's assessment of his prediction skills is spot on when it comes to the stock market! For as we all know, his prediction "OK, it's not going up any time really soon" turned out to be wrong, very wrong. The DJIA Index started climbing in the first week of March 2009, and is up by almost 40% now (Dec 14, 2009). Here is the picture:
Bueno de Mesquita's method is grounded on the assumption that people are basically rational beings:
Now who is rational? A lot of people are worried about what is rationality about? People are rational. Mother Theresa, she was rational. Terrorists, they're rational. Pretty much everybody is rational. I think there are only two exceptions that I'm aware of -- two-year-olds, they are not rational, they have very fickle preferences, they switch what they think all the time, and schizophrenics are probably not rational, but pretty much everybody else is rational. That is, they are just trying to do what they think is in their own best interest.
But isn't this precisely where all the trouble lies? Even if we play down the evidence that people do not always behave rationally (especially under stress), the fundamental difficulty of methods based on this conception of rational behavior is the accurate modeling of :"what is in their best interest"?!
Is it surprising then that Bueno de Mesquita's contentions regarding his models' capabilities to yield concrete results, predicting precise outcomes of conflict situations, have provoked such criticism?
Before I proceed to list some of the arguments criticizing Beuno de Mesquita's method -- discussed in articles published in refereed journals -- here is an [edited] comment on his Feb 2009 Ted.TV talk:
Charles Harpole
Nov 30 2009:
Bruce' Bueno's mumbo jumbo is [...]. If you give me the same info he has, I will equal or beat his math. The reason is that human nature is simply not always rational. I thought everyone knew that. And, that no one can look into another's head with accuracy as he claims. My accurate prediction: by 2026, China will surpass usa as a world economic power and will create a new international money to replace the Dollar... and make it stick. Go chew on that one and I used the most powerful non-rational and also human understanding power of all........ my brain. [...]
Ignoring the uncalled for explitives and crass language, this comment does highlight the basic dificulty that "... no one can look into another's head with accuracy ..." to construct a model that will predict with accuracy their behavior ... not even high-ranking politicians assisted by the most able experts in deicison-making ...
Also, Bueno de Mesquita's theory is based on expected utility theory. It is highly questionable whether this theory is a proper tool for the analysis of the behaviour of complex socieconomic systems that are subjected to severe uncertainty.
Readers who are not familiar with the problematics of expected utility theory may wish to read the paper entitled MAPS OF BOUNDED RATIONALITY: A PERSPECTIVE ON INTUITIVE JUDGMENT AND CHOICE by DANIEL KAHNEMAN (Nobel Prize Lecture, December 8, 2002). Or watch the lecture.
Criticism
To give you an idea of the kind of "formal" criticism that Bueno de Mesquita's theory has provoked, here is a short list of articles published in refereed journals.
- Robert Harrison Wagner (1984), War and Expected-Utility Theory
- Stephen J. Majeski and David J. Sylvan (1984), Simple Choices and Complex Calculations: A Critique of the War Trap
- Robert S. Thompson (1987), Forecasting the future of Hong Kong
- Roslyn Simowitz and Barry L. Price (1990), The Expected Utility Theory of Conflict: Measuring Theoretical Progress
- M. Cristina Molinari (2000), Military Capabilities and Escalation: A Correction to Bueno de Mesquita, Morrow, and Zorick
Let me know of any other articles of this nature, especially more recent ones.
Some background material on Bueno de Mesquita and related topics is provided in the Appendix.
A comment regarding Bueno de Mesquita's (1981) claims concerning the contribution of his theory to the state of the art and the empirical evidence supporting its validity:
These are strong claims, and they merit careful scrutiny. Unfortunately, I do not believe they are fully justified. In the first part of this article, I will argue that Bueno de Mesquita's theory cannot be derived from his assumptions. In the second part, I will examine the operational version of his theory, which is the one that was actually tested. I will argue that it can be more plausibly interpreted as a version of a theory he professes to discredit rather than the theory he professes to deduce from his own assumptions. I will also argue that the empirical evidence offered provides less impressive confirmation even of the theory actually tested than Bueno de Mesquita claims.Wagner (1984, p. 408)
From the Conclusions section:
Contrary to appearances, therefore, The War Trap provides no evidence, one way or the other, on the question of whether theories of individual rational choice can explain foreign policy decisions. Moreover, when one considers how Bueno de Mesquita's analysis would have to be modified to avoid the criticisms made above, it is apparent that it would not be easy to devise as straightforward a test this hypothesis as the book professes to provide.Wagner (1984, p. 423)
There are several problems with the conceptualizations that Bueno de Mesquita employs. First, he drastically reduces the scope of his concepts' meaning, thereby vitiating their significance. Second, his concepts turn out to be too ambiguous or problematic for rigorous inquiry. Finally, Bueno de Mesquita makes assumptions that are unjustified and inconsistent.Majeski and Sylvan (1984, p. 317)
From the discussion so far, it should be apparent that the theory put forward in The War Trap is badly flawed in both conceptualization and measurement. For this reason, the empirical analyses in Chapter 5 must be taken with more than a grain of salt. We therefore restrict ourselves to a brief look at Bueno de Mesquita's empirical tests.Majeski and Sylvan (1984, p. 338)
The Conclusions section is particularly harsh:
It is always easier to criticize than to propose constructive alternatives. For reasons of space the latter are omitted. However, from the above analysis, it is clear that such an alternative would include more realistic theory, subtler and more complex measurement, and "softer" and more micro-level empirical tests. The precise form of such changes is, of course, an open research question; we remain convinced, however, that movement in this direction is a necessity if future research on war initiation is to yield valid and useful results. Unfortunately, the research design of The War Trap rules out any such movement, and pursuing it further would be counterproductive.Majeski and Sylvan (1984, p. 339-340)
Regarding the details of the forecasts:
Such are our authors' forecasts. The acid tests of these predictions, of course, lie in the answers to a series of questions: Are these predictions clear? Are they testable? Do they accord with outside evidence? Do they accord with the authors' own supporting documentation? Does that documentation exist, or is it, at least, presented? And ultimately, are the forecasts correct? Unfortunately, the book falls short on every one of these counts, The trouble begins early on, as the authors present their four general points about the 1984 Declaration. Point Three states that, for both British and Chinese, Hong Kong's stability until 1997 was an important goal. Perhaps so, but this statement is a description of ends and not a prediction of means. Point Two indicates that Chinese and British interests were different - a thought that, again, is scarcely a prediction; and the same is true of Point Four, which alludes to Britain's supposed "impotence". Only Point One is really a forecast, since it tells us that Hong Kong's sovereignty will be transferred to China. The authors fail, however, to define "sovereignty," a critical failure, as we can see if we turn to the specific forecasts for the Joint Declaration.Thompson (1987. p. 188)
The last ten lines of the paper reads as follows:
But whatever the shortcomings of China scholars, the authors' approach is also lacking. By their own admission (p. 13), they select the information demanded by their model and even, it seems possible, invent their facts - at least, they give us no way of checking their facts. In short, they let their method determine their "evidence." But that process is argument by tautology and so has little or nothing to do with the real world. And that is precisely the problem. The authors, and others who would apply mathematical models to Chinese studies, have yet to demonstrate that their approach is not tautological.Thompson (1987. p. 195)
This is one of the most detailed and comprehensive technical analysis of The War Trap. Some quotes:
Internal Conceptual Problems. There are two major kinds of internal conceptual problems that plague Bueno de Mesquita's expected utility theory. The first of these — use of vague and ambiguous concepts — has already been discussed in relation to his work by other authors. Majeski and Sylvan (1984), for example, emphasize the "reductionism" and ambiguity of such concepts as utility, national foreign policy, possible opponents of potential war initiators and the like. They argue that these concepts, as used by Bueno de Mesquita in The War Trap are ambiguous and that the scope of their meanings are reduced in actual use. Luterbacher (1984) argues that by only taking into account what third parties might or might not do, Bueno de Mesquita's conceptualization of risk taking is too restrictive and does not capture most of what we mean by this concept.Simowitz and Price (1990, p. 445)
We have identified the theory's solved empirical problems, as well as theoretical predictions that were either unsupported by the data or were faced with refuting instances. While agreeing with Bueno de Mesquita that refutations do not falsify his theory, we have argued that he needs to treat these anomalous cases as if they were theoretical refutations and at some point attempt to modify his theory to eliminate them. We also agreed that an appraisal of his theory requires Bueno de Mesquita to compare his theoretical predictions and the predictions of rival theories. We found, however, that due to inferential errors in his deductions, his comparisons were seriously flawed. Consequently, we are unable to choose between Bueno de Mesquita's theory and rival theories of conflict on the basis of which one has solved more empirical problems. We are nevertheless able to provide an overall evaluation of Bueno de Mesquita's theory in light of our review of the theory's solved empirical problems and unresolved conceptual and anomalous problems.Simowitz and Price (1990, p. 454)
By adopting Laudan's criteria for theoretical progress, we were led to identify invalid inferences in Bueno de Mesquita's theory, as well as to examine all derivations of the theory confronted with refutations. Just as importantly, we were provided a framework for assessing a wide range of diverse criticisms directed at this theory. By helping us to weigh the strengths and weaknesses of Bueno de Mesquita's work, Laudan's framework led us to recommend pursuing Bueno de Mesquita's theory in spite of its obvious flaws. Whether or not one agrees with this recommendation, however, the fact that Laudan's theory of scientific progress was so helpful in evaluating Bueno de Mesquita's work and the wide range of criticism it has provoked indicates that Laudan's criteria has broad applicability to social science theories.Simowitz and Price (1990, p. 457)
We additionally believe that Bueno de Mesquita has misrepresented the balance of power theory in his presentation of the hypotheses. Specifically, the balance of power theory incorporates both power and alignments as sufficient conditions for war; but Bueno de Mesquita, in his tests of balance of power and alliances, tests the individual impact of each of these variables on war Initiation. Wagner (1984) makes a similar point.Simowitz and Price (1990, Note 13, p. 458)
Commentary on this criticism can be found in
James D. Morrow, Barry L. Price, Roslyn Simowitz
Conceptual Problems in Theorizing About International Conflict
The American Political Science Review, Vol. 85, No. 3 (Sep., 1991), pp. 923-940.
The criticism is technical in nature in that it exposes an obvious error in the mathematical analysis of the proposed model. The important point here is that this error demolishes the foundation of the proposed model.
In "Capabilities, Perception, and Escalation," Bueno de Mesquita, Morrow, and Zorick (1997) model the evolution of disputes between nations as a game of two-sided incomplete information and test empirically their theoretical results. Their main conclusion is that the relationship between the ex ante probability of a crisis escalating to armed conflict and the countries' observable military capabilities is not monotonic; in particular, this probability is initially decreasing and then increasing. In other words, the ex ante probability of an escalation to armed conflict is at a minimum when power, measured by observable military capabilities, is balanced. Although these findings are supported by the data, the analysis of the equilibria contains a mistake that invalidates some of the statements.This note explains the nature of the mistake and discusses how the conclusions change once the correct equilibria are considered. I find that the relationship between the observed military capabilities and the probability of armed conflict given a crisis is constant and equal to one. Therefore, observable military capabilities have no role in explaining differences in the evolution of crises, and we are left with no explanations for the empirical findings of Bueno de Mesquita, Morrow, and Zorick
Molinari (2000, p. 425 )
Bueno de Mesquita, Morrow, and Zorick acknowledge the error in their article.
The CIA connection
Given that Bueno de Mesquita's method is used by organizations such as the CIA to predict the outcome of wars and conflicts, it is in the public interest to know how "good"/"bad" the predictions generated by his models are.
The important point to note in this regard is that as amply made plain by Prof. Bueno de Mesquita himself, the application of his prediction models hangs on the information (expert knowledge) supplied by experts on the subject matter.
But, according to the CIA web site,
Intelligence analysis, like other complex tasks, demands considerable expertise. It requires individuals who can recognize patterns in large data sets, solve complex problems, and make predictions about future behavior or events. To perform these tasks successfully, analysts must dedicate a considerable number of years to researching specific topics, processes, and geographic regions.Paradoxically, it is the specificity of expertise that makes expert forecasts unreliable. While experts outperform novices and machines in pattern recognition and problem solving, expert predictions of future behavior or events are seldom as accurate as simple actuarial tables. In part, this is due to cognitive biases and processing-time constraints. In part, it is due to the nature of expertise itself and the process by which one becomes an expert.
Rob Johnston
Integrating Methodologists into Teams of Substantive Experts
Reducing Analytic Error
Studies in Intelligence
Volume 47, Number 1, 2003.
One imagines that the above statement is based on the vast experience gained at the CIA from a wide-ranging interaction with internal/external experts.
Therefore, the claim that extensive tests conducted by the CIA show that Bueno de Mesquita's prediction models were accurate 90% of the time, is impressive. Indeed, in a sense, it contradicts the CIA assessment regarding the performance of experts in the messy business of forecasting.
If you are familiar with publications providing evidence substantiating this claim, please send me the details and I shall post them here. The main reference I have to this claim is
Feder, Stanley A.
Annual Review of Political Science
5:111-25, 2002
and references cited therein.
On the other hand, the extensive empirical tests reported on in
D. Scott Bennett and Allan C. Stam
The Behavioral Origins of War
University of Michigan Press, 2004
do not paint such a rosy picture.
The n factorial Debacle
In a recent (February 2009) TED lecture Bueno de Mesquita made the contention that the number of possible interactions between n individuals is n factorial, namely n!. He showed a graphic example with n=5 individuals and the ten links between them (6:00 minute into the lecture):
 |
 |
|
| n=5 individuals | n=5 individuals with 10 links |
So far so good.
Then, out of the blue, he went on to claim that there are n! possible interactions between these 5 individuals:
 |
| n=5 individuals, 10 links. How many interactions? |
Clearly, this does not add up. However, as this point was eventually taken up and discussed at some length on the internet (eg. see Abu muqawama and TED), I shall not elaborate on it any further except to note the following.
The number of links between n individuals is "n choose 2" = n!/(2!(n-2)!). See for example the following comment at Abu muqawama:
Apparently Bueno de Mesquita was asked to clarify his assertion that the number of interactions is n!. Unfortunately, the explanation is as confusing as the original statement, if not more so. A clip from Abu muqawama:
In fact, the 'logic' is not "very simple" at all. For, it is hardly clear what kind of "interactions" Bueno de Mesquita has in mind. For instance, consider this (Abu muqawama):
More generally, it is easy to envisage a definition of "interaction" that will yield more than n! possible "interactions" between n individuals. To repeat then: what kind of "interactions" enter into Bueno de Mesquita's analysis of the future of Iran?
The fact is that Bueno de Mesquita should have no problem at all to give us an edifying account of the type of interactions that form part of his analysis because, given that 4!=24, it is rather simple to enumerate all such interactions for n=4.
As things stand then, this claim is manifestly false (unless unbeknownst to us, Bueno de Mesquita is making certain assumptions about the problem). The inference is then that Bueno de Mesquita would do well to provide a clear explanation to resolve this question.
For, wouldn't it be ironic if it turns out that Bueno de Mesquita is in error on such a simple, yet important, matter!
Appendix
New Nostradamuses
One need hardly point out that not only professionals who make "decision under uncertainty" their metier, but also the proverbial "man in the street", take it for granted that the ability to accurately predict future events is one of the most onerous challenges facing humankind — especially persons in authority, persons responsible for the management of business or economic organizations etc.
But apparently no longer.
For, according to Good Magazine, predicting future events — at least in the area of international conflicts — is now possible thanks to the efforts of the New Nostradamus: Prof. Bruce Bueno de Mesquita, a political science professor at New York University and Senior Fellow at the Hoover Institution.
The claim is that this distinguished political scientist can actually predict the future — more specifically the outcome of any international conflcit!
And this he does not with the aid of the age old Crystal Ball, but through the use of a scientific method that, apparently, is grounded in a branch of applied mathematics called Game Theory.
According to GoodReads.com,
" ... Bruce Bueno de Mesquita is a political scientist, professor at New York University, and senior fellow at the Hoover Institution. He specializes in international relations, foreign policy, and nation building. He is also one of the authors of the selectorate theory.
He has founded a company, Mesquita & Roundell, that specializes in making political and foreign-policy forecasts using a computer model based on game theory and rational choice theory. He is also the director of New York University's Alexander Hamilton Center for Political Economy.
He was featured as the primary subject in the documentary on the History Channel in December 2008. The show, titled Next Nostradamus, details how the scientist is using computer algorithms to predict future world events ..."
Here is an interview with Prof. Bueno de Mesquita (with Riz Khan - The art and science of prediction - 09 Jan 08):
And here is a 20-minute lecture on the ... future of Iran (TED, February 2009):
Apparently, all you need to accomplish this is a computer, expert-knowledge on Iran, and game theory!
Some of the predictions attributed to Prof. Bueno de Mesquita are:
- The second Palestinian Intifada and the death of the Mideast peace process, two years before this came to pass.
- The succession of the Russian leader Leonid Brezhnev by Yuri Andropov, who at the time was not even considered a contender.
- The voting out of office of Daniel Ortega and the Sandanistas in Nicaragua, two years before this happened.
- The harsh crack down on dissidents by China's hardliners four months before the Tiananmen Square incident.
- France's hairs-breadth passage of the European Union's Maastricht Treaty.
- The exact implementation of the 1998 Good Friday Agreement between Britain and the IRA.
- China's reclaiming of Hong Kong and the exact manner the handover would take place, 12 years before it happened.
Impressive, isn't it!
As might be expected, these and similar claims by Prof. Bueno de Mesquita have sparked a vigorous debate not only in the professional journals but also on the WWW. Interested readers can consult this material to see for themselves, whether Bueno de Mesquita's claims attest to a major scientific breakthrough or ... voodoo mathematics.
Also, in addition to consulting this material you may want to have a look at a short video clip by Matt Brawn (right) which, he compiled in response to a short note entitled This man can actually predict the future!.
Of particular interest is, of course, the "success" rate of the Prof. Bueno de Mesquita's predictions: over 90% — yes over ninty percent!
Here is Trevor Black's common sense reaction to this claim:
I am a little skeptical about anyone who claims to have a 90% success rate. I just don't buy it. Especially when they say that they can explain away a lot of the other 10%.If you come to me and tell me you have a model that gets it right 60% or 70% of the time, I may listen. Skeptically, but I will listen. 90% and I start to smell something.
All I wish to add here is that Prof. Bueno de Mesquita (left) makes his predictions under conditions of "severe uncertainty" which of course render them hugely vulnerable to what Prof. Naseem Taleb (right) dubs the Black Swan phenomenon.
 |
Hence, the very proposition that such predictions can be made at all, let alone be reliable, is diametrically opposed to Nassim Taleb's categorical rejection of any such position. For his thesis is that Black Swans are totally outside the purview of mathematical treatment, especially by models that are based on expected utility theory and rational choice theory. Interesting, though, this is precisely the stuff that Prof. Bueno de Mesquita's method is made of: expected utility theory and rational choice theory! Even more interesting is the fact that Nassim Taleb (right) and Bueno de Mesquita (left) are staff members of the same academic institution, namely New York University. So, all that's left to say is: Go figure! |
 |
As indicated above, the debate over Bueno de Mesquita's theories is not new. It has been ongoing, in the relevant academic literature, at least since the publication of his book The War Trap (1981).
Note, therefore, that Bueno de Mesquita's work has attracted a considerable amount of criticism. For an idea of the kind of criticism sparked by his work, take a look at the quotes I provide from articles that are critical of Bueno de Mesquita theories.
Of course, there are other New Nostradamuses around.
According to the Associated Press, the latest (Mar 4, 4:39 AM EST) news from Russia about the future of the USA is that
" ... President Barack Obama will order martial law this year, the U.S. will split into six rump-states before 2011, and Russia and China will become the backbones of a new world order ..."Apparently this prediction was made by Igor Panarin (right), Dean of the Russian Foreign Ministry diplomatic academy and a regular on Russia's state-controlled TV channels (see full AP news report).
Regarding the future of Russia,
"You don't sound too hopeful".
"Hopeful? Please, I am Russian. I live in a land of mad hopes, long queues, lies and humiliations. They say about Russia we never had a happy present, only a cruel past and a quite amazing future ..."Malcolm Bradbury
To the Hermitage (2000, p. 347)We should therefore be reminded of J K Galbraith's (1908-2006) poignant observation:
There are two classes of forecasters: those who don't know and those who don't know they don't know.
And in the same vein,
The future is just what we invent in the present to put an order over the past.
Malcolm Bradbury
Doctor Criminale (1992, p. 328)So, we shall have to wait and see.
And how about this more recent piece by Heath Gilmore and Brian Robins in the Sydney Morning Herald (March 27, 2009):
"... COUPLES wondering if the love will last could find out if theirs is a match made in heaven by subjecting themselves to a mathematical test.
A professor at Oxford University and his team have perfected a model whereby they can calculate whether the relationship will succeed.
In a study of 700 couples, Professor James Murray, a maths expert, predicted the divorce rate with 94 per cent accuracy.
His calculations were based on 15-minute conversations between couples who were asked to sit opposite each other in a room on their own and talk about a contentious issue, such as money, sex or relations with their in-laws.
Professor Murray and his colleagues recorded the conversations and awarded each husband and wife positive or negative points depending on what was said. ..."
Such interviews should perhaps be made mandatory for all couples registering their marriage.
More details on the mathematics of marriage can be found in The Mathematics of Marriage: Dynamic Nonlinear Models by J.M. Gottman, J.D. Murray, C. Swanson, R. Tyson, and K.R. Swanson (MIT Press, Cambridge, MA, 2002.)
On a more positive note, though, here is an online Oracle from Melbourne (Australia: the land of the real Black Swan!).
You may wish to consult this friendly 24/7 facility about important "Yes/No" questions that you no doubt have about the future.
The Black Swan
Only time will tell what impact (if any) Nassim Taleb's recent popular and controversial book The Black Swan: The Impact of the Highly Improbable will have on the field of decision-making under severe uncertainty.
I, for one, hope that the issues raised in this book and in its predecessor, Fooled by Randomness: The Hidden Role of Chance in the Markets and in Life, will be instrumental in helping decision-makers to identify voodoo decision theories -- such as Info-Gap decision theory -- that promise robust decisions under severe uncertainty.
I fear though -- in view of my experience of the past 40 years - that the danger is that the huge success of the Black Swan will inspire a new wave of voodoo decision theories, purportedly capable of ... "domesticating" black swans and preempting the discovery of ... purple swans!
We shall have to wait and see.
For those who have "been in hiding" I should note that Taleb has become quite a celebrity. According to the Prudent Investor Newsletters (Tuesday, June 3, 2008):
- Mr. Taleb charges about $60,000 per speaking engagement and does about 30 presentations a year to "to bankers, economists, traders, even to Nasa, the US Fire Administration and the Department of Homeland Security" according to Timesonline’s Bryan Appleyard.
- He recently got $4million as advance payment for his next much awaited book.
- Earned $35-$40 MILLION on a huge Black Swan event-on the biggest stockmarket crash in modern history-Black Monday, October 19,1987.
So, if you haven’t heard him in person you can easily find on the WWW numerous videos of his interviews.
Here is a link to a very short (2:45 min) clip, recorded by Taleb himself, apparently at Heathrow Airport, of 10 tips on how to deal with Black Swans, and life in general.
- Scepticism is effortful and costly. It is better to be sceptical about matters of large consequences, and be imperfect, foolish and human in the small and the aesthetic.
- Go to parties. You can't even start to know what you may find on the envelope of serendipity. If you suffer from agoraphobia, send colleagues.
- It's not a good idea to take a forecast from someone wearing a tie. If possible, tease people who take themselves and their knowledge too seriously.
- Wear your best for your execution and stand dignified. Your last recourse against randomness is how you act -- if you can't control outcomes, you can control the elegance of your behaviour. You will always have the last word.
- Don't disturb complicated systems that have been around for a very long time. We don't understand their logic. Don't pollute the planet. Leave it the way we found it, regardless of scientific 'evidence'.
- Learn to fail with pride -- and do so fast and cleanly. Maximise trial and error -- by mastering the error part.
- Avoid losers. If you hear someone use the words 'impossible', 'never', 'too difficult' too often, drop him or her from your social network. Never take 'no' for an answer (conversely, take most 'yeses' as 'most probably').
- Don't read newspapers for the news (just for the gossip and, of course, profiles of authors). The best filter to know if the news matters is if you hear it in cafes, restaurants ... or (again) parties.
- Hard work will get you a professorship or a BMW. You need both work and luck for a Booker, a Nobel or a private jet.
- Answer e-mails from junior people before more senior ones. Junior people have further to go and tend to remember who slighted them.
It is interesting to juxtapose Prof. Taleb’s thesis in The Black Swan that severe uncertainty makes (reliable) prediction in the Socio/economic/political spheres impossible, with the polar position taken by his colleague, Prof. Bruce Bueno de Mesquita, who actually specializes in predicting the future.
Modern Alchemy, Freudian Slips, Quick-Fixes and Suchlike
If you are taking it for granted that the quest for a magic formula capable of transforming severe lack of knowledge / information into substantial knowledge was abandoned with the Enlightenment, I have news for you!
Apparently, against all scientific odds, Info-Gap scholars were successful in imputing likelihood to results generated by a non-probabilistic model that is completely devoid of any notion of likelihood!
Recall that Info-Gap decision theory prides itself on being non-probabilistic and likelihood-free. Yet, Info-gap scholars -- the Father of Info-Gap included -- now claim that Info-Gap's robustness model is capable of identifying decisions that are most likely to satisfy a given performance requirement.
Consider for instance the following quote from ACERA Endorsed Core Material (emphasis is mine):
Information-gap (henceforth termed 'info-gap') theory was invented to assist decision-making when there are substantial knowledge gaps and when probabilistic models of uncertainty are unreliable (Ben-Haim 2006). In general terms, info-gap theory seeks decisions that are most likely to achieve a minimally acceptable (satisfactory) outcome in the face of uncertainty, termed robust satisficing. It provides a platform for comprehensive sensitivity analysis relevant to a decision.
Burgman, Wintle, Thompson, Moilanen, Runge, and Ben-Haim (2008, p. 8).
Reconciling uncertain costs and benefits in Bayes nets for invasive species management
ACERA Endorsed Core Material: Final Report, Project 0601 - 0611.
(PDF file, Downloaded on March 21, 2009)This is a major scientific breakthrough.
For, until now we have been warned repeatedly by Info-Gap scholars that no likelihood must be attributed to results generated by Info-Gap decision models. Indeed, we have been advised that this would be deceptive and even dangerous (emphasis is mine):
However, unlike in a probabilistic analysis, r has no connotation of likelihood. We have no rigorous basis for evaluating how likely failure may be; we simply lack the information, and to make a judgment would be deceptive and could be deceptive and dangerous. There may definitely be a likelihood of failure associated with any given radial tolerance. However, the available information does not allow one to assess this likelihood with any reasonable accuracy.
Ben-Haim (1994, p. 152)
Convex models of uncertainty: applications and implications
Erkenntnis, 4, 139-156.This point is also made crystal clear in the second edition of the Info-Gap book (emphasis is mine):
In info-gap set models of uncertainty we concentrate on cluster-thinking rather than on recurrence or likelihood. Given a particular quantum of information, we ask: what is the cloud of possibilities consistent with this information? How does this cloud shrink, expand and shift as our information changes? What is the gap between what is known and what could be known. We have no recurrence information, and we can make no heuristic or lexical judgments of likelihood.
Ben-Haim (2006, p. 18)
Info-Gap Decision Theory: Decisions Under Severe uncertainty
Academic Press.So the question is: have Info-gap scholars managed to accomplish a major feat in the area of decision-making under severe uncertainty?
Of course the answer is that this new claim (Burgman et al's 2008) is not due to a breakthrough in decision-making under severe uncertainty, but rather to a serious misrepresentation of Info-gap’s robustness model, culminating in a thoroughly incorrect representation of the results.
My view on this episode -- based as it is on numerous discussions with Info-Gap scholars over the past five years -- is that this new claim is simply -- but not surprisingly -- ... a Freudian slip.
The point is that -- see my FAQs about Info-Gap -- without imputing some sort of "likelihood" to Info-Gap's decision model, Info-Gap decision theory is, and cannot escape being, a voodoo decision theory.
So, all that this Freudian slip manages to do is to extend the already existing error -- an alternative that some Info-Gap scholars seem to prefer to an admission to a mistake.
It is interesting to note, though, that some Info-Gap scholars have taken note of my criticism of Info-Gap's robustness analysis to the effect that they now introduce an assumption that explicitly imputes "likelihood" to Info-Gap's uncertainty model. For instance, consider this (emphasis is mine):
An assumption remains that values of u become increasingly unlikely as they diverge from û.
Hall, J. and Harvey, H. (2009, p. 2)
Decision making under severe uncertainty for flood risk management: a case study of info-gap robustness analysis.
Eighth International Conference on Hydroinformatics
(January 12-16, 2009, Concepcion, Chile)
(PDF file)Although this attempt at a quick-fix fails to fix the problem (see FAQ # 78), it does attest to a recognition that without such an assumption, conducting an analysis of the kind prescribed by Info-Gap's robustness model is utterly senseless.
One can only wonder then: how long will it take other Info-Gap scholars such as Burgman et al (2008) to reach this unavoidable conclusion?
Only time will tell (March 21, 2009).
Recent Articles, Working Papers, Notes
Also, see my complete list of articles '
- Sniedovich M. (2009) Modeling of robustness against severe uncertainty, pp. 33- 42, Proceedings of the 10th International Symposium on Operational Research, SOR'09, Nova Gorica, Slovenia, September 23-25, 2009.
- Caserta, M., Voss, S., Sniedovich, M. (2009) Applying the corridor method to a blocks relocation problem, OR Spectrum in press.
- Sniedovich M.(2009) A Critique of Info-Gap Robustness Model. In: Martorell et al. (eds), Safety, Reliability and Risk Analysis: Theory, Methods and Applications, pp. 2071-2079, Taylor and Francis Group, London.
.
- Sniedovich M.(2009) A Classical Decision Theoretic Perspective on Worst-Case Analysis, Working Paper No. MS-03-09, Department of Mathematics and Statistics, The University of Melbourne, (PDF File)
- Caserta, M., Voss, S., Sniedovich, M. (2008) The corridor method - A general solution concept with application to the blocks relocation problem. In: A. Bruzzone, F. Longo, Y. Merkuriev, G. Mirabelli and M.A. Piera (eds.), 11th International Workshop on Harbour, Maritime and Multimodal Logistics Modeling and Simulation, DIPTEM, Genova, 89-94.
- Sniedovich, M. (2008) FAQS about Info-Gap Decision Theory, Working Paper No. MS-12-08, Department of Mathematics and Statistics, The University of Melbourne, (PDF File)
- Sniedovich, M. (2008) A Call for the Reassessment of the Use and Promotion of Info-Gap Decision Theory in Australia (PDF File)
- Sniedovich, M. (2008) Info-Gap decision theory and the small applied world of environmental decision-making, Working Paper No. MS-11-08
This is a response to comments made by Mark Burgman on my criticism of Info-Gap (PDF file
- Sniedovich, M. (2008) A call for the reassessment of Info-Gap decision theory, Decision Point, 24, 10.
- Sniedovich, M. (2008) From Shakespeare to Wald: modeling wors-case analysis in the face of severe uncertainty, Decision Point, 22, 8-9.
- Sniedovich, M. (2008) Wald's Maximin model: a treasure in disguise!, Journal of Risk Finance, 9(3), 287-291.
- Sniedovich, M. (2008) Anatomy of a Misguided Maximin formulation of Info-Gap's Robustness Model (PDF File)
In this paper I explain, again, the misconceptions that Info-Gap proponents seem to have regarding the relationship between Info-Gap's robustness model and Wald's Maximin model.
- Sniedovich. M. (2008) The Mighty Maximin! (PDF File)
This paper is dedicated to the modeling aspects of Maximin and robust optimization.
- Sniedovich, M. (2007) The art and science of modeling decision-making under severe uncertainty, Decision Making in Manufacturing and Services, 1-2, 111-136. (PDF File)
- Sniedovich, M. (2007) Crystal-Clear Answers to Two FAQs about Info-Gap (PDF File)
In this paper I examine the two fundamental flaws in Info-Gap decision theory, and the flawed attempts to shrug off my criticism of Info-Gap decision theory.
- My reply (PDF File)
This is an exciting development!
- Ben-Haim's response confirms my assessment of Info-Gap. It is clear that Info-Gap is fundamentally flawed and therefore unsuitable for decision-making under severe uncertainty.
- Ben-Haim is not familiar with the fundamental concept point estimate. He does not realize that a function can be a point estimate of another function.
So when you read my papers make sure that you do not misinterpret the notion point estimate. The phrase "A is a point estimate of B" simply means that A is an element of the same topological space that B belongs to. Thus, if B is say a probability density function and A is a point estimate of B, then A is a probability density function belonging to the same (assumed) set (family) of probability density functions.
Ben-Haim mistakenly assumes that a point estimate is a point in a Euclidean space and therefore a point estimate cannot be say a function. This is incredible!
- A formal proof that Info-Gap is Wald's Maximin Principle in disguise. (December 31, 2006)
This is a very short article entitled Eureka! Info-Gap is Worst Case (maximin) in Disguise! (PDF File)
It shows that Info-Gap is not a new theory but rather a simple instance of Wald's famous Maximin Principle dating back to 1945, which in turn goes back to von Neumann's work on Maximin problems in the context of Game Theory (1928).
- A proof that Info-Gap's uncertainty model is fundamentally flawed. (December 31, 2006)
This is a very short article entitled The Fundamental Flaw in Info-Gap's Uncertainty Model (PDF File)
It shows that because Info-Gap deploys a single point estimate under severe uncertainty, there is no reason to believe that the solutions it generates are likely to be robust.
- A math-free explanation of the flaw in Info-Gap. ( December 31, 2006)
This is a very short article entitled The GAP in Info-Gap (PDF File)
It is a math-free version of the paper above. Read it if you are allergic to math.
- A long essay entitled What's Wrong with Info-Gap? An Operations Research Perspective (PDF File)
This is a paper that I presented at the ASOR Recent Advances in Operations Research (PDF File)
Recent Lectures, Seminars, Presentations
If your organization is promoting Info-Gap, I suggest that you invite me for a seminar at your place. I promise to deliver a lively, informative, entertaining and convincing presentation explaining why it is not a good idea to use — let alone promote — Info-Gap as a decision-making tool.
Here is a list of relevant lectures/seminars on this topic that I gave in the last two years.
- A Critique of Info-Gap Decision theory: From Voodoo Decision-Making to Voodoo Economics(PDF File)
(Recent Advances in OR, RMIT, Melbourne, Australia, November 25, 2009)
- Robust decision-making in the face of severe uncertainty(PDF File)
(GRIPS, Tokyo, Japan, October 16, 2009)
- Decision-making in the face of severe uncertainty(PDF File)
(KORDS'09 Conference, Vilnius, Lithuania, September 30 -- OCtober 3, 2009)
- Modeling robustness against severe uncertainty (PDF File)
(SOR'09 Conference, Nova Gorica, Slovenia, September 23-25, 2009)
- How do you recognize a Voodoo decision theory?(PDF File)
(School of Mathematical and Geospatial Sciences, RMIT, June 26, 2009).
- Black Swans, Modern Nostradamuses, Voodoo Decision Theories, Info-Gaps, and the Science of Decision-Making in the Face of Severe Uncertainty (PDF File)
(Department of Econometrics and Business Statistics, Monash University, May 8, 2009).
- The Rise and Rise of Voodoo Decision Theory.
ASOR Recent Advances, Deakin University, November 26, 2008. This presentation was based on the pages on my website (voodoo.moshe-online.com).
- Responsible Decision-Making in the face of Severe Uncertainty (PDF File)
(Singapore Management University, Singapore, September 29, 2008)
- A Critique of Info-Gap's Robustness Model (PDF File)
(ESREL/SRA 2008 Conference, Valencia, Spain, September 22-25, 2008)
- Robust Decision-Making in the Face of Severe Uncertainty (PDF File)
(Technion, Haifa, Israel, September 15, 2008)
- The Art and Science of Robust Decision-Making (PDF File)
(AIRO 2008 Conference, Ischia, Italy, September 8-11, 2008 )
- The Fundamental Flaws in Info-Gap Decision Theory (PDF File)
(CSIRO, Canberra, July 9, 2008 )
- Responsible Decision-Making in the Face of Severe Uncertainty (PDF File)
(OR Conference, ADFA, Canberra, July 7-8, 2008 )
- Responsible Decision-Making in the Face of Severe Uncertainty (PDF File)
(University of Sydney Seminar, May 16, 2008 )
- Decision-Making Under Severe Uncertainty: An Australian, Operational Research Perspective (PDF File)
(ASOR National Conference, Melbourne, December 3-5, 2007 )
- A Critique of Info-Gap (PDF File)
(SRA 2007 Conference, Hobart, August 20, 2007)
- What exactly is wrong with Info-Gap? A Decision Theoretic Perspective (PDF File)
(MS Colloquium, University of Melbourne, August 1, 2007)
- A Formal Look at Info-Gap Theory (PDF File)
(ORSUM Seminar , University of Melbourne, May 21, 2007)
- The Art and Science of Decision-Making Under Severe Uncertainty (PDF File)
(ACERA seminar, University of Melbourne, May 4, 2007)
- What exactly is Info-Gap? An OR perspective. (PDF File)
ASOR Recent Advances in Operations Research mini-conference (December 1, 2006, Melbourne, Australia).
