The Spin Stops Here!
Decision-Making Under Severe Uncertainty  
Faqs | Help | @ | Contact | home  
voodoostan info-gap decision theory info-gap economics severe uncertainty mighty maximin robust decisions responsible decisions

The Power of the (Peer-Reviewed) Word

(This is an advanced draft) Last modified:

For some reason, my presentation On The Power of the (Peer-Reviewed) Word, at the ASOR Recent Advances in Operations Research 2011 meeting, which I posted on this website, has attracted an unusually large number of visitors.

In view of this extraordinary response, I decided to launch a blog on this clearly intriguing topic.

However, as this may take some time to complete, for the time being, visitors will have to make do with this rather lean page. To download a copy of the presentation, simply click here, or on the picture on the first slide of the presentation that will display on the right hand side of the page. This is a 3MB PDF file.

I want to point out to all those visitors to my website, who did not attend my presentation, that the material covered in this presentation will feature as a chapter in my next book: Fooled by Robustness: a Perspective from the Land of the Black Swan.

To give my discussion an Australian flavor I based it on the following quote. Take note that the Federal Government in question is the Australian Government, also known as the Commonwealth Government (emphasis added):

" ... Environmental decision science is young. Prior to AEDA there was no centre in which researchers and practitioners from all the relevant disciplines worked closely together to formulate and solve the pressing environmental problems of the day. The Federal government took a calculated risk investing in a multidisciplinary centre that was very different from traditional ecological science. And what has been the return on that investment? Quite a lot if you consider our achievements (many of which have been presented in Decision Point, see the next page for just a few examples). It's important to note that all of these outputs appeared in the peer-reviewed literature (including some of the top journals like Science and Nature). We often forget that the CERF program is a research program, albeit applied research, and research must eventually be subject to peer review to be credible. ..."
Hugh Possingham
Decision Point, 45 (2010), p. 2

With this as background, the issue that I address in this presentation is the treatment of Type II errors , namely flawed publications that passed muster in the peer-review process. By this I mean "bad apples" that somehow managed to penetrate the defenses set up by the peer-review process, to find their way, as refereed articles, into peer-reviewed journals. See my compilation of review reports on an array of articles and books.

That this happens every so often is a fact. The trouble is, however, that most journals lack adequate procedures to deal with such cases. Thus:

"... Peer review failures occur when a peer-reviewed article contains obvious fundamental errors that undermine at least one of its main conclusions. Many journals have no procedure to deal with peer review failures beyond publishing letters to the editor. ..."
Source: http://en.wikipedia.org/wiki/Peer_review

To make vivid the issue that I am raising here, let us consider a hypothetical case that is in fact based on a real-world situation. For the purposes of this discussion, it suffices to say that the journal in question is a reputable international journal in the area of risk analysis. The dates are fictitious.



Source:
http://mychinaconnection.com/

Case Study 1

Suppose that in March 2003 a peer-reviewed journal, call it XXXX YYYYYYY, accepted for publication an article proposing the use of a theory, call it TTT, to handle a large class of important risk analysis problems. Suppose next that in August 2005 the article was published in the journal. As a matter of fact, assume further that another 5 articles of this type (accepted at different dates) were published by the journal over a period of five years.

Now, suppose that in February 2006 it was established that TTT is a "bad apple" following which articles discussing and demonstrating the flaws in this theory were published in three peer-reviewed journals, refereed conference proceedings, research reports, and were posted on web pages.

If you are not involved in the Publish or Perish sector of the economy, you may not be aware that although cases such as this may not occur very frequently, they do occur from time to time.

The question therefore arising is this:

Whose responsibility is it to inform the readership of XXXX YYYYYYY that TTT is a "bad apple"?

Here are some plausible answers to this fundamental question:

  1. The author(s) of the article.

  2. The referee(s) who reviewed the article.

  3. The Associate Editor who handled the article.

  4. The Chief Editor of XXXX YYYYYYY.

  5. The Editorial Board of XXXX YYYYYYY.

  6. The person(s) who discovered the flaws in TTT.

  7. The United Nations Security Council (UNSC).

  8. No one.

  9. Other (please specify): _________________

I raise this question, and I propose a number of plausible answers to it, because, as indicated above, most journals lack suitable procedures to deal with "bad apples".

And to bring matters to a head, I want to point out that, this fact, is recognized for instance by Wikipedia which relies heavily on peer-reviewed publications for its content. Indeed, peer-reviewed articles are regarded by Wikipedia as reliable and verifiable sources.

Wikipedia's way of dealing with this issue is to invoke the "Majority Vote principle" ...

Wikipedia's reliance on the "Majority Vote principle", is embedded in its so called Neutral point of view (NPOV):

Neutrality requires that each article or other page in the mainspace fairly represents all significant viewpoints that have been published by reliable sources, in proportion to the prominence of each viewpoint. Giving due weight and avoiding giving undue weight means that articles should not give minority views as much of or as detailed a description as more widely held views. Generally, the views of tiny minorities should not be included at all. For example, the article on the Earth does not directly mention modern support for the Flat Earth concept, the view of a distinct minority; to do so would give "undue weight" to the Flat Earth belief.

In articles specifically about a minority viewpoint, such views may receive more attention and space. However, these pages should still make appropriate reference to the majority viewpoint wherever relevant and must not represent content strictly from the perspective of the minority view. Specifically, it should always be clear which parts of the text describe the minority view.

Source: http://en.wikipedia.org/wiki/Wikipedia:Neutral_point_of_view

So, as a result of Wikipedia's reliance on the "Majority Vote principle", the following situation may well occur. If theory TTT boasts 346 peer-reviewed articles as opposed to your say 2 articles showing conclusively (that is, proving formally and rigorously) that TTT is fundamentally flawed, then Wikipedia's policy may well hamper your ability to inform the readers that TTT is fundamentally flawed.

And to further illustrate the effect of this policy, imagine that in some classroom 34 kids decree that 3 + 8 is equal to 12, whereas one little kid insists that this sum is equal to 11 then, on the NPOV policy, much more weight would be given to 3 + 8 = 12 than to 3 + 8 = 11.

Happily, in some circles the "majority vote principle" is not taken at face value to serve as the decisive factor in matters of science. And so, although a person may prove to be a "lone voice" in a certain (scientific) discourse, debate, and so on, the factor deciding the merit of a case is the reasoning underpinning it. For:

Galileo Galilei (1564-1642)
In matters of science, the authority of thousands is not worth the humble reasoning of one single person.

To which I say: Viva Galileo!

The trouble is that the Editorial Boards of some peer-reviewed journals may not have among their ranks, even one single person whose humble reasoning can settle the issue. Worse, they may be reluctant to nominate such a person.

Case Study 2:

Suppose that

To complete the picture, suppose that the other two referees justify your criticism of the theory.

Got the picture?

The question is then:

What can authors do about referees whose review reports are riddled with flaws and whose accusations (such as the above) stem from serious technical misconceptions they have about the subject matter?

I don't know about you, but ... I consider posting my response to the referee's accusations on my website, hoping that (sooner or later) the referee will get the message.

What's next?

Until the envisioned new blog is ready, you may want to do one or more of the following:

  1. Nothing.

  2. Read my compilation of review reports. Most of these reports deal with articles that were published in peer-reviewed journals.

  3. Submit one of your rejected articles to a suitable journal that specializes in publishing rejects. For example, if your article has a mathematical bent, you may try Rejecta Mathematica. This journal accepts any paper that was rejected by a peer-reviewed journal provided that the article is mathematically oriented.

  4. Send your article(s) to the Journal of Universal Rejection. Their instructions for authors are as follows:

    Instructions for Authors

    The JofUR solicits any and all types of manuscript: poetry, prose, visual art, and research articles. You name it, we take it, and reject it. Your manuscript may be formatted however you wish. Frankly, we don't care.

    After submitting your work, the decision process varies. Often the Editor-in-Chief will reject your work out-of-hand, without even reading it! However, he might read it. Probably he'll skim. At other times your manuscript may be sent to anonymous referees. Unless they are the Editor-in-Chief's wife or graduate school buddies, it is unlikely that the referees will even understand what is going on. Rejection will follow as swiftly as a bird dropping from a great height after being struck by a stone. At other times, rejection may languish like your email buried in the Editor-in-Chief's inbox. But it will come, swift or slow, as surely as death. Rejection.

    Submissions should be emailed to editor@universalrejection.org (We have recently become aware that this email address is acting up sometimes, so as a backup you may also use our old address j.universal.rejection@gmail.com.). Small files only, please. Why not just send the first couple pages if it is long? If you are lucky, your eventual rejection letter will appear on the Journal's blog. Please let us know in your cover letter if you would not mind being identified, otherwise most identifying information will be redacted.

I highly recommend the latter to persons who have never received an official rejection letter from a peer-reviewed journal.

Whatever you do, remember this:

Give me 26 lead soldiers and I'll conquer the world

Benjamin Franklin
(1706 -- 1790) 		     The pen is mightier than the sword

Edward Bulwer-Lytton
(1803 -- 1873)

The 26 soldiers referred to in the quote from Franklin are the 26 letters of the English alphabet and the "lead" a printing/writing device. Translating it to our time: with a word processor and a printer you can conquer the world!

In short: the word can conquer the world.

Best wishes

Moshe

  Latest News  
   
  Supplementary Material  

Recent Articles, Working Papers, Notes

Also, see my complete list of articles
    Moshe's new book!
  • Sniedovich, M. (2012) Fooled by local robustness, Risk Analysis, in press.

  • Sniedovich, M. (2012) Black swans, new Nostradamuses, voodoo decision theories and the science of decision-making in the face of severe uncertainty, International Transactions in Operational Research, in press.

  • Sniedovich, M. (2011) A classic decision theoretic perspective on worst-case analysis, Applications of Mathematics, 56(5), 499-509.

  • Sniedovich, M. (2011) Dynamic programming: introductory concepts, in Wiley Encyclopedia of Operations Research and Management Science (EORMS), Wiley.

  • Caserta, M., Voss, S., Sniedovich, M. (2011) Applying the corridor method to a blocks relocation problem, OR Spectrum, 33(4), 815-929, 2011.

  • Sniedovich, M. (2011) Dynamic Programming: Foundations and Principles, Second Edition, Taylor & Francis.

  • Sniedovich, M. (2010) A bird's view of Info-Gap decision theory, Journal of Risk Finance, 11(3), 268-283.

  • Sniedovich M. (2009) Modeling of robustness against severe uncertainty, pp. 33- 42, Proceedings of the 10th International Symposium on Operational Research, SOR'09, Nova Gorica, Slovenia, September 23-25, 2009.

  • Sniedovich M. (2009) A Critique of Info-Gap Robustness Model. In: Martorell et al. (eds), Safety, Reliability and Risk Analysis: Theory, Methods and Applications, pp. 2071-2079, Taylor and Francis Group, London.
    • .
  • Sniedovich M. (2009) A Classical Decision Theoretic Perspective on Worst-Case Analysis, Working Paper No. MS-03-09, Department of Mathematics and Statistics, The University of Melbourne.(PDF File)

  • Caserta, M., Voss, S., Sniedovich, M. (2008) The corridor method - A general solution concept with application to the blocks relocation problem. In: A. Bruzzone, F. Longo, Y. Merkuriev, G. Mirabelli and M.A. Piera (eds.), 11th International Workshop on Harbour, Maritime and Multimodal Logistics Modeling and Simulation, DIPTEM, Genova, 89-94.

  • Sniedovich, M. (2008) FAQS about Info-Gap Decision Theory, Working Paper No. MS-12-08, Department of Mathematics and Statistics, The University of Melbourne, (PDF File)

  • Sniedovich, M. (2008) A Call for the Reassessment of the Use and Promotion of Info-Gap Decision Theory in Australia (PDF File)

  • Sniedovich, M. (2008) Info-Gap decision theory and the small applied world of environmental decision-making, Working Paper No. MS-11-08
    This is a response to comments made by Mark Burgman on my criticism of Info-Gap (PDF file )

  • Sniedovich, M. (2008) A call for the reassessment of Info-Gap decision theory, Decision Point, 24, 10.

  • Sniedovich, M. (2008) From Shakespeare to Wald: modeling wors-case analysis in the face of severe uncertainty, Decision Point, 22, 8-9.

  • Sniedovich, M. (2008) Wald's Maximin model: a treasure in disguise!, Journal of Risk Finance, 9(3), 287-291.

  • Sniedovich, M. (2008) Anatomy of a Misguided Maximin formulation of Info-Gap's Robustness Model (PDF File)
    In this paper I explain, again, the misconceptions that Info-Gap proponents seem to have regarding the relationship between Info-Gap's robustness model and Wald's Maximin model.

  • Sniedovich. M. (2008) The Mighty Maximin! (PDF File)
    This paper is dedicated to the modeling aspects of Maximin and robust optimization.

  • Sniedovich, M. (2007) The art and science of modeling decision-making under severe uncertainty, Decision Making in Manufacturing and Services, 1-2, 111-136. (PDF File) .

  • Sniedovich, M. (2007) Crystal-Clear Answers to Two FAQs about Info-Gap (PDF File)
    In this paper I examine the two fundamental flaws in Info-Gap decision theory, and the flawed attempts to shrug off my criticism of Info-Gap decision theory.

  • My reply (PDF File) to Ben-Haim's response to one of my papers. (April 22, 2007)

    This is an exciting development!

    • Ben-Haim's response confirms my assessment of Info-Gap. It is clear that Info-Gap is fundamentally flawed and therefore unsuitable for decision-making under severe uncertainty.

    • Ben-Haim is not familiar with the fundamental concept point estimate. He does not realize that a function can be a point estimate of another function.

      So when you read my papers make sure that you do not misinterpret the notion point estimate. The phrase "A is a point estimate of B" simply means that A is an element of the same topological space that B belongs to. Thus, if B is say a probability density function and A is a point estimate of B, then A is a probability density function belonging to the same (assumed) set (family) of probability density functions.

      Ben-Haim mistakenly assumes that a point estimate is a point in a Euclidean space and therefore a point estimate cannot be say a function. This is incredible!


  • A formal proof that Info-Gap is Wald's Maximin Principle in disguise. (December 31, 2006)
    This is a very short article entitled Eureka! Info-Gap is Worst Case (maximin) in Disguise! (PDF File)
    It shows that Info-Gap is not a new theory but rather a simple instance of Wald's famous Maximin Principle dating back to 1945, which in turn goes back to von Neumann's work on Maximin problems in the context of Game Theory (1928).

  • A proof that Info-Gap's uncertainty model is fundamentally flawed. (December 31, 2006)
    This is a very short article entitled The Fundamental Flaw in Info-Gap's Uncertainty Model (PDF File) .
    It shows that because Info-Gap deploys a single point estimate under severe uncertainty, there is no reason to believe that the solutions it generates are likely to be robust.

  • A math-free explanation of the flaw in Info-Gap. ( December 31, 2006)
    This is a very short article entitled The GAP in Info-Gap (PDF File) .
    It is a math-free version of the paper above. Read it if you are allergic to math.

  • A long essay entitled What's Wrong with Info-Gap? An Operations Research Perspective (PDF File) (December 31, 2006).
    This is a paper that I presented at the ASOR Recent Advances in Operations Research (PDF File) mini-conference (December 1, 2006, Melbourne, Australia).

Recent Lectures, Seminars, Presentations

If your organization is promoting Info-Gap, I suggest that you invite me for a seminar at your place. I promise to deliver a lively, informative, entertaining and convincing presentation explaining why it is not a good idea to use — let alone promote — Info-Gap as a decision-making tool.

Here is a list of relevant lectures/seminars on this topic that I gave in the last two years.

Disclaimer: This page, its contents and style, are the responsibility of the author (Moshe Sniedovich) and do not represent the views, policies or opinions of the organizations he is associated/affiliated with.


Last modified: