Manipulability of comparative tests.

Abstract

Multiple self-proclaimed experts claim that they know the probabilities of future events. A tester does not know the odds of future events and she also does not know whether, among the multiple experts, there are some who do know the relevant probabilities. So the tester requires each expert to announce, before any data are observed, the probabilities of all future events. A test either rejects or does not reject each expert based on the observed data and the profile of the probabilities announced by the experts. We assume that the test controls for the type I error of rejecting the true probabilities. However, consider the case in which all experts are uninformed (i.e., they do not know anything about true probabilities). We show that they can still independently produce false forecasts that are likely to both pass the test, no matter how the data evolve in the future. Hence, the data may not suffice to effectively discredit uninformed, but strategic, experts.