5

Bayesianism doesn't seem to discriminate against ad hoc hypotheses.

A simple example illustrates this.

Let's assume a person tosses a coin 20 straight times and it lands on heads. They, ad hoc, start postulating whether or not their mind somehow controlled that coin to land on heads. The probability of it landing on heads 20 straight times given the coin can be controlled by their mind to land on heads is literally 1. Most would look at the prior probability of minds being able to control coins being very low and conclude it probably wasn't controlled by their mind (or would at least assume it's rigged).

Note, however, the other scenario where they thought of the hypothesis before they tossed the coin. Even now, if they toss it straight heads 20 times, the probability of it landing on heads 20 straight times given their mind controlled it to land on heads is still 1. And the prior probability of minds controlling coins would still be the same.

According to Bayes theorem, those hypotheses would give you an equivalent probability of the mind-controlling hypothesis given the evidence, even though the former is clearly ad hoc.

This leads to the conclusion that either Bayesianism is wrong in not discriminating against ad hoc hypotheses, or that ad hoc hypotheses aren't really "bad" in explaining things. What are your thoughts?

Community
  • 1
  • 2
    There is no cogent "ad hocness" concept to discriminate against. Traditional accounts slide into appealing to the author's intentions in proposing the hypothesis, which an objective criterion surely should ignore, see Bamford, What is the Problem of Ad Hoc Hypotheses? "Once we have distilled the psychological element from ad hocness... supposedly ad hoc hypotheses can turn out to be true or important or interesting scientific hypotheses... or are better evaluated or eliminated by other criteria." – Conifold Oct 26 '22 at 05:01

4 Answers4

5

The difference between a prediction and an "ad hoc" (really post hoc) hypothesis is in the prior probability you assign to the hypothesis. If you came up with the hypothesis before you saw the data, then heuristically, the prior probability of that hypothesis may be fairly high, simply because it did occur to you as a likely explanation while you were in the "prior to seeing the data" state of mind.

However, if you came up with the hypothesis after you saw the data, then we can't use the same heuristic to estimate a high prior probability. It didn't occur to you while you were in the "prior to seeing the data" state of mind. So we don't have reason to think it had a high prior probability. Perhaps it had a very low prior probability.

In the above discussion, we are thinking of the "true" prior probabilities of each hypothesis as an unknown number, that we try to guess, on the basis of whether the hypothesis occurred to us or not, and when it occurred to us. If the hypothesis occurred to us before seeing any data, this suggests it might have a high prior probability. If the hypothesis only occurred to us after seeing all the data, this is much weaker evidence about the prior probability.

causative
  • 12,714
  • 1
  • 16
  • 50
5

A few things require clarification here:

  • We are really talking about post hoc rather than ad hoc hypothesis here, as other answers have already pointed out.
  • Hypothesis testing, as described, is really Frequentist rather than a Bayesian approach:
    • in Bayesian view we have probabilities of different hypotheses that are updated according to the results of the experiment
    • in Frequentist view we state the null hypothesis (and at least one alternative hypothesis), and then test whether the hypothesis is true or not
  • As already mentioned in the previous bullet, whether we use Bayesian or Frequentist approach, we must postulate not only the null hypothesis, but also an alternative hypothesis - otherwise the whole exercise is meaningless. In this case the alternative hypothesis is the coin is not controlled by one's mind. In Bayesian setting then we should give this hypothesis a very small probability, p, whereas the null hypothesis has less-than-one probability 1-p. Applying Bayesian update would significantly reduce the probability that the coin cannot be controlled by our mind, but it would never be truly zero. If however we took this probability to be zero, then no Bayesian update would cure this situation.
  • In Frequentist approach one cannot prove the null hypothesis, but only disprove it. Thus, if we postulated as null hypothesis that the coin is controlled by one's mind, and it landed 20 times in its head, it does not prove that we can control it - we merely failed to disprove it. (Note that we are talking here about a one in a million event. Note also, that even hypothesis rejection is never definitive - rejecting hypothesis merely says that the likelihood/probability of it being true is less than some specified statistical threshold.)
  • Formulating hypothesis - a hypothesis is not taken out of blue, but based on our empirical/pre-existing knowledge (or our prior information - in Bayesian language.) Coin can be controlled by one's mind is one possible hypothesis, but coin is not fair (it has different probability to land on head than on tail - or perhaps it has two heads?) is another hypothesis, which seems more plausible - that is more consistent with our pre-existing knowledge.
Roger V.
  • 1,565
  • 1
  • 20
  • Where does having a null hypothesis come in with Bayesian inference? I have not needed a null hypothesis in the Bayesian procedures I've used so far. See Statistical Rethinking for what I have been digesting lately. – Galen Oct 26 '22 at 16:54
  • @Galen as I explained in my answer, null hypothesis is a not a part of Bayesian approach, but of the frequentist one. Perhaps, you use "Bayesian" in another sense? - I am talking about the distinction made in statistics here. – Roger V. Oct 26 '22 at 19:28
  • I don't mean "Bayesian" in any other sense than what I have learned from course material in statistics (e.g. Statistical Rethinking videos and textbook). It is your phrasing "[...], whether we use Bayesian or Frequentist approach, we must postulate not only the null hypothesis, [...]" that prompted my question. It sounds like you are saying we postulate a null hypothesis regardless or whether we take a Bayesian or Frequentist approach. Hence my question about where null hypothesis come into Bayesian inference, since in no Bayesian procedure that I've seen have I needed such a postulate. – Galen Oct 26 '22 at 19:36
  • Since you agree that Bayesian inference doesn't require a null hypothesis, I think it is just that I find this phrasing difficult to understand. – Galen Oct 26 '22 at 19:38
  • @Galen in Bayesian approach one postulates several hypotheses on equal footing, so one usually does not call any of them null. However, at least two different hypotheses should be present from the beginning. – Roger V. Oct 26 '22 at 19:46
  • Yes, we never call any of them null in Bayesian inference (to my knowledge). Hence why your phrasing "[...], whether we use Bayesian or Frequentist approach, we must postulate not only the null hypothesis, [...]" confused me. – Galen Oct 26 '22 at 19:48
2

The problem of ad hoc hypothesis is not that they are made after an observation, but that they are made to narrowly address this specific observation, without regard for global consistency.

In your exemple, "maybe I can control the coin with my mind" is not really what we call an ad hoc hypothesis, or at least it's difficult to judge without context. In the scope of our Bayesian analysis, the hypothesis that we can control the coin takes into account the totality of our data, and therefore it can't be considered ad hoc.

But let's say we repeat the experience and this time we get something more expected, like 8 tails 12 heads. Why is the mind control not working anymore? At that time we learn there was a thunderstorm in a distant place during our second experiment, and emit the hypothesis that "its magnetism must have interfered with our mental vibrations". This is an explanation that covers only the last set of data points, we should also do our due diligence and check if there wasn't a storm somewhere the first time we tried. Otherwise, that is ad hoc.

In order to address only the subset of data that went against our initial mental control hypothesis, instead of revising it as would be reasonable, we added a whole lot of assumptions that make it even more brittle and subject to attacks.

I am not familiar enough with bayesian probabilities to give you the proper formula, but I think factoring in the "thunderstorm interferes with mental vibrations" hypothesis could be modeled in Bayesian form and it would lower your overall probability, because you'd have to factor in all the new assumptions.

Actually, justifying the probability of the storm nullifying our control would probably require us to clarify how we think it works, and make it painfully obvious that we don't know what we are talking about.

(It seems you are also assuming your control works 100% of the time by giving it a probability of 1. This is far fetched, no control is a 100% sure. Factoring in the uncertainty if your control power would automatically make the whole hypothesis less probable)

armand
  • 6,280
  • 1
  • 13
  • 36
  • 1
    I can potentially see your logic in terms of it narrowly explaining only one observation, but I don't see how bayesianism arrives at the same conclusion. We are not talking about adjustable assumptions in this case. That may be true for the case in which we are trying to modify a hypothesis after it's been proven wrong.

    But in my example, this doesn't happen. In my example, I simply invent a hypothesis that explains a particular observation. There is nothing being lowered with respect to a prior hypothesis, since that's the ONLY hypothesis.

     –  Oct 26 '22 at 02:18
  • @thinkingman that's my point: your exemple is not an ad hoc hypothesis based on the scope of your context, therefore Bayesian inference shouldn't discriminate against it. I think the confusion comes from the fact that you know it's actually ad hoc because you judge from your experience that coin toss are roughly 50/50. But this experience of yours is not factored in your bayesian modeling, so of course the hypothesis appears just as valid as any other. – armand Oct 26 '22 at 02:24
  • It's an ad hoc hypothesis in that the hypothesis was invented to incorporate the observation that 20 heads occurred. And given that in the example, the priors for mind controlling would be the same, I don't see how Bayesianism gives us any justification to give lower value to the ad hoc theory in this case –  Oct 26 '22 at 02:40
  • @thinkingman that's your misconception: since the hypothesis is consistent with all of your observations, it can't be considered ad hoc yet. It is not a fallacy to come up with an explanation after the fact as long as you don't cherry pick the facts. Also, I failed to mention it but I would never let someone go with a probability of 1 for wether their mind control device is working as intended. – armand Oct 26 '22 at 03:03
1

"According to Bayes theorem, those hypotheses would give you an equivalent probability of the mind-controlling hypothesis given the evidence, even though the former is clearly ad hoc."

This would only be true if the statistician were less than completely competent because there is a hidden "multiple hypothesis testing" issue here that isn't being corrected for.

If you observe a sequence of a million coin flips and wait until you get a run of 20 heads (which is mathematically equivalent to predicting the flip correctly - both are equiprobable Bernoulli trials), you will eventually get a "success" just by random chance. For the post-hoc (rather than ad-hoc) hypothesis, the analysis should include the possibility of their having been previous unsuccessful attempts.

Essentially for the post-hoc hypothesis you are calling success if you find a successful run of predictions, for the other hypothesis you require this run of predictions to be successful. A competent statistician will include that context in the analysis.

An example of this arises in the on-line debate on climate change. Climate skeptics often monitor recent trends in global mean surface temperatures and if they find a period where there is no significant warming trend, they post a blog article about it. But if you have a noisy signal with an underlying trend, you will get that occasionally (and it is also seen i climate model output) - see Easterling and Wehner (2009). The climate skeptics are making the same error, they are not taking the post-hoc nature of the test into account.

Oddly enough, the skeptics never post post-hoc articles about periods where the trend is higher than the rate of warming predicted by the models. I wonder why that is... ;o)

Dikran Marsupial
  • 2,118
  • 10
  • 15