2

Please note in the comments if the question is too long and should be rephrased more concise. I am happy to do so if so wished.

The story:

The name comes from a joke about a Texan who fires some gunshots at the side of a barn, then paints a target centered on the tightest cluster of hits and claims to be a sharpshooter. (0)

I. Sources of description of the fallacy

1. wikipedia.org

stated that

The Texas sharpshooter fallacy is an informal fallacy which is committed when differences in data are ignored, but similarities are overemphasized. From this reasoning, a false conclusion is inferred. (1)

and also states that

It is related to the clustering illusion, which is the tendency in human cognition to interpret patterns where none actually exist. (2)

and

The Texas sharpshooter fallacy often arises when a person has a large amount of data at their disposal, but only focuses on a small subset of that data. (3)

2. rationalwiki.org

describes it as

The Texas sharpshooter fallacy (or clustering fallacy) occurs when the same data is used both to construct and test a hypothesis (4)

and goes on to explain

The fallacy's name comes from a parable in which a Texan fires his gun at the side of a barn, paints a bullseye around the bullet hole, and claims to be a sharpshooter. Though the shot may have been totally random, he makes it appear as though he has performed a highly non-random act. In normal target practice, the bullseye defines a region of significance, and there's a low probability of hitting it by firing in a random direction. However, when the region of significance is determined after the event has occurred, any outcome at all can be made to appear spectacularly improbable. (5)

The Texas sharpshooter fallacy uses the same data to both construct and test a hypothesis. A hypothesis must be constructed before data is collected based on that hypothesis. If one data set is used to construct a hypothesis, then a new data set must be generated (ideally, in a different way, based on predictions made by the hypothesis) to test it. (6)

3. philosophyterms.com

says

A Texas sharpshooter fallacy occurs when someone draws conclusions based on only the consistent data – the data points that are similar to each other — ignoring data that may not support the conclusion. This does not allow the data to paint the full picture of what is really going on. (7)

This fallacy gets its name from a story in which a Texas shooter fired many bullet holes into the side of a barn. He then drew a target around a tight cluster of bullet holes and called himself a “sharpshooter.” He wasn’t necessarily aiming anywhere in particular, but several of his bullets seemed to find a similar position in the barn wall. Drawing a target around the area made it look like he aimed, and succeeded in hitting, that particular area. (8)

II. Analysis

It seems there are multiple fallacies mixed into one. Let's break this down.

However, first we need to state that multiple things are at play here:

  1. data - which is the input into our reasoning process)
  2. method - which is our reasoning process
  3. hypothesis - a model of the world we have, which can be the output of a method (constructing the hypothesis) or the input (testing the hypothesis). If we are testing a hypothesis. The hypothesis can be either be described as a model or as expected output. If it is described as a model and we want to test the hypothesis we need to see how well the data fits the "into" the model. If it is described as expected output, we either to see how similar the expected with the new, real data is.
  4. result - which is the output of our reasoning process

For example we might put data into the method and get a set of clusters as a result. Another example is that we want to see if two things are

1. Fallacy: Similarity Illusion

This fallacy is about seeing a connection between the data and the hypothesis, because we emphasize the similarities between the data (structure) and the hypothesis (structure) and disregard the discrepancies. In practice this means leaving out variables/features that do not support similarities.

The task for the method is to check if two things are similar.

Here our method is not at fault, but the way we are filtering our parts of the data and the hypothesis.

In https://www.clocktimizer.com/a-quick-guide-to-data-fallacies-and-how-to-avoid-them/ this is contrasted to "cherry picking" (selecting results that fit your claim and excluding those results which do not fit your claim), but I do not see the difference.

  • This fallacy is described by (1) and (3).
  • The (3) is missing that the ignoring is selectively done, but fair enough.
  • I think that (5) is aiming to explain this.
  • I think that (7) is aiming to explain this.
  • (8) mentions this when saying "Drawing a target around the area made it look like he aimed, and succeeded in hitting, that particular area."

enter image description here

Making the story work

The story (0) would work if we were to tell it differently:

The shooter shoots at a barn, all over the place (not at a single spot as suggested by some accounts). Then the shooter draws a circle around the place where there are many hits at the same place. Finally, the shooter patches up the other wholes that one cannot see them (the ignoring part).

2. Fallacy: Pattern / Clustering illusion

This fallacy is about seeing patterns in randomness. If one wants to be less general, one could say that a pattern is a set of clusters. Clustering is the task of grouping samples such that samples in the same group are similar to each other and samples in different groups are dissimilar to each other.

Here, we do not have hypothesis, we put the data into the method and get a clustering result.

The task for the method is to check if there are patterns (and which) in data.

The fallacy is with our method: The method returns clusters that actually do not exist. It does so by emphasizing similarities and dissimilarities, but disregard that these similarities and dissimilarities are actually not that great.

  • This fallacy is described by (2).
  • The (4) is mentioning "clustering fallacy" in parenthesis, but is talking about something else otherwise.
  • (8) mentions this when saying that the bullets "seemed to find a similar position"

enter image description here

Making the story work

Not quite sure how...

3. Fallacy: Using the data to construct and test a hypothesis

Here are actually two reasoning processes at play. First we use data to construct the hypothesis than we test the hypothesis with data. If the same data is used in both processes, we commit the fallacy.

  • The (4) and (6) are talking about this fallacy.

enter image description here

Making the story (semi) work

The story (0) would work if we were to tell it differently:

The shooter fixes the gun to a post and shoots the bullets at the same spot. Then he draws a circle around the holes. Finally, he shoots again (into the circle) and claims the gun always shoots into the circle.

4. Fallacy: Using the result as input

This is basically the same as begging the question, which, according to wikipedia, is

a type of circular reasoning: an argument that requires that the desired conclusion be true.

So, we are using the result as part of the premise, we could also say, as part of the reasoning method.

I am not sure what is the case here.

enter image description here

Somehow I first thought of this fallacy when I first heard the story (0), because the original story suggests to me that the point is that the shooter is claiming he aimed THERE and then "showed" that he did by drawing the circle.

I am not sure if Fallacy 1 and 2 are the same as this (I think not quite), but in all three cases it is a type of circular reasoning, it seems to me. However, according to the table at wikipedia, the Texas sharpshooter fallacy does not fall under "Question-begging fallacies" or is a "Circular reasoning" fallacy, but falls under "Questionable cause".

III. Questions

Basically my questions are

  1. what did I get right,
  2. what did I get wrong,
  3. what the Texas sharpshooting fallacy is,
  4. what it is not and
  5. what those fallacies are called that are not the Texas sharpshooting fallacy?
Community
  • 1
Make42
  • 181
  • 4
  • 2
    The Wikipedia quote mentions that this is an "informal fallacy", so I suspect there isn't much further to say about this. – Brian Z Jun 16 '20 at 12:46
  • 2
    Consider making the question very short and posting most of your text as an answer. The current format does not work well with how this site is structured, your question is too broad and confusing. – tkruse Jun 16 '20 at 16:03
  • @tkruse, I disagree, I think the question needs this much detail to make the questioner's uncertainties clear, and to post the body of the content as an answer would be inappropriate given the degrees of variation in their understanding. – Paul Ross Jun 16 '20 at 17:22
  • The distinction you make between "method", "hypothesis", and "result" isn't clear to me--could you give some concrete examples? On the other thread I gave an example of the sharpshooter fallacy where there was a test group that ate a certain food and a control group that did not, then one just looks at a large number of health-related variables until one is found (grip strength in the example) that's different enough that it would be statistically significant if one had been conducting the test specifically to test for it from the start. – Hypnosifl Jun 16 '20 at 22:40
  • (cont.) In this case, would the "result" be the calculation of how statistically significant the association between the food and increased grip strength was, one which ignores the fact that if you look at enough variables you're likely to get some correlations by chance? Would you also call this a "similarity measure"? And is the "hypothesis" the one that eating the food increases grip strength, which was only formed after looking at the data? If not, what would those words denote in this example? And if that's right, what would the "method" be in this example? – Hypnosifl Jun 16 '20 at 22:42
  • @Hypnosifl: Your food-related example would be Fallacy 1. The result is a measurement how much correlation/similarity there is between a health-related variable (e.g. 1 (bad) to 10 (good)) and the food-eating (boolean 0 or 1 saying if the food was eaten) - both are data. The hypothesis is that the variable is improved by the food. The method calculates the correlation between the variable and the boolean and then the similarity between correlation and hypothesis. Is that answering your question? – Make42 Jun 17 '20 at 11:43
  • In your description of the method, how is calculating "the correlation between the variable and the boolean" diff. from calculating "the similarity between correlation and hypothesis", and how are either of these different from the "result"? Are you thinking of any specific calculations other than one where you calculate the improbability that we'd see the observed degree of correlation under a null hypothesis where there is no causal correlation between the food and the health variable? (i.e. the fallacious calculation if you didn't specify you were looking for this correlation in advance) – Hypnosifl Jun 17 '20 at 17:37
  • In general I'd say the Texas sharpshooter fallacy always involves an erroneous appeal to the unlikeliness of some data under a null hypothesis where the correlation/clustering is due to random luck, even if this is implicit--for ex., the actual sharpshooter example is implicitly saying that it's unlikely he would get so many shots clustered in the same "chosen" region under the null hypothesis that he's actually a bad shot whose bullet holes would ordinarily be more randomly distributed, and clustering in that region was just dumb luck (the fallacy is that the 'chosen' region is picked after) – Hypnosifl Jun 17 '20 at 18:05
  • @Hypnosifl: In the scenario of Fallacy 1, we want to check if our hypothesis is correct. So, in the example, the method has two steps, as described, where the "result" is the final result. Thus the "result" is the "the similarity between correlation and hypothesis". The general way to approach the second part of the "method" is using the null-hypothesis in the humanities - as it seems to me. In machine learning research and data science more close to math, this is not necessarily the case and I would strongly suggest to not use this approach. Multiple mathematicians have advocated against it. – Make42 Jun 19 '20 at 13:02
  • So, you could use the null-hyothesis approach (NHA), but I am not assuming it (or any other specific approach) in the framework. But, if you would like to use the NHA, the way you described it would be the standard way to go about it, yes. – Make42 Jun 19 '20 at 13:06
  • When I talked about using a null hypothesis I just meant that you're trying calculating the probability of the observed data under some alternate hypothesis where the apparent pattern was just due to luck (or at least involved a higher degree of luck), that's all I meant by "null hypothesis", not necessarily implying a formal "null hypothesis approach" as distinguished from say Bayesianism (which also involves calculating the probability of the observed data conditional on different background hypotheses). – Hypnosifl Jun 19 '20 at 13:12
  • Either way, I'm still not clear on "result" and "method"--are you saying "method" in this example would not refer to a specific calculation but rather a framework for making probabilistic calculations (like 'the null hypothesis approach', or 'bayesian inference') while the "result" would be the specific calculation of probability done within this framework? – Hypnosifl Jun 19 '20 at 13:14
  • @Hypnosifl: Reg. "null hypothesis": Got it :-). Reg. you last question: "Method" refers to the calculation, "result" is a (mathematical) object, possibly a number, which is not a method. A method always involves descriptions on what to do. However, a method can also be a framework (most methods are frameworks). Calling s.th. a framework means that some steps have been left vague. E.g. a method may contain a step "calculate an average", but does not specify if that must a me a mean or a median or something else. So, we need to plug into that framework how we want to implement that vague step. – Make42 Jun 19 '20 at 13:24
  • Your last post from yesterday says the Texas sharpshooter fallacy (TSF) is about "seeing a pattern which does not exist", so my Fallacy 2. In contrast, your example (in the way you described it) is Fallacy 1. This demonstrates the problem in understanding I have with the TSF - it seems to me that it mixes different fallacies into one - or at least people writing about it are. @PaulRoss and I discussed in the chat a different possibility of interpretation of the TSF. For that one would need to retell the story a bit to make it more precise though. Maybe he can draft up a result. – Make42 Jun 19 '20 at 13:24
  • I don't think I used the phrase "seeing a pattern which does not exist", by "pattern" do you mean just some observed correlation, or do you mean a claim of a causal explanation for such a correlation, or something broader? And with my clarification on the broader meaning of "null hypothesis", do you agree with what I said yesterday about all versions of the TSF involving 'an erroneous appeal to the unlikeliness of some data under a null hypothesis where the correlation/clustering is due to random luck'? – Hypnosifl Jun 19 '20 at 14:23
  • @Hypnosifl True, I meant to use the ", not for quoting you, but for keeping the verbal construct separated from the main text, similarly as I do for single words. I cannot format text slanted here. The "seeing a pattern which does not exist" is a rewrite of your (now I am quoting) "it's unlikely he would get so many shots clustered in the same "chosen" region under the null hypothesis that he's actually a bad shot whose bullet holes would ordinarily be more randomly distributed, and clustering in that region was just dumb luck". – Make42 Jun 22 '20 at 15:40
  • What a "pattern" means: Very good question! In this context I guess, I mean s.th. along the lines of "an observed correlation that is interpreted as having an underlying (possibly indirect) causal explanation". – Make42 Jun 22 '20 at 15:40
  • When you say "all versions of the TSF" - do you mean my Fallacy 1 to 4 (F1-4)? Your generalized null-hypothesis approach (GNHA) is the 'an erroneous appeal ... random luck'. You might be right, that in all F1-4 there is a component where one tries to answer if there is a pattern. E.g. in F2 one makes the fallacy of saying "yes" without checking anything - just assuming. Then one makes the erroneous appeal, even though for very different reasons. But, an erroneous appeal could be just an "honest mistake" that cannot be avoided ("bad luck") while F1-4 make avoidable methodological errors. – Make42 Jun 22 '20 at 15:42
  • Just to make it clear: I think you have very good point though, that might help is solving my question. Alternatively, one could also argue that F1-4 all have an invalid circular reasoning, which would be a very different "common denominator". Differently again, @PaulRoss argued that the F1-4 are not the TSF, but instead the TSF is the error of "building a model, saying the model has something to do with the data and just not checking at all" (this is me paraphrasing, not quoting). That would, again be something very different. – Make42 Jun 22 '20 at 15:44
  • I feel p-hacking is relevant, & the push for pre-registering hypothesees before trials begin https://en.wikipedia.org/wiki/Data_dredging – CriglCragl Nov 10 '21 at 13:05
  • I was looking at our discussion a year and a bit later and I realized I should respond to this comment: 'But, an erroneous appeal could be just an "honest mistake" that cannot be avoided ("bad luck")'. To be clear, in my earlier comments when I referred to the "erroneous appeal", the specific error was defining the "target" in retrospect, after you've already seen the data, and saying that there would be a low probability of the data clustering in the target under a null hypothesis where there's no causal explanation for the clustering, & saying therefore the causal explanation is favored. – Hypnosifl Nov 10 '21 at 14:57
  • Of course there could also be a different kind of "erroneous appeal" where you actually define the target beforehand, and then the data happens to cluster in that target just due to random chance, and you then erroneously appeal to a causal explanation (like the idea that the shooter actually has good aim) for the clustering. But that wouldn't be the Texas Sharpshooter Fallacy as I understand it, where the key error is defining the target (the special type of event you want to calculate the probability of under a null hypothesis) in retrospect, with knowledge of the data. – Hypnosifl Nov 10 '21 at 15:00
  • Very strongly related question: Isn't everything absurdly improbable? – Stef Oct 02 '23 at 12:08

2 Answers2

3

I think all four of your models seem to miss the mark for me. (haha) A consideration which I'd like to introduce here is that there isn't anything in the fallacy that hinges on whether or not the conclusions we draw are true or confirmed by the data; merely that those conclusions are methodologically suspect

Let's start by digging into the "Sharpshooter", who I'll call Ted for the sake of brevity.

Let's suppose that Ted wants to be recognized as an expert shot, so Ted makes a decision to aim at one specific area of the barn. And, in fact, Ted is a sharpshooter, and so Ted manages spectacularly well to intentionally bundle all of the shots within the same small plank of wood on the barn. After having done so, Ted walks up to the cluster of bullet holes and draws the target around it.

All of Ted's shots do indeed form a cluster that would confirm the hypothesis if it were put in place prior to Ted shooting. The clustering and similarity are not illusory (contra 1 and 2), and Ted shooting intentionally at a particular point does not invalidate the data gathering itself (contra 4) (that was the point of the exercise, after all). Ted has still done something shady by drawing the boundaries after the shots were fired - namely, had some shots fallen differently, the target would also have been drawn differently.

The closest thing to this is your Fallacy 3 - Ted's drawn target (the "model") around the bullet holes (the data) is being presented as the shots being on target (the data confirms the model), when it remains an open question as to whether Ted could now continue to hit the target (will subsequent data back up the model).

But even this doesn't quite get the heart of the matter, because your sketch suggests that Ted is feeding the same data into the model, finding that it fits, and arguing that the conclusions are confirmed. That's already a step further than the Texas Sharpshooter takes it; Ted's shady act is drawing the target.

Our statistical counterpart is the idea that we've picked our model in order to post-hoc capture as much of the positive similarities of our data as possible - that is, we have overfit our model to the specifics of our data. We're arguing that our data as we've captured it supports a particular conclusion because everything we've tested fits that model, but we haven't been actually testing that conclusion as a hypothesis - it's just our methodology to by design produce the picture that most closely fits the data we do have.

Had we set out to specifically test the hypothesis that we eventually came up with, we would probably find that our data isn't really even good confirmation for that hypothesis. But the fallacy is in presuming the validity of a conclusion chosen by design to match the data, rather than the assumption of the quality of the data in confirming the hypothesis.

So I can see why the general treatment of the fallacy is as a "Questionable Cause" inference. It's not so much circular reasoning so much as we're just arguing from insufficient data, but holding it to be sufficient because it maximally satisfies our data set.

Paul Ross
  • 5,441
  • 19
  • 39
  • 1
    Who is "they", are there multiple shooters involved? This is confusing. – Make42 Jun 16 '20 at 17:37
  • @Make42, using a gender neutral pronoun. – Paul Ross Jun 16 '20 at 17:38
  • 1
    Aha... well, English is not my mother's tongue. I understand your sentiment of not excluding someone in a text about general - let's say "doctors". But, you are telling a story, right? So can't you just pick a gender for the story (I am happy for it to be a female)? As I male I would still not feel left out. I think "Kim" is a name in English for a female which is short, right? I am not trying to be a offensive, just, I am reading the text already the 5th time, sorry. – Make42 Jun 16 '20 at 17:41
  • 2
    @Make42, the use of "they/their" as pronouns for a generic person is standard English language academic publication practice now, so it would probably be good to practice this type of reference resolution comprehension if you're planning to read and write philosophy. But basically, once Ted has been introduced, assume that "they/their" refers to Ted. You could always copy and paste it and do a "replace all" if you're struggling? – Paul Ross Jun 16 '20 at 17:56
  • @Make42 Actually, what I'll do is remove pronouns altogether. Might make it easier! – Paul Ross Jun 16 '20 at 18:00
  • That is in fact what I did - I copied it to a document and did the replace. Well, so far I am not a professional philosopher, although I am interested in philosophy. Thanks for the heads-up. What I am doing when given a speech or write a text or such, I tell one story with a female and another with a male, one story with doctor, one with an engineer and so on. That why everyone feels recognized at some point, I hope. I at least do not expect to be exactly matched in every story I read, and I am still able to relate. Does that make sense? – Make42 Jun 16 '20 at 18:03
  • Let us continue this discussion in chat. – Paul Ross Jun 16 '20 at 18:07
  • 1
    Ted the sharpshooter reminds me of many researchers in computer science: there is a problem P, and an existing algorithm A1 that solves problem P with a certain accuracy. Researcher invents an algorithm A2, compares its result with algorithm A1 on ten test datasets, then publishes a research paper called "New algorithm A2 performs better than A1" and mentions only one of the ten test datasets. – Stef Oct 02 '23 at 11:04
1

This question is all over the place, so I'm going to cut to the chase and explain the Texas Sharpshooter fallacy in simple terms, and then work from there.

The (not quite) Fallacy.

The Texas Sharpshooter fallacy is more of a caution about logic than a proper fallacy; that's typical of most informal fallacies. It rests on the understanding that there are two qualities involved in being a sharpshooter:

  • Consistency, or the ability to reproduce (replicate) a result multiple times. This is tantamount to the statistical concept of 'variability': low variability means high consistency.
  • Accuracy, or that ability to produce the result one aims at. This is roughly equivalent to the normative statistical concept of 'central tendency'.

Ideally, one expects a marksman to be both accurate and consistent: i.e., to aim at a given point (the central tendency) and produce strikes that are tightly clustered around that point. But it's perfectly possible for someone to be accurate and not consistent (to give a wide spread of strikes centered on the target point), or to be consistent and not accurate (to give a tight cluster of strikes centered on a point far away from the target point. Imagine a man who can consistently produce a tight grouping of strikes, but whose strikes are always clustered two feet to the left of the point he was aiming at. Imagine he then strolls over, moves his target point two feet to the left, and tries to claim the prize for marksmanship. That's the Texas Sharpshooter fallacy in a nutshell.

Now, this is not always the wrong thing to do. In fact, most of statistics and a lot of perfectly valid science uses this process to estimate means of populations by taking the means of sample distributions. The assumption in such cases is that the mean and variance of a sample — when the sample is selected by a properly random method — should serve as good approximations of the mean and variance of the population the sample is drawn from.

However, in the Texas Sharpshooter fallacy this assumption of properly random selection is thrown into question. The Sharpshooter is not using a sample of strikes to estimate an unknown population mean; instead, the joke is that he is changing a 'known' population mean — the actual target point he was aiming at — to match the sample he produced. As such, it's a kind of data fudging: like the (unfortunately common) practice of taking a sample of opinions from one political constituency and claiming that it represents the opinions of the population as a whole. And the difficult part of it, of course, is that unless the Sharpshooter specifies what he is aiming at in advance (or the political pollster specifies what group he is polling in advance) it's almost impossible to determine whether he is being accurate; all we know is that he's being consistent.

Statistical aside: I say 'almost' impossible because — technically speaking — strikes that are skewed significantly away from the mean will show an uneven distribution around this centroid. Variance will be elliptical or ovoid, not circular: greater in the direction of the mean than away from it. That can be detected with a large enough sample size. But does any one really want to know that?

The 'fallacy' here lies in an assumption about intention: is the person justifiably estimating the central tendency from the sample, or is the central tendency a pre-given norm that the person is skewing by using the sample CT instead? It's a sociological misdirection, not a logical (symbolic) error. I mean, there's a natural presumption that someone who can shoot consistently can also shoot accurately, but that natural presumption is not a logical necessity, and we leave ourselves open to misinformation if we don't take the possibility into account.

Ted Wrigley
  • 19,410
  • 2
  • 22
  • 55