1.1 Basic task setup

Participants were randomly assigned to one of two evidence powers (wide, \(q=3\); or narrow, \(q=7\)) with equal probability. Participants were also randomly assigned to one of eight “true” effect sizes. Because the behaviour of participants when there is no true effect was particularly of interest, the probability of assignment to no effect (\(\delta=0\)) was 25%. Across the seven other effect sizes listed in Table 1.1, the remaining 75% probability was evenly distributed.

The cover story (which can be read here) presented a problem in which it was desired to know which of two groups of elves (“Sparklies” or “Jinglies”) was faster. Participants were presented with the results of fictitious experiments as they requested them.

Participants could increase or decrease the sample size for the experiments as well; importantly, they were not aware of the actual sample size. Participants could adjust the sample size with a slider that had 20 divisions. The corresponding 20 hidden sample sizes are shown in Table 1.2.

The results were returned to the participants in a visual fashion: instead of being presented with a test statistic, each result was associated with a color intensity (from white to red) and a horizontal location that we will describe as left (-1) to right (1). Results on the far left were red and were associated with maximum evidence for the Sparklies being faster; results in the center were white were not evidence for either group; and results on the right were again red and were evidence for the Jinglies being faster. The intensity of the red color was defined by a linear gradient on the transparency (alpha) from 0 to -1 or 1 as defined using CSS.¹ The resulting horizintal axis is shown in the figure below.

Figure 1.1: The interface on which fictitious results were presented to the participants.

1.2 Underlying statistical details

Each fictitious result was the result of applying a transformation to randomly sampled \(Z\) statistic. The distribution of the \(Z\) statistic was a function of the randomly-assigned (but unknown to the participant) effect size \(\delta\) and a group sample size \(n\) (adjustable by, but unknown to, the participant).

\[ Z \sim \mbox{Normal}(\delta\sqrt{n/2}, 1) \]

The \(x\) location of the result, and hence the color, was then defined by the transformation: \[ x = \mbox{sgn}(Z)\left[1 - \left(1 - F_{\chi_1^2}\left(Z^2\right)\right)^{\frac{1}{q}}\right] \] where evidence power \(q\in\{3,7\}\), the location \(x \in (-1,1)\), and \(F_{\chi_1^2}\) is the cumulative distribution function of the \(\chi_1^2\) distribution.

The figure below shows the transformation from \(Z\) statistics (top axis) and one-sided \(p\) values (bottom axis) to \(x\) locations.

Figure 1.2: Transformation between traditional test statistics (\(Z\), \(p\)) and the \(x\) location

Figure 1.3: Evidence distributions as functions of underlying effect size, for non-negative effect sizes. Distributions from left to right correspond to increasingly large true effect sizes. Evidence distributions for negative effect sizes were mirror images of these distributions. A: Smallest sample size, wide evidence distribution; B: Largest sample size, wide evidence distribution; C: Smallest sample size, narrow evidence distribution; D: Largest sample size, narrow evidence distribution

Figure 1.4: Selected two-sided \(p\) values for the null distribution of the evidence for the narrow evidence distribution (\(q=7\))

Figure 1.5: Selected two-sided \(p\) values for the null distribution of the evidence for the wide evidence distribution (\(q=3\))

2.1 Video example

2.2 One participant’s final result

Figure 2.1: Selected participant’s random shuffle report samples.

Figure 2.2: Selected participant’s experimental samples.

This participant’s (id: R_0xLf49ng52wHp1n) final decision was “no_detect.”

3.1 Likelihood inference

Consider what would be necessary for a likelihood inference from these data. A likelihoodist would require a model with a parameter \(\theta\): \[ l(\theta; \mathbf x) \propto \prod_{i=1}^M f_{n_i}(x_i;\theta) \] where \(\mathbf x\) is the vector of length \(M\) of all evidence/\(x\)-locations of experimental samples produced by the participant, \(x_i\) represents the \(i\)th element of \(\mathbf x\), and \(f_{n_i}\) represents the density function of experiments for the hidden sample size \(n_i\). The density functions are unknown, as are the sample sizes. There is no obvious measure of effect size on which to build a model.

One might choose an impoverished model that throws out information about \(x\) and only uses the ordering of null samples and experimental samples: \[ \theta = Pr(X>X_0), \] where \(X\) is a draw from an experiment and \(X_0\) is a draw from the shuffle reports. But then \(\theta\) would be dependent on \(n_i\) in an unknown way except under the null hypothesis where \(\theta=0.5\) (because under the null hypothesis experiments and random shuffle reports have the same distribution, by definition). Thus, the null is the only thing the participant knows.

Even a clever participant who knew about the arbitrary transformation from the \(z\) statistic would have difficulty. They might use the following strategy:

1. Sample many random shuffle reports, estimate their distribution
2. Find a transformation \(z_i = g(x_i)\) to take the shuffle reports to standard normal deviates
3. Use this transformation to transform experimental samples to \(z\) statistics
4. Use likelihood inference on the mean of the \(z\) statistics from experimental samples

This strategy would require extensive knowledge of statistical theory and, likely, sophisticated programming skills used during the task. It is difficult to imagine anyone applying it (and certainly no participant reported doing such a strategy). But even this strategy is frustrated, because the transformation conflates the effect size and sample size. Only \(\delta\sqrt{n/2}\) is estimable, and \(n\) is unknown. No inference about \(\delta\) is possible, except that it is different from 0.

3.2 Bayesian inference

“For [statistical] hypotheses, Bayes’ theorem tells us this: Unless the observed facts are absolutely impossible on hypothesis \(H_0\), it is meaningless to ask how much those facts tend ‘in themselves’ to confirm or refute \(H_0\). Not only the mathematics, but also our innate common sense (if we think about it for a moment) tell us that we have not asked any definite, well-posed question until we specify the possible alternatives to \(H_0\).” – Jaynes (2003)

Bayesian inference, being dependent on the likelihood, is saddled with the difficulties outlined above, as well as an additional one: the Bayesian has need of a prior. As Jaynes points out, Bayesian inference requires specified alternative. Typically, this would be a set of alternatives rendered as a prior distribution over some parameter, \(p(\theta)\). In this problem, there is no clear parameter and hence it is not clear what the prior would be placed on. Additionally, the sample sizes are unknown, leading to the problem that it is unclear how a Bayesian would update their prior to the posterior; the sample sizes are clearly relevant information, but hidden from the analyst. The final difficulty is that the likelihood is unknown due to the arbitrary transform applied to the \(z\) statistic.

Suppose, however, that the Bayesian analyst knew about the transformation, and applied the strategy described in the previous section to reverse-engineer the transformation back to \(z\) scores. The inference about \(\delta\) would be dependent on their prior about the sample sizes. As emphasized by Bayesians who appeal to the Jeffreys-Lindley paradox (Lindley 1957), the statistical support for the null hypothesis \(\delta=0\) depends on the sample size such that assuming larger sample sizes guarantees evidence for the null hypothesis. The inference about \(\delta\) is completely confounded with \(n\): assuming larger \(n\) means inferring smaller \(\delta\).

It is possible that a Bayesian making just the right assumptions, and working very hard to reverse-engineer the transformation, could perform the task well. However, their inference would be dependent on these strong assumptions.

No participant reported using such a strategy (or anything like it).

6.1 Coding of strategy-related questions

After they reported their decision, participants were asked three open-ended questions related to their overall experimental strategy:

1. “What facts or observations were most salient in coming to the conclusion that you did (whatever that was)?”
2. “Please describe your experimental strategy, if any, in your own words.”
3. “Did you make use of the ‘random shuffle reports?’ If so, how?”

Authors RM and RH coded the responses as one or more of the following. Responses were judged holistically across the three questions; occasionally a response was unclear to one question, but their response to another question clarified their meaning.

Coding was performed without reference to the participant’s assigned condition and without reference to their decision (to the greatest extent possible; sometimes the text of their answer suggested the condition or decision).

1. Comparison to the random shuffle reports: Participant mentioned comparing the experimental results to the random shuffle reports
   • Example (to Q1): “A comparison of the experimental results when compared to the ‘random shuffle’ arrangements.”
   • Example (to Q2): “…[experiments were] much more extreme than would be predicted by chance given the sampling distribution from the random experiments”
2. Random shuffle reports to assess sampling variability: Participant mentioned using the random shuffle reports as a guide to the expected variability of the evidence/sampling distribution/noise/random error etc.
   • Example (to Q3): “Yup - smashed it to bits to see the variation in difference ‘scores.’”
   • Example (to Q2): “Get an idea of the variability (and expected difference, which seemed to be zero difference) using the ‘random shuffle’ results, and then see how the experimental results varied.”
3. Asymmetry of experiments: Participant noted that the observations tend to be on one side or another, or favoring one team or another; that there was a lack of asymmetry; or that symmetry was in some way considered by the participant.
   • Example (to Q2): “Conducted experiment with max participants 4 times; all observations well to the Jinglies’ side…”
   • Example (to Q2): “take samples see whether consistently left or right of middle…”
   • Example (to Q1): “all [experiments] came out 1 way”
   • Example (to Q1): “Approx same number of wins for each team ; no clear visual assymetry”
4. Increasing asymmetry as N increased: Participant mentioned noting a pattern where the extremeness of results appeared to increase as the sample size increased, a signature of a true effect, or noting that such information would have been of interest.
   • Example (to Q1): “…larger sample sizes tended to show clearer evidence of difference”
5. Response not relevant: Responses in this category did not answer the questions posed.
6. Response missing: No response was given to any of the three questions.
7. Explicitly denied using shuffle reports: In describing their strategy, the participant said that they did not use the shuffle reports or did not find them useful in their decision.

Categories 1, 2, 3, and 4 are consistent with significance testing logic. Categories 1 and 2 represent high-quality SST strategies, whereas Categories 3 and 4 may represent less efficient (e.g., ignoring variance of null distribution), unclear, or invalid SST strategies.

We must note that although many or most of the responses clearly fit into one category or another, sometimes categorization was not clear. We attempted to ensure the most reasonable coding for each participant through multiple rounds of coding and discussions. Although individual participant’s codings might be up for debate, in the aggregate they provide a guide to the strategies deployed by participants and a rough estimate of their frequencies.

6.2 Frequencies of strategies

The figure below shows the frequencies of various self-reported strategies as coded by the authors.

Figure 6.1: Coded frequencies of different strategies in the open-ended responses. A “Strong” indicates use of comparison to the null distribution or its use to assess sampling variability/distribution. A “Weak only” response indicates some indication of symmetry or asymmetry (or its use as a strategy).

7.1 Error rates

We can compare error rates of those who reported using a strong significance-testing strategy versus ones who did not.

Figure 7.1: Proportion who have a “difference” response as a function of effect size, for participants who were coded as reported 0, 1, or 2 strong significance testing strategies. Ribbons are standard errors.

7.2 Sampling behaviour

Did participants who reported more strong significance testing strategies use more null samples, on average? It appears so (left figure below; note the log scale).

Did participants who reported more strong significance testing strategies use more (or fewer) experimental samples, on average? There’s not much evidence either way, but importantly (see plot) those who reported fewer strong significance testing strategies didn’t also sample fewer experiments on average. There is nothing here to suggest they were less engaged on average.

Figure 7.2: Boxes, in order from left to right (red, green, blue) are 0, 1, and 2 strong significance testing strategies, respectively. Lines are robust regression fits. Stars show participants who indicated they did not use the random shuffle reports. Null samples (left) order from bottom to top (0, 1, 2); experimental samples (right) order is (3, 0, 1) from bottom to top.