Inclusion criteria

The following criteria were all required to be met by participants.

Requirements of participation

  • Informed consent. Participants were given the opportunity to stop the experiment if they did not consent to participation. If they declined consent, the experiment ended immediately.
  • A desktop browser. Qualtrics allows the exclusion of users of mobile devices from participation. We disallowed mobile users because the small size of many mobile screens would make the task difficult. Mobile users were shown a message inviting them to participate later from a desktop device.

Requirements of data inclusion

  • First time doing experiment: Must have answered “No, this is my FIRST TIME doing this survey” to the question “Have you completed this survey previously?”
  • Following directions/engagement: Must have performed at least 1 null sample and 1 experimental sample. Participants eliminated by this criterion started the instructions, but did not perform the experimental task.
  • Field is arguably scientific: Must have offered an answer to the question “Is your work in a field that would typically be considered scientific?”, and that answer must not have been “No”.
  • Education at University level: Must have offered an answer to the question “What is the highest level of formal education you have achieved in your scientific field?”, and that answer must not have been “I have no formal scientific education at the University level”.

Deleted/edited records

After data collection was complete, we were made aware of the possibility that our data set included minors. Because our experiment was targeted at scientists, we did not include indication of parental consent for minors in our consent form. Although there is nothing ethically problematic about our experiment, out of an abundance of caution and in consultation with the Cardiff University School of Psychology Board of Ethics we deleted 32 records that may have been minors. We used the following criteria to select these records:

  • The participant indicated no university-level education, and
  • In their reply to the question about their scientific field, their response included any of the following terms: “ap” (as in advanced placement), “student”, “school”, “high”, or “class”.

These records were deleted from Qualtrics. Because none of the deleted records indicated University-level education, this did not affect the data that met the criteria for analysis.

The record deletion was done on 2 May, 2019.

An additional 15 records were deleted on 31 May, 2019 when it was found that some students had indicated their student status in other fields.

One record was further edited to remove personal information on 31 May, 2019. This record (id: R_2a8OIVnAjcZXHyU) now includes a note indicating the redaction.

Demographic and informational questions

This section outlines the responses to the informational questions in the Christmas 2018 statistical cognition experiment. Only the “valid” participants are included below.

Field is scientific

Question: “Is your work in a field that would typically be considered scientific?”

  • Yes
  • No
  • Depends/Maybe/Other [specify]

Education and training

Question: “What is the highest level of formal education you have achieved in your scientific field?”

  • I have no formal scientific education at the University level
  • I have some education at the Bachelor level (or equivalent)
  • I have completed a Bachelor’s degree (or equivalent)
  • I have some education above the Bachelor level (e.g., Master’s, PhD, or equivalent)
  • I have completed a PhD (or equivalent)

Question: “How many years of formal statistical training do you have?”

[enter a number]

#>    Min. 1st Qu.  Median    Mean 3rd Qu.    Max.    NA's 
#>   0.000   1.000   2.000   3.168   4.000  26.000      32

Scientific field

Question: “In what applied field(s) do you use statistics? (check all that apply)”

  • Biological sciences
  • Medical sciences
  • Physical sciences
  • Social/Behavioral sciences
  • Computer science / technology
  • Other [specify]

Role of statistics

Question: “How do statistics play a role in your work? (check all that apply)”

  • Statistics do not play a role in my work
  • I use statistics in practice for the analysis of data
  • I comment on statistical practice, but do not develop them myself
  • I develop statistical methods
  • I comment on the philosophy of statistics
  • Other [specify]

Preferred statistical method

Question: “If you use statistics regularly in your work, what sort of inferential procedures would you typically prefer?”

  • Classical, frequentist, or error statistical
  • Bayesian (of any sort)
  • None; I prefer only descriptive
  • I do not use statistics in my work
  • Other [specify]

Significance testing

Question: “What is your opinion about statistical significance testing? (check all that apply)”

  • I do not have a strong opinion for or against significance testing
  • I think significance testing is necessary for science
  • I think significance testing is fine, but prefer other approaches to statistical inference
  • I think significance testing is fine, but people misunderstand/misuse it
  • I think significance testing should be discontinued or performed very rarely
  • I think the logic of significance testing is fatally flawed
  • I do not understand the question
  • Other [specify]

Data downloads

During debriefing, participants were given the option of downloading their data for their own analysis. The table below shows how many took advantage of the option for both shuffle reports and experimental samples.

  Experiments
Shuffle reports Did not download Downloaded
Did not download 317 36
Downloaded 13 140

Compiled 2020-11-04 16:46:34 (Europe/London) under R version 4.0.3 (2020-10-10).