What Makes Some Evidence Stronger? | Lesson 2.6, Free Course

Four ways evidence gains or loses weight.

The idea

Two things moving together isn't proof that one causes the other. In a well-known example, ice-cream and sunscreen sales rise and fall together across a year, because the season drives both. Sometimes the cause could run either way. Many studies linked minimum wages with lower employment. But it could also run backwards. When jobs are scarce, employers can set lower wages, and that may lead to calls for a higher minimum. And search enough pairs of numbers, and some will match purely by chance. Still, moving together is often a good clue. The mistake is jumping to the cause too fast.
A fair test lets chance decide who gets what. Say you want to know if a study app raises test scores. Chance puts some students on it and some not, then the groups are compared. Letting chance pick the groups balances their hidden differences, on average. 'On average' means the groups end up alike, not identical. Small differences of the kind chance produces aren't a sign the test was unfair. Just watching students who chose the app is weaker. Keen students might pick it, and also study more.
Watching what people already do still has a job. In medicine, it's better at spotting side effects that are rare or show up late. It's also more likely to show what a treatment achieves in everyday care. Its weak spot is judging cause and effect, because the groups may differ in other ways. Picture comparing people who already cycle to work with people who drive. Any difference might come from cycling, or from who chooses to cycle.
A small study with a dramatic result deserves more caution, not less. In a small study, even a statistically clear result is less likely to reflect a real effect. And one study rarely settles anything, either way. A failed repeat doesn't sink a finding for good. A successful repeat doesn't make it certain either. When a hundred psychology studies were repeated, nearly all the originals had a statistically clear result. About a third of the repeats did. People still disagree about what that means.

Want the research, names and dates? They are in Go deeper, at the end.

A rumour, a witness and a recording walk in.

Picture it

Think of a court case. A rumour, one witness and a recording all count as evidence. Nobody weighs them the same, though. The judge asks how each was gathered, and whether anything else could explain it. A recording can still be edited, and a rumour can turn out true. The weight depends on how each came to be.

For a child: Every time Mia wears her red hat, it rains. She decides the hat brings the rain. Her brother points out that she only wears it when the sky already looks grey. On a sunny day, the hat stays on its hook. The grey sky explains both. A third thing can sit behind two things that happen together.

Four questions between a headline and a belief.

How it applies

Weighing evidence comes down to four questions. How was it gathered: did chance decide who got what, or were people simply watched? What else could explain it: a third thing behind both, or a cause running the other way? How big, and how many: how many people, and how large the effect? And has it been repeated? A finding that turns up again, studied in different ways, can carry more weight. Say a town's new speed bumps arrived the year accidents fell. A new traffic light on the same road could explain the drop just as well.

Four questions · Drawing: Hacks Vitae

Lucky socks, a new manager, a tiny study.

Where you’ll see it

At home: a cousin swears his lucky socks win matches. He remembers the wins in those socks, and forgets the losses. Nobody has kept score.
At work: sales rose the month a new manager started, and the manager got the credit. It was also the start of the holiday season.
Online: a "study finds" headline, where the study turns out to be a handful of people and was never repeated. The headline is far more confident than the study.

Signs it’s happening

One striking story is doing the work of a whole study.
Two things rose together, and the cause was simply assumed.
A single result gets quoted as if it settled the matter.

Same headline, a cause or a question.

Before and after

A headline says children who play chess get better grades.

The quick answer

"Right, chess makes kids smarter." The link becomes a cause in one step, and the next headline repeats it.

The second look

"Did chance decide who played, or did keen kids pick chess? What else could explain both?" A school with a chess club might also have smaller classes. A link in a headline is a question, not yet a cause.

A pump handle, and the count that complicated it.

A story from history: John Snow and the Broad Street pump

In 1854, near Broad Street in London, more than 500 people died of cholera in ten days.
The doctor John Snow found that, apart from people who drank from one pump, cholera had not particularly risen there.
On the evening of 7 September he put this to the parish's local board, the Board of Guardians. The pump's handle was removed the next day.
Yet Snow's own count shows the outbreak was already fading. The most attacks in one day came on 1 September, at 143. On 8 September there were 12.
Snow himself wrote that the attacks had fallen so far before the water was stopped. That made it impossible to decide if the pump's well was still dangerous. He added that the toll would have been much greater if so many people had not fled.

Snow weighed his own evidence, including the part that made the story less neat. The pump evidence pointed to the cause, but couldn't show the handle ended it. Evidence can point to a cause without showing what ended an outbreak.

Quick check3 questions · nothing is saved

1. What explains both Mia's red hat and the rain?

2. What does letting chance pick the groups do?

3. What did Snow's own count show about the pump handle?

One question worth keeping.

In one line

How a finding was gathered shapes how much it can carry, and one striking result rarely settles a question.

Which belief of yours rests on one striking story rather than a fair test?

Go deeperThe research, names and dates

Where this comes from

The Australian Bureau of Statistics defines causation as one event being the result of the other. Its example of things moving together without it: ice-cream and sunscreen sales rise and fall with the season. The Internet Encyclopedia of Philosophy notes that correlations are often good evidence of a causal link. The mistake is leaping to the cause hastily.
In a randomised trial, treatments are selected by a random process, such as a random-numbers table. Cochrane's handbook explains why. If randomisation works, it stops known or unknown factors from steering who ends up in which group. Confounding, by contrast, is when something else affects both who ends up in which group and how things turn out.
In an observational study, people may receive treatments, but the investigator does not assign them. The STROBE authors write that such studies are more suitable to detect rare or late adverse effects. They are also more likely to show what is achieved in daily medical care. Cochrane's handbook says their potential for bias, meaning skewed results, is likely greater when judging effects.
A 2013 review by Button and colleagues warned that "low power also reduces the likelihood that a statistically significant result reflects a true effect". In 2015 the Open Science Collaboration repeated 100 psychology studies. Nearly all the originals had statistically significant results; of the repeats, 36% did. What that means is disputed. The US National Academies caution that a single failed replication does not conclusively refute the original claims. In the same sentence, they add that a successful one does not guarantee the original was right.

Randomised trial vs observational study

Compared	Randomised trial	Observational study
Who decides who gets what	A random process	Not the investigator
Main strength	Chance balances known and unknown factors, on average	Better suited to spotting rare or late side effects
Main weak spot	Can't answer every important question	Confounding: other factors often shape who gets what

A meta-analysis can pool either kind. But a 2016 paper argued that the two kinds of pool are not equal. Well-conducted trials with a low risk of skewed results differ from higher-risk observational studies.

More examples

If a headline says 'Dog owners are happier', a question is 'Did dogs cause it, or did happy people choose dogs?'
If an advert says 'Our app works', a question is 'Tested on how many people, and repeated?'
If a report says 'Sales doubled after the new logo', a question is 'What else changed at the same time?'

What if…?

What made Snow's case stronger?: His records also covered two water companies. Snow counted deaths in the opening weeks of the 1854 epidemic. He recorded 71 per 10,000 houses supplied with the Southwark and Vauxhall Company's grossly polluted water. That was 14 times the rate for the rival Lambeth Company's sewage-free water, at 5 per 10,000. The statistician Austin Bradford Hill later used these figures to show what a strong link looks like.
Is a study of many studies always the strongest evidence?: It depends on what went in. Cochrane defines a meta-analysis as the statistical combination of results from two or more separate studies. Its handbook says meta-analyses have the potential to mislead seriously. That is particularly so if the studies' designs and flaws are not carefully considered. A 2016 paper by Murad and colleagues called the usual evidence pyramid intuitive and likely correct in many instances. But it argued that "Study design alone appears to be insufficient on its own as a surrogate for risk of bias", meaning how likely a study is to be skewed.
Is there a short checklist?: Carl Sagan's The Demon-Haunted World has one, in a chapter called "The Fine Art of Baloney Detection". Among its tools are getting independent confirmation of the "facts" wherever possible, and spinning more than one hypothesis. Another is using a measure or a number when there is one. Another is checking that every link in a chain of argument works, including the premise. A last one is asking whether the claim could be shown false, at least in principle.

Words you may meet

randomised controlled trial: A study that splits people into groups by chance, so the groups start out alike and what each group receives can be compared fairly. Glossary →

meta-analysis: A study that pools the results of many earlier studies on the same question, so their findings can be weighed together. Glossary →

replication: Repeating an earlier study to check whether its results hold. Glossary →