Keep this life skill for later
Sign in to save
← All life skills

Learning & thinking · a life skill

Advanced Source Checking

weigh the evidence

Finding the source is half of it. The other half is knowing what the source can bear: a story, a survey and a trial can all be honest and still prove very different things. For thirty days, take one claim a day, write down how sure you are, and find out how much the evidence actually holds.

  • Challenging to start
  • Life-Changing impact
  • Learn it in 30 days
  • One claim a day

Your thirty days

  1. 1
  2. 2
  3. 3
  4. 4
  5. 5
  6. 6
  7. 7
  8. 8
  9. 9
  10. 10
  11. 11
  12. 12
  13. 13
  14. 14
  15. 15
  16. 16
  17. 17
  18. 18
  19. 19
  20. 20
  21. 21
  22. 22
  23. 23
  24. 24
  25. 25
  26. 26
  27. 27
  28. 28
  29. 29
  30. 30
  • Log
  • Ladder
  • Read
  • Look further
  • Score the week
one claim on the evidence ladder
The evidence ladder, with one day's claim placed on it A ladder of five rungs, weakest at the bottom. From the bottom: a story or a post, which shows that something happened once; an expert's opinion, what one person thinks; a study that watched people, which shows that two things go together; a trial that assigned them, which shows that one causes the other, in that group; and many trials reviewed together, which show how big the effect is and how sure we can be. Below the ladder, Monday's log entry: supplement A sharpens memory, ninety percent sure. An arrow places it on the fourth rung, one trial. A dashed arrow points up to the top rung: look for a review next. A card reads the study: forty adults, against a dummy pill, “doubles recall”, which is relative, and paid for by the maker. Beside it, the Sunday score: said eighty or ninety percent on four claims, one held up; said sixty or less on three, all three held. Mon · the log “Supplement A sharpens memory” 90% sure, before checking A [look for a review next] read the study who? 40 adults against? a dummy pill size? “doubles recall” [double of what?] paid by? the maker one trial: a start, not the answer Sunday · the score said 80–90% sure 1 of 4 held up said 60% or less 3 of 3 held up [too sure, too often]

The higher the rung, the more questions the evidence can answer. A lower rung isn't worthless; it just can't carry a claim about cause.

A week of the log

Made up, to show the shape. One line a day, and the “how sure” column filled in before you check.

DayClaim · evidenceSureHeld?
Mon Supplement A sharpens memoryone trial, 40 people 90% yes
Tue Crime in City B doubleda chart in a post 80% no
Wed Habit C adds years of lifea study that watched people 90% no
Thu Drug D lowers blood pressuremany trials, reviewed 60% yes
Fri Method E raises test scoresone trial, 300 pupils 60% yes
Sat Food F causes headachesa friend's story 90% no
Sun Policy G cut traffic deathsa study that watched cities 50% yes

The Sunday score: sure on 4 claims, and 1 held up. Less sure on 3, and all 3 held. That gap is the thing to work on: next week, ask what rung a claim sits on before deciding how sure to feel.

The thirty days

A week for each move.

Keep the log going all month. Each week adds one move on top of the last, and Sundays are for the score.

  1. Week 1

    Log one claim a day

    Write the exact words, where you saw them, and how sure you are, as a percentage, before you check anything. Pick claims you'd act on.

  2. Week 2

    Put it on the ladder

    What kind of evidence stands behind it: a story, an expert's view, a study that watched people, a trial that assigned them, or many trials reviewed? Only trials can show cause.

  3. Week 3

    Read the study itself

    Who, and how many. Compared with what. Is the effect relative (“doubles the risk”) or absolute (“1 in 7,000 to 2”)? Who paid for it.

  4. Week 4

    Look for the rest of the evidence

    Search for a review or a replication, and for the strongest case against the claim from people who know the field. One study is a start.

  5. Every Sunday

    Score how sure you were

    Of the claims you were 90% sure of, how many held up? If it's far from nine in ten, let the rung, not the feeling, set how sure you are.

Why one study isn't enough

Run them again, and many shrink.

The original studies

97 of 100found a significant effect

The same studies, run again

36 of 100found it again

Open Science Collaboration, Science, 2015: 100 published psychology studies, repeated.

In 2015 a large team of psychologists repeated 100 studies from three leading journals, following the original methods as closely as they could. Nearly all the originals had reported a significant result. About a third of the repeats did, and on average the effects they found were about half the size.

That doesn't mean the originals were fraud, or that psychology is broken. It means one study is a first look. Small samples, flexible analyses and journals that prefer surprising results all push the first estimate up. So the question to ask of any single study is: has anyone found it again?

Relative and absolute

Twice the risk, of what?

The newer pills double the risk of blood clots.What women in the UK heard in October 1995, from a safety warning to doctors

Older pill 1 in 7,000

Each dot is a woman taking the pill. One had a clot.

Newer pill 2 in 7,000

“Doubled” is true. So is “one more in seven thousand”.

The warning was accurate, and it was given as a relative risk: 100% higher. Many women stopped taking the pill, and in the following year there were an estimated 13,000 more abortions in England and Wales. Given as an absolute risk, the same finding reads very differently. Whenever a number says “double”, “half” or “40% more”, ask: out of how many?

The idea

Why it works.

Checking a source tells you whether a claim was made honestly. Weighing the evidence tells you how much the claim can hold. The two come apart all the time: a careful study that watched people can be entirely honest and still say nothing about cause.

The log and the Sunday score do the rest. Writing down how sure you were, before you checked, is the only way to find out whether your sense of certainty means anything. Many people find theirs runs high, and that's worth knowing.

While you weigh

Habits that help.

  • Ask “compared with what?” An effect with no comparison group is a story with numbers in it.
  • Turn relative into absolute. “Doubles the risk” means little until you know the risk it doubles.
  • Look for the review before the study. Many trials, weighed together, beat the newest one.
  • Steelman the other side. Find the best case against the claim, from someone who knows the field.

Go deeper

Keep going.

Keep this going in the app.

HourLife turns these ideas into a daily system — goals, habits, focus, and reflection in one private place.

Download on the App Store Download on the Mac App Store See everything HourLife does