Keep this life skill for later
Sign in to save
← All life skills

Learning & thinking · a life skill

Advanced Experiment Design

write the rule first, then look

Coin-flip days take care of chance. They don't take care of you. If you know which days are which, you'll rate them the way you expect to, and if you can look at the numbers as they come in, you'll stop when they say what you hoped. A protocol fixes both: a rule written before day one, blocks a helper shuffles, a blind where one is possible, and an end date you don't move.

  • Challenging to start
  • Life-Changing impact
  • Learn it in 30 days
  • A helper and a sealed rule
a sealed protocol, opened on day 24
A three-arm blinded self-test of the 3pm coffee, from the sealed protocol to the unblinded result The protocol, written before day one: does the 3pm coffee cost me sleep? Three arms: regular and decaf, made blind by a housemate from identical jars, and no 3pm cup, which can't be blind. Sleep rated 0 to 10 on waking. The rule: switch to decaf if regular averages at least 0.5 below decaf. Stop on day 24. Eight shuffled blocks of three days hold each arm once. The blind check: regular and decaf guessed right on 9 of 16 days, about what a coin would get. Made-up results: no cup averaged 7.5, decaf 6.5, regular 6.0. Of the 1.5 point gap, 0.5 came from the caffeine and 1.0 from having the cup at all. The rule is met: switch to decaf, and test the cup itself next. Protocol written Oct 31, before day one SEALED Question does the 3pm coffee cost me sleep? Arms regular · decaf (blind) · no cup (open) Measure sleep, 0 to 10, rated on waking Rule regular 0.5 or more below decaf: switch Stop day 24, no looking before Eight shuffled blocks, three days each D R N N D R R N D D N R R D N N R D D R N R N D Hatched days stay blind until day 24.Guessed regular or decaf right on 9 of 16: about a coin's worth. Each day stands alone: a 3pm cup is gone by the next night. Sleep on waking, unblinded no cup 7.5 decaf 6.5 regular 6.0 4 5 6 7 8 9 Where the gap comes from regular no cup caffeine 0.5 the cup itself 1.0 Rule met: regular ran 0.5 below decaf. Switch. The bigger gap is the cup itself. That's the next protocol. 1 2 3 4 5 6

Made-up numbers, to show the method. The no-cup arm can't be blind, and that's what makes it useful: it splits the gap into what the caffeine did and what having the cup did.

The system

Try it now.

  1. Write the protocol, rule included, before day one

    The question, the arms, the measure, the end date, and the result that would make you change something. Date it and send it to someone, so you can't quietly rewrite it.

  2. Randomise in shuffled blocks

    Each block holds every arm once, in a random order. A bad week then hits all the arms, not just one.

  3. Blind what you can, and keep an open no-change arm

    A helper prepares the look-alikes from a key you don't see. Beside them, run days with nothing at all: they show how much of the effect comes from expecting one. Guess each blind day, to check the blind held.

  4. Fix the end, and don't look before it

    Log the numbers where you can't see the running totals. Stopping early on a good stretch is the commonest way these tests mislead.

  5. Check for carryover

    If one day's arm can still be working the next day, lengthen each spell or add plain days between them, so each day measures only itself.

  6. Unblind, apply the rule, and decide

    Read the key, compare the arms, and do what the rule says, even when it's not what you hoped. Then write down the next question.

When to break the rule: never use this to stop, start or change a prescribed medicine on your own. Take the question to your doctor, who may be able to set up a test like this with you; the trial below was run by doctors, with patients who had already stopped. And if an arm turns out to be making you feel unwell, stop it. No result is worth that.

Your protocol

Write the protocol, check it's long enough, and know the red flags.

Fill in the protocol and press shuffle. Your copy hides which blind day is which; the helper's key is a separate copy. Then see whether your number of blocks can catch the effect you care about, before you spend a month finding out.

Part 1 The protocol

Protocol ·

Sealed
  • Question
  • Arms
  • Measure
  • Rule
  • Stop
  • Key

Your days

Shuffle the blocks to make the schedule.

Part 2 How many blocks?

Before you start, guess how big a difference you care about and how much the number wobbles from day to day. The page runs your protocol 200 times on pretend numbers with that difference built in, and shows how often it would clear your rule.

Part 3 Red flags, mid-test

Press any that's happened. Each one has a fix that keeps the test honest.

    Nothing you type is saved or sent anywhere, and it's gone when you leave. Copy your protocol and the helper's key before you close the page.

    The trial this page is built on

    When people who'd quit statins over side effects took statin, dummy and empty months blind, most of the symptoms came with the dummy pills too.

    The SAMSON trial, run at Imperial College London, is an experiment on one person, done sixty times. Each person had stopped taking statins because of side effects. Each got twelve bottles, one per month: four of atorvastatin, four of identical dummy tablets, and four empty. A computer decided which month got which bottle, and every day they rated their symptoms from 0 to 100 in a phone app.

    Forty-nine finished all twelve months. Symptoms averaged 8.0 in the months with no tablets, 15.4 on the dummy pills and 16.3 on the statin. About 90% of the symptoms people had on the statin also came on the dummy, and they were just as likely to stop a month's tablets early because of side effects. After seeing their own results, half went back on statins.

    The empty months are what make it work. Without them, statin and dummy would look the same, and you couldn't tell "neither does anything" from "both make me feel bad." With them, you can see that taking a tablet, any tablet, was most of it.

    The honest limits: a small trial, only of people who had already stopped statins over side effects, months back to back with no gap between, and symptoms rated by the same people who were living them. It doesn't show that no one gets side effects from statins. It shows how much of what people felt came from expecting them, and that a blinded test on one person can find that out.

    A year of twelve months assigned at random to statin, dummy or empty bottles, and the average symptom score in each One person's year, one bottle a month S E P P S E E S P E S P S statin · P dummy · E empty. The order is our example. Average symptoms, 0 to 100 Empty 8.0 Dummy 15.4 Statin 16.3 came with the dummy too
    Wood et al. (2020), the SAMSON trial, New England Journal of Medicine, as reported by the British Heart Foundation and the American College of Cardiology. Scores are the trial's monthly averages; the order of months in the strip is our example, since each person's was set by a computer.

    The idea

    Why it works.

    Each part of the protocol closes one way to fool yourself. The rule written first stops you deciding afterwards what counts as a result. Blocks stop a bad week landing on one arm. The blind stops your expectations rating the days, and the open arm measures how much those expectations were worth. The fixed end stops you quitting on a lucky run.

    None of it needs a lab. It needs a helper, two jars that look the same, and a willingness to be told no by your own numbers.

    When it goes wrong

    Habits that help.

    • Make the look-alike truly alike. Same cup, same taste, same time, same person handing it over. If you can tell them apart, the blind is gone; note it and treat those arms as open.
    • Guess every blind day. Write "A" or "B" before you unblind. If you're right far more than half the time, the blind didn't hold.
    • Let the helper keep the totals too. Hand over each day's number and don't keep a running average. What you can't see can't tempt you.
    • Write the next question at the end. A good protocol usually answers one thing and turns up another. The open arm in the coffee test is exactly that.

    Go deeper

    Keep going.

    Practise this in HourLife.

    In Experiments: Running experiments. Test one change for 7 to 30 days, rate it each evening, and record your own verdict when the run ends.

    Download on the App Store Download on the Mac App Store How Experiments works in HourLife