Keep this life skill for later
Sign in to save
← All life skills

Learning & thinking · a life skill

Deliberate Experiment Design

let a coin pick the days

A before-and-after test has a weak spot: the "after" weeks are different weeks. Work gets busier, the weather turns, you get a cold. Mixing the days fixes that. A coin decides which days get the change, so whatever else happens lands on both kinds of day, and after a few weeks you can see whether the gap is bigger than chance would make.

  • Moderate to start
  • High impact
  • Learn it in 2-4 weeks
  • Fourteen days, one coin
fourteen coin-flip days
Two weeks of coin-flip days testing a walk before work, compared in a dot plot and against every shuffle of the labels The change is a 20-minute walk before work, and the number is minutes of focused work before lunch. Fourteen days are split into seven pairs, and a coin picks which day of each pair gets the walk. Walk days scored 95, 110, 80, 120, 100, 90, 115, middle 100. Usual days scored 70, 85, 100, 65, 90, 75, 80, middle 80. The two lanes overlap between 80 and 100. Swapping the labels inside each pair every possible way gives 128 shuffles, and 24 of them make a gap of 20 minutes or more either way. The verdict: promising, run fourteen more days. Change a 20-minute walk before work Count focused minutes before lunch Day 1 – 70 2 W 95 3 W 110 4 – 85 5 W 80 6 – 100 7 – 65 8 W 120 9 W 100 10 – 90 11 – 75 12 W 90 13 W 115 14 – 80 One flip per pair: heads, the walk comes first. Two lanes, one number walk usual 60 80 100 120 middles: walk 100, usual 80 · shaded: where they overlap What chance alone does -20 -15 -10 -5 0 +5 +10 +15 +20 gap in middles, for each of the 128 ways to swap the labels 24 of 128 shuffles made a gap this big. Promising, not settled: run fourteen more days. 1 2 3 4 5

Walk days ran 20 minutes higher in the middle. But swapping which days were "walk" gave a gap that big about one time in five, so two weeks isn't enough to call it.

Two to four weeks

Try it now.

Something you can do or skip on any given day, and a number you can write down the same way each time. Set it up tonight; read it in two weeks.

  1. One change, one daily number

    A change you can switch on or off day by day, and a number you can write at the same time each day: minutes, pages, a 1 to 10 rating.

  2. Let a coin pick the days

    Split the days into pairs and flip once for each pair: heads, the change comes first. Each pair gets one of each, so a bad week hits both kinds.

    Write the whole schedule before day one. Deciding each morning lets your mood choose.

  3. Record the same way, and don't look back

    Same time, same scale, one line a day, plus a note on anything odd. Don't read the earlier days until the end.

  4. Compare the middles and the spread

    Put each kind of day on its own line. Where are the middles, and how much do the two lines overlap?

  5. Shuffle the labels, then decide

    Swap which day in each pair was the change, every possible way, and see how often chance makes a gap as big as yours. Then keep it, drop it, or run another two weeks.

A walk before work, two weeks. The coin picked seven walk days and seven usual ones, one of each in every pair. Walk days had a middle of 100 focused minutes before lunch, usual days 80. But 24 of the 128 label shuffles gave a gap that big, so the walk earned two more weeks, not a verdict.

Your schedule

Flip the coins, log each day, and read it at the end.

Name the change and the number, pick a length and press flip. Each pair of days gets one change day and one usual day in a random order. Type each day's number in as you go; the plot stays covered until the last day so it can't tempt you to stop early.

3 · Each day's number

Press "Flip the coins" and your days appear here, two by two.

4 · The read

  • Change days–
  • Usual days–
  • Shuffles as big–

Flip the coins to make a schedule.

Nothing you type is saved or sent anywhere, and it's gone when you leave. Copy the schedule and keep the daily numbers on paper or in your notes.

Why weeks, not one evening

In one classroom study, a lesson on fair tests was neither needed nor enough on its own. The skill came slowly, over weeks of working on problems.

David Dean and Deanna Kuhn followed three groups of fifteen fourth-graders learning the rule the simple version of this skill starts with: to find out what something does, change it and keep the rest the same. One group worked on problems that needed the rule, twelve sessions over ten weeks. A second group did the same, after a session of direct teaching. A third had the teaching session and nothing more.

An earlier study had made the case that direct teaching beats discovery for this strategy. Followed over a longer stretch and a range of similar tasks, the authors concluded that the teaching was neither necessary nor sufficient for the strategy to take hold and to last. What they saw instead was a gradual process, built and consolidated over many sessions.

The honest limits: forty-five children, one strategy, classroom problems, and a published abstract that gives the conclusion rather than the scores we'd want to quote. It doesn't say a lesson is useless. It says that knowing the rule and using it well are different things, and the second one takes repeated goes.

That's the case for running your tests on a schedule of weeks, and running more than one. The first coin-flip test teaches you the method; the third one is where you start to trust it.

What each of the three groups did over ten weeks: practice sessions only, a teaching session then practice, or the teaching session only wk 1 2 3 4 5 6 7 8 9 10 Practice Lesson, then practice Lesson only no further sessions teaching session practice session (12 over 10 weeks)
Dean & Kuhn (2007), Science Education. The three timelines show what each group did, drawn evenly because the abstract gives the count of sessions, not their dates. The abstract reports a conclusion across the groups, not scores, so none are drawn.

The idea

Why it works.

If you choose the days, you'll choose well: the walk happens on mornings with time to spare, and those mornings were going to go better anyway. A coin has no idea what kind of day it is, so good days and bad ones end up on both sides. Pairing makes sure of it, week by week.

The shuffle answers the question every small test raises: would a gap like this turn up if the change did nothing? With labels that mean nothing, it shows you the gaps your own noisy numbers make. If yours sits among them, you haven't learned much yet. If it sits out at the edge, you've learned something.

When it goes wrong

Habits that help.

  • Fix the length before day one. Stopping the day the numbers look good is the most common way to fool yourself. Pick 14, 20 or 28 days and finish them.
  • Pick a change that switches off overnight. A walk, a cup of coffee, a phone in another room. A new skill or a diet carries over into the next day and blurs both sides.
  • Count a missed day as missed. If the coin said walk and you didn't, write the number and mark it. Moving it to the other side quietly undoes the coin.
  • Run the next one. A test that comes out "promising" is a reason to run 14 more days, not to tell people it works.

Go deeper

Keep going.

Practise this in HourLife.

In Experiments: Running experiments. Test one change for 7 to 30 days, rate it each evening, and record your own verdict when the run ends.

Download on the App Store Download on the Mac App Store How Experiments works in HourLife