Sample size, and why it matters in studies of children

One of the first numbers to look for in any study is how many people took part. It's usually in the abstract, and if it isn't, that alone tells you something. The number matters, but how it matters is less obvious than "bigger is better". This guide explains what sample size does, what it doesn't do, and how to read it.

Why small samples give shaky answers

Any group you study is a sample from a larger population. If you measured a different group of children, you'd get slightly different results, just by chance. With a small sample, those chance differences can be large. With a big sample, they shrink.

Think of flipping a coin. Ten flips can easily give seven heads. A thousand flips will land close to half heads almost every time. Studies work the same way: a result based on 20 children can swing a long way from the truth in either direction.

Margins of error in plain numbers

Surveys often report a margin of error. For a simple yes-or-no question with answers near 50%, a rough rule of thumb gives these margins at the usual 95% confidence level:

So if a survey of 100 parents says 40% worry about their child's sleep, the true figure could plausibly be anywhere from about 30% to 50%. Notice that quadrupling the sample only halves the margin. Precision gets expensive quickly, which is why many studies stop at a few hundred or a few thousand people.

These margins only cover random chance. They don't cover a badly worded question or a sample that doesn't represent the population. Those errors don't shrink as the sample grows.

Small studies exaggerate

Here is the less intuitive part. Small studies don't just give noisier results; the small studies that get attention tend to overstate effects.

The reason is filtering. A small study can only reach statistical significance if it happens to find a large effect. Small studies that found modest or no effects are less likely to be published or reported. So the small studies that make the news are disproportionately the ones where chance pushed the result upward. When larger studies follow, the effect often shrinks or disappears.

This is one reason the checklist gives less weight to "first study of its kind" findings, and why replication matters so much.

Big samples have their own trap

At the other extreme, very large studies can detect tiny differences. With 100,000 participants, a difference of a fraction of a point on a wellbeing scale can be "statistically significant". Significant here means unlikely to be pure chance, not large or important.

So for big studies, the question shifts from "is there an effect?" to "how big is it?" Look for the size of the difference in real terms: minutes of sleep, points on a reading test, or how many children out of 100 were affected. The screen-time guide gives an example where a widely reported link turned out to be very small.

Who took part matters more than how many

A sample can be large and still misleading if it isn't the right people. Online surveys that anyone can join tend to attract people with strong views. A study of 5,000 children from private schools in one city tells you about those children, not about children everywhere.

Questions worth asking:

• How were participants recruited? Randomly from a register, from schools, from a clinic, or through social media ads?
• Who said no? If many invited families declined, the ones who agreed may differ.
• Where and when was the study done? Results from one country or decade may not carry over.
• Does the age range match the children you care about?

A well-chosen sample of 800 can tell you more than a self-selected one of 80,000.

Dropouts and missing data

Studies that follow children over time lose some of them along the way. Families move, children change schools, people stop answering. A study can start with 2,000 children and analyze 1,200.

That's normal, but it can bias the results if the families who leave differ from those who stay. Children with more difficulties, for example, are often harder to follow up. Good studies report how many dropped out, compare them with those who stayed and explain how they handled missing data. Look for a flow chart or a sentence that accounts for everyone.

Subgroups shrink the sample

A study of 1,000 teenagers may report a striking result "among girls aged 13 to 14 who use social media for more than three hours a day". That subgroup might be 60 people. All the problems of small samples come back.

Be especially cautious when a study reports many subgroups and only one shows an effect. If you test enough groups, some will look interesting by chance alone. Findings that the authors planned to test in advance, and said so, are more trustworthy than ones found after the fact.

Classrooms and schools count too

In education research, children are often studied in groups. If a trial gives a new reading program to 10 schools and compares them with 10 others, the effective sample is closer to 20 schools than to the 3,000 pupils inside them. Children in the same school share teachers, buildings and neighborhoods, so they aren't independent data points. Good studies account for this; news stories rarely mention it.

Sample size in interview studies

Qualitative studies use small samples on purpose, often between 10 and 50 participants. That's fine, because their aim is depth and explanation, not counting. Judge them by how participants were chosen, whether different kinds of people were included, and whether the authors show their reasoning with examples. Don't use them to say how common something is.

Putting it together

When you look at a sample, ask three questions: how many, who, and how many were left at the end. Then look at the size of the effect, not only whether it was significant. Those four answers usually tell you more than the headline does.

Next, research ethics and consent with minors explains why studies with children are designed the way they are, and why some questions are so hard to study directly.