A study reports that an after-school reading club "significantly improved" children's reading scores. Most readers hear "significantly" and think "a lot". In a research paper the word usually means something narrower: the result passed a statistical test. The two meanings get mixed up in news stories, in school newsletters and sometimes in the papers themselves. This note explains what the word means in research, what the number behind it does, and what it can't tell you.
What does "statistically significant" actually mean?
A result is statistically significant when the data would be fairly unlikely if there were no real effect, judged against a cutoff the researchers chose in advance, usually a p-value below 0.05.
Researchers start from a dull assumption called the null hypothesis: the reading club makes no difference. Then they ask how surprising their data would be if that assumption were true. The p-value is the answer, written as a number between 0 and 1.
Say the p-value is 0.03. In plain terms: if the club really made no difference, and you ran the same study many times, you would see a gap at least this large in about 3 of every 100 runs, just from the luck of which children ended up in which group. That's rare enough to pass the usual 0.05 line, so the authors call the result significant.
The 0.05 line is a convention that researchers adopted many decades ago. It is useful as a shared rule. It is not a law of nature, and a result at 0.049 is not meaningfully stronger than one at 0.051.
What doesn't a p-value tell you?
A p-value doesn't tell you how big an effect is, how likely the finding is to be true, or whether it matters for a particular child.
These are the three most common misreadings:
- "p = 0.03 means a 3% chance the result is a fluke." It doesn't. The p-value is calculated by assuming there is no effect. It can't also tell you the probability that this assumption is right.
- "Significant means important." A difference can pass the test and still be too small to notice in a classroom.
- "Not significant means no effect." It means the study couldn't separate the effect from chance. That's a different statement.
In 2016 the American Statistical Association published a short statement on p-values because these misreadings had become so widespread. Two of its points are worth remembering: a p-value does not measure the size of an effect or the importance of a result, and decisions shouldn't rest only on whether a p-value crosses a threshold.
Why can a tiny effect be "significant"?
With enough participants, almost any difference, however small, will pass the 0.05 test.
Imagine a survey of 50,000 teenagers that finds a gap of a fraction of a point on a 50-point wellbeing scale between two groups. With that many people, chance alone rarely produces even small gaps, so the result can come out highly significant. The difference is still a fraction of a point.
This is common in large studies of screen time, sleep and wellbeing. Our guide on sample size covers the trap in more detail. The fix is simple to state: whenever you see "significant", look for the number it refers to, in units you can picture.
Why can a real effect be "not significant"?
Small studies often miss real effects, so "no significant difference" is weak evidence that nothing is going on.
A pilot study with 30 children might find that a new phonics program raised scores a little, with a p-value of 0.20. That doesn't show the program fails. It shows that 30 children aren't enough to tell a modest improvement apart from noise. A larger study could find the same improvement and call it significant.
The confidence interval is the better tool here. It shows the range of effects that fit the data, from "slightly harmful" to "quite helpful" in a small study, which tells you plainly how little the study could settle.
What should you look at instead?
Look at the size of the effect, the range of uncertainty around it, and whether other studies found the same thing.
- Find the effect size in plain units. Points on a test, minutes of sleep, the number of children out of 100. If you only see percentages, check the starting rate.
- Check the confidence interval. A narrow range means a precise estimate. A wide one means the study left a lot open.
- Count the comparisons. If a study tested 20 outcomes, one of them may pass the 0.05 line by chance alone. Results that were planned in advance carry more weight than ones found after a search.
- Look for repeats. One significant result is a start. Several independent studies that agree are much stronger.
- Check the design. A significant link in a survey still doesn't show cause. The study types guide explains why.
Here is how common phrases translate:
| Phrase in the paper | What it means | What to ask next |
|---|---|---|
| Significant (p < 0.05) | Passed the usual test | How big is the effect? |
| Highly significant (p < 0.001) | Very unlikely if there were no effect | Was the sample huge? Is the effect still small? |
| Marginally significant (p = 0.07) | Did not pass the test | Is there a larger study? |
| No significant difference | Couldn't separate the result from chance | Was the study big enough to find an effect? |
How should you read the word in a news story?
Treat "significant" in a news story as a claim to check, because the reporter may mean "large" while the paper only means "passed a test".
Find the number behind the word. If the story gives none, the abstract of the paper usually does. Then run the study through the study checklist, which asks about effect size and design alongside significance. For a fuller routine, the guide on reading news about a study walks through eight steps.
A significant result tells you the data would be unusual if nothing were going on. Whether the something is big enough to change what you do at home or in class is a separate question, and the p-value can't answer it.
Questions readers ask
- Is p < 0.05 a strict scientific rule?
- No. It's a long-standing convention. Some fields use stricter cutoffs, such as 0.005 or 0.001, and many statisticians argue against treating any single cutoff as a pass or fail line. A result at p = 0.049 and one at p = 0.051 carry almost the same evidence.
- What does "marginally significant" or "a trend toward significance" mean?
- It usually means the p-value was a little above the cutoff, for example 0.06 or 0.08. The result did not pass the test the authors set. Treat it as unconfirmed and look for larger or repeated studies.
- If a study found no significant difference, does that prove there is no effect?
- No. It means the study could not rule out chance. A small study can easily miss a real but modest effect. The confidence interval shows how large an effect the study could have missed.
- Does a smaller p-value mean a bigger effect?
- Not necessarily. A p-value depends on both the size of the effect and the number of participants. A huge study can produce a very small p-value for a difference too small to notice in daily life.