DSRT 734: Interpreting P-Values, Confidence Intervals, and Effect Sizes

What a p-value means

A p-value is calculated under a specified null hypothesis and model. It is the probability, assuming that null model and its conditions, of obtaining a test statistic at least as extreme as the observed one in the direction defined by the test. A smaller value indicates greater incompatibility between the observed statistic and the null model.

This interpretation is conditional. It depends on the test statistic, hypotheses, sampling or assignment process, model assumptions, and analysis choices. A p-value does not measure the size or importance of an effect. It also does not verify data quality, remove bias, repair confounding, or establish that the selected model is correct.

What a p-value does not mean

The p-value is not the probability that the null hypothesis is true, not the probability that the result occurred by chance, and not the probability that the study will replicate. One minus the p-value is not the probability that the alternative is true. A p-value above a threshold does not prove no effect, while a p-value below a threshold does not prove an important or causal effect.

These errors often arise because the p-value is treated as a direct probability statement about hypotheses. Classical hypothesis testing instead evaluates the behavior of a statistic under the null model. Keep that conditioning visible in the written conclusion and pair the test with estimates, intervals, design evidence, and effect magnitude.

Separate the significance level from the observed p-value

The significance level, alpha, is a decision threshold selected before examining the test result. Under repeated use of a valid procedure, alpha controls the long-run Type I error rate for the specified null-testing process. The p-value is calculated from the observed data. Comparing the two supports the procedural decision to reject or fail to reject the null hypothesis.

Alpha is not a universal boundary between truth and falsehood. The costs of errors, research domain, multiplicity, design, and evidence standards should inform its choice. Reporting the exact p-value, estimate, and interval is more informative than labeling a result only “significant” or “not significant.”

Understand Type I error, Type II error, and power

A Type I error occurs when a testing procedure rejects a true null hypothesis. A Type II error occurs when it fails to reject a false null hypothesis for the effect under consideration. Statistical power is the probability that the procedure rejects the null under a specified alternative, design, sample size, variability, alpha, and method. Power is therefore not a fixed property of a test name.

Low power can make real effects difficult to detect and can produce wide intervals. Very high power can make a small, practically unimportant effect statistically significant. Planning requires a meaningful effect size, realistic variability, anticipated missingness, and a credible analysis plan. After a study, the observed estimate and confidence interval usually communicate more than a post hoc power calculation based on the same data.

Read the point estimate and confidence interval

A point estimate is the sample-based best estimate of a population parameter under the chosen method—for example, a mean difference, proportion difference, odds ratio, correlation, or regression slope. A confidence interval surrounds that estimate with a measure of sampling uncertainty. Its width reflects the confidence level, variability, sample information, and method.

A 95 percent confidence procedure is designed so that, across repeated samples under its assumptions, about 95 percent of the constructed intervals cover the true parameter. For one observed frequentist interval, avoid saying there is a 95 percent probability the fixed parameter lies inside it. Instead, describe the interval as the range of parameter values compatible with the data and procedure at that confidence level.

Use the interval to evaluate both direction and practical importance

For a two-sided test, a confidence interval that excludes the null value corresponds to rejection at the matching significance level under the same method. But the interval adds information a yes-or-no test cannot: the range of plausible magnitudes and the precision of the estimate. An interval can exclude zero while containing only effects too small to matter, or include both negligible and important effects.

Compare the interval with a pre-established practical threshold when one exists. If the entire interval lies beyond that threshold, the evidence for practical importance is stronger. If it spans negligible and meaningful values, uncertainty remains decision-relevant even if the p-value crosses a conventional cutoff.

Interpret effect size in a meaningful scale

An effect size describes magnitude. Raw effects, such as a difference in minutes or percentage points, are often easiest to connect to a decision. Standardized effects can support comparison across measures but require context and should not replace the original scale. Ratios such as risk ratios or odds ratios need careful interpretation because equal numerical distances do not necessarily have equal practical meaning.

No universal small, medium, or large label fits every discipline. The same effect can be trivial in one setting and consequential in another. Interpret magnitude with a practical threshold, baseline risk, measurement reliability, costs, benefits, feasibility, and the confidence interval.

Distinguish statistical from practical significance

Statistical significance is a property of the observed evidence relative to a null model, chosen procedure, and threshold. Practical significance asks whether the magnitude matters for the decision. Sample size links the two: with enough precise data, a very small effect may produce a small p-value; with limited data, a meaningful effect may remain uncertain.

A decision should therefore not be written as “the result was significant, so the intervention works.” State the estimated magnitude, uncertainty, test result, design boundary, and practical comparison. The conclusion may support further study, cautious implementation, rejection of an option, or no change, depending on costs and consequences beyond the statistical output.

Why a non-significant result does not prove no effect

Failing to reject the null means the observed evidence did not meet the selected rejection rule under the method. It is not proof that the null is true. The confidence interval may show that the data remain compatible with both negligible and important effects. Limited sample information, high variability, measurement error, weak implementation, and model mismatch can all contribute to an inconclusive result.

If the research goal is to show that effects are sufficiently small, a conventional non-significance result is not enough. Equivalence or noninferiority questions require prespecified margins and appropriate procedures. Otherwise, use language such as “the study did not provide sufficient evidence of a difference” and describe the interval and limitations.

Illustrative result: interpret the complete evidence

Imagine a fictional study estimating that a revised review process reduces average decision time by 3.2 minutes compared with the current process. The 95 percent confidence interval for the mean reduction is 0.6 to 5.8 minutes, the two-sided p-value is 0.018, and the standardized effect is modest. The data and model are invented and are not B03 output or an official assignment.

At alpha 0.05, the result rejects the null of no mean difference under the selected model. The interval suggests reductions from less than one minute to nearly six minutes remain compatible with the evidence. Whether the result matters in practice depends on a decision threshold, implementation cost, quality effects, study design, and assumption checks. If only reductions of at least five minutes justify implementation, the interval includes both practically insufficient and sufficient values, so the decision remains uncertain despite statistical significance.

Report a conclusion without overstating evidence

A complete conclusion identifies the population and comparison, reports the point estimate and confidence interval, states the p-value or testing decision, describes effect magnitude, and names the study-design and assumption boundaries. Use causal language only when the design and analysis support it. Avoid “proved,” “no effect,” “highly important,” or “due to chance” unless the claim is independently justified.

Finish with the decision consequence and remaining uncertainty. A useful template is: the sample estimated a specified difference or association; the interval shows the compatible magnitude range; the test provides a stated level of evidence against the null model; and the practical decision depends on a defined threshold, limitations, and context.

Related DSRT 734 questions

• DSRT 734: Reporting an Independent-Samples T-Test in APA Format
• DSRT 734: Estimating Pearson’s Correlation From a Scatterplot
• DSRT 734: Interpreting an ANOVA and Post Hoc Comparison
• DSRT 734: Reporting a Second Independent-Samples T-Test in APA Format
• DSRT 734: Interpreting a P-Value of 0.051
• DSRT 734: Using a Regression Line to Estimate Manufacturing Time
• DSRT 734: Reporting a Paired-Samples T-Test in APA Format
• DSRT 734: Interpreting an ANOVA With Bonferroni Comparisons
• DSRT 734: Selecting the Correct APA Result for Coffee and Tea Scores
• DSRT 734: Selecting the Correct P-Value for Gender and Daily Exercise

Related DSRT 734 resources

• DSRT 734 Inferential Statistics in Decision-Making at University of the Cumberlands
• DSRT 734: From a Research Question to Hypotheses and Study Design
• DSRT 734: How to Choose an Inferential Statistical Test