Statistical Inference

Population parameter	Sample statistic
Mean μ	Mean x̄
Proportion p	Proportion p̂
Standard deviation σ	Standard deviation s

Confidence level	Approximate z critical value for a two-sided interval
80%	1.282
90%	1.645
95%	1.960
98%	2.326
99%	2.576

 

 

 

 

 

 

 

 

 

 

 

 

The Significance Level

The significance level, written α (alpha), is the threshold used to decide whether a result is statistically significant.

Common choices are:

The significance level should be chosen before examining the results.

At α = 0.05:

• If p ≤ 0.05, reject H₀.

• If p > 0.05, fail to reject H₀.

The value α represents the maximum probability of a Type I error under the test's null model and assumptions.

A smaller α requires stronger evidence before rejecting H₀.

Calculating Test Statistics

A test statistic measures how far a sample result is from the value predicted by H₀, relative to the sampling variability expected under H₀.

A general structure is:

Test statistic = (Observed estimate − Null value) / Standard error

A large absolute test statistic indicates that the observed result is far from the null value relative to its standard error.

Different hypothesis tests use different test statistics.

z-Test for a Population Mean

When the population standard deviation σ is known, a one-sample z-test may be appropriate.

Formula:

z = (x̄ − μ₀)/(σ/√n)

Where:

• x̄ = sample mean.

• μ₀ = population mean specified by H₀.

• σ = known population standard deviation.

• n = sample size.

The test assumes an appropriate random and independent sampling process and a normal or approximately normal sampling distribution of the mean.

Worked Example 3: Calculating a z-Test Statistic

A factory claims that the mean lifetime of its batteries is 500 hours.

A random sample of 64 batteries has:

x̄ = 485 hours.

σ = 40 hours.

Test whether the population mean is less than 500 hours.

Step 1: State the hypotheses

H₀: μ = 500.

H₁: μ < 500.

Step 2: Identify the information

x̄ = 485.

μ₀ = 500.

σ = 40.

n = 64.

Step 3: Calculate the standard error

SE = σ/√n

= 40/√64

= 40/8

= 5 hours.

Step 4: Calculate z

z = (485 − 500)/5

= −15/5

= −3.00.

Answer: z = −3.00.

The sample mean is three standard errors below the null-hypothesis mean.

A z-value of −3 is unusually low under the standard normal distribution.

The Standard Normal Distribution

Under H₀ and the z-test assumptions, the z-statistic follows a standard normal distribution, exactly or approximately.

The standard normal distribution has:

Mean = 0.

Standard deviation = 1.

Its shape is symmetric and bell-shaped.

Values close to zero are common, while values far from zero are less common.

The p-value is calculated from the appropriate tail area or areas of this distribution.

Understanding p-Values

The p-value is the probability, assuming the null hypothesis and the statistical model are correct, of obtaining a test statistic at least as extreme as the one observed, in the direction or directions specified by the alternative hypothesis.

A small p-value means that the observed result would be relatively unusual under H₀.

A large p-value means that the result is not particularly unusual under H₀.

Importantly:

• A p-value is not the probability that H₀ is true.

• A p-value is not the probability that the results occurred purely by chance.

• A small p-value does not measure the size or practical importance of an effect.

• A large p-value does not prove that H₀ is true.

Calculating p-Values from z-Statistics

Let Φ(z) represent the cumulative probability to the left of z under the standard normal distribution.

For a two-tailed test, the probability is calculated using both tails.

Worked Example 4: Completing the Battery Hypothesis Test

Return to the battery investigation.

H₀: μ = 500 hours.

H₁: μ < 500 hours.

Calculated test statistic:

z = −3.00.

Use α = 0.05.

Step 1: Calculate the p-value

Because this is a left-tailed test:

p = P(Z ≤ −3.00)

≈ 0.00135.

Step 2: Compare with α

0.00135 < 0.05.

Therefore, reject H₀.

Step 3: State the conclusion

There is statistically significant evidence at the 5% significance level that the population mean battery lifetime is less than 500 hours.

Step 4: Interpret the p-value

If the population mean really were 500 hours and the model assumptions held, the probability of obtaining a sample mean of 485 hours or less from a sample of 64 batteries would be approximately 0.135%.

This is strong evidence against the manufacturer's claimed mean.

It does not prove that every battery lasts less than 500 hours.

Interactive Hypothesis-Test Explorer

Use the controls to explore how the test statistic, alternative hypothesis, and significance level affect the statistical decision.

Worked Example 5: A Right-Tailed z-Test

A company claims that the mean delivery time is 30 minutes.

A random sample of 100 deliveries gives:

x̄ = 32.4 minutes.

The known population standard deviation is:

σ = 10 minutes.

Test whether the mean delivery time exceeds 30 minutes at α = 0.05.

Step 1: Hypotheses

H₀: μ = 30.

H₁: μ > 30.

Step 2: Standard error

SE = 10/√100 = 1.

Step 3: Test statistic

z = (32.4 − 30)/1

= 2.40.

Step 4: p-value

p = P(Z ≥ 2.40)

≈ 0.0082.

Step 5: Decision

0.0082 < 0.05.

Reject H₀.

Conclusion: There is statistically significant evidence that the population mean delivery time exceeds 30 minutes.

The sample mean suggests an increase of 2.4 minutes. Whether this increase is practically important depends on the business context.

Worked Example 6: A Two-Tailed z-Test

A filling machine is designed to dispense a mean of 250 mL.

A sample of 64 containers has:

x̄ = 247 mL.

σ = 12 mL.

Test whether the mean filling volume differs from 250 mL at α = 0.05.

Step 1: Hypotheses

H₀: μ = 250.

H₁: μ ≠ 250.

Step 2: Standard error

SE = 12/√64

= 1.5 mL.

Step 3: Test statistic

z = (247 − 250)/1.5

= −2.00.

Step 4: Two-tailed p-value

p = 2P(Z ≤ −2.00)

≈ 2(0.02275)

≈ 0.0455.

Step 5: Decision

0.0455 < 0.05.

Reject H₀.

Conclusion: There is statistically significant evidence that the population mean filling volume differs from 250 mL.

The sample indicates that the mean may be lower than the target.

Because this was a two-tailed test, the p-value includes extreme results in both directions.

t-Tests for a Population Mean

In many investigations, the population standard deviation σ is unknown.

Instead, the sample standard deviation s is used.

For a one-sample mean test, the appropriate statistic is usually:

t = (x̄ − μ₀)/(s/√n)

The degrees of freedom are:

df = n − 1

The p-value is calculated using Student's t-distribution.

A t-test requires an appropriate sampling design and distributional conditions. For small samples, the observations should be approximately normally distributed without severe outliers; larger samples allow more flexibility.

Worked Example 7: One-Sample t-Test

A school believes that the population mean mathematics test score is 70 marks.

A random sample of 25 students produces:

x̄ = 74.

s = 10.

Test whether the population mean score is different from 70 at α = 0.05.

Assume the t-test conditions hold.

Step 1: Hypotheses

H₀: μ = 70.

H₁: μ ≠ 70.

Step 2: Degrees of freedom

df = 25 − 1 = 24.

Step 3: Standard error

SE = 10/√25

= 2.

Step 4: Test statistic

t = (74 − 70)/2

= 2.00.

Step 5: p-value

For t = 2.00 with 24 degrees of freedom, using a two-tailed test:

p ≈ 0.057.

Step 6: Decision

0.057 > 0.05.

Fail to reject H₀.

Conclusion: There is insufficient statistical evidence at the 5% significance level that the population mean mathematics score differs from 70 marks.

This does not prove that the population mean is exactly 70.

The result is close to the significance threshold, illustrating why reporting the actual p-value is more informative than reporting only the decision.

Worked Example 8: A Left-Tailed t-Test

A factory claims that its components have a mean breaking strength of 100 N.

A random sample of 16 components has:

x̄ = 94 N.

s = 12 N.

Test whether the population mean breaking strength is less than 100 N.

Assume the population distribution is approximately normal.

Use α = 0.05.

Step 1: Hypotheses

H₀: μ = 100.

H₁: μ < 100.

Step 2: Degrees of freedom

df = 15.

Step 3: Standard error

SE = 12/√16 = 3 N.

Step 4: Test statistic

t = (94 − 100)/3

= −2.00.

Step 5: p-value

For t = −2.00 with df = 15, left-tailed:

p ≈ 0.032.

Step 6: Decision

0.032 < 0.05.

Reject H₀.

Conclusion: There is statistically significant evidence that the population mean breaking strength is below 100 N.

This may justify further quality-control investigation.

Hypothesis Tests for Population Proportions

Hypothesis tests are also used to investigate population proportions.

Examples include the proportion of students who prefer a new timetable, the percentage of defective products, and the proportion of households using renewable energy.

The sample proportion is:

p̂ = x/n

Where x is the number of observations with the characteristic and n is the sample size.

For a one-sample proportion z-test:

z = (p̂ − p₀)/√(p₀(1 − p₀)/n)

Where p₀ is the population proportion specified by H₀.

Notice that the standard error uses p₀, not p̂. This is because the test statistic is calculated under the null hypothesis.

For the normal approximation, common conditions include:

n p₀ ≥ 10

and

n(1 − p₀) ≥ 10.

The observations should also be independent or approximately independent, and the sampling design should be appropriate.

Worked Example 9: Testing a Population Proportion

A school claims that 50% of students prefer digital textbooks.

A random sample of 200 students finds that 120 prefer digital textbooks.

Test whether the population proportion is greater than 50% at α = 0.05.

Step 1: Hypotheses

H₀: p = 0.50.

H₁: p > 0.50.

Step 2: Calculate the sample proportion

p̂ = 120/200

= 0.60.

Step 3: Check the approximation

n p₀ = 200 × 0.50 = 100.

n(1 − p₀) = 100.

Both exceed 10.

Step 4: Calculate standard error under H₀

SE₀ = √(0.50 × 0.50/200)

= √0.00125

≈ 0.03536.

Step 5: Calculate the test statistic

z = (0.60 − 0.50)/0.03536

≈ 2.83.

Step 6: Calculate the p-value

This is a right-tailed test.

p = P(Z ≥ 2.83)

≈ 0.0023.

Step 7: Decision

0.0023 < 0.05.

Reject H₀.

Conclusion: There is statistically significant evidence that more than 50% of students in the population prefer digital textbooks.

Worked Example 10: Testing a Defect Rate

A manufacturer claims that 5% of its products are defective.

A random sample of 400 products contains 30 defective items.

Test whether the population defect rate exceeds 5% at α = 0.05.

Step 1: Hypotheses

H₀: p = 0.05.

H₁: p > 0.05.

Step 2: Sample proportion

p̂ = 30/400

= 0.075.

Step 3: Check conditions

n p₀ = 400 × 0.05 = 20.

n(1 − p₀) = 380.

The normal approximation is reasonable.

Step 4: Standard error

SE₀ = √(0.05 × 0.95/400)

≈ 0.01090.

Step 5: Test statistic

z = (0.075 − 0.05)/0.01090

≈ 2.29.

Step 6: p-value

p ≈ 0.0109.

Step 7: Decision

0.0109 < 0.05.

Reject H₀.

Conclusion: There is statistically significant evidence that the population defect rate exceeds 5%.

This finding could motivate further inspection of the manufacturing process.

Making Statistical Decisions

Once the p-value has been calculated, compare it with the chosen significance level.

We generally avoid saying accept H₀.

A failure to reject may occur because the null hypothesis is correct, because the sample is too small, or because the true effect is difficult to detect.

Worked Example 11: Different Significance Levels

A hypothesis test produces:

p = 0.032.

Determine the decision at three significance levels.

The data and p-value have not changed.

Only the decision threshold has changed.

This demonstrates why a statistical report should include the p-value and the chosen significance level rather than simply stating that a result is significant.

Type I and Type II Errors

Hypothesis testing involves the possibility of incorrect decisions.

Type I Error

A Type I error occurs when H₀ is rejected even though it is true.

This is sometimes called a false positive.

The probability of a Type I error is controlled by the significance level α, under the test assumptions.

Type II Error

A Type II error occurs when H₀ is not rejected even though a particular alternative is true.

This is sometimes called a false negative.

The probability of a Type II error is represented by β (beta), for a specified alternative.

Statistical Power

The power of a test is the probability of rejecting H₀ when a specified alternative is true.

Power = 1 − β

Higher power means a greater chance of detecting a real effect of the specified size.

Power is affected by:

• Sample size.

• Effect size.

• Population variability.

• Significance level.

• Choice of statistical test.

Larger samples generally increase power when other conditions remain unchanged.

Worked Example 12: Understanding Errors in Context

A company tests whether its mean product mass is below the advertised 500 g.

H₀: μ = 500.

H₁: μ < 500.

A Type I error would occur if the company concluded that the population mean was below 500 g when the null value of 500 g was actually correct.

A Type II error would occur if the company failed to detect an actual reduction in mean mass.

The consequences differ.

A Type I error could cause unnecessary investigations or production changes.

A Type II error could allow an underfilling problem to continue.

The appropriate significance level should reflect the consequences of these errors, not merely a conventional choice of 0.05.

Statistical Significance Versus Practical Significance

A statistically significant result is not necessarily important in real life.

Suppose a very large sample finds that a new teaching method increases the mean test score by 0.2 marks.

The result might have a very small p-value because the estimate is extremely precise.

However, an increase of only 0.2 marks may have little practical educational value.

Conversely, a small study might estimate a substantial improvement but fail to reach statistical significance because the estimate is too uncertain.

Worked Example 13: A Large Sample with a Small Effect

A process has a historical mean output of 100 units.

A sample of 10,000 observations produces:

x̄ = 100.5.

Known σ = 10.

Test:

H₀: μ = 100.

H₁: μ ≠ 100.

Step 1: Standard error

SE = 10/√10000

= 0.1.

Step 2: Test statistic

z = (100.5 − 100)/0.1

= 5.00.

Step 3: p-value

The two-tailed p-value is approximately:

p ≈ 0.00000057.

This is statistically significant at α = 0.05.

However, the estimated increase is only 0.5 units, or 0.5% of the historical mean.

Conclusion: The data provide strong statistical evidence of a difference, but the practical importance of that difference must be evaluated separately.

Connecting Hypothesis Tests and Confidence Intervals

Hypothesis testing and confidence intervals are closely related.

For many standard two-sided tests, a hypothesis test at significance level α corresponds to a confidence interval with confidence level:

100(1 − α)%

For example:

A two-sided test at α = 0.05 corresponds to a 95% confidence interval when the test and interval use matching methods and assumptions.

If the null-hypothesis value lies outside the confidence interval, reject H₀.

If the null-hypothesis value lies inside the interval, fail to reject H₀.

This correspondence applies to compatible two-sided procedures; it should not be applied indiscriminately to one-sided tests or intervals constructed using different methods.

Worked Example 14: Testing with a Confidence Interval

A sample gives:

x̄ = 52.

Known σ = 12.

n = 36.

Test:

H₀: μ = 48.

H₁: μ ≠ 48.

Use α = 0.05.

Step 1: Calculate the standard error

SE = 12/√36

= 2.

Step 2: Construct the 95% confidence interval

CI = 52 ± 1.96 × 2

= 52 ± 3.92.

CI = (48.08, 55.92).

Step 3: Compare the null value

The null value is 48.

Since 48 lies outside the interval, reject H₀.

Step 4: Check with a test statistic

z = (52 − 48)/2

= 2.00.

Two-tailed p ≈ 0.0455.

Since p < 0.05, reject H₀.

Conclusion: Both methods agree that there is statistically significant evidence that the population mean differs from 48.

The confidence interval also provides a range of plausible values for the population mean, making it especially useful for communicating the size and uncertainty of the estimated effect.

Choosing the Appropriate Hypothesis Test

The correct test depends on the type of data, the parameter being investigated, and the study design.

Each test has assumptions that must be checked.

For example, a paired t-test is appropriate when the same students are measured before and after an intervention, while a two-sample t-test may be appropriate when comparing independent groups of students.

Worked Example 15: Paired Data

Ten students complete a mathematics assessment before and after a teaching intervention.

For each student, calculate:

Difference = After score − Before score.

The ten differences have:

Mean difference d̄ = 4.5 marks.

Standard deviation of differences s_d = 5 marks.

Test whether the mean improvement is greater than zero.

Assume the differences are approximately normally distributed and students are independently sampled.

Step 1: Hypotheses

Let μ_d be the population mean score difference.

H₀: μ_d = 0.

H₁: μ_d > 0.

Step 2: Standard error

SE = s_d/√n

= 5/√10

≈ 1.581.

Step 3: Test statistic

t = 4.5/1.581

≈ 2.846.

Step 4: Degrees of freedom

df = 10 − 1 = 9.

Step 5: p-value

For a right-tailed test:

p ≈ 0.0096.

Step 6: Decision

0.0096 < 0.05.

Reject H₀.

Conclusion: There is statistically significant evidence that the population mean after-minus-before score difference is positive.

However, without an appropriate comparison group, the improvement cannot automatically be attributed entirely to the teaching intervention. Other influences may have affected the scores.

Communicating the Conclusions of Hypothesis Tests

A complete statistical conclusion should contain four elements:

1. The statistical decision.

2. The significance level.

3. The population parameter and direction of the claim.

4. The strength and limitations of the evidence.

Example: Rejecting H₀

"At the 5% significance level, the test provides statistically significant evidence that the population mean battery lifetime is less than 500 hours (z = −3.00, p ≈ 0.00135)."

Example: Failing to Reject H₀

"At the 5% significance level, there is insufficient statistical evidence that the population mean test score differs from 70 marks (t = 2.00, df = 24, p ≈ 0.057)."

Avoid These Statements

Incorrect: "The null hypothesis is definitely false."

A hypothesis test provides evidence, not certainty.

Incorrect: "The null hypothesis is true because p > 0.05."

Failure to reject H₀ does not establish that H₀ is true.

Incorrect: "There is a 3% chance that the null hypothesis is true."

A p-value of 0.03 does not represent the probability that H₀ is true.

Incorrect: "The result is important because p < 0.05."

Statistical significance does not automatically establish practical importance.

Evaluating the Reliability of a Hypothesis Test

Before accepting a statistical conclusion, consider:

Sampling: Was the sample randomly selected and representative?

Independence: Are the observations independent, or was dependence properly modelled?

Distribution: Are the assumptions of the selected test reasonable?

Sample size: Is the sample large enough to provide useful power and support any approximations?

Measurement: Were the data collected accurately?

Effect size: How large is the observed difference?

Multiple testing: Were many hypotheses tested, increasing the chance of false positives?

Study design: Does the study support a causal conclusion, or only an association?

A small p-value cannot compensate for a fundamentally flawed study design.

Real-World Applications

Integrated Investigation: Does a New Study Programme Improve Results?

A school introduces an optional mathematics study programme.

Historically, the mean score on a standardized assessment is 70 marks.

A random sample of 36 participating students obtains:

x̄ = 75 marks.

s = 12 marks.

The school wants to determine whether the population mean score among programme participants exceeds 70 marks.

Assume the observations are independent and the t-test conditions are appropriate.

Step 1: State the Hypotheses

H₀: μ = 70.

H₁: μ > 70.

Step 2: Select the Test

The population standard deviation is unknown.

Therefore, use a one-sample t-test.

Step 3: Choose the Significance Level

α = 0.05.

Step 4: Calculate the Standard Error

SE = 12/√36

= 2 marks.

Step 5: Calculate the Test Statistic

t = (75 − 70)/2

= 2.50.

Step 6: Degrees of Freedom

df = 36 − 1 = 35.

Step 7: Calculate the p-Value

For t = 2.50, df = 35, right-tailed:

p ≈ 0.0086.

Step 8: Make the Decision

0.0086 < 0.05.

Reject H₀.

Step 9: Communicate the Conclusion

There is statistically significant evidence at the 5% significance level that the population mean score among students participating in the programme exceeds 70 marks.

Step 10: Evaluate the Investigation

This does not prove that the programme caused the higher scores.

Students who volunteered for the programme may already be more motivated or better prepared.

A stronger causal investigation would use random assignment to programme and comparison groups, with appropriate analysis of the resulting outcomes.

This example demonstrates how hypothesis testing combines mathematical calculation, statistical interpretation, and critical evaluation.

Common Mistakes

Mistake 1: Confusing H₀ and H₁

The null hypothesis specifies the reference claim. The alternative describes the effect or difference being investigated.

Mistake 2: Using Sample Statistics in Hypotheses

Hypotheses concern population parameters, such as μ or p, rather than observed sample values such as x̄ or p̂.

Mistake 3: Choosing the Test Direction After Seeing the Data

The alternative hypothesis should be determined from the research question before examining the results.

Mistake 4: Using a Two-Tailed p-Value for a One-Tailed Test

The p-value must correspond to the specified alternative hypothesis.

Mistake 5: Forgetting the Standard Error

A test statistic compares a difference with its expected sampling variability, not merely with the raw standard deviation.

Mistake 6: Using σ When It Is Unknown

For a one-sample mean test with unknown σ, the t-test is generally appropriate.

Mistake 7: Forgetting Degrees of Freedom

For a one-sample t-test, df = n − 1.

Mistake 8: Using p̂ Instead of p₀ in a One-Proportion z-Test Standard Error

The test statistic uses the standard error calculated under H₀.

Mistake 9: Misinterpreting the p-Value

The p-value is not the probability that H₀ is true.

Mistake 10: Saying "Accept H₀"

The standard conclusion is "fail to reject H₀" when the evidence is insufficient.

Mistake 11: Treating Statistical Significance as Proof

Even statistically significant findings can be affected by sampling error, bias, or incorrect assumptions.

Mistake 12: Confusing Statistical Significance with Practical Importance

A very small effect may be statistically significant in a large sample.

Mistake 13: Ignoring Type I and Type II Errors

Both false rejection and failure to detect a real effect have consequences.

Mistake 14: Ignoring Sampling Bias

A large biased sample does not automatically support valid population inference.

Mistake 15: Failing to Check Test Assumptions

Independence, distributional conditions, and sampling design matter.

Mistake 16: Assuming a Nonsignificant Result Proves No Effect

The test may have insufficient power to detect a real effect.

Mistake 17: Using Inappropriate Precision

Report calculated values to a sensible number of decimal places without rounding intermediate calculations excessively.

Mistake 18: Making a Causal Claim from an Observational Study

An association or statistically significant difference does not necessarily establish causation.

Mistake 19: Ignoring Multiple Comparisons

Testing many hypotheses without appropriate adjustment increases the chance of obtaining false-positive findings.

Mistake 20: Giving a Decision Without Context

A complete conclusion should explain what the evidence means for the population and research question.

Key Terms

• Hypothesis Testing: A statistical procedure for evaluating a claim about a population.

• Statistical Hypothesis: A statement about a population parameter or distribution.

• Null Hypothesis (H₀): The reference hypothesis evaluated by a statistical test.

• Alternative Hypothesis (H₁ or Hₐ): The competing claim for which evidence is sought.

• Population Parameter: A numerical characteristic of a population.

• Sample Statistic: A numerical characteristic calculated from sample data.

• Test Statistic: A standardized or otherwise defined measure of evidence against H₀.

• z-Test: A test using the standard normal distribution under appropriate assumptions.

• t-Test: A test using Student's t-distribution.

• One-Tailed Test: A test for an effect in a specified direction.

• Left-Tailed Test: A test with a less-than alternative.

• Right-Tailed Test: A test with a greater-than alternative.

• Two-Tailed Test: A test for a difference in either direction.

• Significance Level (α): The chosen threshold controlling the Type I error probability.

• p-Value: The probability under H₀ of a test statistic at least as extreme as the observed one.

• Statistical Significance: A result meeting the chosen criterion for rejecting H₀.

• Standard Error: The standard deviation of a statistic's sampling distribution, or an estimate of it.

• Critical Value: A cutoff defining a rejection region.

• Rejection Region: Values of a test statistic that lead to rejection of H₀.

• Degrees of Freedom: A parameter determining the relevant t-distribution.

• Type I Error: Rejecting a true null hypothesis.

• Type II Error: Failing to reject a false null hypothesis.

• Statistical Power: The probability of rejecting H₀ under a specified alternative.

• Effect Size: A measure of the magnitude of an effect or difference.

• Practical Significance: The real-world importance of an effect.

• Confidence Interval: An interval estimate with a specified long-run coverage rate.

• Sampling Variability: Differences between statistics obtained from different samples.

• Random Sampling: Selecting observations using a chance-based procedure.

• Sampling Bias: Systematic distortion caused by an unrepresentative selection process.

• Normal Approximation: Using a normal distribution to approximate a sampling distribution.

• Paired t-Test: A t-test applied to differences between paired observations.

• Two-Sample t-Test: A test comparing the means of two independent populations.

• One-Proportion z-Test: A test evaluating a population proportion using a normal approximation.

• Multiple Testing: Conducting several hypothesis tests, potentially increasing false-positive risk.

• Statistical Inference: Drawing conclusions about populations using sample data.

Key Takeaways

1. Hypothesis testing evaluates claims about population parameters.

2. The null hypothesis provides the reference model.

3. The alternative hypothesis specifies the claim being investigated.

4. Tests may be left-tailed, right-tailed, or two-tailed.

5. The test direction should be selected before examining the data.

6. A test statistic measures the observed difference relative to sampling variability.

7. z-tests and t-tests are used under different conditions.

8. One-proportion z-tests use the null proportion when calculating standard error.

9. A p-value measures how extreme the observed statistic is under H₀.

10. A small p-value provides evidence against H₀.

11. The significance level establishes the rejection threshold.

12. When p ≤ α, reject H₀.

13. When p > α, fail to reject H₀.

14. Failing to reject H₀ does not prove H₀ true.

15. Type I errors involve rejecting true null hypotheses.

16. Type II errors involve failing to detect false null hypotheses.

17. Statistical power describes the probability of detecting a specified real effect.

18. Statistical significance and practical significance are different.

19. Confidence intervals and compatible two-sided hypothesis tests are closely connected.

20. A complete conclusion includes the decision, evidence, context, and limitations.

Check Your Understanding

Use α = 0.05 unless another significance level is stated. Show your calculations and explain your decisions in context.

Question 1: Formulating Hypotheses

A company claims that the mean mass of its cereal packets is 400 g.

A researcher suspects that the mean mass is lower.

State H₀ and H₁, and identify the test direction.

Question 2: Two-Tailed Hypotheses

A school wants to investigate whether its population mean mathematics score differs from 75 marks.

Write the hypotheses and explain why the test is two-tailed.

Question 3: Calculating a z-Statistic

A sample of 100 measurements has:

x̄ = 54.

μ₀ = 50.

σ = 20.

Calculate the z-test statistic.

Question 4: Left-Tailed z-Test

A random sample of 64 products has:

x̄ = 98 g.

μ₀ = 100 g.

σ = 8 g.

Test whether the population mean mass is below 100 g.

Use the standard normal distribution.

Question 5: Right-Tailed z-Test

A sample of 100 observations has:

x̄ = 105.

μ₀ = 100.

σ = 20.

Test whether the population mean exceeds 100.

Question 6: Interpreting a p-Value

A hypothesis test produces p = 0.018.

Explain what the p-value means and state the decision at α = 0.05.

Question 7: Different Significance Levels

A test produces p = 0.034.

Determine the decision at:

a) α = 0.10.

b) α = 0.05.

c) α = 0.01.

Question 8: One-Sample t-Test

A random sample of 16 students has:

x̄ = 78.

s = 8.

Test whether the population mean score exceeds 75.

Assume the t-test conditions hold.

Use the t-distribution with 15 degrees of freedom.

Question 9: Population Proportion

A random sample of 200 students finds that 130 prefer digital learning materials.

Test whether the population proportion exceeds 0.50.

Question 10: Defect Rate

A manufacturer claims that 10% of its products are defective.

A random sample of 300 products contains 42 defective products.

Test whether the population defect proportion exceeds 10%.

Question 11: Type I and Type II Errors

A researcher tests whether the population mean concentration of a pollutant exceeds a specified safety threshold.

Explain Type I and Type II errors in this context.

Question 12: Statistical Versus Practical Significance

A large study finds that a new educational intervention increases average test scores by 0.1 marks, with p = 0.001.

Explain the statistical conclusion and why the result may not be practically important.

Question 13: Confidence Interval Connection

A 95% confidence interval for a population mean is:

(48, 56).

For a compatible two-sided test at α = 0.05, determine whether to reject H₀ when:

a) μ₀ = 50.

b) μ₀ = 45.

Question 14: Identifying Flawed Reasoning

A student writes:

"Because p = 0.12, there is a 12% probability that the null hypothesis is true. Therefore, we should accept the null hypothesis."

Identify and correct the errors.

Question 15: Comprehensive Hypothesis Test

A school claims that the population mean time students spend completing a mathematics assessment is 60 minutes.

A random sample of 36 students gives:

x̄ = 64 minutes.

s = 12 minutes.

Assume the t-test conditions hold.

Test whether the population mean completion time differs from 60 minutes.

Use:

α = 0.05.

For 35 degrees of freedom, the two-tailed p-value for t = 2.00 is approximately 0.053.

Calculate and explain:

a) The hypotheses.

b) The appropriate test.

c) The standard error.

d) The test statistic.

e) The p-value.

f) The statistical decision.

g) The conclusion in context.

h) Two limitations of the investigation.

Fully Worked Answers

Answer 1: Formulating Hypotheses

Let μ be the population mean mass of cereal packets.

H₀: μ = 400 g.

H₁: μ < 400 g.

Answer: Left-tailed test, because the researcher suspects that the mean is below 400 g.

Answer 2: Two-Tailed Hypotheses

Let μ be the population mean mathematics score.

H₀: μ = 75.

H₁: μ ≠ 75.

Answer: Two-tailed test.

The school is interested in differences in either direction, not only higher or lower scores.

Answer 3: Calculating a z-Statistic

Given:

x̄ = 54.

μ₀ = 50.

σ = 20.

n = 100.

Step 1: Standard error

SE = 20/√100

= 2.

Step 2: z-statistic

z = (54 − 50)/2

= 2.00.

Answer: z = 2.00.

The sample mean is two standard errors above the null value.

Answer 4: Left-Tailed z-Test

Step 1: Hypotheses

H₀: μ = 100.

H₁: μ < 100.

Step 2: Standard error

SE = 8/√64

= 1 g.

Step 3: Test statistic

z = (98 − 100)/1

= −2.00.

Step 4: p-value

p = P(Z ≤ −2.00)

≈ 0.0228.

Step 5: Decision

0.0228 < 0.05.

Reject H₀.

Conclusion: There is statistically significant evidence that the population mean product mass is below 100 g.

Answer 5: Right-Tailed z-Test

Step 1: Hypotheses

H₀: μ = 100.

H₁: μ > 100.

Step 2: Standard error

SE = 20/√100

= 2.

Step 3: Test statistic

z = (105 − 100)/2

= 2.50.

Step 4: p-value

p = P(Z ≥ 2.50)

≈ 0.0062.

Step 5: Decision

0.0062 < 0.05.

Reject H₀.

Conclusion: There is statistically significant evidence that the population mean exceeds 100.

Answer 6: Interpreting a p-Value

p = 0.018.

This means that, assuming H₀ and the statistical model are correct, the probability of obtaining a test statistic at least as extreme as the observed one is 1.8%, in the direction or directions specified by H₁.

Since:

0.018 < 0.05,

reject H₀.

Answer: The result is statistically significant at the 5% level.

Answer 7: Different Significance Levels

p = 0.034.

a) α = 0.10

0.034 < 0.10.

Reject H₀.

b) α = 0.05

0.034 < 0.05.

Reject H₀.

c) α = 0.01

0.034 > 0.01.

Fail to reject H₀.

Conclusion: The decision depends on the significance threshold, even though the sample evidence remains unchanged.

Answer 8: One-Sample t-Test

Given:

x̄ = 78.

μ₀ = 75.

s = 8.

n = 16.

Step 1: Hypotheses

H₀: μ = 75.

H₁: μ > 75.

Step 2: Standard error

SE = 8/√16

= 2.

Step 3: Test statistic

t = (78 − 75)/2

= 1.50.

Step 4: Degrees of freedom

df = 15.

Step 5: p-value

For t = 1.50, df = 15, right-tailed:

p ≈ 0.077.

Step 6: Decision

0.077 > 0.05.

Fail to reject H₀.

Conclusion: There is insufficient statistical evidence that the population mean score exceeds 75 marks.

Answer 9: Population Proportion

Given:

x = 130.

n = 200.

Step 1: Hypotheses

H₀: p = 0.50.

H₁: p > 0.50.

Step 2: Sample proportion

p̂ = 130/200

= 0.65.

Step 3: Standard error

SE₀ = √(0.50 × 0.50/200)

≈ 0.03536.

Step 4: Test statistic

z = (0.65 − 0.50)/0.03536

≈ 4.24.

Step 5: p-value

The right-tailed p-value is approximately 0.000011.

Step 6: Decision

p < 0.05.

Reject H₀.

Conclusion: There is strong statistical evidence that more than 50% of students in the population prefer digital learning materials.

Answer 10: Defect Rate

Given:

x = 42.

n = 300.

Step 1: Hypotheses

H₀: p = 0.10.

H₁: p > 0.10.

Step 2: Sample proportion

p̂ = 42/300

= 0.14.

Step 3: Check approximation

n p₀ = 30.

n(1 − p₀) = 270.

Both exceed 10.

Step 4: Standard error

SE₀ = √(0.10 × 0.90/300)

= √0.0003

≈ 0.01732.

Step 5: Test statistic

z = (0.14 − 0.10)/0.01732

≈ 2.31.

Step 6: p-value

Right-tailed p ≈ 0.0105.

Step 7: Decision

0.0105 < 0.05.

Reject H₀.

Conclusion: There is statistically significant evidence that the population defect rate exceeds 10%.

Answer 11: Type I and Type II Errors

Suppose the hypotheses are:

H₀: μ = the safety threshold.

H₁: μ > the safety threshold.

Type I error: Concluding that the population mean pollutant concentration exceeds the threshold when the null value is actually correct.

This could lead to unnecessary warnings or interventions.

Type II error: Failing to detect that the population mean pollutant concentration exceeds the threshold when it actually does.

This could allow a genuine environmental problem to remain undetected.

Both errors matter, but their consequences may differ substantially.

Answer 12: Statistical Versus Practical Significance

Given:

p = 0.001.

Since:

0.001 < 0.05,

the result is statistically significant.

There is evidence that the intervention changes the population mean score, assuming the test and study design are valid.

However, the estimated increase is only 0.1 marks.

This may be too small to produce a meaningful educational benefit.

Conclusion: Statistical significance does not necessarily imply practical significance.

Answer 13: Confidence Interval Connection

95% CI = (48, 56).

Use a compatible two-sided test at α = 0.05.

a) μ₀ = 50

50 lies inside the interval.

Therefore:

Fail to reject H₀.

b) μ₀ = 45

45 lies outside the interval.

Therefore:

Reject H₀.

This illustrates the relationship between compatible two-sided hypothesis tests and confidence intervals.

Answer 14: Identifying Flawed Reasoning

The student's statement contains two errors.

Error 1: A p-value of 0.12 does not mean that H₀ has a 12% probability of being true.

It means that, assuming H₀ and the statistical model are correct, the probability of obtaining a result at least as extreme as the observed one is 12%.

Error 2: The correct decision is not to "accept H₀."

At α = 0.05:

0.12 > 0.05.

Therefore:

Fail to reject H₀.

There is insufficient evidence against the null hypothesis, but this does not establish that it is true.

Answer 15: Comprehensive Hypothesis Test

Given:

x̄ = 64 minutes.

μ₀ = 60 minutes.

s = 12 minutes.

n = 36.

α = 0.05.

a) Hypotheses

H₀: μ = 60.

H₁: μ ≠ 60.

The test is two-tailed because the question asks whether the population mean differs from 60 minutes.

b) Appropriate test

Use a one-sample t-test because the population standard deviation is unknown.

c) Standard error

SE = s/√n

= 12/√36

= 12/6

= 2 minutes.

Answer: SE = 2 minutes.

d) Test statistic

t = (64 − 60)/2

= 2.00.

Answer: t = 2.00.

e) p-value

Degrees of freedom:

df = 36 − 1 = 35.

Given:

Two-tailed p ≈ 0.053.

f) Statistical decision

0.053 > 0.05.

Fail to reject H₀.

g) Conclusion in context

At the 5% significance level, there is insufficient statistical evidence that the population mean mathematics assessment completion time differs from 60 minutes.

The sample mean is 64 minutes, but the observed difference is not statistically significant at the chosen threshold.

This does not prove that the true mean is exactly 60 minutes.

h) Two limitations

1. The sample may not fully represent the school population, especially if some student groups are underrepresented.

2. Completion times may be affected by differences in assessment difficulty, testing conditions, or student preparation.

A larger, representative sample and consistent testing conditions could improve the investigation.

Final Course Connections

Hypothesis testing connects probability distributions, sampling variability, standard errors, confidence intervals, and statistical inference.

It provides a structured way to evaluate claims using evidence rather than relying on individual observations or assumptions.

The essential process is to formulate appropriate hypotheses, select a valid statistical test, calculate the test statistic and p-value, make a decision using the significance level, and communicate the conclusion in the context of the original question.

The most important distinction is that hypothesis testing evaluates the strength of statistical evidence; it does not establish absolute certainty.

This completes Hypothesis Testing, covering all five learning targets with 15 worked examples, explanatory diagrams, an interactive z-test explorer, real-world applications, common mistakes, key terms, key takeaways, and 15 practice questions with fully worked answers.

 

Actual situation	Reject H₀	Fail to reject H₀
H₀ is true	Type I error	Correct decision
H₀ is false	Correct rejection	Type II error

Feature	Type I error	Type II error
Statistical decision	Reject H₀	Fail to reject H₀
Reality	H₀ is true	H₀ is false
Common description	False positive	False negative
Probability	α, or at most α	β
Main consequence	Detecting a nonexistent effect	Missing a real effect

Significance level	Interpretation under a true H₀
α = 0.10	Up to 10% probability of a Type I error
α = 0.05	Up to 5% probability of a Type I error
α = 0.01	Up to 1% probability of a Type I error

 

 

 

 

 

 

 

 

 

Characteristic	Population	Sample
Mean	μ	x̄
Standard deviation	σ	s
Proportion	p	p̂
Number of observations	N	n

Data characteristics	Recommended measure
Approximately symmetric, no extreme outliers	Mean
Strongly skewed	Median
Contains extreme outliers	Median
Categorical data	Mode or proportions
Approximately normal numerical data	Mean and standard deviation

 

 

Comparing examination scores using box plots

The box plots indicate that Group B generally has higher scores, but both groups have the same IQR:

IQR = 75 − 55 = 20 for Group A

IQR = 85 − 65 = 20 for Group B

The distributions also overlap.

Histograms

Histograms are useful for comparing distribution shapes, including skewness, clustering, and possible outliers.

When comparing histograms, use consistent bin widths and, where possible, the same horizontal scale.

If sample sizes differ, relative frequencies or densities may be more informative than raw frequencies.

Dot Plots

Dot plots are particularly useful for small datasets because they display individual observations.

Choosing a Graph

Comparing Population Means Using Statistical Inference

Descriptive statistics show what happened in our samples.

Inferential statistics help us decide what the sample results suggest about the populations.

One common method is the two-sample t-test.

The Two-Sample t-Test

A two-sample t-test examines whether there is statistical evidence that two population means differ.

Suppose we want to compare the average mathematics scores of students using two teaching methods.

We begin with two hypotheses.

Null hypothesis (H₀):

μ₁ = μ₂

The population means are equal.

Alternative hypothesis (H₁):

μ₁ ≠ μ₂

The population means are different.

The null hypothesis represents the condition against which we test the evidence.

Worked Example: Comparing Teaching Methods

Suppose two independent random samples produce these results:

We want to determine whether the difference between the population means is statistically significant.

Step 1: Calculate the observed difference

Difference = 78 − 72 = 6

Step 2: Calculate the standard error

For independent samples, Welch's two-sample t-test uses:

SE = √(s₁²/n₁ + s₂²/n₂)

Substitute:

SE = √(10²/40 + 12²/40)

SE = √(2.5 + 3.6)

SE = √6.1

SE ≈ 2.47

Step 3: Calculate the test statistic

t = (x̄₂ − x̄₁) / SE

t = 6 / 2.47

t ≈ 2.43

Step 4: Interpret the result

Using the Welch–Satterthwaite approximation, the degrees of freedom are approximately 75.5.

The two-sided p-value is approximately 0.018.

At the 5% significance level:

p ≈ 0.018 < 0.05

Therefore, we reject the null hypothesis.

There is statistically significant evidence that the population mean examination scores differ.

The sample results suggest Method B is associated with higher scores.

However, this does not establish that Method B caused the improvement. A causal conclusion would require an appropriate experimental design, such as random assignment to teaching methods.

Understanding Statistical Significance

Statistical significance describes whether a result is sufficiently inconsistent with the null hypothesis under a chosen statistical test.

A common significance level is:

α = 0.05

The decision rule is:

• If p ≤ 0.05, reject H₀.

• If p > 0.05, do not reject H₀.

A p-value is the probability, assuming the null hypothesis and the statistical model are true, of obtaining a test statistic at least as extreme as the one observed.

It is not the probability that the null hypothesis is true.

Statistical Significance Versus Practical Significance

A statistically significant result is not necessarily important in practical terms.

For example, a large study may find that one teaching method increases the average examination score by only 0.3 marks.

This difference could be statistically significant because the sample size is very large.

However, an improvement of 0.3 marks might have little educational importance.

Conversely, a small study might identify a potentially important difference but lack sufficient statistical power to establish significance.

Therefore, we should consider:

• The size of the difference.

• The uncertainty in the estimate.

• The sample size.

• The practical consequences.

• The quality of the study design.

Comparing Populations Using Confidence Intervals

A confidence interval provides a range of plausible values for a population parameter, based on the data and statistical model.

When comparing populations, we can construct a confidence interval for the difference between population means.

For independent samples, an approximate confidence interval is:

(x̄₂ − x̄₁) ± t* × SE

Here:

• x̄₂ − x̄₁ is the observed difference.

• t* is the appropriate critical value.

• SE is the standard error of the difference.

Worked Example

Using our previous teaching-method data:

Observed difference = 6

SE ≈ 2.47

For a 95% confidence interval with approximately 75.5 degrees of freedom:

t* ≈ 1.99

Margin of error:

ME = 1.99 × 2.47

ME ≈ 4.91

Confidence interval:

6 ± 4.91

Lower limit:

6 − 4.91 = 1.09

Upper limit:

6 + 4.91 = 10.91

Therefore, the approximate 95% confidence interval is:

(1.09, 10.91) marks

Interpretation

We are 95% confident that the population mean score for Method B is between approximately 1.09 and 10.91 marks higher than that for Method A, under the assumptions of the analysis.

Because the interval does not contain zero, it supports the conclusion that the population means differ at the 5% significance level in a corresponding two-sided test.

A 95% confidence level means that the method used to construct the interval would capture the true population difference in approximately 95% of repeated samples under the model assumptions.

It does not mean that 95% of individual students have scores within this interval.

Understanding Effect Size

Statistical significance indicates whether there is evidence of a difference.

Effect size describes how large that difference is.

One commonly used measure for numerical data is Cohen's d.

For two independent groups:

d = (x̄₂ − x̄₁) / sₚ

where sₚ is the pooled standard deviation.

For equal sample sizes:

sₚ = √[(s₁² + s₂²)/2]

Worked Example

Using the teaching-method data:

sₚ = √[(10² + 12²)/2]

sₚ = √122

sₚ ≈ 11.05

Cohen's d:

d = 6 / 11.05

d ≈ 0.54

This indicates that the difference between the sample means is approximately 0.54 pooled standard deviations.

A commonly used, approximate interpretation is:

These are guidelines rather than universal rules. The importance of an effect depends on its context.

In this example, the observed effect is approximately medium according to these conventional benchmarks.

Comparing Population Proportions

Not every comparison involves numerical averages.

Sometimes we want to compare proportions.

For example, suppose two schools use different revision programs.

School A:

80 students passed out of 100.

School B:

90 students passed out of 100.

Step 1: Calculate Sample Proportions

School A:

p̂₁ = 80/100 = 0.80

School B:

p̂₂ = 90/100 = 0.90

Step 2: Calculate the Difference

p̂₂ − p̂₁ = 0.90 − 0.80

Difference = 0.10

School B has a pass rate 10 percentage points higher.

Step 3: Calculate the Standard Error

For an estimated difference in proportions:

SE = √[p̂₁(1 − p̂₁)/n₁ + p̂₂(1 − p̂₂)/n₂]

SE = √[(0.80 × 0.20)/100 + (0.90 × 0.10)/100]

SE = √(0.0016 + 0.0009)

SE = 0.05

Step 4: Calculate an Approximate 95% Confidence Interval

Difference ± 1.96 × SE

0.10 ± 1.96 × 0.05

0.10 ± 0.098

Confidence interval:

(0.002, 0.198)

This approximate unpooled Wald interval suggests that School B's population pass rate could be between 0.2 and 19.8 percentage points higher than School A's.

However, the lower endpoint is very close to zero, and the Wald method can be inaccurate in some situations. A score-based confidence interval is often preferable for formal analysis.

When testing equality of proportions, a pooled standard error is commonly used instead.

Paired Versus Independent Comparisons

An important decision is whether the observations in two datasets are independent or paired.

Independent Samples

Independent samples contain observations from separate, unrelated individuals or units.

Examples:

• Comparing students from two different schools.

• Comparing patients randomly assigned to different treatments.

• Comparing products from two independent factories.

Common methods include:

• Independent-samples t-test.

• Welch's t-test.

• Mann–Whitney U test, for appropriate distributional comparisons.

• Two-proportion z-test.

Paired Samples

Paired samples contain related observations.

Examples:

• Students' scores before and after a revision course.

• Blood pressure measurements from the same patients before and after treatment.

• Measurements from matched pairs of participants.

For paired samples, we first calculate the difference within each pair.

Difference = After − Before

Then we analyse those differences.

Worked Example: Before-and-After Scores

Five students complete a mathematics revision program.

Mean difference:

d̄ = (8 + 5 + 8 + 8 + 6) / 5

d̄ = 7

The students improved by an average of 7 marks.

A paired t-test could investigate whether there is evidence of a nonzero population mean improvement, provided its assumptions are appropriate.

Using the differences:

5, 6, 8, 8, 8

Sample standard deviation:

s_d = √[Σ(d − 7)² / 4]

s_d = √(8/4)

s_d ≈ 1.41

Standard error:

SE = 1.41 / √5 ≈ 0.632

Test statistic:

t = 7 / 0.632 ≈ 11.07

Degrees of freedom = 4

This provides strong evidence of a positive mean change under the paired t-test assumptions.

However, a small sample of five students limits generalization, and without a control group we cannot conclude that the revision program alone caused the improvement.

Evaluating the Quality of Statistical Evidence

Statistical calculations are only one part of comparing populations.

We must also evaluate whether the evidence is reliable.

Sample Size

Larger samples generally produce more precise estimates, assuming appropriate sampling methods.

Small samples often have greater uncertainty and are more sensitive to unusual observations.

Representative Sampling

A sample should reflect the population being studied.

For example, comparing only the highest-achieving students in one school with randomly selected students in another would produce a biased comparison.

Random sampling can reduce selection bias, although it does not eliminate every source of bias.

Confounding Variables

A confounding variable is a factor related to both the group or exposure being studied and the outcome, potentially distorting their relationship.

For example, differences in examination scores might be associated with:

• Previous academic achievement.

• Access to tutoring.

• Student attendance.

• Socioeconomic conditions.

• Study time.

A comparison that ignores important confounding variables may lead to misleading conclusions.

Assumptions of Statistical Tests

Statistical methods depend on assumptions.

For example, a two-sample t-test generally requires:

• Independent observations within and between groups.

• Numerical outcomes.

• Approximately normal sampling distributions of the means, often supported by sufficiently large samples or suitable underlying distributions.

• No extreme data problems that invalidate the procedure.

Welch's t-test does not require equal population variances.

A paired t-test instead requires independent pairs and a suitable distribution of the within-pair differences.

Outliers

Outliers can substantially affect means and standard deviations.

When outliers are present, consider whether robust methods or comparisons of medians and distributions are more appropriate.

Multiple Comparisons

If researchers perform many statistical tests, the chance of obtaining at least one statistically significant result by chance increases.

Approaches such as Bonferroni correction or false discovery rate control can help address this problem.

Choosing an Appropriate Comparative Statistical Method

Different research questions require different statistical approaches.

The correct method depends on the research question, measurement scale, sampling design, assumptions, and type of comparison.

For example, the Mann–Whitney U test is often described as a test of medians, but that interpretation requires additional distributional assumptions. More generally, it compares relative ranks or stochastic ordering.

Communicating Statistical Findings

A good statistical report should explain both the calculations and their meaning.

A Clear Reporting Structure

When comparing populations, include:

1. Research question: What populations or groups are being compared?

2. Descriptive statistics: What are the sample sizes, means, medians, proportions, and measures of spread?

3. Statistical method: Which test or estimation procedure was used, and why?

4. Results: What were the estimated difference, confidence interval, test statistic, and p-value where appropriate?

5. Interpretation: What do the findings suggest in the context of the investigation?

6. Limitations: What factors could affect the reliability or generalizability of the conclusions?

Example Statistical Report

A comparison of examination scores was conducted using two independent samples of 40 students. Students taught using Method A had a mean score of 72 (SD = 10), while those taught using Method B had a mean score of 78 (SD = 12). A Welch two-sample t-test found a statistically significant difference between the group means, t ≈ 2.43, p ≈ 0.018. The estimated mean difference was 6 marks, with an approximate 95% confidence interval of 1.09 to 10.91 marks. These results suggest that Method B is associated with higher examination scores. However, further information about the sampling and allocation procedures is required before drawing causal conclusions.

Notice that this report does not simply state that Method B is better.

It provides numerical evidence, identifies the statistical method, interprets the results, and acknowledges limitations.

Real-World Applications

Medicine: Comparing Treatments

Researchers may compare blood pressure reductions in patients receiving two different treatments.

They can examine mean reductions, variability, confidence intervals, and possible adverse effects.

A statistically significant difference may not necessarily be clinically meaningful.

Education: Evaluating Teaching Strategies

Schools may compare examination results, attendance rates, or student progress under different teaching approaches.

Researchers must consider previous achievement, class composition, and other potential confounders.

Manufacturing: Comparing Product Quality

Two factories may produce components with the same average dimensions but different standard deviations.

The factory with lower variation may produce more consistently sized components.

Environmental Science: Comparing Pollution Levels

Scientists may compare average particulate pollution concentrations between two cities.

However, seasonal differences, weather, measurement methods, and monitoring locations can affect the comparison.

Business: Comparing Conversion Rates

An online business might compare the proportion of visitors who register for an account under two different website designs.

For example:

Design A: 120 registrations from 2,000 visitors = 6%

Design B: 165 registrations from 2,000 visitors = 8.25%

Observed difference:

8.25% − 6% = 2.25 percentage points

Relative increase:

(8.25 − 6) / 6 × 100% = 37.5%

This is a 37.5% relative increase in the observed registration rate, not an increase of 37.5 percentage points.

A properly randomized A/B test can help determine whether the design change caused an improvement.

Common Mistakes

Mistake 1: Comparing means without considering variation

Two groups can have different means but substantial overlap.

Always examine standard deviations, distributions, and uncertainty.

Mistake 2: Assuming statistical significance means practical importance

A very small difference may be statistically significant with a sufficiently large sample.

Consider effect size and context.

Mistake 3: Interpreting a nonsignificant result as proof of equality

Failing to reject the null hypothesis does not establish that two populations are identical.

The data may be insufficiently precise to detect an important difference.

Mistake 4: Using an independent-samples test for paired data

Before-and-after measurements from the same individuals are related.

A paired analysis is generally more appropriate.

Mistake 5: Ignoring sample bias

A large biased sample may produce a precise estimate of the wrong quantity.

Sampling quality is essential.

Mistake 6: Confusing percentage points and percentage changes

An increase from 20% to 30% is:

• 10 percentage points.

• A 50% relative increase.

These are different descriptions.

Mistake 7: Claiming causation from observational comparisons

A statistically significant association does not necessarily mean that one variable caused another.

Confounding variables and study design must be considered.

Mistake 8: Misinterpreting confidence intervals

A 95% confidence interval does not mean there is a 95% probability that the fixed population parameter lies in the calculated interval.

The 95% refers to the long-run coverage of the interval-building procedure.

Key Terms

• Population: The complete group of individuals or observations of interest.

• Sample: A subset of a population used for statistical analysis.

• Population parameter: A numerical characteristic of a population.

• Sample statistic: A numerical characteristic calculated from a sample.

• Mean: The arithmetic average of a dataset.

• Median: The middle value of an ordered dataset.

• Mode: The most frequently occurring value.

• Range: The difference between the maximum and minimum values.

• Interquartile range (IQR): The difference between the third and first quartiles.

• Standard deviation: A measure of variability around the mean.

• Sampling variability: Variation in sample statistics across different random samples.

• Standard error: The estimated standard deviation of a statistic's sampling distribution.

• Null hypothesis: A statistical claim used as the reference point for a hypothesis test.

• Alternative hypothesis: A competing statistical claim.

• p-value: The probability, under the null hypothesis and model, of obtaining a test statistic at least as extreme as the observed value.

• Significance level: A chosen threshold for deciding whether to reject the null hypothesis.

• Confidence interval: An interval estimate constructed using a method with a specified long-run coverage rate.

• Effect size: A measure of the magnitude of a statistical relationship or difference.

• Independent samples: Samples consisting of unrelated observations.

• Paired samples: Samples containing matched or repeated observations.

• Confounding variable: A variable that can distort the relationship between an exposure and an outcome.

• Statistical significance: Evidence against a null hypothesis according to a specified statistical test and significance level.

• Practical significance: The real-world importance of an observed effect.

Key Takeaways

• Comparing populations involves examining differences in central tendency, spread, shape, and other statistical characteristics.

• Sample statistics provide estimates of unknown population parameters.

• Means and medians describe typical values, while standard deviations and IQRs describe variation.

• Graphs such as box plots, histograms, and dot plots help reveal patterns and differences.

• Statistical tests assess how strongly the data contradict a specified null hypothesis.

• Confidence intervals provide information about both the estimated difference and its uncertainty.

• Effect sizes help describe the magnitude of differences.

• Independent and paired data require different analytical approaches.

• Sample size, bias, confounding variables, and statistical assumptions affect the reliability of conclusions.

• Clear statistical communication includes numerical evidence, contextual interpretation, and appropriate limitations.

Check Your Understanding

Question 1: Comparing Means

Two groups of students receive the following examination scores.

Group A: 60, 70, 75, 80, 85

Group B: 65, 75, 80, 85, 95

Calculate the mean for each group and determine which group has the higher mean.

Answer

Group A:

Mean = (60 + 70 + 75 + 80 + 85) / 5

Mean = 370 / 5 = 74

Group B:

Mean = (65 + 75 + 80 + 85 + 95) / 5

Mean = 400 / 5 = 80

Difference:

80 − 74 = 6

Group B has a mean score 6 marks higher than Group A.

This describes the observed samples but does not by itself establish a population difference.

Question 2: Comparing Variation

Two datasets have the following characteristics:

Dataset A: Mean = 50, SD = 4

Dataset B: Mean = 50, SD = 12

Which dataset has greater variation?

Answer

Dataset B has greater variation because its standard deviation is larger.

The values in Dataset B tend to be more widely dispersed around the mean.

Both datasets have the same mean, but their consistency differs.

Question 3: Interpreting Statistical Significance

A statistical test comparing two population means produces:

p = 0.032

The significance level is:

α = 0.05

What conclusion should be reached?

Answer

Since:

0.032 < 0.05

We reject the null hypothesis.

There is statistically significant evidence of a difference between the population means, assuming the test and its assumptions are appropriate.

This does not necessarily mean that the difference is large or practically important.

Question 4: Confidence Intervals

A study estimates that the mean difference between two populations is 8 units.

The standard error is 3 units.

Calculate an approximate 95% confidence interval using a critical value of 1.96.

Answer

Confidence interval:

Difference ± 1.96 × SE

= 8 ± 1.96 × 3

= 8 ± 5.88

Lower limit:

8 − 5.88 = 2.12

Upper limit:

8 + 5.88 = 13.88

95% confidence interval: (2.12, 13.88)

Because the interval does not include zero, it suggests a positive population mean difference under the normal approximation.

Question 5: Comparing Proportions

School A has 120 students who pass an examination out of 150 students.

School B has 135 students who pass out of 150.

Calculate the pass rates and their difference.

Answer

School A:

120 / 150 × 100% = 80%

School B:

135 / 150 × 100% = 90%

Difference:

90% − 80% = 10 percentage points

School B's observed pass rate is 10 percentage points higher.

Relative increase:

(90 − 80) / 80 × 100% = 12.5%

Therefore, School B's pass rate is 12.5% higher relative to School A's rate.

Question 6: Choosing a Statistical Method

A researcher measures the reaction times of 30 participants before and after drinking coffee.

Should the researcher use an independent-samples t-test or a paired t-test?

Answer

A paired t-test is appropriate because the same participants are measured twice.

The researcher should calculate each participant's change in reaction time and analyse the differences.

This accounts for the relationship between the two measurements.

Question 7: Evaluating Statistical Evidence

A study reports that a new teaching method improves examination scores by an average of 2 marks.

The difference is statistically significant, with p = 0.001.

Does this prove that the teaching method is educationally important?

Answer

No.

The p-value provides evidence against the null hypothesis but does not describe the educational importance of the difference.

A 2-mark improvement may or may not be meaningful, depending on the examination scale, consequences, costs, and other factors.

We should also examine the confidence interval, effect size, study design, and possible confounding variables.

Question 8: Comparing Two Populations

Two independent samples have the following characteristics:

Population A sample:

n₁ = 50, x̄₁ = 100, s₁ = 15

Population B sample:

n₂ = 50, x̄₂ = 108, s₂ = 20

Calculate the standard error of the difference and the Welch t-statistic.

Answer

Step 1: Calculate the standard error.

SE = √(s₁²/n₁ + s₂²/n₂)

SE = √(15²/50 + 20²/50)

SE = √(225/50 + 400/50)

SE = √(4.5 + 8)

SE = √12.5

SE ≈ 3.54

Step 2: Calculate the difference.

Difference = 108 − 100 = 8

Step 3: Calculate the t-statistic.

t = 8 / 3.54

t ≈ 2.26

This statistic can be used with the appropriate Welch degrees of freedom to calculate a p-value.

The result suggests that Population B may have a higher mean, but a complete inferential conclusion requires the test assumptions, degrees of freedom, and chosen significance level.

Question 9: Identifying Bias

A researcher compares the examination scores of students attending two schools.

For School A, the researcher selects only students enrolled in advanced mathematics classes.

For School B, the researcher selects students randomly from all mathematics classes.

Explain why the comparison may be misleading.

Answer

The samples were selected using different procedures.

School A's sample includes only advanced mathematics students, who may have higher achievement than the overall student population.

School B's sample represents a wider range of students.

Therefore, the observed difference may reflect selection bias rather than a genuine difference between the schools.

A better approach would be to use comparable sampling methods and account for relevant student characteristics.

Question 10: Communicating a Statistical Conclusion

A researcher compares the average daily exercise time of two groups.

Group A:

Mean = 35 minutes

Group B:

Mean = 42 minutes

The estimated difference is 7 minutes, with a 95% confidence interval from 2 to 12 minutes.

Write an appropriate statistical conclusion.

Answer

Group B exercised for an average of 7 minutes longer per day than Group A in the samples studied.

The 95% confidence interval suggests that the population mean difference is plausibly between 2 and 12 minutes, under the statistical model.

Because the confidence interval does not include zero, the results provide evidence of a difference between the population means.

However, the practical importance of this difference depends on the research context, and the reliability of the conclusion depends on the study design, sampling methods, and statistical assumptions.

Final Reflection

Comparing populations is one of the most important applications of statistics because it allows us to move beyond individual observations and investigate broader patterns.

A strong statistical comparison does not simply ask, "Which group has the higher average?"

It also asks:

• How large is the difference?

• How much variation exists within each group?

• How reliable is the evidence?

• Could the difference be explained by sampling variability or bias?

• Is the difference meaningful in the real world?

• What conclusions can we reasonably communicate?

By combining descriptive statistics, graphical analysis, statistical inference, and careful interpretation, we can make informed comparisons and communicate findings accurately and responsibly.

A confounding variable can create an association

Other Explanations for Correlation

Reverse causation

Researchers may incorrectly assume the direction of cause and effect.

For example, a study might find that students receiving extra tutoring have lower examination scores.

It would be misleading to conclude that tutoring causes poor performance.

Students who are already struggling may be more likely to receive tutoring.

Coincidence

Two variables may appear correlated simply by chance, especially when many possible relationships are examined.

Confounding variables

Other factors may influence both variables and create or distort the observed relationship.

Selection effects

The way participants are selected can create an apparent relationship that does not exist in the wider population.

Observational Studies and Experiments

The type of investigation strongly affects the conclusions researchers can draw.

Observational Studies

In an observational study, researchers measure variables without assigning participants to treatments or deliberately manipulating the exposure of interest.

For example, researchers might compare examination results among students who independently choose different revision methods.

Observational studies can identify associations and provide valuable evidence.

However, differences between groups may be caused by confounding variables.

Controlled Experiments

In a controlled experiment, researchers deliberately assign treatments or conditions and measure the resulting outcomes.

A randomized controlled experiment is particularly useful for investigating causation.

For example, researchers might randomly assign students to:

• Group A: Traditional revision materials.

• Group B: Interactive revision materials.

Both groups complete the same assessment after four weeks.

Random assignment helps balance both known and unknown confounding variables across groups, especially in sufficiently large studies.

Random Sampling Versus Random Assignment

These concepts serve different purposes.

An experiment can have strong internal validity but limited generalizability if its participants are not representative of the wider population.

Identifying Misleading Statistical Conclusions

Statistical claims can be misleading even when the underlying numbers are mathematically correct.

Misleading Percentages

Suppose an advertisement states:

"Our program increased student success by 100%!"

This sounds impressive.

However, imagine the pass rate increased from 2% to 4%.

Relative increase:

(4 − 2) / 2 × 100% = 100%

Absolute increase:

4% − 2% = 2 percentage points

The statement is mathematically correct but omits important context.

Always ask whether a reported percentage is an absolute change or a relative change.

Misleading Graph Scales

A graph with a truncated vertical axis can make a small difference appear dramatic.

Cherry-Picking Data

Cherry-picking occurs when someone selects only data that support a desired conclusion.

For example, a company might report sales growth during its best three months while ignoring the rest of the year.

A fair analysis should consider the relevant full dataset and explain any exclusions.

Small or Unrepresentative Samples

Suppose a company claims:

"Nine out of ten customers recommend our product."

If only ten customers were surveyed, the estimate is highly uncertain.

If those customers were selected from a group of enthusiastic supporters, the result may also be biased.

Confusing Statistical Significance With Importance

A study may report a statistically significant improvement of 0.2 marks on a 100-mark examination.

This result could be statistically detectable but educationally unimportant.

We must consider the magnitude and consequences of the effect.

Evaluating Evidence Using Descriptive Statistics

Descriptive statistics help summarize the observations in a study.

However, different statistics reveal different aspects of the data.

Worked Example: Comparing Two Teaching Programs

Suppose researchers collect the following results:

Step 1: Compare the means

78 − 72 = 6

Program B has a sample mean 6 marks higher.

Step 2: Compare the medians

79 − 74 = 5

Program B also has a higher median.

Step 3: Compare variation

Program A: SD = 8

Program B: SD = 15

Program B has substantially greater variability.

Step 4: Interpret the findings

Program B has higher typical scores in this sample, but its results are less consistent.

Some students may perform very well under Program B, while others may perform poorly.

Step 5: Evaluate limitations

Before concluding that Program B is more effective, we should ask whether students were randomly assigned, whether the assessments were comparable, and whether other differences between groups could explain the results.

Evaluating Evidence Using Confidence Intervals

A confidence interval provides an estimate of uncertainty around a population parameter.

Suppose a study estimates that a new teaching program improves scores by 5 marks.

Its 95% confidence interval for the mean improvement relative to a comparison program is:

(1, 9)

This suggests that the true mean difference is plausibly between 1 and 9 marks, assuming the statistical model is appropriate.

Because the interval does not contain zero, it provides evidence of a positive difference under a corresponding two-sided 5% test.

Now consider a second study:

Estimated difference = 5 marks

95% confidence interval:

(−3, 13)

Although both studies estimate the same difference, the second interval is much wider and includes zero.

The second study provides less precise evidence about the population difference.

Confidence Interval Width

Narrow confidence intervals generally indicate greater precision.

Wide confidence intervals indicate greater uncertainty.

Confidence interval width depends on factors including sample size, variability, confidence level, and the estimation method.

Evaluating Evidence Using Hypothesis Tests

A hypothesis test evaluates how compatible observed data are with a specified null hypothesis.

Worked Example: A New Learning Program

A study compares two teaching programs.

The null hypothesis is:

H₀: μ₁ = μ₂

The alternative hypothesis is:

H₁: μ₁ ≠ μ₂

The test produces:

p = 0.03

At a significance level of 0.05:

0.03 < 0.05

Therefore, the researchers reject the null hypothesis.

There is statistically significant evidence of a difference between the population means.

However, the p-value does not tell us:

• How large the difference is.

• Whether the difference is educationally important.

• Whether the study was biased.

• Whether one teaching method caused the difference.

• The probability that the null hypothesis is true.

A p-value should be interpreted alongside the effect size, confidence interval, research design, and data quality.

Understanding Study Limitations

Every statistical investigation has limitations.

Recognising these limitations is essential for evaluating the strength of its conclusions.

Sampling Limitations

A sample may not represent the intended population.

For example, results from students at one school may not generalize to students across an entire country.

Measurement Limitations

Measurements may be inaccurate or inconsistent.

Self-reported exercise time, for example, may differ from actual exercise time.

Confounding Variables

Unmeasured variables may influence the results.

An observational study linking screen time to examination performance might be affected by sleep, prior achievement, or home environment.

Small Sample Sizes

Small samples often produce imprecise estimates and can be strongly affected by unusual observations.

Short Study Duration

A short investigation may not reveal long-term effects.

For example, a teaching intervention may improve short-term test scores without improving long-term retention.

Missing Data

If participants drop out or fail to provide information, the remaining data may be unrepresentative.

Lack of Replication

A single study may produce an unusual result by chance.

Replication helps determine whether findings are consistent across different samples and settings.

Generalizability

A conclusion may apply only to the population, conditions, and setting studied.

Researchers should avoid extending results beyond the evidence.

Evaluating Evidence Using Effect Size

Statistical significance tells us whether the data provide evidence against a null hypothesis.

Effect size describes the magnitude of an observed difference or relationship.

For two independent groups, a common standardized effect size is Cohen's d.

d = (x̄₂ − x̄₁) / sₚ

Here, sₚ is the pooled standard deviation.

Worked Example

Suppose:

Group A mean = 70

Group B mean = 76

Pooled standard deviation = 12

Then:

d = (76 − 70) / 12

d = 6 / 12

d = 0.5

The observed mean difference is half a pooled standard deviation.

A conventional interpretation is that this represents a medium-sized effect.

However, whether the effect is practically important depends on the situation.

For example, a modest improvement in a medical treatment could be important if it prevents serious complications, while a similar standardized difference in another context might have little practical consequence.

Evaluating Multiple Statistical Studies

When several studies investigate the same question, their findings may differ.

For example:

All three studies estimate a positive improvement.

However, Study A has a wide confidence interval that includes zero, indicating substantial uncertainty.

Studies B and C provide more precise estimates, assuming the methods and data are reliable.

We should not automatically conclude that Study C is the best study simply because it has the largest sample.

Study quality, participant selection, measurement accuracy, confounding, and statistical methods must also be considered.

Systematic Reviews and Meta-Analyses

A systematic review uses a structured method to identify, evaluate, and summarize relevant studies.

A meta-analysis statistically combines results from multiple studies when appropriate.

These approaches can provide stronger evidence than a single study, but their reliability depends on the quality, comparability, and completeness of the included studies.

Combining several biased studies does not automatically eliminate bias.

Real-World Applications

Health and Medicine

Suppose a newspaper reports:

"People who drink green tea have a lower risk of heart disease."

Before accepting a causal interpretation, we should ask whether the study was observational or experimental, whether other lifestyle factors were considered, and how large and precise the estimated association was.

People who drink green tea may differ from other people in diet, exercise, income, or other characteristics.

Education

A school reports that examination scores increased after introducing a new online learning platform.

However, the increase might also reflect changes in teaching, examination difficulty, student composition, or additional revision time.

A comparison with an appropriate control group would provide stronger evidence about the platform's effect.

Advertising

A company advertises:

"95% of users experienced improvement."

Important questions include how improvement was measured, how many people participated, how participants were selected, and whether the study included a comparison group.

Environmental Science

Researchers report a correlation between air pollution and respiratory illness.

Statistical evidence can help identify an association, but causal interpretation requires careful consideration of other factors, exposure measurement, timing, and the broader body of scientific evidence.

Social Media and Technology

A study reports that students who spend more time on social media have lower average examination scores.

Possible explanations include:

• Social media use affecting study time.

• Students struggling academically spending more time online.

• Other factors influencing both social media use and academic performance.

The association alone cannot establish which explanation is correct.

A Practical Framework for Critically Evaluating Statistical Claims

When reading a statistical study or news article, use the following questions.

A study that performs well on these criteria is generally more trustworthy than one that ignores important methodological weaknesses.

However, no checklist can replace careful consideration of the specific research question and study design.

Common Mistakes

Mistake 1: Assuming correlation proves causation

A relationship between two variables does not automatically mean one causes the other.

Look for confounding variables, reverse causation, and alternative explanations.

Mistake 2: Believing a large sample eliminates all bias

Large samples can reduce random sampling error, but they cannot automatically correct systematic bias.

Mistake 3: Treating a small p-value as proof that a hypothesis is true

A small p-value provides evidence against the null hypothesis under the model assumptions. It does not prove the alternative hypothesis with certainty.

Mistake 4: Ignoring absolute differences when reading percentages

A 100% relative increase may represent a change from 1% to 2%, which is only one percentage point.

Mistake 5: Assuming a statistically significant result must be important

Statistical significance and practical importance are different concepts.

Mistake 6: Ignoring the scale of a graph

A truncated or inconsistent scale can exaggerate or conceal differences.

Mistake 7: Generalizing beyond the population studied

Results from a small or narrowly selected group may not apply to a wider population.

Mistake 8: Accepting conclusions without examining study design

Observational studies, randomized experiments, and surveys support different types of conclusions.

Mistake 9: Ignoring uncertainty

A point estimate without a confidence interval or another measure of uncertainty may give an incomplete picture.

Mistake 10: Selecting only evidence that supports a preferred conclusion

Reliable statistical interpretation considers the relevant evidence as a whole, including contradictory findings and limitations.

Key Terms

• Statistical investigation: A systematic process of collecting, analysing, and interpreting data to answer a research question.

• Population: The complete group of individuals or observations of interest.

• Sample: A subset of a population.

• Representative sample: A sample that adequately reflects the relevant characteristics of its target population.

• Bias: A systematic error that distorts a statistical estimate or conclusion.

• Selection bias: Bias caused by the process used to select participants.

• Response bias: Systematic distortion in participants' answers.

• Nonresponse bias: Bias arising when nonrespondents differ from respondents in relevant ways.

• Confirmation bias: The tendency to favour information supporting existing beliefs.

• Publication bias: Distortion arising when certain study results are more likely to be published.

• Correlation: A statistical relationship between variables.

• Causation: A relationship in which a change in one factor produces a change in another.

• Correlation coefficient: A numerical measure of the direction and strength of a relationship.

• Confounding variable: A variable that can distort an observed relationship between an exposure and an outcome.

• Observational study: A study in which researchers observe variables without assigning the exposure or treatment of interest.

• Controlled experiment: An investigation in which researchers assign conditions or treatments and compare outcomes.

• Random assignment: Allocation of participants to experimental groups using a random process.

• Random sampling: Selection of participants from a population using a random process.

• Null hypothesis: A statistical claim used as the reference point for a hypothesis test.

• p-value: The probability, under the null hypothesis and statistical model, of observing a test statistic at least as extreme as the one obtained.

• Statistical significance: Evidence against a null hypothesis according to a specified statistical test and significance level.

• Confidence interval: An interval estimate constructed using a procedure with a specified long-run coverage rate.

• Effect size: A measure of the magnitude of a difference or relationship.

• Practical significance: The real-world importance of an observed result.

• Generalizability: The extent to which findings apply beyond the individuals or conditions studied.

• Replication: Repeating a study to assess whether findings are consistent.

• Systematic review: A structured synthesis of research evidence.

• Meta-analysis: A statistical method for combining results from multiple studies.

Key Takeaways

• Statistical studies must be evaluated critically rather than accepted solely because they contain numerical evidence.

• Reliable investigations require clear research questions, appropriate samples, accurate measurements, and suitable analytical methods.

• Bias can arise during participant selection, data collection, analysis, and reporting.

• Correlation describes an association between variables, while causation requires evidence that one factor influences another.

• Randomized controlled experiments can provide stronger causal evidence than simple observational comparisons when appropriately designed.

• Misleading statistical claims may involve selective reporting, confusing percentages, inappropriate graphs, or unsupported causal conclusions.

• Confidence intervals, effect sizes, p-values, and descriptive statistics provide different but complementary information.

• Statistical significance does not necessarily imply practical importance.

• Every study has limitations that should be acknowledged when interpreting findings.

• Strong statistical conclusions are supported by reliable evidence, appropriate methods, and clear communication of uncertainty.

Check Your Understanding

Question 1: Identifying Bias

A researcher wants to determine whether students enjoy mathematics.

The researcher surveys 100 students attending an optional mathematics enrichment club.

Explain why the results may be biased.

Answer

The sample may suffer from selection bias because students attending a mathematics enrichment club are likely to have a greater interest in mathematics than the general student population.

Therefore, the sample may overestimate how much students enjoy mathematics.

A better approach would be to select a representative sample from the wider student population.

Question 2: Correlation Versus Causation

A study finds that students who spend more time studying tend to achieve higher examination scores.

The correlation coefficient is:

r = 0.82

What does this indicate? Does it prove causation?

Answer

A correlation coefficient of 0.82 indicates a strong positive linear relationship between study time and examination scores.

Students who study more tend to achieve higher scores in the observed data.

However, correlation does not establish causation.

Other factors, such as motivation, previous academic achievement, and access to learning resources, may influence both study time and performance.

A carefully designed experiment could provide stronger evidence about the causal effect of additional study time.

Question 3: Misleading Percentages

A company reports that its product increased customer satisfaction by 50%.

The original satisfaction rate was 40%.

Calculate the new satisfaction rate if the claim refers to a relative increase.

Answer

Increase = 50% of 40%

Increase = 0.50 × 40% = 20 percentage points

New satisfaction rate:

40% + 20% = 60%

The satisfaction rate increased from 40% to 60%.

This is a relative increase of 50%, but an absolute increase of 20 percentage points.

Question 4: Interpreting a p-value

A statistical test produces:

p = 0.008

The significance level is:

α = 0.05

What conclusion should be reached?

Answer

Since:

0.008 < 0.05

We reject the null hypothesis.

The data provide statistically significant evidence against the null hypothesis under the assumptions of the test.

However, the p-value does not establish the size, practical importance, or causal nature of the effect.

Question 5: Evaluating Study Design

A school introduces a new teaching program.

At the end of the year, average examination scores increase from 70 to 78.

The school concludes that the program caused the improvement.

Explain why this conclusion may be unjustified.

Answer

The improvement could be associated with several other factors, including:

• Changes in student ability.

• Different examination difficulty.

• Additional revision opportunities.

• Changes in teaching staff.

• Differences in attendance.

Without an appropriate comparison group or stronger experimental design, the school cannot confidently attribute the improvement to the new program.

The evidence establishes that average scores increased, but not necessarily why they increased.

Question 6: Interpreting Confidence Intervals

A study estimates that a new treatment reduces recovery time by an average of 3 days compared with an existing treatment.

The 95% confidence interval for the reduction is:

(−1, 7) days

What does this suggest?

Answer

The estimated mean reduction is 3 days, but the confidence interval includes zero.

It also includes negative values, which would indicate that the new treatment increases recovery time.

Therefore, the results do not establish a statistically significant difference at the 5% level under the corresponding two-sided test.

The evidence is compatible with a range of effects, from a small disadvantage to a substantial benefit.

More precise evidence may be needed.

Question 7: Comparing Statistical Evidence

Two studies investigate the same teaching method.

Study A:

n = 30

Estimated improvement = 5 marks

95% confidence interval = (−4, 14)

Study B:

n = 400

Estimated improvement = 5 marks

95% confidence interval = (3, 7)

Which study provides more precise evidence?

Answer

Study B provides more precise evidence because its confidence interval is narrower.

Study A interval width:

14 − (−4) = 18

Study B interval width:

7 − 3 = 4

Both studies estimate the same improvement, but Study B has substantially less uncertainty.

However, greater precision does not automatically mean lower bias. Study design and sampling quality must still be evaluated.

Question 8: Recognising a Confounding Variable

A study finds that people who own expensive sports equipment tend to exercise more frequently.

The researcher concludes that purchasing expensive equipment causes people to exercise more.

Identify a possible confounding variable.

Answer

One possible confounding variable is income.

People with higher incomes may be more likely to purchase expensive equipment and may also have greater access to fitness facilities or recreational opportunities.

Another possible factor is existing interest in fitness.

People who already exercise frequently may be more willing to purchase expensive equipment.

Therefore, the observed association does not prove that buying expensive equipment causes increased exercise.

Question 9: Evaluating Statistical Significance

A large study finds that a new educational program improves examination scores by an average of 0.4 marks out of 100.

The result is statistically significant, with:

p = 0.001

Should the school automatically adopt the program?

Answer

No.

Although the result is statistically significant, the average improvement is very small.

The school should also consider:

• The cost of the program.

• The time required to implement it.

• The confidence interval around the improvement.

• Whether the effect is consistent across different students.

• Whether the study supports a causal conclusion.

• Whether the improvement has meaningful educational consequences.

Statistical significance alone is insufficient for making an informed decision.

Question 10: Critical Evaluation of a Statistical Claim

A news article states:

"Scientists prove that drinking coffee makes students more intelligent."

The article describes an observational study of 200 university students.

Students who drank coffee regularly achieved an average test score of 78, while students who did not drink coffee achieved an average score of 73.

Identify at least four problems with the headline or conclusion.

Answer

Problem 1: Causation is not established.

The study is observational, so it cannot by itself prove that coffee causes higher intelligence or better performance.

Problem 2: Confounding variables may exist.

Coffee consumption may be associated with study habits, sleep, workload, age, or other factors.

Problem 3: Test scores do not directly measure intelligence.

The study measured examination performance, not necessarily intelligence.

Problem 4: Statistical uncertainty is not reported.

The article does not provide standard deviations, confidence intervals, or hypothesis test results.

Problem 5: Generalizability is uncertain.

The sample consists of 200 university students, and the sampling method is unknown.

Improved conclusion:

"In a study of 200 university students, regular coffee drinkers had an average test score 5 marks higher than non-coffee drinkers. However, the observational design does not establish that coffee consumption caused the difference."

This conclusion accurately describes the observed association without making unsupported causal claims.

Final Reflection

Statistics can reveal valuable information about the world, but numbers alone do not guarantee reliable conclusions.

A strong statistical thinker examines not only what the data show, but also how the data were obtained, what assumptions were made, how uncertainty was handled, and whether alternative explanations are possible.

When interpreting any statistical study, three questions are particularly important:

What does the evidence actually show?

What conclusions are justified by the research design and statistical analysis?

What limitations or uncertainties should be considered before accepting the findings?

Developing these habits allows us to distinguish strong evidence from misleading claims and make better-informed decisions in education, science, health, business, and everyday life.