3. Statistics in the Media

Statistics in the Media

Statistics appear constantly in news reports, advertisements, social media posts, political claims, health stories, sports coverage, and business reports.

A headline might claim:

"Sales increased by 50%."

"9 out of 10 people prefer Product A."

"People who do X are twice as likely to experience Y."

These statements sound precise because they contain numbers. However, the presence of a statistic does not automatically make a claim reliable.

Statistical information must be examined critically.

A useful question is:

What evidence actually supports this claim?

Statistical Claims

A statistical claim is a statement based on numerical data or statistical analysis.

Examples include:

• "72% of customers were satisfied."
• "The average household spends $400 per month on transportation."
• "Students who studied longer achieved higher test scores."
• "Unemployment decreased by 10%."
• "Four out of five surveyed participants preferred Option A."

Each statement raises additional questions.

For example:

Who was surveyed?

How many people were surveyed?

How were they selected?

How was the question worded?

What does "average" mean?

Compared with what?

Was the study observational or experimental?

Without this information, a statistic may be difficult to evaluate properly.

Statistics Can Be Correct but Misleading

A statistical statement does not have to be mathematically false to be misleading.

Suppose a company's sales increase from:

100 units to 150 units

The company can correctly say:

"Sales increased by 50%."

But suppose the previous year's sales had fallen from 1,000 to 100 because the company temporarily closed most of its stores.

The 50% increase is mathematically correct, but without context it may create a misleading impression about the company's overall performance.

This is why statistical literacy involves more than checking calculations.

We must also examine context.

Reading Beyond the Headline

Media headlines are designed to attract attention.

A headline might say:

"New Study Finds Exercise Doubles Your Chances of Success!"

Before accepting this claim, we would want to know:

• What was measured?
• How was "success" defined?
• How many people participated?
• How were participants selected?
• What was the original probability?
• Was the study experimental or observational?
• Could another variable explain the result?

A strong statistical reader separates the headline from the evidence.

Absolute Change and Relative Change

One common source of misleading statistical reporting is the difference between absolute change and relative change.

Suppose a rate increases from:

2% to 3%

The absolute increase is:

3% − 2% = 1 percentage point

The relative increase is:

(3 − 2) / 2 × 100%

= 50%

Therefore, both statements are mathematically correct:

"The rate increased by 1 percentage point."

and:

"The rate increased by 50%."

However, they create very different impressions.

Whenever a report gives a large percentage increase, ask:

Increase from what original value?

"Twice as Likely"

Suppose the probability of an event changes from:

1 in 10,000

to:

2 in 10,000

The second probability is twice the first.

So a headline could correctly state:

"Risk doubles."

But the absolute change is only:

1 additional case per 10,000

Both pieces of information matter.

A responsible statistical interpretation considers both relative risk and absolute risk.

Percent Versus Percentage Points

These terms are not interchangeable.

Suppose support for a proposal rises from:

40% to 50%

The increase is:

10 percentage points

The percentage increase relative to the original value is:

(50 − 40) / 40 × 100%

= 25%

Therefore:

40% → 50%

is a:

10 percentage-point increase

but a:

25% relative increase

Confusing these quantities can exaggerate or minimize changes.

Misleading Graphs

Graphs can communicate large amounts of information quickly.

They can also create misleading impressions through:

• truncated axes
• inconsistent scales
• distorted pictures
• unequal intervals
• inappropriate graph types
• missing labels
• selective time periods
• exaggerated three-dimensional effects

The numbers may even be correct while the visual presentation is misleading.

Truncated Axes

Consider two products with satisfaction scores:

Product A = 92%

Product B = 96%

The difference is:

4 percentage points

A bar graph with a vertical axis from:

0% to 100%

would show a relatively small difference.

But suppose the graph starts at:

90%

Now Product B's bar can appear several times taller than Product A's visible bar, even though the actual scores differ by only four percentage points. Truncating a bar chart's y-axis can therefore exaggerate visual differences.

Always examine the axis before interpreting the apparent size of a difference.

Does Every Graph Need to Start at Zero?

Not necessarily.

For bar graphs, starting at zero is particularly important because the length of each bar represents its magnitude.

For some line graphs, scientific measurements, or situations involving small but important changes, a non-zero starting point may be appropriate.

However, the scale must be clearly shown so readers understand what they are seeing.

The problem is not simply:

"The axis does not start at zero."

The better question is:

"Does the scale create a misleading impression of the size of the change?"

Changing the Scale

Even when an axis begins at zero, its maximum value can affect how dramatic a trend appears.

A graph ranging from:

0 to 100

can make a change from 40 to 50 appear substantial.

A graph ranging from:

0 to 10,000

would make the same change appear almost invisible.

Neither scale necessarily changes the underlying numbers.

But the visual impression changes.

This is why readers should examine:

• minimum values
• maximum values
• intervals
• units

rather than relying only on the visual shape.

Unequal Intervals

Suppose a time axis shows:

2020, 2021, 2022, 2026, 2027

but each year is placed the same distance apart.

This visually treats the four-year interval from 2022 to 2026 as if it were only one year.

That can distort the apparent rate of change.

Axis intervals should accurately represent the quantities they describe unless the graph clearly indicates otherwise.

Distorted Pictographs

Pictures can also exaggerate differences.

Suppose one company sold twice as many phones as another.

If an icon representing the second company is made:

twice as tall and twice as wide

its area becomes approximately:

2 × 2 = 4 times as large

The picture can therefore make a twofold difference look like a fourfold difference.

This violates the principle that the visual representation should be proportional to the data.

Three-Dimensional Graphs

Three-dimensional bars and pie charts may look impressive, but perspective can distort comparisons.

Objects closer to the viewer may appear larger.

Some parts of the graph may hide others.

A simple two-dimensional graph is often easier to interpret accurately.

Good statistical graphics should prioritize:

clarity over decoration.

Cherry-Picking Data

Cherry-picking occurs when only selected data are presented because they support a particular conclusion.

Suppose a company's profits are:

A report beginning with 2024 could say:

"Profits have risen by 40%!"

because:

(7 − 5) / 5 × 100% = 40%

That is true.

But looking from 2021 to 2026 shows:

$10 million → $7 million

which tells a very different story.

Choosing the starting and ending points can strongly affect the message.

Ask Why Those Dates Were Chosen

When examining a time-series graph, ask:

Why does the graph begin here?

Why does it end here?

What happened immediately before or after the displayed period?

A carefully selected time window can turn:

• a long-term decline into an apparent increase
• a temporary spike into apparent rapid growth
• a normal seasonal change into an apparent crisis

Context matters.

Mean, Median, and Mode

The word average can also be misleading because several different measures of centre exist.

The mean is:

mean = sum of values / number of values

The median is the middle value when the data are ordered.

The mode is the most frequently occurring value.

These can be very different when data are skewed.

An "Average" Income Example

Suppose five people earn:

$30,000
$32,000
$34,000
$36,000
$368,000

The mean is:

(30,000 + 32,000 + 34,000 + 36,000 + 368,000) / 5

= $100,000

The median is:

$34,000

A headline saying:

"Average income is $100,000"

could create a misleading impression of what a typical person earns.

The mean is mathematically correct, but it is strongly affected by the very high income.

Sample and Population

A population is the complete group we want information about.

A sample is the smaller group from which data are actually collected.

For example:

Population:

all 1,500 students at a school

Sample:

150 students surveyed

We often use the sample to make conclusions about the population.

For this to work well, the sample should reasonably represent the population.

Sample Size

Consider a headline:

"80% of teenagers prefer Product A."

That sounds impressive.

But suppose only:

5 teenagers

were surveyed.

Four preferred Product A.

Then:

4/5 × 100% = 80%

The percentage is correct, but the sample is far too small to support a strong claim about teenagers generally.

Always ask:

How large was the sample?

Representative Samples

A large sample can still be poor if it is not representative.

Suppose a school wants to determine whether students support replacing the football field with a larger library.

The school surveys:

500 members of the school football program

Even though 500 is a large sample, it may not represent the entire school.

The method used to choose participants matters as much as the number of participants.

Selection Bias

Selection bias occurs when the way participants are selected makes some members of the population more likely to be included than others.

For example, a newspaper asks:

"Should our city build more bicycle lanes?"

but surveys only people attending a cycling event.

The results may accurately describe those participants while poorly representing the city's entire population.

Voluntary Response Bias

Suppose a website asks:

"Click here to tell us whether you love or hate the new school timetable."

People with particularly strong opinions may be more likely to respond.

People who do not care may ignore the survey.

This creates voluntary response bias.

The results may therefore overrepresent extreme opinions.

Non-Response Bias

Suppose researchers randomly select 1,000 people for a survey, but only 200 respond.

The original selection may have been excellent.

However, if the 800 non-respondents differ systematically from the 200 respondents, the final results can still be biased.

This is called non-response bias.

Undercoverage

Undercoverage occurs when some groups in the population have little or no chance of being included.

For example, conducting an online-only survey may underrepresent people with limited internet access.

The resulting sample may not accurately represent the target population.

Wording Bias

The wording of a question can influence responses.

Compare:

Question A:
"Do you support the proposed school schedule?"

with:

Question B:
"Do you support the unnecessarily long school schedule that will reduce students' free time?"

Question B contains emotionally loaded language.

A reliable survey should use neutral wording whenever possible.

Leading Questions

A leading question encourages respondents toward a particular answer.

For example:

"Don't you agree that the new recycling program is an excellent idea?"

This wording suggests what the "correct" response should be.

A more neutral question would be:

"Do you support or oppose the new recycling program?"

Who Conducted the Study?

The source of statistical information matters.

Suppose a headline states:

"Study finds Product X is the most effective product available."

We should ask:

Who funded the study?

If the manufacturer of Product X funded it, that does not automatically make the research false.

But it does create a possible conflict of interest that should be considered when evaluating the evidence.

Primary and Secondary Sources

A primary source presents the original research or data.

Examples include:

• research papers
• government databases
• original survey reports
• official statistical publications

A secondary source reports or interprets information from somewhere else.

Examples include:

• newspaper articles
• television reports
• blogs
• social media posts

When possible, checking the original source helps determine whether the media report accurately represents the evidence.

Correlation and Causation

Suppose data show that people who exercise more tend to report better health.

This is a correlation or association.

It does not automatically prove:

Exercise caused every observed difference in health.

Other variables may also matter.

For example:

• age
• diet
• income
• existing health
• access to healthcare
• lifestyle

A correlation identifies a relationship.

A causal claim requires stronger evidence.

Confounding Variables

A confounding variable is another variable related to both variables being studied that can help explain an observed association.

For example, suppose:

ice-cream sales increase

and:

sunburn cases increase

at the same time.

It would be unreasonable to conclude that ice cream causes sunburn.

A likely confounding variable is:

hot, sunny weather

which can increase both ice-cream consumption and sun exposure.

Identifying possible confounding variables is an important part of critically evaluating statistical claims.

Observational Studies

In an observational study, researchers observe what happens without assigning treatments.

For example, researchers might compare people who voluntarily drink coffee with people who do not.

Observational studies can identify associations.

However, other differences between the groups may help explain the result.

Therefore, causal conclusions from observational studies require particular caution.

Experiments

In a well-designed experiment, researchers deliberately apply treatments and compare outcomes.

A strong experiment may include:

• random assignment
• a control group
• consistent procedures
• sufficient sample size
• blinding where appropriate

Random assignment helps distribute other variables between groups and can provide stronger evidence about cause and effect.

Statistical Significance

Media reports sometimes say a result was:

"statistically significant."

This does not necessarily mean:

• the effect was large
• the effect was important
• the result is guaranteed
• the study was perfectly designed

Statistical significance generally indicates that the observed result would be relatively unlikely under a specified null model, according to the statistical test and threshold being used.

A small effect can be statistically significant, especially with a large sample.

Therefore, readers should also consider the size and practical importance of the effect.

Practical Significance

Suppose a study finds that a new method improves average test scores from:

80.0% to 80.2%

With a sufficiently large sample, the difference might potentially be statistically detectable.

But we should still ask:

Is a 0.2 percentage-point improvement educationally important?

This is a question of practical significance.

Statistical evidence should be interpreted in context.

Margin of Error

Opinion polls often report a margin of error.

Suppose a poll reports:

Candidate A = 48%

Candidate B = 46%

with a margin of error of approximately:

±3 percentage points

The observed difference is only two percentage points.

It would therefore be inappropriate to confidently claim that Candidate A definitely has greater population support based on those estimates alone.

The estimates contain sampling uncertainty.

Sample Estimates Are Not Exact Population Values

If 52% of a sample chooses Option A, this does not necessarily mean exactly 52% of the entire population would choose it.

Samples vary.

Another randomly selected sample could produce:

50%

or:

54%

or another nearby result.

Statistical methods help us describe this uncertainty rather than pretending sample estimates are exact.

Missing Denominators

A statistic can sound dramatic when the denominator is omitted.

Consider:

"Cases increased by 100!"

That statement is difficult to interpret.

An increase from:

10 to 110

is very different from an increase from:

1,000,000 to 1,000,100.

Always ask:

100 out of how many?

Rates, percentages, and population sizes often provide necessary context.

Counts Versus Rates

Suppose City A has:

1,000 reported incidents

and City B has:

500 reported incidents

It might seem that City A has a larger problem.

But suppose:

City A population = 2,000,000

City B population = 100,000

Rates provide a more meaningful comparison.

City A:

1,000 / 2,000,000 × 100,000

= 50 incidents per 100,000 people

City B:

500 / 100,000 × 100,000

= 500 incidents per 100,000 people

City B actually has the much higher rate.

Percentages Without Counts

Consider:

"90% of users reported improvement."

This sounds strong.

But compare:

9 out of 10 users

with:

9,000 out of 10,000 users

Both are 90%.

The percentage is identical, but the amount of evidence is very different.

Good statistical reporting should provide enough information to understand both the proportion and the sample size.

Missing Data

Missing observations can also influence conclusions.

Suppose a survey is sent to 10,000 customers.

Only 500 respond.

Of those respondents:

90% report satisfaction.

A headline might state:

"90% of customers are satisfied."

But the evidence actually shows:

90% of the customers who responded were satisfied.

If satisfied customers were more likely to respond, the headline could overstate satisfaction among all customers.

Outliers and Selective Reporting

An outlier is an unusually high or low observation.

Outliers can affect statistics such as the mean.

However, researchers should not simply remove inconvenient observations because they weaken a desired conclusion.

There should be a legitimate methodological reason for excluding data.

Similarly, reporting only analyses that support a preferred conclusion can create reporting bias.

Graphs Should Show Data Honestly

A trustworthy graph should generally have:

• clearly labelled axes
• clearly stated units
• appropriate scales
• proportional visual elements
• meaningful categories
• consistent intervals
• an appropriate graph type
• enough context to understand the data

A graph should help the reader understand the data rather than manipulate the reader's impression.

Evaluating Statistical Evidence

When evaluating evidence in a media report, consider four broad areas.

Source

Who produced the information?

Data

Where did the numbers come from?

Method

How were the data collected and analysed?

Claim

Does the conclusion actually follow from the evidence?

A trustworthy-looking graph does not eliminate the need to ask these questions.

Questions About the Source

Ask:

• Who conducted the study?
• Who funded it?
• Is the original source available?
• Is the organization qualified to conduct the research?
• Is there a possible conflict of interest?
• Does the media report accurately represent the original findings?

The identity of the source does not automatically prove or disprove a claim, but it helps us assess credibility and possible bias.

Questions About the Sample

Ask:

• What population is being studied?
• How large is the sample?
• How were participants selected?
• Is the sample representative?
• Who was excluded?
• How many people failed to respond?

A precise calculation from a biased sample can still produce a misleading conclusion.

Questions About the Measurement

Ask:

• What exactly was measured?
• How was it measured?
• Were the questions neutral?
• Were measurements taken consistently?
• Were important variables ignored?
• Were definitions changed during the study?

If a report says:

"People are happier,"

we should ask:

How was happiness measured?

Questions About the Analysis

Ask:

• Was mean, median, or another statistic used?
• Why was that statistic appropriate?
• Are outliers affecting the result?
• Are percentages being reported without sample sizes?
• Are relative changes being presented without absolute changes?
• Are uncertainty and margins of error reported?
• Were only favourable results selected?

Questions About the Conclusion

Finally ask:

Does the evidence actually support the claim being made?

A study showing correlation should not automatically produce a headline claiming causation.

A small sample should not support sweeping conclusions about millions of people.

A small percentage-point change should not be presented as a dramatic transformation without context.

The strength of the conclusion should match the strength of the evidence.

A Media Claim Example

Suppose an advertisement states:

"Product A improves performance by 200%!"

Before accepting the claim, investigate.

Suppose the average score increased from:

1 point to 3 points

The increase is:

3 − 1 = 2

Relative increase:

2/1 × 100%

= 200%

The percentage is mathematically correct.

But we still need to know:

• What does one point represent?
• How important is a two-point difference?
• How many participants were tested?
• Was there a control group?
• Who funded the study?
• Was the result replicated?

The percentage alone is not enough.

Another Media Claim Example

Headline:

"Students Who Use Tablets Score 15% Higher."

Possible interpretation:

Tablet use improves scores.

But other explanations might include:

• schools with more resources provide more tablets
• those schools may also have smaller classes
• students may have greater access to tutoring
• family income may differ
• the students may have been different before tablet use began

Without stronger evidence, the safest conclusion may be:

Tablet use was associated with higher scores in the observed data.

That is different from saying:

Tablets caused the higher scores.

Social Media Statistics

Social media makes critical statistical thinking particularly important.

A graphic may contain:

• professional-looking charts
• percentages
• scientific terminology
• logos
• references to a "study"

None of these automatically guarantees reliability.

Before sharing a statistical claim, consider finding the original source and checking whether the graphic accurately represents it.

Comparing Two Reports

Imagine two headlines about the same data.

Headline A:
"Support for Proposal Surges by 25%!"

Headline B:
"Support Rises from 40% to 50%."

Both may describe the same change.

Headline A emphasizes the relative increase:

(50 − 40)/40 × 100%

= 25%

Headline B reports the actual percentages and reveals that the change was:

10 percentage points

The second headline provides more immediate context.

This does not make the first mathematically false, but it demonstrates how presentation influences interpretation.

A Critical Statistics Checklist

When encountering statistics in media, ask:

1. What exactly is the claim?
2. What is the original source?
3. Who collected the data?
4. Who funded the research?
5. What population is being discussed?
6. How large was the sample?
7. How was the sample selected?
8. Could selection, response, or wording bias be present?
9. Are percentages given with actual counts?
10. Are relative changes given with starting values?
11. Are the graph's axes and scales appropriate?
12. Has the time period been selectively chosen?
13. Is correlation being presented as causation?
14. Are uncertainty and limitations acknowledged?
15. Does the evidence actually justify the headline?

Common Mistakes

Assuming numbers cannot be misleading

Correct numbers can still be presented without important context.

Looking only at the graph's shape

Always read the axis labels, units, and scale.

Ignoring the starting value

A "100% increase" could mean 1 became 2.

Confusing percent with percentage points

An increase from 20% to 30% is 10 percentage points but a 50% relative increase.

Assuming a large sample is automatically representative

A large biased sample is still biased.

Ignoring non-response

The people who answer a survey may differ from those who do not.

Assuming correlation proves causation

Other variables or reverse causation may explain an association.

Assuming "statistically significant" means "important"

Statistical detectability and practical importance are different ideas.

Accepting "average" without asking which average

Mean and median can tell very different stories.

Ignoring the source

Always consider who collected, funded, analysed, and reported the data.

Treating one statistic as the whole story

Good statistical interpretation considers context, uncertainty, methodology, and alternative explanations.

Key Terms

Statistical claim — a statement supported or presented using numerical data.

Population — the entire group about which a conclusion is intended.

Sample — the subset of the population actually studied.

Representative sample — a sample whose relevant characteristics reasonably reflect the population.

Sampling bias — systematic distortion caused by how a sample is selected.

Selection bias — bias caused when some members of a population have a greater chance of being selected.

Voluntary response bias — bias caused when people choose whether to participate, often overrepresenting strong opinions.

Non-response bias — bias arising when people who do not respond differ meaningfully from those who do.

Undercoverage — failure to adequately include some groups in the population.

Wording bias — influence caused by the way survey questions are phrased.

Cherry-picking — selectively presenting data that support a desired conclusion while ignoring other relevant data.

Truncated axis — an axis that begins or ends in a way that can visually exaggerate or minimize differences.

Absolute change — the numerical difference between two values.

Relative change — change expressed relative to the original value.

Percentage point — the arithmetic difference between two percentages.

Correlation — a statistical association between variables.

Causation — a relationship in which a change in one variable produces a change in another.

Confounding variable — an additional variable that may help explain an observed association.

Margin of error — a measure describing sampling uncertainty around an estimate under a specified statistical method.

Statistical significance — evidence, under a statistical model and testing procedure, that an observed result would be relatively unlikely under the null hypothesis.

Practical significance — whether the size of an effect is meaningful in the real-world context.

Outlier — an observation unusually distant from the general pattern of the data.

Conflict of interest — a situation in which personal, financial, or organizational interests could influence research or reporting.

Key Takeaways

• Statistics in media should be examined critically rather than accepted simply because they contain numbers.
• A statistic can be mathematically correct while still creating a misleading impression.
• Always consider the source, sample, method, context, and conclusion.
• Relative changes should be interpreted alongside the original values and absolute changes.
• Percent changes and percentage-point changes are different.
• Truncated axes and distorted scales can exaggerate or minimize differences.
• Cherry-picked time periods can dramatically change the apparent story.
• Mean and median can produce very different descriptions of skewed data.
• A large sample is useful only if it reasonably represents the population.
• Selection bias, voluntary response bias, non-response bias, undercoverage, and wording bias can distort results.
• Correlation does not by itself establish causation.
• Confounding variables can create or help explain apparent relationships.
• Statistical significance does not necessarily mean that an effect is large or practically important.
• Percentages should be interpreted alongside sample sizes whenever possible.
• Strong statistical reporting acknowledges uncertainty and limitations.
• The strength of a media claim should not exceed the strength of the evidence supporting it.

Check Your Understanding

1. A headline states, "Sales increased by 100%!" Sales actually increased from 5 units to 10 units. Explain why the headline is mathematically correct but could still be misleading.
2. Support for a proposal rises from 30% to 36%. Calculate both the percentage-point increase and the relative percentage increase.
3. A bar graph compares values of 94 and 97 but begins its vertical axis at 93. Explain how this could affect a reader's interpretation.
4. A website reports that 80% of students prefer online homework, but only five students were surveyed. Evaluate the quality of this evidence.
5. A school surveys 600 members of its athletics teams about whether more money should be spent on sports facilities. Explain why the sample may be biased even though it is large.
6. Explain the difference between selection bias and voluntary response bias.
7. A survey asks, "Don't you agree that our excellent new school timetable should be kept?" Identify the problem with this question.
8. Five employees earn $30,000, $32,000, $35,000, $38,000, and $365,000. Determine the mean and median. Which would better represent a typical employee's income? Explain.
9. A report states that people who exercise more frequently have lower stress levels. Explain why this result alone does not prove that exercise caused the difference.
10. Ice-cream sales and sunburn cases both increase during summer. Identify a possible confounding variable and explain its role.
11. A poll reports Candidate A at 48% and Candidate B at 46%, with a margin of error of ±3 percentage points. Explain why the statement "Candidate A is definitely ahead" is too strong.
12. A company's profits fell for four consecutive years and then increased for two years. Its advertisement displays only the last two years and says, "Profits continue their strong upward trend." Identify the statistical reporting problem.
13. A study reports that a treatment increased success from 1% to 2%. Explain the difference between saying "success increased by one percentage point" and "success increased by 100%."
14. A social-media post says, "9 out of 10 people recommend this product." List at least four pieces of information you would want before deciding whether this is strong evidence.
15. Find or imagine a statistical claim from a news article, advertisement, or social-media post. Evaluate it by considering the source, sample, method, graph or numerical presentation, possible bias, uncertainty, and whether the conclusion is justified by the evidence.