Why Experimental Quality Matters

Collecting data does not automatically make an investigation scientifically useful. Scientists must also consider the quality of the evidence.

Two important ideas used to evaluate investigations are:

• Reliability – are the results consistent?
• Validity – does the investigation actually test what it is supposed to test?

A useful distinction is:

Reliability asks: "Can I get consistent results?"

Validity asks: "Am I actually testing the intended question?"

An investigation should ideally be both reliable and valid.

---

What Is Reliability?

Reliability describes the consistency of measurements or results when an investigation is repeated under comparable conditions.

Suppose a student measures the time for a toy car to travel down a ramp three times:

• Trial 1: 2.42 s
• Trial 2: 2.39 s
• Trial 3: 2.41 s

The results are very similar.

This suggests that the measurement is reasonably reliable.

If the results were:

• Trial 1: 1.84 s
• Trial 2: 3.21 s
• Trial 3: 2.46 s

there is much greater variation, so the results are less reliable.

---

Repeated Trials

One of the most important ways to improve reliability is to perform repeated trials.

Instead of collecting one measurement for each condition, scientists collect several.

For example:

The repeated measurements allow the investigator to determine whether the results are consistent.

---

Why Repetition Improves Reliability

Repeated trials help scientists:

• Identify unusual results.
• Estimate how much measurements naturally vary.
• Reduce the influence of random error when calculating a mean.
• Determine whether a pattern appears consistently.
• Detect problems with the experimental method.

One result could occur by chance.

A repeated pattern provides stronger evidence.

---

Calculating a Mean

Repeated measurements can often be summarized by calculating a mean.

Mean = sum of measurements / number of measurements

Suppose three measurements are:

12.4 cm, 12.7 cm and 12.5 cm

Mean = (12.4 + 12.7 + 12.5) / 3

Mean = 37.6 / 3

Mean = 12.53 cm

Depending on the measurement precision, this might reasonably be reported as:

12.5 cm

---

Identifying Anomalies

Repeated trials can also reveal an anomaly.

Consider:

4.21 s, 4.18 s, 7.96 s, 4.20 s

The measurement:

7.96 s

is very different from the others.

Possible explanations include:

• Incorrect timing.
• Equipment malfunction.
• Recording error.
• An inconsistent starting procedure.
• An unexpected event during the trial.

An anomalous result should be investigated rather than automatically removed.

---

Range and Reliability

The spread of repeated measurements can provide information about consistency.

Consider two sets of measurements.

Experiment A

10.2 cm, 10.3 cm, 10.2 cm, 10.4 cm

Range:

10.4 − 10.2 = 0.2 cm

Experiment B

8.7 cm, 10.9 cm, 9.4 cm, 11.2 cm

Range:

11.2 − 8.7 = 2.5 cm

Experiment A shows much less variation between repeated measurements.

This suggests greater repeatability under those conditions.

---

Repeating vs Replicating

There are two useful ways to check scientific findings.

Repetition

The same investigator repeats measurements using the same or very similar equipment and method.

Replication

Other investigators independently perform the investigation.

If different scientists using appropriate methods obtain similar findings, confidence in the result increases.

Replication is particularly important in scientific research.

---

Reliability in Biological Experiments

Living organisms naturally vary.

Two plants of the same species may:

• Grow at different rates.
• Have slightly different masses.
• Produce different numbers of leaves.
• Respond differently to environmental conditions.

Using only one organism can therefore produce misleading results.

Scientists often use larger sample sizes to reduce the influence of individual variation.

---

Sample Size

Imagine testing fertilizer using:

1 fertilized plant + 1 control plant

If the fertilized plant grows faster, this might simply reflect natural differences between the two plants.

A stronger investigation could use:

20 fertilized plants + 20 control plants

The larger sample provides more information about variation within each group.

Increasing sample size does not automatically make an investigation valid, but it can improve the reliability and precision of estimates.

---

Consistent Procedures

Reliability also depends on performing measurements consistently.

For example, when measuring plant height, students should:

• Measure from the same starting point.
• Use the same type of measuring instrument.
• Measure using the same definition of plant height.
• Record measurements consistently.

Changing the measurement method introduces additional variation.

---

What Is Validity?

Validity concerns whether an investigation and its measurements appropriately address the question being investigated.

Suppose the research question is:

How does fertilizer concentration affect plant growth?

A valid investigation should deliberately change fertilizer concentration while controlling other important factors that could affect growth.

If fertilizer concentration changes and water volume changes, the experiment cannot clearly isolate the effect of fertilizer.

This weakens the validity of the investigation.

---

Validity and Variables

A simple controlled experiment usually involves:

Independent Variable

The factor deliberately changed.

Dependent Variable

The factor measured.

Controlled Variables

Other relevant factors kept as constant as reasonably possible.

For a valid investigation:

the independent variable should be manipulated appropriately

and:

the dependent variable should actually measure the outcome relevant to the question

while:

important confounding variables should be controlled where possible.

---

Worked Example: Fertilizer

Question:

How does fertilizer concentration affect plant growth?

Independent Variable

Fertilizer concentration.

Dependent Variable

Change in plant height.

Controlled Variables

• Plant species.
• Starting plant size.
• Soil type.
• Pot size.
• Water volume.
• Light exposure.
• Temperature.
• Growing time.

If these variables are appropriately controlled, differences in growth can more reasonably be associated with fertilizer concentration.

---

An Invalid Comparison

Suppose:

Group A

Receives fertilizer, 200 mL water each day, and 10 hours of light.

Group B

Receives no fertilizer, 50 mL water each day, and 3 hours of light.

If Group A grows more, what caused the difference?

Possibilities include:

• Fertilizer.
• Water.
• Light.
• A combination of factors.

The investigation cannot clearly answer the original question.

Its validity is therefore weak.

---

Confounding Variables

A confounding variable is a factor that changes along with the independent variable and provides another possible explanation for the result.

For example:

Research question:

Does exercise increase heart rate?

Suppose:

• Group A exercises in a cool room.
• Group B exercises outdoors in extreme heat.

Temperature could also influence heart rate.

Temperature is therefore a potential confounding variable.

Confounding variables reduce the ability to determine cause and effect.

---

Measuring the Right Thing

Validity also depends on choosing an appropriate dependent variable.

Suppose the question is:

How does fertilizer affect plant growth?

The student measures:

soil temperature

Soil temperature might be interesting, but it does not directly measure plant growth.

More appropriate measurements might include:

• Change in plant height.
• Change in biomass.
• Leaf area.

The measurement must match the research question.

---

Operational Definitions

Sometimes scientists need to define exactly how something will be measured.

Suppose the investigation asks:

Does fertilizer improve plant growth?

What exactly does growth mean?

It could mean:

• Increase in height.
• Increase in mass.
• Increase in leaf area.
• Increase in root length.

An operational definition states exactly how a variable will be measured.

For example:

Plant growth will be measured as the increase in stem height in centimetres over 21 days.

This makes the method clearer and easier to reproduce.

---

Reliable but Not Valid

An investigation can be reliable without being valid.

Suppose a broken balance always adds 5 g.

A true mass of 50 g might repeatedly be measured as:

• 55.0 g
• 55.0 g
• 55.0 g

The results are extremely consistent.

They are therefore repeatable.

But the measurement is systematically wrong.

This demonstrates an important principle:

consistent results are not automatically correct or valid measurements.

---

Valid Idea but Unreliable Data

An investigation can also be appropriately designed but produce highly variable measurements.

Suppose students correctly investigate:

How does ramp height affect travel time?

They control all relevant variables but use inconsistent hand timing.

Results at one height are:

• 1.4 s
• 2.1 s
• 1.6 s
• 2.5 s
• 1.3 s

The design may address the intended question, but the measurements have poor reliability.

A better timing system could improve the investigation.

---

Accuracy, Reliability, and Validity

These ideas are related but should not be confused.

Accuracy

How close is a measurement to the accepted or true value?

Reliability

How consistent are measurements or findings?

Validity

Does the investigation appropriately measure or test what it claims to?

A high-quality investigation aims for all three where they are applicable.

---

Random Error

Random error causes unpredictable variation between measurements.

Examples include:

• Human reaction time.
• Slight temperature fluctuations.
• Small changes in equipment readings.
• Natural biological variation.

Random errors often reduce reliability.

Repeated measurements can help reveal and reduce their influence on calculated averages.

---

Systematic Error

Systematic error consistently shifts measurements in a particular direction.

Examples include:

• A balance that is incorrectly zeroed.
• A thermometer that consistently reads 2°C too high.
• Measuring length from the damaged end of a ruler.
• A sensor that has been incorrectly calibrated.

Repeating measurements does not usually solve systematic error.

If the same faulty instrument is used repeatedly, it may repeatedly produce the same incorrect result.

---

Worked Example: Reaction Time

A student investigates:

How does temperature affect reaction rate?

The student measures the time required for a reaction to reach a particular endpoint.

At 30°C:

• Trial 1: 42.1 s
• Trial 2: 41.8 s
• Trial 3: 42.3 s

These measurements are closely grouped.

This suggests good repeatability.

However, suppose the student also used twice the reactant concentration at 30°C compared with the other temperatures.

The comparison would not validly isolate the effect of temperature.

This demonstrates:

reliable measurements do not guarantee a valid experiment.

---

Worked Example: Toy-Car Ramp

Question:

How does ramp height affect the time required for a toy car to travel 2.0 m?

Improving Validity

Keep constant:

• Car.
• Ramp.
• Surface.
• Travel distance.
• Release procedure.

Only ramp height should deliberately change.

Improving Reliability

• Repeat each height several times.
• Use a consistent release mechanism.
• Use electronic timing if available.
• Calculate a mean.

The same experiment requires different strategies to improve reliability and validity.

---

Worked Example: Enzyme Investigation

Question:

How does temperature affect enzyme activity?

Students test:

10°C, 20°C, 30°C, 40°C and 50°C

To improve validity, they should control:

• pH.
• Enzyme concentration.
• Substrate concentration.
• Reaction volume.
• Measurement method.

To improve reliability, they should:

• Repeat each temperature.
• Use consistent equipment.
• Calculate appropriate averages.
• Investigate anomalous results.

---

Factors That Reduce Reliability

Reliability can be reduced by:

• Too few repeated trials.
• Very small sample sizes.
• Inconsistent procedures.
• Poorly controlled measurement conditions.
• Human reaction-time differences.
• Low-resolution equipment.
• Natural variation between organisms.
• Inconsistent equipment use.

These factors increase variation in results.

---

Factors That Reduce Validity

Validity can be reduced by:

• Changing several variables simultaneously.
• Failing to control important confounding variables.
• Measuring the wrong dependent variable.
• Using an inappropriate control.
• Using a method that does not address the research question.
• Systematic measurement problems.
• Bias in experimental design.
• Comparing groups under substantially different conditions.

---

Improving Reliability

Possible improvements include:

• Repeat each measurement.
• Increase sample size where appropriate.
• Use a standardized procedure.
• Use more precise or automated equipment.
• Keep measurement methods consistent.
• Train investigators to use equipment correctly.
• Calculate means where appropriate.
• Investigate anomalies.
• Replicate the experiment independently.

---

Improving Validity

Possible improvements include:

• Change only the intended independent variable in a simple fair test.
• Control relevant confounding variables.
• Include appropriate control conditions.
• Measure a dependent variable that directly addresses the question.
• Use an appropriate range of independent-variable values.
• Calibrate measuring equipment.
• Reduce sources of systematic bias.
• Define variables clearly.
• Ensure experimental and control groups are treated consistently apart from the intended difference.

---

Improving Both

Some changes can improve several aspects of experimental quality.

For example, using an electronic light gate instead of a handheld stopwatch in a motion experiment may:

• Reduce human reaction-time variation.
• Improve timing precision.
• Standardize measurement.
• Improve repeatability.

However, better equipment cannot fix every problem.

A highly precise sensor used in a badly designed experiment still produces evidence that may not answer the intended question.

---

Evaluating an Investigation

When evaluating reliability, ask:

Were trials repeated?

How similar were repeated measurements?

Was the sample size appropriate?

Were procedures performed consistently?

Are there unexplained anomalies?

When evaluating validity, ask:

Does the method actually test the research question?

Was the correct independent variable changed?

Was an appropriate dependent variable measured?

Were important confounding variables controlled?

Was an appropriate control used where necessary?

---

Example Evaluation

A student wants to determine whether salt concentration affects seed germination.

They use:

• 10 seeds with pure water.
• 10 seeds with 1% salt solution.
• 10 seeds with 2% salt solution.
• 10 seeds with 3% salt solution.

All seeds have:

• The same species.
• Similar environmental conditions.
• Equal solution volumes.
• The same temperature.
• The same growing medium.
• The same observation period.

The student records the percentage that germinate.

Validity

The design is reasonably valid for the stated question because salt concentration is deliberately changed while other important factors are controlled.

Reliability

Reliability could be improved by:

• Using more seeds.
• Repeating the entire investigation.
• Repeating the experiment with additional batches.

This separates two different questions about experimental quality.

---

Evaluating Conclusions

Scientists must also consider whether conclusions are supported by the evidence.

Suppose the results show:

A reasonable conclusion is:

Within the tested range and conditions, germination decreased as salt concentration increased.

A poor conclusion would be:

Salt always prevents all plants from germinating.

The data do not support such a broad claim.

Validity also involves making conclusions appropriate to the evidence collected.

---

Experimental Quality Is More Than Repetition

Repeating an experiment ten times does not automatically make it a good experiment.

Consider:

poor method × 10 repetitions

The same design problem may simply be repeated ten times.

A strong investigation needs:

appropriate design + suitable controls + reliable measurement + repetition + appropriate analysis

---

Reliability and Scientific Confidence

Science becomes stronger when findings can be reproduced by other investigators.

If many independent researchers obtain similar results using appropriate methods, confidence in the finding increases.

This is why scientists:

• Publish methods.
• Describe equipment.
• Report experimental conditions.
• Share data.
• Replicate studies.
• Compare results.

Science depends on evidence that can be checked.

---

Common Mistakes

"Reliable Means Correct"

Not necessarily.

A measurement can be consistently wrong.

"Valid Means the Results Are Similar"

Similarity between repeated results relates more directly to reliability.

Validity concerns whether the investigation appropriately addresses what it claims to test.

"Repeating an Experiment Makes It Valid"

Repetition improves information about reliability.

It does not automatically fix poor experimental design.

"One Measurement Is Enough"

One measurement provides little information about random variation.

"More Trials Fix Every Error"

Repeated trials can reduce the influence of random error but usually do not remove systematic error.

"A Large Sample Automatically Makes an Experiment Valid"

Large samples can be useful, but they cannot compensate for a fundamentally inappropriate method.

"Anomalies Should Always Be Deleted"

Anomalies should be investigated and handled with scientific justification.

"Controlling More Variables Always Improves the Experiment"

Only variables relevant to the question need to be controlled. Some variables cannot or should not be held constant depending on the investigation.

---

Check Your Understanding

1. Define reliability.

2. Define validity.

3. Explain the difference between reliability and validity.

4. Why are repeated trials important?

5. How can calculating a mean improve the usefulness of repeated measurements?

6. What is an anomaly?

7. Why should anomalies be investigated?

8. How can increasing sample size improve an investigation?

9. What is the difference between repetition and replication?

10. What is a confounding variable?

11. How can a confounding variable reduce validity?

12. Why must the dependent variable match the research question?

13. What is an operational definition?

14. Give an example of an investigation that is reliable but not valid.

15. Can an appropriately designed investigation still produce unreliable measurements? Explain.

16. What is the difference between random and systematic error?

17. Why can repeated trials reduce the influence of random error?

18. Why don't repeated trials usually remove systematic error?

19. Give three ways to improve reliability.

20. Give three ways to improve validity.

21. A student investigates temperature and reaction rate but changes concentration at each temperature. What problem does this create?

22. How could the investigation in Question 21 be improved?

23. A student obtains almost identical measurements using a balance that consistently reads 4 g too high. Are the measurements reliable? Explain.

24. Why doesn't highly precise equipment guarantee a valid experiment?

25. Explain why a scientifically strong investigation should consider both reliability and validity.

---

Key Terms

• Reliability – consistency of measurements or findings when an investigation is repeated under comparable conditions.
• Validity – extent to which an investigation and its measurements appropriately address the intended question.
• Repeated trial – repeating a measurement or procedure under comparable conditions.
• Replication – independent repetition of an investigation, often by other researchers.
• Mean – sum of values divided by the number of values.
• Range – difference between the highest and lowest values.
• Anomaly – result that differs noticeably from the general pattern.
• Sample size – number of experimental units or observations included in an investigation.
• Confounding variable – uncontrolled factor that provides an alternative explanation for a result.
• Operational definition – precise description of how a variable will be measured or identified.
• Random error – unpredictable variation between measurements.
• Systematic error – consistent error that shifts measurements in a particular direction.
• Accuracy – closeness of a measurement to an accepted or true value.
• Precision – closeness of repeated measurements to one another and the resolution of measurement.
• Control – reference condition used to help interpret results.

---

Key Takeaways

• Reliability and validity describe different aspects of experimental quality.
• Reliability concerns consistency.
• Validity concerns whether the investigation appropriately tests what it claims to test.
• Repeated trials help assess reliability.
• Means can reduce the influence of random variation.
• Larger samples can reduce the influence of individual variation.
• Anomalies should be investigated rather than automatically discarded.
• Consistent procedures improve reliability.
• Controlling important confounding variables improves validity.
• The dependent variable must appropriately represent what the investigation is trying to measure.
• Random errors mainly increase variation between measurements.
• Systematic errors can consistently shift results.
• Repetition does not normally eliminate systematic error.
• An experiment can be reliable but not valid.
• An appropriately designed experiment can still produce unreliable measurements.
• Better equipment can improve measurement quality but cannot rescue a fundamentally poor experimental design.
• Scientists improve experimental quality by evaluating both their methods and their data.

The central distinction is:

Reliability = Can I obtain consistent results?

Validity = Am I actually testing what I intend to test?