Experimental Design
| Site: | Young Education |
| Cursus: | Experimental Physics and Scientific Measurement |
| Boek: | Experimental Design |
| Afgedrukt door: | Guest user |
| Datum: | maandag, 9 november 2026, 16:42 |
1. Variables
Learning outcomes
- I can distinguish between independent, dependent, and controlled variables.
- I can identify variables in scientific investigations.
- I can design experiments that manipulate only one independent variable.
- I can explain why controlled variables are important.
- I can write clear hypotheses based on variables.
2. Controls
Learning outcomes
- I can explain the purpose of a control in an experiment.
- I can identify appropriate control groups or control conditions.
- I can distinguish between controlled variables and control groups.
- I can evaluate whether an experiment includes suitable controls.
- I can design experiments with effective controls.
3. Fair tests
Learning outcomes
- I can explain what makes an experiment a fair test.
- I can identify sources of bias in investigations.
- I can design experiments that produce reliable comparisons.
- I can control variables to improve fairness.
- I can evaluate whether an investigation is scientifically valid.
What Makes an Experiment a Fair Test?
A fair test is an investigation designed so that the effect of one factor can be studied without other factors interfering with the comparison.
In a simple controlled experiment:
one independent variable is deliberately changed
while:
other relevant variables are kept as constant as reasonably possible
and:
the dependent variable is measured consistently
A fair test allows scientists to make a meaningful comparison between different experimental conditions.
A useful idea is:
change one factor → control other relevant factors → measure the outcome → compare results
The Three Main Types of Variables
Understanding variables is essential for designing fair tests.
Independent Variable
The independent variable (IV) is the factor deliberately changed by the investigator.
For example:
Does water temperature affect how quickly sugar dissolves?
Independent variable:
water temperature
Dependent Variable
The dependent variable (DV) is the factor measured or observed.
In the sugar investigation:
Dependent variable:
time required for the sugar to dissolve
The dependent variable may change because the independent variable was changed.
Controlled Variables
Controlled variables are factors kept as constant as reasonably possible.
For the sugar investigation, these could include:
- Mass of sugar.
- Volume of water.
- Type of sugar.
- Container.
- Stirring method.
- Method used to determine when the sugar has dissolved.
These controls make the comparison more meaningful.
A Fair-Test Example
Suppose a student asks:
How does ramp height affect the speed of a toy car?
The student changes the ramp height and measures the car's speed.
Independent Variable
Ramp height.
Dependent Variable
Speed of the car.
Controlled Variables
- Same toy car.
- Same ramp.
- Same surface.
- Same travel distance.
- Same release point or release procedure.
- Same timing method.
If the student changes both the ramp height and the car, the comparison becomes less fair because either change could influence the result.
Why Controlled Variables Matter
Imagine testing whether temperature affects how quickly a tablet dissolves.
Trial A
Cold water + one tablet + 100 mL water.
Trial B
Hot water + two crushed tablets + 300 mL water.
This is not a fair comparison.
Several variables changed:
- Temperature.
- Number of tablets.
- Tablet particle size.
- Water volume.
If Trial B behaves differently, we cannot confidently determine which change was responsible.
Improving the Investigation
A better design would use:
Trial A
10°C water + one whole tablet + 100 mL water.
Trial B
30°C water + one whole tablet + 100 mL water.
Trial C
50°C water + one whole tablet + 100 mL water.
Now:
temperature changes
while the other important variables remain constant.
This provides a much fairer comparison.
Fair Does Not Mean Identical
A common misunderstanding is that everything must be kept the same.
If everything were identical, there would be no experiment.
One factor must deliberately change:
the independent variable
Therefore:
IV → changes
DV → measured
controlled variables → kept constant where relevant
Choosing an Appropriate Range
A good fair test usually investigates several useful values of the independent variable.
For example, instead of testing plant growth at only:
10°C and 30°C
a student might test:
10°C, 15°C, 20°C, 25°C and 30°C
This provides more information about the relationship between temperature and growth.
A suitable range should also be:
- Safe.
- Practical.
- Relevant to the research question.
Choosing Suitable Intervals
The values of the independent variable should allow useful comparisons.
Suppose a student investigates how ramp height affects travel time.
They could use:
5 cm, 10 cm, 15 cm, 20 cm, 25 cm
These regular intervals make it easier to identify patterns.
If the student used:
5 cm, 6 cm, 7 cm, 50 cm and 100 cm
the values might provide a less useful picture of the relationship, depending on the investigation.
Measuring the Dependent Variable
A fair test also requires the dependent variable to be measured consistently.
For example, if measuring plant growth, students should decide whether they will measure:
- Plant height.
- Change in height.
- Leaf number.
- Biomass.
They should then use the same method throughout the investigation.
Changing measurement methods halfway through an experiment reduces comparability.
Reliable Comparisons
A reliable comparison is one in which differences between groups are less likely to be caused by inconsistent procedures or random variation.
Ways to improve reliability include:
- Repeating measurements.
- Using multiple samples.
- Using consistent procedures.
- Using appropriate measuring equipment.
- Calculating averages where appropriate.
Fairness and reliability are related, but they are not exactly the same idea.
Repeated Trials
Suppose a student measures the time for a toy car to travel down a ramp.
First result:
1.42 s
Should the student immediately accept this as the correct value?
Probably not.
The student could repeat the measurement:
Trial 1: 1.42 s
Trial 2: 1.38 s
Trial 3: 1.40 s
Average:
(1.42 + 1.38 + 1.40) / 3 = 1.40 s
Repeated trials reduce the influence of random variation.
Identifying Anomalies
An anomaly is a result that differs noticeably from the general pattern.
Suppose the measurements are:
1.42 s, 1.38 s, 2.91 s, 1.40 s, 1.41 s
The value:
2.91 s
appears unusual compared with the others.
The student should investigate possible reasons.
Perhaps:
- The timer was started late.
- The car hit something.
- The car was released incorrectly.
- The measurement was recorded incorrectly.
An anomalous result should not simply be deleted because it is inconvenient.
There should be a scientific reason for excluding or repeating it.
Accuracy
Accuracy describes how close a measurement is to the accepted or true value.
For example, suppose a balance should read:
100.0 g
but consistently reads:
104.0 g
The measurements may be consistent but inaccurate.
Checking and calibrating equipment can improve accuracy.
Precision
Precision refers to how closely repeated measurements agree with one another and, in measurement contexts, the resolution with which measurements are reported.
For example:
20.1 cm, 20.2 cm, 20.1 cm
are closely grouped.
Measurements such as:
18.4 cm, 22.7 cm, 20.0 cm
show much greater spread.
A good investigation aims for measurements that are both accurate and appropriately precise.
Validity
An investigation is valid when its design and measurements allow it to answer the intended research question appropriately.
Suppose the question is:
How does light intensity affect photosynthesis?
If the experiment actually changes both light intensity and temperature, it becomes difficult to determine whether light intensity caused the observed change.
This weakens the validity of the investigation.
Reliability and Validity Are Different
A procedure can produce very consistent results without answering the intended question properly.
For example, students might repeat a poorly designed experiment ten times.
The results could be highly consistent.
But if two important variables were changed simultaneously, the investigation would still have weak validity for isolating the effect of either one.
Therefore:
repetition improves reliability
but:
repetition cannot automatically fix a poor experimental design
What Is Bias?
Bias is a systematic influence that pushes results or interpretations in a particular direction.
Bias can enter an investigation through:
- Sample selection.
- Measurement methods.
- Expectations.
- Experimental procedures.
- Data collection.
- Data interpretation.
Scientists try to identify and reduce sources of bias.
Selection Bias
Selection bias occurs when the sample being studied is not selected in a way that fairly represents the population relevant to the question.
Suppose a student asks:
What is the average height of students at our school?
They measure only members of the senior basketball team.
The sample is unlikely to represent the entire school.
A better approach would involve selecting students from the relevant school population using an appropriate sampling method.
Measurement Bias
Measurement bias can occur when a measuring method consistently favors particular results.
For example, suppose a ruler has a damaged end and every measurement begins at the damaged edge.
All measurements might be systematically too large or too small.
This produces a systematic error.
Observer Bias
Expectations can sometimes influence observations.
Suppose a researcher knows which plants received a new treatment and expects those plants to look healthier.
If "healthier" is judged subjectively, expectations might influence the assessment.
A more objective measurement could be:
- Plant height.
- Dry mass.
- Number of leaves.
In some investigations, researchers can also use blinding so the person making measurements does not know which treatment each sample received.
Confirmation Bias
Confirmation bias is the tendency to focus more strongly on evidence that supports an existing expectation while overlooking contradictory evidence.
For example, a student predicts that fertilizer will increase plant growth.
If one fertilized plant grows poorly, the student should not simply ignore it.
Scientists should examine all relevant data, including results that do not support the original hypothesis.
Random Error
Not every difference is caused by bias.
Random errors produce unpredictable variation in measurements.
Examples include:
- Human reaction time.
- Small differences between biological organisms.
- Slight fluctuations in temperature.
- Small variations in equipment readings.
Repeating measurements can help reduce the influence of random error on the final result.
Systematic Error
A systematic error shifts measurements consistently in the same direction.
For example:
A balance reads:
+2.0 g
even when empty.
Every measurement may therefore be approximately 2.0 g too high unless the balance is correctly zeroed.
Repeating the measurement does not remove this systematic error.
The equipment or method must be corrected.
Random vs Systematic Error
| Random Error | Systematic Error |
|---|---|
| Causes unpredictable variation | Shifts measurements consistently |
| Often reduced by repetition | Usually not removed by repetition |
| Can affect precision | Often affects accuracy |
| Example: reaction-time variation | Example: incorrectly calibrated balance |
Understanding this difference helps students improve experimental design.
Using a Control Group
Some fair tests require a control group.
Suppose the question is:
Does fertilizer increase plant growth?
Experimental Group
Plants receive fertilizer.
Control Group
Plants receive no fertilizer.
All other important conditions should be similar.
The control provides a baseline for comparison.
Control Group vs Controlled Variable
These are different concepts.
Control Group
A comparison group or condition.
Controlled Variable
A factor kept constant.
For the fertilizer investigation:
control group → plants receiving no fertilizer
while:
controlled variables → water, light, soil, temperature, plant species, pot size
Do not confuse the two.
Worked Example: Fertilizer and Plant Growth
Question:
How does fertilizer concentration affect bean plant growth?
Independent Variable
Fertilizer concentration.
Possible values:
0%, 1%, 2%, 3%, 4%
Dependent Variable
Change in plant height over 21 days.
Controlled Variables
- Bean variety.
- Starting size.
- Soil type and amount.
- Pot size.
- Water volume.
- Light exposure.
- Temperature.
- Growing time.
Control Condition
0% fertilizer
Reliability
Use several plants at each concentration.
Measurement
Measure plant height using the same method at regular intervals.
This is much stronger than using one plant for each concentration.
Worked Example: Ramp Investigation
Question:
How does ramp height affect the time a toy car takes to travel 2.0 m?
Independent Variable
Ramp height.
Dependent Variable
Travel time.
Controlled Variables
- Same car.
- Same ramp.
- Same surface.
- Same travel distance.
- Same release procedure.
- Same timing method.
Reliability
Repeat each height at least three times.
Processing Data
Calculate the mean time for each ramp height.
The results can then be graphed.
Worked Example: Temperature and Dissolving
Question:
How does water temperature affect the time required for sugar to dissolve?
Independent Variable
Water temperature.
Dependent Variable
Dissolving time.
Controlled Variables
- Mass of sugar.
- Volume of water.
- Sugar particle size.
- Container.
- Stirring method.
Useful Temperatures
For example:
10°C, 20°C, 30°C, 40°C, 50°C
Reliability
Repeat each temperature several times.
Worked Example: Exercise and Heart Rate
Question:
How does exercise duration affect heart rate immediately after exercise?
Independent Variable
Exercise duration.
Dependent Variable
Heart rate immediately after exercise.
Variables Requiring Control
Depending on the design:
- Type of exercise.
- Exercise intensity.
- Measurement method.
- Timing of heart-rate measurement.
- Recovery conditions.
Biological investigations can be difficult because people naturally differ.
Using repeated measurements or appropriate groups can help account for variation.
Evaluating an Investigation
Consider this investigation:
A student wants to determine whether light affects plant growth.
Plant A
- 8 hours of light per day.
- 100 mL water per day.
- 25°C.
Plant B
- 2 hours of light per day.
- 30 mL water per day.
- 15°C.
Is this a fair test?
No.
Three important variables changed:
- Light.
- Water.
- Temperature.
If Plant A grows more, the student cannot determine which variable caused the difference.
Improving the Investigation
A better design could use:
Plant A
8 hours of light.
Plant B
2 hours of light.
Both receive:
- 100 mL water per day.
- Same temperature.
- Same soil.
- Same plant species.
- Same pot size.
- Same experimental duration.
Now the major deliberate difference is:
light exposure
The comparison is much fairer.
Sample Size
Using only one experimental subject can produce misleading results.
Suppose:
one fertilized plant grows 12 cm
and:
one control plant grows 8 cm
Can we confidently conclude that fertilizer caused the difference?
Not necessarily.
The individual plants may naturally grow at different rates.
Using multiple plants in each condition provides stronger evidence.
Random Assignment
When experimental subjects naturally vary, random assignment can help reduce bias.
Suppose 40 similar plants are divided into:
- Fertilizer group.
- Control group.
If the researcher deliberately places the healthiest-looking plants in the fertilizer group, the experiment becomes biased.
Randomly assigning plants to groups helps reduce this problem.
Standardized Procedures
A standardized procedure means performing measurements and treatments consistently.
For example:
- Measure plants at the same time of day.
- Use the same measuring instrument.
- Use the same starting point.
- Use the same person or clearly defined method when appropriate.
- Follow the same procedure for every trial.
Standardization makes comparisons more reliable.
Designing a Fair Test
A useful procedure is:
Ask a Testable Question
For example:
How does temperature affect enzyme activity?
Identify the Independent Variable
Temperature.
Identify the Dependent Variable
Rate of enzyme activity.
Identify Controlled Variables
Keep relevant factors constant.
Select an Appropriate Range
Choose several suitable temperatures.
Decide How Measurements Will Be Made
Use a consistent and appropriate method.
Include a Control if Appropriate
Choose a meaningful baseline or comparison.
Repeat Trials
Collect enough data to assess variation.
Record All Results
Do not remove inconvenient results without justification.
Analyze the Data
Calculate averages where appropriate and look for patterns.
Evaluate the Method
Identify limitations and suggest realistic improvements.
Evaluating Scientific Validity
When evaluating an investigation, ask:
Does the experiment actually test the stated question?
Is only the intended independent variable deliberately changed?
Are important variables controlled?
Is the dependent variable measured appropriately?
Are measurements collected consistently?
Is the sample size adequate for the question?
Are trials repeated?
Are appropriate controls included where needed?
Could bias have influenced the results?
Are the conclusions supported by the data?
A scientifically strong investigation should survive this type of questioning.
Correlation Is Not Automatically Causation
Sometimes two variables change together.
This is called a correlation.
However, observing a correlation does not automatically prove that one variable caused the other.
For example, suppose researchers observe that two variables increase at the same time.
Possible explanations include:
- Variable A causes B.
- Variable B causes A.
- A third variable influences both.
- The relationship occurred by chance.
Well-controlled experiments can provide stronger evidence for causal relationships.
Fair Tests in Biology
Biological experiments can be challenging because organisms naturally vary.
Scientists may improve comparisons by:
- Increasing sample size.
- Randomly assigning organisms.
- Using organisms of similar age or size where appropriate.
- Maintaining consistent environmental conditions.
- Repeating experiments.
Natural variation cannot always be eliminated, but it can be managed.
Fair Tests in Chemistry
Chemistry experiments may require control of:
- Temperature.
- Concentration.
- Volume.
- Mass.
- Pressure.
- Surface area.
- Reaction time.
For example, when investigating concentration and reaction rate, temperature should normally remain constant unless temperature is the variable being investigated.
Fair Tests in Physics
Physics investigations may require control of:
- Mass.
- Distance.
- Angle.
- Surface.
- Starting position.
- Applied force.
- Equipment.
- Measurement method.
For example, when investigating ramp height and car speed, using a different car for every height could introduce another variable.
Fair Does Not Mean Perfect
Real experiments are rarely perfect.
Scientists may not be able to control every possible factor.
Instead, they try to:
- Control the most important variables.
- Measure carefully.
- Reduce bias.
- Repeat measurements.
- Quantify uncertainty where appropriate.
- Identify limitations.
- Improve future investigations.
Scientific validity comes from thoughtful experimental design and transparent evaluation, not from pretending uncertainty does not exist.
Common Mistakes
Changing More Than One Independent Variable
This makes cause and effect difficult to determine.
Confusing the Independent and Dependent Variables
Remember:
IV → deliberately changed
DV → measured
Confusing Control Groups and Controlled Variables
A control group provides a comparison.
A controlled variable is kept constant.
Using Only One Trial
One result may be affected by random variation.
Using an Extremely Small Sample
Individual differences can strongly influence results.
Ignoring Anomalies
Unexpected results should be investigated rather than automatically deleted.
Thinking Repetition Fixes Systematic Error
Repeating a biased measurement usually reproduces the same bias.
Choosing Samples That Support the Hypothesis
Samples should be selected using an appropriate method rather than chosen to produce a desired outcome.
Changing the Measurement Method
The dependent variable should be measured consistently.
Claiming More Than the Evidence Shows
A conclusion should match the strength and limitations of the data.
Check Your Understanding
1. What is a fair test?
2. What is the independent variable?
3. What is the dependent variable?
4. What is a controlled variable?
5. Why should relevant variables be controlled?
6. Why does a fair test deliberately change one variable?
7. Identify the IV, DV, and three controlled variables in an investigation of ramp height and toy-car travel time.
8. Why are repeated trials useful?
9. What is an anomaly?
10. Should an anomalous result automatically be deleted? Explain.
11. What is the difference between accuracy and precision?
12. What is the difference between random and systematic error?
13. Why doesn't repeating an experiment necessarily remove systematic error?
14. What is experimental bias?
15. Give an example of selection bias.
16. How can observer bias influence results?
17. What is confirmation bias?
18. What is the difference between reliability and validity?
19. Why might a larger sample improve a biological investigation?
20. How can random assignment reduce bias?
21. Why are standardized procedures important?
22. A student changes both temperature and concentration while measuring reaction rate. Why is this not a fair test of temperature alone?
23. How could the investigation in Question 22 be improved?
24. Why does correlation not necessarily demonstrate causation?
25. Describe how you would design a fair test investigating how light exposure affects plant growth.
Key Terms
- Fair test – investigation designed to make meaningful comparisons by deliberately changing an independent variable while controlling other relevant factors.
- Independent variable – factor deliberately changed by the investigator.
- Dependent variable – factor measured or observed.
- Controlled variable – factor kept constant during an investigation.
- Control group – comparison group used to help interpret experimental results.
- Validity – extent to which an investigation appropriately answers the question it was designed to test.
- Reliability – extent to which measurements or findings are consistent when repeated under comparable conditions.
- Accuracy – closeness of a measurement to the accepted or true value.
- Precision – closeness of repeated measurements to one another and the resolution with which measurements are expressed.
- Bias – systematic influence that can push results or interpretations in a particular direction.
- Random error – unpredictable variation between measurements.
- Systematic error – consistent error that shifts measurements in a particular direction.
- Anomaly – result noticeably different from the general pattern.
- Sample size – number of experimental units or observations included.
- Random assignment – assigning experimental units to groups using chance.
- Standardized procedure – method performed consistently across trials or groups.
- Confounding variable – uncontrolled factor that could provide an alternative explanation for an observed result.
- Correlation – relationship in which variables change together.
- Causation – relationship in which a change in one factor produces a change in another.
Key Takeaways
- A fair test allows meaningful comparisons between experimental conditions.
- The independent variable is deliberately changed.
- The dependent variable is measured.
- Other relevant variables should be controlled.
- Fair testing does not mean keeping everything identical—the independent variable must change.
- Appropriate controls can provide useful baselines.
- Repeated trials help assess random variation and improve reliability.
- Larger sample sizes can reduce the influence of individual variation.
- Random assignment can reduce selection bias.
- Standardized procedures make comparisons more consistent.
- Random and systematic errors are different and require different solutions.
- Repetition does not automatically correct systematic error.
- Bias can enter an investigation through sampling, measurement, procedures, or interpretation.
- Scientists should examine evidence that both supports and contradicts their hypotheses.
- Reliability and validity are related but different.
- A reliable investigation is not necessarily valid if it does not properly test the intended question.
- A strong scientific conclusion should be supported by the data and acknowledge important limitations.
- A useful summary of fair testing is:
change one relevant factor → control other important factors → measure consistently → repeat → compare → evaluate
4. Reliability and validity
Learning outcomes
- I can distinguish between reliability and validity.
- I can explain how repeated trials improve reliability.
- I can evaluate whether experimental methods are valid.
- I can identify factors that reduce reliability or validity.
- I can suggest improvements to increase experimental quality.
Why Experimental Quality Matters
Collecting data does not automatically make an investigation scientifically useful. Scientists must also consider the quality of the evidence.
Two important ideas used to evaluate investigations are:
- Reliability – are the results consistent?
- Validity – does the investigation actually test what it is supposed to test?
A useful distinction is:
Reliability asks: "Can I get consistent results?"
Validity asks: "Am I actually testing the intended question?"
An investigation should ideally be both reliable and valid.
What Is Reliability?
Reliability describes the consistency of measurements or results when an investigation is repeated under comparable conditions.
Suppose a student measures the time for a toy car to travel down a ramp three times:
- Trial 1: 2.42 s
- Trial 2: 2.39 s
- Trial 3: 2.41 s
The results are very similar.
This suggests that the measurement is reasonably reliable.
If the results were:
- Trial 1: 1.84 s
- Trial 2: 3.21 s
- Trial 3: 2.46 s
there is much greater variation, so the results are less reliable.
Repeated Trials
One of the most important ways to improve reliability is to perform repeated trials.
Instead of collecting one measurement for each condition, scientists collect several.
For example:
| Ramp Height | Trial 1 | Trial 2 | Trial 3 |
|---|---|---|---|
| 10 cm | 2.51 s | 2.47 s | 2.50 s |
| 20 cm | 2.13 s | 2.10 s | 2.12 s |
| 30 cm | 1.87 s | 1.84 s | 1.86 s |
The repeated measurements allow the investigator to determine whether the results are consistent.
Why Repetition Improves Reliability
Repeated trials help scientists:
- Identify unusual results.
- Estimate how much measurements naturally vary.
- Reduce the influence of random error when calculating a mean.
- Determine whether a pattern appears consistently.
- Detect problems with the experimental method.
One result could occur by chance.
A repeated pattern provides stronger evidence.
Calculating a Mean
Repeated measurements can often be summarized by calculating a mean.
Mean = sum of measurements / number of measurements
Suppose three measurements are:
12.4 cm, 12.7 cm and 12.5 cm
Mean = (12.4 + 12.7 + 12.5) / 3
Mean = 37.6 / 3
Mean = 12.53 cm
Depending on the measurement precision, this might reasonably be reported as:
12.5 cm
Identifying Anomalies
Repeated trials can also reveal an anomaly.
Consider:
4.21 s, 4.18 s, 7.96 s, 4.20 s
The measurement:
7.96 s
is very different from the others.
Possible explanations include:
- Incorrect timing.
- Equipment malfunction.
- Recording error.
- An inconsistent starting procedure.
- An unexpected event during the trial.
An anomalous result should be investigated rather than automatically removed.
Range and Reliability
The spread of repeated measurements can provide information about consistency.
Consider two sets of measurements.
Experiment A
10.2 cm, 10.3 cm, 10.2 cm, 10.4 cm
Range:
10.4 − 10.2 = 0.2 cm
Experiment B
8.7 cm, 10.9 cm, 9.4 cm, 11.2 cm
Range:
11.2 − 8.7 = 2.5 cm
Experiment A shows much less variation between repeated measurements.
This suggests greater repeatability under those conditions.
Repeating vs Replicating
There are two useful ways to check scientific findings.
Repetition
The same investigator repeats measurements using the same or very similar equipment and method.
Replication
Other investigators independently perform the investigation.
If different scientists using appropriate methods obtain similar findings, confidence in the result increases.
Replication is particularly important in scientific research.
Reliability in Biological Experiments
Living organisms naturally vary.
Two plants of the same species may:
- Grow at different rates.
- Have slightly different masses.
- Produce different numbers of leaves.
- Respond differently to environmental conditions.
Using only one organism can therefore produce misleading results.
Scientists often use larger sample sizes to reduce the influence of individual variation.
Sample Size
Imagine testing fertilizer using:
1 fertilized plant + 1 control plant
If the fertilized plant grows faster, this might simply reflect natural differences between the two plants.
A stronger investigation could use:
20 fertilized plants + 20 control plants
The larger sample provides more information about variation within each group.
Increasing sample size does not automatically make an investigation valid, but it can improve the reliability and precision of estimates.
Consistent Procedures
Reliability also depends on performing measurements consistently.
For example, when measuring plant height, students should:
- Measure from the same starting point.
- Use the same type of measuring instrument.
- Measure using the same definition of plant height.
- Record measurements consistently.
Changing the measurement method introduces additional variation.
What Is Validity?
Validity concerns whether an investigation and its measurements appropriately address the question being investigated.
Suppose the research question is:
How does fertilizer concentration affect plant growth?
A valid investigation should deliberately change fertilizer concentration while controlling other important factors that could affect growth.
If fertilizer concentration changes and water volume changes, the experiment cannot clearly isolate the effect of fertilizer.
This weakens the validity of the investigation.
Validity and Variables
A simple controlled experiment usually involves:
Independent Variable
The factor deliberately changed.
Dependent Variable
The factor measured.
Controlled Variables
Other relevant factors kept as constant as reasonably possible.
For a valid investigation:
the independent variable should be manipulated appropriately
and:
the dependent variable should actually measure the outcome relevant to the question
while:
important confounding variables should be controlled where possible.
Worked Example: Fertilizer
Question:
How does fertilizer concentration affect plant growth?
Independent Variable
Fertilizer concentration.
Dependent Variable
Change in plant height.
Controlled Variables
- Plant species.
- Starting plant size.
- Soil type.
- Pot size.
- Water volume.
- Light exposure.
- Temperature.
- Growing time.
If these variables are appropriately controlled, differences in growth can more reasonably be associated with fertilizer concentration.
An Invalid Comparison
Suppose:
Group A
Receives fertilizer, 200 mL water each day, and 10 hours of light.
Group B
Receives no fertilizer, 50 mL water each day, and 3 hours of light.
If Group A grows more, what caused the difference?
Possibilities include:
- Fertilizer.
- Water.
- Light.
- A combination of factors.
The investigation cannot clearly answer the original question.
Its validity is therefore weak.
Confounding Variables
A confounding variable is a factor that changes along with the independent variable and provides another possible explanation for the result.
For example:
Research question:
Does exercise increase heart rate?
Suppose:
- Group A exercises in a cool room.
- Group B exercises outdoors in extreme heat.
Temperature could also influence heart rate.
Temperature is therefore a potential confounding variable.
Confounding variables reduce the ability to determine cause and effect.
Measuring the Right Thing
Validity also depends on choosing an appropriate dependent variable.
Suppose the question is:
How does fertilizer affect plant growth?
The student measures:
soil temperature
Soil temperature might be interesting, but it does not directly measure plant growth.
More appropriate measurements might include:
- Change in plant height.
- Change in biomass.
- Leaf area.
The measurement must match the research question.
Operational Definitions
Sometimes scientists need to define exactly how something will be measured.
Suppose the investigation asks:
Does fertilizer improve plant growth?
What exactly does growth mean?
It could mean:
- Increase in height.
- Increase in mass.
- Increase in leaf area.
- Increase in root length.
An operational definition states exactly how a variable will be measured.
For example:
Plant growth will be measured as the increase in stem height in centimetres over 21 days.
This makes the method clearer and easier to reproduce.
Reliable but Not Valid
An investigation can be reliable without being valid.
Suppose a broken balance always adds 5 g.
A true mass of 50 g might repeatedly be measured as:
- 55.0 g
- 55.0 g
- 55.0 g
The results are extremely consistent.
They are therefore repeatable.
But the measurement is systematically wrong.
This demonstrates an important principle:
consistent results are not automatically correct or valid measurements.
Valid Idea but Unreliable Data
An investigation can also be appropriately designed but produce highly variable measurements.
Suppose students correctly investigate:
How does ramp height affect travel time?
They control all relevant variables but use inconsistent hand timing.
Results at one height are:
- 1.4 s
- 2.1 s
- 1.6 s
- 2.5 s
- 1.3 s
The design may address the intended question, but the measurements have poor reliability.
A better timing system could improve the investigation.
Accuracy, Reliability, and Validity
These ideas are related but should not be confused.
Accuracy
How close is a measurement to the accepted or true value?
Reliability
How consistent are measurements or findings?
Validity
Does the investigation appropriately measure or test what it claims to?
A high-quality investigation aims for all three where they are applicable.
Random Error
Random error causes unpredictable variation between measurements.
Examples include:
- Human reaction time.
- Slight temperature fluctuations.
- Small changes in equipment readings.
- Natural biological variation.
Random errors often reduce reliability.
Repeated measurements can help reveal and reduce their influence on calculated averages.
Systematic Error
Systematic error consistently shifts measurements in a particular direction.
Examples include:
- A balance that is incorrectly zeroed.
- A thermometer that consistently reads 2°C too high.
- Measuring length from the damaged end of a ruler.
- A sensor that has been incorrectly calibrated.
Repeating measurements does not usually solve systematic error.
If the same faulty instrument is used repeatedly, it may repeatedly produce the same incorrect result.
Worked Example: Reaction Time
A student investigates:
How does temperature affect reaction rate?
The student measures the time required for a reaction to reach a particular endpoint.
At 30°C:
- Trial 1: 42.1 s
- Trial 2: 41.8 s
- Trial 3: 42.3 s
These measurements are closely grouped.
This suggests good repeatability.
However, suppose the student also used twice the reactant concentration at 30°C compared with the other temperatures.
The comparison would not validly isolate the effect of temperature.
This demonstrates:
reliable measurements do not guarantee a valid experiment.
Worked Example: Toy-Car Ramp
Question:
How does ramp height affect the time required for a toy car to travel 2.0 m?
Improving Validity
Keep constant:
- Car.
- Ramp.
- Surface.
- Travel distance.
- Release procedure.
Only ramp height should deliberately change.
Improving Reliability
- Repeat each height several times.
- Use a consistent release mechanism.
- Use electronic timing if available.
- Calculate a mean.
The same experiment requires different strategies to improve reliability and validity.
Worked Example: Enzyme Investigation
Question:
How does temperature affect enzyme activity?
Students test:
10°C, 20°C, 30°C, 40°C and 50°C
To improve validity, they should control:
- pH.
- Enzyme concentration.
- Substrate concentration.
- Reaction volume.
- Measurement method.
To improve reliability, they should:
- Repeat each temperature.
- Use consistent equipment.
- Calculate appropriate averages.
- Investigate anomalous results.
Factors That Reduce Reliability
Reliability can be reduced by:
- Too few repeated trials.
- Very small sample sizes.
- Inconsistent procedures.
- Poorly controlled measurement conditions.
- Human reaction-time differences.
- Low-resolution equipment.
- Natural variation between organisms.
- Inconsistent equipment use.
These factors increase variation in results.
Factors That Reduce Validity
Validity can be reduced by:
- Changing several variables simultaneously.
- Failing to control important confounding variables.
- Measuring the wrong dependent variable.
- Using an inappropriate control.
- Using a method that does not address the research question.
- Systematic measurement problems.
- Bias in experimental design.
- Comparing groups under substantially different conditions.
Improving Reliability
Possible improvements include:
- Repeat each measurement.
- Increase sample size where appropriate.
- Use a standardized procedure.
- Use more precise or automated equipment.
- Keep measurement methods consistent.
- Train investigators to use equipment correctly.
- Calculate means where appropriate.
- Investigate anomalies.
- Replicate the experiment independently.
Improving Validity
Possible improvements include:
- Change only the intended independent variable in a simple fair test.
- Control relevant confounding variables.
- Include appropriate control conditions.
- Measure a dependent variable that directly addresses the question.
- Use an appropriate range of independent-variable values.
- Calibrate measuring equipment.
- Reduce sources of systematic bias.
- Define variables clearly.
- Ensure experimental and control groups are treated consistently apart from the intended difference.
Improving Both
Some changes can improve several aspects of experimental quality.
For example, using an electronic light gate instead of a handheld stopwatch in a motion experiment may:
- Reduce human reaction-time variation.
- Improve timing precision.
- Standardize measurement.
- Improve repeatability.
However, better equipment cannot fix every problem.
A highly precise sensor used in a badly designed experiment still produces evidence that may not answer the intended question.
Evaluating an Investigation
When evaluating reliability, ask:
Were trials repeated?
How similar were repeated measurements?
Was the sample size appropriate?
Were procedures performed consistently?
Are there unexplained anomalies?
When evaluating validity, ask:
Does the method actually test the research question?
Was the correct independent variable changed?
Was an appropriate dependent variable measured?
Were important confounding variables controlled?
Was an appropriate control used where necessary?
Example Evaluation
A student wants to determine whether salt concentration affects seed germination.
They use:
- 10 seeds with pure water.
- 10 seeds with 1% salt solution.
- 10 seeds with 2% salt solution.
- 10 seeds with 3% salt solution.
All seeds have:
- The same species.
- Similar environmental conditions.
- Equal solution volumes.
- The same temperature.
- The same growing medium.
- The same observation period.
The student records the percentage that germinate.
Validity
The design is reasonably valid for the stated question because salt concentration is deliberately changed while other important factors are controlled.
Reliability
Reliability could be improved by:
- Using more seeds.
- Repeating the entire investigation.
- Repeating the experiment with additional batches.
This separates two different questions about experimental quality.
Evaluating Conclusions
Scientists must also consider whether conclusions are supported by the evidence.
Suppose the results show:
| Salt Concentration | Germination |
|---|---|
| 0% | 90% |
| 1% | 80% |
| 2% | 55% |
| 3% | 20% |
A reasonable conclusion is:
Within the tested range and conditions, germination decreased as salt concentration increased.
A poor conclusion would be:
Salt always prevents all plants from germinating.
The data do not support such a broad claim.
Validity also involves making conclusions appropriate to the evidence collected.
Experimental Quality Is More Than Repetition
Repeating an experiment ten times does not automatically make it a good experiment.
Consider:
poor method × 10 repetitions
The same design problem may simply be repeated ten times.
A strong investigation needs:
appropriate design + suitable controls + reliable measurement + repetition + appropriate analysis
Reliability and Scientific Confidence
Science becomes stronger when findings can be reproduced by other investigators.
If many independent researchers obtain similar results using appropriate methods, confidence in the finding increases.
This is why scientists:
- Publish methods.
- Describe equipment.
- Report experimental conditions.
- Share data.
- Replicate studies.
- Compare results.
Science depends on evidence that can be checked.
Common Mistakes
"Reliable Means Correct"
Not necessarily.
A measurement can be consistently wrong.
"Valid Means the Results Are Similar"
Similarity between repeated results relates more directly to reliability.
Validity concerns whether the investigation appropriately addresses what it claims to test.
"Repeating an Experiment Makes It Valid"
Repetition improves information about reliability.
It does not automatically fix poor experimental design.
"One Measurement Is Enough"
One measurement provides little information about random variation.
"More Trials Fix Every Error"
Repeated trials can reduce the influence of random error but usually do not remove systematic error.
"A Large Sample Automatically Makes an Experiment Valid"
Large samples can be useful, but they cannot compensate for a fundamentally inappropriate method.
"Anomalies Should Always Be Deleted"
Anomalies should be investigated and handled with scientific justification.
"Controlling More Variables Always Improves the Experiment"
Only variables relevant to the question need to be controlled. Some variables cannot or should not be held constant depending on the investigation.
Check Your Understanding
1. Define reliability.
2. Define validity.
3. Explain the difference between reliability and validity.
4. Why are repeated trials important?
5. How can calculating a mean improve the usefulness of repeated measurements?
6. What is an anomaly?
7. Why should anomalies be investigated?
8. How can increasing sample size improve an investigation?
9. What is the difference between repetition and replication?
10. What is a confounding variable?
11. How can a confounding variable reduce validity?
12. Why must the dependent variable match the research question?
13. What is an operational definition?
14. Give an example of an investigation that is reliable but not valid.
15. Can an appropriately designed investigation still produce unreliable measurements? Explain.
16. What is the difference between random and systematic error?
17. Why can repeated trials reduce the influence of random error?
18. Why don't repeated trials usually remove systematic error?
19. Give three ways to improve reliability.
20. Give three ways to improve validity.
21. A student investigates temperature and reaction rate but changes concentration at each temperature. What problem does this create?
22. How could the investigation in Question 21 be improved?
23. A student obtains almost identical measurements using a balance that consistently reads 4 g too high. Are the measurements reliable? Explain.
24. Why doesn't highly precise equipment guarantee a valid experiment?
25. Explain why a scientifically strong investigation should consider both reliability and validity.
Key Terms
- Reliability – consistency of measurements or findings when an investigation is repeated under comparable conditions.
- Validity – extent to which an investigation and its measurements appropriately address the intended question.
- Repeated trial – repeating a measurement or procedure under comparable conditions.
- Replication – independent repetition of an investigation, often by other researchers.
- Mean – sum of values divided by the number of values.
- Range – difference between the highest and lowest values.
- Anomaly – result that differs noticeably from the general pattern.
- Sample size – number of experimental units or observations included in an investigation.
- Confounding variable – uncontrolled factor that provides an alternative explanation for a result.
- Operational definition – precise description of how a variable will be measured or identified.
- Random error – unpredictable variation between measurements.
- Systematic error – consistent error that shifts measurements in a particular direction.
- Accuracy – closeness of a measurement to an accepted or true value.
- Precision – closeness of repeated measurements to one another and the resolution of measurement.
- Control – reference condition used to help interpret results.
Key Takeaways
- Reliability and validity describe different aspects of experimental quality.
- Reliability concerns consistency.
- Validity concerns whether the investigation appropriately tests what it claims to test.
- Repeated trials help assess reliability.
- Means can reduce the influence of random variation.
- Larger samples can reduce the influence of individual variation.
- Anomalies should be investigated rather than automatically discarded.
- Consistent procedures improve reliability.
- Controlling important confounding variables improves validity.
- The dependent variable must appropriately represent what the investigation is trying to measure.
- Random errors mainly increase variation between measurements.
- Systematic errors can consistently shift results.
- Repetition does not normally eliminate systematic error.
- An experiment can be reliable but not valid.
- An appropriately designed experiment can still produce unreliable measurements.
- Better equipment can improve measurement quality but cannot rescue a fundamentally poor experimental design.
- Scientists improve experimental quality by evaluating both their methods and their data.
The central distinction is:
Reliability = Can I obtain consistent results?
Validity = Am I actually testing what I intend to test?
5. Improving procedures
Learning outcomes
- I can identify weaknesses in an experimental method.
- I can suggest realistic improvements to procedures.
- I can reduce sources of experimental error.
- I can improve data quality through better measurement techniques.
- I can justify recommended improvements using scientific reasoning.