Introduction

Many real-world situations involve relationships between two variables. For example, does spending more time studying lead to higher examination scores? Do taller people tend to have larger shoe sizes? Does the price of a car decrease as it gets older?

Statisticians investigate questions like these using scatter plots and correlation.

A scatter plot is a graph that displays pairs of numerical data. By examining the pattern of points, we can identify relationships, estimate their strength, and make predictions.

However, one of the most important principles in statistics is that correlation does not necessarily mean causation. Two variables may be related without one directly causing the other.

Understanding scatter plots and correlation allows us to analyse data, recognise patterns, evaluate evidence, and avoid misleading conclusions.

What Is a Scatter Plot?

A scatter plot, also called a scatter diagram, displays the relationship between two quantitative variables.

Each observation is represented by a point with coordinates (x, y).

• The horizontal axis (x-axis) usually represents the explanatory or predictor variable.

• The vertical axis (y-axis) usually represents the response or outcome variable.

• Each point represents one pair of measurements.

For example, consider the relationship between hours spent studying and examination scores.

The points generally move upward from left to right. This suggests that students who study for longer tend to achieve higher examination scores in this dataset.

This is an example of positive correlation.

Notice that the graph shows an association. It does not establish that additional study time alone caused the higher scores.

Constructing a Scatter Plot

To construct a scatter plot correctly, follow these steps.

1. Identify the variables. Decide which variable belongs on each axis.

2. Choose suitable scales. Make sure the scales include all the observations and use consistent intervals.

3. Label both axes. Include the variable names and units where appropriate.

4. Plot each coordinate pair. For example, (3, 55) represents 3 hours of study and a score of 55%.

5. Examine the pattern. Look for direction, strength, shape, clusters, and unusual points.

Unlike a line graph showing measurements over time, a scatter plot generally does not connect consecutive points.

Worked Example: Constructing a Scatter Plot

A scientist records the temperature and number of ice creams sold at a shop.

Understanding Correlation

Correlation describes the direction and strength of an association between two variables.

For linear relationships, there are three main directions.

Positive Correlation

Positive correlation occurs when larger values of one variable tend to be associated with larger values of another.

Examples include height and arm span, temperature and ice cream sales, and hours of exercise and calories burned during exercise.

Negative Correlation

Negative correlation occurs when larger values of one variable tend to be associated with smaller values of another.

Examples include vehicle age and resale value, distance remaining and distance travelled along a fixed route, and time spent braking and vehicle speed during a braking event.

No Linear Correlation

Two variables have little or no linear correlation when there is no consistent straight-line trend.

For example, a dataset comparing students' shoe sizes and examination scores might show no meaningful linear association.

Importantly, no linear correlation does not necessarily mean there is no relationship at all. The relationship might be curved.

Strength of Correlation

The strength of linear correlation describes how closely the points follow a straight-line pattern.

• Strong correlation: Points lie close to a straight line.

• Moderate correlation: A general trend is visible, but points are more spread out.

• Weak correlation: Points are widely scattered, with only a slight linear trend.

• Perfect correlation: Every point lies exactly on a straight line.

  

  

  

  Explore the positive, negative, and uncorrelated patterns. Move individual points and observe how the correlation coefficient changes.

  The strength of a relationship is different from its direction. A strong negative correlation can be just as strong as a strong positive correlation.

   

  The Pearson Correlation Coefficient

  Visual inspection of a scatter plot is useful, but statisticians often need a numerical measurement.

  The Pearson correlation coefficient, represented by r, measures the strength and direction of a linear relationship between two quantitative variables.

  Its value is always between −1 and +1.

  

  The main interpretations are:

  Value of r	Interpretation
  +1	Perfect positive linear correlation
  +0.8	Strong positive linear correlation
  +0.4	Moderate positive linear correlation
  0	No linear correlation
  −0.4	Moderate negative linear correlation
  −0.8	Strong negative linear correlation
  −1	Perfect negative linear correlation

  These descriptions are approximate. There is no universally accepted boundary between weak, moderate, and strong correlation; interpretations depend on the field and context.

  Calculating Pearson's r

  The computational formula is:

  \[ r=\frac{n\sum xy-(\sum x)(\sum y)} {\sqrt{[n\sum x^2-(\sum x)^2][n\sum y^2-(\sum y)^2]}} \]

  Where:

  • n = number of paired observations

  • Σx = sum of all x-values

  • Σy = sum of all y-values

  • Σxy = sum of the products of paired values

  • Σx² = sum of the squared x-values

  • Σy² = sum of the squared y-values

  The symbol Σ means "sum of."

  This formula is suitable when both variables vary and the observations are paired.

  Worked Example: Calculating a Correlation Coefficient

  Five students complete a practice exercise. Their study time and scores are recorded.

  x (hours)	y (score)	x²	y²	xy
  1	2	1	4	2
  2	3	4	9	6
  3	5	9	25	15
  4	4	16	16	16
  5	6	25	36	30
  15	20	55	90	69

  Step 1: Identify the totals.

  n = 5

  Σx = 15

  Σy = 20

  Σx² = 55

  Σy² = 90

  Σxy = 69

  Step 2: Substitute into the formula.

  \[ r=\frac{5(69)-(15)(20)} {\sqrt{[5(55)-15^2][5(90)-20^2]}} \]

  Step 3: Simplify.

  \[ r=\frac{345-300}{\sqrt{(275-225)(450-400)}} \]

  \[ r=\frac{45}{\sqrt{2500}}=\frac{45}{50} \]

  \[ \boxed{r=0.90} \]

  Interpretation: There is a strong positive linear correlation between study time and exercise score in this small dataset.

  Students who studied for longer generally achieved higher scores, although the relationship was not perfect.

  Calculating Correlation Using Technology

  For larger datasets, calculations are usually completed using a spreadsheet, graphing calculator, or statistical software.

  In Excel or Google Sheets, if the x-values are in cells A2:A11 and y-values are in B2:B11, enter:

  =CORREL(A2:A11,B2:B11)

  The spreadsheet calculates Pearson's r automatically.

  The correlation coefficient measures association; it does not indicate which variable causes the other.

   

  Interpreting Relationships Between Variables

  A scatter plot provides more information than simply whether a correlation is positive or negative.

  When analysing a scatter plot, consider five important features.

  Direction

  Does the relationship increase, decrease, or show no consistent linear trend?

  Strength

  How closely do the points follow the overall pattern?

  Form

  Is the relationship approximately linear or nonlinear?

  A linear relationship follows an approximately straight-line pattern. A nonlinear relationship may follow a curve.

  

  The points form a U-shaped pattern. There is a clear relationship, even though the Pearson correlation coefficient is zero for this symmetric dataset.

  This demonstrates why we should inspect the scatter plot before relying on r.

  Outliers

  An outlier is a point that lies unusually far from the overall pattern.

  An outlier may result from measurement error, incorrect data entry, an unusual event, or genuine variation.

  Outliers can substantially affect Pearson's correlation coefficient.

  Clusters and Subgroups

  Sometimes a scatter plot contains separate groups of points.

  For example, data from two different age groups might form two clusters. A correlation calculated using all the data may hide important differences within the groups.

  When possible, examine the relationship within relevant subgroups as well as across the full dataset.

   

  Lines of Best Fit

  A line of best fit, or regression line, summarises an approximately linear relationship between two variables.

  It can be used to estimate the value of y for a given x.

  A linear model is commonly written as:

  \[ y=mx+b \]

  Where:

  • m = slope of the line

  • b = y-intercept

  The slope indicates the predicted change in y for each one-unit increase in x.

  Worked Example: Predicting Examination Scores

  Suppose a regression model relating study time to examination score is:

  \[ y=7x+35 \]

  Where x represents study time in hours and y represents predicted examination score.

  Predict the score for a student who studies for 6 hours.

  Substitute x = 6:

  \[ y=7(6)+35 \]

  \[ y=42+35=\boxed{77} \]

  The predicted score is 77%.

  This is a model-based prediction, not a guarantee.

  Interpolation and Extrapolation

  Interpolation means predicting within the range of observed data.

  Extrapolation means predicting beyond the observed range.

  For example, if study times were measured from 1 to 8 hours:

  • Predicting the score for 6 hours is interpolation.

  • Predicting the score for 20 hours is extrapolation.

  Extrapolation is generally less reliable because the relationship may change outside the observed range. A linear examination-score model, for example, cannot reasonably predict scores above 100%.

   

  Correlation Does Not Imply Causation

  One of the most important principles in statistics is that a relationship between two variables does not automatically mean that one variable causes the other.

  Correlation means two variables are statistically associated.

  Causation means a change in one variable produces a change in another.

  Example: Ice Cream Sales and Drowning Incidents

  During hotter weather, both ice cream sales and swimming activity may increase. Drowning incidents may also increase when more people are swimming.

  A statistical study might find a positive correlation between ice cream sales and drowning incidents.

  Does buying ice cream cause drowning?

  No.

  A third variable, such as hot weather, may help explain the association.

  

  Other Reasons Correlation May Not Establish Causation

  Reverse causation: The proposed direction of cause and effect may be incorrect. For example, a relationship between anxiety and poor sleep could involve effects in both directions.

  Confounding variables: A third factor may influence both variables.

  Coincidence: Two variables may be correlated by chance, particularly when many comparisons are tested.

  Selection bias: The way participants or observations are selected may produce a misleading relationship.

  How Can Causation Be Investigated?

  Researchers can strengthen causal conclusions by using:

  • Randomised controlled experiments, where appropriate and ethical.

  • Longitudinal studies that examine how variables change over time.

  • Careful measurement and control of confounding variables.

  • Replication of findings in different populations.

  • Plausible scientific explanations supported by additional evidence.

  Even a very high correlation coefficient, such as r = 0.99, is not by itself proof of causation.

   

  Real-World Applications

  Why Meteorologists Use Multiple Weather Models

   

   

   

  Meteorology

  Scientists investigate relationships between temperature, humidity, rainfall, and atmospheric pressure. Scatter plots can help identify patterns and evaluate weather models.

   

  Understanding Exercise Physiology for Personal Goals - frametether.com

   

   

   

  Sports Science

  Researchers compare exercise intensity with heart rate, oxygen consumption, and recovery time to understand athletic performance.

   

  Peninsula Residential Real Estate - HAYLEN Group

   

   

   

  Economics and Property

  Analysts examine associations between property size, location, age, and selling price. Regression models can support price estimates.

   

  Environmental researcher collecting water samples for field testing

   

   

   

  Environmental Science

  Scientists investigate relationships between pollution levels, water temperature, and aquatic biodiversity. Correlations can help identify environmental questions requiring further investigation.

   

  Common Mistakes

  Common mistake	Correction
  Assuming positive correlation means a strong relationship	Positive describes direction, not strength.
  Thinking r = −0.9 is weaker than r = +0.5	The absolute value of r determines linear strength; 0.9 is stronger than 0.5.
  Assuming r = 0 means there is no relationship	There may be a nonlinear relationship.
  Connecting every point in a scatter plot	Generally plot the points separately and examine the overall pattern.
  Claiming that correlation proves causation	Consider confounding variables, reverse causation, and other explanations.
  Ignoring unusual points	Outliers can greatly change the correlation coefficient.
  Making predictions far beyond observed values	Extrapolation may produce unrealistic predictions.
  Calculating r from unpaired or incorrectly matched observations	Each x-value must be matched with its corresponding y-value.

   

  Key Terms

  • Scatter plot: A graph showing pairs of numerical observations as points.

  • Bivariate data: Data involving two variables measured together.

  • Explanatory variable: A variable used to help explain or predict another variable.

  • Response variable: The variable being investigated or predicted.

  • Correlation: A statistical association between variables.

  • Positive correlation: An association in which larger values of one variable tend to accompany larger values of another.

  • Negative correlation: An association in which larger values of one variable tend to accompany smaller values of another.

  • Linear correlation: An association described by an approximately straight-line pattern.

  • Pearson correlation coefficient (r): A numerical measure of the direction and strength of a linear relationship.

  • Outlier: An observation that lies unusually far from the overall pattern.

  • Line of best fit: A line used to summarise and predict an approximately linear relationship.

  • Regression: A statistical method for modelling relationships between variables.

  • Interpolation: Prediction within the observed data range.

  • Extrapolation: Prediction outside the observed data range.

  • Causation: A relationship in which one factor produces a change in another.

  • Confounding variable: A variable that can distort the apparent relationship between the variables being studied.

   

  Key Takeaways

  1. Scatter plots display paired quantitative data and help reveal relationships between two variables.

  2. Correlation can be positive, negative, or show little or no linear association.

  3. Correlation strength depends on how closely the points follow a linear pattern.

  4. Pearson's correlation coefficient, r, ranges from −1 to +1.

  5. Values of r close to −1 or +1 indicate strong linear correlation; values near zero indicate weak linear correlation.

  6. A scatter plot should be inspected for outliers, clusters, and nonlinear patterns before interpreting r.

  7. Regression lines can be used to make predictions, but predictions outside the observed range require caution.

  8. Correlation alone cannot establish a cause-and-effect relationship.

   

  Check Your Understanding

  Question 1: Identifying Correlation

  A dataset shows that as the age of a car increases, its resale value generally decreases.

  What type of correlation does this represent?

  Question 2: Interpreting Strength

  Three datasets have correlation coefficients:

  • Dataset A: r = 0.92

  • Dataset B: r = −0.75

  • Dataset C: r = 0.18

  Which dataset has the strongest linear correlation? Which has the weakest?

  Question 3: Constructing a Scatter Plot

  The table shows the number of hours students spend exercising each week and their resting heart rates.

  Exercise (hours)	Resting heart rate (bpm)
  1	82
  2	79
  3	75
  4	73
  5	69
  6	66

  Construct a scatter plot, identify the correlation direction, and describe its strength.

  Question 4: Calculating Pearson's r

  Calculate the Pearson correlation coefficient for the following dataset.

  x	y
  1	3
  2	5
  3	4
  4	7

  Interpret your result.

  Question 5: Correlation and Causation

  A researcher discovers that cities with more firefighters tend to experience greater total property damage from fires.

  A newspaper concludes that employing more firefighters causes greater property damage.

  Explain why this conclusion is not justified.

  Question 6: Evaluating a Statistical Relationship

  A scatter plot of x and y shows a clear U-shaped pattern.

  The Pearson correlation coefficient is r = 0.02.

  A student concludes that x and y are unrelated.

  Explain the problem with this conclusion.

  Question 7: Making Predictions

  A regression model predicts electricity consumption using:

  y = 12x + 150

  Where x is the average daily temperature in °C and y is daily electricity consumption in kWh.

  Calculate the predicted electricity consumption at 25°C.

  Explain why using this model to predict consumption at 70°C might be unreliable if the observed temperatures ranged from 15°C to 35°C.

   

  Fully Worked Answers

  Answer 1

  The relationship is a negative correlation.

  As car age increases, resale value generally decreases.

  Answer 2

  Compare the absolute values:

  Dataset	r	Absolute value
  A	0.92	0.92
  B	−0.75	0.75
  C	0.18	0.18

  Dataset A has the strongest linear correlation.

  Dataset C has the weakest linear correlation.

  The negative sign for Dataset B describes direction, not weakness.

  Answer 3

  

  The graph shows a strong negative linear correlation.

  As weekly exercise time increases, resting heart rate tends to decrease in this dataset.

  This does not establish that exercise time alone caused the differences in resting heart rate.

  Answer 4

  First calculate the required totals.

  x	y	x²	y²	xy
  1	3	1	9	3
  2	5	4	25	10
  3	4	9	16	12
  4	7	16	49	28
  10	16	30	99	53

  n = 4

  Substitute:

  \[ r=\frac{4(53)-(10)(16)} {\sqrt{[4(30)-10^2][4(99)-16^2]}} \]

  \[ r=\frac{212-160}{\sqrt{(120-100)(396-256)}} \]

  \[ r=\frac{52}{\sqrt{2800}} \]

  \[ \boxed{r\approx0.983} \]

  The result indicates a very strong positive linear correlation.

  Answer 5

  Cities with more firefighters may be larger or experience more fires, including more serious fires.

  These factors could explain why they also experience greater total property damage.

  The correlation does not demonstrate that firefighters cause damage.

  A better investigation would consider city size, number of fires, fire severity, building density, and other relevant variables.

  Answer 6

  Pearson's r measures linear correlation.

  A U-shaped relationship is nonlinear. Therefore, r can be close to zero even when the variables have a clear mathematical relationship.

  The student should examine the scatter plot and consider a nonlinear model rather than concluding that the variables are unrelated.

  Answer 7

  Given:

  y = 12x + 150

  Substitute x = 25:

  y = 12(25) + 150

  y = 300 + 150

  Predicted electricity consumption = 450 kWh.

  Predicting at 70°C would be extrapolation because 70°C lies far outside the observed range of 15°C to 35°C.

  The original linear relationship may not continue at extreme temperatures, making the prediction unreliable.

   

  Final Reflection

  Scatter plots and correlation provide powerful tools for exploring relationships in data. They help us identify patterns, describe the strength and direction of associations, and make informed predictions.

  However, responsible statistical reasoning requires more than calculating a correlation coefficient. We must examine the shape of the data, recognise unusual observations, evaluate the reliability of predictions, and consider alternative explanations.

  The most important distinction is that a statistical relationship is evidence of association, not automatic proof of cause and effect.

   

Introduction

Imagine that a school wants to investigate whether the number of hours students spend studying can help predict their examination scores. After collecting data, the school notices that students who study longer generally achieve higher scores. However, the points on a scatter plot do not form a perfectly straight line.

How can we describe this relationship mathematically? How can we use it to predict examination results? And how can we determine whether our predictions are reliable?

Linear regression provides a method for answering these questions.

Linear regression is a statistical technique that models the relationship between a numerical explanatory variable and a numerical response variable using a straight line. It allows us to identify trends, make predictions, measure prediction errors, and evaluate how well a mathematical model represents real data.

Linear regression is widely used in science, economics, engineering, medicine, environmental research, and business.

What Is Linear Regression?

Linear regression is a method of finding a straight line that best describes the relationship between two quantitative variables.

This line is called the regression line, or line of best fit.

A scatter plot displays the original observations, while the regression line summarises their overall linear trend.

The points generally rise from left to right, indicating a positive linear association.

A regression line allows us to represent this trend with an equation rather than simply describing it visually.

The Regression Equation

A simple linear regression equation is commonly written as:

Where:

• ŷ (y-hat) = predicted value of the response variable

• x = explanatory or predictor variable

• a = y-intercept of the regression line

• b = slope or regression coefficient

The symbol ŷ is used instead of y because the regression equation produces a prediction, which may differ from the observed value.

Understanding the Slope

The slope, b, describes how much the predicted response changes for each one-unit increase in x.

• If b is positive, the predicted response increases as x increases.

• If b is negative, the predicted response decreases as x increases.

• If b is zero, the model predicts the same response regardless of x.

Understanding the Intercept

The intercept, a, is the predicted response when x = 0.

However, the intercept is not always meaningful in a real-world context. If zero lies outside the observed range, interpreting the intercept may involve unreliable extrapolation.

Determining a Line of Best Fit

A line of best fit can be estimated visually or calculated mathematically.

Method 1: Drawing a Line of Best Fit by Eye

When using a scatter plot:

1. Plot all paired observations accurately.

2. Identify the overall direction of the points.

3. Draw a straight line through the centre of the pattern.

4. Try to balance the points above and below the line.

5. Choose two well-separated points on the line to estimate its slope and intercept.

A visually estimated line is useful for understanding trends, but different people may draw slightly different lines.

For precise statistical analysis, we use the least-squares method.

Method 2: The Least-Squares Regression Line

The least-squares method finds the line that minimises the sum of the squared vertical differences between observed and predicted values.

These vertical differences are called residuals.

Interactive regression model: move the observed points to see how the least-squares line changes. The vertical gaps between points and the fitted line represent residuals.

Why square the residuals?

Squaring ensures that positive and negative errors do not cancel each other. It also gives greater weight to large errors.

The quantity minimised is called the sum of squared errors, or SSE:

The least-squares line has the smallest possible SSE among straight lines with an intercept fitted to the same dataset.

Calculating the Least-Squares Regression Line

For simple linear regression, the slope is:

The intercept is:

Where:

• n = number of paired observations

• Σxy = sum of products of corresponding x- and y-values

• Σx = sum of x-values

• Σy = sum of y-values

• Σx² = sum of squared x-values

• x̄ = mean of x-values

• ȳ = mean of y-values

The slope formula requires variation in x; if all x-values are identical, the usual least-squares slope cannot be calculated.

Worked Example 1: Finding a Regression Equation

A teacher records the number of hours five students spend studying and their scores on a short assessment.

Determine the least-squares regression equation.

Step 1: Construct a calculation table.

Therefore:

n = 5

Σx = 15

Σy = 20

Σx² = 55

Σxy = 69

Step 2: Calculate the slope.

Step 3: Calculate the means.

Step 4: Calculate the intercept.

Step 5: Write the regression equation.

This equation represents the least-squares line of best fit.

Interpreting the Equation

The slope of 0.9 means that each additional hour of study is associated with an increase of 0.9 points in the predicted assessment score.

The intercept of 1.3 means that the model predicts a score of 1.3 points when study time is zero hours.

Because zero hours is outside the observed study-time range of 1–5 hours, the intercept should be interpreted cautiously.

Making Predictions Using Regression

Once a regression equation has been determined, it can be used to estimate the response variable for a particular explanatory-variable value.

Worked Example 2: Predicting an Assessment Score

Using the regression equation:

ŷ = 1.3 + 0.9x

Predict the assessment score for a student who studies for 4 hours.

Step 1: Substitute x = 4.

ŷ = 1.3 + 0.9(4)

Step 2: Simplify.

ŷ = 1.3 + 3.6

The predicted score is 4.9 points.

The actual score of the student who studied for 4 hours was 4 points, illustrating that a regression prediction does not necessarily match an observation exactly.

Worked Example 3: Predicting House Prices

Suppose a property analyst develops the model:

ŷ = 80,000 + 2,500x

Where:

• x = floor area in square metres

• ŷ = predicted house price in dollars

Predict the price of a house with a floor area of 120 m².

ŷ = 80,000 + 2,500(120)

ŷ = 80,000 + 300,000

The model predicts a price of $380,000.

The slope indicates an additional $2,500 in predicted price per square metre of floor area.

This does not mean that increasing a particular house's area by one square metre would necessarily cause its price to rise by exactly $2,500. Other factors, such as location and condition, also matter.

Interpolation and Extrapolation

Regression predictions are not equally reliable for all values of x.

Interpolation

Interpolation occurs when a prediction is made within the range of x-values used to develop the model.

For example, if study times range from 1 to 8 hours, predicting a score for 5 hours is interpolation.

Extrapolation

Extrapolation occurs when a prediction is made outside the observed range.

Predicting the score for a student who studies for 20 hours would be extrapolation.

Extrapolation is often less reliable because the relationship may change outside the observed range.

For example, a model relating study time to examination score cannot reasonably predict examination scores greater than 100% indefinitely.

Understanding Residuals

A regression line does not usually pass through every observation.

The difference between an observed value and its predicted value is called a residual.

The residual formula is:

Where:

• e = residual

• y = observed value

• ŷ = predicted value

A residual measures the prediction error for one observation.

Interpreting Residual Signs

• Positive residual: The observed value is greater than predicted; the point lies above the regression line.

• Negative residual: The observed value is less than predicted; the point lies below the regression line.

• Zero residual: The observed value equals the predicted value; the point lies on the regression line.

  

  Worked Example 4: Calculating Residuals

  Use the regression equation from Worked Example 1:

  ŷ = 1.3 + 0.9x

  Calculate the predicted values and residuals for all five students.

  x	Observed y	Predicted ŷ	Residual (y − ŷ)
  1	2	2.2	−0.2
  2	3	3.1	−0.1
  3	5	4.0	+1.0
  4	4	4.9	−0.9
  5	6	5.8	+0.2

  For the third student:

  e = 5 − 4.0 = +1.0

  The actual score was 1 point higher than predicted.

  For the fourth student:

  e = 4 − 4.9 = −0.9

  The actual score was 0.9 points lower than predicted.

  For an ordinary least-squares regression with an intercept, the residuals sum to zero, apart from rounding.

  Here:

  −0.2 − 0.1 + 1.0 − 0.9 + 0.2 = 0

  This property does not mean the predictions are perfect; positive and negative residuals can cancel.

   

  Residual Plots

  A residual plot displays residuals on the vertical axis and the explanatory variable (or fitted values) on the horizontal axis.

  Residual plots are useful for checking whether a linear model is appropriate.

  For the five-student example:

  

  The residuals appear on both sides of zero without a clear curved trend. This is consistent with a linear model, although five observations are too few to assess model assumptions confidently.

  Recognising Residual Patterns

  

  A good residual plot should generally show points scattered around zero without an obvious systematic pattern.

  A curved pattern suggests that a straight line may not adequately describe the relationship.

  A fan-shaped pattern suggests heteroscedasticity, meaning that the variability of errors changes across the predictor range.

  Residual plots can also help identify unusual observations, although additional diagnostics may be needed.

   

  Evaluating Model Accuracy

  Finding a regression equation is only the beginning. We must also determine how accurately the model describes the data and how useful its predictions are.

  Several statistical measures help us evaluate model performance.

  Sum of Squared Errors (SSE)

  SSE measures the total squared prediction error:

  \[ SSE=\sum (y-\hat y)^2 \]

  Smaller SSE values indicate a closer fit for models evaluated on the same observations and response scale.

  Using the five-student dataset:

  Residual	Residual²
  −0.2	0.04
  −0.1	0.01
  +1.0	1.00
  −0.9	0.81
  +0.2	0.04
  Total	1.90

  Therefore:

  \[ \boxed{SSE=1.90} \]

  Mean Squared Error (MSE)

  MSE is the average squared prediction error.

  For descriptive or predictive evaluation of n observations:

  \[ MSE=\frac{SSE}{n} \]

  For our dataset:

  \[ MSE=\frac{1.90}{5}=\boxed{0.38} \]

  Smaller MSE values generally indicate better predictions on comparable data.

  In statistical inference, a related estimate of residual error variance uses SSE/(n − 2) for simple linear regression with an intercept. This accounts for the two fitted parameters.

  Root Mean Squared Error (RMSE)

  RMSE is the square root of MSE.

  \[ RMSE=\sqrt{\frac{SSE}{n}} \]

  For the five-student dataset:

  \[ RMSE=\sqrt{0.38} \]

  \[ \boxed{RMSE\approx0.616} \]

  The RMSE is approximately 0.616 assessment points.

  Unlike MSE, RMSE is expressed in the same units as the response variable, making it easier to interpret.

  Mean Absolute Error (MAE)

  MAE measures the average absolute prediction error.

  \[ MAE=\frac{\sum |y-\hat y|}{n} \]

  For our dataset:

  \[ MAE=\frac{0.2+0.1+1.0+0.9+0.2}{5} \]

  \[ MAE=\frac{2.4}{5}=\boxed{0.48} \]

  The average absolute prediction error is 0.48 points.

  Unlike MSE and RMSE, MAE does not square errors, so it is less sensitive to a small number of very large errors.

   

  The Coefficient of Determination: R²

  The coefficient of determination, written R², describes the proportion of variation in the response variable explained by the fitted regression model relative to a model that always predicts the sample mean.

  For ordinary least-squares regression with an intercept:

  \[ R^2=1-\frac{SSE}{SST} \]

  Where:

  • SSE = sum of squared residuals

  • SST = total sum of squares

  • SST = Σ(y − ȳ)²

  For simple linear regression with an intercept, R² is also equal to the square of the Pearson correlation coefficient.

  \[ R^2=r^2 \]

  Worked Example 5: Calculating R²

  For our five-student dataset:

  y-values: 2, 3, 5, 4, 6

  Mean score:

  ȳ = 4

  Calculate SST:

  SST = (2 − 4)² + (3 − 4)² + (5 − 4)² + (4 − 4)² + (6 − 4)²

  SST = 4 + 1 + 1 + 0 + 4

  SST = 10

  We previously calculated SSE = 1.90.

  Therefore:

  \[ R^2=1-\frac{1.90}{10} \]

  \[ \boxed{R^2=0.81} \]

  The model accounts for 81% of the variation in assessment scores within this dataset, relative to predicting the mean score.

  The remaining 19% is not explained by the fitted line.

  This does not mean that study time causes exactly 81% of examination performance. R² measures model fit, not causation.

  

  Interpreting R² Carefully

  A high R² indicates that the fitted model explains a large proportion of the observed variation.

  However, a high R² does not guarantee that:

  • The relationship is causal.

  • The model is appropriate for every situation.

  • The model will predict new observations accurately.

  • There are no influential outliers.

  • The model's assumptions are satisfied.

  Similarly, a low R² does not automatically make a model useless. In fields involving highly variable human behaviour, even modest predictive relationships may be informative.

   

  Comparing Measures of Model Accuracy

  Measure	What it measures	Preferred direction
  SSE	Total squared residual error	Lower
  MSE	Average squared prediction error	Lower
  RMSE	Typical error scale, with large errors emphasised	Lower
  MAE	Average absolute prediction error	Lower
  R²	Proportion of sample variation explained by the model	Higher

  These measures should be compared only when appropriate. For example, SSE depends on the number of observations, while MSE and RMSE depend on the scale of the response variable.

  A model should also be tested on data that were not used to fit it.

   

  Training Data and Test Data

  A regression model may fit the observations used to construct it very well but perform poorly on new data.

  This is called overfitting when the model captures patterns or noise that do not generalise.

  To evaluate predictive performance, statisticians often divide data into:

  • Training data: Used to calculate the regression equation.

  • Test data: Used to evaluate predictions on observations not used to fit the model.

  For smaller datasets, cross-validation may provide a more informative assessment than a single train–test split.

  Worked Example 6: Comparing Two Models

  Two regression models are evaluated on the same unseen test dataset.

  Measure	Model A	Model B
  Training RMSE	2.1	1.2
  Test RMSE	2.4	5.8
  Training R²	0.88	0.97

  Model B fits the training data more closely, but Model A makes substantially smaller errors on the test data.

  Therefore, Model A is the better predictive choice based on these test results.

  Model B may have overfitted the training observations.

  This example illustrates why model evaluation should not rely only on training accuracy.

   

  Assumptions of Linear Regression

  Linear regression is most reliable when its underlying assumptions are reasonably satisfied.

  Linearity

  The mean response should have an approximately linear relationship with the explanatory variable.

  A curved residual pattern suggests that a linear model may be inappropriate.

  Independence

  Observations or model errors should be independent where required by the study design.

  For example, measurements collected repeatedly from the same person may be related and require specialised methods.

  Constant Error Variance

  The spread of errors should remain approximately constant across fitted values for standard regression inference.

  A fan-shaped residual plot may indicate that this assumption is violated.

  Approximately Normal Errors for Certain Inference Procedures

  Normality of errors is particularly relevant for exact small-sample confidence intervals and hypothesis tests.

  Normality is not required simply to calculate the least-squares line.

  Absence of Highly Influential Observations

  An extreme observation can substantially change the slope and intercept.

  Researchers should investigate unusual observations rather than automatically deleting them.

  Other Important Considerations

  Even if these assumptions are reasonably satisfied, omitted variables, measurement errors, biased sampling, or inappropriate extrapolation can limit a model's usefulness.

   

  Using Technology for Linear Regression

  Regression calculations can be performed using spreadsheets, graphing calculators, and statistical software.

  In Excel or Google Sheets, suppose the x-values are in A2:A11 and y-values are in B2:B11.

  Purpose	Formula
  Calculate slope	=SLOPE(B2:B11,A2:A11)
  Calculate intercept	=INTERCEPT(B2:B11,A2:A11)
  Calculate R²	=RSQ(B2:B11,A2:A11)
  Calculate Pearson's r	=CORREL(A2:A11,B2:B11)

  To make a prediction, use:

  =INTERCEPT(B2:B11,A2:A11)+SLOPE(B2:B11,A2:A11)*D2

  Where D2 contains the x-value for which a prediction is required.

  For a visual approach, create a scatter plot and add a linear trendline, selecting the options to display the equation and R² value.

  Remember that spreadsheet software may round the displayed equation. Calculations using full-precision coefficients are generally more accurate.

   

  Real-World Applications of Linear Regression

  FRANCE-RESEARCH-CLIMATE

   

   

   

  Climate and Environmental Science

  Researchers use regression to investigate temperature trends, rainfall patterns, pollution concentrations, and environmental changes over time.

   

  Intelligent Infrastructure Laboratory | ICAIR

   

   

   

  Engineering

  Engineers model relationships between applied force and deformation, electrical current and voltage, or material properties and performance.

   

  Become a Financial Analyst After Completing Your Online BCom

   

   

   

  Business and Economics

  Analysts use regression to predict sales, examine relationships between advertising and revenue, and estimate how prices vary with product characteristics.

   

  Clinical Research Course Certificate | Free & Fast Course

   

   

   

  Medicine and Health Research

  Researchers investigate relationships between age, physical measurements, exercise, and health outcomes. Regression helps quantify associations while accounting for uncertainty.

   

  Common Mistakes

  Common mistake	Correction
  Assuming a regression line passes through every observation	The line minimises total squared residuals; individual residuals are usually nonzero.
  Confusing observed y with predicted ŷ	Use y for actual measurements and ŷ for predictions.
  Calculating residuals as ŷ − y	Residual = observed − predicted.
  Interpreting the intercept without considering context	The intercept may be meaningless if x = 0 is outside the observed range.
  Assuming a high R² proves causation	R² describes model fit, not cause and effect.
  Ignoring residual patterns	Curved or fan-shaped patterns can reveal model problems.
  Extrapolating far beyond the data	The relationship may change outside the observed range.
  Comparing SSE values from different-sized datasets without caution	SSE depends on sample size and response scale.
  Assuming low training error guarantees accurate predictions	Evaluate the model on unseen data.
  Automatically removing outliers	Investigate unusual observations and their influence before deciding how to handle them.

   

  Key Terms

  • Linear regression: A statistical method that models a relationship between numerical variables using a straight line.

  • Regression line: A fitted line used to summarise a linear relationship and make predictions.

  • Line of best fit: A line that represents the overall trend in a scatter plot.

  • Least-squares method: A method that minimises the sum of squared residuals.

  • Explanatory variable: A variable used to predict or explain another variable.

  • Response variable: The variable being predicted or studied.

  • Predicted value (ŷ): The response value calculated using the regression equation.

  • Regression coefficient: A parameter of a regression model, such as its slope.

  • Slope: The predicted change in y for a one-unit increase in x.

  • Intercept: The predicted y-value when x = 0.

  • Residual: The difference between an observed and predicted value.

  • Residual plot: A graph used to investigate patterns in regression errors.

  • SSE: Sum of squared errors.

  • MSE: Mean squared error.

  • RMSE: Root mean squared error.

  • MAE: Mean absolute error.

  • R²: Coefficient of determination; a measure of the proportion of variation explained by the fitted model.

  • Interpolation: Prediction within the observed range.

  • Extrapolation: Prediction beyond the observed range.

  • Heteroscedasticity: Unequal error variance across the predictor or fitted-value range.

  • Overfitting: Fitting a model too closely to training data in a way that harms generalisation.

  • Cross-validation: A method for evaluating predictive performance using repeated training and validation partitions.

  • Influential observation: An observation that substantially affects the fitted model.

   

  Key Takeaways

  1. Linear regression models the relationship between an explanatory variable and a response variable using a straight line.

  2. The least-squares method finds the line that minimises the sum of squared residuals.

  3. The slope describes the predicted change in the response for a one-unit increase in the explanatory variable.

  4. The intercept represents the predicted response when the explanatory variable is zero, although it may not always be meaningful.

  5. Regression equations allow us to make predictions, but extrapolation is less reliable than interpolation.

  6. A residual is the difference between an observed value and its predicted value.

  7. Residual plots help identify nonlinearity, unequal error variance, and unusual observations.

  8. SSE, MSE, RMSE, MAE, and R² provide different ways of evaluating model performance.

  9. A high R² does not prove causation or guarantee accurate future predictions.

  10. Reliable regression analysis requires appropriate assumptions, careful interpretation, and evaluation using new data.

   

  Check Your Understanding

  Question 1: Understanding Regression Equations

  A regression equation is:

  ŷ = 15 + 4x

  Identify the slope and intercept.

  Explain what each represents.

  Question 2: Making Predictions

  A business uses the regression equation:

  ŷ = 120 + 35x

  Where x is advertising expenditure in hundreds of dollars and ŷ is predicted sales revenue in hundreds of dollars.

  Predict the sales revenue when advertising expenditure is $500.

  Question 3: Calculating Residuals

  A regression model predicts a student's score to be 72 points.

  The student actually scores 78 points.

  Calculate the residual and interpret its sign.

  Question 4: Finding a Regression Line

  Use the following data to calculate the least-squares regression equation.

  x	y
  1	3
  2	5
  3	4
  4	8

  Calculate the slope and intercept.

  Question 5: Evaluating R²

  A regression model has R² = 0.64.

  Explain what this means and identify one conclusion that cannot be justified from R² alone.

  Question 6: Interpreting a Residual Plot

  A residual plot shows a clear U-shaped pattern.

  What does this suggest about the regression model?

  Question 7: Comparing Models

  Two regression models predict the same response variable on the same test dataset.

  Measure	Model A	Model B
  RMSE	4.2	2.7
  MAE	3.6	2.1

  Which model has better predictive accuracy according to these measures? Explain.

  Question 8: Extrapolation

  A scientist develops a regression model using temperatures between 10°C and 40°C.

  The model is then used to predict a response at 85°C.

  Explain why this prediction may be unreliable.

   

  Fully Worked Answers

  Answer 1

  The equation is:

  ŷ = 15 + 4x

  The slope is 4.

  This means the predicted response increases by 4 units for each one-unit increase in x.

  The intercept is 15.

  This means the predicted response is 15 when x = 0.

  Whether this intercept has practical meaning depends on the context and observed x-range.

  Answer 2

  Advertising expenditure is $500.

  Since x is measured in hundreds of dollars:

  x = 5

  Substitute:

  ŷ = 120 + 35(5)

  ŷ = 120 + 175

  ŷ = 295

  Since revenue is also measured in hundreds of dollars:

  295 × 100 = 29,500

  Predicted sales revenue = $29,500.

  Answer 3

  Residual = observed − predicted

  e = 78 − 72

  \[ \boxed{e=+6} \]

  The positive residual means the student's actual score was 6 points higher than the model predicted.

  Answer 4

  Construct the calculation table.

  x	y	x²	xy
  1	3	1	3
  2	5	4	10
  3	4	9	12
  4	8	16	32
  10	20	30	57

  n = 4

  Calculate the slope:

  \[ b=\frac{4(57)-(10)(20)}{4(30)-10^2} \]

  \[ b=\frac{228-200}{120-100} =\frac{28}{20} \]

  \[ b=1.4 \]

  Calculate the means:

  x̄ = 10/4 = 2.5

  ȳ = 20/4 = 5

  Calculate the intercept:

  a = 5 − 1.4(2.5)

  a = 5 − 3.5

  a = 1.5

  Therefore:

  \[ \boxed{\hat y=1.5+1.4x} \]

  Answer 5

  R² = 0.64 means the fitted model explains 64% of the variation in the observed response variable, relative to predicting its sample mean.

  The remaining 36% is not explained by the model.

  This does not prove that the explanatory variable causes changes in the response variable, nor does it guarantee accurate predictions on new data.

  Answer 6

  A U-shaped residual pattern suggests that the underlying relationship may be nonlinear.

  A straight-line model is systematically overpredicting or underpredicting in different regions.

  A nonlinear model, such as a quadratic regression model, may describe the relationship more appropriately.

  Answer 7

  Model B has the lower RMSE and MAE.

  RMSE:

  • Model A = 4.2

  • Model B = 2.7

  MAE:

  • Model A = 3.6

  • Model B = 2.1

  Both measures favour Model B.

  Therefore, Model B has better predictive accuracy on the given test dataset according to these measures.

  Answer 8

  The prediction at 85°C is extrapolation because 85°C lies outside the observed temperature range of 10°C to 40°C.

  The relationship may not remain linear at higher temperatures.

  The model may therefore produce an inaccurate or unrealistic prediction.

  Additional observations at higher temperatures would be needed to evaluate the relationship more reliably.

   

  Final Reflection

  Linear regression transforms patterns in scatter plots into mathematical models that can be interpreted, tested, and used for predictions.

  However, calculating a line of best fit is only part of a complete statistical investigation. We must also examine residuals, measure prediction errors, check model assumptions, and determine whether predictions are being made within a reasonable range.

  A useful regression model is not simply one that fits existing data closely. It is one that represents the relationship appropriately and makes sufficiently reliable predictions for its intended purpose.

Introduction

In many real-world situations, an outcome is influenced by several factors at the same time. A student's examination score might depend on study time, attendance, sleep, and previous knowledge. The price of a house might depend on its size, location, age, and condition. Scientists studying plant growth might investigate sunlight, water, temperature, and soil nutrients.

Multiple variable analysis is the process of investigating relationships among three or more variables. Rather than examining only one possible influence, we consider how several variables interact and contribute to an outcome.

This approach helps us identify important relationships, recognise misleading conclusions, make predictions, and evaluate statistical evidence.

By the end of this lesson, you will be able to identify relationships among multiple variables, recognise confounding variables, interpret multivariable data, evaluate statistical models, and communicate statistical reasoning clearly.

Understanding Multiple Variables

A variable is a characteristic or quantity that can change or take different values.

In a statistical investigation, variables often have different roles.

• Response variable (dependent variable): The outcome we want to explain or predict.

• Explanatory variable (independent variable): A variable that may help explain changes in the response.

• Control variable: A variable deliberately kept constant in an experiment.

• Confounding variable: A variable that complicates the interpretation of a relationship because it is associated with both the explanatory variable and the outcome.

Consider a study investigating examination performance.

A simple analysis might compare examination scores with study time.

A multiple variable analysis considers study time, attendance, sleep, and previous achievement together.

This provides a more complete picture of student performance.

Identifying Relationships Among Multiple Variables

When analysing multiple variables, we look for patterns that explain how variables are related.

Positive relationships

A positive relationship occurs when higher values of one variable tend to be associated with higher values of another.

For example:

• More study time may be associated with higher examination scores.

• Greater work experience may be associated with higher salaries.

• Higher temperatures may be associated with greater electricity use for cooling.

Negative relationships

A negative relationship occurs when higher values of one variable tend to be associated with lower values of another.

For example:

• Greater distance from a city centre may be associated with lower property prices.

• More hours of exercise may be associated with a lower resting heart rate.

• Higher product prices may be associated with lower demand.

Nonlinear relationships

Not all relationships follow straight lines.

For example, examination scores may improve rapidly with additional study time initially, but the benefits may become smaller after many hours.

A model that assumes a constant increase might therefore be inappropriate.

Relationships involving interactions

Sometimes the relationship between two variables depends on a third variable.

For example, the effect of study time on examination scores might depend on sleep duration.

Students who study for four hours but sleep very little may perform differently from students who study for four hours and get sufficient sleep.

This is called an interaction effect.

The graph suggests that both study time and sleep may be related to performance. However, it does not prove that either variable causes the differences.

Recognising Confounding Variables

A confounding variable is an additional variable that can distort the apparent relationship between an explanatory variable and an outcome.

Confounding is particularly important in observational studies.

Example: Ice cream sales and drowning incidents

Suppose a researcher discovers that months with high ice cream sales also have more drowning incidents.

Does eating ice cream cause drowning?

No.

A possible explanation is that both variables are influenced by temperature.

Introduction

How do scientists predict tomorrow's temperature? How do businesses estimate future sales? How can researchers investigate the factors that influence examination scores, disease rates, or population growth?

All these questions involve statistical modelling.

A statistical model is a mathematical representation of patterns and relationships in data. It allows us to describe observed behaviour, investigate relationships between variables, estimate unknown quantities, and make predictions.

Unlike a purely deterministic mathematical formula, a statistical model recognises that real-world data contain variation and uncertainty. Two students who study for the same number of hours may receive different examination scores. Two houses with similar characteristics may sell for different prices.

Statistical models help us understand these differences while identifying useful patterns.

In this lesson, we will explore how to develop statistical models, assess their assumptions, evaluate their performance, interpret their outputs, and apply them to real-world problems.

What Is a Statistical Model?

A statistical model is a mathematical description of how one or more variables are related, often including a component representing random variation.

A simple statistical model can be written as:

Where:

• Y = observed response variable

• X = explanatory variable

• f(X) = systematic relationship between the variables

• ε = random error or unexplained variation

For example, examination scores may be related to study time.

A possible model is:

This suggests that the expected examination score increases by 8 percentage points for each additional hour of study, within the range where the model is appropriate.

The error term accounts for other influences such as sleep, motivation, previous knowledge, and random variation.

Why do we use statistical models?

Statistical models can serve several purposes.

• Description: Summarising patterns in a dataset.

• Explanation: Investigating relationships among variables.

• Prediction: Estimating future or unknown outcomes.

• Inference: Drawing conclusions about a population from sample data.

• Decision-making: Using evidence to compare possible actions.

A model designed for prediction may differ from one designed to estimate causal effects.

Developing a Statistical Model

Statistical modelling involves several connected stages.

Introduction

Imagine that a school conducts a survey to investigate whether students who sleep longer achieve higher examination scores. After collecting data from 200 students, the researchers discover a positive relationship between sleep duration and academic performance.

But how should they communicate this discovery?

Should they simply announce that students who sleep longer achieve better grades? Should they present a table containing all 200 observations? Or should they use graphs, statistical summaries, and carefully explained conclusions?

Collecting and analysing data are only part of a statistical investigation. The results must also be communicated in ways that are clear, accurate, meaningful, and supported by evidence.

Statistical communication involves selecting appropriate graphs and numerical summaries, explaining patterns, justifying conclusions, acknowledging uncertainty, and making recommendations based on reliable evidence.

These skills are essential in science, business, education, medicine, economics, environmental research, and public policy.

A well-communicated statistical investigation allows readers to understand not only what the data suggest, but also how the conclusions were reached and how much confidence should be placed in them.

The Purpose of Statistical Communication

Statistical communication is the process of presenting data, analyses, interpretations, and conclusions so that an audience can understand and evaluate them.

Effective statistical communication answers five important questions:

1. What was investigated? State the research question and purpose.

2. How was the investigation conducted? Explain the data collection and analysis methods.

3. What did the data show? Present important numerical results and visualisations.

4. What do the results mean? Interpret the findings in context.

5. How reliable are the conclusions? Discuss uncertainty, limitations, and possible sources of error.

Statistical communication should be understandable without oversimplifying or misrepresenting the evidence.

Description, interpretation, and conclusion

These three elements serve different purposes.

Consider a hypothetical investigation of study time and examination performance.

A statistical report should distinguish clearly between what was observed and what can reasonably be concluded.

Presenting Statistical Findings Clearly

The most effective way to present statistical information depends on the type of data, the purpose of the analysis, and the intended audience.

Choosing appropriate graphs

Graphs should make important relationships easier to understand, not simply make a report more colourful.

Example: Comparing examination results

Suppose two classes complete the same examination.

A suitable description is:

"Class B achieved a higher mean examination score (78%) than Class A (72%). Class B also had a lower standard deviation (7 compared with 10 percentage points), indicating less variation in scores as measured by standard deviation."

This statement identifies both the difference in average performance and the difference in variability.

It does not claim that Class B's teacher or teaching method caused the difference.

Principles of effective graphs

A well-designed statistical graph should:

• Have a clear, informative title.

• Label axes and include measurement units.

• Use an appropriate scale.

• Distinguish groups clearly.

• Avoid unnecessary decoration.

• Include sample sizes when relevant.

• Explain uncertainty where appropriate.

• Accurately represent the underlying data.

For bar charts, a zero baseline is generally important because bar length represents magnitude. Scatter plots and line graphs may use narrower scales when clearly labelled and appropriate.

Avoiding Misleading Statistical Visualisations

Graphs can influence how an audience interprets results.

Poor graph design may exaggerate differences, hide variation, or suggest relationships that are not supported by the data.

Example: Misleading axis scales

Suppose a company's sales increase from $100,000 to $110,000.

The actual percentage increase is:

A bar chart beginning at $95,000 could make the difference appear enormous because the second bar would be three times the visible height of the first.

A bar chart beginning at zero represents the relative magnitudes more faithfully.