How do we get the “best” values for the slope and intercept?
How would you draw the “best” line?
How would you draw the “best” line?
Least squares regression
Remember the residual is the difference between the actual value and the predicted value
The regression line minimizes the sum of squared residuals.
Least squares regression
Residual for each point is: \(e_i = y_i - \hat{y}_i\)
Least squares regression line minimizes \(\sum_{i = 1}^n e_i^2\).
Why do we square the residual?
Why not take absolute value?
Principle: larger penalty for residuals further away
Math: makes the math easier and some nice properties (not our concern here…)
Least squares regression
Very Simple Example
What should the slope and intercept be?
Example
\(\hat{Y} = 0 + 1*X\)
Example
What is the sum of squared residuals?
Example
What is sum of squared residuals for \(y = 0 + 0*X\)?
Example
What is sum of squared residuals for \(y = 0 + 0*X\)?
(1-0)^2+ (2-0)^2+ (3-0)^2
[1] 14
Example
What is sum of squared residuals for \(y = 0 + 2*X\)?
Example
What is sum of squared residuals for \(y = 0 + 2*X\)?
(1-2)^2+ (2-4)^2+ (3-6)^2
[1] 14
One more…
What is sum of squared residuals for \(y = 0 + -1*X\)?
One more…
What is sum of squared residuals for \(y = 0 + -1*X\)?
(1+1)^2+ (2+2)^2+ (3+3)^2
[1] 56
Cost Function
Sum of Squared Residuals as function of possible values of \(b\)
Least Squares Regression
When we estimate a least squares regression, it is looking for the line that minimizes sum of squared residuals
In the simple example, I set \(a=0\) to make it easier. More complicated when searching for combination of \(a\) and \(b\) that minimize, but same basic idea
Least Squares Regression
There is a way to solve for this analytically for linear regression (i.e., by doing math…)
– They made us do this in grad school…
In machine learning, people also use gradient descent algorithm in which the computer searches over possible combinations of \(a\) and \(b\) until it settles on the lowest point.
Least Squares Regression
Least Squares Regression
Your Turn
Are democracies less corrupt?
V-Dem includes a Political Corruption Index, which aggregates corruption in a number of spheres (see codebook for details).
The variable name is: v2x_corr : lower values mean less corruption
