Minimizing RSS

Simple Linear Regression with distinct x values – Everything Is OK

Suppose we have 3 observations:

• \(y_1 = 2\), \(x_1 = 1\) (first observation is in the first group)
• \(y_2 = 4\), \(x_2 = 2\) (second observation is in the second group)
• \(y_3 = 5\), \(x_3 = 3\) (third observation is in the second group)

Our model is

\[ y_i = \beta_0 + \beta_1 x_i + \varepsilon_i \\ \varepsilon_i \sim \text{Normal}(0, \sigma^2) \]

The design matrix is \(X = \begin{bmatrix} 1 & 1 \\ 1 & 2 \\ 1 & 3 \end{bmatrix}\)

We know that we can use the following code to find the estimates of \(\beta_0\) and \(\beta_1\):

X <- cbind(
  c(1, 1, 1),
  c(1, 2, 3)
)
y <- matrix(c(2, 4, 5))

beta_hat <- solve(t(X) %*% X) %*% t(X) %*% y
beta_hat

##           [,1]
## [1,] 0.6666667
## [2,] 1.5000000

Here is a picture of the RSS as a function of \(\beta_0\) and \(\beta_1\), with our estimates \((\hat{\beta}_0, \hat{\beta}_1)\) shown with a red point:

\[RSS = \sum_{i = 1}^n (y_i - \hat{y}_i)^2 = \{2 - (\beta_0 + \beta_1 \cdot 1)\}^2 + \{4 - (\beta_0 + \beta_1 \cdot 2)\}^2 + \{5 - (\beta_0 + \beta_1 \cdot 3)\}^2\]

Simple Linear Regression with one x value – Everything Is Broken

Suppose we have 3 observations:

• \(y_1 = 2\), \(x_1 = 2\) (first observation is in the first group)
• \(y_2 = 4\), \(x_2 = 2\) (second observation is in the second group)
• \(y_3 = 5\), \(x_3 = 2\) (third observation is in the second group)

Our model is

\[ y_i = \beta_0 + \beta_1 x_i + \varepsilon_i \\ \varepsilon_i \sim \text{Normal}(0, \sigma^2) \]

The design matrix is \(X = \begin{bmatrix} 1 & 2 \\ 1 & 2 \\ 1 & 2 \end{bmatrix}\)

We know that there is not a unique \(\hat{\beta}\) that minimizes RSS because the columns of \(X\) are not linearly independent.

Here is a picture of the RSS as a function of \(\beta_0\) and \(\beta_1\):

\[RSS = \sum_{i = 1}^n (y_i - \hat{y}_i)^2 = \{2 - (\beta_0 + \beta_1 \cdot 2)\}^2 + \{4 - (\beta_0 + \beta_1 \cdot 2)\}^2 + \{5 - (\beta_0 + \beta_1 \cdot 2)\}^2\]

Note that there is no unique pair \((\beta_0, \beta_1)\) that minimizes RSS.