Xrfgtjtk

Like a lot of people, I understand how to run a linear regression, I understand how to interpret its output, and I understand its limitations.

My understanding of the mathematical underpinnings of linear regression, however, are less developed. In particular, I do not understand the logic behind how we estimate beta using the following formula:

$$ beta = (X'X)^{-1}X'Y $$

Would anyone care to offer an intuitive explanation as to why/how this process works? For example, what function each step in the equation performs and why it is necessary.

Suppose you have a model of the form:
$$X beta= Y$$
where X is a normal 2-D matrix, for ease of visualisation.
Now, if the matrix $X$ is square and invertible, then getting $beta$ is trivial:
$$beta= X^{-1}Y$$
And that would be the end of it.

If this is not the case, to get $beta$ you’ll have to find a way to “approximate” the result of an inverse matrix. $X^dagger = (X'X)^{-1}X'$ is called the (left)-pseudoinverse, and it has some nice properties that make it useful for this application.

In particular, it is unique, and $XX^dagger X=X$, so it kind of works like an inverse matrix would $(XX^{-1}X = XI = X)$. Also, for an invertible and square matrix (i.e. if the inverse matrix exists), it is equal to $X^{-1}$.

Also it gets the shape of the matrix right: If $X$ has order $n times m$, our pseudoinverse should be $m times n$ so we can multiply it with $Y$. This is achieved by multiplying $(X'X)^{-1}$, which is square $(m times m)$, with X' $(m times n)$.

If you look at sources such as wikipedia, there are some good explanations for where this comes from. Here are some cores ideas:

1. OLS is aiming to minimize the error $||y-Xbeta||$.




2. The norm of a vector is minimized when its derivative is perpendicular to the vector. (Since you asked for ELI5, I won't go into a rigorous formulation of "derivative" in this context.)




3. The error is given in terms of $y$, $X$, and $beta$. The first two are constants; we're varying only $beta$. Thus, the derivative can be treated as being $Xbeta'$, so we're looking for $(Xbeta')^T(y-Xbeta)=0$. This is equivalent to $(beta')^TX^Ty=(beta')^TX^TXbeta$. If we cancel the $(beta')^T$ from both sides (normally in linear algebra, you can't just go around canceling things, but I'm not aiming for perfect rigor here, so I won't get into the justification), we're left with $X^Ty=X^TXbeta$. Now, $X^T$ isn't invertible (it isn't even square), so we can't cancel it out, but it does turn out that $X^TX$ must be invertible (assuming that the features are linearly independent). So we can get $beta = (X^TX)^{-1}X^Ty$.

Going back to $X^Ty=X^TXbeta$, recall that $Xbeta$ is the the estimate $hat y$ that is calculated from a given $beta$. $X^Ty$ is a vector in which each entry is the dot product of one of the features with the response. So we have that $X^Ty=X^That y$, i.e., for each feature, the dot product between that feature and the actual response is equal to the dot product between that feature and the estimated response. $forall i, x_i^Ty=x_i^That y$. We can view OLS, then, as solving $n$ equations $x_i^Ty=x_i^That y$, where $n$ is the number of features. So to see why this works, we just need to show that a solution exists, and that any estimate of the response other than this solution will have larger squared error.