Ridge Regression and the Influence of a Single Point

Lagrange multipliers, Cook’s distance, and Sherman–Morrison

Will Gao · March 2025

Contents

1 Linear Regression
1.1 Probabilistic Formulation
2 Regularization and Influence Diagnostics
3 Ridge Regression
3.1 Constrained Formulation
3.2 A Geometric View of Lagrange Multipliers
3.3 Unconstrained Formulation
3.4 Relation Between the Constrained and Penalized Forms
4 Cook’s Distance
4.1 Matrix Determinant Lemma
4.2 Sherman–Morrison Formula
4.3 Derivation of the Influence Formula
5 Autoregression

These notes are adapted from Prof. Alexis Drouot’s MATH 480 course at the University of Washington, Seattle (Winter 2025). They focus on several topics around linear regression: its probabilistic formulation, Ridge regularization, influence diagnostics through Cook’s distance, and the representation of autoregressive models as linear regressions. I have done my best to ensure correctness, but errors may remain.

1Linear Regression

1.1Probabilistic Formulation

Consider a dataset of $N$ observations

\{(x_i, y_i)\}_{i=1}^{N},

where $x_i \in \mathbb{R}^d$ are the predictors and $y_i \in \mathbb{R}$ is the response.

In the probabilistic formulation of linear regression, we assume

y_i = x_i^{\top} a + \epsilon_i,

where $a \in \mathbb{R}^d$ is the parameter vector and

\epsilon_i \stackrel{\text{i.i.d.}}{\sim} \mathcal{N}(0, \sigma^2).

Hence

p(y_i \mid x_i, a, \sigma^2) \;=\; \frac{1}{\sqrt{2\pi\sigma^2}} \exp\!\left( -\frac{(y_i - x_i^{\top} a)^2}{2\sigma^2} \right).

Assuming the observations are independent,

p(y \mid X, a, \sigma^2) \;=\; \prod_{i=1}^{N} \frac{1}{\sqrt{2\pi\sigma^2}} \exp\!\left( -\frac{(y_i - x_i^{\top} a)^2}{2\sigma^2} \right).

Taking logarithms,

\log p(y \mid X, a, \sigma^2) \;=\; -\frac{N}{2}\log(2\pi\sigma^2) \;-\; \frac{1}{2\sigma^2} \sum_{i=1}^{N} (y_i - x_i^{\top} a)^2 .

Therefore, for fixed $\sigma^2$ , maximizing the likelihood with respect to $a$ is equivalent to minimizing

\sum_{i=1}^{N} (y_i - x_i^{\top} a)^2 .

In matrix notation,

L(a) = \lVert Xa - y \rVert^2 ,

where $X \in \mathbb{R}^{N \times d}$ is the design matrix and $y \in \mathbb{R}^{N}$ is the response vector.

The gradient and Hessian are

\nabla L(a) = 2X^{\top}(Xa - y), \qquad \nabla^2 L(a) = 2X^{\top}X .

If $X$ has linearly independent columns, then $X^{\top}X$ is positive definite, so $L$ is strictly convex and the OLS minimizer is unique:

a_{\mathrm{OLS}} = (X^{\top}X)^{-1} X^{\top} y .

If the Gaussian i.i.d. noise assumption is changed, the corresponding likelihood—and hence the natural loss function—may also change.

2Regularization and Influence Diagnostics

OLS can be sensitive to influential observations because squared residuals place substantial weight on large errors. Ridge regression does not make least squares robust in the usual statistical sense; instead, it stabilizes coefficient estimates by shrinking their magnitude and is especially useful in settings involving multicollinearity, ill-conditioning, or high variance.

Influential observations can be studied separately using diagnostics such as Cook’s distance. Such diagnostics are best interpreted as tools for identifying observations that deserve further investigation, rather than as automatic rules for deleting data.

We first study Ridge regression and then return to influence diagnostics.

3Ridge Regression

3.1Constrained Formulation

A constrained form of Ridge regression is

(M_t) \qquad \min_{a} \; \lVert Xa - y \rVert^2 \quad \text{subject to} \quad \lVert a \rVert^2 \le t,

where $t \ge 0$ .

The feasible set

B_t = \{ a \in \mathbb{R}^d : \lVert a \rVert^2 \le t \}

is compact, and the objective is continuous, so a minimizer exists.

If $a_{\mathrm{OLS}}$ satisfies

\lVert a_{\mathrm{OLS}} \rVert^2 \le t,

then the constraint is inactive and the constrained solution is simply $a_{\mathrm{OLS}}$ .

Otherwise, the optimum must lie on the boundary:

\lVert a \rVert^2 = t .

Because the problem is convex, the Karush–Kuhn–Tucker conditions characterize the optimum. Introduce

\mathcal{L}(a, \lambda) = \lVert Xa - y \rVert^2 + \lambda\big( \lVert a \rVert^2 - t \big),

with $\lambda \ge 0$ .

The KKT conditions are

\nabla_a \mathcal{L}(a_*, \lambda_*) = 0, \qquad \lVert a_* \rVert^2 \le t, \qquad \lambda_* \ge 0,

and

\lambda_*\big( \lVert a_* \rVert^2 - t \big) = 0 .

The stationarity condition gives

2X^{\top}(Xa_* - y) + 2\lambda_* a_* = 0,

(X^{\top}X + \lambda_* I_d)\, a_* = X^{\top} y .

If $\lambda_* > 0$ , then $X^{\top}X + \lambda_* I_d$ is positive definite because, for any nonzero $v$ ,

v^{\top}(X^{\top}X + \lambda_* I_d) v = \lVert Xv \rVert^2 + \lambda_* \lVert v \rVert^2 > 0 .

Hence it is invertible, even if $X$ does not have linearly independent columns.

Therefore,

a_* = (X^{\top}X + \lambda_* I_d)^{-1} X^{\top} y .

The complementary-slackness condition separates the two cases:

If $\lambda_* = 0$ , the constraint is inactive and $a_* = a_{\mathrm{OLS}}$ , assuming the OLS solution is uniquely defined.
If $\lambda_* > 0$ , then the constraint is active: $\lVert a_* \rVert^2 = t$ .

3.2A Geometric View of Lagrange Multipliers

For completeness, consider the equality-constrained problem

\min_{a} f(a) \quad \text{subject to} \quad g(a) = 0 .

Let

S = \{ a \in \mathbb{R}^d : g(a) = 0 \},

and suppose $a_* \in S$ is a local extremum with $\nabla g(a_*) \neq 0$ .

The tangent space at $a_*$ can be described by

T_{a_*} S = \{ \gamma'(0) : \gamma : (-\varepsilon, \varepsilon) \to S, \; \gamma(0) = a_* \},

where $\gamma$ is $C^1$ .

For any such curve,

\left. \frac{d}{dt} f(\gamma(t)) \right|_{t=0} = \nabla f(a_*)^{\top} \gamma'(0) .

Since $a_*$ is a local extremum of $f$ restricted to $S$ ,

\nabla f(a_*)^{\top} v = 0 \qquad \forall\, v \in T_{a_*} S .

By the regular-level-set theorem,

T_{a_*} S = \big( \operatorname{span}\{ \nabla g(a_*) \} \big)^{\perp} .

Thus

\nabla f(a_*) \in \operatorname{span}\{ \nabla g(a_*) \},

so there exists $\lambda_* \in \mathbb{R}$ such that

\nabla f(a_*) + \lambda_* \nabla g(a_*) = 0 .

This is the usual Lagrange-multiplier condition. It is a necessary condition for regular constrained local extrema, not in general a sufficient condition for global optimality.

3.3Unconstrained Formulation

The more common form of Ridge regression is

(M_\lambda) \qquad \min_{a} \; \lVert Xa - y \rVert^2 + \lambda \lVert a \rVert^2 ,

where $\lambda > 0$ .

Define

L_\lambda(a) = \lVert Xa - y \rVert^2 + \lambda \lVert a \rVert^2 .

Then

\nabla L_\lambda(a) = 2X^{\top}(Xa - y) + 2\lambda a, \qquad \nabla^2 L_\lambda(a) = 2X^{\top}X + 2\lambda I_d .

For any nonzero $h$ ,

h^{\top} \nabla^2 L_\lambda(a)\, h = 2\lVert Xh \rVert^2 + 2\lambda \lVert h \rVert^2 > 0 .

Thus $L_\lambda$ is strictly convex for every $\lambda > 0$ , regardless of whether $X$ has full column rank. Consequently, it has a unique minimizer satisfying

(X^{\top}X + \lambda I_d)\, a_* = X^{\top} y,

namely

a_* = (X^{\top}X + \lambda I_d)^{-1} X^{\top} y .

3.4Relation Between the Constrained and Penalized Forms

The constrained and penalized formulations describe the same family of nonzero Ridge solutions, with some care at the endpoints.

Suppose $t > 0$ , and let $a_t$ solve

\min_{a} \lVert Xa - y \rVert^2 \quad \text{subject to} \quad \lVert a \rVert^2 \le t .

By the KKT conditions, there exists $\lambda_t \ge 0$ such that

(X^{\top}X + \lambda_t I_d)\, a_t = X^{\top} y .

Therefore, $a_t$ also minimizes

\lVert Xa - y \rVert^2 + \lambda_t \lVert a \rVert^2 .

Conversely, for any $\lambda > 0$ , let

a_\lambda = (X^{\top}X + \lambda I_d)^{-1} X^{\top} y

and set $t = \lVert a_\lambda \rVert^2$ . Then $a_\lambda$ solves the constrained problem with radius $t$ .

There are two endpoint subtleties:

If the constraint is sufficiently loose, then $\lambda = 0$ and the constrained solution is OLS.
If $t = 0$ , the constrained solution is $a = 0$ . In general this is not obtained for any finite $\lambda$ , but $a_\lambda \to 0$ as $\lambda \to \infty$ .

4Cook’s Distance

Cook’s distance is an influence diagnostic that measures how much the fitted regression model changes when a single observation is removed.

Assume $X$ has full column rank. Let

a = (X^{\top}X)^{-1} X^{\top} y

be the OLS estimator using all observations.

For observation $j$ , let $X_{-j}$ and $y_{-j}$ denote the dataset with the $j$ -th observation removed, and define the leave-one-out estimator

a^{(-j)} = (X_{-j}^{\top} X_{-j})^{-1} X_{-j}^{\top} y_{-j},

assuming $X_{-j}$ also has full column rank.

To compare the two fitted models on the same $N$ design points, define the unscaled influence measure

D_j = \lVert Xa - Xa^{(-j)} \rVert^2 .

Let

H = X(X^{\top}X)^{-1}X^{\top}

be the hat matrix. We will show that

D_j = \frac{H_{jj}}{(1 - H_{jj})^2} \, (y_j - x_j^{\top} a)^2 .

4.1Matrix Determinant Lemma

Lemma 1. Suppose $A$ is an invertible $n \times n$ matrix and $u, v \in \mathbb{R}^n$ . Then

\det(A + uv^{\top}) = (1 + v^{\top} A^{-1} u) \det(A) .

Proof. First consider $A = I$ . We have

\begin{pmatrix} I & 0 \\ v^{\top} & 1 \end{pmatrix}\begin{pmatrix} I + uv^{\top} & u \\ 0 & 1 \end{pmatrix}\begin{pmatrix} I & 0 \\ -v^{\top} & 1 \end{pmatrix}=\begin{pmatrix} I & u \\ 0 & 1 + v^{\top} u \end{pmatrix} .

The first and third matrices have determinant $1$ , so

\det(I + uv^{\top}) = 1 + v^{\top} u .

For general $A$ ,

A + uv^{\top} = A(I + A^{-1}uv^{\top}),

and therefore

\det(A + uv^{\top}) = \det(A)\det\!\big(I + (A^{-1}u)v^{\top}\big) = \det(A)\,(1 + v^{\top}A^{-1}u). \qquad \Box

4.2Sherman–Morrison Formula

Lemma 2. Let $A$ be invertible and suppose $1 + v^{\top}A^{-1}u \neq 0$ . Then

(A + uv^{\top})^{-1} = A^{-1} - \frac{A^{-1}uv^{\top}A^{-1}}{1 + v^{\top}A^{-1}u} .

Proof. By the matrix determinant lemma, $A + uv^{\top}$ is invertible. Define

B = A^{-1} - \frac{A^{-1}uv^{\top}A^{-1}}{1 + v^{\top}A^{-1}u} .

A direct multiplication shows

(A + uv^{\top})B = I .

Hence $B = (A + uv^{\top})^{-1}$ . $\Box$

4.3Derivation of the Influence Formula

Since removing the $j$ -th observation gives

X_{-j}^{\top} X_{-j} = X^{\top}X - x_j x_j^{\top},

the Sherman–Morrison formula yields

(X_{-j}^{\top} X_{-j})^{-1} = (X^{\top}X)^{-1} + \frac{(X^{\top}X)^{-1} x_j x_j^{\top} (X^{\top}X)^{-1}}{1 - x_j^{\top}(X^{\top}X)^{-1}x_j},

provided

1 - x_j^{\top}(X^{\top}X)^{-1}x_j \neq 0 .

Also,

X_{-j}^{\top} y_{-j} = X^{\top} y - x_j y_j .

Let

h_{jj} = x_j^{\top}(X^{\top}X)^{-1}x_j = H_{jj} .

A calculation gives

a - a^{(-j)} = \frac{(X^{\top}X)^{-1}x_j}{1 - h_{jj}} \, (y_j - x_j^{\top}a) .

Therefore

Xa - Xa^{(-j)} = \frac{X(X^{\top}X)^{-1}x_j}{1 - h_{jj}} \, (y_j - x_j^{\top}a),

\begin{aligned}D_j = \lVert Xa - Xa^{(-j)} \rVert^2&= \frac{\lVert X(X^{\top}X)^{-1}x_j \rVert^2}{(1 - h_{jj})^2}\,(y_j - x_j^{\top}a)^2 \\[4pt]&= \frac{x_j^{\top}(X^{\top}X)^{-1}X^{\top}X(X^{\top}X)^{-1}x_j}{(1 - h_{jj})^2}\,(y_j - x_j^{\top}a)^2 \\[4pt]&= \frac{h_{jj}}{(1 - h_{jj})^2}\,(y_j - x_j^{\top}a)^2 .\end{aligned}

This is an unscaled version of Cook’s distance. A common conventional definition additionally divides by a factor involving the number of fitted parameters and an estimate of the error variance. The unscaled expression above is still useful for seeing how leverage and residual size jointly determine influence.

5Autoregression

Autoregressive models use past values of a time series to predict its current value.

A zero-intercept AR( $p$ ) model is

X_t = \phi_1 X_{t-1} + \phi_2 X_{t-2} + \cdots + \phi_p X_{t-p} + \epsilon_t,

where

$X_t$ is the value of the time series at time $t$ ,
$\phi_1, \ldots, \phi_p$ are autoregressive coefficients,
$\epsilon_t$ is a white-noise error term with mean zero and constant variance.

If a nonzero intercept is desired, one can instead write

X_t = c + \phi_1 X_{t-1} + \cdots + \phi_p X_{t-p} + \epsilon_t .

For simplicity, consider the zero-intercept case. Given

X_1, X_2, \ldots, X_T,

define

Y = \begin{pmatrix} X_{p+1} \\ X_{p+2} \\ \vdots \\ X_T \end{pmatrix}

and the lagged design matrix

Z = \begin{pmatrix}X_p & X_{p-1} & \cdots & X_1 \\X_{p+1} & X_p & \cdots & X_2 \\\vdots & \vdots & \ddots & \vdots \\X_{T-1} & X_{T-2} & \cdots & X_{T-p}\end{pmatrix} .

Let

\Phi = \begin{pmatrix} \phi_1 \\ \phi_2 \\ \vdots \\ \phi_p \end{pmatrix},\qquad\epsilon = \begin{pmatrix} \epsilon_{p+1} \\ \epsilon_{p+2} \\ \vdots \\ \epsilon_T \end{pmatrix} .

Then the AR model can be written as

Y = Z\Phi + \epsilon .

Thus estimation of the AR coefficients can be viewed as a linear regression problem on lagged observations. The notation $Z$ is used for the design matrix to avoid confusing it with the time-series variable $X_t$ .