Lecture 10: Gaussian Processes

The isonormal process, the Karhunen–Loève expansion, and Stein’s lemma

Published

October 19, 2026

Several objects from earlier lectures are random functions of a parameter. The Gaussian width of Lecture 3 is the expected supremum of \(t\mapsto\langle g,t\rangle\) over a set \(T\). The singular values of a Gaussian matrix are the extreme values of \((u,v)\mapsto\langle Au,v\rangle\) over pairs of unit vectors. In each case the random function is jointly Gaussian, and Lecture 9 ended by claiming that the sharpest results about such functions come not from discretizing the parameter set but from comparing one Gaussian function to another. This lecture sets up the language for that: Gaussian processes, the metric they induce on their index set, and the two tools, the isonormal process and Stein’s lemma, on which the next two lectures rest.

1 The multivariate normal

Definition 1 (Gaussian random vector) A random vector \(X\) in \(\mathbb R^d\) is Gaussian, written \(X\sim N(\mu,\Sigma)\), if there are \(\mu\in\mathbb R^d\) and a positive semidefinite \(\Sigma\in\mathbb R^{d\times d}\) such that \[ \mathbb Ee^{i\langle\xi,X\rangle}=\exp\Big(i\langle\xi,\mu\rangle-\frac12\langle\xi,\Sigma\xi\rangle\Big) \qquad\text{for every }\xi\in\mathbb R^d . \] Then \(\mu=\mathbb EX\) and \(\Sigma=\mathbb E(X-\mu)(X-\mu)^{\mathsf T}\).

The definition through the characteristic function makes the basic manipulations immediate, because the characteristic function determines the law and behaves well under linear maps.

Proposition 1 (Linear images and the square-root representation)  

  1. If \(X\sim N(\mu,\Sigma)\) and \(A\in\mathbb R^{k\times d}\), \(b\in\mathbb R^k\), then \(AX+b\sim N(A\mu+b,\ A\Sigma A^{\mathsf T})\).
  2. If \(Z=(Z_1,\ldots,Z_d)\) has i.i.d. \(N(0,1)\) coordinates, then \(Z\sim N(0,I_d)\), and \(\mu+AZ\sim N(\mu,AA^{\mathsf T})\) for every \(A\).
  3. Conversely, if \(X\sim N(\mu,\Sigma)\), then \(X\) has the same law as \(\mu+\Sigma^{1/2}Z\), where \(\Sigma^{1/2}\) is the positive semidefinite square root of \(\Sigma\). In particular \(N(\mu,\Sigma)\) exists for every \(\mu\) and every positive semidefinite \(\Sigma\).
  4. If \(\Sigma\) is invertible, \(X\) has density \((2\pi)^{-d/2}(\det\Sigma)^{-1/2}\exp\big(-\frac12\langle x-\mu,\Sigma^{-1}(x-\mu)\rangle\big)\).

For (1), \[ \mathbb Ee^{i\langle\xi,AX+b\rangle} =e^{i\langle\xi,b\rangle}\,\mathbb Ee^{i\langle A^{\mathsf T}\xi,X\rangle} =\exp\Big(i\langle\xi,A\mu+b\rangle-\frac12\langle A^{\mathsf T}\xi,\Sigma A^{\mathsf T}\xi\rangle\Big), \] and \(\langle A^{\mathsf T}\xi,\Sigma A^{\mathsf T}\xi\rangle=\langle\xi,A\Sigma A^{\mathsf T}\xi\rangle\). For (2), independence factors the characteristic function of \(Z\) into \(\prod_je^{-\xi_j^2/2}=e^{-\|\xi\|_2^2/2}\), and then (1) applies. For (3), by (2) the vector \(\mu+\Sigma^{1/2}Z\) is \(N(\mu,\Sigma)\), and two random vectors with the same characteristic function have the same law. For (4), change variables in the density of \(Z\) using the map \(z\mapsto\mu+\Sigma^{1/2}z\), whose Jacobian is \((\det\Sigma)^{1/2}\).

Two consequences are used constantly.

Rotation invariance. If \(U\) is orthogonal and \(Z\sim N(0,I_d)\), then \(UZ\sim N(0,UU^{\mathsf T})=N(0,I_d)\). The standard Gaussian vector has no preferred direction.

Independence is a matter of covariance. If \(X\sim N(\mu,\Sigma)\) and \(\Sigma\) is block diagonal with respect to a splitting of the coordinates, then the characteristic function factors over the blocks, so the corresponding sub-vectors of \(X\) are independent. For jointly Gaussian variables, uncorrelated means independent. This is false without joint Gaussianity, and it is the reason the joint law matters in the definition of a Gaussian process below.

For a standard Gaussian vector \(g\sim N(0,I_d)\) and fixed \(u,v\in\mathbb R^d\), part (1) gives \[ \langle g,u\rangle\sim N(0,\|u\|_2^2), \qquad \mathbb E\langle g,u\rangle\langle g,v\rangle=\langle u,v\rangle . \] So the linear forms of a standard Gaussian vector reproduce the Euclidean geometry of \(\mathbb R^d\) inside a space of random variables. This observation is the seed of the whole lecture.

2 Gaussian processes and the canonical metric

Definition 2 (Gaussian process) Let \(T\) be a set. A collection of random variables \((X_t)_{t\in T}\) on a common probability space is a Gaussian process if for every finite \(T'\subseteq T\) the vector \((X_t)_{t\in T'}\) is Gaussian. It is centered if \(\mathbb EX_t=0\) for every \(t\).

Two functions describe a Gaussian process completely, since they determine every finite-dimensional law: \[ \mu(t):=\mathbb EX_t, \qquad K(s,t):=\mathbb E\big(X_s-\mu(s)\big)\big(X_t-\mu(t)\big), \] the mean and the covariance. The covariance is positive semidefinite in the sense that for all \(t_1,\ldots,t_n\in T\) and real \(a_1,\ldots,a_n\), \[ \sum_{i,j}a_ia_jK(t_i,t_j)=\operatorname{Var}\Big(\sum_ia_iX_{t_i}\Big)\geq0 . \] A third function is a repackaging of these two, but it is the one that carries the geometry.

Definition 3 (The canonical metric) The canonical metric of \((X_t)_{t\in T}\) is \[ d(s,t):=\big(\mathbb E(X_s-X_t)^2\big)^{1/2} =\big((\mu(s)-\mu(t))^2+K(s,s)+K(t,t)-2K(s,t)\big)^{1/2}. \]

This is the \(L^2(\mathbb P)\) distance between the random variables \(X_s\) and \(X_t\), so it satisfies the triangle inequality. It is a pseudometric: \(d(s,t)=0\) means \(X_s=X_t\) almost surely, which can happen for \(s\neq t\), and we identify such indices when convenient. The geometry of \((T,d)\) is the input to every estimate on \(\sup_tX_t\) in the rest of the course.

Theorem 1 (Existence) Let \(T\) be a set, \(\mu:T\to\mathbb R\) any function, and \(K:T\times T\to\mathbb R\) symmetric and positive semidefinite. Then there is a probability space carrying a Gaussian process with mean \(\mu\) and covariance \(K\).

For each finite \(F\subseteq T\), the restriction of \(K\) to \(F\times F\) is a positive semidefinite matrix, so by Proposition 1 the law \(N(\mu|_F,K|_{F\times F})\) on \(\mathbb R^F\) exists. These laws are consistent: if \(F\subseteq F'\), the marginal of \(N(\mu|_{F'},K|_{F'\times F'})\) on the coordinates in \(F\) is \(N(\mu|_F,K|_{F\times F})\), by part (1) of Proposition 1 applied to the coordinate projection. The Kolmogorov extension theorem then provides a probability measure on \(\mathbb R^T\), with the product \(\sigma\)-algebra, whose finite-dimensional marginals are these laws. The coordinate maps \(X_t(\omega)=\omega(t)\) are the process.

All examples are centered.

The Gaussian width process. For \(T\subseteq\mathbb R^d\) and \(g\sim N(0,I_d)\), \(X_t=\langle g,t\rangle\) has \(K(s,t)=\langle s,t\rangle\) and \(d(s,t)=\|s-t\|_2\). The canonical metric is the Euclidean metric, and \(\mathbb E\sup_tX_t=w(T)\) is the Gaussian width of Lecture 3.

Brownian motion. \(T=[0,\infty)\) and \(K(s,t)=\min\{s,t\}\). Positive semidefiniteness follows from \(\min\{s,t\}=\int_0^\infty\mathbb 1_{[0,s]}\mathbb 1_{[0,t]}\), which exhibits \(K\) as a Gram matrix. Here \(d(s,t)=|s-t|^{1/2}\). Restricting to \(T=\mathbb N_0\) gives the Gaussian random walk, the partial sums of i.i.d. standard Gaussians.

Branching random walk. Let \(T\) be the set of vertices of the rooted binary tree, and attach independent \(N(0,1)\) weights to the edges. Let \(X_v\) be the sum of the weights along the path from the root to \(v\). Then \(K(v,w)\) is the depth of the most recent common ancestor of \(v\) and \(w\), and \(d(v,w)^2\) is the number of edges on the path from \(v\) to \(w\): the canonical metric is the square root of the graph distance on the tree. The metric tree of Lecture 2 is this geometry.

The Gaussian matrix process. For an \(m\times n\) matrix \(A\) with i.i.d. \(N(0,1)\) entries and \(T=S^{n-1}\times S^{m-1}\), \(X_{(u,v)}=\langle Au,v\rangle\) has \(K((u,v),(w,z))=\langle u,w\rangle\langle v,z\rangle\) and \(d((u,v),(w,z))=\|uv^{\mathsf T}-wz^{\mathsf T}\|_F\). Its supremum is \(s_{\max}(A)\). Lecture 12 computes \(\mathbb E\sup X\) by comparing this process with a simpler one on the same index set.

For an uncountable \(T\) the supremum \(\sup_{t\in T}X_t\) need not be measurable, and it can depend on which version of the process is used. We adopt the following convention, which covers every process in this course. Say that \((T,d)\) is separable if it has a countable dense subset \(T_0\). Then define \[ \sup_{t\in T}X_t:=\sup_{t\in T_0}X_t . \] This is a random variable, and it does not depend on the choice of \(T_0\) up to almost sure equality, because for \(t\in T\) and \(t_k\to t\) in \(T_0\) we have \(X_{t_k}\to X_t\) in \(L^2\), hence along a subsequence almost surely. When the process has continuous sample paths, as \(t\mapsto\langle g,t\rangle\) does, the convention agrees with the pointwise supremum.

The expectation \(\mathbb E\sup_tX_t\) can be \(+\infty\). If \(T=\mathbb N\) and the \(X_t\) are i.i.d. \(N(0,1)\), then \(d(s,t)=\sqrt2\) for \(s\neq t\), the metric space is not totally bounded, and \(\sup_tX_t=+\infty\) almost surely. Sudakov’s inequality of Lecture 3 already says that finiteness of \(\mathbb E\sup X\) forces \((T,d)\) to be totally bounded.

3 The isonormal process

The linear forms \(\langle g,u\rangle\) of a standard Gaussian vector realize the inner product of \(\mathbb R^d\) as a covariance. The same construction works in any separable Hilbert space, and the resulting process is the universal example.

Definition 4 (The isonormal process) Let \(H\) be a separable Hilbert space with orthonormal basis \((e_j)_{j\geq1}\), and let \(Z_1,Z_2,\ldots\) be i.i.d. \(N(0,1)\). For \(h=\sum_j\hat h_je_j\in H\) define \[ g(h):=\sum_{j\geq1}\hat h_jZ_j , \] the series converging in \(L^2(\mathbb P)\). The process \((g(h))_{h\in H}\) is the isonormal process on \(H\).

Proposition 2 (Properties of the isonormal process) The series defining \(g(h)\) converges in \(L^2(\mathbb P)\), and \((g(h))_{h\in H}\) is a centered Gaussian process with \[ \mathbb Eg(h)g(h')=\langle h,h'\rangle_H \qquad\text{for all }h,h'\in H . \] Its canonical metric is the norm distance \(d(h,h')=\|h-h'\|_H\). The map \(h\mapsto g(h)\) is linear: \(g(\alpha h+\beta h')=\alpha g(h)+\beta g(h')\) almost surely.

Let \(g_n(h)=\sum_{j\leq n}\hat h_jZ_j\). For \(m<n\), by independence and \(\mathbb EZ_j^2=1\), \[ \mathbb E\big(g_n(h)-g_m(h)\big)^2=\sum_{j=m+1}^n\hat h_j^2 , \] which tends to zero since \(\sum_j\hat h_j^2=\|h\|_H^2<\infty\). So \((g_n(h))_n\) is Cauchy in \(L^2(\mathbb P)\) and converges. Each \(g_n(h)\) is Gaussian with variance \(\sum_{j\leq n}\hat h_j^2\), and for finitely many \(h^{(1)},\ldots,h^{(k)}\) the vector \((g_n(h^{(i)}))_i\) is Gaussian by Proposition 1, being a linear image of \((Z_1,\ldots,Z_n)\). Characteristic functions pass to \(L^2\) limits, since \(|\mathbb Ee^{i\langle\xi,Y_n\rangle}-\mathbb Ee^{i\langle\xi,Y\rangle}|\leq\|\xi\|_2\,\mathbb E\|Y_n-Y\|_2\), so the limit vector is Gaussian too. The covariance is \[ \mathbb Eg(h)g(h')=\lim_n\mathbb Eg_n(h)g_n(h')=\lim_n\sum_{j\leq n}\hat h_j\hat h'_j=\langle h,h'\rangle_H . \] Linearity holds for each \(g_n\) and passes to the limit.

Two instances are already familiar. For \(H=\mathbb R^d\) with the standard basis, \(g(h)=\langle g,h\rangle\) with \(g=(Z_1,\ldots,Z_d)\), which is the Gaussian width process. For \(H=L^2[0,\infty)\), the process \(B_t:=g(\mathbb 1_{[0,t]})\) has covariance \(\langle\mathbb 1_{[0,s]},\mathbb 1_{[0,t]}\rangle=\min\{s,t\}\), which is Brownian motion, and more generally \(g(f)\) is the stochastic integral \(\int f\,dB\) for deterministic \(f\).

The point of the isonormal process is that it is, up to relabeling, the only Gaussian process.

Theorem 2 (Every separable Gaussian process is isonormal) Let \((X_t)_{t\in T}\) be a centered Gaussian process whose canonical metric space \((T,d)\) is separable. Then there are i.i.d. \(N(0,1)\) variables \(Z_1,Z_2,\ldots\) and vectors \(x(t)=(\hat x(t,j))_{j\geq1}\in\ell^2\), one for each \(t\in T\), such that \[ X_t=\sum_{j\geq1}\hat x(t,j)Z_j \qquad\text{almost surely, for every }t\in T, \] with the series converging in \(L^2(\mathbb P)\). Consequently \[ K(s,t)=\langle x(s),x(t)\rangle_{\ell^2} \qquad\text{and}\qquad d(s,t)=\|x(s)-x(t)\|_{\ell^2}. \] In other words, \(X_t=g(x(t))\) for the isonormal process \(g\) on \(\ell^2\), and \(t\mapsto x(t)\) is an isometry from \((T,d)\) onto the subset \(x(T)\) of \(\ell^2\).

Let \(H_X\subseteq L^2(\mathbb P)\) be the closed linear span of \(\{X_t:t\in T\}\). Since \((T,d)\) is separable and \(t\mapsto X_t\) is an isometry into \(L^2(\mathbb P)\), the set \(\{X_t\}\) is separable in \(L^2(\mathbb P)\), so \(H_X\) is a separable Hilbert space and has a countable orthonormal basis \((Z_j)_{j\geq1}\).

Each \(Z_j\) is an \(L^2\) limit of finite linear combinations of the \(X_t\), so finitely many of the \(Z_j\) form a Gaussian vector, by the same limiting argument as in the proof of Proposition 2. They are orthonormal in \(L^2(\mathbb P)\), that is, they have variance \(1\) and are pairwise uncorrelated, so they are i.i.d. \(N(0,1)\) by the independence remark after Proposition 1.

Expanding \(X_t\in H_X\) in the basis, \(X_t=\sum_j\langle X_t,Z_j\rangle_{L^2}Z_j\) in \(L^2(\mathbb P)\). Set \(\hat x(t,j):=\langle X_t,Z_j\rangle_{L^2}=\mathbb EX_tZ_j\). Parseval gives \(\|x(t)\|_{\ell^2}^2=\mathbb EX_t^2<\infty\), and the covariance and metric identities follow from Parseval applied to \(\mathbb EX_sX_t=\langle X_s,X_t\rangle_{L^2}\).

The representation \(X_t=\sum_j\hat x(t,j)Z_j\) is called the Karhunen–Loève expansion. Its conceptual content is this. A centered Gaussian process on a separable index set is the same thing as a subset \(x(T)\) of a Hilbert space, and the process is the restriction of the isonormal process to that subset. Questions about \(\sup_tX_t\) are questions about the Gaussian width of \(x(T)\), in the sense of Lecture 3 but now in infinite dimensions. The Gaussian width process was not one example among many; it was the general case in disguise.

The theorem does not single out a basis. For Brownian motion on \([0,1]\) a natural choice diagonalizes the covariance operator \(f\mapsto\int_0^1\min\{s,t\}f(s)\,ds\), whose eigenfunctions are \(\sqrt2\sin\big((j-\tfrac12)\pi t\big)\) with eigenvalues \(\lambda_j=\big((j-\tfrac12)\pi\big)^{-2}\). Then \[ B_t=\sum_{j\geq1}\sqrt{\lambda_j}\,\sqrt2\sin\big((j-\tfrac12)\pi t\big)Z_j , \] and the decay of \(\lambda_j\) measures how well Brownian motion is approximated by finitely many Gaussians. The expansion in Theorem 2 is exact but says nothing about which coordinates matter most; that is a question about the geometry of \(x(T)\).

4 Stein’s lemma

The second tool is an integration by parts formula. It is what lets us differentiate expectations of functions of Gaussians, and it is the engine behind the concentration and comparison inequalities of the next two lectures.

Lemma 1 (Stein’s lemma, one variable) Let \(X\sim N(0,\sigma^2)\) and let \(f:\mathbb R\to\mathbb R\) be Lipschitz. Then \[ \mathbb E\,Xf(X)=\sigma^2\,\mathbb Ef'(X). \]

Here \(f\) is differentiable almost everywhere with \(|f'|\leq\|f\|_{\mathrm{Lip}}\), by Rademacher’s theorem, and \(f\) is the integral of \(f'\), so the right-hand side is well defined.

Let \(\varphi_\sigma(x)=(2\pi\sigma^2)^{-1/2}e^{-x^2/(2\sigma^2)}\) be the density of \(X\). Its defining property for this proof is \[ x\varphi_\sigma(x)=-\sigma^2\varphi_\sigma'(x). \] Hence, integrating by parts, \[ \mathbb EXf(X)=\int_{\mathbb R}xf(x)\varphi_\sigma(x)\,dx =-\sigma^2\int_{\mathbb R}f(x)\varphi_\sigma'(x)\,dx =\sigma^2\int_{\mathbb R}f'(x)\varphi_\sigma(x)\,dx . \] The boundary terms vanish because \(|f(x)|\leq|f(0)|+\|f\|_{\mathrm{Lip}}|x|\) grows at most linearly while \(\varphi_\sigma\) decays like a Gaussian, and the integration by parts is legitimate for the absolutely continuous function \(f\).

The identity characterizes the Gaussian: if \(\mathbb EXf(X)=\sigma^2\mathbb Ef'(X)\) for all bounded Lipschitz \(f\), then \(X\sim N(0,\sigma^2)\). Taking \(f(x)=x^{k-1}\) shows that the identity forces all moments to be those of \(N(0,\sigma^2)\), which determines the law. We do not need this converse, but it explains why the lemma is so tightly bound to Gaussian structure.

Lemma 2 (Stein’s lemma, several variables) Let \(X\sim N(0,\Sigma)\) in \(\mathbb R^d\) and let \(f:\mathbb R^d\to\mathbb R\) be Lipschitz. Then for every \(j\), \[ \mathbb E\,X_jf(X)=\mathbb E\langle\Sigma e_j,\nabla f(X)\rangle =\sum_{k=1}^d\Sigma_{jk}\,\mathbb E\,\partial_kf(X). \] Equivalently, \(\mathbb E\,Xf(X)=\Sigma\,\mathbb E\nabla f(X)\) as vectors.

By Proposition 1 we may take \(X=\Sigma^{1/2}Z\) with \(Z\sim N(0,I_d)\). Write \(S=\Sigma^{1/2}\), so that \(X_j=\sum_iS_{ji}Z_i\), and set \(F(z)=f(Sz)\), which is Lipschitz with \(\partial_iF(z)=\sum_kS_{ki}\,\partial_kf(Sz)\) by the chain rule, valid almost everywhere.

For a single coordinate \(Z_i\), condition on the other coordinates: they are independent of \(Z_i\), and \(z_i\mapsto F(z)\) is Lipschitz, so the one-variable lemma with \(\sigma=1\) gives \(\mathbb E\,Z_iF(Z)=\mathbb E\,\partial_iF(Z)\). Therefore \[ \mathbb EX_jf(X)=\sum_iS_{ji}\,\mathbb EZ_iF(Z) =\sum_iS_{ji}\sum_kS_{ki}\,\mathbb E\partial_kf(X) =\sum_k(SS^{\mathsf T})_{jk}\,\mathbb E\partial_kf(X), \] and \(SS^{\mathsf T}=\Sigma\).

For the Gaussian width process this reads \(\mathbb E\langle g,t\rangle f(g)=\mathbb E\langle t,\nabla f(g)\rangle\). The next lecture applies it with \(f\) an exponential of a Lipschitz function to prove that Lipschitz functions of a Gaussian vector concentrate, and Lecture 12 applies it along an interpolation between two covariances to compare the suprema of two processes. Both proofs are the same computation with different choices of \(f\).

5 Where we go next

A centered Gaussian process is a subset of a Hilbert space together with the isonormal process on it, and the object of interest is \(\sup_{t\in T}X_t\), the Gaussian width of that subset. Two questions remain. How concentrated is the supremum around its mean? Lecture 11 shows that the fluctuations are governed by \(\sup_t\operatorname{Var}X_t\) alone, whatever the geometry of \(T\). And how large is the mean? Lecture 12 begins the answer with comparison inequalities, which bound \(\mathbb E\sup X\) by \(\mathbb E\sup Y\) whenever the canonical metric of \(X\) is dominated by that of \(Y\), and uses them to compute the norm of a Gaussian matrix.