📐 Concept diagram

08-05 — Inner Product Spaces

Phase: 8 — Linear Algebra (Rigorous) Subject: 08-05 Prerequisites: 08-04 — Matrices as Linear Transformations Next subject: 08-06 — Orthogonal Projections

Learning Objectives

By the end of this subject, you will be able to:

1. State the axioms of an inner product and verify whether a given operation satisfies them
2. Compute norms, distances, and angles from an inner product, and prove the Cauchy-Schwarz and Triangle inequalities
3. Determine whether vectors are orthogonal, and decompose a vector into orthogonal components
4. Apply the Gram-Schmidt process to convert any basis into an orthonormal basis
5. Recognize orthogonal complements and use the decomposition V = W ⊕ W^⟂

Core Content

1. Definition of Inner Product

An inner product on a real vector space V is a function ⟨·, ·⟩ : V × V → R satisfying:

1. Symmetry: ⟨u, v⟩ = ⟨v, u⟩
2. Linearity in first argument: ⟨au + v, w⟩ = a⟨u, w⟩ + ⟨v, w⟩
3. Positive definiteness: ⟨v, v⟩ ≥ 0, and ⟨v, v⟩ = 0 iff v = 0

CRITICAL — Foundational: An inner product gives a vector space GEOMETRY — length, angle, distance, orthogonality. Cauchy-Schwarz $|⟨u,v⟩| ≤ ‖u‖‖v‖$ is the most important inequality — it justifies everything.

(For complex vector spaces, replace symmetry with conjugate symmetry: ⟨u, v⟩ = ⟨v, u⟩̅.)

Standard inner products: - R^n (dot product): ⟨x, y⟩ = x·y = Σ x_i y_i = x^T y - C[a, b] (functions): ⟨f, g⟩ = ∫_a^b f(x)g(x) dx - M_{m×n} (matrices): ⟨A, B⟩ = tr(A^T B) = Σ_i Σ_j a_ij b_ij - Weighted inner product on R^n: ⟨x, y⟩_W = x^T W y where W is symmetric positive definite

The dot product in R^n is an inner product: - Symmetry: x·y = y·x ✓ - Linearity: (ax+z)·y = a(x·y) + z·y ✓ - Positive definiteness: x·x = Σ x_i² ≥ 0, and = 0 only if all x_i = 0 ✓

2. Norm, Distance, Angle

From an inner product we derive geometric concepts:

Norm (length): ‖v‖ = √⟨v, v⟩

Properties of norm (derived from inner product): - ‖v‖ ≥ 0, and ‖v‖ = 0 iff v = 0 - ‖cv‖ = |c| ‖v‖

Distance: d(u, v) = ‖u − v‖

Angle between u and v (both nonzero):

$cos θ = ⟨u, v⟩ / (‖u‖ ‖v‖)
$

This is well-defined because |⟨u, v⟩| ≤ ‖u‖ ‖v‖ (Cauchy-Schwarz).

Cauchy-Schwarz Inequality: |⟨u, v⟩| ≤ ‖u‖ ‖v‖

Proof sketch for general inner product: For any t ∈ R, ‖u + tv‖² = ⟨u+tv, u+tv⟩ = ‖u‖² + 2t⟨u,v⟩ + t²‖v‖² ≥ 0. This quadratic in t has discriminant ≤ 0: 4⟨u,v⟩² − 4‖u‖²‖v‖² ≤ 0 → ⟨u,v⟩² ≤ ‖u‖²‖v‖². ∎

Triangle Inequality: ‖u + v‖ ≤ ‖u‖ + ‖v‖

Proof: ‖u+v‖² = ⟨u+v, u+v⟩ = ‖u‖² + 2⟨u,v⟩ + ‖v‖² ≤ ‖u‖² + 2‖u‖‖v‖ + ‖v‖² = (‖u‖+‖v‖)².

3. Orthogonality

Definition: u and v are orthogonal if ⟨u, v⟩ = 0. Notation: u ⟂ v.

Orthogonal set: A set S = {v₁, ..., v_k} where ⟨v_i, v_j⟩ = 0 for all i ≠ j.

Orthonormal set: An orthogonal set where additionally ‖v_i‖ = 1 for all i.

Theorem: Every orthogonal set of nonzero vectors is linearly independent. Proof: If Σ c_i v_i = 0, take inner product with v_j: ⟨Σ c_i v_i, v_j⟩ = c_j ⟨v_j, v_j⟩ = c_j ‖v_j‖² = 0. Since ‖v_j‖² ≠ 0, c_j = 0. ∎

Orthogonal decomposition in R^n: For any nonzero vector u, any v can be decomposed as:

$v = v_∥ + v_⟂
v_∥ = proj_u(v) = (⟨v, u⟩ / ‖u‖²) u    (parallel component)
v_⟂ = v − v_∥                          (orthogonal component)
$

These satisfy v_∥ ⟂ v_⟂.

Pythagorean Theorem (inner product version): If u ⟂ v, then ‖u + v‖² = ‖u‖² + ‖v‖².

4. Gram-Schmidt Orthogonalization

Given a basis {b₁, ..., b_n} of V, Gram-Schmidt produces an ORTHOGONAL basis {v₁, ..., v_n}:

Common Pitfall: Classical Gram-Schmidt is numerically unstable. Use Modified Gram-Schmidt or Householder reflections for computation. Also, the output depends on input ORDER — different orderings give different orthogonal bases.

$v₁ = b₁

v₂ = b₂ − proj_{v₁}(b₂) = b₂ − (⟨b₂, v₁⟩ / ‖v₁‖²) v₁

v₃ = b₃ − proj_{v₁}(b₃) − proj_{v₂}(b₃)
   = b₃ − (⟨b₃, v₁⟩ / ‖v₁‖²) v₁ − (⟨b₃, v₂⟩ / ‖v₂‖²) v₂

...

v_k = b_k − Σ_{i=1}^{k-1} proj_{v_i}(b_k)
$

To get an orthonormal basis: normalize each v_k: u_k = v_k / ‖v_k‖.

Why it works: At each step, we subtract the projection of b_k onto all previously computed v_i, which are already mutually orthogonal. The result is orthogonal to all previous v_i.

Example (R^3): b₁ = (1,1,0), b₂ = (1,0,1), b₃ = (0,1,1).

v₁ = (1,1,0)

proj_{v₁}(b₂) = (⟨(1,0,1),(1,1,0)⟩ / 2) (1,1,0) = (1/2)(1,1,0) = (1/2, 1/2, 0) v₂ = (1,0,1) − (1/2, 1/2, 0) = (1/2, −1/2, 1)

proj_{v₁}(b₃) = (⟨(0,1,1),(1,1,0)⟩ / 2)(1,1,0) = (1/2)(1,1,0) = (1/2, 1/2, 0) proj_{v₂}(b₃) = ⟨(0,1,1),(1/2,−1/2,1)⟩ / ‖v₂‖² · v₂. ‖v₂‖² = 1/4+1/4+1 = 3/2. Inner product = 0−1/2+1 = 1/2. proj = (1/2)/(3/2) v₂ = (1/3)(1/2, −1/2, 1) = (1/6, −1/6, 1/3)

v₃ = (0,1,1) − (1/2,1/2,0) − (1/6,−1/6,1/3) = (−2/3, 2/3, 2/3)

Normalize: u₁ = (1,1,0)/√2, u₂ = (1/2,−1/2,1)/√(3/2), u₃ = (−2/3,2/3,2/3)/√(4/3).

5. Orthogonal Complement

Definition: For a subspace W ⊆ V, its orthogonal complement is:

W^⟂ = {v ∈ V : ⟨v, w⟩ = 0 for all w ∈ W}

Properties: - W^⟂ is always a subspace - V = W ⊕ W^⟂ (direct sum; every v ∈ V decomposes uniquely as w + w^⟂) - (W^⟂)^⟂ = W - dim(W^⟂) = dim(V) − dim(W)

Example: In R^3, W = span{(1, 1, 0)}. Then W^⟂ = {(x, y, z) : x + y = 0} — the plane perpendicular to the line.

Connection to fundamental subspaces of a matrix A: - N(A) = C(A^T)^⟂ - C(A) = N(A^T)^⟂

Key Terms

• Legendre polynomials

Worked Examples

Example 1: Verifying an Inner Product

Problem: Show ⟨p, q⟩ = p(0)q(0) + p(1)q(1) + p(2)q(2) defines an inner product on P₂.

Solution:

Symmetry: ⟨p, q⟩ = p(0)q(0) + p(1)q(1) + p(2)q(2) = q(0)p(0) + q(1)p(1) + q(2)p(2) = ⟨q, p⟩. ✓

Linearity: ⟨ap+q, r⟩ = (ap+q)(0)r(0) + (ap+q)(1)r(1) + (ap+q)(2)r(2) = a[p(0)r(0)+p(1)r(1)+p(2)r(2)] + [q(0)r(0)+q(1)r(1)+q(2)r(2)] = a⟨p,r⟩ + ⟨q,r⟩. ✓

Positive definiteness: ⟨p, p⟩ = p(0)² + p(1)² + p(2)² ≥ 0. If ⟨p,p⟩ = 0, then p(0) = p(1) = p(2) = 0. A degree-≤2 polynomial with three distinct roots is the zero polynomial. ✓

Yes, this is a valid inner product.

Example 2: Orthogonal Decomposition

Problem: Decompose v = (5, 2) into components parallel and perpendicular to u = (3, 1).

Solution:

‖u‖² = 3² + 1² = 10 ⟨v, u⟩ = 5·3 + 2·1 = 17

v_∥ = (⟨v,u⟩/‖u‖²) u = (17/10)(3, 1) = (51/10, 17/10) = (5.1, 1.7) v_⟂ = v − v_∥ = (5, 2) − (5.1, 1.7) = (−0.1, 0.3)

Check orthogonality: v_∥ · v_⟂ = 5.1·(−0.1) + 1.7·0.3 = −0.51 + 0.51 = 0. ✓

Example 3: Gram-Schmidt with Non-Standard Inner Product

Problem: On P₂, use inner product ⟨p, q⟩ = ∫₀¹ p(x)q(x) dx. Apply Gram-Schmidt to {1, x, x²}.

Solution:

v₁ = 1, ‖v₁‖² = ∫₀¹ 1 dx = 1.

v₂ = x − proj_{v₁}(x). ⟨x, 1⟩ = ∫₀¹ x dx = 1/2. proj = (1/2)/1 · 1 = 1/2. v₂ = x − 1/2.

v₃ = x² − proj_{v₁}(x²) − proj_{v₂}(x²). ⟨x², 1⟩ = ∫₀¹ x² dx = 1/3 → proj = 1/3. ⟨x², x−1/2⟩ = ∫₀¹ x²(x−1/2) dx = ∫₀¹ (x³ − x²/2) dx = [x⁴/4 − x³/6]₀¹ = 1/4 − 1/6 = 1/12. ‖v₂‖² = ∫₀¹ (x−1/2)² dx = [x³/3 − x²/2 + x/4]₀¹ = 1/3 − 1/2 + 1/4 = 1/12. proj = (1/12)/(1/12) (x−1/2) = x − 1/2.

v₃ = x² − 1/3 − (x − 1/2) = x² − x + 1/6.

These are the first three Legendre polynomials (up to scaling), orthogonal on [0,1].

Example 4: Orthogonal Complement Computation

Problem: In R^4, W = span{(1, 1, 0, 0), (0, 0, 1, 2)}. Find W^⟂.

Solution: v ∈ W^⟂ iff v·w = 0 for all w ∈ W, equivalently v is orthogonal to both basis vectors.

v = (x₁, x₂, x₃, x₄):

$x₁ + x₂ = 0  →  x₂ = −x₁
x₃ + 2x₄ = 0 →  x₃ = −2x₄
$

Free variables: x₁ = s, x₄ = t. So v = s(1, −1, 0, 0) + t(0, 0, −2, 1).

W^⟂ = span{(1, −1, 0, 0), (0, 0, −2, 1)}. dim(W^⟂) = 2.

Check: dim(W) + dim(W^⟂) = 2 + 2 = 4 = dim(R^4). ✓

Practice Problems

(Answers are below. Try each problem before checking.)

Problem 1: Show that ⟨x, y⟩ = 2x₁y₁ + 3x₂y₂ is an inner product on R^2. What is the norm of (1, 1) under this inner product?

Problem 2: Prove the Cauchy-Schwarz inequality for the dot product in R^n: |x·y| ≤ ‖x‖ ‖y‖.

Problem 3: In R^3, determine whether u = (1, 2, −1) and v = (3, −1, 1) are orthogonal under the standard dot product. Find the angle between them.

Problem 4: Apply Gram-Schmidt to {(1, 1, 1), (1, 2, 0), (0, 1, −1)} to obtain an orthonormal basis of R^3.

Problem 5: In R^4, find the orthogonal complement of W = span{(1, 2, 3, 4), (1, 0, 1, 0)}.

Problem 6: Verify the Pythagorean theorem holds for u = (3, 0, 4) and v = (−4, 0, 3) in R^3 under the standard dot product.

Problem 7: Show that if {v₁, ..., v_n} is an orthonormal basis of V, then for any v ∈ V, v = Σ ⟨v, v_i⟩ v_i and ‖v‖² = Σ ⟨v, v_i⟩².

Summary

1. An inner product generalizes the dot product to abstract vector spaces via three axioms: symmetry, linearity, and positive definiteness; it endows a vector space with geometry (length, angle, distance)
2. Cauchy-Schwarz (|⟨u,v⟩| ≤ ‖u‖ ‖v‖) and the Triangle Inequality (‖u+v‖ ≤ ‖u‖+‖v‖) are the foundational inequalities of inner product spaces and underpin all of analysis
3. Orthogonal vectors (⟨u,v⟩ = 0) behave like perpendicular vectors in geometry; every vector decomposes uniquely into parallel + perpendicular components relative to any subspace
4. Gram-Schmidt converts any basis to an orthogonal (then orthonormal) one: at each step, subtract the projection onto all previously computed orthogonal vectors
5. The orthogonal complement W^⟂ together with W decomposes the space: V = W ⊕ W^⟂; this is fundamental to least-squares, projections, and the four fundamental subspaces of a matrix

Pitfalls

1. Forgetting to verify positive definiteness. The third inner product axiom (⟨v,v⟩ ≥ 0 with equality iff v = 0) is the most frequently overlooked. An operation that satisfies symmetry and linearity but fails positive definiteness — such as ⟨x,y⟩ = x₁y₁ on R² — is NOT an inner product.

2. Treating Cauchy-Schwarz as an equality. |⟨u,v⟩| ≤ ‖u‖‖v‖ is an INEQUALITY. Equality holds only when u and v are linearly dependent (collinear). Students often write |⟨u,v⟩| = ‖u‖‖v‖ by reflex, leading to incorrect angle calculations.

3. Losing orthogonality in Gram-Schmidt due to numerical instability. Classical Gram-Schmidt subtracts projections against the original a_j vectors, accumulating round-off error. In practice, use Modified Gram-Schmidt (subtract projections against the current v sequentially) or Householder reflections for better numerical stability.

4. Applying Gram-Schmidt to dependent sets. Gram-Schmidt requires a linearly independent input set. If the input vectors are dependent, one of the v_k will become zero — this is a signal, not an error to ignore. The process naturally detects linear dependence.

5. Forgetting that orthogonal vectors must be nonzero. Every orthogonal set of nonzero vectors is linearly independent, but the zero vector is orthogonal to everything. A set containing the zero vector cannot be a basis, even if all nonzero members are mutually orthogonal.

Quiz

Answer each question, then read the explanation for your choice.

Q1: Which of the following is NOT required for an inner product on a real vector space?

A) ⟨u, v⟩ = ⟨v, u⟩ B) ⟨u + v, w⟩ = ⟨u, w⟩ + ⟨v, w⟩ C) ⟨u, u⟩ = 1 for all u D) ⟨u, u⟩ ≥ 0 with equality only when u = 0

Q2: The Cauchy-Schwarz inequality guarantees that:

A) ⟨u, v⟩ ≤ ‖u‖ + ‖v‖ B) |⟨u, v⟩| ≤ ‖u‖ ‖v‖ C) ‖u + v‖ ≥ ‖u‖ + ‖v‖ D) ⟨u, v⟩ = ‖u‖ ‖v‖ always

Q3: Two vectors are orthogonal if:

A) They are linearly independent B) ⟨u, v⟩ = 0 C) ‖u‖ = ‖v‖ D) u + v = 0

Q4: In the Gram-Schmidt process, at step k we compute v_k by:

A) Adding all previous v_i to b_k B) Subtracting from b_k its projections onto v₁, ..., v_{k-1} C) Computing the cross product of b_k with all previous vectors D) Computing the eigenvalues of b_k

Q5: In R^3, W^⟂ for W = span{(1, 0, 0), (0, 1, 0)} is:

A) The origin B) The z-axis C) The xy-plane D) All of R^3

Q6: What is the dimension of W^⟂ if W is a 3-dimensional subspace of R^7?

A) 0 B) 3 C) 4 D) 7

Q7: Under the standard dot product, ‖u + v‖² = ‖u‖² + ‖v‖² always holds when:

A) u and v are parallel B) u and v are orthogonal C) u and v are linearly dependent D) u = v

Q8: After Gram-Schmidt on a basis of R^4, how many orthonormal vectors do we obtain?

A) 2 B) 3 C) 4 D) Can't say without knowing the basis

Q9: For the inner product ⟨f, g⟩ = ∫₀¹ f(x)g(x) dx on C[0,1], what is the distance between f(x) = x and g(x) = x²?

A) 1/12 B) √(1/30) C) 1/6 D) √(1/6)

Q10: Parseval's identity for an orthonormal basis {v_i} states:

A) Σ ⟨v, v_i⟩ = ‖v‖ B) Σ ⟨v, v_i⟩² = ‖v‖² C) ⟨v, v⟩ = Σ v_i D) Σ ‖v_i‖² = dim(V)

Next Steps

Move on to 08-06 — Orthogonal Projections to learn about projecting vectors onto lines and subspaces, orthogonal matrices, least-squares solutions, and QR decomposition — powerful computational tools built on inner product geometry.