📐 Concept diagram

08-06 — Orthogonal Projections

Phase: 8 — Linear Algebra (Rigorous) Subject: 08-06 Prerequisites: 08-05 — Inner Product Spaces Next subject: 08-07 — Determinants (Deep)

Learning Objectives

By the end of this subject, you will be able to:

1. Compute the orthogonal projection of a vector onto a line and onto a general subspace using projection matrices
2. Characterize orthogonal matrices (Q^T Q = I) and apply them to preserve lengths and angles under transformation
3. Solve least-squares problems Ax ≈ b via the normal equations A^T A x̂ = A^T b
4. Compute and interpret the QR decomposition A = QR, and use it to solve least-squares problems efficiently
5. Understand the geometric meaning of projections: the projection minimizes distance, and the residual is orthogonal to the subspace

Core Content

1. Orthogonal Projection onto a Line

Given a nonzero vector a ∈ R^n, the orthogonal projection of b onto the line span{a} is:

$p = proj_a(b) = (a·b / a·a) a = (a^T b / a^T a) a
$

CRITICAL — Foundational: Orthogonal projection is the geometric foundation of least squares, Fourier series, PCA, and ML. Key property: the residual b − p is ORTHOGONAL to the subspace — this makes the projection the 'closest point.'

This can be written as p = P b where P is the projection matrix:

$P = (a a^T) / (a^T a)      (n × n matrix, rank 1)
$

Properties of a projection matrix P: - P² = P (idempotent: projecting again doesn't change anything)

Common Pitfall: P² = P alone does NOT guarantee orthogonal projection — oblique projections also satisfy this. Orthogonal projection requires BOTH P² = P AND P^T = P. - P^T = P (symmetric) - P projects onto C(P) = span{a} - I − P projects onto (span{a})^⟂

Example: Project b = (3, 1, 0) onto a = (1, 0, 2).

$a·b = 1·3 + 0·1 + 2·0 = 3
a·a = 1 + 0 + 4 = 5
p = (3/5)(1, 0, 2) = (0.6, 0, 1.2)
$

The residual e = b − p = (3, 1, 0) − (0.6, 0, 1.2) = (2.4, 1, −1.2) is orthogonal to a: a·e = 1·2.4 + 0·1 + 2·(−1.2) = 2.4 − 2.4 = 0. ✓

2. Orthogonal Projection onto a Subspace

Let W be a subspace of R^n with basis. We can find the projection of b onto W by two methods:

Method 1 — Orthogonal basis: If {u₁, ..., u_k} is an ORTHOGONAL basis of W, then:

$proj_W(b) = Σ_{i=1}^k (b·u_i / u_i·u_i) u_i
$

Method 2 — Matrix (normal equations): Let A be an n × k matrix whose columns are a basis of W. The projection p = A x̂ where x̂ solves:

$A^T A x̂ = A^T b    (normal equations)
$

Since columns of A are independent, A^T A is invertible, so:

$x̂ = (A^T A)⁻¹ A^T b
p = A (A^T A)⁻¹ A^T b = P b
$

The projection matrix is P = A(A^T A)⁻¹ A^T.

Properties: P² = P, P^T = P, rank(P) = k = dim(W).

Geometric interpretation: p is the unique point in W closest to b. That is:

$‖b − p‖ = min_{w ∈ W} ‖b − w‖
$

3. Orthogonal Matrices

Definition: A square matrix Q is orthogonal if Q^T Q = I (equivalently, Q⁻¹ = Q^T).

Properties: - Columns of Q form an orthonormal basis of R^n - Rows of Q also form an orthonormal basis - Q preserves inner product: ⟨Qx, Qy⟩ = (Qx)^T(Qy) = x^T Q^T Q y = x^T y = ⟨x, y⟩ - Q preserves norm: ‖Qx‖ = ‖x‖ - Q preserves angles - det(Q) = ±1 (rotation: +1; reflection: −1)

Examples: - Rotation matrix: Q = [cos θ −sin θ; sin θ cos θ], Q^T Q = I ✓ - Permutation matrix: swaps coordinates, orthogonal - Householder reflection: Q = I − 2uu^T where ‖u‖ = 1

4. QR Decomposition

Any m × n matrix A with linearly independent columns can be factored as:

$A = Q R
$

where: - Q is m × n with orthonormal columns (Q^T Q = I_n) - R is n × n upper triangular with positive diagonal entries

Computation via Gram-Schmidt:

Let columns of A be a₁, ..., a_n. Orthonormalize to get q₁, ..., q_n (columns of Q). Then R entries are:

r_ij = q_i^T a_j  for i < j    (projection coefficients)
r_jj = ‖a_j − Σ_{i<j} r_ij q_i‖  (length of orthogonalized vector)

Solving Ax = b via QR:

$Ax = b  →  QRx = b  →  Rx = Q^T b
$

Since R is upper triangular, solve by back-substitution.

Least-squares via QR:

$A^T A x̂ = A^T b  ⇔  R^T Q^T Q R x̂ = R^T Q^T b  ⇔  R^T R x̂ = R^T Q^T b
$

Since R is invertible: R x̂ = Q^T b. Solve by back-substitution — no need to form A^T A!

5. Least-Squares Problems

Given A (m × n, m > n) and b ∈ R^m, find x̂ minimizing ‖Ax − b‖².

Geometric interpretation: Find the point A x̂ in C(A) closest to b — the orthogonal projection of b onto C(A).

Normal equations: A^T A x̂ = A^T b.

Solution exists: If columns of A are independent, x̂ is unique: x̂ = (A^T A)⁻¹ A^T b.

The residual: e = b − A x̂ is in N(A^T) = C(A)^⟂. In fact, A^T e = 0 (this is just the normal equations).

Example: Fit a line y = mx + c to data points (1, 1), (2, 2), (3, 3.5).

For each point: mx_i + c = y_i. In matrix form:

$[1 1] [c]   [1]
[2 1] [m] = [2]
[3 1]       [3.5]
$

A = [1 1; 2 1; 3 1], b = (1, 2, 3.5)^T.

A^T A = [1 2 3; 1 1 1] [1 1; 2 1; 3 1] = [14 6; 6 3] A^T b = [1 2 3; 1 1 1] [1; 2; 3.5] = [1+4+10.5; 1+2+3.5] = [15.5; 6.5]

Solve: [14 6; 6 3][c; m] = [15.5; 6.5] → 14c + 6m = 15.5, 6c + 3m = 6.5

From second: 2c + m = 6.5/3? Let's do: 6c + 3m = 6.5 → divide by 3: 2c + m = 13/6. From first minus 2×(second): (14c+6m) − (12c+6m) = 15.5 − 13 = 2.5 → 2c = 2.5 → c = 1.25. Then m = 13/6 − 2c = 13/6 − 2.5 = 13/6 − 15/6 = −2/6 ≈ −0.333. Line: y = −0.333x + 1.25.

Key Terms

• 08 06 Orthogonal Projections
• Correct: B) A rotation
• Correct: B) P² = P and P^T = P
• Correct: B) Q^T Q = I_n
• Correct: B) W^⟂
• Correct: B) e ∈ W^⟂
• Correct: B) ‖Ax − b‖²
• Correct: C) (a·b / a·a) a
• Correct: D) Either 1 or −1
• End-of-Subject Quiz
• Example 1: Projection onto a Plane
• Example 2: QR Decomposition

Worked Examples

Example 1: Projection onto a Plane

Problem: Find the projection of b = (6, 0, 0) onto the plane W = span{(1, 1, 0), (0, 1, 1)}.

Solution: Use Method 2 (normal equations).

A = [1 0; 1 1; 0 1] (columns are the basis vectors).

A^T A = [1 1 0; 0 1 1] [1 0; 1 1; 0 1] = [2 1; 1 2] A^T b = [1 1 0; 0 1 1] [6; 0; 0] = [6; 0]

Solve [2 1; 1 2] x̂ = [6; 0]: (2×eq2 − eq1): (2−2)x̂₁ + (4−1)x̂₂ = 0−6 → 3x̂₂ = −6 → x̂₂ = −2. Then x̂₁ = (6 − x̂₂)/2 = (6+2)/2 = 4.

p = A x̂ = [1 0; 1 1; 0 1][4; −2] = (4, 4−2, −2) = (4, 2, −2).

Residual: e = b − p = (6−4, 0−2, 0−(−2)) = (2, −2, 2).

Check orthogonality: e·(1,1,0) = 2−2+0 = 0, e·(0,1,1) = 0−2+2 = 0. ✓

Example 2: QR Decomposition

Problem: Find QR decomposition of A = [1 1; 1 0; 0 1].

Solution (Gram-Schmidt):

Col1: a₁ = (1, 1, 0). ‖a₁‖ = √2. q₁ = (1/√2, 1/√2, 0). r₁₁ = √2.

Col2: compute r₁₂ = q₁^T a₂ = (1/√2, 1/√2, 0)·(1, 0, 1) = 1/√2. v₂ = a₂ − r₁₂ q₁ = (1, 0, 1) − (1/√2)(1/√2, 1/√2, 0) = (1, 0, 1) − (1/2, 1/2, 0) = (1/2, −1/2, 1). ‖v₂‖² = 1/4 + 1/4 + 1 = 3/2, ‖v₂‖ = √(3/2). r₂₂ = ‖v₂‖ = √(3/2). q₂ = v₂/‖v₂‖ = (1/2, −1/2, 1)/√(3/2) = (1/√6, −1/√6, 2/√6).

$Q = [1/√2   1/√6 ]
    [1/√2  -1/√6 ]
    [0      2/√6 ]

R = [√2   1/√2  ]
    [0    √(3/2)]
$

Check: Q^T Q = I₂ and A = QR. ✓

Example 3: Orthogonal Matrix Properties

Problem: Show that if Q is orthogonal, then ‖Qx‖ = ‖x‖ for all x.

Solution: ‖Qx‖² = (Qx)^T (Qx) = x^T Q^T Q x = x^T I x = x^T x = ‖x‖². Taking square roots: ‖Qx‖ = ‖x‖. ✓

Corollary: det(Q) = ±1 because det(Q^T Q) = det(I) → det(Q)^2 = 1 → det(Q) = ±1.

Example 4: Least-Squares Parabola Fit

Problem: Fit y = ax² + bx + c to (−1, 2), (0, 1), (1, 3), (2, 6).

Solution: For each (x, y): ax² + bx + c = y.

Matrix form:

$[1  -1   1] [a]   [2]
[0   0   1] [b] = [1]
[1   1   1] [c]   [3]
[4   2   1]       [6]
$

A^T A = [1 0 1 4; −1 0 1 2; 1 1 1 1] [1 −1 1; 0 0 1; 1 1 1; 4 2 1] = [18 8 6; 8 6 2; 6 2 4]

A^T b = [1 0 1 4; −1 0 1 2; 1 1 1 1] [2; 1; 3; 6] = [2+3+24; −2+3+12; 2+1+3+6] = [29; 13; 12]

Solve [18 8 6; 8 6 2; 6 2 4] x̂ = [29; 13; 12]: x̂ ≈ (1, 0, 1). So y ≈ x² + 1.

Check values: f(−1)=2, f(0)=1, f(1)=2 (actual 3), f(2)=5 (actual 6). The fit isn't perfect but minimizes squared error.

Quiz

Q1: The orthogonal projection of b onto the line through a nonzero vector a is:

A) (a·b) a B) (a·b / b·b) a C) (a·b / a·a) a D) (a·b) / ‖a‖

Correct: C)

• If you chose C: p = (a·b / a·a) a — the scalar (a·b)/(a·a) scales a to land at the orthogonal projection point. The denominator must be a·a (‖a‖²). Correct!
• If you chose A: Missing the division by a·a; the result would overshoot or undershoot the projection point.
• If you chose B: Normalizes by the wrong length (b·b instead of a·a).
• If you chose D: Divides by ‖a‖ instead of ‖a‖², giving the wrong magnitude.

Q2: An orthogonal projection matrix P must satisfy:

A) P² = P only B) P² = P and P^T = P C) P^T P = I D) det(P) = 1

Correct: B)

• If you chose B: Idempotency (P² = P) means projecting twice doesn't change anything. Symmetry (P^T = P) ensures the projection is orthogonal, not oblique. Both are required. Correct!
• If you chose A: Idempotency alone allows oblique projections; symmetry is also needed for orthogonality.
• If you chose C: That defines an orthogonal matrix, not a projection matrix.
• If you chose D: Projection matrices can have determinant 0 (if they project onto a proper subspace).

Q3: A square matrix Q is orthogonal if:

A) Q has orthonormal rows B) Q^T Q = I C) Q⁻¹ = Q^T D) Both B and C (which are equivalent for square matrices)

Correct: D)

• If you chose D: Q^T Q = I means the columns are orthonormal. For a square matrix, this implies Q Q^T = I and Q⁻¹ = Q^T. Correct!
• If you chose A: Orthonormal rows are a consequence but not the definition.
• If you chose B: True, but C is also equivalent.
• If you chose C: True, but B is also equivalent.

Q4: In the QR decomposition A = QR, what is R?

A) A diagonal matrix B) A lower triangular matrix C) An upper triangular matrix with positive diagonal entries D) The identity matrix

Correct: C)

• If you chose C: Gram-Schmidt produces R upper triangular because aⱼ involves only q₁,...,qⱼ. Diagonal entries rⱼⱼ = ‖aⱼ − projection onto earlier columns‖ > 0. Correct!
• If you chose A: R is triangular, not diagonal (unless columns of A are already orthogonal).
• If you chose B: R is upper, not lower triangular.
• If you chose D: R = I only if A already has orthonormal columns.

Q5: The least-squares solution x̂ to Ax ≈ b (with A having independent columns) satisfies:

A) Ax̂ = b exactly B) A^T A x̂ = A^T b (the normal equations) C) A^T x̂ = b D) x̂ = A⁻¹b

Correct: B)

• If you chose B: The normal equations A^T A x̂ = A^T b arise from setting the gradient of ‖Ax − b‖² to zero. Geometrically, the residual b − Ax̂ is orthogonal to C(A). Correct!
• If you chose A: The system is overdetermined (m > n) — an exact solution generally doesn't exist.
• If you chose C: A^T is n × m; A^T x̂ = b doesn't make sense dimensionally unless n = m.
• If you chose D: A is not square (m > n), so A⁻¹ doesn't exist.

Q6: After projecting b onto subspace W, the residual e = b − p satisfies:

A) e ∈ W B) e ∈ W^⟂ (the orthogonal complement of W) C) e = 0 always D) e is parallel to the projection p

Correct: B)

• If you chose B: The residual is orthogonal to every vector in W — this is the defining property of orthogonal projection and why p is the closest point in W to b. Correct!
• If you chose A: The projection p ∈ W; the residual is what remains after subtracting the projection.
• If you chose C: e = 0 only if b was already in W.
• If you chose D: e is orthogonal to p (since p ∈ W and e ⟂ W).

Practice Problems

(Answers are below. Try each problem before checking.)

Problem 1: Project b = (4, −1, 5) onto the line through a = (2, 1, 0). Find the projection matrix P.

Problem 2: Find the orthogonal projection of b = (1, 1, 1, 1) onto W = span{(1, 0, 1, 0), (0, 1, 0, 1)}.

Problem 3: Determine whether Q = [3/5 4/5; 4/5 −3/5] is orthogonal.

Problem 4: Find the QR decomposition of A = [1 2; 0 1; 1 0].

Problem 5: Solve the least-squares problem: minimize ‖Ax − b‖² where A = [1 0; 1 1; 1 2] and b = (0, 1, 3)^T.

Problem 6: Show that if P is an orthogonal projection matrix, then I − P is also an orthogonal projection matrix.

Problem 7: Prove that Q^T is orthogonal when Q is orthogonal.

$Q = [1/√2   1/√3 ]
    [0      1/√3 ]
    [1/√2  -1/√3 ]

R = [√2   √2 ]
    [0    √3 ]
$

Summary

1. The orthogonal projection of b onto a subspace W gives the unique closest point in W to b; the projection matrix P = A(A^T A)⁻¹A^T is symmetric and idempotent (P² = P)
2. Orthogonal matrices (Q^T Q = I) preserve inner products, norms, and angles — they represent rotations and reflections; det(Q) = ±1
3. QR decomposition A = QR (Gram-Schmidt in matrix form) enables solving Ax = b and least-squares problems via back-substitution on R x = Q^T b, avoiding the formation of A^T A
4. The least-squares solution to Ax ≈ b minimizes ‖Ax − b‖² and satisfies the normal equations A^T A x̂ = A^T b; the residual b − A x̂ is orthogonal to C(A)
5. Projection is the geometric foundation of least-squares approximation, Fourier series, and many numerical linear algebra algorithms

Pitfalls

1. Assuming P² = P is sufficient for orthogonal projection. Idempotency (P² = P) defines a projection, but it could be oblique. An orthogonal projection additionally requires symmetry: P^T = P. Always check both conditions.

2. Forming A^T A explicitly for least squares. Computing A^T A squares the condition number, turning a mildly ill-conditioned problem into a severely ill-conditioned one. Use QR decomposition instead: solve R x = Q^T b by back-substitution, which preserves the original condition number.

3. Confusing Q^T Q = I_n with Q Q^T = I_m for tall matrices. When Q is m×n with m > n and orthonormal columns, Q^T Q = I_n (n×n identity), but Q Q^T ≠ I_m. The product Q Q^T is the projection matrix onto C(Q), not the identity.

4. Forgetting that least-squares requires independent columns for uniqueness. If columns of A are linearly dependent, A^T A is singular and the normal equations have infinitely many solutions. The minimum-norm solution then requires the pseudoinverse.

5. Not verifying orthogonality of the residual. The defining property of the least-squares solution is that the residual b − A x̂ is orthogonal to C(A). If you forget to check A^T(b − A x̂) = 0, you may have computed a non-orthogonal (and therefore non-optimal) projection.

Next Steps

Move on to 08-07 — Determinants (Deep) to explore the rigorous definition of determinants via permutations and parity, cofactor expansion, the full list of determinant properties, Cramer's rule, and the geometric interpretation as signed volume.