114 Linear Algebra Interview Questions and Answers (2026)

They are the same idea (a collection of numbers) at increasing dimensionality: a scalar is a single number, a vector is a 1D array, a matrix is a 2D grid, and a tensor generalizes all of these to any number of dimensions.

• Scalar (rank 0): A single value with magnitude only, e.g. a learning rate or a temperature.

• Vector (rank 1): An ordered list of numbers indexed by one axis, e.g. the features of one sample.

• Matrix (rank 2): A rectangular grid indexed by two axes (rows and columns), e.g. a dataset or a linear transformation.

• Tensor (rank n):

  • A generalization to any number of axes; a scalar, vector, and matrix are just tensors of rank 0, 1, and 2.

  • Example: an RGB image batch is a rank-4 tensor (batch, height, width, channels).

Because data, models, and their operations are naturally expressed as vectors and matrices, and linear algebra gives both the compact notation and the efficient, parallelizable computations that make ML feasible at scale.

• It represents everything uniformly: Data becomes matrices, model parameters become vectors/matrices, and predictions become matrix products.

• It compresses math into clean notation: A whole layer of a network is one expression like Wx + b, instead of many scalar loops.

• It enables speed: Vectorized matrix ops map directly onto optimized BLAS/GPU hardware, far faster than element-wise loops.

• It underpins core algorithms: PCA (eigen/SVD), regression (least squares), embeddings, and gradients in backprop are all linear-algebra operations.

A dataset is stored as a 2D matrix where each row is one sample (observation) and each column is one feature (variable), typically written as X with shape (n_samples, n_features).

• Rows = samples: Each row is a feature vector describing one example (one customer, one image, one row of a spreadsheet).

• Columns = features: Each column holds the same measured attribute across all samples (age, pixel intensity, etc.).

• Labels are separate: Targets are usually a vector y of length n_samples, or a matrix for multi-output problems.

• Why this shape matters: Predictions become a single product X @ w, so consistent (rows=samples) orientation keeps dimensions aligned.

A vector is an ordered list of numbers that represents a point or direction in an n-dimensional space; in ML it is the standard way to encode a single data point, a set of parameters, or a learned representation.

• Two views of a vector:

  • Geometric: an arrow with magnitude and direction in space.

  • Algebraic: an ordered n-tuple of components you can compute with.

• Uses in ML:

  • Feature vector: all attributes of one sample.

  • Weight vector: the parameters of a linear model.

  • Embedding: a dense learned vector for a word, user, or item.

• Key operations: The dot product measures similarity/alignment, and the norm measures length (used in distances and regularization).

A scalar is a single number (magnitude only), while a vector is an ordered collection of numbers that carries both magnitude and direction across multiple dimensions.

• Dimensionality: A scalar is rank 0 (one value); a vector is rank 1 (n values along one axis).

• Information carried: A scalar has magnitude only (temperature, learning rate); a vector encodes magnitude and direction (velocity, a feature vector).

• How they interact: Scalar multiplication scales a vector's length; multiplying by a scalar stretches or shrinks every component uniformly.

• In ML: A loss value is a scalar; the gradient of that loss is a vector.

A matrix is a rectangular array of numbers arranged in rows and columns; it is central because it does double duty: it stores data and it represents linear transformations that act on vectors.

• Two roles:

  • As data: a grid of values (a dataset, an image, an adjacency table).

  • As an operator: multiplying by a matrix rotates, scales, projects, or otherwise transforms vectors.

• Matrix multiplication is the workhorse: Composing transformations = multiplying their matrices, which powers layers of neural networks and systems of equations.

• Structure reveals properties: Rank, determinant, inverse, and eigenvalues describe what the transformation does (invertibility, stretching directions).

• Notation and shape: An m×n matrix maps an n-dimensional input to an m-dimensional output; matching these dimensions is what makes products valid.

In ML, a tensor is a multi-dimensional array: the generalization of scalars, vectors, and matrices to any number of axes (dimensions). Frameworks like PyTorch and TensorFlow use tensors as the fundamental data structure for all computation.

• Rank = number of axes: Rank 0 scalar, rank 1 vector, rank 2 matrix, rank 3+ higher tensors.

• Shape describes the size along each axis: e.g. a batch of color images is shape (batch, height, width, channels), a rank-4 tensor.

• Why ML uses them:

  • They naturally hold batched, multi-dimensional data (sequences, images, video) in one object.

  • They run on GPUs and support autograd, so gradients flow through tensor operations automatically.

• Practical note: The ML use of "tensor" means multi-dimensional array; it is looser than the physics/math notion of a tensor as a coordinate-invariant object.

A row vector is a 1×n matrix (one row, n columns) and a column vector is an n×1 matrix (n rows, one column); the distinction matters because it determines whether matrix multiplication is dimensionally valid and what result you get.

• Orientation: Row vector lies horizontally (shape 1×n); column vector lies vertically (shape n×1); transposing swaps between them.

• Why it changes the math:

  • Inner dimensions must match: a row (1×n) times a column (n×1) gives a 1×1 scalar (the dot product).

  • Reversed, a column (n×1) times a row (1×n) gives an n×n outer-product matrix.

• Convention: Math and ML usually default to column vectors, so Ax is well-defined and x^T A x forms a scalar.

• Practical trap: In NumPy a 1D array is neither; shape (n,) vs (n,1) vs (1,n) can silently broadcast wrong, so being explicit avoids bugs.

A matrix's shape is given as rows by columns, written m × n: the number of horizontal rows first, then the number of vertical columns.

• Convention: rows then columns: An m × n matrix has m rows and n columns; an element is indexed A[i][j] (row i, column j).

• Reading the numbers: A 3 × 2 matrix has 3 rows and 2 columns, so 6 total entries.

• Special shapes: Square when m = n; a column vector is n × 1, a row vector is 1 × n.

• Why order matters: Multiplication is only defined when inner dimensions match: (m × n)(n × p) = (m × p).

A unit vector has length (magnitude) exactly 1; you normalize any nonzero vector by dividing it by its own norm, which preserves direction but rescales length to 1.

• Definition: A vector v is a unit vector when ‖v‖ = 1, where the norm is ‖v‖ = sqrt(v·v).

• Normalizing: Compute û = v / ‖v‖; the result points the same way but has length 1.

• Why it's useful: Isolates pure direction, so comparisons (like cosine similarity) don't depend on magnitude.

• Caveat: The zero vector cannot be normalized: its norm is 0 and direction is undefined.

The dot product multiplies two vectors component-wise and sums the results into a single scalar; geometrically it measures how much the two vectors point in the same direction.

• Algebraic form: a·b = a₁b₁ + a₂b₂ + … + aₙbₙ, producing a scalar.

• Geometric form: a·b = ‖a‖‖b‖cos(θ), so it combines both magnitudes and the angle between them.

• Interpreting the sign: Positive: vectors point in a similar direction; zero: perpendicular; negative: opposing directions.

• Projection view: It equals the length of a's projection onto b times ‖b‖, i.e. how much of one vector lies along the other.

Two vectors are orthogonal when they are perpendicular, which algebraically means their dot product is zero.

• Test: a·b = 0 implies the angle between them is 90° (since cos(90°) = 0).

• Meaning: Neither vector has any component along the other; they carry independent, non-overlapping directions.

• Orthonormal: Vectors that are both orthogonal and unit length; these form convenient bases (e.g. the standard axes).

• Edge case: The zero vector is technically orthogonal to everything since its dot product is always 0, though it has no direction.

The dot product equals the product of the two vectors' magnitudes times the cosine of the angle between them, so you can solve for that angle directly from the dot product.

• The identity: a·b = ‖a‖‖b‖cos(θ), so cos(θ) = (a·b) / (‖a‖‖b‖).

• Isolating direction: Dividing out the magnitudes strips away length, leaving only how aligned the directions are.

• Range of cos(θ): +1 means identical direction, 0 means perpendicular, −1 means exactly opposite.

• Practical link: For unit vectors ‖a‖ = ‖b‖ = 1, the dot product is the cosine itself.

A linear combination of vectors is any sum of those vectors each scaled by a scalar: it is the fundamental way to build new vectors from a set.

• Definition: Given vectors v₁, v₂, ..., vₙ and scalars c₁, ..., cₙ, the combination is c₁v₁ + c₂v₂ + ... + cₙvₙ.

• Why it's central:

  • The span of a set is the collection of all its linear combinations.

  • Linear independence asks whether the only combination giving the zero vector uses all-zero scalars.

  • Matrix-vector product Ax is exactly a linear combination of A's columns weighted by x.

• Geometric picture: Combinations of two independent vectors fill a plane; of one nonzero vector, a line.

The cross product of two 3D vectors produces a new vector perpendicular to both, whose length equals the area of the parallelogram they span. It is defined only in 3D (and 7D).

• Key properties:

  • a × b is orthogonal to both a and b, with direction from the right-hand rule.

  • Magnitude: |a × b| = |a||b|sin θ, zero when the vectors are parallel.

  • Anti-commutative: a × b = -(b × a).

• How to compute: Via a symbolic determinant with unit vectors i, j, k in the first row.

• When it's used: Computing surface normals in graphics, torque and angular momentum in physics, and testing coplanarity/area.

A vector's norm is a measure of its length or magnitude. The most common is the Euclidean (L2) norm: the square root of the sum of squared components.

• L2 (Euclidean) norm:

  • ||v||₂ = √(v₁² + v₂² + ... + vₙ²), equivalently √(v·v).

  • Represents straight-line distance from the origin to the vector's tip.

• Other norms:

  • L1: Σ|vᵢ| (sum of absolute values, "taxicab" distance).

  • L∞: max|vᵢ| (largest component).

• What it represents: A notion of size used to measure distance (||a - b||), normalize vectors to unit length, and regularize models.

The transpose Aᵀ flips a matrix across its diagonal, turning rows into columns: (Aᵀ)ᵢⱼ = Aⱼᵢ. The reversal (AB)ᵀ = BᵀAᵀ happens because transposing swaps the roles of rows and columns in the product.

• What transpose does: An m×n matrix becomes n×m; the (i,j) entry moves to (j,i).

• Why the order reverses:

  • Entry (AB)ᵢⱼ is row i of A dotted with column j of B.

  • Transposing means (AB)ᵀ entry (i,j) equals (AB)ⱼᵢ: row j of A times column i of B.

  • That same value is column i of Bᵀ dotted with row j of Aᵀ, which is exactly (BᵀAᵀ)ᵢⱼ.

  • Dimensionally it must reverse too: AB is m×p, so (AB)ᵀ is p×m, which only BᵀAᵀ (not AᵀBᵀ) can produce.

Matrix multiplication C = AB defines each entry Cᵢⱼ as the dot product of row i of A with column j of B. It's row-times-column because it composes the two linear transformations they represent.

• The rule: Cᵢⱼ = Σₖ Aᵢₖ Bₖⱼ; inner dimensions must match (m×n times n×p gives m×p).

• Why row-times-column:

  • Matrices represent linear maps; multiplying them should represent applying one map then the other: (AB)x = A(Bx).

  • B first sends x to a combination of B's columns, then A acts on that; working out the entries forces the row-column dot product.

• Two useful views:

  • Column view: each column of AB is A times the corresponding column of B.

  • Composition, not entrywise: it is not the same as multiplying matching entries.

Matrix multiplication is not commutative because it composes transformations, and the order in which you apply transformations changes the result: AB means "do B, then A," which generally differs from BA.

• Order of operations matters: Rotating then scaling a vector rarely equals scaling then rotating; the maps compose differently.

• Dimensions may not even allow both: If A is m×n and B is n×p, AB is defined but BA may not be.

• Even for square matrices:

  • AB and BA usually give different results.

  • Special cases do commute: with the identity, with a matrix's own inverse or powers, and among simultaneously diagonalizable matrices.

Two matrices can be multiplied only when the inner dimensions match: the number of columns of the left matrix equals the number of rows of the right matrix. The result takes the outer dimensions.

• Conformability rule: For A (m×n) times B (p×q), the product AB exists only if n = p.

• Result dimensions:

  • The product AB is m×q: rows come from the left, columns from the right.

  • Mnemonic: (m×n)(n×q) → the inner n's cancel, outer m and q remain.

• Order matters: AB being defined does not imply BA is defined; even when both exist they usually differ.

• Each entry: (AB)ᵢⱼ is the dot product of row i of A with column j of B, so it needs equal-length row/column, which is exactly the conformability condition.

The Hadamard product multiplies matrices entry-by-entry and requires identical shapes, while matrix multiplication is the row-times-column dot-product operation representing composition of linear maps. They are fundamentally different operations.

• Hadamard product A ∘ B:

  • Both operands must be the same shape m×n, and the result is also m×n.

  • (A∘B)ᵢⱼ = Aᵢⱼ · Bᵢⱼ: purely local, no mixing across positions.

  • It is commutative.

• Matrix multiplication AB:

  • Requires conformable inner dimensions, not equal shapes.

  • Each entry sums products across a row and column, so information is mixed.

  • Generally non-commutative; represents composition of transformations.

• Where each shows up:

  • Hadamard: gating, masks, element-wise scaling (e.g. neural network activations).

  • Matmul: applying/composing transformations, projections, systems of equations.

Matrix addition and subtraction are done entry-by-entry, and both matrices must have exactly the same dimensions. The result keeps those same dimensions.

• Same-shape requirement: A and B must both be m×n; otherwise the operation is undefined.

• Entry-wise rule: (A ± B)ᵢⱼ = Aᵢⱼ ± Bᵢⱼ.

• Properties:

  • Commutative and associative: A + B = B + A, (A + B) + C = A + (B + C).

  • Identity is the zero matrix; subtraction is A + (−B).

Matrix multiplication is associative and distributive and respects scalar factors, but it is not commutative. These properties follow directly from its meaning as composition of linear maps.

• Associative: (AB)C = A(BC): lets you regroup chained products (and choose an efficient multiplication order).

• Distributive over addition: A(B + C) = AB + AC and (A + B)C = AC + BC.

• Scalar compatibility: c(AB) = (cA)B = A(cB).

• NOT commutative: AB ≠ BA in general; one may be defined while the other is not.

• Identity and transpose:

  • IA = AI = A with the identity matrix.

  • (AB)ᵀ = BᵀAᵀ: order reverses under transpose (and under inverse).

The entry in row i, column j of a product is simply the dot product of row i of the left matrix with column j of the right matrix. So matrix multiplication is a structured batch of all such row-column dot products computed at once.

• Per-entry definition: (AB)ᵢⱼ = rowᵢ(A) · colⱼ(B), a single dot product.

• Batch view:

  • For A (m×n) and B (n×p), you compute all m·p dot products, one per (row, column) pair.

  • This is why the shared dimension n must match: it is the length of every dot product.

• Why this framing is useful:

  • Explains vectorization: a matmul packs many dot products so hardware can parallelize them.

  • If B is a single column, it reduces to m dot products, i.e. a matrix-vector product.

A symmetric matrix is a square matrix that equals its own transpose: A = Aᵀ, so entries mirror across the main diagonal (aᵢⱼ = aⱼᵢ).

• Definition: Must be square; the (i,j) and (j,i) entries are always equal.

• Key spectral properties:

  • All eigenvalues are real.

  • Eigenvectors for distinct eigenvalues are orthogonal, so it is orthogonally diagonalizable: A = QΛQᵀ (the spectral theorem).

• Why it matters:

  • Covariance matrices, Gram matrices, and Hessians are symmetric, so these clean guarantees apply widely.

  • If additionally positive definite, it has all positive eigenvalues and a Cholesky factorization.

The identity matrix I is the square matrix with 1s on the main diagonal and 0s elsewhere; it is the multiplicative identity, leaving any conformable matrix or vector unchanged.

• Defining property: AI = IA = A and Ix = x: it acts like the number 1 for matrices.

• Geometric meaning: Represents the transformation that maps every vector to itself (does nothing).

• Where it appears:

  • Defines the inverse: AA⁻¹ = I.

  • Anchors eigenvalue problems via (A − λI)x = 0.

  • Used in regularization (e.g. adding λI) and as the starting point for many iterative algorithms.

The difference is how many entries are nonzero and how the matrix is stored: a dense matrix stores every entry, while a sparse matrix has mostly zeros and is stored by recording only the nonzeros.

• Dense matrix:

  • Most entries are nonzero; stored as a full 2D array using O(mn) memory.

  • Standard for small or fully populated data.

• Sparse matrix:

  • Vast majority of entries are zero; stored in formats like CSR/CSC/COO that keep only nonzero values plus their positions.

  • Memory and compute scale with the number of nonzeros, not the full size.

• Practical implications:

  • Sparse arises in graphs, text (TF-IDF), and finite-element problems; specialized solvers exploit the zeros.

  • Some operations (e.g. certain factorizations) cause fill-in, destroying sparsity, so algorithm choice matters.

A diagonal matrix is a square matrix whose only nonzero entries lie on the main diagonal; it scales each coordinate axis independently, which makes many operations trivial.

• Structure: aᵢⱼ = 0 for i ≠ j; fully described by its diagonal vector.

• Why it's convenient:

  • Multiplying Dx just scales each entry of x by the corresponding diagonal value.

  • Its inverse is the reciprocals on the diagonal; determinant is the product of diagonal entries; eigenvalues are the diagonal entries.

  • Powers are entrywise: Dᵏ raises each diagonal entry to the k-th power.

• Central to diagonalization: A = PDP⁻¹ turns hard matrix operations into easy diagonal ones in the eigenbasis; the Λ in eigen- and SVD decompositions is diagonal.

The determinant is a single scalar computed from a square matrix that measures how the matrix scales volume, and its sign tells you whether orientation is preserved or flipped.

• Geometric meaning: volume scaling factor:

  • In 2D it's the area of the parallelogram formed by the transformed unit vectors; in 3D, the volume of the parallelepiped.

  • A determinant of 2 means the transformation doubles areas/volumes.

• Sign encodes orientation: Positive: orientation preserved; negative: space is flipped/mirrored.

• Zero determinant = collapse: The transformation squashes space to a lower dimension (zero volume), meaning the matrix is singular.

• Useful properties:

  • det(AB) = det(A)·det(B), and det(A⁻¹) = 1/det(A).

  • Product of the eigenvalues equals the determinant.

The inverse of a square matrix A is the matrix A⁻¹ that undoes A: multiplying them gives the identity. It exists only when A is square and full rank (nonzero determinant).

• Definition:

  • A⁻¹ satisfies A·A⁻¹ = A⁻¹·A = I, the identity matrix.

  • It represents the reverse transformation.

• Existence conditions (all equivalent):

  • A must be square.

  • det(A) ≠ 0.

  • A is full rank; columns are linearly independent.

  • The only solution to Ax = 0 is x = 0.

• Why it matters:

  • Ax = b has the unique solution x = A⁻¹b when the inverse exists.

  • In practice, solve the system directly (e.g. LU/factorization) rather than forming A⁻¹ explicitly, which is slower and less stable.

An invertible matrix has an inverse and represents a reversible transformation; a singular matrix has no inverse because it collapses space and loses information. They are opposite, mutually exclusive cases for a square matrix.

• Invertible (nonsingular):

  • det ≠ 0, full rank, independent columns.

  • Ax = b has exactly one solution for every b.

• Singular:

  • det = 0, rank deficient, columns linearly dependent.

  • Ax = b has either no solution or infinitely many, never a unique one.

  • Has at least one zero eigenvalue.

• Intuition: Invertible = you can always undo it; singular = it flattens space so the original cannot be reconstructed.

You derive a distance metric from a norm by measuring the norm of the difference between two vectors: d(x, y) = ‖x − y‖. Applying the L2 norm gives Euclidean distance.

• The construction: Euclidean distance: d(x, y) = √(Σ(xᵢ − yᵢ)²), which is ‖x − y‖₂.

• Why it's a valid metric:

  • Non-negativity and identity: d(x, y) ≥ 0, zero iff x = y (from norm positivity).

  • Symmetry: ‖x − y‖ = ‖y − x‖ (from homogeneity with scalar −1).

  • Triangle inequality: d(x, z) ≤ d(x, y) + d(y, z) (from the norm's triangle inequality).

• General fact: Any norm induces a metric this way; L1 gives Manhattan distance, L∞ gives Chebyshev distance.

A dense layer applies an affine transform: it multiplies the input vector by a weight matrix, adds a bias, then applies a nonlinearity. So y = f(Wx + b), where the linear part Wx + b mixes inputs into outputs and the activation f adds nonlinearity.

• Shapes:

  • Input x is d-dimensional; W is m x d; bias b is m; output is m-dimensional.

  • Row i of W is the weights of neuron i: its output is a dot product wᵢ·x + bᵢ.

• Batching: Stack N samples as rows of X (N x d) and compute XWᵀ + b: one matrix multiply handles the whole batch (GPU-friendly).

• Why nonlinearity matters: Composing affine maps is still affine, so without f a deep net collapses to a single linear map.

• Other layers are structured matrices too: Convolutions and attention are also linear operations, just with weight-sharing or dynamic weights, expressible as matrix multiplies.

The span of a set of vectors is every point you can reach by their linear combinations; the vectors are linearly independent if none of them is a redundant combination of the others.

• Span:

  • All vectors of the form c₁v₁ + c₂v₂ + … + cₖvₖ for scalars cᵢ; it is the subspace those vectors generate.

  • Example: two non-parallel vectors in 3D span a plane, not all of space.

• Linear independence:

  • The only solution to c₁v₁ + … + cₖvₖ = 0 is all cᵢ = 0; no vector can be written using the others.

  • If some combination gives zero with nonzero coefficients, the set is dependent (has redundancy).

• Why they connect: An independent set that spans a space is a basis: the minimal vectors needed to reach every point exactly once.

Cosine similarity is the cosine of the angle between two vectors: it scores how aligned their directions are, ignoring magnitude, which makes it ideal for comparing embeddings or documents of different sizes.

• Formula: cos_sim(a, b) = (a·b) / (‖a‖‖b‖), ranging from −1 to 1.

• Magnitude invariance: A long document and a short one on the same topic can still score high, since only direction matters.

• Why it fits embeddings: Semantic models encode meaning in a vector's direction, so directional closeness reflects similarity of meaning.

• Note: On already-normalized vectors, cosine similarity reduces to a plain dot product, which is cheaper to compute at scale.

The inner product takes two vectors and returns a single scalar, while the outer product takes two vectors and returns a full matrix: they collapse versus expand dimensions.

• Inner product:

  • For column vectors, aᵀb multiplies (1 × n)(n × 1) giving a 1 × 1 scalar (the dot product).

  • Measures alignment/similarity between the vectors.

• Outer product:

  • abᵀ multiplies (n × 1)(1 × m) giving an n × m matrix where entry (i,j) = aᵢbⱼ.

  • Always rank-1; used to build matrices from vectors (e.g. in SVD, projections).

• Memory aid: Inner shrinks to a number, outer grows to a grid; the order of the transpose determines which you get.

Projecting vector a onto vector b gives the component of a that points along b: the "shadow" a casts on the line through b.

• The formula:

  • proj_b(a) = (a·b / b·b) · b: the scalar a·b / b·b is how many copies of b fit into the shadow.

  • If b is a unit vector, this simplifies to (a·b) · b.

• Geometric meaning:

  • The projection is the closest point on the line spanned by b to the tip of a.

  • The leftover part, a - proj_b(a), is orthogonal to b.

• Why it matters: It's the core operation behind decomposing vectors, Gram-Schmidt orthogonalization, and least-squares regression.

The trace of a square matrix is the sum of its diagonal entries. Despite its simple definition, it equals the sum of the eigenvalues and behaves nicely under many operations.

• Definition: tr(A) = Σ Aᵢᵢ, the sum of entries on the main diagonal.

• Key properties:

  • Linear: tr(A + B) = tr(A) + tr(B) and tr(cA) = c·tr(A).

  • Cyclic: tr(AB) = tr(BA) (even when AB ≠ BA).

  • Invariant under transpose: tr(Aᵀ) = tr(A).

  • Equals the sum of eigenvalues, so it's invariant under similarity tr(P⁻¹AP) = tr(A).

• Why it's useful: A cheap, basis-independent scalar summarizing a matrix, appearing in derivatives, covariance, and physics.

If a matrix represents a linear map, then multiplying two matrices produces the matrix of the composed map: applying one transformation after another. This is precisely why matrix multiplication is defined the way it is.

• Matrices are linear maps: A maps a vector x to Ax; B maps x to Bx.

• Composition becomes a product:

  • Apply B first, then A: A(Bx) = (AB)x, so the single matrix AB encodes doing B then A.

  • Read right-to-left, mirroring function composition f∘g.

• Why the definition works:

  • Dimensions must chain: B's output space is A's input space, matching the conformability rule.

  • Non-commutativity is natural: rotating then scaling generally differs from scaling then rotating.

• Associativity is free: (AB)C = A(BC) because composing functions is associative.

A matrix-vector product Ax is a linear transformation because it takes a vector as input and outputs another vector while preserving vector addition and scalar multiplication. The key insight: Ax is a linear combination of the columns of A weighted by the entries of x.

• Column-combination view:

  • Ax = x₁·col₁ + x₂·col₂ + … + xₙ·colₙ: x tells how much of each column to add.

  • The columns of A are exactly the images of the standard basis vectors.

• Linearity holds: A(x + y) = Ax + Ay and A(cx) = c(Ax), the two defining properties of a linear map.

• Geometric meaning:

  • A (m×n) sends vectors from ℝⁿ to ℝᵐ, stretching, rotating, projecting, or shearing space.

  • Every linear map between finite-dimensional spaces can be written as such a matrix once bases are fixed.

Naive matrix multiplication of two n×n matrices costs O(n³) scalar operations, while addition and matrix-vector products are much cheaper. Asymptotically faster multiplication algorithms exist but are rarely used in practice below very large sizes.

• Multiplication:

  • General (m×n)(n×p): O(m·n·p); for square n×n this is O(n³).

  • Each of the m·p output entries is a length-n dot product.

• Cheaper operations:

  • Addition/subtraction: O(m·n), one pass over entries.

  • Matrix-vector product: O(m·n).

• Sub-cubic algorithms:

  • Strassen: ~O(n^2.807); theoretical bounds go below O(n^2.37).

  • Large hidden constants and numerical stability limit real-world use.

• Practical note: Real performance is dominated by memory access and cache behavior; optimized BLAS libraries far outrun naive triple loops.

A triangular matrix has all its nonzero entries on one side of the diagonal (upper or lower); this structure lets you solve linear systems directly by substitution instead of full elimination.

• Two kinds:

  • Upper triangular: zeros below the diagonal.

  • Lower triangular: zeros above the diagonal.

• Why solving is easy:

  • Back substitution (upper) or forward substitution (lower) solves one unknown at a time in O(n²) work.

  • Determinant is just the product of the diagonal entries.

• Role in factorization:

  • LU decomposition writes A = LU, reducing Ax = b to two easy triangular solves.

  • The same idea powers Cholesky (A = LLᵀ) and the R factor in QR.

A permutation matrix is a square 0/1 matrix with exactly one 1 in each row and column; it is the identity with its rows (or columns) reordered, and multiplying by it simply rearranges rows or columns.

• What multiplication does:

  • PA permutes the rows of A.

  • AP permutes the columns of A.

• Key properties:

  • Orthogonal: P⁻¹ = Pᵀ, so it just relabels coordinates without stretching.

  • Determinant is +1 or −1 (sign of the permutation).

• Where it's used: Partial pivoting in LU gives PA = LU, where P tracks the row swaps for numerical stability.

As a matrix object, the covariance matrix Σ is a square matrix whose (i,j) entry is the covariance between feature i and feature j; the diagonal holds each feature's variance. It summarizes how variables vary together.

• What the entries mean:

  • Diagonal: variances (always ≥ 0).

  • Off-diagonal: pairwise covariances (sign shows direction of joint variation).

• Structural properties:

  • Symmetric: Σ = Σᵀ since cov(i,j) = cov(j,i).

  • Positive semidefinite: vᵀΣv ≥ 0 for all v, so all eigenvalues are ≥ 0.

• Why it matters:

  • Its eigen-decomposition gives the principal components: eigenvectors are the directions of variance, eigenvalues the variance along them (basis of PCA).

  • Computed as Σ = (1/(n−1)) XᵀX on centered data.

An eigenvector of a matrix A is a nonzero vector whose direction is unchanged when A is applied to it; the eigenvalue is the scalar by which it is stretched or shrunk. Formally Av = λv.

• The equation: Av = λv with v ≠ 0; found by solving det(A − λI) = 0 for λ, then (A − λI)v = 0 for v.

• Geometric meaning:

  • Eigenvectors are the axes the transformation leaves pointing the same way (or flipped).

  • The eigenvalue is the scaling along that axis: |λ|>1 stretches, |λ|<1 shrinks, negative λ flips direction, λ=0 collapses it.

• Why they matter:

  • Diagonalization A = PDP⁻¹ expresses A in its eigenbasis, making powers and dynamics easy.

  • They drive PCA (directions of variance), stability analysis, and PageRank (dominant eigenvector).

The trace equals the sum of the eigenvalues and the determinant equals their product (each counted with algebraic multiplicity). These give quick sanity checks and links to invertibility.

• Trace = sum of eigenvalues: tr(A) = Σλᵢ, also equal to the sum of diagonal entries.

• Determinant = product of eigenvalues: det(A) = Πλᵢ.

• Practical consequences:

  • det(A) = 0 iff some eigenvalue is 0 iff the matrix is singular (non-invertible).

  • For a 2×2 matrix, this lets you find eigenvalues directly: they solve λ² - tr(A)λ + det(A) = 0.

• Why it holds: Both are coefficients of the characteristic polynomial, so they are invariant under similarity transforms (basis changes don't alter them).

A matrix is diagonalizable if it is similar to a diagonal matrix: A = PDP⁻¹, which happens exactly when it has n linearly independent eigenvectors that form a basis.

• The condition: n × n matrix needs n independent eigenvectors; equivalently geometric multiplicity = algebraic multiplicity for every eigenvalue.

• What P and D are: Columns of P are the eigenvectors; D holds the corresponding eigenvalues on its diagonal.

• Sufficient shortcuts:

  • n distinct eigenvalues guarantees diagonalizability (but is not necessary).

  • Every real symmetric matrix is diagonalizable (spectral theorem).

• When it fails: Defective matrices with repeated eigenvalues lacking enough eigenvectors: they need Jordan normal form instead.

• Why it's useful: Makes powers and functions of the matrix easy: Aᵏ = PDᵏP⁻¹.

Eigenvalues satisfy several algebraic properties that let you predict how transforming a matrix transforms its spectrum, without recomputing from scratch. The eigenvectors often stay the same while eigenvalues change predictably.

• Aggregate identities: Sum of eigenvalues = tr(A); product = det(A).

• Transformation rules (same eigenvectors):

  1. Aᵏ has eigenvalues λᵏ.

  2. A⁻¹ has eigenvalues 1/λ (requires λ ≠ 0).

  3. A + cI shifts eigenvalues to λ + c.

  4. cA scales them to cλ.

• Structural facts:

  • A and Aᵀ share eigenvalues; triangular matrices show eigenvalues on the diagonal.

  • Similar matrices (B = P⁻¹AP) have identical eigenvalues.

  • Symmetric matrices have real eigenvalues; orthogonal matrices have |λ| = 1.

Least squares finds the x that minimizes ‖Ax − b‖². Geometrically, when b lies outside the column space of A, we project b orthogonally onto that column space; Ax̂ is that projection, and the residual (b − Ax̂) is perpendicular to every column of A.

• The setup:

  • Ax can only reach vectors in Col(A); if b isn't there, no exact solution exists.

  • Best we can do: pick the point in Col(A) closest to b, i.e. its orthogonal projection.

• The orthogonality condition:

  • The residual must be orthogonal to Col(A): Aᵀ(b − Ax̂) = 0.

  • Rearranging gives the normal equations AᵀAx̂ = Aᵀb.

• Projection matrix: The projection is Ax̂ = P b with P = A(AᵀA)⁻¹Aᵀ, and P is symmetric and idempotent (P² = P).

To solve Ax = b, reduce the augmented matrix [A | b] with Gaussian elimination and back-substitute. Existence and uniqueness follow from comparing ranks: a solution exists when b lies in the column space of A, and it's unique when A's columns are linearly independent.

• How to solve: Row-reduce [A | b] to echelon form, then back-substitute; or use A = LU / QR factorizations for stability and reuse.

• Existence (consistency):

  • A solution exists iff rank(A) = rank([A | b]), equivalently b ∈ Col(A).

  • If elimination yields a row [0 … 0 | c] with c ≠ 0, the system is inconsistent.

• Uniqueness:

  • Unique when rank(A) equals the number of unknowns (no free columns), i.e. full column rank.

  • If there are free variables (rank < #unknowns), there are infinitely many solutions.

• Square case: If A is n×n and invertible (det ≠ 0), there's exactly one solution x = A⁻¹b for every b.

Gaussian elimination systematically applies row operations to transform a matrix into an upper-triangular (echelon) form, making a system easy to solve by back-substitution. Pivots are the leading nonzero entries that anchor each step, and RREF (reduced row echelon form) is the fully simplified, unique end state.

• The process:

  • Use three row operations (swap, scale, add a multiple of one row to another) to zero out entries below each pivot.

  • Result is row echelon form: staircase of leading entries, zeros beneath them.

• Pivots:

  • The first nonzero in each row; their count equals rank(A).

  • Pivot columns correspond to basic variables; non-pivot columns give free variables.

  • In practice, partial pivoting (choosing the largest-magnitude candidate) improves numerical stability.

• RREF:

  • Each pivot is 1 and is the only nonzero in its column; the form is unique for a given matrix.

  • Reads off solutions, rank, and null space directly.

Over-determined systems have more equations than unknowns (tall A) and usually have no exact solution, so we seek a best fit; under-determined systems have fewer equations than unknowns (wide A) and usually have infinitely many solutions, so we pick a preferred one.

• Over-determined (m > n):

  • b typically lies outside Col(A), so Ax = b has no exact solution.

  • Solve with least squares: minimize ‖Ax − b‖ via normal equations or QR/SVD.

• Under-determined (m < n):

  • Fewer constraints than unknowns leaves free variables, hence a whole solution subspace.

  • Pick a canonical answer: minimum-norm solution x = Aᵀ(AAᵀ)⁻¹b, or add regularization / sparsity constraints.

• Unifying view: The pseudoinverse A⁺ handles both: least squares for tall, minimum-norm for wide.

The normal equations are AᵀAx = Aᵀb, obtained by requiring the least-squares residual to be orthogonal to the column space of A. They convert the (possibly unsolvable) least-squares problem into a square, solvable system.

• Where they come from:

  • Minimizing ‖Ax − b‖² and setting the gradient to zero gives Aᵀ(Ax − b) = 0.

  • Geometrically: the residual must be perpendicular to Col(A).

• When to use:

  • Over-determined systems and regression fitting where no exact solution exists.

  • If A has full column rank, AᵀA is invertible and x̂ = (AᵀA)⁻¹Aᵀb is unique.

• Caveat: Forming AᵀA squares the condition number, hurting numerical accuracy; prefer QR or SVD in practice.

An orthogonal matrix is a square matrix whose columns (and rows) form an orthonormal set, so QᵀQ = QQᵀ = I, meaning Qᵀ = Q⁻¹. "Orthonormal matrix" is often used loosely for the same thing, or for tall matrices with orthonormal columns but not necessarily orthonormal rows.

• Defining property:

  • Columns are mutually orthogonal unit vectors, so QᵀQ = I.

  • For square Q this also gives QQᵀ = I and cheap inverse Q⁻¹ = Qᵀ.

• Isometry (preserves geometry):

  • Preserves lengths and dot products: ‖Qx‖ = ‖x‖ and (Qx)·(Qy) = x·y.

  • Geometrically these are rotations and reflections.

• Determinant: det(Q) = ±1: +1 is a proper rotation, -1 includes a reflection.

• Numerical stability: Condition number is 1, so they don't amplify errors: the basis of QR and SVD algorithms.

• Tall case (orthonormal columns only): If Q is m×n with m>n, QᵀQ = I but QQᵀ is only a projection, not the identity.

Gram–Schmidt is a procedure that turns a set of linearly independent vectors into an orthonormal basis spanning the same subspace, by successively subtracting off components along the vectors already chosen and normalizing.

• The core step:

  1. Take v₁, normalize to q₁.

  2. For each later vₖ, subtract its projections onto all previous qⱼ, leaving a component orthogonal to them.

  3. Normalize the remainder to get qₖ.

• Result:

  • The qᵢ are orthonormal and span(q₁..qₖ) = span(v₁..vₖ) at every step.

  • This directly produces the QR factorization: A = QR with R upper triangular holding the coefficients.

• Numerical caveat: Classical Gram–Schmidt loses orthogonality with rounding; use Modified Gram–Schmidt or Householder reflections in practice.

Any linear geometric transformation is represented by a matrix whose columns are the images of the basis vectors; applying the transformation is just matrix-vector multiplication. Rotation, scaling, shear, and reflection are the standard building blocks, and composing them means multiplying their matrices.

• Scaling: Diagonal matrix diag(sx, sy) stretches each axis independently; det is the area/volume factor.

• Rotation: In 2D, [[cosθ, -sinθ], [sinθ, cosθ]]; orthogonal with det = 1, preserves lengths and angles.

• Reflection: Orthogonal with det = -1; e.g. diag(1, -1) flips across the x-axis.

• Shear: Off-diagonal entry like [[1, k], [0, 1]] slides points parallel to one axis; det = 1 (area preserved) but angles are not.

• Reading the matrix: Column j is where basis vector eⱼ lands, which makes the geometry inspectable.

• Composition and translation:

  • Combine transforms by multiplying matrices (order matters, they don't generally commute).

  • Translation is not linear; use homogeneous coordinates (an extra dimension) to fold it into a single matrix.

An orthonormal basis is a basis whose vectors are mutually orthogonal and each of unit length. It combines two nice properties: the vectors point in independent directions and each has length one, which makes coordinates and projections trivial to compute.

• Two conditions:

  • Orthogonal: qᵢ·qⱼ = 0 for i ≠ j.

  • Normalized: qᵢ·qᵢ = 1.

• Compact form: Stacking the vectors as columns of Q gives QᵀQ = I.

• Why it's useful:

  • Coordinates are just dot products: the ith coordinate of x is qᵢ·x.

  • No matrix inversion is needed to change into the basis, since Q⁻¹ = Qᵀ.

• How to get one: apply Gram–Schmidt, QR, or take orthonormal eigenvectors of a symmetric matrix.

The rank of a matrix is the number of linearly independent rows (equivalently columns): the dimension of its column space. It measures how much genuine information the matrix carries and governs solvability of linear systems.

• Definition: Rank = number of pivots after row reduction = dimension of column space = dimension of row space.

• Full rank vs deficient: A matrix has full rank when rank equals min(rows, columns); rank deficiency signals redundancy or dependence.

• Why it matters:

  • Determines whether Ax = b is solvable and whether the solution is unique.

  • A square matrix is invertible exactly when it has full rank; deficient rank means a nonzero null space.

  • Central to least squares, dimensionality reduction (low-rank approximation), and numerical stability.

Rank-nullity ties it together: rank + nullity = number of columns.

A basis is a minimal set of vectors that spans a vector space and is linearly independent; the dimension is the number of vectors in any basis, a fixed count for the space.

• Basis: two conditions:

  1. Spanning: every vector in the space is a linear combination of the basis vectors.

  2. Independence: no basis vector is a combination of the others (no redundancy).

• Uniqueness of coordinates: Given a basis, every vector has exactly one representation as a coordinate tuple.

• Dimension:

  • The size of any basis; all bases of a space have the same number of vectors.

  • Example: R^3 has dimension 3 with standard basis e1, e2, e3.

• Practical view: A basis is a coordinate system; dimension is the number of independent degrees of freedom.

Rank is the maximum number of linearly independent columns, which equals the dimension of the column space. Remarkably, the row rank always equals the column rank, so there is a single number called the rank.

• Two views, one number: Column rank = dim of column space; row rank = dim of row space; both equal the number of pivots.

• Why they're equal (row reduction argument):

  • Row operations preserve the row space and don't change linear dependence among columns, so pivots count both independent rows and independent columns.

  • Each pivot marks one independent row and one pivot column simultaneously.

• Factorization argument: Any A can be written A = CR where C has r independent columns and R has r rows; this same r bounds both row and column counts, forcing equality.

• Consequence: rank(A) = rank(Aᵀ), so transposing never changes the rank.

The null space (kernel) of A is the set of all vectors x satisfying Ax = 0. It captures the directions the matrix collapses to zero and directly determines whether solutions to a linear system are unique.

• Definition: N(A) = { x : Ax = 0 }; it is a subspace of R^n (contains 0, closed under sums and scaling).

• Dimension = nullity: nullity = n − rank; equals the number of free variables in row reduction.

• What it tells you about solutions:

  • Only the zero vector (trivial null space) means columns are independent and solutions are unique.

  • A nonzero null space means infinitely many solutions: any particular solution plus any null-space vector still solves Ax = b.

• General solution form: x = x_particular + x_null, so the null space is the homogeneous part of the full solution set.

The column space of A is the span of its columns: all vectors of the form Ax. It is exactly the set of right-hand sides b for which Ax = b has a solution.

• Definition: C(A) = { Ax : x in R^n }, a subspace of R^m; its dimension is the rank r.

• Link to solvability:

  • Ax = b is solvable iff b lies in C(A); otherwise no exact solution exists.

  • Multiplying Ax is just taking a linear combination of columns with weights x, so reachable outputs are precisely the column span.

• When b is not in C(A): Least squares finds the x minimizing ‖Ax − b‖ by projecting b onto C(A).

• Full column-space case: If C(A) = R^m (rank = m), every b is reachable, so the system is always solvable.

A subspace is a subset of a vector space that is itself a vector space under the same operations. In practice you only need to check three closure conditions.

• Three conditions:

  1. Contains the zero vector.

  2. Closed under addition: u, v in S implies u + v in S.

  3. Closed under scalar multiplication: c·u in S for any scalar c.

• Shortcut: Conditions 2 and 3 combine: S is a subspace iff it's nonempty and closed under linear combinations.

• Examples: Lines and planes through the origin in R^3; the span of any set of vectors; the null space and column space of a matrix.

• Non-examples: A line not through the origin fails the zero-vector test; the first quadrant fails scalar multiplication by negatives.

A matrix is full rank when its rank equals the maximum possible (the smaller of its row and column count), meaning its rows or columns are as linearly independent as they can be; it's rank deficient when some rows/columns are linearly dependent, collapsing information.

• Rank = number of linearly independent rows (= independent columns): It's the dimension of the space the matrix's columns actually span (the column space).

• Full rank:

  • For an m×n matrix, rank = min(m, n): no redundant directions.

  • A square full-rank matrix is invertible (nonzero determinant).

• Rank deficient:

  • Rank < min(m, n): at least one row/column is a combination of others.

  • The transformation squashes space onto a lower-dimensional subspace, so information is lost and cannot be recovered.

• Why it matters:

  • Full rank guarantees unique solutions to linear systems and invertibility.

  • Rank deficiency signals multicollinearity, non-unique or no solutions, and instability in numerical methods (e.g. regression, least squares).

A set of vectors forms a basis if it is both linearly independent and spans the space: every vector in the space can be written as exactly one unique combination of them.

• Linearly independent:

  • No vector is a combination of the others; the only way to combine them to get the zero vector is with all-zero coefficients.

  • Ensures no redundancy.

• Spanning:

  • Their linear combinations reach every vector in the space.

  • Ensures nothing is missing.

• Correct count: A basis for an n-dimensional space has exactly n vectors; too few can't span, too many can't be independent.

• Consequence: uniqueness: Because it spans and is independent, every vector has one and only one coordinate representation in that basis.

A zero determinant means the matrix collapses space to a lower dimension (zero volume), so the transformation cannot be reversed, which is exactly what singular means.

• Determinant = volume scaling factor: If it's zero, the output volume is zero: the transformed vectors are squashed onto a line or plane.

• Collapse implies dependence: Zero volume means the columns are linearly dependent, so the matrix is rank deficient.

• No inverse possible:

  • Many input vectors map to the same output, so there's no unique way to reverse the map (no inverse).

  • Formally, A⁻¹ would require dividing by det(A); dividing by zero is undefined.

• Also: a zero determinant means zero is an eigenvalue, another signature of singularity.

The inverse of a product is the product of the inverses in reverse order: (AB)⁻¹ = B⁻¹A⁻¹. The order flips because to undo a sequence of operations you must undo the last one first.

• The rule: (AB)⁻¹ = B⁻¹A⁻¹, provided both A and B are invertible.

• Why the order reverses:

  • Applying AB to a vector means B acts first, then A: like putting on socks then shoes.

  • To reverse it, undo the last action first: take shoes off, then socks, i.e. A⁻¹ then B⁻¹.

• Verification:

  • (AB)(B⁻¹A⁻¹) = A(BB⁻¹)A⁻¹ = A·I·A⁻¹ = AA⁻¹ = I.

  • The naive A⁻¹B⁻¹ generally fails because matrix multiplication is not commutative.

• Same idea applies to transposes: (AB)ᵀ = BᵀAᵀ.

Each of the three elementary row operations affects the determinant in a specific, predictable way: swapping negates it, scaling a row scales it, and adding a multiple of one row to another leaves it unchanged.

• Row swap: Multiplies the determinant by -1 (each swap flips the sign).

• Scale a row by k:

  • Multiplies the determinant by k.

  • So factoring a constant out of one row divides the determinant accordingly.

• Add a multiple of one row to another:

  • Leaves the determinant unchanged.

  • This is why Gaussian elimination to upper-triangular form lets you compute det as the product of the pivots (tracking sign changes from swaps).

• Practical use: Reduce to triangular form, multiply the diagonal, then apply a sign for the number of swaps and undo any scaling factors.

The determinant is multiplicative: for square matrices of the same size, det(AB) = det(A)·det(B). The determinant of a product equals the product of the determinants.

• Geometric intuition: det measures how a transformation scales signed volume; composing two transformations multiplies their scaling factors.

• Key consequences:

  • Inverse: det(A⁻¹) = 1/det(A), so a matrix is invertible iff det(A) ≠ 0.

  • Powers: det(Aⁿ) = det(A)ⁿ.

  • Similarity invariant: det(P⁻¹AP) = det(A), since the P factors cancel.

• Caveats:

  • There is no simple rule for det(A + B); multiplicativity applies only to products.

  • For a scalar multiple, det(cA) = cⁿ·det(A) for an n×n matrix.

The sign of the determinant tells you whether the transformation preserves or reverses orientation. A positive determinant preserves orientation, a negative one flips it (like a mirror reflection), and zero collapses space to a lower dimension.

• Positive determinant: Orientation preserved: a right-handed basis stays right-handed (rotations, scalings).

• Negative determinant: Orientation reversed: the space is flipped/mirrored, as in a reflection.

• Zero determinant: The transformation is singular: it squashes volume to zero, mapping onto a lower-dimensional subspace, and is not invertible.

• Magnitude vs sign: The absolute value gives the volume scaling factor; the sign carries only the orientation information.

A vector norm is a function that assigns a non-negative "length" to a vector, satisfying non-negativity, absolute homogeneity, and the triangle inequality. The L1, L2, and L∞ norms differ in how they aggregate the components' magnitudes.

• L1 norm (Manhattan):

  • Sum of absolute values: ‖x‖₁ = Σ|xᵢ|.

  • Encourages sparsity; used in Lasso regularization.

• L2 norm (Euclidean):

  • Square root of sum of squares: ‖x‖₂ = √(Σxᵢ²).

  • The ordinary straight-line length; rotation-invariant.

• L∞ norm (max/Chebyshev):

  • Largest absolute component: ‖x‖∞ = maxᵢ|xᵢ|.

  • Dominated by the single biggest coordinate.

• Unit-ball shapes: L1 is a diamond, L2 is a circle, L∞ is a square: a useful mental picture of how each measures distance.

The Lp norm is the general family ‖x‖ₚ = (Σ|xᵢ|ᵖ)^(1/p) for p ≥ 1. The value of p controls how much large components dominate the total, interpolating between summing all magnitudes and taking only the maximum.

• Special cases:

  • p = 1: sum of absolute values (Manhattan).

  • p = 2: Euclidean length.

  • p → ∞: the max term dominates, giving ‖x‖∞.

• Effect of increasing p: Larger p weights big components more heavily and downplays small ones; the unit ball inflates from a diamond toward a square.

• The p < 1 caveat:

  • For p < 1 the formula violates the triangle inequality, so it is a quasi-norm, not a true norm.

  • The "L0 norm" (count of nonzeros) is a common misnomer: it isn't actually a norm.

The triangle inequality states that ‖x + y‖ ≤ ‖x‖ + ‖y‖: the length of a sum is at most the sum of the lengths. A valid norm must satisfy it because it encodes the fundamental geometric fact that a direct path is never longer than a detour.

• Geometric meaning: Any side of a triangle is no longer than the sum of the other two; equality holds only when the vectors point the same direction.

• Why norms require it:

  • It is one of the three defining axioms (with non-negativity and homogeneity); without it the function isn't a norm.

  • It guarantees the induced distance d(x, y) = ‖x − y‖ is a proper metric.

• Consequences:

  • Gives the reverse triangle inequality |‖x‖ − ‖y‖| ≤ ‖x − y‖, which implies the norm is a continuous function.

  • Enables convergence arguments and makes optimization well-behaved (convexity of the norm).

The Frobenius norm is the matrix analogue of the Euclidean vector norm: the square root of the sum of the squares of all entries, ‖A‖_F = √(Σᵢⱼ aᵢⱼ²). It treats the matrix as one long vector.

• Equivalent formulations:

  • Trace form: ‖A‖_F = √(trace(AᵀA)).

  • Singular values: ‖A‖_F = √(Σσᵢ²), the root-sum-of-squares of the singular values.

• Interpretation: Overall "size" or total energy of a matrix; it measures the aggregate magnitude of all components at once.

• Uses:

  • Measures reconstruction error between matrices (e.g. in low-rank approximation).

  • Easy and cheap to compute; unitarily invariant (rotations don't change it).

PCA finds orthogonal directions of maximum variance. Those directions are the eigenvectors of the data covariance matrix (largest eigenvalues = most variance), and they can equivalently be obtained from the SVD of the centered data matrix, avoiding forming the covariance explicitly.

• Setup:

  • Center the data: subtract the mean from each column of X (n samples x d features).

  • Covariance matrix C = (1/(n-1)) XᵀX is d x d, symmetric, positive semidefinite.

• Eigen formulation:

  • Solve C vᵢ = λᵢ vᵢ: eigenvectors vᵢ are principal components, eigenvalues λᵢ are variance along them.

  • Symmetry guarantees real eigenvalues and orthogonal eigenvectors (spectral theorem).

  • Sort by λ descending; keep top k for a k-dimensional projection.

• SVD formulation (preferred numerically):

  • Factor the centered data X = U Σ Vᵀ.

  • Columns of V are the principal components; C = V (Σ²/(n-1)) Vᵀ, so eigenvalues are σᵢ²/(n-1).

  • Works directly on X, more stable than squaring into XᵀX (which worsens conditioning).

• Projection: Reduced data = X Vₖ (equivalently Uₖ Σₖ), the coordinates in the new basis.

Recommendation reduces to filling in a huge, mostly-empty user-item ratings matrix. Matrix factorization approximates that matrix as the product of two low-rank matrices (user factors and item factors), so a missing rating is predicted by the dot product of a user's and an item's latent vectors.

• The model:

  • Ratings matrix R (users x items) ≈ U Vᵀ, where U is users x k and V is items x k.

  • Prediction r̂ᵤᵢ = uᵤ · vᵢ: alignment in a k-dimensional latent space of tastes/genres.

• Why low rank:

  • Assumes preferences are driven by few hidden factors, so k ≪ users, items.

  • Compresses the matrix and generalizes to unseen (user, item) pairs.

• How it's solved:

  • Pure SVD needs a complete matrix, but ratings are sparse; instead minimize squared error over observed entries only, with regularization.

  • Typical optimizers: SGD or alternating least squares (ALS), often adding user/item bias terms.

• Payoff: Latent vectors capture similarity, so items near a user's vector are recommended.

I'd frame it as finding a low-dimensional subspace that preserves the most important structure of the data, then projecting onto it. The workhorse is a variance-preserving linear projection via PCA/SVD; I'd extend or swap methods depending on whether structure is linear, discriminative, or nonlinear.

• Baseline: PCA via SVD:

  1. Center (and usually standardize) the data.

  2. Compute X = U Σ Vᵀ and take the top k right singular vectors.

  3. Project: Xₖ = X Vₖ.

• Choosing k: Keep enough components for a target explained-variance ratio (e.g. 95%), read from σᵢ² cumulative sums.

• Alternatives by goal:

  • Supervised separation: LDA maximizes between-class vs within-class scatter (a generalized eigenproblem).

  • Very large/sparse data: truncated/randomized SVD for scalability.

  • Nonlinear manifolds: kernel PCA, or graph/eigenmap methods (still eigenproblems on a similarity matrix).

• Validation: Check reconstruction error and downstream task performance, not just variance retained.

Consider a regression model with many correlated features (multicollinearity). The design matrix is near-rank-deficient, so ordinary least squares weights blow up and generalize poorly. Linear algebra fixes this directly: decorrelate/reduce features and regularize the normal equations to stabilize the solution.

• The problem, in linear algebra terms:

  • OLS solves XᵀX w = Xᵀy; if XᵀX is ill-conditioned (near-singular), tiny data changes swing w wildly.

  • Symptom: huge, high-variance coefficients, overfitting.

• Fix 1: ridge regularization: Solve (XᵀX + λI) w = Xᵀy: adding λI lifts eigenvalues away from zero, improving conditioning and reducing variance.

• Fix 2: PCA / SVD preprocessing:

  • Project onto top principal components to drop redundant directions before fitting.

  • Small singular values (noise directions) are truncated, so the model focuses on signal.

• Result: Lower variance and better test accuracy, at the cost of a little bias: the classic bias-variance trade-off enabled purely by linear algebra.

Relational data maps naturally onto matrices: a table becomes a matrix of rows x attributes, and relationships between two entity types become an incidence/adjacency matrix. Once in matrix form, joins, aggregations, and link analysis become linear algebra operations.

• Tables as matrices:

  • Numeric columns form a feature matrix (rows = records, columns = attributes).

  • Categorical fields become one-hot / indicator vectors, so a table is a sparse 0/1 matrix.

• Relationships as matrices:

  • A many-to-many relation (e.g. users-purchases-products) is a bipartite adjacency matrix A (users x products).

  • Foreign-key links between entities of one type form a graph adjacency matrix.

• Operations become linear algebra:

  • A join/composition of relations ≈ matrix product: A B chains user→product→category.

  • Aggregations (counts, sums) are matrix-vector products with an all-ones or weight vector.

  • Co-occurrence/similarity: AᵀA gives product-product co-purchase counts.

• Why it's useful: Enables factorization, spectral clustering, and link analysis (PageRank is an eigenvector of the link matrix) on relational data.

An image is just a matrix (or a stack of matrices for color channels) of pixel intensities, so image processing is largely applying linear operators to that matrix: filters via convolution, transforms via matrix products, and compression/denoising via decompositions.

• Representation: Grayscale = an m x n matrix; RGB = three such matrices (a tensor).

• Filtering as convolution:

  • Blur, sharpen, edge detection (Sobel) slide a small kernel matrix over the image: a localized linear operation.

  • Convolution can be written as one big sparse matrix-vector product (Toeplitz form).

• Geometric transforms: Rotation, scaling, shear, and translation are matrix multiplications on pixel coordinates (homogeneous coordinates for affine).

• Compression and denoising:

  • SVD: keep top-k singular values to approximate the image with far less data (low-rank approximation).

  • Small singular values often hold noise, so truncation denoises.

• Analysis: Eigen-decompositions (e.g. eigenfaces via PCA) reduce image sets to a compact basis for recognition.

A symmetric matrix A is positive definite if xᵀAx > 0 for every nonzero vector x (positive semidefinite if ≥ 0). Intuitively it curves upward in every direction, which is exactly what makes optimization well-behaved and covariance/kernel matrices valid.

• Equivalent characterizations:

  • All eigenvalues are strictly positive.

  • All leading principal minors are positive (Sylvester's criterion).

  • Has a Cholesky factorization A = LLᵀ with positive diagonal.

• Properties:

  • Invertible, with a positive definite inverse.

  • Defines a valid inner product / norm, and a unique global minimum for the quadratic xᵀAx.

  • PSD is the boundary case (some eigenvalue = 0, singular).

• Why it matters in ML:

  • A Hessian that is PD means a strictly convex objective, so gradient methods converge to the unique minimum.

  • Covariance matrices are PSD by construction; PD is needed to invert them (e.g. Gaussians, Mahalanobis distance).

  • Kernel matrices must be PSD (Mercer's condition) for SVMs and Gaussian processes to be well-posed.

  • Cholesky of a PD matrix gives fast, stable linear solves and sampling.

SVD factors any real (or complex) matrix A into A = UΣVᵀ, where U and V are orthogonal and Σ is diagonal with non-negative singular values. It is fundamental because it reveals the matrix's rank, energy distribution, and best low-rank structure, and it works for any matrix (rectangular, singular, or noisy).

• The three factors:

  • V (right singular vectors): an orthonormal basis of the input space.

  • Σ (singular values): non-negative scale factors, sorted largest to smallest.

  • U (left singular vectors): an orthonormal basis of the output space.

• Geometric meaning: Every linear map is a rotation/reflection, then an axis-aligned scaling, then another rotation/reflection.

• Why it's universal:

  • Exists for every matrix, unlike eigendecomposition; numerically stable to compute.

  • Rank = number of nonzero singular values; the largest ones capture the dominant structure.

• Where it powers ML: PCA, low-rank approximation, pseudo-inverse and least squares, latent semantic analysis, recommender systems, and denoising.

SVD is the computational engine of PCA: applying SVD to a centered data matrix directly gives the principal directions and the amount of variance each captures, and keeping only the top components achieves dimensionality reduction and noise removal.

• PCA via SVD:

  • Center the data X, then compute X = UΣVᵀ.

  • Columns of V are principal directions; σᵢ²/(n-1) are the variances (eigenvalues of the covariance matrix).

  • Projections (scores) are UΣ, i.e. XV.

• Why SVD instead of eigen-decomposing the covariance: More numerically stable: it avoids forming XᵀX, which squares the condition number.

• Dimensionality reduction: Keep the top k singular directions; project data into that k-dimensional subspace that retains the most variance.

• Noise reduction: Signal concentrates in large singular values, noise in small ones; zeroing small singular values reconstructs a cleaner, lower-rank matrix.

Singular values are the non-negative scale factors σᵢ in A = UΣVᵀ, and the singular vectors are the orthonormal input directions (right, in V) and output directions (left, in U) they connect. Together they describe exactly how a matrix stretches space: A vᵢ = σᵢ uᵢ.

• Right singular vectors (columns of V): Orthonormal directions in the input/domain space, the axes the map acts on cleanly.

• Left singular vectors (columns of U): Orthonormal directions in the output/range space where those inputs land.

• Singular values (σᵢ ≥ 0):

  • The stretch factor along each paired direction; A maps unit vector vᵢ to σᵢ uᵢ.

  • Ordered largest to smallest; the largest is the operator (spectral) norm of A.

• What they represent:

  • Relation to eigenvalues: σᵢ are the square roots of the eigenvalues of AᵀA (and of AAᵀ).

  • Count of nonzero singular values equals the rank; near-zero ones flag redundancy or noise.