Inner Products, Norms & Distances | CSCI 8945

1. Inner product — measuring alignment

A vector $\mathbf{u} \in \mathbb{R}^n$ is a list of $n$ numbers. The inner product (also called the dot product) of two vectors $\mathbf{u}$ and $\mathbf{v}$ is:

$\mathbf{u} \cdot \mathbf{v} \;=\; u_1 v_1 + u_2 v_2 + \cdots + u_n v_n$

That’s the formula, but the geometric meaning is what really counts. The inner product is tied to the angle $\theta$ between the two vectors:

$\mathbf{u} \cdot \mathbf{v} \;=\; \|\mathbf{u}\|\,\|\mathbf{v}\|\cos\theta$

Reading this left to right: when $\theta$ is small (vectors point the same way), $\cos\theta \approx 1$ and the inner product is large and positive. When $\theta = 90°$, $\cos\theta = 0$ and the inner product is exactly zero — the vectors are perpendicular. When $\theta > 90°$, $\cos\theta < 0$ and the inner product is negative.

Intuition. The inner product measures alignment. Think of it as asking: “how much of $\mathbf{v}$ points in the same direction as $\mathbf{u}$?” It’s the length of $\mathbf{v}$’s shadow cast onto $\mathbf{u}$, scaled by $\|\mathbf{u}\|$.

What makes something an inner product?

The standard dot product is the most familiar choice, but the term “inner product” refers to any operation $\langle\cdot,\cdot\rangle$ satisfying three rules. These rules are what make geometry work — you need them for angles, projections, and orthogonality to behave sensibly:

Symmetric: $\langle \mathbf{u}, \mathbf{v} \rangle = \langle \mathbf{v}, \mathbf{u} \rangle$. Order doesn’t matter.
Linear: $\langle \alpha\mathbf{u} + \beta\mathbf{w},\, \mathbf{v} \rangle = \alpha\langle\mathbf{u},\mathbf{v}\rangle + \beta\langle\mathbf{w},\mathbf{v}\rangle$. Scaling or adding vectors in the first slot scales or adds the result.
Positive-definite: $\langle \mathbf{u}, \mathbf{u} \rangle > 0$ for any $\mathbf{u} \neq \mathbf{0}$. A non-zero vector always has positive “self-alignment” — it has real length.

Any operation satisfying these three rules is a valid inner product and induces its own geometry. A well-known alternative to the standard dot product is the weighted inner product:

$\langle \mathbf{u}, \mathbf{v} \rangle_W \;=\; \mathbf{u}^\top W \mathbf{v}$

where $W$ is any positive-definite matrix. This stretches or rotates space — some directions count more than others. Setting $W = \Sigma^{-1}$ (the inverse covariance of the data distribution) gives the Mahalanobis inner product. Its induced distance automatically accounts for correlations between dimensions and is scale-invariant: a feature measured in kilometers and another in grams get put on equal footing. It’s one of the workhorses of metric learning.

Vector u

u₁ 3.0

u₂ 1.0

Vector v

v₁ 1.0

v₂ 3.0

u · v = —
angle θ = —
—

2. Norms — the length of a vector

The inner product of a vector with itself gives its squared length: $\mathbf{u} \cdot \mathbf{u} = u_1^2 + u_2^2 + \cdots + u_n^2$. Taking the square root gives the Euclidean norm ($L^2$ norm):

$\|\mathbf{u}\|_2 \;=\; \sqrt{\mathbf{u} \cdot \mathbf{u}} \;=\; \sqrt{u_1^2 + u_2^2 + \cdots + u_n^2}$

This is just the Pythagorean theorem generalized to $n$ dimensions. But $L^2$ is not the only option. Different norms measure “size” differently, and this choice matters in ML:

Name	Formula	Also called	Intuition
$L^1$ norm	$\sum_i \|u_i\|$	Manhattan / taxicab	Total distance walking on a city grid — no diagonals allowed
$L^2$ norm	$\sqrt{\sum_i u_i^2}$	Euclidean	Straight-line (as-the-crow-flies) distance from the origin
$L^\infty$ norm	$\max_i \|u_i\|$	Chebyshev	The largest coordinate — worst-case deviation across all dimensions

Why norms matter in ML. Regularization penalizes weight vectors with large norms. $L^1$ regularization (Lasso) drives many weights to exactly zero — it produces sparse solutions. $L^2$ regularization (Ridge / weight decay) keeps all weights small but spread out. The choice of norm directly shapes what the model can learn.

Notice that the $L^2$ norm arises directly from the inner product — it is the only norm that “knows about angles.” The $L^1$ and $L^\infty$ norms do not come from any inner product; there is no $\langle\cdot,\cdot\rangle$ that produces them. A quick check: a norm comes from an inner product if and only if it satisfies the parallelogram law: $\|\mathbf{u}+\mathbf{v}\|^2 + \|\mathbf{u}-\mathbf{v}\|^2 = 2(\|\mathbf{u}\|^2 + \|\mathbf{v}\|^2)$. The $L^1$ norm fails this, which means there is no notion of “angle” in $L^1$ space.

3. Distance — how far apart are two points?

Given a norm, the distance between two points is the norm of their difference. The Euclidean distance is:

$d_2(\mathbf{u}, \mathbf{v}) \;=\; \|\mathbf{u} - \mathbf{v}\|_2 \;=\; \sqrt{\sum_{i=1}^{n}(u_i - v_i)^2}$

Swapping in the $L^1$ norm gives Manhattan distance, which counts the total coordinate-wise displacement:

$d_1(\mathbf{u}, \mathbf{v}) \;=\; \|\mathbf{u} - \mathbf{v}\|_1 \;=\; \sum_{i=1}^{n}|u_i - v_i|$

Any reasonable distance function (a metric) obeys three common-sense rules:

Non-negative, zero iff identical: $d(\mathbf{u}, \mathbf{v}) \geq 0$, and $d(\mathbf{u}, \mathbf{v}) = 0$ only when $\mathbf{u} = \mathbf{v}$. Two distinct points can’t be at distance zero.
Symmetric: $d(\mathbf{u}, \mathbf{v}) = d(\mathbf{v}, \mathbf{u})$. The distance from A to B equals the distance from B to A.
Triangle inequality: $d(\mathbf{u}, \mathbf{w}) \leq d(\mathbf{u}, \mathbf{v}) + d(\mathbf{v}, \mathbf{w})$. A detour through a third point is never shorter than going directly.

4. The hierarchy of structure

Euclidean space $\mathbb{R}^n$ is not just any space — it has an unusually rich geometric structure. The concepts above form a strict hierarchy: each level adds more tools but also requires more assumptions about the space.

Metric space Examples: word edit distance, graph shortest-path, dynamic time warping (DTW) Tools: nearest-neighbor search, clustering by distance, triangle inequality bounds

Normed space (vector space + norm) Examples: $\mathbb{R}^n$ with $L^1$ norm, $L^p$ spaces + Tools: vector arithmetic, regularization, interpolation between points

Inner product space (norm comes from an inner product) Examples: $\mathbb{R}^n$ with dot product, Hilbert spaces, function spaces + Tools: angles, projections, orthogonality, PCA, cosine similarity

Euclidean space $\mathbb{R}^n$ Flat, finite-dimensional, standard basis vectors $\mathbf{e}_1, \ldots, \mathbf{e}_n$ The default workspace of most ML algorithms

Why this matters. The structure your data lives in determines what tools are available. Cosine similarity and PCA require an inner product. $k$-means and interpolation need a vector space with a norm. $k$-NN search only requires a metric. If your data has a natural non-Euclidean metric — graph distances, string edit distances, tree distances — many standard ML algorithms silently assume flat Euclidean space and give wrong results. The field of metric learning is largely about choosing or learning the right notion of distance for a given task.

One important consequence: not every metric comes from a norm, and not every norm comes from an inner product. So a method that requires angles (like PCA or cosine similarity) cannot be directly applied to, say, a graph-distance metric space. Embedding such data into $\mathbb{R}^n$ — which is exactly what representation learning does — is one way to work around this: you find coordinates that approximately preserve the original distances, gaining access to all the inner-product tools in the process.

5. Cosine similarity — direction without magnitude

From the geometric form of the inner product, dividing by both magnitudes isolates just the angle:

$\text{sim}_{\cos}(\mathbf{u}, \mathbf{v}) \;=\; \frac{\mathbf{u} \cdot \mathbf{v}}{\|\mathbf{u}\|_2\,\|\mathbf{v}\|_2} \;=\; \cos\theta \;\in [-1,\, +1]$

Cosine similarity measures direction only. Two vectors that point the same way have cosine similarity $+1$ regardless of how long they are. Two perpendicular vectors have cosine $0$. Two opposite vectors have cosine $-1$.

Concrete example. Let $\mathbf{u} = (1, 2)$ and $\mathbf{v} = (3, 6)$. The L2 distance is $\sqrt{(3-1)^2+(6-2)^2} = \sqrt{20} \approx 4.47$ — they look far apart. But $\cos\theta = (1 \cdot 3 + 2 \cdot 6)\,/\,(\sqrt{5} \cdot \sqrt{45}) = 15/15 = 1.0$ — they point in exactly the same direction. In word embedding spaces, “run” and “running” often differ in magnitude but are nearly co-directional: cosine similarity correctly says they’re close.

Unit-sphere trick. If you $L^2$-normalize all embeddings before computing distances (i.e., divide each vector by its own norm so $\|\hat{\mathbf{u}}\| = 1$), then the L2 distance and cosine similarity give exactly the same ranking, because $\|\hat{\mathbf{u}} - \hat{\mathbf{v}}\|_2^2 = 2 - 2\cos\theta$. This is why many modern models (CLIP, SimCLR) normalize embeddings before computing losses.

Measure	Sensitive to magnitude?	Range	Common use in ML
Dot product $\mathbf{u} \cdot \mathbf{v}$	Yes	$(-\infty, +\infty)$	Attention scores, unnormalized logits
Cosine similarity	No (direction only)	$[-1, +1]$	Text & image retrieval, CLIP, sentence embeddings
L2 distance	Yes	$[0, +\infty)$	k-NN, metric learning, Gaussian RBF kernel

6. Beyond flat space: distances on manifolds

Everything so far assumes we are in flat, infinite $\mathbb{R}^n$. But real high-dimensional data rarely fills all of space — it tends to cluster near a lower-dimensional curved surface.

The manifold hypothesis — one of the central ideas of this course — says that data like images or speech signals, while formally living in a very high-dimensional space (millions of pixels or samples), actually lies near a much lower-dimensional manifold. The genuine degrees of freedom are small; most of that ambient high-dimensional space is empty. A face image has millions of pixels, but the space of realistic face images is far smaller — parameterized by pose, expression, lighting, identity, and a handful of other factors.

On a curved manifold, the shortest path between A and B follows the surface (geodesic), not the straight Euclidean chord. The Euclidean distance underestimates the true distance.

On a curved surface, Euclidean distance (the straight-line path through the ambient space) may be physically unreachable — you’d have to leave the surface. The geodesic distance is the length of the shortest path that stays on the surface. On Earth, the geodesic between two cities is a great-circle arc, not a chord through the globe.

Consequence for embeddings. Two points can be nearby in Euclidean distance but far apart on the manifold (they lie on different “sheets” that happen to be close in the ambient space). Manifold learning methods like Isomap and UMAP estimate geodesic distances rather than Euclidean distances, then find a flat embedding that preserves them as well as possible. This is a core theme we’ll return to throughout the course.

Riemannian geometry: a local inner product

A smooth manifold can be equipped with an inner product at each point. This is called the Riemannian metric tensor $g_p$: it lives in the tangent space at $p$ — the flat plane that just touches the manifold there. The inner product tells you how to measure lengths and angles in that local neighborhood.

Crucially, $g_p$ can vary from point to point. Flat Euclidean space is the special case where $g_p = I$ everywhere — the same identity matrix at every point. On a curved manifold, $g_p$ changes, which is why straight lines are no longer shortest paths.

In practice, many deep learning models implicitly learn something like a Riemannian structure: the embedding space gets “bent” by the model so that regions of high data density are stretched apart (more discriminative) while sparse regions are compressed. Hyperbolic embeddings — used for hierarchical data like knowledge graphs and taxonomies — are a concrete example of deliberately non-Euclidean geometry in ML. We will revisit these ideas when we discuss manifold learning and subspace structures later in the course.

7. So what? Connections to this course

Every representation learning method encodes a notion of similarity through the geometry of its embedding space. The choice of metric — which distances are small, which are large — is what the model learns to shape. As you go through the course, notice which layer of the hierarchy each method assumes and what happens when that assumption breaks.

Contrastive learning

Pull embeddings of similar items close (small L2 or large cosine); push dissimilar items apart. The loss is literally a distance computation in embedding space.

$k$-NN retrieval

Classify or retrieve by finding the $k$ nearest neighbors in embedding space. The choice of metric determines what “nearest” means — and can be learned.

Attention

In Transformers, attention scores are dot products (scaled by $1/\sqrt{d_k}$). A large dot product means “attend more to this token” — pure inner-product geometry.

PCA / MDS

PCA finds orthogonal directions of maximum variance — it requires an inner product. MDS embeds data by preserving pairwise distances — only a metric is needed.

Kernel methods

A kernel $k(\mathbf{x},\mathbf{y})$ is an inner product in disguise: $k(\mathbf{x},\mathbf{y}) = \langle\phi(\mathbf{x}),\phi(\mathbf{y})\rangle$ for some feature map $\phi$. SVMs and kernel PCA work in potentially infinite-dimensional spaces without ever computing $\phi$ explicitly — the geometry is defined entirely by the kernel.

Manifold learning

Isomap, LLE, UMAP approximate geodesic distances on the data manifold, then embed into flat $\mathbb{R}^k$. The metric is non-Euclidean; the result is Euclidean. This is the bridge between Sections 4 and 6 above.

References

Wikipedia articles for each concept covered above:

Relevant sections in the course reading materials:

Goodfellow, Bengio & Courville, Deep Learning — Appendix, Part I: Applied Math and Machine Learning Basics (linear algebra, probability). Ch. 2: Linear Algebra.
Blum, Hopcroft & Kannan, Foundations of Data Science — Ch. 2: High-Dimensional Space (norms, distances, geometry of high dimensions).