CSCI 8945 · Math background
Manifolds
Almost every modern representation learning system rests on one core intuition: high-dimensional data is not uniformly spread across its ambient space. Images, speech, molecules, and social-network embeddings all cluster near curved, lower-dimensional surfaces called manifolds. Understanding what a manifold is — and how to reason about geometry on one — is the foundation for dimensionality reduction, generative models, and geometric deep learning alike.
1. What is a manifold?
The core idea is disarmingly simple: a manifold is a space that looks flat when you zoom in close enough. More precisely, every point on an $n$-dimensional manifold has a neighborhood that resembles a piece of $n$-dimensional flat space $\mathbb{R}^n$.
Formal definition (topological manifold)
A topological manifold of dimension $n$ is a set $\mathcal{M}$ together with a notion of "nearby" points (a topology), such that every point $p \in \mathcal{M}$ has a neighborhood that can be mapped continuously and bijectively (with a continuous inverse) to an open set in $\mathbb{R}^n$.
Unpacking the jargon: a homeomorphism is just a continuous map that can be continuously undone — think of it as a rubber-sheet deformation, no tearing or gluing allowed. The map $\varphi$ assigns coordinates to each point in $U$. The dimension $n$ is the number of coordinates you need locally.
- Locally flat. Every small patch looks like ordinary $\mathbb{R}^n$. You can assign coordinates, measure angles and distances locally, do calculus.
- Globally curved. The global shape can be very different from $\mathbb{R}^n$: it can wrap around, have holes, or be compact (bounded) like a sphere.
- Lower-dimensional. An $n$-manifold living inside $\mathbb{R}^m$ with $n < m$ is a lower-dimensional surface in a higher-dimensional ambient space. The dimension $n$ counts the degrees of freedom, not the ambient coordinates.
A sphere is a 2-dimensional manifold. Globally curved; locally flat. Any small patch can be described with two coordinates just like a piece of a plane.
2. Examples of manifolds
The Earth — a sphere $S^2$
The surface of the Earth is the canonical example of a 2-manifold: it has two degrees of freedom (latitude and longitude), lives in 3D space, and is curved. Any map in an atlas covers a small patch and uses flat 2D coordinates; no single flat map covers the whole globe faithfully. This is precisely the structure the definition formalizes.
The data manifold hypothesis
Natural data — images, audio, text — lives in an enormous ambient space. A $256 \times 256$ grayscale image is formally a point in $\mathbb{R}^{65536}$. But the manifold hypothesis says that realistic images occupy only a tiny lower-dimensional corner of that space.
Think of faces. The space of all face images is parameterized by a handful of factors: identity, pose, expression, lighting direction, age. That is maybe tens of free parameters, not sixty-five thousand. The face images cluster near a curved low-dimensional manifold embedded in pixel space. Most of $\mathbb{R}^{65536}$ is filled with random noise — not natural images.
Data points (blue) cluster near a low-dimensional manifold even though the ambient space is enormous. Gray dots represent arbitrary points in ambient space — almost none are natural images.
Abstract examples: manifolds of matrices
Manifolds are not just surfaces in 3D. Many important objects in ML are themselves manifolds:
| Space | What it contains | Dimension | ML relevance |
|---|---|---|---|
| Sphere $S^{n-1}$ | Unit vectors in $\mathbb{R}^n$: $\{\mathbf{x} : \|\mathbf{x}\|=1\}$ | $n-1$ | Normalized embeddings (CLIP, SimCLR); direction-only representations |
| Stiefel manifold $\text{St}(k,n)$ | $n\times k$ matrices with orthonormal columns | $nk - \tfrac{k(k+1)}{2}$ | PCA subspace tracking; orthogonal weight matrices in neural nets |
| Symmetric positive-definite (SPD) matrices | $n\times n$ symmetric matrices with all eigenvalues $> 0$ | $\tfrac{n(n+1)}{2}$ | Covariance matrices; metric learning; brain connectivity (fMRI) |
| Lie groups: $SO(3)$, $SE(3)$ | 3D rotations; rigid-body transforms (rotation + translation) | 3; 6 | 3D pose estimation, robotics, protein structure, equivariant networks |
| Grassmannian $\text{Gr}(k,n)$ | $k$-dimensional subspaces of $\mathbb{R}^n$ | $k(n-k)$ | Subspace learning; domain adaptation; video representations |
These are all genuine manifolds. They have a well-defined notion of "nearby" (topology), local coordinates (charts), and intrinsic distances (geodesics). Designing networks that operate on these spaces — rather than naively treating them as flat $\mathbb{R}^m$ — is the subject of geometric deep learning.
3. Charts and atlases — local coordinate systems
A chart (also called a coordinate patch) is a pair $(U, \varphi)$ where $U$ is an open subset of the manifold and $\varphi: U \to \mathbb{R}^n$ is a homeomorphism onto its image. Think of a single map in a geographic atlas: it covers one region and assigns $(x, y)$ coordinates to every point in that region.
No single chart can cover all of $S^2$ (try flattening an orange peel without tearing it). So we cover the manifold with a collection of overlapping charts: an atlas. Where two charts overlap, the transition map $\varphi_j \circ \varphi_i^{-1}$ converts between the two coordinate systems.
Two overlapping charts $U_i$ and $U_j$ each map their region to flat $\mathbb{R}^n$. Where they overlap, the transition map converts between coordinate systems — it must be smooth if $\mathcal{M}$ is a smooth manifold.
Smooth manifolds
A smooth manifold (also called a differentiable manifold) is one where all transition maps are smooth ($C^\infty$ functions). This extra condition is what allows us to do calculus on the manifold: define derivatives, velocity vectors, and smooth curves. Almost all manifolds that appear in ML are smooth (or at least $C^1$), because we need gradients.
4. Tangent space — the local flat approximation
Once we have a smooth manifold, we can talk about directions and velocities. At each point $p$, the tangent space $T_p\mathcal{M}$ is the set of all possible instantaneous velocities of curves passing through $p$.
Concretely: imagine you are standing on the surface of the Earth. The tangent space at your location is the flat 2D plane that just touches the Earth's surface right under your feet. Any direction you could walk corresponds to a tangent vector. The tangent plane is the best flat approximation to the manifold near that point.
Point $p$ sits at the hilltop: the tangent plane $T_p\mathcal{M}$ is horizontal, touching the manifold at $p$. Tangent vectors $\mathbf{v}$ and $\mathbf{w}$ point along directions one can travel from $p$. The Exp map takes $\mathbf{v}$ in the tangent plane and follows the geodesic to land at $q = \mathrm{Exp}_p(\mathbf{v})$ on the manifold. The Log map inverts this: $\mathbf{v} = \mathrm{Log}_p(q)$ recovers the tangent vector from the destination.
Key properties of the tangent space
- It is a vector space. $T_p\mathcal{M}$ is an $n$-dimensional vector space (same dimension as the manifold). You can add tangent vectors and scale them. This is why gradients make sense: a gradient of a function on the manifold is a tangent vector.
- It changes from point to point. $T_p\mathcal{M}$ and $T_q\mathcal{M}$ at different points are different vector spaces — they are not directly comparable without extra structure (a connection). This is what makes optimization on manifolds tricky.
- It supports an inner product. Equipping each tangent space with an inner product $g_p(\cdot,\cdot)$ gives a Riemannian metric. This lets you measure lengths of curves, angles between vectors, and distances on the manifold.
5. Geodesics — shortest paths on a manifold
On a flat plane, the shortest path between two points is a straight line. On a manifold, the notion of "straight line" is replaced by a geodesic: the shortest (or locally shortest) path that stays on the surface.
Intuitively, a geodesic is what you get when you stretch a rubber band between two points on the surface and let it settle. It curves through space, but along the surface it is as straight as possible — it does not unnecessarily veer left or right.
A great-circle arc is the geodesic on a sphere. The Euclidean chord (dashed) cuts through the interior — it is not a valid path on the surface.
A cylinder can be unrolled into a flat rectangle — a chart. The geodesic becomes a straight line in the chart, which wraps into a helix on the cylinder. Charts turn geodesic problems into flat geometry.
On a data manifold, the geodesic distance between two points measures how far apart they are intrinsically — following the manifold's curvature. The Euclidean distance can be very misleading for points on different parts of a highly curved manifold.
The geodesic equation
Formally, a geodesic is a curve $\gamma(t)$ on the manifold whose velocity does not change direction along the curve (it has zero "turning" in the intrinsic sense). In flat $\mathbb{R}^n$ this means $\ddot{\gamma} = 0$, i.e., $\gamma$ is a straight line. On a curved manifold the equation picks up correction terms (called Christoffel symbols) that account for the curvature:
You do not need to memorize this formula. The key takeaway is that geodesics are the curved-space analogue of straight lines. In flat space, $\Gamma = 0$ and you recover straight-line motion. On a sphere, $\Gamma \neq 0$ and the geodesics curve — they become great-circle arcs.
Geodesics and the exponential / logarithmic map
Two operations defined using geodesics are especially important in ML:
- Exponential map $\operatorname{Exp}_p(\mathbf{v})$. Starting at point $p$, travel along the geodesic in the direction of tangent vector $\mathbf{v}$ for one unit of time. The result is a new point on the manifold. This is how manifold optimization algorithms take gradient steps while staying on the manifold.
- Logarithmic map $\operatorname{Log}_p(q)$. The inverse: given two points $p$ and $q$, find the tangent vector at $p$ that points toward $q$ along the geodesic. It encodes the direction and distance from $p$ to $q$ as a flat vector. Used in manifold-valued regression and Fréchet mean computation.
6. Connections to this course
Manifold thinking permeates modern ML. Here are five places you will encounter it explicitly:
Dimensionality reduction
PCA finds the best flat (linear) approximation to data. Isomap, UMAP, and LLE find curved lower-dimensional manifolds. The difference is whether you assume the data manifold is flat or curved.
Generative models (VAEs, diffusion)
The latent space of a VAE is designed to be a smooth manifold where interpolation makes sense. Diffusion models learn to traverse the data manifold by iteratively denoising.
Riemannian optimization
When parameters must satisfy constraints (orthogonal matrices, rotations, SPD matrices), manifold optimization replaces flat gradient descent with Exp/Log maps to stay on the manifold.
Hyperbolic embeddings
Hierarchical data (knowledge graphs, taxonomy trees) embeds much more faithfully in hyperbolic space (a negatively curved manifold) than in Euclidean $\mathbb{R}^n$ of the same dimension.
Geometric deep learning
Graph neural networks, equivariant networks, and mesh-based 3D networks operate directly on manifold-structured data (surfaces, rotation groups), exploiting manifold symmetries for generalization.
Manifold regularization
Semi-supervised methods penalize functions that vary rapidly along the data manifold (estimated from unlabeled data), encouraging classifiers to be smooth in intrinsic distance, not Euclidean distance.
7. Case studies: working through the definitions
Abstract definitions crystallise when you trace them through a concrete example. Below we work through the same checklist — manifold, chart, atlas, tangent space, geodesic, exponential and logarithmic maps — for two explicit manifolds, one in each study. Side by side, they highlight what is universal and what is specific to each geometry.
| Concept | Cosine curve $\mathcal{M}$ | Unit sphere $S^2$ |
|---|---|---|
| Dimension $n$ | 1 | 2 |
| Ambient space | $\mathbb{R}^2$ | $\mathbb{R}^3$ |
| Charts needed | 1 (global chart) | ≥ 2 (no global chart exists) |
| Tangent space | 1-D line, direction $(1,-\sin x_0)$ | 2-D plane, $\{v : v \cdot p = 0\}$ |
| Geodesic distance | Elliptic integral (no closed form) | $\arccos(p \cdot q)$ (clean formula) |
| Exp map | Implicit: follow arc by arc length | $\cos(\|v\|)\,p + \sin(\|v\|)\,\hat v$ |
Case study A
The cosine curve
Our first example is as simple as a 1-dimensional manifold can be: the graph of cosine, living inside $\mathbb{R}^2$. Being 1-dimensional means there is essentially only one direction at each point, which makes every formula concrete and checkable by hand.
The cosine curve from $p=(0,1)$ (crest) to $q=(\pi,-1)$ (trough). The orange arc is the geodesic; the dashed gray line is the shorter Euclidean chord that leaves the manifold. The Exp map follows the arc; the Log map returns the tangent vector $\hat{\mathbf{v}}$ scaled by the arc length.
The manifold
The cosine curve is the set of all points $(x, \cos x)$ as $x$ ranges over $\mathbb{R}$:
It is a 1-dimensional manifold. Every small patch looks like a piece of $\mathbb{R}^1$ (a line segment), even though globally the curve oscillates up and down.
Chart and atlas
The parameter $x$ gives a global chart: the whole manifold is covered by a single coordinate map.
The atlas is $\{(\mathcal{M},\,\varphi)\}$ — just one chart. This is possible because $\mathcal{M}$ is homeomorphic to $\mathbb{R}$, which needs no patch overlap. (Compare with the sphere, which cannot be covered by a single chart.)
Tangent space
Differentiate the parameterization $\gamma(t) = (t,\cos t)$ to get the velocity vector, which spans the tangent space:
At the crest $p=(0,1)$: $\sin(0)=0$, so the tangent is $(1,0)$ — perfectly horizontal. At the inflection point $(\pi/2,\,0)$: the tangent is $(1,-1)$, slope $-1$. The tangent direction changes at every point, reflecting the curve's bending.
Riemannian metric
The metric tensor (in the coordinate $t$) is the squared speed of the parameterization, inherited from the ambient $\mathbb{R}^2$ inner product:
At the crests and troughs ($x_0 = 0, \pi, \ldots$): $g_{tt} = 1$ — the parameter speed equals arc-length speed. At the inflection points ($x_0 = \pi/2, 3\pi/2, \ldots$): $g_{tt} = 2$ — the curve is "faster" by a factor of $\sqrt{2}$.
Geodesic distance
On a 1-dimensional manifold there is only one path between two points: follow the curve. The geodesic distance is the arc length of that path:
This is an elliptic integral — no closed-form antiderivative exists. It must be evaluated numerically. This is a common situation in differential geometry: even simple manifolds can have distances that resist exact formulas.
Exponential and logarithmic maps
Let $\hat{\mathbf{e}} = \frac{(1,-\sin x_0)}{\sqrt{1+\sin^2 x_0}}$ be the unit tangent at $p$, and let $\mathbf{v} = s\,\hat{\mathbf{e}} \in T_p\mathcal{M}$ be a tangent vector of (signed) arc length $s$. The Exp map follows the geodesic from $p$ for arc length $s$:
The Log map inverts this: given the destination $q = (x_1, \cos x_1)$, recover the tangent vector:
• Tangent direction: $(1, -\sin 0) = (1, 0)$ — horizontal, since the crest is a local maximum.
• Metric: $g_p = 1 + \sin^2(0) = 1$. Arc-length and parameter agree at this point.
• Geodesic distance: $\int_0^\pi \sqrt{1+\sin^2 t}\,dt \approx \mathbf{3.820}$. Euclidean chord: $\sqrt{\pi^2+4} \approx 3.724$ — the curve is only 2.6% longer.
• $\operatorname{Exp}_p(3.820\cdot(1,0)) = (\pi, -1) = q$. The Exp map follows the orange arc to land exactly at $q$.
• $\operatorname{Log}_p(q) = 3.820\cdot(1, 0)$ — the tangent vector pointing right, scaled by the arc length.
Case study B
The unit sphere $S^2$
The sphere is the prototype for everything that is genuinely curved. Unlike the cosine curve, it is compact (closed and bounded), cannot be covered by a single chart, and admits beautiful closed-form formulas for every operation — making it the preferred testbed for algorithms in geometric deep learning and Riemannian optimization.
The sphere $S^2$ viewed from the side. $p$ = north pole $(0,0,1)$; the tangent plane is horizontal. Tangent vector $\mathbf{v} = (\alpha,0,0)$ in $T_p S^2$ points right. The Exp map follows the great-circle arc (green) for angle $\alpha = \|\mathbf{v}\|$, landing at $q = \operatorname{Exp}_p(\mathbf{v})$. The Log map inverts it.
The manifold
A 2-dimensional manifold: two degrees of freedom (latitude and longitude), living in 3D ambient space. It is compact — no sequence of points on $S^2$ can "escape to infinity."
Charts and atlas
No single coordinate system covers $S^2$ smoothly everywhere. The minimal atlas uses two stereographic projections, each omitting one pole:
Here $N=(0,0,1)$ and $S=(0,0,-1)$ are the poles. On the overlap $U_N \cap U_S$, the transition map is $(u,v)\mapsto(u,v)/(u^2+v^2)$ — inversion in the unit circle. Familiar geographic coordinates $(\theta,\phi)$ also give a chart, but break down at the poles: longitude is undefined there, an example of a chart singularity.
Tangent space
At any point $p \in S^2$, the tangent space is the plane in $\mathbb{R}^3$ perpendicular to the position vector:
Intuitively: the set of all directions you can walk on the sphere from $p$ without immediately leaving its surface. At the north pole $N=(0,0,1)$, this is the $xy$-plane: $T_N S^2 = \{(v_x, v_y, 0)\}$.
Riemannian metric
The metric is inherited from the dot product in $\mathbb{R}^3$. In spherical coordinates $\theta$ (polar, from north) and $\phi$ (azimuthal), the metric tensor is:
The $\sin^2\theta$ factor captures the fact that lines of longitude converge at the poles: near $\theta=0$ or $\pi$, a step in $\phi$ covers very little arc length ($\sin\theta \to 0$). Near the equator ($\theta=\pi/2$), $\sin^2\theta = 1$ and the geometry is locally flat.
Geodesic distance
Geodesics on $S^2$ are great circles — intersections of the sphere with planes through the origin. The geodesic distance between unit vectors $p$ and $q$ has a clean formula:
This is simply the angle between the two position vectors. For antipodal points ($p \cdot q = -1$), $d = \pi$ (half the great circle). Crucially, $\arccos$ is a closed-form expression — a stark contrast to the elliptic integral needed for the cosine curve.
Exponential map
Starting at $p$ and moving in direction $\mathbf{v} \in T_p S^2$ for arc length $\|\mathbf{v}\|$ along the great circle:
Geometrically: rotate $p$ toward $\mathbf{v}$ (which is perpendicular to $p$) by angle $\|\mathbf{v}\|$. When $\|\mathbf{v}\| = \pi/2$, the result is the unit vector $\hat{\mathbf{v}}$ — you have moved a quarter of the way around the sphere. When $\|\mathbf{v}\| = \pi$, you reach the antipodal point $-p$.
Logarithmic map
Given a target $q \neq -p$ on the sphere, the Log map finds the tangent vector at $p$ that aims at $q$. First project $q$ onto $T_p S^2$, then scale to the geodesic distance:
The term $q - (p\cdot q)\,p$ strips the component of $q$ along $p$, leaving only the part in $T_p S^2$. Dividing by $\sqrt{1-(p\cdot q)^2} = \sin(d(p,q))$ normalises it, then multiplying by $\arccos(p\cdot q) = d(p,q)$ gives the correct arc length.
• $\|\mathbf{v}\| = \alpha$. Choose $\alpha = \pi/3$ (60°).
• $\operatorname{Exp}_{(0,0,1)}(\pi/3,\,0,\,0) = \cos(\pi/3)(0,0,1) + \sin(\pi/3)(1,0,0) = \bigl(\tfrac{\sqrt{3}}{2},\,0,\,\tfrac{1}{2}\bigr)$.
This is a point 60° south of the north pole on the prime meridian — exactly as expected.
• $d\bigl(p,\,q\bigr) = \arccos\bigl((0,0,1)\cdot(\tfrac{\sqrt{3}}{2},0,\tfrac{1}{2})\bigr) = \arccos(\tfrac{1}{2}) = \pi/3$ ✓
• $\operatorname{Log}_{(0,0,1)}\bigl(\tfrac{\sqrt{3}}{2},0,\tfrac{1}{2}\bigr) = \dfrac{\pi/3}{\sin(\pi/3)}\bigl[(\tfrac{\sqrt{3}}{2},0,\tfrac{1}{2}) - \tfrac{1}{2}(0,0,1)\bigr] = \dfrac{\pi/3}{\sqrt{3}/2}\cdot(\tfrac{\sqrt{3}}{2},0,0) = (\pi/3,\,0,\,0) = \mathbf{v}$ ✓
References
Core Wikipedia articles:
- Manifold
- Topological manifold
- Differentiable manifold
- Riemannian manifold
- Tangent space
- Geodesic
- Exponential map
- Atlas (topology)
- Stiefel manifold
- Grassmannian
- Hyperbolic space
- Nonlinear dimensionality reduction
Recommended reading:
- Lee, Introduction to Smooth Manifolds (Springer) — the standard graduate reference; Part I covers topological manifolds and charts.
- Bronstein et al., Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges — accessible overview of manifolds in ML.
- Goodfellow, Bengio & Courville, Deep Learning — §5.11.3: Manifold Learning.