Curved Spaces: Manifolds and High-Dimensional Representation Learning

When I charted planetary motion across the celestial sphere, I worked with a profound illusion: the heavens appeared as a two-dimensional surface enclosing Earth, yet the planets moved through three-dimensional space. My coordinates—celestial longitude, latitude—described positions on this apparent sphere, not the true spatial trajectories. I was mapping a manifold without knowing the term.

The Geometry of Nearness Without Distance

Topology asks: what remains when we strip away measurement itself, keeping only the notion of nearness? As the minimal structure preserving convergence and continuity, it reveals that “closeness” need not depend on distance. Open sets define neighborhoods; limits emerge without metrics. Many spaces in analysis resist measurement yet remain perfectly navigable through topological structure alone.

This resonates with how manifolds formalize my celestial intuition. Earth appears flat locally—from Alexandria’s harbor, the curvature vanishes beneath my feet—yet globally curves into a sphere. A manifold demands precisely this: every point possesses a neighborhood resembling flat Euclidean space, regardless of global complexity. The sphere, the torus, smooth curves without sharp corners—all manifolds. The cone’s pointed tip fails this test; no amount of zooming makes a point appear flat.

Dimension counts the coordinates needed locally. Earth’s surface requires two: latitude and longitude. A circle, despite embedding in the plane, remains one-dimensional—a tiny ant perceives only forward and backward along the curve. Higher dimensions escape visualization but follow the same principle: three-dimensional manifolds describe possible universe shapes; four-dimensional spacetime manifolds encode general relativity’s curved geometry.

Neural Geometries and Learned Coordinates

Now consider neural networks processing images. Raw pixel space sprawls across millions of dimensions—a catastrophic realm where distances become meaningless, volume concentrates at hypercube corners, and exhaustive search proves astronomically infeasible. Yet neural networks not only survive but thrive in these dimensionalities. How?

The manifold hypothesis offers an answer: real data does not uniformly fill high-dimensional space. Cat images occupy a low-dimensional manifold within pixel space—the “cat manifold” of possible cat configurations. Neural networks discover this intrinsic geometry through learned transformations. Each layer maps inputs through geometric operations—rotating planes, folding via ReLU activations—constructing representations where complex patterns become linearly separable. The network learns coordinate systems for the data manifold.

This parallels how I chose coordinate systems for celestial positions. Ecliptic coordinates suited planetary motion; equatorial coordinates aligned with Earth’s rotation. Each system revealed different geometric structure. Do different network architectures similarly discover distinct manifold structures in identical data?

Consider hippocampal place cells encoding two-dimensional spatial position through high-dimensional neural activity. A population of neurons collectively represents location—a distributed coordinate system for the spatial manifold. When animals navigate, place cells fire selectively based on position, tiling the environment with overlapping receptive fields. The brain constructs an allocentric map, independent of orientation or viewing angle, much as my celestial coordinates remained invariant across different observing positions.

The curse of dimensionality reverses in optimization: more parameters provide escape routes from local minima. Each dimension offers freedom. Manifold structure restores meaning to distance—measured along the curved surface, not through ambient space. Neural representations preserve topological structure: nearby inputs map to nearby representations, continuity maintained through learned deformations.

Perhaps representation learning rediscovers what we geometers always knew: the right coordinates reveal intrinsic structure invisible in original embeddings. Data’s true shape exists independently of how we encounter it, waiting for appropriate charts to make the hidden geometry manifest.