Computational capabilities in large AI systems emerge from the collective dynamics and interactions of basic neural elements under simple optimization rules. What are the fundamental principles driving this? How do interactions, dynamics, and data structure conspire to yield these abilities? We’ll revisit a classic setting - that of learning word embeddings (such as word2vec) from natural language. We’ll show how the dynamics of such algorithms can be distilled into a simpler solvable theory. Such algorithms give rise to a well-known linear computational structure in their learned representations that enables analogical reasoning, one of the simplest examples of an “emergent” capability. We propose a solvable microscopic theory of how word correlations in language can arise from factorization in latent space and drive the emergence of this linear computational structure.