The river that runs through the old market town rises in a chain of low hills to the north, where the rain falls on peat and moss and drains slowly into a hundred small streams. By the time the water reaches the first stone bridge it has gathered enough force to turn a mill wheel, and for three centuries it did exactly that. The mill ground wheat and barley for the farms of the valley, and the miller kept a ledger in which every sack was written down with the name of the farmer, the weight of the grain and the price he charged. Those ledgers survive in the county archive, and they tell a quiet story of good harvests and bad ones, of prices that rose in the years of war and fell again when the soldiers came home.

Historians who study such records are careful not to read too much into a single entry. A low figure might mean a poor harvest, but it might also mean that the farmer sold most of his grain somewhere else, or that the miller made a mistake, or that a page was copied badly a century later. Only when many independent sources agree does a pattern become trustworthy. The same rule holds in most fields that depend on measurement. One reading of a thermometer says little about the climate; ten thousand readings taken over decades, from instruments that were checked against each other, can say a great deal.

Consider how a modern weather service builds a forecast. Thousands of stations report temperature, pressure, humidity and wind every hour. Balloons carry instruments into the upper air twice a day. Satellites measure the radiation leaving the top of the atmosphere at many wavelengths, and aircraft report conditions along their routes. None of these observations is complete, and each has its own kind of error. A numerical model of the atmosphere, solved on a grid that covers the whole planet, is used to combine them: the model's previous forecast is nudged toward the new observations in proportion to how much each one is trusted. The result is an estimate of the present state of the air that is better than any single source, and it becomes the starting point for the next forecast.

The equations that govern the atmosphere are well known. They describe how air moves under the influence of pressure differences, how it warms and cools as it rises and sinks, and how water changes between vapour, liquid and ice. What makes forecasting hard is not ignorance of the physics but the sensitivity of the system to small differences in its starting point. Two forecasts that begin from nearly identical states can diverge after a week or two until they bear no resemblance to each other. For this reason forecasters run many versions of the model at once, each starting from a slightly different state, and they report not a single answer but a range of outcomes with their probabilities.

In computing, a similar idea appears whenever a large problem is split across many machines. A single processor can only do so much arithmetic in a second, so engineers divide the work. Some problems divide easily: rendering the frames of a film, or checking millions of independent records, needs little communication between the workers. Others divide only with effort, because each piece of the answer depends on many others. Training a large neural network is of the second kind. The model's parameters must be updated using information gathered from every example in a batch, and every worker needs the updated parameters before it can continue.

There are several ways to share that work. In the simplest, every worker holds a full copy of the model and processes its own slice of the batch; the gradients are then averaged across workers so that every copy receives the same update. When the model becomes too large for one device, its parameters, gradients and optimizer state can be split across the workers instead, and gathered piece by piece only when they are needed. Another approach splits each large matrix multiplication itself, so that every device computes part of every layer, at the cost of exchanging partial results after each one. A third places different layers on different devices and passes activations from one to the next, like an assembly line. Real systems combine these methods, and choosing the combination is a matter of balancing memory, arithmetic and communication on the particular hardware at hand.

Serving a trained model to many users raises a different set of trade-offs. Each request arrives with a prompt, and the system must first read the whole prompt before it can produce the first word of the answer. After that it generates one token at a time, and each step depends on everything that came before. Reading the prompt is dominated by arithmetic; generating tokens is dominated by moving the model's weights and the stored context through memory. A server that batches many users together can share the cost of reading the weights, but every user then waits for the slowest part of the batch. Good schedulers interleave the two phases carefully, so that long prompts do not stall the users who are already receiving answers.

The kitchen of a busy restaurant offers a useful comparison. Orders arrive at unpredictable times. Some dishes take a long time to prepare and a short time to finish; others are the reverse. A head chef who assigns every cook to one table at a time wastes their hands while the oven is busy. One who batches every order of the same dish may serve a crowd quickly but leave a single diner waiting far too long. The best kitchens watch both the average time to serve a plate and the longest wait at any table, because a customer remembers the worst evening, not the average one.

Engineers describe these promises with service level objectives. A provider might commit that the first word of an answer appears within a second for ninety-five requests out of a hundred, and that each following word arrives within a few hundredths of a second. Meeting such a promise at low load is easy. The real question is how much traffic a system can carry before the promise breaks, and that depends on the model, the hardware, the length of the prompts and answers, and dozens of settings that interact in ways that are hard to predict without measuring.

Measurement itself requires care. A benchmark that sends requests one after another, waiting for each to finish, measures something different from one that sends them at random times, as real users do. A benchmark that always uses the same prompt may be helped by caches that would not help real traffic. A result measured on one day, with one version of the software, may not hold on another. Careful laboratories run the old and new configurations side by side, on the same machine, in the same hour, and they repeat the old one at the end to check that nothing drifted while they were working.

Back in the market town, the mill stopped turning in the nineteen fifties, when a larger mill downstream began to grind grain with electric motors. The building became a store, then a workshop, and finally a small museum. On summer weekends a volunteer opens the sluice and lets the river turn the wheel again for an hour, and children stand on the gallery to watch the great stones move. The ledgers are kept in a glass case by the door, open at a page from the autumn of a year when the harvest was so good that the miller wrote, in a margin, that he had never seen so much grain.

Some of the visitors are engineers, and they tend to linger by the gears. The wooden teeth were cut by hand and fitted so that they would wear evenly, and the miller adjusted the gap between the stones with a lever that was sensitive enough to change the texture of the flour. He knew, without any equations, that grinding too fast would heat the flour and spoil it, and that grinding too slowly would leave the farmers waiting at his door. He balanced throughput against quality every day of his working life, and his ledgers are the record of how well he did it.
