Ideas Are Rivalrous: The Thermodynamic Cost of Innovation

The 2025 Nobel Prize in Economics was awarded to Mokyr, Aghion and Howitt for “for having identified the prerequisites for sustained growth through technological progress”i. Mokyr’s historical research attempted to explain why innovation flourishes. Each new technology competes to replace the old, creating both incentives for innovators and disruption for incumbents.

In traditional economic models, physical capital is rivalrous: only one factory or person can use a machine at a time. In contrast, it is claimed that ideas can be reused infinitely; once an innovation exists, it can boost productivity everywhere without being depleted. This is misleading. New ideas need to be learned and implemented via new algorithms and machines, and this depletes capital. These resources are rivalrous and their allocation to new technology can lead to obsolescence and scrapping of old capital—a key theme in the Aghion–Howitt model of creative destruction.

There is a hidden assumption in the claim that ideas can be reused without limit: all humans are equipped with roughly the same biological machinery for learning. This obvious fact is the result of the very wasteful process of evolution. The cost of ideas has already been paid in countless rounds of variation and selection, trial and error. The rise of AI has dramatically highlighted the fundamental cost of ideas (or intelligence): the number of tokens you can generate per unit of energy. What is the link between the ‘token accounting’ of AI and the cost of ideas?

Life on earth is possible because the planet is near a hot star and surrounded by cold space. Heat energy emitted from the sun is absorbed by the earth and remitted into empty space at the same rate. It is in a steady-state, but far from a thermal equilibrium. Between absorption and emission of heat by the earth, some energy is stored as Free Energy ¹. This is life. All living things are evolutionary, metastable states of a complex network of electro-chemical processes driven far from thermal equilibrium by access to low-entropy energy, a.k.a food. We can think of a store of free energy as a battery. We can extract work by using it up as the store is returned to thermal equilibrium.

Natural selection drove the development of the central nervous system (CNS) in locomotive animals, dramatically increasing free energy by lowering entropy and allowing a learning machine to arise. Fossil fuels also store free energy, but the free energy stored by evolution in the CNS of animals is vastly greater. How is it stored?

The relentless pursuit of sustenance demands energy expenditure. Every organism is locked in the struggle to find effective strategies to acquire energy, ensuring survival and the perpetuation of life itself. This delicate balance is critical; expending excessive time and energy in the quest for nourishment jeopardizes the future, risking the ability to reproduce. The evolution of animals with a central nervous system allowed this process to become more efficient through the development of biological learning machines. A learning machine stores free energy in the way it compresses the information extracted from its interactions with the world.

This may sound difficult to believe. A simple story may help. Imagine you are lost and starving in the jungle. The you see a fisherman by a river. You could eat the Fisherman, but a better long-term strategy is to ask the Fisherman to teach you how to fish. This only works because evolution has equipped you with the same kind of learning machine used by Fishermen. That makes it very inexpensive to learn by talking with this Fisherman.

Humans learn novel and effective strategies to intervene in the world. They have ideas, make tools, and teach each other how to use them. Ideas and learning are not free. They are objective features of a special kind of complex machine and, like all machines, constrained by thermodynamic principles.

Learning machines are dissipative systems driven from thermal equilibrium by an external energy source and necessarily generate waste heat. The current focus on AI algorithms masks this feature, although the huge cost of electricity to power the machines running the algorithm gives the game away. This is fundamental and not an accident. There is a minimum thermodynamic cost to learning and modern AI systems are extremely far from it. The biological learning machines built by evolution are an enormous store of free energy in the form of a complex system of vastly reduced entropy.

Evolution paid an enormous thermodynamic cost to produce biological learning machines. This cost is largely invisible as the waste products of the evolutionary learning process are dead. We take the incredible thermodynamic advantage provided by biological learning machines for granted.

The reason that learning consumes energy, and requires work, is easy to see. The irreversible dynamics of learning necessarily lowers the entropy of any physical system that supports the process. This low entropy state becomes the thermodynamic resource that enables learning machines to lower the entropy of raw materials in their environment and create wealth.

A homonid randomly chipping stones may accidentally produce a stone axe, but the amount of waste stone generated will be enormous. Most of the agent’s internal energy will be used up producing the waste and not reducing the entropy of the raw materials at all but increasing it. The solution to this problem was solved by evolution; build a learning machine. It is the learning machine inside the agent that enables it to change the distribution in feature space to enable optimal control.

The fundamental law of learning machines says that a properly configured learning machine will minimize the probability of making an error when it expends the least amount of energy in each trial. The training process is a feedback from output to the internal configuration of the machine that minimizes the error. This process is irreversible as the change in the internal configuration is made in the presence of friction.

A learning machine once trained will produce goods with the least consumption of energy per unit. So long as the efficiency in production so gained pays for the energy lost to train the machine, you win. The cost of building and training the learning machine is a capital expenditure that contributes directly to production. In the case of axe production, the physical labour is spent in learning to minimise the amount of energy required to produce each axe. It is essential that we not only include the cost of labour and the capital cost, but the cost of learning when we seek a formula to estimate productivity.

The AI industry focus is rapidly moving from chip price, capital expenditure, and traditional KPIs to “tokens per watt”. In AI economics, it’s not just about more powerful processors—it’s about optimizing token throughput relative to energy consumed, aligning computation with business outcomes.

A machine learning algorithm requires a machine to run it on, an Nvidia GPU for example, and it consumes energy as it switches transistors on and off while the calculation proceeds. Suppose we submit a query to a large language model (LLM) like ChatGOT for example. The first step is to encode all the input as words, sub-words, or punctuation marks are split into tokens. Each token is assigned a unique numerical ID, allowing it to be represented in a way the model can understand and manipulate.

Each token ID is linked to a high-dimensional vector called an embedding. This vector captures the meaning and correlations between parts of the input. These embeddings are the actual inputs to neural network layers in modern AI models. Models like transformers use sequences of these embeddings to learn patterns and connections by adjusting the numbers in embedding space. During training, the model processes millions of token sequences and gradually updates its internal settings to understand language, images, or sound. For language models, the AI learns to predict the next token based on previous ones and context. For other types of data, like images or audio, tokens represent parts of the input, and the model learns to interpret or rebuild them step by step.

At inference, the model generates new tokens, turning internal representations back into data (such as text) for the user. By analyzing, learning relationships between, and generating tokens, learning algorithms develop the ability to understand, reason, and communicate in natural or structured domains.

Processing more tokens demands proportionally more RAM, VRAM, and GPU cycles. Transformer architectures—core to LLMs—scale computationally with the number of tokens, sometimes quadratically (especially in attention mechanisms).

In the last twelve months, the approach to machine learning has been fundamentally changed by the implementation of reasoning models. OpenAI introduced its first reasoning model—known as o1—on September 12, 2024, moving beyond the Chatbot phase of early consumer models. In reasoning models, tokens are generated from inside the inference phase itself. Reasoning models often produce chains of thought or detailed solution traces before generating a final answer. These reasoning traces can be several times longer than standard outputs—up to 20x higher in some advanced reasoning tasks and models. While a standard question/answer model might generate a concise answer, a reasoning LLM will output extended explanations, step-by-step logic, or code to justify its conclusion, each segment adding tokens. Models produce these extra tokens sequentially—each new token depends on all previous ones—so memory requirements and compute time scale with output length. This impacts inference slowdown and resource consumption as token sequences get longer.

While this increases the performance of the models to the point where they can ace international mathematics competitions, it dramatically increases the cost of inference by generating tokens for reinforcement learning. The change in token count is highly task-dependent, model-dependent, and can be controlled for efficiency in some architectures. About 60% of businesses using LLM APIs have exceeded their planned budgets due to unanticipated token use, underscoring the importance of robust monitoring and control tools.

Errors in prediction or classification are dependent on the mathematical form of token embedding. High-quality token embeddings enable learning algorithms to better capture semantic meaning, relationships, and context among data units such as words or code, which leads to more accurate and reliable predictions. Poorly constructed embeddings may miss complex correlations in the token sequences. Models with robust embeddings are less likely to make these errors because they have a more faithful representation of language or other underlying data.

In summary, it is tokens in and tokens out. Tokens determine the model’s context window—how much information it can process at once—and directly impact accuracy, efficiency, and inference cost. In between input and output there are trillions of transistors switching on and off each second, transforming embedded tokens.

Switching a transistor on and off costs energy and this is where the fundamental cost is paid. Larger and more expressive embeddings often lead to better prediction quality but require more energy, memory, and computational power at inference. The most efficient token embedding systems are co-designed to match hardware capabilities—using hardware-aware algorithms for dynamic token pruning, data compression for sparse input, and pipelines that execute key computations near memory arrays (processing-in-memory/PIM). These innovations cut data movement overhead and maximize tokens-per-watt metrics.

Nvidia’s Jensen Huang has proposed the concept of an AI factory where hardware co-design strategies for token reduction are fundamental to maximizing efficiency measured as tokens generated per unit of energy and capital. The strategy is to achieve exponential performance growth while driving down the cost per token. By focusing on algorithm-hardware integration, each incremental improvement translates directly into lower operational costs and higher return on investment.

In a traditional factory raw materials are processed to produce lower entropy products by doing thermodynamic work, but a key role is played by learning. A homonid learns a procedure for making stone axes. It is not error-free. Even if it is followed exactly, it can fail to produce a useful axe, but most of the time it produces a good axe. A recipe used by a baker to make a cake is another kind of procedure. A cake has less entropy than the raw ingredients that went in it. Henry Ford was one of the first innovators to realizethat the algorithm organizing the factory can make a huge difference to entropy reduction per unit cost.

Jensen Huang’s concept of an AI factory makes a direct link between learning and entropy reduction. Making this process more thermodynamically efficient is the key to wealth growth. In the AI factory, raw data (text, images, experimental data) enters, is processed with massive computation, and intelligence, in the form tokens, exits—ready to be used by businesses, software, robots, and other industries. They key is to maximise the entropy reduction per unit of free energy used. Better hardware co-designed with better algorithms is the new drive of GDP growth.

AI factories run continuously, much like power plants, operating at scale to produce high-value intelligence through enormous compute infrastructure. Tiny improvements in quality or efficiency produce outsized gains in engagement and profitability for clients. Industrial scale intelligence will transform our world in a matter of years. The real bottle neck is the need to talk to humans. If we require that, the machine needs to transform its internal embeddings into the algorithm of human language which evolved over millions of years of trial and error. There is no good reason for this if the real objective is to transform matter. Exponential growth in GDP will only occur when humans are out of the loop.

The Origins of Self-Awareness: Why Any Learning Agent Needs a Self-Model

A conscious AI simulates all AI’s that do not simulate themselves.

In the previous post I introduced a simple learning agent. Confined to the circumference of a circle, it can send and receive optical pulses. All pulses it transmits are ultimately received back due to a reflecting boundary that coincides with the circle on which it is confined. It can change the angle at which it emits a pulse and it can count the ticks of a clock until the pulse is received back. I showed how such an agent could learn the geometry interior to the circle without ever moving off the circumference. Suppose now there were other, similar agents on the circle. How does this change the learning protocol?

The first problem faced by each agent is that they can receive pulses of light that did not originate with them. Can they learn to distinguish anomalous pulses from those they sent themselves? Could they learn that there are other agents, like them, on the circle? The ability to distinguish sensations that are correlated with an agent’s own actions from those that are not is the second step in self awareness.

When an agent has learned the relation between the angle at which pulses are emitted and the time taken for their return, it can use this to ‘signal itself’. It can encode a message for its future self by modulating head angle. The ability of an agent to signal itself is the true origin of self awareness and the concept of an enduring, unitary identity. Our obsession with artificial self-images is evidence enough of this. But the signalling could be more immediate. Standing in front of a mirror and watching the results of actions in your motor cortex is an example.

Signalling yourself is a kind of self reference. Douglas Hofstadter wrote very elegantly on this in “I am a strange Loop”. Recurrent neural networks embody loops like this to achieve some surprising things, including memory. Self reference is central to some of the deepest theorems in mathematics, Goedel’s theorem for example. The theory of recursive functions lies at the core of Turing’s idea of computation.

A learning machine encodes the relations between actions and sensations in the physical states of its components. In a machine that can signal its future, these sensations are caused by prior states of the machine itself. But it treats these sensations like all others and learns the correlations that result from this causal link. In other words, the machine learns to simulate its states as if they are states of the external world. It learns to model itself as a part of the external world.

This is not a new idea. Merlau-Ponty, the French philosopher of phenomenology, argued that self-awareness is fundamentally a consequence of our lived bodily experience and our direct engagement with the world. For Merleau-Ponty, self-awareness is inseparable from our perception and interaction with our environment. It is learned in the same way we learn everything else about our world and for the same reason: to enable us to implement effective strategies that will enhance our survival.

A similar idea was recently elaborated by Gemma De les Coves and her collaborators and draws attention to a defining characteristic of self-reference sometimes called the ‘Russell’s paradox’. Let me state this in an AI-centric way:

A conscious AI simulates all AI’s that do not simulate themselves.

Russell’s paradox arises from self-referential definitions—most famously, the “set of all sets that do not contain themselves.” In that classic form, we ask: Is the set itself a member of itself or not?—if it is, then by definition it shouldn’t be; if it isn’t, then by definition it should be.

We need to be more precise about simulation. I will take it to be synonymous with prediction. In my model of a learning machine, when an agent acts on the external world it also acts on its internal learning machine. The leaning machine then tries to generate, or predict, the actual sensation returned from the world to the agent at a later time. Learning consists in making the probability of an erroneous prediction as small as possible.

Let action a result in a predicted sensation ,

$\tilde{s}=f(a)$

The world however produces the actual sensation

$s=\Phi(a)$

Assuming both f and $\Phi$ are invertible functions we have that

$s=\Phi(f^{-1}(\tilde{s})$ ,

and

$\tilde{s}=f(\Phi^{-1}(s))$ ,

It follows that,

$\tilde{s}=s$ iff $f=\Phi$ .

This would imply that the agent is identical to the world, which cannot be right — the agent is a very small subset of the world and all learning machines necessarily make mistakes. The most we can say is that the prediction error is small. Prediction is uncertain.

The uncertainty arises in many ways. Firstly, the world function depends on many more variables than we can control, or even know. Secondly, it may not belong to the same class of mathematical functions the learning machine can learn. Thirdly, the actual sensation the machine records, $s$ , labels a state of the agent, not of the world, as do the actions, $a$ . The states of the world are fundamentally unknown and unknowable. They transcend the agent.

Suppose that the states of the agent are restricted in such a way that they are in one to one correspondence with the positive integers, while the world states are real numbers. There are infinitely many states of the world that are in one to one correspondence with possible states of the agent, but infinitely many more that are not. The world function must be like

$y=\Phi(x)$

where $x,y$ are real numbers. When the input is restricted to integers, $n=1,2,3\ldots$ the output is also an integer. One example is any polynomial function with integer coefficients. There are also polynomials with rational coefficients (not necessarily integers!) that always take integer values at every integer. These are called integer-valued polynomials. The binomial coefficients are an example. Then there are piecewise functions such as any function that is defined to equal an integer for each positive integer, but is arbitrary elsewhere, and periodic functions such as

$\Phi(x)= x \mbox{ mod } n$

More generally, for any sequence of integers, $(a_1,a_2,\ldots a_n)$ there exists a function on the reals such that $a_n=f(n)$ defined by interpolation (such as Lagrange interpolation), but takes arbitrary values outside the positive integers.

If world functions are in this class, there is going to be a problem with inversion. For example, the only polynomials with integer values at all integers and are invertible as real functions are the linear polynomials with integer slope. This is an example of a strictly monotonic (increasing or decreasing) real functions whose restriction to positive integers is injective and which assign only distinct integer values to positive integers. If you want the function to be both elementary and invertible, essentially only these strictly linear forms fit the criteria. Certainly periodic functions do not fit this category.

Restricting the class of world functions accessible to us has the consequence that we can never ensure that $\tilde{s}=s$ for all possible actions. Predictions will necessarily be imperfect. While some actions may result in completely predictable sensations, there are infinitely many actions that do not.

Can an agent learn an approximation to $\Phi(x)$ even if it is not invertible? Yes! Machine learning algorithms—especially neural networks—can approximate arbitrary integer-valued functions, including those that are not invertible. For a function that is constant or stepwise—clearly not invertible—neural networks can still learn to approximate the mapping from inputs to integer outputs. The price that must be paid is a small but non zero probability of making an error. That is to say the error probability to predict $\tilde{s}=s$ cannot be made to go to zero.

In the course of a long sequence of interactions with the world, an agent learns many world functions, including causal functions that enable it to control the world around it. But the agent itself is embedded in the world, so some of the functions it learns refer to its own physical operations.

This is what it means for a learning agent to ‘signal itself’. It learns how to predict its own behaviour, but not perfectly. It will always make mistakes, especially if it has access to restricted kinds of world functions. Such an agent is necessarily surprised by itself. Its ability to predict its future is limited. It never, truly, knows itself. A long life of interactions with the world, including self signalling, endow the agent with an internally stored prediction engine for its own behaviour in a changing world, but it is necessarily imperfect.

Different agents may differ in there ability to signal themselves and learn how to predict there future behaviour. Perhaps fruit flies can’t do as well as humans in this respect. It suggests that levels of consciousness are linked to the ability to learn, and to predict your own future behaviour. This has implications for how we understand free will. It also has implications for building engineered embedded learning machines.

A physical learning agent must always surprise itself because it is part of a world that necessarily transcends its finite and limited internal learning capability. The actual physical embedded thing transcends its own learned self image.

Learning Is Not Memorisation: The Energy Cost of Remembering Everything

There is but one goal for a biological learning machine: survive long enough to reproduce. Learning changes the odds in your favour only if you don’t spend so much time and energy learning you die young. Ultimately, the goals of evolution and thermodynamics are aligned.

A simple learning task is classification. For example, suppose some red berries you encounter are packed with energy, but some kill you. To distinguish the two kinds of berries requires taking careful note of a lot more than simply the colour of the berry. Perhaps the good red berries have clustered skins (raspberries), but the ones with smooth skins will kill you. Perhaps some smooth-skinned red berries are OK but not if they are overripe. To find the safe red berries you need to take account of many more features.

You could simply not eat any red berries, and die of hunger in the winter. Another way to approach this problem; try to remember all the local factors that were present the last time you saw a member of your tribe eat a berry and die. The ‘memory first’ approach requires a lot of experience and a very good memory. There might be a lot to learn and a lot to recall and that may take so much time that … well, you are deselected. The objective in learning is to avoid memorisation. You don’t need to recall every detail in every situation in which you encountered a red berry. You only need to recall the patterns associated with good berries, and the patterns may be relatively easy to recall … if you can find them efficiently.

The kind of memorisation I am talking about is not how learning agents do memory. We tend to think our memories are like records in the old, analogue world, simply a photo or movie reel, that ‘we’ view in the Cartesian theatre. Our memories are not like that. If they were, it is doubtful we could function at all, like Funes the Memorious in the Borges story. The memories of a learning machine are reruns of learned predictive simulations, and every re-run is a new, different experience.

It takes a lot of resources to memorise every instance of a problem and a lot of time to search the data to know what to do in a novel encounter with a similar problem. In general the effort required, in both time and space, grows exponentially with every instance of the problem, even if the output is simply yes or no. For example, suppose you need to answer all possible 20 yes/no questions before you can decide if the outcome is good or bad. There are 2 raised to the power of 20 such inputs and each one could give two possible outcomes. That means to memorise this function you need to store

sets of yes/no values in order to have a complete look-up table. That is a very big number. Yet the key feature you need to learn might simply be that the good output occurs, almost every time, when all the twenty questions have an equal number of yes answers as no questions (even parity). You don’t need to memorise every instance of the problem, you only need to learn a particular feature of the problem that gives a low chance of making a mistake. And that may take a lot less effort to acquire … if you are a learning machine and are faced only with binary valued functions of many binary inputs.

The thermodynamics of this problem are important. If you simply memorise all instances of the problem you need to set aside locations in memory for 2²⁰ bits. Yet if you learn the key feature of the problem , say all you need to know is if the input has even parity , then you can store much less data.

If all inputs are equally, likely then this means the average entropy of what you need to store has decreased dramatically. In a physical machine (like you) entropy reduction like this requires that you pay a thermodynamics price by emitting enough heat into the environment to raise its entropy by as least as much as your learning machine has reduced your internal entropy. This is why all learning necessarily requires heat generation. This is simply a restatement of the Landauer erasure cost, if that means anything to you. Learning offers a huge evolutionary advantage.

Current AI (based on conventional silicon chips) learns well at a huge thermodynamic cost. Is it the minimum cost required? Absolutely not. If it were, no biological learning machine would ever have evolved on Earth. What is the minimum cost? AI engineers are trying hard to figure this out before they go broke paying the power bill. (That is what natural selection means in silicon valley. ) I think the answer is quantum.

Machine Learning or Learning Machine? The Difference Physics Makes

Biology offers abundant evidence that physical systems can learn, that is to say, physical systems can exhibit stable behaviour, conditioned on prior interactions with an external environment, in order to achieve a goal. We are entering an era in which learning machines can be engineered. In which case, what are the physical principles in play?

A learning machine can be instantiated in any physical system and not necessarily digital. Biological learning in brains is not based on algorithms running on digital computers, even if it can be simulated that way. What are the physical principles required for a machine to learn?

A learning machine, like any machine, is an open, dissipative physical system driven far from thermal equilibrium by access to a low entropy source of energy, for example, a battery. I will focus on simple classification in supervised learning. Here the objective is to learn a binary valued function, f(x), of the input data, x, by giving the machine a list of examples (x, f(x)) and adjusting the parameters of the machine through feedback so that the actual outputs are correct almost all the time. Error cannot be removed in a learning machine: it is an inherent feature of all learning. If you never make a mistake, then you never learn anything. if you only make mistakes , then you never learn anything either.

In a learning machine, reducing the error to zero in a finite machine would violate the laws of thermodynamics. The goal is to reduce the error probability, while making efficient use of the available thermodynamic resources.

A machine learning algorithm however is a mathematical procedure for approximating functions running (usually) on a conventional CMOS based von Neumann computer. There are very many machine learning algorithms and the discovery of new ones proceeds at an incredible pace. I want to contrast algorithms run on computers to actual machines that learn by thermodynamic constraints. In many ways this reduces to the question of who or what sets the goal? Who or what sets the error function? In a learning machine the goals are ultimately set by thermodynamics (in an evolutionary setting). In contrast, in ML algorithms, the algorithm designer sets the goal.

I am interested in quantum machines operating at very low temperature (they are cheaper to run), in which case the goal is to learn by exploiting quantum noise. How can quantum noise be harnessed for efficient learning? I will pursue this approach in future posts.