A conscious AI  simulates all AI’s that do not simulate themselves.   

In the previous post I introduced a simple learning agent. Confined to the circumference of a circle, it can send and receive optical pulses.  All pulses it transmits are ultimately received back due to a  reflecting boundary that coincides with the circle on which it is confined.  It can change the angle at which it emits a pulse and it can count the ticks of a clock until the pulse is received back. I showed how such an agent could learn the geometry interior to the circle without ever moving off the circumference. Suppose now there were other, similar agents on the circle. How does this change the learning protocol?

The first problem faced by each agent is that they can receive pulses of light that did not originate with them. Can they learn to distinguish anomalous pulses from those they sent themselves? Could they learn that there are other agents, like them, on the circle? The ability to distinguish sensations that are correlated with an agent’s own actions from those that are not is the second step in self awareness. 

When an agent has learned the relation between the angle at which pulses are emitted and the time taken for their return, it can use this to ‘signal itself’. It can encode a message for its future self by modulating head angle. The ability of an agent to  signal itself is the true origin of self awareness and the concept of an enduring, unitary identity. Our obsession with artificial self-images is evidence enough of this. But the signalling could be more immediate. Standing in front of a mirror and watching the results of actions in your motor cortex is an example. 

Signalling yourself is a kind of self reference.  Douglas Hofstadter wrote very elegantly on this in “I am a strange Loop”. Recurrent neural networks embody loops like this to achieve some surprising things, including memory. Self reference is central to some of the deepest theorems in mathematics, Goedel’s theorem for example. The theory of recursive functions lies at the core of Turing’s idea of computation. 

A learning machine encodes the relations between actions and sensations in the physical states of its components. In a machine that can signal its future, these sensations are caused by prior states of the machine itself. But it treats these sensations like all others and learns the correlations that result from this causal link.  In other words, the machine learns to simulate its states as if they are states of the external world. It learns to model itself as a part of the external world.

This is not a new idea. Merlau-Ponty, the French philosopher of phenomenology, argued that self-awareness is fundamentally a consequence of our lived bodily experience and our direct engagement with the world. For Merleau-Ponty, self-awareness is inseparable from our perception and interaction with our environment. It is learned in the same way we learn everything else about our world and for the same reason: to enable us to implement effective strategies that will enhance our survival. 

A similar idea was recently elaborated by Gemma De les Coves and her collaborators and draws attention to a defining characteristic of self-reference sometimes called the ‘Russell’s paradox’. Let me state this in an AI-centric way:

A conscious AI  simulates all AI’s that do not simulate themselves.   

Russell’s paradox arises from self-referential definitions—most famously, the “set of all sets that do not contain themselves.” In that classic form, we ask: Is the set itself a member of itself or not?—if it is, then by definition it shouldn’t be; if it isn’t, then by definition it should be.

We need to be more precise about simulation. I will take it to be synonymous with prediction. In my model of a learning machine, when an agent acts on the external world it also acts on its  internal learning machine. The leaning machine then tries to generate, or predict, the actual sensation returned from the world to the agent at a later time. Learning consists in making the probability of an erroneous prediction as small as possible. 

Let action a result in a predicted sensation ,

The world however produces the actual sensation 

Assuming both f and are invertible functions we have that 

,

and 

,

It follows that, 

  iff  .

This would imply that the agent is identical to the world, which cannot be right — the agent is a very small subset of the world and all learning machines necessarily make mistakes. The most we can say is that the prediction error is small. Prediction is uncertain.

The uncertainty arises in many ways. Firstly, the world function depends on many more variables than we can control, or even know. Secondly, it may not belong to the same class of mathematical functions the learning machine can learn.  Thirdly, the actual sensation the machine records, ,  labels a state of the agent, not  of the world, as do the actions, . The states of the world are fundamentally unknown and unknowable. They transcend the agent.  

Suppose that the states of the agent are restricted in such a way that they are in one to one correspondence with the positive integers, while the world states are real numbers. There are infinitely many states of the world that are in one to one correspondence with possible states of the agent, but infinitely many more that are not. The world function must be like  

where are  real numbers. When the input is restricted to integers, the output is also an integer. One example is any polynomial function with integer coefficients. There are also polynomials with rational coefficients (not necessarily integers!) that always take integer values at every integer. These are called integer-valued polynomials.  The binomial coefficients are an example. Then there are piecewise functions such as any function that is defined to equal an integer for each positive integer, but is arbitrary elsewhere,  and periodic functions such as 

More generally, for any sequence of integers, there exists a function on the reals such that defined by interpolation (such as Lagrange interpolation), but takes arbitrary values outside the positive integers. 

If world functions are in this class, there is going to be a problem with inversion. For example, the  only polynomials with integer values at all integers and are invertible as real functions are the linear polynomials with integer slope. This is an example of a strictly monotonic (increasing or decreasing) real functions whose restriction to positive integers is injective and which assign only distinct integer values to positive integers. If you want the function to be both elementary and invertible, essentially only these strictly linear forms fit the criteria. Certainly periodic functions do not fit this category. 

Restricting the class of world  functions accessible to us has the consequence that we can never ensure that for all possible actions. Predictions will necessarily be imperfect.  While some actions may result in completely predictable sensations, there are infinitely many actions that do not. 

Can an agent learn an approximation to even if it is not invertible? Yes! Machine learning algorithms—especially neural networks—can approximate arbitrary integer-valued functions, including those that are not invertible. For a function that is constant or stepwise—clearly not invertible—neural networks can still learn to approximate the mapping from inputs to integer outputs. The price that must be paid is a small but non zero probability of making an error.  That is to say the error probability to predict  cannot be made to go to zero. 

 In the course of a long sequence of interactions with the world, an agent learns many world functions, including causal functions that enable it to control the world around it. But the agent itself is embedded in the world, so some of the functions it learns refer to its own physical operations.  

This is what it means for a learning agent to ‘signal itself’. It learns how to predict its own behaviour, but not perfectly. It will always make mistakes, especially if it has access to restricted kinds of  world functions. Such an agent is necessarily surprised by itself. Its ability to predict its future is limited. It never, truly, knows itself.  A long life of interactions with the world, including self signalling, endow the agent with an internally stored prediction engine for its own behaviour in a changing world, but it is necessarily imperfect.  

Different agents may differ in there ability to signal themselves and learn how to predict there future behaviour. Perhaps fruit flies can’t do as well as humans in this respect. It suggests that levels of consciousness are linked to the ability to learn, and to predict your own future behaviour. This has implications for how we understand free will. It also has implications for building engineered embedded learning machines. 

A physical learning agent must always surprise itself because it is part of a world that necessarily transcends its finite and limited internal learning capability. The actual physical embedded thing transcends its own learned self image.