a not-so-short introduction to biologically plausible artificial neural networks

including a neural network implemented in javascript

Author: R. Zboray

University of Amsterdam - Department of Physics

---

---

Abstract

We construct a virtual ecosystem consisting of Agents and rules that define an action-result space: the environment. The agents' behaviour is guided by neural networks, which are subject to the constraint that they must conform to biophysically realistic, or at least plausible, rules. We use multilayer feed-forward recurrent neural networks with inhibitory and excitory connections. We use Hebbian-like learning rules for the Agents, in combination with genetic algorithms that determine the initial topology of the neural networks.

Within this ecosystem we perform several experiments with which we determine how different topological developments of the neural networks affect the evolutionary success of the agents. We look at the effects of death and generation of neurons and what happens to the agents with changes in the environment.

For a complex ecosystem we will find that the following choices have a strong positive influence on the agents' success: Dividing the global neural network (metanet) into several specialised subnetworks; Having a subnet using genetic learning algorithms determining the success-factor $\rho$ for the rest of the neural net; A mixture of exitory and inhibitory neurons; step-memory in neurons; a mechanism of attention; the pruning of connections and neurons; complex emotions; a mechanism of imitation of other agents; social behaviour; a mechanism of communication; a mechanism of identification; short-term memory; self-awareness; a mechanism of prediction, thinking and planning;

The strongest influences stem in large part from the step memory in neurons, giving us $\nabla \rho$ improvement during learning; social behaviour of Agents allowing their neural networks to escape entrapment in a local maximum that is not the global maximum of $\rho$; the combination of self-awareness and social behaviour to allow rapid traversal of the entire input-output pattern space $\Upsilon$ and replicating succesfull model patterns. Naturally these factors are only beneficial if the environment is of such complexity that they are usefull. Otherwise a negative selection pressure exists against them.

---

Chapter 2 - Biological Neural Networks

The brain, or maybe rather the central nervous system, is largely a complex collection of various specialised glia and neuronal cells, intricately yet profusely interconnected. The connections between the neurons form networks in such a way that these can perform specialised tasks. These different networks again are interconnected so that a complete system is formed to allow the necessary information processing necessary for the successfull reproductive behaviour that natural selection constraints the larger animals with. To begin to understand its working we shall start with its basic building block: the neuron.

The neuron

Although the morphology of various types of neurons differs in some respects, they Almost all neurons contain four functionally distinct element: the cell body or soma, the dendrites, the axon, and the axon terminals.

The soma

The soma contains the cell nucleus and most of the other organelles, so that this is where most of the proteins which the neuron uses are synthesised. These can then be transported to wherever they are needed within the neuron, for example within the axon. The soma not only functions as a chemical factory, but also serves as the neuron's waste treatment facility, where damaged proteins and other elements are broken down to be reused or discarded.

The axon

From the soma springs a single axon, a typically long, widely branching fiber which is specialized to conduct signals away from the neuron (eferrent) toward the axon terminus, where the axon connects with other neurons.

The axon develops through growth: the tip of the axon, the growth cone, will move forward by extending the microtubuli skeleton and the membrane. The direction of growth is usually along some chemical gradient. Therefore a source of these signalling chemicals -for example another neuron- can repel or attract a developing axon. In this way growing axons may move towards specific target areas, so that for example a neuron from the inner ear may seek contact with neurons within the temporal cortex [1].

Usually the axon connects with multiple target neurons. This it achieves through branching out during growth. The branches can continue developing, each growing and branching again, untill they find targets in the dendrites or soma of some other neuron. Because they need to seek out targets which may be a long distance from their soma, axons tend to be long, and usually are a lot longer than the dendrites. Within our own body axons can reach a length of up to a meter (connecting spine and toes for example), which is quite impressive when one considers that the average diameter of a body cell is about $10^{-6}m$, with variations according to species and tissue type [2].

The dendrites

Most neurons have multiple dendrites emanating from the soma, quickly branching out. They are very similar to the axon, except that they are specialised to receive signals from the axons of other neurons and conduct these towards (aferrent) the soma.

Because they need to receive these signals from other neurons, the dendrites have specific structures embedded within their membrane: the dendritic spines. At these loci another neuron's axon terminus may attach itself to the dendrite and form a synaps. It is within the synapses that signals are transmitted from the eferrent axon to the aferrent dendrite.

Because the dendrites develop away from the soma, and because they can strongly branch out, they quickly and cheaply increase the neuron's surface area, so that there is a lot of room where the axon termini of other neurons can attach themselves. Not only this, but they effectively provide an enormous increase in the neuron's functional volume, where an axon may detect -and connect with- the neuron, without having to actually build or maintain an entire cell of the same volume. Thus not only is there a metabolic benefit, but also do the small soma and dendrites not hinder the development of dendrites and axons of other neurons, which a large-sized soma without dendrites certainly would. In the end this results in neurons which connect to -and are connected by- about $10^3$ other neurons (giving an approximate total of $10^{14}$ connections within our brain).

The cell membrane

The membrane which constitutes the surface of the cell is a normal double-lipid cell membrane, which holds its shape through hydrostatic pressure and a skeleton of microtubuli: small interconnected protein-based tubes providing structural integrity. The membrane has some special properties which allow it to conduct the signals of the neurons. These properties stem from the various protein structures that riddle the membrane, poking through it, allowing communication and transport of chemicals between the inside of the cell and the medium in which it resides.

Electrochemical activity

There are three important elements that form the basis on which the signalling mechanism is built. Firstly, some of the protein structures in the cell membrane act as portals for certain ions. These portals can be opened or closed to allow or deny the transportation of these substances between the inside and the outside of the cell. This is not unique to neurons, but the specific use of these portal structures is, as they form an important part of the signalling mechanism. This signalling mechanism depends mostly on the way the portals handle certain ion such as $Na^{+}$ and $K^{+}$, and on the influence of the incoming signals of other neurons.

Now, what you would expect is that through the membrane gates the ion densities within the cell and in the outside medium even out, both because of statistical mechanics and because of the added influence of the electric field¹. And indeed, this is what happens. However, and this is the second piece of the puzzle, there are other protein structures present in the cell membrane which actively, through the use of energy in the form of ATP², move ions between the inside and the outside environment of the cell. These structures are the ion pumps.

This active transport of ions creates of strengthens an ion gradient. Because the ions are electrically charged this ion gradient creates an electric field between the inside and outside environments of the cell. This results in an electric field potential $(V_m)$ through the cell membrane³. The exact value of $V_m$ depends on species and cell type, but is usually between $-50mV$ and $-90mV$, with the potential of the outside medium chosen to equal $0$.

While the ion pumps actively move ions through the membrane, ions leak back through some open ion gates. The stronger the gradient (and the membrane potential) the more ions leak back, untill both ion streams balance each other and the ion gradient, the electric field and $V_m$ remain constant. This point we call the resting potential, because it is the potential the neuron maintains without outside stimuli.

This finally brings us to the third piece of the puzzle, as the description above is only accurate for a neuron without outside stimuli. What kind of stimuli are we talking about? We already know the axon of one neuron can contact another neuron and the two neurons form a synaps. If the axon relays a signal of the presynaptic, eferrent neuron, it releases chemicals called neurotransmitters which attach to a third kind of protein structure in the membrane of the postsynaptic neuron's half of the synaps. These protein structures are connected to ion gates, and if combined with neurotransmitters they can open these gates. These protein structures can therefore be seen as a lock on the ion gates, where the neurotransmitters function as keys.

Now what happens if a signal arrives in the synaps, triggering the release of neurotransmitters which can bind to the postsynaptic lock structures? Well, the lock structures cause more ion gates to open, which allows a greater ion-flux. This in greater or lesser degree depolarizes the neuron ($V_m$ goes to $0$). This change is not localised, but almost instantaneously spreads within the soma, as variations in the ion density are quickly resolved (as we earlier mentioned) through statistical mechanics and electric repulsion.

The action potential

The electric perturbations generated in the dendrites or soma spread to the location of the axon hillock. When the neuron is at rest only a few of the axon's ion portals are open. The percentage of open portals depends on the membrane potential $V_m$. If $V_m$ exceeds a certain threshold value more portals will open, which allows $Na^+$ ions to rush into the neuron, further driving up $V_m$, which opens more portals, letting in more $Na^+$ and in a short time causing $V_m$ to actually overshoot $0$.

As $V_m$ overshoots $0$ the $Na^+$ portals start closing, while $K^+$ portals open up, causing $K^+$ ions to rush out of the neuron. In a short span of time the two ion streams balance each other and the membrane potential reaches its maximum value $V_{\mbox{max}}$. As the $N^+$ flux drops off and the $K^+$ flux increases the neuron polarizes again, even hyperpolarising (pushing $V_m$ down below $V_{\mbox{rest}}$. Eventually the continued action of the ion pumps restores the normal ion distribution and therefore the rest potential $V_m$. The entire sequence of events, from $V$ exceeding the threshold untill restoration of $V_m$ is called the action potential.

If a neuron develops an action potential on its axon hillock it takes some time for the entire activity cycle to run. It is however not limited to this one place on the membrane. As the ion gates open and close and the ion gradients, and with it the potential, at the generation spot change, the potential at the gates neighbouring these also changes (albeit with a slight delay), so that those neighbouring ion gates also start switching. And as this further changes their local potential, their neighbours also experience this changed potential and start switching.

In this manner the activity of the action potential moves over the entire axon, untill it finally extinguishes at the axon terminus. However before this termination it has encountered many of the loci where the axon connects with other neurons, forming synapses.

If the action potential reaches a synaps, the increase in membrane potential of the axon causes the axon to release neurotransmitters, which are molecules that are stored in vesicles in the axon. These neurotransmitters cross the thin gap between axon and dendrite (the synaptic cleft) and are picked up by neuroreceivers in the dendrite's membrane. These then open up ion gates, increasing the membrane potential of the postsynaptic neuron. If enough neurotransmitters are released in a short enough timeframe, the postsynaptic membrane potential at the synaps is increased so strongly that a new action potential is generated within the postsynaptic neuron, recreating the process that just occured in the presynaptic neuron. The remaining neurotransmitter is quickly recycled.

But we already know most of this from what we have written before. However, there are some subtleties we skipped earlier because we wanted to concentrate on the generation of the action potential. For example, there are several types of neurotransmitter, which bind to different receptors and may have different effects.

Each neuron's axon in general releases a single type of neurotransmitter. If this axon were to form a synaps with a neuron that does not respond to this neurotransmitter no signals can be transmitted. This usually results in the eventual breakdown of the synaps. Thus, by forming a sort of error-correction mechanism the various neurotransmitters and their associated receptors help in the formation of the desired functional neural networks.

Furthermore, the exact electrochemical effects may differ between neurotransmitters. While we usually think of neurotransmitters exciting a neuron by depolarising it untill it fires an action potential, it is also possible for the neurotransmitters to hyperpolarise a neuron, thus inhibiting its ability to fire. It is of course more correct to speak of the receptors hyperpolarising the neuron, but since neurotransmitter and receptor are intimately linked, and neurotransmitters are usually easier to discern, this kind of vocabulary creeps in.

Besides these normal excitory and inhibitory connections, it is also possible for chemicals to alter the signal transmission within the synapses. They may for example block the receptor proteins, lowering the effects of the neurotransmitters; They may block the reuptake mechanism, so that the neurotransmitter remains longer within the synaptic cleft, bonding to more receptors, strengthening its effects. This is for example how cocaine works. In fact, almost all psychoactive drugs work by influencing the signalling behaviour of various neuronal connections in the brain. Anyway, this is a specific example of a more general type of moderation; that the possibility exists to influence the signalling mechanism, by blocking, weakening, strengthening or replacing signals between neurons.

Neuronal networks

We now know much of the basic functional mechanisms of biological neural networks as realised through the neuron. We have a reasonable idea of how a neuron may generate a signal in the form of an action potential, and how this can move over the neuron's axon and through other neurons. What we do not yet know however is how these interconnecting neurons can form a neural network that is actually usefull. How does a tangle of connected neurons form a functional whole that benefits the organism it belongs to?

Let's backtrack a little bit. Consider what are neural networks actually for? If we carefully observe organisms, for example in a PET-scan, we will notice that biological neural networks gather information from the senses and control (many of) the physiological actions of the organism they belong to. Evolutionary selective pressure causes these two functions to combine in such a way as to maximise the organism's evolutionary success. In other words, a neural network collects information, and uses that information to instigate those actions which produce the most beneficial result. This may seem trivial, but it is the basic principle on which we build this work, and I will repeat it from time to time.

The question that immediately arises is how does a neural network collect information and how does it use this information to favour certain actions over others? After all, we have not seen anything yet but the neurons and their ability to send action potentials to each other. The answer is that the signals embody the information which is encoded through the specific way the signals flow through the network. This signal flow depends on the properties and shape of the network as defined through the connections between the neurons and the type of neurons: the topology of the network.

Neuronal network topology

What a neuronal network does is take information from sensory cells and with this information compute and implement a course of action. In effect, it maps information from the Input space (the senses) to the Output space (muscles, glands etc).

This mapping can be considered in two forms: physical and logical. The physical map is the actual mesh which is formed by all the neuronal cells and their connections. The logical map is the pathway information takes through the physical map. While these two are obviously closely related, they need not be identical. This is because some connections are weak, so that information is not always transmitted there, or some piece of information may be more important than another, causing the latter to be blocked somewhere through inhibition or lack of amplification. Some neuronal cells may only fire in certain intervals, or they may be in the refractionary period because of previous activity and unable to fire.

It would seem tempting to draw an analogy between this and how computer software and hardware function together, but that analogy would be flawed. The similarities are only superficial, because neuronal networks as we find them generally cannot be described as Turing Universal Computing Machines. This is a bit of a tricky point, not only if you do not know what a Universal Computing Machine is, but maybe even more if you do. Suffice it to say that most neuronal networks are in practice quite inflexible, limited to a specific set of tasks, even when capable of learning⁴.

Let us look at an example of a biological neural network that shows this behaviour: the retina of the eye, which forms a part of the visual subsystem of the nervous system. The visual subsystem takes cues from the environment in the form of light quanta or photons which are picked up by photoreceptors (rod and cone cells) in the retina. This raw data in itself is useless to the organism. It needs to process it to extract relevant information, such as the presence of food and other opportunities, or dangers such as predators. There are several algorithms which are used to extract this relevant information, some of which are even implemented in the retina itself⁵. This, I think, is not only interesting for the reasons we are discussing here, but also for the influence this has on the outside world: visual elements such as the stripes of zebra or tigers exist not because they are pretty, but because they exploit flaws in the algorithms used to extract information from the visual data, which gives the animal in question an evolutionary advantage.

The retina contains photoreceptors (the so-called rod and cone cells) which are stimulated by light, and transmit the information the eye receives through the optic nerve into the brain. What is less known is that if you look at all of the information these photoreceptors pick up, only about one-tenth that amount of information is actually transmitted to the brain [3]. Now how is this possible?

The rods and cones are not directly linked to the optic nerve. Their signals first pass through a few layers of neuronal cells which do a lot of preprocessing. From the raw data of the signals they extract features such as borders, contours, and (the direction of) motion. This stems from the combination of signals from several adjacent photoreceptors in a single ganglion cell, the photoreceptors all being located in that ganglion cell's receptive field. Typically a receptive field is organised in a center disk and a peripheral concentric ring. The activity of the ganglion cell is a function of the signals that are received in these two areas, so that for example some "on center" ganglion cells become very active when the center disk is stimulated and the concentric ring is not while the opposite is true for some "off center" ganglions. Through the overlapping of receptive fields, different types of ganglion cells and the combination of their activities this results in the features such as edge detection that allow

[4]

---

---

---

The retina achieves its enormous and near-instantaneous data-compression through a clever yet simple massively-parallelised algorithm where the information from a single rod or cone is combined with that of those around it, to detect features such as edges, gradients, movement and more. It are these features which are actually transmitted to the brain, which consequently reconstructs the environmental information these features represent. For example, if you look at figure 2a, you will see a simple solid square. However, the information that is actually transmitted from your eye to your brain, is the colour of the square, and its edges. Despite of its efficiency, or rather because of its efficiency, the algorithm has flaws, as you probably can discern in figure 5b.

The brain is so well adapted to restoring the environmental information we need for evolutionary success from these incomplete cues, that we almost never notice the blind spot within our vision which is caused by the optic nerve which gathers the information passing through the retina⁶. Here, take a look. Just close your right eye, focus on the x, keep your eye very close to the screen and slowly move away. You should notice first the b and after that the a fade away into the background:

a

b

x

Now the retina is just a nice example of a relatively small neural net, because you can easily show a bit of how it works, and because a lot is known about it as it is relatively easy to study and dissect...

There are many specialised systems such as the retina in our nervous system, each specialised for its own task(s). The various sytems that make up our complete nervous system, and there are a lot of them, manage to work together to provide a functional individual. Some of the systems seem to form meticulously along some programmed line, while others are dependent on a process of learning. It is however in the interplay between systems that the greatest achievements arise.

Development

Learning

The central nervous system

Have you ever heard the expression: the whole is greater than the sum of its parts? Of course you have. Well, that is what the nervous system is. All the little networks with their intricate methods of solving their particular problems; it is all meaningless if they are not working together to form an individual who can reproduce. But if we look at an individual in this (admittedly rather restricted) way, we must consider that an individual cannot be seen as distinct from the environment, as it is the environment that the process of evolution has shaped the individual to survive, to conquer and to use. Just as it is a part of the environment of everything else, there to be survived or to yield to.

While we could spend quite some time going over what we know about all these magnificent systems within the nervous system, that is not what we are particularly interested in. The greater functionalities are obvious: to gather information about the environment, to analyse it, and to implement some course of action. The first of these comes from the senses, the last (in humans) from the small brain. What is interesting to us is how all the different parts manage to work together. How do they learn and where do their abilities come from?

Let us start with a hypothesis: It is possible for organisms to have some sort of instruction set which determines the properties of some neuronal network within that organism, and this instruction set can be passed on to the organisms progeny through some hereditary mechanism (for example by being encoded within the genes). If this hypothesis is incorrect, you would expect closely related organisms to have very different topologies in their neuronal networks. What we find however, is that the hypothesis is not incorrect: there are many species where the neuronal networks between individuals are practically identical. However, we must acknowledge that these are mostly those species which have very small neuronal networks. We do not find the same level of identicality in creatures with larger nervous systems, such as man. And we can's find that: the numbers do simply not add up. Look at our own species. Our brains consists of approximately $10^{10} - 10^{11}$ neurons, which average $10^3-10^4$ connections with other neurons for a total of $10^{13}-10^{15}$ connections. Meanwhile the number of basepairs in our genetic material is about $3 \cdot 10^9$. While it is not impossible some terribly-clever data-compression scheme is used, it is simply far more likely that not every aspect of our brains is directly encoded into our genetic material.

Does this mean our brains develop randomly? Off course not. They are far too structured and similar for that. To our good fortune I may add, as otherwise something like brain surgery would be an expensive form of russian roulette. The brain develops according to certain rules, or algorithms, which are simple in nature, but can produce a dazzling complexity. The best-known type of such algorithms are fractals. With a very limited amount of instructions, which are repeatedly applied, with some feedback, can produce stunning forms of geometry which have the interesting property that a part is similar to the whole. For a particularly nice example, take a look at John Walker's webpage on fractal food.

Mechanisms similar to fractals, such as Lindenmayer systems, share this property of simplicity in algorithm resulting in seemingly complex structures. And indeed, if you take a good look at these L-systems, which were specifically designed to describe patterns in nature, you may note some similarity not only between them and the branches of branches of trees or the roots of plants, but also the branching pattern of a neuron's extremities.

Now, while a neuron may show some fractal-like behaviour, this does not mean their development is fully controlled by it. The developing axon for example in its growth often follows a chemical gradient, which it can use both to move away from the originating soma (which it would be useless to attach to), and to find a desired target neuron.

---

Chapter 3 - Static Artificial Neural Networks

INTRODUCTION GOES HERE

Neural nodes

The behaviour of neural nodes is easy to quantify if we keep in mind that a neural node becomes active if its potential exceeds the firing threshold:

The connection with the neuronal cell is simple, in that $ \chi_{j} $ represents the electrochemical activity (the action potentials), $h_j$ represents the transmembrane potential $V_M$, and $\theta_j$ represents the firing threshold. The variable time is, naturally, represented by $t$.

If one wants to calculate this activity $\chi_{j}$ very accurately, one must consider the location where $h_j$ is calculated (the axon hillock), and compute the exact function of $h_{j}(t)$ as dependant on incoming signals $\sigma$ over time, the exact morphology of the neuron and a function $g(t)$ that represents the restorative activity of the ion pumps, so that we would have to work with something like this:

Now, this is obviously a bit of a problem. Given that every neuron has a unique shape, and that even if this shape is known calculating the integral over it would be a formidable task, the amount of computing that would be required for every single individual neuron makes this approach completely unfeasable. Luckily we can simplify matters while losing little of practical value. Let us start by assuming that the signals spread very fast within the neuron to the axon hillock.

And let us remember that the potential $h$ denotes a relative value. Even the biological value of the resting potential $h_0$ of around $-70mV$ only has that value relative to the outside medium. We are perfectly free to redefine $h$, and therefore $h_0$ and $\theta$ if we want, which we do in such a way that $h_0=0$

It might also help to remember that a neuron returns to the resting potential if it fires, so that we can reduce the time integral:

While this is certainly easier to work with, it is still far from easily digestible, so there is no getting around it; We are going to do something really sneaky, and lay aside the consequences until later. So, let us begin: While we already consider only the time domain since the last time the neuron fired an action potential, we are going to expand on this and assume that all the incoming signals are synchronised. We can now move $g$ out of the integral and replace the integral with a simple summation:

But wait! After a neuron fires there is a period of time during which it cannot fire again (the refractory period). Because of the demand that the incoming signals are synchronised, the activity of the presynaptic neurons must also be synchronised. And because of regression, all neurons within a layer must have synchronised activities. That means that when a neuron receives some signals at $t_n$, there will not be any new signals to receive untill the refractory period has passed.

After the moment of signalling, the maximum value of $h_j$ is $h_j = \theta_j$. If the refractory period is long, and $g$ is strong, the neuron might always reach the resting potential before the next activity round. This is for example true if $g = - \frac {t \theta}{t_{\mbox{refrac}}}$, and this is exactly what we choose $g$ to be.

From this follows that if we only look at those points in time when signals arrive, we can ignore the influence of $g$, and find:

But now we have to start wondering: what exactly is $\sigma$? Since it represents the influence of the incoming signals, it is obvious we need to have it depend on the activity of the presynaptic neurons ($\chi_i$).

We also know that each synaptic connection has a certain strength, which we denote with $w_{ji}$, which of course only matters if the presynaptic neuron is active, so we find:

But let us keep in mind that the signals do not directly influence each other -every signal has its own synaps- but it is the influence on neuron-wide factors that consitutes their involvement with each other. Thus we can take $f$ out of the summation, so that:

Or, if we are feeling lazy, we can use Einstein notation:

However, as this notation is not that widely known, we shall not use it in the rest of this thesis. I would like if at all possible high-schoolers -with mathematics in their curriculum- to be able to understand most of what I write, even if they (quite wisely) decided to skip these formulae.

Now, all we are left to do, is find some plausible function $f$, and our work creating a neural node that models the neuronal cell is complete. Let us examine the behaviour of $h_j$ in a little more detail. If $h_j$ exceeds the threshold value $\theta_j$ an action potential develops. Because of the synchronisation we can reduce the complex action potential cycle to:

This peaking of $h_j$ is something we have ignored up till now, and to be honest, we will in practice largely ignore it again. After all, there is no more information than is already included in $\chi_j$. No, the real question is whether or not $f$ is a linear function.

Let us think it through. Suppose we were to have some biological neuron $j$, with a resting potential $V_\mbox{rest} = -100mV$ and a threshold $\theta_j = -50mV$. The membrane potential $V_m$ depends directly on the electric field, which depends on the difference in concentration of ions. If the ion concentration difference is halved, than so must be both the electric field and the gradient pressure. You would therefore expect $f$ to be a nonlinear function. However, let us not forget that it is possible that the stochastic voltage control of the ion gates means that an increased opening of gates offsets the smaller ion flux per gate. And there are -within limits- some signs that this is actually true [5] [6].

We choose to approach $f$ as a linear function, with boundary condition $f(0)=0$. The upper boundary condition is already implicitly defined in (reference to formula). This way we can get rid of $f$ by incorporating it into $w_{ji}$. This we do by redefining $w_{ji}$, applying some scale factor, which makes no practical difference for now, since we have not yet defined the value of $w_{ji}$. So we find:

If we combine this with (reference to formula) and the Heaviside step function, we get:

What amazing luck! Starting with a headache-inducing integral, we end up with something that is actually easy to calculate! And we have even divided the time $t$ up into discrete steps. Admittedly, we had to employ some chicanery to get there, giving up some not-all-that-inconsequential biophysical properties. Consequently, this relationship only holds in very specific circumstances. But still, at least it is something we can actually work with. And let us be frank, since it is obvious anyway, that is what we set out to do.

The reason we can afford to do all this is simply that we are not all that interested in the behaviour of the neural node an sich. We are interested in the behaviour of a network of nodes. This behaviour obviously has its basis in the propterties of the nodes, but is certainly not completely determined by this, as the most interesting elements of the network stem from its ability to adapt. This it does by changing the connections between the nodes.

I fully admit it would be preferable to fully simulate the behaviour of neuronal cells, but aside from the fact that this would be an impressive feat in and of itself, to computer-simulate even a single neuronal cell would gobble up most of the CPU-power available to us. Since our goal is not to create good simulations of neuronal cells, but to create biophysically plausible artificial neural networks, we simply have to make the choices we did. However, because of these choices we are limited in the kind of tasks our networks will be able to efficiently execute. For example, to catch a thrown ball one has to be able to quickly make a correct assessment of its velocity. For that small timing differences between action impulses would be a tremendous benefit. The work we will do here will simply not result in an artificial mind that is very good at playing tennis.

Still, one may ask: why work with computer simulations at all? It is possible to create clockless neural networks in hardware, or even using actual neuronal cells. However, these physical implementations have a severe problem that does (supposedly, if one is carefull) not occur in computer software: they are not exactly reproducible. Minute differences, such as in timing as mentioned above, may result in functionally different systems. Moreover, pure software is at this moment simply a lot more flexible, even if there is a severe reduction in processing speed. In the end, doing simulations in computer software is the best choice for us, but it is important to know it is not the only, nor always the best choice. [7] [8]

A simple artificial neural network

The behaviour of the neural nodes an sich is not very interesting to us, as we are not trying to accurately simulate biological neuronal cells (which would be an Herculean task in itself). The behaviour that we find really interesting arises because the nodes combine together to form neural networks.

In a biological network the neuronal cells connect to each other at the synapses, which are formed where axons and dendrites meet. This we model by stating that there is a connection between a presynaptic neuron $i$ and a postsynaptic neuron $j$, and there is a weight $w_{ji}$ associated to this connection between $i$ and $j$ that represents the strength of the synaptic signal transmission. The larger $w_{ji}$, the greater the chance that an active presynaptic node $i$ causes a postsynaptic node $j$ to also become active. This can be visually represented as in Figure unknown

This may seem trivial if you have read all the preceding parts; I work under the assumption some of the readers may skip a part or two, which forces me to be thorough. Please forgive me whenever this leads to repetitiveness.

Suppose we do not have just two nodes. Suppose we have a whole bunch of them, for example as in Figure unknown. It is quite difficult to see what exactly is going on here. Luckily, we have a mechanism present that allows us to impose some order: the synchronisation between nodes. Because of this, and the refractory period, we can surmise that the nodes that fire simultaneously can be grouped together, and that the nodes within one group have no functional interconnections.

As we shall keep things simple for now, we shall add an extra restriction: all the nodes in any one group can only send signals to the nodes in one single other group. Thus we have an automatic division of the nodes into $M$ groups, each of which is ordered in order of signal transmission, so that the nodes in $M_l$ send their signals to those in $M_{l+1}$. Because there are no connections within a group, it is possible to plot the groups as Layers of nodes, which makes a visual analysis of the network's topology simpler⁷. The ordering of the nodes within a layer can be freely chosen, as long as one sticks with that choice.

There are only two obvious ground-state topologies possible for this network: One open, where the "line" of layers has a definite beginning and an end, and one closed, where the layers are organised in a circle. Now if you could somehow introduce an activity pattern somewhere in the closed topology, it is quite possible -depending on the connections $w_{ji}$ between the nodes- for the activity to keep moving along the circular path without ever extinguishing. That creates interesting possibilities that unfortunately we shall right now not discuss any further, although I will get back to that later. For now let us concentrate on the open topology.

Take a look again at figure 8a. The presynaptic node $i$ is active, but what makes it so? After all, there seem to be no other nodes connecting to it that would be able to excite it. Even if there were, without infinite regression there must be some prime mover somewhere that causes the nodes to become active. Well, not surprisingly, there is just such a first cause in the neural network. This task is performed by the senses, such as the eyes, ears and all other systems that take information about the environment (including the body), translate this information into an activity pattern of the sensory nodes. Therefore we can say the sensory nodes belong to the first layer $L_1$ in the network, although it is common to call this the Input Layer, and we shall follow this habit. The activity pattern of all the sensory nodes is therefore called the Input Pattern $\Upsilon$.

Similarly, for an open network consisting of $M$ layers, the last layer $L_m$, in which the signals terminate, is called the Output Layer, and its activity pattern the Outupt Pattern $\Omega$. Like the Input layer, which consists of sensory nodes that take information from the environment, the Output layer consists (mostly) of motor nodes, which cause the agent to perform actions within the environment. Thus, a whole network forms an Input-Output construct where information from the environment causes the Agent to act in some specific manner: a classic stimulus-response system.⁸

We have talked about activity patterns, and you can see them visually represented in various figures. It will be obvious that it is not always best practice to make a new drawing for every activity pattern we talk about. Let's see what we can do with some activity pattern $\zeta_l$ of a layer $l$ consisting of $n_l=5$ nodes. If the first two nodes are active, while the rest are inactive this leads to an activity pattern $\zeta_l = 11000$. Similarly, if we look at the neural network in figure 9a we can see $\zeta_1 = \Upsilon = 01100$. The connections between the sensory nodes in the Input layer and the next layer are such that this $\Upsilon$ results in an Output pattern $\Omega = 00011$.

What is important to understand is that different Input patterns do not necessarily have to lead to different Output patterns. For example, consider the simple case of a network with two different Input patterns, $\Upsilon_a = 11$, $\Upsilon_b = 10$. If the second node is not (or only very weakly) connected to another node, it does not contribute to the output of the network, so that $\Omega_a = \Omega_b$. Since the flow of activity introduces an asymmetry in the network, it is trivial to see that no input pattern can generate more than one ouput pattern:

There is still one little thing we better fix before continuing further: While it may be an improvement over graphically displaying everything, working with long lists of numbers is not all that appealing either. Especially if the number of nodes $n_l$ grows very large. Luckily these patterns can be considered as an expression in binary⁹. Thus, the activity patterns of figure 9a we mentioned above, can also be expressed as $\zeta_1 = \Upsilon = 12$ and $\zeta_2 = \Omega = 3$.

You will not have to worry about the specifics, as long as you know we can translate a binary activity pattern into a number. Where we actively concern ourselves with what $\zeta$ actually is, we use this number as a smaller, simpler way of denoting that pattern.

---

Chapter 4 - The ecosystem of the neural network Agent

We now have some general idea of how a neural network functions, and we have had some not-so-subtle hints on what the purpose is of neural networks in nature. Using a few steps in reasoning we have created a set of constraints, and within this framework we have constructed our first few networks. It is high time we spend some effort on contemplating the purpose of neural networks, and for this we shall not study the networks themselves, but rather the systems they arise in.

Simple competitive systems

Zero sumness

Consider the world and all the life it contains. Where does this life arive from? Well, to answer that we first must have a working definition of life. I have spent some time at this, and I was forced to admit that to construct a complete yet not overly broad definition lies outside my scope of competence when a friend pointed out to me that my then-current hypothesis would include the phenomena of fire and radioactive decay into the realm of living things. Therefore I shall for now resort to the overly limited hypothesis that life is any process which is adaptively self-perpetuating, and functions against entropy through the consumption of energy.

This means it might be quite usefull to identify the available sources of energy. Without delving too deeply into the subject, let us simply state that the available sources of energy here are electromagnetic radiation (mostly from the sun), gravitational energy (mostly from the moon) and radioactive decay (which includes various types of radiation and heat).

All this energy can be used untill it is finally emitted into the universe as blackbody-radiation. The total amount of available energy is therefore fixed; the exact value depends on the equilibrium conditions. Since the total amount of available energy is fixed there is only a limited amount available for every living organism.

There are two methods that can be used for an organism to be self-perpetuating. First, it can continuously try to detect and repair damage when it occurs, thus keeping the individual alive. The second method is to continuously replicate the organism. Since the former must fail eventually, it is always combined with the latter. Consequently, the amount of living organisms increases untill the limited available energy to them is used up. Since this does not stop orgnisms from procreating, they must compete for available energy and other necessary resources. This competition forms the selection mechanism of fitness that drives the evolutionary process.

This system where there is a fixed amount of energy that organisms must compete for, and that has both winners that survive and losers which go extinct, is an example of what is known as a zero-sum game. As you may guess, the term stems from game theory. In a zero-sum game the total amount of whatever is at stake always remains constant, so that the only way for one to win something is for another to lose an equal amount (which, when summed, equals zero). I like the example of a party in your youth, where the more of your mother's delicious home-baked apple pie with extra walnuts, raisins, cinnamon, ice and whipped cream was eaten by everybody else, the less there was for you.

TODO: fix above figure so that it uses -1,1 instead of 0,1

As we use it, the term "game" denotes a system which, loosely interpreted, is a self-contained universe that is inhabited by Agents and is governed by rules that fully describe the environment. If you take a look at the game that is described in figure 10a you can see it consists of two Agents (Alice and Bernadette) who must choose between three possible actions (rock, paper, scissors), and who win (1) or lose (-1) according to what the combination of their actions entails. While Alice and Bernadette may lead interesting and rewarding lives outside of the game, Alice running a manufacturing company in avionics and Bernadette being a theremin virtuoso, it has no bearing on how the game is played, nor on who wins or loses. The game is, or at least appears for now, fully self-contained.

Zero-sum games such as these could in a very real sense be described as systems wherein the reward of the Agents shows a perfect negative correlation ($r=-1$). This implies that it is conceivable to create systems that do not need to display this negative correlation between the outcomes of the Agents. As we do not want to be too clever for our own good, let us call such systems by their standard name, which is non-zero-sum games.

Simple competitive systems - Non-zero sumness

The non-zero systems described above are actually aberrations. Constructs of the human mind such as games or bets, that are generally not found in nature. This of course does not mean that there is no competition; it can well be argued that nature is competition. It means that the benefit that goes to the winner need not equal the loss of the loser. Typically the loss tends to be greater than the benefit. For example a tiger who happens to catch a langur monkey will be fed for a few days at most, while the monkey will be dead for the rest of time.

Despite the fact that the individual Agent, be it a monkey, an ant or a citrus tree, will often end up losing very badly in the challenges that the game of life presents, and in the case of the monkey and probably the ant will undoubtedly experience a short but rather intense vexation on this point, on the grander scale of things this isn't all that bad. After all, this is part of the selection mechanism that drives evolution, creating an arms race between species which adapt to circumstances, finding and forming new niches, which over times causes biological complexity to increase untill very complex creatures such as ourselves may arise, for which we may be gratefull.¹⁰ The trick here is that the monkey and the tiger are not the true Agents playing the game, and the real game that is being played is not the trite eat-or-be-eaten. In reality, as beautifully argued by Richard Dawkins [9] they are mere pawns or maybe one should say avatars for the true Agents, which are the genetic and other hereditary material, and the game played is the game of evolution.

There are quite a lot of examples of cooperation that have arisen in the evolutionary system. It might even be somewhat difficult to find examples of organisms that would be capable of functioning wholly on their own. For cooperation does not only exist in the form of one monkey giving another a leg-up to get the banana, nor of ants acting as floorboards to increase the amount of food being delivered to the colony. Every organism with the entire ecosystem depends upon other organisms to systain the niche it is in. The only possible exceptions there might be the creatures at the "bottom" of the food chain, and even they usually are dependant on the continual recycling of materials that are temporarily locked up in the organisms higher up the pecking order of the eat-and-be-eaten mechanism. And there are systems cooperating within an organism to provide it what it needs to be an evolutionary success.

What is really interesting is the realization that even those systems that are part of the same organism may not always cooperate all that well. Sometimes they compete. And they compete in the only place that it truly matters for them: in reproduction. Because not all genetic material is inherited equally (Y-chromosoms only being inherited by males, mitochondrial DNA only by females), these different systems have different interests in reproduction. And so they compete. Even though it is a silly anthropomorphisation, I like to imagine these little genes in a small rickety boat, each trying very carefully to push the others overboard, all the while still needing to make sure the ship still reaches port safely. And there are actual examples of these genes trying to sabotage each other, locked in an arms race that is really detrimental to the organism they are part of, but not so detrimental that the benefits for them do not pay off.

Enough of that. Even though I understand the advantages, I cannot bring myself to really be happy with the eat-or-be-eaten system, as it (on the surface) seems such a waste: in every prey-to-predator step a large amount of energy is lost due to inefficiences. Admittedly, this frees room for new prey and increases the total energy-content of the system, but feelings may sometimes run counter to objective logic. No, I cannot really be happy with these win-little, lose-lots systems. What does get my heart beating faster is the possibility of another kind of non-zero-sumness: Win-win scenarios.

Suppose we play the game illustrated in TODO: add game payout matrix. This is a game that is played by two Agents, Aetelstan and Bor. If both Agents honestly cooperate with each other they share a total reward $R_{tot}$. Alas, the arbiter of the game is corrupt and willing to take bribes from the Agents, and he has let both of them know this. If one Agent decides to cheat by bribing the corrupt arbiter, he gets the entire reward for himself, minus a slice for the arbiter. If both bribe the arbiter, they again share the reward, which is greatly diminished with the bribes. What do the Agents do?

Consider the game from the point of view of Bor. We assume, as is standard in these scenarios, that both players will try to maximise their profits. This is Bor's motivation, and he assumes the same is true for Aetelstan. By working together both he and Aetelstan get a decent payout. By cheating, Bor can increase his payout if Aetelstan remains honest. However, Aetelstan knows this, and Bor knows Aetelstan knows this. Therefore Aetelstan is likely to also cheat, which reduces the reward for both, defeating the purpose of cheating in the first place. Clearly it is better for Bor to cooperate. Except, of course, that if Bor is honest, while Aetelstan cheats, Bor ends up with nothing. Given that for Bor there are only two scenarios: one where Aetelstan is honest, and one where he cheats, and having just deduced that in both circumstances Bor is better off by cheating himself, he will cheat. Given the symmetry of the game, Aetelstan will also cheat. And despite the fact that both know they will both cheat, while knowing both would be better off not cheating, they still cheat, robbing themselves of potential profit.

In effect the players are trapped in their own logic, as they must rationally choose this poor-result action to maximize their profit. To denote this point a physicist would probably say it is a local minimum, just as (s)he would describe a ball stuck in a well. Fortunately this system was not invented by a physicist, and the usual term to denote this point of logical (if silly) behaviour is known as the Nash Equilibrium. It is not always true that the Nash equilibrium is at a point where everybody is worst off, but this is a nice example to show that it is certainly possible.

Warning: Missing argument 2 for Document::add_subsubsection(), called in /var/www/htdocs/rpzboray/webpages/neural_network/teksten/english/Game theory.php on line 64 and defined in /var/www/htdocs/rpzboray/webpages/neural_network/php_functies/class_document.php on line 307

experiments

Inhibition

The time has come to start to implement and test some of these ideas. We will start with the basic principles, and see where they lead us. I have to warn you in advance that this at first is going to produce some wacky results. There's no need to be alarmed. The problems, and their solutions, we will encounter will (hopefully) give you a better understanding of what we are doing, and why we are doing it in the specific way we will end up with.

Let us start by returning to an earlier point: Neural networks exist only to help an organism (the Agent) adapt to the specifics of its environment, and so increase that organism's evolutionary fitness. If we are to create a neural network, we first must have an Agent and especially an environment for it to function in.

The Agent collects information from its environment which it uses as Input for its neural network. Its actions, which are determined by its neural network, affect change in this environment. This change may produce some result that increases (good) or decreases (bad) the Agent's fitness. We want a simple system that captures the essentials of this process, so we create a small, abstract game.

What shall be the nature of the game? Given that the foundation of our argument is formed by competition, we want a competitive game. We also want to keep things as simple as possible, so let us choose a 2-player game. Then we can also use the outcome of the game to easily compare the Agents with each other. So that we do not deviate from this simplicity, we shall make the game zero-sum: one winner, one loser. We also want to make sure the game is fair, i.e. both players have the same chance of winning, to avoid an extra burden of statistics.

We first construct the simplest fair non-zero game of choice for two players we can think of. The game is defined by:

$\displaystyle{ \left( \begin{array}{c c} [-1,1] & [1,-1] \\ [1,-1] & [-1,1] \end{array}\right) $

Which can be reduced to

$\displaystyle{ A = \left( \begin{array}{c c} -1 & 1 \\ 1 & -1 \end{array}\right) $

which is the payout matrix for player A, which -because this is a zero-sum game- automagically includes the payout matrix for player B, as $B = -A$.

Now we want some Agents to play the game, so we need to create them. This game constitutes a choice between two possible actions, which can be represented by a single Output node where (in)activity means choice $1$ ($2$). As for the information the Agent can use, it is strictly limited to the payout matrix. Because of the symmetry of the game we can get away, just now, with using only two Input nodes. These two nodes can represent either row or column of the payout matrix, and with this information the rest of the game can be derived. We want everything as simple as possible, so we will not include any Hidden nodes, and simply directly connect the Input and Output nodes. We could set the strength of each connection by hand, and probably get a very good result. However, as we will make more networks than just this one (and we physicists like to be as lazy as we can be) we better set up a system that readily generalizes. Therefore we assign to each of these connections a random strength ($w_{ji} = [0,\theta] \in \mathbb{R}$).

To play the game we need at least two Agents. But there is no actual need to stop at that, since it costs no more effort to create a thousand Agents than one, because of the random generation of their properties. We shall keep ourselves to the old motto "waste not, want not", and create only hundred Agents. We let every Agent play against every other Agent for a grand total of 9800 games (remember, the Agents cannot play against themselves, so for $N$ Agents, we have $N \times (N-1) $ games). If we look at the results as plotted in Figure unknown it all seems perfectly clear: each Agent makes the same choice in each game, and the choices are evenly divided (the discrepancy is within the error margin) between the Agents, which on average leads to an equal number of games lost and won.

That is all very interesting, but let's see what happens when we scale up a bit. We want the Agents to play gamess that are somewhat more complex, so how are we going to do that? As all truly lazy people everywhere we want to achieve maximum result with a minimum of effort. Thus we want to be able to randomly generate games. We still want these games to be both zero-sum and fair. This can be done by assigning random payouts for the actions of the Agents, under the restriction that:

$\displaystyle{ A = \left( \begin{array}{c c c} A_{11} & \ldots & A_{1n_g} \\ \vdots & \ddots & \vdots \\ A_{m_g 1} & \ldots & A_{m_g n_g} \end{array}\right) = \left( \begin{array}{c c c} -A_{11} & \ldots & -A_{n_g 1} \\ \vdots & \ddots & \vdots \\ -A_{1 m_g} & \ldots & -A_{n_g m_g} \end{array}\right) = -A^T $

where $A_{ef}$ represents the payout for player $A$, which automatically gives the payout for player $B$, as that equals $-A_{ef}$. Therefore $A=-B$ and it is obvious that

$ A_{ef} = -A_{fe} = B_{fe} = -B_{ef}$

In other words, the payout matrix of our zero-sum games is skew-symmetric, and thus $m_g=n_g$ and $A_{ef}=0\mbox{, if } e=f$. This ensures that not only is the game zero-sum, it is also fair, as each participant has, ceteris paribus, an equal chance of winning.

For this new situation we will have to make some adjustments in the networks of the Agents, as we must account for the new number of choices ($n_g$) the Agent can make, and the possibly greater amount of information ($n{_g}^{2}$) available about the game. For now we are only interested in the Agents winning or losing each game, and not in how much they win or lose, so we do not need more than $n_g^2$ Input nodes. When the Agents could choose between two actions we could encode this into a single Output node. We can scale this up, letting the activity pattern $\Omega$ of the Output nodes represent a binary number that determines the choice made. We shall assume for now the Agent has some system in place to translate $\Omega$ into a distinct action it can take. Unfortunately what we do here leads to a problem, given that that number of choices represented by $\Omega$ is some power of two ($2^{N_{\Omega}}$), while there are $n_g$ possible actions. While any possible action can be represented by $\Omega$ by setting an appropriate $N_{\Omega}$, the reverse is not true, as $\Omega$ may represent a choice that is not available, by overshooting the number of available choices. For example, in a game with 713 choices availble, the minimum value for $N_{\Omega}$ is 10. If so, assuming an equal probability of any choice, there is an approximate $\frac{1}{3}$ chance of the Agent making an inadmissible choice.

Well, how about we try having the activity of each Output node represent a single choice. We shall call this the "direct-choice" method, as opposed to the "binary-choice" described above. Then, in the game with 713 choices, we need a network with 713 Output nodes. That may be all nice and dandy, but what if multiple Output nodes are active together, which is not at all unlikely? This would amount to the Agent making several choices at once, which goes against the nature of the games, so these cases must also be inadmissible. We have quite a conundrum here, and we do not quite know how to solve it. So, for now, let us simply ignore it. In every game we shall simply disqualify any Agent who makes an inadmissible choice and see where that leads us.

This time we shall create 200 Agents, 100 for each method of determining choice, and plot the results of every Agent playing ten different 10-choice games against every other Agent within the same group in Figure unknown. As you can see the percentage of disqualifications in the direct-choice scenario depends rather heavily on the number of possible choices within each game. The number of disqualifications in the binary-choice model however seems different. Let us examine this a bit further. We create 100 different games, and for each create 100 binary-choice Agents. If we plot the disqualifications (Figure unknown), is seems the number of disqualifications always averages about $\frac{1}{3}$, which seems a bit odd. Well, maybe it is not so odd. After all, for every $N_{\Omega}$ nodes at least half the possible activity patterns will always represent valid choices within the game. As for the other half, each game may have any number of choices greater than $2^{N_{\Omega}-1}$, and each number of choices is equally likely. Therefore, we know half of all choices will always be valid, and of the other half of choices on average half (or $\frac{1}{4}$) will be valid, which amounts to an average of $\frac{2}{3}$ valid, and therefore $\frac{1}{3}$ invalid choices.

Now let us take a closer look at the problem with the binary-choice method. It seemed to work so well in our first game, so where do the problems come from in games with $n_g \lt 2$? To be more specific, why does there seem to be a relationship between $n_g$ and the number of active Output nodes? Let's think about it; a node becomes active if its potential exceeds the threshold $\theta$. The potential depends on the number of active presynaptic nodes, and the strength of the connections. With an average connection strength $\bar{w_{ji}} = \frac{\theta}{2}$, which results from our random distribution of weights the activity of a node depends on the amount of presynaptic nodes. With 2 presynaptic nodes the chance an Output node is active, when averaged over a great many Agents, is $\frac{1}{2}$, which through happy coincidence is the correct method of playing our original game. Unfortunately this is an exception, not the rule. For an $n_g$-choice game with $n{_g}^{2}$ payouts there are also $n{_g}^{2}$ Input nodes. All of these may contribute to the potential of the Output nodes, so that the chance of an inadmissible choice grows with $n_{g}^2$.

So what do we do? We have two possible methodologies, and both lead to absurd results. We need to fix them, but how? First we shall look at the binary-choice problem, which is that $\Omega$ may represent a larger number than the amount of available choices. We could assign each Agent with a randomly determined default choice, so that an inadmissible choice revertes to this default. That might actually work, but what possible mechanism could there be which would realize this strategy? Hmmm, given our current understanding I cannot think of anything. We could try something else; we could have each $\Omega$ not determine any specific choice but simly a choice, with some additional mechanism translating this choice into a specific value, assigning every $\Omega$ to a specific choice with a probability $\frac{2^{N_\Omega}}{n_g}$. Unfortunately this means we need some additional mechanism, and the only type of mechanism we are familiar with are neural networks. Which means we face the problem of infinite regression that is such a heavy burden in several other topics of study. Maybe we should just abandon this approach and focus on the direct-choice mechanism.

The problem we have with the direct-choice mechanism within the games we have is that the average amount of active Output nodes increases linearly with the amount of Input nodes, and therefore with the square of the amount of possible choices. How can we fix this? The high activity levels stem from the high potentials in the Output nodes. How about we decouple these from each other? We could simply pick the node with the highest potential to represent the choice made by the network. That's what often if not usually happens in the field of artificial intelligence. Unfortunately there is a problem, in that there is no easy mechanism to quickly and reliably determine which node has the highest potential, because the only fast mechanism of communication the nodes have is their activity, and presto: faster than a priest decrying the need for a prime mover we have locked ourselves into circular reasoning. What else can we do? Well, instead of choosing the node with the highest potential, we could lower the potential of all Output nodes equally, untill only one remains active. This could actually be done, by having the Output nodes connect to some gland or other system that releases inhibitory neurotransmitters back onto the Output nodes, untill only one remained active. It would be cumbersome, but feasible, so have we found our solution? I am afraid not. I hate to use this fallback position, but this system is simply not present in biological neural networks. We need something else.

Given that we cannot alter the potential of the Output nodes post hoc, maybe we can do something about it beforehand. We might tweak the strength of the connections, so that at least on average the Agent generates a single choice. This can be done quite easily by normalisation of $w_{ji}$ so that $w_{ji}=\frac{c}{N_{\Upsilon}} \mbox{, with }c=[0,\theta]$. This does result, as you can see in Figure unknown, in the Agents on average choosing a single action. Unfortunately averages are not all they are cracked up to be. The number of Agents choosing no action, and those who still choose multiple actions is simply unacceptable.

We could attempt another method of reducing the potential in the Output nodes; instead of manipulating the strength of the connections we could alter the amount of presynaptic nodes. While we cannot change the number of Input nodes -nor the number of Output nodes for that matter- we can add a new layer of nodes in-between. In this Hidden layer $L_2$ we can choose any amount of nodes we like. Therefore $N_{L_2}$ can be chosen so that the Output activity expectancy equals 1. Alas, the problem that just now thwarted our designs again raises its ugly head: while we can get the desired average Output activity, the actual level of activity deviates all too often.

TODO: plot results for various $N_L_2$, with appropriate $w_{ji}$.

None of our approaches works! We need something else, something radically different. Since we cannot prevent the network from producing invalid choices, we need to somehow give the network the ability to adapt $\Omega$ if it is invalid. And if we are adapting $\Omega$ anyway, we might as well try to make it so that it creates a good choice.

Untill now we have been working with an open network, i.e. one where the activity of the nodes originates in the Input layer and terminates as an Output pattern. If we however close the network so that the activity flow represents a loop, it is conceivable that $\Omega$ changes over time. Consider the Input layer at $t_0$. Some Input nodes are activated because of stimuli originating in the environment, forming an Input pattern $\Upsilon(t_0)$. The connections between the Input and Output nodes map $\Upsilon(t_0)$ to $\Omega(t_0)$. Because the network constitutes a closed loop, $\Omega$ can be considered to be fed back into the Input nodes. This can be done in two ways. First, the active Output nodes may excite the Input nodes directly by acting as presynaptic nodes. In the case of some Input node $j$:

$\begin{array}{r c l} \chi_{j_\Upsilon}(t_{n+1}) & = &\Theta \left(h_{j_{\Upsilon}}(t_{n+1})\right) \\ & = & \Theta \left(h_{j_{\Upsilon}}(t_n) + \Delta h_{j_{\Upsilon}}(t_{n+1})\right) \\ & = & \Theta \left(h_{j_{\Upsilon}}(t_n) + \Sigma_{i=1}^{N_{\Omega}} w_{ji}\chi_i(t_n) \right) \end{array} $

where $i$ here somewhat counterintuively denotes the Output nodes, as follows from our convention of having $i,j,k$ denoting nodes in order of the signal flow.

The behaviour we would in general expect would be for $\Omega(0)$ to increase the total potential of the input nodes $h_{\Upsilon_{tot}}(1)=\sum_{j=1}^{N_{\Upsilon}}h_j(1)$, which in turn is likely to increase the number of active output nodes, which will further increase $h_{\Upsilon_{tot}$ in a cascade that continues untill the point is reached where $\Omega(t_{n+1})=\Omega(t_{n})$. This will usually be with every Output node in the network actived, which obviously cannot be right. Fortunately there is a way to fix this.

Our efforts up to now to construct a functional and biologically plausible system to make choices failed as with our current abilities we have no good system to restrict the Output the network generates. The constraint that we created in the restrictions on Output activity may on first appearances look a little silly, but they helped us discover that our current systems are inadequate. After all, the neural networks should be able to operate under this constraint, as biological networks (sometimes) do. The fact that often results of biological neural networks are a superposition of Output patterns, each directing a different action which occur simultaneously is therefore irrelevant. The system we are currently considering consists of a closed network that with each cycle adjusts its output untill some stable state results. Unfortunately the actual adjustments only consist of increasing the Output activity, often untill every Output node is active. This is an undesireable result, so we need some mechanism that allows the network to decrease the activity in its nodes.

Let us return for a short span to the biological neuron. It innervates cells it links to through the release of neurotransmitters. These neurotransmitters induce a potential change in their targets by switching ion gates on or off. There are several switching mechanisms, and different neurotransmitters for each. Some neurotransmitters operate on gates that lower the cell's potential, inhibiting its likelyhood to fire. We have not yet accounted for this ability to lower the potential of a connected cell; let us see how far we can get if we do include this mechanism.

We have a closed network, consisting of two layers of excitory nodes, that receives input which resembles the possible results of various actions in a game. This leads to the activity of the Output nodes increasing from $t=0$ untill the point where $\Omega(t_n)=\Omega(t_{n+1})$, usually with every Output node active. We can plot the increase in Output activity over time for some network $A$ in Figure unknown. Now let's copy this network $A$ into several new networks $B_i$, with various percentages of the Output nodes set to become inhibitory. Before we look at the actual results, let us think about what might happen. If the fraction of inhibitory nodes is small ($N_{\Omega_{\mbox{inhib}}} \ll N_{\Omega_{\mbox{excite}}}$) you would expect them to have a minimal or even no influence, possibly slowing down the advance in activity over time or stopping this advance at an earlier stage. If however this fraction is large ($N_{\Omega_{\mbox{inhib}}} \gg N_{\Omega_{\mbox{excite}}}$) what you would expect is that the Input nodes would become inhibited, and therefore less active; this would lead to a greatly reduced activity in the Output nodes, which would therefore nullify their inhibition of the Input nodes. These then resume their original activity pattern, and we find ourselves with an alternating pattern of some $\Omega \neq 0$ and $\Omega = 0$. What then if the fraction between inhibitory and excitory Output nodes is neither large nor small ($N_{\Omega_{\mbox{inhib}}} \approx N_{\Omega_{\mbox{excite}}}$)? You would expect some behaviour inbetween the two extremes we just described, with three possible outcomes: an continuous increase of Output activity untill a steady state is reached; a continuous alternating between a state of zero output activity and some output activity larger than zero, and an Output activty that changes over time but eventually repeats itself. And if we actually do the experiment (Figure unknown) you can see that this is approximately what we find.

Of course this gives us a new problem as except in the steady-state solution we do not have a single choice for the Agent in the game, and that solution is invalid, as we already determined. We might make some argument that the two-state strongly-inhibitory solution could be valid by ignoring $\Omega=0$ but that seems rather contrived. We have to deal with things as we find them, not as we would like them to be; at least if there is a functional difference between the two. The remaining option of a periodicity in $\Omega$ provides us with the conundrum which of the varying $\Omega(t)$ values should be considered its actual output. But perhaps this is not really a problem; after all, the virtual environment (the games) lack a property that is distinct in every "real" environment which are permanent -albeit not necessarily static- in time. If the environment changes with time it would make sense the "optimal" choice changes as well, and therefore ideally the choice the Agent makes should change with it. We should update our games to reflect this new view, so that they become functions of time;

$\displaystyle{ A(t) = \left( \begin{array}{c c c} 0 & \ldots & A_{1n_g}(t) \\ \vdots & \ddots & \vdots \\ A_{m_g 1}(t) & \ldots & 0 \end{array}\right) $

goto learning.

---

Chapter 5 - Learning

Intro: Agent in environment has poor performance.

Learning: changing of behaviour.

Restrictions because we are working with physical neurons (bastiaan wemmenhove).

Hebbian and antihebbian learning.

Test in zero-sum and non-zero-sum games.

Long period of Antihebbian learning leads to certain average activity (bedeaux)

Can fix this by changing parameters for desired average activity, either through cheating or through genetic algorithm.

Instead, add inhibitory nodes: increase variation in activity during antihebbian learning.

Test in zero-sum and non-zero-sum games.

Various distributions of excitory-inhibitory nodes possible. Test various distributions in various games -> find highest variation during antihebbian learning for some given topology, find distributions in each of various games that on average results in the quickest finding of a local maximum.

Explain about local maximum vs global maximum, example balloon floating on ceiling during party.

Plot $\Upsilon-\Omega-\rho$ spaces.

Talk about making plot more meaningfull: hypercube, "dinges"-reeksen. Stik. Todo: juiste naam opzoeken in paar honderd artikelen. Hint: zoek naar hypercube.

Plot $\Upsilon-\Omega-\rho$ subspaces from Agent generation to Local maxima.

Note that the shorter the distance from generation to local maximum, the more likely to be the actual local maximum the neural network ends up in.

-- Learning during sleep to work with better data (averaged, remove misleading data) => During non-sleep the reward-mechanism can be utilised to judge outcomes of estimated outcomes of plans/future events -- -- Learn small action-result units. From these learn result-action units. To get from situation A to situation B determine the difference between both situations, and construct a string of actions that should result in A->B. --

---

Chapter 6 - Improving Agent fitness by changing node properties

Direct learning along $\nabla \rho$ in the $\Upsilon-\Omega-\rho$ space, by adding memory about w_{ji} at last best $\rho$ to node, and during antihebbian learning every $d$ timesteps reset to these weights.

Plot results

If stuck at local maximum with insufficient $\rho$, increase $d$.

Plot results

Logarithmic learning: small differences in reward when reward is not enough for survival are more important than differences in reward when survival is achieved: Weigh the reward factor on a logarithmic scale. This allows the net to better escape a local maximum.

Plot results.

Discuss why results are practically equal to previous results.

Plot lifespan-energy etc. Discuss why these _are_ different. Discuss competing on long (evolutionary) timescales, so that in the long run, the logarithmic reward scale does lead to more success.

Set up a new game, specifically tailored for exponential increases in reward. Discuss logic.

Plot results.

Antihebbian learning following $\nabla \rho$ in the $\Psi\mbox{-space}$

The result of a prolonged period of antihebbian learning is in essence a strongly randomized $\Psi$-space, although one or two things can be said on its average activity. This situation, while functional in such systems as we have hitherto seen, seems contrary to common sense. After all, there is little to no guarantee that a positive result emerges from such a period of learning. Indeed, given a nearly infinite amount of possible actions, and usually a rather limited amount of usefull actions, it would seem that a negative outcome would be where a smart investor would place his bet.

This is a problem, as it would seem that most biological organisms are too succesfull¹¹ when compared to the theory. Most people will be familiar with nature documentary footage of some newly-born herbivore herd animal being born and learning to stand and walk in a manner of hours (something which takes us many months). While it is possible to place responsibility for this purely with hereditary factors (instinct), that to me seems a weak response.

I propose a different system, wherein only improvements are actualised and antihebbian learning steps which produce inferior results are discarded. For this system to work every neuron needs to remember the connection strengths $w_{ji}$ at the time of the highest reward factor (and therefore also the height of $r_{\mbox{last max}}$). While this requires an added complexity to the theoretical framework, I feel it is a price worth paying because the endresult is a steady, albeit slow, improvement of $\Psi$ along $\nabla r$, until $r$ reaches its (local) maximum.

For this system to work in biological reality it would require the synapses to be rapidly informed of the success of the network compared to the speed with which the short-term connection strength changes. There is an inverse relationship between the rapidity of learning and the "correctness" with which each learning step follows $\nabla r$ which depends on the amount of antihebbian short-term connection strength changes that are applied before long-term potentiation takes over.

Despite the obviously enormous advantage of automatically and continuously increasing $r$, the system does have its drawbacks (other than the already mentioned increase in complexity). The disadvantage lies not so much in the slowness of the learning process (as normal antihebbian learning can be extremely slow as well, depending on the size of $\Upsilon$ and $\Omega$) although that may matter in specific circumstances, but in the fact that the connections of the neural network may get trapped in a suboptimal local maximum of $r$, with true excellence ever out of reach.

---

Chapter 7 - Improving Agent fitness by division of labour within the neural network

Re-discuss changing environments, the difference between genetic (instinct) and learning for agents within such an environment.

Hypothesis: combine properties of both for improved result.

Divide network into two networks. One "normal" or "thought", one "overmind" or "emotion" whose output determines $\rho$ for "normal".

Plot results for several generations for pure genetic, pure learning, combined for the environments: stable, slowly changing, completely chaotic. Last one maybe keep out? It was silly waste of CPU time anyway.

*briefly* Discuss biological reality.

Discuss that in a stable environment, a genetic learning rule will eventually outcompete "normal" learning. Propose experiment with predicted outcome that creatures in stable environments are worse at learning than those that evolved in less stable environments, for the same brain mass/number of neurons. Discuss what exactly the equalizer should be.

Set up a new game. This game has increased complexity, in that it is simple to play a good strategy ($\rho$ such that one can survive easily. There are many good strategies, which may be developed into superior strategies. The game is played with several agents.

Add a new subnet. This subnet can compare two patterns, and determine if they equal one another or not.

Add a new subnet, this subnet can compare two numbers, and determine if one is equal, greater or smaller than the other

Add a new subnet. This subnet can temporarily store information. Two possible methods: use learning to store the pattern in the weights, or use a closed loop net that continually has output identical to a previous input, untill it is inhibited away by a new pattern. The former works for very small patterns, the latter is more sensitive to disruptions.

Add a new subnet that continually counts upwards in binary, resetting to zero if the maximum is reached.

This system allows to reliably and somewhat rapidly test various Output responses to Input from the environment. The best response can be found by storing at each improvement the Output pattern in memory. Once the whole range of Output possibilities has been gone through, the global maximum -for that particular Input- has been found in one of the patterns.

Now let the action net use transhebbian learning. Use feedback from the pattern-comparing subnet, which compares the action output pattern with the stored output pattern.

plot results

Discuss the possibilities of using different pattern-generating subnets, having them learn to generate only a few, usually successfull patterns. No experiments done. Hypothesis: this would very strongly improve success.

Allow emotional output to be used as reward for emotional network.

This leads, predictably, to failure.

plot results

Introduce new, secondary emotional subnet.

This subnet learns to match emotional Output with environmental Input. Thus it can associate positive and negative environmental factors with high/low reward that are not in the original emotional subnet.

Use several algorithms to switch from primary to secondary emotional subnet over time.

Plot results

Discuss possibilities of several stages of learning, each employing a new emotional subnet, which was trained (auto-associated) with the Output from the previous one. No further experiments done. Hypothesis: this may explain periods during which strong changes take place, such as in puberty. Dit dient beter geformuleerd.

Allow this to associate with output from original and environmental input when success is being achieved. If $\rho$ is stable, switch to secondary emotional system. Periodically check if with this second system $\int_t \rho$ in the original increases. If not, use antihebbian learning on it. If $\rho$ decreases, let the secondary system re-associate with the original. If they are present in the environement (and in the new game this is true), new/better reward factors will be found.

Introduce critical periods of learning for emotional network. Allow it to auto-associate when it learns a new succesfull strategy. Use time, use events as indicators to start new critical period.

Discuss similarity with sleep cycles? Maybe later.

This leads to improvement through association.

Antihebbian learning following $\nabla \rho$ in the $\Psi\mbox{-space}$

The result of a prolonged period of antihebbian learning is in essence a strongly randomized $\Psi$-space, although one or two things can be said on its average activity. This situation, while functional in such systems as we have hitherto seen, seems contrary to common sense. After all, there is little to no guarantee that a positive result emerges from such a period of learning. Indeed, given a nearly infinite amount of possible actions, and usually a rather limited amount of usefull actions, it would seem that a negative outcome would be where a smart investor would place his bet.

This is a problem, as it would seem that most biological organisms are too succesfull¹² when compared to the theory. Most people will be familiar with nature documentary footage of some newly-born herbivore herd animal being born and learning to stand and walk in a manner of hours (something which takes us many months). While it is possible to place responsibility for this purely with hereditary factors (instinct), that to me seems a weak response.

I propose a different system, wherein only improvements are actualised and antihebbian learning steps which produce inferior results are discarded. For this system to work every neuron needs to remember the connection strengths $w_{ji}$ at the time of the highest reward factor (and therefore also the height of $r_{\mbox{last max}}$). While this requires an added complexity to the theoretical framework, I feel it is a price worth paying because the endresult is a steady, albeit slow, improvement of $\Psi$ along $\nabla r$, until $r$ reaches its (local) maximum.

For this system to work in biological reality it would require the synapses to be rapidly informed of the success of the network compared to the speed with which the short-term connection strength changes. There is an inverse relationship between the rapidity of learning and the "correctness" with which each learning step follows $\nabla r$ which depends on the amount of antihebbian short-term connection strength changes that are applied before long-term potentiation takes over.

Despite the obviously enormous advantage of automatically and continuously increasing $r$, the system does have its drawbacks (other than the already mentioned increase in complexity). The disadvantage lies not so much in the slowness of the learning process (as normal antihebbian learning can be extremely slow as well, depending on the size of $\Upsilon$ and $\Omega$) although that may matter in specific circumstances, but in the fact that the connections of the neural network may get trapped in a suboptimal local maximum of $r$, with true excellence ever out of reach.

---

Superior innate value systems - exit Homunculus

Learning in stages (staggered learning?)

Imitation

---

Chapter 8 - Improving Agent fitness through social factors

Warning: Missing argument 2 for Document::add_subsection(), called in /var/www/htdocs/rpzboray/webpages/neural_network/teksten/english/artificial neural networks - improvements through social factors.php on line 3 and defined in /var/www/htdocs/rpzboray/webpages/neural_network/php_functies/class_document.php on line 286

Emotions

Warning: Missing argument 2 for Document::add_subsection(), called in /var/www/htdocs/rpzboray/webpages/neural_network/teksten/english/artificial neural networks - improvements through social factors.php on line 5 and defined in /var/www/htdocs/rpzboray/webpages/neural_network/php_functies/class_document.php on line 286

Family relationships - emergent family relationships

Warning: Missing argument 2 for Document::add_subsection(), called in /var/www/htdocs/rpzboray/webpages/neural_network/teksten/english/artificial neural networks - improvements through social factors.php on line 7 and defined in /var/www/htdocs/rpzboray/webpages/neural_network/php_functies/class_document.php on line 286

Direct and indirect communication

Warning: Missing argument 2 for Document::add_subsection(), called in /var/www/htdocs/rpzboray/webpages/neural_network/teksten/english/artificial neural networks - improvements through social factors.php on line 9 and defined in /var/www/htdocs/rpzboray/webpages/neural_network/php_functies/class_document.php on line 286

Culture and fashions

Warning: Missing argument 2 for Document::add_subsection(), called in /var/www/htdocs/rpzboray/webpages/neural_network/teksten/english/artificial neural networks - improvements through social factors.php on line 11 and defined in /var/www/htdocs/rpzboray/webpages/neural_network/php_functies/class_document.php on line 286

Trade and treachery

Language in learning. Lossy medium. Consciousness / memory is a lossy medium. Describe reality => incomplete. Literature is an example of lossy communication.

---

Chapter 9 - Society

Structured society

The great source of both the misery and disorders of human life, seems to arise from over-rating the difference between one permanent situation and another. Avarice over-rates the difference between poverty and riches: ambition, that between a private and a public station: vain-glory, that between obscurity and extensive reputation. The person under the influence of any of those extravagant passions, is not only miserable in his actual situation, but is often disposed to disturb the peace of society, in order to arrive at that which he so foolishly admires. The slightest observation, however, might satisfy him, that, in all the ordinary situations of human life, a well-disposed mind may be equally calm, equally cheerful, and equally contented. Some of those situations may, no doubt, deserve to be preferred to others: but none of them can deserve to be pursued with that passionate ardour which drives us to violate the rules either of prudence or of justice; or to corrupt the future tranquillity of our minds, either by shame from the remembrance of our own folly, or by remorse from the horror of our own injustice. ADAM SMITH

Affective forecasting

If you ask people after their goals in life, amazingly few seem to answer "happiness". When using the Socratic method, you will probably get this response eventually, even though the person in question tends to dismiss it as a sort of ultimate underlying assumption that is so trivial it does not even need to be mentioned. I wonder however if many people do not tend to forget that this is often the actual goal one tries to attain or keep. Because you see, while what I just wrote is purely anecdotal, it turns out that people on the whole tend to be rather bad at predicting what makes them happy. [10]

I wonder if too strong a bias on certain elements was conducive to our survival long ago: Striving harder, even at great cost, for some future success might impel us to store food, prepare for winter, all those things that make life possible in harsh or even temperate climates for those whose physiology appears to lack usefull little tricks such as hibernation.

Suppose you have two individuals living in the primeval Rhine delta. Let us call them Arnold and Berthold. It is springtime, and both know it will not be all that long untill winter. So, both prepare. Arnold does nothing all year round but improving his shelter, gathering fuel and storing food. Berthold does the same, except that he also takes care to spend some quality time with his family, to enjoy an afternoon off, and to basically remember what is really important in life. Next winter is brutal. It lasts six months of bitter, biting cold. Which individual is more likely to survive?

One might conject that if this general idea is true, it might be that over time, those societies which consist out of Arnolds would experience a faster technological advance than those societies of Bertholds. And given that societies compete, much like individuals or species, one may consider the possibility that over time a society of Arnolds will outcompete the Bertholds. In this case one might expect to find neurological differences in affective forecasting between populations which have persisted for a long time in harsh conditions, and those in optimal conditions.

However, even if this conjecture were true, it is altogether likely that not enough time has passed for our own species for the difference to be noticeable, or that technological advances spread more quickly through populations than they can compete. Still, it would be interesting to see what result simulations would produce, even though this lies outside the scope of the work I have done here.

Economics

What is interesting to me is how much of our economic system seems to depend upon the failure in affective forecasting. Our society is driven by economics, while economics are driven by consumer behaviour. But consumer behaviour is not driven by needs, as within the developed nations most actual needs such as for food and shelter tend to be fulfilled.

Fashions

Determining relative success

Women rate men more attractive when other women smile at these men (article-link?), might lead to extremes (popstars)

Conclusions

---

Principles of Neural Science, Eric R. Kandel, James H. Schwartz, Thomas M. Jessell

artikelen: language: Piraha; Everett, Dan; Linguists, Linguistics; The Amazon; Tribes, Tribal; Brazil dan gilbert: happiness

Informatie Verwerking

Dit netwerk probeert om voor een gegeven patroon ${{\rm Y}}_C$ van activiteit in de Input neuronen een specifiek patroon ${\Omega }_C$ van activiteit in de Output neuronen te leren. In het voorbeeld hierboven zitten de Input neuronen (zintuigen) in de bovenste laag, en de Output neuronen (handelingen) in de onderste laag. De overige neuronen worden omdat in een biologisch neuraal netwerk hun activiteit slecht waarneembaar is meestal de Hidden of verborgen neuronen genoemd. U kunt zien hoe het netwerk hierboven op dit moment aan het leren is (tenminste: als het nog niet klaar is) en hoe de activiteit van de neuronen verandert totdat het netwerk het gewenste Output patroon genereert.

• Huidig Input patroon ${{\rm Y}}_C$: als U dit ziet is het programma dat U gebruikt om dit document te bekijken daar niet geschikt voor.
• Huidig Output patroon $\Omega$: Dit document maakt gebruik van de volgende standaard elementen: XML, XHTML, MATHML, SVG,top Ecmascript/javascript. Het kan zijn dat U javascript uit heeft staan. Misschien helpt het dan om dit (weer) aan te zetten.
• Gewenst Output patroon ${\Omega }_C$: Dit programma is succesvol getest onder firefox 2.0.0.2

Voor het gedrag van het neurale netwerk op de lange termijn zijn twee zaken belangrijk: Ten eerste de omgeving waarin het netwerk verkeert, zoals het die begrijpt via de zintuigen die verschillende eigenschappen van de omgeving omzetten in een Input patroon ${\rm Y}$. Ten tweede is van belang welke handelingen (zoals weergegeven in het Output patroon $\Omega$) beloond worden, en welke afgestraft. In het eenvoudige voorbeeld hierboven wordt slechts één enkel Output patroon ${\Omega }_C$ beloond, en alle anderen worden afgestraft.

Vanuit een praktisch standpunt is het interessante aan dit systeem dat het werkt als een black box: zonder zelf iets te doen aan de interne werking van het systeem, maar door het enkel informatie te voeren en een gewenst eindresultaat te specificeren krijg je (uiteindelijk) een systeem dat dit gewenste resultaat voor je zal bereiken zonder dat menselijk ingrijpen nodig is. In principe kan je een neuraal netwerk inzetten om taken uit te voeren waar mensen niet zo goed in zijn, zoals het vliegen van een onbemand vliegtuigje. In de praktijk is het waarschijnlijk nuttiger om deze systemen te gebruiken voor taken die mensen wel kunnen uitvoeren, maar waar het economischer zal zijn om geautomatiseerde neurale netwerken in te zetten. Bijvoorbeeld het analyseren en het voorspellen van de koers van aandelen op de beurs; het inzetten van industriele robots in productieprocessen zonder dat deze geprogrammeerd hoeven te worden; het vertalen van documenten zoals op grote schaal nodig in de Europese Unie. Voor deze voorbeelden heb je natuurlijk wel neurale netwerken nodig die zowel groter en sneller als wat subtieler zijn dan degene die je hierboven ziet.

Hoe werkt het

Als een neuron $i$ in een rij $m_i$ actief wordt stuurt het een signaal naar de neuronen $\mathrm{j }(j=1,2\mathrm{,...,}{N}_J)$ in rij $m_j (m_j=m_i+1)$ $$ waarmee het verbonden is. De kracht van een signaal hangt af van de sterkte wji van de verbinding tussen $i$ en $j$. Als in een neuron j de gecombineerde binnenkomende signalen sterk genoeg zijn wordt j zelf actief en stuurt het een signaal naar de neuronen $\mathrm{k }(k=1,2\mathrm{,...,}{N}_K)$ waar het zelf mee verbonden is.

Op deze wijze lopen (in dit netwerk) de signalen van de bovenste rij met de Input neuronen (m=1) naar de onderste rij met de Output neuronen (hier m=5). Ieder patroon van activiteit in de Input neuronen genereert dan een resulterend activiteitspatroon in de Output neuronen.

Wat als het resultaat van het netwerk anders is dan wat je wilt hebben? We nemen als voorbeeld een neuraal netwerk dat gebouwd wordt om winst te maken op de aandelenbeurs. Stel je geeft je neurale netwerk als Input informatie over beurskoersen, de wereldeconomie, nieuwsberichten en dergelijke. Vervolgens koppel je (zogenaamd) de Output aan opdrachten om aandelen te kopen, te verkopen of te houden. Vervolgens laat je het neurale netwerk lekker handelen, waarbij je het afstraft als het verlies maakt, en beloont als het winst maakt. Op een gegeven moment zal het neurale netwerk de patronen en correlaties leren kennen en consistent winst gaan maken. Hierbij maken we gebruik van de aanname dat er een onderliggende logica is in de beurskoersen, en dat deze een weerspiegeling zijn van een statistisch proces dat op voorspelbare wijze wordt beinvloed door factoren zoals de groei van de wereldeconomie.

Als je het neurale netwerk straft (omdat het verlies maakt bijvoorbeeld) moet het net zijn gedrag veranderen. Dit doet het door de sterkte van de verbindingen tussen neuronen die op dat moment actief zijn te verzwakken (hierdoor wordt het bestaande gedrag uitgeveegd), en de sterkte van de verbindingen tussen actieve presynaptische neuronen (laag m_i) en inactieve postsynaptische neuronen (laag m_i + 1) te versterken (hierdoor wordt nieuw gedrag aangeleerd). Vervolgens gaat het neurale netwerk weer handelen, en als het nog steeds verlies maakt straf je het opnieuw, en zal het wederom de verbindingen tussen de neuronen gaan aanpassen.

Uiteindelijk zal het netwerk succes hebben (winst maken). In plaats van het netwerk te straffen beloon je het, waarbij de verbindingen tussen neuronen die actief zijn versterkt worden. Het netwerk zal dit gedrag dan vaker gaan vertonen. Ook zal het netwerk als het af en toe verlies maakt het goede gedrag behouden terwijl het overgebleven slecht gedrag afleert. Zodra het netwerk (bijna) nooit meer slecht gedrag vertoon, en dus bijna altijd winst maakt koppel je het netwerk niet meer zogenaamd maar echt aan de beurs en wacht je rustig met een drankje in de zon af terwijl er geld voor je wordt verdiend.

NOTA BENE: Het voorbeeld van handelen op de beurs heb ik gekozen omdat het mensen waarschijnlijk zal aanspreken. In werkelijkheid gedragen de dagelijkse wisselingen op de beurs zich behoorlijk chaotisch en voor wie het zich niet kan veroorloven met gigantische bedragen te gokken kan vrijwel iedere beurs strategie zich tegen hem of haar keren. Dus ook deze! Dit voorbeeld gaat direct in tegen het adagio "resultaten behaald in het verleden bieden geen garantie voor de toekomst", omdat het juist bouwt op het zoeken naar patronen die zich in het verleden hebben voorgedaan. Meer praktische maar minder romantische voorbeelden zijn uit de robotica, de wapenindustrie en dergelijke.

Leerregel

De manier waarop een organisme tijdens het leren de verbindingen $w_{\mathrm{ji}}$ tussen de neuronen aanpast wordt wiskundig beschreven door een zogenaamde leerregel. Hiermee bereken je het verschil in verbindingssterkte in de tijd:

$$\Delta w_{\mathrm{ji}}\left(t_p\right)=w_{\mathrm{ji}}\left(t_{p+1}\right)-w_{\mathrm{ji}}\left(t_p\right)$$

De studie van Kunstmatige Intelligentie heeft vele verschillende leerregels opgeleverd die allen hun eigen sterktes en zwaktes hebben. Biologische neurale netwerken zijn echter beperkt doordat ze zijn opgebouwd uit hersencellen. De hersencellen kunnen bijvoorbeeld (op korte termijn) alleen informatie uitwisselen door actief of inactief te worden. Ze kunnen niet met elkaar overleggen welke van hen actief moet worden.

Door deze en andere beperkingen kunnen biologische neuronen in principe alleen gebruik maken van een enkel type leerregel*. In deze leerregel is $\Delta w_{\mathrm{ji}}$ afhankelijk van de activiteit van zowel het eferrent als het aferrent neuron, en van de beloningsfactor $r$ die het succes van het netwerk weergeeft:

$$\Delta w_{\mathrm{ji}}={\chi }_i\left\{\alpha r\right(2{\chi }_j-1)+\beta (1-r\left)\right(1-2{\chi }_j\left)\right\}$$

Waar a en b positieve constanten zijn. Deze formule geeft aan dat de verbindingssterkte verandert als neuron $i$ actief is (${\chi }_i=1$); dat wanneer het netwerk goed presteert ($r=1$) en neuron $j$ actief is (${\chi }_j=1$), of het netwerk slecht presteert ($r=0$) en neuron $j$ inactief is (${\chi }_j=0$) de verbinding tussen $i$ en $j$ versterkt wordt ($\Delta w_{\mathrm{ji}}>0$); tot slot geeft deze formule aan dat wanneer het netwerk goed presteert ($r=1$) en neuron $j$ niet actief is (${\chi }_j=0$), of het netwerk slecht presteert ($r=0$) en neuron $j$ wel actief is (${\chi }_j=1$) de verbinding tussen $i$ en $j$ verzwakt wordt ($\Delta w_{\mathrm{ji}}<0$). Hier zien we waarom formules soms beter zijn dan tekst, helemaal als we die formules nog verder kunnen vereenvoudigen:

$$\Delta w_{\mathrm{ji}}=[\alpha r-\beta (1-r\left)\right](2{\chi }_j-1){\chi }_i$$

* Een gecombineerde vorm van de Hebbian en AntiHebbian leerregels

OVERIG - Welk Output patroon ontstaat hangt af van de verschillende sterktes van de verbindingen tussen de neuronen. Net zoals de Input neuronen gelijk zijn aan de zintuigen, vormen de Output neuronen het uitvoerend deel van het zenuwstelsel. Ze zijn de motorische neuronen die onze spieren aansturen, ze regelen de snelheid van onze ademhaling en sturen in het algemeen alle processen aan die niet zelfstandig verlopen.