it is widely accepted that symbolic systems are useful in understanding the working of the brain, and there are many symbolic models of functions of the brain. This is based on the assumption, commonly implicit, that in the brain itself there is a symbolic system. In this article I challenge this belief, by showing that symbolic systems cannot be implemented by neurons in the brain. I based the argument on textbook knowledge from neurobiology, and the basic requirements for implementing symbolic systems. In particular, I show that there is no way to implement symbol tokens in neuronal substrate, where the individual connections of individual neurons (as opposed to cell populations) are not well defined.
It is common for cognitive scientists and researchers in related areas to assume that the human cognition can be described as a symbolic system. For example, Eysneck and Keane (1989), in a `Student's Handbook', list the basic characteristics of the information-processing framework, which they say is agreed as the appropriate way to study human cognition (p. 9). The second and the third items on this list are:
(they do mention connectionism later).
Other cognitive psychology textbooks are less blunt, but they also tend to regard the symbolic view as the appropriate way of looking at the brain, with connectionism as an alternative. For example, Stillings et al(1995), spend more than 2/3 of the chapter titled "The architecture of mind" (pp. 15-63) on discussion of the "symbolic paradigm", and the rest of the chapter (pp. 63-86) on connectionism and comparison between the two approach. Note that this is also "an introduction" book, rather than a speculative effort. In line with all other textbooks, they don't discuss the question of implementation in the brain at all.
The general appeal of symbolic systems stems mainly from two related reasons:
1) They are in general relatively simple, i.e. easier to understand than other models.
2) They are easy to implement on computers. This is because computers are themselves symbolic system, in the sense that addresses in computer satisfy the requirements of `symbolic tokens' (see next section).
However, this two characteristics are irrelevant to the brain, because brain systems are not necessarily simple or easy to implement on computers. The processing in the brain is done by neurons (possibly with some modulation by neuroglia), and every mechanism the brain is using must be implemented by neurons. Therefore, models of the mechanisms of the brain must be, in principle, implementable by neurons with the characteristics of the neurons in the brain. In this article I will show that symbolic systems are in principle unimplementable with neurons with these characteristics, and hence that symbolic systems are unlikely to be relevant to the brain.
Symbolic systems are based on symbols. Symbols are, according to Newell(1990) (p. 77)
"Patterns that provide access to distal structures" and "A symbol token is the occurrence of a pattern in a structure". Thus the implementation of a symbolic system requires tokens, which must have this two characteristics:
For a symbolic system to work, the operation of storing a token must satisfy two requirement:
This requirements are normally less explicit in discussions of symbolic systems. The first of them is most explicitly stated in Newell and Simon (1976, P.116): "A symbol can be used to designate any expression whatever". It is sometimes expressed in other terms (e.g. Newell (1990) talks about completeness, P. 77). The second one is taken for granted. Nevertheless, they are essential, and all implementation of symbolic models use them. It is this two requirements which are I will argue are not implementable by neurons in the brain. Systems which require only one of these, or none, are not discussed in this article.
As mentioned above, the question of implementation of symbol tokens is rarely even mentioned. For example, in Newell & reviewers (1992), which discuss the symbolic system SOAR (Newell, 1990), none of the participants raises the question of implementation of SOAR in real neurons. in Vera & Simon and reviewers(1993), which is also a multi- author discussion concerning symbolic systems, it is mentioned briefly: "The way in which symbols are represented in the brain is not known. Presumably they are patterns of neural arrangement of some kind" (Vera & Simon 1993b, p. 9).
However, implicitly it is assumed that symbolic systems are implemented in the brain, as Vera & Simon (1993c, P. 120) say: "The symbolic theories implicitly assert that there are also symbol structures (essentially changing patterns of neurons and neuronal relations) in the human brain that bear one-to-one relations to the symbols of category 4 [symbols in computer memory] in the corresponding program." These authors say later in the same article (P.126): "We are aware of no evidence (nor does Clancey provide any) that research at neuropsychological level is in conflict with the symbol system hypothesis". In the next four sections I will show that our knowledge at the neurobiological level is in conflict with the symbol system hypothesis.
It is not my intention to give a full description of what is know about neurons in the brain. The interested reader can find more in any textbook about the brain (for example, Brodar 1992, Dowling 1992, Gutnick & Mody 1995, Kandel et al 1991, Nicholls, Martin & Wallace 1992, Shepherd (ed.)1990, Shepherd 1994). Instead, I will list those characteristics which are essential to my argument. It is important to note that the characteristics listed here are `textbook' knowledge, supported by large body of consistent experimental evidence, accumulated by over 100 years of research.
In the following text, I use the term `brain' to mean `vertebrate brain', and the characteristics listed here are not necessarily true for simpler brains. When numerical values are mentioned, they are mainly based on the structure of the cerebral cortex, which is the main site of thinking in the brain. Other parts of the brain deviate from this values, but these deviations do not introduce any new principles.
The characteristics which are relevant to the argument are:
At the scale of organs, brains have a well defined structure. Parts of the brain have a reasonable well defined structure in smaller scale, in the region of 1mm. The connectivity at lower scale (low-level connectivity), however, is not well specified.
For example, When an axon from the Lateral Geniculate Nucleus enter the visual cortex, it is directed to some location in the cortex, to preserve the topographic mapping of the information. This is commonly given as an example of highly ordered connection (e.g. Shepherd (1990), p.395). However, in the cortex the axon branch to an `axon tree' which span more than 1mm squared, and is made of hundreds of branches (Shepherd (1990), p.396). Within this region the neuron forms contacts with only part of the neurons, depending on the type of the target neuron and location of its dendrites (mostly layer 4, in this case). This still leaves a choice of several tens of thousands of neurons to choose from (or even more), and the axon forms connections with few thousands of these. The selection of these few thousands is essentially stochastic.
The evidence for this is from comparison of the axon trees of different neurons, within the same brain and from brains of different animals of the same species. It is clear that the structure of the axon trees of individual neurons are not well specified. When it come to comparison between brains, it is not even possible to match individual neurons between brains, because they are too different. Since the low-level connectivity is different between individuals, it cannot be specified during development (by the genes or otherwise), and hence must be stochastic. Thus the connectivity of the brain is specified at the level of neuron populations, but not individual neurons.
It can be argued is that even though the connectivity as defined by the axon trees are not well defined, some process reduce the strength of irrelevant synapses, so they become insignificant. The problem with this possibility, however, is that synapse strength modification is mainly dependent on the activity of the individual neurons which form it. Since the initial low-level connectivity is stochastic, the activity of each individual neuron is stochastic (i.e. different in different brains), and the modification of the strength of each synapse is also stochastic.
In the Peripheral Nervous System (PNS) The individual connections are less stochastic, but even there, in most of the cases, the low-level connectivity is not well specified. For example, normally each muscle fibre is innervated by a single axon. Initially the fibre is innervated by several axons, and then there is a process of selection, which causes all of these, except one, to retract. Which axon stays is a stochastic choice, a conclusion which is again based on comparison between individual animals.
The stochastic nature of the low-level connectivity is almost never mentioned explicitly in neurobiological textbooks, probably because they don't believe this fact have any consequences. Instead, these books emphasize the order that exists in coarser resolution, many times in a confusing way.
For example, Nicholls, Martin & Wallace (1992, p. 341) ask: "What cellular mechanism enable one neuron to select another out of myriad of choices, to grow toward it, and to form synapses?". They later bring examples of specific connectivity. However, in all the examples which concern vertebrate Central Nervous System (CNS), the specificity is in the level of cell populations, rather than individual connections. Thus the answer to the question is that in the CNS a neuron does not "select another". Rather, it selects a region and cell types, which still leaves quiet a large spectrum for individual choices.
Maybe the worst example is in Kandel et al (1991). On page 20 appears, as part of the `principle of connectional specificity' which is supposed to be general property of neurons, this assertion: ".. (3) Each cell makes specific connections of precise and specialized points of synaptic contacts - with some postsynaptic target cells but not with others." The `specific connections' is true in some invertebrate systems, but it is simply false when applied to the vertebrate brain. In chapter 58, `Cell migration and axon guidance', the author tries to support this assertion, but all the examples of specific connectivity are from invertebrates. There are some examples from vertebrate, but they all show connectivity between cell populations, rather than individual cells. In addition, they are all about peripheral neural system, except one example from the spine of bullfrog. The vertebrate brain is not even mentioned in this chapter. It is obvious that this is because there are no example of specific connectivity there, but the text does not actually say this. The next chapter, `Neural Survival and synapse formation', discusses only neuron-muscle junctions, and there is no further discussion on the question of specific connectivity.
Disappointingly, This is true even in books that are explicitly about the computational aspect of the brain, e.g. Churchland & Sejnowski (1992), Baron (1987), Gutnick & Mody (Eds.)(1995). For example, in Gutnick & Mody (Eds.)(1995), Section iii is about "The Cortical Neuron as Part of a Network". However, only the chapter about modelling this network (Bush & sejnowski, 1995) mention individual connections, by saying that they assume them to be random in their simulations (P. 187). Even they don't actually discuss the point, and none of the other chapters in this section, or in the rest of the book, touches the point.
Even though the stochastic nature is not explicitly stated, it is clear from the data that is presented in these books that this is the case. One of the `distal' targets of the this article is to show the significance of this fact, and hence to convince neurobiologists (and others) to pay attention to it.
How are symbol tokens implemented?
Since it must be possible to store symbol tokens in arbitrary structures during computation (in other words, they are dynamic), they cannot be implemented by static features. This means that symbol tokens cannot be implemented by patterns of neurons and the connections between them, because these are static in the time scale of thinking. The dynamic features of the brain are the activity of neurons, and to some extent the strength of the synapses. Thus symbol tokens must be implemented by patterns of activity or strength of synapses, or both.
First, let us assume that patterns of activity are used, and see if they can fulfil the requirements for symbol tokens (section 2 above).
To store a token in some arbitrary structure, it would require to take the token, i.e. the pattern of activity, and propagate it to the appropriate `location'. Note that the need to propagate the pattern is true whether the `location' to store in is a physical location, or specified in any other way. The propagation must happen by the pattern of activity activating other pattern of activity, because there is no other way in which a pattern of activity can have any effect (in the time-scale of thinking).
However, in a system with a stochastic low-level connectivity, the pattern of activity at a time t+1 (p1)relates to the pattern of activity at t (p0) in a stochastic way, i.e. the transformation p0->p1 is not related in a consistent way to other transformations, either inside the same system or in other systems. In particular, the transformation is dependent on the pattern itself, in a stochastic way. Therefore each pattern propagates through a series of stochastic transformations, dependent on the pattern, and the result of propagation has no consistent relation to the original pattern. Thus it is not possible to propagate a pattern of activity, unless the way to do it have been specified in some way.
The stochastic transformation of patterns of activity is the most crucial point to grasp in the whole argument. It is worth noting here that this is in stark contrast with the situation in computers (more generally, artificial devices). In these, the connectivity is defined exactly and completely, and the relation between the activity at time t+1 and the activity at t is well specified. As a result, it is possible to propagate any pattern, to any place, without any restriction, and without changing the pattern itself.
The fact that propagation of arbitrary data on computer is not a problem is probably the reason that most of people intuitively assume that there is no problem to implement symbolic systems in the brain. The problem with this intuition is that it does not take the stochastic nature of the low- level connectivity in the brain into account.
It can be argued that the way to propagate symbol tokens is learned, or acquired by some other process. This, however, would require some part of the brain to know (in some sense) in advance the appropriate transformation of each pattern between each pair of locations, so it can direct the acquisition process (an "appropriate transformation" is a transformation such that the transformed pattern in the target location points to the same location the original pattern points to in the original location). In a system with stochastic low-level connectivity, there is no way to know this transformations in advance, so this is not a possible explanation.
Hence there is no way to propagate patterns of activity to arbitrary locations, so they cannot be used to implement symbol tokens. This also means that pattern of synapse strengths cannot be used, because the only way to propagate them is through activity.
The `cope out' solution, of regarding any pattern in the target location as a `copy' of the source pattern is obviously unacceptable, because this will not fulfil the other requirement of symbol tokens, i.e that they point (i.e. allow access) to some structure. Two patterns which are related to each other in a stochastic way cannot, in general, point to the same structure.
Thus we have reached the conclusion that there isn't any feature in the brain that can be used as symbol tokens. It is important to note that the argument is general, and is not dependent on a specific implementation details.
A possible objection to the argument above is that there may be a higher level of organization that may support symbol tokens. However, this is clearly not the case. Up to a level of ~1mm, the connectivity is clearly stochastic, so the argument in section 5 applies. This eliminates any implementation that relies on primitive elements which are smaller than ~1mm, whether localized or distributed. Thus implementing symbol tokens in higher levels of organizations means that it is based on primitive elements of dimension of 1mm or larger, and the total activity of the whole element, rather than its pattern at higher resolution, is the significant variable.
At that level, however, we can easily tell that there is no coherent connectivity between different elements. By `coherent connectivity' I mean a connectivity that allows one primitive element to affect separately other elements. To do this, the output from an element to other elements have to be separatable in some way, so it can be controlled separately. However, when we look at any 1mm square of the cortex, the neurons that send processes to other elements are all mixed up together, on a very small scale (tens of microns, at most). Because at that level the connectivity is stochastic, these neurons cannot be controlled separately.
It is important to note that this is true for all the connections inside the cortex. Hence it is independent of what are the actual elements that are postulated to be the base for implementation of the symbolic system, provided these elements are large (1mm or larger).
The lack of coherent connectivity means that the state of the element as a whole cannot be propagated inside the cortex. Instead, it is distributed approximately equally to all its neighbours, and sometimes to further elements by intracortical projections. In the neighbours, or the further elements connected by projections, it is mixed with local activity and activity from other neighbours, in a stochastic fashion. As a result, an element which is further away, and is not connected by a projection, can never `see' (be affected by) the activity in the original element. Instead, it always `sees' a stochastic mixture of activity of many elements. In that sense, the connectivity at the 1mm level is stochastic as well, and the argument of section 5 apply to that level as well.
A more coherent connectivity is seen outside the cortex, and in connections in and out of the cortex, mainly sensory input and motor output. However, these connections clearly are not coherent enough to transfer specific activity across the cortex (in the case of sensory and motor connections, it does not transfer activity across the cortex at all).
Hence, in the brain, there is no higher level of organization that can support symbol tokens.
The person as a whole is different qualitatively from the components of the brain in at least two fundamental properties:
The sensory systems differ from neurons in the brain in that their direct source of input (e.g the objects that emit the photons that the eyes receive) is very variable. In the case of the eyes, the direct source of input changes in time scales of several 10ms, both because the eyes move, and because objects in the world (or images of objects in TV) are moving. In other senses the changes are slower, but the even the slowest changes (in taste and smell) happen many times in each day.
In contrast, the direct source of information to any neuron, or a group of neurons, in the human CNS, is almost static (with very few exceptions). Once the development of the brain stops in the first year of life, the set of neurons that deliver information to any group of neurons in the brain is essentially constant for the rest of the life of this person. Major changes are rare, and do not happen during normal computation.
This means that argument that relies on the question of propagation of information is not applicable to the whole person as it is applicable to the brain itself. In particular, if we try to apply the argument in section 5 to the whole person, it will fail on the fact that every visual information in the world can be propagated to the eyes without any transformation (and the same for auditory and tactile information).
We don't know much about the details of this system, but we know it works. This system is capable of learning new skills, by trial and error learning, by analysing situations and deciding on the right actions, and by receiving communication from other people. These new skills are not limited to the capabilities that are inherited in the genetic make-up of the person.
Components in the brain do not have these learning capabilities, so they are limited in a way that the whole person isn't. The possibility of learning how to deal with symbol tokens inside the brain is discussed in section 5.
Since components of the brain do not have sensory input and the learning capabilities of the whole person, there are many tasks that the whole person can do that components cannot do. Thus, that the person can perform some task (e.g written communication, symbolic operations) does not prove that components of the brain can do it.
As stated, this statement is true. However, the implied conclusion, that we can ignore neurobiology, is false. While we cannot build models directly from neurobiology, the models that we do build have to be compatible with it.
All the `evidence' for symbolic systems is that they can be used to model parts of human behaviour. However, without considering implementation, symbolic systems are open ended, and can be used to model any behaviour whatsoever (as long it is expressed in symbols). Thus the fact that symbolic systems can be used to model parts of human behaviour does not show anything, and cannot show anything even in principle.
The only way to support symbolic systems from psychological experiments is to show that they can predict human behaviour, better than can be done by common sense and extrapolation of existing data. Symbolic models flatly fails to do that.
The only argument that I know for this is the density of information at the computation centre (Newell 1990, P. 75). This relies on the assumption that the brain has a computation centre. This is unfounded assumption, and is in conflict with evidence from brain damage.
Because in language the `value' of a `symbol' (i.e. the meaning of a word) is static in the time scale of thinking, handling language does not require any dynamic mechanism. Hence, it does not require symbolic systems as defined by symbolic systems theorists and modellers.
In addition, explained in section 7 above, the person as a whole can do things that components of the barin cannot, even in principle.
This does not solve the problem of the order at individual neuron level. There are two main alternatives:
It should be noted that none of the people that suggest additional features has come up with an evidence that is in conflict with the textbook neurobiology.
`Contents addressable memory' is actually a misnomer, used to describe pattern matching systems (Baron 1987, Ch. 2 and 3). These systems cannot implement association between a pattern and arbitrary value (which is what needed for implementing symbol tokens), unless the underlying structure already supports this (e.g. when they are implemented on computers).
This is the result of the hegemony of symbolic models. As long as implementation in the brain is not considered important, while simplicity and computer implementability are major considerations, symbol systems have a huge head start, and swamp all other models. The connectionist models (Rumelhart & McClelland, 1986) have succeeded to get noticed only when they became computer implementable, even though this was done by using a technique which is clearly irrelevant to the brain (backward error propagation). Until the evaluation of models will be based solely on brain related parameters (including human behaviour), realistic models of the cognitive functions of the brain will never get noticed, so there will not be an alternative.
The analysis has quite wide implications, and can be used to reject any model which relies on accurate low-level connectivity, even if it is not symbolic. However, I am aware of at least one model that can explain thinking and is compatible with this analysis (mine), and there are probably others. As explained in the previous paragraph, these are currently always in the shadow of symbolic systems.
The discussion in the sections 4-6 is fairly short and straightforward, based on well known facts, and leads to quite important conclusion. Why wasn't it noticed before?
First, it is not true that it was not noticed before. It is more accurate to say that it was never put so explicitly. For example, Robinson(1995) reaches a similar conclusion. The main difference is that in this text I explicitly discuss the neurobiological evidence that I used to reach my conclusion, while Robinson(1995) assume more or less the same without giving neurobiological evidence. This allows me to reach much stronger conclusion, and open the way for a discussion of the validity of the argument.
The connectionists (Rumelhart & McClelland, 1986) obviously see problems with symbolic systems, but I haven't seen anyone explicitly stating the impossibility of implementing symbolic systems. For example, Churchland and Sejnowski (1992) discuss the computational aspects of the brain, but do not touch the question of determination of individual connections. On the other hand, there were several efforts to merge the two approaches (e.g. Smolensky 1988).
Even some of those who believe in symbolic systems seem to realize that there are some problems. For example, The last item in the list of the main characteristics of the information- processing framework which is given by Eysneck and Keane (1989) (p. 9, see in the introduction above) reads: "This symbol system depends on a neurological substrate, but is not wholly constrained by it". It is not obvious what `not wholly constrained' in this statement means, but a plausible interpretation is that the authors realize that symbolic system cannot be implemented by neurons, but don't think it is an important point.
The question why the argument was not put explicitly still remains. I think the best explanation is that the argument requires understanding both neurobiology and implementation of symbolic systems. Neurobiologists do not realize the importance of the stochastic connectivity for theories of cognition, while computer scientists don't know enough neurobiology to realize that there is a problem.
The latter is not helped by the fact that neurobiological texts tend to strongly emphasize the coarser order, and use terms like "specific" and "precise" to describe it. For computer scientists, these terms imply that all the connections are well specified, the way they are in computers. Clarification of these terms would be a great help in understanding this domain.
In sections 4-6 it was shown that there is no way to implement symbol tokens in the brain, so it cannot be a symbolic system. This means that the brain is not a symbolic system, and theoretical analysis of symbolic systems is not applicable to it. However, it can still be argued that experimenting with symbolic systems is useful for understanding the brain. A typical argument would be: Both the brain and the symbolic models are information-processing systems, so experimenting with symbolic systems will tell us something about the brain.
This argument is flawed, because there is no general way to know which of the features of symbolic systems are applicable to information- processing systems in general, and therefore to the brain. Hence every feature that is found in symbolic systems have to be first tested on the brain (possibly indirectly through behaviour) before we know if it is applicable to the brain.
In theory, symbolic systems can still be used to direct research on the brain by suggesting hypotheses which are worth testing, and the argument in sections 4-6 is silent about this possibility. This, however, is a heuristic approach, which may or may not work. The experience with symbolic systems in the last ~40 years suggests that this approach does not work.
In general, a model is useful when it generates useful insights into the system under investigation. Symbolic systems clearly did not generate any insight into the neurobiology or anatomy of the brain, but it can be claimed that it generated useful insights into human thinking.
It is problematic to decide what is a 'useful insight', but a pluasible heuristic is that useful insights will be mentioned in the basic textbooks of the relevant subject. Inspection of textbooks in cognitive psychology (Eysneck and Keane 1989, Matlin 1994, Mayer 1992, Stillings et al 1995) and even more symbolic systems model specific books (e.g. Baars 1988, Johnson-Laird 1993, Newell 1990) does not show any insight into human behaviour or thinking which was generated by testing symbolic system models hypotheses. These books are full of models of human behaviour, but in all cases the behaviour was first noticed, or postulated based on the researcher's knowledge, and then modelled.
When it comes to Artificial Intelligent, the argument in section 4-6 is of no great consequences. Even if the brain is not a symbolic system, symbolic systems may still be the best way of building artificial systems. It is also possible in principle that there are living intelligent creatures somewhere in the universe that have thinking systems based on symbols.
When it comes to research about the way the brain works, the argument has crucial implications. It shows that symbolic systems are incompatible with what we currently know about the brain
Thus, these systems need very strong supporting evidence before they can be regarded as real candidates for modelling brain mechanisms. Since this evidence is lacking, symbolic systems do not deserve the attention they get, and researchers of the brain would do better to explore other avenues.
---------------------------------------------------------------------------
Baars, Bernard J. (1988), A Cognitive Theory Of Consciousness. Cambridge, UK: Cambridge University Press.
Baron, Robert J. (1987), The Cereblar Computer: An introduction to the computational structure of the human brain. Hillsdale, NJ: Lawrence Erlbaum Associates.
Brodar, Per(1992) The Central Nervous System, structure and function. New York, NY: Oxford University Press.
Bush, Paul & Sejnowski, Terrence J. (1995), `Models of Cortical Networks', in Gutnick, Michael J. & Mody, Istvan (Eds.) The cortical neuron. New york, NY: Oxford University Press, pp.174-189.
Churchland, Patricia S. & Sejnowski, Terrence J. (1992). The Computational Brain. Cambridge, MA:MIT Press.
Dowling, John E. (1992). Neurons and Networks: An introduction to neuroscience. Cambridge, MA: Harvard University Press.
Eysneck, Michael W. & Keane, Mark T. (1990),Cognitive Psychology A Student's Handbook. Hove and London: Lawrence Erlbaum Associates.
Gutnick, Michael J. & Mody, Istvan (Eds.)(1995), The cortical neuron. New york, NY: Oxford University Press.
Johnson-Laird, Philip (1993), The Computer And The Mind (second edition). London, UK: Fontana Press.
Kandel, Eric P., Schwartz, James H., Jessel, Thomas M. (Eds.) (1991), Principles of Neural Sciences (third edition). New York: Elsevier.
Matlin, Margaret W. (1994), Cognition (third edition). Fort Worth: Harcourt Brach Publishers.
Mayer, Richard E. (1992). Thinking, problem solving, cognition (second edition). New York: W.H. Freeman and Company.
Newell, A., & Simon, H.A. (1976), `Computer Science as Empirical Enquiry: Symbols and Search.' Communications of the association for computing machinery 19, pp. 113-126.
Newell, Allen (1990), Unified Theories of Cognition. Cambridge, MA: Harvard University Press.
Newell Allen and reviewers(1992), `Precis of Unified theories of Cognition'. Brain and Behaviour Science 15, pp. 425-492.
Nicholls, John G., Martin, Robert A. & Wallace, Bruce G. (1992), From Neuron to Brain (third edition). Sunderland, MA: Snauer Associates Inc.
Robinson, William S. (1995), `Brain Symbols and Computationalist explanation'. Minds and Machines 5 pp. 25-44.
Rumelhart, David E., & McClelland, James E. (1986), Parallel Distributed Processing. Cambridge, MA: MIT Press.
Shepherd, Gordon M. (Ed.)(1990), The synaptic organization of the brain (third edition). New York, NY: Oxford University Press.
Shepherd, Gordon M. (1994). Neurobiology (third edition). New York, NY: Oxford University Press.
Smolensky, Paul (1988) `On the proper treatment of connectionism'. Brain and Behaviour Science 11, 1-74.
Stillings et al. (1995), Cognitive Science an introduction. Cambridge, MA: MIT press.
Vera, A. & Simon H.A. and reviewers(1993a). `Special Issue: Situated Action.' Cognitive Science, 17, pp. 1-133.
Vera, A. & Simon H.A. (1993b), `Situated Action: A symbolic interpretation.' Cognitive Science 17, pp. 7-48.
Vera, A. & Simon H.A. (1993c), `Situated Action: Reply to Clancey.' Cognitive Science 17, pp. 117-133.