The modeling of multiple-task behaviors has been explored extensively in EPIC (Meyer & Kieras, 1997), and SOAR (Newell, 1990; Laird, Newell, & Rosenbloom, 1987) has also been adapted to model multiple-task behaviors. In particular, Meyer and Kieras (1997) report considerable success in developing a production rule-based model of the psychological refractory period (PRP) procedure. The basic components of their model are a cognitive processor comprised a production rule interpreter with inputs from long-term and production memory, and a working memory, with auditory and visual processor inputs that interact with the production rule interpreter. The model relies heavily on a centralized, synchronous production rule framework. A production rule-based executive process administers the task scheduling strategy for regulating competing task execution. The implementation is just one of a theoretically infinite number of computational frameworks that might give rise to the desired human-like multiple-task behaviors. In building the OMAR framework, particular attention has been paid to developing multiple-task behaviors from an assembly of concurrently operating functional centers absent the executive or central controller.

The motivation for this approach to human performance modeling, derived from a selective a reading in several disciplines, is outlined in the Section 2. Section 3 provides background on the aircrew/ATC domain and describes implementation of the models of aircrew in-person conversations and their interruptions by ATC directives. Section 4 provides a description of the computational elements for constructing OMAR human performance models.
 
 

Over the years, experimental psychologists have conducted extensive experiments providing a wealth of interpreted data, philosophical discussion dates back through the millennia, and more recently, cognitive neuroscience and clinical studies have provided electroencephalography (EEG), magnetoencephalography (MEG), positron emission tomography (PET) and functional magnetic resonance (fMRI) images of the brain at work (Posner, 1993; Raichle, 1994) identifying the locus of specific perceptual, cognitive and motor functionalities. There is a vast literature, but no road map for modeling human performance. The computational architecture for models developed in the OMAR framework differs from that of EPIC and SOAR in fundamental ways: (1) stimuli impinge directly on, activate, and propagate through long term procedural memory—the knowledge of how to do things (see Figure 1); (2) tasks, skilled cognitively-driven behaviors, are accomplished through the coordinated actions of function-specific procedures representing the contributions of specific brain areas; (3) to the extent that the resulting behaviors may be considered intelligent, that intelligence is the product of the pattern matching implicit in the changing sensitivities of the network of procedures as stimuli evoke responses at network nodes; (4) task contention outcome, rather than being determined by a central executive, is mediated on a pair-wise basis among contending tasks. The foundations for these choices are briefly in this section.
 

The now routine accounts of the early PET studies and the more recent fMRI studies portray the execution of each experiment as being the product of a small number of brain centers—small areas of activity at widely dispersed major brain centers. Posner, Peterson, Fox, and Raichle (1988) draw on the evidence of a series of their PET experiments to suggest that "the mental operations that form the basis of cognitive analysis are localized in the human brain." To further support their assertion of the localization of cognitive function, they cite studies of patients with lesions and their related deficits. Based on these studies, the basic architectural framework seems reasonable well established. Tasks, made up of perceptual, cognitive and motor components, appear to be accomplished through the collective actions of small specialized areas of activity that take place in each of several widely dispersed brain centers.

On a closely related but more conjectural plane, Edelman (1987) discusses the psychological functions of "development, perception (in particular, perceptual categorization), memory, and learning" and how they relate to the brain. Edelman (1989) extends his analysis to consider "perceptual experience—the interaction of memory with the present awareness of the individual animal," that is, perceptual awareness and conscious experience. He describes neural maps as the ordered arrangement and activity of large groups of neurons as distinct from single-neuron connections. They are highly and individually variant in their intrinsic connectivity. Changes in the behavior of the network are the result of changes within particular populations of synapses. "These structures provide the basis for the formation of large numbers of degenerate neuronal groups in different repertoires linked in ways that permit reentrant signaling" (Edelman, 1987, p. 240) where, in degenerate systems, functional elements in a repertoire may perform more than one function and a function may be performed by more than one element (Edelman, 1987, p. 57). Reentry is a basic mechanism suitable for synchronizing the neuronal activity across the mappings at diverse hierarchical levels. Global mappings have a dynamic structure that reaches across reentrant local maps and unmapped regions of the brain to account for the flow from perception to action. Motor activity, an essential input to perceptual categorization, closes the loop.

Taken together, Posner et al. and Edelman present a picture of the execution of a task as the coordinated activities of small, specialized local sites operating at several remotely located brain centers. In Edelman’s terms, reentrant signals link the components within the local sites, while global mappings connect the activities of the broadly dispersed major centers. The OMAR models attempt to emulate this basic computational framework. That the smallest operating units are large groups of neurons is taken as license to build the models at a symbolic level.

Edelman, referencing Bartlett (1932), goes on to present a view of memory as process. For him, memory is the "ability to categorize or generalize associatively" (Edelman’s italics, 1987, p. 241). Categorization occurs at the level of a global map and is degenerate. Edelman is well aware of the distinctions between declarative and procedural memory, but he is also quick to point out that these distinctions may be less than generally assumed. He suggests that there may be a procedural base supporting declarative memory.

In Edelman’s view of memory as process, perception, categorization, generalization, and memory are closely linked. "Memory is a form of recategorization based upon current input; as such, it is transformational rather than replicative" (Edelman, 1987, p. 265). Memory is an active process of classification leading to recategorization and, thus, a partitioning of the world that is presented as one "without labels." Storage, to the extent that it exists, is one of procedures for mapping inputs to responses; hence, full representations of objects are neither stored nor required: "It is the complex of capacities to carry out a particular set of procedures (or acts) leading to recategorization that is recollected" (Edelman’s italics 1987, p. 267). This view contrasts sharply with memory cast as data residing in a data base, where content is passive, references are made to it, it may fade with time, and in the case of short-term memory, new memories may reinforce or replace existing memories. In such schemes, something operates on memory as data, reinforcing some of it and degrading other parts of it. In the models developed, memory is an integral part of the processes that employ it.

Production rules have played a central role in cognitive modeling systems (e.g., EPIC and SOAR). As a computational tool they are convenient and have been exploited to build some simple models of human learning (Laird, Rosenbloom, & Newell, 1986). On the other hand they have all the trappings of an executive—in their conditions they may have oversight of one or more active tasks and memory stores, while in their actions they may initiate, interrupt or terminate tasks and execute operations on memory or other capabilities central to the functioning of a model. The existence of such a brain center is certainly open to question. Neither the clinical research that has extended over many years, nor the more recent PET and fMRI imaging studies have identified a potential location for such an omnipotent brain function. Dennett (1991) expresses considerable concern over such homuncular theories. Centering his discussion around the metaphor of the Cartesian Theater where everything comes together, he suggests that theater provides catchall for awkward elements leading to the failure to address difficult underlying questions. Dennett offers a Multiple Drafts model of consciousness in which "all varieties of perception—indeed all varieties of thought or mental activity—are accomplished in the brain by parallel, multi-track processes of interpretation and elaboration of sensory inputs." He speaks in terms of an on-going process of "editorial revision." Dennett reinforces parallel processing as essential to modeling task execution and reminds us to be firm in our disavowal of homuncular concepts in modeling human performance. Following Dennett’s admonition, the models do not employ an executive or controlling process.
 
 

3.1 Aircrew/Air-Traffic-Control Communication

Conversation on the flight deck between the captain and first officer is the more typical person-to-person conversation of everyday life, but it is subject to interruption by ATC communication. The interruptions may take the form of directives addressed to their aircraft or to another aircraft under control of the ATC. In the interests of clarity and efficiency, most of the aircrew/ATC communications are highly stylized exchanges initiated with a directive or a question and completed by an acknowledgment of the directive or a response to the question. Established policy plays an important role in these exchanges. Verbal transactions between aircrew members must be suspended for ATC-initiated communication, even when the communication is directed to another aircraft. The crew must remember to resume the transaction on completion of the ATC interruption. An aircrew member wishing to initiate a communication with an ATC must wait for the completion of an on-going transaction before initiating the communication. Typical directives to an aircraft might involve changes in heading, altitude and airspeed. The crew member handling the communication will acknowledge the communication and monitor the execution of the directive by the other crew member. Policy dictates cross checking—each crew member’s expectations of exactly what the other crew member will do must be confirmed or the exception addressed. The domain is a fertile one in which to examine multi-tasking.

Figure 2 provides an example of a aircrew conversation interrupted once by ATC directive that they must attend to and then by an ATC directive for another aircraft causing them to further delay the resumption of their conversation. Jim, the captain of flight DAL100 has just initiated a conversation with his first officer Joe, when they are interrupted by a communication from the ATC. Jim acknowledges the ATC directive and Joe, having initiated the flight level change, resumes the in-person conversation, but it is immediately interrupted by another ATC communication, this time directed to Jane and Bill’s flight UAL10. Jim must again pause before once again picking up the interrupted communication with his first officer.
 

3.2 Modeling Contention Between Tasks and its Resolution

The OMAR simulator is an event-based simulator to accommodate the particular and varied time steps at which each of several concurrent processes can be expected to operate. An aircrew member may initiate the action required by the change-altitude portion of an ATC directive (perhaps by setting the new altitude on the mode control panel (MCP)), while continuing to attend to subsequent speed and heading directives. These activities go on concurrently, each implemented as task with appropriate time frames. Established policy dictates that an in-person aircrew conversation be deferred at the onset of an ATC communication. In OMAR, rather than being the subject of a rule-based decision, established policy-driven behaviors are viewed as a cognitive form of automaticity (Logan, 1988). The priority of the aircrew "listen to the ATC" task is simply higher than the aircrew "in-person conversation" task. The onset of "listen to the ATC" task interrupts the "in-person conversation" task based on its priority. In like manner, the aircrew "listen to other ATC transaction" has higher priority than "initiate ATC communication." An aircrew member will wait for the completion of an on-going party-line transaction to complete before initiating a new transaction. Policy-based decisions are viewed, not as the product of a centralized executive process (for which there is little evidence), but rather as the outcome of contention among the particular subset of tasks competing to execute in response to events initiated externally or internally. Events, be they externally or internally initiated, impinge, not on short-term memory, but on activated long term memories in the form of schemata with well established policy-based priorities. In acting on an the initial directive of an ATC directive while attending to subsequent directives there may be no contention, but when contention is present, as in initiating a party-line communication, policy-based priorities mediate action. Given that several dispersed functional components may contribute to each of the contending tasks, when the contention is resolved, the component functions must act in accordance with the resolution.

3.3 Modeling Three Functional Areas in Listening

As currently implemented, the listening tasks themselves have two additional components. The listening task complex is initiated by a verbal message. A separate procedure for processing the auditory input, activated through a different path, awaits the onset of an auditory message. Shortly after the auditory message onset, a speech understanding procedure is invoked to develop the propositional form that the attended cognitive task will operate with. In the simulation, the message is simply conveyed as an object and the auditory and speech understanding processes are just time-consuming stubs. The development of the listening task posits three distinct functional areas of processing. Separate goal and subgoal trees set up each of the functional capabilities. The onset of the auditory message initiates the processing with the activities of the three functional areas coordinated through a series of messages, or signals, as they are termed within OMAR. The functional areas and signals are a specific symbolic analogue of Edelman’s (1987) re-entrant nets. The procedural bias in the modeling approach is taken a step further. Short-term memory (Martin, 1993), rather than being treated as a faculty in its own right, is modeled as a set of distinct capabilities distributed (Schneider & Detweiller, 1987) among a family of functional procedures. The motivation for this approach is once again derived from Edelman’s (1987) process view of memory and reinforced by his references to Bartlett (1932). Auditory memory of a verbal message is a component of the auditory process, while the propositional memory of a message is a component of the language understanding process. Their persistence, clearly different for each modality, is envisioned as, but not yet implemented as, a product of the persistence of their enclosing procedures.

Given a task, postulated to be the product of contributions from several dispersed functional capabilities, the event of the interruption of that tasks can be explored in many ways. Exactly what occurs in the auditory and language understanding components of the "listening" task, indeed, even that this is a correct functional component allocation is not precisely established. The model makes explicit a proposal for how the component functionalities might be coordinated during task execution and more interestingly, when a task is interrupted.
 
 

4.1 Concurrent Task Execution and Mediating Task Contention

The contention between procedures is a more complex issue. At least three levels of contention can be envisioned. Thoughtful, attended deliberation can lead to the selection of one course of action over another. The deliberation process that should probably be explicitly modeled is not addressed here. The concern in the current effort has been with the simpler cases of policy-driven decisions as described in the aircrew scenarios above and the still simpler contention based on access to particular, identifiable resources. The contention between tasks can occur high in the goal tree as in the contention between "listen to ATC" and "in-person conversation" or near the leaves of the tree as in contention between tasks for access to the dominant hand for a skilled manual operation. All SCORE procedures are SFL concepts and may be classified as tasks that contend with particular other tasks (as in the case of "listen to ATC" and "in-person conversation") or with other instances of their own class (as in the case of the dominant hand requirement). A new task about to run must either establish that it does not contend with a running task or that if it does, that it has sufficient priority to block the execution of the running task. If a new task has sufficient priority, it begins execution and execution of the contending task is halted until execution of the new task has completed. At this point, barring intervening events effecting these tasks, the original task resumes execution. If the priority of the new task is not sufficient to block the running task, it must wait for the running task to terminate. Tasks priorities are computed dynamically and tasks contention is revisited as priorities change.

4.2 Pattern Matching in Coordinated Functional Component Execution

Signal based coordination of executing procedures bears a strong resemblance to a data flow architecture (Arvind & Culler, 1983), while differing significantly from object-oriented message passing. A procedure issuing a signal continues operation and does not receive any returned values. The issuing procedure has no knowledge of the other procedures that have enqueued on the signal. There may any number of procedures enqueued on it or none at all. A given procedure may enqueue on a signal once or in each of two or parallel threads employed to explore different patterns of events that each include this particular event.

Signal processing forms the core of the implementation of the functional capabilities that make up the multiple-task model of human performance. Signals are the representation of external events that trigger the model’s human receptors (eyes and ears in the current implementation) and they are the basis for the subsequent internal cascade of events produced in developing the coordinated multiple level response to those external events. The network of activation of with-signal forms changes rapidly over time to reflect the occurrence of external events and the many procedures representing the functional capabilities that combine to form tasks governing the response to those events. The changing network of active procedures, each sensitive to particular external or internal events, forms a pattern matcher that determines the behaviors of the model. The proactive component of the behaviors is provided by the goals and subgoals that govern the initial activation of the procedures. Each of the behaviors in the performance of a task is the result of a mix of the proactive and reactive components of the task. The signal-driven activation of network nodes representing functional capabilities provides an emulation of Edelman’s reentrant maps.