Discourse for Human-Computer Dialog

Motivation/Background

Autonomous systems, such as collaborative robots, can greatly extend the reach and capability of military forces and substantially reduce risk to humans, while substantially reducing manning requirements. However, autonomous systems still need human intervention and/or require interaction to compensate for incomplete information, disambiguate various elements of the dialog, repair misunderstandings, and adapt to unexpected changes. Two critical factors—robust linguistic information and a model of human cognitive behaviors—are needed to enable effective human-machine interactions and communications. To compensate for faulty or misunderstood data and information, humans frequently negotiate with each other during a dialog to repair those misunderstandings, to resolve ambiguities and provide clarifications where necessary. Likewise, when corrective or supplemental information is exchanged in ways similar to the way humans do in everyday dialog, the reliability, effectiveness, and usability of computer systems will increase, ultimately resulting in more efficient and effective task completion.

While humans are autonomous, they rely on collaborative techniques using linguistic information to communicate with other humans. Autonomous systems, no matter how intelligent or autonomous they may be, will still have to interact with humans at times, just as humans find it necessary to interact with each other. Such systems should, therefore, incorporate linguistic information and cognitive behaviors that characterize human-human communication. We will develop a system that acts and interacts in ways that humans expect, thereby lessening the cognitive load on the human user and making the system a more efficient and effective means of information interchange, as well as a collaborator in task accomplishment.

In our previous work in Coordinated Teams of Autonomous Systems (CTAS), we examined the underlying principles for using computational cognitive models as reasoning mechanisms for autonomous vehicles, the resulting decrease in cognitive load, and increasing team productivity. In this work, an architecture was developed for combining cognitive skills and reactive behaviors for autonomous vehicles. Previous work on natural language interaction also produced a parsing strategy that permitted natural language and gestural interactions with a mobile robot. We focused on spatial language and reasoning. However, interactions in this research were limited. The present research is based on the desire to leverage the previous cognitive, linguistic and communication work, to integrate these efforts further, making them more robust, and to produce an autonomous system that is capable of interfacing and dialoging with humans in a collaborative and cooperative manner in both proximate and distal environments.

Objective

The goal of this project is to find a possible solution to a problem that constantly plagues interactive autonomous systems, and AI systems in general, particularly those that attempt to incorporate context, dialog issues, and cognitive capabilities; namely, the problem of having to use a domain-specific model of a particular application.  Existing systems simply cannot cope with the amount and depth of information necessary to interact fluently with human users (and warfighters) in even one domain, let alone be flexible enough to interact with humans as richly and widely as humans do within domains and across domains.  Consequently, existing systems settle for constrained and sometimes highly specific and narrow knowledge and information. This, unfortunately, causes problems for real-time applications when information previously not entered into the system becomes crucial in solving a problem.  Solving this “brittleness” problem by enabling a more robust interaction between systems and users—particularly those that attempt to incorporate context, dialog issues, and cognitive capabilities—is the main goal of this research.

The research will augment previously researched and integrated components of an autonomous system  (specifically a mobile robotic system and team of mobile robots) with a dialog system that behaves in a cognitively plausible fashion when communicating and interacting with people. While domain-specific information will be used, this research will not rely on it to solve all of the problems that confront natural interactions, for, if anything, reliance on domain-specificity is the very cause of "brittleness."  On the other hand, we will use this information initially, but when novel or unforeseen situations arise and the domain-specific information does not resolve an ambiguity, for example, the system will engage in a negotiation to determine the unknown elements of the dialog with which to proceed. The new system will, furthermore, recognize when an agent is experiencing atypical behavior, and it will suggest possible solutions to rectify the situation, or seek additional information from human partners.  These capabilities will be built into a dialog management component integrating meta-cognitive and linguistic capabilities.

Our work directly addresses the aforementioned issues of required cognitive skills and natural interactions between the users of the autonomous systems and the autonomous systems themselves. In this work, we extend and apply previously discovered principles in hybrid reactive/cognitive architectures to embodied autonomous systems that work collaboratively with humans, and demonstrate the resulting system in a relevant task and environment.  While we will be investigating interactions that occur when autonomous systems and humans are in close proximity to each other, another important aspect of our work will focus on distant communication between autonomous systems and humans. Intelligent autonomy, coupled with more robust cooperation, collaboration, and interaction, will enable human operators to act as supervisors for individual unmanned vehicles and/or teams of cooperating unmanned vehicles.  It will also provide an enabling technology for affordable unmanned vehicles with a high degree of reliability, survivability, and mission effectiveness.

Technical Approach:

The research develops algorithms for dialog and multi-modal interaction that incorporate more general knowledge to enable more robust interaction between the autonomous system and humans.  For this purpose, we are collaborating with researchers at the University of Maryland, College Park and Baltimore Campuses.  Our combined research on meta-cognition and natural language understanding will be tested on platforms located at NRL.  The resulting system incorporates linguistic and cognitive behaviors characteristic of human-human communications promoting collaborative interactions between human users and autonomous systems.  In one phase of the work, when both users and platforms are proximate, the research will focus on natural interactions, dialog management, and gestures employed for disambiguation.  We will be leveraging other funded NRL work on the use of natural and symbolic gestures for communication in this phase of the research.   For example, linguistic and gestural capabilities will be combined in proximate situations so that an utterance such as “Put this over there” can be interpreted correctly and appropriate action taken.  Not only must the system be able to parse and interpret the utterance, resolving the ambiguity of what is being referred to, but the gesture component must also be able to resolve the visual referent and location.  If either component has difficulty in resolving either the referent or the location, a session of negotiation and repair must be entered into so that the system can acquire the necessary information to complete the request.   After successive testing, we can then generalize negotiation strategies across several/various domains.

In the second phase of the research, we will address situations in which both users and platforms are removed from each other by some distance, and in many cases do not have a direct line-of-sight and may not have any direct awareness of each other’s local environment.  Such conditions can influence the kinds of interactions, negotiations, repairs, etc. that the system and the human user must utilize in order to achieve effective and efficient communication.  For this research, a simulated world (MarsWorld) will be created at order to foster distance communication between humans located at the university and robots located at NRL. We will investigate how dialog and adaptation in communication occurs when some distance separates human and robotic teams.  For example, human users will need to formulate their commands and queries based on images sent back by the robot from and about the hypothetical MarsWorld.  Further, since the robot is in a discovery scenario, the amount of information that can be pre-programmed into the system will be minimal.  A great deal of the interaction that will occur in this scenario will depend on what is established in real time through experience and then negotiation.  Human users will have to react to what is being encountered in real time by the system, and the system will have to adapt and interact with the human users based on newly encountered objects and events.  Additionally, we will be able to simulate corrupted data, such as when transmission of information is either garbled or lost.  Again, the system and the user will have to repair the faulty communication for continued interaction. 

As a task domain, we will focus on spatial cognition/navigation and human-robot communication. Tasks will include both proximal and distal communications demonstrating interactions of one or more robots in two separate but related phases of the research.  In both domains, knowing how the person thinks and/or what type of behavior is expected in a given situation (e.g., knowing where a person or an object is or might be, knowing what a person might see, and knowing what to do when something unforeseen happens) is critical to successful task completion.  Being able to act upon such contingencies requires a more developed dialog component that is capable of the meta-cognitive and linguistic (dialog) skills mentioned here.

Contact:

Dennis Perzanowski, Ph.D. (Co-PI)

Section Head, Interactive Systems

Code 5512

Naval Research Laboratory

Washington, DC  20375

Phone: 202-767-9005              Fax: 202-767-3172

Email: w5512 at aic.nrl.navy.mil

J. Gregory Trafton, Ph.D. (Co-PI)

Section Head, Intelligent Systems

Code 5515

Naval Research Laboratory

Washington, DC 20375

Phone: 202-767-3479              Fax: 202-767-3172

Email: w5515 at aic.nrl.navy.mil

Alan Schultz, (Co-PI)

Director, Navy Center for Applied Research in Artificial Intelligence

Code 5510

Naval Research Laboratory

Washington, DC 20375

Phone: 202-767-2684              Fax: 202-767-3172

Email: w5510 at aic.nrl.navy.mil