Four decades of asking whether automation can be a team player

Before the metaphor, machines replaced tasks

The first framing treated human and machine as separate components. A chapter by Klein and colleagues traces automation to the goal of replacing "whenever feasible any task currently performed by a human with a machine." It cites Fitts (1951) for the first systematic comparison of human and machine strengths, which became the discipline of function allocation.

Function allocation turned out not to be "a simple process of transferring responsibilities from one component to another," because automated assistance "changes the nature of the task itself."

Three problem statements from this period are later cited by the team-player authors.

Norman's argument reappears in the reference list of the 2004 Ten Challenges paper. The Royal Society meeting where it was presented ran 28-29 June 1989.

One system with two cognitive agents

Hollnagel and Woods's "Cognitive Systems Engineering: New Wine in New Bottles" is the root of the later team-player thread. Woods recalls writing it in the second half of 1981. A report version (Riso-M-2330) appeared in February 1982. The journal copy was received on 26 March 1982, revised that October, and printed in volume 18 of the International Journal of Man-Machine Studies, whose volume year is 1983.

"The man-machine interface is gradually becoming the interaction of two cognitive systems."Hollnagel & Woods, 1983

The paper defines a cognitive system as an adaptive system that "functions using knowledge about itself and the environment in the planning and modification of actions." It holds that a man-machine system "regarded as a whole is definitely a cognitive system." Woods later describes the core idea of CSE as "the base unit of analysis is the Joint Cognitive System."

Woods, 1985: who controls the interaction

"Cognitive Technologies: The Design of Joint Human-Machine Cognitive Systems" (AI Magazine 6(4)) is dated 15 December 1985 by the journal site. Woods cites it himself as 1986. Its abstract proposes that the key to applying computational technology is to conceive, model, design, and evaluate "the joint human-machine cognitive system." It names three design problems:

1. machine control of the interaction;
2. the danger of a responsibility-authority double bind;
3. the unsupported task of filtering poor machine solutions.

• team 0
• teammate 0
• cooperat 0
• collaborat 0

The vocabulary was "joint cognitive system" and "cognitive coupling." The article states that in its hypothetical computer consultant, "control of the interaction in this type of joint system resides with the machine." Later reviews still treat it as the origin of a lineage. McCarthy (2024) says the joint-activity perspective "dates back to Woods (1985)."

The evidence behind it: Roth, Bennett and Woods, 1987

"Human interaction with an 'intelligent' machine" studied an expert system built on the machine-as-prosthesis paradigm.

Technicians who were passive data gatherers degraded joint performance. Active technicians recognized unproductive paths and redirected the system. The machine, meanwhile, often could not recover once it was off track. Woods and Roth followed with "Cognitive Engineering: Human Problem Solving with Tools" (Human Factors, August 1988) and a handbook chapter the same year.

Two communities in the same years

NASA writes the metaphor into a design brief

In 1990 a NASA study "was initiated to provide guidance in designing intelligent systems that are effective team members in flight operations support." It was funded for the AI Division of the Office of Aeronautics, Explorations, and Technology. The resulting report, dated September 1991, examined fifteen aerospace intelligent systems.

"The human is the team leader of the cooperative effort and the intelligent system assists the human."Malin et al., NASA TM-104738, September 1991

The report's problem statement is "to make artificial intelligence (AI) systems team players." That requires treating "the coupling between human and intelligent system" as part of design itself. Both user and system are "agents" responsible for specific fault-management tasks, so the user becomes "an essential part of the system design." The authors caution against building a stand-alone problem solver and then "decorating it" with interface features.

This is a design philosophy, not yet a critique of existing automation. The concept of how the system assists the person constrains the architecture of the system itself.

1. 1990CHI '90 workshop on Computer-Human Interaction in Aerospace Systems. Malin and Schreckenghost present a framework paper. Remington and Shao present companion papers.
2. Sep 1991Making Intelligent Systems Team Players: Case Studies and Design Issues, Volume 1. The first verified title using "Team Players." Malin, Schreckenghost, Woods, Potter, Johannesen, Holloway and Forbus.NASA TM-104738
3. 1992Making Intelligent Systems Team Players: Overview for Designers. A follow-on, not the starting point. The month and contents were not retrieved.NASA TM-104751
4. Jul 1995A Guide to Developing Intelligent Monitoring Systems. A practitioner guide in the same series by Land, Malin, Thronesbery and Schreckenghost.NASA TM-104807

Team play becomes a diagnosis

Nadine Sarter, Woods's collaborator at Ohio State, supplied much of the empirical base. Her studies of pilots and flight management systems found that pilots had difficulty tracking automation status and behavior, and that opaque interfaces were implicated. Woods's 1994 keynote then states the team-player diagnosis directly.

"Systems that automate some aspects of cognitive work are often strong, silent and non-directable. In other words, automation often does not function as a team player within the larger ensemble."Woods, keynote to the Cognitive Science Society, 1994

The same keynote states: "Automation surprises are one class of symptoms of a deeper problem - the automated systems are not working as team players (Malin et al., 1991)." It also reframes automation design as "the design of a new human-machine cooperative system."

Woods's 1996 chapter "Decomposing Automation: Apparent Simplicity, Real Complexity" is described by a NASA review as treating automation as a "wrapped package" that changes tasks, cognitive demands and roles at once. Technology change often yields automated systems that are "strong, silent, clumsy, and difficult to direct," which "are not team players." O'Neill et al. quote Woods (1996) saying that introducing automated agents "is a change in the team composition."

The phrase "team play" in the Sarter and Woods titles was a research label for coordination breakdown. It was not yet a design checklist. Meanwhile, Klein's group was studying human teams and command-and-control cognition, and the 1997 Zsambok and Klein volume consolidated the NDM paradigm. That human-team strand is what the 2004 collaboration later brought to bear on automation.

The diagnosis becomes two design properties

"How to Make Automated Systems Team Players," by Christoffersen and Woods, first appeared as a 2000 Ohio State report and was published as a chapter in 2002. It starts by challenging a habit of explanation: "Traditionally, the assumption has been that if a joint system fails to perform adequately, the cause can be traced to so-called 'human error.'"

Digging deeper, adequate performance often depends on human "resourcefulness and adaptability" in the face of "uncommunicative and uncooperative machine agents." Joint-system performance "depends intimately on the ability of the human and machine agents to coordinate and capitalize upon the unique abilities and information to which each agent has access."

Secondary summaries describe the chapter as a literature review plus argument, with no original research. It covers the difference between information and mere data availability, human-human coordination cues as an analogy, ownership of the problem, the substitution myth, and the rejection of binary "more automation versus less automation" thinking.

Around it

Dekker and Woods's "MABA-MABA or Abracadabra?" (Cognition, Technology & Work, November 2002) argued that substitution-based function allocation cannot make progress, because automation's effects are qualitative. They proposed asking, "How do we make them get along together?" That reframes function allocation as coordination design.

Hoffman, Klein and Laughery (January/February 2002) listed among their lessons that "Machines should adapt to people, not the other way around," and that "the modern 'sociotechnical' workplace is characterized by changing collaborative mixes of humans and machines." Their 2003 "Macrocognition" paper is the explicit bridge between NDM and CSE. It says any CSE methodology "will be incomplete unless they include a description of the cognition that is needed to accomplish the work."

The chapter was applied almost at once. Lutzhoft and Dekker's maritime paper states, "The question is how to turn automated systems into effective team players," and argues that syntactic communication such as "I am now in DR mode" is not enough for common ground on intentions.

The Ten Challenges

"Ten Challenges for Making Automation a 'Team Player' in Joint Human-Agent Activity" (IEEE Intelligent Systems 19(6), 91-95) was written by Gary Klein, David Woods, Jeffrey Bradshaw, Robert Hoffman and Paul Feltovich. Klein Associates, Ohio State and IHMC prepared it under the Army Research Laboratory's Advanced Decision Architectures Collaborative Technology Alliance. It was adapted from a longer chapter on common ground and coordination, whose draft is dated 7 June 2004.

The draft states: "The ultimate desire of researchers and developers is to make automation a team player." The published paper moves the burden from a feature list for the machine to the conditions of joint activity. The draft chapter defines joint activity with two criteria: the parties "have to intend to work together, and their work has to be interdependent." All participants must then meet four requirements.

• REQUIREMENT 1Basic CompactAn often tacit agreement to coordinate and prevent breakdowns.
• REQUIREMENT 2Mutual predictabilityPartners can anticipate what each other will do.
• REQUIREMENT 3Mutual directabilityEach can redirect the other.
• REQUIREMENT 4Common groundMaintained by all participants throughout the joint activity.

The Compact is an agreement "to facilitate coordination, work toward shared goals, and prevent breakdowns in team coordination." It "is not a once-and-for-all prerequisite to be satisfied but rather has to be continuously reinforced or renewed."

1. 1

   "To be a team player, an intelligent agent must fulfill the requirements of a Basic Compact to engage in common-grounding activities."

   Accept joint goals and roles. Signal inability or unwillingness to participate.

2. 2

   "Intelligent agents must be able to adequately model the other participants' intentions and actions vis-a-vis the joint activity's state and evolution."

   Detect when partners are in trouble, on a standard path, or at an impasse.

3. 3

   "Human-agent team members must be mutually predictable."

   Act neither capriciously nor unobservably.

4. 4

   "Agents must be directable."

   People regulate agent behavior dynamically, for example through policies changeable without code changes.

5. 5

   "Agents must be able to make pertinent aspects of their status and intentions obvious to their teammates."

   Make targets, states, capacities, intentions, changes and upcoming actions obvious.

6. 6

   "Agents must be able to observe and interpret pertinent signals of status and intentions."

   Interpret teammates' signals, including pauses and pacing.

7. 7

   "Agents must be able to engage in goal negotiation."

   Convey current and potential goals so the team can adapt.

8. 8

   "Support technologies for planning and autonomy must enable a collaborative approach."

   Treat understanding, problem solving and execution as incremental, negotiable and tentative.

9. 9

   "Agents must be able to participate in managing attention."

   Direct attention in a context-sensitive way without overwhelming others.

10. 10

    "All team members must help control the costs of coordinated activity."

    Coordination "doesn't come for free." Agents must conform to operators' needs.

What the paper concedes

Agents "might eventually be fellow team members with humans in the way a young child or a novice can be," subject to "brittle and literal-minded interpretation of language and events."Klein et al., 2004

The authors say some agents might eventually enter a Basic Compact "with diminished capability," and describe their position as walking "a fine line" between AI that emulates human capability and human-centered computing that extends it. The quotation marks around "Team Player" in the title mark it as a metaphor. A neighboring 2004 chapter by Bradshaw and colleagues argued that agent autonomy must be "bounded by an explicit enforceable policy."

Autonomy, and the argument over the word

Hollnagel and Woods's Joint Cognitive Systems: Foundations of Cognitive Systems Engineering appeared in February 2005. A recent critique describes it as synthesizing more than twenty years of work from 1983 and moving the field "from interaction to integration."

Then the focus moved to autonomy. Bradshaw, Hoffman, Johnson and Woods (2013) critiqued the myths of "autonomous systems." Among their seven are "Autonomy is unidimensional" and "Once achieved, full autonomy obviates need for human collaboration." Johnson, Bradshaw, Hoffman, Feltovich and Woods published "Seven Cardinal Virtues of Human-Machine Teamwork" in November/December 2014. Woods's "The Risks of Autonomy" followed in June 2016. Only its title and date were confirmed for the report.

The term "human-autonomy teaming" gained currency in the late 2010s. O'Neill et al. note that the concept "is not new" and attribute its spread to advances in AI, machine learning and cognitive modeling. Their review lists Johannesen et al. (1994), Malin et al. (1991), Sarter and Woods (1995) and Woods (1994, 1996) as 1990s precursors.

Five positions, 2020-2024

McCarthy (2024) sorts the field into tool, teammate and joint-activity perspectives, and places Woods (1985) at the root of the third. The panel slides do not state a session date. The dates above come from the conference program, and Klein's statement comes from slides, not verified spoken remarks.

What the trail does not show

The report searched for a workshop devoted specifically to "automation as a team player" between 1980 and 2006 and found none. The thread emerged through NATO institutes, a CHI 1990 workshop, NDM conferences and an AAAI symposium.

Earlier uses of "team" for machines outside NASA and CSE, such as in AI planning or the Pilot's Associate programme, were not exhaustively checked. Hoffman mentions the Pilot's Associate and Intelligence Analyst's Associate as earlier "machine associates." The contents and month of TM-104751 remain unconfirmed, and Woods's "Origins" chapter is a pre-publication copy whose recollections are best treated as retrospective testimony.

Dates that need care

Item	Issue	Resolution
Woods, "Cognitive Technologies"	Cited as both 1985 and 1986	The AAAI site dates it 15 December 1985. Woods's 1994 keynote cites 1986.
Hollnagel & Woods, "New wine"	1982 report, 1983 journal, 1999 reprint	Report February 1982. Journal received 26 March 1982. Volume year 1983. Reprint 1999.
Malin & Schreckenghost TM-104751	Often cited as the first "team players" item	TM-104738 (September 1991) precedes it. Only the year 1992 is confirmed.
Christoffersen & Woods	2000 report versus 2002 chapter	Both exist. The 2002 chapter is the published version. The first author is Klaus, not Kim, Christoffersen.
Klein et al., "Common Ground"	Draft dated 7 June 2004, cited as 2005	The Ten Challenges paper cites it as forthcoming in 2005.
Billings, aviation automation	Cited as 1996 by Klein et al. and 1997 elsewhere	Both appear in the sources. The book itself was not checked.

Questions to bring to a "teammate" claim

These prompts are this page's reading aid, not part of the report. Each draws on the four joint-activity requirements and the challenges named beside it, turned into a question about a product.

• Basic Compact . Challenge 1Can it decline?

  Can the agent accept a joint goal and its role, and signal when it cannot or will not take part?

• Mutual predictability . Challenges 3 and 5Can the team see what it will do next?

  Does it make its status, intentions and upcoming actions obvious, and act in ways people can anticipate?

• Mutual directability . Challenge 4Can people steer it mid-task?

  Can responders regulate its behavior as it works, for example through policies that change without code changes?

• Common ground . Challenges 2, 6 and 10Who pays for the coordination?

  Does it notice when people are in trouble, interpret their signals, and conform to operators' needs, or do operators conform to it?