More than Task Performance: Developing New Criteria for Successful Human-AI Teaming Using the Cooperative Card Game Hanabi
General
Art der Publikation: Conference Paper
Veröffentlicht auf / in: CHI EA '24: Extended Abstracts of the 2024 CHI Conference on Human Factors in Computing Systems
Jahr: 2024
DOI: https://doi.org/10.1145/3613905.3650853
Authors
Patricia Wollstadt
Christiane Wiebel-Herboth
Abstract
As we shift to designing AI agents as teammates rather than tools, the social aspects of human-AI interaction become more pronounced. Consequently, to develop agents that are able to navigate the social dynamics that accompany cooperative teamwork, evaluation criteria that refer only to objective task performance will not be sufficient. We propose perceived cooperativity and teaming perception as subjective metrics for investigating successful human-AI teaming. Corresponding questionnaire scales were developed and tested in a pilot study employing the collaborative card game Hanabi, which has been identified as a unique setting for investigating human-AI teaming. Preliminary descriptive results suggest that rule-based and reinforcement learning-based agents differ in terms of perceived cooperativity and teaming perception. Future work will extend the results in a large user study to psychometrically evaluate the scales and test a conceptual framework that includes further aspects related to social dynamics in human-AI teaming.
Downloads
Download Publication![](/sites/default/files/extern.png)