digital empathy

human-centered benchmark for measuring empathy in human-AI interactions

Empathy Scorecard evaluates agent empathy across 7 dimensions- Affective Understanding, Cognitive Understanding, Response Appropriateness, Prosocial Behavior, Curiosity and Attention, Contextual Understanding, and Conversational Memory.

Project Overview

The rising popularity of AI chatbots for mental health has led to increasing concerns about the lack of empathy in conversational AI agents. However, prior work on emotionally intelligent agents has focused on simulating human empathy, which has raised ethical concerns about deception and inauthenticity. In contrast, our work investigates how empathic human-AI interactions can benefit humans.

Approach

We proposed a multidimensional framework of perceived empathic behaviors that includes 7 dimensions that are inspired by theories of cognitive and affective empathy in psychology but reframed as interactional, behaviorally observable phenomenon in AI agents.

To understand how individual and contextual differences influence perceptions of empathy dimensions in various AI systems, we conducted a study with 151 participants who engaged with one out of 4 different AI agents and assessed their perception of empathic behaviors in that agent. From each conversation, we obtained first-person and per-turn labels of perceived empathic agent responses.

In our study, participants will chat with an AI agent and then rate the agent's responses as good, neutral, or poor examples of Digital Empathy.

Outcomes

The main contributions of this work include our 7-dimensional digital empathy scale and an annotated dataset of empathic behaviors in conversational AI (paper in preparation).