Get to the heart of AI! Developing an evaluation platform for AI to understand emotions

Professor Shirataka of Faculty of Data Science received the Best Paper Award at the International Conference on Innovative Computing, Information and Control (ICICICIC2025) held in Kitakyushu from August 26 to 29, 2025. This award was highly acclaimed for groundbreaking research to scientifically elucidate how far AI can understand human emotions.

❓ Why did you need this study?

AI technologies, especially large language models (LLMs) such as ChatGPT, are becoming an integral part of our lives. However, the question "Does AI really understand people's feelings?" has not yet been fully elucidated.

Previous research focused on individual evaluation of the performance of specific AI models, and it was difficult to compare multiple different AIs side by side. This is because the API specifications provided by each company were different, so it was not possible to evaluate fairly in a unified manner.

In order to solve this problem, Prof. Shirataka developed an evaluation platform that can be fairly compared with AI models of different companies on the same playing field. As a result, large-scale and unified experiments have become possible, and the overall picture of AI's emotional comprehension capabilities, which were previously unknown, was revealed.

How did you proceed with your research?

In this study, we used the evaluation base we developed to conduct large-scale comparison experiments on 36 leading AI models from seven of the world’s leading companies (OpenAI, Google, Anthropic, xAI, DeepSeek, Meta, Alibaba).

In the experiment, AI was asked to read Japanese literary works (works by Hisasaku Yumeno and Kotaro Takamura) and evaluated "How much fun, surprise, sadness, and anger of this work" on a 100-point scale. In addition, we introduced a unique method of giving AI four different "personalities (personas)" such as "university students", "literary researchers", "poeters", and "robots" to analyze emotions from each perspective.

Discovered by surprise!

This large-scale experiment revealed the following important facts:

1. The way you feel emotions depends on AI!

Emotional evaluation scores differed greatly for each AI model. This suggests that AI has a variety of ways to interpret emotions, and that no AI has fully captured all human emotions.

2. Reliable AI, Still Developing AI

Models from Anthropic, xAI and DeepSeek showed perfect reliability without failing to evaluate them in this experiment. On the other hand, some models did not return the evaluation numerically, and it was also found that there were some developing parts. This is an important indicator of which model you should choose when you actually use AI in your business or service.

3. AI’s thinking patterns are clear!

By allowing AI to explain why such feelings were evaluated, the pattern of "thinking" was revealed. I found that one model was interpreted from a different perspective even if I read the same work, such as in-depth analysis using literary jargon, while another model responded in an expression close to the reader's emotions.

This research

The evaluation base developed in this study crosses the barriers of different APIs offered by each company, making it a groundbreaking "measuring" that measures AI's ability to understand emotions on a unified basis. This is expected to accelerate the development of AI that is more human-like and empathetic.

In the future, these results will be applied to the development of services tailored to the emotions of individual users (AI that supports mental health, AI that enriches reading experiences, etc.). In addition, TUFS students will be able to experience such cutting-edge research and acquire the ability to create the next generation of AI technologies on their own.

Comments

"I am very honored to receive such a wonderful award. This research is the first step toward AI becoming a partner that enables richer communication, rather than just a tool. I would like to express my sincere gratitude to the collaborators who supported my research and the students who continue to study with a spirit of exploration every day.

Data science is an academic study that considers how the power of AI can be used for the future of society. We hope that the results of this research will resonate in the hearts of everyone aiming to become a data scientist in the future, and will be a catalyst for learning at our university."

Paper information

• Title of the paper: Development of an Experimental Platform for Evaluating LLM Emotion Understanding
• Author: Naruki Shirahama (SHIRAHAMA Naruki), Naofumi Nakaya (Nakatani), Satoshi Watanabe (Shi Watanabe)
• Award Name: Best Paper Award

[Contact information]
Shimonoseki City University
Corporate Planning Department Public Relations and Branding Strategy Division
TEL. 083-253-8967