Torin Hopkins, PhD

AI & Music · Health & Learning

LivePoem

Improving the learning experience of classical Chinese poetry with AI-generated musical storyboards.

Read the paper (PDF)Code
LivePoem

LivePoem generates musical storyboards (illustrated storyboards with AI-composed background music) to help learners understand classical Chinese poetry. In a study with 25 learners, the storyboards kept the learning outcomes of textbooks while making learning more engaging and enjoyable.

Motivation

Textbook reading dominates classical poetry education in Chinese-speaking communities, but extensive text-based learning can lead to boredom and disengagement. Research in the learning sciences suggests that audiovisual media, such as music, pictures, and video, can support language skills, comprehension, and imagery.

The system

LivePoem uses a pretrained diffusion model to generate storyboard images and a Transformer-based, prosody-aware poem-to-melody generator to compose the music. Both were validated with standard objective metrics.

In a within-subjects study, 25 non-native Chinese learners compared textbook reading with musical storyboards. We measured learning with standardized reading comprehension tests and measured experience with the Self-Assessment Manikin and a thematic analysis of open-ended feedback.

Key findings

  • Musical storyboards retained the learning outcomes of textbooks.
  • Learners found the storyboards significantly more pleasant and engaging.
  • We recommend combining traditional textbooks with multimedia materials in Chinese poetry teaching.

Publications

  1. LivePoem: Improving the Learning Experience of Classical Chinese Poetry with AI-Generated Musical Storyboards

    Qihao Liang, Xichu Ma, Torin Hopkins, and Ye Wang

    International Joint Conference on Artificial Intelligence (IJCAI), Montreal, Canada, 2025. 4% acceptance rate

    AbstractPDFCode