Virtual puppets developed by kinetic imaging professor help older adults feel more comfortable telling their stories

VoicingElder avatars mimic the motions of users. Credit: Virginia Commonwealth University

Semi Ryu had performed "Parting on Z"—her play about a farewell between lovers—a couple of times before that 2013 night in London. Ryu doesn't know what made this performance different, but something unexpected happened: She found herself sobbing in the middle of it.

09 mar 2017--"It was not acting," she said. "Something was touching me so deeply about it."
Ryu, an associate professor of kinetic imaging in the Virginia Commonwealth University School of the Arts, played both parts in the play about a man and woman ending their affair because of a difficult family disagreement. She physically played the female lover and spoke through an avatar, a virtual puppet, as the male lover.
"Sometimes I act as the female lover and then after that I activate a microphone on the avatar and then my voice becomes the male lover's voice," she said. "So we exchanged a dialogue of farewell and we drink together ... and then I took off my ring and placed it [in his hand to] keep it forever for your memory.
"It was extremely sad and I was really crying," she said. "I didn't expect to cry in front of this live audience and live performance setting and I [felt] so unprofessional. … I had to control myself but I couldn't control my emotions because there is something really overwhelming in this situation—[the] relationship between [the] virtual body and me."
She felt as if she was saying farewell to a part of herself, a symbolic farewell that made her think about how the self has different aspects.

VCUarts Associate Professor Semi Ryu, right, demonstrates how the on-screen avatar mimics the user’s motions. Credit: Virginia Commonwealth University

"There are multiple selves, but what is left out is maybe my social self," Ryu said. "Something defined by [society] as a role, as a woman … but I had to [say farewell to] my virtual body. … It was really about self-exploration and it made me cry."
She was embarrassed, but couldn't stop. The audience, equally moved, started crying as well. As the play ended, Ryu realized her primary emotion was no longer embarrassment, but rather relief. Speaking through the avatar had exposed suppressed feelings and given her emotional release.

'The puppet is speaking to me'

Having grown up in a Korean Confucian society, Ryu was discouraged from expressing emotion. Somehow this particular performance had revealed a lifetime of suppressed feelings. Ryu attributes those feelings of repression to han, the Korean cultural concept that encompasses feelings of oppression and grief with a strong hope and desire to overcome the impossible. Using the virtual puppet had allowed a different side of Ryu to emerge in the storytelling.
"I felt like something unresolved was resolved and then I felt this could also benefit … somebody else who has some kind of communication dilemma in society," Ryu said, realizing that han can be universal.
She immediately thought of the older adult population. As people get older, their image of themselves differs from how other people see them.
"They have ageless bodies happening in their mind but biologically it's degeneration and downgrading," she said. "There is an interesting tension happening, a kind of dilemma between what they want to be [and how they have been portrayed in the] media as powerless, hopeless and helpless individuals. Those images never fit [how] older adults perceive themselves. In this sense, they have han, and maybe we can … process a kind of ritual with older adults so they can start to release what they want to express."
Ryu was inspired by her brother, an occupational therapist who volunteered at area nursing homes and regularly talked about his amazing experiences working with older adults. She applied for the VCU Presidential Research Quest Fund in 2015 and started collaborating with Tracey Gendron, Ph.D., an associate professor in the VCU Department of Gerontology who also serves as the department's director of community engagement and research. Yet Ryu's motivation was always as an artist.
After years of using technology to explore the relationship between herself as the user and the virtual puppet, she became interested in what happens psychologically to users when they speak through the avatar.
"I was always curious about users' psyche…," Ryu said. "What kind of psychological relationship is possible between the user and avatar? That has been my primary research focus for a long time—more than 10 years. It seems to me it's such a different experience when I talk with or without 'puppet me.' [With the puppet], the puppet is speaking to me and so a different part of myself is emerging during the process of storytelling."

Developing VoicingElder

For years, Ryu has collaborated with Stefano Faralli of University of Manheim, Germany, to develop the interactive technology for her virtual puppetry. Together, they developed VoicingElder, an avatar system with four different development stages: childhood, teenage years, young adulthood and older adulthood, with four female and four male avatars. They implemented lip-sync technology so the avatars can speak in real time, giving the illusion that the puppet is the speaker. Users are then in the position of listening to their own stories even as they are telling them.

The VoicingElder program uses avatars from four different development stages: childhood, teenage, young adulthood and older adulthood. Credit: Virginia Commonwealth University

"It's an interesting psychology, looking at the puppet and the puppet is actually telling the story to them. They're going to get some distance from their story so they can easily talk about an emotional story … with more safe distance," Ryu said.
They also created a sentiment analysis algorithm, which detects emotional content from live speech and provides a virtual background to match. Happy stories have bright sunlight, while thunderstorms accompany angry stories. Other emotions detected are sadness, anxiety, fear and laugher.
"The VoicingElder project uses an artistic framework to combine technology and storytelling in an innovative way that actively engages older adult users in a virtual platform that provides the opportunity to share their life stories and experiences," said Gendron.
In March 2016, Ryu was ready to put the system to use. She worked with 10 residents of the Gayton Terrace Senior Living Center for three months, visiting every week or two. At the beginning, the seniors brought scripts with prepared stories, because they were self-conscious about speaking in front of each other. But around the fifth week, they began improvising and volunteering stories.
"It's wonderful because mainly your face is not there," said Connie Cole, 86. "It's a caricature there, so that takes the personal part away from it. … I think the fact that you're using the caricature and our voices and our stories blend beautifully for someone who's having dementia or some other elderly problem. … The fact the avatar's disconnected from me, it was easier for me to talk."
Vikki Fleming, resident program director at the center, said the project is a tool for residents to connect emotionally with important stories that are meaningful to them.

Connie Cole gestures as she tells a story through the VoicingElder system. Credit: Virginia Commonwealth University

"It's an interesting concept and it's fun to see residents experimenting with technology and getting to try new things because they want to continue to learn new things and this is one way of doing that," she said.
Ryu recorded the avatars at each session and at the end of the three months screened the film for residents, who all asked for a copy to share with their families.
"This can contribute to intergenerational relationships, so some story of the grandparents can be delivered to the grandsons … and it can work as a family legacy," she said.
Ryu is applying for a National Endowment for the Arts grant this spring and is pursuing National Institutes of Health grants as well. She has already presented the VoicingElder project at national and international conferences, including the National Center for Creative Aging in Washington, D.C., and Arts in Society as well as gerontology and drama therapy conferences and an upcoming Human-Computer Interaction International Conference. Many attendees have introduced the prototype system—which is very low-tech, requiring a simple setup: a computer with a decent processor, a connected camera and a wireless microphone—into their programs.
Ryu foresees extending the program to other populations and has begun collaboration with Tony Gentry, Ph.D., in the VCU Department of Occupational Therapy to apply the program to people with memory loss and PTSD.
"It's not just for older people but for [those who are] disabled, autistic [or have] communications difficulties," she said. "It can also be used to address diversity in society to increase awareness of racial, cultural and gender issues. It can help us be more empathetic to each other."


Provided by Virginia Commonwealth University