Tuning the Face: Modulating Facial Expressions for Realistic Self-Avatars in Virtual Reality
Yang Lu,
Jiamu Tang,
Jiankun Yang,
Shijian Luo,
Chenliang Xu,
Yukang Yan.
Published at
DIS
2026
Abstract
Photorealistic self-avatars with facial tracking bring VR communication closer to face-to-face interaction, but also expose a fundamental limitation: people often lack accurate awareness and control of their own facial expressions. As a result, directly mirroring facial movements may not reliably convey users’ intended expressions in social VR. We propose modulating rendered expressions on avatars to improve social interaction in VR (e.g., exaggerating positive ones). To design effective modulation strategies and understand how modulation impacts user behavior and perception, we invited 18 participants to perform emotional expressions under varying modulation levels. We collected their self-reported ratings on accuracy and social appropriateness, as well as naturalness ratings and emotion recognition results from 18 additional observers who viewed the expressions. Results show that suppressing rendered expressions amplified participants’ actual facial movements, with effects differing by expression type. Based on our findings, we calculated the modulation ranges for different expressions and developed a real-time modulation system that exaggerates users’ facial expressions with a mild amplitude. We evaluated the system through a VR speech task where 12 participants delivered speeches with and without modulation. Results showed that the applied modulation improved emotional expressiveness and performance. Finally, we developed three application prototypes to illustrate the broader practical implications of our findings.
Materials
Bibtex
@inproceedings{10.1145/3800645.3812871, author = {Lu, Yang and Tang, Jiamu and Yang, Jiankun and Luo, Shijian and Xu, Chenliang and Yan, Yukang}, title = {Tuning the Face: Modulating Facial Expressions for Realistic Self-Avatars in Virtual Reality}, year = {2026}, isbn = {9798400725630}, publisher = {Association for Computing Machinery}, address = {New York, NY, USA}, url = {https://doi.org/10.1145/3800645.3812871}, doi = {10.1145/3800645.3812871}, abstract = {Photorealistic self-avatars with facial tracking bring VR communication closer to face-to-face interaction, but also expose a fundamental limitation: people often lack accurate awareness and control of their own facial expressions. As a result, directly mirroring facial movements may not reliably convey users’ intended expressions in social VR. We propose modulating rendered expressions on avatars to improve social interaction in VR (e.g., exaggerating positive ones). To design effective modulation strategies and understand how modulation impacts user behavior and perception, we invited 18 participants to perform emotional expressions under varying modulation levels. We collected their self-reported ratings on accuracy and social appropriateness, as well as naturalness ratings and emotion recognition results from 18 additional observers who viewed the expressions. Results show that suppressing rendered expressions amplified participants’ actual facial movements, with effects differing by expression type. Based on our findings, we calculated the modulation ranges for different expressions and developed a real-time modulation system that exaggerates users’ facial expressions with a mild amplitude. We evaluated the system through a VR speech task where 12 participants delivered speeches with and without modulation. Results showed that the applied modulation improved emotional expressiveness and performance. Finally, we developed three application prototypes to illustrate the broader practical implications of our findings.}, booktitle = {Proceedings of the 2026 Designing Interactive Systems Conference}, pages = {1419–1436}, numpages = {18}, keywords = {Photorealistic avatar, Facial expression modulation, Social Virtual Reality}, location = { }, series = {DIS '26} }