SemanticAdapt: Optimization-based Adaptation of Mixed Reality Layouts Leveraging Virtual-Physical Semantic Connections

We present an optimization-based approach that automatically adapts Mixed Reality (MR) interfaces to different physical environments. Current MR layouts, including the position and scale of virtual interface elements, need to be manually adapted by users whenever they move between environments, and whenever they switch tasks. This process is tedious and time consuming, and arguably needs to be automated by MR systems for them to be beneficial for end users. We contribute an approach that formulates this challenges as combinatorial optimization problem and automatically decides the placement of virtual interface elements in new environments. In contrast to prior work, we exploit the semantic association between the virtual interface elements and physical objects in an environment. Our optimization furthermore considers the utility of elements for users' current task, layout factors, and spatio-temporal consistency to previous environments. All those factors are combined in a single linear program, which is used to adapt the layout of MR interfaces in real time. We demonstrate a set of application scenarios, showcasing the versatility and applicability of our approach. Finally, we show that compared to a naive adaptive baseline approach that does not take semantic association into account, our approach decreased the number of manual interface adaptations by 37%.

More information

Source code available at https://github.com/ycheng14799/SemanticAdapt

@inproceedings { Cheng2021, 
 author = {Cheng, Yi Fei and Yan, Yukang and Yi, Xin and Shi, Yuanchun, and Lindlbauer, David}, 
 title = {SemanticAdapt: Optimization-based Adaptation of Mixed Reality Layouts Leveraging Virtual-Physical Semantic Connections}, 
 year = {2021}, 
 isbn = {9781450375146}, 
 publisher = {Association for Computing Machinery}, 
 address = {New York, NY, USA}, 
 url = {https://doi.org/10.1145/3472749.3474750}, 
 doi = {10.1145/3472749.3474750}, 
 keywords = {Mixed Reality, Computational interaction, Adaptive user interfaces}, 
 location = {Virtual Event, USA}, 
 series = {UIST '2021} 
 }