CMU at UIST 2026
New research advances activity recognition, mixed reality interfaces, context-aware applications and more
*UIST 2026 is going on now in Detroit, MI. This preview article will be updated*
Carnegie Mellon University authors contributed to more than two dozen papers, demos, and posters at the 2026 Association of Computing Machinery (ACM) Symposium on User Interface Software and Technology (UIST).
For many UIST (pronounced “wist”) attendees, the interdisciplinary conference’s annual Demo Reception is a hands-on, fan favorite event. CMU authors co-authored nine demos and four posters this year. Don't miss the Posters, Demos, and Student Innovation Contest Reception on Tuesday, November 3 at 6:00 p.m. for the opportunity to interact with them.
CMU researchers contributed to 13 full papers and three short papers, as well as participated in three workshops and the doctoral consortium. Jump below for a condensed list of research, or search for Carnegie Mellon in the UIST 2026 program to see involvement along with the schedule.
Papers
🏅 Honorable Mention Award
Object-Agent Twins (OATs): Expressive Micro-Agent Avatars for Inanimate Everyday Objects (Paper & Demo)
Daehwa Kim, Chris Harrison
We introduce Object-Agent Twins (OATs), a toolkit for creating Tamagotchi-like companions for inanimate everyday objects. OATs allow non-expert users to easily author small personality-bearing agents tied to real-world artifacts such as a tea kettle, pet bowls, and a trash can. These OATs act as expressive proxies for physical objects, using real-time sensed data to drive simple needs, moods, and behaviors.
OATs web page
Camera-Free Object Discovery and Interaction with GazeTags (Paper & Demo)
Vimal Mollyn, Chris Harrison
We present GazeTags, a camera- and computer-vision-free approach for worn head-gaze object awareness. Our small electronic tags cost roughly $2 each and can operate for more than a year on a coin cell battery. They work in concert with a user-worn device (smartglasses, hearables, etc.) that periodically emits a focused infrared signal aligned with the user’s head direction.
:30 video
FlexiFab: Laser-Cutting Volumetric Compliant Structures with Tunable-Stiffness (Paper & Demo)
Deying Pan, Fanqi Zhou, Yitao Fan, Guanyun Wang, Alexandra Ion
Laser cutting is a powerful fabrication tool for rapidly creating 3D objects from 2D sheet materials. While most research has focused on making these constructions robust and rigid, we expand this area by introducing robust yet compliant laser-cut structures. We present FlexiFab, a method for fabricating load-bearing volumetric objects with tunable stiffness.
FlexiFab web page / :30 video
A Benchmarking Framework for Context-aware XR Interfaces
Hyunsung Cho, Sarah Yewon Yun, Nancy Ruonan Sun, Ben Lafreniere, Mark Parent, Kashyap Todi, Tanya R Jonker, Hrvoje Benko, Tongshuang Wu, David Lindlbauer
Everyday Extended Reality (XR) systems aim to provide context-aware access to the right functionalities at the right time and place, with minimal manual reconfiguration as users switch context. Yet these interfaces are hard to evaluate: current prototyping and user-study workflows offer no systematic, repeatable way to compare adaptation methods across users and scenarios. We present ContextXR, a novel benchmarking framework for context-aware XR interfaces.
Content XR web page / :30 video
Co-opting Mobile Device Cellular RF Front Ends for HCI Sensing
Daehwa Kim, Chris Harrison
Modern smartphones contain sophisticated cellular modems that can tune their antennas. We show how this latent internal capability, intended to minimize power consumption while maximizing antenna performance, can be co-opted as a sensing capability in already ubiquitous mobile devices. For example, we can sense electromagnetic loading from a user’s hand, surrounding objects, and nearby materials, enabling diverse applications such as detecting grip pose, worn device placement, and passive interactive widgets.
:30 video
Explorable Proofs: Making Written Proofs Explorable by Grounding Them in Formal Representations
Hita Kambhamettu, Will Crichton, Sean Welleck (CMU LTI), Harrison Goldstein, Andrew Head
LLM-generated explanations can make technical content more accessible, but there is a ceiling on what they can support interactively. Because LLM outputs are static text, they cannot be executed, tinkered with, or stepped through. In this paper, we explore how grounding explanations in a formalized representation enables interactive affordances beyond what static text supports.
:30 video
MolecularCanvas: LLM-assisted Small-Molecule Drug Discovery via Structure-Guided Constraints
Haoyu Dong, Rui Sheng, Shuhao Zhang (CMU), Yushi Sun, Dingyang Wu, Hanxiang Chao, Olexandr Isayev (CMU), Huamin Qu, Yuyang Wu (CMU), Yanna Lin
Small-molecule drug discovery relies on iterative molecular optimization, where chemists repeatedly modify candidate compounds to balance multiple competing properties such as efficacy, toxicity, and solubility. Recent advances in generative AI (GenAI), have shown promise in accelerating this process by automatically proposing new molecular structures or targeted modifications… we introduce MolecularCanvas, an interactive system that enables users to iteratively construct an optimization context by integrating high-level goals, structure-level annotations, property constraints, and reference-based preferences.
:30 video
MultiVerse: A Creator-Centered Approach for Authoring Context-Adaptive Lyrics
Alexander Wang, Chris Donahue (CMU CSD), David Lindlbauer
Generative AI may enable new forms of context-aware creative expression by dynamically tailoring media content to its consumption context. For instance, AI systems could adapt song lyrics to the listener and their current activity. However, existing media adaptation systems primarily optimize for audience experience, often neglecting artists' intent, style, and preference. We address this challenge by introducing a novel creator-centered approach to adaptive media authoring and present MultiVerse, a system that instantiates this approach for steering adaptive lyrics.
MultiVerse web page / :30 video / MultiVerse Keeps Creative Intent Alive
PersonaTeaming: Supporting Persona-Driven Red-Teaming for Generative AI
Wesley H Deng, Mingxi Yan, Sunnie S. Y. Kim, Akshita Jha, Lauren Wilcox, Kenneth Holstein, Motahhare Eslami, Leon Gatys
Recent developments in AI safety research have called for red-teaming methods---the practice of probing an AI system with adversarial inputs in order to surface harmful, biased, or otherwise problematic behaviors before deployment---that effectively surface potential risks posed by generative AI models, with growing emphasis on how red-teamers' backgrounds and perspectives shape their strategies and the risks they uncover. While automated red-teaming approaches promise to complement human red-teaming through larger-scale exploration, existing automated approaches do not account for human identities and rarely incorporate human inputs. In this work, we explore persona-driven red-teaming to advance both automated red-teaming and human-AI collaboration.
PersonaTeaming paper / :30 video
Priors of U: Dynamic Preference Learning for Embodied Interaction
Violet Yinuo Han, Alexandra Ion
People have their own habits; from how they organize their workspace, over which mug they reach for in the morning, to how they prepare for cooking. Such preferences are context-dependent, dynamic, and often difficult to articulate. As embodied AI agents, such as robots, enter personal spaces, effective interaction requires understanding these implicit and evolving preferences; otherwise, such systems risk being more hindering than helpful. We propose an architecture that learns user preferences from real-world behavioral observation without requiring explicit instruction or feedback.
Priors of U web page / :30 video
TransfHAR: Self-Supervised Wrist Representations for On-Demand Activity Recognition
Aidan Bradshaw, Riku Arakawa, Xin Liu, Karan Ahuja
Fine-grained wrist activity recognition can support applications such as procedural step guidance and context-aware assistance, yet acquiring labeled data for every new task, user, and activity granularity remains a bottleneck. We present TransfHAR, a self-supervised wrist IMU framework for on-demand, fine-grained activity recognition by learning transferable motion priors from global, unlabeled activities. We show that self-supervised pretraining on coarse wrist IMU activities (e.g., sitting, walking, exercise) learns motion structure rich enough to transfer to fine-grained manipulative, gestural, and procedural activities (e.g., snapping, stirring, waving) that are absent from pretraining.
TransfHAR paper / :30 video
WhatIf: Interactive Exploration of Large-Scale LLM Agent Simulations for Policy Reasoning
Yuxuan Li, Kyzyl Monteiro, Hirokazu Shirado, Sauvik Das
Policymakers in domains such as emergency management, public health, and urban planning must make decisions under deep uncertainty, where outcomes depend on how large populations interpret information, coordinate, and adopt over time. Existing tools only partially support this process: tabletop exercises enable collaborative discussion but lack dynamic feedback, while computational simulations capture population dynamics but are designed for offline analysis. We present WhatIf, an interactive system that enables policymakers to steer, inspect, and compare LLM-powered social simulations in real time.
WhatIf paper / :30 video
While You're Still Busy: Context-Aware Approach Optimization for Proactive Mobile Robots
Ziru Wei, Di Wen, Leyuan Chen, Hassan Shahzad, Alexandra Ion
Robot assistants are increasingly capable of assisting users in everyday environments, yet they don’t take users’ cognitive context into consideration. As a result, proactive behaviors may interrupt users during cognitively demanding tasks, before they can acknowledge or respond. We identify this pre-acknowledgment window as a critical phase and introduce our real-time optimization approach that adapts how robots approach and communicate during this window, using user engagement signals and task context.
Robot Approach web page / :30 video
Demos
A Demonstration of Lightweight Generative UI Personalization
Yi-Hao Peng, Jeffrey P Bigham, Jason Wu
Camera-Free Object Discovery and Interaction with GazeTags
Vimal Mollyn, Chris Harrison
Contexty: Capturing and Organizing In-situ Thoughts for Context-Aware AI Support
Yoonsu Kim, Chanbin Park, Kihoon Son, Saelyne Yang, Juho Kim
Demo of Object-Agent Twins (OATs): Expressive Micro-Agent Avatars for Inanimate Everyday Objects
Daehwa Kim, Chris Harrison
Demonstrating Roomify: Spatially-Grounded Style Transformation for Immersive Virtual Environments
Xueyang Wang, Qinxuan Cen, Weitao Bi, Yunxiang Ma, Xin Yi, Robert Xiao, Xinyi Fu, Hewu Li
Sensorimotor Stickies: A Reconfigurable On-Body Platform for Closed-Loop Sensorimotor Training
Tianhong C Yu, Jiwei Zheng, Chi-Jung Lee, Qifeng Yang, Peter He, Zhuoqi Li (CMU ECE), Tingyu Cheng, Qiuyue (Shirley) Xue, Cheng Zhang, Yiyue Luo
Sketch Oversight: Drawing Decision Boundaries for Delegated AI Agent Actions
Kyzyl Monteiro, Takeshi Koey (CMU IS), Sauvik Das
Short Papers
AtmoSteer: Personalized Scene Control with Reusable Atmospheric Recipes
Nancy Ruonan Sun, Hyunsung Cho, David Lindlbauer
Generating scenes that convey atmospheres such as “cozy” or “contemplative” is difficult because people interpret them subjectively and often struggle to express them concretely. We present AtmoSteer, a human-in-the-loop approach that captures a user’s personal interpretation of an atmosphere as an atmospheric recipe: a weighted set of descriptive tags that can be reused across scenes for tasks such as image generation and image search.
AtmoSteer web page / :30 video
HIDAgent: A Platform-Agnostic Hardware Bridge for Personal Computer Use Agents
Jeffrey P Bigham
Computer Use Agents powered by performant VLMs are increasingly capable of visually interpreting GUIs and executing actions like humans do. However, their method of controlling UIs remains tethered to system-specific APIs or VNC connections, limiting agents to specific platforms and open OS environments. This paper introduces HIDAgent, a platform-agnostic hardware bridge that allows agents to observe any device via HDMI capture and control it via emulated HID mouse and keyboard events—all over a single physical connection.
HIDAgent paper / :30 video
MetaBeads: Enabling Shape- and Stiffness-Changing Interfaces through Beaded Metamaterials
Yujia Liu, Yuyu Lin, Xie He, Victoria Sun, Trevor J Jones, Alexandra Ion
As interactive systems increasingly extend into the physical world, designers need accessible ways to prototype interfaces that can change both shape and stiffness. In this paper, we investigate how simple, widely available materials—beads and string—can be used to construct shape- and stiffness-changing interfaces without specialized machinery. We introduce a set of reusable structural mechanisms that enable sturdy, wearable, and load-bearing structures from bead-based assemblies, each supporting a distinct mode of deformation and stiffness change, such as transforming from a flexible sheet into a jammed cuboid, a stiff tube, or a bistable curvature cell.
MetaBeads web page / :30 video
Workshops
Cyber-Physical Systems for Accessibility and Ability Augmentation: Bridging Diverse Communities
Shuchang Xu, Riku Arakawa, Mina Huh, Nandi Zhang, Tianyu Zhang, Wazeer Zulfikar, Ruei-Che Chang, Yotam Sechayk, Huamin Qu, Amy Pavel, Franklin Mingzhe Li, Yukang Yan, Brian A Smith, Pattie Maes
Program page
Drawing on Materials and Processes: A Hands-on Workshop on Building Creative Tools with Grain Spaces
Lea Albaugh, Ilan E Moyer, Tucker Rae-Grant, Anup Sathya, Ken Nakagaki
Program page
Revibing HCI: Agentic Reimplementation as Community Practice
Yoonjoo Lee, Tae Soo Kim, Kevin Pu, Mira Dontcheva, Bjoern Hartmann, Toby J Li, Brad A Myers, Jeffrey Nichols, April Y Wang, Xuhai "Orson" Xu, Eytan Adar
Program page
Posters
Anteroom: Bounded-Choice Message Triage for Executive and Motor Access
Atieh Taheri, Patrick Carrington, Jeffrey P Bigham
ReVision: Supporting Designers’ Interpretation and Exploration of Visuals in Concepts and Forms
Yaqing Yang, Mei Xi Chia, Aniket Kittur
Towards Benchmarking Screen Reader Use in Computer Use Agents
Valerie Sazonova, Saelyne Yang, Jeffrey P Bigham, Yi-Hao Peng
TractorBeam: Personalized AI Sensemaking Support via Collaborative Machine Annotation
Sireesh Gururaja (CMU LTI), Jordan Taylor, Emma Strubell (CMU LTI)