CMU at Virtual CHI 2021

May 10, 2021
CHI 2021 logo features an ocean wave


More than one hundred researchers from Carnegie Mellon will log on to virtually attend the international ACM Conference on Human Factors in Computing Systems (also known as CHI) conference this week from May 8-13, 2021.

CHI 2021 was scheduled to take place in Yokohama, Japan, but the premier international conference on Human-Computer Interaction will now be held virtually due to the coronavirus pandemic.

Despite the countless challenges we all faced this year, our local HCII community has continued to support each other.

"I am happy to see that even in the face of the pandemic, the HCII had a productive year for research, which is reflected in our numerous and diverse contributions to the CHI 2021 conference," said Jodi Forlizzi, professor and Geschke Director of the HCII.

We highlight the awards and research from Carnegie Mellon University authors that were accepted to CHI 2021, including:

SIGCHI Lifetime Research Award (2021)
Professor Scott Hudson received the 2021 ACM CHI Lifetime Research Award in recognition for his outstanding contributions to the study of human-computer interaction. As a part of that award, Hudson was invited to give a talk during the conference. His talk, "The Future Is Not What It Used to Be: What’s Changed, What’s the Same, and Why the Fun Stuff in Technical HCI is All Ahead of Us," will take place on Monday, May 10, and will also be available via recording. 

CHI Academy Award (2020)
Professor Jason Hong was named to the 2020 CHI Academy, an honorary group of individuals who have made substantial contributions to the field of human-computer interaction (HCI). The distribution of these 2020 CHI Academy awards took place at a virtual ceremony on Thursday, May 6, 2021.

Accepted Papers, Virtual Talks, Journals and Blog Posts
Carnegie Mellon University authors contributed to more than 30 accepted papers this year, including one Best Paper and nine Honorable Mention awards. A list of papers with CMU contributing authors is available below.


Accepted Papers

"Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels" -- Best Paper Award


Xiaoyi Zhang, Lilian de Greef, Amanda Swearngin, Samuel White, Kyle Murray, Lisa Yu, Qi Shan, Jeffrey Nichols, Jason Wu, Chris Fleizach, Aaron Everitt, Jeffrey P. Bigham


Many accessibility features available on mobile platforms require applications (apps) to provide complete and accurate metadata describing user interface (UI) components. Unfortunately, many apps do not provide sufficient metadata for accessibility features to work as expected. In this paper, we explore inferring accessibility metadata for mobile apps from their pixels, as the visual interfaces often best reflect an app's full functionality. We trained a robust, fast, memory-efficient, on-device model to detect UI elements using a dataset of 77,637 screens (from 4,068 iPhone apps) that we collected and annotated. To further improve UI detections and add semantic information, we introduced heuristics (e.g., UI grouping and ordering) and additional models (e.g., recognize UI content, state, interactivity). We built Screen Recognition to generate accessibility metadata to augment iOS VoiceOver. In a study with 9 screen reader users, we validated that our approach improves the accessibility of existing mobile apps, enabling even previously inaccessible apps to be used.

"'It's Complicated': Negotiating Accessibility and (Mis)Representation in Image Descriptions of Race, Gender, and Disability" -- Honorable Mention Award


Cynthia L Bennett, Cole Gleason, Morgan Klaus Scheuerman, Jeffrey P. Bigham, Anhong Guo, Alexandra To


Content creators are instructed to write textual descriptions of visual content to make it accessible; yet existing guidelines lack specifics on how to write about people's appearance, particularly while remaining mindful of consequences of (mis)representation. In this paper, we report on interviews with screen reader users who were also Black, Indigenous, People of Color, Non-binary, and/or Transgender on their current image description practices and preferences, and experiences negotiating theirs and others' appearances non-visually. We discuss these perspectives, and the ethics of humans and AI describing appearance characteristics that may convey the race, gender, and disabilities of those photographed. In turn, we share considerations for more carefully describing appearance, and contexts in which such information is perceived salient. Finally, we offer tensions and questions for accessibility research to equitably consider politics and ecosystems in which technologies will embed, such as potential risks of human and AI biases amplifying through image descriptions.


"'You Gotta Watch What You Say': Surveillance of Communication with Incarcerated People" -- Honorable Mention Award


Kentrell Owens, Camille Cobb, Lorrie Cranor


Surveillance of communication between incarcerated and non-incarcerated people has steadily increased, enabled partly by technological advancements. Third-party vendors control communication tools for most U.S. prisons and jails and offer surveillance capabilities beyond what individual facilities could realistically implement. Frequent communication with family improves mental health and post-carceral outcomes for incarcerated people, but does discomfort about surveillance affect how their relatives communicate with them? To explore this and the understanding, attitudes, and reactions to surveillance, we conducted 16 semi-structured interviews with participants who have incarcerated relatives. Among other findings, we learn that participants communicate despite privacy concerns that they felt helpless to address. We also observe inaccuracies in participants’ beliefs about surveillance practices. We discuss implications of inaccurate understandings of surveillance, misaligned incentives between end-users and vendors, how our findings enhance ongoing conversations about carceral justice, and recommendations for more privacy-sensitive communication tools.


"Engineering Multifunctional Spacer Fabrics by Machine Knitting" -- Honorable Mention Award


Lea Albaugh, James McCann, Scott E. Hudson, and Lining Yao


Machine knitting is an increasingly accessible fabrication technology for producing custom soft goods. However, recent machine knitting research has focused on knit shaping, or on adapting hand-knitting patterns. We explore a capability unique to machine knitting: producing multilayer spacer fabrics. These fabrics consist of two face layers connected by a monofilament filler yarn which gives the structure stiffness and volume. We show how to vary knit patterning and yarn parameters in spacer fabrics to produce tactile materials with embedded functionality for forming soft actuated mechanisms and sensors with tunable density, stiffness, material bias, and bristle properties. These soft mechanisms can be rapidly produced on a computationally-controlled v-bed knitting machine and integrated directly into soft objects.


"Medical Maker Response to COVID-19: Distributed Manufacturing Infrastructure for Stop Gap Protective Equipment" -- Honorable Mention Award


Udaya Lakshmi, Megan Hofmann, Kelly Mack, Scott E. Hudson, Jennifer Mankoff, Rosa I. Arriaga


Unprecedented maker efforts arose in response to COVID-19 medical supply gaps worldwide. Makers in the U.S., participated in peer-production activities to manufacture personal protective equipment (PPE). Whereas, medical makers, who innovate exclusively for points of care, pivoted towards safer, reliable PPE. What were their efforts to pivot medical maker infrastructure towards reliable production of safe equipment at higher volumes? We interviewed 13 medical makers as links between institutions, maker communities, and wider regional industry networks. These medical makers organized stopgap manufacturing in institutional spaces to resolve acute shortages (March--May) and chronic shortages (May--July). They act as intermediaries in efforts to prototype and produce devices under regulatory, material, and human constraints of a pandemic. We re-frame their making efforts as repair work to offer an alternate critical view of optimism around making for crisis. We contribute an understanding of these efforts to inform infrastructure design for making with purpose and safety leading to opportunities for community production of safe devices at scale.


"PrivacyMic: Utilizing Inaudible Frequencies for Privacy Preserving Daily Activity Recognition" -- Honorable Mention Award


Yasha Iravantchi, Karan Ahuja, Mayank Goel, Chris Harrison, Alanson Sample


Sound presents an invaluable signal source that enables computing systems to perform daily activity recognition. However, microphones are optimized for human speech and hearing ranges: capturing private content, such as speech, while omitting useful, inaudible information that can aid in acoustic recognition tasks. We simulated acoustic recognition tasks using sounds from 127 everyday household/workplace objects, finding that inaudible frequencies can act as a substitute for privacy-sensitive frequencies. To take advantage of these inaudible frequencies, we designed a Raspberry Pi-based device that captures inaudible acoustic frequencies with settings that can remove speech or all audible frequencies entirely. We conducted a perception study, where participants "eavesdropped" on PrivacyMic’s filtered audio and found that none of our participants could transcribe speech. Finally, PrivacyMic’s real-world activity recognition performance is comparable to our simulated results, with over 95% classification accuracy across all environments, suggesting immediate viability in performing privacy-preserving daily activity recognition.


"Screen2Vec: Semantic Embedding of GUI Screens and GUI Components" -- Honorable Mention Award


Toby Jia-Jun Li, Lindsay Popowski, Tom M. Mitchell, Brad A. Myers


Representing the semantics of GUI screens and components is crucial to data-driven computational methods for modeling user-GUI interactions and mining GUI designs. Existing GUI semantic representations are limited to encoding either the textual content, the visual design and layout patterns, or the app contexts. Many representation techniques also require significant manual data annotation efforts. This paper presents Screen2Vec, a new self-supervised technique for generating representations in embedding vectors of GUI screens and components that encode all of the above GUI features without requiring manual annotation using the context of user interaction traces. Screen2Vec is inspired by the word embedding method Word2Vec, but uses a new two-layer pipeline informed by the structure of GUIs and interaction traces and incorporates screen- and app-specific metadata. Through several sample downstream tasks, we demonstrate Screen2Vec's key useful properties: representing between-screen similarity through nearest neighbors, composability, and capability to represent user tasks.


"Significant Otter: Understanding the Role of Biosignals in Communication" -- Honorable Mention Award


Fannie Liu, Chunjong Park, Yu Jiang Tham, Tsung-yu Tsai, Laura Dabbish, Geoff Kaufman, Andrés Monroy-Hernández


With the growing ubiquity of wearable devices, sensed physiological responses provide new means to connect with others. While recent research demonstrates the expressive potential for biosignals, the value of sharing these personal data remains unclear. To understand their role in communication, we created Significant Otter, an Apple Watch/iPhone app that enables romantic partners to share and respond to each other’s biosignals in the form of animated otter avatars. In a one-month study with 20 couples, participants used Significant Otter with biosignals sensing OFF and ON. We found that while sensing OFF enabled couples to keep in touch, sensing ON enabled easier and more authentic communication that fostered social connection. However, the addition of biosignals introduced concerns about autonomy and agency over the messages they sent. We discuss design implications and future directions for communication systems that recommend messages based on biosignals.

More information:

Link to paper

Medium post


"The Right to Help and the Right Help: Fostering and Regulating Collective Action in a Medical Making Reaction to COVID-19" -- Honorable Mention Award


Megan Hofmann, Udaya Lakshmi, Kelly Mack, Scott E. Hudson, Rosa I. Arriaga, Jennifer Mankoff


Medical making intersects opposing value systems of a medical ``do no harm'' ethos and makers' drive to innovate. Since March 2020, online maker communities have formed to design, manufacture, and distribute personal protective equipment (PPE) and other medical devices needed to fight the COVID-19 pandemic. We present a participant observation study of 14 maker communities, which have developed differing driving principles for efforts with varied access to interdisciplinary expertise on online platforms that mutually shape collective action. Over time, these communities unintentionally align towards action-oriented or regulated practices because they often lack higher level insight and agency in choosing communication platforms. In response, we recommend: regulatory bodies to build coalitions with makers, online platforms to give communities more control over the presentation of information, and repositories to balance needs to distribute information while limiting the spread of misinformation.


"When the Tab Comes Due: Challenges in the Cost Structure of Browser Tab Usage" -- Honorable Mention Award


Joseph Chee Chang, Nathan Hahn, Yongsung Kim, Julina Coupland, Bradley Breneisen, Hannah S Kim, John Hwong, Aniket Kittur


Tabs have become integral to browsing the Web yet have changed little since their introduction nearly 20 years ago. In contrast, the internet has gone through dramatic changes, with users increasingly moving from navigating to websites to exploring information across many sources to support online sensemaking. This paper investigates how tabs today are overloaded with a diverse set of functionalities and issues users face when managing them. We interviewed ten information workers asking about their tab management strategies and walk through each open tab on their work computers four times over two weeks. We uncovered competing pressures pushing for keeping tabs open (ranging from interaction to emotional costs) versus pushing for closing them (such as limited attention and resources). We then surveyed 103 participants to estimate the frequencies of these pressures at scale. Finally, we developed design implications for future browser interfaces that can better support managing these pressures.


"'I Choose Assistive Devices That Save My Face' A Study on Perceptions of Accessibility and Assistive Technology Use Conducted in China"


Franklin Mingzhe Li, Di Laura Chen, Mingming Fan, Khai N Truong


Despite the potential benefits of assistive technologies (ATs) for people with various disabilities, only around 7% of Chinese with disabilities have had an opportunity to use ATs. Even for those who have used ATs, the abandonment rate was high. Although China has the world's largest population with disabilities, prior research exploring how ATs are used and perceived, and why ATs are abandoned have been conducted primarily in North America and Europe. In this paper, we present an interview study conducted in China with 26 people with various disabilities to understand their practices, challenges, perceptions, and misperceptions of using ATs. From the study, we learned about factors that influence AT adoption practices (e.g., misuse of accessible infrastructure, issues with replicating existing commercial ATs), challenges using ATs in social interactions (e.g., Chinese stigma), and misperceptions about ATs (e.g., ATs should overcome inaccessible social infrastructures). Informed by the findings, we derive a set of design considerations to bridge the existing gaps in AT design (e.g., manual vs. electronic ATs) and to improve ATs' social acceptability in China.


"'It Feels Like I am Talking into a Void': Understanding Interaction Gaps in Synchronous Online Classrooms"


Matin Yarmand, Jaemarie Solyst, Scott Klemmer, Nadir Weibel


This paper investigates in-class interactions in synchronous online classrooms when the choice of modality is discretionary, such that students choose when and if they turn on their cameras and microphones. Instructor interviews (N = 7) revealed that most students preferred not to share videos and verbally participate. This hindered instructors' ability to read their classrooms and make deeper connections with students. Survey results (N = 102) suggested that students felt a lacking sense of community in online vs. in-person lectures. Some students felt uncomfortable broadcasting their appearances to everyone in the class, and some were unaware of the benefits for instructors. Most students favored using the text chat to participate. Considering the needs of both instructors and students, we propose recommendations to mitigate the loss of classroom interactions by collecting and presenting less invasive social cues in an aggregated format, and incorporating opportunities for informal exchanges and individual control to spark peer bonding.


"'Brilliant AI Doctor' in Rural China: Tensions and Challenges in AI-Powered CDSS Deployment"


Dakuo Wang, Liuping Wang, Zhan Zhang, Ding Wang, Haiyi Zhu, Yvonne Gao, Xiangmin Fan, and Feng Tian


Artificial intelligence (AI) technology has been increasingly used in the implementation of advanced Clinical Decision Support Systems (CDSS). Research demonstrated the potential usefulness of AI-powered CDSS (AI-CDSS) in clinical decision making scenarios. However, post-adoption user perception and experience remain understudied, especially in developing countries. Through observations and interviews with 22 clinicians from 6 rural clinics in China, this paper reports the various tensions between the design of an AI-CDSS system ("Brilliant Doctor'') and the rural clinical context, such as the misalignment with local context and workflow, the technical limitations and usability barriers, as well as issues related to transparency and trustworthiness of AI-CDSS. Despite these tensions, all participants expressed positive attitudes toward the future of AI-CDSS, especially acting as "a doctor's AI assistant'' to realize a Human-AI Collaboration future in clinical settings. Finally we draw on our findings to discuss implications for designing AI-CDSS interventions for rural clinical contexts in developing countries.


"A Human-AI Collaborative Approach for Clinical Decision Making on Rehabilitation Assessment"


Min Hun Lee, Daniel P. Siewiorek, Asim Smailagic, Alexandre Bernardino, Sergi Bermúdez i Badia


Advances in artificial intelligence (AI) have made it increasingly applicable to supplement expert's decision-making in the form of a decision support system on various tasks. For instance, an AI-based system can provide therapists quantitative analysis on patient's status to improve practices of rehabilitation assessment. However, there is limited knowledge on the potential of these systems. In this paper, we present the development and evaluation of an interactive AI-based system that supports collaborative decision making with therapists for rehabilitation assessment. This system automatically identifies salient features of assessment to generate patient-specific analysis for therapists, and tunes with their feedback. In two evaluations with therapists, we found that our system supports therapists significantly higher agreement on assessment (0.71 average F1-score) than a traditional system without analysis (0.66 average F1-score, $p < 0.05$). After tuning with therapist’s feedback, our system significantly improves its performance from 0.8377 to 0.9116 average F1-scores ($p < 0.01$). This work discusses the potential of a human-AI collaborative system to support more accurate decision making while learning from each other's strengths.


"Classroom Digital Twins with Instrumentation-Free 6DOF Gaze Tracking"


Karan Ahuja, Deval Shah, Sujeath Pareddy, Franceska Xhakaj, Amy Ogan, Yuvraj Agarwal, Chris Harrison


Classroom sensing is an important and active area of research with great potential to improve instruction. Complementing professional observers - the current best practice - automated pedagogical professional development systems can attend every class and capture fine-grained details of all occupants. One particularly valuable facet to capture is class gaze behavior. For students, certain gaze patterns have been shown to correlate with interest in the material, while for instructors, student-centered gaze patterns have been shown to increase approachability and immediacy. Unfortunately, prior classroom gaze-sensing systems have limited accuracy and often require specialized external or worn sensors. In this work, we developed a new computer-vision-driven system that powers a 3D “digital twin” of the classroom and enables whole-class, 6DOF head gaze vector estimation without instrumenting any of the occupants. We describe our open source implementation, and results from both controlled studies and real-world classroom deployments.


"Enabling Personal Computational Handweaving with a Low-Cost Jacquard Loom"


Lea Albaugh, James McCann, Lining Yao and Scott Hudson


We present an inexpensive tabletop loom that offers fully computational patterning while maintaining the flexibility of handweaving. Our loom can be assembled for under US\$200 with 3D printed parts, and it can be controlled straightforwardly over USB. Our loom is explicitly a \emph{hand} loom: that is, a weaver is required to operate the weaving process and may mediate row-by-row patterning and material specifics like yarn tension. This approach combines the flexibility of fully analog handweaving with the computational affordances of digital fabrication: it enables the incorporation of special techniques and materials, as well as allowing for the possibility of computational and creative interventions in the weaving process itself -- for skill-building, for interactive design, or for creative reflection. We describe the mechanical and electronic implementation of our loom and show examples of its use for personal fabrication.


"Firefox Voice: An Open and Extensible Voice Assistant Built Upon the Web"


Julia Cambre, Alex C. Williams, Afsaneh Razi, Ian Bicking, Abraham Wallin, Janice Tsai, Chinmay Kulkarni, Jofish Kaye


Voice assistants are fundamentally changing the way we access information. However, voice assistants still leverage little about the web beyond simple search results. We introduce Firefox Voice, a novel voice assistant built on the open web ecosystem with an aim to expand access to information available via voice. Firefox Voice is a browser extension that enables users to use their voice to perform actions such as setting timers, navigating the web, and reading a webpage’s content aloud. Through an iterative development process and use by over 12,000 active users, we find that users see voice as a way to accomplish certain browsing tasks efficiently, but struggle with discovering functionality and frequently discontinue use. We conclude by describing how Firefox Voice enables the development of novel, open web-powered voice-driven experiences.


"FlexTruss: A Computational Threading Method for Multi-material, Multi-form and Multi-use Prototyping"


Lingyun Sun, Jiaji Li, Yu Chen, Yue Yang, Zhi Yu, Danli Luo, Jianzhe Gu, Lining Yao, Ye Tao, Guanyun Wang


3D printing, as a rapid prototyping technique, usually fabricates objects that are difficult to modify physically. This paper presents FlexTruss, a design and construction pipeline based on the assembly of modularized truss-shaped objects fabricated with conventional 3D printers and assembled by threading. To create an end-to-end system, a parametric design tool with an optimal Euler path calculation method is developed, which can support both inverse and forward design workflow and multi-material construction of modular parts. In addition, the assembly of truss modules by threading is evaluated with a series of application cases to demonstrate the affordance of FlexTruss. We believe that FlexTruss extends the design space of 3D printing beyond typically hard and fixed forms, and it will provide new capabilities for designers and researchers to explore the use of such flexible truss structures in human-object interaction.


"Fostering Equitable Help-Seeking for K-3 Students in Low Income and Rural Contexts"


Judith Uchidiuno, Jessica Hammer, Ken Koedinger, Amy Ogan


Adaptive Collaborative Learning Support (ACLS) systems improve collaboration and learning for students over individual work or collaboration with non-adaptive support. However, many ACLS systems are ill-suited for rural contexts where students often need multiple kinds of support to complete tasks, may speak languages unsupported by the system, and require more than pre-assigned tutor-tutee student pairs for more equitable learning. We designed an intervention that fosters more equitable help-seeking by automatically detecting student struggles and prompts them to seek help from specific peers that can help. We conducted a mixed-methods experimental study with 98 K-3 students in a rural village in Tanzania over a one-month period, evaluating how the system affects student interactions, system engagement, and student learning. Our intervention increased student interactions by almost 4 times compared to the control condition, increased domain knowledge interactions, and propelled students to engage in more cognitively challenging activities.


"Freeform Fabrication of Fluidic Edible Materials"


Humphrey Yang, Danli Luo, Kuanren Qian, Lining Yao


From providing nutrition to being social platforms, food plays an essential role in our daily lives and cultures. In HCI, we are interested in using food as an interaction medium and a context of personal fabrication. Yet, the design space of available food printing methods is limited to shapes with minimal overhangs and materials that have a paste-like consistency. In this work, we seek to expand this design space by adapting support bath-assisted printing to the food context. The bath scaffolds the embedded materials and preserves shapes during the printing processes, enabling us to create freeform food with fluid-like materials. We provide users guidelines for choosing the appropriate support bath type and processing methods depending on the printing material’s properties. A design tool suite and application examples, including confectionery arts, 4D printed food, and edible displays are also offered to demonstrate the enabled interaction design space.


"From Tactile to NavTile: Opportunities and Challenges for Multi-Modal Feedback in Guiding Surfaces during Non-Visual Navigation"


Sai Swaminathan, Yellina Yim, Scott Hudson, Cynthia Bennett, Patrick Carrington


Tactile guiding surfaces in the built environment have held a contentious place in the process of navigation by people who are blind or visually impaired. Despite standards for tactile guiding surfaces, problems persist with inconsistent implementation, perception, and geographic orientation. We investigate the role of tactile cues in non-visual navigation and attitudes surrounding guiding surfaces through a survey of 67 people with vision impairments and ten interviews with navigation and public accessibility experts. Our participants revealed several opportunities to augment existing tactile surfaces while envisioning novel multimodal feedback solutions in immediately relevant contexts. We also propose an approach for designing and exploring low cost, multimodal tactile surfaces, which we call navtiles. Finally, we discuss practical aspects of implementation for new design alternatives such as standardization, installation, movability, discoverability, and a need for transparency. Collectively, these insights contribute to the production and implementation of novel multimodal navigation aids.


"Homecoming: Exploring Returns to Long-Term Single Player Games"


Noor Hammad, Owen Brierley, Zach McKendrick, Patrick Finn, Sowmya Somanath, Jessica Hammer, Ehud Sharlin


We present an autobiographical design journey exploring the experience of returning to long-term single player games. Continuing progress from a previously saved game, particularly when substantial time has passed, is an understudied area in games research. To begin our exploration in this domain, we investigated what the return experience is like first-hand. By returning to four long-term single player games played extensively in the past, we revealed a phenomenon we call The Pivot Point, a ‘eureka’ moment in return gameplay. The pivot point anchors our design explorations, where we created prototypes to leverage the pivot point in reconnecting with the experience. These return experiences and subsequent prototyping iterations inform our understanding of how to design better returns to gameplay, which can benefit both producers and consumers of long-term single player games.


"Mapping Design Spaces for Teaching Audience Participation in Game Live Streaming"


Alina Striner, Andrew M. Webb, Jessica Hammer, Amy Cook


Live streaming sites such as Twitch offer new ways for remote audiences to engage with and affect gameplay. While research has considered how audiences interact with games, HCI lacks clear demarcations of the potential design spaces for audience participation. This paper introduces and validates a theme map of audience participation in game live streaming for student designers. This map is a lens that reveals relationships among themes and sub-themes of Agency, Pacing, and Community---to explore, reflect upon, describe, and make sense of emerging, complex design spaces. We are the first to articulate such a lens, and to provide a reflective tool to support future research and education. To create the map, we perform a thematic analysis of design process documents of a course on audience participation for Twitch, using this analysis to visually coordinate relationships between important themes. To help student designers analyze and reflect on existing experiences, we supplement the theme map with a set of mapping procedures. We validate the applicability of our map with a second set of student designers, who found the map useful as a comparative and reflective tool.


"Pose-on-the-Go: Approximating Partial User Pose with Smartphone Sensor Fusion and Inverse Kinematics"


Karan Ahuja, Sven Mayer, Mayank Goel, Chris Harrison


We present Pose-on-the-Go, a full-body pose estimation system that uses sensors already found in today’s smartphones. This stands in contrast to prior systems, which require worn or external sensors. We achieve this result via extensive sensor fusion, leveraging a phone’s front and rear cameras, the user-facing depth camera, touchscreen, and IMU. Even still, we are missing data about a user’s body (e.g., angle of the elbow joint), and so we use inverse kinematics to estimate and animate probable body poses. We provide a detailed evaluation of our system, benchmarking it against a professional-grade Vicon tracking system. We conclude with a series of demonstration applications that underscore the unique potential of our approach, which could be enabled on many modern smartphones with a simple software update.


"Priorities, Technology & Power: Co-Designing an Inclusive Transit Agenda in Kampala, Uganda"


Lynn Kirabo, Elizabeth J. Carter, Devon Barry, Aaron Steinfeld


There is considerable effort within the HCI community to explore, document, and advocate for the lived experiences of persons with disabilities (PWDs). However, PWDs from the Global South, particularly Africa, are underrepresented in this scholarship. We contribute to closing this gap by investigating the unmet transit needs and characterization of technology within the disability community in Kampala, Uganda. We investigated transportation due to the increase in ride-share solutions created by widespread mobile computing and the resulting disruption of transportation worldwide. We hosted co-design sessions with disability advocates and adapted the stakeholder tokens method from the value-sensitive design framework to map the stakeholder ecosystem. Our key insight is the identification of a new group of non-traditional core stakeholders who highlight the values of inclusion, mobility, and safety within the ecosystem. Finally, we discuss how our findings engage with concepts of disability justice and perceptions of power.


"Reducing Uncertainty and Offering Comfort: Designing Technology for Coping with Interpersonal Racism"


Alexandra To, Hillary Carey, Geoff Kaufman, Jessica Hammer


Ranging from subtle to overt, unintentional to systemic, navigating racism is additional everyday work for many people. Yet the needs of people who experience racism have been overlooked as a fertile ground for better technology. Through a series of workshops we call Foundational Fiction, we engaged BIPOC (Black, Indigenous, People of Color) in participatory design to identify qualities of technology that can support people coping before, during, and after a racist interaction. Participants developed storyboards for digital tools that offer advice, predict consequences, identify racist remarks and intervene, educate both targets and perpetrators about interpersonal and systemic racism, and more. In the paper we present our workshop method utilizing interactive fiction, participants' design concepts, prevalent themes (reducing uncertainty and offering comfort), and we provide critical analysis of the complexity of technology in these contexts. This work identifies specific opportunities for exploring anti-racist social tools.


"Say It All: Feedback for Improving Non-Visual Presentation Accessibility"


Yi-Hao Peng, JiWoong Jang, Jeffrey P. Bigham, Amy Pavel


Presenters commonly use slides as visual aids for informative talks.When presenters fail to verbally describe the content on their slides,blind and visually impaired audience members lose access to necessary content, making the presentation difficult to follow. Our analysis of 90 existing presentation videos revealed that 72% of 610 visual elements (e.g., images, text) were insufficiently described. To help presenters create accessible presentations, we introduce Presentation A11y, a system that provides real-time and post-presentation accessibility feedback. Our system analyzes visual elements on the slide and the transcript of the verbal presentation to provide element-level feedback on what visual content needs to be further described or even removed. Presenters using our system with their own slide-based presentations described more of the content on their slides, and identified 3.26 times more accessibility problems to fix after the talk than when using a traditional slide-based presentation interface. Integrating accessibility feedback into content creation tools will improve the accessibility of informational content for all.


"ShrinCage: 4D Printing Accessories that Self-Adapt"


Lingyun Sun, Yue Yang, Yu Chen, Jiaji Li, Danli Luo, Haolin Liu, Lining Yao, Ye Tao, Guanyun Wang


3D printing technology makes Do-It-Yourself and reforming everyday objects a reality. However, designing and fabricating attachments that can seamlessly adapt existing objects to extended functionality is a laborious process, which requires accurate measuring, modeling, manufacturing, and assembly. This paper presents ShrinCage, a 4D printing system that allows novices to easily create shrinkable adaptations to fit and fasten existing objects. Specifically, the design tool presented in this work aid in the design of attachment that adapts to irregular morphologies, which accommodates the variations in measurements and fabrication, subsequently simplifying the modeling and assembly processes. We further conduct mechanical tests and user studies to evaluate the availability and feasibility of this method. Numerous application examples created by ShrinCage prove that it can be adopted by aesthetic modification, assistive technology, repair, upcycling, and augmented 3D printing.


"Soliciting Stakeholders’ Fairness Notions in Child Maltreatment Predictive Systems"


Hao-Fei Cheng, Logan Stapleton, Ruiqi Wang, Paige Bullock, Alexandra Chouldechova, Zhiwei Steven Wu, and Haiyi Zhu


Recent work in fair machine learning has proposed dozens of technical definitions of algorithmic fairness and methods for enforcing these definitions. However, we still lack an understanding of how to develop machine learning systems with fairness criteria that reflect relevant stakeholders' nuanced viewpoints in real-world contexts. To address this gap, we propose a framework for eliciting stakeholders' subjective fairness notions. Combining a user interface that allows stakeholders to examine the data and the algorithm's predictions with an interview protocol to probe stakeholders' thoughts while they are interacting with the interface, we can identify stakeholders' fairness beliefs and principles. We conduct a user study to evaluate our framework in the setting of a child maltreatment predictive system. Our evaluations show that the framework allows stakeholders to comprehensively convey their fairness viewpoints. We also discuss how our results can inform the design of predictive systems.


"Stitching Together the Experiences of Disabled Knitters"


Taylor Gotfridm, Kelly Mack, Kathryn J Lum, Evelyn Yang, Jessica Hodgins, Scott E Hudson, Jennifer Mankoff


Knitting is a popular craft that can be used to create customized fabric objects such as household items, clothing and toys. Knitting is also a relaxing and calming exercise for many knitters. Little is known about how disabled knitters use and benefit from knitting, and what accessibility solutions and challenges they create and encounter. We conducted interviews with 16 experienced, disabled knitters and analyzed 20 threads from five forums that discussed accessible knitting to identify how and why disabled knitters knit, and what accessibility concerns remain. We additionally conducted an iterative design case study developing knitting tools for a knitter who found existing solutions insufficient. Our innovations improved the range of stitches she could produce. We conclude by arguing for the importance of improving tools for both pattern generation and modification as well as adaptations or modifications to existing tools such as looms to make it easier to track progress.


"Super-Resolution Capacitive Touchscreens"


Sven Mayer, Xiangyu Xu, Chris Harrison


Capacitive touchscreens are near-ubiquitous in today's touch-driven devices, such as smartphones and tablets. By using rows and columns of electrodes, specialized touch controllers are able to capture a 2D image of capacitance at the surface of a screen. For over a decade, capacitive "pixels" have been around 4 millimeters in size – a surprisingly low resolution that precludes a wide range of interesting applications. In this paper, we show how super-resolution techniques, long used in fields such as biology and astronomy, can be applied to capacitive touchscreen data. By integrating data from many frames, our software-only process is able to resolve geometric details finer than the original sensor resolution. This opens the door to passive tangibles with higher-density fiducials and also recognition of every-day metal objects, such as keys and coins. We built several applications to illustrate the potential of our approach and report the findings of a multipart evaluation.


"Synthesizing Doppler Radar Data from Videos for Training Privacy-Preserving Activity Recognition"


Karan Ahuja, Yue Jiang, Mayank Goel, Chris Harrison


Millimeter wave (mmWave) Doppler radar is a new and promising sensing approach for human activity recognition, offering signal richness approaching that of microphones and cameras, but without many of the privacy-invading downsides. However, unlike audio and computer vision approaches that can draw from huge libraries of videos for training deep learning models, Doppler radar has no existing large datasets, holding back this otherwise promising sensing modality. In response, we set out to create a software pipeline that converts videos of human activities into realistic, synthetic Doppler radar data. We show how this cross-domain translation can be successful through a series of experimental results. Overall, we believe our approach is an important stepping stone towards significantly reducing the burden of training such as human sensing systems, and could help bootstrap uses in human-computer interaction.


"Toggles, Dollar Signs, and Triangles: How to (In)Effectively Convey Privacy Choices with Icons and Link Texts"


Hana Habib, Yixin Zou, Yaxing Yao, Alessandro Acquisti, Lorrie Faith Cranor, Joel Reidenberg, Norman Sadeh, and Florian Schaub


Increasingly, icons are being proposed to concisely convey privacy-related information and choices to users. However, complex privacy concepts can be difficult to communicate. We investigate which icons effectively signal the presence of privacy choices. In a series of user studies, we designed and evaluated icons and accompanying textual descriptions (link texts) conveying choice, opting-out, and sale of personal information --- the latter an opt-out mandated by the California Consumer Privacy Act (CCPA). We identified icon-link text pairings that conveyed the presence of privacy choices without creating misconceptions, with a blue stylized toggle icon paired with "Privacy Options" performing best. The two CCPA-mandated link texts ("Do Not Sell My Personal Information" and "Do Not Sell My Info") accurately communicated the presence of do-not-sell opt-outs with most icons. Our results provide insights for the design of privacy choice indicators and highlight the necessity of incorporating user testing into policy making.


"Vibrosight++: City-Scale Sensing Using Existing Retroreflective Signs and Markers"


Yang Zhang, Sven Mayer, Jesse Gonzalez, Chris Harrison


Today's smart cities use thousands of physical sensors distributed across the urban landscape to support decision making in areas such as infrastructure monitoring, public health, and resource management. These weather-hardened devices require power and connectivity, and often cost thousands just to install, let alone maintain. In this paper, we show how long-range laser vibrometry can be used for low-cost, city-scale sensing. Although typically limited to just a few meters of sensing range, the use of retroreflective markers can boost this to 1km or more. Fortuitously, cities already make extensive use of retroreflective materials for street signs, construction barriers, road studs, license plates, and many other markings. We describe how our prototype system can co-opt these existing markers at very long ranges and use them as unpowered accelerometers for use in a wide variety of sensing applications.


"What Makes Videos Accessible to Blind and Visually Impaired People?"


Xingyu Liu, Patrick Carrington, Xiang 'Anthony' Chen, Amy Pavel


Videos on sites like YouTube have become a primary source for information online. User-generated videos almost universally lack audio descriptions, making most videos inaccessible to blind and visually impaired (BVI) consumers. Our formative studies with BVI people revealed that they used a time-consuming trial-and-error approach when searching for videos: clicking on a video, watching a portion, leaving the video, and repeating the process to find videos that would be accessible — or understandable without additional description of the visual content. BVI people also reported video accessibility heuristics that characterize accessible and inaccessible videos. We instantiate 7 of the identified heuristics (2 audio-related, 2 video-related, and 3 audio-visual) as automated metrics to assess video accessibility. Our automated video accessibility metrics correlate with BVI people’s perception of video accessibility (Adjusted R-squared = 0.642). We augment a video search interface with our video accessibility metrics and find that our system improves BVI peoples’ efficiency in finding accessible videos. With accessibility metrics, participants found videos 40% faster and clicked 54% less videos in our user study. By integrating video accessibility metrics, video hosting platforms could help people surface accessible videos and encourage content creators to author more accessible products, improving video accessibility for all.



Late Breaking Work and Journal

"Counterspace Games for BIWOC STEM Students" -- Late Breaking Work


Erica Principe Cruz, Nalyn Sriwattanakomen, Jessica Hammer, Geoff Kaufman

"Black, Indigenous, and other Women of Color (BIWOC) studying STEM are underrepresented in STEM and subject to its "chilly" climate; it is unsurprising that BIWOC STEM students report weaker senses of belonging and higher rates of attrition. Counterspaces, or spaces for mutual support for BIWOC at the margins of STEM, have long combated dominant STEM culture to support BIWOC to thrive and persist in STEM. Digital game design and playful interactions to counter oppression can be leveraged to create digital games that function as counterspaces for BIWOC STEM students to playfully cultivate their belonging and persistence. Our exploratory game design research aims to co-design counterspaces games with BIWOC STEM students, and here we present our initial focus group designs centered on exploring existing BIWOC counterspace practices, preliminary data and insights, and promising directions for developing game design strategies to support BIWOC belonging and persistence in STEM."


"Designing a self-efficacy game for health literacy in marginalized communities" -- Late Breaking Work


Morgan C Evans, Adela Kapuscinska, Maya Greenholt, Junchao Lin, Xuanyuan Liu, Tianyi Zhang, Jessica Hammer, Geoff Kaufman


"Health inequity is a critical problem in the United States and one that primarily affects marginalized communities. One critical aspect to interventions addressing this issue is the aim to increase health literacy, so that members of these communities can make informed decisions and feel empowered to take charge of their personal health. Our team developed a transformational game that uses self-efficacy to address players’ health literacy in context. Using properties from both visual novel and strategy simulation genres, we present a game in which players take on the role of a community manager who works to better their community by completing quests from non-player character (NPC) community members. The current paper contributes our research and iterative design processes, and highlights future directions utilizing focus groups and playtesting with community members, game development, and evaluation studies to assess impact."


"ExoForm: Shape Memory and Self-Fusing Semi-Rigid Wearables" -- Late Breaking Work


Fang Qin, Huai-Yu Cheng, Rachel Sneeringer, Maria Vlachostergiou, Sampada Acharya, Haolin Liu, Carmel Majidi, Mohammad Islam, Lining Yao


"Semi-rigid and rigid structures have been utilized in many on-body applications including musculoskeletal support (e.g., braces and splints). However, most of these support structures are not very compliant, so effortless custom fitting becomes a unique design challenge. Furthermore, the weight and space needed to transport these structures impede adoption in mobile environments. Here, we introduce ExoForm, a compact, customizable, and semi-rigid wearable material system with self-fusing edges that can semi-autonomously assemble on-demand while providing integrated sensing, control, and mobility. We present a comprehensive and holistic engineering strategy that includes optimized material composition, computationally-guided design and fabrication, semi-autonomous self-morphing assembly and fusing steps, heating control, and sensing for our easy-to-wear ExoForm. Finally, we fabricate wearable braces using the ExoForm method as a demonstration along with preliminary evaluation of ExoForm’s performance."


"Hashtag-Forget: Using Social Media Ephemerality to Support Evolving Identities" -- Late Breaking Work


Michal Luria, Nate Foulds


"This work looks into the current state of sharing ephemeral versus permanent content on common social media platforms. Previous research has indicated that ephemerality of content, content that has an expiration date or time, can support users' identity construction. However, not much is known about whether current ephemeral interactions on social media are successful in doing so, or whether there are ephemerality-related design opportunities for social media platforms that can improve identity expression. In an 8-day qualitative diary study, participants reported when they posted on social media, and responded to questions about the type of content they shared, their motivation, and the content's ideal duration. We discuss our findings about short-term and long-term ephemerality as part of the social media experience, and the potential impact on the evolving identities of teenagers and young adults."



"Towards Examining The Effects of Live Streaming an Educational Game" -- Late Breaking Work


Noor Hammad, Erik Harpstead, Jessica Hammer


"We propose a study on the effects of live streaming on an educational game's learning outcomes. The COVID-19 pandemic has strengthened the call for interactive online learning experiences. There is a growing body of literature examining learning on platforms such as Twitch, and studies have shown that enhancing in-game performance is possible from viewing a stream. However, little work has explored whether learning from live streaming educational games, where in-game performance relates to educational outcomes outside of the game context, is possible. We share the details of our proposed study, in which an educational game called Angle Jungle will streamed to participants, and learning gains will be compared to three non-live streamed conditions. By executing this study, we can understand the benefits and shortcomings of current live streaming interfaces in supporting educational games, paving the way for the design of novel and interactive learning experiences built for live streaming platforms."



"Detecting Depression and Predicting its Onset Using Longitudinal Symptoms Captured by Passive Sensing: A Machine Learning Approach With Robust Feature Selection" -- Journal



We present a machine learning approach that uses data from smartphones and fitness trackers of 138 college students to identify students that experienced depressive symptoms at the end of the semester and students whose depressive symptoms worsened over the semester. Our novel approach is a feature extraction technique that allows us to select meaningful features indicative of depressive symptoms from longitudinal data. It allows us to detect the presence of post-semester depressive symptoms with an accuracy of 85.7% and change in symptom severity with an accuracy of 85.4%. It also predicts these outcomes with an accuracy of >80%, 11-15 weeks before the end of the semester, allowing ample time for preemptive interventions. Our work has significant implications for the detection of health outcomes using longitudinal behavioral data and limited ground truth. By detecting change and predicting symptoms several weeks before their onset, our work also has implications for preventing depression.

Chikersal, P., Doryab, A., Tumminia, M., Villalba, D. K., Dutcher, J. M., Liu, X., Cohen, S., Creswell, K. G., Mankoff, J., Creswell, J. D., Goel, M., & Dey, A. K. (2021) Detecting Depression and Predicting its Onset Using Longitudinal Symptoms Captured by Passive Sensing: A Machine Learning Approach With Robust Feature Selection. ACM Transactions on Computer-Human Interaction (TOCHI), 28, 1, Article 3 (January 2021).