Human-Computer Interaction Ph.D. Thesis Defense - Franklin Mingzhe Li

Many Activities of Daily Living (ADLs) and Instrumental Activities of Daily Living (IADLs), such as cooking, are highly visual and create barriers for people with vision impairments. With over 2.2 billion people globally affected by vision loss, these challenges significantly impact autonomy, health, and overall quality of life. Cooking, in particular, involves navigating the kitchen, identifying ingredients, and following recipes, which often leads to reliance on unhealthy, costly pre-packaged meals. This contributes to a 150% higher obesity rate among people with vision impairments. Despite advancements in assistive technology, there remains a critical gap in providing solutions that address non-visual interaction with the physical information space required for complex tasks like cooking.

This thesis addresses that gap by asking how AI-enabled assistive technologies can support non-visual interaction with physical activities, using cooking as a concrete and demanding setting. For an AI-based system to be useful in a kitchen, it needs structured information about objects, environments, and user actions; a generalized text or vision model alone cannot supply the contextual information a blind cook actually needs at the moment they need it. I argue that progress requires first architecting a physical information space for non-visual cooking, a structured account of what to track, why it matters, and how it should be communicated, and then building AI-driven systems on top of that representation.

The thesis is structured around six research efforts. First, a content analysis of 122 YouTube videos and semi-structured interviews with 12 blind cooks established the practices and challenges of non-visual cooking, identifying eight key challenges that current technology fails to address. Second, an interview study with 20 blind cooks and four rehabilitation instructors produced a design framework for accessible recipes, characterizing what content, structure, and technological support a recipe needs in order to be usable non-visually. Third, an in-home contextual inquiry with 12 blind cooks produced a taxonomy of the contextual information about kitchen objects (five primary, two secondary, and one application-specific category) that is required to act safely and confidently in the kitchen. Fourth, OSCAR (Object Status Context Awareness for Recipes) operationalized one piece of that information space: it extracts object-status transitions from recipes, aligns them with cooking video, and uses a time-causal model to track recipe progress, improving step-prediction accuracy by more than 20% over text-only baselines on both YouCook2 and a newly collected dataset of 12 real-world non-visual cooking sessions.

Building on these four studies, the thesis introduces the Object Dependency Graph (ODG), a unifying representation of a recipe that integrates object status, prerequisites, safety, non-visual cues, measurements, progress tracking, and parallelism into a single, machine-readable structure. The ODG is decomposed into seven design rules (R1—R7), each explicitly traceable to one of the prior chapters, and is implemented as a voice-first cooking assistant paired with a researcher dashboard. I evaluated the ODG-based assistant in a within-subjects remote study with 10 blind participants comparing it against a screen-reader baseline across nine recipes spanning multi-stage, linear, and high-parallelism cooking flows. The study used matched task probes, custom Likert measures mapped to specific ODG rules, and a closing semi-structured interview.

The evaluation shows that the ODG materially improves the dimensions of non-visual cooking that screen readers currently fail to support: hazard awareness, parallel-task identification, and measurement and technique clarity all improved substantially under the ODG condition, while progress orientation showed a mixed pattern that points to concrete onboarding and verbosity refinements. Participants also articulated when they would and would not adopt a voice-first assistant, surfacing populations the system does not yet serve well. Together, the chapters of this thesis show that non-visual cooking is best supported not by better screen readers or better vision models in isolation, but by an explicit information space, the ODG, that ties what a system can perceive to what a blind cook actually needs to know while cooking.