HCII Ph.D. Thesis Proposal: Alex Cabrera
When
-
Description
Time and Location:
July 20, 2023 - 12pm ET
Gates & Hillman Centers (GHC) 6121
Zoom Link
Thesis Committee:
Adam Perer (Co-Chair), HCII, CMU
Jason I. Hong (Co-Chair), HCII, CMU
Kenneth Holstein, HCII, CMU
Ameet Talwalkar, MLD, CMU
Aditya Parameswaran, EECS, UC Berkeley
Abstract:
Massive AI systems that can be guided with text prompts or fine-tuned with small datasets are enabling millions of people to prototype complex AI products. But it remains a significant challenge to go from an initial prototype to a robust, deployable model that is equitable, safe, and works for most users and edge cases. This challenge is compounded by the increasingly complex tasks for which AI is used, such as text and image generation, which do not have clearly defined metrics or evaluation methods. As it becomes easier and faster to create candidate AI systems, the brunt of development work moves from getting a system working towards the design problem of which AI system should be built and how it should behave.
This thesis proposes an AI development philosophy called behavior-driven AI development (BDAI) that centers the AI development lifecycle on the desired behaviors of complex AI systems. By centering development on a model's desired behaviors instead of the training data and model architecture, developers can focus on creating responsible AI systems that best fulfill end-users' needs. This dissertation presents four major research contributions defining and validating the BDAI framework. I first describe qualitative interview studies with 27 practitioners investigating how they understand and improve behaviors of complex AI systems. Next, I introduce a theoretical framework that describes this process as a form of sensemaking and show how the framework can be used to describe and create AI development tools. I further show how insights into model behavior can be shown to end users to improve human-AI collaboration by calibrating end-users reliance on model outputs. Lastly, I use the sensemaking framework to create a general-purpose evaluation tool, Zeno, built specifically for behavior-driven development. To complete my dissertation, I propose conducting longitudinal case studies with practitioners using Zeno in deployment to understand better how BDAI evolves across an AI system's lifecycle.
Proposal document:
https://drive.google.com/file/d/1p_gp1Rhhd18fUGstPL5BXqufWzGowCdE/view?…