Building Effective Unstructured Data Systems (and other PhD reflections)

Event details Date: Tuesday, July 28, 2026 Time: 11:00 AM to 11:00 AM Location: Gateway 2133 Type: Lecture / Workshop Audience: Faculty,Students About this event Databases and other data systems have successfully democratized data-oriented computation across domains, thanks to decades of research in system internals and end-user interfaces. However, such systems center on structured (i.e., tabular) data; unstructured data—the vast majority of data—has largely been ignored. Large language models (LLMs) now give us a building block for unstructured data analysis, and we face the same questions as in the early days of data systems—e.g., how should users author queries? How do we efficiently execute queries at scale?—but many well-established tenets from traditional data systems no longer hold. In my talk, I will present DocETL, a system I developed for unstructured data analysis. I will discuss how we had to rethink query optimization under these new assumptions, optimizing user-written pipelines for both accuracy and efficiency—as well as end-user interfaces for authoring, iterating on, and debugging pipelines. DocETL is open-source with 3.5k+ GitHub stars; our hosted interface has supported 4.1k+ pipelines across 30+ S&P-500 industries. Query optimization ideas from our work have been adopted in databases such as Snowflake and BigQuery, and our interface design principles have been adopted by companies like LangChain and OpenAI. The advent of LLMs was very serendipitous; it allowed me and colleagues to help develop a new subfield in our community and, in turn, get a job. I will conclude my talk with a brief retrospective, for those considering going on the faculty job market. Official event details: https://events.berkeley.edu/eecs/event/324464-building-effective-unstructured-data-systems-and