| layout | title |
|---|---|
notes |
presentations |
Links to many of my presentations
- Using LLMs to bridge data-driven models and scientific theories in language neuroscience
UCSF talk Title: Using LLMs to bridge data-driven models and scientific theories in language neuroscience
Abstract: Science faces an explainability crisis: data-driven deep learning methods are proving capable of *predicting* many natural phenomena but not at *explaining* them. One emblematic field is language neuroscience, where LLMs are highly effective at predicting human brain responses to natural language, but are virtually impossible to interpret or analyze by hand. To overcome this challenge, we introduce a framework that translates deep learning models of language selectivity in the brain into concise verbal explanations, and then designs follow-up experiments to verify that these explanations are causally related to brain activity. This approach is successful at explaining selectivity both in individual voxels and cortical regions of interest, demonstrating that LLMs can be used to bridge the widening gap between data-driven models and formal scientific theories. This talk covers 2 papers: Benara et al. ([NeurIPS 2024](https://arxiv.org/abs/2405.16714v1)) & Antonello et al. ([arXiv, 2024](https://arxiv.org/abs/2410.00812)). -MSR project green talk
Title: From Data-Driven Models to Scientific Theories: A Case Study in Language Neuroscience
Abstract: Modern data-driven methods are proving capable of predicting many natural phenomena, but not at explaining them. This talk will cover a case study where LLMs can be carefully used to convert predictive models of the human brain into interpretable, testable scientific theories. -
-
- Title: Uniting Large Language Models and Decision Trees
-
Abstract 1
Decision trees are the cornerstone for a wide range of applications, especially in tabular data, where they are often used as a transparent model However, decision trees can fail to model complex interactions and dependencies, an area where modern large language models (LLMs) excel. In this talk, I will discuss recent works that unite decision trees and LLMs to bring out the best in both for NLP applications. Specifically, I will discuss how decision trees can be used to steer LLMs by structuring sequential prompted calls, and how LLMs can be used to improve transparent decision trees by augmenting individual nodes with relevant features. -
Abstract 2
Decision trees are a pillar of modern machine learning, forming a foundation for transparent, accurate decision making. However, decision trees often fail to model complex interactions, an area where modern large language models (LLMs) excel. In this talk, I will discuss our recent works that unite decision trees and LLMs to bring out the best in both for NLP applications. Specifically, I will discuss (1) how how LLMs can be used to improve transparent decision trees by augmenting individual nodes with relevant features and (2) how decision trees can be used to steer LLMs by structuring sequential prompted calls.
- animated thesis
- thesis slides
- Title: Useful interpretations for machine learning in science & medicine
-
Abstract Machine-learning models have achieved impressive predictive performance by learning complex functions of many variables. However, this increase in complexity has often come at the cost of interpretability, a crucial consideration in many domains, such as science and medicine. In this talk, I will discuss recent work that extracts useful interpretations from machine-learning models by scoring and distilling interactions. Then, I will showcase how we have worked with domain experts to improve models using these interpretations in different computer vision settings, including bioimaging and cosmology. - research overview
- overview slide
- title slide
Abstract
Deep learning models have achieved impressive predictive performance by learning complex functions of many variables, often at the cost of interpretability. I will discuss a recent line of work aiming to interpret neural networks by attributing importance to features and feature interactions for individual predictions. Importantly, the proposed methods disentangle the importance of features in isolation and the interactions between groups of features. These attributions significantly enhance interpretability and can be used to directly improve generalization in interesting ways. I will showcase how we have worked with domain experts to make these attributions useful in different computer vision settings, including in bioimaging and cosmology.- qual slides
- acd paige slides
- Title: Disentangled interpretations for deep learning
-
Abstract
Deep learning models have achieved impressive predictive performance by learning complex functions of many variables, often at the cost of interpretability. I will discuss a recent line of work aiming to interpret neural networks by attributing importance to features and feature interactions for individual predictions. Importantly, the proposed methods disentangle the importance of features in isolation and the interactions between groups of features. These attributions significantly enhance interpretability and can be used to directly improve generalization in interesting ways. I will showcase how we have worked with domain experts to make these attributions useful in different computer vision settings, including in bioimaging and cosmology.
- dl joint reading group slides
- Title: Disentangled interpretations for deep learning with ACD
-
Abstract
Deep learning models have achieved impressive predictive performance by learning complex functions of many variables, often at the cost of interpretability. I will discuss our recent works aiming to interpret neural networks by attributing importance to features and feature interactions for individual predictions. Importantly, the proposed method (named agglomerative contextual decomposition, or ACD) disentangles the importance of features in isolation and the interactions between groups of features. These attributions yield insights across domains, including in NLP/computer vision and can be used to directly improve generalization in interesting ways.
-
- Paper links: hierarchical interpretations [(ICLR 2019)](https://openreview.net/pdf?id=SkEqro0ctQ), interpreting transformations in cosmology [(ICLR workshop 2020)](https://arxiv.org/abs/2003.01926), penalizing explanations [(ICML 2020)](https://github.com/laura-rieger/deep-explanation-penalization)
Summary
We focus on a problem in cosmology, where it is crucial to interpret how a model trained on simulations predicts fundamental cosmological parameters. By extending ACD to interpret transformations of input features, we vet the model by analyzing attributions in the frequency domain. Finally, we discuss ongoing work using ACD to develop simple transformations (e.g. adaptive wavelets) which can be both predictive and interpretable for cosmological parameter prediction. - bair sem slides
- Title: Interpreting and Improving Neural Networks via Disentangled Attributions
-
Abstract Machine learning models have achieved impressive predictive performance by learning complex functions of many variables. However, the inability to effectively interpret these functions has limited their use across many fields, such as science and medicine. This talk will cover a recent line of work aiming to interpret models by attributing importance to features / feature groups for a single prediction. Importantly, the proposed attributions disentangle the importance of features in isolation and the interactions between groups of features. These attributions are shown to yield insights across domains, including an NLP classification task and in understanding cosmological models. Moreover, these attributions can be used during training to improve generalization in interesting ways, such as forcing an image classification model to focus on shape instead of texture. The talk will place a large emphasis on rigorously evaluating and testing the proposed attributions to assure they are properly describing the model. - acd + interpretable ml talk
- nlp version
- interpretable ml (discussion slides for group meeting)
- 15 min overall talk
- cd + acd + cdep 5 min talk
- biohub meeting (w/ wooseok)
- dudoit group meeting
- simons 2020 talk (announcement)
- Title: Disentangled interpretations and how we can use them
-
Abstract Recent machine learning models have achieved impressive predictive performance by learning complex functions of many variables, often at the cost of interpretability. This talk will cover recent work aiming to interpret models by attributing importance to features / feature groups for a single prediction. Importantly, the proposed attributions disentangle the importance of features in isolation and the interactions between groups of features. These attributions are shown to yield insights across domains, including an NLP classification task and in understanding cosmological models. Moreover, these attributions can be used during training to directly improve generalization in interesting ways. The talk will place a large emphasis on how to cogently define and evauate the proposed interpretations.
- acd paige slides
- cogsci acd slides
- acd white-theme slides for bin
- acd bavrd (3 min pres. given at bavrd 2019)
- cdep-focused slides 40 min
- trim_cosmology_group_meeting (03/17/21)
- awd (neurips 15 min)
- pdr
- auxilin prediction
- interpretation to causation
- covid-forecasting
-
- hierarchical shrinkage (group meeting)
-
iai (group meeting)
- iai-rulevetting (215A presentation)
- faces final pres
- sensible local interpretations (5 min update for pacmed internship)
- semantic segmentation (facebook pres)
- dnn experiments
- alphafold 2
- vissci ovw
- harms of ai (1 hr talk given in possible minds course 2019)
- hummingbird tracking
- wsimule urds (5 min talk given at URDS 2017)
- wsimule tomtom (10 min talk given at Tom Tom 2017)
- wsimule slide (1 min pres. given at AMLCID Neurips 2017 workshop)
- linearization (5 min talk given at URDS 2017)
- sparse coding class pres
- talks
- other talks list
- trustml symposium talk (2022)
- frontiers in deep learning workshop (msr, 2022)
- cnls physics-informed ml conference (2022)
- disentangled interpretations and how we can use them (simons workshop, 2020)
- tom tom founder's ml conference (2017)
- misc releases
One-paragraph bio (nov 2023)
Chandan is a senior researcher at Microsoft Research, where he works on interpretable machine learning with the broad goal of improving science and medicine using data. Recently, he has focused on language models and how they can be used to directly explain data or to improve transparent models. He completed his Computer Science PhD from UC Berkeley in 2022.One-paragraph bio (jan 2022)
Deep learning models have achieved impressive predictive performance by learning complex functions of many variables, often at the cost of interpretability. I will discuss a recent line of work aiming to interpret neural networks by attributing importance to features and feature interactions for individual predictions. Importantly, the proposed methods disentangle the importance of features in isolation and the interactions between groups of features. These attributions significantly enhance interpretability and can be used to directly improve generalization in interesting ways. I will showcase how we have worked with domain experts to make these attributions useful in different computer vision settings, including in bioimaging and cosmology.2-sentence bio (feb 2022)
Chandan is a fifth and final-year PhD student in computer science. He hopes to build on recent advances in machine-learning to improve the world of healthcare. His research focuses on how to interpret machine-learning models with the goal of ensuring that they can be reliably used when someone’s health is at stake.4-paragraph research overview (feb 2022)
🔎 My research focuses on how we can build trustworthy machine-learning systems by making them interpretable. In my work, interpretability is grounded seriously via close collaboration with domain experts, e.g. medical doctors or cell biologists. These collaborations have given rise to useful methodology, roughly split into two areas: (1) building more effective transparent models and (2) improving the trustworthiness of black-box models. Going forward, I hope to help bridge the gap between transparent models and black-box models to improve real-world healthcare.
🌳 Whever possible, building transparent models is the most effective route towards ensuring interpretability. Transparent models are interpretable by design, including models such as (concise) decision trees, rule lists, and linear models. My work in this area was largely motivated by the problem of clinical decision-rule development. Clinical decision rules (especially those used in emergency medicine), need to be extremely transparent so they can be readily audited and used by physicians making split-second decisions. To this end, we have developed methodology for enhancing decision trees. For example, replacing the standard CART algorithm with a novel greedy algorithm for tree-sums can substantially improve predictive performance without sacrificing predictive performance. Additionally, hierarchical regularization can improve the predictions of an already fitted model without altering its interpretability. Despite their effectiveness, transparent models such as these often get overlooked in favor of black-box models; to address thie issue, we've spent a lot of time curating imodels, an open-source package for fitting state-of-the-art transparent models.
🌀 My second line of my work focuses on interpreting and improving black-box models, such as neural networks, for the cases when a transparent model simply can't predict well enough. Here, I work closely on real-world problems such as analyzing imaging data from cell biology and cosmology. Interpretability in these contexts demands more nuanced information than standard notions of "feature importance; common in the literature. As a result, we have developed methods to characterize and summarize the interactions in a neural network, particularly in transformed domains (such as the Fourier domain), where domain interpretations can be more natural. I'm particularly interested in how we can ensure that these interpretations are useful, either by using them to embed prior knowledge into a model or identify when it can be trusted.
🤝 There is a lot more work to do on bridging the gap between transparent models and black-box models in the real world. One promising avenue is distillation, whereby we can use a black-box model to build a better transparent model. For example, in one work we were able to distill state-of-the-art neural networks in cell-biology and cosmology into transparent wavelet models with <40 parameters. Despite this huge size reduction, these models actually improve prediction performance. By incorporating close domain knowledge into models and the way we approach problems, I believe interpretability can help unlock many benefits of machine-learning for improving healthcare and science.
Read research overview (interpretable modeling)
🔎 My research focuses on how we can build trustworthy machine-learning systems by making them interpretable. In my work, interpretability is grounded seriously via close collaboration with domain experts, e.g. medical doctors or cell biologists. These collaborations have given rise to useful methodology, roughly split into two areas: (1) building more effective transparent models and (2) improving the trustworthiness of black-box models. Going forward, I hope to help bridge the gap between transparent models and black-box models to improve real-world healthcare.
🌳 Whenever possible, building transparent models is the most effective route towards ensuring interpretability. Transparent models are interpretable by design, including models such as (concise) decision trees, rule lists, and linear models. My work in this area was largely motivated by the problem of clinical decision-rule development. Clinical decision rules (especially those used in emergency medicine), need to be extremely transparent so they can be readily audited and used by physicians making split-second decisions. To this end, we have developed methodology for enhancing decision trees. For example, replacing the standard CART algorithm with a novel greedy algorithm for tree-sums can substantially improve predictive performance without sacrificing predictive performance. Additionally, hierarchical regularization can improve the predictions of an already fitted model without altering its interpretability. Despite their effectiveness, transparent models such as these often get overlooked in favor of black-box models; to address this issue, we've spent a lot of time curating imodels, an open-source package for fitting state-of-the-art transparent models.
🌀 My second line of my work focuses on interpreting and improving black-box models, such as neural networks, for the cases when a transparent model simply can't predict well enough. Here, I work closely on real-world problems such as analyzing imaging data from cell biology and cosmology. Interpretability in these contexts demands more nuanced information than standard notions of "feature importance" common in the literature. As a result, we have developed methods to characterize and summarize the interactions in a neural network, particularly in transformed domains (such as the Fourier domain), where domain interpretations can be more natural. I'm particularly interested in how we can ensure that these interpretations are useful, either by using them to embed prior knowledge into a model or identify when it can be trusted.
🤝 There is a lot more work to do on bridging the gap between transparent models and black-box models in the real world. One promising avenue is distillation, whereby we can use a black-box model to build a better transparent model. For example, in one work we were able to distill state-of-the-art neural networks in cell-biology and cosmology into transparent wavelet models with <40 parameters. Despite this huge size reduction, these models actually improve prediction performance. By incorporating close domain knowledge into models and the way we approach problems, I believe interpretability can help unlock many benefits of machine-learning for improving healthcare and science.
1000-character summary of MSR work (may 2024)
LLM Interpretability/Steering I led the Aug-imodels paper, which leverages LLMs to build transparent models, achieving dramatic (>1000x) efficiency improvements for a large set of text-classification tasks. I have additionally led efforts to steer LLMs, both for black-box models (Tree-Prompt) and for open-source models (Attention steering, intern's paper).
Next-generation transformers [still exploratory] I have been exploring how to extract useful insights from brain representations of language (through SASC and 2 major upcoming submissions). My current focus is on how to leverage these insights in "Hyperdimensional LMs", an architecture that heavily uses retrieval to build CPU-only LMs. Early results are showing some potential, and I'm hoping to get GPT-2 level performance in the next few months, which would be a huge cost-saving for many simple tasks that Microsoft could use in deployment.
I have additionally been involved in broader efforts, including an ongoing collaboration with health futures on using LLMs on clinical text (which has produced a couple papers already) and some one-off ideas (e.g. learning a decision tree algorithm with transformers).