Publications

Nugget-based evaluation methods have emerged as the standard for measuring relevance of long-form RAG responses. We argue that nuggets are valuable also for guiding retrieval and generation, and we present a nugget-centric RAG system that automatically constructs its own nugget bank and uses it as a control signal throughout the pipeline.

We present the first pipeline for automatic, difficulty-controlled creation of uncheatable, realistic, unanswerable, and multi-hop queries (CRUMQs), adaptable to any corpus and domain. This offers a simple way to enhance benchmark difficulty and realism and drive development of more capable RAG systems.

Report generation (RG) is a RAG task that aims to produce a long-form, citation-attributed response to a complex user query. We present the first public, automated, LLM-based implementation of the ARGUE evaluation framework for RG.

LLM benchmarks are essential for tracking progress and ensuring safety in AI, but most benchmarks don’t measure what matters, as suggested by our systematic review of 445 LLM benchmarks from top AI conferences. A taxonomy of these failures was therefore built and translated into an operational checklist to help future benchmark authors demonstrate construct validity.

For LLMs to be deployed reliably and responsibly, it is essential that their linguistically expressed confidence faithfully reflect their internal uncertainty. This paper presents the first study to systematically and comprehensively benchmark faithful calibration of LLMs and proposes MetaFaith, the first method to improve faithful calibration of any instruction-following LLM in a task-agnostic manner.

Multi-document (MD) processing is crucial for LLMs to handle real-world tasks across large sets of documents. This paper introduces MDCure, an effective and scalable procedure for generating high-quality multi-document instruction tuning data to improve MD capabilities of any base LLM.

Metacognition of emotion refers to the ability to evaluate success at identifying one’s own emotional feelings and adjusting them accordingly. This paper establishes the first experimentally-validated procedure to reliably assess and quantify metacognition of emotion.

This paper discusses the social implications of reinforcement learning with human feedback (RLHF), identifying key social and ethical issues and discussing social impacts for stakeholders. Seven impact areas are examined, including misinformation, AI value-alignment, bias, AI access, cross-cultural dialogue, industry, and workforce.

An undergraduate-level curriculum that cultivates understanding of the robot rights debate as informed by Western and Eastern principles, stepping toward filling existing gaps in knowledge and discourse and uniting the global community. Constructs and works through a foundational framework to engage students in critical thinking, understanding, and discussion of decisions regarding robot rights.

We define the Dynamical Chinese Restaurant Process (Dynamical CRP), a novel stochastic process that provides a non-stationary prior over cluster assignments and yields an efficient streaming variational inference algorithm. Experiments show the Dynamical CRP can be applied on diverse synthetic and real data with Gaussian and non-Gaussian likelihoods.

R-IBP is a novel recursive form of the Indian Buffet Process that makes feature models applicable to streaming data. It enables creation of new features online and in a probabilistic, principled manner. As a prior for feature models, R-IBP yields efficient inference over an unbounded number of latent features, with quasilinear average time complexity and logarithmic average space complexity.

The first experimentally quantitative index of metacognition of emotion. Advancing understanding of awareness toward subjective feelings.

Weight friction draws from principles in neuroscience and physics to help mitigate catatstrophic forgetting in neural networks. It operates over multiple task domains and has convergence comparable to SGD.

GFCCs can be powerful speech features for use in speech emotion recognition. This preliminary study suggests they are more suitable versus MFCCs for emotion and intensity classification tasks with neural networks.