Dr. Bhavana Dalvi Mishra is a Lead Research Scientist at the Allen Institute for AI (Ai2). Her research focuses on NLP, interactive reasoning, and scientific discovery. She obtained her Ph.D. in Computer Science from Carnegie Mellon University in 2015 and earned her Master's in Computer Science from the Indian Institute of Technology, Bombay in 2007. She has received several awards, including two Best Paper runner-up awards, Google Ph.D. fellowship, and Barbara Lazarus Women@IT Fellowship from CMU for her contributions to NLP and AI.

Despite the physiological limitations of the human brain, humans are remarkably efficient thinkers, in large part because they can learn from experience, allowing them to avoid prior reasoning errors and quickly jump to conclusions that previously took substantial effort. Similarly, language models (LMs) can rapidly improve their inference-time efficiency through inference-time learning, supplementing lower-level methods like fast decoding and caching. I'll describe two agent-based systems (CLIN and SSO) that do this, using an external RAG (retrieval-augmented generation) memory to help the agent navigate a complex, virtual environment. Unlike typical RAG systems, the memory is dynamic and updated after each task (including forgetting unhelpful learnings). In addition, unlike reinforcement-based continual learning techniques, these systems rapidly learn from just a handful of examples by exploiting LMs to conjecture useful generalizations of past experiences. I'll outline three critical activities in this process - what to remember, how to index those memories, and how to retrieve from that index - and how those choices impact the effectiveness of the resulting agent. While this concept of efficiency is a little different to foundational architectural considerations, I'll show that it is nonetheless powerful, and an important additional tool in the toolbox for efficient future applications.

Dr. Lili Mou is an Assistant Professor at the Department of Computing Science, University of Alberta. He is also an Alberta Machine Intelligence Institute (Amii) Fellow and a Canada CIFAR AI (CCAI) Chair. Lili received his BS and PhD degrees in 2012 and 2017, respectively, from School of EECS, Peking University. After that, he worked as a postdoctoral fellow at the University of Waterloo. His research interests mainly lie in designing novel machine learning algorithms and frameworks for NLP. He has publications at top conferences and journals, including ACL, EMNLP, TACL, ICML, ICLR, and NeurIPS. He also presented tutorials at EMNLP'19 and ACL'20. He received a AAAI New Faculty Highlight Award in 2021.

Knowledge distillation (KD) aims to transfer the knowledge in a large model (called a teacher) into a small one (called a student), and has become an emerging research topic as the sizes of deep learning models keep growing. Today, there are abundant readily available large models, such as ChatGPT, LLaMa, and T5. It then becomes natural to ask: Can we distill the knowledge from multiple teachers? At first glance, it appears easy to perform multi-teacher KD, as we can simply train the student from the union of teachers’ predictions. However, I would argue that such a naïve attempt may not work well for multi-teacher KD. This is because traditional KD adopts the cross-entropy loss, which tends to yield a smooth distribution. In this talk, I will present a novel ensemble-then-distill approach, which builds an ensemble of teacher models to train the student. I will also discuss applications to text generation and syntactic parsing.

Tri Dao is an Assistant Professor at Princeton University and chief scientist of Together AI. He completed his PhD in Computer Science at Stanford. He works at the intersection of machine learning and systems, and his research interests include sequence models with long-range memory and structured matrices for compact deep learning models. His work has received the COLM 2024 Outstanding paper award and ICML 2022 Outstanding paper runner-up award.

Transformers are slow and memory-hungry on long sequences, since the time and memory complexity of self-attention are quadratic in sequence length. We describe recent progress on subquadratic-time architectures such structured state space models (SSMs). We identify that a key weakness of such models is their inability to perform content-based reasoning, and propose a selection mechanism to address this shortcoming. Though this change prevents the use of efficient convolutions, we design a hardware-aware parallel algorithm in recurrent mode. We integrate these selective SSMs into a simplified end-to-end neural network architecture without attention or even MLP blocks. The resulting architecture (Mamba and Mamba-2) matches or exceeds the performance of strong modern Transformers on language modeling, validated at 3B scales on both pretraining and downstream evaluation, while enjoying 5x higher inference throughput and linear scaling in sequence length. Hybridizing Mamba layers with a 2-4 attention layers leads to state-of-the-art models, excelling at long context and fast inference.

Navdeep Jaitly is a Research Scientist at Apple Machine Learning Research (MLR) where he leads a team of researchers working on fundamental techniques in Machine Learning with an emphasis on speech and language. He got his PhD from University of Toronto, under the supervision of Geoffrey Hinton in the foundational days of Deep Learning. During a PhD internship at Google in 2011 he demonstrated how Deep Neural Networks would revolutionize speech recognition replacing HMM systems that were in use before. After his PhD he joined Google Brain working on sequence models and techniques such as Listen Attend and Spell, Adversarial Autoencoders and Pointer Networks. He has also held machine learning research positions at Nvidia, Google Brain Robotics (initiating robotic ping pong), D. E. Shaw and the National Labs.

It is well accepted now that speech needs to be tokenized before it can be modeled with transformer based generative models. In fact there is a rich body of intricate work using semantic and other acoustic tokens for speech modeling. In this talk we show how tokenization may not be necessary and that, indeed, a simple way of discretizing Mel-spectrograms (which we call d-Mel) is enough to build generative models with transformers. We show how we can build conditional generative models of speech (text-to-speech) using d-Mel and transformer based models. We also demonstrate that the same technique can be applied to multi-modal generation of speech conditioned on text and video. It is our hope that this leads to more exploration on minimal preprocessing of speech for use in generative modeling.

Danqi Chen is an assistant professor of Computer Science at Princeton University and co-leads the Princeton NLP group. She is also an Associate Director of Princeton Language and Intelligence. Her recent research focuses on training, adapting and understanding large language models, especially with the goal of making them more accessible to academia. Before joining Princeton, Danqi was a visiting scientist at Facebook AI Research. She received her Ph.D. from Stanford University (2018) and her B.E. from Tsinghua University (2012), both in Computer Science. Her research was recognized by a Sloan Fellowship, an NSF CAREER award, a Samsung AI Researcher of the Year award, and outstanding paper awards from ACL and EMNLP.

Training large language models relies heavily on the quality and composition of data, yet optimizing data selection and utilization remains a significant challenge in the field. In this talk, I will outline several key ideas to enhance training efficiency through better data use and cover several findings from my lab on selecting high-quality datasets and optimizing data compositions. I will also introduce a simple yet powerful pre-training approach that conditions on meta-data information associated with training data. This approach is remarkably straightforward to implement, incurs minimal computational overhead, and yields significant efficiency gains.

Woomin Song (KAIST),Jihoon Tack (KAIST),Sangwoo Mo (University of Michigan),Seunghyuk Oh (KAIST),Jinwoo Shin (KAIST)

State-space models (SSMs) offer a promising architecture for sequence modeling, providing an alternative to Transformers by replacing expensive self-attention with linear recurrences. In this paper, we propose a simple yet effective trick to enhance SSMs within given computational budgets by sparsifying them. Our intuition is that tokens in SSMs are highly redundant due to gradual recurrent updates, and dense recurrence operations block the delivery of past information. In particular, we observe that upper layers of SSMs tend to be more redundant as they encode global information, while lower layers encode local information. Motivated by this, we introduce Simba, a hierarchical sparsification method for SSMs based on token pruning. Simba sparsifies upper layers more than lower layers, encouraging the upper layers to behave like highways. To achieve this, we propose a novel token pruning criterion for SSMs, measuring the global impact of tokens on the final output by accumulating local recurrences. We demonstrate that Simba outperforms the baseline model, Mamba, with the same FLOPS in various natural language tasks. Moreover, we illustrate the effect of highways, showing that Simba not only enhances efficiency but also improves the information flow across long sequences.

Bo Liu (University of Texas, Austin), Rui Wang (Helixon),Lemeng Wu (University of Texas, Austin),Yihao Feng (University of Texas, Austin ),Peter Stone (University of Texas at Austin and Sony AI),Qiang Liu (UT Austin)

Modern large language models are built on sequence modeling via next-token prediction.
While the Transformer remains the dominant architecture for sequence modeling, its quadratic decoding complexity in sequence length poses a major limitation. State-space models (SSMs) present a competitive alternative, offering linear decoding efficiency while maintaining parallelism during training. However, most existing SSMs rely on linear recurrence designs that appear somewhat ad hoc.
In this work, we explore SSM design through the lens of online learning, conceptualizing SSMs as meta-modules for specific online learning problems. This approach links SSM design to formulating precise online learning objectives, with state transition rules derived from solving these objectives.
Based on this insight, we introduce a novel deep SSM architecture, Longhorn, whose update resembles the closed-form solution for solving the online associative recall problem. Our experimental results show that Longhorn outperforms state-of-the-art SSMs, including the Mamba model, on standard sequence modeling benchmarks, language modeling, and vision tasks. Specifically, Longhorn achieves a 1.8x improvement in sample efficiency compared to Mamba, and can extrapolate over contexts that are up to 16x longer during inference.

Hao Kang (Georgia Institute of Technology),Qingru Zhang (Georgia Institute of Technology)*,Souvik Kundu (Intel Labs),Geonhwa Jeong (Georgia Institute of Technology),Zaoxing Liu (University of Maryland),Tushar Krishna (Georgia Institute of Technology),Tuo Zhao (Gatech)

Key-value (KV) caching has become the de-facto technique to accelerate generation speed for large language models (LLMs) inference. However, the growing cache demand with increasing sequence length has transformed LLM inference to be a memory bound problem, significantly constraining the system throughput. Existing methods rely on dropping unimportant tokens or quantizing entries group-wise. Such methods, however, often incur high approximation errors to represent the compressed matrices. The autoregressive decoding process further compounds the error of each step, resulting in critical deviation in model generation and deterioration of performance. To tackle this challenge, we propose GEAR, an efficient error reduction framework that augments a quantization scheme with two error reduction components and achieves near-lossless performance at high compression ratios. GEAR first applies quantization to majority of entries of similar magnitudes to ultra-low precision. It then employs a low-rank matrix to approximate the quantization error, and a sparse matrix to remedy individual errors from outlier entries. By adeptly integrating three techniques, GEAR is able to fully exploit their synergistic potentials. Our experiments show that GEAR can maintain similar accuracy to that of FP16 cache with improvement up to 24.42% over the SOTA baselines at 2-bit compression. Additionally, compared to LLM inference with FP16 KV cache, GEAR can reduce peak-memory of up to $2.39\times$, bringing $2.1\times\sim 5.07\times$ throughput improvement. Our code will be publicly available.

Edoardo Cetin (Sakana AI)*,Qi Sun (tokyo institute of technology),Tianyu Zhao (Sakana AI),Yujin Tang (Sakana AI)

We introduce Neural Attention Memory Models (NAMMs) to improve the performance and efficiency of transformer foundation models. NAMMs are evolved atop pre-trained transformers to provide different latent contexts containing the most relevant information for individual layers and attention heads. NAMMs are universally applicable to any model using self-attention as they condition exclusively on the attention matrices produced in each layer. NAMMs learned on a relatively small set of problems substantially improve performance across multiple unseen long-context language tasks while cutting the model's input contexts up to a fraction of the original sizes, setting them apart from prior hand-designed KV cache eviction strategies that only aim to preserve model behavior. We show the generality of our conditioning enables zero-shot transfer of NAMMs trained only on language to entirely new transformer architectures even across input modalities, with their benefits carrying over to vision and reinforcement learning.

Niklas Muennighoff (Contextual AI/Allen Institute for Artificial Intelligence)*,Luca Soldaini (Allen Institute for Artificial Intelligence),Dirk Groeneveld (Allen Institute for Artificial Intelligence),Kyle Lo (Allen Institute for Artificial Intelligence),Jacob Morrison (Allen Institute for AI),Sewon Min (University of Washington),Weijia Shi (University of Washington),Pete Walsh (Allen Institute for Artificial Intelligence),Oyvind Tafjord (AI2),Nathan Lambert (Allen Institute for Artificial Intelligence),Yuling Gu (Allen Institute for Artificial Intelligence),Shane Arora (Allen Institute for Artificial Intelligence),Akshita Bhagia (Allen Institute for Artificial Intelligence),Dustin Schwenk (Allen Institute for Artificial Intelligence),David Wadden (Allen Institute for Artificial Intelligence),Alexander Wettig (Princeton University),Binyuan Hui (Alibaba Group),Tim Dettmers (Allen Institute for Artificial Intelligence),Douwe Kiela (Contextual AI),Noah Smith (Allen Institute for AI, University of Washington),Pang Wei Koh (Allen Institute for AI, University of Washington),Amanpreet Singh (Contextual AI),Hannaneh Hajishirzi (Allen Institute for AI, University of Washington)

We introduce OLMoE, a fully open, state-of-the-art language model leveraging sparse Mixture-of-Experts (MoE). OLMoE-1B-7B has 7 billion (B) parameters but uses only 1B per input token. We pretrain it on 5 trillion tokens and further adapt it to create OLMoE-1B-7B-Instruct. Our models outperform all available models with similar active parameters, even surpassing larger ones like Llama2- 13B-Chat and DeepSeekMoE-16B. We present novel findings on MoE training, define and analyze new routing properties showing high specialization in our model, and open-source all our work: model weights, training data, code, and logs.

Di Liu (Shanghai Jiao Tong University),Meng Chen (Fudan University),Baotong Lu (Microsoft Research)*,Huiqiang Jiang (Microsoft Research Asia),Zhenhua Han (Microsoft),Qianxi Zhang (MSRA),Qi Chen (Microsoft Research Asia),Chengruidong Zhang (MSFT),Bailu Ding (Microsoft Research),Kai Zhang (Fudan University),Chen Chen (Shanghai Jiao Tong University),Fan Yang (MSRA),Yuqing Yang (Microsoft),Lili Qiu (Microsoft Research Asia

Transformer-based Large Language Models (LLMs) have become increasingly important. However, due to the quadratic time complexity of attention computation, scaling LLMs to longer contexts incurs extremely slow inference latency and high GPU memory consumption for caching key-value (KV) vectors. This paper proposes RetrievalAttention, a training-free approach to both accelerate attention computation and reduce GPU memory consumption. By leveraging the dynamic sparsity of attention mechanism, RetrievalAttention proposes to use approximate nearest neighbor search (ANNS) indexes for KV vectors in CPU memory and retrieves the most relevant ones with vector search during generation. Unfortunately, we observe that the off-the-shelf ANNS indexes are often ineffective for such retrieval tasks due to the out-of-distribution (OOD) between query vectors and key vectors in attention mechanism. RetrievalAttention addresses the OOD challenge by designing an attention-aware vector search algorithm that can adapt to the distribution of query vectors. Our evaluation shows that RetrievalAttention only needs to access 1--3% of data while maintaining high model accuracy. This leads to significant reduction in the inference cost of long-context LLMs with much lower GPU memory footprint. In particular, RetrievalAttention only needs a single NVIDIA RTX4090 (24GB) for serving 128K tokens in LLMs with 8B parameters, which is capable of generating one token in 0.188 seconds.

Vui Seng Chua (Intel Corporation),Yujie Pan (Intel)*,Nilesh Jain (Intel)

We present Statistical Calibrated Activation Pruning (SCAP), a post-training activation pruning framework that (1) generalizes sparsification by input activations of Fully-Connected layers for generic and flexible application across Transformers, and (2) features a simple Mode-Centering technique to pre-calibrate activation distributions for maximizing post-training sparsity. Our results demonstrate robust Pareto efficiency compared to prior methods, translating to a 1.5× additional LLM decoding speedup against CATS[12] at iso model quality. SCAP effectiveness is empirically verified across a wide range of models, including recent Transformer Decoders, MoE, Mamba2, Encoding Transformer, and pre-quantized models, highlighting its practicality and scalability. The code is available at https://github.com/IntelLabs/SCAP.

Niklas Schmidinger (Johannes Kepler University Linz)*,Lisa Schneckenreiter (Johannes Kepler University, Linz),Philipp Seidl (JKU Linz),Johannes Schimunek (Johannes Kepler University Linz),Pieter-Jan Hoedt (LIT AI Lab, Institute for Machine Learning, Johannes Kepler University Linz, Austria),Johannes Brandstetter (LIT AI Lab, Institute for Machine Learning, Johannes Kepler University Linz, Austria),Andreas Mayr (Johannes Kepler University Linz),Sohvi Luukkonen (Johannes Kepler University),Sepp Hochreiter (LIT AI Lab, Institute for Machine Learning, Johannes Kepler University, NXAI GmbH, Linz, Austria),Guenter Klambauer (LIT AI Lab)

Language models for biological and chemical sequences enable crucial applications such as drug discovery, protein engineering, and precision medicine. Currently, these language models are predominantly based on Transformer architectures. While Transformers have yielded impressive results, their quadratic runtime dependency on sequence length complicates their use for long genomic sequences and in-context learning on proteins and chemical sequences. Recently, the recurrent xLSTM architecture has been shown to perform favorably compared to Transformers and modern state-space models (SSMs) in the natural language domain. Similar to SSMs, xLSTMs have linear runtime dependency and allow for constant-memory decoding at inference time, which makes them prime candidates for modeling long-range dependencies in biological and chemical sequences. In this work, we tailor xLSTM towards these domains and we propose a suite of language models called Bio-xLSTM. Extensive experiments in three large domains, genomics, proteins, and chemistry, were performed to assess xLSTM's ability to model biological and chemical sequences. The results show that Bio-xLSTM is a highly proficient generative model for DNA, protein, and chemical sequences, learns rich representations, and can perform in-context learning for proteins and small molecules.

Shashank Rajput (Databricks)*,Ying Sheng (NA),Sean Owen (Databricks),Vitaliy Chiley (Cerebras)

The size of the key-value (KV) cache plays a critical role in determining both the maximum context length and the number of concurrent requests supported during inference in modern language models. The KV cache size grows proportionally with the number of attention heads and the tokens processed, leading to increased memory consumption and slower inference for long inputs. In this work, we explore the use of MixAttention, a model architecture modification closely related to a blog published by Character.AI. MixAttention combines sliding window attention, where only a small subset of recent tokens is stored in the KV cache, with KV cache sharing across layers. Our experiments demonstrate that MixAttention significantly reduces memory usage and improves inference speed without sacrificing model performance in both short and long-context tasks. We also explore various configurations of this architecture, identifying those that maintain quality across evaluation metrics while optimizing resource efficiency.

Fabian Paischer (ELLIS Unit Linz, LIT AI Lab, Institute for Machine Learning, Johannes Kepler University Linz, Austria)*,Lukas Hauzenberger (ELLIS Unit Linz, LIT AI Lab, Institute for Machine Learning, Johannes Kepler University Linz),Thomas Schmied (ELLIS Unit Linz, LIT AI Lab, Institute for Machine Learning, Johannes Kepler University Linz),Benedikt Alkin (Institute for Machine Learning),Marc Deisenroth (University College London),Sepp Hochreiter (LIT AI Lab, In

Abstract TBD

Weizhu Chen Weizhu Chen is the Vice President leading the Microsoft GenAI modeling team, driving innovation in large-scale AI model training, including pre-training, post-training, and evaluation for both Microsoft and OpenAI. Under his leadership, the team has pioneered groundbreaking advancements such as LoRA, DeBERTa, and Phi-3 models. With over 19 years at Microsoft, Weizhu has held pivotal roles in shaping AI and machine learning technologies. Previously, he served as Partner Science Manager at Microsoft Azure AI and led teams in the Business Applications Group and Research divisions, focusing on deep learning, NLP, and distributed machine learning at cloud scale. Before joining Microsoft, he contributed to research on information retrieval at IBM Research. Weizhu’s career reflects a deep commitment to advancing the state of AI, making a significant impact on the field and enabling transformative technologies.

Abstract TBD

Hananeh Hajishirzi, Hanna Hajishirzi is the Torode Family Associate Professor in the Allen School of Computer Science and Engineering at the University of Washington and a Senior Director of NLP at AI2. Her current research delves into various domains within Natural Language Processing (NLP) and Artificial Intelligence (AI), with a particular emphasis on accelerating the science of language modeling, broadening their scope, and enhancing their applicability and usefulness for human lives. She has published over 140 scientific articles in prestigious journals and conferences across ML, AI, NLP, and Computer Vision. She is the recipient of numerous awards, including the Sloan Fellowship, NSF CAREER Award, Intel Rising Star Award, Allen Distinguished Investigator Award, Academic Achievement UIUC Alumni Award, and Innovator of the Year Award by GeekWire. The work from her lab has been nominated for or has received best paper awards at various conferences and has been featured in numerous magazines and newspapers.

Language models (LMs) have become ubiquitous in both AI research and commercial product offerings. As their commercial importance has surged, the most powerful models have become closed off, gated behind proprietary interfaces, with important details of their training data, architectures, and development undisclosed. In this talk, I present our OLMo project aimed at building strong language models and making them fully accessible to researchers along with open-source code for data, training, and inference. Training language models are expensive, therefore we optimize for quality vs. compute cost. I focus on how data, architecture, and training improvements advance models at pre-training and post-training stages with less compute cost.