The fourth version of the Efficient Natural Language and Speech Processing (ENLSP-IV) workshop will focus on how to make large language and foundation models more efficient in terms of Architecture, Training, and Inference in their real-world applications. This year, following the trend of industry and academia, we put more emphasis on investigating new architectures to make future language and foundation models more efficient. Moreover, we highlight the importance of comprehensive evaluation and benchmarking new efficient models from different practical aspects. The workshop program offers an interactive platform for gathering experts and talents from academia and industry through invited talks, panel discussion, paper submission, reviews, interactive poster sessions, oral presentations and a couple of mentorship sessions for new researchers. This will be a unique opportunity to discuss and share challenging problems, build connections, exchange ideas and brainstorm, and foster future collaborations. The topics of this workshop can be of interest for people working on general machine learning, deep learning, hardware, optimization, theory and applications.
Overview
As large language models (e.g. GPT-3, GPT-4, Llama 3, PALM, Gemini, and Pangu-∑), pre-trained speech models (e.g. wav2vec, Hubert, wavLM, Whisper, Conformer-1 and Conformer-2 ) and other foundation models (e.g. GPT-4o, and Stable Diffusion) have advanced rapidly and become more prominent and widespread, improving their efficiency would be more crucial. While it is true that the computational power and GPU resources have played a significant role in the success of these models, we need to also be aware that using more computational resources can result in: (a) increasing the cost of training and deploying such models, (b) making the models less accessible, (c) less contribution from the research community, and (d) increasing the environmental costs of the models. Moreover, it is evident that most of these pre-trained models are largely over-parameterized and their efficiency is under question. Lack of efficiency can largely limit the application of these advanced models in practice.
Building upon the framework of our previous three editions, this workshop remains dedicated to investigating solutions for enhancing the efficiency of pre-trained language and foundation models but with introducing some fresh and important new topics to the community and encouraging their contributions. Just to highlight a few: (1) Despite the ubiquitous usage of Transformers, they suffer from quadratic computational complexity which limits their efficiency especially for longer sequence lengths. Should we improve the efficiency of Transformers (e.g. in Hedgehog, Gated Linear Attention) or look for other architectures (e.g. Mamba, Jamba, RVKW, xLSTM, and SSMs)? (2) For accelerating training, we have seen the significant impact of designing hardware efficient implementations such as in Flash Attention. Should we focus more on these hardware-aware solutions or more on new/improved architectures? (3) For efficient inference, there are solutions such as: Speculative Decoding [Link1] [Link2] where the performance is strongly model and task-dependent and the draft and target models should have the same vocabulary (tokenizer); improved KV-caching (e.g. [Link]) which has a limited speed-up; and many-in-one models such as SortedNet, MatFormer, and LayerSkip but the performance of sub-models drops compared to their corresponding individual models. (4) While there are many so-called efficient solutions in the literature, there is no fair, comprehensive and practical evaluation of these models and their comparison to each other. For example, we do not know the hallucination extent of the new architectures vs. the transformer model (e.g. in [Link]).
Call for Papers
Investing in the future of language and foundation models requires a concrete effort to enhance their efficiency across multiple dimensions (including architecture, training, and inference) and having a comprehensive evaluation framework.
To encourage engagement from the NeurIPS community, we present several active research topics in this field that invite participation and contributions. The scope of this workshop includes, but not limited to, the following topics:
Efficient Architectures Proposing alternative architectures that are more efficient than Transformers (in terms of computational complexity, memory footprint, handling longer sequence lengths ) or modifying Transformer architectures to make them more efficient
- Linear and sub-quadratic Transformers , sparse attention Transformers
- New architures for LLMs and foundation models and their scalability
- Evaluation and benchmarking of new architectures (fair comparison of different models)
- Long sequence modeling
- Dense vs. sparse architectures (MoEs)
- More efficient pre-training solutions, from better initialization and hyper-parameter tuning to better optimization which lowers the cost of pre-training
- Parameter efficient fine-tuning (PEFT) solutions for large pre-trained models
- Efficient instruction tuning, prompt engineering and in-context learning
- Hardware-aware solutions (e.g. better CUDA kernels), memory read/write aware solutions
- Data-efficient training, reducing the requirement for labeled data, data compression and distillation
- Improved speculative sampling for LLMs, self-speculative sampling, selecting among multiple drafts, one draft model for different heterogeneous target models
- Neural model compression techniques such as quantization, pruning, and knowledge distillation
- Improved KV-caching solutions for Transformers
- Distributed inference of large pre-trained models
- Serving many target devices with one model, many-in-one models, early exiting, elastic networks
- Datasets, benchmarks, leaderboards for evaluating efficient models
- Benchmarking the performance of efficient models from different perspectives such as reasoning, hallucination, understanding, and generation quality
- Benchmarking efficiency of models in terms of their memory footprint, training time, inference time, different target hardware devices and inference platforms (e.g. GPU vs. CPU)
- Efficiency of foundational or pre-trained models in multi-modal set-up and other modalities (beyond NLP and Speech) such as biology, chemistry, computer vision, and time series
- Efficient representations (e.g. Matryoshka representation) and models in dense retrieval and search
- Efficient Federated learning, lower communication costs, tackling heterogeneous data and models
- Efficient graph and LLM joint learning
Submission Instructions
You are invited to submit your papers in our CMT submission portal (Link). All the submitted papers have to be anonymous for double-blind review. We expect each paper will be reviewed by at least three reviewers. The content of the paper (excluding the references and supplementary materials) should not be more than 8 pages for Long Papers and 4 pages for Short Papers, strictly following the NeurIPS template style (Link). Please be advised that the NeurIPS submission checklist is not needed for our workshop submissions.
Authors can submit up to 100 MB of supplementary materials separately. Authors are highly encouraged to submit their codes for reproducibility purposes. According to the guideline of the NeurIPS workshops, already published papers are not encouraged for submission, but you are allowed to submit your ArXiv papers or the ones which are under submission (for example any NeurIPS submissions can be submitted concurrently to workshops ). Moreover, a work that is presented at the main NeurIPS conference should not appear in a workshop. Please make sure to indicate the complete list of conflict of interests for all the authors of your paper. To encourage higher quality submissions, our sponsors are offering the Best Paper and the Best Poster Awards to qualified outstanding original oral and poster presentations (upon nomination of the reviewers). Bear in mind that our workshop is not archival, but the accepted papers will be hosted on the workshop website. Moreover, we are currently negotiating with a publisher to host opt-in accepted papers in a special issue proceeding for our workshop.
Important Dates:
Special NeurIPS Fast Track Submission Deadline: September 30, 2024 Anywhere on Earth (AOE)Submission Deadline: September 15, 2024 Anywhere on Earth (AOE)Acceptance Notification: October 09, 2024 AOE- Camera-Ready Submission: October 25, 2024 AOE
- Workshop Date: December 14, 2024
Keynote Speakers
Danqi Chen
Princeton
Peter Clark
Allen Institute for AI
Weizhu Chen
Microsoft
Tri Dao
Princeton/Together AI
Hananeh Hajishirzi
University of Washington
Navdeep Jaitly
Apple
Lili Mou
University of Alberta
Panelists
Marjan Ghazvini Nejad
Meta
Lu Hou
Huawei
Joel Hestness
Cerebras
Katie Derthick
Microsoft
Tentative Schedule
Organizers
Volunteers
Technical Committee
|
|
Diamond Sponsors
Platinum Sponsor
Gold Sponsors
