Natural language processing tasks, such as question answering, machine translation,
reading comprehension, and summarization, are typically approached with supervised�…

Large language models are trained on massive scrapes of the web, which are often
unstructured, noisy, and poorly phrased. Current scaling laws show that learning from such�…

Abstract Language model pre-training has been shown to capture a surprising amount of
world knowledge, crucial for NLP tasks such as question answering. However, this�…

When pre-trained on large unsupervised textual corpora, language models are able to store
and retrieve factual knowledge to some extent, making it possible to use them directly for�…

Abstract Language models (LMs) exhibit remarkable abilities to solve new tasks from just a
few examples or textual instructions, especially at scale. They also, paradoxically, struggle�…

Coupled with the availability of large scale datasets, deep learning architectures have
enabled rapid progress on the Question Answering task. However, most of those datasets�…

Large language models are commonly trained on a mixture of filtered web data and
curated``high-quality''corpora, such as social media conversations, books, or technical�…

Learning what to share between tasks has been a topic of great importance recently, as
strategic sharing of knowledge has been shown to improve downstream task performance�…

Transfer learning, where a model is first pre-trained on a data-rich task before being fine-
tuned on a downstream task, has emerged as a powerful technique in natural language�…

Intermediate-task training---fine-tuning a pretrained model on an intermediate task before
fine-tuning again on the target task---often improves model performance substantially on�…