Must Read NLP Papers from the Last 12 Months

The era of large language models is here now

Photo by Anil Sharma on Pexels

Since the groundbreaking release of BERT in October 2018, machine learning has achieved ever greater heights through clever optimization and augmented compute. BERT, which stands for Bidirectional Encoder Representations from Transformers, introduced a new paradigm in neural network architecture. The transformer has served as a significant unlock in machine learning capabilities.

Further advancements in the field of Natural Language Processing (NLP) have improved foreign language translation, enhanced no-code applications, increased the fluency of chatbots, and very quickly set new standards for an array of state-of-the art benchmarks.

Alongside these remarkable accomplishments, the development of large language models (LLMs) has not been without controversy. In the 2021 "Stochastic Parrots" paper, a team of researchers including machine learning engineer and ethicist Timnit Gebru criticized these models for:

• Levying a damning environmental cost
• Excluding marginalized voices through inelegant curation of the training data set
• Plagiarizing internet content and stealing from human writers