In the previous articles, NLP: a comprehensive guide [Part 1] and NLP: a comprehensive guide [Part 2] , we introduced what natural language processing is, its possible applications, and the main techniques used to manipulate text and generate models. To conclude this overview we will explore some of the most important models and which programming languages and libraries to use to handle and generate NLP models. Finally, we will address some controversies that are emerging regarding the use of these models.
Six important models of natural language processing
Over the years, many NLP models have caused a stir within the AI community and some have made headlines. The most famous have been chatbots and language models. Here are some of them.
Eliza
This software was developed in the mid-1960s to try to solve the Turing Test, that is, to fool people into thinking they are conversing with another human being and not a machine. Eliza used pattern comparison and a set of rules without coding the context of the language.
Tay
This was the first chatbot launched by Microsoft in 2016. It was supposed to tweet like a teenager and learn from conversations with real users on Twitter. The bot adopted phrases from users who tweeted sexist and racist comments, and Microsoft deactivated it not long after. Tay illustrates some of the points of the “Stochastic Parrots” paper, particularly the danger of not analyzing the data.
BERT
Many deep learning models for NLP are named after Muppet characters, including ELMo, BERT, Big BIRD, ERNIE, Kermit, Grover, RoBERTa, and Rosita. Most of these models can provide contextual embeddings and better knowledge representation.
Generative Pre-Trained Transformer 3 (GPT-3)
GPT-3 is a 175 billion parameter model capable of writing original prose with human-equivalent fluency in response to an input request. The model is based on the transformer architecture. The previous version, GPT-2, is open source. Microsoft acquired an exclusive license to access the underlying model of GPT-3 from its developer OpenAI, but other users can interact with it through an application programming interface (API). Several groups, including EleutherAI and Meta, have released open source interpretations of GPT-3. To date, its GPT-4 version (paid only) is used in many application contexts.
Language Model for Dialogue Applications (LaMDA)
LaMDA is a conversational chatbot developed by Google. Its model is based on transformers trained on dialogue rather than the usual web text. The system aims to provide meaningful and specific responses to conversations. Blake Lemoine, a Google developer, has come to believe that LaMDA is sentient. Lemoine has had detailed conversations with the artificial intelligence about its rights and personality. During one of these conversations, the AI changed Lemoine’s mind about Isaac Asimov’s third law of robotics. Lemoine claimed that LaMDA was sentient, but the idea was disputed by many observers and commentators. Later, Google put Lemoine on administrative leave for distributing proprietary information and finally fired him.
Mixture of experts (MoE)
While most deep learning models use the same set of parameters to process each input, MoE models aim to provide different parameters for different inputs based on efficient routing algorithms to achieve higher performance. Switch Transformer is an example of a MoE approach that aims to reduce communication and computational costs.
Programming languages, libraries and frameworks for natural language processing
Many languages and libraries support NLP. Here are some of the most widely used.
Python
It is definitely the most widely used programming language to handle NLP tasks. Most of the libraries and frameworks for deep learning are written for Python. Here are some very useful libraries if you want to work in this field.
Natural Language Toolkit (NLTK)
It is one of the first NLP libraries written in Python. It provides easy-to-use interfaces to corpora and lexical resources such as WordNet. It also provides a suite of text processing libraries for classification, tagging, stemming, parsing and semantic reasoning.
spaCy
spaCy is one of the most versatile open source NLP libraries as it supports more than 66 languages. It also provides pre-trained word vectors and implements many popular patterns such as BERT. It can be used to build production-ready systems for named entity recognition, part-of-speech tagging, dependency parsing, sentence segmentation, text classification, lemmatization, morphological analysis, and entity recognition.
Deep learning libraries
Popular deep learning libraries include TensorFlow and PyTorch, which facilitate the creation of models with features such as automatic differentiation. These libraries are the most common tools for developing NLP models.
Hugging Face
This platform offers open-source implementations of more than 135 state-of-the-art models. The repository allows easy customization and definition of models.
Gensim
This library provides vector space modeling and topic modeling algorithms.
R
Many early NLP models were written in R, which is still widely used by data scientists and statisticians. Libraries in R for NLP include TidyText, Weka, Word2Vec, SpaCyR, TensorFlow and PyTorch. Despite this, the performance of this language compared to Python is significantly lower.
Many other languages, including JavaScript, Java, and Julia, have libraries that implement NLP methods.
Controversies over natural language processing
NLP has been at the center of numerous controversies. Some focus directly on the models and their results, others on second-order issues, such as access to these systems and the impact of their training on the natural world.
Stochastic parrots
A 2021 paper entitled “On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?” by Emily Bender, Timnit Gebru, Angelina McMillan-Major and Margaret Mitchell examines how language models can repeat and amplify biases found in their training data. The authors point out that the huge unedited datasets collected from the Web are bound to include social biases and other undesirable information, and the models that are trained on them will absorb these flaws. The researchers call for greater care in curating and documenting datasets, assessing the potential impact of a model before it is developed, and encouraging research in directions other than designing ever-larger architectures to ingest ever-larger datasets.
Consistency versus sensitivity
Recently, a Google engineer assigned to evaluate the LaMDA language model was so impressed by the quality of its chat output that he believed it was sentient. The fallacy of attributing human-like intelligence to AIs dates back to some of the earliest NLP experiments.
Environmental impact
Large language models require a lot of energy for both training and inference. One study estimated that training a single large language model can emit five times as much carbon dioxide as a single car over its operating life. Another study found that models consume even more energy during inference than during training. As for solutions, the researchers proposed the use of cloud servers located in countries with lots of renewable energy as one way to offset this impact.
High costs exclude non-corporate researchers
The computational requirements needed to train or deploy large language models are too expensive for many small companies. Some experts fear that this may block many capable engineers from contributing to innovation in AI.
Black box
When a deep learning model produces an output, it is difficult or impossible to know why it generated that particular result. While traditional models such as logistic regression allow engineers to examine the impact of individual features on the output, neural network methods in natural language processing are essentially black boxes. Such systems are called “unexplained” because we cannot explain how they arrived at their output. An effective approach to achieving model interpretability is particularly important in areas such as banking, where regulators want to confirm that a natural language processing system does not discriminate against certain groups of people, and law enforcement, where models trained on historical data can perpetuate historical biases against certain groups.
“Nonsense on stilts.”
Writer Gary Marcus has criticized NLP-based deep learning for generating sophisticated language that leads users to believe that natural language algorithms understand what they say and to mistakenly assume that they are capable of more sophisticated reasoning than is currently possible.
