Pre-trained models: Past, present and future

The early efforts of pre-training are mainly involved in transfer learning (Thrun and Pratt, 1998). The study of transfer learning is heavily motivated by the fact that people can rely on previously learned knowledge to solve new problems and even achieve better results. More formally, transfer learning aims to capture important knowledge from multiple source tasks and then apply the knowledge to a target task.

In transfer learning, source tasks and target tasks may have completely different data domains and task settings, yet the knowledge required to handle these tasks is consistent (Pan and Yang, 2009). It is thus important to select a feasible method to transfer knowledge from source tasks to target tasks. To this end, various pre-training methods have been proposed to work as the bridge between source and target tasks. Specifically, these methods first pre-train models on the data of multiple source tasks to pre-encode knowledge and then transfer the pre-encoded knowledge to train models for target tasks.

Generally, two pre-training approaches are widely explored in transfer learning: feature transfer and parameter transfer. Feature transfer methods pre-train effective feature representations to pre-encode knowledge across domains and tasks (Johnson and Zhang, 2005; Evgeniou and Pontil, 2007; Dai et al., 2007; Raina et al., 2007). By injecting these pre-trained representations into target tasks, model performance of target tasks can be significantly improved. Parameter transfer methods follow an intuitive assumption that source tasks and target tasks can share model parameters or prior distributions of hyper-parameters. Therefore, these methods pre-encode knowledge into shared model parameters (Lawrence and Platt, 2004; Evgeniou and Pontil, 2004; Williams et al., 2007; Gao et al., 2008), and then transfer the knowledge by fine-tuning pre-trained parameters with the data of target tasks.

Since AlexNet (Krizhevsky et al., 2012), a series of deep neural networks have been developed for AI tasks. As compared with those conventional machine learning models, deep neural models have more parameters and show better capabilities of fitting complex data. Therefore, from AlexNet to later VGG (Simonyan and Zisserman, 2015) and GoogleNet (Szegedy et al., 2015), the architecture of these neural networks becomes deeper and deeper, and their performance accordingly becomes better and better. Although the network depth is important, training a deep network is not easy, as stacking more network layers inevitably brings the problem of vanishing or exploding gradients (Bengio et al., 1994). Besides the gradient issues, model performance may soon meet a ceiling and then degrade rapidly with continually increasing network depths.

By adding normalization to parameter initialization (LeCun et al., 2012; Saxe et al., 2013) and hidden states (Ioffe and Szegedy, 2015), and introducing shortcut connections with residual layers, ResNet (He et al., 2016) effectively tackles these problems. As we mentioned before, deep neural networks require large amounts of data for training. To provide sufficient data to train deep models, some large-scale supervised datasets have also been built (Russakovsky et al., 2015; Lin et al., 2014; Krishna et al., 2017; Chen et al., 2015; Cordts et al., 2016), and the most representative one is ImageNet. ImageNet contains millions of images divided into thousands of categories, representing a wide variety of everyday objects. Based on the combination of effective model ResNet, informative dataset ImageNet, as well as mature knowledge transfer methods, a wave of pre-training models on labeled data emerges.

The CV community benefits a lot from this wave. By applying ResNet pre-trained on ImageNet as the backbone, various CV tasks have been quickly advanced, like image classification (He et al., 2016; Lee et al., 2015), object detection (Ren et al., 2016; Sermanet et al., 2014; Gidaris and Komodakis, 2015), image segmentation (Long et al., 2015; Zheng et al., 2015), image caption (Vinyals et al., 2015; Johnson et al., 2016), visual question answering (Antol et al., 2015; Gao et al., 2015; Xiong et al., 2016), etc. Utilizing PTMs like ResNet50³ has proven to be a crucial step to obtain highly accurate results on most CV tasks. Inspired by the success of PTMs for CV tasks, some NLP researchers also explore supervised Pre-training, and the most representative work is CoVE (McCann et al., 2017). CoVE adopts machine translation as its pre-training objective. After pre-training, the encoder of source languages can work as a powerful backbone for downstream NLP tasks.

Among these four settings, the inductive and transductive settings are the core of research, as these two settings aim to transfer knowledge from supervised source tasks to target tasks. Although supervised learning is always one of the core issues of machine learning research, the scale of unlabeled data is much larger than that of manually labeled data. Recently, more and more researchers have noticed the importance of large-scale unlabeled data and are committed to extracting information from unlabeled data. Self-supervised learning has been proposed to extract knowledge from large-scale unlabeled data by leveraging input data itself as supervision.

Self-supervised learning and unsupervised learning have many similarities in their settings. To a certain extent, self-supervised learning can be regarded as a branch of unsupervised learning because they both apply unlabeled data. However, unsupervised learning mainly focuses on detecting data patterns (e.g., clustering, community discovery, and anomaly detection), while self-supervised learning is still in the paradigm of supervised settings (e.g., classification and generation) (Liu et al., 2020b).

The early PTMs for NLP tasks exist in the form of well-known word embeddings (Collobert and Weston, 2008; Mikolov et al., 2013b; Pennington et al., 2014), which apply self-supervised methods to transform words into distributed representations. As these pre-trained word representations capture syntactic and semantic information in the text, they are often used as input embeddings and initialization parameters for NLP models and offer significant improvements over random initialization parameters (Turian et al., 2010). Since these word-level models often suffer from the word polysemy, Peters et al. (2018) further adopt a sequence-level neural model to capture complex word features across different linguistic contexts and generates context-aware word embeddings. Using word embeddings as the input of neural models has almost become the common mode for NLP tasks.

After Vaswani et al. (2017) propose Transformers to deal with sequential data, PTMs for NLP tasks have entered a new stage, because it is possible to train deeper language models compared to conventional CNNs and RNNs. Different from those word-level PTMs used as input features, the Transformer-based PTMs such as GPT and BERT can be used as the model backbone of various specific tasks. After pre-training these Transformer-based PTMs on large-scale textual corpora, both the architecture and parameters of PTMs can serve as a starting point for specific NLP tasks, i.e., just fine-tuning the parameters of PTMs for specific NLP tasks can achieve competitive performance. So far, these Transformer-based PTMs have achieved state-of-the-art results on almost all NLP tasks. Inspired by GPT and BERT, many more effective PTMs for NLP tasks have also been proposed, like XLNET (Yang et al., 2019), RoBERTa (Liu et al., 2020d), BART (Lewis et al., 2020a), and T5 (Raffel et al., 2020).

With the recent advance of PTMs for NLP tasks, applying Transformer-based PTMs as the backbone of NLP tasks has become a standard procedure. Motivated by the success of self-supervised learning and Transformers in NLP, some researchers explore self-supervised learning (Wu et al., 2018; Chen et al., 2020d; Chen and He, 2020; He et al., 2020) and Transformers (Carion et al., 2020; Liu et al., 2021c) for CV tasks. These preliminary efforts have shown that self-supervised learning and Transformers can outperform conventional supervised CNNs. Furthermore, Transformer-based multimodal PTMs (Lu et al., 2019; Li et al., 2019; Tan and Bansal, 2019) have also been proposed and shown promising results. After the last wave of supervised pre-training, self-supervised pre-training has become the focus of current AI research.

Looking back at the pre-training in the AI spectrum, it is not difficult to find that pre-training has been developed for decades, focusing on how to acquire versatile knowledge for various downstream tasks. Next, we will comprehensively introduce the latest breakthroughs of PTMs in this wave of self-supervised pre-training. Considering that almost all the latest PTMs are related to pre-trained language models, “PTMs” in the following sections refers to pre-trained language models or multimodal models. For those conventional PTMs based on supervised pre-training, we refer to the papers of He et al. (2019) and Zoph et al. (2020).

Before Transformer, RNNs have long been a typical tool for processing sequential data, especially for processing natural languages. As RNNs are equipped with sequential nature, they read a word at each time step in order. For each word, RNNs refer to all hidden states of its previous words to process it. Such a mechanism is considered to be difficult to take advantage of the parallel capabilities of high-performance computing devices such as GPUs and TPUs.

Attention Layer. Self-attention layers are the key to the success of Transformer. Formally, given a query set $\mathcal{Q}=\left\{{\mathbf{q}}_1,\dots ,{\mathbf{q}}_n\right\}$, a key set $\mathcal{K}=\left\{{\mathbf{k}}_1,\dots ,{\mathbf{k}}_m\right\}$, a value set $\mathcal{V}=\left\{{\mathbf{v}}_1,\dots ,{\mathbf{v}}_m\right\}$, each query vector ${\mathbf{q}}_i\in {\mathbb{R}}^{d_k}$, each key vector ${\mathbf{k}}_i\in {\mathbb{R}}^{d_k}$, and each value vector ${\mathbf{v}}_i\in {\mathbb{R}}^{d_v}$, the scaled dot-product attention is defined as(1)$\begin{array}{cc}& \left\{{\mathbf{h}}_1,\dots ,{\mathbf{h}}_n\right\}=\mathtt{A}\mathtt{T}\mathtt{T}\left(\mathcal{Q},\mathcal{K},\mathcal{V}\right),\\ & {\mathbf{h}}_i=\sum _{j=1}^m{a}_{ij}{\mathbf{v}}_j,\\ & a_{ij}=\frac{\exp \left.\mathtt{A}\mathtt{T}\mathtt{T}-\mathtt{M}\mathtt{a}\mathtt{s}\mathtt{k}\left.\frac{{\mathbf{q}}_i\cdot {\mathbf{k}}_j}{\sqrt{d_k}}\right)\right)}{{\sum }_{l=1}^m\exp \left.\mathtt{A}\mathtt{T}\mathtt{T}-\mathtt{M}\mathtt{a}\mathtt{s}\mathtt{k}\left.\frac{{\mathbf{q}}_i\cdot {\mathbf{k}}_l}{\sqrt{d_k}}\right)\right)}.\end{array}$

Intuitively, $\mathcal{Q}$ is the set of vectors to calculate the attention for, $\mathcal{K}$ is the set of vectors to calculate the attention against. As a result of dot-product multiplication, we can get the weight a_ij to indicate how attended the query vector q_i against the key vector k_j. Finally, we can calculate the weighted mean of value vectors as the final result of the attention layer. Note that, the masking function $\mathtt{A}\mathtt{T}\mathtt{T}-\mathtt{M}\mathtt{a}\mathtt{s}\mathtt{k}\left(\cdot \right)$ is used to restrict which key-value pairs each query vector can attend. If we do not want q_i to attend k_j, $\mathtt{A}\mathtt{T}\mathtt{T}-\mathtt{M}\mathtt{a}\mathtt{s}\mathtt{k}\left(x\right)=-\infty$, otherwise $\mathtt{A}\mathtt{T}\mathtt{T}-\mathtt{M}\mathtt{a}\mathtt{s}\mathtt{k}\left(x\right)=x$.

By respectively packing $\mathcal{Q},\mathcal{K},\mathcal{V}$ into matrix representations $\mathbf{Q}\in {\mathbb{R}}^{n\times d_k},\mathbf{K}\in {\mathbb{R}}^{m\times d_k},\mathbf{V}\in {\mathbb{R}}^{m\times d_v}$, the attention can be simplified to(2)$\begin{array}{cc}& \mathbf{H}=\mathtt{A}\mathtt{T}\mathtt{T}\left(\mathbf{Q},\mathbf{K},\mathbf{V}\right)=\mathbf{A}\mathbf{V},\\ & \mathbf{A}=\mathtt{S}\mathtt{o}\mathtt{f}\mathtt{t}\mathtt{m}\mathtt{a}\mathtt{x}\left.\mathtt{A}\mathtt{T}\mathtt{T}-\mathtt{M}\mathtt{a}\mathtt{s}\mathtt{k}\left.\frac{\mathbf{Q}{\mathbf{K}}^{\top }}{\sqrt{d_k}}\right)\right),\end{array}$where $\mathtt{S}\mathtt{o}\mathtt{f}\mathtt{t}\mathtt{m}\mathtt{a}\mathtt{x}\left(\cdot \right)$ is applied in a row-wise manner, $\mathbf{A}\in {\mathbb{R}}^{n\times m}$ is the attention matrix, $\mathbf{H}\in {\mathbb{R}}^{n\times d_v}$ is the result.

Instead of using the vanilla scaled dot-product attention, Transformer applies a multi-head attention layer defined as follows,(3)$\begin{array}{cc}\mathbf{H}& =\mathtt{M}\mathtt{H}-\mathtt{A}\mathtt{T}\mathtt{T}\left(\mathbf{Q},\mathbf{K},\mathbf{V}\right)\\ & =\mathtt{C}\mathtt{o}\mathtt{n}\mathtt{c}\mathtt{a}\mathtt{t}\left({\mathbf{H}}_1,\dots ,{\mathbf{H}}_h\right){\mathbf{W}}^O,\\ {\mathbf{H}}_i& =\mathtt{A}\mathtt{T}\mathtt{T}\left(\mathbf{Q}{\mathbf{W}}_i^Q,\mathbf{K}{\mathbf{W}}_i^K,\mathbf{V}{\mathbf{W}}_i^V\right),\end{array}$where h is the head number. ${\mathbf{W}}_i^Q$, ${\mathbf{W}}_i^K$, ${\mathbf{W}}_i^V$ are respectively used to project the input Q, K, V into the feature space of the i-th head attention. After concatenating all head outputs by $\mathtt{C}\mathtt{o}\mathtt{n}\mathtt{c}\mathtt{a}\mathtt{t}\left(\cdot \right)$, the multi-head attention layer applies W^O to project the concatenation into the final output space.

Position-Wise Feed-Forward Layer. Besides attention layers, each block of Transformer also contains a position-wise feed-forward layer. Given the packed input matrix $\mathbf{X}\in {\mathbb{R}}^{n\times d_i}$ indicating a set of input vectors, d_i is the vector dimension, a position-wise feed-forward layer is defined as(4)$\mathbf{H}=\mathtt{F}\mathtt{F}\mathtt{N}\left(\mathbf{X}\right)=\sigma \left(\mathbf{X}{\mathbf{W}}_1+{\mathbf{b}}_1\right){\mathbf{W}}_2+{\mathbf{b}}_2,$where σ(⋅) is the activation function (usually the ReLU function). ${\mathbf{W}}_1\in {\mathbb{R}}^{d_i\times d_f}$, ${\mathbf{b}}_1\in {\mathbb{R}}^{d_f}$, ${\mathbf{W}}_2\in {\mathbb{R}}^{d_f\times d_o}$, ${\mathbf{b}}_2\in {\mathbb{R}}^{d_o}$ are all learnable parameters for projection. $\mathbf{H}\in {\mathbb{R}}^{n\times d_o}$ is the final result of the feed-forward layer. Empirically, d_i is set equal to d_o, d_f is set to be much larger than d_i and d_o.

Residual Connection and Normalization As we mentioned before, Transformer applies residual connection and layer normalization between various neural layers, making the architecture of Transformer possible to be deep. Formally, given a neural layer f(⋅), the residual connection and normalization layer is defined as(5)$\mathbf{H}=\mathtt{A}\mathtt{\&}\mathtt{N}\left(\mathbf{X}\right)=\mathtt{L}\mathtt{a}\mathtt{y}\mathtt{e}\mathtt{r}\mathtt{N}\mathtt{o}\mathtt{r}\mathtt{m}\left(f\left(\mathbf{X}\right)+\mathbf{X}\right),$where $\mathtt{L}\mathtt{a}\mathtt{y}\mathtt{e}\mathtt{r}\mathtt{N}\mathtt{o}\mathtt{r}\mathtt{m}\left(\cdot \right)$ denotes the layer normalization operation.

As shown in Fig. 5, there are three variants of the multi-head attention in Transformer:

• (1)
  Self-attention is used in the encoder, which uses the output of the previous layer as Q, K, V. In the encoding phase, given a word, the self-attention computes its attention scores by comparing it with all words in the input sequence. And such attention scores indicate how much each of the other words should contribute to the next representation of the given word. We give an example in Fig. 6, where the self-attention accurately captures the referential relationships between “Jack” and “he”, generating the highest attention score.

  

  1. Download: Download high-res image (306KB)
  2. Download: Download full-size image

  Fig. 6. An illustration of the self-attention mechanism of Transformer. The figure shows the self-attention results when encoding the word “he”, where the darker the color of the square is, the larger the corresponding attention score is.

• (2)
  Masked self-attention is used in the decoder, whose attention matrix satisfies A_ij = 0, i > j. This attention is beneficial to autoregressive language modeling. In the decoding phase, the self-attention is similar to the encoding, except that it only decodes one representation from left to right at one time. Since each step of the decoding phase only consults the previously decoded results, we thus require to add the masking function into the self-attention.
• (3)
  Cross-attention is also used in the decoder, which uses the output of the previous decoder block as Q as well as the output of the encoder as K and V. Such a procedure is essentially an aggregation of the information of the whole input sequence, and it will be applied to all the words to generate in the decoding phase. Taking advantage of the input context is of great significance to some seq2seq tasks such as machine translation and text summarization.

For more details of Transformer, please refer to its original paper (Vaswani et al., 2017) and the survey paper (Lin et al., 2021). Due to the prominent nature, Transformer gradually becomes a standard neural structure for natural language understanding and generation. Moreover, it also serves as the backbone neural structure for the subsequently derived PTMs. Next, we will introduce two landmarks that completely open the door towards the era of large-scale self-supervised PTMs, GPT and BERT. In general, GPT is good at natural language generation, while BERT focuses more on natural language understanding.

As introduced in Section 2, PTMs typically consist of two phases, the pre-training phase and the fine-tuning phase. Equipped by the Transformer decoder as the backbone,⁵ GPT applies a generative pre-training and a discriminative fine-tuning. Theoretically, compared to precedents of PTMs, GPT is the first model that combines the modern Transformer architecture and the self-supervised pre-training objective. Empirically, GPT achieves significant success on almost all NLP tasks, including natural language inference, question answering, commonsense reasoning, semantic similarity and classification.

The adaptation procedure of GPT to specific tasks is fine-tuning, by using the pre-trained parameters of GPT as a start point of downstream tasks. In the fine-tuning phase, passing the input sequence through GPT, we can obtain the representations of the final layer of the GPT Transformer. By using the representations of the final layer and task-specific labels, GPT optimizes standard objectives of downstream tasks with simple extra output layers. As GPT has hundreds of millions of parameters, it is trained for 1 month on 8 GPUs, which is fairly the first “large-scale” PTM in the history of NLP. And undoubtedly, the success of GPT pave the way for the subsequent rise of a series of large-scale PTMs. In the next part, we will introduce another most representative model BERT.

In the pre-training phase, BERT applies autoencoding language modeling rather than autoregressive language modeling used in GPT. More specifically, inspired by cloze (Taylor, 1953), the objective masked language modeling (MLM) is designed. As shown in Fig. 7, in the procedure of MLM, tokens are randomly masked with a special token $\left[\mathtt{M}\mathtt{A}\mathtt{S}\mathtt{K}\right]$, the objective is to predict words at the masked positions with contexts. Compared with standard unidirectional autoregressive language modeling, MLM can lead to a deep bidirectional representation of all tokens. Formally, given a corpus consisting of tokens $\mathcal{X}=\left\{x_0,x_1,\dots ,x_n,x_n+1\right\}$, BERT randomly masks m tokens in $\mathcal{X}$ and then maximizes the following log-likelihood:(7)$\mathcal{L}\left(\mathcal{X}\right)=\sum _{i=1}^m\log P\left({\left[\mathtt{M}\mathtt{a}\mathtt{s}\mathtt{k}\right]}_i=y_i|\tilde{\mathcal{X}};\mathrm{\Theta }\right),$where the probability P is modeled by the Transformer encoder with parameters Θ, $\tilde{\mathcal{X}}$ is the result after masking some tokens in $\mathcal{X}$, ${\left[\mathtt{M}\mathtt{a}\mathtt{s}\mathtt{k}\right]}_i$ is the i-th masked position, and y_i is the original token at this position.

Besides MLM, the objective of next sentence prediction (NSP) is also adopted to capture discourse relationships between sentences for some downstream tasks with multiple sentences, such as natural language inference and question answering. For this task, a binary classifier is used to predict whether two sentences are coherent. In the pre-training phase, MLM and NSP work together to optimize the parameters of BERT.

After pre-training, BERT can obtain robust parameters for downstream tasks. By modifying inputs and outputs with the data of downstream tasks, BERT could be fine-tuned for any NLP tasks. As shown in Fig. 8, BERT could effectively handle those applications with the input of a single sentence or sentence pairs. For the input, its schema is two sentences concatenated with the special token $\left[\mathtt{S}\mathtt{E}\mathtt{P}\right]$, which could represent: (1) sentence pairs in paraphrase, (2) hypothesis-premise pairs in entailment, (3) question-passage pairs in question answering, and (4) a single sentence for text classification or sequence tagging. For the output, BERT will produce a token-level representation for each token, which can be used to handle sequence tagging or question answering, and the special token [CLS] can be fed into an extra layer for classification. After GPT, BERT has further achieved significant improvements on 17 different NLP tasks, including SQuAD (better than human performance), GLUE (7.7% point absolute improvements), MNLI (4.6% point absolute improvements), etc.

After GPT and BERT, some of their improvements have been proposed, such as RoBERTa and ALBERT. RoBERTa(Liu et al., 2020d) is one of the success variants of BERT, which mainly has four simple and effective changes: (1) Removing the NSP task; (2) More training steps, with bigger batch size and more data; (3) Longer training sentences; (4) Dynamically changing the $\left[\mathtt{M}\mathtt{A}\mathtt{S}\mathtt{K}\right]$ pattern. RoBERTa achieves impressive empirical results on the basis of BERT. Moreover, RoBERTa has pointed out that the NSP task is relatively useless for the training of BERT. ALBERT(Lan et al., 2019) is another important variant of BERT, which provides several interesting observations on reducing parameters. First, it factorizes the input word embedding matrix into two smaller ones. Second, it enforces parameter-sharing between all Transformer layers to significantly reduce parameters. Third, it proposes the sentence order prediction (SOP) task to substitute BERT's NSP task. As a sacrifice to its space efficiency, ALBERT has a slower fine-tuning and inference speed.

Nevertheless, the differences between them are not so significant. As Feynman's saying goes, “What I cannot create, I do not understand”. On one hand, a model that can not understand must not fluently generate; on the other hand, we can easily turn understanding tasks into generation tasks (Schick and Schütze, 2020). Recent studies also show that GPTs can achieve similar and even better performance on understanding benchmarks than BERTs (Liu et al., 2021b). The boundary between understanding and generation is vague.

Based on the observation, a bunch of novel architectures has been seeking for unifying different types of language tasks with one PTM. We will take a look over its development and discuss the inspirations they bring towards a unified foundation of natural language processing.

Combining Autoregressive and Autoencoding Modeling. The pioneer work to unify GPT-style unidirectional generation and BERT-style bidirectional understanding is XLNet (Yang et al., 2019), which proposes the permutated language modeling. The masked-recover strategy in BERT naturally contradicts with its downstream application, where there is no $\left[\mathtt{M}\mathtt{A}\mathtt{S}\mathtt{K}\right]$ in input sentences. XLNet solves the problem by permutating tokens' order in the pre-training and then applying the autoregressive prediction paradigm, which endows XLNet with the ability for both understanding and generation. An important follower of permutation language modeling is MPNet (Song et al., 2020), which amends the XLNet's discrepancy that in pre-training XLNet does not know the sentence's length while in downstream it knows.

Besides permutated language modeling, another stream would be multi-task training. UniLM (Dong et al., 2019) proposes to jointly train different language modeling objectives together, including unidirectional, bidirectional, and seq2seq objectives. This can be achieved by changing the attention masks in Transformers. UniLM performs quite well in generative question answering and abstract summarization.

Recently, GLM (Du et al., 2021) proposes a more elegant approach for combining autoregressive and autoencoding. Given a variable-length masked span, instead of providing the number of $\left[\mathtt{M}\mathtt{A}\mathtt{S}\mathtt{K}\right]$ to model as BERT and SpanBERT (Joshi et al., 2020) do, GLM asks Transformer blocks to autoregressively generate the masked tokens. And to preserve the information of $\left[\mathtt{M}\mathtt{A}\mathtt{S}\mathtt{K}\right]$s’ number, GLM proposes a 2D positional encoding strategy. GLM is the first model to achieve the best performance on all types of tasks including natural language understanding, conditional generation, and unconditional generation at the same time.

Applying Generalized Encoder-Decoder. Before GLM, both encoder structure (e.g., BERT) or decoder structure (e.g., GPT) can not solve an important problem: to fill in blanks with variable lengths (Du et al., 2021; Shen et al., 2020b). The decoder-based models can not make it because they can only generate at the end of the sequence and neither the encoder-based models because the number of $\left[\mathtt{M}\mathtt{A}\mathtt{S}\mathtt{K}\right]$s will leak information. A natural idea is to turn to encoder-decoder architectures originally designed for machine translation, which would produce variable lengths of target sequences conditioned on the sources.

The pioneer of this genre is MASS (Song et al., 2019), which introduces the masked-prediction strategy into the encoder-decoder structure. However, MASS does not touch the problem of filling variable-length blanks. T5 (Raffel et al., 2020) solves the problem by masking a variable-length of span in text with only one mask token and asks the decoder to recover the whole masked sequence. BART (Lewis et al., 2020a) introduces the interesting idea of corrupting the source sequence with multiple operations such as truncation, deletion, replacement, shuffling, and masking, instead of mere masking. There are following works that specify in typical seq2seq tasks, such as PEGASUS (Zhang et al., 2020a) and PALM (Bi et al., 2020).

However, several challenges lie in front of encoder-decoder architectures. First, the encoder-decoder introduces much more parameters compared to a single encoder/decoder. Although this problem could be alleviated by parameter-sharing of the encoder and decoder, its parameter-efficiency is still doubtful. Second, encoder-decoder structures generally do not perform very well on natural language understanding. Despite reported improvements over similar-sized vanilla BERT, well-trained RoBERTa or GLM encoder performs much better than them.

Is the current Transformer a good enough implementation of human beings' cognitive system? Of course not. Attention mechanism, the core module in the Transformer architecture, is inspired by the micro and atom operation of the human's cognitive system and only responsible for the perceptive function. However, human-level intelligence is far more complex than the mere understanding of the association between different things.

In pursuit for human-level intelligence, understanding the macro architecture of our cognitive functions including decision making, logical reasoning, counterfactual reasoning and working memory (Baddeley, 1992) is crucial. In this subsection, we will take a look over the novel attempts inspired by advances of cognitive science, especially on maintainable working memory and sustainable long-term memory.

Maintainable Working Memory. A natural problem of Transformer is its fixed window size and quadratic space complexity, which significantly hinders its applications in long document understanding and generation.

Despite the bunch of modifications on approximate computing of the quadratic growing point-wise attention (Tay et al., 2020), a question is that we humans do not present such a long-range attention mechanism. As an alternative, cognitive scientists have revealed that humans could maintain a working memory (Baddeley, 1992; Brown, 1958; Barrouillet et al., 2004; Wharton et al., 1994), which not only memorizes and organizes but also forgets. The conventional long-short term memory (LSTM) network is an exemplar practice for such a philosophy.

For Transformer-based architectures, the Transformer-XL (Dai et al., 2019) is the first to introduce segment-level recurrence and relative positional encoding to fulfill this goal. However, the recurrence only implicitly models the working memory. As a more explicit solution, CogQA (Ding et al., 2019) proposes to maintain a cognitive graph in the multi-hop reading. It is composed of two systems: System 1 based on PTMs and System 2 based on GNNs to model the cognitive graph for multi-hop understanding.

A limitation of CogQA is that its use of System 1 is still based on fixed window size. To endow working memory with the ability to understand long documents, CogLTX (Ding et al., 2020) leverages a Mem Recall language model to select sentences that should be maintained in the working memory and task-specific modules for answering or classification.

Sustainable Long-Term Memory. The success of GPT-3 and recent studies on language models’ ability in recalling factual knowledge (Petroni et al., 2019; Wang et al., 2020a; Liu et al., 2021b) has revealed the fact that Transformers can memorize. How does Transformers make it?

In Lample et al. (2019), the authors provide some inspiring evidences on how Transformers memorize. They replace the feed-forward networks in a Transformer layer with large key-value memory networks, and find it to work pretty well. This somehow proves that the feed-forward networks in Transformers is equivalent to memory networks.

Nevertheless, the memory capacity in Transformers is quite limited. For human intelligence, besides working memory for deciding and reasoning, the long-term memory also plays a key role in recalling facts and experiences. REALM (Guu et al., 2020) is a pioneer to explore how to construct a sustainable external memory for Transformers. The authors tensorize the whole Wikipedia sentence by sentence, and retrieve relevant sentences as context for masked pre-training. The tensorized Wikipedia is asynchronously updated for a given number of training steps. RAG (Lewis et al., 2020b) extends the masked pre-training to autoregressive generation, which could be better than extractive question answering.

Besides tensorizing the textual corpora (Verga et al., 2020; Févry et al., 2020), propose to tensorize entities and triples in existing knowledge bases. When entities appear in contexts, they replace entity tokens’ embedding in an internal Transformer layer with the embedding from outer memory networks (Dhingra et al., 2020; Sun et al., 2021). maintain a virtual knowledge from scratch, and propose a differentiable reasoning training objective over it. All of these methods achieve promising improvement on many open-domain question answering benchmarks.

Besides the practice to unify sequence modeling and construct cognitive-inspired architectures, most current studies focus on optimizing BERT's architecture to boost language models' performance on natural language understanding.

A stream of work aims at improving the masking strategy, which could be regarded as a certain kind of data augmentation (Gu et al., 2020). SpanBERT (Joshi et al., 2020) shows that masking a continuous random-length span of tokens with a span boundary objective (SBO) could improve BERT's performance. Similar ideas have also been explored in ERNIE (Sun et al., 2019c,d) (where a whole entity is masked), NEZHA (Wei et al., 2019), and Whole Word Masking (Cui et al., 2019).

Another interesting practice is to change the masked-prediction objective to a harder one. ELECTRA (Clark et al., 2020) transform MLM to a replace token detection (RTD) objective, in which a generator will replace tokens in original sequences and a discriminator will predict whether a token is replaced.

Language models trained on large-scale English corpora have achieved great success in many benchmarks. However, we live in a multilingual world, and training a large language model for each language is not an elegant solution because of the cost and the amount of data required. In fact, although people from all over the world use different languages, they can express the same meaning. This may indicate that semantics is independent of symbol systems. Additionally, some researchers found that they could get even better performance on benchmarks when training one model with several languages comparing with training several monolingual models (Lample and Conneau, 2019; Huang et al., 2020b). Hence, training one model to learn multilingual representations rather than monolingual representations may be a better way.

Before BERT, some researchers have explored multilingual representations. There are mainly two ways to learn multilingual representations. One way is to learn through parameter sharing. For example, training multilingual LSTMs with several language pairs together achieves multilingual translation. Another way is to learn language-agnostic constraints, such as decoupling language representations into language-specific and language-agnostic representations utilizing the WGAN (Arjovsky et al., 2017) framework. Both of these two ways enable models to be applied to multilingual scenarios, but only for specific tasks. The model in each of them is trained with one specific task from beginning to end, and cross-lingual knowledge cannot be generalized to other tasks. Hence, for any other multilingual tasks, training new models from scratch is still required. Learning new models from scratch needs a large volume of task-specific data.

The appearance of BERT shows that the framework of pre-training with general self-supervised tasks and then fine-tuning on specific downstream tasks is feasible. This motivates researchers to design tasks to pre-train versatile multilingual models. Multilingual tasks could be divided into understanding tasks and generation tasks according to task objectives. Understanding tasks focus on sentence-level or word-level classification, and are of help for downstream classification tasks such as natural language inference (Conneau et al., 2018b). Generation tasks focus on sentence generation, and are crucial in downstream generation tasks such as machine translation.

Some understanding tasks are first used to pre-train multilingual PTMs on non-parallel multilingual corpora. For example, multilingual BERT (mBERT) released by Devlin et al. (2019) is pre-trained with the multilingual masked language modeling (MMLM) task using non-parallel multilingual Wikipedia corpora in 104 languages. The research conducted by Pires et al. (2019) shows that mBERT has the ability to generalize cross-lingual knowledge in zero-shot scenarios. This indicates that even with the same structure of BERT, using multilingual data can enable the model to learn cross-lingual representations. XLM-R (Conneau et al., 2020) builds a non-parallel multilingual dataset called CC-100, which supports 100 languages. The scale of CC-100 is much larger than the Wikipedia corpora used by mBERT, especially for those low-resource languages. XLM-R is pre-trained with MMLM as the only task on CC-100 and gets better performance on several benchmarks than mBERT, which indicates that a larger scale of multilingual corpora can bring better performance.

Besides TLM, there are some other effective methods to learn multilingual representations from parallel corpora. Unicoder (Huang et al., 2019a) provides two novel pre-training tasks based on parallel corpora: cross-lingual word recovery (CLWR) and cross-lingual paraphrase classification (CLPC). CLWR uses target language embeddings to represent source language embeddings by leveraging attention mechanisms, and its objective is to recover the source language embeddings. This task enables models to learn word-level alignments between different languages. CLPC treats aligned sentences as positive pairs and samples misaligned sentences as negative pairs to perform sentence-level classification, letting models predict whether the input pair is aligned or not. With CLPC, models can learn sentence-level alignments between different languages. ALM (Yang et al., 2020) automatically generates code-switched sequences from parallel sentences and performs MLM on it, which forces models to make predictions based only on contexts of other languages. InfoXLM (Chi et al., 2020b) analyzes MMLM and TLM from the perspective of information theory, and encourages models to distinguish aligned sentence pairs with misaligned negative examples under the framework of contrastive learning. HICTL (Wei et al., 2021) extends the idea of using contrastive learning to learn both sentence-level and word-level cross-lingual representations. ERNIE-M (Ouyang et al., 2020) proposes back-translation masked language modeling (BTMLM), and expands the scale of parallel corpora through back-translation mechanisms. These works show that leveraging parallel corpora can bring much help towards learning cross-lingual representations.

Researches have also widely explored generative models for multilingual PTMs. Normally, a generative model consists of a Transformer encoder and a Transformer decoder. For example, MASS (Song et al., 2019) extends MLM to language generation. It randomly masks a span of tokens in the input sentence and predicts the masked tokens in an autoregressive manner. Denoising autoencoding (DAE) is a typical generation task, which applies noise functions to the input sentence and then restores the original sentence with the decoder. The noise functions of DAE usually contain two operations: replacing a span of tokens with a mask token as well as permuting the order of tokens. mBART (Liu et al., 2020c) extends DAE to support multiple languages by adding special symbols. It adds a language symbol both to the end of the encoder input and the beginning of the decoder input. This enables models to know the languages to be encoded and generated.

Although DAE in mBART (Liu et al., 2020c) is trained with multiple languages, the encoding input and the decoding output are always in the same language. This leads models to capture spurious correlations between language symbols and generated sentences. In other words, models may ignore the given language symbols and directly generate sentences in the same language of the input. To address this issue, XNLG (Chi et al., 2020a) proposes the cross-lingual autoencoding (XAE) task. Different from DAE, the encoding input and the decoding output of XAE are in different languages, which is similar to machine translation. In addition, XNLG optimizes parameters in a two-stage manner. It trains the encoder with the MLM and TLM tasks in the first stage. Then, it fixes the encoder and trains the decoder with the DAE and XAE tasks in the second stage. All parameters are well pre-trained by this way, and the gap between pre-training with MLM and fine-tuning with autoregressive decoding is also filled.

Large-scale pre-training and its downstream applications have cascaded impactful research and development with diverse real-world modalities. As human beings, we are exposed to different modalities—we see objects, hear sounds and speak languages. Modalities, such as audio, video, image and text, refer to how something happens or is experienced. Recent years have witnessed an upsurging interest in cross-modal tasks that involves multiple modalities. More recently, large-scale PTMs have enhanced research interests in the intersection of multiple modalities, such as the intersection of image and text, or the intersection of video and text. Most of these cross-modal works can be classified as vision and language (V&L), considering that images and videos belong to vision as well as text and speech (audio) belong to language. Specifically, V&L tasks can be further divided into image-text-based tasks, video-text-based tasks, and video-audio-based tasks according to their specific modalities being used. In this section, we present an overview of existing works in pre-training on V&L modalities. Existing cross-modal pre-training PTMs mainly focus on (1) improving model architecture, (2) utilizing more data, and (3) designing better pre-training tasks.

For image-text-based PTMs, most current works are based on the architecture of visual-linguistic BERT. The main challenge lies in the alignment of visual and textual content in a unified semantic space (i.e. V&L grounding). To this end, there are mainly two kinds of model architecture designs: two-stream and single-stream. As a representative work of two-stream models, ViLBERT (Lu et al., 2019) processes image regions and text tokens with two separate streams, and fuses them with specifically designed co-attention transformer blocks. In comparison, LXMERT (Tan and Bansal, 2019) first processes two modalities separately and then conducts a late fusion with a cross-modality encoder. In single-stream models, such as VisualBERT (Li et al., 2019), Unicoder-VL (Li et al., 2020a), B2T2 (Alberti et al., 2019), the image region features and word embeddings are usually concatenated and fed into a single transformer. Researchers have not reached a consensus on which design is better (Lu et al., 2019; Su et al., 2020) on the V&L grounding ability. Considering model simplicity and parameter efficiency, current works mainly adopt the single-stream design.

In cross-modal pre-training, data resources are also of vital significance. The most widely used corpora are image-text pairs collected from web including Conceptual Captions (Sharma et al., 2018), SBU Captions (Ordonez et al., 2011) or existing V&L datasets designed for specific tasks including COCO (Lin et al., 2014), Flicker30 K (Plummer et al., 2015), GQA (Hudson and Manning, 2019), VQA (Antol et al., 2015) and Visual Genome (Krishna et al., 2017). Directly increasing the scale of image-text data is useful for better V&L grounding. UNITER (Chen et al., 2020f) combines several above-mentioned datasets, resulting in 5.6 million image-text pairs for training. Sufficient training data helps UNITER achieve impressive results on downstream tasks. Similar to UNITER in architecture and pre-training tasks, Image BERT (Qi et al., 2020) further constructs a dataset containing 10 million web image-text pairs and uses it as a pre-training dataset, leading to a better performance than UNITER on image-text retrieval tasks. In addition to parallel image-text data, VL-BERT (Su et al., 2020) finds that incorporating extra text-only corpora like Books Corpus (Zhu et al., 2015) and Wikipedia is helpful for text understanding, especially for tasks with long and complex sentences like visual commonsense reasoning. Different from works using only easily collected data like image-text pairs or textual corpora, Lu et al. (2020) identifies the contribution of dedicated datasets by conducting a joint multi-task training on nearly all kinds of V&L tasks.

Given data resources, it is also important to design corresponding pre-training tasks or strategies to utilize the information efficiently. For V&L understanding tasks, the most widely used pre-training tasks are MLM, sentence-image alignment (SIA), masked region classification (MRC), masked region feature regression (MRFR), and directly incorporating downstream tasks. Similar to MLM for NLP, MLM for V&L aims to recover masked tokens in captions with the help of visual and textual context. SIA is designed to judge whether image-text pairs are matched. MRC can be considered as the visual MLM, requiring V&L models to predict the categories of masked objects. MRFR further requires V&L models to recover the visual features of masked object regions. There are also models directly conducting downstream V&L understanding tasks in the pre-training stage. For example, LXMERT employs VQA as a pre-training task. Lu et al. (2020) trains all downstream tasks jointly. To learn the fine-grained alignment between image regions and words, UNITER further proposes a word-region alignment task in the way of Optimal Transport (Chen et al., 2020c), which first finds a sparse matching between image regions and words, and then minimizes the alignment distance. However, most of these works ignore the object tags’ function as a kind of explicit bridges between image regions and text tokens. Therefore, Oscar (Li et al., 2020e) proposes to concatenate the object tags with original image-text pairs as anchors to learn the alignment between V&L modalities, and designs a new pre-training task for image-tag sequence-caption alignment judgment. In this way, Oscar achieves SOTA results on most V&L tasks compared with the aforementioned models on both V&L understanding and generation tasks. Besides pre-training tasks designed for V&L understanding tasks, there are also some pre-training tasks targeting at V&L generation tasks. For example, VLP (Zhou et al., 2020a) and X-GPT (Xia et al., 2020) employ seq2seq MLM as their pre-training tasks.

Instead of designing delicate pre-training tasks, recent works CLIP (Radford et al., 2021) and WenLan (Huo et al., 2021) choose to grasp the V&L grounding ability in a simple and holistic regime. They encode images and captions into holistic visual and text representations rather than separated region features and word embeddings, and then only conduct an image-text retrieval task. The success of this kind of holistic alignment can be largely attributed to the enlarging scale of web data, which is 400 million image-text pairs for CLIP and 30 million for WenLan.

Previous works mentioned above are specialized for V&L understanding or only image captioning tasks, but are not capable of image generation. Recently, a bigger step towards conditional image generation is taken by DALLE (Ramesh et al., 2021) and CogView (Ding et al., 2021a). DALLE is the first transformer-based text-to-image PTM, with around 10 billion parameters. It shows the potential of multi-modal PTMs in bridging the gap between text descriptions and image generation, especially the excellent ability in combining different objects, such as “an armchair in the shape of an avocado”. CogView further improves the numerical precision and training stability by introducing sandwich transformer and sparse attention mechanism, and surpasses the DALLE in Fréchet Inception Distance (FID) (Heusel et al., 2017) on blurred COCO.

In addition to image-text PTMs, there are also PTMs for other modalities, such as video and audio. VideoBERT (Sun et al., 2019a) conducts pre-training on Cooking312 K video dataset (Sun et al., 2019a) and validates the model on zero-shot action classification task and video captioning task. SpeechBERT (Chuang et al., 2019) first encodes the continuous audio signal into several phonetic-semantic word embeddings, and then uses MLM on both text and audio modalities as pre-training tasks. After pre-training, spoken question answering (SQA) task is used for evaluation.

PTMs can extract plenty of statistical information from large amounts of data. Besides, external knowledge, such as knowledge graphs, domain-specific data and extra annotations of pre-training data, is the outcome of human wisdom which can be a good prior to the modeling of statistics. In this subsection, we classify external knowledge according to the knowledge format and introduce several methods attempting to combine knowledge with PTMs.

The typical form of structured knowledge is knowledge graphs. Many works try to enhance PTMs by integrating entity and relation embeddings (Zhang et al., 2019b; Liu et al., 2020a; Peters et al., 2019; Sun et al., 2020; Rosset et al., 2020; Qin et al., 2021) or their alignments with the text (Xiong et al., 2019; Sun et al., 2019c). However, real-world knowledge graphs like Wikidata contain more information than entities and relations. Wang et al. (2021b) pre-train models based on the descriptions of Wikidata entities, by incorporating a language model loss and a knowledge embedding loss together to get knowledge-enhanced representations. Some works regard the paths and even sub-graphs in knowledge graphs as a whole, and directly model them and the aligned text to retain more structural information. Since aligning entities and relations to raw text is often troublesome and can introduce noise in data pre-processing, another line of works (Bosselut et al., 2019; Guan et al., 2020; Chen et al., 2020e) can directly convert structural knowledge into the serialized text and let models learn knowledge-text alignments by themselves. An interesting attempt is OAG-BERT (Liu et al., 2021a), which integrates heterogeneous structural knowledge in the open academic graph (OAG) (Zhang et al., 2019a), which covers 0.7 billion heterogeneous entities and 2 billion relations.

Compared to structured knowledge, unstructured knowledge is more intact but also noisier. How to effectively model this kind of knowledge from the data is also worth being explored. The data of a specific domain or task can be considered as a kind of unstructured knowledge. Many works (Beltagy et al., 2019; Lee et al., 2020) further pre-train the general PTMs on this data to get better domain-specific or task-specific models. Since there are some domain-specific and task-specific human annotations, Ke et al. (2020) incorporate these extra annotations to get better domain-specific and task-specific language representations. For all the above-mentioned works, knowledge is implicitly stored in their model parameters. To model external knowledge in a more interpretable way, some works (Lewis et al., 2020b; Guu et al., 2020) design retrieval-based methods to use structured knowledge on downstream tasks. Another kind of works (Wang et al., 2020b) can use adapters trained on different knowledge sources with extra annotations to distinguish where the knowledge is from.

An effective and practical way to reduce computational requirements is system-level optimization towards computational efficiency and memory usage. System-level optimization methods are often model-agnostic and do not change underlying learning algorithms. Therefore, they are widely used in training large-scale PTMs. Generally, these methods can be divided into single-device optimization methods and multi-device optimization ones.

Single-Device Optimization. Current large-scale PTMs usually cost a lot of memory for pre-training. This is mainly due to the redundant representation of floating-point numbers. Modern deep learning systems are mainly based on a single-precision floating-point format (FP32). However, the weights of models usually fall in a limited range, and using a half-precision floating-point format (FP16) can accomplish most of the computation with little precision loss (Gupta et al., 2015).

However, in some cases, training models in FP16 may fail because of the floating-point truncation and overflow. To tackle this problem, mixed-precision training methods (Micikevicius et al., 2018) have been proposed, which preserve some critical weights in FP32 to avoid the floating-point overflow and use dynamic loss scaling operations to get rid of the floating-point truncation. Sufficient experiments have shown that mixed-precision training methods are more stable than directly training models in FP16. Although mixed-precision training methods can significantly reduce the training time and memory usage, they still face some challenges. When model parameters are not initialized well, mixed-precision methods may still cause unstable training. All these challenges still require to be further explored.

Besides the redundant representation of floating-point numbers, the activation states saved for computing gradients are also redundant. For example, in Transformer-based models, apart from the weights of attention layers and linear layers, computational devices also store the hidden states of each layer for the efficiency of the chain rule used in the gradient back-propagation. As compared with model parameters, these hidden states can consume even much more memory. To handle redundant activation states, gradient check pointing methods (Rasley et al., 2020) have been used to save memory by storing only a part of the activation states after forward pass. The discarded activation states are recomputed during the backward steps if necessary.

When pre-training models with billions to trillions of parameters, traditional data parallelism brings challenges of fitting whole model parameters into a single GPU, even with half-precision or mixed-precision training. Although this problem can be solved by using a GPU with larger memory, the expenses can be hard to afford, limiting the use of PTM by ordinary researchers. Model parallelism is an effective way to tackle this problem (Shazeer et al., 2018). As shown in Fig. 11, when conducting model parallelism, model parameters can be distributed to multiple nodes. The communication operations between these nodes like reduce-scatter and all-gather guarantee the correctness of forward pass and backward pass. Megatron-LM (Shoeybi et al., 2019) adopts model parallelism to Transformer-based PTMs. It splits self-attention heads as well as feed-forward layers into different GPUs, reducing the memory burden of a single GPU. Mesh-Tensorflow (Shazeer et al., 2018) also enables users to split tensors along any tensor dimensions, which can bring more customized options for model parallelism.

Although model parallelism enables different computational nodes to store different parts of model parameters, it has to insert collective communication primitives during both forward pass and backward pass, which can not be overlapped by device computation. On the contrary, the all-reduce collective communication operation in data parallelism usually can be overlapped by the backward computation. As a result, data parallelism is preferred as long as it can conquer the excessive requirement of memory capacity. In the standard implementation of data parallelism, optimizer states are usually copied along different nodes to guarantee synchronized optimization across data parallelism units. This redundancy leads to the additional overhead of GPU memory, especially when models are trained in a mixed-precision manner because the optimizer needs to store 32-bit master states of these models to ensure accuracy. To eliminate the redundancy brought by optimizer states and parameters, ZeRO optimizer (Rajbhandari et al., 2020) methods equally partition and distribute optimizer states to each node of data parallelism, such that each node only updates the optimizer states corresponding to its partition. At the end of a training step, all optimizer states are gathered across data parallelism nodes.

Besides some system-level optimization methods, various efforts have been devoted to exploring more efficient pre-training methods, so that we can pre-train large-scale PTMs with a lower cost solution.

Efficient Training Methods. Conventional pre-training tasks can be sample-inefficient. For example, for MLM which is widely used to pre-train recent PTMs, models are required to predict masked tokens according to contexts. The masked tokens are usually a subset (typically 15%) of input tokens, i.e., models can only learn from a small set of input tokens. To tackle this problem, ELECTRA (Clark et al., 2020) applies the replaced token detection task. This task forces models to distinguish whether an input token is replaced by a generator. This task can leverage more supervision information from each sample since all input tokens need to be distinguished. ELECTRA takes much fewer pre-training steps when it reaches similar performance to those MLM models. Furthermore, traditional MLM randomly masks tokens in a document to predict. Since the difficulty of predicting different tokens varies a lot, the random masking strategy makes the training process aimless and inefficient. Therefore, some works selectively mask tokens based on their importance (Gu et al., 2020) or gradients (Chen et al., 2020b) in back-propagation to speed up model training.

Apart from the pre-training tasks, the current pre-training dynamics are also sub-optimal. Recent large-scale PTMs usually require a large batch size. But in an early work (Goyal et al., 2017), researchers find that naively increasing the batch size may cause difficulty in optimization. Therefore, they propose a warmup strategy that linearly increases the learning rate at the beginning of training. This strategy is commonly used in recent large-scale PTMs. Another feature of recent PTMs is that they are usually composed of multiple stacks of a base structure like Transformers. The conventional training paradigm optimizes each layer simultaneously using the same hyper-parameters. However, some recent works study Transformer-based models and claim that different layers can share similar self-attention patterns. Therefore, a shallow model can firstly be trained and then duplicated to construct a deep model (Gong et al., 2019). Some layers can also be dropped during training to reduce the complexity of back-propagation and weight update (Zhang and He, 2020). In addition, You et al. (2017) and You et al. (2020) find that adaptively using different learning rates at different layers can also speed up convergence when the batch size is large.

Efficient Model Architectures. Besides efficient pre-training methods, more variants of model architectures can also reduce the computational complexity to improve the efficiency of training PTMs. For most Transformer-based PTMs, as their input sequence goes longer, their efficiency is limited by the computation of attention weights due to its quadratic time and space complexity of the sequence length. Therefore, many works attempt to reduce the complexity of Transformers. Some works (Peng et al., 2021; Choromanski et al., 2021; Wang et al., 2020c; Katharopoulos et al., 2020) design low-rank kernels to theoretically approximate the original attention weights and result in linear complexity. Some works (Child et al., 2019) introduce sparsity into attention mechanisms by limiting the view of each token to a fixed size and separating tokens into several chunks so that the computation of attention weights takes place in every single chunk rather than a complete sequence. Compared to predefined chunks, some works (Roy et al., 2021; Kitaev et al., 2020) find that using learnable parameters to assign tokens into chunks results in better performance. Another kind of methods (Guo et al., 2019; Lee et al., 2019; Beltagy et al., 2020; Ainslie et al., 2020; Zaheer et al., 2020) combine global and local attention mechanisms, and then use global nodes to gather tokens in a sequence. In this way, the long sequence is compressed into a small number of elements so that we can reduce the complexity.

Keeping the same theoretical computation complexity as the original Transformer, more variants of the model structure can also accelerate the model convergence. Mix-of-experts (MoE) has been proved early (Shazeer et al., 2017) to increase the parameters of deep neural models while keeping the computational overhead nearly unchanged. Recently, Switch Transformers (Fedus et al., 2021) employ this technique in pre-training. They add multiple experts to each layer of Transformers. During each forward and backward step, they select only one expert for computation, and thus the training and inference time remain similar to the ordinary Transformers without experts. Some experimental results show that MoE-based models converge faster than the ordinary ones due to the significantly larger model capacity brought by multiple experts. Some efficient open-source toolkits (He et al., 2021) are also developed to train large-scale MoE-based models.

Another important approach to improve the efficiency of PTMs is model compression. In this setting, large models are compressed to small ones to meet the demand for faster inference and deployment on resource-constrained devices.

Parameter Sharing. PTMs can be compressed with sharing parameters across similar units. ALBERT (Lan et al., 2019) uses factorized embedding parameterization and cross-layer parameter sharing to reduce the parameters of PTMs. Using same weights across all Transformer layers, ALBERT achieves a significant parameter reduction based on the BERT model, and meanwhile has the same or even better performance. This indicates that PTMs can be extremely over-parameterized.

Model Pruning. To take more advantage of the over-parameterized feature of current PTMs, another method to reduce model parameters is model pruning, which cuts off some useless parts in PTMs to achieve accelerating while maintaining the performance. In (Fan et al., 2019), Transformer layers are selectively dropped during training, resulting in a more shallow model during inference. In (Michel et al., 2019; Voita et al., 2019), and (Zhang et al., 2021b), researchers study the redundancy of the attention heads in Transformers and find that only a small part of them is enough for good performance. Most of these heads can be removed with little impact on the accuracy. Other trials such as CompressingBERT (Gordon et al., 2020) try to prune the weights of attention layers and linear layers to reduce the number of parameters in PTMs, while maintaining the comparable performance to the original model.

Knowledge Distillation. Although ALBERT saves the memory usage of PTMs, its inference time is not significantly decreased since features still need to go through its layers with the same number as the original model. Knowledge distillation aims at training a small model to reproduce the behavior of a large teacher model. The memory usage and the time overhead are both decreased when using a small distilled model for inference. There are some typical works employing knowledge distillation for PTMs, such as DistillBERT (Sanh et al., 2019), TinyBERT (Jiao et al., 2019), BERT-PKD (Sun et al., 2019b) and MiniLM (Wang et al., 2020d). In these works, a small student model is trained to mimic the output probability, the hidden states, and the attention matrices of a large teacher model during both the pre-training and fine-tuning stages. With knowledge distillation, the modelegy in the teacher model is transferred into the student model, which can lead to increasing performance compared to training a student model alone. However, the knowledge distillation methods mentioned above require the data used for pre-training the teacher model, which is usually not released in consideration of the data copyright and privacy. Moreover, the teacher model needs to forward over the entire pre-training data to produce logits or intermediate representations for knowledge distillation, causing an even longer training time.

Model Quantization. To get a more compressed model, model quantization is also a useful technique, which has been widely explored in some CNN-based models (Stock et al., 2020; Polino et al., 2018). Model quantization refers to the compression of higher-precision floating-point parameters to lower-precision floating-point ones. Conventional PTMs are usually represented in 32 bits or 16 bits, while models after quantization can be in 8 bits or even 1 or 2 bits. For recent Transformer-based models, 8-bit quantization has been proved to be effective for model compression in Q8BERT (Zafrir et al., 2019), with little impact on the model performance. Despite this, training 1 or 2 Bits models remains challenging due to the significant decrease in model capacity. To alleviate the performance degradation, other methods to preserve the accuracy can also be employed. Q-BERT (Shen et al., 2020a) uses mixed-bits quantization in which the parameters with higher Hessian spectrum require higher precision while those parameters with lower Hessian spectrum need lower precision. TernaryBERT (Zhang et al., 2020b) applies knowledge distillation in quantization, forcing low-bit models to imitate full-precision models. Both Q-BERT and TernaryBERT result in ultra low-bit models. However, low-bit representation is a highly hardware-related technique, which means quantization often requires specific hardware and can not generalize to other devices.

The implicit knowledge captured by PTMs can be roughly divided into two categories: linguistic knowledge and world knowledge.

Linguistic Knowledge. The linguistic knowledge of PTMs attracts most of attentions among all topics of PTMs' interpretation. Compared to conventional neural models such as CNNs and RNNs which have fewer layers and parameters, large-scale PTMs can learn rich linguistic knowledge from massive pre-training data. In order to study PTMs’ linguistic knowledge, researcher design several approaches: (1) Representation Probing: Fix the parameters of PTMs and train a new linear layer on the hidden representations of PTMs for a specific probing task. It is the most popular approach because it can be easily adapted to any probing task without particular design. (2) Representation Analysis: Use the hidden representations of PTMs to compute some statistics such as distances or similarities. According to these statistics, we can construct the relation between different words, phrases, or sentences. (3) Attention analysis: similar to representation analysis, attention analysis compute statistics about attention matrices and is more suitable to discover the hierarchical structure of texts. (4) Generation Analysis: Use language models to directly estimate the probabilities of different sequences or words. The target texts could be correct or incorrect in some linguistic phenomenons.

Representation probing have been widely applied to analyze NLP neural models from word embeddings to PTMs (Köhn, 2015; Ettinger et al., 2016; Shi et al., 2016; Adi et al., 2017; Conneau et al., 2018a; Hewitt and Manning, 2019; Glavaš and Vulić, 2021). Liu et al. (2019) conduct comprehensive probing experiments on 11 linguistic tasks and find that the representations given by large-scale PTMs are competitive compared to previous task-specific models, which indicates that the models have already learned knowledge about tokens, chunks, and pairwise relations. To further investigate how PTMs represent sentence structures about syntactic, semantic, local, and long-range information, Tenney et al. (2019b) design a new edge probing task and examine PTMs on a broad suite of sub-sentence tasks and show that PTMs have strong ability to encode syntactic informative while they bring little improvement on semantic tasks. Similarly, several works also reveal the strong syntax encoding of PTMs (Vilares et al., 2020; Warstadt and Bowman, 2020; Hewitt and Manning, 2019). To analyze the function of different layers, Jawahar et al. (2019a) and Tenney et al. (2019a) show that PTMs encode linguistic information with phrase features at the bottom, syntactic features in the middle and semantic features at the top. Compared to non-contextual representations (e.g., word2vec), PTMs' representations are better in encoding sentence-level properties (Miaschi and Dell’Orletta, 2020). Furthermore, Manning et al. (2020) explore to reconstruct the sentence tree structures given by linguists using a linear transformation of PTMs' embeddings and achieve promising results.

Besides representation probing, researchers try to uncover the structure and relation among different representations. Kim et al. (2020) propose to leverage the concept of Syntactic Distance to construct the constituency trees of sentences from word representations. Rosa and Mareček (2019) analyze how the deletion of one word in a sentence changes representations of other words to reveal the influence of one word on other words.

There are also several works on interpreting PTMs via attention matrices. Lin et al. (2019) quantitatively evaluate attention matrices for subject-verb agreement and anaphor-antecedent dependencies, and show that PTMs tend to encode positional information in lower layers and capture hierarchical information in higher layers. To better characterize the behaviors of PTMs’ attention matrices, Htut et al. (2019) propose to take the maximum attention weight and compute the maximum spanning tree as two statistics. Based on the experimental results, they find that fine-tuning has little impact on the self-attention patterns.

Since PTMs can be directly used to generate tokens or estimate the probabilities of different sentences, it is intuitive to construct analysis tasks based on generation (Goldberg, 2019). Perturbed Masking (Wu et al., 2020) recovers syntactic trees from PTMs without any extra parameter and the structure given by PTMs are competitive with a human-designed dependency schema in some downstream tasks. To analysis the gain of pre-training on estimating the probabilities of ungrammatical words, Schijndel et al. (2019) show that expanding the training corpora yields diminishing returns and the training corpora would need to be unrealistically large to make PTMs match human performance.

World Knowledge. In addition to linguistic knowledge, PTMs also learn rich world knowledge from pre-training, mainly including commonsense knowledge and factual knowledge (Zhou et al., 2020b; Bouraoui et al., 2020).

For the commonsense knowledge, Ettinger (2020) first evaluates PTMs' knowledge in the aspect of psycholinguists and find that the models perform well in the situation of shared category or role reversal but fail with challenging inferences and role-based event. Then, to extract commonsense from PTMs, Davison et al. (2019) propose to first transform relational triples into masked sentences and then rank these sentences according to the mutual information given by PTMs. In the experiments, the PTM-based extraction method without further training even generalizes better than current supervised approaches. Similarly, Da and Kasai (2019) also find that PTMs have learned various commonsense features in their representation space based on a series of probing tasks. In addition to the commonsense features/attributes, the implicit relations between different attributes are important and Forbes et al. (2019) show that current PTMs’ representations cannot model the implicit relations well, which requires further exploration.

For factual knowledge, Petroni et al. (2019) propose to formulate the relational knowledge generation as the completion of fill-in-the-blank statements. According to the experimental results, they find that PTMs significantly outperform previous supervised baselines on this task without any fine-tuning. However, the construction of these fill-in-the-blank statements is non-trivial. To extract more factual knowledge from PTMs, LPAQA (Jiang et al., 2020b) have been propose to automatically search better statements/prompts through mining-based and paraphrasing-based methods. AutoPrompt (Shin et al., 2020) proposes to train discrete prompts for knowledge probing. In P-tuning (Liu et al., 2021b), the authors discover that the better prompts lie in continuous embedding space, rather than discrete space. The P-tuning boosts the P@1 performance on LAMA to 64%, which is 20% higher than AutoPrompt. Moreover, Roberts et al. (2020) fine-tune PTMs for the task of open-domain question answering and find that fine-tuning can further benefit the knowledge generation of PTMs. However, Pörner et al. (2020) find that the success of knowledge generation may rely on learning neural stereotypical associations, i.e., a person with an Italian-sounding name will be predicted to Italian by PTMs. For understanding the number in texts, Wallace et al. (2019c) find that ELMo captures numeracy the best for all pre-trained methods, which is a character-based model, but BERT, which uses sub-word units, is less exact (Wang et al., 2020a). investigates the knowledge stored in Transformer's feed-forward attention matrices and proposes a framework to construct open knowledge graphs using PTMs.

Recent works have identified the severe robustness problem in PTMs using adversarial examples. Adversarial attacks aims to generate new samples, which are mis-classified by models, by small perturbation on the original inputs. For example, PTMs can be easily fooled by synonym replacement (Jin et al., 2020; Zang et al., 2020; Wang et al., 2021a). Meanwhile, irrelevant artifacts such as form words can mislead the PTMs into making wrong predictions (Niven and Kao, 2019; Wallace et al., 2019a). Current works mainly utilize the model prediction, prediction probabilities, and model gradients of the models to search adversarial examples. However, it is difficult to maintain the quality of the adversarial examples generated by machines. Recently, human-in-the-loop methods (Wallace et al., 2019b; Nie et al., 2020) have been applied to generate more natural, valid, and diverse adversarial examples, which brings larger challenge and expose more properties and problems of PTMs. In conclusion, the robustness of PTMs has become a serious security threat when people deploy PTMs for real-world applications.

Following BERT, most PTMs adopt Transformer as the architecture backbone. Although people can easily train a deep Transformer and achieve significant improvement over previous works using CNN and RNN, Transformer meets the problem of over-parameterization. Researchers have shown that the multi-head attention structures are redundant in the tasks of machine translation (Michel et al., 2019), abstractive summarization (Baan et al., 2019), and language understanding (Kovaleva et al., 2019), i.e., when removing part of attention heads, we can achieve better performance. This phenomenon is consistent to the observation in (Clark et al., 2019) where they find that most heads in the same layer have similar self-attention patterns. Furthermore, Kovaleva et al. (2019) conduct a qualitative and quantitative analysis of the information encoded by PTMs’ heads. Their findings suggest that the attention behaviors of different heads can be categorized into a limited set of patterns. Besides the multi-head attention, several other works explore to identify the sparsity of parameters. Gordon et al. (2020) show that low levels of pruning (30–40%) do not affect pre-training loss or the performance on downstream tasks at all. Targeting the sparsity during fine-tuning, Prasanna et al. (2020) validate the lottery ticket hypothesis on PTMs and find that it is possible to find sub-networks achieving performance that is comparable with that of the full model. Surprisingly, Kao et al. (2020) show that we can improvement the performance by simply duplicating some hidden layers to increase the model capacity, which suggests that the redundant parameters may benefit the fine-tuning.

Since pre-training has achieved great success in deep learning, researchers try to investigate how pre-training works, especially unsupervised pre-training. In the early days of deep learning, people found that it is effective to train a deep belief network by greedy layer-wise unsupervised pre-training followed by supervised fine-tuning (Hinton et al., 2006). Recently, pre-training based on contrast learning including language modeling has become the mainstream approach. In this section, we will introduce some theoretical explanatory hypotheses or frameworks for pre-training.

Towards the recent development of pre-training objective, Saunshi et al. (2019) conduct a theoretical analysis of contrastive unsupervised representation learning. Contrastive learning treats the pairs of text/images appearing in the same context as the semantically similar pairs and the randomly sampled pairs as the semantically dissimilar pairs. Then, the distance between the similar pair should be close and the distance between the dissimilar pair should be distant. In the prediction process of language modeling, the context and the target word are the similar pair and the other words are negative samples (Kong et al., 2020). Saunshi et al. (2019) first provide a new conceptual framework to bridge the gap between pre-training and fine-tuning. Specifically, they introduce the concept of latent classes and the semantically similar pairs are from the same latent class. For example, the latent class can be “’happy” to include all texts including happy sentiments. The latent classes cover all possible classes and the classes defined by downstream tasks are from the set of latent classes. Then, they prove that the loss of contrastive learning is the upper bound of the downstream loss. Hence, when optimizing the pre-training loss, we can expect a lower loss in downstream tasks.

From the aspect of architectures and pre-training methods, we believe the following problems worth further exploring in the future:

New Pre-Training Tasks. The general-purpose PTMs are always our pursuits for learning the intrinsic universal knowledge of languages (even world knowledge). However, such PTMs usually need deeper architecture, larger corpora and challenging pre-training tasks. All these requirements further result in higher training costs. Moreover, training huge models is also a challenging problem, which needs sophisticated and efficient training techniques such as distributed training, mixed-precision training, etc. Therefore, a more practical direction is to design more efficient self-supervised pre-training tasks and training methods according to the capabilities of existing hardware and software. ELECTRA (Clark et al., 2020) is a good attempt towards this direction.

Beyond Fine-Tuning. Currently, fine-tuning is the dominant method to transfer the knowledge of PTMs to downstream tasks but one deficiency is its parameter inefficiency: every downstream task has its own fine-tuned parameters. An improved solution is to fix the original parameters of PTMs and add small fine-tunable adaption modules for specific tasks. Thus, we can use a shared PTM to serve multiple downstream tasks. Recently, with the emerging of GPT-3, a novel genre for model tuning, namely prompt tuning, is getting more and more attention. By designing, generating and searching discrete (Petroni et al., 2019; Gao et al., 2021; Hu et al., 2021) or continuous (Liu et al., 2021b; Han et al., 2021; Lester et al., 2021) prompts and using MLM for specific downstream tasks, these models could (1) bridge the gap between pre-training and fine-tuning, and thereby perform better on downstream tasks; (2) reduce the computational cost on fine-tuning the tremendous amounts of parameters. To sum up, prompt tuning is a promising way to stimulate the linguistic and world knowledge distributed in PTMs.

Reliability. The reliability of PTMs is also becoming an issue of great concern with the extensive use of PTMs in production systems. The studies of adversarial attacks (Li et al., 2020b,c; Zhang et al., 2021c) against PTMs help us understand their capabilities by fully exposing their vulnerabilities. Adversarial defenses (Si et al., 2020; Yao et al., 2021; Li and Qiu, 2021) for PTMs are also promising, which can improve the robustness of PTMs and make them immune against adversarial attacks. Overall, as a key component in many NLP applications, the interpretability and reliability of PTMs remain to be further explored, which will help us understand how PTMs work and provide guidance for better use and further improvement of PTMs.

Although multimodal and multilingual PTMs have witnessed numerous advances in the last two years, they still have the following ongoing research lines:

More Modalities. In addition to image and text, video and audio can also be exploited for multimodal pre-training. The main challenge thus lies in how to model temporal contexts involved in these two modalities. In particular, for large-scale pre-training over video-text pairs, the conventional self-supervised learning methods are not suitable due to their high computational costs. To handle this problem, it is important to develop more effective and efficient self-supervised learning methods for more complex modalities.

More Insightful Interpretation. It is still unknown why bridging vision and language works. For example, regardless of the advantages brought by multimodal pre-training, does it lead to any harm to the single modality (image or text)? If the answer is yes, can we overcome this drawback during multimodal pre-training? Along this research line, the latest visualization tools for deep learning can be exploited for the interpretation of multimodal pre-training.

More Downstream Applications. It is well-known that multimodal pre-training can be applied to image-text retrieval, image-to-text generation, text-to-image generation and other downstream tasks. However, it is still challenging to find a “true” real-world application scenario for multimodal pre-training, since many effective engineering tricks can be leveraged instead (even with less cost). A closer collaboration with the industry is thus needed.

Transfer Learning. Currently, to make multimodal multilingual models handle different languages, data for each language is required during pre-training. It is not flexible to add unseen languages during pre-training. Therefore, a new pre-training framework should be explored to easily adapt to those unseen languages. Besides, current multimodal multilingual models are not able to process audio data. For example, to translate English audio to Chinese audio, we need to first transfer English audio to English text by an extra speech recognition system. After translation with a cross-lingual model, we need to further transfer Chinese text to Chinese audio by an extra text-to-speech tool. How to directly transfer the source language audio to the target language text or target language audio by multimodal multilingual PTMs is also worth exploring.

Deep learning models have become increasingly complicated and large (Devlin et al., 2019; Brown et al., 2020; Kaplan et al., 2020; Fedus et al., 2021) in the recent years. The novel requirements of large-scale deep learning models bring severe challenges to the existing deep learning frameworks such as TensorFlow (Abadi et al., 2016) and PyTorch (Paszke et al., 2019), which were designed in the early days without initially foreseeing the emerging requirements such as model/pipeline parallelism of large models (Brown et al., 2020; Huang et al., 2019b; Wang et al., 2019). To develop more efficient frameworks, the following directions are helpful.

Data Movement. Developing an efficient distributed deep learning framework faces various challenges. One has to carefully manage the data movement between devices, which may otherwise become the performance bottleneck (Narayanan et al., 2019; Jiang et al., 2020a). A well-defined parallelism strategy is needed to place and schedule computational tasks on inter-connected devices, by minimizing the communication cost, maximizing the computational and memory resources, and optimizing the computation-communication overlap. In the best case, this efficient parallelism strategy can be generated automatically.

Parallelism Strategies. Particular to the choice of parallelism strategy, data parallelism, model parallelism, pipeline parallelism, and various hybrid parallelism approaches can find their best usage depending on the structure of neural networks and hardware configuration (Ben-Nun and Hoefler, 2019). Data parallelism is especially suitable for deep learning models with a relatively small set of parameters (usually less than tens of million parameters) where near-linear speed-up can be achieved when the back-propagation maximally overlaps with the gradient/parameter communication (Hashemi et al., 2019; Peng et al., 2019; Jiang et al., 2020a). Model parallelism and pipeline parallelism are for models with a more significant number of parameters, which probably cannot fit into a single device. In current practice, a user must thoroughly consider the network structure given a deep learning model and the inter-device communication bandwidth to decide the most appropriate parallelism strategies or switch between different strategies (Shazeer et al., 2018).

Large-Scale Training. Given the poor support to model parallelism and pipeline parallelism by existing deep learning frameworks, some emerging open-source projects develop dedicated frameworks for large-scale training. For example, HugeCTR (Oldridge et al., 2020) is used for large-scale click-through rate estimation. Megatron-LM (Shoeybi et al., 2019; Narayanan et al., 2021) and Deep Speed (Rajbhandari et al., 2020, 2021) target at training large-scale NLP PTMs. Insight Face (ins, 2021) trains large-scale face recognition models. However, these frameworks are restricted to limited application cases and cannot serve as a general solution. Further, these approaches cannot work together to constitute a complete solution due to the compatibility issue.

Wrappers and Plugins. Without a mechanism to support model parallelism and pipeline parallelism, one has to develop various libraries dedicated to some particular algorithms via inserting the data routing operations by hand between computing operations on top of existing frameworks. Further, communication and computation need to be manually overlapped to maximize the system throughput. Manually programming communication operations is prohibitively complicated and only can solve problems case by case, leading to a significant obstacle in applying parallelism strategies to new deep learning models. If communication operations can be automatically managed transparently to users by deep learning frameworks, more models and applications can benefit from the distributed training.

To support more complicated parallelism strategies, many schemes are used as wrappers or plugins based on some mainstream deep learning frameworks such as TensorFlow and PyTorch. Mesh-TensorFlow (Shazeer et al., 2018), FlexFlow (Jia et al., 2019), OneFlow (one, 2021), MindSpore (min, 2021) and GShard (Lepikhin et al., 2021) provide APIs for developers to express a wide range of parallel computation patterns for different components of deep neural models. The SBP configuration in OneFlow could be still too complex for users to set. However, directly programming with communication primitives for a different kind of parallelism is more complicated. OneFlow transforms the manually programming to just setting SBP signatures. Moreover, in OneFlow, the user could just set the SBP signatures of a subset of operations instead of the whole set, and leave the rest SBP to be inferred with heuristic approaches like GShard (Lepikhin et al., 2021), in which users provide some initial annotations or use default annotations as seed, then the algorithm propagates the sharding information to the un-annotated tensors. The approach in FlexFlow (Jia et al., 2019) can also be used here. The automatic scheduling of parallelism strategies is the trend of distributed training in the future.

In this subsection, we analyze the future directions in a more fundamental way. In the aspect of theoretical foundation, we discuss the following research problems.

Uncertainty. One under-addressed issue with PTMs (as well as other deep neural networks) is that they are often over-confident in predictions, i.e., these models do not know what they do not know. For instance, GPT-3 can be used to answer questions with promising performance on benchmark datasets. However, if you ask a simple question like “How many eyes does my foot have?”, GPT-3 would certainly produce an answer like “Your foot has two eyes”, which looks counter-intuitive.⁶ Of course, the above question is not often asked by normal human beings. It is generally a challenging task to deal with such out-of-distribution (OOD) data in machine learning.

To address the above challenge, one promising direction is to adopt Bayesian methods that explore probabilistic tools to capture the uncertainty of both data and model (also known as aleatoric uncertainty and epistemic uncertainty respectively) (Der Kiureghian and Ditlevsen, 2009) or derive some testing statistics. Such uncertainty or statistics is helpful to detect outliers (Wang et al., 2020f). Recently, much work has been done on the theory, algorithms and programming libraries of Bayesian deep learning, which conjoins Bayesian methods and deep networks (e.g., see (Shi et al., 2017) for more details). Such progress can be further extended to large-scale PTMs to properly characterize uncertainty and avoid over-confident outputs. Of course, improving the computational efficiency of Bayesian deep learning is a key factor to address the above challenge.

Generalization and Robustness. Another important issue with PTMs is on generalization. As an important advancement of deep learning, it inherits the advantages as well as challenges of deep neural networks. It has been observed that classical learning theory is not sufficient to understand the behavior of deep networks (Zhang et al., 2017), thereby calling for new tools in learning theory. As for PTMs, besides theoretical understanding of the neural models themselves (e.g., Transformer and BERT), new questions arise. For example, it is important to theoretically understand the roles of pre-training in improving the generalization of downstream tasks. The recent work (Saunshi et al., 2019) provides a fruitful attempt at understanding contrastive learning with particular assumptions. However, it is still largely open to analyze PTMs under more realistic settings.

As we mentioned before, the adversarial robustness also raises new questions. In previous work, it was shown that a higher sample complexity is needed in order to achieve adversarial robustness for neural networks (Schmidt et al., 2018). Such analysis has inspired further improvements (e.g. (Pang et al., 2020),). However, it is generally unknown how large-scale PTMs can help in this aspect. Are there effective ways to explore PTMs as extra data resources to improve the robustness of downstream tasks? Also, the robustness of PTMs themselves is an unsolved issue, as mentioned before.

As introduced in section 7, PTMs can achieve a surge of improvements for a wide range of NLP tasks because they learn versatile knowledge from large unlabeled corpora. As opposed to the knowledge represented by discrete symbols, which is interpretable to human beings, the knowledge stored in PTMs is represented as real-valued vectors. For example, given a triple 〈h, r, t〉 in a knowledge graph, it is easy to know that the head entity h has a relation r to the tail entity t. In contrast, you seem to have difficulty knowing what a representation produced by a PTM means. Therefore, we can refer to the knowledge stored in PTMs as “modeledge”, which is distinguished from the discrete symbolic knowledge formalized by human beings.

Knowledge-Aware Tasks. While the use of symbolic knowledge is effective, it is time-consuming and labor-intensive to manually organize this discrete knowledge such as building various knowledge bases. With the rapid advance of researches on PTMs, there emerge various PTMs such as GPT, BERT and BART. More and more researchers have probed into what knowledge do PTMs learn from the data, and why they perform so well on downstream tasks (Jawahar et al., 2019b; Ethayarajh, 2019). Petroni et al. (2019) state that PTMs can be seen as knowledge bases and study how to apply PTMs to the knowledge completion task. Ethayarajh (2019) also claim that PTMs would be open knowledge graphs and propose an unsupervised method to build knowledge graphs based on PTMs. From all these knowledge-aware tasks, we can find that a wealth of human knowledge is captured by PTMs and stored in the form of modeledge. How to stimulate the modeledge of PTMs is worth further exploring in the future.

Modeledge Storage and Management. As existing PTMs are built on varying architectures and may be trained with different corpora, they contain diverse modeledge. As a result, how to store and manage various continuous modeledge in PTMs becomes a new challenge. There are two kinds of straightforward ideas. The first is to pre-train a huge model on extra-large scale data. Then, PTMs will have the extraordinary ability to cover almost all modeledge in existing PTMs. This method is simple and effective while it requires extremely high computational power and storage resources. For example, GPT-3 uses about 175 billion parameters. The second is to combine multiple models into one large model based on the mixture of experts (MoE) (Jacobs et al., 1991). For example, Fedus et al. (2021) improve MoE to propose Switch Transformers. This method is easy to contain new models but the requirement of memory grows as the number of models increases.

Considering that there are both similarities and differences among existing PTMs, we have an important question that needs to be answered: is it possible to build a universal continuous knowledge base (UCKB) that stores modeledge from various PTMs? The UCKB can not only store continuous modeledge imported from existing PTMs but also can blend different modeledge and then export the fused modeledge to a model to make it more powerful. Chen et al. (2020a) first propose the concept of UCKB and make some preliminary explorations. They regard neural networks as parameterized functions and use knowledge distillation (Hinton et al., 2014) to import and export modeledge. UCKB overcomes the redundancy of model storage and stores the modeledge of various models into a common continuous knowledge base. However, how to design more effective architectures for the storage and interface of UCKB still remains challenging.

Making PTMs more knowledgeable is an important topic for the future of PTMs. We divide the future development of knowledgeable PTMs into the following three approaches:

Knowledge Augmentation. For an input text, there is rich related external knowledge, which can be used to augment the input. Considering the formats of knowledge and plain text are very different, it is important to bridge the gap between text representations and knowledge representations (including symbols or vectors) and use their information uniformly as input. The solution to this problem requires both unified model architectures and knowledge-guided pre-training objectives.

Knowledge Support. Current model architectures are manually designed and usually very regular. With prior knowledge about the input, we can train different sub-module to process different kinds of input, which may accelerate the process of training and inference and benefit the model efficiency. This process is similar to human behavior where different brain regions correspond to different activity functions.

Knowledge Supervision. Knowledge bases store amounts of structural data, which can be used as a complementary source during pre-training. By learning from both knowledge bases and large-scale corpora, PTMs can have better language understanding and generation abilities compared to only using plain text. Through these three directions, we hope the future PTMs can easily understand the meanings beyond words and achieve better performance on various downstream tasks.

In terms of cognitive PTMs, we believe the following approaches would be helpful:

Cognitive Architecture. Since neural networks are inspired by the micro structure of the human neural system, it is expected to see how the macro function and organization of human cognitive system can enlighten the design of the next generation of intelligence system, such as the Global Working Theory (GWT). The success of CogQA and CogLTX may provide some thoughts on this challenge.

Explicit and Controllable Reasoning. While deep learning has achieved success in many perceptive tasks, how to conduct complex decision making and efficient multi-step reasoning is still unsolved, which may require machines to automatically plan the decision making process into a cognitive graph and do explicit reasoning over the factors in graphs as human do. Methods such as Inverse Prompting (Zou et al., 2021) which shows supreme ability in controlling theme-related text generation would provide some thoughts.

Interactions of Knowledge. Though our PTMs are getting bigger and more general, what knowledge it has learned from pre-training is largely unexplored. Moreover, since our brains are working with the collaboration of different function zones, it is important to see if our PTMs have shaped different inner function modules and how they would interact with each other.

PTMs have been successfully applied in a wide variety of domains and tasks. In this section, we will highlight some of these applications.

Dialog Systems. Many recent open-domain dialog systems are built upon large-scale transformer structures. These examples include Meena (Adiwardana et al., 2020), Blender (Roller et al., 2021), CDial-GPT (Wang et al., 2020e), Plato (Bao et al., 2020) and Plato-2 (Bao et al., 2021), which are trained on large-scale conversation data, commonly with the seq2seq framework. These models have shown capabilities of delivering natural and engaging conversations, some of which have been reported to be close to human-level performance (Adiwardana et al., 2020). However, dialog-specific pre-training tasks are yet to be explored, comparing to pre-training tasks for other applications.

Domain-Specific PTMs. When large-scale domain-specific corpora are cheaply available, we can train domain-specific PTMs on such data. Some notable works include BioBERT (Lee et al., 2020) and SciBERT (Beltagy et al., 2019), which are trained respectively on the biological and scientific literature text. These models are expected and verified to learn more domain-specific knowledge and language use than those trained on the general text. Such domain expertise is usually regarded as important for solving many domain-specific problems.

Domain Adaptation and Task Adaptation. Large-scale PTMs learn general knowledge from the large-scale general text, providing a good initial point to further learn domain-specific knowledge by fine-tuning or other techniques. Although PTMs are becoming larger and larger, the domain-specific data are always limited. Therefore, domain adaptation is becoming crucial for domain-specific applications. It has been evident that the simple fine-tuning of large-scale PTMs is not sufficient for domain-specific applications (Gururangan et al., 2020; Ke et al., 2020). The most essential reason for this is the distribution shift: the data distribution in a specific domain may be substantially different from that in the general pre-training text. Another important issue for the success of domain-specific applications goes to task adaptation. Most often, domain applications have a small set of labeled data, which can empower supervised learning to learn domain expertise more efficiently. However, for super-large PTMs, simply fine-tuning on labeled data seems to be inefficient in computation, nor effective in performance. Thus, how to bridge the gap between pre-training and task-specific fine-tuning becomes crucial. Moreover, efficient and effective task-specific fine-tuning is also an important research direction for the future application of PTMs (Soares et al., 2019; Ding et al., 2021b).