WIXLIB

Token Output(i)Token OutputAggregatorInformationFusion(i)e1Entity ̃1(i)e2(i)ẽ2(i)Entity OutputAggregatorK-Encoder MxInformation rT-Encoder Nx(i)w4(i)w̃4(i)Entity InputMulti-Head AttentionMulti-Head AttentionMulti-HeadAttention(i 1)Token Inputw1bob(i 1)w2dylan(i 1)w3wrote(i 1)w4blow···wn(i1)1962(i 1)e1(i 1)e2Entity InputBob Dylan Blowin’ in the WindToken InputBob Dylan wrote Blowin’ in the Wind in 1962(a) Model Achitecture(b) AggregatorFigure 2: The left part is the architecture of ERNIE. The right part is the aggregator for the mutualintegration of the input of tokens and entities. Information fusion layer takes two kinds of input: one is thetoken embedding, and the other one is the concatenation of the token embedding and entity embedding.After information fusion, it outputs new token embeddings and entity embeddings for the next layer.inference (Chen et al., 2018), knowledge acquisition (Han et al., 2018a), and dialog systems (Madotto et al., 2018). Hence, we argue thatextra knowledge information can effectively benefit existing pre-training models. In fact, somework has attempted to joint representation learning of words and entities for effectively leveraging external KGs and achieved promising results (Wang et al., 2014; Toutanova et al., 2015;Han et al., 2016; Yamada et al., 2016; Cao et al.,2017, 2018). Sun et al. (2019) propose the knowledge masking strategy for masked language modelto enhance language representation by knowledge 1 . In this paper, we further utilize both corpora and KGs to train an enhanced language representation model based on BERT.3.1We denote a token sequence as {w1 , . . . , wn } 2 ,where n is the length of the token sequence.Meanwhile, we denote the entity sequence aligning to the given tokens as {e1 , . . . , em }, where mis the length of the entity sequence. Note that mis not equal to n in most cases, as not every token can be aligned to an entity in KGs. Furthermore, we denote the whole vocabulary containingall tokens as V, and the entity list containing allentities in KGs as E. If a token w V has a corresponding entity e E, their alignment is definedas f (w) e. In this paper, we align an entity tothe first token in its named entity phrase, as shownin Figure 2.3.23MethodologyIn this section, we present the overall frameworkof ERNIE and its detailed implementation, including the model architecture in Section 3.2, the novelpre-training task designed for encoding informative entities and fusing heterogeneous informationin Section 3.4, and the details of the fine-tuningprocedure in Section 3.5.1It is a coincidence that both Sun et al. (2019) and wechose ERNIE as the model names, which follows the interesting naming habits like ELMo and BERT. Sun et al. (2019)released their code on March 16th and submitted their paperto Arxiv on April 19th while we submitted our paper to ACLwhose deadline is March 4th.NotationsModel ArchitectureAs shown in Figure 2, the whole model architecture of ERNIE consists of two stacked modules:(1) the underlying textual encoder (T-Encoder)responsible to capture basic lexical and syntactic information from the input tokens, and (2) theupper knowledgeable encoder (K-Encoder) responsible to integrate extra token-oriented knowledge information into textual information from theunderlying layer, so that we can represent heterogeneous information of tokens and entities into aunited feature space. Besides, we denote the number of T-Encoder layers as N , and the number14432In this paper, tokens are at the subword level.

of K-Encoder layers as M .To be specific, given a token sequence{w1 , . . . , wn } and its corresponding entity sequence {e1 , . . . , em }, the textual encoder firstlysums the token embedding, segment embedding,positional embedding for each token to computeits input embedding, and then computes lexicaland syntactic features {w1 , . . . , wn } as follows,Then, the i-th aggregator adopts an informationfusion layer for the mutual integration of the tokenand entity sequence, and computes the output embedding for each token and entity. For a token wjand its aligned entity ek f (wj ), the informationfusion process is as follows,(i)(i)(i)hj σ(W̃t w̃j W̃e(i) ẽk b̃(i) ),(i)(i)(i)(4)wj σ(Wt hj bt ),{w1 , . . . , wn } T-Encoder({w1 , . . . , wn }),(1)where T-Encoder(·) is a multi-layer bidirectional Transformer encoder. As T-Encoder(·) isidentical to its implementation in BERT and BERTis prevalent, we exclude a comprehensive description of this module and refer readers to Devlinet al. (2019) and Vaswani et al. (2017).After computing {w1 , . . . , wn }, ERNIE adoptsa knowledgeable encoder K-Encoder to inject the knowledge information into languagerepresentation.To be specific, we represent {e1 , . . . , em } with their entity embeddings{e1 , . . . , em }, which are pre-trained by the effective knowledge embedding model TransE (Bordes et al., 2013). Then, both {w1 , . . . , wn } and{e1 , . . . , em } are fed into K-Encoder for fusing heterogeneous information and computing final output embeddings,{w1o , . . . , wno }, {eo1 , . . . , eon } K-Encoder({w1 , . . . , wn }, {e1 , . . . , em }).(2){w1o , . . . , wno } and {eo1 , . . . , eon } will be used asfeatures for specific tasks. More details of theknowledgeable encoder K-Encoder will be introduced in Section 3.3.3.3Knowledgeable EncoderAs shown in Figure 2, the knowledgeable encoder K-Encoder consists of stacked aggregators, which are designed for encoding both tokens and entities as well as fusing their heterogeneous features. In the i-th aggregator, the in(i 1)(i 1)put token embeddings {w1, . . . , wn } and(i 1)(i 1)entity embeddings {e1, . . . , em } from thepreceding aggregator are fed into two multi-headself-attentions (MH-ATTs) (Vaswani et al., 2017)respectively,(i)(i 1){w̃1 , . . . , w̃n(i) } MH-ATT({w1(i)(i 1)(i){ẽ1 , . . . , ẽm} MH-ATT({e1(i)ek, . . . , w(i 1)}),n σ(We(i) hj b(i)e ).where hj is the inner hidden state integrating theinformation of both the token and the entity. σ(·)is the non-linear activation function, which usually is the GELU function (Hendrycks and Gimpel, 2016). For the tokens without correspondingentities, the information fusion layer computes theoutput embeddings without integration as follows,(i)(i)hj σ(W̃t w̃j b̃(i) ),(i)(i)(i)(5)wj σ(Wt hj bt ).For simplicity, the i-th aggregator operation isdenoted as follows,(i)(i){w1 , . . . , wn(i) }, {e1 , . . . , e(i)m } Aggregator((i 1){w1(i 1), . . . , wn(i 1) }, {e1, . . . , e(i 1)}).m(6)The output embeddings of both tokens and entitiescomputed by the top aggregator will be used asthe final output embeddings of the knowledgeableencoder K-Encoder.3.4Pre-training for Injecting KnowledgeIn order to inject knowledge into language representation by informative entities, we propose anew pre-training task for ERNIE, which randomlymasks some token-entity alignments and then requires the system to predict all corresponding entities based on aligned tokens. As our task issimilar to training a denoising auto-encoder (Vincent et al., 2008), we refer to this procedure asa denoising entity auto-encoder (dEA). Considering that the size of E is quite large for the softmax layer, we thus only require the system to predict entities based on the given entity sequence instead of all entities in KGs. Given the token sequence {w1 , . . . , wn } and its corresponding entitysequence {e1 , . . . , em }, we define the aligned entity distribution for the token wi as follows,(i 1), . . . , em}).(3)1444exp(linear(wio ) · ej )p(ej wi ) Pm,ok 1 exp(linear(wi ) · ek )(7)

Mark Twain wrote The Million Pound Bank Note in 1893.Input for Common NLP themillionpoundbanknotein1893.[SEP]Input for Entity Typing:[CLS][ENT]marktwain[ENT]Input for Relation Figure 3: Modifying the input sequence for the specific tasks. To align tokens among different types ofinput, we use dotted rectangles as placeholder. The colorful rectangles present the specific mark tokens.where linear(·) is a linear layer. Eq. 7 will beused to compute the cross-entropy loss functionfor dEA.Considering that there are some errors in tokenentity alignments, we perform the following operations for dEA: (1) In 5% of the time, for a giventoken-entity alignment, we replace the entity withanother random entity, which aims to train ourmodel to correct the errors that the token is alignedwith a wrong entity; (2) In 15% of the time, wemask token-entity alignments, which aims to trainour model to correct the errors that the entity alignment system does not extract all existing alignments; (3) In the rest of the time, we keep tokenentity alignments unchanged, which aims to encourage our model to integrate the entity information into token representations for better languageunderstanding.Similar to BERT, ERNIE also adopts themasked language model (MLM) and the next sentence prediction (NSP) as pre-training tasks to enable ERNIE to capture lexical and syntactic information from tokens in text. More details of thesepre-training tasks can be found from Devlin et al.(2019). The overall pre-training loss is the sum ofthe dEA, MLM and NSP loss.to fine-tune ERNIE for relation classification isto apply the pooling layer to the final output embeddings of the given entity mentions, and represent the given entity pair with the concatenationof their mention embeddings for classification. Inthis paper, we design another method, which modifies the input token sequence by adding two marktokens to highlight entity mentions. These extramark tokens play a similar role like position embeddings in the conventional relation classificationmodels (Zeng et al., 2015). Then, we also take the[CLS] token embedding for classification. Notethat we design different tokens [HD] and [TL] forhead entities and tail entities respectively.The specific fine-tuning procedure for entitytyping is a simplified version of relation classification. As previous typing models make full use ofboth context embeddings and entity mention embeddings (Shimaoka et al., 2016; Yaghoobzadehand Schütze, 2017; Xin et al., 2018), we arguethat the modified input sequence with the mentionmark token [ENT] can guide ERNIE to combineboth context information and entity mention information attentively.3.5In this section, we present the details of pretraining ERNIE and the fine-tuning results onfive NLP datasets, which contain both knowledgedriven tasks and the common NLP tasks.Fine-tuning for Specific TasksAs shown in Figure 3, for various common NLPtasks, ERNIE can adopt the fine-tuning proceduresimilar to BERT. We can take the final output embedding of the first token, which corresponds tothe special [CLS] token, as the representation ofthe input sequence for specific tasks. For someknowledge-driven tasks (e.g., relation classification and entity typing), we design special finetuning procedure:For relation classification, the task requires systems to classify relation labels of given entity pairsbased on context. The most straightforward way4Experiments4.1Pre-training DatasetThe pre-training procedure primarily acts in accordance with the existing literature on pre-traininglanguage models. For the large cost of trainingERNIE from scratch, we adopt the parameters ofBERT released by Google3 to initialize the Transformer blocks for encoding tokens. Since pre-14453https://github.com/google-research/bert

training is a multi-task procedure consisting ofNSP, MLM, and dEA, we use English Wikipediaas our pre-training corpus and align text to Wikidata. After converting the corpus into the formatted data for pre-training, the annotated input hasnearly 4, 500M subwords and 140M entities, anddiscards the sentences having less than 3 entities.Before pre-training ERNIE, we adopt theknowledge embeddings trained on Wikidata4 byTransE as the input embeddings for entities. Tobe specific, we sample part of Wikidata whichcontains 5, 040, 986 entities and 24, 267, 796 facttriples. The entity embeddings are fixed duringtraining and the parameters of the entity encodingmodules are all initialized randomly.4.2Parameter Settings and Training DetailsIn this work, we denote the hidden dimension oftoken embeddings and entity embeddings as Hw ,He respectively, and the number of self-attentionheads as Aw , Ae respectively. In detail, we havethe following model size: N 6, M 6, Hw 768, He 100, Aw 12, Ae 4. The totalparameters are about 114M.The total amount of parameters of BERTBASEis about 110M, which means the knowledgeablemodule of ERNIE is much smaller than the language module and has little impact on the run-timeperformance. And, we only pre-train ERNIE onthe annotated corpus for one epoch. To acceleratethe training process, we reduce the max sequencelength from 512 to 256 as the computation of selfattention is a quadratic function of the length. Tokeep the number of tokens in a batch as same asBERT, we double the batch size to 512. Exceptfor setting the learning rate as 5e 5 , we largelyfollow the pre-training hyper-parameters used inBERT. For fine-tuning, most hyper-parameters arethe same as pre-training, except batch size, learning rate, and number of training epochs. We findthe following ranges of possible values work wellon the training datasets with gold annotations, i.e.,batch size: 32, learning rate (Adam): 5e 5 , 3e 5 ,2e 5 , number of epochs ranging from 3 to 10.We also evaluate ERNIE on the distantly supervised dataset, i.e., FIGER (Ling et al., 2015). Asthe powerful expression ability of deeply stackedTransformer blocks, we found small batch sizewould lead the model to overfit the training data.Hence, we use a larger batch size and less train4https://www.wikidata.org/DatasetFIGEROpen 05632,0001136Table 1: The statistics of the entity typing datasetsFIGER and Open Entity.ModelAcc.MacroMicroNFGEC (Attentive)NFGEC 71.63ERNIE57.1976.5173.39Table 2: Results of various models on FIGER (%).ing epochs to avoid overfitting, and keep the rangeof learning rate unchanged, i.e., batch size: 2048,number of epochs: 2, 3.As most datasets do not have entity annotations,we use TAGME (Ferragina and Scaiella, 2010) toextract the entity mentions in the sentences andlink them to their corresponding entities in KGs.4.3Entity TypingGiven an entity mention and its context, entity typing requires systems to label the entity mentionwith its respective semantic types. To evaluateperformance on this task, we fine-tune ERNIE ontwo well-established datasets FIGER (Ling et al.,2015) and Open Entity (Choi et al., 2018). Thetraining set of FIGER is labeled with distant supervision, and its test set is annotated by human.Open Entity is a completely manually-annotateddataset. The statistics of these two datasets areshown in Table 1. We compare our model withthe following baseline models for entity typing:NFGEC. NFGEC is a hybrid model proposedby Shimaoka et al. (2016). NFGEC combines therepresentations of entity mention, context and extra hand-craft features as input, and is the stateof-the-art model on FIGER. As this paper focuseson comparing the general language representationabilities of various neural models, we thus do notuse the hand-craft features in this work.UFET. For Open Entity, we add a new hybridmodel UFET (Choi et al., 2018) for comparison.UFET is proposed with the Open Entity dataset,which uses a Bi-LSTM for context representationinstead of two Bi-LSTMs separated by entity mentions in NFGEC.Besides NFGEC and UFET, we also report theresult of fine-tuning BERT with the same inputformat introduced in Section 3.5 for fair com-1446

ModelPRF1NFGEC 8.0073.56ERNIE78.4272.9075.56ModelTable 3: Results of various models on Open Entity Table 5: Results of various models on FewRel and TACRED (%).4.4Table 4: The statistics of the relation classificationdatasets FewRel and TACRED.parison. Following the same evaluation criteriaused in the previous work, we compare NFGEC,BERT, ERNIE on FIGER, and adopt strict accuracy, loose macro, loose micro scores for evaluation. We compare NFGEC, BERT, UFET, ERNIEon Open Entity, and adopt precision, recall, microF1 scores for evaluation.The results on FIGER are shown in Table 2.From the results, we observe that: (1) BERTachieves comparable results with NFGEC on themacro and micro metrics. However, BERT haslower accuracy than the best NFGEC model. Asstrict accuracy is the ratio of instances whose predictions are identical to human annotations, it illustrates some wrong labels from distant supervision are learned by BERT due to its powerfulfitting ability. (2) Compared with BERT, ERNIEsignificantly improves the strict accuracy, indicating the external knowledge regularizes ERNIE toavoid fitting the noisy labels and accordingly benefits entity typing.The results on Open Entity are shown in Table 3. From the table, we observe that: (1) BERTand ERNIE achieve much higher recall scores thanthe previous entity typing models, which meanspre-training language models make full use ofboth the unsupervised pre-training and manuallyannotated training data for better entity typing. (2)Compared to BERT, ERNIE improves the precision by 2% and the recall by 2%, which means theinformative entities help ERNIE predict the labelsmore precisely.In summary, ERNIE effectively reduces thenoisy label challenge in FIGER, which is awidely-used distantly supervised entity typingdataset, by injecting the information from KGs.Besides, ERNIE also outperforms the baselines onOpen Entity which has gold annotations.Relation ClassificationRelation classification aims to determine the correct relation between two entities in a given sentence, which is an important knowledge-drivenNLP task. To evaluate performance on thistask, we fine-tune ERNIE on two well-establisheddatasets FewRel (Han et al., 2018c) and TACRED (Zhang et al., 2017). The statistics of thesetwo datasets are shown in Table 4. As the original experimental setting of FewRel is few-shotlearning, we rearrange the FewRel dataset for thecommon relation classification setting. Specifically, we sample 100 instances from each class forthe training set, and sample 200 instances for thedevelopment and test respectively. There are 80classes in FewRel, and there are 42 classes (including a special relation “no relation”) in TACRED. We compare our model with the followingbaseline models for relation classification:CNN. With a convolution layer, a max-poolinglayer, and a non-linear activation layer, CNN getsthe output sentence embedding, and then feeds itinto a relation classifier. To better capture the position of head and tail entities, position embeddingsare introduced into CNN (Zeng et al., 2015; Linet al., 2016; Wu et al., 2017; Han et al., 2018b).PA-LSTM. Zhang et al. (2017) propose PALSTM introducing a position-aware attentionmechanism over an LSTM network, which evaluates the relative contribution of each word in thesequence for the final sentence representation.C-GCN. Zhang et al. (2018) adopt the graphconvolution operations to model dependency treesfor relation classification. To encode the wordorder and reduce the side effect of errors in dependency parsing, Contextualized GCN (C-GCN)firstly uses Bi-LSTM to generate contextualizedrepresentations as input for GCN models.In addition to these three baselines, we also finetune BERT with the same input format introducedin Section 3.5 for fair comparison.1447

84.89ERNIEw/o entitiesw/o ble 7: Ablation study on FewRel (%).Table 6: Results of BERT and ERNIE on different tasksof GLUE (%).As FewRel does not have any null instancewhere there is not any relation between entities,we adopt macro averaged metrics to present themodel performances. Since FewRel is built bychecking whether the sentences contain facts inWikidata, we drop the related facts in KGs before pre-training for fair comparison. From Table 5, we have two observations: (1) As the training data does not have enough instances to trainthe CNN encoder from scratch, CNN just achievesan F1 score of 69.35%. However, the pre-trainingmodels including BERT and ERNIE increase theF1 score by at least 15%. (2) ERNIE achieves anabsolute F1 increase of 3.4% over BERT, whichmeans fusing external knowledge is very effective.In TACRED, there are nearly 80% nullinstances so that we follow the previouswork (Zhang et al., 2017) to adopt microaveraged metrics to represent the model performa

most recently proposed models, obtains the state-of-the-art results on various NLP applications by simple fine-tuning, including named entity recog-nition (Sang and De Meulder,2003), question indicates equal contribution yCorresponding author: Z.Liu(l