ERLKG

CORD-19 requires pre-processing for downstream NLP tasks such as tokenization at both sentence (sentence tokenization) and word (word tokenization) level as well as addition of POS (Parts of Speech) tags for richer information per token. We used the Spacy library for this purpose.

Proteins, chemicals and diseases are the biomedical entities of interest that need to be extracted from running text. We therefore use SciBERT^[2] on the CORD-19 dataset to tag said entities. SciBERT is a BERT variant which has been trained on 1.14M full text scientific research papers of which 18% articles are from the computer science domain and 82% are from the broad biomedical domain. SciBERT is a perfect fit for this task since most CORD-19 papers belong to the biomedical domain.

In order to achieve better performance, we finetuned SciBERT on three different datasets, namely JNLPBA^[3] (for proteins), NCBI-disease corpus^[4] (for disease) and CHEMDNER^[5] (for chemicals).

Based on the now tagged CORD-19 corpus, it is necessary to understand how every entity is related to each other. Accordingly, we will connect them via edges and construct our Knowledge Graph (KG). We use SciBERT for this task as well with only sentences having more than one tagged entity being fed to the mode.

We want to target only those entity pairs that have chemical-protein and chemical-disease relationships (protein-disease can be found indirectly). To achieve better performance, we again finetune SciBERT on CHEMPROT^[6] (for chemical-protein examples) and BC5CDR^[7] (for chemical-disease examples) datasets as they contain the said relationships.

From the NER module, we obtained biomedical entities that formed the nodes of a graph while, from the RE module, we obtained the relationship types among said entities which formed the edges between nodes. This gives us a huge graph with multiple disconnected components and repeated nodes. In order to form a single KG, all repeated nodes are merged and edges are handled accordingly. An abstraction of the process is provided in Fig. 5.

Now that our KG is constructed, we need to take advantage of the rich information it holds. There are many ways to do this, but one of the proven methods is to learn vector representations of each node using the type of incoming and outgoing edges as well as neighbourhood information for a given node.

As stated earlier, in order to take advantage of the rich information contained in our KG, one of the proven methods is to learn dense vector representations of the nodes. Just like word2vec^[8] learns vector embeddings of words and groups similar terms together, example, "house" and "home", we need a method that can provide such vector representations for nodes in our KG so that similar entities can be grouped together. This is exactly what Knowledge Graph embedding techniques achieve.

We explored many KG embedding techniques like TransD^[9], TransE^[10], RotatE^[11], etc and compared the results. The basic idea of these embedding techniques is to keep vectors of connected nodes closer to each other in comparison to all other nodes in a graph thus segregating unrelated entities from related ones. We use OpenKE^[12] which is an Open-source Framework for trying different KG embedding techniques. Fig. 6 gives an abstract overview of such methods.

After learning the latent representations we now have embeddings for each entity, including COVID-19. Also the embeddings are such that similar entities are present closer to each other in the learned vector space while entities with dissimilar properties are present further apart. Thus, in order to find entities associated with COVID-19, we simply take the cosine similarity of COVID-19's vector with the vectors of all other biomedical entities, rank them and pick the top k as our final list. This k can be decided by the user.

Our evaluation strategy for ERLKG was to perform both extrinsic evaluation in the form of link prediction task as well as intrinsic evaluation.

Link prediction is the task of predicting the existence of an edge between two entities. If our generated KG is correct, we should be able to achieve a good score on said task by feeding the vectors learned by graph embedding techniques into a classification model.

The test and validation set is created by deleting edges with the addition of equal number of randomly sampled pairs of false links (nodes that did not have connections in the graph). The test and validation sets have 10 percent and 5 percent of true links, respectively. We use GCN-AE^[13] as our model for the task. The Average Precision and ROC score of each setting is noted and used to benchmark these embedding types as can be seen in Table 1. We can see that RotatE and TransD perform the best.

From the link prediction task, we know that both TransD and RotatE are good graph embedding techniques but in order to find a clear winner, we decided to perform another round of evaluation. This time, we went for intrinsic evaluation which is the process of evaluating a model with the direct involvement of human judgement. Similar to our KG predicament, there is no intrinsic evaluation dataset for COVID-19. We thus propose two new datasets, COV19_729 and COV19_25, that we created with the help of medical experts. Following are the details on how we developed the datasets:

• COV19_729: After generating the KG, a list of all entities are supplied to a medical practitioner, who clubbed the terms into 3 groups based on their relatedness to COVID-19, i.e., NOT RELATED, PARTIALLY RELATED and HIGHLY RELATED. It was identified that the number of entities in the HIGHLY RELATED group are much less in comparison to the other two categories. Thus, in order to reduce bias, the expert sampled nearly equal number of entities from each group, resulting in a final dataset comprising of 729 entities. This dataset was then shuffled and passed on to two independent medical exprts, who provided ratings to each sample indicating how related an entity is to COVID-19 on a scale of 0 (NOT RELATED) to 5 (HIGHLY RELATED).

  The inter-rater agreement score (kappa score) was found to be 0.5116, which lies in the moderate agreement range. We, therefore, averaged out the ratings and propose a relatively large, intrinsic evaluation dataset called COV19_729 for benchmarking COVID-19 related embedding techniques. Table 2 shows a snapshot of COV19_729.

  Entity	Cosine with COVID-19	Rating by medical expert
  retinoic acid inducible gene-1 (protein)	-0.0799	0
  hydroxyprolinol (chemical)	-0.0183	2
  acute asthma attacks (disease)	0.0530	1
  pc18 (chemical)	0.1540	1
  immunodominant epitopes (protein)	0.1894	2
  receptor binding domain (protein)	0.4644	5
  spike glycoprotein (protein)	0.4138	4

  
  
  Table 2: A snapshot of the COV19_729 dataset. The second column denotes cosine scores generated by TransD.



• COV19_25: The top 100 entities given by TransD and RotatE were selected and an intersection of the generated entities was taken, which was then passed on to a medical practitioner. The expert recommended a list of 25 relevant entities, out of the provided set. This list was then sent to another expert who rated the entities based on their relatedness to COVID-19. This was named COV19_25.

  If you want to know the reason behind proposing two separate intrinsic evaluation datasets, then please do give the paper a read.

Once the datasets were complete, we performed intrinsic evalaution by generating cosine similarity scores from both TransD and RotatE for the biomedical entity pairs in each dataset and finding their Pearson and Spearman correlation values between generated score and the expert ratings. The cosine similarity scores for each entity was generated with respect to the COVID19 embedding vector obtained from our proposed pipeline. As can be seen from Table 3, TransD outperformed RotatE on both datasets and hence is the selected graph embedding technique in our case.

Using ERLKG we are able to rapidly mine important biomedical entities that are highly associated with COVID-19. The final result of chemical entities are given in Fig. 7. Entities color coded “Blue” signify a higher cosine similarity value compared to entities color coded “Yellow”. We can see entities like “Flutamide”, “Lopinavir”, “Carfilzomib”, etc. are highly associated with COVID-19. On further investigation we found out that most of the entities are actually being considered for COVID-19 treatment.

For further results we highly encourage the readers to check out the full paper.