{"cells":[{"metadata":{"trusted":true},"cell_type":"code","source":"import os\nimport json\nimport pandas as pd\nimport numpy as np\nimport nltk\nnltk.download('stopwords')\nfrom nltk.corpus import stopwords\nstop_words = stopwords.words('english')\nimport torch\nfrom transformers import BertTokenizer, BertModel\nfrom nltk import word_tokenize ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def removeStopwords(x):\n    x = word_tokenize(x);\n    x = [i for i in x if i not in stop_words]\n    return ' '.join(x)   ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def preprocess(data):\n    data['selected_features'] = data['selected_features'].apply(lambda x: \" \".join(x.lower() for x in x.split()))\n    data['selected_features'] = data['selected_features'].str.replace(r\"\\d+\", \"\")\n    data['selected_features'] = data['selected_features'].str.replace('[^\\w\\s]','')\n    data['selected_features'] = data['selected_features'].str.replace(r\"[︰-＠]\", \"\")\n    data['selected_features'] = data['selected_features'].apply(lambda x: removeStopwords(x))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The function get_word_encodings uses pretrained BERT model SciBERT which is trained on scientific data including PMC data. It gives out word embedding vectors for the input data X with output length 786. In interest of time and resources, the SciBERT has not been fine-tuned. But it is possible to fine tune it for the corpus formed from our dataset. Also, due to resource limitation, the encodings are processed for each record separately in a loop. However, with more memory, batches could be used to do the same."},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_word_encodings(x):\n    features = []\n    model_version = 'allenai/scibert_scivocab_uncased'\n    do_lower_case = True\n    model = BertModel.from_pretrained(model_version)\n    tokenizer = BertTokenizer.from_pretrained(model_version, do_lower_case=do_lower_case)\n    for i in x:\n        inputs = tokenizer(i, return_tensors = \"pt\", padding = True)\n        doc_embeddings = model(**inputs)\n        feature  = doc_embeddings[0][:,0,:].cpu().detach().numpy().squeeze()\n        features.append(feature)\n    return np.array(features)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The following cell loads data from metadata.csv. In the interset of time and resources, the title and abstract of the files is only considered. The text in the data files can also be appended to these two features for future experiments. Here, in doc_embeddings, we store the encodings of 100 records. Due to processing time, complete data set has not been used as of now."},{"metadata":{"trusted":true},"cell_type":"code","source":"df_docs = pd.read_csv( '/kaggle/input/trec-covid-information-retrieval/CORD-19/CORD-19/metadata.csv')\ndf_docs['title'] = df_docs['title'].fillna('');\ndf_docs['abstract'] = df_docs['abstract'].fillna('')\ndf_title_abstract = pd.DataFrame(df_docs['title'] + \" \" + df_docs['abstract'], columns = ['selected_features'])\npreprocess(df_title_abstract)\nx = df_title_abstract['selected_features'].to_list()\ndoc_embeddings = get_word_encodings(x)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The following cell stores the embeddings of query + narrative for all topics. Experimentally, question can also be concatenated for the same."},{"metadata":{"trusted":true},"cell_type":"code","source":"df_query = pd.read_csv( '/kaggle/input/trec-covid-information-retrieval/topics-rnd3.csv')\ndf_query_narrative = pd.DataFrame(df_query['query']+\" \" + df_query['narrative'], columns = ['selected_features']);\npreprocess(df_query_narrative)\nx = df_query_narrative['selected_features'].to_list()\nquery_embedding = get_word_encodings(x)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Finally, dot product is used to calculate cosine similarity since the encoded vectors lie in similar range. Index of data frame is the document id and column names are topic ids."},{"metadata":{"trusted":true},"cell_type":"code","source":"similarity_matrix = np.dot(doc_embeddings, np.transpose(query_embedding))\nsimilarity_df = pd.DataFrame(similarity_matrix,index = df_docs['cord_uid'].values, columns=df_query['topic-id'].values)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The similairy score for each topic is sorted in descending order and top 10 records are stored with doc ids and topic ids in submission.csv"},{"metadata":{"trusted":true},"cell_type":"code","source":"information_retrieved_df = pd.DataFrame(columns=['topic_id','cord_id'])\nrow= 0;\nfor column in similarity_df.columns:\n    for cordID in similarity_df.sort_values(column, ascending=False).head(1000).index:\n        a = {}\n        a['topic-id'] = column\n        a['cord-id'] = cordID\n        information_retrieved_df.loc[row, :] = dict(topic_id=column, cord_id=cordID)\n        row+=1\ninformation_retrieved_df.to_csv('submission.csv',index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}