{"cells":[{"metadata":{},"cell_type":"markdown","source":"# TREC-COVID Search Index\n\nThis notebook builds a search index over the TREC-COVID dataset. Background on how this search index works can be found in the [CORD-19 Analysis with Sentence Embeddings](https://www.kaggle.com/davidmezzetti/cord-19-analysis-with-sentence-embeddings) notebook.\n\n## Install environment\nInstall the cord19q library and scispacy.","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"# Install cord19q project\n!pip install git+https://github.com/neuml/cord19q\n\n# Install scispacy model\n!pip install https://s3-us-west-2.amazonaws.com/ai2-s2-scispacy/releases/v0.2.4/en_core_sci_md-0.2.4.tar.gz","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Build articles database and embeddings index\n\nThe following section loads the necessary backing datasets, builds a SQLite database and an embeddings index to support searching. When complete, the output directory will have an cord19q directory with the search index.","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import os\nimport shutil\n\nfrom cord19q.etl.execute import Execute as Etl\nfrom cord19q.index import Index\n\n# Copy study design models locally\nos.mkdir(\"cord19q\")\nshutil.copy(\"../input/cord19-study-design/attribute\", \"cord19q\")\nshutil.copy(\"../input/cord19-study-design/design\", \"cord19q\")\n\n# Build SQLite database for metadata.csv and json full text files\nEtl.run(\"../input/trec-covid-information-retrieval/CORD-19/CORD-19\", \"cord19q\", \"../input/cord-19-article-entry-dates/entry-dates.csv\", False)\n\n# Copy vectors locally for predictable performance\nshutil.copy(\"../input/cord19-fasttext-vectors/cord19-300d.magnitude\", \"/tmp\")\n\n# Build the embeddings index\nIndex.run(\"cord19q\", \"/tmp/cord19-300d.magnitude\")\n","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}