{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-12-26T21:01:02.661805Z","iopub.execute_input":"2022-12-26T21:01:02.663544Z","iopub.status.idle":"2022-12-26T21:01:02.683758Z","shell.execute_reply.started":"2022-12-26T21:01:02.663449Z","shell.execute_reply":"2022-12-26T21:01:02.682945Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from matplotlib.colors import ListedColormap, LinearSegmentedColormap\nfrom matplotlib.offsetbox import AnnotationBbox, OffsetImage\nfrom sklearn.decomposition import TruncatedSVD\nfrom matplotlib.patches import Rectangle\nimport matplotlib.patches as patches\nimport matplotlib.pyplot as plt\nfrom tqdm import tqdm\nimport seaborn as sns\nimport pandas as pd\nimport numpy as np\nimport itertools\nimport warnings\nimport os\nimport gc\n!pip install scanpy\nimport scanpy as sc\nimport anndata\nimport time\nt0start = time.time()\nimport sys","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In this notebook I would like to further investigate defining cellular population dynamics at single-cell resolution during hematopoiesis.\nDuring the competion we were looking at followng cell types over a certain time period and across a number of different donors:\n\nMasP = Mast Cell Progenitor\nMkP = Megakaryocyte Progenitor\nNeuP = Neutrophil Progenitor\nMoP = Monocyte Progenitor\nEryP = Erythrocyte Progenitor\nHSC = Hematoploetic Stem Cell\nBP = B-Cell Progenitor\n\nFrom the competition: Following the central dogma of molecular biology: DNA --> RNA-->Protein, your task is as follows:\n\nFor the Multiome samples: given chromatin accessibility, predict gene expression.\nFor the CITEseq samples: given gene expression, predict protein levels.\n\nGene expression will not always correlate with protein expression. Ofcourse, you will need some levels of expression from the Gene to generate protein, however if the gene is being expressed you will not necessarily get protein due to post-transcriptional and post-translational modifications. \n\nCell types can be defined by both gene and protein expression. I looked at the paper provided: https://www.nature.com/articles/ncb3493\nand Figure 3A defines genes associated with the HSC and progenitor populations:\n\n\n\n\n","metadata":{}},{"cell_type":"code","source":"import pandas as pd\ndfa = pd.read_csv('../input/dfa123/dfa.csv')","metadata":{"execution":{"iopub.status.busy":"2022-12-26T21:28:42.187713Z","iopub.execute_input":"2022-12-26T21:28:42.189457Z","iopub.status.idle":"2022-12-26T21:28:42.248592Z","shell.execute_reply.started":"2022-12-26T21:28:42.189393Z","shell.execute_reply":"2022-12-26T21:28:42.246801Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The table below shows genes associated with each cell type (dfa). If we had the 10X filtered feature barcode matrix for each sample we would be able to determine the clusters representing each cell type using this gene information using the tools in the anaysis package Scanpy https://scanpy.readthedocs.io/en/stable/tutorials.html#clustering. \nIn the protein marker list for the CITE-seq (dfb) we do not see all these markers. ITGA2B gene corresponds to the CD41 protein which is present in the CITE_seq protein marker list(dfb), however the majority of other proteins for the corresponding cell type association gene(s) are not there. \nITGA2B(gene) and CD41(protein) in the MKp population may show stronger gene-protein association due to the fact they are a marker for the MKp population.","metadata":{}},{"cell_type":"code","source":"print(dfa)","metadata":{"execution":{"iopub.status.busy":"2022-12-26T21:29:01.586323Z","iopub.execute_input":"2022-12-26T21:29:01.586804Z","iopub.status.idle":"2022-12-26T21:29:01.627221Z","shell.execute_reply.started":"2022-12-26T21:29:01.586768Z","shell.execute_reply":"2022-12-26T21:29:01.625917Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dfb = pd.read_csv('../input/dfb123/dfb.csv')","metadata":{"execution":{"iopub.status.busy":"2022-12-26T21:46:35.269378Z","iopub.execute_input":"2022-12-26T21:46:35.271413Z","iopub.status.idle":"2022-12-26T21:46:35.290655Z","shell.execute_reply.started":"2022-12-26T21:46:35.271358Z","shell.execute_reply":"2022-12-26T21:46:35.289405Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(dfb)","metadata":{"execution":{"iopub.status.busy":"2022-12-26T21:46:41.637041Z","iopub.execute_input":"2022-12-26T21:46:41.637495Z","iopub.status.idle":"2022-12-26T21:46:41.655899Z","shell.execute_reply.started":"2022-12-26T21:46:41.637459Z","shell.execute_reply":"2022-12-26T21:46:41.654322Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Test Hypothesis: CD41 in the MKp population may show stronger gene-protein association due to the fact that CD41 is a marker for the MKp population.","metadata":{}},{"cell_type":"code","source":"\nMAIN_DIRECT = '../input/open-problems-multimodal'\nmeta_data_path = os.path.join(MAIN_DIRECT + '/metadata.csv')\neval_data_path  = os.path.join(MAIN_DIRECT + '/evaluation_ids.csv')\nsubmission_path = os.path.join(MAIN_DIRECT + '/sample_submission.csv')\n\ntest_cite_path = os.path.join(MAIN_DIRECT + '/test_cite_inputs.h5')\ntest_cite_path_day2 = os.path.join(MAIN_DIRECT + '/test_cite_inputs_day_2_donor_27678.h5')\ntest_multi_path = os.path.join(MAIN_DIRECT + '/test_multi_inputs.h5')\n\n\ntrain_cite_path = os.path.join(MAIN_DIRECT + '/train_cite_inputs.h5')\ntrain_cite_target_path = os.path.join(MAIN_DIRECT + '/train_cite_targets.h5')\ntrain_multi_path = os.path.join(MAIN_DIRECT + '/train_multi_inputs.h5')\ntrain_multi_target_path = os.path.join(MAIN_DIRECT + '/train_multi_targets.h5')","metadata":{"execution":{"iopub.status.busy":"2022-12-26T22:04:37.924254Z","iopub.execute_input":"2022-12-26T22:04:37.925964Z","iopub.status.idle":"2022-12-26T22:04:37.934683Z","shell.execute_reply.started":"2022-12-26T22:04:37.925897Z","shell.execute_reply":"2022-12-26T22:04:37.933601Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"meta_data = pd.read_csv(meta_data_path)\ndisplay(meta_data,'Meta Data')","metadata":{"execution":{"iopub.status.busy":"2022-12-26T22:04:42.482290Z","iopub.execute_input":"2022-12-26T22:04:42.482803Z","iopub.status.idle":"2022-12-26T22:04:42.988799Z","shell.execute_reply.started":"2022-12-26T22:04:42.482765Z","shell.execute_reply":"2022-12-26T22:04:42.987023Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"citeseq_data = meta_data[meta_data['technology'] == 'citeseq']\ndisplay(citeseq_data,'CITEseq Data')","metadata":{"execution":{"iopub.status.busy":"2022-12-26T22:04:47.524643Z","iopub.execute_input":"2022-12-26T22:04:47.525949Z","iopub.status.idle":"2022-12-26T22:04:47.594305Z","shell.execute_reply.started":"2022-12-26T22:04:47.525884Z","shell.execute_reply":"2022-12-26T22:04:47.592591Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"MkP_data = citeseq_data[citeseq_data['cell_type'] == 'MkP']\ndisplay(MkP_data,'MkP Data')","metadata":{"execution":{"iopub.status.busy":"2022-12-26T22:06:43.924004Z","iopub.execute_input":"2022-12-26T22:06:43.924569Z","iopub.status.idle":"2022-12-26T22:06:43.959161Z","shell.execute_reply.started":"2022-12-26T22:06:43.924528Z","shell.execute_reply":"2022-12-26T22:06:43.957695Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_hdf_cite = pd.read_hdf(train_cite_path)","metadata":{"execution":{"iopub.status.busy":"2022-12-26T22:08:53.764744Z","iopub.execute_input":"2022-12-26T22:08:53.765262Z","iopub.status.idle":"2022-12-26T22:09:47.354603Z","shell.execute_reply.started":"2022-12-26T22:08:53.765225Z","shell.execute_reply":"2022-12-26T22:09:47.353275Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_hdf_cite.info(memory_usage='deep')","metadata":{"execution":{"iopub.status.busy":"2022-12-26T22:09:47.356557Z","iopub.execute_input":"2022-12-26T22:09:47.356891Z","iopub.status.idle":"2022-12-26T22:09:49.359187Z","shell.execute_reply.started":"2022-12-26T22:09:47.356860Z","shell.execute_reply":"2022-12-26T22:09:49.357992Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_hdf_cite_target = pd.read_hdf(train_cite_target_path)\ndisplay(train_hdf_cite_target,'Train HDF Target Data')","metadata":{"execution":{"iopub.status.busy":"2022-12-26T22:09:49.361153Z","iopub.execute_input":"2022-12-26T22:09:49.361606Z","iopub.status.idle":"2022-12-26T22:09:50.112430Z","shell.execute_reply.started":"2022-12-26T22:09:49.361560Z","shell.execute_reply":"2022-12-26T22:09:50.111007Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df2 = pd.merge(train_hdf_cite_target, MkP_data, on=[\"cell_id\"])","metadata":{"execution":{"iopub.status.busy":"2022-12-26T22:09:58.283050Z","iopub.execute_input":"2022-12-26T22:09:58.283970Z","iopub.status.idle":"2022-12-26T22:09:58.331688Z","shell.execute_reply.started":"2022-12-26T22:09:58.283898Z","shell.execute_reply":"2022-12-26T22:09:58.330549Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(df2)","metadata":{"execution":{"iopub.status.busy":"2022-12-26T22:10:07.636041Z","iopub.execute_input":"2022-12-26T22:10:07.636621Z","iopub.status.idle":"2022-12-26T22:10:07.662603Z","shell.execute_reply.started":"2022-12-26T22:10:07.636567Z","shell.execute_reply":"2022-12-26T22:10:07.661154Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas as pd\ndf6 = pd.merge(train_hdf_cite, train_hdf_cite_target, on='cell_id')","metadata":{"execution":{"iopub.status.busy":"2022-12-26T22:12:31.996658Z","iopub.execute_input":"2022-12-26T22:12:31.998366Z","iopub.status.idle":"2022-12-26T22:12:39.068431Z","shell.execute_reply.started":"2022-12-26T22:12:31.998280Z","shell.execute_reply":"2022-12-26T22:12:39.067171Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(df6)","metadata":{"execution":{"iopub.status.busy":"2022-12-26T22:12:45.300347Z","iopub.execute_input":"2022-12-26T22:12:45.300810Z","iopub.status.idle":"2022-12-26T22:12:45.335666Z","shell.execute_reply.started":"2022-12-26T22:12:45.300773Z","shell.execute_reply":"2022-12-26T22:12:45.334549Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"ENSG00000108798_ABI3 = CD41","metadata":{}},{"cell_type":"code","source":"df7 = df6[['ENSG00000108798_ABI3']]","metadata":{"execution":{"iopub.status.busy":"2022-12-26T22:17:52.063006Z","iopub.execute_input":"2022-12-26T22:17:52.063425Z","iopub.status.idle":"2022-12-26T22:18:00.881873Z","shell.execute_reply.started":"2022-12-26T22:17:52.063393Z","shell.execute_reply":"2022-12-26T22:18:00.880533Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(df7)","metadata":{"execution":{"iopub.status.busy":"2022-12-26T22:18:10.661759Z","iopub.execute_input":"2022-12-26T22:18:10.662227Z","iopub.status.idle":"2022-12-26T22:18:10.671406Z","shell.execute_reply.started":"2022-12-26T22:18:10.662185Z","shell.execute_reply":"2022-12-26T22:18:10.670271Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df8 = pd.merge(df7, df2, on=[\"cell_id\"])","metadata":{"execution":{"iopub.status.busy":"2022-12-26T22:19:11.598100Z","iopub.execute_input":"2022-12-26T22:19:11.599606Z","iopub.status.idle":"2022-12-26T22:19:11.649696Z","shell.execute_reply.started":"2022-12-26T22:19:11.599551Z","shell.execute_reply":"2022-12-26T22:19:11.648589Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(df8)","metadata":{"execution":{"iopub.status.busy":"2022-12-26T22:19:22.855282Z","iopub.execute_input":"2022-12-26T22:19:22.855677Z","iopub.status.idle":"2022-12-26T22:19:22.876644Z","shell.execute_reply.started":"2022-12-26T22:19:22.855645Z","shell.execute_reply":"2022-12-26T22:19:22.875527Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df10 = df8[['ENSG00000108798_ABI3', 'CD41']]","metadata":{"execution":{"iopub.status.busy":"2022-12-26T22:20:18.420865Z","iopub.execute_input":"2022-12-26T22:20:18.421425Z","iopub.status.idle":"2022-12-26T22:20:18.434654Z","shell.execute_reply.started":"2022-12-26T22:20:18.421380Z","shell.execute_reply":"2022-12-26T22:20:18.433129Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(df10)","metadata":{"execution":{"iopub.status.busy":"2022-12-26T22:20:26.473798Z","iopub.execute_input":"2022-12-26T22:20:26.474228Z","iopub.status.idle":"2022-12-26T22:20:26.484678Z","shell.execute_reply.started":"2022-12-26T22:20:26.474193Z","shell.execute_reply":"2022-12-26T22:20:26.483364Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport matplotlib as mpl\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n","metadata":{"execution":{"iopub.status.busy":"2022-12-26T22:32:03.954279Z","iopub.execute_input":"2022-12-26T22:32:03.954765Z","iopub.status.idle":"2022-12-26T22:32:03.960789Z","shell.execute_reply.started":"2022-12-26T22:32:03.954726Z","shell.execute_reply":"2022-12-26T22:32:03.959358Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.regplot(x=\"ENSG00000108798_ABI3\", y=\"CD41\", data=df10);","metadata":{"execution":{"iopub.status.busy":"2022-12-26T22:32:06.153618Z","iopub.execute_input":"2022-12-26T22:32:06.154064Z","iopub.status.idle":"2022-12-26T22:32:06.846936Z","shell.execute_reply.started":"2022-12-26T22:32:06.154031Z","shell.execute_reply":"2022-12-26T22:32:06.846134Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Not a strong positive correlation between CD41 gene and protein for majority of the MkP population. a small population (8 cells) does show dramatic increase in CD41 protein as CD41 gene increases from 3-4.\npaper(https://www.ahajournals.org/doi/10.1161/ATVBAHA.119.312129) Megakaryocyte-like HSCs defined a subset of HSCs expressing the megakaryocyte-specific integrin CD41.CD41-expressing HSCs have a myeloid-bias, increase with age, and are associated with the megakaryocyte-specific transcription factor, GATA1.","metadata":{}}]}