{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Open Problems - Multimodal Single-Cell Integration - 📊 EDA","metadata":{}},{"cell_type":"markdown","source":"#### I am a Data Science practitioner that is very passionate and motivated. In this notebook, I will focus on analyzing the data pertaining to the Open Problems - Multimodal Single-Cell Integration my progress step by step until I present my own conclusion for this problem.\n\n#### If you found this Kernel insightful, please consider upvoting it 👍\n\n#### I am all ears if you have any advice :)","metadata":{}},{"cell_type":"markdown","source":"\"Titanic\"","metadata":{}},{"cell_type":"markdown","source":"
\n\n

\n THE NOTEBOOK CONTAINS THE FOLLOWING SECTIONS:\n

\n \n
","metadata":{}},{"cell_type":"markdown","source":"
\n\n

\n Problem Description\n

\n
","metadata":{}},{"cell_type":"markdown","source":"
\n\n

\nObjective ⛳\n

\n
","metadata":{}},{"cell_type":"markdown","source":"The Open Problems - Multimodal Single-Cell Integration Competition presents goals such as predicting how DNA, RNA, and protein measurements co-vary in single cells as bone marrow stem cells develop into more mature blood cells.\n\nIn this competition, participants must seek to develop a model trained on a subset of 300,000-cell time course dataset of CD34+ hematopoietic stem and progenitor cells (HSPC) from four human donors at five time points generated for this competition specifically. One of the main challenges is taht the test set contains data from a much later time compared to the training data\n\n","metadata":{}},{"cell_type":"markdown","source":"\"Titanic\"","metadata":{}},{"cell_type":"markdown","source":"#### Context \nIn the past decade, the advent of single-cell genomics has enabled the measurement of DNA, RNA, and proteins in single cells. These technologies allow the study of biology at an unprecedented scale and resolution. Among the outcomes have been detailed maps of early human embryonic development, the discovery of new disease-associated cell types, and cell-targeted therapeutic interventions. Moreover, with recent advances in experimental techniques it is now possible to measure multiple genomic modalities in the same cell.\n\nWhile multimodal single-cell data is increasingly available, data analysis methods are still scarce. Due to the small volume of a single cell, measurements are sparse and noisy. Differences in molecular sampling depths between cells (sequencing depth) and technical effects from handling cells in batches (batch effects) can often overwhelm biological differences. When analyzing multimodal data, one must account for different feature spaces, as well as shared and unique variation between modalities and between batches. Furthermore, current pipelines for single-cell data analysis treat cells as static snapshots, even when there is an underlying dynamical biological process. Accounting for temporal dynamics alongside state changes over time is an open challenge in single-cell data science.\n\nGenerally, genetic information flows from DNA to RNA to proteins. DNA must be accessible (ATAC data) to produce RNA (GEX data), and RNA in turn is used as a template to produce protein (ADT data). These processes are regulated by feedback: for example, a protein may bind DNA to prevent the production of more RNA. This genetic regulation is the foundation for dynamic cellular processes that allow organisms to develop and adapt to changing environments. In single-cell data science, dynamic processes have been modeled by so-called pseudotime algorithms that capture the progression of the biological process. Yet, generalizing these algorithms to account for both pseudotime and real time is still an open problem.\n\nCompetition host Open Problems in Single-Cell Analysis is an open-source, community-driven effort to standardize benchmarking of single-cell methods. The core efforts of Open Problems include the formalization of existing challenges into measurable tasks, a collection of high-quality datasets, centralized benchmarking of community-contributed methods, and community-focused events that bring together diverse method developers to improve single-cell algorithms. They're excited to be partnering with Cellarity, Chan Zuckerbeg Biohub, the Chan Zuckerberg Initiative, Helmholtz Munich, and Yale to see what progress can be made in predicting changes in genetic dynamics over time through interdisciplinary collaboration.\n\nThere are approximately 37 trillion cells in the human body, all with different behaviors and functions. Understanding how a single genome gives rise to a diversity of cellular states is the key to gaining mechanistic insight into how tissues function or malfunction in health and disease. You can help solve this fundamental challenge for single-cell biology. Being able to solve the prediction problems over time may yield new insights into how gene regulation influences differentiation as blood and immune cells mature.","metadata":{}},{"cell_type":"markdown","source":"
\n\n

\nAbout the given Datasets 📁📂\n

\n
","metadata":{}},{"cell_type":"markdown","source":"#### Cell Types\n* MasP = Mast Cell Progenitor\n* MkP = Megakaryocyte Progenitor\n* NeuP = Neutrophil Progenitor\n* MoP = Monocyte Progenitor\n* EryP = Erythrocyte Progenitor\n* HSC = Hematoploetic Stem Cell\n* BP = B-Cell Progenitor","metadata":{}},{"cell_type":"markdown","source":"#### metadata.csv\n* cell_id - A unique identifier for each observed cell.\n* donor - An identifier for the four cell donors.\n* day - The day of the experiment the observation was made.\n* technology - Either citeseq or multiome.\n* cell_type - One of the above cell types or else hidden.","metadata":{}},{"cell_type":"markdown","source":"##### The experimental observations are located in several large arrays and are provided in HDF5 format.\n#### HDF5 Arrays:\n* Multiome\n* CITEseq\n* Splits\n##### We will talk about the content of these Arrays in a later section.\n\n","metadata":{}},{"cell_type":"markdown","source":"
\n\n

\nEvaluation Metrics Used 📏\n

\n
","metadata":{}},{"cell_type":"markdown","source":"### Based on the problem, we will use the Pearson Correlation Coefficient metric (PCC).\n\n#### PCC is a measure of linear correlation between two sets of data and gives an with range: [a, b]\n\n
\n
\n\"Titanic\"\n\n$ PCC = \\frac{Cov(x, y}{\\sigma_x \\sigma_y}$\n\n#### If you want to learn more about PCC: \nhttps://www.analyticsvidhya.com/blog/2021/01/beginners-guide-to-pearsons-correlation-coefficient/\n","metadata":{}},{"cell_type":"markdown","source":"$$f(X,n) = X_n + X_{n-1}$$","metadata":{}},{"cell_type":"markdown","source":"
\n\n

\n Work 🔨\n

\n
","metadata":{}},{"cell_type":"markdown","source":"
\n\n

\nImporting relevant libraries 📚📚📚\n

\n
","metadata":{}},{"cell_type":"code","source":"! pip install -q tables # Necessary to load hdf5 files","metadata":{"execution":{"iopub.status.busy":"2022-08-28T00:25:50.703138Z","iopub.execute_input":"2022-08-28T00:25:50.703624Z","iopub.status.idle":"2022-08-28T00:26:05.956897Z","shell.execute_reply.started":"2022-08-28T00:25:50.703521Z","shell.execute_reply":"2022-08-28T00:26:05.954859Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport os\nfrom termcolor import colored\nimport warnings\nimport h5py\nwarnings.simplefilter('ignore')","metadata":{"execution":{"iopub.status.busy":"2022-08-28T00:26:05.960201Z","iopub.execute_input":"2022-08-28T00:26:05.960685Z","iopub.status.idle":"2022-08-28T00:26:06.831173Z","shell.execute_reply.started":"2022-08-28T00:26:05.960632Z","shell.execute_reply":"2022-08-28T00:26:06.829835Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n

\nReading Data 📖\n

\n
","metadata":{}},{"cell_type":"markdown","source":"#### Due to memory constaints, some of the data will not be evaluated.","metadata":{}},{"cell_type":"code","source":"metadata = pd.read_csv('../input/open-problems-multimodal/metadata.csv').set_index('cell_id')\nevaluation_ids = pd.read_csv('../input/open-problems-multimodal/evaluation_ids.csv').set_index('row_id')","metadata":{"execution":{"iopub.status.busy":"2022-08-28T00:26:06.832862Z","iopub.execute_input":"2022-08-28T00:26:06.833360Z","iopub.status.idle":"2022-08-28T00:27:05.809320Z","shell.execute_reply.started":"2022-08-28T00:26:06.833315Z","shell.execute_reply":"2022-08-28T00:27:05.808027Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# test_cite_inputs = pd.read_hdf('../input/open-problems-multimodal/test_cite_inputs.h5', nrows=400).astype(np.float8)\n# train_cite_inputs = pd.read_hdf('../input/open-problems-multimodal/train_cite_inputs.h5', nrows=400).astype(np.float8)\n# #test_multi_inputs = pd.read_hdf('../input/open-problems-multimodal/test_multi_inputs.h5', nrows=800).astype(np.float16)\n# train_multi_inputs = pd.read_hdf('../input/open-problems-multimodal/train_multi_inputs.h5', nrows=100).astype(np.float16)\ntrain_cite_targets = pd.read_hdf('../input/open-problems-multimodal/train_cite_targets.h5', nrows=800).astype(np.float16)\n# train_multi_targets = pd.read_hdf('../input/open-problems-multimodal/train_multi_targets.h5', nrows=800).astype(np.float16)","metadata":{"execution":{"iopub.status.busy":"2022-08-28T00:42:35.200480Z","iopub.execute_input":"2022-08-28T00:42:35.201377Z","iopub.status.idle":"2022-08-28T00:42:36.101622Z","shell.execute_reply.started":"2022-08-28T00:42:35.201326Z","shell.execute_reply":"2022-08-28T00:42:36.100554Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n
\n \n
\n
\n\n

\nMetadata 📁\n

\n
\n
","metadata":{}},{"cell_type":"code","source":"metadata.head()","metadata":{"execution":{"iopub.status.busy":"2022-08-28T00:27:05.818690Z","iopub.execute_input":"2022-08-28T00:27:05.819040Z","iopub.status.idle":"2022-08-28T00:27:05.844700Z","shell.execute_reply.started":"2022-08-28T00:27:05.819008Z","shell.execute_reply":"2022-08-28T00:27:05.843504Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n
\n \n
\n
\n\n

\nEvaluation ID's\n

\n
\n
","metadata":{}},{"cell_type":"code","source":"evaluation_ids.head()","metadata":{"execution":{"iopub.status.busy":"2022-08-28T00:27:05.846268Z","iopub.execute_input":"2022-08-28T00:27:05.847029Z","iopub.status.idle":"2022-08-28T00:27:05.857710Z","shell.execute_reply.started":"2022-08-28T00:27:05.846984Z","shell.execute_reply":"2022-08-28T00:27:05.856836Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## For now we will only cover Metadata and Evaluation ID's","metadata":{}},{"cell_type":"code","source":"#train_multi_targets.head()","metadata":{"execution":{"iopub.status.busy":"2022-08-25T01:10:57.530499Z","iopub.execute_input":"2022-08-25T01:10:57.530967Z","iopub.status.idle":"2022-08-25T01:10:57.539582Z","shell.execute_reply.started":"2022-08-25T01:10:57.530941Z","shell.execute_reply":"2022-08-25T01:10:57.538859Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n

\nExploratory Data Analysis and Visualization 🔍 👀\n

\n
","metadata":{}},{"cell_type":"markdown","source":"
\n\n

\nView Data Types of Predictors and Target Variables\n

\n
","metadata":{}},{"cell_type":"markdown","source":"#### Metadata","metadata":{}},{"cell_type":"code","source":"print(colored(f'DATA TYPES:', 'cyan', attrs=['bold', 'underline']))\nmetadata.dtypes","metadata":{"execution":{"iopub.status.busy":"2022-08-28T00:27:05.859268Z","iopub.execute_input":"2022-08-28T00:27:05.859916Z","iopub.status.idle":"2022-08-28T00:27:05.874848Z","shell.execute_reply.started":"2022-08-28T00:27:05.859883Z","shell.execute_reply":"2022-08-28T00:27:05.873907Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Evaluation ID's","metadata":{}},{"cell_type":"code","source":"print(colored(f'DATA TYPES:', 'cyan', attrs=['bold', 'underline']))\nevaluation_ids.dtypes","metadata":{"execution":{"iopub.status.busy":"2022-08-28T00:27:05.876295Z","iopub.execute_input":"2022-08-28T00:27:05.877209Z","iopub.status.idle":"2022-08-28T00:27:05.889384Z","shell.execute_reply.started":"2022-08-28T00:27:05.877169Z","shell.execute_reply":"2022-08-28T00:27:05.888168Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n

\nAnalysis of Missing Values ⚠️\n

\n
","metadata":{}},{"cell_type":"code","source":"nan_cols_metadata = metadata.columns[metadata.isna().any()].tolist()\nnan_cols_evaluation_ids = evaluation_ids.columns[evaluation_ids.isna().any()].tolist()\nprint(nan_cols_metadata, nan_cols_evaluation_ids)","metadata":{"execution":{"iopub.status.busy":"2022-08-28T00:27:05.890779Z","iopub.execute_input":"2022-08-28T00:27:05.891150Z","iopub.status.idle":"2022-08-28T00:27:11.694509Z","shell.execute_reply.started":"2022-08-28T00:27:05.891117Z","shell.execute_reply":"2022-08-28T00:27:11.693062Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n

\nConclusions 📍\n

\n
\n\n* Metadata and Evaluation ID's don't have any missing values","metadata":{}},{"cell_type":"markdown","source":"
\n\n

\n Data Analysis of Individual Predictors 🔮\n

\n
","metadata":{}},{"cell_type":"markdown","source":"#### I used pie-charts because of inspiration from this notebook: https://www.kaggle.com/code/jirkaborovec/mmscel-inst-eda-stat-predictions","metadata":{}},{"cell_type":"markdown","source":"\n
\n
\n \n
\n
\n\n

\nDay\n

\n
\n
","metadata":{}},{"cell_type":"code","source":"fig,ax = plt.subplots(nrows=1, ncols=2, figsize=(12, 5))\nsns.countplot(x='day', data=metadata, palette='mako', ax=ax[0])\nmetadata[['day']].value_counts().plot.pie(autopct='%1.1f%%', ylabel='day', ax=ax[1])\nplt.show()\nprint(colored(f'Day counts:', 'cyan', attrs=['bold', 'underline']))\nmetadata.day.value_counts().sort_index()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-08-28T00:39:26.120516Z","iopub.execute_input":"2022-08-28T00:39:26.120938Z","iopub.status.idle":"2022-08-28T00:39:26.454473Z","shell.execute_reply.started":"2022-08-28T00:39:26.120896Z","shell.execute_reply":"2022-08-28T00:39:26.453043Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n

\nConclusions 📍\n

\n
\n\n* Day 4 had the most donations","metadata":{}},{"cell_type":"markdown","source":"\n
\n
\n \n
\n
\n\n

\nDonor 🧑\n

\n
\n
","metadata":{}},{"cell_type":"code","source":"fig, ax = plt.subplots(nrows=1, ncols=2, figsize=(12, 5))\nsns.countplot(x='donor', data=metadata, palette='mako', ax=ax[0])\nmetadata[['donor']].value_counts().plot.pie(autopct='%1.1f%%', ylabel='donor', ax=ax[1])\nplt.show()\nprint(colored(f'Donor counts:', 'cyan', attrs=['bold', 'underline']))\nmetadata.donor.value_counts().sort_index()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-08-28T00:39:10.680105Z","iopub.execute_input":"2022-08-28T00:39:10.681177Z","iopub.status.idle":"2022-08-28T00:39:11.034273Z","shell.execute_reply.started":"2022-08-28T00:39:10.681128Z","shell.execute_reply":"2022-08-28T00:39:11.032994Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n

\nConclusions 📍\n

\n
\n\n* Donor 31800 donated the most cells","metadata":{}},{"cell_type":"markdown","source":"\n
\n
\n \n
\n
\n\n

\nCell Type 🧫\n

\n
\n
","metadata":{}},{"cell_type":"code","source":"fig, ax = plt.subplots(nrows=1, ncols=2, figsize=(12, 5))\nsns.countplot(x='cell_type', data=metadata, palette='mako', ax=ax[0])\nmetadata[['cell_type']].value_counts().plot.pie(autopct='%1.1f%%', ylabel='cell_type', ax=ax[1])\nplt.show()\nprint(colored(f'Cell Type counts:', 'cyan', attrs=['bold', 'underline']))\nmetadata.cell_type.value_counts().sort_index()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-08-28T00:38:51.573781Z","iopub.execute_input":"2022-08-28T00:38:51.574570Z","iopub.status.idle":"2022-08-28T00:38:52.192475Z","shell.execute_reply.started":"2022-08-28T00:38:51.574528Z","shell.execute_reply":"2022-08-28T00:38:52.191026Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n

\nConclusions 📍\n

\n
\n\n* The most common cell types are HSC and hidden","metadata":{}},{"cell_type":"markdown","source":"\n
\n
\n \n
\n
\n\n

\nTechnology 👩‍💻\n

\n
\n
","metadata":{}},{"cell_type":"code","source":"fig, ax = plt.subplots(nrows=1, ncols=2, figsize=(12, 5))\nsns.countplot(x='technology', data=metadata, palette='mako', ax=ax[0])\nmetadata[['technology']].value_counts().plot.pie(autopct='%1.1f%%', ylabel='technology', ax=ax[1])\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-08-28T00:27:13.056267Z","iopub.execute_input":"2022-08-28T00:27:13.056825Z","iopub.status.idle":"2022-08-28T00:27:13.526844Z","shell.execute_reply.started":"2022-08-28T00:27:13.056793Z","shell.execute_reply":"2022-08-28T00:27:13.526021Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Lets have a look at citeseq and multiome and generate a plot based on teh donor","metadata":{}},{"cell_type":"code","source":"metadata_cite = metadata[metadata.technology=='citeseq']\nmetadata_multiome = metadata[metadata.technology=='multiome']\n_, ax = plt.subplots(1,2,figsize=(24,12))\nsns.countplot(x='day', hue='donor', data=metadata_cite, palette='mako', ax=ax[0])\nsns.countplot(x='day', hue='donor', data=metadata_multiome, palette='mako', ax=ax[1])\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2022-08-28T00:33:28.312622Z","iopub.execute_input":"2022-08-28T00:33:28.313074Z","iopub.status.idle":"2022-08-28T00:33:29.020311Z","shell.execute_reply.started":"2022-08-28T00:33:28.313036Z","shell.execute_reply":"2022-08-28T00:33:29.019176Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n

\nConclusions 📍\n

\n
\n\n* Most of the cells were sample using the Multiome kit\n* On day 4 donor 27678 did not donate using the multiome approach, whule the rest of the donors every single day used both technologies\n* For the CITEseq kit day 4 had the most samples, while for the Multiome kit day 3 seems to have the most samples","metadata":{}},{"cell_type":"markdown","source":"\n
\n
\n \n
\n
\n\n

\nMetadata + CITEseq KIT\n

\n
\n
","metadata":{}},{"cell_type":"code","source":"metadata = metadata.join(train_cite_targets, how='right')","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, axarr = plt.subplots(nrows=2, ncols=3)\nfor i, col in enumerate([\"day\", \"donor\", \"cell_type\"]):\n _= metadata[[col]].value_counts().plot.pie(ax=axarr[0][i], autopct='%1.1f%%', ylabel=col)\n sns.countplot(x=col, data=metadata, palette='mako', ax=axarr[1][i])","metadata":{"execution":{"iopub.status.busy":"2022-08-28T00:48:34.556044Z","iopub.execute_input":"2022-08-28T00:48:34.556818Z","iopub.status.idle":"2022-08-28T00:48:35.272257Z","shell.execute_reply.started":"2022-08-28T00:48:34.556771Z","shell.execute_reply":"2022-08-28T00:48:35.270978Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n

\n WORK IN PROGRESS ⚠️\n

\n \n
\n\n### Feel free to comment what you would like me to include in this Kernel. ","metadata":{}}]}