{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Heatmap (imshow) ATAC-seq (Multiome) data\n\nAll credits to NORIFUMI IRIE  https://www.kaggle.com/code/norifumiirie/heatmap-of-multiome-data\n\nJust use another ways to load data. \n\nWe look on imshow for ATAC-seq data (Multiome) . \n\nWhat we can see - non-zero regions does not change much from cell to cell. \n\n### Versions:\n\n### 6 suspect - horizontal stripes - correspond to \"days\" - so it is quite strong \"batch effect\"\n\n    Looked on the pattern of days and donors in metadata file - it seems it quite correspond to pattern of the strips in ATAC-seq. \n    In theory it can be cell types but cell types for progenitors are typically changing continously - but here we see quite sharp change. \n    \n    To chekc that more precisely we need to check the order of cells agrees in metadata file and in multiome files - but multiome are too large to load them into 16G Kaggle RAM - so we will resolve it later. \n\n#### 5 cosmetics changes\n\n#### 4 use plt.spy - heatmap for sparse matrices \n    Mysterious structure appears ! (Same as seen in NORIFUMI IRIE scripts - from where we take plt.spy)\n    (We cannot see that structure  using imshow for standard (non-sparse) matrices - that is strange)\n    \n    What can it be DAY ? cell type (hardly). \n    \n    Technicality: important to rescrict to not more than 5000 intervals - otherwise plot it fully filled - can see nothing.\n    also markersize=0.01 - important param for plt.spy\n    \n#### 3 cosmetics changes\n#### 2 Sparse matrices\n\n    Using decoding to sparse matrices done by SBUNZINI:\n    https://www.kaggle.com/datasets/sbunzini/open-problems-msci-multiome-sparse-matrices\n    We can work with full data not crashing RAM.\n    \n    Unlike the NORIFUMI IRIE https://www.kaggle.com/code/norifumiirie/heatmap-of-multiome-data\n    at the moment we do see clear horizontal stripes in ATAC-seq data - that is quite stange.\n    (Except clear difference in chromose Y - which is due to \"3 man and 1 woman\": https://www.kaggle.com/competitions/open-problems-multimodal/discussion/348761\n    \n    Technicality - we check that columns order in sparse matrix is the same as in the original matrix \n    \n    \n\n#### 1 first look \n    confirm NORIFUMI IRIE observations that non-zero regions often to do not change much.\n    We load directly non-sparse matrices - so can load only 4000 cells. \n    Create plots for most of the chromosomes \n    \n    \n","metadata":{"execution":{"iopub.status.busy":"2022-09-12T18:30:56.899011Z","iopub.execute_input":"2022-09-12T18:30:56.899526Z","iopub.status.idle":"2022-09-12T18:30:56.907709Z","shell.execute_reply.started":"2022-09-12T18:30:56.899490Z","shell.execute_reply":"2022-09-12T18:30:56.905939Z"}}},{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-09-13T13:44:55.591338Z","iopub.execute_input":"2022-09-13T13:44:55.591892Z","iopub.status.idle":"2022-09-13T13:44:55.630199Z","shell.execute_reply.started":"2022-09-13T13:44:55.591784Z","shell.execute_reply":"2022-09-13T13:44:55.628633Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport os","metadata":{"execution":{"iopub.status.busy":"2022-09-13T13:44:56.330538Z","iopub.execute_input":"2022-09-13T13:44:56.331001Z","iopub.status.idle":"2022-09-13T13:44:57.447615Z","shell.execute_reply.started":"2022-09-13T13:44:56.330961Z","shell.execute_reply":"2022-09-13T13:44:57.445973Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#If you see a urllib warning running this cell, go to \"Settings\" on the right hand side, \n#and turn on internet. Note, you need to be phone verified.\n!pip install --quiet tables","metadata":{"execution":{"iopub.status.busy":"2022-09-13T13:44:57.449805Z","iopub.execute_input":"2022-09-13T13:44:57.450710Z","iopub.status.idle":"2022-09-13T13:45:10.330040Z","shell.execute_reply.started":"2022-09-13T13:44:57.450656Z","shell.execute_reply":"2022-09-13T13:45:10.328438Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"DATA_DIR = \"/kaggle/input/open-problems-multimodal/\"\nFP_CELL_METADATA = os.path.join(DATA_DIR,\"metadata.csv\")\n\nFP_CITE_TRAIN_INPUTS = os.path.join(DATA_DIR,\"train_cite_inputs.h5\")\nFP_CITE_TRAIN_TARGETS = os.path.join(DATA_DIR,\"train_cite_targets.h5\")\nFP_CITE_TEST_INPUTS = os.path.join(DATA_DIR,\"test_cite_inputs.h5\")\n\nFP_MULTIOME_TRAIN_INPUTS = os.path.join(DATA_DIR,\"train_multi_inputs.h5\")\nFP_MULTIOME_TRAIN_TARGETS = os.path.join(DATA_DIR,\"train_multi_targets.h5\")\nFP_MULTIOME_TEST_INPUTS = os.path.join(DATA_DIR,\"test_multi_inputs.h5\")\n\nFP_SUBMISSION = os.path.join(DATA_DIR,\"sample_submission.csv\")\nFP_EVALUATION_IDS = os.path.join(DATA_DIR,\"evaluation_ids.csv\")","metadata":{"execution":{"iopub.status.busy":"2022-09-13T13:45:10.333874Z","iopub.execute_input":"2022-09-13T13:45:10.334418Z","iopub.status.idle":"2022-09-13T13:45:10.343466Z","shell.execute_reply.started":"2022-09-13T13:45:10.334361Z","shell.execute_reply":"2022-09-13T13:45:10.342151Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_cell = pd.read_csv(FP_CELL_METADATA)\ndf_cell","metadata":{"execution":{"iopub.status.busy":"2022-09-13T16:23:59.712889Z","iopub.execute_input":"2022-09-13T16:23:59.716789Z","iopub.status.idle":"2022-09-13T16:24:00.189355Z","shell.execute_reply.started":"2022-09-13T16:23:59.716584Z","shell.execute_reply":"2022-09-13T16:24:00.187623Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Look on order of days/donors in metadata files - seems quite correspond to strips in ATAC-seq \n\nneed to check futher. \n","metadata":{}},{"cell_type":"code","source":"d = df_cell[df_cell['technology'] == 'multiome']\nplt.plot(d['day'].values)\nplt.show()\nplt.plot(d['donor'].values)\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2022-09-13T16:27:49.730148Z","iopub.execute_input":"2022-09-13T16:27:49.730651Z","iopub.status.idle":"2022-09-13T16:27:50.205954Z","shell.execute_reply.started":"2022-09-13T16:27:49.730605Z","shell.execute_reply":"2022-09-13T16:27:50.204899Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Load ATAC-seq data in sparse data format. ","metadata":{}},{"cell_type":"code","source":"%%time\nimport scipy.sparse \nfn = '/kaggle/input/open-problems-msci-multiome-sparse-matrices/train_multiome_input_sparse.npz'\ntrain_inputs = scipy.sparse.load_npz(fn)# \"../input/multimodal-single-cell-as-sparse-matrix/train_multi_inputs_values.sparse.npz\")","metadata":{"execution":{"iopub.status.busy":"2022-09-13T13:45:10.345038Z","iopub.execute_input":"2022-09-13T13:45:10.345517Z","iopub.status.idle":"2022-09-13T13:46:10.803643Z","shell.execute_reply.started":"2022-09-13T13:45:10.345466Z","shell.execute_reply":"2022-09-13T13:46:10.802760Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('8G consumed 1min to load')","metadata":{"execution":{"iopub.status.busy":"2022-09-13T13:46:30.098996Z","iopub.execute_input":"2022-09-13T13:46:30.099476Z","iopub.status.idle":"2022-09-13T13:46:30.107088Z","shell.execute_reply.started":"2022-09-13T13:46:30.099427Z","shell.execute_reply":"2022-09-13T13:46:30.105483Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Sanity check - need to check that order of columns in sparse matrix is the same as in the original","metadata":{}},{"cell_type":"code","source":"train_inputs","metadata":{"execution":{"iopub.status.busy":"2022-09-13T13:46:34.059845Z","iopub.execute_input":"2022-09-13T13:46:34.060306Z","iopub.status.idle":"2022-09-13T13:46:34.071377Z","shell.execute_reply.started":"2022-09-13T13:46:34.060265Z","shell.execute_reply":"2022-09-13T13:46:34.070111Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_inputs[:1,-100:].toarray()","metadata":{"execution":{"iopub.status.busy":"2022-09-13T13:46:34.861231Z","iopub.execute_input":"2022-09-13T13:46:34.861688Z","iopub.status.idle":"2022-09-13T13:46:34.872872Z","shell.execute_reply.started":"2022-09-13T13:46:34.861648Z","shell.execute_reply":"2022-09-13T13:46:34.871761Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_inputs[1:2,-100:].toarray()","metadata":{"execution":{"iopub.status.busy":"2022-09-13T13:46:36.117165Z","iopub.execute_input":"2022-09-13T13:46:36.117632Z","iopub.status.idle":"2022-09-13T13:46:36.126744Z","shell.execute_reply.started":"2022-09-13T13:46:36.117591Z","shell.execute_reply":"2022-09-13T13:46:36.125791Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Load small chunk of train data (not sparse - not make sanity check and compare with sparse)","metadata":{}},{"cell_type":"code","source":"%%time\nSTART = int(0)\nSTOP = START+10# 4_000 # Bigger chunk will crash 16G memory at plt.imshow()\n\ndf_multi_train_x = pd.read_hdf(FP_MULTIOME_TRAIN_INPUTS,start=START,stop=STOP)\ndisplay( df_multi_train_x.head() )","metadata":{"execution":{"iopub.status.busy":"2022-09-13T13:46:37.754926Z","iopub.execute_input":"2022-09-13T13:46:37.756182Z","iopub.status.idle":"2022-09-13T13:46:38.643179Z","shell.execute_reply.started":"2022-09-13T13:46:37.756139Z","shell.execute_reply":"2022-09-13T13:46:38.641892Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_multi_train_x.sum(axis=1)","metadata":{"execution":{"iopub.status.busy":"2022-09-13T13:46:39.459774Z","iopub.execute_input":"2022-09-13T13:46:39.460232Z","iopub.status.idle":"2022-09-13T13:46:39.985074Z","shell.execute_reply.started":"2022-09-13T13:46:39.460189Z","shell.execute_reply":"2022-09-13T13:46:39.983703Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_inputs[:10,:].toarray().sum(axis = 1)","metadata":{"execution":{"iopub.status.busy":"2022-09-13T13:46:40.981131Z","iopub.execute_input":"2022-09-13T13:46:40.981598Z","iopub.status.idle":"2022-09-13T13:46:40.994964Z","shell.execute_reply.started":"2022-09-13T13:46:40.981545Z","shell.execute_reply":"2022-09-13T13:46:40.993827Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('More or less coincide - up to machine precision')","metadata":{"execution":{"iopub.status.busy":"2022-09-13T13:46:42.433385Z","iopub.execute_input":"2022-09-13T13:46:42.434609Z","iopub.status.idle":"2022-09-13T13:46:42.440231Z","shell.execute_reply.started":"2022-09-13T13:46:42.434554Z","shell.execute_reply":"2022-09-13T13:46:42.439230Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Plot for Train","metadata":{}},{"cell_type":"code","source":"list_col_names = (df_multi_train_x.columns)","metadata":{"execution":{"iopub.status.busy":"2022-09-13T13:46:44.665900Z","iopub.execute_input":"2022-09-13T13:46:44.667117Z","iopub.status.idle":"2022-09-13T13:46:44.671812Z","shell.execute_reply.started":"2022-09-13T13:46:44.667064Z","shell.execute_reply":"2022-09-13T13:46:44.670802Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"list_col_names3 = [ t[:3] for t in list_col_names ]\nnp.unique(list_col_names3)","metadata":{"execution":{"iopub.status.busy":"2022-09-13T13:46:46.163782Z","iopub.execute_input":"2022-09-13T13:46:46.165156Z","iopub.status.idle":"2022-09-13T13:46:46.290374Z","shell.execute_reply.started":"2022-09-13T13:46:46.165093Z","shell.execute_reply":"2022-09-13T13:46:46.288938Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_inputs.shape","metadata":{"execution":{"iopub.status.busy":"2022-09-13T13:47:26.338439Z","iopub.execute_input":"2022-09-13T13:47:26.338920Z","iopub.status.idle":"2022-09-13T13:47:26.351753Z","shell.execute_reply.started":"2022-09-13T13:47:26.338880Z","shell.execute_reply":"2022-09-13T13:47:26.350470Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Plots for special chromosomes - X,Y, GL0 , KI2","metadata":{}},{"cell_type":"markdown","source":"# Chromosome Y","metadata":{}},{"cell_type":"code","source":"%%time\ni = 'Y'\nl = ['chr'+str(i)+':'in x for x in list_col_names ]\n#if np.sum(l) == 0: continue\nI = np.where(l)[0]    \nprint(i,np.sum(l)) #  , df_multi_test_x.columns[l],     \n# Visualize it using plt.imshow\n\nplt.figure(figsize = (20,12))\nplt.imshow(train_inputs[:105942,I].toarray(), cmap='Blues', aspect='auto', vmin=0, vmax=1)\nplt.title('chr'+str(i), fontsize = 20)\nplt.show()\n\nplt.figure(figsize = (20,12))\n#plt.imshow(train_inputs[:105942,I].toarray(), cmap='Blues', aspect='auto', vmin=0, vmax=1)\nplt.spy(train_inputs[:,I], aspect='auto',  markersize=0.01) #  vmin=0, vmax=1)\nplt.title('chr'+str(i), fontsize = 20)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-09-13T15:02:20.219617Z","iopub.execute_input":"2022-09-13T15:02:20.220115Z","iopub.status.idle":"2022-09-13T15:02:33.515738Z","shell.execute_reply.started":"2022-09-13T15:02:20.220073Z","shell.execute_reply":"2022-09-13T15:02:33.514338Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Chromosome X ","metadata":{}},{"cell_type":"code","source":"%%time\ni = 'X'\nl = ['chr'+str(i)+':'in x for x in list_col_names ]\n#if np.sum(l) == 0: continue\nI = np.where(l)[0]    \nprint(i,np.sum(l)) #  , df_multi_test_x.columns[l],     \n# Visualize it using plt.imshow\n\nplt.figure(figsize = (20,12))\nplt.imshow(train_inputs[:105942,I].toarray(), cmap='Blues', aspect='auto', vmin=0, vmax=1)\nplt.title('chr'+str(i), fontsize = 20)\nplt.show()\n\nplt.figure(figsize = (20,12))\n#plt.imshow(train_inputs[:105942,I].toarray(), cmap='Blues', aspect='auto', vmin=0, vmax=1)\nplt.spy(train_inputs[:,I], aspect='auto',  markersize=0.01) #  vmin=0, vmax=1)\nplt.title('chr'+str(i), fontsize = 20)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-09-13T15:01:26.635113Z","iopub.execute_input":"2022-09-13T15:01:26.635581Z","iopub.status.idle":"2022-09-13T15:02:07.002430Z","shell.execute_reply.started":"2022-09-13T15:01:26.635542Z","shell.execute_reply":"2022-09-13T15:02:07.000190Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"d = df_cell[df_cell['technology'] == 'multiome']\nplt.plot(d['day'].values)\nplt.show()\nplt.plot(d['donor'].values)\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2022-09-13T16:30:40.215341Z","iopub.execute_input":"2022-09-13T16:30:40.215911Z","iopub.status.idle":"2022-09-13T16:30:40.707536Z","shell.execute_reply.started":"2022-09-13T16:30:40.215865Z","shell.execute_reply":"2022-09-13T16:30:40.705913Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"'GL0', 'KI2', 'chr'","metadata":{"execution":{"iopub.status.busy":"2022-09-13T13:48:17.441477Z","iopub.execute_input":"2022-09-13T13:48:17.441916Z","iopub.status.idle":"2022-09-13T13:48:17.449672Z","shell.execute_reply.started":"2022-09-13T13:48:17.441876Z","shell.execute_reply":"2022-09-13T13:48:17.448446Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# GL0 region of DNA","metadata":{}},{"cell_type":"code","source":"%%time\ni = 'GL0' # 'X'\nl = [ str(i) in x for x in list_col_names ]\n#if np.sum(l) == 0: continue\nI = np.where(l)[0]    \nprint(i,np.sum(l)) #  , df_multi_test_x.columns[l],     \n# Visualize it using plt.imshow\n\nplt.figure(figsize = (20,12))\nplt.imshow(train_inputs[:105942,I].toarray(), cmap='Blues', aspect='auto', vmin=0, vmax=1)\nplt.title('chr'+str(i), fontsize = 20)\nplt.show()\n\n\nplt.figure(figsize = (20,12))\n#plt.imshow(train_inputs[:105942,I].toarray(), cmap='Blues', aspect='auto', vmin=0, vmax=1)\nplt.spy(train_inputs[:,I],  aspect='auto', markersize=0.1)# , vmin=0, vmax=1)\nplt.title('chr'+str(i), fontsize = 20)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-09-13T14:58:52.249489Z","iopub.execute_input":"2022-09-13T14:58:52.250623Z","iopub.status.idle":"2022-09-13T14:59:05.066451Z","shell.execute_reply.started":"2022-09-13T14:58:52.250575Z","shell.execute_reply":"2022-09-13T14:59:05.065407Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# KI2 region of DNA","metadata":{}},{"cell_type":"code","source":"%%time\ni = 'KI2' # 'X'\nl = [ str(i) in x for x in list_col_names ]\n#if np.sum(l) == 0: continue\nI = np.where(l)[0]    \nprint(i,np.sum(l)) #  , df_multi_test_x.columns[l],     \n# Visualize it using plt.imshow\n\nplt.figure(figsize = (20,12))\nplt.imshow(train_inputs[:105942,I].toarray(), cmap='Blues', aspect='auto', vmin=0, vmax=1)\nplt.title('chr'+str(i), fontsize = 20)\nplt.show()\n\n\n\nplt.figure(figsize = (20,12))\n#plt.imshow(train_inputs[:105942,I].toarray(), cmap='Blues', aspect='auto', vmin=0, vmax=1)\nplt.spy(train_inputs[:,I], aspect='auto',  markersize=0.5) #  vmin=0, vmax=1)\nplt.title('chr'+str(i), fontsize = 20)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-09-13T15:00:42.758255Z","iopub.execute_input":"2022-09-13T15:00:42.758682Z","iopub.status.idle":"2022-09-13T15:00:55.419281Z","shell.execute_reply.started":"2022-09-13T15:00:42.758646Z","shell.execute_reply":"2022-09-13T15:00:55.418057Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('10.8G consumed, 14G max')","metadata":{"execution":{"iopub.status.busy":"2022-09-13T13:49:08.526635Z","iopub.execute_input":"2022-09-13T13:49:08.527155Z","iopub.status.idle":"2022-09-13T13:49:08.533781Z","shell.execute_reply.started":"2022-09-13T13:49:08.527112Z","shell.execute_reply":"2022-09-13T13:49:08.532623Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Chromose by chromose (chromosomes 1-22)","metadata":{}},{"cell_type":"code","source":"import gc\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2022-09-13T13:56:34.816093Z","iopub.execute_input":"2022-09-13T13:56:34.816523Z","iopub.status.idle":"2022-09-13T13:56:34.972493Z","shell.execute_reply.started":"2022-09-13T13:56:34.816487Z","shell.execute_reply":"2022-09-13T13:56:34.971002Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\nfor i in  list(range(50)):\n    l = ['chr'+str(i)+':'in x for x in list_col_names ]\n    if np.sum(l) == 0: continue\n    I = np.where(l)[0]    \n    print(i,np.sum(l)) #  , df_multi_test_x.columns[l],     \n    # Visualize it using plt.imshow\n \n    b = 0\n    plt.figure(figsize = (20,12))\n    #plt.imshow(train_inputs[b:b+30_000,I].toarray(), cmap='Blues', aspect='auto', vmin=0, vmax=1)\n    plt.spy(train_inputs[:,I[:5000]], aspect='auto', markersize=0.005)\n    plt.title('chr'+str(i)+ ' First 5000 positions. Plot for sparse matrices', fontsize = 20)\n    plt.ylabel('cell counter',fontsize = 20)\n    plt.xlabel('interval (ATAC-seq) counter',fontsize = 20)\n    plt.show()\n    gc.collect()\n    \n    for b in [0]:\n        plt.figure(figsize = (20,4))\n        plt.imshow(train_inputs[b:b+20_000,I[:5000]].toarray(), cmap='Blues', aspect='auto', vmin=0, vmax=1)\n        #plt.spy(train_inputs[:,I], aspect='auto', markersize=0.1)\n        plt.title('chr'+str(i)+ ' First 20000cells. First 5000 positions. Plot for standard matrices', fontsize = 20)\n        plt.ylabel('cell counter',fontsize = 20)\n        plt.xlabel('interval (ATAC-seq) counter',fontsize = 20)\n        plt.show()\n        gc.collect()\n    \n    \n    if 0:\n        for b in [0,30_000,60_000,90_000]:\n            plt.figure(figsize = (20,4))\n            plt.imshow(train_inputs[b:b+30_000,I].toarray(), cmap='Blues', aspect='auto', vmin=0, vmax=1)\n            #plt.spy(train_inputs[:,I], aspect='auto', markersize=0.1)\n            plt.title('chr'+str(i), fontsize = 20)\n            plt.show()\n            gc.collect()\n    #break","metadata":{"execution":{"iopub.status.busy":"2022-09-13T15:45:30.166391Z","iopub.execute_input":"2022-09-13T15:45:30.166949Z","iopub.status.idle":"2022-09-13T15:51:04.443009Z","shell.execute_reply.started":"2022-09-13T15:45:30.166902Z","shell.execute_reply":"2022-09-13T15:51:04.441772Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Plot for sparse matrices - plt.ply - non zero elements ","metadata":{}},{"cell_type":"code","source":"%%time\n\ni = 'X'\nl = ['chr'+str(i)+':'in x for x in list_col_names ]\n#if np.sum(l) == 0: continue\nI = np.where(l)[0]    \nprint(i,np.sum(l)) #  , df_multi_test_x.columns[l],     \n# Visualize it using plt.imshow\n\nplt.figure(figsize = (20,12))\n#plt.imshow(train_inputs[:105942,I].toarray(), cmap='Blues', aspect='auto', vmin=0, vmax=1)\nplt.spy(train_inputs[:105942,I], aspect='auto', markersize=0.01)\n\nplt.title('chr'+str(i), fontsize = 20)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-09-13T14:51:39.486937Z","iopub.execute_input":"2022-09-13T14:51:39.487368Z","iopub.status.idle":"2022-09-13T14:51:47.295455Z","shell.execute_reply.started":"2022-09-13T14:51:39.487327Z","shell.execute_reply":"2022-09-13T14:51:47.294169Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Chromose Y compare sparse and standard plots","metadata":{"execution":{"iopub.status.busy":"2022-09-13T14:34:35.545827Z","iopub.execute_input":"2022-09-13T14:34:35.546278Z","iopub.status.idle":"2022-09-13T14:34:35.551932Z","shell.execute_reply.started":"2022-09-13T14:34:35.546234Z","shell.execute_reply":"2022-09-13T14:34:35.550712Z"}}},{"cell_type":"code","source":"%%time\n\ni = 'Y'\nl = ['chr'+str(i)+':'in x for x in list_col_names ]\n#if np.sum(l) == 0: continue\nI = np.where(l)[0]    \nprint(i,np.sum(l)) #  , df_multi_test_x.columns[l],     \n# Visualize it using plt.imshow\n\nplt.figure(figsize = (20,6))\nplt.imshow(train_inputs[:105942,I].toarray(), cmap='Blues', aspect='auto', vmin=0, vmax=1)\n#plt.spy(train_inputs[:105942,I], aspect='auto', markersize=0.1)\n\nplt.title('NON sparse plot. chr'+str(i), fontsize = 20)\nplt.show()\n\ni = 'Y'\nl = ['chr'+str(i)+':'in x for x in list_col_names ]\n#if np.sum(l) == 0: continue\nI = np.where(l)[0]    \nprint(i,np.sum(l)) #  , df_multi_test_x.columns[l],     \n# Visualize it using plt.imshow\n\nplt.figure(figsize = (20,6))\n#plt.imshow(train_inputs[:105942,I].toarray(), cmap='Blues', aspect='auto', vmin=0, vmax=1)\nplt.spy(train_inputs[:105942,I], aspect='auto', markersize=0.1)\n\nplt.title('Plot for sparse matrices. chr'+str(i), fontsize = 20)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-09-13T14:34:06.242200Z","iopub.execute_input":"2022-09-13T14:34:06.242683Z","iopub.status.idle":"2022-09-13T14:34:19.425716Z","shell.execute_reply.started":"2022-09-13T14:34:06.242643Z","shell.execute_reply":"2022-09-13T14:34:19.424247Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# From verision 1 notebook - only 4000 first rows - loaded without use of sparse matrcies ","metadata":{}},{"cell_type":"code","source":"# Visualize it using plt.imshow\nplt.figure(figsize = (20,12))\n\nplt.imshow(df_multi_train_x.values, cmap='Blues', aspect='auto', vmin=0, vmax=1)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-09-12T19:53:35.837492Z","iopub.execute_input":"2022-09-12T19:53:35.837809Z","iopub.status.idle":"2022-09-12T19:54:44.873204Z","shell.execute_reply.started":"2022-09-12T19:53:35.837779Z","shell.execute_reply":"2022-09-12T19:54:44.871980Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del df_multi_train_x\nimport gc\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2022-09-12T19:54:44.874629Z","iopub.execute_input":"2022-09-12T19:54:44.875494Z","iopub.status.idle":"2022-09-12T19:54:45.045359Z","shell.execute_reply.started":"2022-09-12T19:54:44.875454Z","shell.execute_reply":"2022-09-12T19:54:45.044095Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Plot for test","metadata":{}},{"cell_type":"code","source":"df_multi_test_x = pd.read_hdf(FP_MULTIOME_TRAIN_INPUTS,start=START,stop=STOP)\n\ndisplay( df_multi_test_x.head() )","metadata":{"execution":{"iopub.status.busy":"2022-09-12T19:54:45.046654Z","iopub.execute_input":"2022-09-12T19:54:45.046997Z","iopub.status.idle":"2022-09-12T19:55:01.442187Z","shell.execute_reply.started":"2022-09-12T19:54:45.046964Z","shell.execute_reply":"2022-09-12T19:55:01.440997Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Visualize it using plt.imshow\nplt.figure(figsize = (20,12))\nplt.imshow(df_multi_test_x.values, cmap='Blues', aspect='auto', vmin=0, vmax=1)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-09-12T19:55:01.446164Z","iopub.execute_input":"2022-09-12T19:55:01.446620Z","iopub.status.idle":"2022-09-12T19:56:14.350222Z","shell.execute_reply.started":"2022-09-12T19:55:01.446582Z","shell.execute_reply":"2022-09-12T19:56:14.349274Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_multi_test_x.shape","metadata":{"execution":{"iopub.status.busy":"2022-09-12T19:56:14.351733Z","iopub.execute_input":"2022-09-12T19:56:14.352118Z","iopub.status.idle":"2022-09-12T19:56:14.359451Z","shell.execute_reply.started":"2022-09-12T19:56:14.352081Z","shell.execute_reply":"2022-09-12T19:56:14.358140Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Visualize it using plt.imshow\nplt.figure(figsize = (20,12))\nplt.imshow(df_multi_test_x.values[:,:30_000], cmap='Blues', aspect='auto', vmin=0, vmax=1)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-09-12T19:56:14.360812Z","iopub.execute_input":"2022-09-12T19:56:14.361203Z","iopub.status.idle":"2022-09-12T19:56:28.893264Z","shell.execute_reply.started":"2022-09-12T19:56:14.361163Z","shell.execute_reply":"2022-09-12T19:56:28.892038Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Plot by chromosomes","metadata":{}},{"cell_type":"code","source":"%%time\nfor i in range(50):\n    l = ['chr'+str(i)+':'in x for x in df_multi_test_x.columns ]\n    if np.sum(l) == 0: continue\n    I = np.where(l)[0]    \n    print(i,np.sum(l)) #  , df_multi_test_x.columns[l],     \n    # Visualize it using plt.imshow\n    plt.figure(figsize = (20,12))\n    plt.imshow(df_multi_test_x.values[:,I], cmap='Blues', aspect='auto', vmin=0, vmax=1)\n    plt.title('chr'+str(i), fontsize = 20)\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2022-09-12T20:03:00.338718Z","iopub.execute_input":"2022-09-12T20:03:00.339218Z","iopub.status.idle":"2022-09-12T20:05:06.094374Z","shell.execute_reply.started":"2022-09-12T20:03:00.339176Z","shell.execute_reply":"2022-09-12T20:05:06.093168Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"    i = 'Y'\n    l = ['chr'+str(i)+':'in x for x in df_multi_test_x.columns ]\n    #if np.sum(l) == 0: continue\n    I = np.where(l)[0]    \n    print(i,np.sum(l)) #  , df_multi_test_x.columns[l],     \n    # Visualize it using plt.imshow\n    plt.figure(figsize = (20,12))\n    plt.imshow(df_multi_test_x.values[:,I], cmap='Blues', aspect='auto', vmin=0, vmax=1)\n    plt.title('chr'+str(i), fontsize = 20)\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2022-09-12T20:05:46.992826Z","iopub.execute_input":"2022-09-12T20:05:46.993276Z","iopub.status.idle":"2022-09-12T20:05:47.865957Z","shell.execute_reply.started":"2022-09-12T20:05:46.993239Z","shell.execute_reply":"2022-09-12T20:05:47.865136Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}