{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# results\n0. \n    a) number of clusters = 8\n    b) clusters obtained by kmeans, regarding all the cells:\n    4 ['CD3', 'CD7', 'CD194', 'CD16', 'CD25', 'Mouse-IgG1', 'Mouse-IgG2a', 'Mouse-IgG2b', 'Rat-IgG2b', 'CD196', 'CD69', 'CD27', 'CD64', 'CD35', 'CD39', 'CD42b', 'CD62P', 'Rat-IgG1', 'Rat-IgG2a', 'IgD', 'CD22', 'CD49a'] \n\n    3 ['CD40', 'CD8', 'CD56', 'CD19', 'CD4', 'CD14', 'CD20', 'CD146', 'IgM', 'CD195', 'CD161', 'CD1c', 'CD11b', 'CD21', 'CD79b', 'CD169', 'integrinB7', 'CD268', 'CD122', 'CD2', 'CD38', 'CD127', 'CD172a', 'CD93', 'CD73', 'TCRVa7.2', 'TCRVd2', 'CD158e1', 'CD352', 'CD94'] \n\n    5 ['CD154', 'CD105', 'CD45RO', 'CD279', 'TIGIT', 'CD185', 'CD223', 'KLRG1', 'CD134', 'CD141', 'CD314', 'CX3CR1', 'CD24', 'CD119', 'CD192', 'CD163', 'CD83', 'CD303', 'CD304', 'CD142', 'CD319', 'CD23', 'HLA-E'] \n\n    7 ['CD33', 'CD49f', 'CD58', 'CD29', 'CD63', 'CD49d', 'CD162', 'CD224'] \n\n    2 ['CD47', 'CD48', 'HLA-A-B-C', 'CD45RA', 'CD123', 'CD44', 'CD31', 'CD62L', 'CD95', 'HLA-DR', 'CD11a', 'CD244', 'CD13', 'CD49b', 'CD81', 'CD18', 'CD45'] \n\n    6 ['CD11c', 'CD54', 'CD72', 'CD9', 'CD328', 'CD101'] \n\n    1 ['Podoplanin', 'CD5', 'CD103', 'CD152', 'CD107a', 'CD1d', 'CD57', 'CD272', 'CD278', 'TCR', 'FceRIa', 'CD137', 'CD124', 'CD226', 'CD28', 'CD26', 'CD115', 'CD158', 'LOX-1', 'CD158b', 'CD85j', 'CD82'] \n\n    0 ['CD41', 'CD71', 'CD36', 'CD88'] \n    \n    \n1. \n    a) stability of clusters methods. \n        DBSCAN is extremely sensetive to epsilon parameter, which correspods to the maximum distance between two samples for one to be considered as in the neighborhood of the other. generally, it's possible to obtain the same or very similar clusters, in comparison to ones produced by KMeans. however, with regard to specific cell type/day, both methods are not applicable, since they result in one large cluster of size 100-130 samples and a few small clusters. perhaps, optimal parameter choice will lead to result enhancement, but optimization procedure will not be convenient for big data. moreover, high sensetivity of DBSCAN doesn't allow to estimate the reasonability of the provided clustering. for now, graph clustering is under consideration\n    b) pearson/spearman correlation.\n        pearson coefficient represents linear correlation, while spearman coef represents non-linear one. by comparing both of them, it's observed that there's no significant difference between them for most of cases. that could be the sign of quasi-linear relation. though there are exceptions: \n        correlation difference between: \n        CD48 and CD32 = 0.2136\n        CD7 and CD328 = 0.4134\n        CD304 and CD101 = 0.2784\n\n        none of these genes are highly correlated\n        \n        mean error = E[pearson]-E[spearman] = 0.131222-0.130632 = 0.00059 (all genes)\n        \n        top groups by correlation:\n        \n        0.5 correlation_threshold \n            Number of weakly connected compoenents 98\n            Top 5 cluster sizes: [39  4  2  1  1]\n            50 Genes in largest correlated group:\n            ['CD155' 'CD112' 'CD47' 'CD48' 'CD33' 'HLA-A-B-C' 'CD45RA' 'CD123' 'CD49f'\n             'CD44' 'CD31' 'Podoplanin' 'CD32' 'CD62L' 'CD107a' 'CD95' 'HLA-DR' 'CD1d'\n             'CD272' 'CD58' 'CD11a' 'CD244' 'FceRIa' 'CD137' 'CD13' 'CD29' 'CD49b'\n             'CD81' 'CD18' 'CD45' 'CD71' 'CD26' 'CD115' 'CD63' 'CD49d' 'CD162' 'CD85j'\n             'CD88' 'CD224']\n\n            0.6 correlation_threshold \n            Number of weakly connected compoenents 122\n            Top 5 cluster sizes: [14  5  2  1  1]\n            50 Genes in largest correlated group:\n            ['CD155' 'CD112' 'CD47' 'HLA-A-B-C' 'CD45RA' 'CD31' 'CD11a' 'CD13' 'CD29'\n             'CD81' 'CD18' 'CD45' 'CD49d' 'CD162']\n\n            0.7 correlation_threshold \n            Number of weakly connected compoenents 137\n            Top 5 cluster sizes: [3 2 1 1 1]\n            50 Genes in largest correlated group:\n            ['CD71' 'CD115' 'CD88']\n        \n\n\n2. stability of clustering methods under variation of day/cell type. DBSCAN and KMeans make one large cluster comprising 100+ samples and a few small ones. the composition of the clusters are similar. by variation of epsilon parameter we can achieve similar distribution. it's like stable but pointless. graph clustering shows more reasonable results, and it is likely robust to the variation of the day and cell type. it saves large cluster (marked as red), while changes the distribution of small clusters. there are a few exceptional cases: BP cell type and MoP cell type\n\n3. correspondace between kmeans clustering and seaborn one.\n\nthey create similar cluster with discrepancy about 10-20% (can be optimized providing additional kmeans parameters)","metadata":{}},{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2023-02-19T21:23:48.385568Z","iopub.execute_input":"2023-02-19T21:23:48.385973Z","iopub.status.idle":"2023-02-19T21:23:48.410502Z","shell.execute_reply.started":"2023-02-19T21:23:48.385945Z","shell.execute_reply":"2023-02-19T21:23:48.408967Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_protein = pd.read_hdf('/kaggle/input/open-problems-multimodal/train_cite_targets.h5')\ndf_meta = pd.read_csv('/kaggle/input/feature-shop-for-multimodal-singlecell-competition/_citeseq_meta_all_text_also.csv')","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:23:48.412819Z","iopub.execute_input":"2023-02-19T21:23:48.413983Z","iopub.status.idle":"2023-02-19T21:23:49.242669Z","shell.execute_reply.started":"2023-02-19T21:23:48.413922Z","shell.execute_reply":"2023-02-19T21:23:49.241283Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_protein.head()","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:23:49.244043Z","iopub.execute_input":"2023-02-19T21:23:49.244443Z","iopub.status.idle":"2023-02-19T21:23:49.270497Z","shell.execute_reply.started":"2023-02-19T21:23:49.244392Z","shell.execute_reply":"2023-02-19T21:23:49.269235Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"protein_corr_pearson = df_protein.corr(method='pearson')\nprotein_corr_spearman = df_protein.corr(method='spearman')","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:23:49.273532Z","iopub.execute_input":"2023-02-19T21:23:49.273971Z","iopub.status.idle":"2023-02-19T21:23:56.714803Z","shell.execute_reply.started":"2023-02-19T21:23:49.273934Z","shell.execute_reply":"2023-02-19T21:23:56.713825Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Full correlation matrix","metadata":{}},{"cell_type":"markdown","source":"**Pearson (linear) correlation**","metadata":{}},{"cell_type":"code","source":"import seaborn as sns\nimport matplotlib.pyplot as plt\n\nsns.clustermap(np.abs(protein_corr_pearson),cmap='vlag', figsize=(30,30), xticklabels=protein_corr_pearson.columns, yticklabels=protein_corr_pearson.columns);","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:23:56.715810Z","iopub.execute_input":"2023-02-19T21:23:56.716045Z","iopub.status.idle":"2023-02-19T21:24:00.413439Z","shell.execute_reply.started":"2023-02-19T21:23:56.716021Z","shell.execute_reply":"2023-02-19T21:24:00.412155Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\nv = protein_corr_pearson.values[np.triu_indices(len(protein_corr_pearson),k=1)]\nprint(len(v), len(protein_corr_pearson)*(len(protein_corr_pearson)-1)/2 )\nplt.hist(v, bins = 1000)#  , color='k')\nplt.show()\nprint(pd.Series(v).describe() )","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:24:00.415052Z","iopub.execute_input":"2023-02-19T21:24:00.415932Z","iopub.status.idle":"2023-02-19T21:24:02.136867Z","shell.execute_reply.started":"2023-02-19T21:24:00.415894Z","shell.execute_reply":"2023-02-19T21:24:02.135450Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**spearman (non-linear) correlation**","metadata":{}},{"cell_type":"code","source":"sns.clustermap(np.abs(protein_corr_spearman),cmap='vlag', figsize=(30,30), xticklabels=protein_corr_spearman.columns, yticklabels=protein_corr_spearman.columns);","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:24:02.140638Z","iopub.execute_input":"2023-02-19T21:24:02.140986Z","iopub.status.idle":"2023-02-19T21:24:05.855082Z","shell.execute_reply.started":"2023-02-19T21:24:02.140960Z","shell.execute_reply":"2023-02-19T21:24:05.853968Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**scipy cluster**","metadata":{}},{"cell_type":"code","source":"import scipy.cluster.hierarchy as sch\n\nwork_mode_correlation = 'Load precalculated data'\n\nclusters_list = []\n\nfor corr_method in ['pearson', 'spearman', 'kendall']:\n    \n    df_corr = df_protein.corr(method= corr_method) ;print();  print(corr_method +' correlations')\n        \n    d = sch.distance.pdist(df_corr)\n    L = sch.linkage(d, method='complete')\n    clusters = sch.fcluster(L, 0.4*d.max(), 'distance')\n    for i,cluster in enumerate(clusters):\n        clusters_list.append([df_corr.index[i], cluster, corr_method])\n    \n    print(corr_method )\n    sns.clustermap(np.abs(df_corr),cmap='vlag', figsize=(25, 25), xticklabels=df_corr.index, yticklabels=df_corr.index); plt.show()\ndf_clusters = pd.DataFrame(clusters_list, columns=['cd', 'cluster', 'corr_method'])","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:24:05.856412Z","iopub.execute_input":"2023-02-19T21:24:05.856697Z","iopub.status.idle":"2023-02-19T21:27:51.565231Z","shell.execute_reply.started":"2023-02-19T21:24:05.856670Z","shell.execute_reply":"2023-02-19T21:27:51.564304Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_clusters.head(20)","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:27:51.566491Z","iopub.execute_input":"2023-02-19T21:27:51.567075Z","iopub.status.idle":"2023-02-19T21:27:51.582854Z","shell.execute_reply.started":"2023-02-19T21:27:51.567040Z","shell.execute_reply":"2023-02-19T21:27:51.580972Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pearson_clusters = df_clusters[df_clusters['corr_method'] == 'pearson']\nspearman_clusters = df_clusters[df_clusters['corr_method'] == 'spearman']\nkendall_clusters = df_clusters[df_clusters['corr_method'] == 'kendall']","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:27:51.587154Z","iopub.execute_input":"2023-02-19T21:27:51.587501Z","iopub.status.idle":"2023-02-19T21:27:51.598363Z","shell.execute_reply.started":"2023-02-19T21:27:51.587471Z","shell.execute_reply":"2023-02-19T21:27:51.597251Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pearson_v_counts = pearson_clusters['cluster'].value_counts().head(10)\n\npearson_df = pearson_clusters.merge(pearson_v_counts.to_frame(),\n                                left_on='cluster',\n                                right_index=True)","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:27:51.599380Z","iopub.execute_input":"2023-02-19T21:27:51.599700Z","iopub.status.idle":"2023-02-19T21:27:51.618706Z","shell.execute_reply.started":"2023-02-19T21:27:51.599665Z","shell.execute_reply":"2023-02-19T21:27:51.617348Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pearson_group = pearson_df.groupby('cluster')\n\npearson_df2 = pearson_group.apply(lambda x: x['cd'].unique())\ns = pearson_df2.str.len().sort_values(ascending=False).index\npearson_df2 = pearson_df2.reindex(s)\npearson = pearson_df2.reset_index(drop=True)\n\nfor x in pearson:\n    print('число элементов в кластере: ', len(x), ', кластер: ', x)","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:27:51.620708Z","iopub.execute_input":"2023-02-19T21:27:51.621089Z","iopub.status.idle":"2023-02-19T21:27:51.638167Z","shell.execute_reply.started":"2023-02-19T21:27:51.621062Z","shell.execute_reply":"2023-02-19T21:27:51.637550Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**stats**","metadata":{}},{"cell_type":"code","source":"%%time\nv = protein_corr_spearman.values[np.triu_indices(len(protein_corr_spearman),k=1)]\nprint(len(v), len(protein_corr_spearman)*(len(protein_corr_spearman)-1)/2 )\nplt.hist(v, bins = 1000)#  , color='k')\nplt.show()\nprint(pd.Series(v).describe() )","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:27:51.639349Z","iopub.execute_input":"2023-02-19T21:27:51.640179Z","iopub.status.idle":"2023-02-19T21:27:53.166544Z","shell.execute_reply.started":"2023-02-19T21:27:51.640152Z","shell.execute_reply":"2023-02-19T21:27:53.165003Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"best cor","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Correlation clusters**","metadata":{}},{"cell_type":"markdown","source":"**similarity to distance**","metadata":{}},{"cell_type":"code","source":"import math\nfrom scipy.cluster.hierarchy import median, fcluster\nfrom scipy.spatial.distance import pdist\n\nnum_features = 139\ngamma = 1/num_features\n\nsim_dist_scalar = protein_corr_pearson.apply(lambda x: np.sqrt(2*(1-abs(x))), axis = 0)\nsim_dist_exp = protein_corr_pearson.apply(lambda x: -x*gamma, axis = 0)\n\ny = pdist(protein_corr_pearson)\nZ = median(y)","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:27:53.167898Z","iopub.execute_input":"2023-02-19T21:27:53.168230Z","iopub.status.idle":"2023-02-19T21:27:53.276732Z","shell.execute_reply.started":"2023-02-19T21:27:53.168203Z","shell.execute_reply":"2023-02-19T21:27:53.275686Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from scipy.spatial import distance\nfrom scipy.cluster import hierarchy\n\ncorrelations = sim_dist_scalar.corr()\ncorrelations_array = np.asarray(sim_dist_scalar.corr())\n\nrow_linkage = hierarchy.linkage(\n    distance.pdist(correlations_array), method='average')\n\ncol_linkage = hierarchy.linkage(\n    distance.pdist(correlations_array.T), method='average')\n\nprint(row_linkage)","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:27:53.279067Z","iopub.execute_input":"2023-02-19T21:27:53.279428Z","iopub.status.idle":"2023-02-19T21:27:53.307039Z","shell.execute_reply.started":"2023-02-19T21:27:53.279402Z","shell.execute_reply":"2023-02-19T21:27:53.305509Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.neighbors import NearestNeighbors\nimport matplotlib.pyplot as plt\n\nneighbors = NearestNeighbors(n_neighbors=num_features)\nneighbors_fit = neighbors.fit(Z)\ndistances, indices = neighbors_fit.kneighbors(Z)\n\ndistances = np.sort(distances, axis=0)\ndistances = distances[:,1]\nplt.plot(distances)","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:27:53.309048Z","iopub.execute_input":"2023-02-19T21:27:53.309738Z","iopub.status.idle":"2023-02-19T21:27:53.481065Z","shell.execute_reply.started":"2023-02-19T21:27:53.309708Z","shell.execute_reply":"2023-02-19T21:27:53.479285Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"f = fcluster(Z, 1.5, criterion='distance')","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:27:53.482270Z","iopub.execute_input":"2023-02-19T21:27:53.482548Z","iopub.status.idle":"2023-02-19T21:27:53.488603Z","shell.execute_reply.started":"2023-02-19T21:27:53.482522Z","shell.execute_reply":"2023-02-19T21:27:53.487252Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(np.unique(f))","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:27:53.490511Z","iopub.execute_input":"2023-02-19T21:27:53.490928Z","iopub.status.idle":"2023-02-19T21:27:53.504579Z","shell.execute_reply.started":"2023-02-19T21:27:53.490892Z","shell.execute_reply":"2023-02-19T21:27:53.502643Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sim_dist_scalar.head()","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:27:53.506556Z","iopub.execute_input":"2023-02-19T21:27:53.507062Z","iopub.status.idle":"2023-02-19T21:27:53.533669Z","shell.execute_reply.started":"2023-02-19T21:27:53.507035Z","shell.execute_reply":"2023-02-19T21:27:53.532544Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**louvain partition**","metadata":{}},{"cell_type":"code","source":"!pip install leidenalg\n!pip install cairocffi","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:27:53.535608Z","iopub.execute_input":"2023-02-19T21:27:53.536187Z","iopub.status.idle":"2023-02-19T21:28:11.553636Z","shell.execute_reply.started":"2023-02-19T21:27:53.536151Z","shell.execute_reply":"2023-02-19T21:28:11.552502Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import igraph\nimport cairocffi\nimport leidenalg as la\n\ng = igraph.Graph().Adjacency(sim_dist_scalar)\ng.to_undirected(mode = 'collapse')\npartition = la.find_partition(g, la.CPMVertexPartition,resolution_parameter = 0.997)\nigraph.plot(partition, bbox = (400,400)) ","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:11.556393Z","iopub.execute_input":"2023-02-19T21:28:11.556759Z","iopub.status.idle":"2023-02-19T21:28:12.166929Z","shell.execute_reply.started":"2023-02-19T21:28:11.556732Z","shell.execute_reply":"2023-02-19T21:28:12.165678Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**KMeans**","metadata":{}},{"cell_type":"code","source":"from sklearn import cluster\n\nn_clusters = 8\n\nk_means = cluster.KMeans(n_clusters = n_clusters)\nk_means.fit(sim_dist_scalar)\n    # apply the labels\ntrain_labels = k_means.labels_","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:12.168620Z","iopub.execute_input":"2023-02-19T21:28:12.168943Z","iopub.status.idle":"2023-02-19T21:28:12.678587Z","shell.execute_reply.started":"2023-02-19T21:28:12.168916Z","shell.execute_reply":"2023-02-19T21:28:12.677711Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sim_dist_scalar['clusters_kmeans'] = train_labels\ntrain_labels.shape","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:12.680704Z","iopub.execute_input":"2023-02-19T21:28:12.682167Z","iopub.status.idle":"2023-02-19T21:28:12.693351Z","shell.execute_reply.started":"2023-02-19T21:28:12.682128Z","shell.execute_reply":"2023-02-19T21:28:12.692197Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"genes = sim_dist_scalar.columns.tolist()\ngenes.remove('clusters_kmeans')\nclusters = dict()\n\nfor i in range(len(train_labels)):\n    if train_labels[i] in clusters:\n        clusters[train_labels[i]].append(genes[i])\n    else:\n        clusters[train_labels[i]] = []\n\nfor key in clusters.keys():\n    print(key, clusters[key],'\\n')","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:12.699647Z","iopub.execute_input":"2023-02-19T21:28:12.700301Z","iopub.status.idle":"2023-02-19T21:28:12.713084Z","shell.execute_reply.started":"2023-02-19T21:28:12.700271Z","shell.execute_reply":"2023-02-19T21:28:12.712154Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"protein_corr_pearson.head()","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:12.715562Z","iopub.execute_input":"2023-02-19T21:28:12.717047Z","iopub.status.idle":"2023-02-19T21:28:12.755175Z","shell.execute_reply.started":"2023-02-19T21:28:12.717007Z","shell.execute_reply":"2023-02-19T21:28:12.754284Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"clusters_corr_list = list()\n\nfor key in clusters.keys():\n    l = list(set(protein_corr_pearson.columns).intersection(set(clusters[key])))\n    print(l)\n    cluster_corr = protein_corr_pearson[l]\n    #cluster_corr = select_rows(protein_corr_pearson,clusters[key])\n    #cluster_corr = protein_corr_pearson[np.equal.outer(protein_corr_pearson.to_numpy(copy=False),  clusters[key]).any(axis=1).all(axis=1)]\n    v = cluster_corr.abs().values[np.triu_indices(len(cluster_corr.columns),k=1)].mean().mean()\n    print(v.sum())\n    clusters_corr_list.append(v.sum()/len(cluster_corr))","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:12.757901Z","iopub.execute_input":"2023-02-19T21:28:12.758260Z","iopub.status.idle":"2023-02-19T21:28:12.771922Z","shell.execute_reply.started":"2023-02-19T21:28:12.758232Z","shell.execute_reply":"2023-02-19T21:28:12.771201Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**DBSCAN**","metadata":{}},{"cell_type":"markdown","source":"eps boundaries: 1.87 - 2.57","metadata":{}},{"cell_type":"code","source":"dbscan = cluster.DBSCAN(eps = 1.99)\ndbscan.fit(sim_dist_scalar)\n# apply the labels\ntrain_labels = dbscan.labels_\nif -1 not in train_labels:\n    print(set(train_labels))","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:12.773132Z","iopub.execute_input":"2023-02-19T21:28:12.773554Z","iopub.status.idle":"2023-02-19T21:28:12.788118Z","shell.execute_reply.started":"2023-02-19T21:28:12.773530Z","shell.execute_reply":"2023-02-19T21:28:12.786892Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sim_dist_scalar = sim_dist_scalar.copy()\nsim_dist_scalar['clusters_dbscan'] = train_labels\nsim_dist_scalar['clusters_dbscan'].unique()","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:12.803058Z","iopub.execute_input":"2023-02-19T21:28:12.804239Z","iopub.status.idle":"2023-02-19T21:28:12.821888Z","shell.execute_reply.started":"2023-02-19T21:28:12.804203Z","shell.execute_reply":"2023-02-19T21:28:12.820569Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"no_noise = sim_dist_scalar.loc[(sim_dist_scalar['clusters_dbscan'] != -1)]","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:12.824011Z","iopub.execute_input":"2023-02-19T21:28:12.824355Z","iopub.status.idle":"2023-02-19T21:28:12.836658Z","shell.execute_reply.started":"2023-02-19T21:28:12.824314Z","shell.execute_reply":"2023-02-19T21:28:12.835478Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"d_dbscan = []\nd_kmeans = []\nfor i in range(8):\n    d_dbscan.append(sim_dist_scalar['clusters_dbscan'][sim_dist_scalar['clusters_dbscan']==i].size)\n    d_kmeans.append(sim_dist_scalar['clusters_kmeans'][sim_dist_scalar['clusters_kmeans']==i].size)\nsns.histplot(d_dbscan, bins=8, color='red')\nsns.histplot(d_kmeans, bins=8)","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:12.838209Z","iopub.execute_input":"2023-02-19T21:28:12.838923Z","iopub.status.idle":"2023-02-19T21:28:13.080629Z","shell.execute_reply.started":"2023-02-19T21:28:12.838887Z","shell.execute_reply":"2023-02-19T21:28:13.079846Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Clusters distribution","metadata":{}},{"cell_type":"markdown","source":"**Check if rat/mouse genes in one cluster**","metadata":{}},{"cell_type":"code","source":"methods = ['clusters_dbscan', 'clusters_kmeans']\nmethod = 'clusters_kmeans'\nnum_clusters_rat_mouse = 0\nclusters_rat_mouse = []\nrat_mouse_genes = ['Mouse-IgG1', 'Mouse-IgG2a', 'Mouse-IgG2b', 'Rat-IgG2b', 'Rat-IgG1', 'Rat-IgG2a']\n\nfor i in sim_dist_scalar[method]:\n    for gene in rat_mouse_genes:\n        if gene in sim_dist_scalar[method][sim_dist_scalar[method]==i]:\n            clusters_rat_mouse.append(i)\n\nprint(set(clusters_rat_mouse))","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:13.082065Z","iopub.execute_input":"2023-02-19T21:28:13.082396Z","iopub.status.idle":"2023-02-19T21:28:13.243492Z","shell.execute_reply.started":"2023-02-19T21:28:13.082362Z","shell.execute_reply":"2023-02-19T21:28:13.242818Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import igraph\nlist_X_column_names = list(df_protein.columns)\ni=0\ndf_stat = pd.DataFrame()\nfor correlation_threshold in [0.2, 0.3, 0.4, 0.5, 0.6, 0.7]:\n    print()\n    print(correlation_threshold , 'correlation_threshold ')\n    corr_matr_abs_bool = protein_corr_pearson > correlation_threshold\n    corr_matr_abs_bool = np.triu(corr_matr_abs_bool,1) # Take upper triangular part \n    g = igraph.Graph().Adjacency(corr_matr_abs_bool.tolist())\n    g.to_undirected(mode = 'collapse')\n    print('Number of weakly connected compoenents', len( g.clusters(mode='WEAK')))\n\n    list_clusters_nodes_lists = list( g.clusters(mode='WEAK') )\n    list_clusers_size = [len(t) for t in list_clusters_nodes_lists ]\n    list_clusers_size = np.sort(list_clusers_size)[::-1]\n    print('Top 5 cluster sizes:', list_clusers_size[:5])\n    #dict_save_largest_component_size[correlation_threshold ] = list_clusers_size[0]\n    for t  in list_clusters_nodes_lists:\n        if len(t) == list_clusers_size[0]:\n            print('50 Genes in largest correlated group:')\n            print(np.array(list_X_column_names)[t[:50]])\n    i += 1\n    df_stat.loc[i,'correlation threshold'] = correlation_threshold\n    df_stat.loc[i,'Largest Component Size'] = list_clusers_size[0]\n\n    \ndf_stat","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:13.244648Z","iopub.execute_input":"2023-02-19T21:28:13.245042Z","iopub.status.idle":"2023-02-19T21:28:13.277464Z","shell.execute_reply.started":"2023-02-19T21:28:13.245017Z","shell.execute_reply":"2023-02-19T21:28:13.276101Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Spearman (nonlinear) correlation**","metadata":{}},{"cell_type":"code","source":"sns.clustermap(np.abs(protein_corr_spearman),cmap='vlag', figsize=(30,30), xticklabels=protein_corr_pearson.columns, yticklabels=protein_corr_pearson.columns);","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:13.278944Z","iopub.execute_input":"2023-02-19T21:28:13.279202Z","iopub.status.idle":"2023-02-19T21:28:17.002252Z","shell.execute_reply.started":"2023-02-19T21:28:13.279179Z","shell.execute_reply":"2023-02-19T21:28:17.000436Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"cd 155 cd81 cd13 cd31 \nCD270          1\nCD155          1\nCD154          1\nCD33           1\nCD45RO         1\nCD279          1\nTIGIT          1\nMouse-IgG1     1\nMouse-IgG2a    1\nMouse-IgG2b    1\nRat-IgG2b      1\nCD134          1\nCD141          1\nCD314          1\nCX3CR1         1\nCD24           1\nCD119          1\nRat-IgG1       1\nRat-IgG2a      1\nCD192          1\nCD163          1\nCD83           1\nCD29           1\nCD38           1\nCD63           1\nCD49d          1\nCD162          1\nCD23           1\nHLA-E          1\nCD224          1\nName: clust","metadata":{}},{"cell_type":"markdown","source":"**Genes with significant difference between linear and nonlinear correlation**","metadata":{}},{"cell_type":"code","source":"diff = protein_corr_spearman-protein_corr_pearson\ndiff = diff.abs()\nmin_max_corr_diff = (diff.max().max()-diff.min().min())/2\nsignificant_corr_diff = [{gene_id: diff[gene_id].max()} for gene_id in diff.columns if diff[gene_id].max()>min_max_corr_diff]\nprint(significant_corr_diff)","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:17.003842Z","iopub.execute_input":"2023-02-19T21:28:17.004200Z","iopub.status.idle":"2023-02-19T21:28:17.034507Z","shell.execute_reply.started":"2023-02-19T21:28:17.004156Z","shell.execute_reply":"2023-02-19T21:28:17.033167Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\ntop_diff_pear_spear = dict()\nfor i in range(len(significant_corr_diff)):\n    for j in range (len(significant_corr_diff),0):\n        print(significant_corr_diff[i])\n        if significant_corr_diff[i].values() == significant_corr_diff[j].values():\n            top_diff_pear_spear[significant_corr_diff[i].values()] = significant_corr_diff[i].keys() + significant_corr_diff[j].keys()\n\nprint(top_diff_pear_spear)\n        ","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:17.036916Z","iopub.execute_input":"2023-02-19T21:28:17.037327Z","iopub.status.idle":"2023-02-19T21:28:17.045744Z","shell.execute_reply.started":"2023-02-19T21:28:17.037300Z","shell.execute_reply":"2023-02-19T21:28:17.044400Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Check if highly correlated genes show significant difference in pearson and spearman correlation**","metadata":{}},{"cell_type":"code","source":"corr_threshold_05 = ['CD155', 'CD112', 'CD47', 'CD48', 'CD33', 'HLA-A-B-C', 'CD45RA', 'CD123', 'CD49f',\n 'CD44', 'CD31', 'Podoplanin', 'CD32', 'CD62L', 'CD107a', 'CD95', 'HLA-DR', 'CD1d',\n 'CD272', 'CD58', 'CD11a', 'CD244', 'FceRIa', 'CD137', 'CD13', 'CD29', 'CD49b',\n 'CD81', 'CD18', 'CD45', 'CD71', 'CD26', 'CD115', 'CD63', 'CD49d', 'CD162', 'CD85j',\n 'CD88', 'CD224']\nprint([gene for gene in corr_threshold_05 if gene in significant_corr_diff])   # no significant difference","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:17.047735Z","iopub.execute_input":"2023-02-19T21:28:17.048118Z","iopub.status.idle":"2023-02-19T21:28:17.059150Z","shell.execute_reply.started":"2023-02-19T21:28:17.048092Z","shell.execute_reply":"2023-02-19T21:28:17.057938Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Correlation matrix of specific gene type","metadata":{}},{"cell_type":"code","source":"cols_to_delete = list(df_meta.columns)\nfor el in cols_to_delete:\n    if el in ['cell_id', 'cell_type', 'day']:\n        cols_to_delete.remove(el)\ndf_meta.drop(cols_to_delete, inplace=True, axis = 1)\ndf_meta.head()","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:17.060105Z","iopub.execute_input":"2023-02-19T21:28:17.060371Z","iopub.status.idle":"2023-02-19T21:28:17.084666Z","shell.execute_reply.started":"2023-02-19T21:28:17.060346Z","shell.execute_reply":"2023-02-19T21:28:17.083188Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_protein.head()","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:17.086425Z","iopub.execute_input":"2023-02-19T21:28:17.086750Z","iopub.status.idle":"2023-02-19T21:28:17.112088Z","shell.execute_reply.started":"2023-02-19T21:28:17.086723Z","shell.execute_reply":"2023-02-19T21:28:17.110713Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df = df_protein.merge(df_meta, on='cell_id')\ndf.head()","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:17.113323Z","iopub.execute_input":"2023-02-19T21:28:17.114013Z","iopub.status.idle":"2023-02-19T21:28:17.251104Z","shell.execute_reply.started":"2023-02-19T21:28:17.113985Z","shell.execute_reply":"2023-02-19T21:28:17.250091Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"d = dict()\nfor cell_type in df['cell_type'].unique():\n    df_type = df.loc[lambda df1: df1['cell_type'] == cell_type, :]\n    protein_corr_pearson = df_type.corr(method='pearson')\n    d[f'{cell_type}_pearson']=protein_corr_pearson\n    protein_corr_spearman = df_type.corr(method='spearman')\n    d[f'{cell_type}_spearman']=protein_corr_spearman\n","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:17.252765Z","iopub.execute_input":"2023-02-19T21:28:17.253087Z","iopub.status.idle":"2023-02-19T21:28:25.640576Z","shell.execute_reply.started":"2023-02-19T21:28:17.253056Z","shell.execute_reply":"2023-02-19T21:28:25.638707Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Top 10 CD by AUC","metadata":{}},{"cell_type":"code","source":"cell_type_auc = {\n    'EryP': ['CD82', 'CD26', 'CD272', 'CD36', 'FceRIa', 'CD32', 'CD41', 'CD115', 'CD71', 'CD88'],\n    'HSC' : ['CD72', 'CD9', 'CD45',  'CD31', 'CD13', 'CD54', 'CD11a', 'CD123', 'CD62L', 'CD49b'],\n    'MkP' : ['FceRIa', 'CD40', 'CD88', 'CD62P',  'CD63', 'KLRG1', 'CD9', 'CD71', 'CD82', 'CD41'],\n    'MasP' : ['CD26', 'CD158', 'FceRIa', 'CD49d', 'CD142', 'CD162', 'CD33', 'CD38', 'CD82', 'CD32'],\n    'NeuP' : ['HLA-A-B-C',  'CD112', 'CD64', 'CD49d', 'CD162', 'CD33', 'CD95', 'CD38', 'CD45RA', 'CD48'],\n    'MoP' : ['CD36', 'CD54', 'CD11a', 'CD9', 'CD32', 'CD304', 'CD86', 'CD45RA', 'CD48', 'CD101'],\n    'BP' : ['CD52', 'CD47', 'CD9', 'IgD', 'CD72', 'CD11a', 'CD18', 'CD22', 'CD54', 'CD45RA']\n    \n}","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:25.642551Z","iopub.execute_input":"2023-02-19T21:28:25.642972Z","iopub.status.idle":"2023-02-19T21:28:25.650468Z","shell.execute_reply.started":"2023-02-19T21:28:25.642943Z","shell.execute_reply":"2023-02-19T21:28:25.649486Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cluster_cell_type_match = pd.DataFrame(columns=cell_type_auc.keys(), index=clusters.keys())\nfor cluster_ in clusters.keys():\n    for cell_type in cell_type_auc.keys():\n        match = len(set(clusters[cluster_]).intersection(set(cell_type_auc[cell_type])))\n        cluster_cell_type_match[cell_type][cluster_] = match/len(clusters[cluster_])\ncluster_cell_type_match.head(50)","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:25.652425Z","iopub.execute_input":"2023-02-19T21:28:25.653740Z","iopub.status.idle":"2023-02-19T21:28:25.688454Z","shell.execute_reply.started":"2023-02-19T21:28:25.653664Z","shell.execute_reply":"2023-02-19T21:28:25.687309Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**HSC**","metadata":{}},{"cell_type":"code","source":"sns.clustermap(np.abs(d['HSC_pearson']),cmap='vlag', figsize=(30,30), xticklabels=protein_corr_pearson.columns, yticklabels=protein_corr_pearson.columns);","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:25.690140Z","iopub.execute_input":"2023-02-19T21:28:25.690753Z","iopub.status.idle":"2023-02-19T21:28:29.606491Z","shell.execute_reply.started":"2023-02-19T21:28:25.690715Z","shell.execute_reply":"2023-02-19T21:28:29.605274Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\nv = d['HSC_pearson'].values[np.triu_indices(len(d['HSC_pearson']),k=1)]\nprint(len(v), len(d['HSC_pearson'])*(len(d['HSC_pearson'])-1)/2 )\nplt.hist(v, bins = 1000)#  , color='k')\nplt.show()\nprint(pd.Series(v).describe() )","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:29.607989Z","iopub.execute_input":"2023-02-19T21:28:29.608318Z","iopub.status.idle":"2023-02-19T21:28:31.031351Z","shell.execute_reply.started":"2023-02-19T21:28:29.608289Z","shell.execute_reply":"2023-02-19T21:28:31.029837Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"2 ['CD40', 'CD3', 'CD19', 'CD11c', 'CD4', 'CD14', 'CD16', 'CD25', 'CD20', 'IgM', 'CD196', 'CD69', 'CD27', 'CD1c', 'CD11b', 'CD64', 'CD35', 'CD39', 'CD21', 'CD79b', 'CD268', 'CD62P', 'IgD', 'CD38', 'CD22', 'CD172a', 'CD93', 'CD49a', 'CD352', 'CD328', 'CD101'] \n\n0 ['CD56', 'CD105', 'CD279', 'TIGIT', 'CD195', 'CD185', 'CD161', 'CD223', 'KLRG1', 'CD134', 'CX3CR1', 'CD24', 'CD169', 'CD122', 'CD163', 'CD303', 'CD127', 'CD304', 'CD73', 'CD158e1', 'CD142', 'CD319', 'CD94', 'CD23', 'HLA-E'] \n\n5 ['CD154', 'CD45RO', 'CD335', 'Podoplanin', 'CD5', 'CD103', 'CD152', 'CD107a', 'CD141', 'CD1d', 'CD314', 'CD57', 'CD272', 'CD278', 'TCR', 'CD192', 'FceRIa', 'CD137', 'CD83', 'CD124', 'CD226', 'CD28', 'CD26', 'CD115', 'CD158', 'LOX-1', 'CD158b', 'CD85j', 'CD82', 'CD88'] \n\n7 ['CD49f', 'CD58', 'CD119', 'CD29', 'CD63', 'CD49d', 'CD224'] \n\n1 ['CD47', 'CD52', 'HLA-A-B-C', 'CD123', 'CD44', 'CD31', 'CD62L', 'CD95', 'HLA-DR', 'CD11a', 'CD244', 'CD54', 'CD13', 'CD49b', 'CD81', 'CD45', 'CD9'] \n\n6 ['CD33', 'CD45RA', 'CD18', 'CD72', 'CD162'] \n\n3 ['CD7', 'CD194', 'Mouse-IgG1', 'Mouse-IgG2a', 'Mouse-IgG2b', 'Rat-IgG2b', 'CD146', 'integrinB7', 'CD42b', 'Rat-IgG1', 'Rat-IgG2a', 'CD2', 'TCRVa7.2', 'TCRVd2'] \n\n4 ['CD41', 'CD71', 'CD36'] ","metadata":{}},{"cell_type":"code","source":"num_features = 139\ngamma = 1/num_features\n\nsim_dist_scalar = d['HSC_pearson'].corr().apply(lambda x: np.sqrt(2*(1-abs(x))), axis = 0)\nsim_dist_exp = d['HSC_pearson'].corr().apply(lambda x: -x*gamma, axis = 0)\n\ny = pdist(d['HSC_pearson'].corr())\nZ = median(y)","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:31.032988Z","iopub.execute_input":"2023-02-19T21:28:31.033276Z","iopub.status.idle":"2023-02-19T21:28:31.149715Z","shell.execute_reply.started":"2023-02-19T21:28:31.033253Z","shell.execute_reply":"2023-02-19T21:28:31.148956Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.clustermap(np.abs(d['HSC_spearman']),cmap='vlag', figsize=(30,30), xticklabels=protein_corr_spearman.columns, yticklabels=protein_corr_spearman.columns);","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:31.150892Z","iopub.execute_input":"2023-02-19T21:28:31.151311Z","iopub.status.idle":"2023-02-19T21:28:35.039465Z","shell.execute_reply.started":"2023-02-19T21:28:31.151286Z","shell.execute_reply":"2023-02-19T21:28:35.038424Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\nv = d['HSC_spearman'].values[np.triu_indices(len(d['HSC_spearman']),k=1)]\nprint(len(v), len(d['HSC_spearman'])*(len(d['HSC_spearman'])-1)/2 )\nplt.hist(v, bins = 1000)#  , color='k')\nplt.show()\nprint(pd.Series(v).describe() )","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:35.040748Z","iopub.execute_input":"2023-02-19T21:28:35.041201Z","iopub.status.idle":"2023-02-19T21:28:36.521183Z","shell.execute_reply.started":"2023-02-19T21:28:35.041175Z","shell.execute_reply":"2023-02-19T21:28:36.520268Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"protein_corr_pearson = d['HSC_pearson'].corr()\nsim_dist_scalar = protein_corr_pearson.apply(lambda x: np.sqrt(2*(1-abs(x))), axis = 0)\ng = igraph.Graph().Adjacency(sim_dist_scalar)\ng.to_undirected(mode = 'collapse')\npartition = la.find_partition(g, la.CPMVertexPartition,resolution_parameter = 0.997)\nigraph.plot(partition, bbox = (400,400)) ","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:36.522642Z","iopub.execute_input":"2023-02-19T21:28:36.523215Z","iopub.status.idle":"2023-02-19T21:28:37.142791Z","shell.execute_reply.started":"2023-02-19T21:28:36.523178Z","shell.execute_reply":"2023-02-19T21:28:37.140991Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"protein_corr_pearson = d['HSC_pearson'].corr()\nsim_dist_scalar_hsc = d['HSC_pearson'].apply(lambda x: np.sqrt(2*(1-abs(x))), axis = 0)\n\nn_clusters = 7\n\nk_means = cluster.KMeans(n_clusters = n_clusters)\nk_means.fit(sim_dist_scalar_hsc)\n    # apply the labels\ntrain_labels = k_means.labels_\n\n\nsim_dist_scalar_hsc['clusters_kmeans'] = train_labels\n\ngenes = sim_dist_scalar_hsc.columns.tolist()\ngenes.remove('clusters_kmeans')\nclusters_hsc = dict()\n\nfor i in range(len(train_labels)):\n    if train_labels[i] in clusters_hsc:\n        clusters_hsc[train_labels[i]].append(genes[i])\n    else:\n        clusters_hsc[train_labels[i]] = []\n\nfor key in clusters_hsc.keys():\n    print(key, clusters_hsc[key],'\\n')","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:37.144300Z","iopub.execute_input":"2023-02-19T21:28:37.144634Z","iopub.status.idle":"2023-02-19T21:28:37.822584Z","shell.execute_reply.started":"2023-02-19T21:28:37.144603Z","shell.execute_reply":"2023-02-19T21:28:37.821632Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('\\n *** 0.3 threshold ***\\n')\nfor cluster_hsc in clusters_hsc.keys():\n    for cluster_ in clusters.keys():\n        inter = set(clusters_hsc[cluster_hsc]).intersection(set(clusters[cluster_]))\n        if list(inter)!=[] and len(inter)/max(len(clusters[cluster_]), len(clusters_hsc[cluster_hsc]))>0.3:\n            print(f'{len(inter)/max(len(clusters[cluster_]), len(clusters_hsc[cluster_hsc]))}')\n            print(f'all cell types clusters:  {clusters[cluster_]}')\n            print(f'HSC clusters:  {clusters[cluster_]}')\n            print(f'intersection:  {inter}')\nprint('\\n *** 0.5 threshold ***\\n')\nfor cluster_hsc in clusters_hsc.keys():\n    for cluster_ in clusters.keys():\n        inter = set(clusters_hsc[cluster_hsc]).intersection(set(clusters[cluster_]))\n        if list(inter)!=[] and len(inter)/max(len(clusters[cluster_]), len(clusters_hsc[cluster_hsc]))>=0.5:\n            print(f'{len(inter)/max(len(clusters[cluster_]), len(clusters_hsc[cluster_hsc]))}')\n            print(f'all cell types clusters:  {clusters[cluster_]}')\n            print(f'HSC clusters:  {clusters[cluster_]}')\n            print(f'intersection:  {inter}')","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:49:42.388060Z","iopub.execute_input":"2023-02-19T21:49:42.388419Z","iopub.status.idle":"2023-02-19T21:49:42.399186Z","shell.execute_reply.started":"2023-02-19T21:49:42.388389Z","shell.execute_reply":"2023-02-19T21:49:42.397737Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**EryP**","metadata":{}},{"cell_type":"markdown","source":"**pearson**","metadata":{}},{"cell_type":"code","source":"sns.clustermap(np.abs(d['EryP_pearson']),cmap='vlag', figsize=(30,30), xticklabels=protein_corr_pearson.columns, yticklabels=protein_corr_pearson.columns);","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:37.864826Z","iopub.status.idle":"2023-02-19T21:28:37.868498Z","shell.execute_reply.started":"2023-02-19T21:28:37.868127Z","shell.execute_reply":"2023-02-19T21:28:37.868165Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\nv = d['EryP_pearson'].values[np.triu_indices(len(d['EryP_pearson']),k=1)]\nprint(len(v), len(d['EryP_pearson'])*(len(d['EryP_pearson'])-1)/2 )\nplt.hist(v, bins = 1000)#  , color='k')\nplt.show()\nprint(pd.Series(v).describe() )","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:37.873383Z","iopub.status.idle":"2023-02-19T21:28:37.875265Z","shell.execute_reply.started":"2023-02-19T21:28:37.874872Z","shell.execute_reply":"2023-02-19T21:28:37.874916Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**spearman**","metadata":{}},{"cell_type":"code","source":"sns.clustermap(np.abs(d['EryP_spearman']),cmap='vlag', figsize=(30,30), xticklabels=protein_corr_spearman.columns, yticklabels=protein_corr_spearman.columns);","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:37.878991Z","iopub.status.idle":"2023-02-19T21:28:37.879647Z","shell.execute_reply.started":"2023-02-19T21:28:37.879460Z","shell.execute_reply":"2023-02-19T21:28:37.879479Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\nv = d['EryP_spearman'].values[np.triu_indices(len(d['EryP_spearman']),k=1)]\nprint(len(v), len(d['EryP_spearman'])*(len(d['EryP_spearman'])-1)/2 )\nplt.hist(v, bins = 1000)#  , color='k')\nplt.show()\nprint(pd.Series(v).describe() )","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:37.883479Z","iopub.status.idle":"2023-02-19T21:28:37.883990Z","shell.execute_reply.started":"2023-02-19T21:28:37.883733Z","shell.execute_reply":"2023-02-19T21:28:37.883756Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"protein_corr_pearson = d['EryP_pearson'].corr()\nsim_dist_scalar = protein_corr_pearson.apply(lambda x: np.sqrt(2*(1-abs(x))), axis = 0)\ng = igraph.Graph().Adjacency(sim_dist_scalar)\ng.to_undirected(mode = 'collapse')\npartition = la.find_partition(g, la.CPMVertexPartition,resolution_parameter = 0.997)\nigraph.plot(partition, bbox = (400,400)) ","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:37.885857Z","iopub.status.idle":"2023-02-19T21:28:37.886280Z","shell.execute_reply.started":"2023-02-19T21:28:37.886117Z","shell.execute_reply":"2023-02-19T21:28:37.886133Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"protein_corr_pearson = d['EryP_pearson'].corr()\nsim_dist_scalar = protein_corr_pearson.apply(lambda x: np.sqrt(2*(1-abs(x))), axis = 0)\n\nn_clusters = 8\n\nk_means = cluster.KMeans(n_clusters = n_clusters)\nk_means.fit(sim_dist_scalar)\n    # apply the labels\ntrain_labels = k_means.labels_\nprint(train_labels)\n\n\nsim_dist_scalar['clusters_kmeans'] = train_labels\n\ndbscan = cluster.DBSCAN(eps = 1.99)\ndbscan.fit(sim_dist_scalar)\n# apply the labels\ntrain_labels = dbscan.labels_\nif -1 not in train_labels:\n    print(set(train_labels))\nnum_clusters_db = len(train_labels)\nn_clusters = max(n_clusters,num_clusters_db)\n    \nsim_dist_scalar['clusters_dbscan'] = train_labels\nsim_dist_scalar.head()\n\n#Plot \nd_dbscan = []\nd_kmeans = []\nfor i in range(n_clusters):\n    d_dbscan.append(sim_dist_scalar['clusters_dbscan'][sim_dist_scalar['clusters_dbscan']==i].size)\n    d_kmeans.append(sim_dist_scalar['clusters_kmeans'][sim_dist_scalar['clusters_kmeans']==i].size)\nsns.histplot(d_dbscan, bins=n_clusters, color='red')\nsns.histplot(d_kmeans, bins=n_clusters)","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:37.888336Z","iopub.status.idle":"2023-02-19T21:28:37.889124Z","shell.execute_reply.started":"2023-02-19T21:28:37.888890Z","shell.execute_reply":"2023-02-19T21:28:37.888915Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**NeuP**","metadata":{}},{"cell_type":"markdown","source":"**pearson**","metadata":{}},{"cell_type":"code","source":"sns.clustermap(np.abs(d['NeuP_pearson']),cmap='vlag', figsize=(30,30), xticklabels=protein_corr_pearson.columns, yticklabels=protein_corr_pearson.columns);","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:37.890378Z","iopub.status.idle":"2023-02-19T21:28:37.891055Z","shell.execute_reply.started":"2023-02-19T21:28:37.890843Z","shell.execute_reply":"2023-02-19T21:28:37.890865Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\nv = d['NeuP_pearson'].values[np.triu_indices(len(d['NeuP_pearson']),k=1)]\nprint(len(v), len(d['NeuP_pearson'])*(len(d['NeuP_pearson'])-1)/2 )\nplt.hist(v, bins = 1000)#  , color='k')\nplt.show()\nprint(pd.Series(v).describe() )","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:37.892299Z","iopub.status.idle":"2023-02-19T21:28:37.893029Z","shell.execute_reply.started":"2023-02-19T21:28:37.892760Z","shell.execute_reply":"2023-02-19T21:28:37.892833Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**spearman**","metadata":{}},{"cell_type":"code","source":"sns.clustermap(np.abs(d['NeuP_spearman']),cmap='vlag', figsize=(30,30), xticklabels=protein_corr_spearman.columns, yticklabels=protein_corr_spearman.columns);","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:37.894290Z","iopub.status.idle":"2023-02-19T21:28:37.894990Z","shell.execute_reply.started":"2023-02-19T21:28:37.894746Z","shell.execute_reply":"2023-02-19T21:28:37.894787Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\nv = d['NeuP_spearman'].values[np.triu_indices(len(d['NeuP_spearman']),k=1)]\nprint(len(v), len(d['NeuP_spearman'])*(len(d['NeuP_spearman'])-1)/2 )\nplt.hist(v, bins = 1000)#  , color='k')\nplt.show()\nprint(pd.Series(v).describe() )","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:37.896276Z","iopub.status.idle":"2023-02-19T21:28:37.896978Z","shell.execute_reply.started":"2023-02-19T21:28:37.896717Z","shell.execute_reply":"2023-02-19T21:28:37.896738Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"protein_corr_pearson = d['NeuP_pearson'].corr()\nsim_dist_scalar = protein_corr_pearson.apply(lambda x: np.sqrt(2*(1-abs(x))), axis = 0)\ng = igraph.Graph().Adjacency(sim_dist_scalar)\ng.to_undirected(mode = 'collapse')\npartition = la.find_partition(g, la.CPMVertexPartition,resolution_parameter = 0.997)\nigraph.plot(partition, bbox = (400,400))","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:37.898224Z","iopub.status.idle":"2023-02-19T21:28:37.898916Z","shell.execute_reply.started":"2023-02-19T21:28:37.898666Z","shell.execute_reply":"2023-02-19T21:28:37.898688Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"protein_corr_pearson = d['NeuP_pearson'].corr()\nsim_dist_scalar = protein_corr_pearson.apply(lambda x: np.sqrt(2*(1-abs(x))), axis = 0)\n\nn_clusters = 8\n\nk_means = cluster.KMeans(n_clusters = n_clusters)\nk_means.fit(sim_dist_scalar)\n    # apply the labels\ntrain_labels = k_means.labels_\nprint(train_labels)\n\n\nsim_dist_scalar['clusters_kmeans'] = train_labels\n\ndbscan = cluster.DBSCAN(eps = 1.99)\ndbscan.fit(sim_dist_scalar)\n# apply the labels\ntrain_labels = dbscan.labels_\nif -1 not in train_labels:\n    print(set(train_labels))\nnum_clusters_db = len(train_labels)\nn_clusters = max(n_clusters,num_clusters_db)\n    \nsim_dist_scalar['clusters_dbscan'] = train_labels\nsim_dist_scalar.head()\n\n#Plot \nd_dbscan = []\nd_kmeans = []\nfor i in range(n_clusters):\n    d_dbscan.append(sim_dist_scalar['clusters_dbscan'][sim_dist_scalar['clusters_dbscan']==i].size)\n    d_kmeans.append(sim_dist_scalar['clusters_kmeans'][sim_dist_scalar['clusters_kmeans']==i].size)\nsns.histplot(d_dbscan, bins=n_clusters, color='red')\nsns.histplot(d_kmeans, bins=n_clusters)","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:37.900169Z","iopub.status.idle":"2023-02-19T21:28:37.900847Z","shell.execute_reply.started":"2023-02-19T21:28:37.900609Z","shell.execute_reply":"2023-02-19T21:28:37.900630Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**MasP**","metadata":{}},{"cell_type":"code","source":"sns.clustermap(np.abs(d['MasP_pearson']),cmap='vlag', figsize=(30,30), xticklabels=protein_corr_pearson.columns, yticklabels=protein_corr_pearson.columns);","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:37.902023Z","iopub.status.idle":"2023-02-19T21:28:37.902505Z","shell.execute_reply.started":"2023-02-19T21:28:37.902350Z","shell.execute_reply":"2023-02-19T21:28:37.902366Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\nv = d['MasP_pearson'].values[np.triu_indices(len(d['MasP_pearson']),k=1)]\nprint(len(v), len(d['MasP_pearson'])*(len(d['MasP_pearson'])-1)/2 )\nplt.hist(v, bins = 1000)#  , color='k')\nplt.show()\nprint(pd.Series(v).describe() )","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:37.903400Z","iopub.status.idle":"2023-02-19T21:28:37.903906Z","shell.execute_reply.started":"2023-02-19T21:28:37.903724Z","shell.execute_reply":"2023-02-19T21:28:37.903740Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.clustermap(np.abs(d['MasP_spearman']),cmap='vlag', figsize=(30,30), xticklabels=protein_corr_spearman.columns, yticklabels=protein_corr_spearman.columns);","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:37.904792Z","iopub.status.idle":"2023-02-19T21:28:37.905299Z","shell.execute_reply.started":"2023-02-19T21:28:37.905145Z","shell.execute_reply":"2023-02-19T21:28:37.905161Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\nv = d['MasP_spearman'].values[np.triu_indices(len(d['MasP_spearman']),k=1)]\nprint(len(v), len(d['MasP_spearman'])*(len(d['MasP_spearman'])-1)/2 )\nplt.hist(v, bins = 1000)#  , color='k')\nplt.show()\nprint(pd.Series(v).describe() )","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:37.906211Z","iopub.status.idle":"2023-02-19T21:28:37.906688Z","shell.execute_reply.started":"2023-02-19T21:28:37.906534Z","shell.execute_reply":"2023-02-19T21:28:37.906550Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"protein_corr_pearson = d['MasP_pearson'].corr()\nsim_dist_scalar = protein_corr_pearson.apply(lambda x: np.sqrt(2*(1-abs(x))), axis = 0)\ng = igraph.Graph().Adjacency(sim_dist_scalar)\ng.to_undirected(mode = 'collapse')\npartition = la.find_partition(g, la.CPMVertexPartition,resolution_parameter = 0.997)\nigraph.plot(partition, bbox = (400,400)) ","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:37.907600Z","iopub.status.idle":"2023-02-19T21:28:37.908108Z","shell.execute_reply.started":"2023-02-19T21:28:37.907953Z","shell.execute_reply":"2023-02-19T21:28:37.907969Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"protein_corr_pearson = d['MasP_pearson'].corr()\nsim_dist_scalar = protein_corr_pearson.apply(lambda x: np.sqrt(2*(1-abs(x))), axis = 0)\n\nn_clusters = 8\n\nk_means = cluster.KMeans(n_clusters = n_clusters)\nk_means.fit(sim_dist_scalar)\n    # apply the labels\ntrain_labels = k_means.labels_\nprint(train_labels)\n\n\nsim_dist_scalar['clusters_kmeans'] = train_labels\n\ndbscan = cluster.DBSCAN(eps = 1.99)\ndbscan.fit(sim_dist_scalar)\n# apply the labels\ntrain_labels = dbscan.labels_\nif -1 not in train_labels:\n    print(set(train_labels))\nnum_clusters_db = len(train_labels)\nn_clusters = max(n_clusters,num_clusters_db)\n    \nsim_dist_scalar['clusters_dbscan'] = train_labels\nsim_dist_scalar.head()\n\n#Plot \nd_dbscan = []\nd_kmeans = []\nfor i in range(n_clusters):\n    d_dbscan.append(sim_dist_scalar['clusters_dbscan'][sim_dist_scalar['clusters_dbscan']==i].size)\n    d_kmeans.append(sim_dist_scalar['clusters_kmeans'][sim_dist_scalar['clusters_kmeans']==i].size)\nsns.histplot(d_dbscan, bins=n_clusters, color='red')\nsns.histplot(d_kmeans, bins=n_clusters)","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:37.909060Z","iopub.status.idle":"2023-02-19T21:28:37.909535Z","shell.execute_reply.started":"2023-02-19T21:28:37.909385Z","shell.execute_reply":"2023-02-19T21:28:37.909401Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**MkP**","metadata":{}},{"cell_type":"code","source":"sns.clustermap(np.abs(d['MkP_pearson']),cmap='vlag', figsize=(30,30), xticklabels=protein_corr_pearson.columns, yticklabels=protein_corr_pearson.columns);","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:37.910436Z","iopub.status.idle":"2023-02-19T21:28:37.910942Z","shell.execute_reply.started":"2023-02-19T21:28:37.910763Z","shell.execute_reply":"2023-02-19T21:28:37.910805Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\nv = d['MkP_pearson'].values[np.triu_indices(len(d['MkP_pearson']),k=1)]\nprint(len(v), len(d['MkP_pearson'])*(len(d['MkP_pearson'])-1)/2 )\nplt.hist(v, bins = 1000)#  , color='k')\nplt.show()\nprint(pd.Series(v).describe() )","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:37.911850Z","iopub.status.idle":"2023-02-19T21:28:37.912321Z","shell.execute_reply.started":"2023-02-19T21:28:37.912171Z","shell.execute_reply":"2023-02-19T21:28:37.912186Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.clustermap(np.abs(d['MkP_spearman']),cmap='vlag', figsize=(30,30), xticklabels=protein_corr_spearman.columns, yticklabels=protein_corr_spearman.columns);","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:37.913151Z","iopub.status.idle":"2023-02-19T21:28:37.913983Z","shell.execute_reply.started":"2023-02-19T21:28:37.913820Z","shell.execute_reply":"2023-02-19T21:28:37.913840Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\nv = d['MkP_spearman'].values[np.triu_indices(len(d['MkP_spearman']),k=1)]\nprint(len(v), len(d['MkP_spearman'])*(len(d['MkP_spearman'])-1)/2 )\nplt.hist(v, bins = 1000)#  , color='k')\nplt.show()\nprint(pd.Series(v).describe() )","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:37.914994Z","iopub.status.idle":"2023-02-19T21:28:37.915303Z","shell.execute_reply.started":"2023-02-19T21:28:37.915142Z","shell.execute_reply":"2023-02-19T21:28:37.915160Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"protein_corr_pearson = d['MkP_pearson'].corr()\nsim_dist_scalar = protein_corr_pearson.apply(lambda x: np.sqrt(2*(1-abs(x))), axis = 0)\ng = igraph.Graph().Adjacency(sim_dist_scalar)\ng.to_undirected(mode = 'collapse')\npartition = la.find_partition(g, la.CPMVertexPartition,resolution_parameter = 0.997)\nigraph.plot(partition, bbox = (400,400)) ","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:37.916214Z","iopub.status.idle":"2023-02-19T21:28:37.916501Z","shell.execute_reply.started":"2023-02-19T21:28:37.916363Z","shell.execute_reply":"2023-02-19T21:28:37.916377Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"protein_corr_pearson = d['MkP_pearson'].corr()\nsim_dist_scalar = protein_corr_pearson.apply(lambda x: np.sqrt(2*(1-abs(x))), axis = 0)\n\nn_clusters = 8\n\nk_means = cluster.KMeans(n_clusters = n_clusters)\nk_means.fit(sim_dist_scalar)\n    # apply the labels\ntrain_labels = k_means.labels_\nprint(train_labels)\n\n\nsim_dist_scalar['clusters_kmeans'] = train_labels\n\ndbscan = cluster.DBSCAN(eps = 1.99)\ndbscan.fit(sim_dist_scalar)\n# apply the labels\ntrain_labels = dbscan.labels_\nif -1 not in train_labels:\n    print(set(train_labels))\nnum_clusters_db = len(train_labels)\nn_clusters = max(n_clusters,num_clusters_db)\n    \nsim_dist_scalar['clusters_dbscan'] = train_labels\nsim_dist_scalar.head()\n\n#Plot \nd_dbscan = []\nd_kmeans = []\nfor i in range(n_clusters):\n    d_dbscan.append(sim_dist_scalar['clusters_dbscan'][sim_dist_scalar['clusters_dbscan']==i].size)\n    d_kmeans.append(sim_dist_scalar['clusters_kmeans'][sim_dist_scalar['clusters_kmeans']==i].size)\nsns.histplot(d_dbscan, bins=n_clusters, color='red')\nsns.histplot(d_kmeans, bins=n_clusters)","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:37.917255Z","iopub.status.idle":"2023-02-19T21:28:37.917541Z","shell.execute_reply.started":"2023-02-19T21:28:37.917403Z","shell.execute_reply":"2023-02-19T21:28:37.917418Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**BP**","metadata":{}},{"cell_type":"code","source":"sns.clustermap(np.abs(d['BP_pearson']),cmap='vlag', figsize=(30,30), xticklabels=protein_corr_pearson.columns, yticklabels=protein_corr_pearson.columns);","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:37.918933Z","iopub.status.idle":"2023-02-19T21:28:37.919239Z","shell.execute_reply.started":"2023-02-19T21:28:37.919081Z","shell.execute_reply":"2023-02-19T21:28:37.919095Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\nv = d['BP_pearson'].values[np.triu_indices(len(d['BP_pearson']),k=1)]\nprint(len(v), len(d['BP_pearson'])*(len(d['BP_pearson'])-1)/2 )\nplt.hist(v, bins = 1000)#  , color='k')\nplt.show()\nprint(pd.Series(v).describe() )","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:37.920412Z","iopub.status.idle":"2023-02-19T21:28:37.920693Z","shell.execute_reply.started":"2023-02-19T21:28:37.920555Z","shell.execute_reply":"2023-02-19T21:28:37.920568Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.clustermap(np.abs(d['BP_spearman']),cmap='vlag', figsize=(30,30), xticklabels=protein_corr_spearman.columns, yticklabels=protein_corr_spearman.columns);","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:37.921452Z","iopub.status.idle":"2023-02-19T21:28:37.921724Z","shell.execute_reply.started":"2023-02-19T21:28:37.921588Z","shell.execute_reply":"2023-02-19T21:28:37.921601Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\nv = d['BP_spearman'].values[np.triu_indices(len(d['BP_spearman']),k=1)]\nprint(len(v), len(d['BP_spearman'])*(len(d['BP_spearman'])-1)/2 )\nplt.hist(v, bins = 1000)#  , color='k')\nplt.show()\nprint(pd.Series(v).describe() )","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:37.922938Z","iopub.status.idle":"2023-02-19T21:28:37.923231Z","shell.execute_reply.started":"2023-02-19T21:28:37.923093Z","shell.execute_reply":"2023-02-19T21:28:37.923107Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"protein_corr_pearson = d['BP_pearson'].corr()\nsim_dist_scalar = protein_corr_pearson.apply(lambda x: np.sqrt(2*(1-abs(x))), axis = 0)\ng = igraph.Graph().Adjacency(sim_dist_scalar)\ng.to_undirected(mode = 'collapse')\npartition = la.find_partition(g, la.CPMVertexPartition,resolution_parameter = 0.997)\nigraph.plot(partition, bbox = (400,400)) ","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:37.924814Z","iopub.status.idle":"2023-02-19T21:28:37.925118Z","shell.execute_reply.started":"2023-02-19T21:28:37.924975Z","shell.execute_reply":"2023-02-19T21:28:37.924989Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"protein_corr_pearson = d['BP_pearson'].corr()\nsim_dist_scalar = protein_corr_pearson.apply(lambda x: np.sqrt(2*(1-abs(x))), axis = 0)\n\nn_clusters = 8\n\nk_means = cluster.KMeans(n_clusters = n_clusters)\nk_means.fit(sim_dist_scalar)\n    # apply the labels\ntrain_labels = k_means.labels_\nprint(train_labels)\n\n\nsim_dist_scalar['clusters_kmeans'] = train_labels\n\ndbscan = cluster.DBSCAN(eps = 1.99)\ndbscan.fit(sim_dist_scalar)\n# apply the labels\ntrain_labels = dbscan.labels_\nif -1 not in train_labels:\n    print(set(train_labels))\nnum_clusters_db = len(train_labels)\nn_clusters = max(n_clusters,num_clusters_db)\n    \nsim_dist_scalar['clusters_dbscan'] = train_labels\nsim_dist_scalar.head()\n\n#Plot \nd_dbscan = []\nd_kmeans = []\nfor i in range(n_clusters):\n    d_dbscan.append(sim_dist_scalar['clusters_dbscan'][sim_dist_scalar['clusters_dbscan']==i].size)\n    d_kmeans.append(sim_dist_scalar['clusters_kmeans'][sim_dist_scalar['clusters_kmeans']==i].size)\nsns.histplot(d_dbscan, bins=n_clusters, color='red')\nsns.histplot(d_kmeans, bins=n_clusters)","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:37.926113Z","iopub.status.idle":"2023-02-19T21:28:37.926409Z","shell.execute_reply.started":"2023-02-19T21:28:37.926258Z","shell.execute_reply":"2023-02-19T21:28:37.926272Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**MoP**","metadata":{}},{"cell_type":"code","source":"sns.clustermap(np.abs(d['MoP_pearson']),cmap='vlag', figsize=(30,30), xticklabels=protein_corr_pearson.columns, yticklabels=protein_corr_pearson.columns);","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:37.927405Z","iopub.status.idle":"2023-02-19T21:28:37.927686Z","shell.execute_reply.started":"2023-02-19T21:28:37.927547Z","shell.execute_reply":"2023-02-19T21:28:37.927560Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\nv = d['MoP_pearson'].values[np.triu_indices(len(d['MoP_pearson']),k=1)]\nprint(len(v), len(d['MoP_pearson'])*(len(d['MoP_pearson'])-1)/2 )\nplt.hist(v, bins = 1000)#  , color='k')\nplt.show()\nprint(pd.Series(v).describe() )","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:37.928599Z","iopub.status.idle":"2023-02-19T21:28:37.928914Z","shell.execute_reply.started":"2023-02-19T21:28:37.928741Z","shell.execute_reply":"2023-02-19T21:28:37.928754Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.clustermap(np.abs(d['MoP_spearman']),cmap='vlag', figsize=(30,30), xticklabels=protein_corr_spearman.columns, yticklabels=protein_corr_spearman.columns);","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:37.929888Z","iopub.status.idle":"2023-02-19T21:28:37.930169Z","shell.execute_reply.started":"2023-02-19T21:28:37.930029Z","shell.execute_reply":"2023-02-19T21:28:37.930044Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\nv = d['MoP_spearman'].values[np.triu_indices(len(d['MoP_spearman']),k=1)]\nprint(len(v), len(d['MoP_spearman'])*(len(d['MoP_spearman'])-1)/2 )\nplt.hist(v, bins = 1000)#  , color='k')\nplt.show()\nprint(pd.Series(v).describe() )","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:37.931220Z","iopub.status.idle":"2023-02-19T21:28:37.931503Z","shell.execute_reply.started":"2023-02-19T21:28:37.931365Z","shell.execute_reply":"2023-02-19T21:28:37.931379Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"protein_corr_pearson = d['MoP_pearson'].corr()\nsim_dist_scalar = protein_corr_pearson.apply(lambda x: np.sqrt(2*(1-abs(x))), axis = 0)\ng = igraph.Graph().Adjacency(sim_dist_scalar)\ng.to_undirected(mode = 'collapse')\npartition = la.find_partition(g, la.CPMVertexPartition,resolution_parameter = 0.997)\nigraph.plot(partition, bbox = (400,400)) ","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:37.935031Z","iopub.status.idle":"2023-02-19T21:28:37.935356Z","shell.execute_reply.started":"2023-02-19T21:28:37.935211Z","shell.execute_reply":"2023-02-19T21:28:37.935225Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"protein_corr_pearson = d['MoP_pearson'].corr()\nsim_dist_scalar = protein_corr_pearson.apply(lambda x: np.sqrt(2*(1-abs(x))), axis = 0)\n\nn_clusters = 8\n\nk_means = cluster.KMeans(n_clusters = n_clusters)\nk_means.fit(sim_dist_scalar)\n    # apply the labels\ntrain_labels = k_means.labels_\nprint(train_labels)\n\n\nsim_dist_scalar['clusters_kmeans'] = train_labels\n\ndbscan = cluster.DBSCAN(eps = 1.7)\ndbscan.fit(sim_dist_scalar)\n# apply the labels\ntrain_labels = dbscan.labels_\nif -1 not in train_labels:\n    print(set(train_labels))\nnum_clusters_db = len(train_labels)\nn_clusters = max(n_clusters,num_clusters_db)\n    \nsim_dist_scalar['clusters_dbscan'] = train_labels\nsim_dist_scalar.head()\n\n#Plot \nd_dbscan = []\nd_kmeans = []\nfor i in range(n_clusters):\n    d_dbscan.append(sim_dist_scalar['clusters_dbscan'][sim_dist_scalar['clusters_dbscan']==i].size)\n    d_kmeans.append(sim_dist_scalar['clusters_kmeans'][sim_dist_scalar['clusters_kmeans']==i].size)\nsns.histplot(d_dbscan, bins=n_clusters, color='red')\nsns.histplot(d_kmeans, bins=n_clusters)","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:37.936459Z","iopub.status.idle":"2023-02-19T21:28:37.936758Z","shell.execute_reply.started":"2023-02-19T21:28:37.936614Z","shell.execute_reply":"2023-02-19T21:28:37.936628Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Correlation matrix of a specific day","metadata":{}},{"cell_type":"code","source":"d_day = dict()\nfor day in df['day'].unique():\n    df_day = df.loc[lambda df1: df1['day'] == day, :]\n    df_day.drop('day', inplace=True, axis=1)\n    protein_corr_pearson = df_day.corr(method='pearson')\n    protein_corr_spearman = df_day.corr(method='spearman')\n    d_day[f'{day}_pearson']=protein_corr_pearson\n    d_day[f'{day}_spearman']=protein_corr_spearman","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:37.937540Z","iopub.status.idle":"2023-02-19T21:28:37.937862Z","shell.execute_reply.started":"2023-02-19T21:28:37.937678Z","shell.execute_reply":"2023-02-19T21:28:37.937692Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(d_day.keys())","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:37.938863Z","iopub.status.idle":"2023-02-19T21:28:37.939159Z","shell.execute_reply.started":"2023-02-19T21:28:37.939010Z","shell.execute_reply":"2023-02-19T21:28:37.939027Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"d_day['2_pearson'].head()","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:37.940210Z","iopub.status.idle":"2023-02-19T21:28:37.940501Z","shell.execute_reply.started":"2023-02-19T21:28:37.940359Z","shell.execute_reply":"2023-02-19T21:28:37.940373Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Day 2**","metadata":{}},{"cell_type":"code","source":"sns.clustermap(np.abs(d_day['2_pearson']),cmap='vlag', figsize=(30,30), xticklabels=protein_corr_pearson.columns, yticklabels=protein_corr_pearson.columns);","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:37.941839Z","iopub.status.idle":"2023-02-19T21:28:37.942127Z","shell.execute_reply.started":"2023-02-19T21:28:37.941986Z","shell.execute_reply":"2023-02-19T21:28:37.942001Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\nv = d_day['2_pearson'].values[np.triu_indices(len(d_day['2_pearson']),k=1)]\nprint(len(v), len(d_day['2_pearson'])*(len(d_day['2_pearson'])-1)/2 )\nplt.hist(v, bins = 1000)#  , color='k')\nplt.show()\nprint(pd.Series(v).describe() )","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:37.943388Z","iopub.status.idle":"2023-02-19T21:28:37.943695Z","shell.execute_reply.started":"2023-02-19T21:28:37.943549Z","shell.execute_reply":"2023-02-19T21:28:37.943563Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.clustermap(np.abs(d_day['2_spearman']),cmap='vlag', figsize=(30,30), xticklabels=protein_corr_spearman.columns, yticklabels=protein_corr_spearman.columns);","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:37.945641Z","iopub.status.idle":"2023-02-19T21:28:37.946077Z","shell.execute_reply.started":"2023-02-19T21:28:37.945905Z","shell.execute_reply":"2023-02-19T21:28:37.945923Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\nv = d_day['2_spearman'].values[np.triu_indices(len(d_day['2_spearman']),k=1)]\nprint(len(v), len(d_day['2_spearman'])*(len(d_day['2_spearman'])-1)/2 )\nplt.hist(v, bins = 1000)#  , color='k')\nplt.show()\nprint(pd.Series(v).describe() )","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:37.947667Z","iopub.status.idle":"2023-02-19T21:28:37.948121Z","shell.execute_reply.started":"2023-02-19T21:28:37.947946Z","shell.execute_reply":"2023-02-19T21:28:37.947964Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"protein_corr_pearson = d_day['2_pearson'].corr()\nsim_dist_scalar = protein_corr_pearson.apply(lambda x: np.sqrt(2*(1-abs(x))), axis = 0)\ng = igraph.Graph().Adjacency(sim_dist_scalar)\ng.to_undirected(mode = 'collapse')\npartition = la.find_partition(g, la.CPMVertexPartition,resolution_parameter = 0.997)\nigraph.plot(partition, bbox = (400,400)) ","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:37.949594Z","iopub.status.idle":"2023-02-19T21:28:37.949969Z","shell.execute_reply.started":"2023-02-19T21:28:37.949809Z","shell.execute_reply":"2023-02-19T21:28:37.949826Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"protein_corr_pearson = d_day['2_pearson'].corr()\nsim_dist_scalar = protein_corr_pearson.apply(lambda x: np.sqrt(2*(1-abs(x))), axis = 0)\n\nn_clusters = 2\n\nk_means = cluster.KMeans(n_clusters = n_clusters)\nk_means.fit(sim_dist_scalar)\n    # apply the labels\ntrain_labels = k_means.labels_\nprint(train_labels)\n\n\nsim_dist_scalar['clusters_kmeans'] = train_labels\n\ndbscan = cluster.DBSCAN(eps = 0.01)\ndbscan.fit(sim_dist_scalar)\n# apply the labels\ntrain_labels = dbscan.labels_\nif -1 not in train_labels:\n    print(set(train_labels))\nnum_clusters_db = len(train_labels)\nn_clusters = max(n_clusters,num_clusters_db)\n    \nsim_dist_scalar['clusters_dbscan'] = train_labels\nsim_dist_scalar.head()\n\n#Plot \nd_dbscan = []\nd_kmeans = []\nfor i in range(n_clusters):\n    d_dbscan.append(sim_dist_scalar['clusters_dbscan'][sim_dist_scalar['clusters_dbscan']==i].size)\n    d_kmeans.append(sim_dist_scalar['clusters_kmeans'][sim_dist_scalar['clusters_kmeans']==i].size)\nsns.histplot(d_dbscan, bins=n_clusters, color='red')\nsns.histplot(d_kmeans, bins=n_clusters)","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:37.951060Z","iopub.status.idle":"2023-02-19T21:28:37.952716Z","shell.execute_reply.started":"2023-02-19T21:28:37.952377Z","shell.execute_reply":"2023-02-19T21:28:37.952416Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Day 3**","metadata":{}},{"cell_type":"code","source":"sns.clustermap(np.abs(d_day['3_pearson']),cmap='vlag', figsize=(30,30), xticklabels=protein_corr_pearson.columns, yticklabels=protein_corr_pearson.columns);","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:37.954182Z","iopub.status.idle":"2023-02-19T21:28:37.954528Z","shell.execute_reply.started":"2023-02-19T21:28:37.954373Z","shell.execute_reply":"2023-02-19T21:28:37.954387Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\nv = d_day['3_pearson'].values[np.triu_indices(len(d_day['3_pearson']),k=1)]\nprint(len(v), len(d_day['3_pearson'])*(len(d_day['3_pearson'])-1)/2 )\nplt.hist(v, bins = 1000)#  , color='k')\nplt.show()\nprint(pd.Series(v).describe() )","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:37.955552Z","iopub.status.idle":"2023-02-19T21:28:37.955872Z","shell.execute_reply.started":"2023-02-19T21:28:37.955708Z","shell.execute_reply":"2023-02-19T21:28:37.955722Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.clustermap(np.abs(d_day['3_spearman']),cmap='vlag', figsize=(30,30), xticklabels=protein_corr_spearman.columns, yticklabels=protein_corr_spearman.columns);","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:37.957052Z","iopub.status.idle":"2023-02-19T21:28:37.957375Z","shell.execute_reply.started":"2023-02-19T21:28:37.957221Z","shell.execute_reply":"2023-02-19T21:28:37.957235Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\nv = d_day['3_spearman'].values[np.triu_indices(len(d_day['3_spearman']),k=1)]\nprint(len(v), len(d_day['3_spearman'])*(len(d_day['3_spearman'])-1)/2 )\nplt.hist(v, bins = 1000)#  , color='k')\nplt.show()\nprint(pd.Series(v).describe() )","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:37.958387Z","iopub.status.idle":"2023-02-19T21:28:37.958681Z","shell.execute_reply.started":"2023-02-19T21:28:37.958538Z","shell.execute_reply":"2023-02-19T21:28:37.958552Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"protein_corr_pearson = d_day['3_pearson'].corr()\nsim_dist_scalar = protein_corr_pearson.apply(lambda x: np.sqrt(2*(1-abs(x))), axis = 0)\ng = igraph.Graph().Adjacency(sim_dist_scalar)\ng.to_undirected(mode = 'collapse')\npartition = la.find_partition(g, la.CPMVertexPartition,resolution_parameter = 0.997)\nigraph.plot(partition, bbox = (400,400)) ","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:37.959440Z","iopub.status.idle":"2023-02-19T21:28:37.959713Z","shell.execute_reply.started":"2023-02-19T21:28:37.959574Z","shell.execute_reply":"2023-02-19T21:28:37.959587Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Day 4**","metadata":{}},{"cell_type":"code","source":"sns.clustermap(np.abs(d_day['4_pearson']),cmap='vlag', figsize=(30,30), xticklabels=protein_corr_pearson.columns, yticklabels=protein_corr_pearson.columns);","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:37.960633Z","iopub.status.idle":"2023-02-19T21:28:37.960988Z","shell.execute_reply.started":"2023-02-19T21:28:37.960804Z","shell.execute_reply":"2023-02-19T21:28:37.960821Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\nv = d_day['4_pearson'].values[np.triu_indices(len(d_day['4_pearson']),k=1)]\nprint(len(v), len(d_day['4_pearson'])*(len(d_day['4_pearson'])-1)/2 )\nplt.hist(v, bins = 1000)#  , color='k')\nplt.show()\nprint(pd.Series(v).describe() )","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:37.961915Z","iopub.status.idle":"2023-02-19T21:28:37.962208Z","shell.execute_reply.started":"2023-02-19T21:28:37.962061Z","shell.execute_reply":"2023-02-19T21:28:37.962074Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.clustermap(np.abs(d_day['4_spearman']),cmap='vlag', figsize=(30,30), xticklabels=protein_corr_spearman.columns, yticklabels=protein_corr_spearman.columns);","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:37.963039Z","iopub.status.idle":"2023-02-19T21:28:37.963330Z","shell.execute_reply.started":"2023-02-19T21:28:37.963180Z","shell.execute_reply":"2023-02-19T21:28:37.963200Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\nv = d_day['4_spearman'].values[np.triu_indices(len(d_day['4_spearman']),k=1)]\nprint(len(v), len(d_day['4_spearman'])*(len(d_day['4_spearman'])-1)/2 )\nplt.hist(v, bins = 1000)#  , color='k')\nplt.show()\nprint(pd.Series(v).describe() )","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:37.964316Z","iopub.status.idle":"2023-02-19T21:28:37.964593Z","shell.execute_reply.started":"2023-02-19T21:28:37.964455Z","shell.execute_reply":"2023-02-19T21:28:37.964469Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"protein_corr_pearson = d_day['4_pearson'].corr()\nsim_dist_scalar = protein_corr_pearson.apply(lambda x: np.sqrt(2*(1-abs(x))), axis = 0)\ng = igraph.Graph().Adjacency(sim_dist_scalar)\ng.to_undirected(mode = 'collapse')\npartition = la.find_partition(g, la.CPMVertexPartition,resolution_parameter = 0.997)\nigraph.plot(partition, bbox = (400,400)) ","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:37.965983Z","iopub.status.idle":"2023-02-19T21:28:37.966275Z","shell.execute_reply.started":"2023-02-19T21:28:37.966130Z","shell.execute_reply":"2023-02-19T21:28:37.966144Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Save to csv","metadata":{}},{"cell_type":"code","source":"for cell_type in d:\n    d[cell_type].to_csv(f'Cell_type_{cell_type}.csv')   # cell type\n\nfor day in d_day:\n    d_day[day].to_csv(f'Day_{day}.csv')   # day \n    \nprotein_corr_spearman.to_csv('full_df_spearman.csv')   # full dataframe\nprotein_corr_pearson.to_csv('full_df_pearson.csv')","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:37.967114Z","iopub.status.idle":"2023-02-19T21:28:37.967393Z","shell.execute_reply.started":"2023-02-19T21:28:37.967254Z","shell.execute_reply":"2023-02-19T21:28:37.967267Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pd.read_csv('/kaggle/working/Day_4_pearson.csv').head()","metadata":{"execution":{"iopub.status.busy":"2023-02-19T21:28:37.968326Z","iopub.status.idle":"2023-02-19T21:28:37.968594Z","shell.execute_reply.started":"2023-02-19T21:28:37.968463Z","shell.execute_reply":"2023-02-19T21:28:37.968476Z"},"trusted":true},"execution_count":null,"outputs":[]}]}