{"cells":[{"metadata":{},"cell_type":"markdown","source":"See leak explanation in [this post](https://www.kaggle.com/c/recursion-cellular-image-classification/discussion/102905).\n\nRPE-03 and HUVEC-07 have the same pattern of controls, but not only… also the treatments are in the same pattern, only the plates are rotated.\n\nNow, there is an experiment in the test set that has the same pattern of controls… HUVEC-18. See images below to demonstrate what I said.\n\nDoes HUVEC-18 also have the same pattern of treatments in some plate rotation?"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport os\nimport matplotlib as mpl\nfrom matplotlib import pyplot as plt\nimport sys\n\ndf_train = pd.read_csv(\"../input/train.csv\")\ndf_train_contr = pd.read_csv(\"../input/train_controls.csv\").drop(labels=\"well_type\", axis=1)\ndf_test = pd.read_csv(\"../input/test.csv\")\ndf_test_contr = pd.read_csv(\"../input/test_controls.csv\").drop(labels=\"well_type\", axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train.head(2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train_contr.head(2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# extract row and column number from well\ndf_train[\"row\"] = df_train[\"well\"].apply(lambda x: ord(x[0].lower()) - 96)\ndf_train[\"col\"] = df_train[\"well\"].apply(lambda x: int(x[1:]))\ndf_train_contr[\"row\"] = df_train_contr[\"well\"].apply(lambda x: ord(x[0].lower()) - 96)\ndf_train_contr[\"col\"] = df_train_contr[\"well\"].apply(lambda x: int(x[1:]))\ndf_test_contr[\"row\"] = df_test_contr[\"well\"].apply(lambda x: ord(x[0].lower()) - 96)\ndf_test_contr[\"col\"] = df_test_contr[\"well\"].apply(lambda x: int(x[1:]))\ndf_train.sample(n=5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# create ordered list of treatment sirnas with group1+group2+...+group4\nsirnas = []\nexp = \"HEPG2-03\" # select experiment that has all sirnas\ndf_exp = df_train.groupby(\"experiment\").get_group(exp)\nfor plate, df_exp_pl in df_exp.groupby(\"plate\"):\n    ss = sorted(df_exp_pl[\"sirna\"].unique())\n    sirnas += ss\n    print(\"Plate {} has {} sirnas.\".format(plate, df_exp_pl[\"sirna\"].nunique()))\n    print(\"   First 10 in ordered group:\", ss[:10])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# write sirna groups to a dataframe to save as output\npd.DataFrame(data={\"sirna\" : sirnas, \n                   \"group\" : [i for i in range(1,5) for j in range(277)]}).to_csv(\"sirna_groups.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# assign unique colors to treatment sirnas\nsirnas_colormaps = {1 : \"Blues\", 2 : \"Greens\", 3 : \"Purples\", 4 : \"Reds\"}\ncolors = []\nfor plate in [1,2,3,4]:\n    colors += [mpl.cm.get_cmap(sirnas_colormaps[plate])(i) for i in np.linspace(0., 1., 277)]\n    \nsirnas_colors_dict = dict(zip(sirnas, colors))\ndf_train[\"color\"] = df_train[\"sirna\"].map(sirnas_colors_dict)\ndf_train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# create ordered list of control sirnas\ncontr_sirnas = sorted(df_train_contr[\"sirna\"].unique())\nprint(f\"{len(contr_sirnas)} control sirnas.\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# assign unique colors to control sirnas\ncolors = [mpl.cm.get_cmap(\"hsv\")(i) for i in np.linspace(0., 1., 31)]\ncontr_sirnas_colors_dict = dict(zip(contr_sirnas, colors))\ndf_train_contr[\"color\"] = df_train_contr[\"sirna\"].map(contr_sirnas_colors_dict)\ndf_test_contr[\"color\"] = df_test_contr[\"sirna\"].map(contr_sirnas_colors_dict)\ndf_train_contr.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Find frequency of group assignment patterns."},{"metadata":{"trusted":true},"cell_type":"code","source":"df_pattern = pd.DataFrame(index=df_train[\"experiment\"].unique())\ndf_pattern[\"pattern\"] = \"\"\nfor exp, df_exp in df_train.groupby(\"experiment\"):\n    pattern = \"\"\n    for plate, df_exp_pl in df_exp.groupby(\"plate\"):\n        sirna_sample = df_exp_pl[\"sirna\"].values[0]\n        group_sirna_sample = sirnas.index(sirna_sample) // 277 + 1\n        pattern += str(group_sirna_sample)\n    df_pattern.loc[exp, \"pattern\"] = pattern\n\ndf_pattern.reset_index(inplace=True)\ndf_pattern.columns = [\"experiment\", \"pattern\"]\ndf_pattern","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_pattern.groupby(\"pattern\").size()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Colorful visualization\nBelow the visualization. Experiments are grouped by their pattern.\n\nEach group of 277 treatment siRNAs is visualized using the same nuances of a color (blue, green, purple, red).\nControl siRNAs are always colored with the same nuance of the jet colormap.\n\n### Can you find a pattern in the location of the controls??\nNote: the hot red dot always close to the origin of the plot is the negative control (1138). Sometimes there's more than one negative per plate."},{"metadata":{"trusted":true},"cell_type":"code","source":"for pattern, df_pattern_pattern in df_pattern.groupby(\"pattern\"):\n    print(\"====================================================================\")\n    print(f\"Pattern {pattern}\\n\")\n    experiments = df_pattern_pattern[\"experiment\"]\n    for exp in experiments:\n        df_exp = df_train.groupby(\"experiment\").get_group(exp)\n        fig, axs = plt.subplots(1, 8, figsize=(16,3))\n        for plate, df_exp_pl in df_exp.groupby(\"plate\"):\n            if plate == 1:\n                axs[plate-1].set_ylabel(exp)\n            axs[plate-1].scatter(df_exp_pl[\"row\"], df_exp_pl[\"col\"], color=df_exp_pl[\"color\"], s=30)\n            axs[plate-1].set_title(f\"PL. {plate} treat\")\n            df_exp_pl_contr = df_train_contr[(df_train_contr[\"experiment\"]==exp) & (df_train_contr[\"plate\"]==plate)]\n            axs[plate-1+4].scatter(df_exp_pl_contr[\"row\"], df_exp_pl_contr[\"col\"], color=df_exp_pl_contr[\"color\"], s=30)\n            axs[plate-1+4].set_title(f\"PL. {plate} contr\")\n        plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Check if there's any pattern in the controls."},{"metadata":{"trusted":true},"cell_type":"code","source":"# controls seem to appear in a random order in the same \"scheme\" of wells\n# sometimes negative controls appear in a well that is normally dedicated to a treatment (it's a failed treatment)\n# pick \"real\" control wells from a plate that seems not to have failed treatments or anomalies\ncontrol_wells = df_train_contr.loc[(df_train_contr[\"experiment\"]==\"RPE-06\") & (df_train_contr[\"plate\"]==1), \"well\"].values\ncontrol_wells","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"experiment_plate_contr_list = []\npattern_contr_list = []\nfor exp_pl, df_exp_pl in df_train_contr[df_train_contr[\"well\"].isin(control_wells)].groupby([\"experiment\", \"plate\"]):\n    experiment_plate_contr_list.append(exp_pl)\n    # df_exp_pl is already sorted by row and column\n    pattern_contr_list.append(\"_\".join(df_exp_pl[\"sirna\"].astype(\"str\").values.tolist()))\ndf_pattern_contr = pd.DataFrame(data={\"experiment_plate\" : experiment_plate_contr_list,\n                                      \"pattern\" : pattern_contr_list})\ndf_pattern_contr.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_pattern_contr.groupby(\"pattern\").size().sort_values(ascending=False).head(5)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The first two patterns are in fact almost the same (note that 1119 in the first pattern becomes a negative 1138 in the second pattern).\n1138_1108_1109_1110_1111_1112_1113_1114_1115_1116_1117_1118_**1119**_1120_1121_1122_1123_1124_1125_1126_1127_1128_1129_1130_1131_1132_1133_1134_1135_1136_1137\n1138_1108_1109_1110_1111_1112_1113_1114_1115_1116_1117_1118_**1138**_1120_1121_1122_1123_1124_1125_1126_1127_1128_1129_1130_1131_1132_1133_1134_1135_1136_1137\n\nI inspected the remaining patterns by eye but couldn't find any hint of a scheme or something."},{"metadata":{},"cell_type":"markdown","source":"Below the experiments and plates with the same control pattern. All plates of HUVEC-07 and RPE-03 have the same \"scheme\" of control siRNAs."},{"metadata":{"trusted":true},"cell_type":"code","source":"p1 = \"1138_1108_1109_1110_1111_1112_1113_1114_1115_1116_1117_1118_1119_1120_1121_1122_1123_1124_1125_1126_1127_1128_1129_1130_1131_1132_1133_1134_1135_1136_1137\"\np2 = \"1138_1108_1109_1110_1111_1112_1113_1114_1115_1116_1117_1118_1138_1120_1121_1122_1123_1124_1125_1126_1127_1128_1129_1130_1131_1132_1133_1134_1135_1136_1137\"\ndf_pattern_contr[df_pattern_contr[\"pattern\"].isin([p1,p2])]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### What happens in the test set?"},{"metadata":{"trusted":true},"cell_type":"code","source":"# visualize controls in test\nfor exp, df_exp in df_test_contr.groupby(\"experiment\"):\n    fig, axs = plt.subplots(1, 4, figsize=(8,3))\n    for plate, df_exp_pl in df_exp.groupby(\"plate\"):\n        if plate == 1:\n            axs[plate-1].set_ylabel(exp)\n        axs[plate-1].scatter(df_exp_pl[\"row\"], df_exp_pl[\"col\"], color=df_exp_pl[\"color\"], s=30)\n        axs[plate-1].set_title(f\"PL. {plate} contr\")\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"experiment_plate_contr_test_list = []\npattern_contr_test_list = []\nfor exp_pl, df_exp_pl in df_test_contr[df_test_contr[\"well\"].isin(control_wells)].groupby([\"experiment\", \"plate\"]):\n    experiment_plate_contr_test_list.append(exp_pl)\n    # df_exp_pl is already sorted by row and column\n    pattern_contr_test_list.append(\"_\".join(df_exp_pl[\"sirna\"].astype(\"str\").values.tolist()))\ndf_pattern_contr_test = pd.DataFrame(data={\"experiment_plate\" : experiment_plate_contr_test_list,\n                                           \"pattern\" : pattern_contr_test_list})\ndf_pattern_contr_test.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_pattern_contr_test.groupby(\"pattern\").size().sort_values(ascending=False).head(5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"p1 = \"1138_1108_1109_1110_1111_1112_1113_1114_1115_1116_1117_1118_1119_1120_1121_1122_1123_1124_1125_1126_1127_1128_1129_1130_1131_1132_1133_1134_1135_1136_1137\"\ndf_pattern_contr_test[df_pattern_contr_test[\"pattern\"].isin([p1])]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"HUVEC-18 has also the same \"scheme\" of control siRNAs."},{"metadata":{},"cell_type":"markdown","source":"### Attempt to find the same scheme of treatment siRNAs in train set"},{"metadata":{"trusted":true},"cell_type":"code","source":"experiment_plate_treat_list = []\npattern_treat_list = []\nfor exp_pl, df_exp_pl in df_train.groupby([\"experiment\", \"plate\"]):\n    experiment_plate_treat_list.append(exp_pl)\n    # df_exp_pl is already sorted by row and column\n    pattern_treat_list.append(\"_\".join(df_exp_pl[\"sirna\"].astype(\"str\").values.tolist()))\ndf_pattern_treat = pd.DataFrame(data={\"experiment_plate\" : experiment_plate_treat_list,\n                                      \"pattern\" : pattern_treat_list})\ndf_pattern_treat.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_pattern_treat.groupby(\"pattern\").size().sort_values(ascending=False).head(5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from difflib import SequenceMatcher\n\ndef similar(a, b):\n    return SequenceMatcher(None, a, b).ratio()\n\nm = np.zeros((len(pattern_treat_list), len(pattern_treat_list)))\nfor i, p1 in enumerate(pattern_treat_list):\n    for j, p2 in enumerate(pattern_treat_list):\n        if i<j:\n            s = similar(p1, p2)\n            m[i,j] = s\n            if s>0.1:\n                print(f\"- Match found at {experiment_plate_treat_list[i]} and {experiment_plate_treat_list[j]}\")\n                print(f\"     p1 = {p1}\")\n                print(f\"     p2 = {p2}\")\n                print(f\"     simil = {s}\")","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}