{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Control Wells Present on All Plates\n\nAn interesting factor in this competition is the presence of control experiments in each plate. Experimental controls provide a way of calibrating the outcome of an experiment against expected variance between experiments. The goal of this competition is to create a model that is robust to a high level of variance between plates. I think making intelligent use of controls will be an important factor in creating a high accuracy model.\n\nThere's one little wrinkle though. Not every control is present in every plate. The code in this notebook finds the control experiments that are common across every plate in both the training and testing set. The code also gathers image channel stats for the common control wells for each plate, which could be a useful feature."},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\nimport os\nfrom pathlib import Path\nimport numpy as np","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"path = Path('../input')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def _load_dataset(base_path, dataset, include_controls=True):\n    df =  pd.read_csv(os.path.join(base_path, dataset + '.csv'))\n    if include_controls:\n        controls = pd.read_csv(\n            os.path.join(base_path, dataset + '_controls.csv'))\n        df['well_type'] = 'treatment'\n        df = pd.concat([controls, df], sort=True)\n    df['cell_type'] = df.experiment.str.split(\"-\").apply(lambda a: a[0])\n    df['dataset'] = dataset\n    dfs = []\n    for site in (1, 2):\n        df = df.copy()\n        df['site'] = site\n        dfs.append(df)\n    res = pd.concat(dfs).sort_values(\n        by=['id_code', 'site']).set_index('id_code')\n    return res","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def combine_metadata(base_path=path,\n                     include_controls=True):\n    df = pd.concat(\n        [\n            _load_dataset(\n                base_path, dataset, include_controls=include_controls)\n            for dataset in ['test', 'train']\n        ],\n        sort=True)\n    return df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"md = combine_metadata()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"neg_ctrls = []\npos_ctrls = []\nfor experiment in md.experiment.unique():\n    for plate in range(1, 5):\n        negs = set(md[(md.experiment == experiment) & \n                      (md.plate == plate) & (md.well_type == 'negative_control')].sirna)\n        pos = set(md[(md.experiment == experiment) & \n                      (md.plate == plate) & (md.well_type == 'positive_control')].sirna)\n        neg_ctrls.append(negs)\n        pos_ctrls.append(pos)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"positive_controls = set.intersection(*pos_ctrls)\nnegative_controls = set.intersection(*neg_ctrls)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Positive control sirnas present in every plate. Note that the same sirna control might be present in a different well between two plates. The important thing to keep track of is the sirna itself."},{"metadata":{"trusted":true},"cell_type":"code","source":"positive_controls","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The same negative control (no sirna) is present in all plates"},{"metadata":{"trusted":true},"cell_type":"code","source":"negative_controls","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Overall there are 18 controls (positive + negative) present in all plates"},{"metadata":{"trusted":true},"cell_type":"code","source":"controls = list(negative_controls) + list(positive_controls)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"stats = pd.read_csv(path/'pixel_stats.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"md.reset_index(inplace=True)\nmd['code_site'] = md.apply(lambda row: row['id_code'] + '_' + str(row['site']), axis=1)\nstats['code_site'] = stats.apply(lambda row: row['id_code'] + '_' + str(row['site']), axis=1)\n\nmerged = pd.merge(md, stats[['mean', 'std', 'median', 'min', 'max', 'code_site']], on='code_site')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Control well stats for each plate are saved as a numpy array"},{"metadata":{"trusted":true},"cell_type":"code","source":"control_stats = []\ncontrol_ids = []\n\nfor experiment in merged.experiment.unique():\n    for plate in range(1, 5):\n        plate_data = []\n        # looping through controls for each plate is slow, but ensures all controls are stored in the same order\n        for rna in controls:\n            data = merged[(merged.experiment == experiment) &\n                          (merged.plate == plate) &\n                          (merged.sirna == rna)][['mean', 'std', 'median', 'min', 'max']].values\n            \n            plate_data.append(data)\n            \n        data = np.concatenate(plate_data)\n        control_stats.append(data)\n        control_ids.append(experiment + '_' + str(plate))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The data is stored in a 216x5 matrix for each plate. (18 controls/plate * 2 sites/control * 6 channels/site image = 216)"},{"metadata":{"trusted":true},"cell_type":"code","source":"control_stats[0].shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"control_ids[0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}