{"cells":[{"metadata":{},"cell_type":"markdown","source":"### Stratified K-fold for this competition\n\nSharing the k-fold splits could be nice to team up later.  If team mates have been using the same k-fold splits at the time of the team merge, they can easily do a second level model.\n\nSo I encourage you to use this k-fold split. Perhaps, when you find a team, you are surprised that your teammates were using it too!\n\nRegarding the dataset, it is quite imbalance.  On one hand, we have only 79 ETT-Abnormal cases. In the other hand, we have 21324 CVC-Normal cases\n\nThe goals of this notebook is to split the data, satisfiying:\n\n    1. Group by patient. Usually, if you have 5 photos of an item in a catalog, you have to ensure that they must be toguether in the same fold.  I suppose it must be appliable to this case.\n\n\t2. All the classes are represented in the same number accross folds\n    \n\t\t· specially less represented classes\n        \n    3. All the annotations are represented in the same number accross folds (there are labeld images in train set without annotations)\n        \n    4. Have the same number of observations in each fold\n    \n\n    \nThis script ensures to satisfy the first point (group by patient), and performs a heuristic effort to optimize the other points.\n\nIt has been perfomed in an iterative way. Before asigning an observation to a fold, it tries and evaluates how much does it hurts these assignment for each fold, to use the less bad option.\n\nSince row order matters to this heuristic, it could have been improved by repeating the iterative algorithm N times, and random shuffling the rows on each execution, to get pick the best k-folds split.\n\nAlso, it could be probably better to use a genetic algorithm instead of an iterative heuristic.\n\n\nNotebook Version 3:  Takes into account the number of annotations for each fold and labeled class.  \n\nFuture improvements:\n  - give more importance for more difficult classes to be preddicted, according to confussion matrix\n    "},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"NUM_FOLDS=5\n\nimport pandas as pd\nfrom tqdm import tqdm\nimport numpy as np\n\n\n\n\ndata_dir='../input/ranzcr-clip-catheter-line-classification/'\n\ntarget_col = [\n    'ETT - Abnormal',\n    'ETT - Borderline',\n    'ETT - Normal',\n    'NGT - Abnormal',\n    'NGT - Borderline',\n    'NGT - Incompletely Imaged',\n    'NGT - Normal',\n    'CVC - Abnormal',\n    'CVC - Borderline',\n    'CVC - Normal',\n    'Swan Ganz Catheter Present'\n]\n   \ndef annotation_col_name(x):\n    x=x[1]\n    return \"annotation-\"+\"\".join([c for c in x if c.upper()==c and c!=' '])\nannotations_df = pd.read_csv(data_dir + 'train_annotations.csv', usecols=['StudyInstanceUID', 'label'])\nannotations_df['ones']=1\nannotations_df=pd.pivot_table(annotations_df, values=['ones'], index=['StudyInstanceUID'], columns=['label'], aggfunc=np.sum).fillna(0)\nannotations_df.columns=[annotation_col_name(col_name) for col_name in annotations_df.columns]\n\n   \ntrain_gt_df = pd.read_csv(data_dir + '/train.csv')\ntrain_gt_df=train_gt_df.merge(annotations_df, on='StudyInstanceUID', how='left' ).fillna(0)\n\ntarget_col+=[x for x in annotations_df.columns]\n\nnum_target = len(target_col)\nkfold_df=train_gt_df.copy().drop(['StudyInstanceUID'],axis=1)\n\n\nprint(len(train_gt_df.PatientID.unique()), \"patients\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Loss function to evaluate K-fold splits\n\nJust using the rule of the thumb, a little bit of this, a little bit of that, and manually tunning some parameters\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"patient_kfold_df=kfold_df.groupby('PatientID').sum()\n\n\npriority_df=patient_kfold_df.sum().to_frame().reset_index().rename(columns={'index':'label',0:'score',})\npriority_df['score']=[(s*4 if 'annotation' in l else s )for l,s in zip(priority_df['label'],priority_df['score'])]\npriority_df.sort_values(by='score',inplace=True)\n\nlabel_priority=list(priority_df['label'])\nlabel_importance=[(0.1+patient_kfold_df.sum().max()-x)**2 for x in priority_df['score']]\nlabel_importance[0]=3*label_importance[0]\n\n\n\n\npatient_kfold_df=patient_kfold_df.sort_values(by=label_priority, ascending=False)\n\ndef kfold_quality(folds): #(greater is worse)\n    patient_kfold_df['fold']=folds+([-1]*(len(patient_kfold_df)-len(folds)))\n    scores=patient_kfold_df[:len(folds)][['fold']+target_col].groupby('fold').sum()\n    \n    if scores.shape[0]<NUM_FOLDS:\n        scores.reset_index(inplace=True)\n        for f in range(NUM_FOLDS):\n            if f not in scores.index:\n                scores.loc[len(scores)] = 0\n                scores.loc[len(scores)-1,'fold'] = f\n        scores.set_index('fold', drop=True,inplace=True)\n\n    # penalizes not having the same number of observations per fold\n    score=scores.sum(axis=1).std()\n    \n    # penalizes not having the same number of observations per fold and class\n    # using weights to give more importance to unbalances in less represented classes\n    for label, importance in zip (label_priority,label_importance):\n        scores[label]*=importance\n        score+=scores[label].max()-scores[label].min()\n    return score","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Iterative heuristic"},{"metadata":{"trusted":true},"cell_type":"code","source":"current_kfold=[]\nfor _ in tqdm(range(len(patient_kfold_df))):\n    fold_scores=[(f, kfold_quality(current_kfold+[f])) for f in range(NUM_FOLDS)]\n    best_fold=sorted(fold_scores, key=lambda x: x[1])[0][0]\n    current_kfold.append(best_fold)\npatient_kfold_df['fold']=current_kfold","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Save results\n\npatient_kfold_df=patient_kfold_df.reset_index()\n\ntrain_gt_df=train_gt_df.merge(patient_kfold_df[['PatientID', 'fold']], on='PatientID', how='left')\n\npatient_kfold_df[['PatientID', 'fold']].to_csv(f'stratified_{NUM_FOLDS}_folds_by_patient_id.csv', index=False)\ntrain_gt_df[['StudyInstanceUID', 'fold']].to_csv(f'stratified_{NUM_FOLDS}_folds.csv', index=False)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Results"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"train_gt_df[['fold']+target_col].groupby('fold').sum().transpose()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Images per fold:"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"images_per_fold=train_gt_df[['fold', 'StudyInstanceUID']].groupby('fold').count().rename(columns={'StudyInstanceUID':'Num. images'})\nimages_per_fold","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Patients per fold:"},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":false,"trusted":true},"cell_type":"code","source":"patients_per_fold=train_gt_df[['fold', 'PatientID']].groupby('fold').agg({'PatientID': 'nunique'}).rename(columns={'PatientID':'Num. patients'})\npatients_per_fold","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}