{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# import libraries\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# import data visualization\nimport matplotlib.pyplot as plt\nfrom matplotlib.font_manager import FontProperties\n\n# import StratifiedKFold\nfrom sklearn.model_selection import StratifiedKFold","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Bengali GroupShuffleSplit\n\nBengaliGroupShuffleSplit is a custom made validation strategy that random shuffles groups while making sure that graphemes in validation set may be constructed from root-consonant-vowel components present in training set.\n\nAs competition hosts clarified [here](https://www.kaggle.com/c/bengaliai-cv19/discussion/123002#707668) that test includes graphemes that are not present in the train but are made of train root-consonant-vowel components. Furthermore, in data decription hosts specify that [\"the goal of competition is recognition of grapheme components rather than on recognizing whole graphemes.\"](https://www.kaggle.com/c/bengaliai-cv19/data)"},{"metadata":{},"cell_type":"markdown","source":"## Original GroupShuffleSplit\nfrom [sklearn](https://scikit-learn.org/stable/modules/cross_validation.html)\nThe GroupShuffleSplit iterator behaves as a combination of ShuffleSplit and LeavePGroupsOut, and generates a sequence of randomized partitions in which a subset of groups are held out for each split.\n\n![](https://scikit-learn.org/stable/_images/sphx_glr_plot_cv_indices_008.png)"},{"metadata":{"trusted":true},"cell_type":"code","source":"# setup the input data folder\nDATA_PATH = '../input/bengaliai-cv19/'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# load the dataframes with labels\ntrain_labels = pd.read_csv(DATA_PATH + 'train.csv')\ntest_labels = pd.read_csv(DATA_PATH + 'test.csv')\nclass_map = pd.read_csv(DATA_PATH + 'class_map.csv')\nsample_submission = pd.read_csv(DATA_PATH + 'sample_submission.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"NUM_FOLDS = 5","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Observation 1: Groups are well balanced.\nEvery grapheme is a Root-Vowel-Consonant combination. Every combination corresponds to a group of roughly 150 images. So image groups are very well balanced; GroupShuffleSplit staregy should work well without rebalancing groups."},{"metadata":{"trusted":true},"cell_type":"code","source":"#Lets find total number of combinations:\ncombinations = train_labels.groupby(by=['vowel_diacritic','consonant_diacritic', 'grapheme_root'])\\\n            .count().reset_index().drop(['grapheme'],axis=1)\ncombinations.rename(columns={'image_id': 'image_count'}, inplace=True)\ncombinations.image_count.describe()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Observation 2: Special case of Single Combination Roots\nGroupShuffleSplit cannot be applied to roots that are present in a single combination (*protected_roots*). If all such combinations are excluded from train, model wont be able to recognise the root coorrectly as it has never seen its label. Alternatively, if we exclude the combinations from validations, we will not know know how well model performs for these components.\n\nFor *protected_roots* we will use **graphemewise or rootwise** [StratifiedKFold](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.StratifiedKFold.html#sklearn.model_selection.StratifiedKFold), so that we have *protected_roots* both in train and validation set.\n\n![](https://scikit-learn.org/stable/_images/sphx_glr_plot_cv_indices_003.png)"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Add 'fold' column to train_labels set to store fold info:\ntrain_labels['fold'] = np.nan","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Separate protected_roots:\nimg_roots = combinations.groupby(by=['grapheme_root']).count()\nprotected_roots = img_roots[img_roots.image_count == 1].index.values\nprotected_roots","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Rootwise StratifiedKFold for images containing protected_roots:\nprotected_imgs = train_labels[train_labels['grapheme_root'].isin(protected_roots)]['grapheme_root']\n\nX = protected_imgs.index.values\ny = protected_imgs.values\nskf = StratifiedKFold(n_splits=NUM_FOLDS, shuffle=True)\nskf.get_n_splits(X, y)\nprint(skf)\n\ni_fold = 0\nfor train_index, test_index in skf.split(X, y):\n    train_labels.loc[X[test_index],'fold'] = i_fold\n    i_fold += 1","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_labels.groupby(by=['fold']).count()['image_id']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# BengaliGroupShuffleSplit\nA handmade *BengaliGroupShuffleSplit* is needed because GroupShuffleSplit does not check if combinations in validation set can be composed of components given in the train set."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Drop combinations with protected_roots from combinations pool:\ncombinations = combinations[~combinations['grapheme_root'].isin(protected_roots)]\nindex_pool = set(combinations.index.values)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Suppose we want to do 5 folds, so we need to save this many combinations for validation:\nCOMBS_PER_FOLD = combinations.shape[0]//NUM_FOLDS\nCOMBS_PER_FOLD","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Split all combinations by folds, while checking that all elements in validation set are present in train set.\nimport random\ncombinations['fold'] = 0\nunused_idx = index_pool.copy()\n\nfor i in range(NUM_FOLDS):\n    counter = COMBS_PER_FOLD\n    valid_set = set()\n    protected_set = set()\n    loop_pool = index_pool.copy()\n    while (counter and unused_idx):\n        # get a random grapheme candidate from pool of candidate graphemes\n        candidate = random.sample(unused_idx.intersection(loop_pool), 1)[0]\n        loop_pool.remove(candidate)\n        # check if all components of a candidate grapheme are present in the pool\n        if (combinations.loc[loop_pool]['consonant_diacritic'].isin([combinations['consonant_diacritic'][candidate]]).any() and \n            combinations.loc[loop_pool]['vowel_diacritic'].isin([combinations['vowel_diacritic'][candidate]]).any() and\n            combinations.loc[loop_pool]['grapheme_root'].isin([combinations['grapheme_root'][candidate]]).any()):\n                \n                # if TRUE add candidate grapheme to validation set\n                valid_set.add(candidate)\n                counter -= 1\n        else:\n            # if any of the components are missing from candidates pool, \"leave it in training set\"\n            protected_set.add(candidate)\n            \n    if i==4: valid_set |= unused_idx\n    unused_idx -= valid_set\n    \n    combinations.loc[valid_set,'fold'] = i\n\n    print('Loop: {} Number of unused idx: {} Len protected_Set: {}'\\\n          .format(i, len(unused_idx), len(protected_set)))\n\n#Slow. Got to be an elegant way to do this faster.","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Make sure folds have equal number of groups:\ncombinations.groupby(by=['fold']).count()['image_count']\n\n## 3 extra combinations in fold 4 could have been randomly added to different folds.","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\nfor i in range(combinations.shape[0]):\n    i_fold = combinations.iloc[i,:]['fold']\n    i_root = combinations.iloc[i,:]['grapheme_root']\n    i_vowel = combinations.iloc[i,:]['vowel_diacritic']\n    i_consonant = combinations.iloc[i,:]['consonant_diacritic']\n\n    i_mask = (train_labels['grapheme_root'] == i_root) &\\\n            (train_labels['vowel_diacritic'] == i_vowel) &\\\n            (train_labels['consonant_diacritic'] == i_consonant) \n    \n    train_labels.loc[train_labels[i_mask].index,'fold'] = i_fold","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# take a look at folds\ntrain_labels.groupby(by=['fold']).count()['image_id']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_labels[train_labels['fold'].isna()].shape[0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#output\ntrain_labels['fold'] = train_labels['fold'].astype('int')\ntrain_labels.to_csv('train_BengaliGroupShuffle.csv', index = False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}