{"cells":[{"metadata":{},"cell_type":"markdown","source":"### Acknowledgement\n\nIn this notebook, we will use the implementation of Stratified Group K-fold cross validation presented in [this kernel](https://www.kaggle.com/jakubwasikowski/stratified-group-k-fold-cross-validation) by [Jakub Wąsikowski](https://www.kaggle.com/jakubwasikowski).","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### The purpose of this notebook\n\nOne of the challenges that we are facing in this competition is finding the best way to implement a K-fold cross-validation. On one hand, we have a very imbalanced dataset, so some sort of stratification seems to be in order here. On the other hand, we know that some of the patients present in the dataset are represented by multiple images and it is probably a good idea not to mix images corresponding to the same `patient_id` in our training and validation sets. The latter can be achieved by using sklearn Group K-Fold split. Unfortunately, sklearn does not have an implementation of Stratified Group K-Folds, so we have to make one from scratch. Fortunately, one such implementation of Stratified Group K-Folds was already developed in the PetFinder competition (see the \"Acknowledgement\" section), so we can just take it and use it here. This is what we do in this notebook.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Setting the number of folds\n\nTo illustrate the main idea we will be using an example of 5 folds. The number of folds can be easily adjusted by changing the parameter `NFOLDS` below.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"NFOLDS=5","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Loading libraries","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import random\nimport numpy as np\nimport pandas as pd\nfrom pathlib import Path\nfrom collections import Counter, defaultdict","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Function for implementing Stratified Group k-Fold split","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def stratified_group_k_fold(X, y, groups, k, seed=None):\n    labels_num = np.max(y) + 1\n    y_counts_per_group = defaultdict(lambda: np.zeros(labels_num))\n    y_distr = Counter()\n    for label, g in zip(y, groups):\n        y_counts_per_group[g][label] += 1\n        y_distr[label] += 1\n\n    y_counts_per_fold = defaultdict(lambda: np.zeros(labels_num))\n    groups_per_fold = defaultdict(set)\n\n    def eval_y_counts_per_fold(y_counts, fold):\n        y_counts_per_fold[fold] += y_counts\n        std_per_label = []\n        for label in range(labels_num):\n            label_std = np.std([y_counts_per_fold[i][label] / y_distr[label] for i in range(k)])\n            std_per_label.append(label_std)\n        y_counts_per_fold[fold] -= y_counts\n        return np.mean(std_per_label)\n    \n    groups_and_y_counts = list(y_counts_per_group.items())\n    random.Random(seed).shuffle(groups_and_y_counts)\n\n    for g, y_counts in sorted(groups_and_y_counts, key=lambda x: -np.std(x[1])):\n        best_fold = None\n        min_eval = None\n        for i in range(k):\n            fold_eval = eval_y_counts_per_fold(y_counts, i)\n            if min_eval is None or fold_eval < min_eval:\n                min_eval = fold_eval\n                best_fold = i\n        y_counts_per_fold[best_fold] += y_counts\n        groups_per_fold[best_fold].add(g)\n\n    all_groups = set(groups)\n    for i in range(k):\n        train_groups = all_groups - groups_per_fold[i]\n        test_groups = groups_per_fold[i]\n\n        train_indices = [i for i, g in enumerate(groups) if g in train_groups]\n        test_indices = [i for i, g in enumerate(groups) if g in test_groups]\n\n        yield train_indices, test_indices","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Loading data","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"PATH=Path('/kaggle/input/siim-isic-melanoma-classification/')\ntrain=pd.read_csv(PATH/'train.csv')\nprint(f\"The shape of the `train` is {train.shape}.\\n\")\nprint(f\"The columns present in `train` are {train.columns.values}.\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_x = train['image_name']\ntrain_y = train['target'].values\ngroups = np.array(train['patient_id'].values)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_distribution(y_vals):\n        y_distr = Counter(y_vals)\n        y_vals_sum = sum(y_distr.values())\n        return [f'{y_distr[i] / y_vals_sum:.5%}' for i in range(np.max(y_vals) + 1)]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"distrs = [get_distribution(train_y)]\nindex = ['training set']\n\nfor fold_ind, (dev_ind, val_ind) in enumerate(stratified_group_k_fold(train_x, train_y, \n                                                                      groups, k=NFOLDS), 1):\n    dev_y, val_y = train_y[dev_ind], train_y[val_ind]\n    dev_groups, val_groups = groups[dev_ind], groups[val_ind]\n    \n    # making sure that train and validation group do not overlap:\n    assert len(set(dev_groups) & set(val_groups)) == 0\n    \n    distrs.append(get_distribution(dev_y))\n    index.append(f'development set - fold {fold_ind}')\n    distrs.append(get_distribution(val_y))\n    index.append(f'validation set - fold {fold_ind}')\n\ndisplay('Distribution per class:')\npd.DataFrame(distrs, index=index, columns=[f'Label {l}' for l in range(np.max(train_y) + 1)])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As you can see, it seems to be working pretty well -- each fold has almost the same percentages of 0's and 1's and the unique `patient_id` values do not overlap between different folds.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Saving fold indicies\n\nNow, let's save the fold indicies into NumPy arrays. ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"for fold_ind, (dev_ind, val_ind) in enumerate(stratified_group_k_fold(train_x, train_y, \n                                                                      groups, k=NFOLDS), 1):\n    \n    dev_ind=np.array(dev_ind)\n    val_ind=np.array(val_ind)\n    \n    np.save(f\"train_idx_fold_{fold_ind}.npy\", dev_ind)\n    np.save(f\"val_idx_fold_{fold_ind}.npy\", val_ind)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Final remark\n\nI can think of two possible ways how these folds can be utilized: we can use them to do cross-validation on jpeg files or we can make a new set of tfrecord files based on these folds. In the latter case, the number of folds should probably be ajusted. ","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}