{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Note: Be Careful to Use This Notebook...\n\n* The strategy used in this split is NOT TESTED. It could cause LEAKS. I recommend you to test before using it to your experiment.\n\nFor detail, refer to the below comment:\n\n[1] https://www.kaggle.com/tatamikenn/balanced-fold-splitting-algorithm/comments#1627417","metadata":{}},{"cell_type":"markdown","source":"# What is the Notebook for?\n\nNaively applying GroupKFold results in biased fold sets -- the #COTS in the frame, #frames per fold much differs in different folds.\n\nTo handle this unbalances, I came up with the idea splitting the sequences into sub-sequences. \nSplitting into short banch of frames, the statistics of each folds are expected to be more balanced.\n\nOne problem is, how to split the sequence. Spliting naively may cause a serious validation reaks.\nSo I hypothesized a simple rule: \"**the same individual of COTSs are only appeares in the consecutive annotated frames.**\" That is, no same individual of COTSs are appeared in two frame sets which are sepalated by non-annotated cuts.\n\nI conducted below studies in the notebook:\n\n* visualize how inballanced when splitting folds in a naiive GroupKFold\n* introduce a simple sequence split algorithm and cut sequence into tiny sub-sequences\n* allocating sub-sequences to each fold using a simple round-robin algorithm\n\n## Limitation\n\n* currently each sub-sequences only includes annotated frames (i.e. no backgroud frame are considered)\n\n## Change Note\n\n* background image is added to the validation data (foreground + background = 20:80)","metadata":{}},{"cell_type":"code","source":"!pip install nb-black > /dev/null\n%load_ext lab_black","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-12-24T00:08:53.685299Z","iopub.execute_input":"2021-12-24T00:08:53.685646Z","iopub.status.idle":"2021-12-24T00:09:04.578206Z","shell.execute_reply.started":"2021-12-24T00:08:53.685554Z","shell.execute_reply":"2021-12-24T00:09:04.57729Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nfrom itertools import cycle\nimport matplotlib.pylab as plt\nfrom matplotlib.patches import Rectangle\n\nplt.style.use(\"ggplot\")\ncolor_pal = plt.rcParams[\"axes.prop_cycle\"].by_key()[\"color\"]\ncolor_cycle = cycle(plt.rcParams[\"axes.prop_cycle\"].by_key()[\"color\"])","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2021-12-24T00:09:04.580512Z","iopub.execute_input":"2021-12-24T00:09:04.580736Z","iopub.status.idle":"2021-12-24T00:09:04.596446Z","shell.execute_reply.started":"2021-12-24T00:09:04.58071Z","shell.execute_reply":"2021-12-24T00:09:04.595367Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = pd.read_csv(\"../input/tensorflow-great-barrier-reef/train.csv\")\ntest = pd.read_csv(\"../input/tensorflow-great-barrier-reef/test.csv\")\nss = pd.read_csv(\"../input/tensorflow-great-barrier-reef/example_sample_submission.csv\")\n\ntrain.shape, test.shape","metadata":{"execution":{"iopub.status.busy":"2021-12-24T00:09:04.598026Z","iopub.execute_input":"2021-12-24T00:09:04.599036Z","iopub.status.idle":"2021-12-24T00:09:04.684881Z","shell.execute_reply.started":"2021-12-24T00:09:04.598979Z","shell.execute_reply":"2021-12-24T00:09:04.684311Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Add Additional Columns\n\n* `n_annotaions`: Number of annotations in the frame\n* `video_sequence`: `video_id` + '_' + `sequence`","metadata":{}},{"cell_type":"code","source":"train[\"sum_cots\"] = train[\"annotations\"].apply(lambda x: len(eval(x)))\ntrain[\"video_sequence\"] = (\n    train[\"video_id\"].astype(\"str\") + \"_\" + train[\"sequence\"].astype(\"str\")\n)","metadata":{"execution":{"iopub.status.busy":"2021-12-24T00:09:04.686793Z","iopub.execute_input":"2021-12-24T00:09:04.687044Z","iopub.status.idle":"2021-12-24T00:09:04.94519Z","shell.execute_reply.started":"2021-12-24T00:09:04.687015Z","shell.execute_reply":"2021-12-24T00:09:04.944491Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# How Naive GroupKFold Creates Unbalanced Folds","metadata":{}},{"cell_type":"code","source":"def plot_folds(df):\n    df = df.copy()\n    plt.style.use(\"ggplot\")\n    df = df.groupby(\"fold_id\").agg(\n        sum_cots=(\"sum_cots\", \"sum\"), duration=(\"fold_id\", \"count\")\n    )\n    df[\"mean_cots\"] = df.sum_cots / df.duration\n    fig, axs = plt.subplots(1, 3, figsize=(15, 4))\n    df.sum_cots.plot(kind=\"bar\", ax=axs[0])\n    df.duration.plot(kind=\"bar\", ax=axs[1])\n    df.mean_cots.plot(kind=\"bar\", ax=axs[2])\n    axs[0].set_title(\"#COTS\")\n    axs[1].set_title(\"#Frames\")\n    axs[2].set_title(\"#COTS/frame\")\n    return df","metadata":{"execution":{"iopub.status.busy":"2021-12-24T00:09:04.946575Z","iopub.execute_input":"2021-12-24T00:09:04.946959Z","iopub.status.idle":"2021-12-24T00:09:04.970285Z","shell.execute_reply.started":"2021-12-24T00:09:04.946912Z","shell.execute_reply":"2021-12-24T00:09:04.969437Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.model_selection import GroupKFold\n\n\ndef allocate_group_k_fold(df, n_split):\n    df = df.copy()\n    kf = GroupKFold(n_splits=n_split)\n    df[\"fold_id\"] = -1\n    for fold, (train_idx, val_idx) in enumerate(kf.split(df, groups=df.video_sequence)):\n        df.loc[val_idx, \"fold_id\"] = fold\n    return df","metadata":{"execution":{"iopub.status.busy":"2021-12-24T00:09:04.971636Z","iopub.execute_input":"2021-12-24T00:09:04.971843Z","iopub.status.idle":"2021-12-24T00:09:05.986181Z","shell.execute_reply.started":"2021-12-24T00:09:04.971818Z","shell.execute_reply":"2021-12-24T00:09:05.985253Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"n_split = 5\ndf = train.copy()\n# df = df.query(\"sum_cots > 0\")  # select only annotated frames\ndf.reset_index(inplace=True)\ngroup_k_alloc_df = allocate_group_k_fold(df, n_split)\ndf = plot_folds(group_k_alloc_df)\nplt.suptitle(\"Visualization of Statistics of each Folds - GroupKFold\", fontsize=16)\ndf, df.std()","metadata":{"execution":{"iopub.status.busy":"2021-12-24T00:17:57.479334Z","iopub.execute_input":"2021-12-24T00:17:57.479625Z","iopub.status.idle":"2021-12-24T00:17:57.994056Z","shell.execute_reply.started":"2021-12-24T00:17:57.479597Z","shell.execute_reply":"2021-12-24T00:17:57.993263Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In the avobe graph, frame number are almost perfectly balanced, but **the number of COTSs are poorly balanced**.","metadata":{}},{"cell_type":"markdown","source":"# Fold Splitted by Sequence","metadata":{}},{"cell_type":"code","source":"import matplotlib.patches as patches\n\n\ndef visualize_fold(df, sequence_df, label):\n    fig, axs = plt.subplots(3, 1, figsize=(18, 10))\n    max_cots = df[\"sum_cots\"].max()\n    for (video_id, sequence), d in df.groupby([\"video_id\", \"sequence\"]):\n        ax = axs[video_id].plot(\n            d[\"video_frame\"], d[\"sum_cots\"] / max_cots, c=\"black\", linewidth=0.5\n        )\n\n    fold_colors = [\"red\", \"blue\", \"green\", \"orange\", \"purple\"]\n    for item in sequence_df.itertuples():\n        ax = axs[item.video_id]\n        rect = patches.Rectangle(\n            (item.start_frame, 0),\n            item.duration,\n            1,\n            facecolor=fold_colors[item.fold_id % len(fold_colors)],\n            alpha=0.3,\n        )\n        ax.add_patch(rect)\n        middle = (item.start_frame + item.end_frame) // 2\n        ax.text(middle - 40, 0.9, f\"{item.fold_id}\", c=\"gray\")\n\n    [axs[idx].set_title(f\"Video: {idx}\") for idx in range(3)]\n    plt.suptitle(label, fontsize=16)\n    plt.tight_layout()","metadata":{"execution":{"iopub.status.busy":"2021-12-24T00:18:11.394822Z","iopub.execute_input":"2021-12-24T00:18:11.395248Z","iopub.status.idle":"2021-12-24T00:18:11.451851Z","shell.execute_reply.started":"2021-12-24T00:18:11.395186Z","shell.execute_reply":"2021-12-24T00:18:11.451122Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sequence_df = group_k_alloc_df.groupby(\"sequence\").agg(\n    start_frame=(\"video_frame\", \"min\"),\n    end_frame=(\"video_frame\", \"max\"),\n    duration=(\"video_frame\", lambda x: x.max() - x.min() + 1),\n    video_id=(\"video_id\", \"max\"),\n    fold_id=(\"fold_id\", \"min\"),\n)\nsequence_df","metadata":{"execution":{"iopub.status.busy":"2021-12-24T00:18:11.713162Z","iopub.execute_input":"2021-12-24T00:18:11.71426Z","iopub.status.idle":"2021-12-24T00:18:11.759298Z","shell.execute_reply.started":"2021-12-24T00:18:11.71419Z","shell.execute_reply":"2021-12-24T00:18:11.758578Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# df = train.query('sum_cots > 0')\ndf = train.copy()\nvisualize_fold(df, sequence_df, label=\"Folds Splitted by Sequence\")","metadata":{"execution":{"iopub.status.busy":"2021-12-24T00:18:28.834654Z","iopub.execute_input":"2021-12-24T00:18:28.835294Z","iopub.status.idle":"2021-12-24T00:18:29.564475Z","shell.execute_reply.started":"2021-12-24T00:18:28.835257Z","shell.execute_reply":"2021-12-24T00:18:29.563424Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"You can see how each sequences are distributed to folds in an un-balanced way.","metadata":{}},{"cell_type":"markdown","source":"# Spliting Sequence into Tiny Sub-Sequences\n\nHypothesis:\n\n* two consecutive annotated frames may include the same individual COTS.\n* two frames in the same sequence don't include the same individual of COTS when these are sepalated by non-annotated frames. That means all the individuals of COTS are traced (annotated) consecutively.\n\nSo I introduced a simple sequence-spliting algorithm:\n1. for each sequence, add the label 'annotated' to each frames which means more than one annotation(s) are inclued.\n2. split the sequence into parts which have consecutive 'annotated' frames","metadata":{}},{"cell_type":"code","source":"df = train.copy()\n\n# make annotated flag\ndf[\"annotated\"] = df[\"sum_cots\"].apply(lambda x: min(x, 1))\n\ndfs = []\n\n# calculate non-annotated frame sub_sequence\nfor i, d in df.groupby(\"video_id\"):\n    ad = d.groupby((d[\"annotated\"] != d[\"annotated\"].shift()).cumsum(), as_index=False)[\n        [\"video_frame\", \"annotated\", \"sum_cots\"]\n    ].agg(\n        annotated=(\"annotated\", \"first\"),\n        start_frame=(\"video_frame\", \"first\"),\n        end_frame=(\"video_frame\", \"last\"),\n        sum_cots=(\"sum_cots\", \"sum\"),\n        mean_cots=(\"sum_cots\", \"mean\"),\n    )\n    ad[\"video_id\"] = i\n    dfs.append(ad)\n\ndf_annot = pd.concat(dfs)\ndf_annot[\"duration\"] = df_annot[\"end_frame\"] - df_annot[\"start_frame\"] + 1\nsub_sequence = df_annot.query(\"annotated == 1\")\n\nsub_sequence.reset_index(drop=True)\n\nlast_sub_sequence_end = -1\nsub_sequence_id = 0\nsub_sequence_ids = []\ncontinuous = False\nprev_video_id = 0\nfor idx, (\n    annotated,\n    start_frame,\n    end_frame,\n    sum_cots,\n    mean_cots,\n    video_id,\n    duration,\n) in sub_sequence.iterrows():\n    sub_sequence_ids.append(sub_sequence_id)\n    last_sub_sequence_end = end_frame\n    prev_video_id = video_id\n    if not (prev_video_id == video_id and last_sub_sequence_end + 1 == start_frame):\n        sub_sequence_id += 1\n\nsub_sequence.loc[:, \"sub_sequence_id\"] = sub_sequence_ids\nsub_sequence.drop(\"annotated\", axis=1, inplace=True)\nsub_sequence.reset_index(drop=True, inplace=True)\nsub_sequence","metadata":{"execution":{"iopub.status.busy":"2021-12-24T00:18:46.759235Z","iopub.execute_input":"2021-12-24T00:18:46.759895Z","iopub.status.idle":"2021-12-24T00:18:46.884395Z","shell.execute_reply.started":"2021-12-24T00:18:46.759854Z","shell.execute_reply":"2021-12-24T00:18:46.883578Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Visualize Sub-Sequences","metadata":{}},{"cell_type":"code","source":"import matplotlib.patches as mpatches\n\n\nfig, axes = plt.subplots(3, 1, figsize=(15, 8), sharex=True, sharey=True)\naxes = axes.ravel()\nmax_annotation = df[\"sum_cots\"].max()\nfor i, d in df.groupby([\"video_id\", \"sequence\"]):\n    video_id = d[\"video_id\"].values[0]\n    ax = axes[video_id]\n    d.set_index(\"video_frame\")[\"sum_cots\"].apply(lambda x: x / max_annotation).plot(\n        ax=ax, c=\"black\", linewidth=0.5\n    )\n\n    ax.set_title(f\"Video ID: {video_id}\")\n\n\n# visualize clippable interval\nfor (\n    annotated,\n    start_frame,\n    end_frame,\n    sum_cots,\n    mean_cots,\n    video_id,\n    duration,\n    sub_sequence_id,\n) in sub_sequence.itertuples():\n    ax = axes[int(video_id)]\n    rect = mpatches.Rectangle(\n        (start_frame, 0), duration, 1, alpha=0.3, facecolor=\"gray\"\n    )\n    ax.add_patch(rect)\n\nfig.suptitle(\"Sub-Sequences Visualized\", fontsize=15)\nplt.tight_layout()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-12-24T00:18:51.930282Z","iopub.execute_input":"2021-12-24T00:18:51.93107Z","iopub.status.idle":"2021-12-24T00:18:52.912438Z","shell.execute_reply.started":"2021-12-24T00:18:51.931028Z","shell.execute_reply":"2021-12-24T00:18:52.911546Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Fold Split Algorithm (Sub-Sequence Allocation)\n\nTo balance the statistics (like number of COTS, frame length etc.) of folds, I used simle round-robin algorithm to allocate sub-sequences to each folds:\n\n1. sort sub_sequence in descending order for total COTS in the sub_sequences\n2. allocate sub_sequences for each folds in order","metadata":{}},{"cell_type":"code","source":"def allocate_fold(df, n_split, key=\"sum_cots\"):\n    df = df.copy()\n    assert key in df.columns\n    df.sort_values(key, ascending=False, inplace=True)\n    df[\"fold_id\"] = -1\n    for fold_id in range(n_split):\n        index = df.iloc[fold_id::n_split].index\n        df.loc[index, \"fold_id\"] = fold_id\n\n    return df","metadata":{"execution":{"iopub.status.busy":"2021-12-10T05:00:15.550997Z","iopub.execute_input":"2021-12-10T05:00:15.551257Z","iopub.status.idle":"2021-12-10T05:00:15.557987Z","shell.execute_reply.started":"2021-12-10T05:00:15.551228Z","shell.execute_reply":"2021-12-10T05:00:15.556991Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def plot_folds_sub_sequence(df):\n    df = df.copy()\n    plt.style.use('ggplot')\n    df = df.groupby('fold_id').agg(\n        sum_cots=('sum_cots', 'sum'), duration=('duration', 'sum'))\n    df['mean_cots'] = df.sum_cots / df.duration\n    fig, axs = plt.subplots(1, 3, figsize=(15, 5))\n    df.sum_cots.plot(kind='bar', ax=axs[0])\n    df.duration.plot(kind='bar', ax=axs[1])\n    df.mean_cots.plot(kind='bar', ax=axs[2])\n    axs[0].set_title('#COTS')\n    axs[1].set_title('#Frames')\n    axs[2].set_title('#COTS/frame')\n    \n    return df","metadata":{"execution":{"iopub.status.busy":"2021-12-10T05:00:15.559198Z","iopub.execute_input":"2021-12-10T05:00:15.559427Z","iopub.status.idle":"2021-12-10T05:00:15.573612Z","shell.execute_reply.started":"2021-12-10T05:00:15.5594Z","shell.execute_reply":"2021-12-10T05:00:15.571994Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df = allocate_fold(sub_sequence, n_split=5)\ndf = plot_folds_sub_sequence(df)\nplt.suptitle('Statistics of Folds by Round-Robin Algorithm', fontsize=16)\nplt.tight_layout()\ndf, df.agg('std')","metadata":{"execution":{"iopub.status.busy":"2021-12-10T05:06:13.345201Z","iopub.execute_input":"2021-12-10T05:06:13.345859Z","iopub.status.idle":"2021-12-10T05:06:13.998748Z","shell.execute_reply.started":"2021-12-10T05:06:13.345809Z","shell.execute_reply":"2021-12-10T05:06:13.997557Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"By splitting into subsequences and applying subsequence-allocation algorighm, the standard deviations of #COTS and #COTS/frame are decreased, which means the unbalance is mitigated.\n\n* std(#COTS): 955 -> 787\n* std(#COTS/frame): 0.89 -> 0.42","metadata":{}},{"cell_type":"markdown","source":"## Find Optimal Number of Split\n\nSince the algorithm introduced before is dependent on the number of splits, I searched the most balanced split number.","metadata":{}},{"cell_type":"code","source":"def calc_split_statistics(sub_sequence, n_split):\n    df = allocate_fold(sub_sequence, n_split)\n    df = df.groupby('fold_id').agg(\n    sum_cots=('sum_cots', 'sum'), duration=('duration', 'sum'))\n    df['mean_cots'] = df.sum_cots / df.duration\n    return df\n\n\ndeviations = {\"sum_cots\": [], \"duration\": [], \"mean_cots\": []}\nn_splits = np.arange(3, 11)\nfor i in n_splits:\n    data = calc_split_statistics(sub_sequence, i).std()\n    for key in data.keys():\n        deviations[key].append(data[key])\n\nfig, ax = plt.subplots(1, 3, figsize=(15, 4))\nfor i, key in enumerate(deviations.keys()):\n    ax[i].plot(n_splits, deviations[key], label=key)\n    ax[i].set_ylim(bottom=0)\n    ax[i].set_title(key)\n    ax[i].set_xlabel(\"n_splits\")\nplt.suptitle(\"Standard Deviation vs. #Splits\", fontsize=15)","metadata":{"execution":{"iopub.status.busy":"2021-12-10T05:00:17.168335Z","iopub.execute_input":"2021-12-10T05:00:17.169191Z","iopub.status.idle":"2021-12-10T05:00:17.824872Z","shell.execute_reply.started":"2021-12-10T05:00:17.16914Z","shell.execute_reply":"2021-12-10T05:00:17.823764Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In terms of minimizing standard deviation of #COTS, the best balanced setting is `n_split == 4`.","metadata":{}},{"cell_type":"markdown","source":"# Visualization of Folds in Video Frames","metadata":{}},{"cell_type":"code","source":"n_split=5\ndf = allocate_fold(sub_sequence, n_split)\nvisualize_fold(train, df, 'Folds Splitted by Sub-Sequence')","metadata":{"execution":{"iopub.status.busy":"2021-12-10T05:14:01.80249Z","iopub.execute_input":"2021-12-10T05:14:01.802798Z","iopub.status.idle":"2021-12-10T05:14:02.982935Z","shell.execute_reply.started":"2021-12-10T05:14:01.802765Z","shell.execute_reply":"2021-12-10T05:14:02.981849Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now you san see how sub-sequences are distributed to each folds in more balanced way.","metadata":{}},{"cell_type":"markdown","source":"# Merge Fold Column to the Train Metadata ","metadata":{}},{"cell_type":"code","source":"# concatenate sub_sequence table and train table\n\nn_split = 5\ndf = train.copy()\ndfs = []\nalloc_df = allocate_fold(sub_sequence, n_split)\nbb = alloc_df.copy()\nfor video_id, d in df.groupby(\"video_id\"):\n    a = d[\"video_frame\"].values\n    b = bb.query(\"video_id == @video_id\").drop(\"video_id\", axis=1)\n    sub_sequence_low = b[\"start_frame\"].values\n    sub_sequence_high = b[\"end_frame\"].values\n\n    i, j = np.where((a[:, None] >= sub_sequence_low) & (a[:, None] <= sub_sequence_high))\n    dfs.append(\n        pd.DataFrame(\n            np.column_stack([d.values[i], b.values[j]]),\n            columns=d.columns.append(b.columns),\n        )\n    )\n\ndf = pd.concat(dfs)\ndf = df.loc[:, ~df.columns.duplicated()] # remove duplicated columns\n\nfor column in alloc_df.columns:\n    if column != \"mean_cots\":\n        df[column] = df[column].astype(int)\n        \ndf.to_csv(\"train_split_balanced_fold.csv\", index=False)\ndf[:3]","metadata":{"execution":{"iopub.status.busy":"2021-12-10T05:00:19.74622Z","iopub.execute_input":"2021-12-10T05:00:19.746461Z","iopub.status.idle":"2021-12-10T05:00:19.881113Z","shell.execute_reply.started":"2021-12-10T05:00:19.746434Z","shell.execute_reply":"2021-12-10T05:00:19.880275Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"group_k_alloc_df.to_csv(\"train_split_group_k_fold.csv\", index=False)","metadata":{},"execution_count":null,"outputs":[]}]}