{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Freesound Audio Shaking 2019 \n\n\"drawing\""},{"metadata":{},"cell_type":"markdown","source":"In this kernel, I will do a naive experiment to estimate the scale of LB shakeup. \nWe can explore that using `Out Of Fold (OOF)` prediction on train_curated data, \nand estimate discrepancy between public LB score and private LB score with 2 LB split cases. \nDue to annotation noises on train_noisy data, here we will use only train_curated data.\n\n---\n\nShort summary \n( For more detail, please read to the end ) \n\n1. In 2 cases (random LB split and multi-label stratified LB split), \nlwlrap discrepancy between public LB and private LB is about **1% (1 sigma)** or so.\n\n2. But discrepancy will depend on models. \n"},{"metadata":{},"cell_type":"markdown","source":"##
1. Load Library
"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport warnings\nimport matplotlib\nimport matplotlib.pyplot as plt\n\nfrom iterstrat.ml_stratifiers import MultilabelStratifiedKFold\n\n%matplotlib inline\nmatplotlib.style.use('ggplot')\nwarnings.filterwarnings(\"ignore\", category=FutureWarning) ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"##
2. Load Data
\n\nWe will use train_curated data for this experiment and 2 OOF predictions. \n( NOTE : To get OOF predictions, I used 5-fold CV on a simple CNN. )"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"train = pd.read_csv('../input/freesound-audio-tagging-2019/train_curated.csv')\ntest = pd.read_csv('../input/freesound-audio-tagging-2019/sample_submission.csv')\nsub1 = pd.read_csv('../input/fs2019/oof_sub1.csv')\nsub2 = pd.read_csv('../input/fs2019/oof_sub2.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for c in test.columns[1:]:\n cc = c.replace('(', '\\(').replace(')', '\\)')\n train.loc[:, c] = train['labels'].str.contains(cc).astype(int)\n if (train.loc[:, c] > 1).sum():\n raise Exception(\n 'label key \"{}\" are duplicated in train_cur !'.format(c))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train = train.query('fname != \"1d44b0bd.wav\"') # remove silent audio\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub1.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub2.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"##
3. Evaluation Metric : lwlrap
\n\nhttps://colab.research.google.com/drive/1AgPdhSp7ttY18O3fEoHOQKlt_3HJDLi8"},{"metadata":{"trusted":true},"cell_type":"code","source":"def _one_sample_positive_class_precisions(scores, truth):\n \"\"\" Calculate precisions for each true class for a single sample.\n This metric is MAP@K like.\n\n Args:\n scores:\n np.array of (num_classes,) giving the individual classifier scores.\n truth:\n np.array of (num_classes,) bools indicating which classes are true.\n\n Returns:\n pos_class_indices:\n np.array of indices of the true classes for this sample.\n pos_class_precisions:\n np.array of precisions corresponding to each of those classes.\n \"\"\"\n num_classes = scores.shape[0]\n pos_class_indices = np.flatnonzero(truth > 0)\n\n # Only calculate precisions if there are some true classes.\n if not len(pos_class_indices):\n return pos_class_indices, np.zeros(0)\n\n # Retrieval list of classes for this sample.\n retrieved_classes = np.argsort(scores)[::-1]\n\n # class_rankings[top_scoring_class_index] == 0 etc.\n class_rankings = np.zeros(num_classes, dtype=np.int)\n class_rankings[retrieved_classes] = range(num_classes)\n\n # Which of these is a true label?\n retrieved_class_true = np.zeros(num_classes, dtype=np.bool)\n retrieved_class_true[class_rankings[pos_class_indices]] = True\n\n # Num hits for every truncated retrieval list.\n retrieved_cumulative_hits = np.cumsum(retrieved_class_true)\n\n # Precision of retrieval list truncated at each hit, in order of pos_labels.\n precision_at_hits = (\n retrieved_cumulative_hits[class_rankings[pos_class_indices]] /\n (1 + class_rankings[pos_class_indices].astype(np.float))\n )\n return pos_class_indices, precision_at_hits\n\ndef calculate_per_class_lwlrap(truth, scores):\n \"\"\"\n Calculate label-weighted label-ranking average precision.\n\n Arguments:\n truth:\n np.array of (num_samples, num_classes) giving boolean ground-truth\n of presence of that class in that sample.\n scores:\n np.array of (num_samples, num_classes) giving the classifier-under-\n test's real-valued score for each class for each sample.\n\n Returns:\n per_class_lwlrap:\n np.array of (num_classes,) giving the lwlrap for each class.\n weight_per_class:\n np.array of (num_classes,) giving the prior of each\n class within the truth labels.\n Then the overall unbalanced lwlrap is\n simply np.sum(per_class_lwlrap * weight_per_class).\n \"\"\"\n assert truth.shape == scores.shape\n num_samples, num_classes = scores.shape\n\n # Space to store a distinct precision value for each class on each sample.\n # Only the classes that are true for each sample will be filled in.\n precisions_for_samples_by_classes = np.zeros((num_samples, num_classes))\n for sample_num in range(num_samples):\n pos_class_indices, precision_at_hits = (\n _one_sample_positive_class_precisions(scores[sample_num, :],\n truth[sample_num, :])\n )\n precisions_for_samples_by_classes[sample_num, pos_class_indices] = (\n precision_at_hits)\n\n # Compute weight per class\n labels_per_class = np.sum(truth > 0, axis=0)\n weight_per_class = labels_per_class / float(np.sum(labels_per_class))\n\n # Form average of each column,\n # i.e. all the precisions assigned to labels in a particular class.\n per_class_lwlrap = (np.sum(precisions_for_samples_by_classes, axis=0) /\n np.maximum(1, labels_per_class))\n\n return per_class_lwlrap, weight_per_class\n\ndef lwlrap(actual, pred):\n per_class_lwlrap, weight_per_class = calculate_per_class_lwlrap(actual, pred)\n return np.sum(per_class_lwlrap * weight_per_class)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"##
4. Naive Experiment
"},{"metadata":{"trusted":true},"cell_type":"markdown","source":"###
4.1 Label Imbalance on train_curated
\n\nSome labels are less amount of records than others. \nIf private test dataset are imbalanced, our evaluation metric will fluctuate unless our models are perfect."},{"metadata":{"trusted":true},"cell_type":"code","source":"train.loc[:, 'Accelerating_and_revving_and_vroom':].sum().plot(\n kind='barh', \n title=\"Number of Audio Samples per Category\", \n color='deeppink', \n figsize=(15,25));","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"###
4.2 lwlrap on all train_curated data
"},{"metadata":{"trusted":true},"cell_type":"code","source":"lwlrap_1 = lwlrap(\n train.loc[:, 'Accelerating_and_revving_and_vroom':].values,\n sub1.loc[:, 'Accelerating_and_revving_and_vroom':].values\n)\nprint('lwlrap on sub1 : {:.5f}'.format(lwlrap_1))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"lwlrap_2 = lwlrap(\n train.loc[:, 'Accelerating_and_revving_and_vroom':].values,\n sub2.loc[:, 'Accelerating_and_revving_and_vroom':].values\n)\nprint('lwlrap on sub2 : {:.5f}'.format(lwlrap_2))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We have 2 OOF train_curated predictions, both lwlrap value are enough different. \nIn this case sub1 is better than sub2. "},{"metadata":{"trusted":true},"cell_type":"markdown","source":"###
4.3 Lwlrap Difference Distribution With Random Sampling
\n\nThe private test set is approximately three times the size of the public. \n ( public : 1120 records, private : 1120 * 3 records ) \nSo splitting train_curated data into 1:3 (public : 1242 records, private : 3727 records) would be reasonable. \nLet's simulate lwlrap differences between public and private. "},{"metadata":{"trusted":true},"cell_type":"code","source":"sub1_diff = []\nsub2_diff = []\nfor i in range(1000):\n seed = 2019 + i\n fname_sel = train.fname.sample(int(len(train)/4), random_state=seed)\n public_lwlrap1 = lwlrap(\n train.loc[train.fname.isin(fname_sel), 'Accelerating_and_revving_and_vroom':].values,\n sub1.loc[sub1.fname.isin(fname_sel), 'Accelerating_and_revving_and_vroom':].values\n )\n private_lwlrap1 = lwlrap(\n train.loc[~train.fname.isin(fname_sel), 'Accelerating_and_revving_and_vroom':].values,\n sub1.loc[~sub1.fname.isin(fname_sel), 'Accelerating_and_revving_and_vroom':].values\n )\n sub1_diff.append(private_lwlrap1 - public_lwlrap1)\n public_lwlrap2 = lwlrap(\n train.loc[train.fname.isin(fname_sel), 'Accelerating_and_revving_and_vroom':].values,\n sub2.loc[sub2.fname.isin(fname_sel), 'Accelerating_and_revving_and_vroom':].values\n )\n private_lwlrap2 = lwlrap(\n train.loc[~train.fname.isin(fname_sel), 'Accelerating_and_revving_and_vroom':].values,\n sub2.loc[~sub2.fname.isin(fname_sel), 'Accelerating_and_revving_and_vroom':].values\n )\n sub2_diff.append(private_lwlrap2 - public_lwlrap2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(2, 1, figsize=(16,8))\nax[0].hist(sub1_diff, bins=25, rwidth=0.5, color='deeppink')\nax[1].hist(sub2_diff, bins=25, rwidth=0.5, color='darkslateblue')\nax[0].set_xlim(-0.04, 0.04)\nax[1].set_xlim(-0.04, 0.04)\nax[0].set_title('sub1 (higher lwlrap)')\nax[1].set_title('sub2 (lower lwlrap)')\nplt.suptitle('lwlrap difference between public and private', ha='center');","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pd.Series(sub1_diff).describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pd.Series(sub2_diff).describe()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In the case when public and private split is random : \n\n* Higher lwlrap model is more stable. \n* 1 sigma is ~1% in higher lwlrap model, ~1.5% in lower one. "},{"metadata":{},"cell_type":"markdown","source":"###
4.4 Lwlrap Difference Distribution With Balanced Sampling
\n\nNext, let's consider the case when both public and private dataset contain balanced amount of labels. \nTo get the stratified fold in multi-labeled data, we will use `iterative-stratification` package. \nhttps://github.com/trent-b/iterative-stratification"},{"metadata":{"trusted":true},"cell_type":"code","source":"sub1_diff = []\nsub2_diff = []\nfor i in range(int(1000 / 4)):\n seed = 2019 + i\n mskf = MultilabelStratifiedKFold(n_splits=4, random_state=seed)\n for private_idx, public_idx in mskf.split(\n np.zeros(len(train)), train.loc[:, 'Accelerating_and_revving_and_vroom':].values):\n public_lwlrap1 = lwlrap(\n train.iloc[public_idx, 2:].values, sub1.iloc[public_idx, 1:].values\n )\n private_lwlrap1 = lwlrap(\n train.iloc[private_idx, 2:].values, sub1.iloc[private_idx, 1:].values\n )\n sub1_diff.append(private_lwlrap1 - public_lwlrap1)\n public_lwlrap2 = lwlrap(\n train.iloc[public_idx, 2:].values, sub2.iloc[public_idx, 1:].values\n )\n private_lwlrap2 = lwlrap(\n train.iloc[private_idx, 2:].values, sub2.iloc[private_idx, 1:].values\n )\n sub2_diff.append(private_lwlrap2 - public_lwlrap2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(2, 1, figsize=(16,8))\nax[0].hist(sub1_diff, bins=25, rwidth=0.5, color='deeppink')\nax[1].hist(sub2_diff, bins=25, rwidth=0.5, color='darkslateblue')\nax[0].set_xlim(-0.04, 0.04)\nax[1].set_xlim(-0.04, 0.04)\nax[0].set_title('sub1 (higher lwlrap)')\nax[1].set_title('sub2 (lower lwlrap)')\nplt.suptitle('lwlrap difference between public and private', ha='center');","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pd.Series(sub1_diff).describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pd.Series(sub2_diff).describe()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In the case when public and private split is multi-label stratified : \n\n* Higher lwlrap model is less stable. This result is contradictory to previous result. \n\n* 1 sigma is ~1.1% in higher lwlrap model, ~0.8% in lower one. "}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}