{"cells":[{"metadata":{"_cell_guid":"d384e5ae-a017-40a5-83d3-b7b2ca5787cb","_uuid":"069b3d9945db6c38cb9ff5dd1583f2ad2436860a","slideshow":{"slide_type":"slide"}},"cell_type":"markdown","source":"# Freesound Kaggle Presentation\n\n### Matt Potma\n\n### May 3, 2018\n\nThis notebook has been written for a presentation happening at approximately 6:30pm PDT today, May 3rd. A link to the livestream will be posted here shortly before it begins, so anyone can view it. Without watching the presentation, and even with, the flow of the notebook may be a bit awkward, but the presentation will be left up on YouTube to be viewed at any time. At some point, I may edit this notebook to provide more context, but will be left as-is for now.\n\nUnfortunately, despite the notebook running from scratch on my laptop in about 2 hours, trying to Commit & Run the notebook ran out of time. I've added two flags at the beginning of the second code cell, `cache` and `run_full_notebook`. Both are set to `False` in this notebook, hoping that it can fully execute on this kernel. The `cache` flag saves intermediate results and calculates spectral features on audio files with silence trimmed off, and `run_full_notebook` trains a couple of extra LGBM models. By setting both of those flags to `True` and running the notebook locally, the last submission file is good for 0.836 on the Public Leaderboard, with lots that can be improved upon.\n\nA link to my presentation where I go through this notebook can be seen here https://www.youtube.com/watch?v=3CtPuwok7Nw\n\n"},{"metadata":{"_cell_guid":"0eec6742-d21e-4368-bc9f-9fd0fbe9eee3","_uuid":"3fb30e845a9234641b85a02aef5cfe2aa5b4df94","slideshow":{"slide_type":"slide"}},"cell_type":"markdown","source":"### Miscellaneous imports"},{"metadata":{"_cell_guid":"24194e3c-416c-431b-8207-e67a836ac16c","_uuid":"940641c04f640a54e86f0ca28bba96eb174e2174","slideshow":{"slide_type":"-"},"trusted":true,"scrolled":true},"cell_type":"code","source":"import os, random, math\n\nimport pandas as pd\nimport numpy as np\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n%matplotlib inline\n\nimport lightgbm as lgb\n\nimport librosa\nimport librosa.display\n\nfrom scipy.stats import skew, kurtosis\nfrom sklearn.model_selection import train_test_split\n\nfrom prettytable import PrettyTable\nfrom tqdm import tqdm_notebook, tqdm_pandas\ntqdm_notebook().pandas(smoothing=0.7)\n\nimport IPython\nimport IPython.display as ipd\n\nimport matplotlib as mpl\nmpl.rcParams['font.size'] = 14","execution_count":61,"outputs":[]},{"metadata":{"_cell_guid":"f4b4fe34-ca49-46c1-a3ae-445966ba393c","_uuid":"7a86204c86e2d87a77645a8c4e71cb89e2ccbb93","slideshow":{"slide_type":"slide"}},"cell_type":"markdown","source":"### Setting up directories\n\nIf `cache` is set to `True`, calculated features will be saved to disk for use later. In addition, the spectral features will be calculated on trimmed audio files, instead of the original files."},{"metadata":{"_cell_guid":"71bbfce6-d760-4dc3-bf86-a5cb00193e66","_uuid":"45af454148413acb11ea0a23f7883907d1ac2e2d","collapsed":true,"slideshow":{"slide_type":"-"},"trusted":true},"cell_type":"code","source":"cache = False\nrun_full_notebook = False\n\ntrain_root = '../input/audio_train/'\ntest_root = '../input/audio_test/'\n\nif cache:\n    train_root_trimmed = '../input/audio_train_trimmed/'\n    test_root_trimmed = '../input/audio_test_trimmed/'\n\n    os.makedirs('../input/audio_train_trimmed', exist_ok=True)\n    os.makedirs('../input/audio_test_trimmed', exist_ok=True)\n    \n    os.makedirs('../cache', exist_ok=True)\n    os.makedirs('../output', exist_ok=True)\n\nelse:\n    train_root_trimmed = train_root\n    test_root_trimmed = test_root","execution_count":62,"outputs":[]},{"metadata":{"_cell_guid":"8f16979f-6a7c-4874-a644-5fbbce776b25","_uuid":"7bc2a56cdb3c3d77c607b61185b9b5557eaf0edc","slideshow":{"slide_type":"slide"}},"cell_type":"markdown","source":"### Load and view data"},{"metadata":{"_cell_guid":"a87ee15b-9624-4341-99fd-b4b4aa83903c","_uuid":"9b0bbc666aedc9316c22d4dda2d7f2b2dfb65248","slideshow":{"slide_type":"-"},"trusted":true},"cell_type":"code","source":"test_df = pd.read_csv(\"../input/sample_submission.csv\")\ntrain_df = pd.read_csv(\"../input/train.csv\")\n\ntrain_df.head()","execution_count":63,"outputs":[]},{"metadata":{"_cell_guid":"6dfef7df-3a1c-4370-bb0b-3549ec4b178a","_uuid":"235d6fe83cb9432ad2d6857c126c18b4588e82bb","slideshow":{"slide_type":"slide"}},"cell_type":"markdown","source":"### Data counts"},{"metadata":{"_cell_guid":"d5d2e393-14ed-4e44-a741-a42a8b66cbab","_uuid":"006205b21aff711a21edf7468dce2b4a91aaf95e","slideshow":{"slide_type":"-"},"trusted":true},"cell_type":"code","source":"n_test = test_df.shape[0]\nn_training = train_df.shape[0]\nn_categories = len(train_df.label.unique())\n\nprint(\"Number of training examples: {}\".format(n_training))\nprint(\"Number of testing examples: {}\".format(n_test))\nprint(\"Number of unique categories: {}\".format(n_categories))","execution_count":64,"outputs":[]},{"metadata":{"_cell_guid":"e039c324-c665-4ac7-b434-e63888eea91e","_uuid":"d66583a878968e6a997c433d71edfd8d04a88c27","slideshow":{"slide_type":"slide"},"trusted":true},"cell_type":"code","source":"# Plot a pie chart\nmpl.rcParams['font.size'] = 16\nplt.figure(figsize=(10, 10))\nplt.pie([n_training - train_df.manually_verified.sum(), train_df.manually_verified.sum()],\n        labels=[\"Not Verified ({:.0f}%)\".format(100*(n_training - train_df.manually_verified.sum())/n_training),\n                \"Verified ({:.0f}%)\".format(100*train_df.manually_verified.sum()/n_training)])\n\n# Turn the pie chart into a donut chart\np = plt.gcf()\np.gca().add_artist(plt.Circle((0, 0), 0.6, color='white'))\nplt.axis('equal')\n\nplt.show()","execution_count":65,"outputs":[]},{"metadata":{"_cell_guid":"a7905027-520d-48bc-a76b-e7af6fe17b86","_uuid":"05e1b73f8b7d59d60d496d38c3591c2c4527af13","collapsed":true,"slideshow":{"slide_type":"slide"},"trusted":true},"cell_type":"code","source":"def play_audio(wavfile, dset='train'):\n    print(wavfile)\n    fname = '../input/audio_{}/{}'.format(dset, wavfile)\n    IPython.display.display(ipd.Audio(fname))\n    \n    x, sr = librosa.load(fname)\n    \n    plt.figure(figsize=(14, 5))\n    librosa.display.waveplot(x, sr=sr)\n    \n    X = librosa.stft(x)\n    Xdb = librosa.amplitude_to_db(abs(X))\n    plt.figure(figsize=(14, 5))\n    librosa.display.specshow(Xdb, sr=sr, x_axis='time', y_axis='hz')","execution_count":66,"outputs":[]},{"metadata":{"_cell_guid":"598ac374-9cad-4398-9e69-f9bb86a7db9b","_uuid":"201f2aad3bd9f8966937ad4584fa6db136aa53f3","slideshow":{"slide_type":"slide"},"trusted":true,"scrolled":true},"cell_type":"code","source":"random_int = random.randint(0, n_training)\nrandom_wavfile = train_df.fname.iloc[random_int]\nprint(train_df.label.iloc[random_int])\n\nplay_audio(random_wavfile)","execution_count":67,"outputs":[]},{"metadata":{"_cell_guid":"02be8b99-2490-4393-9a66-a40fa51353fa","_uuid":"0a025e5e6566b17ac69f4903641e79ddcaa5c9c9","slideshow":{"slide_type":"slide"}},"cell_type":"markdown","source":"## List of categories\n\n| Instruments   | Instruments Continued | Human        | Actions | Other |\n| :------------ | :-------              | :------      | :----   | :---  |\n| Hi-hat        | Bass_drum             | Laughter     | Knock   | Gunshot_or_gunfire |\n| Saxophone     | Harmonica             | Finger_snapping | Drawer_open_or_close  | Bus |\n| Trumpet       | Gong                  | Fart         | Computer_keyboard | Telephone |\n| Glockenspiel  | Double_bass           | Cough        | Tearing | Squeak |\n| Cello         | Tambourine            | Applause     | Shatter | Scissors |\n| Clarinet      | Cowbell               | Burping_or_eructation  | Keys_jangling | Microwave_oven |\n| Snare_drum    | Electric_piano        |              | Writing | Bark |\n| Oboe          | Acoustic_guitar       |  |  | Meow |\n| Flute         | Voilin_or_fiddle      |              | | Fireworks |\n| Chime         |  | |  | |\n\n"},{"metadata":{"_cell_guid":"5f9afd89-db4c-41bf-8ab7-561c61db0e1e","_uuid":"a165c9b667d352c21c5eeb39696ebfee845ac77f","slideshow":{"slide_type":"slide"}},"cell_type":"markdown","source":"### Label Distributions"},{"metadata":{"_cell_guid":"a5c01b8d-b73c-4096-98ce-74dc27091ea5","_uuid":"d8c2a1d7645c0296e7209cee8d5be94ac54dfa42","slideshow":{"slide_type":"-"},"trusted":true},"cell_type":"code","source":"def plot_label_distributions():\n    plot = train_df.groupby(['label', 'manually_verified'])['label'].count().unstack('label').transpose()\n    plot['total'] = plot[0] + plot[1]\n    plot.sort_values(['total', 1], ascending=[0, 1], inplace=True)\n    plot.drop('total', axis=1, inplace=True)\n    plot.plot(kind='bar', stacked=True, figsize=(22, 7), fontsize=18)\n\nplot_label_distributions()","execution_count":68,"outputs":[]},{"metadata":{"_cell_guid":"e426f5e1-8606-4f0b-a4eb-9d9f83b936be","_uuid":"b1f1b36225d03be0a1bef0fe3d95a58ce023be0f","slideshow":{"slide_type":"slide"}},"cell_type":"markdown","source":"### First attempt at basic feature engineering\nFrom the data, derive simple statistical features, such as mean, min, max, standard deviation, length, rms, skewness and kurtosis."},{"metadata":{"_cell_guid":"a46b7add-9d48-4757-9bb0-fdd1a0e4767f","_uuid":"7b01b9b7c0178083deb7fb422354a96d4be55396","collapsed":true,"slideshow":{"slide_type":"-"},"trusted":true},"cell_type":"code","source":"def wavfile_stats(fname, root):\n    try:\n        data, fs = librosa.core.load(root + fname, sr=None)\n        mean = np.mean(data)\n        minimum = np.min(data)\n        maximum = np.max(data)\n        std = np.std(data)\n        length = len(data)\n        rms = np.sqrt(np.mean(data**2))\n        skewness = skew(data)\n        kurt = kurtosis(data)\n\n        return pd.Series([length, mean, minimum, maximum, std, rms, skewness, kurt])\n    except ValueError:\n        print(\"Bad file at {}\".format(fname))\n        return pd.Series([0, 0, 0, 0, 0, 0, 0, 0])","execution_count":69,"outputs":[]},{"metadata":{"_cell_guid":"c0c4ec96-a6d6-4741-a7b1-71431025342c","_uuid":"429c99cdd11685b5c55528df6f4e8d0c708ed84a","slideshow":{"slide_type":"slide"}},"cell_type":"markdown","source":"### Apply to the DataFrame\n\nUse pandas' progress apply to calculate the simple statistical features.\n\nIf possible, load features from a cached csv file."},{"metadata":{"_cell_guid":"7640764e-db5b-4aab-b14b-b731ec1ac6a3","_uuid":"129c82b5fede25d26a7eebc8991238c3f439cf7d","scrolled":true,"slideshow":{"slide_type":"-"},"trusted":true},"cell_type":"code","source":"if os.path.isfile('../cache/train_1.csv') and cache:\n    train_df = pd.read_csv('../cache/train_1.csv')\n    test_df = pd.read_csv('../cache/test_1.csv')\n    assert len(train_df.index) == n_training\n    assert len(test_df.index) == n_test\n    print(\"Files loaded from cache\")\n\nelse:\n    train_df[['length', 'data_mean', 'data_min', 'data_max', 'data_std', 'data_rms', 'skewness', 'kurtosis']] = \\\n        train_df['fname'].progress_apply(wavfile_stats, root=train_root)\n    test_df[['length', 'data_mean', 'data_min', 'data_max', 'data_std', 'data_rms', 'skewness', 'kurtosis']] = \\\n        test_df['fname'].progress_apply(wavfile_stats, root=test_root)\n    \n    if cache:\n        train_df.to_csv('../cache/train_1.csv', index=False)\n        test_df.to_csv('../cache/test_1.csv', index=False)","execution_count":70,"outputs":[]},{"metadata":{"_cell_guid":"d8c39200-ce02-4c00-9d33-dcbdacfa9f1a","_uuid":"4a43ea2e25b5bb72fd317f712d7e80a488fa17ff","slideshow":{"slide_type":"slide"}},"cell_type":"markdown","source":"### A couple of other miscellaneous features"},{"metadata":{"_cell_guid":"f30a4477-0145-4382-82ed-952d1a8f1f69","_uuid":"83913a875953914c74550388987dc819776e658b","collapsed":true,"slideshow":{"slide_type":"-"},"trusted":true},"cell_type":"code","source":"train_df['rms_std'] = train_df['data_rms'] / train_df['data_std']\ntest_df['rms_std'] = test_df['data_rms'] / test_df['data_std']\n\ntrain_df['max_min'] = train_df['data_max'] / train_df['data_min']\ntest_df['max_min'] = test_df['data_max'] / test_df['data_min']","execution_count":71,"outputs":[]},{"metadata":{"_cell_guid":"0e54d9d8-877f-4d2a-be95-a76b8d9d59dc","_uuid":"306ef8308a9691130be37f6941ec572148305e92","slideshow":{"slide_type":"slide"}},"cell_type":"markdown","source":"### Make it easy to view your data\n\nHere are two simple functions for producing either a histogram or boxplot of any variable in the DataFrame"},{"metadata":{"_cell_guid":"31458724-63f6-4dba-82f4-700fc2d791f3","_uuid":"02db75791f09e6ae1f51f5e7664f96188174dfab","collapsed":true,"slideshow":{"slide_type":"-"},"trusted":true},"cell_type":"code","source":"def plot_hist(feature_name, bins=50, log=False):\n    \"\"\"Plot feature histogram with pandas.\"\"\"\n    data = train_df[feature_name].values\n    plt.hist(data, bins=bins, log=log)\n    plt.grid()\n    plt.show()\n\ndef plot_box(feature_name):\n    \"\"\"Plot boxplot of variable with pandas.\"\"\"\n    props = dict(linewidth=3)\n    train_df.boxplot(column=feature_name, by='label', rot=90, figsize=(20, 7), sym='', grid=False, boxprops=props)\n    plt.title('{} boxplot'.format(feature_name))\n    plt.suptitle('')","execution_count":72,"outputs":[]},{"metadata":{"_cell_guid":"3abe2c27-8b2b-44b1-ab26-0fa14e4080ec","_uuid":"333af20a99c26365cdb6b88714741ae474034287","slideshow":{"slide_type":"slide"},"trusted":true},"cell_type":"code","source":"feature = 'kurtosis'\n\n# plot_hist(feature, log=True)\nplot_box(feature)","execution_count":73,"outputs":[]},{"metadata":{"_cell_guid":"75b0bb18-f69f-4f3a-b863-99a9ed4730bb","_uuid":"9c2cc15b47ed70e5363cbf95e95e689305334cf3","slideshow":{"slide_type":"slide"}},"cell_type":"markdown","source":"### Removal of NaNs, etc"},{"metadata":{"_cell_guid":"88598805-5525-4d42-8701-d3ce3d0919d5","_uuid":"f8b8bf8ffd356ebf8bc5b37a05ce7d1ad1c01b79","slideshow":{"slide_type":"-"},"trusted":true},"cell_type":"code","source":"test_df[pd.isnull(test_df).any(axis=1)].head()","execution_count":74,"outputs":[]},{"metadata":{"_cell_guid":"cc2fc4e9-f3e4-42ba-b90d-3d376a70481e","_uuid":"bcf02c670500fa0edbcb41dc05c9911e09571e37","collapsed":true,"slideshow":{"slide_type":"-"},"trusted":true},"cell_type":"code","source":"test_df.fillna(0, inplace=True)","execution_count":75,"outputs":[]},{"metadata":{"_cell_guid":"1593246e-2356-4f2c-9ea6-36c0f5e62594","_uuid":"66a5d3136c37d72b8af58344fcfe3f1f1ff36bd0","slideshow":{"slide_type":"slide"}},"cell_type":"markdown","source":"### Split data into training, validation, and testing sets"},{"metadata":{"_cell_guid":"cbc81e08-8ce3-4237-bf34-e9f4337de957","_uuid":"0d2736698fc2d14e6ba90e7bec36130ebdf38dbc","collapsed":true,"slideshow":{"slide_type":"-"},"trusted":true},"cell_type":"code","source":"def data_split(train_df, test_df, shuffle=True, test_size=0.25, random_state=0, verbose=True):\n    # Get numpy array of X data\n    X_train = train_df.drop(['fname', 'label', 'manually_verified'], axis=1).values\n    X_test = test_df.drop(['fname', 'label'], axis=1).values\n    feature_names = list(test_df.drop(['fname', 'label'], axis=1).columns.values)\n\n    # Get numpy array of y data\n    y_train = pd.get_dummies(train_df.label)\n    labels = y_train.columns.values\n    y_train = y_train.values\n\n    y_train = [np.argmax(row) for row in y_train]\n    X_train, X_valid, y_train, y_valid = train_test_split(X_train, y_train, test_size=test_size,\n                                                          random_state=random_state, shuffle=shuffle)\n    if verbose:\n        print(\"Train X shape = {}\\nTrain y shape = {}\".format(X_train.shape, len(y_train)))\n        print(\"\\nValid X shape = {}\\nValid y shape = {}\".format(X_valid.shape, len(y_valid)))\n\n    assert X_train.shape[1] == X_valid.shape[1] == X_test.shape[1]\n    assert len(y_train) == X_train.shape[0]\n    assert len(y_valid) == X_valid.shape[0]\n    \n    return X_train, X_valid, y_train, y_valid, X_test, feature_names, labels","execution_count":76,"outputs":[]},{"metadata":{"_cell_guid":"0a38b74c-46a5-4a7e-b1ce-fa81f17b08f4","_uuid":"fb4d008968092ee37bb160328c82dfbaca24776a","slideshow":{"slide_type":"slide"},"trusted":true},"cell_type":"code","source":"X_train, X_valid, y_train, y_valid, X_test, feature_names, labels = data_split(train_df, test_df)","execution_count":77,"outputs":[]},{"metadata":{"_cell_guid":"19c902c8-3019-4e3b-abfc-0e58c42ace0f","_uuid":"048f483d4b84b135e3de489c044b588a788cdd6d","slideshow":{"slide_type":"slide"}},"cell_type":"markdown","source":"### Prepare dataset for LightGBM"},{"metadata":{"_cell_guid":"6dad776a-76c9-4df3-9589-241aaa5dbb6d","_uuid":"a927f88f2e64300ab19b4147f6c574fdcdd26783","collapsed":true,"slideshow":{"slide_type":"-"},"trusted":true},"cell_type":"code","source":"def lgb_dset(X_train, X_valid, y_train, y_valid, feature_names):\n    \n    d_train = lgb.Dataset(X_train, label=y_train, feature_name=feature_names)\n    d_valid = lgb.Dataset(X_valid, label=y_valid, feature_name=feature_names)\n\n    params = {\n        'boosting_type': 'gbdt',\n        'objective': 'multiclass',\n        'metric': 'multi_logloss',\n        'max_depth': 5,\n        'num_leaves': 31,\n        'learning_rate': 0.025,\n        'feature_fraction': 0.85,\n        'bagging_fraction': 0.85,\n        'bagging_freq': 5,\n        'num_threads': os.cpu_count(),\n        'lambda_l2': 1.0,\n        'min_gain_to_split': 0,\n        'num_class': n_categories,\n    }\n    \n    return d_train, d_valid, params","execution_count":78,"outputs":[]},{"metadata":{"_cell_guid":"3e81d902-d16b-4826-a2ac-a371cb43c4da","_uuid":"b797957987bdfee900b5c36f113504d5301589a0","slideshow":{"slide_type":"slide"}},"cell_type":"markdown","source":"### Train our first model"},{"metadata":{"_cell_guid":"fa575314-815c-485b-a4d8-8f2593e941bc","_uuid":"3be3ebb2de0cbaaf1eac097860ce1995ad726062","slideshow":{"slide_type":"-"},"trusted":true},"cell_type":"code","source":"d_train, d_valid, params = lgb_dset(X_train, X_valid, y_train, y_valid, feature_names)\nclf = lgb.train(params, d_train, num_boost_round=500, valid_sets=d_valid, verbose_eval=100, early_stopping_rounds=100)","execution_count":79,"outputs":[]},{"metadata":{"_cell_guid":"ee5bc396-bf9e-4349-a6f1-31c3bc761b13","_uuid":"bca56a0b451310a3d08232f50f673ee4c43593dd","slideshow":{"slide_type":"slide"}},"cell_type":"markdown","source":"### Calculation of MAP@3\nDemo available here https://www.kaggle.com/wendykan/map-k-demo"},{"metadata":{"_cell_guid":"43529d8c-8a30-4333-acd6-d16373592ff7","_uuid":"d15fe03f95d5260e053236c29461decf17ad1f4b","collapsed":true,"slideshow":{"slide_type":"-"},"trusted":true},"cell_type":"code","source":"def apk(actual, predicted, k=10):\n    if len(predicted)>k:\n        predicted = predicted[:k]\n\n    score = 0.0\n    num_hits = 0.0\n\n    for i,p in enumerate(predicted):\n        if p in actual and p not in predicted[:i]:\n            num_hits += 1.0\n            score += num_hits / (i+1.0)\n\n    if not actual:\n        return 0.0\n\n    return score / min(len(actual), k)\n\ndef mapk(actual, predicted, k=10):\n    return np.mean([apk(a,p,k) for a,p in zip(actual, predicted)])","execution_count":80,"outputs":[]},{"metadata":{"_cell_guid":"f8f22d2a-d234-4894-ab0a-696ffe72d135","_uuid":"d622f593fc8aa8c618ed88fb8d94d6fde3877753","slideshow":{"slide_type":"slide"}},"cell_type":"markdown","source":"### Predict and score validation set"},{"metadata":{"_cell_guid":"e67308a2-e55d-46da-8268-27c30f8a8dd8","_uuid":"dd6b90c86f3f6d456f504823be7312ee7aa0e9e5","slideshow":{"slide_type":"-"},"trusted":true},"cell_type":"code","source":"p = clf.predict(X_valid, num_iteration=clf.best_iteration)\n\npredictions = [list(np.argsort(p[i])[::-1][:3]) for i in range(len(p))]\nactual = [[i] for i in y_valid]\n\nvalid_score = mapk(actual, predictions, k=3)\n\nprint(\"Score = {:.4f}\".format(valid_score))","execution_count":81,"outputs":[]},{"metadata":{"_cell_guid":"b3b9fffe-644a-4e99-9f3f-0a3208042d99","_uuid":"a5f977d606ba2ae236764edbe4001a6a1ffc7a0d","slideshow":{"slide_type":"slide"}},"cell_type":"markdown","source":"### LightGBM feature importances"},{"metadata":{"_cell_guid":"0e0f04c9-12b2-404d-a9a9-ce1a6ed2f320","_uuid":"81b85404e6b77ea37c7ff5a714c5fd41de2d775b","slideshow":{"slide_type":"-"},"trusted":true},"cell_type":"code","source":"ax = lgb.plot_importance(clf, max_num_features=10, grid=False, height=0.8, figsize=(20, 6))\nplt.show()","execution_count":82,"outputs":[]},{"metadata":{"_cell_guid":"1d3f7064-3711-4fb9-8437-e50f3fd908ad","_uuid":"4154fda7e94671aa97e7c89540595e9b48a28d30","slideshow":{"slide_type":"slide"}},"cell_type":"markdown","source":"### View some sample predictions"},{"metadata":{"_cell_guid":"15003ce8-e7e7-4c51-a36d-3fde31d04445","_uuid":"a255b487aef6aa949c631a9e57f8d5a598aa2c13","slideshow":{"slide_type":"-"},"trusted":true},"cell_type":"code","source":"def preds_to_labels(p, labels):\n    predictions = [list(np.argsort(p[i])[::-1][:3]) for i in range(len(p))]\n    prediction_labels = []\n    \n    for pred in predictions:\n        label_list = []\n        for output in pred:\n            label_list.append(labels[output])\n        prediction_labels.append(label_list)\n    return prediction_labels\n\nlab = preds_to_labels(p, labels)\n\nt = PrettyTable(['Truth', 'Prediction'])\n[t.add_row([labels[l[1][0]], l[0]]) for l in zip(lab[:10], actual[:10])]\nprint(t)","execution_count":83,"outputs":[]},{"metadata":{"_cell_guid":"5a13f1b9-190e-4257-adc7-e10c61ccf6a2","_uuid":"3c9a1fa3df7ae9d8d05b36dcd8dc475f375ffcd2","slideshow":{"slide_type":"slide"}},"cell_type":"markdown","source":"### Predict test set"},{"metadata":{"_cell_guid":"b65a50cb-a6b4-4edb-8dd3-c5286c53ea0e","_uuid":"cfbfadd76fd586a887f75e4241a0d4eacdbd587c","collapsed":true,"slideshow":{"slide_type":"-"},"trusted":true},"cell_type":"code","source":"preds = clf.predict(X_test, num_iteration=clf.best_iteration)\nlab = preds_to_labels(preds, labels)","execution_count":84,"outputs":[]},{"metadata":{"_cell_guid":"d2360b07-0a03-4e13-9f69-ea199f7019b2","_uuid":"cb58b4fdecec681e01fb27c86a0f0dc49605711b","slideshow":{"slide_type":"slide"}},"cell_type":"markdown","source":"### Spot check test set"},{"metadata":{"_cell_guid":"c755ece5-fb67-4e42-bca9-c2211fab6079","_uuid":"950dab8e7559bd53839fe69903437b18ad492d6d","slideshow":{"slide_type":"-"},"trusted":true},"cell_type":"code","source":"random_int = random.randint(0, n_test)\nrandom_wavfile = test_df.fname.iloc[random_int]\nprint(lab[random_int])\nplay_audio(test_df.fname.iloc[random_int], dset='test')","execution_count":85,"outputs":[]},{"metadata":{"_cell_guid":"33940bc5-2c77-4e2b-bd38-8cd333e126b2","_uuid":"5bab1f9ac3d46444bb63b662640efcf45bc92a78","slideshow":{"slide_type":"slide"}},"cell_type":"markdown","source":"### Create submission"},{"metadata":{"_cell_guid":"1ba4ef82-0a9f-42ef-a8df-43d408952d09","_uuid":"1f22fe9ea759686df99ecffc95a84752f756b4da","slideshow":{"slide_type":"-"},"trusted":true,"collapsed":true},"cell_type":"code","source":"def create_submission(predictions, name='submission.csv'):\n    predictions = ['{} {} {}'.format(x[0], x[1], x[2]) for x in predictions]\n    submission = pd.read_csv('../input/sample_submission.csv')\n    submission.label = predictions\n    submission.to_csv('{}'.format(name), index=False)\n    print(\"Submission saved to '{}'\".format(name))\n\n# LB score = 0.445, CV score = 0.4500\nif run_full_notebook:\n    create_submission(lab, 'submission-{:.4f}.csv'.format(valid_score))","execution_count":86,"outputs":[]},{"metadata":{"_cell_guid":"ac9083b2-8e2b-4fa2-ae60-4a714ce0ad5c","_uuid":"da8e3126683a636fbb758e2e650c938282a3a7c7","slideshow":{"slide_type":"slide"}},"cell_type":"markdown","source":"### Trim silence from files"},{"metadata":{"_cell_guid":"69a42785-f98d-48b1-a0d1-2e22c3f1d939","_uuid":"d0e37753fcf069aee265c2565541dbe5e3386bdb","collapsed":true,"slideshow":{"slide_type":"-"},"trusted":true},"cell_type":"code","source":"def trim_silence(fname, root, window_length=0.5):\n    try:\n        trimmed_ends = 0\n        trimmed_int = 0\n        \n        data, fs = librosa.core.load(root + fname, sr=None)\n        length = len(data)\n        \n        # Trim silence from ends\n        data, _ = librosa.effects.trim(data, top_db=40)\n        length_int = len(data)\n        ratio_int = length_int/length\n        \n        # Split file into non-silent chunks and recombine\n        splits = librosa.effects.split(data, top_db=40)\n        if len(splits) > 1:\n            data = np.concatenate([data[x[0]:x[1]] for x in splits])    \n        \n        length_final = len(data)\n        ratio_final = length_final/length_int     \n\n        if cache:\n            # Save file and return new features\n            librosa.output.write_wav('{}_trimmed/{}'.format(root[:-1], fname), data, fs)\n        return pd.Series([length_int, length_final, ratio_int, ratio_final])\n       \n    except ValueError:\n        print(\"Bad file at {}\".format(fname))\n        return pd.Series([0, 0, 0, 0])  ","execution_count":87,"outputs":[]},{"metadata":{"_cell_guid":"d3a7faaa-92f9-4a7a-b3e2-952081fbdbfd","_uuid":"ebfb679aefe9acf0782ae44d37cdf59f5dc8e16d","slideshow":{"slide_type":"slide"}},"cell_type":"markdown","source":"### Apply silence trimming to DataFrame"},{"metadata":{"_cell_guid":"6acb8a85-a307-4bb0-8473-5d5c597a2908","_uuid":"81fd2522d183bb0c6e2343da4feafef5fd70cdec","slideshow":{"slide_type":"-"},"trusted":true},"cell_type":"code","source":"if os.path.isfile('../cache/train_2.csv') and cache:\n    train_df = pd.read_csv('../cache/train_2.csv')\n    test_df = pd.read_csv('../cache/test_2.csv')\n    assert len(train_df.index) == n_training\n    assert len(test_df.index) == n_test\n    print(\"Files loaded from cache\")\n\nelse:\n    train_df[['length_int', 'length_final', 'ratio_int', 'ratio_final']] = \\\n        train_df['fname'].progress_apply(trim_silence, root=train_root)\n    test_df[['length_int', 'length_final', 'ratio_int', 'ratio_final']] = \\\n        test_df['fname'].progress_apply(trim_silence, root=test_root)\n\n    if cache:\n        train_df.to_csv('../cache/train_2.csv', index=False)\n        test_df.to_csv('../cache/test_2.csv', index=False)","execution_count":88,"outputs":[]},{"metadata":{"_cell_guid":"d272757c-1332-469d-b1e7-0e43a72ce00b","_uuid":"d1fdfce7c38bd7c4e6532509f3d2041705cbee53","trusted":true},"cell_type":"code","source":"trimmed_ends = 100*train_df.ratio_final[train_df.ratio_final < 1.0].count()/len(train_df.index)\ntrimmed_int = 100*train_df.ratio_int[train_df.ratio_int < 1.0].count()/len(train_df.index)\n\ntrimmed_ends_test = 100*test_df.ratio_final[test_df.ratio_final < 1.0].count()/len(test_df.index)\ntrimmed_int_test = 100*test_df.ratio_int[test_df.ratio_int < 1.0].count()/len(test_df.index)\n\nt = PrettyTable(['Dataset', 'Ends Trimmed', 'Intermediate Trimmed'])\nt.add_row(['Training', '{:.1f}%'.format(trimmed_ends), '{:.1f}%'.format(trimmed_int)])\nt.add_row(['Testing', '{:.1f}%'.format(trimmed_ends_test), '{:.1f}%'.format(trimmed_int_test)])\nprint(t)","execution_count":89,"outputs":[]},{"metadata":{"_cell_guid":"08773f70-4832-4b82-ae9e-39a80100b702","_uuid":"451840dc6ceaa82699cdeb9c8ab914f636e5755c","slideshow":{"slide_type":"slide"}},"cell_type":"markdown","source":"### Create dataset with new features and train"},{"metadata":{"_cell_guid":"8b7f23ef-5fc4-48bd-a9c2-851c18d51fdd","_uuid":"8bffaa3762ad19013fca55fffb0dfa77683d963b","collapsed":true,"slideshow":{"slide_type":"-"},"trusted":true},"cell_type":"code","source":"if run_full_notebook:\n    X_train, X_valid, y_train, y_valid, X_test, feature_names, labels = data_split(train_df, test_df)\n\n    d_train, d_valid, params = lgb_dset(X_train, X_valid, y_train, y_valid, feature_names)\n    clf = lgb.train(params, d_train, num_boost_round=1000, valid_sets=d_valid, verbose_eval=100, early_stopping_rounds=100)","execution_count":90,"outputs":[]},{"metadata":{"_cell_guid":"a68deaa0-5ec4-451b-8260-d7e7518db617","_uuid":"c1e886b148a39190569f280421b560d80b75b5f0","slideshow":{"slide_type":"slide"}},"cell_type":"markdown","source":"### Check our new validation score"},{"metadata":{"_cell_guid":"363c2405-f62e-4d12-93a8-a0775ae76150","_uuid":"c65627d02e75d32dae32df3f97ad575f70c526c9","collapsed":true,"slideshow":{"slide_type":"-"},"trusted":true},"cell_type":"code","source":"if run_full_notebook:    \n    p = clf.predict(X_valid, num_iteration=clf.best_iteration)\n\n    predictions = [list(np.argsort(p[i])[::-1][:3]) for i in range(len(p))]\n    actual = [[i] for i in y_valid]\n\n    valid_score = mapk(actual, predictions, k=3)\n\n    print(\"Score = {:.4f}\".format(valid_score))","execution_count":91,"outputs":[]},{"metadata":{"_cell_guid":"7cdc0c0e-fe03-42a9-8cb9-9667f3b708ca","_uuid":"0150d858efbec9eb42df1fec9336471d1e8f3fb5","slideshow":{"slide_type":"slide"}},"cell_type":"markdown","source":"### Check new feature importances"},{"metadata":{"_cell_guid":"7ce8fafe-53ad-4424-9ea0-6f9154532ce8","_uuid":"b5bf0f3a4d7178c51c8562f51a8656117a80ec19","collapsed":true,"slideshow":{"slide_type":"-"},"trusted":true},"cell_type":"code","source":"if run_full_notebook:\n    ax = lgb.plot_importance(clf, max_num_features=10, grid=False, height=0.8, figsize=(20, 6))\n    plt.show()","execution_count":92,"outputs":[]},{"metadata":{"_cell_guid":"3c940bf1-4c73-4975-b940-7e547a69d46a","_uuid":"05b292081e71754671b341ed8e2cd2a047df7836","slideshow":{"slide_type":"slide"}},"cell_type":"markdown","source":"### MFCC\n\n- Similar to log-magnitude spectrograms\n- Use non-linaer Mel scale frequencies, $M(f) = 1125*ln(1+f/700)$\n- Use discrete cosine transformation, retaining only the lower coefficients"},{"metadata":{"_cell_guid":"5e621474-0c72-4a3f-84c9-74df95540513","_uuid":"610c7f798944e70582ece350db37ce2984fc3931","collapsed":true,"slideshow":{"slide_type":"slide"},"trusted":true},"cell_type":"code","source":"def spectral_features(fname=None, root=None, n_mfcc=20, return_fnames=False):\n    feature_names = []\n    for i in ['mean', 'std', 'min', 'max', 'skew', 'kurt']:\n        for j in range(n_mfcc):\n            feature_names.append('mfcc_{}_{}'.format(j, i))\n        feature_names.append('centroid_{}'.format(i))\n        feature_names.append('bandwidth_{}'.format(i))\n        feature_names.append('contrast_{}'.format(i))\n        feature_names.append('rolloff_{}'.format(i))\n        feature_names.append('flatness_{}'.format(i))\n        feature_names.append('zcr_{}'.format(i))\n    \n    if return_fnames:\n        return feature_names\n\n    spectral_features = [\n        librosa.feature.spectral_centroid,\n        librosa.feature.spectral_bandwidth,\n        librosa.feature.spectral_contrast,\n        librosa.feature.spectral_rolloff,\n        librosa.feature.spectral_flatness,\n        librosa.feature.zero_crossing_rate]\n     \n    try:\n        data, fs = librosa.core.load(root + fname, sr=None)\n        M = librosa.feature.mfcc(data, sr=fs, n_mfcc=n_mfcc)\n        data_row = np.hstack((np.mean(M, axis=1), np.std(M, axis=1), np.min(M, axis=1),\n                              np.max(M, axis=1), skew(M, axis=1), kurtosis(M, axis=1)))\n        \n        for feat in spectral_features:\n            S = feat(data)[0]\n            data_row = np.hstack((data_row, np.mean(S), np.std(S), np.min(S),\n                                  np.max(S), skew(S), kurtosis(S)))\n\n        return pd.Series(data_row)\n        \n    except (ValueError, RuntimeError):\n        print(\"Bad file at {}\".format(fname))\n        return pd.Series([0]*len(feature_names))  ","execution_count":93,"outputs":[]},{"metadata":{"_cell_guid":"9383fa46-652c-4745-a09b-53cd416fad6a","_uuid":"f8ecf6f894b577c5ff0cf08e7e6a0c29020ded02","slideshow":{"slide_type":"slide"}},"cell_type":"markdown","source":"### Apply spectral features to DataFrame"},{"metadata":{"_cell_guid":"f07fcd70-2504-4cf9-9cbd-9d327dcb46a7","_uuid":"4a1da3fac2e9f99863b0496fa0f2c27d56fac1ec","slideshow":{"slide_type":"-"},"trusted":true},"cell_type":"code","source":"if os.path.isfile('../cache/train_spectral.csv') and cache:\n    train_df = pd.read_csv('../cache/train_spectral.csv')\n    test_df = pd.read_csv('../cache/test_spectral.csv')\n    assert len(train_df.index) == n_training\n    assert len(test_df.index) == n_test\n    print(\"Files loaded from cache\")\n\nelse:\n    feature_names = spectral_features(return_fnames=True)\n    train_df[feature_names] = train_df['fname'].progress_apply(spectral_features, root=train_root_trimmed)\n    test_df[feature_names] = test_df['fname'].progress_apply(spectral_features, root=test_root_trimmed)\n    if cache:\n        train_df.to_csv('../cache/train_spectral.csv', index=False)\n        test_df.to_csv('../cache/test_spectral.csv', index=False)","execution_count":94,"outputs":[]},{"metadata":{"_cell_guid":"54a89da3-0b2a-4f4d-adc2-db5f3ddbd6c9","_uuid":"5a44233202a6dcd65254ebe31e0ac906f2dceb14","slideshow":{"slide_type":"slide"}},"cell_type":"markdown","source":"### Train with spectral features"},{"metadata":{"_cell_guid":"10e12e18-3ddb-4986-a3e1-6b666f40e6aa","_uuid":"138ef412e123fa7f48f9b97e1612da805056fa83","slideshow":{"slide_type":"-"},"trusted":true},"cell_type":"code","source":"# Create dataset\nX_train, X_valid, y_train, y_valid, X_test, feature_names, labels = data_split(train_df, test_df, verbose=False)\nd_train, d_valid, params = lgb_dset(X_train, X_valid, y_train, y_valid, feature_names)\n\n# Train and predict\nclf = lgb.train(params, d_train, num_boost_round=2000, valid_sets=d_valid, verbose_eval=200, early_stopping_rounds=100)\np = clf.predict(X_valid, num_iteration=clf.best_iteration)\n\n# Score\npredictions = [list(np.argsort(p[i])[::-1][:3]) for i in range(len(p))]\nactual = [[i] for i in y_valid]\nvalid_score = mapk(actual, predictions, k=3)\nprint(\"\\nScore = {:.4f}\".format(valid_score))","execution_count":95,"outputs":[]},{"metadata":{"_cell_guid":"fab6e2ce-a0c2-4859-a332-c90066da301d","_uuid":"4f80d5e09c10fd51e912037a2b099f4586ae28ba","slideshow":{"slide_type":"slide"}},"cell_type":"markdown","source":"### Feature importance with spectral features"},{"metadata":{"_cell_guid":"92a72524-1638-44a3-b3fe-54a9e997de37","_uuid":"8797fbc888382635ecc590a35e0d39d74b315571","slideshow":{"slide_type":"-"},"trusted":true},"cell_type":"code","source":"# Plot importances\nax = lgb.plot_importance(clf, max_num_features=10, grid=False, height=0.8, figsize=(16, 8))\nplt.show()","execution_count":96,"outputs":[]},{"metadata":{"_cell_guid":"7d52cd7c-5c2f-4718-8da1-e00974ff2e3d","_uuid":"52253bb63ce1e5551bd2210bb0cc8cb0abe20ecb","slideshow":{"slide_type":"slide"}},"cell_type":"markdown","source":"### Create submission with spectral features"},{"metadata":{"_cell_guid":"4722838f-8657-423b-be2e-3db03f526ef9","_uuid":"e3825e42171a7a2c2c6c9086cedd91a367839564","collapsed":true,"slideshow":{"slide_type":"-"},"trusted":true},"cell_type":"code","source":"# CV = 0.7854, LB = 0.835\np = clf.predict(X_test, num_iteration=clf.best_iteration)\nlab = preds_to_labels(p, labels)\ncreate_submission(lab, 'submission-{:.4f}.csv'.format(valid_score))","execution_count":97,"outputs":[]},{"metadata":{"_cell_guid":"2e2a3d5c-b5bd-42ea-ba56-5c21fc979bb2","_uuid":"cfc2efc3ae75fb6f4c7ba3dbab2b5fefc9940553","slideshow":{"slide_type":"slide"}},"cell_type":"markdown","source":"### Prepare to train on full dataset"},{"metadata":{"_cell_guid":"2a4cebd2-a3a3-4573-90f0-bb73d8b1113a","_uuid":"d1d3ba01fcb97481f012a51b4005e4149606d57c","collapsed":true,"slideshow":{"slide_type":"-"},"trusted":true},"cell_type":"code","source":"if run_full_notebook:\n    params = {\n        'boosting_type': 'gbdt',\n        'objective': 'multiclass',\n        'metric': 'multi_logloss',\n        'max_depth': 5,\n        'num_leaves': 31,\n        'learning_rate': 0.025,\n        'feature_fraction': 0.85,\n        'bagging_fraction': 0.85,\n        'bagging_freq': 5,\n        'num_threads': os.cpu_count(),\n        'lambda_l2': 1.0,\n        'min_gain_to_split': 0,\n        'num_class': n_categories,\n    }\n\n    # Create dataset\n    X_train, X_valid, y_train, y_valid, X_test, feature_names, labels = \\\n        data_split(train_df, test_df, test_size=0)\n    d_train = lgb.Dataset(X_train, label=y_train, feature_name=feature_names)","execution_count":98,"outputs":[]},{"metadata":{"_cell_guid":"c168a25a-0699-40c5-804f-ce41aa45eb40","_uuid":"61c886486f08b1a2412014ca8e226c427b9354a4","slideshow":{"slide_type":"slide"}},"cell_type":"markdown","source":"### Train, predict, create submission"},{"metadata":{"_cell_guid":"784b972e-cc20-4fb4-8dfd-8a3055efd218","_uuid":"c19636eb26cce3caf469eec6cdcacf64914c304a","collapsed":true,"slideshow":{"slide_type":"-"},"trusted":true},"cell_type":"code","source":"if run_full_notebook:\n    # Train and predict\n    print(\"Begin training...\")\n    clf = lgb.train(params, d_train, num_boost_round=1135)\n\n    print(\"Begin test predictions...\")\n    p = clf.predict(X_test)\n    lab = preds_to_labels(p, labels)\n\n    create_submission(lab, 'submission-test.csv')\n    print(\"Submission created.\")\n\n    # 0.836 LB","execution_count":99,"outputs":[]},{"metadata":{"_cell_guid":"e23cf88a-45b8-426a-a221-c32a99b16769","_uuid":"f5d351c3a658041002cd14740d786ac4a1a0cbdd","slideshow":{"slide_type":"slide"}},"cell_type":"markdown","source":"## Next steps beginner\n- Run this notebook yourself and submit the predictions\n- Kaggle user Zafar has a great notebook, based on neural networks\n    - https://www.kaggle.com/fizzbuzz/beginner-s-guide-to-audio-data\n    - Run his notebook and combine the predictions\n    - How do different methods of combing results effect your score? (mean, geometric mean, voting, ??)\n        - This could net you a top-5 place right now\n- Identify and modify any hyperparameters in this notebook"},{"metadata":{"_cell_guid":"43155c54-ffc7-44fd-8635-ccc442df11b8","_uuid":"2ac6956f0c944ea9ba49d741a784603d1d331499","slideshow":{"slide_type":"slide"}},"cell_type":"markdown","source":"## Next steps intermediate\n- Try different algorithms\n- Normalization of MFCC coefficients\n- New/more/less features\n- Add more data\n    - Either external datasets or by adding noise\n- Add test data with high confidence predictions to training data"},{"metadata":{"_cell_guid":"52acbfdd-a0ba-432f-ae51-135b416f3664","_uuid":"9669cd676ede9e810d9d002e3103c6ade0c556e5","slideshow":{"slide_type":"slide"}},"cell_type":"markdown","source":"## Next steps advanced\n- Apply a sliding window to augment data\n    - Can assume label is characterized by entire signal and average outputs\n    - Can also assume label is characterized best by single frame and use highest output\n- Build a neural network on MFCC's or spectrograms and either\n    - A) Append calculated features to dense layer of neural network\n    - B) Use low dimension bottleneck of neural network as features in other models\n- Ignore me and do something better"},{"metadata":{"_cell_guid":"e5f57d54-9a6d-4554-8fad-f39885ca766e","_uuid":"d0fc7d1870f104ba2c49030cb517561e35ab56ae","collapsed":true,"slideshow":{"slide_type":"skip"},"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"celltoolbar":"Slideshow","kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.5","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}