{"cells":[{"metadata":{},"cell_type":"markdown","source":"So, during the [Freesound General-Purpose Audio Tagging Challenge](Freesound General-Purpose Audio Tagging Challenge) I got into top 11% by using LightGBM and CatBoost, and running those on a large amount of manually extracted features. So, how well will this approach work this time? (Old code can be found [here](https://github.com/knstmrd/kagglefreesound))\nNot as well, unfortunately, due to a few reasons:\n1. The main reason is that previously I used VGGish, a pre-trained Tensorflow network which uses a VGG-type CNN to produce a 128-dimensional feature vector for an audiofile. Since it was pre-trained, it's not possible to use it here; and those features turned out to be one of the most important ones\n2. I used YAAFE and Essentia, two fast toolboxes that are unavailable in Kaggle kernels; so not only I'm limited by the speed of the feature extraction process, I'm also more limited in the features I can extract\n3. 2-hour kernel runtime limit; but it's still possible to get a somewhat passable result in this timeframe, and do all the feature extraction and training and prediction in under two hours (without any CV though); obviously, the pre-processing step and training steps can be separated from the classification step (pre-process, save dataset, train and save classifiers), but where's the fun in that?\n\nSo what I'm doing is\n1. Extract a lot of features (both from the raw wav file and from a spectrogram)\n2. Most of these are time-series; take various percentiles of these\n3. For time series also take time derivative and take various percentiles of that\n4. Remove some of the highly-correlated features\n\nTo account the multiple-labels, I did the following:\n1. Extract features for the train and test sets\n2. For each train example with N labels, add N-1 copies (each with a new label) to the train set\n3. During train, set sample weights like 1 / (1 + w * label_secondary), where w is some parameter, and label_secondary is either 1 or 0 (depending on whether it's the main or additional label); this basically weights the examples with more than 1 label somewhat lower if their non-main label is used\n\nAnd then just run LightGBM and XGB and average.\n\n**Possible improvements**\n1. Of course, one can fine-tune more classifier parameters; find better blending weights, etc., etc.\n2. Train a neural net and use some intermediate layer output as a feature set\n3. Forget about this whole approach altogether :)\n4. Find a way to utilize the noisy training dataset (some preliminary runs I did with it were pretty terrible)"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\nimport os\nprint(os.listdir(\"../input\"))\nimport librosa as lr\nimport glob\nimport lightgbm as lgbm\nimport xgboost as xgb\nimport time\nfrom scipy import stats\nfrom sklearn.model_selection import StratifiedKFold\nfrom sklearn.model_selection import ParameterSampler\nfrom sklearn.preprocessing import LabelEncoder\nfrom scipy.signal import filtfilt, argrelmax, find_peaks, correlate\nfrom scipy.stats import skew, kurtosis","execution_count":1,"outputs":[{"output_type":"stream","text":"['train_noisy.csv', 'test', 'train_curated', 'train_noisy', 'sample_submission.csv', 'train_curated.csv']\n","name":"stdout"}]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"srate = 44100  # loading without re-sampling is faster\nuse_noisy = True\nuse_noisy = False\n\ndo_cv = True\n# do_cv = False\n\nn_folds = 10\nactual_folds = 4\n\nn_mfcc = 20  # number of MFCC coefficients to use\n\nexpand_multi_label = True\n# expand_multi_label = False\nmulti_label_weight_multiplier = 0.5 # (w = 1 / (1+w.mult))","execution_count":26,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"# from official code https://colab.research.google.com/drive/1AgPdhSp7ttY18O3fEoHOQKlt_3HJDLi8#scrollTo=cRCaCIb9oguU\ndef _one_sample_positive_class_precisions(scores, truth):\n    \"\"\"Calculate precisions for each true class for a single sample.\n\n    Args:\n      scores: np.array of (num_classes,) giving the individual classifier scores.\n      truth: np.array of (num_classes,) bools indicating which classes are true.\n\n    Returns:\n      pos_class_indices: np.array of indices of the true classes for this sample.\n      pos_class_precisions: np.array of precisions corresponding to each of those\n        classes.\n    \"\"\"\n    num_classes = scores.shape[0]\n    pos_class_indices = np.flatnonzero(truth > 0)\n    # Only calculate precisions if there are some true classes.\n    if not len(pos_class_indices):\n        return pos_class_indices, np.zeros(0)\n    # Retrieval list of classes for this sample.\n    retrieved_classes = np.argsort(scores)[::-1]\n    # class_rankings[top_scoring_class_index] == 0 etc.\n    class_rankings = np.zeros(num_classes, dtype=np.int)\n    class_rankings[retrieved_classes] = range(num_classes)\n    # Which of these is a true label?\n    retrieved_class_true = np.zeros(num_classes, dtype=np.bool)\n    retrieved_class_true[class_rankings[pos_class_indices]] = True\n    # Num hits for every truncated retrieval list.\n    retrieved_cumulative_hits = np.cumsum(retrieved_class_true)\n    # Precision of retrieval list truncated at each hit, in order of pos_labels.\n    precision_at_hits = (\n            retrieved_cumulative_hits[class_rankings[pos_class_indices]] /\n            (1 + class_rankings[pos_class_indices].astype(np.float)))\n    return pos_class_indices, precision_at_hits\n\n\ndef calculate_per_class_lwlrap(truth, scores):\n    \"\"\"Calculate label-weighted label-ranking average precision.\n\n    Arguments:\n      truth: np.array of (num_samples, num_classes) giving boolean ground-truth\n        of presence of that class in that sample.\n      scores: np.array of (num_samples, num_classes) giving the classifier-under-\n        test's real-valued score for each class for each sample.\n\n    Returns:\n      per_class_lwlrap: np.array of (num_classes,) giving the lwlrap for each\n        class.\n      weight_per_class: np.array of (num_classes,) giving the prior of each\n        class within the truth labels.  Then the overall unbalanced lwlrap is\n        simply np.sum(per_class_lwlrap * weight_per_class)\n    \"\"\"\n    assert truth.shape == scores.shape\n    num_samples, num_classes = scores.shape\n    # Space to store a distinct precision value for each class on each sample.\n    # Only the classes that are true for each sample will be filled in.\n    precisions_for_samples_by_classes = np.zeros((num_samples, num_classes))\n    for sample_num in range(num_samples):\n        pos_class_indices, precision_at_hits = (\n            _one_sample_positive_class_precisions(scores[sample_num, :],\n                                                  truth[sample_num, :]))\n        precisions_for_samples_by_classes[sample_num, pos_class_indices] = (\n            precision_at_hits)\n    labels_per_class = np.sum(truth > 0, axis=0)\n    weight_per_class = labels_per_class / float(np.sum(labels_per_class))\n    # Form average of each column, i.e. all the precisions assigned to labels in\n    # a particular class.\n    per_class_lwlrap = (np.sum(precisions_for_samples_by_classes, axis=0) /\n                        np.maximum(1, labels_per_class))\n    # overall_lwlrap = simple average of all the actual per-class, per-sample precisions\n    #                = np.sum(precisions_for_samples_by_classes) / np.sum(precisions_for_samples_by_classes > 0)\n    #           also = weighted mean of per-class lwlraps, weighted by class label prior across samples\n    #                = np.sum(per_class_lwlrap * weight_per_class)\n    return per_class_lwlrap, weight_per_class\n\n\n# Wrapper for fast.ai library\ndef lwlrap(scores, truth, **kwargs):\n    score, weight = calculate_per_class_lwlrap(truth, scores)\n    return (score * weight).sum()","execution_count":3,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"t1 = time.time()\n\ndf_train_noisy = pd.read_csv('../input/train_noisy.csv')\ndf_train_curated = pd.read_csv('../input/train_curated.csv')\ndf_test = pd.read_csv('../input/sample_submission.csv')\nprint(len(df_test.columns) - 1, 'categories')\ndf_test = df_test[['fname']]\n\nprint(df_train_noisy.shape, df_train_curated.shape)\ntime_to_load_df = time.time() - t1\nprint('Time to load dataframes: ', time_to_load_df)","execution_count":4,"outputs":[{"output_type":"stream","text":"80 categories\n(19815, 2) (4970, 2)\nTime to load dataframes:  0.05104851722717285\n","name":"stdout"}]},{"metadata":{"trusted":true},"cell_type":"code","source":"t1 = time.time()\n\nlabels = []\n\nfor row in df_train_noisy['labels'].values:\n    for sublabel in row.split(','):\n        labels.append(sublabel)\n\ndf_train_noisy['labels_list'] = df_train_noisy['labels'].apply(lambda x: x.split(','))\ndf_train_noisy['label0'] = df_train_noisy['labels_list'].apply(lambda x: x[0])\ndf_train_noisy['labels_count'] = df_train_noisy['labels_list'].apply(lambda x: len(x))\ndf_train_noisy['label_secondary'] = 0\n\nfor row in df_train_curated['labels'].values:\n    for sublabel in row.split(','):\n        labels.append(sublabel)\n\ndf_train_curated['labels_list'] = df_train_curated['labels'].apply(lambda x: x.split(','))\ndf_train_curated['label0'] = df_train_curated['labels_list'].apply(lambda x: x[0])\ndf_train_curated['labels_count'] = df_train_curated['labels_list'].apply(lambda x: len(x))\ndf_train_curated['label_secondary'] = 0\n\nn_labels_max_curated = df_train_curated['labels_count'].max()\nn_labels_max_noisy = df_train_noisy['labels_count'].max()\nprint('Max labels, curated, noisy', n_labels_max_curated, n_labels_max_noisy)\n\nlabels = set(labels)\nprint(len(labels))\nlabels = list(labels)\n\nlenc = LabelEncoder()\nlenc.fit(labels)\n\ndf_train_curated['int_label0'] = lenc.transform(df_train_curated['label0'])\ndf_train_noisy['int_label0'] = lenc.transform(df_train_noisy['label0'])\n\n\ndf_train_curated['labels_list_int'] = df_train_curated['labels_list'].apply(lambda x: lenc.transform(x))\ndf_train_noisy['labels_list_int'] = df_train_noisy['labels_list'].apply(lambda x: lenc.transform(x))\n\n\n\ndf_train_curated['full_fname'] = df_train_curated['fname'].apply(lambda x: '../input/train_curated/' + x)\ndf_train_noisy['full_fname'] = df_train_noisy['fname'].apply(lambda x: '../input/train_noisy/' + x)\n\nprint(df_train_curated['int_label0'].nunique(), df_train_noisy['int_label0'].nunique())\n\ntime_to_process_labels = time.time() - t1\nprint('Time to process labels: ', time_to_process_labels)","execution_count":6,"outputs":[{"output_type":"stream","text":"Max labels, curated, noisy 6 7\n80\n78 80\nTime to process labels:  1.4047446250915527\n","name":"stdout"}]},{"metadata":{"trusted":true},"cell_type":"code","source":"def load_file_librosa(fname, sr=44100):\n#     print(lr.core.load(fname, sr=sr)[0].shape)\n    return lr.core.load(fname, sr=sr)[0]","execution_count":7,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def length(array, sample_rate=44100):\n    return array.shape[0] / sample_rate","execution_count":8,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def wav_features(array):\n    if len(array) == 0:\n        return [0.] * 16\n    else:\n        argmin = np.argmin(array)\n        argmax = np.argmax(array)\n        std = np.std(array, ddof=1)\n        if array[argmin] == 0.0:\n            min_arr_corr = array[argmin] + 1e-10\n        else:\n            min_arr_corr = array[argmin]\n        return [argmin / array.shape[0], argmax / array.shape[0], # 2\n                array[argmin], array[argmax], # 4\n                np.mean(array), # 5\n                np.percentile(array, 10), np.percentile(array, 25), # 7\n                np.percentile(array, 50),  # 8\n                np.percentile(array, 75), np.percentile(array, 90),  # 10\n                skew(array), kurtosis(array),  # 12\n                std,  # 13\n                array[argmax] / min_arr_corr]  # 15","execution_count":9,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_zcr(array, sample_rate=44100):\n    return np.sum(lr.core.zero_crossings(array)) / array.shape[0]","execution_count":10,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def wav_autocorrelation(array):\n    try:\n        if array.shape[0] > 3 * srate:\n            tmp_arr = array[:3 * srate]\n        else:\n            tmp_arr = array\n        autocorr = correlate(tmp_arr, tmp_arr)\n        autocorr = autocorr[autocorr.shape[0]//2:]\n        peaks = find_peaks(autocorr[:800])\n        if len(peaks[0]) == 0:\n            peakpos = 1000\n            peakval = 1\n        else:\n            peakpos = peaks[0][0]\n            peakval = autocorr[peaks[0][0]] / autocorr[0]\n        return {'wav_autocorr_peak_position': peakpos / srate,\n                'wav_autocorr_peak_value_normalized': peakval,\n                'wav_autocorr_ZCR': np.sum(lr.core.zero_crossings(autocorr[:800])) / 800}\n    except ValueError as e:\n        return {'wav_autocorr_peak_position': 1000 / srate,\n                'wav_autocorr_peak_value_normalized': 1,\n                'wav_autocorr_ZCR': 0}","execution_count":11,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def spec_features(spec, srate):\n    spectroid = lr.feature.spectral_centroid(sr=srate, S=spec)[0, :]\n    rolloff = lr.feature.spectral_rolloff(sr=srate, S=spec, roll_percent=0.85)[0, :]\n    rolloff50 = lr.feature.spectral_rolloff(sr=srate, S=spec, roll_percent=0.5)[0, :]\n    spec_max = np.max(spectroid)\n    spec_min = np.min(spectroid)\n    \n    roll_max = np.max(rolloff)\n    roll_min = np.min(rolloff)\n    \n    roll_max50 = np.max(rolloff50)\n    roll_min50 = np.min(rolloff50)\n    \n    if spectroid.shape[0] > 1:\n        grad_spec = spectroid[1:] - spectroid[:-1]\n    else:\n        grad_spec = [0.]\n    \n    grad_spec_max = np.max(grad_spec)\n    grad_spec_min = np.min(grad_spec)\n    \n    if rolloff.shape[0] > 1:\n        grad_rolloff = rolloff[1:] - rolloff[:-1]\n        grad_rolloff50 = rolloff50[1:] - rolloff50[:-1]\n    else:\n        grad_rolloff = [0.]\n        grad_rolloff50 = [0.]\n        \n    grad_rolloff_max = np.max(grad_rolloff)\n    grad_rolloff_min = np.min(grad_rolloff)\n    \n    grad_rolloff_max50 = np.max(grad_rolloff50)\n    grad_rolloff_min50 = np.min(grad_rolloff50)\n    \n    rms = lr.feature.rms(S=spec)\n    \n    rms = rms[0, :]\n    \n    if rms.shape[0] > 2:\n        grad_rms = rms[1:] - rms[:-1]\n    else:\n        rms = [0., 0.]\n        grad_rms = [0., 0.]\n    \n    if spec.shape[1] > 2:\n        sflux_a = spec[:, 1:]\n        sflux_b = spec[:, :-1]\n        \n        sflux = sflux_a / np.max(sflux_a, axis=0) - sflux_b / np.max(sflux_b, axis=0)\n        sflux = np.sum(sflux**2, axis=0)\n        \n    else:\n        sflux = [0., 0.]\n    \n    sflux = np.nan_to_num(sflux)\n    \n    return {'rms_kurtosis': kurtosis(rms),\n            'rms_skew': skew(rms),\n            'rms_mean': np.mean(rms),\n            'rms_std': np.std(rms, ddof=1),\n            'rms_median': np.median(rms),\n            \n            'd_rms_kurtosis': kurtosis(grad_rms),\n            'd_rms_skew': skew(grad_rms),\n            'd_rms_mean': np.mean(grad_rms),\n            'd_rms_std': np.std(grad_rms, ddof=1),\n            'd_rms_median': np.median(grad_rms),\n            \n            'sflux_perc10': np.percentile(sflux, 10),\n            'sflux_perc25': np.percentile(sflux, 25),\n            'sflux_perc75': np.percentile(sflux, 75),\n            'sflux_mean': np.mean(sflux),\n            'sflux_median': np.median(sflux),\n            'sflux_skew': skew(sflux),\n            'sflux_kurtosis': kurtosis(sflux),\n            \n            'spectroid_mean': np.mean(spectroid), 'spectroid_std': np.std(spectroid, ddof=1),\n            'spectroid_max_div_min': spec_max / (spec_min + 1e-20),\n            'spectroid_max': spec_max, 'spectroid_min': spec_min,\n            'spectroid_median': np.median(spectroid),\n            'spectroid_perc10': np.percentile(spectroid, 10),\n            'spectroid_perc25': np.percentile(spectroid, 25),\n            'spectroid_perc75': np.percentile(spectroid, 75),\n            'rolloff_mean': np.mean(rolloff), 'rolloff_std': np.std(rolloff, ddof=1),\n            'rolloff_max_div_min': roll_max / (roll_min + 1e-20),\n            'rolloff_max': roll_max, 'rolloff_min': roll_min,\n            'rolloff_median': np.median(rolloff),\n            'rolloff_perc10': np.percentile(rolloff, 10),\n            'rolloff_perc25': np.percentile(rolloff, 25),\n            'rolloff_perc75': np.percentile(rolloff, 75),\n            'd_spectroid_mean': np.mean(grad_spec), 'd_spectroid_std': np.std(grad_spec, ddof=1),\n            'd_spectroid_max_div_min': grad_spec_max / (grad_spec_min + 1e-20),\n            'd_spectroid_max': grad_spec_max, 'd_spectroid_min': grad_spec_min,\n            'd_spectroid_median': np.median(grad_spec),\n            'd_spectroid_perc10': np.percentile(grad_spec, 10),\n            'd_spectroid_perc25': np.percentile(grad_spec, 25),\n            'd_spectroid_perc75': np.percentile(grad_spec, 75),\n            'd_rolloff_mean': np.mean(grad_rolloff), 'd_rolloff_std': np.std(grad_rolloff, ddof=1),\n            'd_rolloff_max_div_min': grad_rolloff_max / (grad_rolloff_min + 1e-20),\n            'd_rolloff_max': grad_rolloff_max, 'd_rolloff_min': grad_rolloff_min,\n            'd_rolloff_median': np.median(grad_rolloff),\n            'd_rolloff_perc10': np.percentile(grad_rolloff, 10),\n            'd_rolloff_perc25': np.percentile(grad_rolloff, 25),\n            'd_rolloff_perc75': np.percentile(grad_rolloff, 75),\n        \n            'rolloff_mean50': np.mean(rolloff50),\n            'rolloff_std50': np.std(rolloff50, ddof=1),\n            'rolloff_max50': roll_max50,\n            'rolloff_min50': roll_min50,\n            'rolloff_median50': np.median(rolloff50),\n            'rolloff_perc1050': np.percentile(rolloff50, 10),\n            'rolloff_perc2550': np.percentile(rolloff50, 25),\n            'rolloff_perc7550': np.percentile(rolloff50, 75),\n            'd_rolloff_mean50': np.mean(grad_rolloff50),\n            'd_rolloff_std50': np.std(grad_rolloff50, ddof=1),\n            'd_rolloff_max50': grad_rolloff_max50,\n            'd_rolloff_min50': grad_rolloff_min50,\n            'd_rolloff_median50': np.median(grad_rolloff50),\n            'd_rolloff_perc1050': np.percentile(grad_rolloff50, 10),\n            'd_rolloff_perc2550': np.percentile(grad_rolloff50, 25),\n            'd_rolloff_perc7550': np.percentile(grad_rolloff50, 75),\n           }","execution_count":12,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def mfcc_features(spec, srate):\n    \n    mfcc = lr.feature.mfcc(sr=srate, S=spec, n_mfcc=n_mfcc)\n    output = []\n    \n    for i in range(n_mfcc):\n        output.append(np.mean(mfcc[i, :]))\n        output.append(np.std(mfcc[i, :], ddof=1))\n        output.append(skew(mfcc[i, :]))\n        output.append(kurtosis(mfcc[i, :]))\n    \n    return output","execution_count":13,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def extract_features_from_file_v1(fname):\n    audio = load_file_librosa(fname, sr=srate)\n    output = {'length, s': length(audio)}\n    output['ZCR'] = get_zcr(audio, sample_rate=srate)\n        \n    output['wav features'] = wav_features(audio)\n    output['wav autocorr'] = wav_autocorrelation(audio)\n    \n    if len(audio) < 3:\n        d_audio = np.zeros(2)\n    else:\n        d_audio = audio[1:] - audio[:-1]\n        \n    \n    output['d_wav features'] = wav_features(d_audio)\n    output['d_wav autocorr'] = wav_autocorrelation(d_audio)\n    \n    spec = np.abs(lr.core.stft(audio))\n    spec_mel = lr.feature.melspectrogram(sr=srate, S=spec**2)\n    output['spec features'] = spec_features(spec, srate)\n    output['mfcc features'] = mfcc_features(spec_mel, srate)\n    return output","execution_count":14,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def process_names_v1(df):\n    all_names = list(df['feats'][0].keys())\n    \n    for featname in ['length, s', 'ZCR']:\n        df[featname] = df['feats'].apply(lambda x: x[featname])\n    \n    for featname in ['wav_autocorr_peak_position', 'wav_autocorr_peak_value_normalized',\n                     'wav_autocorr_ZCR']:\n        df[featname] = df['feats'].apply(lambda x: x['wav autocorr'][featname])\n    \n    \n    \n    for i, funcname in enumerate(['argmin_rel', 'argmax_rel', 'min', 'max',\n                                  'mean', 'perc10', 'perc25', 'perc50',\n                                  'perc75', 'perc90', 'skew', 'kurtosis', 'std',\n                                  'max_div_min']):\n        df[' '.join(('wav', funcname))] = df['feats'].apply(lambda x: x['wav features'][i])\n        \n        \n    # dw/dt\n    for featname in ['wav_autocorr_peak_position', 'wav_autocorr_peak_value_normalized',\n                     'wav_autocorr_ZCR']:\n        df[' '.join(('d_wav', funcname))] = df['feats'].apply(lambda x: x['d_wav autocorr'][featname])\n    \n    \n    \n    for i, funcname in enumerate(['argmin_rel', 'argmax_rel', 'min', 'max',\n                                  'mean', 'perc10', 'perc25', 'perc50',\n                                  'perc75', 'perc90', 'skew', 'kurtosis', 'std',\n                                  'max_div_min']):\n        df[' '.join(('d_wav', funcname))] = df['feats'].apply(lambda x: x['d_wav features'][i])\n        \n    for i, funcname in enumerate(['rms_kurtosis', 'rms_skew',\n                                  'rms_mean', 'rms_median', 'rms_std',\n                                  'd_rms_kurtosis', 'd_rms_skew',\n                                  'd_rms_mean', 'd_rms_median', 'd_rms_std',\n                                  'sflux_perc10',\n                                  'sflux_perc25',\n                                  'sflux_perc75',\n                                  'sflux_mean',\n                                  'sflux_median',\n                                  'sflux_skew',\n                                  'sflux_kurtosis',\n                                  'spectroid_mean', 'spectroid_std',\n                                  'spectroid_max_div_min',\n                                  'spectroid_max', 'spectroid_min',\n                                  'rolloff_mean', 'rolloff_std',\n                                  'rolloff_max_div_min',\n                                  'rolloff_max', 'rolloff_min',\n                                  'd_spectroid_mean', 'd_spectroid_std',\n                                  'd_spectroid_max_div_min',\n                                  'd_spectroid_max', 'd_spectroid_min',\n                                  'd_rolloff_mean', 'd_rolloff_std',\n                                  'd_rolloff_max_div_min',\n                                  'd_rolloff_max', 'd_rolloff_min',\n                                  'spectroid_median', 'rolloff_median',\n                                  'd_spectroid_median', 'd_rolloff_median',\n                                  'spectroid_perc10', 'spectroid_perc25', 'spectroid_perc75',\n                                  'rolloff_perc10', 'rolloff_perc25', 'rolloff_perc75',\n                                  'd_spectroid_perc10', 'd_spectroid_perc25', 'd_spectroid_perc75',\n                                  'd_rolloff_perc10', 'd_rolloff_perc25', 'd_rolloff_perc75',\n                \n                                  'rolloff_mean50', 'rolloff_std50',\n                                  'rolloff_max50', 'rolloff_min50',\n                                  'rolloff_median50', 'd_rolloff_median50',\n                                  'd_rolloff_mean50', 'd_rolloff_std50',\n                                  'd_rolloff_max50', 'd_rolloff_min50',\n                                  'rolloff_perc1050', 'rolloff_perc2550', 'rolloff_perc7550',\n                                  'd_rolloff_perc1050', 'd_rolloff_perc2550', 'd_rolloff_perc7550'\n                                 ]):\n        df[' '.join(('spec', funcname))] = df['feats'].apply(lambda x: x['spec features'][funcname])\n    \n    counter = 0\n    for i in range(n_mfcc):\n        for k in ['mean', 'std', 'skew', 'curtosis']:\n            df[' '.join(('mfcc', str(i), k))] = df['feats'].apply(lambda x: x['mfcc features'][counter])\n            counter += 1\n    \n    df.drop(['feats'], axis=1, inplace=True)","execution_count":15,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if use_noisy:\n    df_train = pd.concat([df_train_curated, df_train_noisy], axis=0)\nelse:\n    df_train = df_train_curated\nprint(len(df_train))","execution_count":16,"outputs":[{"output_type":"stream","text":"4970\n","name":"stdout"}]},{"metadata":{"trusted":true},"cell_type":"code","source":"t1 = time.time()\n\ndf_train['feats'] = df_train['full_fname'].apply(lambda x: extract_features_from_file_v1(x))\nprocess_names_v1(df_train)\n\ntime_to_extract_v1_train = time.time() - t1\nprint('Time to extract v1 features from train set:', time_to_extract_v1_train)","execution_count":17,"outputs":[{"output_type":"stream","text":"/opt/conda/lib/python3.6/site-packages/scipy/signal/signaltools.py:491: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.\n  return x[reverse].conj()\n","name":"stderr"},{"output_type":"stream","text":"Time to extract v1 features from train set: 922.3083081245422\n","name":"stdout"}]},{"metadata":{"trusted":true},"cell_type":"code","source":"t1 = time.time()\n\ndf_test['feats'] = df_test['fname'].apply(lambda x: extract_features_from_file_v1('../input/test/' + x))\nprocess_names_v1(df_test)\n\ntime_to_extract_v1_test = time.time() - t1\nprint('Time to extract v1 features from test set:', time_to_extract_v1_test)","execution_count":18,"outputs":[{"output_type":"stream","text":"/opt/conda/lib/python3.6/site-packages/scipy/signal/signaltools.py:491: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.\n  return x[reverse].conj()\n","name":"stderr"},{"output_type":"stream","text":"Time to extract v1 features from test set: 247.37632632255554\n","name":"stdout"}]},{"metadata":{"trusted":true},"cell_type":"code","source":"feat_names = [col for col in df_train.columns if not (col.startswith('label')\n                                                      or col.endswith('fname') or col.startswith('int_label'))]\nprint('Total features, ', len(feat_names))","execution_count":19,"outputs":[{"output_type":"stream","text":"Total features,  182\n","name":"stdout"}]},{"metadata":{"trusted":true},"cell_type":"code","source":"if expand_multi_label:\n    df_train_multilabel = df_train[df_train['labels_count'] > 1]\n\n    startpos = len(df_train)\n    print(startpos, len(df_train_multilabel))\n    for row in df_train_multilabel.iterrows():\n        for i in range(1,row[1]['labels_count']):\n            df_train = df_train.append(row[1])\n            df_train.iloc[startpos, df_train.columns.get_loc('label0')] = row[1]['labels_list'][i]\n            df_train.iloc[startpos, df_train.columns.get_loc('int_label0')] = row[1]['labels_list_int'][i]\n            df_train.iloc[startpos, df_train.columns.get_loc('label_secondary')] = 1\n            startpos += 1","execution_count":20,"outputs":[{"output_type":"stream","text":"4970 701\n","name":"stdout"}]},{"metadata":{"trusted":true},"cell_type":"code","source":"real_classes = df_train['int_label0'].unique()\nreal_classes.sort()","execution_count":21,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"corr = df_train[feat_names].corr()\ncorr = corr.abs()\n\ncorrelation_threshold = 0.95\n\nupper = corr.where(np.triu(np.ones(corr.shape), k=1).astype(np.bool))\ncolumns_to_remove = [col for col in upper.columns if any(upper[col] > correlation_threshold)]\nprint(len(feat_names), columns_to_remove, len(columns_to_remove))\n\nfeat_names = [x for x in feat_names if x not in columns_to_remove]\nprint(len(feat_names), 'features left after removing highly correlated features')","execution_count":22,"outputs":[{"output_type":"stream","text":"182 ['wav perc50', 'wav perc90', 'd_wav max', 'd_wav perc75', 'd_wav perc90', 'spec rms_mean', 'spec rms_median', 'spec rolloff_max_div_min', 'spec spectroid_median', 'spec rolloff_median', 'spec spectroid_perc75', 'spec rolloff_perc10', 'spec rolloff_mean50', 'spec rolloff_max50', 'spec rolloff_median50', 'spec rolloff_perc7550', 'mfcc 1 mean', 'mfcc 1 std', 'mfcc 2 std', 'mfcc 3 std', 'mfcc 4 std', 'mfcc 5 std', 'mfcc 6 std', 'mfcc 7 std', 'mfcc 8 std', 'mfcc 9 std', 'mfcc 10 std', 'mfcc 11 std', 'mfcc 12 std', 'mfcc 13 std', 'mfcc 14 std', 'mfcc 15 std', 'mfcc 16 std', 'mfcc 17 std', 'mfcc 18 std', 'mfcc 19 std'] 36\n146 features left after removing highly correlated features\n","name":"stdout"}]},{"metadata":{"trusted":true},"cell_type":"code","source":"clf1 = lgbm.LGBMClassifier(n_estimators=1100, num_leaves=25, learning_rate=0.005,\n                           colsample_bytree=0.75, objective='multiclass', random_state=47,\n                           reg_alpha=0.09, reg_lambda=0.05)\n\nclf2 = xgb.XGBClassifier(max_depth=5, learning_rate=0.005, n_estimators=850, colsample_bytree=0.8,\n                         colsample_bylevel=0.95, object='multi:softmax')\n\nclf1.fit(df_train[feat_names], df_train['int_label0'],\n         sample_weight=1.0/(1.0 + multi_label_weight_multiplier * df_train['label_secondary']))\noutput = 0.3 * clf1.predict_proba(df_test[feat_names])\n\nclf2.fit(df_train[feat_names], df_train['int_label0'],\n         sample_weight=1.0/(1.0 + multi_label_weight_multiplier * df_train['label_secondary']))\noutput += 0.7 * clf2.predict_proba(df_test[feat_names])\n\nprint(output.shape, len(labels))\n    ","execution_count":28,"outputs":[{"output_type":"stream","text":"0\n","name":"stdout"},{"output_type":"error","ename":"KeyboardInterrupt","evalue":"","traceback":["\u001b[0;31m---------------------------------------------------------------------------\u001b[0m","\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)","\u001b[0;32m<ipython-input-28-783d0a517e0a>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m     37\u001b[0m             clf2.fit(X_train, y_train,\n\u001b[1;32m     38\u001b[0m                      sample_weight=1.0/(1.0 + multi_label_weight_multiplier\n\u001b[0;32m---> 39\u001b[0;31m                                         * df_train['label_secondary'].values[train_index]))\n\u001b[0m\u001b[1;32m     40\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     41\u001b[0m             \u001b[0mpreds_lgb\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mclf1\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpredict_proba\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX_test\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n","\u001b[0;32m/opt/conda/lib/python3.6/site-packages/xgboost/sklearn.py\u001b[0m in \u001b[0;36mfit\u001b[0;34m(self, X, y, sample_weight, eval_set, eval_metric, early_stopping_rounds, verbose, xgb_model, sample_weight_eval_set, callbacks)\u001b[0m\n\u001b[1;32m    711\u001b[0m                               \u001b[0mevals_result\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mevals_result\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mobj\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mobj\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfeval\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mfeval\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    712\u001b[0m                               \u001b[0mverbose_eval\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mverbose\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mxgb_model\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mxgb_model\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 713\u001b[0;31m                               callbacks=callbacks)\n\u001b[0m\u001b[1;32m    714\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    715\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mobjective\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mxgb_options\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"objective\"\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n","\u001b[0;32m/opt/conda/lib/python3.6/site-packages/xgboost/training.py\u001b[0m in \u001b[0;36mtrain\u001b[0;34m(params, dtrain, num_boost_round, evals, obj, feval, maximize, early_stopping_rounds, evals_result, verbose_eval, xgb_model, callbacks, learning_rates)\u001b[0m\n\u001b[1;32m    214\u001b[0m                            \u001b[0mevals\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mevals\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    215\u001b[0m                            \u001b[0mobj\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mobj\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfeval\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mfeval\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 216\u001b[0;31m                            xgb_model=xgb_model, callbacks=callbacks)\n\u001b[0m\u001b[1;32m    217\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    218\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n","\u001b[0;32m/opt/conda/lib/python3.6/site-packages/xgboost/training.py\u001b[0m in \u001b[0;36m_train_internal\u001b[0;34m(params, dtrain, num_boost_round, evals, obj, feval, xgb_model, callbacks)\u001b[0m\n\u001b[1;32m     72\u001b[0m         \u001b[0;31m# Skip the first update if it is a recovery step.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     73\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mversion\u001b[0m \u001b[0;34m%\u001b[0m \u001b[0;36m2\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 74\u001b[0;31m             \u001b[0mbst\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mupdate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdtrain\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mi\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mobj\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     75\u001b[0m             \u001b[0mbst\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msave_rabit_checkpoint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     76\u001b[0m             \u001b[0mversion\u001b[0m \u001b[0;34m+=\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n","\u001b[0;32m/opt/conda/lib/python3.6/site-packages/xgboost/core.py\u001b[0m in \u001b[0;36mupdate\u001b[0;34m(self, dtrain, iteration, fobj)\u001b[0m\n\u001b[1;32m   1108\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mfobj\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1109\u001b[0m             _check_call(_LIB.XGBoosterUpdateOneIter(self.handle, ctypes.c_int(iteration),\n\u001b[0;32m-> 1110\u001b[0;31m                                                     dtrain.handle))\n\u001b[0m\u001b[1;32m   1111\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1112\u001b[0m             \u001b[0mpred\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpredict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdtrain\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n","\u001b[0;31mKeyboardInterrupt\u001b[0m: "]}]},{"metadata":{"trusted":true},"cell_type":"code","source":"output_df = pd.DataFrame({'fname': df_test['fname']}) \n        \nfor i in range(output.shape[1]):\n    real_col_name = lenc.inverse_transform([real_classes[i]])[0]\n    output_df[real_col_name] = output[:, i]\n\nif output.shape[1] < len(labels):\n    for i in range(len(labels)):\n        if i not in real_classes:\n            real_col_name = lenc.inverse_transform([i])[0]\n            output_df[real_col_name] = np.zeros(output.shape[0])\n            \noutput_df.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}