{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Introduction\n\n**see V56 for the best result of LB632 -- Finally I beat the current best public kernel using Keras :) -- This probably be my last update on this kernel -- If you find this kernel helpful, please upvote**\n\n**Version upto V60 have a silly bug of 'if <-- elif' so that model selection is wrong **\n\nThis is my effort to do a `Keras` replication with comparable baseline to the great kernel of @mhiro2 https://www.kaggle.com/mhiro2/simple-2d-cnn-classifier-with-pytorch (and further improved by @peining), which in turns use the excellent pre-processed data of @daisukelab https://www.kaggle.com/daisukelab/creating-fat2019-preprocessed-data) -- Note that to inference to the private data in stage-2, you have to preprocess data yourself.\n\nOne change I made in a Keras version, in addition to a simple conv net, we can also use a pre-defined architectures [trained from scratch] `MobileNetV2`, `InceptionV3` and `Xception` where you can choose in the kernel. Also, many ideas borrow from a nice kernel of @voglinio https://www.kaggle.com/voglinio/keras-2d-model-5-fold-log-specgram-curated-only , I also borrow the SoftMax+BCE loss & TTA ideas from Giba's kernel (BTW, we all know Giba without having to mention his user :).\n\nI apologize that my code is not at all clean; some of the `pytorch` code is still here albeit not used.\n\n## Major Updates\n* V1 [CV680, LB574]\n* V4 [CV66x, LB576]\n* V5 [] Add image augmentation module\n* V9 [CV679] Add lwlrap TF metric (credit @rio114 : https://www.kaggle.com/rio114/keras-cnn-with-lwlrap-evaluation )\n* V11 [] Employ list of augmentations mentioned in https://github.com/sainathadapa/kaggle-freesound-audio-tagging/blob/master/approaches_all.md\n* V16 [] Add BCEwithLogits (use only with ACTIVATION = 'linear')\n* V17 add SimpleCNN similar to the pytorch baseline\n* V22 add Curated-Only, Train-augment options\n* V23 add CRNN model\n* **V30 LB598 with shallow CNN in 400s, set iteration to 150**\n* **V39 LB608 with CoarseDropout Augmentation**\n* V40 Simple Snapshot (Checkpoint) Ensemble\n* **V52 [CV811, LB616] MixUp+CoarseDropout : credit https://www.kaggle.com/mathormad/resnet50-v2-keras-focal-loss-mix-up **\n* **V56 [CV830, LB632] Change Architecture to get the best result **\n* V61 fix silly bugs on model selection\n"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import gc\nimport os\nimport pickle\nimport random\nimport time\nfrom collections import Counter, defaultdict\nfrom functools import partial\nfrom pathlib import Path\nfrom psutil import cpu_count\nimport matplotlib.pyplot as plt\n\nimport librosa\nimport numpy as np\nimport pandas as pd\nfrom PIL import Image\nfrom sklearn.model_selection import train_test_split\nfrom imgaug import augmenters as iaa\n#from skmultilearn.model_selection import iterative_train_test_split\n\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nfrom fastprogress import master_bar, progress_bar\nfrom torch.optim import Adam\nfrom torch.optim.lr_scheduler import CosineAnnealingLR\nfrom torch.utils.data import Dataset, DataLoader\nfrom torchvision.transforms import transforms","execution_count":1,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### utils"},{"metadata":{"trusted":true},"cell_type":"code","source":"NUM_CLASSES = 80\nSIZE=128\ncheckpoint_file = ['model_best1.h5', 'model_best2.h5', 'model_best3.h5']\n# See Version40 for 3 snapshots (or you can use only 1 which is normal run)\nEPOCHS = [432, 0, 0] #150 for inception, 100 for xception\nTTA = [19, 0, 0] #Number of test-time augmentation\nBATCH_SIZE = 32\n\nLR = 4e-4\nPATIENCE = 10 #ReduceOnPlateau option\nLR_FACTOR = 0.8 #ReduceOnPlateau option\nCURATED_ONLY = True # use only curated data for training\nTRAIN_AUGMENT = True # use augmentation for training data?\nVALID_AUGMENT = False\nMODEL = 'mobile' #'cnn8th' # choose among 'xception', 'inception', 'mobile', 'crnn', 'simple'\nSEED = 520\n\nUSE_MIXUP = True\nMIXUP_PROB = 0.275\n\n# No K-Fold implementation yet\n# NUM_K_FOLDS = 5 # how many folds (K) you gonna splits\n# NUM_MODEL_RUN = 5 # how many models (<= K) you gonna train [e.g. set to 1 for a simple train/test split]\n\n# if use BCEwithLogits loss, use Activation = 'linear' only\nACTIVATION = 'linear' \n# ACTIVATION = 'softmax'\n# ACTIVATION = 'sigmoid'\n\n# LOSS = 'categorical_crossentropy'\n# LOSS = 'binary_crossentropy' \nLOSS = 'BCEwithLogits' ","execution_count":2,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def seed_everything(seed):\n random.seed(seed)\n os.environ['PYTHONHASHSEED'] = str(seed)\n np.random.seed(seed)\n torch.manual_seed(seed)\n torch.cuda.manual_seed(seed)\n torch.backends.cudnn.deterministic = True\n\nseed_everything(SEED)","execution_count":3,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"# from official code https://colab.research.google.com/drive/1AgPdhSp7ttY18O3fEoHOQKlt_3HJDLi8#scrollTo=cRCaCIb9oguU\ndef _one_sample_positive_class_precisions(scores, truth):\n \"\"\"Calculate precisions for each true class for a single sample.\n\n Args:\n scores: np.array of (num_classes,) giving the individual classifier scores.\n truth: np.array of (num_classes,) bools indicating which classes are true.\n\n Returns:\n pos_class_indices: np.array of indices of the true classes for this sample.\n pos_class_precisions: np.array of precisions corresponding to each of those\n classes.\n \"\"\"\n num_classes = scores.shape[0]\n pos_class_indices = np.flatnonzero(truth > 0)\n # Only calculate precisions if there are some true classes.\n if not len(pos_class_indices):\n return pos_class_indices, np.zeros(0)\n # Retrieval list of classes for this sample.\n retrieved_classes = np.argsort(scores)[::-1]\n # class_rankings[top_scoring_class_index] == 0 etc.\n class_rankings = np.zeros(num_classes, dtype=np.int)\n class_rankings[retrieved_classes] = range(num_classes)\n # Which of these is a true label?\n retrieved_class_true = np.zeros(num_classes, dtype=np.bool)\n retrieved_class_true[class_rankings[pos_class_indices]] = True\n # Num hits for every truncated retrieval list.\n retrieved_cumulative_hits = np.cumsum(retrieved_class_true)\n # Precision of retrieval list truncated at each hit, in order of pos_labels.\n precision_at_hits = (\n retrieved_cumulative_hits[class_rankings[pos_class_indices]] /\n (1 + class_rankings[pos_class_indices].astype(np.float)))\n return pos_class_indices, precision_at_hits\n\n\ndef calculate_per_class_lwlrap(truth, scores):\n \"\"\"Calculate label-weighted label-ranking average precision.\n\n Arguments:\n truth: np.array of (num_samples, num_classes) giving boolean ground-truth\n of presence of that class in that sample.\n scores: np.array of (num_samples, num_classes) giving the classifier-under-\n test's real-valued score for each class for each sample.\n\n Returns:\n per_class_lwlrap: np.array of (num_classes,) giving the lwlrap for each\n class.\n weight_per_class: np.array of (num_classes,) giving the prior of each\n class within the truth labels. Then the overall unbalanced lwlrap is\n simply np.sum(per_class_lwlrap * weight_per_class)\n \"\"\"\n assert truth.shape == scores.shape\n num_samples, num_classes = scores.shape\n # Space to store a distinct precision value for each class on each sample.\n # Only the classes that are true for each sample will be filled in.\n precisions_for_samples_by_classes = np.zeros((num_samples, num_classes))\n for sample_num in range(num_samples):\n pos_class_indices, precision_at_hits = (\n _one_sample_positive_class_precisions(scores[sample_num, :],\n truth[sample_num, :]))\n precisions_for_samples_by_classes[sample_num, pos_class_indices] = (\n precision_at_hits)\n labels_per_class = np.sum(truth > 0, axis=0)\n weight_per_class = labels_per_class / float(np.sum(labels_per_class))\n # Form average of each column, i.e. all the precisions assigned to labels in\n # a particular class.\n per_class_lwlrap = (np.sum(precisions_for_samples_by_classes, axis=0) /\n np.maximum(1, labels_per_class))\n # overall_lwlrap = simple average of all the actual per-class, per-sample precisions\n # = np.sum(precisions_for_samples_by_classes) / np.sum(precisions_for_samples_by_classes > 0)\n # also = weighted mean of per-class lwlraps, weighted by class label prior across samples\n # = np.sum(per_class_lwlrap * weight_per_class)\n return per_class_lwlrap, weight_per_class","execution_count":4,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"import tensorflow as tf\n\n\n\n# from https://www.kaggle.com/rio114/keras-cnn-with-lwlrap-evaluation/\ndef tf_one_sample_positive_class_precisions(y_true, y_pred) :\n num_samples, num_classes = y_pred.shape\n \n # find true labels\n pos_class_indices = tf.where(y_true > 0) \n \n # put rank on each element\n retrieved_classes = tf.nn.top_k(y_pred, k=num_classes).indices\n sample_range = tf.zeros(shape=tf.shape(tf.transpose(y_pred)), dtype=tf.int32)\n sample_range = tf.add(sample_range, tf.range(tf.shape(y_pred)[0], delta=1))\n sample_range = tf.transpose(sample_range)\n sample_range = tf.reshape(sample_range, (-1,num_classes*tf.shape(y_pred)[0]))\n retrieved_classes = tf.reshape(retrieved_classes, (-1,num_classes*tf.shape(y_pred)[0]))\n retrieved_class_map = tf.concat((sample_range, retrieved_classes), axis=0)\n retrieved_class_map = tf.transpose(retrieved_class_map)\n retrieved_class_map = tf.reshape(retrieved_class_map, (tf.shape(y_pred)[0], num_classes, 2))\n \n class_range = tf.zeros(shape=tf.shape(y_pred), dtype=tf.int32)\n class_range = tf.add(class_range, tf.range(num_classes, delta=1))\n \n class_rankings = tf.scatter_nd(retrieved_class_map,\n class_range,\n tf.shape(y_pred))\n \n #pick_up ranks\n num_correct_until_correct = tf.gather_nd(class_rankings, pos_class_indices)\n\n # add one for division for \"presicion_at_hits\"\n num_correct_until_correct_one = tf.add(num_correct_until_correct, 1) \n num_correct_until_correct_one = tf.cast(num_correct_until_correct_one, tf.float32)\n \n # generate tensor [num_sample, predict_rank], \n # top-N predicted elements have flag, N is the number of positive for each sample.\n sample_label = pos_class_indices[:, 0] \n sample_label = tf.reshape(sample_label, (-1, 1))\n sample_label = tf.cast(sample_label, tf.int32)\n \n num_correct_until_correct = tf.reshape(num_correct_until_correct, (-1, 1))\n retrieved_class_true_position = tf.concat((sample_label, \n num_correct_until_correct), axis=1)\n retrieved_pos = tf.ones(shape=tf.shape(retrieved_class_true_position)[0], dtype=tf.int32)\n retrieved_class_true = tf.scatter_nd(retrieved_class_true_position, \n retrieved_pos, \n tf.shape(y_pred))\n # cumulate predict_rank\n retrieved_cumulative_hits = tf.cumsum(retrieved_class_true, axis=1)\n\n # find positive position\n pos_ret_indices = tf.where(retrieved_class_true > 0)\n\n # find cumulative hits\n correct_rank = tf.gather_nd(retrieved_cumulative_hits, pos_ret_indices) \n correct_rank = tf.cast(correct_rank, tf.float32)\n\n # compute presicion\n precision_at_hits = tf.truediv(correct_rank, num_correct_until_correct_one)\n\n return pos_class_indices, precision_at_hits\n\ndef tf_lwlrap(y_true, y_pred):\n num_samples, num_classes = y_pred.shape\n pos_class_indices, precision_at_hits = (tf_one_sample_positive_class_precisions(y_true, y_pred))\n pos_flgs = tf.cast(y_true > 0, tf.int32)\n labels_per_class = tf.reduce_sum(pos_flgs, axis=0)\n weight_per_class = tf.truediv(tf.cast(labels_per_class, tf.float32),\n tf.cast(tf.reduce_sum(labels_per_class), tf.float32))\n sum_precisions_by_classes = tf.zeros(shape=(num_classes), dtype=tf.float32) \n class_label = pos_class_indices[:,1]\n sum_precisions_by_classes = tf.unsorted_segment_sum(precision_at_hits,\n class_label,\n num_classes)\n labels_per_class = tf.cast(labels_per_class, tf.float32)\n labels_per_class = tf.add(labels_per_class, 1e-7)\n per_class_lwlrap = tf.truediv(sum_precisions_by_classes,\n tf.cast(labels_per_class, tf.float32))\n out = tf.cast(tf.tensordot(per_class_lwlrap, weight_per_class, axes=1), dtype=tf.float32)\n return out","execution_count":5,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras import backend as k\ndef BCEwithLogits(y_true, y_pred):\n return K.mean(K.binary_crossentropy(y_true, y_pred, from_logits=True), axis=-1)","execution_count":6,"outputs":[{"output_type":"stream","text":"Using TensorFlow backend.\n","name":"stderr"}]},{"metadata":{},"cell_type":"markdown","source":"### dataset"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"dataset_dir = Path('../input/freesound-audio-tagging-2019')\npreprocessed_dir = Path('../input/fat2019_prep_mels1')","execution_count":7,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"csvs = {\n 'train_curated': dataset_dir / 'train_curated.csv',\n #'train_noisy': dataset_dir / 'train_noisy.csv',\n 'train_noisy': preprocessed_dir / 'trn_noisy_best50s.csv',\n 'sample_submission': dataset_dir / 'sample_submission.csv',\n}\n\ndataset = {\n 'train_curated': dataset_dir / 'train_curated',\n 'train_noisy': dataset_dir / 'train_noisy',\n 'test': dataset_dir / 'test',\n}\n\nmels = {\n 'train_curated': preprocessed_dir / 'mels_train_curated.pkl',\n 'train_noisy': preprocessed_dir / 'mels_trn_noisy_best50s.pkl',\n 'test': preprocessed_dir / 'mels_test.pkl', # NOTE: this data doesn't work at 2nd stage\n}","execution_count":8,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_curated = pd.read_csv(csvs['train_curated'])\ntrain_noisy = pd.read_csv(csvs['train_noisy'])\nif CURATED_ONLY:\n train_df = train_curated\nelse:\n train_df = pd.concat([train_curated, train_noisy], sort=True, ignore_index=True)\ntrain_df.head()","execution_count":9,"outputs":[{"output_type":"execute_result","execution_count":9,"data":{"text/plain":" fname labels\n0 0006ae4e.wav Bark\n1 0019ef41.wav Raindrop\n2 001ec0ad.wav Finger_snapping\n3 0026c7cb.wav Run\n4 0026f116.wav Finger_snapping","text/html":"
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
fnamelabels
00006ae4e.wavBark
10019ef41.wavRaindrop
2001ec0ad.wavFinger_snapping
30026c7cb.wavRun
40026f116.wavFinger_snapping
\n
"},"metadata":{}}]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df = pd.read_csv(csvs['sample_submission'])\ntest_df.head()","execution_count":10,"outputs":[{"output_type":"execute_result","execution_count":10,"data":{"text/plain":" fname ... Zipper_(clothing)\n0 000ccb97.wav ... 0\n1 0012633b.wav ... 0\n2 001ed5f1.wav ... 0\n3 00294be0.wav ... 0\n4 003fde7a.wav ... 0\n\n[5 rows x 81 columns]","text/html":"
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
fnameAccelerating_and_revving_and_vroomAccordionAcoustic_guitarApplauseBarkBass_drumBass_guitarBathtub_(filling_or_washing)Bicycle_bellBurping_and_eructationBusBuzzCar_passing_byCheeringChewing_and_masticationChild_speech_and_kid_speakingChink_and_clinkChirp_and_tweetChurch_bellClappingComputer_keyboardCrackleCricketCrowdCupboard_open_or_closeCutlery_and_silverwareDishes_and_pots_and_pansDrawer_open_or_closeDripElectric_guitarFartFemale_singingFemale_speech_and_woman_speakingFill_(with_liquid)Finger_snappingFrying_(food)GaspGlockenspielGong...HarmonicaHi-hatHissKeys_janglingKnockMale_singingMale_speech_and_man_speakingMarimba_and_xylophoneMechanical_fanMeowMicrowave_ovenMotorcyclePrinterPurrRace_car_and_auto_racingRaindropRunScissorsScreamingShatterSighSink_(filling_or_washing)SkateboardSlamSneezeSqueakStreamStrumTapTick-tockToilet_flushTraffic_noise_and_roadway_noiseTrickle_and_dribbleWalk_and_footstepsWater_tap_and_faucetWaves_and_surfWhisperingWritingYellZipper_(clothing)
0000ccb97.wav000000000000000000000000000000000000000...0000000000000000000000000000000000000000
10012633b.wav000000000000000000000000000000000000000...0000000000000000000000000000000000000000
2001ed5f1.wav000000000000000000000000000000000000000...0000000000000000000000000000000000000000
300294be0.wav000000000000000000000000000000000000000...0000000000000000000000000000000000000000
4003fde7a.wav000000000000000000000000000000000000000...0000000000000000000000000000000000000000
\n
"},"metadata":{}}]},{"metadata":{"trusted":true},"cell_type":"code","source":"labels = test_df.columns[1:].tolist()\nlabels[:10]","execution_count":11,"outputs":[{"output_type":"execute_result","execution_count":11,"data":{"text/plain":"['Accelerating_and_revving_and_vroom',\n 'Accordion',\n 'Acoustic_guitar',\n 'Applause',\n 'Bark',\n 'Bass_drum',\n 'Bass_guitar',\n 'Bathtub_(filling_or_washing)',\n 'Bicycle_bell',\n 'Burping_and_eructation']"},"metadata":{}}]},{"metadata":{"trusted":true},"cell_type":"code","source":"num_classes = len(labels)\nnum_classes","execution_count":12,"outputs":[{"output_type":"execute_result","execution_count":12,"data":{"text/plain":"80"},"metadata":{}}]},{"metadata":{"trusted":true},"cell_type":"code","source":"y_train = np.zeros((len(train_df), num_classes)).astype(int)\nfor i, row in enumerate(train_df['labels'].str.split(',')):\n for label in row:\n idx = labels.index(label)\n y_train[i, idx] = 1\n\ny_train.shape","execution_count":13,"outputs":[{"output_type":"execute_result","execution_count":13,"data":{"text/plain":"(4970, 80)"},"metadata":{}}]},{"metadata":{"trusted":true},"cell_type":"code","source":"with open(mels['train_curated'], 'rb') as curated, open(mels['train_noisy'], 'rb') as noisy:\n x_train = pickle.load(curated)\n if CURATED_ONLY == False:\n x_train.extend(pickle.load(noisy))\n\nwith open(mels['test'], 'rb') as test:\n x_test = pickle.load(test)\n \nlen(x_train), len(x_test)","execution_count":14,"outputs":[{"output_type":"execute_result","execution_count":14,"data":{"text/plain":"(4970, 1120)"},"metadata":{}}]},{"metadata":{"trusted":true},"cell_type":"code","source":"\nfor ii in range(5):\n print(x_train[ii].shape) #x_train is of shape (TRAIN_NUM,128,LEN,3) [4D Tensor]\n print(x_test[ii].shape,'\\n') #x_test of shape (TEST_NUM,128,LEN,3) [4D Tensor]","execution_count":15,"outputs":[{"output_type":"stream","text":"(128, 448, 3)\n(128, 128, 3) \n\n(128, 131, 3)\n(128, 1021, 3) \n\n(128, 128, 3)\n(128, 300, 3) \n\n(128, 1623, 3)\n(128, 1146, 3) \n\n(128, 128, 3)\n(128, 1442, 3) \n\n","name":"stdout"}]},{"metadata":{},"cell_type":"markdown","source":"### model"},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"from keras.layers import *\nfrom keras.models import Sequential, load_model, Model\nfrom keras import metrics\nfrom keras.optimizers import Adam \nfrom keras import backend as K\nimport keras\nfrom keras.models import Model\nfrom keras.applications.inception_v3 import InceptionV3\nfrom keras.applications.inception_v3 import preprocess_input as preprocess_inception\nfrom keras.applications.mobilenet_v2 import MobileNetV2\nfrom keras.applications.mobilenet_v2 import preprocess_input as preprocess_mobile\nfrom keras.applications.xception import Xception\nfrom keras.applications.xception import preprocess_input as preprocess_xception\n\nfrom keras.utils import Sequence\nfrom sklearn.utils import shuffle\ndef create_model_inception(n_out=NUM_CLASSES):\n\n base_model =InceptionV3(weights=None, include_top=False)\n \n x0 = base_model.output\n x1 = GlobalAveragePooling2D()(x0)\n x2 = GlobalMaxPooling2D()(x0)\n x = Concatenate()([x1,x2])\n \n x = BatchNormalization()(x)\n x = Dropout(0.5)(x)\n \n x = Dense(256, activation='relu')(x)\n x = BatchNormalization()(x)\n x = Dropout(0.5)(x)\n\n \n predictions = Dense(n_out, activation=ACTIVATION)(x)\n\n # this is the model we will train\n model = Model(inputs=base_model.input, outputs=predictions)\n return model","execution_count":16,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def create_model_xception(n_out=NUM_CLASSES):\n\n base_model = Xception(weights=None, include_top=False)\n \n x0 = base_model.output\n x1 = GlobalAveragePooling2D()(x0)\n x2 = GlobalMaxPooling2D()(x0)\n x = Concatenate()([x1,x2])\n \n x = BatchNormalization()(x)\n x = Dropout(0.5)(x)\n \n x = Dense(256, activation='relu')(x)\n x = BatchNormalization()(x)\n x = Dropout(0.5)(x)\n\n# x = Dense(128, activation='relu')(x)\n# x = BatchNormalization()(x)\n# x = Dropout(0.3)(x)\n \n predictions = Dense(n_out, activation=ACTIVATION)(x)\n\n # this is the model we will train\n model = Model(inputs=base_model.input, outputs=predictions)\n return model","execution_count":17,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"def create_model_mobile(n_out=NUM_CLASSES):\n\n base_model =MobileNetV2(weights=None, include_top=False)\n \n x0 = base_model.output\n x1 = GlobalAveragePooling2D()(x0)\n x2 = GlobalMaxPooling2D()(x0)\n x = Concatenate()([x1,x2])\n \n x = BatchNormalization()(x)\n x = Dropout(0.5)(x)\n \n x = Dense(256, activation='relu')(x)\n x = BatchNormalization()(x)\n x = Dropout(0.5)(x)\n\n# x = Dense(128, activation='relu')(x)\n# x = BatchNormalization()(x)\n# x = Dropout(0.25)(x)\n\n \n predictions = Dense(n_out, activation=ACTIVATION)(x)\n\n # this is the model we will train\n model = Model(inputs=base_model.input, outputs=predictions)\n return model","execution_count":18,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def conv_simple_block(x, n_filters):\n \n x = Convolution2D(n_filters, (3,1), padding=\"same\")(x)\n x = BatchNormalization()(x)\n x = Activation(\"relu\")(x)\n \n x = Convolution2D(n_filters, (3,1), padding=\"same\")(x)\n x = BatchNormalization()(x)\n x = Activation(\"relu\")(x)\n x = AveragePooling2D()(x)\n\n return x\n\ndef create_model_simplecnn(n_out=NUM_CLASSES):\n \n inp = Input(shape=(128,128,3))\n# inp = Input(shape=(None,None,3))\n x = conv_simple_block(inp,64)\n x = conv_simple_block(x,128)\n x = conv_simple_block(x,256)\n x = conv_simple_block(x,128)\n \n# x1 = GlobalAveragePooling2D()(x)\n# x2 = GlobalMaxPooling2D()(x)\n# x = Add()([x1,x2])\n\n x = Flatten()(x)\n x = Dropout(0.2)(x)\n\n x = Dense(128, activation='linear')(x)\n x = PReLU()(x)\n x = BatchNormalization()(x)\n x = Dropout(0.2)(x)\n predictions = Dense(n_out, activation=ACTIVATION)(x)\n\n model = Model(inputs=inp, outputs=predictions)\n return model","execution_count":19,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def output_of_lambda(input_shape):\n return (input_shape[0], input_shape[2], input_shape[3])\n\ndef my_max(x):\n return K.max(x, axis=1, keepdims=False)\n\ndef crnn_simple_block(x, n_filters):\n \n x = Convolution2D(n_filters, (3,1), padding=\"same\")(x)\n x = Activation(\"relu\")(x)\n \n x = Convolution2D(n_filters, (3,1), padding=\"same\")(x)\n x = Activation(\"relu\")(x)\n x = MaxPooling2D()(x)\n x = Dropout(0.2)(x)\n\n return x\n\ndef create_model_crnn(n_out=NUM_CLASSES):\n \n# inp = Input(shape=(128,128,3))\n inp = Input(shape=(128,None,3))\n x = crnn_simple_block(inp,64)\n x = crnn_simple_block(x,128)\n x = crnn_simple_block(x,256)\n \n # eliminate the frequency dimension, x = (batch, time, channels)\n x = Lambda(my_max, output_shape=output_of_lambda)(x)\n \n x = Bidirectional(CuDNNGRU(128, return_sequences=True))(x)\n# x = Bidirectional(CuDNNLSTM(64, return_sequences=True))(x)\n x = GlobalMaxPooling1D()(x)\n x = Dense(128, activation='linear')(x)\n x = PReLU()(x)\n x = BatchNormalization()(x)\n x = Dropout(0.2)(x)\n predictions = Dense(n_out, activation=ACTIVATION)(x)\n\n model = Model(inputs=inp, outputs=predictions)\n return model","execution_count":20,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# from the 8th solution in 2018 competition\n# https://github.com/sainathadapa/kaggle-freesound-audio-tagging\ndef create_model_cnn8th(n_out=NUM_CLASSES):\n regu=0\n inp = Input(shape=(128,128,3))\n\n x = Conv2D(48, 11, strides=(1,1),kernel_initializer='he_uniform', activation='relu', padding='same',kernel_regularizer=regularizers.l2(regu))(inp)\n x = BatchNormalization()(x)\n x = Conv2D(48, 11, strides=(2,3),kernel_initializer='he_uniform', activation='relu', padding='same',kernel_regularizer=regularizers.l2(regu))(x)\n x = MaxPooling2D(3, strides=(1,2))(x)\n x = BatchNormalization()(x)\n\n x = Conv2D(128, 5, strides=(1,1),kernel_initializer='he_uniform', activation='relu', padding='same',kernel_regularizer=regularizers.l2(regu))(x)\n x = BatchNormalization()(x)\n x = Conv2D(128, 5, strides=(2,3),kernel_initializer='he_uniform', activation='relu', padding='same',kernel_regularizer=regularizers.l2(regu))(x)\n x = MaxPooling2D(3, strides=2)(x)\n x = BatchNormalization()(x)\n\n x = Conv2D(192, 3, strides=1,kernel_initializer='he_uniform', activation='relu', padding='same')(x)\n x = BatchNormalization()(x)\n x = Conv2D(192, 3, strides=1,kernel_initializer='he_uniform', activation='relu', padding='same')(x)\n x = BatchNormalization()(x)\n x = Conv2D(128, 3, strides=1,kernel_initializer='he_uniform', activation='relu', padding='same',kernel_regularizer=regularizers.l2(regu))(x)\n x = MaxPooling2D(3, strides=(1,2))(x)\n x = BatchNormalization()(x)\n\n x = Flatten()(x)\n x = Dense(256, activation='relu')(x)\n x = Dropout(0.5)(x)\n x = Dense(256, activation='relu')(x)\n x = Dropout(0.5)(x)\n\n predictions = Dense(n_out, activation=ACTIVATION)(x)\n\n model = Model(inputs=inp, outputs=predictions)\n return model","execution_count":21,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"K.clear_session()\n'''Choose your model here'''\nif MODEL == 'xception':\n preprocess_input = preprocess_xception\n model = create_model_xception(n_out=NUM_CLASSES)\nelif MODEL == 'inception':\n preprocess_input = preprocess_inception\n model = create_model_inception(n_out=NUM_CLASSES)\nelif MODEL == 'mobile':\n preprocess_input = preprocess_mobile\n model = create_model_mobile(n_out=NUM_CLASSES)\nelif MODEL == 'crnn':\n preprocess_input = preprocess_mobile\n model = create_model_crnn(n_out=NUM_CLASSES)\nelif MODEL == 'cnn8th':\n preprocess_input = preprocess_mobile\n model = create_model_cnn8th(n_out=NUM_CLASSES)\nelse:\n preprocess_input = preprocess_mobile\n model = create_model_simplecnn(n_out=NUM_CLASSES)\n\nprint(MODEL)\nmodel.summary()","execution_count":23,"outputs":[{"output_type":"stream","text":"mobile\n__________________________________________________________________________________________________\nLayer (type) Output Shape Param # Connected to \n==================================================================================================\ninput_1 (InputLayer) (None, None, None, 3 0 \n__________________________________________________________________________________________________\nConv1_pad (ZeroPadding2D) (None, None, None, 3 0 input_1[0][0] \n__________________________________________________________________________________________________\nConv1 (Conv2D) (None, None, None, 3 864 Conv1_pad[0][0] \n__________________________________________________________________________________________________\nbn_Conv1 (BatchNormalization) (None, None, None, 3 128 Conv1[0][0] \n__________________________________________________________________________________________________\nConv1_relu (ReLU) (None, None, None, 3 0 bn_Conv1[0][0] \n__________________________________________________________________________________________________\nexpanded_conv_depthwise (Depthw (None, None, None, 3 288 Conv1_relu[0][0] \n__________________________________________________________________________________________________\nexpanded_conv_depthwise_BN (Bat (None, None, None, 3 128 expanded_conv_depthwise[0][0] \n__________________________________________________________________________________________________\nexpanded_conv_depthwise_relu (R (None, None, None, 3 0 expanded_conv_depthwise_BN[0][0] \n__________________________________________________________________________________________________\nexpanded_conv_project (Conv2D) (None, None, None, 1 512 expanded_conv_depthwise_relu[0][0\n__________________________________________________________________________________________________\nexpanded_conv_project_BN (Batch (None, None, None, 1 64 expanded_conv_project[0][0] \n__________________________________________________________________________________________________\nblock_1_expand (Conv2D) (None, None, None, 9 1536 expanded_conv_project_BN[0][0] \n__________________________________________________________________________________________________\nblock_1_expand_BN (BatchNormali (None, None, None, 9 384 block_1_expand[0][0] \n__________________________________________________________________________________________________\nblock_1_expand_relu (ReLU) (None, None, None, 9 0 block_1_expand_BN[0][0] \n__________________________________________________________________________________________________\nblock_1_pad (ZeroPadding2D) (None, None, None, 9 0 block_1_expand_relu[0][0] \n__________________________________________________________________________________________________\nblock_1_depthwise (DepthwiseCon (None, None, None, 9 864 block_1_pad[0][0] \n__________________________________________________________________________________________________\nblock_1_depthwise_BN (BatchNorm (None, None, None, 9 384 block_1_depthwise[0][0] \n__________________________________________________________________________________________________\nblock_1_depthwise_relu (ReLU) (None, None, None, 9 0 block_1_depthwise_BN[0][0] \n__________________________________________________________________________________________________\nblock_1_project (Conv2D) (None, None, None, 2 2304 block_1_depthwise_relu[0][0] \n__________________________________________________________________________________________________\nblock_1_project_BN (BatchNormal (None, None, None, 2 96 block_1_project[0][0] \n__________________________________________________________________________________________________\nblock_2_expand (Conv2D) (None, None, None, 1 3456 block_1_project_BN[0][0] \n__________________________________________________________________________________________________\nblock_2_expand_BN (BatchNormali (None, None, None, 1 576 block_2_expand[0][0] \n__________________________________________________________________________________________________\nblock_2_expand_relu (ReLU) (None, None, None, 1 0 block_2_expand_BN[0][0] \n__________________________________________________________________________________________________\nblock_2_depthwise (DepthwiseCon (None, None, None, 1 1296 block_2_expand_relu[0][0] \n__________________________________________________________________________________________________\nblock_2_depthwise_BN (BatchNorm (None, None, None, 1 576 block_2_depthwise[0][0] \n__________________________________________________________________________________________________\nblock_2_depthwise_relu (ReLU) (None, None, None, 1 0 block_2_depthwise_BN[0][0] \n__________________________________________________________________________________________________\nblock_2_project (Conv2D) (None, None, None, 2 3456 block_2_depthwise_relu[0][0] \n__________________________________________________________________________________________________\nblock_2_project_BN (BatchNormal (None, None, None, 2 96 block_2_project[0][0] \n__________________________________________________________________________________________________\nblock_2_add (Add) (None, None, None, 2 0 block_1_project_BN[0][0] \n block_2_project_BN[0][0] \n__________________________________________________________________________________________________\nblock_3_expand (Conv2D) (None, None, None, 1 3456 block_2_add[0][0] \n__________________________________________________________________________________________________\nblock_3_expand_BN (BatchNormali (None, None, None, 1 576 block_3_expand[0][0] \n__________________________________________________________________________________________________\nblock_3_expand_relu (ReLU) (None, None, None, 1 0 block_3_expand_BN[0][0] \n__________________________________________________________________________________________________\nblock_3_pad (ZeroPadding2D) (None, None, None, 1 0 block_3_expand_relu[0][0] \n__________________________________________________________________________________________________\nblock_3_depthwise (DepthwiseCon (None, None, None, 1 1296 block_3_pad[0][0] \n__________________________________________________________________________________________________\nblock_3_depthwise_BN (BatchNorm (None, None, None, 1 576 block_3_depthwise[0][0] \n__________________________________________________________________________________________________\nblock_3_depthwise_relu (ReLU) (None, None, None, 1 0 block_3_depthwise_BN[0][0] \n__________________________________________________________________________________________________\nblock_3_project (Conv2D) (None, None, None, 3 4608 block_3_depthwise_relu[0][0] \n__________________________________________________________________________________________________\nblock_3_project_BN (BatchNormal (None, None, None, 3 128 block_3_project[0][0] \n__________________________________________________________________________________________________\nblock_4_expand (Conv2D) (None, None, None, 1 6144 block_3_project_BN[0][0] \n__________________________________________________________________________________________________\nblock_4_expand_BN (BatchNormali (None, None, None, 1 768 block_4_expand[0][0] \n__________________________________________________________________________________________________\nblock_4_expand_relu (ReLU) (None, None, None, 1 0 block_4_expand_BN[0][0] \n__________________________________________________________________________________________________\nblock_4_depthwise (DepthwiseCon (None, None, None, 1 1728 block_4_expand_relu[0][0] \n__________________________________________________________________________________________________\nblock_4_depthwise_BN (BatchNorm (None, None, None, 1 768 block_4_depthwise[0][0] \n__________________________________________________________________________________________________\nblock_4_depthwise_relu (ReLU) (None, None, None, 1 0 block_4_depthwise_BN[0][0] \n__________________________________________________________________________________________________\nblock_4_project (Conv2D) (None, None, None, 3 6144 block_4_depthwise_relu[0][0] \n__________________________________________________________________________________________________\nblock_4_project_BN (BatchNormal (None, None, None, 3 128 block_4_project[0][0] \n__________________________________________________________________________________________________\nblock_4_add (Add) (None, None, None, 3 0 block_3_project_BN[0][0] \n block_4_project_BN[0][0] \n__________________________________________________________________________________________________\nblock_5_expand (Conv2D) (None, None, None, 1 6144 block_4_add[0][0] \n__________________________________________________________________________________________________\nblock_5_expand_BN (BatchNormali (None, None, None, 1 768 block_5_expand[0][0] \n__________________________________________________________________________________________________\nblock_5_expand_relu (ReLU) (None, None, None, 1 0 block_5_expand_BN[0][0] \n__________________________________________________________________________________________________\nblock_5_depthwise (DepthwiseCon (None, None, None, 1 1728 block_5_expand_relu[0][0] \n__________________________________________________________________________________________________\nblock_5_depthwise_BN (BatchNorm (None, None, None, 1 768 block_5_depthwise[0][0] \n__________________________________________________________________________________________________\nblock_5_depthwise_relu (ReLU) (None, None, None, 1 0 block_5_depthwise_BN[0][0] \n__________________________________________________________________________________________________\nblock_5_project (Conv2D) (None, None, None, 3 6144 block_5_depthwise_relu[0][0] \n__________________________________________________________________________________________________\nblock_5_project_BN (BatchNormal (None, None, None, 3 128 block_5_project[0][0] \n__________________________________________________________________________________________________\nblock_5_add (Add) (None, None, None, 3 0 block_4_add[0][0] \n block_5_project_BN[0][0] \n__________________________________________________________________________________________________\nblock_6_expand (Conv2D) (None, None, None, 1 6144 block_5_add[0][0] \n__________________________________________________________________________________________________\nblock_6_expand_BN (BatchNormali (None, None, None, 1 768 block_6_expand[0][0] \n__________________________________________________________________________________________________\nblock_6_expand_relu (ReLU) (None, None, None, 1 0 block_6_expand_BN[0][0] \n__________________________________________________________________________________________________\nblock_6_pad (ZeroPadding2D) (None, None, None, 1 0 block_6_expand_relu[0][0] \n__________________________________________________________________________________________________\nblock_6_depthwise (DepthwiseCon (None, None, None, 1 1728 block_6_pad[0][0] \n__________________________________________________________________________________________________\nblock_6_depthwise_BN (BatchNorm (None, None, None, 1 768 block_6_depthwise[0][0] \n__________________________________________________________________________________________________\nblock_6_depthwise_relu (ReLU) (None, None, None, 1 0 block_6_depthwise_BN[0][0] \n__________________________________________________________________________________________________\nblock_6_project (Conv2D) (None, None, None, 6 12288 block_6_depthwise_relu[0][0] \n__________________________________________________________________________________________________\nblock_6_project_BN (BatchNormal (None, None, None, 6 256 block_6_project[0][0] \n__________________________________________________________________________________________________\nblock_7_expand (Conv2D) (None, None, None, 3 24576 block_6_project_BN[0][0] \n__________________________________________________________________________________________________\nblock_7_expand_BN (BatchNormali (None, None, None, 3 1536 block_7_expand[0][0] \n__________________________________________________________________________________________________\nblock_7_expand_relu (ReLU) (None, None, None, 3 0 block_7_expand_BN[0][0] \n__________________________________________________________________________________________________\nblock_7_depthwise (DepthwiseCon (None, None, None, 3 3456 block_7_expand_relu[0][0] \n__________________________________________________________________________________________________\nblock_7_depthwise_BN (BatchNorm (None, None, None, 3 1536 block_7_depthwise[0][0] \n__________________________________________________________________________________________________\nblock_7_depthwise_relu (ReLU) (None, None, None, 3 0 block_7_depthwise_BN[0][0] \n__________________________________________________________________________________________________\nblock_7_project (Conv2D) (None, None, None, 6 24576 block_7_depthwise_relu[0][0] \n__________________________________________________________________________________________________\nblock_7_project_BN (BatchNormal (None, None, None, 6 256 block_7_project[0][0] \n__________________________________________________________________________________________________\nblock_7_add (Add) (None, None, None, 6 0 block_6_project_BN[0][0] \n block_7_project_BN[0][0] \n__________________________________________________________________________________________________\nblock_8_expand (Conv2D) (None, None, None, 3 24576 block_7_add[0][0] \n__________________________________________________________________________________________________\nblock_8_expand_BN (BatchNormali (None, None, None, 3 1536 block_8_expand[0][0] \n__________________________________________________________________________________________________\nblock_8_expand_relu (ReLU) (None, None, None, 3 0 block_8_expand_BN[0][0] \n__________________________________________________________________________________________________\nblock_8_depthwise (DepthwiseCon (None, None, None, 3 3456 block_8_expand_relu[0][0] \n__________________________________________________________________________________________________\nblock_8_depthwise_BN (BatchNorm (None, None, None, 3 1536 block_8_depthwise[0][0] \n__________________________________________________________________________________________________\nblock_8_depthwise_relu (ReLU) (None, None, None, 3 0 block_8_depthwise_BN[0][0] \n__________________________________________________________________________________________________\nblock_8_project (Conv2D) (None, None, None, 6 24576 block_8_depthwise_relu[0][0] \n__________________________________________________________________________________________________\nblock_8_project_BN (BatchNormal (None, None, None, 6 256 block_8_project[0][0] \n__________________________________________________________________________________________________\nblock_8_add (Add) (None, None, None, 6 0 block_7_add[0][0] \n block_8_project_BN[0][0] \n__________________________________________________________________________________________________\nblock_9_expand (Conv2D) (None, None, None, 3 24576 block_8_add[0][0] \n__________________________________________________________________________________________________\nblock_9_expand_BN (BatchNormali (None, None, None, 3 1536 block_9_expand[0][0] \n__________________________________________________________________________________________________\nblock_9_expand_relu (ReLU) (None, None, None, 3 0 block_9_expand_BN[0][0] \n__________________________________________________________________________________________________\nblock_9_depthwise (DepthwiseCon (None, None, None, 3 3456 block_9_expand_relu[0][0] \n__________________________________________________________________________________________________\nblock_9_depthwise_BN (BatchNorm (None, None, None, 3 1536 block_9_depthwise[0][0] \n__________________________________________________________________________________________________\nblock_9_depthwise_relu (ReLU) (None, None, None, 3 0 block_9_depthwise_BN[0][0] \n__________________________________________________________________________________________________\nblock_9_project (Conv2D) (None, None, None, 6 24576 block_9_depthwise_relu[0][0] \n__________________________________________________________________________________________________\nblock_9_project_BN (BatchNormal (None, None, None, 6 256 block_9_project[0][0] \n__________________________________________________________________________________________________\nblock_9_add (Add) (None, None, None, 6 0 block_8_add[0][0] \n block_9_project_BN[0][0] \n__________________________________________________________________________________________________\nblock_10_expand (Conv2D) (None, None, None, 3 24576 block_9_add[0][0] \n__________________________________________________________________________________________________\nblock_10_expand_BN (BatchNormal (None, None, None, 3 1536 block_10_expand[0][0] \n__________________________________________________________________________________________________\nblock_10_expand_relu (ReLU) (None, None, None, 3 0 block_10_expand_BN[0][0] \n__________________________________________________________________________________________________\nblock_10_depthwise (DepthwiseCo (None, None, None, 3 3456 block_10_expand_relu[0][0] \n__________________________________________________________________________________________________\nblock_10_depthwise_BN (BatchNor (None, None, None, 3 1536 block_10_depthwise[0][0] \n__________________________________________________________________________________________________\nblock_10_depthwise_relu (ReLU) (None, None, None, 3 0 block_10_depthwise_BN[0][0] \n__________________________________________________________________________________________________\nblock_10_project (Conv2D) (None, None, None, 9 36864 block_10_depthwise_relu[0][0] \n__________________________________________________________________________________________________\nblock_10_project_BN (BatchNorma (None, None, None, 9 384 block_10_project[0][0] \n__________________________________________________________________________________________________\nblock_11_expand (Conv2D) (None, None, None, 5 55296 block_10_project_BN[0][0] \n__________________________________________________________________________________________________\nblock_11_expand_BN (BatchNormal (None, None, None, 5 2304 block_11_expand[0][0] \n__________________________________________________________________________________________________\nblock_11_expand_relu (ReLU) (None, None, None, 5 0 block_11_expand_BN[0][0] \n__________________________________________________________________________________________________\nblock_11_depthwise (DepthwiseCo (None, None, None, 5 5184 block_11_expand_relu[0][0] \n__________________________________________________________________________________________________\nblock_11_depthwise_BN (BatchNor (None, None, None, 5 2304 block_11_depthwise[0][0] \n__________________________________________________________________________________________________\nblock_11_depthwise_relu (ReLU) (None, None, None, 5 0 block_11_depthwise_BN[0][0] \n__________________________________________________________________________________________________\nblock_11_project (Conv2D) (None, None, None, 9 55296 block_11_depthwise_relu[0][0] \n__________________________________________________________________________________________________\nblock_11_project_BN (BatchNorma (None, None, None, 9 384 block_11_project[0][0] \n__________________________________________________________________________________________________\nblock_11_add (Add) (None, None, None, 9 0 block_10_project_BN[0][0] \n block_11_project_BN[0][0] \n__________________________________________________________________________________________________\nblock_12_expand (Conv2D) (None, None, None, 5 55296 block_11_add[0][0] \n__________________________________________________________________________________________________\nblock_12_expand_BN (BatchNormal (None, None, None, 5 2304 block_12_expand[0][0] \n__________________________________________________________________________________________________\nblock_12_expand_relu (ReLU) (None, None, None, 5 0 block_12_expand_BN[0][0] \n__________________________________________________________________________________________________\nblock_12_depthwise (DepthwiseCo (None, None, None, 5 5184 block_12_expand_relu[0][0] \n__________________________________________________________________________________________________\nblock_12_depthwise_BN (BatchNor (None, None, None, 5 2304 block_12_depthwise[0][0] \n__________________________________________________________________________________________________\nblock_12_depthwise_relu (ReLU) (None, None, None, 5 0 block_12_depthwise_BN[0][0] \n__________________________________________________________________________________________________\nblock_12_project (Conv2D) (None, None, None, 9 55296 block_12_depthwise_relu[0][0] \n__________________________________________________________________________________________________\nblock_12_project_BN (BatchNorma (None, None, None, 9 384 block_12_project[0][0] \n__________________________________________________________________________________________________\nblock_12_add (Add) (None, None, None, 9 0 block_11_add[0][0] \n block_12_project_BN[0][0] \n__________________________________________________________________________________________________\nblock_13_expand (Conv2D) (None, None, None, 5 55296 block_12_add[0][0] \n__________________________________________________________________________________________________\nblock_13_expand_BN (BatchNormal (None, None, None, 5 2304 block_13_expand[0][0] \n__________________________________________________________________________________________________\nblock_13_expand_relu (ReLU) (None, None, None, 5 0 block_13_expand_BN[0][0] \n__________________________________________________________________________________________________\nblock_13_pad (ZeroPadding2D) (None, None, None, 5 0 block_13_expand_relu[0][0] \n__________________________________________________________________________________________________\nblock_13_depthwise (DepthwiseCo (None, None, None, 5 5184 block_13_pad[0][0] \n__________________________________________________________________________________________________\nblock_13_depthwise_BN (BatchNor (None, None, None, 5 2304 block_13_depthwise[0][0] \n__________________________________________________________________________________________________\nblock_13_depthwise_relu (ReLU) (None, None, None, 5 0 block_13_depthwise_BN[0][0] \n__________________________________________________________________________________________________\nblock_13_project (Conv2D) (None, None, None, 1 92160 block_13_depthwise_relu[0][0] \n__________________________________________________________________________________________________\nblock_13_project_BN (BatchNorma (None, None, None, 1 640 block_13_project[0][0] \n__________________________________________________________________________________________________\nblock_14_expand (Conv2D) (None, None, None, 9 153600 block_13_project_BN[0][0] \n__________________________________________________________________________________________________\nblock_14_expand_BN (BatchNormal (None, None, None, 9 3840 block_14_expand[0][0] \n__________________________________________________________________________________________________\nblock_14_expand_relu (ReLU) (None, None, None, 9 0 block_14_expand_BN[0][0] \n__________________________________________________________________________________________________\nblock_14_depthwise (DepthwiseCo (None, None, None, 9 8640 block_14_expand_relu[0][0] \n__________________________________________________________________________________________________\nblock_14_depthwise_BN (BatchNor (None, None, None, 9 3840 block_14_depthwise[0][0] \n__________________________________________________________________________________________________\nblock_14_depthwise_relu (ReLU) (None, None, None, 9 0 block_14_depthwise_BN[0][0] \n__________________________________________________________________________________________________\nblock_14_project (Conv2D) (None, None, None, 1 153600 block_14_depthwise_relu[0][0] \n__________________________________________________________________________________________________\nblock_14_project_BN (BatchNorma (None, None, None, 1 640 block_14_project[0][0] \n__________________________________________________________________________________________________\nblock_14_add (Add) (None, None, None, 1 0 block_13_project_BN[0][0] \n block_14_project_BN[0][0] \n__________________________________________________________________________________________________\nblock_15_expand (Conv2D) (None, None, None, 9 153600 block_14_add[0][0] \n__________________________________________________________________________________________________\nblock_15_expand_BN (BatchNormal (None, None, None, 9 3840 block_15_expand[0][0] \n__________________________________________________________________________________________________\nblock_15_expand_relu (ReLU) (None, None, None, 9 0 block_15_expand_BN[0][0] \n__________________________________________________________________________________________________\nblock_15_depthwise (DepthwiseCo (None, None, None, 9 8640 block_15_expand_relu[0][0] \n__________________________________________________________________________________________________\nblock_15_depthwise_BN (BatchNor (None, None, None, 9 3840 block_15_depthwise[0][0] \n__________________________________________________________________________________________________\nblock_15_depthwise_relu (ReLU) (None, None, None, 9 0 block_15_depthwise_BN[0][0] \n__________________________________________________________________________________________________\nblock_15_project (Conv2D) (None, None, None, 1 153600 block_15_depthwise_relu[0][0] \n__________________________________________________________________________________________________\nblock_15_project_BN (BatchNorma (None, None, None, 1 640 block_15_project[0][0] \n__________________________________________________________________________________________________\nblock_15_add (Add) (None, None, None, 1 0 block_14_add[0][0] \n block_15_project_BN[0][0] \n__________________________________________________________________________________________________\nblock_16_expand (Conv2D) (None, None, None, 9 153600 block_15_add[0][0] \n__________________________________________________________________________________________________\nblock_16_expand_BN (BatchNormal (None, None, None, 9 3840 block_16_expand[0][0] \n__________________________________________________________________________________________________\nblock_16_expand_relu (ReLU) (None, None, None, 9 0 block_16_expand_BN[0][0] \n__________________________________________________________________________________________________\nblock_16_depthwise (DepthwiseCo (None, None, None, 9 8640 block_16_expand_relu[0][0] \n__________________________________________________________________________________________________\nblock_16_depthwise_BN (BatchNor (None, None, None, 9 3840 block_16_depthwise[0][0] \n__________________________________________________________________________________________________\nblock_16_depthwise_relu (ReLU) (None, None, None, 9 0 block_16_depthwise_BN[0][0] \n__________________________________________________________________________________________________\nblock_16_project (Conv2D) (None, None, None, 3 307200 block_16_depthwise_relu[0][0] \n__________________________________________________________________________________________________\nblock_16_project_BN (BatchNorma (None, None, None, 3 1280 block_16_project[0][0] \n__________________________________________________________________________________________________\nConv_1 (Conv2D) (None, None, None, 1 409600 block_16_project_BN[0][0] \n__________________________________________________________________________________________________\nConv_1_bn (BatchNormalization) (None, None, None, 1 5120 Conv_1[0][0] \n__________________________________________________________________________________________________\nout_relu (ReLU) (None, None, None, 1 0 Conv_1_bn[0][0] \n__________________________________________________________________________________________________\nglobal_average_pooling2d_1 (Glo (None, 1280) 0 out_relu[0][0] \n__________________________________________________________________________________________________\nglobal_max_pooling2d_1 (GlobalM (None, 1280) 0 out_relu[0][0] \n__________________________________________________________________________________________________\nconcatenate_1 (Concatenate) (None, 2560) 0 global_average_pooling2d_1[0][0] \n global_max_pooling2d_1[0][0] \n__________________________________________________________________________________________________\nbatch_normalization_1 (BatchNor (None, 2560) 10240 concatenate_1[0][0] \n__________________________________________________________________________________________________\ndropout_1 (Dropout) (None, 2560) 0 batch_normalization_1[0][0] \n__________________________________________________________________________________________________\ndense_1 (Dense) (None, 256) 655616 dropout_1[0][0] \n__________________________________________________________________________________________________\nbatch_normalization_2 (BatchNor (None, 256) 1024 dense_1[0][0] \n__________________________________________________________________________________________________\ndropout_2 (Dropout) (None, 256) 0 batch_normalization_2[0][0] \n__________________________________________________________________________________________________\ndense_2 (Dense) (None, 80) 20560 dropout_2[0][0] \n==================================================================================================\nTotal params: 2,945,424\nTrainable params: 2,905,680\nNon-trainable params: 39,744\n__________________________________________________________________________________________________\n","name":"stdout"}]},{"metadata":{},"cell_type":"markdown","source":"### train"},{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np\nxx = np.random.rand(1)\nprint(xx.shape,xx)\n\nxx = np.random.rand(1,1)\nprint(xx.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# If you want, you can try more advanced augmentation like this\naugment_img = iaa.Sequential([\n# iaa.ContrastNormalization((0.9, 1.1)),\n# iaa.Multiply((0.9, 1.1), per_channel=0.2),\n iaa.Fliplr(0.5),\n# iaa.GaussianBlur(sigma=(0, 0.1)),\n# iaa.Affine( # x-shift\n# translate_percent={\"x\": (-0.1, 0.1), \"y\": (-0.0, 0.0)},\n# ),\n iaa.CoarseDropout(0.12,size_percent=0.05) # see examples : https://github.com/aleju/imgaug\n ], random_order=True)\n\n\n\n# Or you can choose this simplest augmentation (like pytorch version)\n# augment_img = iaa.Fliplr(0.5)\n\n# This is my ugly modification; sorry about that\nclass FATTrainDataset(Sequence):\n\n def mix_up(x, y):\n x = np.array(x, np.float32)\n lam = np.random.beta(1.0, 1.0)\n ori_index = np.arange(int(len(x)))\n index_array = np.arange(int(len(x)))\n np.random.shuffle(index_array) \n \n mixed_x = lam * x[ori_index] + (1 - lam) * x[index_array]\n mixed_y = lam * y[ori_index] + (1 - lam) * y[index_array]\n \n return mixed_x, mixed_y\n \n def getitem(image):\n # crop 2sec\n\n base_dim, time_dim, _ = image.shape\n crop = random.randint(0, time_dim - base_dim)\n image = image[:,crop:crop+base_dim,:]\n\n image = preprocess_input(image)\n \n# label = self.labels[idx]\n return image\n def create_generator(train_X, train_y, batch_size, shape, augument=False, shuffling=False, test_data=False, mixup=False, mixup_prob=0.3):\n assert shape[2] == 3\n while True:\n if shuffling:\n train_X,train_y = shuffle(train_X,train_y)\n\n for start in range(0, len(train_y), batch_size):\n end = min(start + batch_size, len(train_y))\n batch_images = []\n X_train_batch = train_X[start:end]\n if test_data == False:\n batch_labels = train_y[start:end]\n \n for i in range(len(X_train_batch)):\n image = FATTrainDataset.getitem(X_train_batch[i]) \n if augument:\n image = FATTrainDataset.augment(image)\n batch_images.append(image)\n \n if (mixup and test_data == False):\n dice = np.random.rand(1)\n if dice > mixup_prob:\n batch_images, batch_labels = FATTrainDataset.mix_up(batch_images, batch_labels) \n \n if test_data == False:\n yield np.array(batch_images, np.float32), batch_labels\n else:\n yield np.array(batch_images, np.float32)\n return image\n \n def augment(image):\n\n image_aug = augment_img.augment_image(image)\n return image_aug","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras.callbacks import (ModelCheckpoint, LearningRateScheduler,\n EarlyStopping, ReduceLROnPlateau,CSVLogger)\n \nfrom sklearn.model_selection import train_test_split,KFold\n\n\nreduceLROnPlat = ReduceLROnPlateau(monitor='val_tf_lwlrap', factor=LR_FACTOR, patience=PATIENCE, \n verbose=1, mode='max', min_delta=0.0001, cooldown=2, min_lr=1e-5 )\n\ncsv_logger = CSVLogger(filename='../working/training_log.csv',\n separator=',',\n append=True)\n\ncheckpoint = ModelCheckpoint(checkpoint_file[0], monitor='val_tf_lwlrap', verbose=1, \n save_best_only=True, mode='max', save_weights_only = False)\ncallbacks_list = [checkpoint, csv_logger, reduceLROnPlat]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# split data into train, valid\nx_trn, x_val, y_trn, y_val = train_test_split(x_train, y_train, test_size=0.2, random_state=SEED)\n\n# create train and valid datagens\ntrain_generator = FATTrainDataset.create_generator(\n x_trn, y_trn, BATCH_SIZE, (SIZE,SIZE,3), augument=TRAIN_AUGMENT, shuffling=True, mixup = USE_MIXUP, mixup_prob = MIXUP_PROB)\nvalidation_generator = FATTrainDataset.create_generator(\n x_val, y_val, BATCH_SIZE, (SIZE,SIZE,3), augument=VALID_AUGMENT, shuffling=False)\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\ntrain_steps = np.ceil(float(len(x_trn)) / float(BATCH_SIZE))\nval_steps = np.ceil(float(len(x_val)) / float(BATCH_SIZE))\ntrain_steps = train_steps.astype(int)\nval_steps = val_steps.astype(int)\nprint(train_steps, val_steps)\nprint(len(x_trn))\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(LOSS)\nif LOSS=='BCEwithLogits':\n model.compile(loss=BCEwithLogits,\n optimizer=Adam(lr=LR),\n metrics=[tf_lwlrap,'categorical_accuracy'])\nelse:\n model.compile(loss=LOSS,\n optimizer=Adam(lr=LR),\n metrics=[tf_lwlrap,'categorical_accuracy'])\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\nprint(LR, PATIENCE, LR_FACTOR,BATCH_SIZE, TRAIN_AUGMENT, USE_MIXUP, MIXUP_PROB)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\n\nhist = model.fit_generator(\n train_generator,\n steps_per_epoch=train_steps,\n validation_data=validation_generator,\n validation_steps=val_steps,\n epochs=EPOCHS[0],\n verbose=1,\n callbacks=callbacks_list)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(K.eval(model.optimizer.lr))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# if LOSS=='BCEwithLogits':\n# model.compile(loss=BCEwithLogits,\n# optimizer=Adam(lr=3e-4),\n# metrics=[tf_lwlrap,'categorical_accuracy'])\n# else:\n# model.compile(loss=LOSS,\n# optimizer=Adam(lr=3e-4),\n# metrics=[tf_lwlrap,'categorical_accuracy'])\n\n# train_generator = FATTrainDataset.create_generator(\n# x_trn, y_trn, BATCH_SIZE, (SIZE,SIZE,3), augument=TRAIN_AUGMENT, \n# shuffling=True, mixup = False, mixup_prob=0.1)\n\n# EPOCHS = [100, 66, 0]\n\n# print(K.eval(model.optimizer.lr))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if EPOCHS[1] > 0:\n checkpoint = ModelCheckpoint(checkpoint_file[1], monitor='val_tf_lwlrap', verbose=1, \n save_best_only=True, mode='max', save_weights_only = False)\n callbacks_list = [checkpoint, csv_logger, reduceLROnPlat]\n \n hist = model.fit_generator(\n train_generator,\n steps_per_epoch=train_steps,\n validation_data=validation_generator,\n validation_steps=val_steps,\n epochs=EPOCHS[1],\n verbose=1,\n callbacks=callbacks_list)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(K.eval(model.optimizer.lr))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if EPOCHS[2] > 0:\n checkpoint = ModelCheckpoint(checkpoint_file[2], monitor='val_tf_lwlrap', verbose=1, \n save_best_only=True, mode='max', save_weights_only = False)\n callbacks_list = [checkpoint, csv_logger, reduceLROnPlat]\n \n hist = model.fit_generator(\n train_generator,\n steps_per_epoch=train_steps,\n validation_data=validation_generator,\n validation_steps=val_steps,\n epochs=EPOCHS[2],\n verbose=1,\n callbacks=callbacks_list)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(K.eval(model.optimizer.lr))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1, 2, figsize=(15,5))\nax[0].set_title('loss')\nax[0].plot(hist.epoch, hist.history[\"loss\"], label=\"Train loss\")\nax[0].plot(hist.epoch, hist.history[\"val_loss\"], label=\"Validation loss\")\nax[1].set_title('categorical_accuracy')\nax[1].plot(hist.epoch, hist.history[\"categorical_accuracy\"], label=\"Train categorical_accuracy\")\nax[1].plot(hist.epoch, hist.history[\"val_categorical_accuracy\"], label=\"Validation categorical_accuracy\")\nax[0].legend()\nax[1].legend()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1, 2, figsize=(15,5))\nax[0].set_title('tf_lwlrap')\nax[0].plot(hist.epoch, hist.history[\"tf_lwlrap\"], label=\"Train lwlrap\")\nax[0].plot(hist.epoch, hist.history[\"val_tf_lwlrap\"], label=\"Validation lwlrap\")\nax[1].set_title('categorical_accuracy')\nax[1].plot(hist.epoch, hist.history[\"categorical_accuracy\"], label=\"Train categorical_accuracy\")\nax[1].plot(hist.epoch, hist.history[\"val_categorical_accuracy\"], label=\"Validation categorical_accuracy\")\nax[0].legend()\nax[1].legend()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Calculate Validation Score using TTA\nNote that we have to initiate validation_generation everytime before doing a new prediction as `model.fit_generator` will mis-index examples at the end of epoch (and you will get random score)"},{"metadata":{"trusted":true},"cell_type":"code","source":"model.load_weights(checkpoint_file[0])\n\nvalidation_generator = FATTrainDataset.create_generator(\n x_val, y_val, BATCH_SIZE, (SIZE,SIZE,3), augument=False, shuffling=False)\npred_val_y = model.predict_generator(validation_generator,steps=val_steps,verbose=1)\n\nfor kk in range(len(TTA)):\n \n for ii in range(TTA[kk]):\n validation_generator = FATTrainDataset.create_generator(\n x_val, y_val, BATCH_SIZE, (SIZE,SIZE,3), augument=False, shuffling=False)\n \n pred_val_y += model.predict_generator(validation_generator,steps=val_steps,verbose=1)\n \n if kk+1 < len(TTA) and TTA[kk+1] > 0:\n model.load_weights(checkpoint_file[kk+1])\n\n'''Since the score is based on ranking, we do not need to normalize the prediction'''\n# pred_val_y = pred_val_y/10\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_generator = FATTrainDataset.create_generator(\n x_trn, y_trn, BATCH_SIZE, (SIZE,SIZE,3), augument=False, shuffling=False)\npred_train_y = model.predict_generator(train_generator,steps=train_steps,verbose=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import sklearn.metrics\ndef calculate_overall_lwlrap_sklearn(truth, scores):\n \"\"\"Calculate the overall lwlrap using sklearn.metrics.lrap.\"\"\"\n # sklearn doesn't correctly apply weighting to samples with no labels, so just skip them.\n sample_weight = np.sum(truth > 0, axis=1)\n nonzero_weight_sample_indices = np.flatnonzero(sample_weight > 0)\n overall_lwlrap = sklearn.metrics.label_ranking_average_precision_score(\n truth[nonzero_weight_sample_indices, :] > 0, \n scores[nonzero_weight_sample_indices, :], \n sample_weight=sample_weight[nonzero_weight_sample_indices])\n return overall_lwlrap","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(pred_val_y.shape, y_val.shape)\nprint(np.sum(pred_val_y), np.sum(y_val))\n# for ii in range(len(y_val)):\n# print(np.sum(pred_val_y[ii]), np.sum(y_val[ii]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"lwlrap from sklearn.metrics for training data =\", calculate_overall_lwlrap_sklearn(y_trn, pred_train_y))\nprint(\"val lwlrap from sklearn.metrics =\", calculate_overall_lwlrap_sklearn(y_val, pred_val_y/10))\n\nscore, weight = calculate_per_class_lwlrap(y_val, pred_val_y)\nlwlrap = (score * weight).sum()\nprint('direct calculation of val lwlrap : %.4f' % (lwlrap))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Simple Error Analysis"},{"metadata":{"trusted":true},"cell_type":"code","source":"idx = np.sum(y_val,axis=1) > 1\nprint(y_val[idx, :].shape, y_val[idx==False, :].shape)\n\nprint(\"val lwlrap for multi-labels =\", calculate_overall_lwlrap_sklearn(y_val[idx], pred_val_y[idx]))\nprint(\"val lwlrap for single-label =\", calculate_overall_lwlrap_sklearn(y_val[idx==False], pred_val_y[idx==False]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"## Predict Test Data with TTA"},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_steps = np.ceil(float(len(x_test)) / float(BATCH_SIZE)).astype(int)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model.load_weights(checkpoint_file[0])\n\ntest_generator = FATTrainDataset.create_generator(\n x_test, x_test, BATCH_SIZE, (SIZE,SIZE,3), augument=False, shuffling=False, test_data=True)\npred_test_y = model.predict_generator(test_generator,steps=test_steps,verbose=1)\n\nfor kk in range(len(TTA)):\n for ii in range(TTA[kk]):\n test_generator = FATTrainDataset.create_generator(\n x_test, x_test, BATCH_SIZE, (SIZE,SIZE,3), augument=False, shuffling=False, test_data=True)\n \n pred_test_y += model.predict_generator(test_generator,steps=test_steps,verbose=1)\n \n if kk+1 < len(TTA) and TTA[kk+1] > 0:\n model.load_weights(checkpoint_file[kk+1])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":" sort_idx = np.argsort(labels).astype(int)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(sort_idx)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_sub = pd.read_csv('../input/freesound-audio-tagging-2019/sample_submission.csv')\ntest_Y_sort = pred_test_y[:, sort_idx]\nsample_sub.iloc[:, 1:] = test_Y_sort\nsample_sub.to_csv('submission.csv', index=False)\n\nsample_sub.head()","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}