{"cells":[{"metadata":{},"cell_type":"markdown","source":"# CNN 2D Basic Solution #2.5\n\nThis is 2nd version of https://www.kaggle.com/daisukelab/cnn-2d-basic-solution-powered-by-fast-ai.\n\n- Based on fast.ai library, solving issue as image multi-label classification.\n- Using mel-spectrogram preprocessing.\n- Using preprocessed dataset additionally. https://www.kaggle.com/daisukelab/fat2019_prep_mels1\n\n## _What's new in 14-Apr version_\n\n- _Smaller subset of noisy training data was added; The best 50s. (I should call random 50 actually...)_\n- _Training uses both curated and this subset of noisy data._\n\n# Important notice for final submission\n\nThis kernel is not preprocessing __test set__, then this doesn't work for final submission.\nWe should be ready for preprocessing new test set used in re-evaluation after deadline:\n\n_\"Note, as this competition is a Kernels-only, two-stage competition, following the final submission deadline for the competition, your kernel code will be re-run on a privately-held test set that is not provided to you. It is your model's score against this private test set that will determine your ranking on the private leaderboard and final standing in the competition. The leaderboard will be updated in the days following the competition's completion, and our team will announce that the re-run has been completed and leaderboard finalized with an announcement made on the competition forums.\"_\n\nLink: https://www.kaggle.com/c/freesound-audio-tagging-2019/overview/timeline"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nfrom pathlib import Path\nimport matplotlib.pyplot as plt\nfrom tqdm import tqdm_notebook\nimport IPython\nimport IPython.display\nimport PIL\nimport pickle\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nimport os\nprint(os.listdir(\"../input\"))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"! ls ../input/fat2019_prep_mels1","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## File/folder definitions\n\n- `df` will handle training data.\n- `test_df` will handle test data."},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"DATA = Path('../input/freesound-audio-tagging-2019')\nPREPROCESSED = Path('../input/fat2019_prep_mels1')\nWORK = Path('work')\nPath(WORK).mkdir(exist_ok=True, parents=True)\n\nCSV_TRN_CURATED = DATA/'train_curated.csv'\nCSV_TRN_NOISY = DATA/'train_noisy.csv'\nCSV_TRN_NOISY_BEST50S = PREPROCESSED/'trn_noisy_best50s.csv'\nCSV_SUBMISSION = DATA/'sample_submission.csv'\n\nMELS_TRN_CURATED = PREPROCESSED/'mels_train_curated.pkl'\nMELS_TRN_NOISY = PREPROCESSED/'mels_train_noisy.pkl'\nMELS_TRN_NOISY_BEST50S = PREPROCESSED/'mels_trn_noisy_best50s.pkl'\nMELS_TEST = PREPROCESSED/'mels_test.pkl'\n\ntrn_curated_df = pd.read_csv(CSV_TRN_CURATED)\ntrn_noisy_df = pd.read_csv(CSV_TRN_NOISY)\ntrn_noisy50s_df = pd.read_csv(CSV_TRN_NOISY_BEST50S)\ntest_df = pd.read_csv(CSV_SUBMISSION)\n\n#df = pd.concat([trn_curated_df, trn_noisy_df], ignore_index=True) # not enough memory\ndf = pd.concat([trn_curated_df, trn_noisy50s_df], ignore_index=True, sort=True)\ntest_df = pd.read_csv(CSV_SUBMISSION)\n\nX_train = pickle.load(open(MELS_TRN_CURATED, 'rb')) + pickle.load(open(MELS_TRN_NOISY_BEST50S, 'rb'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Custom `open_image` for fast.ai library to load data from memory\n\n- Important note: Random cropping 1 sec, this is working like augmentation."},{"metadata":{"trusted":true},"cell_type":"code","source":"from fastai import *\nfrom fastai.vision import *\nfrom fastai.vision.data import *\nimport random\n\nCUR_X_FILES, CUR_X = list(df.fname.values), X_train\n\ndef open_fat2019_image(fn, convert_mode, after_open)->Image:\n    # open\n    idx = CUR_X_FILES.index(fn.split('/')[-1])\n    x = PIL.Image.fromarray(CUR_X[idx])\n    # crop\n    time_dim, base_dim = x.size\n    crop_x = random.randint(0, time_dim - base_dim)\n    x = x.crop([crop_x, 0, crop_x+base_dim, base_dim])    \n    # standardize\n    return Image(pil2tensor(x, np.float32).div_(255))\n\nvision.data.open_image = open_fat2019_image","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Follow multi-label classification\n\n- Almost following fast.ai course: https://nbviewer.jupyter.org/github/fastai/course-v3/blob/master/nbs/dl1/lesson3-planet.ipynb\n- But `pretrained=False`"},{"metadata":{"trusted":true},"cell_type":"code","source":"tfms = get_transforms(do_flip=True, max_rotate=0, max_lighting=0.1, max_zoom=0, max_warp=0.)\nsrc = (ImageList.from_csv(WORK, Path('..')/CSV_TRN_CURATED, folder='trn_curated')\n       .split_by_rand_pct(0.2)\n       .label_from_df(label_delim=',')\n)\ndata = (src.transform(tfms, size=128)\n        .databunch(bs=64).normalize(imagenet_stats)\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data.show_batch(3)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"f_score = partial(fbeta, thresh=0.2)\nlearn = cnn_learner(data, models.resnet18, pretrained=False, metrics=[f_score])\nlearn.unfreeze()\n\nlearn.lr_find()\nlearn.recorder.plot()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.fit_one_cycle(5, slice(1e-6, 1e-1))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.lr_find()\nlearn.recorder.plot()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.fit_one_cycle(100, slice(1e-6, 1e-2))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.save('fat2019_fastai_cnn2d_stage-2')\nlearn.export()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Test prediction and making submission file simple\n- Switch to test data.\n- Overwrite results to sample submission; simple way to prepare submission file."},{"metadata":{"trusted":true},"cell_type":"code","source":"del X_train\nX_test = pickle.load(open(MELS_TEST, 'rb'))\nCUR_X_FILES, CUR_X = list(test_df.fname.values), X_test\n\ntest = ImageList.from_csv(WORK, Path('..')/CSV_SUBMISSION, folder='test')\nlearn = load_learner(WORK, test=test)\npreds, _ = learn.get_preds(ds_type=DatasetType.Test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df[learn.data.classes] = preds\ntest_df.to_csv('submission.csv', index=False)\ntest_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Visualizing activations"},{"metadata":{"trusted":true},"cell_type":"code","source":"del X_test\nX_train = pickle.load(open(MELS_TRN_CURATED, 'rb'))\n\nCUR_X_FILES, CUR_X = list(df.fname.values), X_train\nlearn = cnn_learner(data, models.resnet18, pretrained=False, metrics=[f_score])\nlearn.load('fat2019_fastai_cnn2d_stage-2');","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Thanks to https://nbviewer.jupyter.org/github/fastai/course-v3/blob/master/nbs/dl1/lesson6-pets-more.ipynb\nfrom fastai.callbacks.hooks import *\n\ndef visualize_cnn_by_cam(learn, data_index):\n    x, _y = learn.data.valid_ds[data_index]\n    y = _y.data\n    if not isinstance(y, (list, np.ndarray)): # single label -> one hot encoding\n        y = np.eye(learn.data.valid_ds.c)[y]\n\n    m = learn.model.eval()\n    xb,_ = learn.data.one_item(x)\n    xb_im = Image(learn.data.denorm(xb)[0])\n    xb = xb.cuda()\n\n    def hooked_backward(cat):\n        with hook_output(m[0]) as hook_a: \n            with hook_output(m[0], grad=True) as hook_g:\n                preds = m(xb)\n                preds[0,int(cat)].backward()\n        return hook_a,hook_g\n    def show_heatmap(img, hm, label):\n        _,axs = plt.subplots(1, 2)\n        axs[0].set_title(label)\n        img.show(axs[0])\n        axs[1].set_title(f'CAM of {label}')\n        img.show(axs[1])\n        axs[1].imshow(hm, alpha=0.6, extent=(0,img.shape[0],img.shape[0],0),\n                      interpolation='bilinear', cmap='magma');\n        plt.show()\n\n    for y_i in np.where(y > 0)[0]:\n        hook_a,hook_g = hooked_backward(cat=y_i)\n        acts = hook_a.stored[0].cpu()\n        grad = hook_g.stored[0][0].cpu()\n        grad_chan = grad.mean(1).mean(1)\n        mult = (acts*grad_chan[...,None,None]).mean(0)\n        show_heatmap(img=xb_im, hm=mult, label=str(learn.data.valid_ds.y[data_index]))\n\nfor idx in range(10):\n    visualize_cnn_by_cam(learn, idx)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# https://discuss.pytorch.org/t/how-to-visualize-the-actual-convolution-filters-in-cnn/13850\nfrom sklearn.preprocessing import minmax_scale\n\ndef visualize_first_layer(learn):\n    conv1 = list(learn.model.children())[0][0]\n    weights = conv1.weight.data.cpu().numpy()\n    weights_shape = weights.shape\n    weights = minmax_scale(weights.ravel()).reshape(weights_shape)\n    fig, axes = plt.subplots(8, 8, figsize=(8,8))\n    for i, ax in enumerate(axes.flat):\n        ax.imshow(np.rollaxis(weights[i], 0, 3))\n        ax.get_xaxis().set_visible(False)\n        ax.get_yaxis().set_visible(False)\n\nvisualize_first_layer(learn)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Fiters are not matured yet..."},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}