{"nbformat": 4, "nbformat_minor": 1, "cells": [{"cell_type": "markdown", "source": ["\n", "\n", "## PyTorch Speech Recognition Challenge\n", "\n", "https://www.kaggle.com/c/tensorflow-speech-recognition-challenge\n", "\n", "\n", "Notebooks: <a href=\"https://github.com/QuantScientist/Deep-Learning-Boot-Camp/blob/master/Kaggle-PyTorch/tf/PyTorch%20Speech%20Recognition%20Challenge%20Starter.ipynb\"> On GitHub</a>\n", "\n", "\n", "#### References:\n", "\n", "- http://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html\n", "\n", "- https://www.bountysource.com/issues/44576966-a-tutorial-on-writing-custom-datasets-samplers-and-using-transforms\n", "\n", "- https://medium.com/towards-data-science/my-first-kaggle-competition-9d56d4773607\n", "\n", "- https://github.com/sohyongsheng/kaggle-planet-forest\n", "\n", "- https://github.com/rwightman/pytorch-planet-amazon/blob/master/dataset.py\n", "\n", "- https://www.kaggle.com/c/tensorflow-speech-recognition-challenge/discussion/43624\n", "\n", "## PyTorch dada sets\n", "\n", "- Convert the audio files into images (spectogram) \n", "- Create a CSV file consisting of the good labels \n", "- Then write a custom PyTorch data loader\n", "- Simple CNN\n", "\n", "## Issues:\n", "- Problem with the loss function for the multi-class case during training, loss is negative\n", "\n", "#### Shlomo Kashani"], "metadata": {"_uuid": "f51d80c1d9fafa2ee077ac806ee5b05bb84956ab", "_cell_guid": "91828084-4f86-4c4f-a4bf-18fced929057", "slideshow": {"slide_type": "slide"}}}, {"cell_type": "markdown", "source": ["# PyTorch Imports\n"], "metadata": {"_uuid": "fe12b6b44c03cf6201b0530a1806494e8ee942b6", "_cell_guid": "4130290b-62a9-4f95-b8bc-541766f0396a", "slideshow": {"slide_type": "slide"}, "collapsed": true}}, {"cell_type": "code", "outputs": [], "execution_count": null, "source": ["% reset -f\n", "import torch\n", "import sys\n", "import torch\n", "from torch.utils.data.dataset import Dataset\n", "from torch.utils.data import DataLoader\n", "from torchvision import transforms\n", "from torch import nn\n", "import torch.nn.functional as F\n", "import torch.optim as optim\n", "from torch.autograd import Variable\n", "\n", "from sklearn import cross_validation\n", "from sklearn import metrics\n", "from sklearn.metrics import roc_auc_score, log_loss, roc_auc_score, roc_curve, auc\n", "from sklearn.cross_validation import StratifiedKFold, ShuffleSplit, cross_val_score, train_test_split\n", "\n", "print('__Python VERSION:', sys.version)\n", "print('__pyTorch VERSION:', torch.__version__)\n", "print('__CUDA VERSION')\n", "from subprocess import call\n", "# call([\"nvcc\", \"--version\"]) does not work\n", "! nvcc --version\n", "print('__CUDNN VERSION:', torch.backends.cudnn.version())\n", "print('__Number CUDA Devices:', torch.cuda.device_count())\n", "print('__Devices')\n", "# call([\"nvidia-smi\", \"--format=csv\", \"--query-gpu=index,name,driver_version,memory.total,memory.used,memory.free\"])\n", "print('Active CUDA Device: GPU', torch.cuda.current_device())\n", "\n", "print ('Available devices ', torch.cuda.device_count())\n", "print ('Current cuda device ', torch.cuda.current_device())\n", "\n", "import numpy\n", "import numpy as np\n", "\n", "use_cuda = torch.cuda.is_available()\n", "FloatTensor = torch.cuda.FloatTensor if use_cuda else torch.FloatTensor\n", "LongTensor = torch.cuda.LongTensor if use_cuda else torch.LongTensor\n", "Tensor = FloatTensor\n", "\n", "import pandas\n", "import pandas as pd\n", "\n", "import logging\n", "handler=logging.basicConfig(level=logging.INFO)\n", "lgr = logging.getLogger(__name__)\n", "%matplotlib inline\n", "\n", "# !pip install psutil\n", "import psutil\n", "import os\n", "def cpuStats():\n", "        print(sys.version)\n", "        print(psutil.cpu_percent())\n", "        print(psutil.virtual_memory())  # physical memory usage\n", "        pid = os.getpid()\n", "        py = psutil.Process(pid)\n", "        memoryUse = py.memory_info()[0] / 2. ** 30  # memory use in GB...I think\n", "        print('memory GB:', memoryUse)\n", "\n", "cpuStats()"], "metadata": {"_uuid": "4036a0228bd117adf9fe5063f04626ba46ddcc3b", "_cell_guid": "ce7b1f1f-05e4-4b44-b9ad-0bdfb1e8a0b2", "slideshow": {"slide_type": "-"}, "collapsed": true}}, {"cell_type": "code", "outputs": [], "execution_count": null, "source": ["use_cuda = torch.cuda.is_available()\n", "# use_cuda = False\n", "\n", "FloatTensor = torch.cuda.FloatTensor if use_cuda else torch.FloatTensor\n", "LongTensor = torch.cuda.LongTensor if use_cuda else torch.LongTensor\n", "Tensor = FloatTensor"], "metadata": {"_uuid": "0b624b4c246113a00e58ca38337159708fa3173a", "_cell_guid": "6dbb23ed-a978-4851-8291-ef6c1374f2d7", "collapsed": true}}, {"cell_type": "markdown", "source": ["# Setting up global variables\n", "\n", "- Root folder\n", "- Audio folder\n", "- Audio Label folder"], "metadata": {"_uuid": "d80155836fdfa3709e1fd56a49cd09f225fe02a8", "_cell_guid": "31653fdb-0f86-4a1c-8aa9-de6e855a6c3e", "slideshow": {"slide_type": "slide"}, "collapsed": true}}, {"cell_type": "code", "outputs": [], "execution_count": null, "source": ["DATA_ROOT ='d:/db/data/tf/'\n", "IMG_PATH = DATA_ROOT + '/picts/train/'\n", "IMG_EXT = '.png'\n", "IMG_DATA_LABELS = DATA_ROOT + '/train_v2.csv'"], "metadata": {"_uuid": "e91e36bb8dffd45b7400b35c7498080503cf500e", "_cell_guid": "28130de5-6522-4147-8525-6177d572bd66", "collapsed": true}}, {"cell_type": "markdown", "source": ["# Turn WAV into Images\n", "- See https://www.kaggle.com/timolee/audio-data-conversion-to-images-eda\n"], "metadata": {"_uuid": "69a15db0297f518b1ebde26e43aed6f943b4ec3e", "_cell_guid": "9afd28df-baa9-4085-93ee-ebfda7c9795a", "slideshow": {"slide_type": "slide"}, "collapsed": true}}, {"cell_type": "code", "outputs": [], "execution_count": null, "source": ["audio_path = 'd:/db/data/tf/train/audio/'\n", "pict_Path = 'd:/db/data/tf//picts/train/'\n", "test_pict_Path = 'd:/db/data/tf//picts/test/'\n", "test_audio_path = 'd:/db/data/tf//test/audio/'\n", "samples = []\n", "\n", "\n", "if not os.path.exists(pict_Path):\n", "    os.makedirs(pict_Path)\n", "\n", "if not os.path.exists(test_pict_Path):\n", "    os.makedirs(test_pict_Path)\n", "    \n", "subFolderList = []\n", "\n", "for x in os.listdir(audio_path):\n", "    if os.path.isdir(audio_path + '/' + x):\n", "        subFolderList.append(x)\n", "        if not os.path.exists(pict_Path + '/' + x):\n", "            os.makedirs(pict_Path +'/'+ x)"], "metadata": {"_uuid": "1c6637a424d7ca830eb15f692b611bd2f933e344", "_cell_guid": "63d23f3f-3fb9-4c45-921b-4e75dfbe4f17", "collapsed": true}}, {"cell_type": "code", "outputs": [], "execution_count": null, "source": ["# #### Function: convert audio to spectogram images\n", "\n", "# def wav2img(wav_path, targetdir='', figsize=(4,4)):\n", "#     \"\"\"\n", "#     takes in wave file path\n", "#     and the fig size. Default 4,4 will make images 288 x 288\n", "#     \"\"\"\n", "#     fs = 44100 # sampling frequency\n", "    \n", "#     # use soundfile library to read in the wave files\n", "#     test_sound, samplerate = sf.read(wav_path)\n", "    \n", "#     # make the plot\n", "#     fig = plt.figure(figsize=figsize)\n", "#     S, freqs, bins, im = plt.specgram(test_sound, NFFT=1024, Fs=samplerate, noverlap=512)\n", "#     plt.show\n", "#     plt.axis('off')\n", "    \n", "#     ## create output path\n", "#     output_file = wav_path.split('/')[-1].split('.wav')[0]\n", "#     output_file = targetdir +'/'+ output_file\n", "#     plt.savefig('%s.png' % output_file)\n", "#     plt.close()\n", "\n", "\n", "# def wav2img_waveform(wav_path, targetdir='', figsize=(4,4)):\n", "#     test_sound, samplerate = sf.read(sample_audio[0])\n", "#     fig = plt.figure(figsize=figsize)\n", "#     plt.plot(test_sound)\n", "#     plt.axis('off')\n", "#     output_file = wav_path.split('/')[-1].split('.wav')[0]\n", "#     output_file = targetdir +'/'+ output_file\n", "#     plt.savefig('%s.png' % output_file)\n", "#     plt.close()\n", "\n", "# ### Convert Training Audio\n", "# #### Loop through source audio and save as pictures \n", "# # (may take a while) may also consider running at commandline. \n", "# # Code is limited to 3 folders and 10 files each, get rid of array limits to process the entire directory\n", "\n", "# # c:\\Anaconda3\\lib\\site-packages\\matplotlib\\axes\\_axes.py:7221: RuntimeWarning: divide by zero encountered in log10\n", "# #   Z = 10. * np.log10(spec)\n", "\n", "# for i, x in enumerate(subFolderList):\n", "#     print(i, ':', x)\n", "#     # get all the wave files\n", "#     all_files = [y for y in os.listdir(audio_path + x) if '.wav' in y]\n", "#     for file in all_files:\n", "#         try:\n", "#             wav2img(audio_path + x + '/' + file, pict_Path + x)                \n", "#         except Exception:\n", "#             pass\n", "            "], "metadata": {"_uuid": "6aec2eafabf2a9d43be0dd5ad605303562162c38", "_cell_guid": "9c112ee9-3c76-4a09-8393-671637f74245", "collapsed": true}}, {"cell_type": "markdown", "source": ["# Generate lables into a CSV, which is easier for PyTorch Dataset class"], "metadata": {"_uuid": "e46e266f764375ac3c1bf767afd52cffeec70604", "_cell_guid": "9bb329c3-2076-438f-8e8f-ee58991e1370", "slideshow": {"slide_type": "slide"}, "collapsed": true}}, {"cell_type": "code", "outputs": [], "execution_count": null, "source": ["# from sklearn.preprocessing import MultiLabelBinarizer, LabelBinarizer\n", "# from sklearn.preprocessing import LabelEncoder\n", "# from sklearn.pipeline import Pipeline\n", "# from collections import defaultdict\n", "# d = defaultdict(LabelEncoder)\n", "\n", "# # Build the pictures path\n", "# subFolderList = []\n", "# for x in os.listdir(pict_Path):\n", "#     if os.path.isdir(pict_Path + '/' + x):\n", "#         subFolderList.append(x)        \n", "            \n", "# good_labels=['yes', 'no', 'up', 'down', 'left', 'right', 'on', 'off', 'stop', 'go']\n", "# POSSIBLE_LABELS = 'yes no up down left right on off stop go silence unknown'.split()\n", "\n", "# # print (type(POSSIBLE_LABELS))\n", "# # print (type(good_labels))\n", "# columns = ['img', 'label-str','fullpath']\n", "# df_pred=pd.DataFrame(data=np.zeros((0,len(columns))), columns=columns)\n", "# # df_pred.id.astype(int)\n", "\n", "# for i, x in enumerate(subFolderList):\n", "#     if (x in POSSIBLE_LABELS):\n", "#     #     print(i, ':', x)\n", "#         # get all the wave files\n", "#         all_files = [y for y in os.listdir(pict_Path + x) if '.png' in y]\n", "#         for file in all_files:\n", "#     #         print (audio_path + x + '/' + file, pict_Path + x)\n", "#             fullPath=pict_Path + x + '/' + file\n", "#     #         print (fullPath)\n", "#             df_pred = df_pred.append({'img':file, 'label-str':x,'fullpath':fullPath},ignore_index=True)\n", "#     #         print (pict_Path + x)    \n", "    \n", "\n", "# # Encode the categorical labels as numeric data\n", "# df_pred['label'] = LabelEncoder().fit_transform(df_pred['label-str'])\n", "# # Make sure we dont save the header\n", "# df_pred.to_csv(IMG_DATA_LABELS, columns=('img','label-str','fullpath', 'label'), index=None, header=False)\n", "# df_pred.to_csv(IMG_DATA_LABELS +'_header', columns=('img','label-str','fullpath', 'label'), index=None, header=True)\n", "\n", "# # img,label,fullpath\n", "# # 00176480_nohash_0.wav,down,d:/db/data/tf/train/audio/down/00176480_nohash_0.wav\n", "    \n", "# df_pred.head(3)"], "metadata": {"_uuid": "ac9e5923538f8f3d5a2f85d7bbe3dac4c4e65ea6", "_cell_guid": "bd31b132-1900-4dad-8ce6-6bd5745e5398", "collapsed": true}}, {"cell_type": "markdown", "source": ["# The Torch Dataset Class"], "metadata": {"_uuid": "1a45aaa6ff0da2ecd7d365b839736b70efec74ec", "_cell_guid": "0cbfbf6c-3d9e-4f9f-87c3-22278aa0eac2", "slideshow": {"slide_type": "slide"}, "collapsed": true}}, {"cell_type": "code", "outputs": [], "execution_count": null, "source": ["import time\n", "from sklearn.preprocessing import MultiLabelBinarizer, LabelBinarizer\n", "from sklearn.preprocessing import LabelEncoder\n", "from sklearn.feature_extraction import DictVectorizer\n", "from sklearn.pipeline import Pipeline\n", "from collections import defaultdict\n", "d = defaultdict(LabelEncoder)\n", "\n", "# Encoding the variable\n", "# X_df_train_SINGLE = X_df_train_SINGLE.apply(lambda x: d[x.name].fit_transform(x))\n", "# X_df_train_SINGLE=X_df_train_SINGLE.apply(LabelEncoder().fit_transform)\n", "# Inverse the encoded\n", "# fit.apply(lambda x: X_df_train_SINGLE[x.name].inverse_transform(x))\n", "# Using the dictionary to label future data\n", "# df.apply(lambda x: X_df_train_SINGLE[x.name].transform(x))\n", "# answers_1_SINGLE = list (X_df_train_SINGLE[singleResponseVariable].values)\n", "# answers_1_SINGLE= map(int, answers_1_SINGLE)\n", "\n", "def encode_onehot(df, cols):  \n", "    vec = DictVectorizer()    \n", "    vec_data = pd.DataFrame(vec.fit_transform(df[cols].to_dict(outtype='records')).toarray())\n", "    vec_data.columns = vec.get_feature_names()\n", "    vec_data.index = df.index\n", "    \n", "    df = df.drop(cols, axis=1)\n", "    df = df.join(vec_data)\n", "    return df\n", "\n", "try:\n", "    from PIL import Image\n", "except ImportError:\n", "    import Image\n", "    \n", "class GenericImageDataset(Dataset):    \n", "\n", "    def __init__(self, csv_path, img_path, img_ext, transform=None):\n", "        \n", "        t = time.time()        \n", "        lgr.info('CSV path {}'.format(csv_path))\n", "        lgr.info('IMG path {}'.format(img_path))        \n", "        \n", "        assert img_ext in ['.png']\n", "        \n", "        tmp_df = pd.read_csv(csv_path, header=None) # img,label,fullpath\n", "                        \n", "        self.mlb = MultiLabelBinarizer()\n", "        self.img_path = img_path\n", "        self.img_ext = img_ext\n", "        self.transform = transform\n", "\n", "        # Encoding the variables                \n", "        lgr.info(\"DF CSV:\\n\" + str (tmp_df.head(3)))\n", "                        \n", "        self.X_train = tmp_df[2]        \n", "        \n", "        self.y_train = self.mlb.fit_transform(tmp_df[1].str.split()).astype(np.float32)           \n", "        self.y_train=self.y_train.reshape((self.y_train.shape[0]*10,1)) # Must be reshaped for PyTorch!                \n", "        \n", "#         y_df = encode_onehot(tmp_df, cols=[tmp_df[1]])\n", "#         self.y_train = y_df \n", "        \n", "        lgr.info('y_train {}'.format(self.y_train))\n", "                \n", "#         self.y_train = tmp_df[3].astype(np.float32)                          \n", "#         self.y_train = self.mlb.fit_transform(tmp_df[1].str.split()).astype(np.float32)\n", "#         self.y_train = tmp_df[3].astype(np.float32)       \n", "#         d = defaultdict(LabelEncoder)\n", "#         self.y_train =tmp_df[1].apply(lambda x: d[x].fit_transform(x))\n", "    \n", "#         tmp_df=one_hot(tmp_df,tmp_df[1])\n", "#         self.y_train = tmp_df[1].astype(np.float32)       \n", "#         encoder = LabelEncoder()\n", "#         encoder.fit(tmp_df[1])\n", "#         self.y_train = encoder.transform(tmp_df[1]).astype(np.float32)\n", "#         self.y_train=self.y_train.reshape((self.y_train.shape[0],1)) # Must be reshaped for PyTorch!\n", "                \n", "        lgr.info('[*]Dataset loading time {}'.format(time.time() - t))\n", "        lgr.info('[*] Data size is {}'.format(len(self)))\n", "        \n", "        lgr.info(\"DF CSV:\\n\" + str (tmp_df.head(5)))\n", "        \n", "        print ()\n", "\n", "    def __getitem__(self, index):\n", "#         lgr.info (\"__getitem__:\" + str(index))\n", "        path=self.img_path + self.X_train[index]\n", "        path=self.X_train[index]\n", "#         lgr.info (\" --- get item path:\" + path)\n", "        img = Image.open(path)\n", "        img = img.convert('RGB')\n", "        if self.transform is not None: # TypeError: batch must contain tensors, numbers, or lists; \n", "                                     #found <class 'PIL.Image.Image'>\n", "            img = self.transform(img)\n", "#             print (str (type(img))) # <class 'torch.FloatTensor'>                \n", "#         label = torch.from_numpy(self.y_train[index])\n", "        label = (self.y_train[index])\n", "        return img, label\n", "\n", "    def __len__(self):\n", "        l=len(self.X_train.index)\n", "#         lgr.info (\"Lenght:\" +str(l))\n", "        return (l)       \n", "\n", "    @staticmethod        \n", "    def imshow(img):\n", "        img = img / 2 + 0.5     # unnormalize\n", "        npimg = img.numpy()\n", "        plt.imshow(np.transpose(npimg, (1, 2, 0)))\n", "\n", "    @staticmethod    \n", "    def flaotTensorToImage(img, mean=0, std=1):\n", "        \"\"\"convert a tensor to an image\"\"\"\n", "        img = np.transpose(img.numpy(), (1, 2, 0))\n", "        img = (img*std+ mean)*255\n", "        img = img.astype(np.uint8)    \n", "        return img    \n", "    \n", "    @staticmethod\n", "    def toTensor(img):\n", "        \"\"\"convert a numpy array of shape HWC to CHW tensor\"\"\"\n", "        img = img.transpose((2, 0, 1)).astype(np.float32)\n", "        tensor = torch.from_numpy(img).float()\n", "        return tensor/255.0    "], "metadata": {"_uuid": "8a92488bf0787dc10a6ff340404b816fda454ef5", "_cell_guid": "d8a7753d-b4cd-442c-8251-fba82506cce7", "slideshow": {"slide_type": "-"}, "collapsed": true}}, {"cell_type": "markdown", "source": ["# The Torch transforms.ToTensor() methood\n", "\n", "- Converts: a PIL.Image or numpy.ndarray (H x W x C) in the range [0, 255] to a torch.FloatTensor of shape (C x H x W) in the range [0.0, 1.0]."], "metadata": {"_uuid": "36f3b7cdf9046745072ff3cc9173b38bea67a504", "_cell_guid": "7f7eb854-aaac-470a-89ae-9791f5f977f8", "slideshow": {"slide_type": "slide"}, "collapsed": true}}, {"cell_type": "code", "outputs": [], "execution_count": null, "source": ["transformations = transforms.Compose([transforms.ToTensor()])"], "metadata": {"_uuid": "9d0ac620ca746ba9b384d881c14bb2a1263e6982", "_cell_guid": "546c67b9-9324-44e5-bad1-ac90630a23c3", "collapsed": true}}, {"cell_type": "markdown", "source": ["# The Torch DataLoader Class\n", "\n", "- Will load our GenericImageDataset\n", "- Can be regarded as a list (or iterator, technically). \n", "- Each time it is invoked will provide a minibatch of (img, label) pairs."], "metadata": {"_uuid": "076ad9caa14596353917e1db32907e0fe181ac85", "_cell_guid": "b7795718-f8fe-47b7-b153-61184fd3a1d2", "slideshow": {"slide_type": "slide"}, "collapsed": true}}, {"cell_type": "code", "outputs": [], "execution_count": null, "source": ["dset_train = GenericImageDataset(IMG_DATA_LABELS,IMG_PATH,IMG_EXT,transformations)"], "metadata": {"_uuid": "2260abd160dcebfa4237ab47a217f740eee70fc4", "_cell_guid": "26c85bf0-9f65-4cf9-b107-6437854202fb", "slideshow": {"slide_type": "-"}, "collapsed": true}}, {"cell_type": "markdown", "source": ["# Train Validation Split\n", "\n", "- Since there is no train_test_split method in PyTorch, we have to split a Training dataset into training and validation sets."], "metadata": {"_uuid": "3dab99bf60fc9867843a05ae1ebb4b15e62a6000", "_cell_guid": "59b28c4b-b78a-4761-b1fc-f5b26afcfd98", "slideshow": {"slide_type": "slide"}, "collapsed": true}}, {"cell_type": "code", "outputs": [], "execution_count": null, "source": ["batch_size = 16 # on GTX 1080\n", "global_epoches = 10\n", "LR = 0.0005\n", "MOMENTUM = 0.95\n", "validationRatio=0.11    \n", "\n", "class FullTrainningDataset(torch.utils.data.Dataset):\n", "    def __init__(self, full_ds, offset, length):\n", "        self.full_ds = full_ds\n", "        self.offset = offset\n", "        self.length = length\n", "        assert len(full_ds)>=offset+length, Exception(\"Parent Dataset not long enough\")\n", "        super(FullTrainningDataset, self).__init__()\n", "        \n", "    def __len__(self):\n", "        return self.length\n", "    \n", "    def __getitem__(self, i):\n", "        return self.full_ds[i+self.offset]\n", "    \n", "\n", "\n", "def trainTestSplit(dataset, val_share=validationRatio):\n", "    val_offset = int(len(dataset)*(1-val_share))\n", "    print(\"Offest:\" + str(val_offset))\n", "    return FullTrainningDataset(dataset, 0, val_offset), FullTrainningDataset(dataset, val_offset, len(dataset)-val_offset)\n", "\n", " \n", "train_ds, val_ds = trainTestSplit(dset_train)\n", "\n", "train_loader = torch.utils.data.DataLoader(train_ds, batch_size=batch_size, shuffle=False, num_workers=0)\n", "val_loader = torch.utils.data.DataLoader(val_ds, batch_size=batch_size, shuffle=False, num_workers=0)\n", "\n", "print(train_loader)\n", "print(val_loader)"], "metadata": {"_uuid": "c063690c0e852860c1161a20dbce72e3feac71fc", "_cell_guid": "848d5ac9-6b9c-4b5f-bde5-794ba6b3a087", "collapsed": true}}, {"cell_type": "markdown", "source": ["# Test the DataLoader Class"], "metadata": {"_uuid": "3934bafd96ee333e0173c265c24b6c99b056060c", "_cell_guid": "16f240ce-3b33-43f9-939a-a6eefbbb448f", "slideshow": {"slide_type": "slide"}, "collapsed": true}}, {"cell_type": "code", "outputs": [], "execution_count": null, "source": ["%matplotlib inline\n", "import matplotlib.pyplot as plt\n", "\n", "imagesToShow=4\n", "\n", "for i, data in enumerate(train_loader, 0):\n", "    lgr.info('i=%d: '%(i))            \n", "    images, labels = data            \n", "    num = len(images)\n", "    \n", "    ax = plt.subplot(1, imagesToShow, i + 1)\n", "    plt.tight_layout()\n", "    ax.set_title('Sample #{}'.format(i))\n", "    ax.axis('off')\n", "    \n", "    for n in range(num):\n", "        image=images[n]\n", "        label=labels[n]\n", "        plt.imshow (GenericImageDataset.flaotTensorToImage(image))\n", "        \n", "    if i==imagesToShow-1:\n", "        break    "], "metadata": {"_uuid": "aa32003befe28499d06703faca0daa0c28f6ee30", "_cell_guid": "a1b47af1-003e-47e9-80ed-fa8425028cb6", "collapsed": true}}, {"cell_type": "markdown", "source": ["# The NN model\n", "\n", "- We will use a several CNNs with conv(3x3) -> bn -> relu -> pool(4x4) -> fc.\n", "\n", "- In PyTorch, a model is defined by a subclass of nn.Module. It has two methods:\n", "\n", "- `__init__: constructor. Create layers here. Note that we don't define the connections between layers in this function.`\n", "\n", "\n", "- `forward(x): forward function. Receives an input variable x. Returns a output variable. Note that we actually connect the layers here dynamically.` "], "metadata": {"_uuid": "b3bdab4ecde189976828f219a89bee4b054ce33e", "_cell_guid": "cea5830f-ced0-464a-82b0-47c3f9169186", "slideshow": {"slide_type": "slide"}, "collapsed": true}}, {"cell_type": "code", "outputs": [], "execution_count": null, "source": ["dropout = torch.nn.Dropout(p=0.30)\n", "class ConvRes(nn.Module):\n", "    def __init__(self, insize, outsize):\n", "        super(ConvRes, self).__init__()\n", "        drate = .3\n", "        self.math = nn.Sequential(\n", "            nn.BatchNorm2d(insize),\n", "            # nn.Dropout(drate),\n", "            torch.nn.Conv2d(insize, outsize, kernel_size=2, padding=2),\n", "            nn.PReLU(),\n", "        )\n", "\n", "    def forward(self, x):\n", "        return self.math(x)\n", "\n", "\n", "class ConvCNN(nn.Module):\n", "    def __init__(self, insize, outsize, kernel_size=7, padding=2, pool=2, avg=True):\n", "        super(ConvCNN, self).__init__()\n", "        self.avg = avg\n", "        self.math = torch.nn.Sequential(\n", "            torch.nn.Conv2d(insize, outsize, kernel_size=kernel_size, padding=padding),\n", "            torch.nn.BatchNorm2d(outsize),\n", "            torch.nn.LeakyReLU(),\n", "            torch.nn.MaxPool2d(pool, pool),\n", "        )\n", "        self.avgpool = torch.nn.AvgPool2d(pool, pool)\n", "\n", "    def forward(self, x):\n", "        x = self.math(x)\n", "        if self.avg is True:\n", "            x = self.avgpool(x)\n", "        return x\n", "\n", "\n", "class Net(nn.Module):\n", "    def __init__(self):\n", "        super(Net, self).__init__()\n", "\n", "        self.avgpool = nn.AdaptiveAvgPool2d(1)\n", "\n", "        self.cnn1 = ConvCNN(3, 32, kernel_size=7, pool=4, avg=False)\n", "        self.cnn2 = ConvCNN(32, 32, kernel_size=5, pool=2, avg=True)\n", "        self.cnn3 = ConvCNN(32, 32, kernel_size=5, pool=2, avg=True)\n", "\n", "        self.res1 = ConvRes(32, 64)\n", "\n", "        self.features = nn.Sequential(\n", "            self.cnn1, dropout,\n", "            self.cnn2,\n", "            self.cnn3,\n", "            self.res1,\n", "        )\n", "\n", "        self.classifier = torch.nn.Sequential(\n", "            nn.Linear(3136, 1),\n", "        )\n", "        self.sig = nn.Sigmoid()\n", "  \n", "    def forward(self, x):\n", "        x = self.features(x)\n", "#         print (x.data.shape)\n", "        x = x.view(x.size(0), -1)\n", "#         print (x.data.shape)\n", "        x = self.classifier(x)\n", "#         print (x.data.shape)\n", "        x = self.sig(x)\n", "        return x\n", "\n", "    \n", "if use_cuda:\n", "    lgr.info (\"Using the GPU\")\n", "    model = Net().cuda() # On GPU\n", "else:\n", "    lgr.info (\"Using the CPU\")\n", "    model = Net() # On CPU\n", "\n", "lgr.info('Model {}'.format(model))\n", "\n"], "metadata": {"_uuid": "22ecd9a16aeed8b1daabf1b5b315fabe50c38542", "_cell_guid": "35d28804-1afa-472c-ada3-81d49f803e92", "collapsed": true}}, {"cell_type": "markdown", "source": ["#  Loss and Optimizer\n", "\n", "- Select a loss function and the optimization algorithm."], "metadata": {"_uuid": "dff173f6c61ac634706092470c9b19e4c5cd6894", "_cell_guid": "8bb5ff35-50b2-4c85-bcc4-0b83f6a024b3", "slideshow": {"slide_type": "slide"}, "collapsed": true}}, {"cell_type": "code", "outputs": [], "execution_count": null, "source": ["loss_func=torch.nn.BCELoss()\n", "loss_func = nn.MultiLabelSoftMarginLoss()\n", "# loss_func = torch.nn.CrossEntropyLoss()\n", "# NN params\n", "LR = 0.005\n", "MOMENTUM= 0.9\n", "optimizer = torch.optim.Adam(model.parameters(), lr=LR,weight_decay=5e-5) #  L2 regularization\n", "if use_cuda:\n", "    lgr.info (\"Using the GPU\")    \n", "    model.cuda()\n", "    loss_func.cuda()\n", "\n", "lgr.info (optimizer)\n", "lgr.info (loss_func)"], "metadata": {"_uuid": "b2fa99ff652c742f2852c9aa4444d536d9526991", "_cell_guid": "a5f733d8-8b5a-4d81-90d9-5e8a874a01e4", "collapsed": true}}, {"cell_type": "markdown", "source": ["# Start training in Batches\n", "\n", "See example here:\n", "http://codegists.com/snippet/python/pytorch_mnistpy_kernelmode_python\n", "\n", "https://github.com/pytorch/examples/blob/53f25e0d0e2710878449900e1e61d31d34b63a9d/mnist/main.py"], "metadata": {"_uuid": "1cb9ffdd6df6c2d96227fa3fdc3f938a182e932f", "_cell_guid": "3f3fe771-b888-4fb0-b53b-4745300ba9cd", "slideshow": {"slide_type": "slide"}, "collapsed": true}}, {"cell_type": "code", "outputs": [], "execution_count": null, "source": ["import torch\n", "import torch.nn as nn\n", "import torch.nn.functional as F\n", "import torch.optim as optim\n", "from torchvision import datasets, transforms\n", "from torch.autograd import Variable\n", " \n", "\n", "    \n", "clf=model \n", "opt= optimizer\n", "loss_history = []\n", "acc_history = []\n", " \n", "def train(epoch):\n", "    clf.train() # set model in training mode (need this because of dropout)\n", "     \n", "    # dataset API gives us pythonic batching \n", "    for batch_idx, (data, target) in enumerate(train_loader):\n", "        \n", "        if use_cuda:\n", "            data, target = Variable(data.cuda(async=True)), Variable(target.cuda(async=True)) # On GPU                \n", "        else:            \n", "            data, target = Variable(data), Variable(target) # RuntimeError: expected CPU tensor (got CUDA tensor)                           \n", "                 \n", "        # forward pass, calculate loss and backprop!\n", "        opt.zero_grad()\n", "        preds = clf(data)\n", "        if use_cuda:\n", "            loss = loss_func(preds, target).cuda()\n", "#             loss = F.log_softmax(preds).cuda() # TypeError: log_softmax() takes exactly 1 argument (2 given)\n", "#             loss = F.nll_loss(preds, target).cuda() # https://github.com/torch/cutorch/issues/227\n", "            \n", "        else:\n", "            loss = loss_func(preds, target)\n", "#             loss = F.log_softmax(preds)\n", "#             loss = F.nll_loss(preds, target.long()) # RuntimeError: multi-target not supported at /pytorch/torch/lib/THNN/generic/ClassNLLCriterion.c:22\n", "        loss.backward()\n", "        \n", "        opt.step()\n", "        \n", "        \n", "        if batch_idx % 100 == 0:\n", "            loss_history.append(loss.data[0])\n", "            lgr.info('Train Epoch: {} [{}/{} ({:.0f}%)]\\tLoss: {:.6f}'.format(\n", "            epoch, batch_idx * len(data), len(train_loader.dataset),\n", "            100. * batch_idx / len(train_loader), loss.data[0]))              \n", "\n", "            \n", "start_time = time.time()    \n", "\n", "for epoch in range(1, 5):\n", "    print(\"Epoch %d\" % epoch)\n", "    train(epoch)    \n", "end_time = time.time()\n", "print ('{} {:6.3f} seconds'.format('GPU:', end_time-start_time))\n", "%matplotlib inline\n", "import matplotlib.pyplot as plt\n", "plt.plot(loss_history)\n", "plt.show()"], "metadata": {"_uuid": "928dd9a8fd67d6d900499f03e5d565521644f89b", "_cell_guid": "2b0eef4c-4ef0-4adb-b57e-2268d0018283", "collapsed": true}}, {"cell_type": "code", "outputs": [], "execution_count": null, "source": ["criterion = loss_func\n", "all_losses = []\n", "val_losses = []\n", "\n", "\n", "if __name__ == '__main__':\n", "\n", "    for epoch in range(global_epoches):\n", "        print('Epoch {}'.format(epoch + 1))\n", "        print('*' * 5 + ':')\n", "        running_loss = 0.0\n", "        running_acc = 0.0\n", "        for i, data in enumerate(train_loader, 1):\n", "    \n", "            img, label = data\n", "            if use_cuda:\n", "                img, label = Variable(img.cuda(async=True)), Variable(label.cuda(async=True))  # On GPU\n", "            else:\n", "                img, label = Variable(img), Variable(\n", "                    label)  # RuntimeError: expected CPU tensor (got CUDA tensor)\n", "    \n", "            out = model(img)\n", "            loss = criterion(out, label)\n", "            running_loss += loss.data[0] * label.size(0)\n", "    \n", "            optimizer.zero_grad()\n", "            loss.backward()\n", "            optimizer.step()\n", "    \n", "            if i % 100 == 0:\n", "                all_losses.append(running_loss / (batch_size * i))\n", "                print('[{}/{}] Loss: {:.6f}'.format(\n", "                    epoch + 1, global_epoches, running_loss / (batch_size * i),\n", "                    running_acc / (batch_size * i)))\n", "                \n", "    \n", "#                 loss_cost = loss.data[0]                                \n", "#                 # RuntimeError: can't convert CUDA tensor to numpy (it doesn't support GPU arrays). \n", "#                 # Use .cpu() to move the tensor to host memory first.        \n", "#                 prediction = (model(img).data).float() # probabilities         \n", "#         #         prediction = (net(X_tensor).data > 0.5).float() # zero or one\n", "#         #         print (\"Pred:\" + str (prediction)) # Pred:Variable containing: 0 or 1\n", "#         #         pred_y = prediction.data.numpy().squeeze()            \n", "#                 pred_y = prediction.cpu().numpy().squeeze()\n", "#                 target_y = label.cpu().data.numpy()\n", "\n", "#                 tu = (log_loss(target_y, pred_y),roc_auc_score(target_y,pred_y ))\n", "#                 print ('LOG_LOSS={}, ROC_AUC={} '.format(*tu))  \n", "        \n", "    \n", "        print('Finish {} epoch, Loss: {:.6f}'.format(epoch + 1, running_loss / (len(train_ds))))\n", "    \n", "        model.eval()\n", "        eval_loss = 0\n", "        eval_acc = 0\n", "        for data in val_loader:\n", "            img, label = data\n", "    \n", "            if use_cuda:\n", "                img, label = Variable(img.cuda(async=True), volatile=True),Variable(label.cuda(async=True), volatile=True)  # On GPU\n", "            else:\n", "                img = Variable(img, volatile=True)\n", "                label = Variable(label, volatile=True)\n", "    \n", "            out = model(img)\n", "            loss = criterion(out, label)\n", "            eval_loss += loss.data[0] * label.size(0)\n", "    \n", "        print('VALIDATION Loss: {:.6f}'.format(eval_loss / (len(val_ds))))\n", "        val_losses.append(eval_loss / (len(val_ds)))\n", "        print()\n", "    "], "metadata": {"_uuid": "ced00ba1944c77f41d5a6460b975c6243860b895", "_cell_guid": "e68df111-a0d5-4b45-ace6-d34ef0fe608c", "collapsed": true}}, {"cell_type": "code", "outputs": [], "execution_count": null, "source": ["%%bash\n", "jupyter nbconvert \\\n", "    --to=slides \\\n", "    --reveal-prefix=https://cdnjs.cloudflare.com/ajax/libs/reveal.js/3.2.0/ \\\n", "    --output=py09.html \\\n", "    './09 PyTorch Kaggle Image Data-set loading with CNN'"], "metadata": {"_uuid": "c3bb8afb7f0950cf125b3665690fd5f2697c676a", "_cell_guid": "63425b81-ec11-4888-a936-76cc8af78ca7", "slideshow": {"slide_type": "skip"}, "collapsed": true}}, {"cell_type": "code", "outputs": [], "execution_count": null, "source": [], "metadata": {"_uuid": "bc52dab88821701fad652059f59ee61fb1a3581b", "_cell_guid": "23d06878-b2ea-428c-97e9-0a51e52b67c7", "collapsed": true}}], "metadata": {"language_info": {"nbconvert_exporter": "python", "file_extension": ".py", "codemirror_mode": {"name": "ipython", "version": 3}, "version": "3.6.3", "name": "python", "mimetype": "text/x-python", "pygments_lexer": "ipython3"}, "celltoolbar": "Slideshow", "livereveal": {"mouseWheel": "true", "scroll": "true", "start_slideshow_at": "selected", "controls": "true", "progress": "true", "history": "true", "overview": "true"}, "kernelspec": {"display_name": "Python 3", "language": "python", "name": "python3"}}}