{"cells":[{"metadata":{},"cell_type":"markdown","source":"### FINAL PROJECT"},{"metadata":{},"cell_type":"markdown","source":"#### all the necesassry libraries. Numpy for arrays to extract data. pyplot to plot the validation curves. torch used for models. torch.utils.data used for loading data into dataloader for easier manipulations. "},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"import os\nimport numpy as np\nfrom matplotlib import pyplot as plt\nimport torch\nfrom torch.utils.data import Dataset, DataLoader\n\n\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"##### Reading the data from npy files. These files are taken from other competitor. This data mostly used in kaggle because it is like in cifar10 converted to 32x32x3 dimensions. The code below loads data and splits it using only numpy functions"},{"metadata":{"trusted":true},"cell_type":"code","source":"x_train = np.load('../input/reducing-image-sizes-to-32x32/X_train.npy').astype('float32') \nx_test = np.load('../input/reducing-image-sizes-to-32x32/X_test.npy').astype('float32') \ny_train = np.load('../input/reducing-image-sizes-to-32x32/y_train.npy').astype('float32') \n\nx_train, x_val =np.split(x_train, [int(.8 * len(x_train))])\ny_train, y_val =np.split(y_train, [int(.8 * len(y_train))])\nprint('x_train shape:', x_train.shape)\nprint('x_val shape:', x_val.shape)\nprint('y_train shape:', y_train.shape)\nprint('y_val shape:', y_val.shape)\nprint(x_train.shape[0], 'train samples')\nprint(x_test.shape[0], 'test samples')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"##### As was mentioned in the report data is not configured in right way. It has form of ndarray with length of 14. For example label for category corresponding to 4 would look like [0 0 0 1 0 0..] the 3rd index is equal to one which means that the item category is 4. The loop below converts this form into integer number. The first category actually starts from 0. "},{"metadata":{"trusted":true},"cell_type":"code","source":"y_train_new = []\nfor i in range(len(y_train)):\n    y_train_new.append(np.where(y_train[i] == 1)[0][0])\ny_val_new = []\nfor i in range(len(y_val)):\n    y_val_new.append(np.where(y_val[i] == 1)[0][0])\n    ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"##### Converting list to numyp array"},{"metadata":{"trusted":true},"cell_type":"code","source":"y_tr = np.array(y_train_new)\ny_vl = np.array(y_val_new)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"##### Normalizing data"},{"metadata":{"trusted":true},"cell_type":"code","source":"x_train = x_train.astype('float32')\nx_test = x_test.astype('float32')\nx_train /= 255.\nx_test /= 255.\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"##### The code below is a custom made class to feed the data to pytorch DataLoader. Dataloader allows to better loops through train data with shuffling and specified batch size. This functionality is very useful"},{"metadata":{"trusted":true},"cell_type":"code","source":"\nclass MyDataset(Dataset):\n    def __init__(self, data, target, transform=None):\n        self.data = torch.from_numpy(data).float()\n        self.target = torch.from_numpy(target).long()\n        self.transform = transform\n        \n    def __getitem__(self, index):\n        x = self.data[index]\n        y = self.target[index]\n        \n        if self.transform:\n            x = self.transform(x)\n        \n        return x, y\n    \n    def __len__(self):\n        return len(self.data)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"##### Loading all the datasets"},{"metadata":{"trusted":true},"cell_type":"code","source":"dataset = MyDataset(x_train, y_tr)\nloader = DataLoader(dataset,batch_size=4,shuffle=True)\ntestSet = MyDataset(x_val, y_vl)\ntestloader = DataLoader(testSet,batch_size=4,shuffle=True)\nvalset = MyDataset(x_val, y_vl)\nvaloader = DataLoader(valset,batch_size=4,shuffle=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"##### This the first model. More explanation is given in the report. The libraries used Con2d, MaxPool2d, Linear and activation function relu. Source: https://www.stefanfiott.com/machine-learning/cifar-10-classifier-using-cnn-in-pytorch/"},{"metadata":{"trusted":true},"cell_type":"code","source":"import torch.nn as nn\nimport torch.nn.functional as F\n\nclass Net(nn.Module):\n    def __init__(self):\n        super(Net, self).__init__()\n        self.conv1 = nn.Conv2d(3, 6, 5)\n        self.pool = nn.MaxPool2d(2, 2)\n        self.conv2 = nn.Conv2d(6, 16, 5)\n        self.fc1 = nn.Linear(16 * 5 * 5, 120)\n        self.fc2 = nn.Linear(120, 84)\n        self.fc3 = nn.Linear(84, 14)\n\n    def forward(self, x):\n        x = self.pool(F.relu(self.conv1(x)))\n        x = self.pool(F.relu(self.conv2(x)))\n        x = x.view(-1, 16 * 5 * 5)\n        x = F.relu(self.fc1(x))\n        x = F.relu(self.fc2(x))\n        x = self.fc3(x)\n        return x\n    \nnet = Net()\ndevice = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")  ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"##### The code below specifies the way loss is calculated and also the optimizer which job is optimization and assigning of weights."},{"metadata":{"trusted":true},"cell_type":"code","source":"import torch.optim as optim\n\ncriterion = nn.CrossEntropyLoss()\noptimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9, weight_decay=5e-4)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"##### The variables where losses and accuracies will be stored"},{"metadata":{"trusted":true},"cell_type":"code","source":"# setting up lists for handling loss/accuracy\ntrain_acc, train_loss = [], []\nvalid_acc, valid_loss = [], []\ntest_acc, test_loss = [], []\ncur_loss = 0\nlosses = []\nnum_epochs =5\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"##### The actual training of the model. This is standard code for all the models. All needed for this is dataloader, input forwarded to the net, outputs generated and compared to original labels, the accuracy score computed. The same procedure done on validation set. Then we do not train model we evaluate its accuracy on training set and validation set"},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\n\nfor epoch in range(num_epochs):  # loop over the dataset multiple times\n    running_loss = 0.0\n    validation_loss = 0.0\n    for i, data in enumerate(loader, 0):\n        # get the inputs\n        inputs, labels = data\n        inputs = np.transpose(inputs, (0,3,2,1))\n        # zero the parameter gradients\n        optimizer.zero_grad()\n\n        # forward + backward + optimize\n        outputs = net(inputs)\n        loss = criterion(outputs, labels)\n        loss.backward()\n        optimizer.step()\n\n        # print statistics\n        running_loss += loss.item()\n        if i % 2000 == 1999:    # print every 2000 mini-batches\n            print('[%d, %5d] loss: %.3f' %(epoch + 1, i + 1, running_loss / 2000))\n            losses.append(running_loss)\n            running_loss = 0.0\n    for i, data in enumerate(valoader, 0):\n        # get the inputs\n        inputs, labels = data\n        inputs = np.transpose(inputs, (0,3,2,1))\n        # zero the parameter gradients\n        optimizer.zero_grad()\n\n        # forward + backward + optimize\n        outputs = net(inputs)\n        loss1 = criterion(outputs, labels)\n        loss1.backward()\n        optimizer.step()\n\n        # print statistics\n        validation_loss += loss1.item()\n        if i % 2000 == 1999:    # print every 2000 mini-batches\n            print('[%d, %5d] loss_valid: %.3f' %(epoch + 1, i + 1, validation_loss / 2000))\n            valid_loss.append(validation_loss)\n            validation_loss = 0.0\n    net.eval()\n    total = 0\n    correct = 0\n\n    with torch.no_grad():\n        for batch_idx, data in enumerate(loader):\n            inputs, targets = data\n            inputs = np.transpose(inputs, (0,3,2,1))\n            outputs = net(inputs)\n            _, predicted = torch.max(outputs.data, 1)\n            total += targets.size(0)\n            correct += predicted.eq(targets.data).cpu().sum()\n\n        print('Epoch : %d Train Acc : %.3f' % (epoch, 100.*correct/total))\n        print('--------------------------------------------------------------')\n        train_acc.append(100.*correct/total)\n        total = 0\n        correct = 0\n        for batch_idx, data in enumerate(valoader):\n            inputs, targets = data\n            inputs = np.transpose(inputs, (0,3,2,1))\n            outputs = net(inputs)\n            _, predicted = torch.max(outputs.data, 1)\n            total += targets.size(0)\n            correct += predicted.eq(targets.data).cpu().sum()\n\n        print('Epoch : %d Val Acc : %.3f' % (epoch, 100.*correct/total))\n        valid_acc.append(100.*correct/total)\n        print('--------------------------------------------------------------')\n    net.train() \n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"##### This code plots the graph Train Accuracy and Validation Accuracy"},{"metadata":{"trusted":true},"cell_type":"code","source":"epoch = np.arange(len(train_acc))\nplt.figure()\nplt.plot(epoch, train_acc, 'r', epoch, valid_acc, 'b')\nplt.legend(['Train Accucary','Validation Accuracy'])\nplt.xlabel('Updates'), plt.ylabel('Acc')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"x = np.arange(len(losses))\nplt.plot(x, losses, 'r', label='Train Loss')\ny = np.arange(len(valid_loss))\nplt.plot(y, valid_loss, 'b', label='Val Loss')\nplt.legend()\nplt.xlabel('Updates')\nplt.ylabel('Loss')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"##### This is the code where accuracy estimated on test set. It achieved accuracy of 70%. "},{"metadata":{"trusted":true},"cell_type":"code","source":"total_correct = 0\ntotal_images = 0\nconfusion_matrix = np.zeros([10,10], int)\nwith torch.no_grad():\n    for data in testloader:\n        images, labels = data\n        images = np.transpose(images, (0,3,2,1))\n        outputs = net(images)\n        _, predicted = torch.max(outputs.data, 1)\n        total_images += labels.size(0)\n        total_correct += (predicted == labels).sum().item()\n\n\nmodel_accuracy = total_correct / total_images * 100\nprint('Model accuracy on {0} test images: {1:.2f}%'.format(total_images, model_accuracy))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"##### Residual network, source: https://blog.paperspace.com/pytorch-101-building-neural-networks/\n##### The explnanation on the network is given in the report. Residual Block is the additional class to implement shortcut connection in the network"},{"metadata":{"trusted":true},"cell_type":"code","source":"class ResidualBlock(nn.Module):\n    def __init__(self, in_channels, out_channels, stride=1):\n        super(ResidualBlock, self).__init__()\n        \n        # Conv Layer 1\n        self.conv1 = nn.Conv2d(\n            in_channels=in_channels, out_channels=out_channels,\n            kernel_size=(3, 3), stride=stride, padding=1, bias=False\n        )\n        self.bn1 = nn.BatchNorm2d(out_channels)\n        \n        # Conv Layer 2\n        self.conv2 = nn.Conv2d(\n            in_channels=out_channels, out_channels=out_channels,\n            kernel_size=(3, 3), stride=1, padding=1, bias=False\n        )\n        self.bn2 = nn.BatchNorm2d(out_channels)\n    \n        # Shortcut connection to downsample residual\n        # In case the output dimensions of the residual block is not the same \n        # as it's input, have a convolutional layer downsample the layer \n        # being bought forward by approporate striding and filters\n        self.shortcut = nn.Sequential()\n        if stride != 1 or in_channels != out_channels:\n            self.shortcut = nn.Sequential(\n                nn.Conv2d(\n                    in_channels=in_channels, out_channels=out_channels,\n                    kernel_size=(1, 1), stride=stride, bias=False\n                ),\n                nn.BatchNorm2d(out_channels)\n            )\n\n    def forward(self, x):\n        out = nn.ReLU()(self.bn1(self.conv1(x)))\n        out = self.bn2(self.conv2(out))\n        out += self.shortcut(x)\n        out = nn.ReLU()(out)\n        return out","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"##### The whole residual network. It starts with convolutional layer and the followed by 4 block where 2 additional convolutional layers and shortcut connection provided. Also in here batch normalization used. "},{"metadata":{},"cell_type":"markdown","source":"![](http://)"},{"metadata":{"trusted":true},"cell_type":"code","source":"class ResNet(nn.Module):\n    def __init__(self, num_classes=14):\n        super(ResNet, self).__init__()\n        \n        # Initial input conv\n        self.conv1 = nn.Conv2d(\n            in_channels=3, out_channels=64, kernel_size=(3, 3),\n            stride=1, padding=1, bias=False\n        )\n\n        self.bn1 = nn.BatchNorm2d(64)\n        \n        # Create blocks\n        self.block1 = self._create_block(64, 64, stride=1)\n        self.block2 = self._create_block(64, 128, stride=2)\n        self.block3 = self._create_block(128, 256, stride=2)\n        self.block4 = self._create_block(256, 512, stride=2)\n        self.linear = nn.Linear(512, num_classes)\n    \n    # A block is just two residual blocks for ResNet18\n    def _create_block(self, in_channels, out_channels, stride):\n        return nn.Sequential(\n            ResidualBlock(in_channels, out_channels, stride),\n            ResidualBlock(out_channels, out_channels, 1)\n        )\n\n    def forward(self, x):\n\t# Output of one layer becomes input to the next\n        out = nn.ReLU()(self.bn1(self.conv1(x)))\n        out = self.block1(out)\n        out = self.block2(out)\n        out = self.block3(out)\n        out = self.block4(out)\n        out = nn.AvgPool2d(4)(out)\n        out = out.view(out.size(0), -1)\n        out = self.linear(out)\n        return out","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"##### This code provides the loss function and type of optimizer. Also it has scheduler whose job is to change learning rate each epoch. This is used because we want optimal learning. We do not want too smal steps or too big in exploring the space. "},{"metadata":{"trusted":true},"cell_type":"code","source":"device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")     #Check whether a GPU is present.\n\nresnet = ResNet()\n\ncriterion = nn.CrossEntropyLoss()\noptimizer = optim.SGD(resnet.parameters(), lr=0.1, momentum=0.9, weight_decay=5e-4)\nscheduler = torch.optim.lr_scheduler.MultiStepLR(optimizer, milestones=[150, 200], gamma=0.1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"##### Additional initialization of loader test for 2rd model where we change the batch size and allow parallel memory usage"},{"metadata":{"trusted":true},"cell_type":"code","source":"trainloader = torch.utils.data.DataLoader(dataset, batch_size=128, shuffle=True, num_workers=2)\n# setting up lists for handling loss/accuracy\ntrain_acc, train_loss = [], []\nvalid_acc = []\nlosses = []","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"##### The training for 2nd model. The code is the same as for previous model"},{"metadata":{"trusted":true},"cell_type":"code","source":"for epoch in range(5):\n    scheduler.step()\n    # Train\n    running_loss = 0.0\n    validation_loss = 0.0\n    for batch_idx, data in enumerate(loader):\n        inputs, targets = data\n        break\n        optimizer.zero_grad()                 # Zero the gradients\n        inputs = np.transpose(inputs, (0,3,2,1))\n        outputs = resnet(inputs)                 # Forward pass\n        loss = criterion(outputs, targets)    # Compute the Loss\n        loss.backward()                       # Compute the Gradients\n\n        optimizer.step()                      # Updated the weights\n        losses.append(loss.item())\n        if batch_idx % 100 == 0:\n            print('Batch Index : %d Loss : %.3f' % (batch_idx, np.mean(losses)))\n\n    # Evaluate\n    resnet.eval()\n    total = 0\n    correct = 0\n    with torch.no_grad():  \n        for batch_idx, data in enumerate(loader):\n            inputs, targets = data\n            inputs = np.transpose(inputs, (0,3,2,1))\n            outputs = resnet(inputs)\n            _, predicted = torch.max(outputs.data, 1)\n            total += targets.size(0)\n            correct += predicted.eq(targets.data).cpu().sum()\n\n        print('Epoch : %d Train Acc : %.3f' % (epoch, 100.*correct/total))\n        print('--------------------------------------------------------------')\n        train_acc.append(100.*correct/total)\n        total = 0\n        correct = 0\n        for batch_idx, data in enumerate(valoader):\n            inputs, targets = data\n            inputs = np.transpose(inputs, (0,3,2,1))\n            outputs = resnet(inputs)\n            _, predicted = torch.max(outputs.data, 1)\n            total += targets.size(0)\n            correct += predicted.eq(targets.data).cpu().sum()\n\n        print('Epoch : %d Val Acc : %.3f' % (epoch, 100.*correct/total))\n        valid_acc.append(100.*correct/total)\n        print('--------------------------------------------------------------')\n    tesnet.train() ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"LeNet"},{"metadata":{},"cell_type":"markdown","source":"##### This is the 3rd model. The 3rd model architecture is the same as for 1st. However it contains more parameters."},{"metadata":{"trusted":true},"cell_type":"code","source":"class LeNet(nn.Module):  \n    def __init__(self):  \n        super().__init__()  \n        self.conv1=nn.Conv2d(3,20,5,1)  \n        self.conv2=nn.Conv2d(20,50,5,1)  \n        self.fully1=nn.Linear(5*5*50,500)  \n        self.dropout1=nn.Dropout(0.5)   \n        self.fully2=nn.Linear(500,14)  \n    def forward(self,x):  \n        x=F.relu(self.conv1(x))  \n        x=F.max_pool2d(x,2,2)  \n        x=F.relu(self.conv2(x))  \n        x=F.max_pool2d(x,2,2)  \n        x=x.view(-1,5*5*50) #Reshaping the output into desired shape  \n        x=F.relu(self.fully1(x)) #Applying relu activation function to our first fully connected layer  \n        x=self.dropout1(x)  \n        x=self.fully2(x)    #We will not apply activation function here because we are dealing with multiclass dataset  \n        return x      ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"##### Initializaation of loss it is actually the same as previous. Adam optimizer. Also variables to store accuracy and loss scores. "},{"metadata":{"trusted":true},"cell_type":"code","source":"lenet=LeNet()\ncriteron=nn.CrossEntropyLoss()  \noptimizer=torch.optim.Adam(lenet.parameters(),lr=0.00001, weight_decay = 1e-4)   \nepochs=12  \nloss_history=[]  \ncorrect_history=[]  \nval_loss_history=[]  \nval_correct_history=[]  \ntrainloader = torch.utils.data.DataLoader(dataset, batch_size=100, shuffle=True, num_workers=2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"##### The training for the 3rd model"},{"metadata":{"trusted":true},"cell_type":"code","source":"for e in range(epochs):  \n    loss=0.0  \n    correct=0.0  \n    val_loss=0.0  \n    val_correct=0.0  \n    for input,labels in trainloader:  \n        input = np.transpose(input, (0,3,2,1))\n        outputs=lenet(input)  \n        loss1=criteron(outputs,labels)  \n        optimizer.zero_grad()  \n        loss1.backward()  \n        optimizer.step()  \n        _,preds=torch.max(outputs,1)  \n        loss+=loss1.item()  \n        correct+=torch.sum(preds==labels.data)  \n        if i % 2000 == 1999:    # print every 2000 mini-batches\n            print('[%d, %5d] loss: %.3f' %(epoch + 1, i + 1, running_loss / 2000))\n            losses.append(running_loss)\n            running_loss = 0.0\n    for i, data in enumerate(valoader, 0):\n        # get the inputs\n        inputs, labels = data\n        inputs = np.transpose(inputs, (0,3,2,1))\n        # zero the parameter gradients\n        optimizer.zero_grad()\n\n        # forward + backward + optimize\n        outputs = net(inputs)\n        loss1 = criterion(outputs, labels)\n        loss1.backward()\n        optimizer.step()\n\n        # print statistics\n        validation_loss += loss1.item()\n        if i % 2000 == 1999:    # print every 2000 mini-batches\n            print('[%d, %5d] loss_valid: %.3f' %(epoch + 1, i + 1, running_loss / 2000))\n            valid_loss.append(validation_loss)\n            validation_loss = 0.0\n    else:\n        with torch.no_grad():  \n            for val_input,val_labels in valoader:  \n                val_input = np.transpose(val_input, (0,3,2,1))\n                val_outputs=lenet(val_input)  \n                val_loss1=criteron(val_outputs,val_labels)   \n                _,val_preds=torch.max(val_outputs,1)  \n                val_loss+=val_loss1.item()  \n                val_correct+=torch.sum(val_preds==val_labels.data)  \n        epoch_loss=loss/len(trainloader)  \n        epoch_acc=correct.float()/len(trainloader)  \n        loss_history.append(epoch_loss)  \n        correct_history.append(epoch_acc)  \n        val_epoch_loss=val_loss/len(valoader)  \n        val_epoch_acc=val_correct.float()/len(valoader)  \n        val_loss_history.append(val_epoch_loss)  \n        val_correct_history.append(val_epoch_acc)  \n        print('training_loss:{:.4f},{:.4f}'.format(epoch_loss,epoch_acc.item()))  \n        print('validation_loss:{:.4f},{:.4f}'.format(val_epoch_loss,val_epoch_acc.item()))  ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":4}