{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Digit Recognition with Convolutional Neural Network","metadata":{}},{"cell_type":"markdown","source":"Hello everyone,\n\nWhen I first started Digit Recognizer project, I implemented a simple ANN with only 2 hidden linear layers, which you can find [here](https://www.kaggle.com/ahmetcelik158/digit-recognition-with-a-simple-pytorch-model). By editing that notebook, I am now planning to implement **Convolutional Neural Network (CNN)** using **PyTorch** in this notebook.\n\nWhat is new?\n* Convolutional Neural Network\n* KFold Cross Validation\n* Scheduler\n\n---\n\n**Index**\n\n1. [Data Preparation](#1.-Data-Preparation)\n\n2. [CNN Model](#2.-CNN-Model)\n\n3. [Function Definitions - Validation, Training and Prediction](#3.-Function-Definitions---Validation,-Training-and-Prediction)\n\n4. [Training the Model](#4.-Training-the-Model)\n\n5. [Prediction](#5.-Prediction)","metadata":{}},{"cell_type":"code","source":"# importing packages\nimport math\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nfrom sklearn.model_selection import StratifiedKFold\n%matplotlib inline\n\nimport torch\nfrom torch import nn\nfrom torch import optim\nfrom torch.utils.data import DataLoader, TensorDataset\n\ntrain = pd.read_csv(\"../input/digit-recognizer/train.csv\")\ntest = pd.read_csv(\"../input/digit-recognizer/test.csv\")\ntrain.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 1. Data Preparation","metadata":{}},{"cell_type":"code","source":"train_X = train.drop(\"label\", axis=1)\ntrain_y = train[\"label\"]\n\nprint(\"Shape of training set: {}\".format(train_X.shape))\nprint(\"Shape of test set: {}\".format(test.shape))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Since the input of a CNN is a 2D image, i have to reshape datasets:\ntrain_X = train_X.values.reshape(-1,1,28,28)\ntest = test.values.reshape(-1,1,28,28)\n\nprint(\"After Reshape\")\nprint(\"Shape of training set: {}\".format(train_X.shape))\nprint(\"Shape of test set: {}\".format(test.shape))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Lets plot some images from training data and see the labels\nindices = [42, 314, 2022, 33333]\nf,ax = plt.subplots(1, len(indices))\nfor i in range(len(indices)):\n    title = \"Label: {}\".format(train_y.iloc[indices[i]])\n    ax[i].imshow( train_X[indices[i],0] )\n    ax[i].set_title(title)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Converting train, validation and test image data into tensors\n# dividing by 255 is for normalization\ntrain_X_tensor = torch.tensor(train_X)/255.0\ntest_tensor = torch.tensor(test)/255.0\ntrain_y_tensor = torch.tensor(train_y.values)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 2. CNN Model\n\nIn this section, I will define my **CNN model**. While creating my model, I am inspired from the structure of AlexNet model. Using torchvision.models package, a pre-defined AlexNet model can be used directly. The pre-defined model is for 3 layered RGB images. Since we have black and white images with 28x28 pixels, I defined less connections compared to AlexNet.","metadata":{}},{"cell_type":"code","source":"# Definition of a CNN Model class\nclass CNN(nn.Module):\n    def __init__(self, output_size=100, dropout=0.5):\n        super(CNN, self).__init__()\n        self.conv = nn.Sequential(nn.Conv2d(1, 32, kernel_size=5, stride=2, padding=1),\n                                  nn.ReLU(inplace=True),\n                                  nn.MaxPool2d(kernel_size=2, stride=2),\n                                  nn.Conv2d(32, 96, kernel_size=4, stride=1, padding=1),\n                                  nn.ReLU(inplace=True),\n                                  nn.MaxPool2d(kernel_size=2, stride=2),\n                                  nn.Conv2d(96, 192, kernel_size=3, stride=1, padding=1),\n                                  nn.ReLU(inplace=True),\n                                  nn.Conv2d(192, 128, kernel_size=3, stride=1, padding=1),\n                                  nn.ReLU(inplace=True),\n                                  nn.Conv2d(128, 128, kernel_size=3, stride=1, padding=1),\n                                  nn.ReLU(inplace=True),\n                                  nn.MaxPool2d(kernel_size=2, stride=2),\n                                 )\n        self.avg_pool = nn.AdaptiveAvgPool2d(output_size=(6, 6))\n        self.fc = nn.Sequential(nn.Linear(128*6*6, 2048),\n                                nn.ReLU(inplace=True),\n                                nn.Dropout(p=dropout),\n                                nn.Linear(2048, 1024),\n                                nn.ReLU(inplace=True),\n                                nn.Dropout(p=dropout),\n                                nn.Linear(1024, output_size),\n                                nn.LogSoftmax(dim=1)\n                               )\n        return\n    \n    def forward(self, x):\n        x = self.conv(x)\n        x = self.avg_pool(x)\n        x = torch.flatten(x, 1)\n        x = self.fc(x)\n        return x","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 3. Function Definitions - Validation, Training and Prediction\n\nIn this sections I will define the following functions:\n* **validation**: returns the average loss and the accuracy of the model for the given data.\n* **train_model**: trains the model for the desired number of epochs, after each epoch calculates accuracy, at the end loads the model with the best validation accuracy\n* **prediction**: returns the predictions for the given data","metadata":{}},{"cell_type":"code","source":"### VALIDATION FUNCTION\ndef validation(model, loader, criterion, device=\"cpu\"):\n    model.eval()\n    loss = 0\n    acc = 0\n    \n    with torch.no_grad():\n        for images, labels in loader:\n            images, labels = images.to(device), labels.to(device)\n            \n            output = model.forward(images)\n            loss += criterion(output,labels).item()\n            \n            probs = torch.exp(output)\n            equality = (labels.data == probs.max(dim=1)[1])\n            acc += equality.type(torch.FloatTensor).mean()\n    res_loss = loss/len(loader)\n    res_acc = (acc.item())/len(loader)\n    return res_loss, res_acc","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"### TRAINING FUNCTION\ndef train_model(model, trainloader, validloader, criterion, optimizer,\n                scheduler, epochs=10, print_every=1, device=\"cpu\"):\n    model.to(device)\n    best_acc = 0\n    best_epoch = 0\n    \n    for e in range(epochs):\n        model.train()        \n        for images, labels in trainloader:\n            images, labels = images.to(device), labels.to(device)\n            \n            # Training \n            optimizer.zero_grad()\n            output = model.forward(images)\n            loss = criterion(output, labels)\n            loss.backward()\n            optimizer.step()\n            scheduler.step()\n        # at the end of each epoch, calculating accuracy:\n        model.eval()\n        train_loss, train_accuracy = validation(model, trainloader, criterion, device)\n        valid_loss, valid_accuracy = validation(model, validloader, criterion, device)\n        if valid_accuracy > best_acc:\n            best_acc = valid_accuracy\n            best_epoch = e\n            torch.save(model.state_dict(), \"best-state.pt\")\n        if e % print_every == 0:\n            to_print = \"Epoch: \"+str(e+1)+\" of \"+str(epochs)\n            to_print += \".. Train Loss: {:.4f}\".format(train_loss)\n            to_print += \".. Valid Loss: {:.4f}\".format(valid_loss)\n            to_print += \".. Valid Accuracy: {:.3f}\".format(valid_accuracy)\n            print(to_print)\n    # After Training:\n    model.load_state_dict(torch.load(\"best-state.pt\"))\n    to_print = \"\\nTraining completed. Best state dict is loaded.\\n\"\n    to_print += \"Best Valid Acc is: {:.4f} after {} epochs\".format(best_acc,best_epoch+1)\n    print(to_print)\n    return","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"### PREDICTION FUNCTION\ndef prediction(model, loader, device=\"cpu\"):\n    model.to(device)\n    model.eval()\n    preds_all = torch.LongTensor()\n    \n    with torch.no_grad():\n        for images in loader:\n            images = images.to(device)\n            \n            output = model.forward(images)            \n            probs = torch.exp(output)\n            pred = probs.to('cpu').max(dim=1)[1]\n            preds_all = torch.cat((preds_all, pred), dim=0)\n    return preds_all","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 4. Training the Model\n\nIn this section, I will\n* initiate CNN model\n* define criterion, optimizer and schedular\n* train the model\n\nI used a schedular to adjust learning rate. With OneCycleLR, learning rate will increase at the beginning until max learning rate, and then start decreasing. I used pct_start = 0.2, so learning rate will reach to max value at the 0.2 of the learning phase.","metadata":{}},{"cell_type":"code","source":"model_recognizer = CNN(output_size=10, dropout=0.5)\nprint(model_recognizer)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Hyperparameters:\nmax_learning_rate = 0.001\nn_epochs = 10\nkfold_n_split = 10\ntrain_batch_size = 32\nvalid_batch_size = 16\nsteps_per_epoch = math.ceil(len(train_X_tensor)*(1-1/kfold_n_split)/train_batch_size)\n\n# Criterion, optimizer, scheduler:\ncriterion = nn.CrossEntropyLoss()\noptimizer = optim.Adam(model_recognizer.parameters(), lr=max_learning_rate)\t\nscheduler = optim.lr_scheduler.OneCycleLR(optimizer,\n                                          max_lr = max_learning_rate,\n                                          epochs = n_epochs*kfold_n_split,\n                                          steps_per_epoch = steps_per_epoch,\n                                          pct_start = 0.2,\n                                          anneal_strategy = \"cos\")\n\n# Checking if GPU is available\nif torch.cuda.is_available():\n    my_device = \"cuda\"\n    print(\"GPU is enabled\")\nelse:\n    my_device = \"cpu\"\n    print(\"No GPU :(\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cv = StratifiedKFold(n_splits=kfold_n_split, shuffle=True, random_state=42)\n\nfor fold, (train_ind, val_ind) in enumerate(cv.split(train_X_tensor, train_y_tensor)):\n    \n    train_X_cv, val_X_cv = train_X_tensor[train_ind], train_X_tensor[val_ind]\n    train_y_cv, val_y_cv = train_y_tensor[train_ind], train_y_tensor[val_ind]\n    \n    # Creating dataloaders\n    train_tensor = TensorDataset(train_X_cv, train_y_cv)\n    valid_tensor = TensorDataset(val_X_cv, val_y_cv)\n    train_loader = DataLoader(train_tensor, batch_size=train_batch_size, shuffle=True)\n    valid_loader = DataLoader(valid_tensor, batch_size=valid_batch_size)\n    \n    # Training:\n    print(f\"\\n--- Fold: {fold+1} of {cv.n_splits} ---\\n\")\n    train_model(model = model_recognizer, trainloader = train_loader,\n                validloader = valid_loader, criterion = criterion,\n                optimizer = optimizer, scheduler = scheduler,\n                epochs = n_epochs, print_every=3, device = my_device)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 5. Prediction","metadata":{}},{"cell_type":"code","source":"test_loader = DataLoader(test_tensor, batch_size=16)\ny_pred = prediction(model_recognizer, test_loader)\nprint(\"Prediction completed...\")\n\n# Creating a dataframe for results\ntest = pd.read_csv(\"../input/digit-recognizer/test.csv\")\nresult = pd.DataFrame({'ImageId': test.index, 'Label': y_pred})\nresult[\"ImageId\"] += 1\nresult.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Plot images from test data and see predictions\nindices = [42, 314, 2022, 22111]\nf,ax = plt.subplots(1, len(indices))\nfor i in range(len(indices)):\n    title = \"Label: {}\".format(int(y_pred[indices[i]]))\n    ax[i].imshow( test_tensor[indices[i],0] )\n    ax[i].set_title(title)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"result.to_csv('submission.csv', index=False)\nprint(\"Resuls are saved to submission.csv\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]}]}