{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"

MNIST πŸ”’ – Getting with start image preprocessing & CV with PyTorch

\n\n
\n \n
","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"}},{"cell_type":"markdown","source":"Copyright [2022] [AI Engineer: [Ahmed](https://www.kaggle.com/dsxavier/)]\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this file except in compliance with the License.\nYou may obtain a copy of the License at\n\n http://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nSee the License for the specific language governing permissions and\nlimitations under the License.","metadata":{}},{"cell_type":"markdown","source":"# πŸ“–Overview\n\nMNIST (\"Modified National Institute of Standards and Technology\") is the de facto β€œhello world” dataset of computer vision. Since its release in 1999, this classic dataset of handwritten images has served as the basis for benchmarking classification algorithms. As new machine learning techniques emerge, MNIST remains a reliable resource for researchers and learners alike.\n\nIn this competition, your goal is to correctly identify digits from a dataset of tens of thousands of handwritten images. We’ve curated a set of tutorial-style kernels which cover everything from regression to neural networks. We encourage you to experiment with different algorithms to learn first-hand what works well and how techniques compare.","metadata":{}},{"cell_type":"markdown","source":"# πŸ“Acknowledgments\n\nMore details about the dataset, including algorithms that have been tried on it and their levels of success, can be found at http://yann.lecun.com/exdb/mnist/index.html. The dataset is made available under a Creative Commons Attribution-Share Alike 3.0 license.","metadata":{}},{"cell_type":"markdown","source":"# πŸ“Proof of Work\n\n**MNIST** considers being the first case study the scientists start doing their experiments on it during the 70s, and now it is one of the essential datasets that most scientists are using to verify their networks. \n\nIf you want to start exploring the field of image processing and computer vision **(CV)**, you start from the right place. \n\nIt is not just working on code more than exploring the field with you, guys. I would like to put what I have learnt into this kernel step by step so that we can come up with something simple and efficient to show you how beautiful this field is and how it can help you in further industries. Thank you for following up with my work!","metadata":{}},{"cell_type":"markdown","source":"# πŸ“šData Dictionary\n\nThe data files `train.csv` and `test.csv` contain **gray-scale images of hand-drawn digits**, from zero through nine.\n\nEach image is **28 pixels in height and 28 pixels** in width, for a total of **784 pixels** in total. Each pixel has a single pixel-value associated with it, indicating the lightness or darkness of that pixel, with higher numbers meaning darker. This pixel-value is an integer between 0 and 255, inclusive.\n\nThe training data set, (train.csv), has 785 columns. The first column, called \"label\", is the digit that was drawn by the user. The rest of the columns contain the pixel-values of the associated image.\n\nEach pixel column in the training set has a name like pixelx, where x is an integer between 0 and 783, inclusive. To locate this pixel on the image, suppose that we have decomposed x as x = i * 28 + j, where i and j are integers between 0 and 27, inclusive. Then pixelx is located on row i and column j of a 28 x 28 matrix, (indexing by zero).\n\nFor example, pixel31 indicates the pixel that is in the fourth column from the left, and the second row from the top, as in the ascii-diagram below.\n\nVisually, if we omit the \"pixel\" prefix, the pixels make up the image like this:\n\n```markdown\n000 001 002 003 ... 026 027\n028 029 030 031 ... 054 055\n056 057 058 059 ... 082 083\n | | | | ... | |\n728 729 730 731 ... 754 755\n756 757 758 759 ... 782 783 \n```\n\nThe test data set, (test.csv), is the same as the training set, except that it does not contain the \"label\" column.\n\nYour submission file should be in the following format: For each of the 28000 images in the test set, output a single line containing the ImageId and the digit you predict. For example, if you predict that the first image is of a 3, the second image is of a 7, and the third image is of a 8, then your submission file would look like:\n\n```markdown\nImageId,Label\n1,3\n2,7\n3,8 \n(27997 more lines)\n```","metadata":{}},{"cell_type":"markdown","source":"# πŸ”–Dependencies","metadata":{}},{"cell_type":"code","source":"%config Completer.use_jedi = False","metadata":{"execution":{"iopub.status.busy":"2022-04-15T19:13:07.342405Z","iopub.execute_input":"2022-04-15T19:13:07.343189Z","iopub.status.idle":"2022-04-15T19:13:07.375644Z","shell.execute_reply.started":"2022-04-15T19:13:07.343089Z","shell.execute_reply":"2022-04-15T19:13:07.374984Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# (A) Install Dependencies","metadata":{}},{"cell_type":"code","source":"!pip3 install -q graphviz\n!pip install -q -U git+https://github.com/waleedka/hiddenlayer.git","metadata":{"_kg_hide-input":false,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-04-15T19:13:07.423947Z","iopub.execute_input":"2022-04-15T19:13:07.424142Z","iopub.status.idle":"2022-04-15T19:13:29.304269Z","shell.execute_reply.started":"2022-04-15T19:13:07.424108Z","shell.execute_reply":"2022-04-15T19:13:29.303458Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# (B) Import Libraries","metadata":{}},{"cell_type":"code","source":"import os\nimport warnings\nimport math\nimport re\nimport time\nimport json\nimport random as python_random\n\nfrom IPython.display import clear_output, display\nfrom itertools import product\nfrom collections import namedtuple, OrderedDict\nfrom typing import OrderedDict, Text, Optional, Union, List, NamedTuple, Tuple\nfrom numbers import Number\nfrom tqdm.auto import tqdm\nfrom tqdm.auto import trange\n\nimport pandas as pd \nimport numpy as np\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import (f1_score, \n confusion_matrix,\n auc,\n roc_auc_score,\n classification_report,\n ConfusionMatrixDisplay)\n\nimport torch\nimport torchvision\nimport torchvision.transforms as transforms\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport torch.optim as optim\n\nimport hiddenlayer as hl\nimport seaborn as sns\nfrom matplotlib import gridspec\nimport matplotlib.pylab as plt\n\ntorch.set_printoptions(linewidth=150)\n%matplotlib inline","metadata":{"execution":{"iopub.status.busy":"2022-04-15T19:13:29.307119Z","iopub.execute_input":"2022-04-15T19:13:29.307813Z","iopub.status.idle":"2022-04-15T19:13:31.726848Z","shell.execute_reply.started":"2022-04-15T19:13:29.307781Z","shell.execute_reply":"2022-04-15T19:13:31.726043Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Main Params for Matplotlib**","metadata":{}},{"cell_type":"code","source":"plt.rcParams['figure.figsize'] = [18, 15]\nplt.style.use('fivethirtyeight')","metadata":{"execution":{"iopub.status.busy":"2022-04-15T19:13:31.728068Z","iopub.execute_input":"2022-04-15T19:13:31.730227Z","iopub.status.idle":"2022-04-15T19:13:31.734772Z","shell.execute_reply.started":"2022-04-15T19:13:31.730182Z","shell.execute_reply":"2022-04-15T19:13:31.734048Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Main Params for Seaborn**","metadata":{}},{"cell_type":"code","source":"plt.rcParams['figure.figsize'] = [18, 15]\nplt.style.use('fivethirtyeight')","metadata":{"execution":{"iopub.status.busy":"2022-04-15T19:13:31.736776Z","iopub.execute_input":"2022-04-15T19:13:31.737224Z","iopub.status.idle":"2022-04-15T19:13:31.744607Z","shell.execute_reply.started":"2022-04-15T19:13:31.737182Z","shell.execute_reply":"2022-04-15T19:13:31.743911Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Tabular Data Configuration**","metadata":{}},{"cell_type":"code","source":"pd.set_option('display.max_colwidth', 500)\npd.set_option('display.width', 1000)","metadata":{"execution":{"iopub.status.busy":"2022-04-15T19:13:31.745651Z","iopub.execute_input":"2022-04-15T19:13:31.746277Z","iopub.status.idle":"2022-04-15T19:13:31.752348Z","shell.execute_reply.started":"2022-04-15T19:13:31.746236Z","shell.execute_reply":"2022-04-15T19:13:31.751458Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# (C) Hardware Specifications","metadata":{}},{"cell_type":"code","source":"# https://stackoverflow.com/a/20354129/14473118\nimport multiprocessing\npool = multiprocessing.Pool()\nprint(f'Number of workers for parallelism: {pool._processes}')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-04-15T19:13:31.753774Z","iopub.execute_input":"2022-04-15T19:13:31.754259Z","iopub.status.idle":"2022-04-15T19:13:31.785663Z","shell.execute_reply.started":"2022-04-15T19:13:31.754218Z","shell.execute_reply":"2022-04-15T19:13:31.784645Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!nvidia-smi","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-04-15T19:13:31.788498Z","iopub.execute_input":"2022-04-15T19:13:31.788849Z","iopub.status.idle":"2022-04-15T19:13:32.538111Z","shell.execute_reply.started":"2022-04-15T19:13:31.788802Z","shell.execute_reply":"2022-04-15T19:13:32.53731Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---","metadata":{}},{"cell_type":"markdown","source":"# 1. Data Preparation\n\n- we will start with preparaing the data. To prepare our data, we will be following what is loosely known as an [ETL process](https://en.wikipedia.org/wiki/Extract,_transform,_load)\n\n- **Extract** data from a data source.\n- **Transform** data into a desirable format.\n- **Load** data into a suitable structure.","metadata":{}},{"cell_type":"markdown","source":"# (A) Extract, Transform, and Load (ETL)\n\n For these purposes, PyTorch provides us with two classes: \n\n\n\n \n \n \n \n\n\n \n \n \n \n \n \n \n \n\n
ClassDescription
torch.utils.data.DatasetAn abstract class for representing a dataset.
torch.utils.data.DataLoaderWraps a dataset and provides access to the underlying data.
","metadata":{}},{"cell_type":"markdown","source":"**Adding Control seed**","metadata":{}},{"cell_type":"code","source":"# Controlling random samples\ndef set_seed():\n np.random.seed(123)\n python_random.seed(123)\n torch.manual_seed(1234)\n if torch.cuda.is_available():\n torch.cuda.manual_seed_all(1234)\nset_seed()","metadata":{"execution":{"iopub.status.busy":"2022-04-15T19:13:32.540024Z","iopub.execute_input":"2022-04-15T19:13:32.54031Z","iopub.status.idle":"2022-04-15T19:13:32.59771Z","shell.execute_reply.started":"2022-04-15T19:13:32.540272Z","shell.execute_reply":"2022-04-15T19:13:32.597013Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### (I) Extract","metadata":{}},{"cell_type":"code","source":"train_data = pd.read_csv('../input/digit-recognizer/train.csv')\ntest_data = pd.read_csv('../input/digit-recognizer/test.csv')","metadata":{"execution":{"iopub.status.busy":"2022-04-15T19:13:32.599364Z","iopub.execute_input":"2022-04-15T19:13:32.599566Z","iopub.status.idle":"2022-04-15T19:13:37.147323Z","shell.execute_reply.started":"2022-04-15T19:13:32.599529Z","shell.execute_reply":"2022-04-15T19:13:37.1466Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_data.head()","metadata":{"execution":{"iopub.status.busy":"2022-04-15T19:13:37.150883Z","iopub.execute_input":"2022-04-15T19:13:37.151091Z","iopub.status.idle":"2022-04-15T19:13:37.178887Z","shell.execute_reply.started":"2022-04-15T19:13:37.151066Z","shell.execute_reply":"2022-04-15T19:13:37.178231Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_data.info()","metadata":{"execution":{"iopub.status.busy":"2022-04-15T19:13:37.180271Z","iopub.execute_input":"2022-04-15T19:13:37.180521Z","iopub.status.idle":"2022-04-15T19:13:37.231181Z","shell.execute_reply.started":"2022-04-15T19:13:37.180488Z","shell.execute_reply":"2022-04-15T19:13:37.230125Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"images = train_data.loc[:, train_data.columns != 'label']\nlabel = train_data.loc[:, 'label']","metadata":{"execution":{"iopub.status.busy":"2022-04-15T19:13:37.232237Z","iopub.execute_input":"2022-04-15T19:13:37.232641Z","iopub.status.idle":"2022-04-15T19:13:37.317485Z","shell.execute_reply.started":"2022-04-15T19:13:37.2326Z","shell.execute_reply":"2022-04-15T19:13:37.316505Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"images","metadata":{"execution":{"iopub.status.busy":"2022-04-15T19:13:37.319868Z","iopub.execute_input":"2022-04-15T19:13:37.320513Z","iopub.status.idle":"2022-04-15T19:13:37.346149Z","shell.execute_reply.started":"2022-04-15T19:13:37.320465Z","shell.execute_reply":"2022-04-15T19:13:37.345456Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"label","metadata":{"execution":{"iopub.status.busy":"2022-04-15T19:13:37.347257Z","iopub.execute_input":"2022-04-15T19:13:37.347671Z","iopub.status.idle":"2022-04-15T19:13:37.356305Z","shell.execute_reply.started":"2022-04-15T19:13:37.347634Z","shell.execute_reply":"2022-04-15T19:13:37.355453Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"np.unique(images.to_numpy())","metadata":{"execution":{"iopub.status.busy":"2022-04-15T19:13:37.359083Z","iopub.execute_input":"2022-04-15T19:13:37.359287Z","iopub.status.idle":"2022-04-15T19:13:38.212031Z","shell.execute_reply.started":"2022-04-15T19:13:37.359261Z","shell.execute_reply":"2022-04-15T19:13:38.211275Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### (II) Transform","metadata":{}},{"cell_type":"markdown","source":"#### Data Normalization\n\nThe idea of data [normalization](https://en.wikipedia.org/wiki/Normalization_(statistics)) is an general concept that refers to the act of transforming the original values of a dataset to new values. The new values are typically encoded relative to the dataset itself and are scaled in some way.","metadata":{}},{"cell_type":"markdown","source":"We're going to use `transforms.Normalize`. We need the `mean` and `std` of the dataset so we can apply the normalization.\n\n**what is the point of doing that?**\n\nOur goal is to normalize the range of the colours in the dataset from [0, 255] to be in mean in **zero** and **std** on **1**. **Standard Deviation** summarises the data and controls the spread of the data, thus we would have higher accuracy!\n\n","metadata":{}},{"cell_type":"code","source":"images.values","metadata":{"execution":{"iopub.status.busy":"2022-04-15T19:13:38.213246Z","iopub.execute_input":"2022-04-15T19:13:38.21358Z","iopub.status.idle":"2022-04-15T19:13:38.219312Z","shell.execute_reply.started":"2022-04-15T19:13:38.213526Z","shell.execute_reply":"2022-04-15T19:13:38.21859Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def normalization(images: pd.DataFrame) -> Tuple[float, float]:\n # let's get the number of pixels of our dataset\n num_of_pixels = len(images) * len(images.columns.values.tolist()) # No. of images * flatten pixels (784)\n\n # summation of the images' colours\n total_sum = images.to_numpy().sum()\n\n mean = total_sum / num_of_pixels\n\n sum_of_squard_error = np.power(images.to_numpy() - mean, 2).sum()\n sum_of_squard_error\n\n std = np.sqrt(sum_of_squard_error/num_of_pixels)\n return mean, std","metadata":{"execution":{"iopub.status.busy":"2022-04-15T19:13:38.220486Z","iopub.execute_input":"2022-04-15T19:13:38.221142Z","iopub.status.idle":"2022-04-15T19:13:38.228058Z","shell.execute_reply.started":"2022-04-15T19:13:38.221097Z","shell.execute_reply":"2022-04-15T19:13:38.22737Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(normalization(images))","metadata":{"execution":{"iopub.status.busy":"2022-04-15T19:13:38.229246Z","iopub.execute_input":"2022-04-15T19:13:38.229872Z","iopub.status.idle":"2022-04-15T19:13:39.412049Z","shell.execute_reply.started":"2022-04-15T19:13:38.22983Z","shell.execute_reply":"2022-04-15T19:13:39.411239Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.hist(images.to_numpy().flatten())\nplt.axvline(images.to_numpy().mean(), color='DarkTurquoise')\nplt.title('Mean of our images before normalization', fontsize=24, color='MediumSlateBlue')\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-04-15T19:13:39.413421Z","iopub.execute_input":"2022-04-15T19:13:39.413702Z","iopub.status.idle":"2022-04-15T19:13:40.378801Z","shell.execute_reply.started":"2022-04-15T19:13:39.413665Z","shell.execute_reply":"2022-04-15T19:13:40.37812Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we can see in the plot, the **mean** is `~33.4` and **standard deviation** is `~78.67`, which is not good at all. Let's try to normalize our data and see the `mean` and `std` again! ","metadata":{}},{"cell_type":"markdown","source":"Let's split our dataset into train, validation","metadata":{}},{"cell_type":"code","source":"(train_images, valid_images,\n train_label, valid_label) = train_test_split(images, \n label,\n test_size=0.3,\n random_state=123,\n stratify=label) ","metadata":{"execution":{"iopub.status.busy":"2022-04-15T19:13:40.380256Z","iopub.execute_input":"2022-04-15T19:13:40.380716Z","iopub.status.idle":"2022-04-15T19:13:40.641961Z","shell.execute_reply.started":"2022-04-15T19:13:40.380678Z","shell.execute_reply":"2022-04-15T19:13:40.641204Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

To know more about Stratify and why we used it, you can visit Titanic - How Data Scientist applies Analysis & ML – \n2. Train & Test Split

\n\n","metadata":{}},{"cell_type":"markdown","source":"Initialize the **Transformer** – this allows us to create various tasks simultaneously without overloading the data into multiple processes. For more info. [Transforms](https://pytorch.org/tutorials/beginner/basics/transforms_tutorial.html)","metadata":{}},{"cell_type":"code","source":"len(train_images), len(valid_images), len(train_label), len(valid_label)","metadata":{"execution":{"iopub.status.busy":"2022-04-15T19:13:40.643191Z","iopub.execute_input":"2022-04-15T19:13:40.643443Z","iopub.status.idle":"2022-04-15T19:13:40.65132Z","shell.execute_reply.started":"2022-04-15T19:13:40.643408Z","shell.execute_reply":"2022-04-15T19:13:40.650475Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# obtain mean, and std from our dataset\ntrain_normalization_values = normalization(train_images)\nvalid_normalization_values = normalization(valid_images)","metadata":{"execution":{"iopub.status.busy":"2022-04-15T19:13:40.652651Z","iopub.execute_input":"2022-04-15T19:13:40.653056Z","iopub.status.idle":"2022-04-15T19:13:41.586315Z","shell.execute_reply.started":"2022-04-15T19:13:40.653019Z","shell.execute_reply":"2022-04-15T19:13:41.585598Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def transform(images: np.ndarray,\n normalization_values: float) -> torch.Tensor:\n transformer = transforms.Compose([\n transforms.ToTensor(),\n transforms.Normalize(*normalization_values)\n ])\n return transformer(images)\n\ntensor_train_images = transform(train_images.to_numpy().astype(np.float64), train_normalization_values)\ntensor_valid_images = transform(valid_images.to_numpy().astype(np.float64), valid_normalization_values)","metadata":{"execution":{"iopub.status.busy":"2022-04-15T19:13:41.587619Z","iopub.execute_input":"2022-04-15T19:13:41.587958Z","iopub.status.idle":"2022-04-15T19:13:42.12207Z","shell.execute_reply.started":"2022-04-15T19:13:41.587922Z","shell.execute_reply":"2022-04-15T19:13:42.121345Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tensor_train_images","metadata":{"execution":{"iopub.status.busy":"2022-04-15T19:13:42.123277Z","iopub.execute_input":"2022-04-15T19:13:42.123644Z","iopub.status.idle":"2022-04-15T19:13:42.158107Z","shell.execute_reply.started":"2022-04-15T19:13:42.123606Z","shell.execute_reply":"2022-04-15T19:13:42.157445Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.hist(tensor_train_images.numpy().flatten())\nplt.axvline(tensor_train_images.numpy().mean(), color='DarkTurquoise')\nplt.title('Mean of our images after normalization', fontsize=24, color='MediumSlateBlue')\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-04-15T19:13:42.159241Z","iopub.execute_input":"2022-04-15T19:13:42.159467Z","iopub.status.idle":"2022-04-15T19:13:42.836302Z","shell.execute_reply.started":"2022-04-15T19:13:42.159434Z","shell.execute_reply":"2022-04-15T19:13:42.83562Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, we have our images' colours normalized and ready to load!","metadata":{}},{"cell_type":"markdown","source":"### (III) Load","metadata":{}},{"cell_type":"code","source":"torch.from_numpy(train_label.to_numpy()).shape","metadata":{"execution":{"iopub.status.busy":"2022-04-15T19:13:42.837441Z","iopub.execute_input":"2022-04-15T19:13:42.838187Z","iopub.status.idle":"2022-04-15T19:13:42.845021Z","shell.execute_reply.started":"2022-04-15T19:13:42.838144Z","shell.execute_reply":"2022-04-15T19:13:42.843965Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tensor_train_images.shape","metadata":{"execution":{"iopub.status.busy":"2022-04-15T19:13:42.847364Z","iopub.execute_input":"2022-04-15T19:13:42.847752Z","iopub.status.idle":"2022-04-15T19:13:42.854619Z","shell.execute_reply.started":"2022-04-15T19:13:42.847712Z","shell.execute_reply":"2022-04-15T19:13:42.853814Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"First, we're going to build our dataset using `torch.utils.data.TensorDataset` for representing our `train_dataset` and `validation_dataset`.","metadata":{}},{"cell_type":"code","source":"train_dataset = torch.utils.data.TensorDataset(tensor_train_images.squeeze(0),\n torch.from_numpy(train_label.to_numpy()))\nvalid_dataset = torch.utils.data.TensorDataset(tensor_valid_images.squeeze(0),\n torch.from_numpy(valid_label.to_numpy()))","metadata":{"execution":{"iopub.status.busy":"2022-04-15T19:13:42.860592Z","iopub.execute_input":"2022-04-15T19:13:42.861163Z","iopub.status.idle":"2022-04-15T19:13:42.866976Z","shell.execute_reply.started":"2022-04-15T19:13:42.86112Z","shell.execute_reply":"2022-04-15T19:13:42.866203Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 2. Data Analysis\n\nHere, we're going to analyse our dataset and check the proportion and balance of our dataset. ","metadata":{}},{"cell_type":"markdown","source":"# (A) Analysing images\n","metadata":{}},{"cell_type":"code","source":"len(train_dataset)","metadata":{"execution":{"iopub.status.busy":"2022-04-15T19:13:42.869685Z","iopub.execute_input":"2022-04-15T19:13:42.870131Z","iopub.status.idle":"2022-04-15T19:13:42.877241Z","shell.execute_reply.started":"2022-04-15T19:13:42.870092Z","shell.execute_reply":"2022-04-15T19:13:42.876429Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_dataset.tensors","metadata":{"execution":{"iopub.status.busy":"2022-04-15T19:13:42.879669Z","iopub.execute_input":"2022-04-15T19:13:42.879967Z","iopub.status.idle":"2022-04-15T19:13:42.88844Z","shell.execute_reply.started":"2022-04-15T19:13:42.87993Z","shell.execute_reply":"2022-04-15T19:13:42.88756Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's check the number of samples in each class. It is important to have the same length of examples in each class","metadata":{}},{"cell_type":"code","source":"train_dataset.tensors[1].bincount()","metadata":{"execution":{"iopub.status.busy":"2022-04-15T19:13:42.890377Z","iopub.execute_input":"2022-04-15T19:13:42.890661Z","iopub.status.idle":"2022-04-15T19:13:42.8976Z","shell.execute_reply.started":"2022-04-15T19:13:42.890613Z","shell.execute_reply":"2022-04-15T19:13:42.896696Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It seems like our dataset is **unbalanced dataset**.\n\n**A systematic study of the class imbalance problem in convolutional neural networks4**\n\n> *The default version of oversampling is to increase the number of cases in the minority classes\nso that the number matches the majority classes. Similarly, the default of undersampling is to\ndecrease the number of cases in the majority classes to match the minority classes. However,\na more moderate version of these algorithms could be applied. For the case of MNIST with\nimbalance ratio of 1 000 we have tried to gradually decrease the imbalance with oversampling\nand undersampling.*","metadata":{}},{"cell_type":"markdown","source":"

Number of samples per class

","metadata":{}},{"cell_type":"code","source":"plt.title('Number of samples in each class')\nsns.countplot(x=train_dataset.tensors[1].numpy())\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-04-15T19:13:42.900424Z","iopub.execute_input":"2022-04-15T19:13:42.900885Z","iopub.status.idle":"2022-04-15T19:13:43.155069Z","shell.execute_reply.started":"2022-04-15T19:13:42.900845Z","shell.execute_reply":"2022-04-15T19:13:43.154379Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's check our images","metadata":{}},{"cell_type":"markdown","source":"

Random samples from MNIST Dataset

","metadata":{}},{"cell_type":"code","source":"fig, axis = plt.subplots(5, 5, figsize=(14, 16))\n\nfig.suptitle('MNIST Number Images and Labels', fontsize=21, color='MediumSlateBlue')\nfig.tight_layout(pad=1.6)\n# flatten the multi-dimentionality of `subplots`\naxis = axis.ravel()\n\nfor i in range(25):\n sample = python_random.randint(0, len(train_dataset))\n axis[i].imshow(train_dataset[sample][0].numpy().reshape(28, 28))\n axis[i].get_xaxis().set_visible(False)\n axis[i].get_yaxis().set_visible(False)\n axis[i].set_title(train_dataset[sample][1].numpy(),\n fontsize=18, color='DarkSlateBlue')\nplt.subplots_adjust(hspace=0.2)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-04-15T19:13:43.156302Z","iopub.execute_input":"2022-04-15T19:13:43.156534Z","iopub.status.idle":"2022-04-15T19:13:44.594262Z","shell.execute_reply.started":"2022-04-15T19:13:43.156501Z","shell.execute_reply":"2022-04-15T19:13:44.591727Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# (B) Analysing Model layers\n\nFor Analysing Model layers – the **PyTorch** debugging system is different compared with **TensorFlow**. It is beneficial because we can test our layers on our images and see the effect of each layer on our image. Our images are in the same dimensionality so that we won't face any issues if we choose the right layers to build our network.","metadata":{}},{"cell_type":"markdown","source":"First, I need to know if **MaxPooling** would affect our images. Our images are small in size, and as we know **MaxPooling** considers reducing the size of the images.","metadata":{}},{"cell_type":"code","source":"train_dataset[0][0].shape","metadata":{"execution":{"iopub.status.busy":"2022-04-15T19:13:44.595527Z","iopub.execute_input":"2022-04-15T19:13:44.595874Z","iopub.status.idle":"2022-04-15T19:13:44.60297Z","shell.execute_reply.started":"2022-04-15T19:13:44.595835Z","shell.execute_reply":"2022-04-15T19:13:44.602281Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's check the image that we're going to work with","metadata":{}},{"cell_type":"markdown","source":"

Exprimenting with MaxPooling

","metadata":{}},{"cell_type":"code","source":"plt.title(f\"Image Label: {train_dataset[0][1]}\", fontsize=21)\nplt.imshow(train_dataset[0][0].reshape(28, 28), interpolation=None)\nplt.grid(False)\nax = plt.gca()\nax.axes.xaxis.set_visible(False)\nax.axes.yaxis.set_visible(False)\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-04-15T19:13:44.604231Z","iopub.execute_input":"2022-04-15T19:13:44.60462Z","iopub.status.idle":"2022-04-15T19:13:44.8019Z","shell.execute_reply.started":"2022-04-15T19:13:44.604585Z","shell.execute_reply":"2022-04-15T19:13:44.8011Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"maxpooling = nn.MaxPool2d(kernel_size=5)\nplt.imshow(maxpooling(train_dataset[0][0].reshape(1, 28, 28)).squeeze(0))\nplt.title(f\"Image after applying MaxPooling with kernel = 5\", fontsize=21)\nplt.grid(False)\nax = plt.gca()\nax.axes.xaxis.set_visible(False)\nax.axes.yaxis.set_visible(False)\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-04-15T19:13:44.803254Z","iopub.execute_input":"2022-04-15T19:13:44.803512Z","iopub.status.idle":"2022-04-15T19:13:45.02277Z","shell.execute_reply.started":"2022-04-15T19:13:44.803477Z","shell.execute_reply":"2022-04-15T19:13:45.021954Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**That's interesting...**","metadata":{}},{"cell_type":"markdown","source":"We can see that the image after the **max-pooling** has been corrupted. This corruption leads to misclassification. Therefore, it is better not to use **Maxpooling** in our network.","metadata":{}},{"cell_type":"markdown","source":"Let's change the `kernel_size`","metadata":{}},{"cell_type":"code","source":"maxpooling = nn.MaxPool2d(kernel_size=2)\nplt.imshow(maxpooling(train_dataset[0][0].reshape(1, 28, 28)).squeeze(0))\nplt.title(f\"Image after applying MaxPooling with kernel = 5\", fontsize=21)\nplt.grid(False)\nax = plt.gca()\nax.axes.xaxis.set_visible(False)\nax.axes.yaxis.set_visible(False)\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-04-15T19:13:45.024313Z","iopub.execute_input":"2022-04-15T19:13:45.024814Z","iopub.status.idle":"2022-04-15T19:13:45.232574Z","shell.execute_reply.started":"2022-04-15T19:13:45.024773Z","shell.execute_reply":"2022-04-15T19:13:45.23186Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's check the proper `kernel_size` for the convolution layer","metadata":{}},{"cell_type":"markdown","source":"As we know, the input to our **CNN Network** has to be in shape of `4d Tensor` – this indicates the [batch, n_channels, height, width]\n\nTo convert our image from `3d Tensor` to `4d Tensor` of one image, we `unsqueeze` to give us batch with `1` size, or by using `reshape`.","metadata":{}},{"cell_type":"code","source":"train_dataset[0][0].reshape(1, 1, 28, 28).shape","metadata":{"execution":{"iopub.status.busy":"2022-04-15T19:13:45.233997Z","iopub.execute_input":"2022-04-15T19:13:45.234406Z","iopub.status.idle":"2022-04-15T19:13:45.240993Z","shell.execute_reply.started":"2022-04-15T19:13:45.234365Z","shell.execute_reply":"2022-04-15T19:13:45.240087Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Convolution Layer** has to provide 6 different types of channels based on the one channel that we're going to provide, let's see how it will display 6 different channels from our image. This task has to show us if the `kernel_size` that we provide will affect the image or not.","metadata":{}},{"cell_type":"code","source":"conv1 = nn.Conv2d(in_channels=1,\n out_channels=6,\n kernel_size=5)\n\nconv1(train_dataset[0][0].reshape(1, 1, 28, 28).float()).detach().shape","metadata":{"execution":{"iopub.status.busy":"2022-04-15T19:13:45.242605Z","iopub.execute_input":"2022-04-15T19:13:45.242973Z","iopub.status.idle":"2022-04-15T19:13:45.295059Z","shell.execute_reply.started":"2022-04-15T19:13:45.242874Z","shell.execute_reply":"2022-04-15T19:13:45.294368Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

Exprimenting with Convolution Layer

","metadata":{}},{"cell_type":"code","source":"fig, axis = plt.subplots(2, 3, figsize=(16, 12))\nfig.suptitle(f'Number of channels for Number: {train_dataset[0][1].detach().numpy()}',\n fontsize=24, color='MediumSlateBlue')\n\nfig.tight_layout(pad=1.2)\n\n# flatten the multi-dimentionality of `subplots`\naxis = axis.ravel()\nfor i in range(6):\n axis[i].imshow(conv1(train_dataset[0][0].reshape(1, 1, 28, 28).float()).detach().squeeze(0)[i])\n axis[i].get_xaxis().set_visible(False)\n axis[i].get_yaxis().set_visible(False)\n axis[i].set_title(f\"Image Label: {train_dataset[0][1].numpy()} | Channel: {i+1}\",\n fontsize=18, color='DarkSlateBlue')\nplt.subplots_adjust(hspace=0.2)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-04-15T19:13:45.296274Z","iopub.execute_input":"2022-04-15T19:13:45.296508Z","iopub.status.idle":"2022-04-15T19:13:45.861642Z","shell.execute_reply.started":"2022-04-15T19:13:45.296477Z","shell.execute_reply":"2022-04-15T19:13:45.860907Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 3. Modelling","metadata":{}},{"cell_type":"markdown","source":"# (A) Building Model\n\n\n- Let's now build our model. Since we're going to build two models for a testing network with `with_maxpooling` and `without_maxpooling`.","metadata":{}},{"cell_type":"code","source":"class NetworkFactory:\n @staticmethod # running without creating class instance\n def get_network(name):\n if name == 'with_MaxPooling':\n return nn.Sequential(\n nn.Conv2d(in_channels=1, out_channels=6, kernel_size=5),\n nn.ReLU(),\n nn.MaxPool2d(kernel_size=2, stride=2),\n nn.BatchNorm2d(num_features=6), # It is important for prevent exploring the gradient by normalizing the weights\n nn.Conv2d(in_channels=6, out_channels=12, kernel_size=5),\n nn.ReLU(),\n nn.MaxPool2d(kernel_size=2, stride=2),\n nn.Flatten(start_dim=1),\n nn.Linear(in_features= 12 * 4 * 4, out_features=120),\n nn.ReLU(),\n nn.BatchNorm1d(num_features=120),\n nn.Linear(in_features=120, out_features=60),\n nn.ReLU(),\n nn.Linear(in_features=60, out_features=10)\n )\n elif name == 'without_MaxPooling':\n return nn.Sequential(\n nn.Conv2d(in_channels=1, out_channels=6, kernel_size=5),\n nn.ReLU(),\n# nn.MaxPool2d(kernel_size=2, stride=2),\n nn.BatchNorm2d(num_features=6), # It is important for prevent exploring the gradient by normalizing the weights\n nn.Conv2d(in_channels=6, out_channels=12, kernel_size=5),\n nn.ReLU(),\n# nn.MaxPool2d(kernel_size=2, stride=2),\n nn.Flatten(start_dim=1),\n nn.Linear(in_features= 12 * 20 * 20, out_features=120),\n nn.ReLU(),\n nn.BatchNorm1d(num_features=120),\n nn.Linear(in_features=120, out_features=60),\n nn.ReLU(),\n nn.Linear(in_features=60, out_features=10)\n )","metadata":{"execution":{"iopub.status.busy":"2022-04-15T19:13:45.863109Z","iopub.execute_input":"2022-04-15T19:13:45.863384Z","iopub.status.idle":"2022-04-15T19:13:45.875786Z","shell.execute_reply.started":"2022-04-15T19:13:45.863347Z","shell.execute_reply":"2022-04-15T19:13:45.874979Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### ***Analysing our Network***","metadata":{}},{"cell_type":"markdown","source":"Now, let's take it step by step and see how did we end up with `Linear(12 * 4 * 4, 120)` or `Linear(12 * 20 * 20, 120)`?\n\nFirst, we need to visualize one of the models that we created, then we're going to move in into each of the CNN layers, mathematically.\n\nFor visualizing our model in **PyTorch** – there's no directed way to do that, but on the other hand, this gives us the capability to play with our model smoothly.\n\nWe're going to use a GitHub repository called [HiddenLayer](https://github.com/waleedka/hiddenlayer). \n\nI'm impressed by what these guys were able to contribute to bringing us these tools in their package.","metadata":{}},{"cell_type":"code","source":"warnings.filterwarnings('ignore')\n# https://stackoverflow.com/a/62458882/14473118\ntrans = [ hl.transforms.Prune('Constant') ] # Removes Constant nodes from graph.\nhl_graph = hl.build_graph(NetworkFactory.get_network('with_MaxPooling'),\n train_dataset[:3][0].reshape(3, 1, 28, 28).float(),\n transforms=trans)\n\nhl_graph.theme = hl.graph.THEMES[\"blue\"].copy()\nhl_graph","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-04-15T19:13:45.877367Z","iopub.execute_input":"2022-04-15T19:13:45.877862Z","iopub.status.idle":"2022-04-15T19:13:46.653867Z","shell.execute_reply.started":"2022-04-15T19:13:45.877825Z","shell.execute_reply":"2022-04-15T19:13:46.653129Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Starting from the image – we converted its flattened values into 3 dimensions `(n_channels, height, width)`, then we sent it to the first **Convolution Layer**, which takes 4 dimensions `(batch_size, n_channels, height, width)`.\n\n
$\\frac{n \\text{ -} f \\text{ + } 2\\times\\text{ p}}{s}+1$
\n\n**Convolution Layer No.01**:\n\n--> Input: image (28px, 28px)\n\n--> Process: $\\frac{size\\_pixels \\text{ -} kernel\\_size \\text{ + } 2\\times\\text{ padding}}{stride}+1= \\frac{28 \\text{ -} 5 \\text{ + } 2\\times\\text{ 0}}{1}+1 = 24px$\n\n--> Output: image (24px, 24px)\n\n**MaxPool No. 01**:\n\n--> Input: image (24px, 24px)\n\n--> Process: $\\frac{size\\_pixels \\text{ -} kernel\\_size \\text{ + } 2\\times\\text{ padding}}{stride}+1= \\frac{24 \\text{ -} 2 \\text{ + } 2\\times\\text{ 0}}{2}+1 = 12px$\n\n--> Output: image (12px, 12px)\n\n**Convolution Layer No.02**:\n\n--> Input: image (12px, 12px)\n\n--> Process: $\\frac{size\\_pixels \\text{ -} kernel\\_size \\text{ + } 2\\times\\text{ padding}}{stride}+1= \\frac{12 \\text{ -} 5 \\text{ + } 2\\times\\text{ 0}}{1}+1 = 8px$\n\n--> Output: image (8px, 8px)\n\n**MaxPool No. 02**:\n\n--> Input: image (8px, 8px)\n\n--> Process: $\\frac{size\\_pixels \\text{ -} kernel\\_size \\text{ + } 2\\times\\text{ padding}}{stride}+1= \\frac{8 \\text{ -} 2 \\text{ + } 2\\times\\text{ 0}}{2}+1 = 4px$\n\n--> Output: image (4px, 4px)\n\n**Flatten**:\n\n--> Input: image (4px, 4px)\n\n--> process: flatten vector :- 4*4 = 16\n\n**Linear**:\n\n--> in_features: 12 `n_channels`, 16 flatten image\n\n--> Process: 12 * 16 = 192\n\n --> Output: 120\n","metadata":{}},{"cell_type":"markdown","source":"# (B) Building Run Builder\n\n- This module will allow us to run multiple trials.","metadata":{}},{"cell_type":"code","source":"class RunBuilder:\n @staticmethod\n def get_runs(params: OrderedDict[Text,\n Optional[Union[Number,\n Text]]]) -> List[NamedTuple]:\n # take each key in the dict and add it as a class name for Run\n Run = namedtuple('Run', params.keys())\n \n # a list that reserves all the combinations from the product module out of the\n ## params dict.\n runs = []\n for value in product(*params.values()):\n runs.append(Run(*value))\n \n return runs","metadata":{"execution":{"iopub.status.busy":"2022-04-15T19:13:46.655169Z","iopub.execute_input":"2022-04-15T19:13:46.655376Z","iopub.status.idle":"2022-04-15T19:13:46.66163Z","shell.execute_reply.started":"2022-04-15T19:13:46.65535Z","shell.execute_reply":"2022-04-15T19:13:46.660953Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# (C) Training Loop Refactoring\n\n- This phase aims to have an encapsulated Run Manager that helps us to clear the complexity in our Huperparameter and Monitoring Phase and also be able to help us to trace all the information we need without affecting the structure of our code.","metadata":{}},{"cell_type":"markdown","source":"#### ***Variable Refactoring***","metadata":{}},{"cell_type":"code","source":"class Epoch:\n def __init__(self) -> None:\n self.count = 0\n self.best_epoch = 0\n self.loss = 0\n self.num_correct = 0\n self.f1_score = 0\n self.roc_auc = 0\n self.cm = []\n self.start_time = 0","metadata":{"execution":{"iopub.status.busy":"2022-04-15T19:13:46.662795Z","iopub.execute_input":"2022-04-15T19:13:46.663436Z","iopub.status.idle":"2022-04-15T19:13:46.670963Z","shell.execute_reply.started":"2022-04-15T19:13:46.663284Z","shell.execute_reply":"2022-04-15T19:13:46.670196Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class Run:\n def __init__(self) -> None:\n self.params = None\n self.count = 0\n self.best_run = 0\n self.data = []\n self.start_time = None\n self.num_workers = 0","metadata":{"execution":{"iopub.status.busy":"2022-04-15T19:13:46.672105Z","iopub.execute_input":"2022-04-15T19:13:46.672812Z","iopub.status.idle":"2022-04-15T19:13:46.678806Z","shell.execute_reply.started":"2022-04-15T19:13:46.672775Z","shell.execute_reply":"2022-04-15T19:13:46.678123Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class Evaluate:\n def __init__(self) -> None:\n self.loss = 0\n self.num_correct = 0\n self.best_accuracy = 0\n self.f1_score = 0\n self.best_f1 = 0\n self.roc_auc = 0\n self.cm = []","metadata":{"execution":{"iopub.status.busy":"2022-04-15T19:13:46.680199Z","iopub.execute_input":"2022-04-15T19:13:46.680449Z","iopub.status.idle":"2022-04-15T19:13:46.686585Z","shell.execute_reply.started":"2022-04-15T19:13:46.680416Z","shell.execute_reply":"2022-04-15T19:13:46.68584Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### ***Visualization Canvas***","metadata":{}},{"cell_type":"code","source":"class MyCanvas(hl.Canvas):\n \"\"\"Extending Canvas to add a pie chart method.\"\"\"\n\n def draw_image(self, metric, limit=3):\n \"\"\"Display a series of images at different time steps.\"\"\"\n rows = 2\n cols = limit\n self.ax.axis(\"off\")\n self.ax.set_title('Convolution Layer Channels',\n fontsize=18,\n color='DarkSlateBlue',\n pad=30)\n \n self.figure.tight_layout(pad=1.2)\n # Take the Axes gridspec and divide it into a grid\n gs = gridspec.GridSpecFromSubplotSpec(\n rows, cols, subplot_spec=self.gs)\n \n # Loop through images in last few steps\n for i in range(rows * cols):\n ax = self.figure.add_subplot(gs[i])\n ax.axis('off')\n ax.set_title(f'Channel: {i+1}', fontsize=14, color='MediumSlateBlue')\n ax.imshow(metric.data[-1][i])\n \n \n def draw_pie(self, metric, val_metric):\n # Method name must start with 'draw_' for the Canvas to automatically manage it\n # Use the provided matplotlib Axes in self.ax\n self.ax.axis(\"off\")\n self.ax.set_title('Accuracy of the training & validation datasets',\n fontsize=18,\n color='DarkSlateBlue',\n pad=30)\n # Get latest value of the metric\n value = np.clip(metric.data[-1], 0, 1)\n val_value = np.clip(val_metric.data[-1], 0, 1)\n \n # Take the Axes gridspec and divide it into a grid\n gs = gridspec.GridSpecFromSubplotSpec(\n 1, 2, subplot_spec=self.gs)\n \n ax1 = self.figure.add_subplot(gs[0])\n ax2 = self.figure.add_subplot(gs[1])\n \n # Draw accuracy pie chart\n ax1.pie([value, 1-value],\n labels=[\"Accuracy\", \"\"],\n autopct='%1.1f%%')\n \n ax2.pie([val_value, 1-val_value],\n labels=[\"Val_Accuracy\", \"\"],\n autopct='%1.1f%%')\n \n def draw_confusion_matrix(self, cm_matric, val_cm_matric):\n # Use the provided matplotlib Axes in self.ax\n self.ax.axis(\"off\")\n self.ax.set_title('Plotting Confusion Matrix',\n fontsize=18,\n color='DarkSlateBlue',\n pad=30)\n # https://stackoverflow.com/a/57452776/14473118\n cm = cm_matric.data[-1]\n norm_cm = np.round(cm / cm.sum(axis=1)[:np.newaxis], 3)\n val_cm = val_cm_matric.data[-1]\n norm_valid_cm = np.round(val_cm / val_cm.sum(axis=1)[:np.newaxis], 3)\n \n # Take the Axes gridspec and divide it into a grid\n gs = gridspec.GridSpecFromSubplotSpec(1, 2,\n subplot_spec=self.gs,\n hspace=0.2,\n wspace=0.6,\n width_ratios=[21, 21])\n \n self.figure.tight_layout(pad=1.2)\n self.figure.set_size_inches(14, 18)\n ax1 = self.figure.add_subplot(gs[0])\n ax2 = self.figure.add_subplot(gs[1])\n \n self.figure.gca().invert_yaxis()\n \n \n ax1.set_title('Training Confusion Matrix',\n fontsize=14,\n color='MediumSlateBlue',\n pad=10)\n ax1.grid(False)\n sns.heatmap(norm_cm,\n vmin=0,\n vmax=1,\n square=True,\n cmap=\"Blues\",\n linewidths=0.1,\n annot=True,\n annot_kws={\"fontsize\":8},\n ax=ax1)\n \n ax1.set_xticklabels(list(range(0, 10)))\n ax1.set_yticklabels(list(range(0, 10)))\n \n ax2.set_title('Validation Confusion Matrix',\n fontsize=12,\n color='MediumSlateBlue',\n pad=10)\n ax2.grid(False)\n sns.heatmap(norm_valid_cm,\n vmin=0,\n vmax=1,\n square=True,\n cmap=\"Blues\",\n linewidths=0.3,\n annot=True,\n annot_kws={\"fontsize\":8},\n ax=ax2)\n ax2.set_xticklabels(list(range(0, 10)))\n ax2.set_yticklabels(list(range(0, 10)))\n \n plt.gcf().set_tight_layout(True)","metadata":{"execution":{"iopub.status.busy":"2022-04-15T19:45:01.75821Z","iopub.execute_input":"2022-04-15T19:45:01.75849Z","iopub.status.idle":"2022-04-15T19:45:01.781209Z","shell.execute_reply.started":"2022-04-15T19:45:01.758458Z","shell.execute_reply":"2022-04-15T19:45:01.780499Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### ***Run Manager***","metadata":{}},{"cell_type":"code","source":"class RunManager:\n def __init__(self) -> None:\n self.epoch = Epoch()\n self.run = Run()\n self.eval = Evaluate()\n \n self.device = None\n self.network = None \n self.best_network = None\n \n self.train_loader = None\n self.eval_loader = None\n \n def begin_run(self, run, network, train_loader, eval_loader, device) -> None:\n self.run.start_time = time.time()\n \n self.run.params = run\n self.run.count += 1\n \n self.device = device\n self.network = network\n self.network.canvas = MyCanvas()\n self.network.history = hl.History()\n \n self.train_loader = train_loader\n self.eval_loader = eval_loader\n \n def end_run(self) -> None:\n self.epoch.count = 0\n \n def begin_epoch(self) -> None:\n self.epoch.start_time = time.time()\n self.epoch.count += 1\n\n self.epoch.loss = 0\n self.epoch.num_correct = 0\n self.epoch.f1_score = 0\n self.epoch.roc_auc = 0\n self.epoch.cm = []\n\n self.eval.loss = 0\n self.eval.num_correct = 0 \n self.eval.f1_score = 0\n self.eval.roc_auc = 0\n self.eval.cm = []\n \n def end_epoch(self) -> None:\n epoch_duration = time.time() - self.epoch.start_time\n run_duration = time.time() - self.run.start_time\n \n loss = self.epoch.loss / len(self.train_loader.dataset)\n accuracy = self.epoch.num_correct / len(self.train_loader.dataset)\n f1 = (self.epoch.f1_score / len(self.train_loader.dataset)) * self.train_loader.batch_size\n roc_auc = (self.epoch.roc_auc / len(self.train_loader.dataset)) * self.train_loader.batch_size\n cm = np.sum(np.array(self.epoch.cm), 0)\n\n # ===== Evaluate our network ===== #\n self.evaluate(self.device)\n val_loss = self.eval.loss / len(self.eval_loader.dataset)\n val_accuracy = self.eval.num_correct / len(self.eval_loader.dataset)\n val_f1 = (self.eval.f1_score / len(self.eval_loader.dataset)) * self.train_loader.batch_size\n val_roc_auc = (self.eval.roc_auc / len(self.eval_loader.dataset)) * self.train_loader.batch_size\n val_cm = np.sum(np.array(self.eval.cm), 0)\n \n ## Check the best model\n if (val_accuracy >= self.eval.best_accuracy) and (val_f1 >= self.eval.best_f1):\n self.eval.best_accuracy = val_accuracy\n self.eval.best_f1 = val_f1\n self.epoch.best_epoch = self.epoch.count\n self.run.best_run = self.run.count\n self.best_network = self.network\n \n # ===== Adding the Statistics & Summary ===== #\n # Logging loss and accuracy\n self.network.history.log((self.epoch.count, run.batch_size),\n loss=loss,\n accuracy=accuracy,\n val_accuracy=val_accuracy,\n cm=cm,\n val_cm=val_cm,\n conv1_weight=self.network[0].weight.data.permute(0, 2, 3, 1))\n \n # ===== Tracing our training loop performace ===== #\n \n results = OrderedDict()\n results['run'] = self.run.count\n results['epoch'] = self.epoch.count\n results['loss'] = loss\n results['val_loss'] = val_loss\n results['accuracy'] = accuracy\n results['val_accuracy'] = val_accuracy\n results['f1'] = f1\n results['val_f1'] = val_f1\n results['roc'] = roc_auc\n results['val_roc'] = val_roc_auc\n results['epoch duration'] = epoch_duration\n results['run duration'] = run_duration\n \n \n # adding our hyperparameters to the results' dictionary\n for key, value in self.run.params._asdict().items(): results[key] = value\n \n if not torch.cuda.is_available():\n results['device'] = 'cpu'\n \n \n # Append all these information inside the run list to display\n self.run.data.append(results)\n \n # Display the results data using pandas dataframe\n df = pd.DataFrame.from_dict(self.run.data,\n orient='columns').style.background_gradient(cmap='ocean')\n \n # Clean the kernel every end of epoch to display the update\n clear_output(wait=True)\n # Display\n with self.network.canvas:\n self.network.canvas.draw_pie(self.network.history[\"accuracy\"],\n self.network.history[\"val_accuracy\"])\n self.network.canvas.draw_confusion_matrix(self.network.history['cm'],\n self.network.history['val_cm'])\n self.network.canvas.draw_plot([self.network.history[\"accuracy\"],\n self.network.history[\"loss\"]])\n self.network.canvas.draw_image(self.network.history[\"conv1_weight\"])\n display(df)\n \n def trace_loss(self, loss) -> None:\n self.epoch.loss += loss.item() * self.train_loader.batch_size\n\n def eval_trace_loss(self, loss) -> None:\n self.eval.loss += loss.item() * self.eval_loader.batch_size\n \n def trace_num_correct(self, pred, labels) -> None:\n self.epoch.num_correct += self._get_num_correct(pred, labels)\n\n def eval_trace_num_correct(self, pred, labels) -> None:\n self.eval.num_correct += self._get_num_correct(pred, labels)\n \n \n @torch.no_grad()\n def _get_num_correct(self, pred, labels):\n return pred.argmax(dim=1).eq(labels).sum().item()\n \n @torch.no_grad()\n def scores(self,\n pred: torch.Tensor,\n labels: torch.Tensor,\n mode: str='TRAIN') -> None:\n \n if mode == 'TRAIN':\n self.epoch.f1_score += f1_score(labels.cpu().detach().numpy(),\n torch.argmax(pred, dim=1).cpu().detach().numpy(),\n average='macro')\n \n self.epoch.roc_auc += roc_auc_score(labels.cpu().detach().numpy(),\n F.softmax(pred).cpu().detach().numpy(),\n average=\"macro\",\n multi_class='ovr')\n \n self.epoch.cm.append(confusion_matrix(labels.cpu().detach().numpy(),\n torch.argmax(pred, dim=1).cpu().detach().numpy(),\n labels=list(range(0, 10))).tolist())\n \n elif mode == \"EVAL\":\n self.eval.f1_score += f1_score(labels.cpu().detach().numpy(),\n torch.argmax(pred, dim=1).cpu().detach().numpy(),\n average='macro')\n\n self.eval.roc_auc += roc_auc_score(labels.cpu().detach().numpy(),\n F.softmax(pred).cpu().detach().numpy(),\n average=\"macro\",\n multi_class='ovr')\n self.eval.cm.append(confusion_matrix(labels.cpu().detach().numpy(),\n torch.argmax(pred, dim=1).cpu().detach().numpy(),\n labels=list(range(0, 10))).tolist())\n \n @torch.no_grad()\n def evaluate(self, device) -> None:\n self.network.eval()\n for batch in self.eval_loader:\n images = batch[0].float().to(device)\n labels = batch[1].to(device)\n \n # Calculate logits\n logits = self.network(images)\n\n # Calculate Loss\n loss = F.cross_entropy(logits, labels)\n \n # calculate the performace\n self.eval_trace_loss(loss)\n self.eval_trace_num_correct(logits, labels)\n self.scores(logits, labels, mode='EVAL')\n\n def save(self, filename: Text) -> None:\n # Save training report\n if not os.path.isdir('training_report'):\n os.mkdir('training_report')\n \n if not os.path.isdir('model'):\n os.mkdir('model')\n\n print('Saving the best model...')\n # Save Model\n torch.save(self.best_network, f'model/model_run_{self.run.best_run}_epoch_{self.epoch.best_epoch}_val_accuracy_{round(self.eval.best_accuracy, 3)}_val_f1_{round(self.eval.best_f1, 3)}.pth')\n print(f\"Model saved in model/{os.listdir('model')[0]}\")\n # save as CSV\n pd.DataFrame.from_dict(\n self.run.data, orient='columns',\n ).to_csv(f'training_report/{filename}.csv', index=False)\n \n # save as JSON\n with open(f'training_report/{filename}.json', 'w', encoding='utf-8') as f:\n json.dump(self.run.data, f, ensure_ascii=False, indent=2)","metadata":{"execution":{"iopub.status.busy":"2022-04-15T19:45:09.622907Z","iopub.execute_input":"2022-04-15T19:45:09.623283Z","iopub.status.idle":"2022-04-15T19:45:09.668353Z","shell.execute_reply.started":"2022-04-15T19:45:09.62325Z","shell.execute_reply":"2022-04-15T19:45:09.667565Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### ***Hyperparameter Initialization***","metadata":{}},{"cell_type":"code","source":"EPOCH = 8\n\nparams = OrderedDict(\n lr = [5e-3, 1e-3, 1e-4],\n batch_size=[100, 500, 1000],\n num_workers=[3],\n device=['cuda'],\n network=['with_MaxPooling', 'without_MaxPooling']\n)","metadata":{"execution":{"iopub.status.busy":"2022-04-15T19:13:46.75919Z","iopub.execute_input":"2022-04-15T19:13:46.759587Z","iopub.status.idle":"2022-04-15T19:13:46.768777Z","shell.execute_reply.started":"2022-04-15T19:13:46.759538Z","shell.execute_reply":"2022-04-15T19:13:46.768047Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# (D) Training & Evaluating\n","metadata":{}},{"cell_type":"markdown","source":"

Training & Validation Visualization

","metadata":{}},{"cell_type":"code","source":"train_dataset = torch.utils.data.TensorDataset(tensor_train_images.squeeze(0).view(-1, 1, 28, 28),\n torch.from_numpy(train_label.to_numpy()))\nvalid_dataset = torch.utils.data.TensorDataset(tensor_valid_images.squeeze(0).view(-1, 1, 28, 28),\n torch.from_numpy(valid_label.to_numpy()))\n\nlen(train_dataset), len(valid_dataset)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-04-15T19:13:46.771277Z","iopub.execute_input":"2022-04-15T19:13:46.7718Z","iopub.status.idle":"2022-04-15T19:13:46.780355Z","shell.execute_reply.started":"2022-04-15T19:13:46.771764Z","shell.execute_reply":"2022-04-15T19:13:46.779617Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"run_manager = RunManager()\nprint(' Start the training '.center(100, '='))\n\nfor index, run in enumerate(RunBuilder.get_runs(params)):\n ## +++++ [Torch GPU] (1) Detect the device +++++\n if torch.cuda.is_available():\n device= torch.device(run.device)\n ## +++++ [Multiple Training Network] Using NetworkFactory +++++\n network = NetworkFactory().get_network(run.network).to(device)\n else:\n device = torch.device('cpu')\n network = NetworkFactory().get_network(run.network).to(device)\n \n #Initialize the train & eval batches & Optimizer\n train_loader = torch.utils.data.DataLoader(train_dataset,\n batch_size=run.batch_size,\n num_workers=run.num_workers)\n \n eval_loader = torch.utils.data.DataLoader(valid_dataset,\n batch_size=run.batch_size,\n num_workers=run.num_workers)\n \n optimizer = optim.Adam(network.parameters(), lr=run.lr)\n ## ===== (1) Begin Run =====\n run_manager.begin_run(run, network, train_loader, eval_loader, device)\n \n print(f' Start the training No. [{index+1}] '.center(100, '+'))\n\n for epoch in range(EPOCH):\n run_manager.network.train()\n ## ===== (2) Begin Epoch =====\n run_manager.begin_epoch()\n \n # Looping for single epoch over total number of batches\n for batch in train_loader:\n images = batch[0].float().to(device)\n labels = batch[1].to(device)\n\n preds = network(images) # pass batch of images\n loss = F.cross_entropy(preds, labels) # calculate the loss\n\n optimizer.zero_grad() # reset the gradient accumilation\n loss.backward() # Calculate the gradient along with the network\n optimizer.step() # update the network with the updated gradients\n\n ## ===== (3) Trace the performance =====\n run_manager.trace_loss(loss)\n run_manager.trace_num_correct(preds, labels)\n run_manager.scores(preds, labels, mode='TRAIN')\n ## ===== (4) End Epoch =====\n run_manager.end_epoch()\n ## ===== (5) End Run =====\n run_manager.end_run()\n ## ===== (6) Save Results =====\nrun_manager.save('training_report')\nprint(' Finish the training '.center(125, '='))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-04-15T19:45:13.056877Z","iopub.execute_input":"2022-04-15T19:45:13.057129Z","iopub.status.idle":"2022-04-15T19:59:58.1062Z","shell.execute_reply.started":"2022-04-15T19:45:13.057099Z","shell.execute_reply":"2022-04-15T19:59:58.105417Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"***Sort By Accuracy***","metadata":{}},{"cell_type":"code","source":"pd.DataFrame.from_dict(run_manager.run.data, orient='columns').sort_values('val_accuracy', ascending=False).style.background_gradient(cmap='Blues')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-04-15T20:01:06.686234Z","iopub.execute_input":"2022-04-15T20:01:06.6865Z","iopub.status.idle":"2022-04-15T20:01:06.848748Z","shell.execute_reply.started":"2022-04-15T20:01:06.68647Z","shell.execute_reply":"2022-04-15T20:01:06.848114Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see that the network `without_MaxPooling` gave better accuracy compared with `with_MaxPooling`.","metadata":{}},{"cell_type":"markdown","source":"# (E) Batch Inference\n\n- We're going to load the inference data for predictions.","metadata":{}},{"cell_type":"code","source":"test_images = pd.read_csv('../input/digit-recognizer/test.csv')\ntest_images","metadata":{"execution":{"iopub.status.busy":"2022-04-15T20:01:14.879966Z","iopub.execute_input":"2022-04-15T20:01:14.880224Z","iopub.status.idle":"2022-04-15T20:01:16.543445Z","shell.execute_reply.started":"2022-04-15T20:01:14.880194Z","shell.execute_reply":"2022-04-15T20:01:16.542768Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- Apply Normalization & Transformation","metadata":{}},{"cell_type":"code","source":"norm_test_data = normalization(test_images)","metadata":{"execution":{"iopub.status.busy":"2022-04-15T20:01:17.116623Z","iopub.execute_input":"2022-04-15T20:01:17.117609Z","iopub.status.idle":"2022-04-15T20:01:17.753925Z","shell.execute_reply.started":"2022-04-15T20:01:17.117534Z","shell.execute_reply":"2022-04-15T20:01:17.753165Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tran_nrom_test_data = transform(test_images.to_numpy().astype(np.float64),\n norm_test_data)","metadata":{"execution":{"iopub.status.busy":"2022-04-15T20:01:19.560128Z","iopub.execute_input":"2022-04-15T20:01:19.560386Z","iopub.status.idle":"2022-04-15T20:01:19.888027Z","shell.execute_reply.started":"2022-04-15T20:01:19.560357Z","shell.execute_reply":"2022-04-15T20:01:19.887293Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- Prepare test dataset","metadata":{}},{"cell_type":"code","source":"test_dataset = torch.utils.data.TensorDataset(tran_nrom_test_data.squeeze(0).view(-1, 1, 28, 28))\ntest_loader = torch.utils.data.DataLoader(test_dataset,\n batch_size=500,\n num_workers=1)","metadata":{"execution":{"iopub.status.busy":"2022-04-15T20:01:21.717882Z","iopub.execute_input":"2022-04-15T20:01:21.718522Z","iopub.status.idle":"2022-04-15T20:01:21.723452Z","shell.execute_reply.started":"2022-04-15T20:01:21.718484Z","shell.execute_reply":"2022-04-15T20:01:21.722808Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"next(iter(test_loader))[0].shape","metadata":{"execution":{"iopub.status.busy":"2022-04-15T20:01:23.589786Z","iopub.execute_input":"2022-04-15T20:01:23.590325Z","iopub.status.idle":"2022-04-15T20:01:23.702534Z","shell.execute_reply.started":"2022-04-15T20:01:23.590281Z","shell.execute_reply":"2022-04-15T20:01:23.701786Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- Import Model","metadata":{}},{"cell_type":"code","source":"os.listdir('model')","metadata":{"execution":{"iopub.status.busy":"2022-04-15T20:01:44.206436Z","iopub.execute_input":"2022-04-15T20:01:44.207176Z","iopub.status.idle":"2022-04-15T20:01:44.213466Z","shell.execute_reply.started":"2022-04-15T20:01:44.207136Z","shell.execute_reply":"2022-04-15T20:01:44.212416Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Load model\nmodel = torch.load(f=os.path.join('model', os.listdir('model')[0]))\n# Add model into device\nmodel.to(device)\n# Check the eval structure\nmodel.eval()","metadata":{"execution":{"iopub.status.busy":"2022-04-15T20:02:05.260913Z","iopub.execute_input":"2022-04-15T20:02:05.261199Z","iopub.status.idle":"2022-04-15T20:02:05.478045Z","shell.execute_reply.started":"2022-04-15T20:02:05.261165Z","shell.execute_reply":"2022-04-15T20:02:05.477382Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"@torch.no_grad()\ndef batch_predictions(model, test_loader) -> pd.DataFrame:\n \n prediction_list = []\n \n for batch in tqdm(test_loader, desc='Batch Predictions'):\n images = batch[0].float().to(device)\n \n # predict logits: un-normalized data given by the model\n logits = model(images)\n \n # convert logits into real values predictions\n pred = torch.argmax(logits, dim=1).cpu().detach().numpy().tolist()\n \n # add the vector of predictions into a list\n prediction_list += pred\n \n # Read submission csv file to append the prediction inside\n sub_df = pd.read_csv('../input/digit-recognizer/sample_submission.csv')\n sub_df = sub_df.drop(columns='Label', axis=1)\n \n # Convert our list of predictions into pandas csv \n pred_df = pd.DataFrame(prediction_list, columns=['Label'])\n\n # Concatenate both dataset\n submission = pd.concat([sub_df, pred_df], axis=1)\n \n # save the sumbission\n submission.to_csv('submission.csv', index=False)\n \n return submission","metadata":{"execution":{"iopub.status.busy":"2022-04-15T20:17:56.168715Z","iopub.execute_input":"2022-04-15T20:17:56.169029Z","iopub.status.idle":"2022-04-15T20:17:56.180347Z","shell.execute_reply.started":"2022-04-15T20:17:56.168994Z","shell.execute_reply":"2022-04-15T20:17:56.179608Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"batch_predictions(model, test_loader)","metadata":{"execution":{"iopub.status.busy":"2022-04-15T20:17:57.760584Z","iopub.execute_input":"2022-04-15T20:17:57.761148Z","iopub.status.idle":"2022-04-15T20:17:58.518042Z","shell.execute_reply.started":"2022-04-15T20:17:57.761108Z","shell.execute_reply":"2022-04-15T20:17:58.517042Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

I would be more than happy If you share with me your thoughts in the comments section!!!

","metadata":{}},{"cell_type":"markdown","source":"---","metadata":{}},{"cell_type":"markdown","source":"# β€»References\n\n- Course: **[PyTorch - Python Deep Learning Neural Network API](https://www.youtube.com/playlist?list=PLZbbT5o_s2xrfNyHZsM6ufI0iZENK9xgG)**\n\n- [Digit Recognizer 0.998](https://www.kaggle.com/code/dkurbatovv/digit-recognizer-0-998)\n\n- [10 PyTorch Transformations you need to know!](https://www.analyticsvidhya.com/blog/2021/04/10-pytorch-transformations-you-need-to-know/)\n\n- **Mateusz Buda, Atsuto Maki2, Maciej A. Mazurowski, A systematic study of the class imbalance problem in convolutional neural networks, 4.5 Undersampling and oversampling to smaller imbalance ratio, page 14**\n\n- [25 Million Images! [0.99757] MNIST](https://www.kaggle.com/code/cdeotte/25-million-images-0-99757-mnist)\n\n- [\nHiddenLayer Training Demo - PyTorch](https://github.com/waleedka/hiddenlayer/blob/master/demos/pytorch_train.ipynb)","metadata":{}},{"cell_type":"markdown","source":"
\n________________________________\n
","metadata":{}},{"cell_type":"markdown","source":"

Thanks for reaching this level of expermenting\nthe idea of

\n

MNIST πŸ”’ – Getting with start image preprocessing & CV with PyTorch

\n

AI Engineer: Ahmed

\n

Created at: 2022-04-08
","metadata":{}}]}