{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"\n#  MNIST Classification with CNN\n## Table of Contents\n* [1. Setup](#1.)\n* [2. Loading data](#2.)\n* [3. Exploratory Data Analysis](#3.)\n\t* [3.1 Correlation Score](#3.1)\n\t* [3.2 Distribution of label](#3.2)\n\t* [3.3 Mean image for different labels](#3.3)\n* [4. Model Development](#4.)\n* [5. Model Training](#5.)\n* [6. Model Evaluation](#6.)\n\t* [6.1 Confusion Matrix](#6.1)\n\t* [6.2 Accuracy Score](#6.2)\n\t* [6.3 Classification Report](#6.3)\n* [7. Submission](#7.)\n\n<a id=\"1.\"></a>\n## 1. Setup","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport tensorflow as tf\nfrom tensorflow.keras.models import Sequential\nfrom tensorflow.keras.layers import Conv2D, MaxPooling2D, Flatten, Dense, Dropout, BatchNormalization\nimport pandas as pd\nimport os\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nfrom sklearn import metrics\nfrom sklearn.model_selection import train_test_split, KFold\n","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class Config:\n    is_kaggle_platform = os.path.exists(\"/kaggle/input\")\n    dataset_name = \"digit-recognizer\"\n    data_path = \"/kaggle/input/%s/\"%(dataset_name) if is_kaggle_platform else \"\"\n    submit_filename = \"submission.csv\"\n    image_width = 28\n    image_height = 28\n    image_channel = 1\n    num_classes = 10\n    epochs = 10\n    label_name = \"label\"\n    model_path = \"model.h5\"\nconfig = Config()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if not config.is_kaggle_platform:\n  try:\n    import kaggle\n  except:\n    !pip install kaggle\n  if not os.path.exists(\"/root/.kaggle/kaggle.json\"):\n    !echo \"{\"username\":\"{You username}\",\"key\":\"{Your APIKey}\"}\" >> /root/.kaggle/kaggle.json\n    !chmod 600 /root/.kaggle/kaggle.json\n  !kaggle competitions download -c $config.dataset_name\n  !unzip test.csv.zip\n  !unzip train.csv.zip","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"2.\"></a>\n## 2. Loading data","metadata":{}},{"cell_type":"code","source":"train_df = pd.read_csv(config.data_path + 'train.csv')\ntest_df = pd.read_csv(config.data_path + 'test.csv')\nsample_submission =  pd.read_csv(config.data_path + 'sample_submission.csv')\ntrain = np.array(train_df)\ntest = np.array(test_df)\ntrain_images = train[:, 1:].reshape((train.shape[0], config.image_width, config.image_height, config.image_channel))\ntrain_labels = train[:, 0].astype(np.uint8)\ntest_images = test.reshape((test.shape[0], config.image_width, config.image_height, config.image_channel))","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"3.\"></a>\n## 3. Exploratory Data Analysis","metadata":{}},{"cell_type":"code","source":"train_df.describe()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.info()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"3.1\"></a>\n### 3.1 Correlation Score","metadata":{}},{"cell_type":"code","source":"correlation_score = train_df.corr()\ncorrelated_features = correlation_score[\"label\"].sort_values(ascending=False).dropna()\ncorrelated_columns = list(correlated_features[correlated_features.abs() > 0.2].index)\ncorrelated_columns.remove(config.label_name)\nprint(correlated_columns)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"3.2\"></a>\n### 3.2 Distribution of label","metadata":{}},{"cell_type":"code","source":"sns.countplot(x=config.label_name, data=train_df)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"3.3\"></a>\n### 3.3 Mean image for different labels\nCalcuate mean image for different labels, they look exactly the label they belong to.","metadata":{}},{"cell_type":"code","source":"mean_images = [np.mean(train_images[train_labels == i].reshape(-1, config.image_width, config.image_height), axis=0) for i in range(10)]\nconcat_image = np.concatenate(mean_images, axis=1)\nprint(concat_image.shape)\nplt.imshow(concat_image)\nplt.show()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"4.\"></a>\n## 4. Model Development","metadata":{}},{"cell_type":"code","source":"def get_model(config):\n    inputs = tf.keras.Input((config.image_width, config.image_height, config.image_channel))\n    x = Conv2D(16, (3, 3), activation='relu')(inputs)\n    x = MaxPooling2D(2, 2)(x)\n    x = BatchNormalization()(x)\n    x = Conv2D(32, (3, 3), activation='relu')(x)\n    x = MaxPooling2D(2, 2)(x)\n    x = BatchNormalization()(x)\n    x = Conv2D(64, (3, 3), activation='relu')(x)\n    x = MaxPooling2D(2, 2)(x)\n    x = BatchNormalization()(x)\n    x = tf.keras.layers.GlobalAveragePooling2D()(x)\n    x = Dropout(0.3)(x)\n    output = Dense(config.num_classes)(x)\n    return tf.keras.Model(inputs=inputs, outputs=output)\n","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model = get_model(config)\ntf.keras.utils.plot_model(model)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"5.\"></a>\n## 5. Model Training","metadata":{}},{"cell_type":"code","source":"\nmodel_checkpoint = tf.keras.callbacks.ModelCheckpoint(\n    filepath=config.model_path,\n    save_weights_only=True,\n    monitor='val_accuracy',\n    mode='max',\n    save_best_only=True\n)\nearly_stopping = tf.keras.callbacks.EarlyStopping(monitor='val_accuracy', patience=4)\ntrain_images, val_images, train_labels, val_labels = train_test_split(train_images, train_labels)\nloss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)\nmodel.compile(\n    optimizer='adam',\n    loss=loss,\n    metrics=['accuracy']\n)\nhistory = model.fit(train_images, train_labels, epochs=config.epochs, validation_data=(val_images, val_labels), callbacks=[model_checkpoint, early_stopping], verbose=1)\npd.DataFrame(history.history).plot()\nplt.show()\nmodel.load_weights(config.model_path)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"6.\"></a>\n## 6. Model Evaluation","metadata":{}},{"cell_type":"code","source":"y_pred = np.argmax(model.predict(val_images), axis=-1)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"6.1\"></a>\n### 6.1 Confusion Matrix","metadata":{}},{"cell_type":"code","source":"import sklearn\ncm = sklearn.metrics.confusion_matrix(val_labels, y_pred)\nsns.heatmap(cm, annot=True)\nplt.show()\nprint(sklearn.metrics.classification_report(val_labels, y_pred))","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"6.2\"></a>\n### 6.2 Accuracy Score","metadata":{}},{"cell_type":"code","source":"test_accuracy = metrics.accuracy_score(val_labels, y_pred)\nprint(\"Test Accuracy: %.2f\"%(test_accuracy))","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"6.3\"></a>\n### 6.3 Classification Report","metadata":{}},{"cell_type":"code","source":"cls_report = metrics.classification_report(val_labels, y_pred)\nprint(cls_report)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"7.\"></a>\n## 7. Submission","metadata":{}},{"cell_type":"code","source":"test_labels = np.argmax(model.predict(test_images), axis=-1)\nsample_submission[\"Label\"] = test_labels.reshape(-1)\nsample_submission.to_csv(config.submit_filename, index=False)\nif not config.is_kaggle_platform:\n  !kaggle competitions submit $config.dataset_name -m \"Submission\" -f $config.submit_filename","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**If you found my notebook useful, give me an upvote.**\n\nWhat can you learn from small datasets such as MNIST, Cifar10? Checkout following notebooks:\n- [MNIST Classification with CatBoost](https://www.kaggle.com/lonnieqin/catboost-mnist-classification)\n- [MNIST Classification with TF-DF](https://www.kaggle.com/lonnieqin/tf-df-mnist-classification)\n- [MNIST Classification with Data Augmentation](https://www.kaggle.com/lonnieqin/mnist-classifier-with-data-augmentation)\n- [Fashion MNIST Classification with 3DCNN](https://www.kaggle.com/lonnieqin/fashion-mnist-classification-with-3dcnn)\n- [Fashion MNIST Classification with TensorFlow](https://www.kaggle.com/lonnieqin/fashion-mnist-classification-with-tensorflow)\n- [Cifar10 Image Classification with PyTorch](https://www.kaggle.com/lonnieqin/cifar10-image-classification-with-pytorch)\n- [MNIST classification with KerasTuner HyperModels](https://www.kaggle.com/lonnieqin/mnist-classification-with-kerastuner-hypermodels)\n- [Cifar10 Classification: Vision Transformer](https://www.kaggle.com/lonnieqin/cifar10-classification-vision-transformer)\n- [Fashion MNIST Classification: Vision Transformer](https://www.kaggle.com/lonnieqin/fashion-mnist-classification-vision-transformer)\n- [Fashion MNIST: Getting Started with Trax](https://www.kaggle.com/lonnieqin/fashion-mnist-getting-started-with-trax)\n","metadata":{}}]}