{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Final Project\n## Author: Shaurya Chandhoke\n### Description: Based on a collection of over 200k images provided from Metropolitan Museum of Art in New York, this notebook builds a multilabel image classification model in R and labels images based on their country, culture, tags, dimensions, and other labels defined by _labels.csv_"},{"metadata":{},"cell_type":"markdown","source":"**NOTE: This notebook was built in Kaggle as it provided the images and datasets for this project. Due to this, some paths may need to be reconfigured to and images may need to be downloaded if you want to run the project locally**\n\n**NOTE: This notebook was programmed and ran using Kaggle services. Recreating the performance of this notebook requires similar computer specs to Kaggle service. The Kaggle services provided gave me 4.9Gb of disk space, 16Gb of RAM, and CPU often spiked to 100% usage. Depending on the specs of machine that will run this notebook, performance might be better or worse. Entire notebook finished in Kaggle at around roughly 10-25 min depending on if I used GPU services.**\n\n**NOTE: You may see warnings like `Warning message in grayscale(.): “Image appears to already be in grayscale mode”` I don't know how to suppress these warning as it comes from the `imager` library and there's no way to disable this. As a result, I've lowered the sample collection sizes to reduce the number of times the warning pops up**"},{"metadata":{},"cell_type":"markdown","source":"## Prerequisites\n"},{"metadata":{},"cell_type":"markdown","source":"### Package Installation\nIf running on local machine, some packages may need to be installed manually using `install.packages(<package>)`"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Keras is used to build the project in R, which has dependencies that are installed first. For ease of use, a virtualenv is created\nlibrary(reticulate)\nreticulate::virtualenv_install(\"pandas\", envname = \"r-reticulate\")\nlibrary(keras)\ninstall_keras()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"library(tidyverse) # Meta package that configures a set of other packages. \nlibrary(stringr)\nlibrary(imager)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Path Configuration"},{"metadata":{"trusted":true},"cell_type":"code","source":"# These are paths to the Kaggle images. Feel free to change these paths to the correct locations\nINPUT_PATH = \"../input/imet-2020-fgvc7/\"\nOUTPUT_PATH = \"/kaggle/working/\"\n\nTRAIN_IMG_DIR = paste(INPUT_PATH, \"train/\", sep=\"\")\nTEST_IMG_DIR = paste(INPUT_PATH, \"test/\", sep=\"\")\n\nTRAINCSV = paste(INPUT_PATH, \"train.csv\", sep=\"\")\nLABELSCSV = paste(INPUT_PATH, \"labels.csv\", sep=\"\")\nOUTPUT_SUBMISSION_FILENAME = paste(OUTPUT_PATH, \"submission.csv\", sep=\"\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Data Collection and Sample Size Configuration"},{"metadata":{"trusted":true},"cell_type":"code","source":"labelsFrame = read.csv(LABELSCSV, header=TRUE)\n\nfullTrainingFrame = read.csv(TRAINCSV, header=TRUE, stringsAsFactors=FALSE)\nfullTrainingFrame$id = paste(TRAIN_IMG_DIR, fullTrainingFrame$id, \".png\", sep=\"\")\nfullTrainingFrame$attribute_ids = strsplit(fullTrainingFrame$attribute_ids, \" \")\n\nTOTALLABELS = nrow(labelsFrame)\n\n# Adjust for how many samples to train/test/evaluate\nTRAIN_SAMPLING=1000        # How many rows in split training data frame\nTEST_SAMPLING=1000         # How many rows in split testing data frame\nTESTING_SAMPLING=1000      # How many rows of testing images data frame\nIMGSIZE=100                # How big in pixels should image be (makes square image)\nACCEPTANCE_PERCENTAGE=0.15 # The percentage threshold that will accept model's prediction for each label\nEPOCHS=30                  # How many epochs the model should run for\nSTEPS_PER_EPOCH=10         # How many steps should be in each epoch","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"set.seed(1)\ntrainingFrame = sample_n(fullTrainingFrame, TRAIN_SAMPLING)\ntestingFrame = sample_n(fullTrainingFrame, TEST_SAMPLING)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"head(trainingFrame)\nhead(testingFrame)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Helper Functions\nThese function will assist in some tedious procedures to be done throughout the notebook"},{"metadata":{"trusted":true},"cell_type":"code","source":"#'@param file: Relative path of image filename'\n#'@return img: A grayscaled and resized image converted into a 1D vector'\nIMG_AUGMENT = function(file){\n    img = load.image(file)\n    img = img %>% grayscale() %>% resize(IMGSIZE, IMGSIZE) %>% as.matrix(byrow=TRUE) %>% as.vector()\n    \n    return(img)\n}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#'@param classes: A list of classes the file is associated with'\n#'@return classVector: A 1D hot encoded binary vector where each class has a 1 at its location(index + 1). \n#'                     Ex. If class id = 64, 1D vector will have 1 at index location 65'\nENCODING = function(classes){\n    classList = as.integer(unlist(classes))\n    classVector = rep(0, TOTALLABELS)\n    \n    for(class in classList){\n        classVector[class + 1] = 1\n    }    \n    return(classVector)\n}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#'@param prediction: Matrix row of predictions for one image'\n#'@return: Vector of best predictions based on ACCEPTANCE_PERCENTAGE'\nRETRIEVE_PREDICTIONS = function(prediction){\n    prediction = as.vector(prediction)\n    sortedPredictions = sort(prediction, decreasing=TRUE)\n    bestPredictions = sortedPredictions[sortedPredictions > ACCEPTANCE_PERCENTAGE]\n    predictedLabels = match(bestPredictions, prediction) - 1\n    return(predictedLabels)    \n}","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Data Cleaning and Augmenting to Support Model Params"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Convert each image into consistent image size and color channel (grayscale)\ntrain_X = lapply(trainingFrame$id, IMG_AUGMENT)\ntest_X = lapply(testingFrame$id, IMG_AUGMENT)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Convert train_X into array of images\ntrain_X = unlist(train_X, use.names=FALSE)\ntrain_X = as.array(train_X)\ntrain_X = array_reshape(train_X, c(TRAIN_SAMPLING, IMGSIZE, IMGSIZE, 1))\n\n# Convert test_X into array of images\ntest_X = unlist(test_X, use.names = FALSE)\ntest_X = as.array(test_X)\ntest_X = array_reshape(test_X, c(TEST_SAMPLING, IMGSIZE, IMGSIZE, 1))\n\n\n# To show that the images still exist but are simply in an array collection, plot sample images\npar(mfrow=c(2,2))\nplot(as.cimg(t(train_X[1,,,])))\nplot(as.cimg(t(train_X[2,,,])))\nplot(as.cimg(t(train_X[3,,,])))\nplot(as.cimg(t(train_X[4,,,])))\n\npar(mfrow=c(2,2))\nplot(as.cimg(t(test_X[1,,,])))\nplot(as.cimg(t(test_X[2,,,])))\nplot(as.cimg(t(test_X[3,,,])))\nplot(as.cimg(t(test_X[4,,,])))\n\n\n# Convert array of images into special keras_array compatible with keras model\ntrain_X = keras_array(train_X)\ntrain_X$shape\n\ntest_X = keras_array(test_X)\ntest_X$shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Create matrix of 1D hot encoded classes. Each row of matrix corresponds to an image\ntrain_y = lapply(trainingFrame$attribute_ids, ENCODING)\ntrain_y = unlist(train_y)\ntrain_y = matrix(train_y, ncol=TOTALLABELS, nrow=TRAIN_SAMPLING, byrow=TRUE)\n\n# Create matrix of 1D hot encoded classes. Each row of matrix corresponds to an image\ntest_y = lapply(testingFrame$attribute_ids, ENCODING)\ntest_y = unlist(test_y)\ntest_y = matrix(test_y, ncol=TOTALLABELS, nrow=TEST_SAMPLING, byrow=TRUE)\n\n\n\n# Convert matrix of 1D hot encoded classes into special keras_array compatible with keras model \ntrain_y = keras_array(train_y)\ntrain_y$shape\n\n# Convert matrix of 1D hot encoded classes into special keras_array compatible with keras model\ntest_y = keras_array(test_y)\ntest_y$shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Model Creation\n### The model created does the following\n1. Repeat 4 times:\n\n\n* Model will take as input keras_array of dimension `keras_array$shape` (shown above for `train_X` and `test_X`)\n* Model will take an array of image and perform dot product calculation of square matrix kernel size (3x3)\n* Model will normalize matrix of dot product and repeat dot product of square matrix kernel size (2x2)\n* Model will drop values below either one of `c(0.3, 0.4, 0.5)` and flatten matrix into 1D vector and pass into next hidden layer\n\n\n2. Model will flatten final output layer into 1D vector of size `TOTALLABELS` that are mutually exclusive to give independent probabilities of each label"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Creating sequential model\nmodel = keras_model_sequential()\nmodel %>%\nlayer_conv_2d(16, c(3,3), activation='relu', input_shape=train_X[1]$shape) %>%\nlayer_batch_normalization() %>%\nlayer_max_pooling_2d(pool_size=c(2,2)) %>%\nlayer_dropout(0.3) %>%\nlayer_conv_2d(32, c(3,3), activation='relu') %>%\nlayer_batch_normalization() %>%\nlayer_max_pooling_2d(pool_size=c(2,2)) %>%\nlayer_dropout(0.3) %>%\nlayer_conv_2d(64, c(3,3), activation='relu') %>%\nlayer_batch_normalization() %>%\nlayer_max_pooling_2d(pool_size=c(2,2)) %>%\nlayer_dropout(0.4) %>%\nlayer_conv_2d(128, c(3,3), activation='relu') %>%\nlayer_batch_normalization() %>%\nlayer_max_pooling_2d(pool_size=c(2,2)) %>%\nlayer_dropout(0.5) %>%\nlayer_flatten() %>%\nlayer_dense(128, activation='relu') %>%\nlayer_batch_normalization() %>%\nlayer_dropout(0.5) %>%\nlayer_dense(128, activation='relu') %>%\nlayer_batch_normalization() %>%\nlayer_dropout(0.5) %>%\nlayer_dense(TOTALLABELS, activation='sigmoid')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"summary(model)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Compile the model to be used. \n#### Optimizer `adam` was found to be most popular.\n#### Because train_y is binary vector of 1's or 0's depending on classes, `binary_crossentropy` is used\n#### Model is tested for `accuracy` metric"},{"metadata":{"trusted":true},"cell_type":"code","source":"model %>% compile(optimizer='adam', loss='binary_crossentropy', metrics=c('accuracy'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Fit Model and Plot Accuracy and Loss Graph\nFeel free to adjust epochs and epoch steps if model fitting turns out to be too slow. Kaggle provided with 30 hrs of free GPU usage which fit model in under 10 min with defined epochs and epoch steps. Without GPU, this model took around 15-20 min to train."},{"metadata":{"trusted":true},"cell_type":"code","source":"history = model %>% fit(train_X, train_y, epochs=EPOCHS, steps_per_epoch=STEPS_PER_EPOCH, validation_data=c(test_X, test_y), verbose=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot(history)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model %>% evaluate(test_X, test_y)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Analyzing Model Against Sample Image"},{"metadata":{"trusted":true},"cell_type":"code","source":"SAMPLEINDEX = 151\nfile = testingFrame[SAMPLEINDEX,1]\nplot(load.image(file))\nimg = IMG_AUGMENT(file)\nimg = unlist(img, use.names=FALSE)\nimg = as.array(img)\nimg = array_reshape(img, c(1, IMGSIZE, IMGSIZE, 1))\nimg = keras_array(img)\n\nfileLabel = lapply(testingFrame[SAMPLEINDEX,2], ENCODING)\nfileLabel = unlist(fileLabel)\nfileLabelIndices = which(fileLabel == 1) - 1\nlabelsFrame[fileLabelIndices, ] \n\npredictions = as.vector(model %>% predict(img))\nmaxPredictions = sort(predictions, decreasing=TRUE)\nmaxPredictions = maxPredictions[maxPredictions > ACCEPTANCE_PERCENTAGE]\nindices = match(maxPredictions, predictions) - 1\nlabelsFrame[indices, ]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Running Model Against Test Images"},{"metadata":{},"cell_type":"markdown","source":"### Data Collection"},{"metadata":{"trusted":true},"cell_type":"code","source":"set.seed(2)\ntestingImages = list.files(TEST_IMG_DIR)\ntestingImages = data.frame('id'=unlist(testingImages), stringsAsFactors=FALSE)\ntestingImages$id = paste(TEST_IMG_DIR, testingImages$id, sep=\"\")\ntestingImages = sample_n(testingImages, TESTING_SAMPLING)\nhead(testingImages)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Data Cleaning and Augmenting to Support Model"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Convert each image into consistent image size and color channel (grayscale)\nimages_X = lapply(testingImages$id, IMG_AUGMENT)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Convert images_X into array of images\nimages_X = unlist(images_X, use.names=FALSE)\nimages_X = as.array(images_X)\nimages_X = array_reshape(images_X, c(TESTING_SAMPLING, IMGSIZE, IMGSIZE, 1))\n\n# To show that the images still exist but are simply in an array collection, plot sample images\npar(mfrow=c(2,2))\nplot(as.cimg(t(images_X[1,,,])))\nplot(as.cimg(t(images_X[2,,,])))\nplot(as.cimg(t(images_X[3,,,])))\nplot(as.cimg(t(images_X[4,,,])))\n\n# Convert array of images into special keras_array compatible with keras model\nimages_X = keras_array(images_X)\nimages_X$shape\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Writing Predictions to CSV"},{"metadata":{"trusted":true},"cell_type":"code","source":"predictions = model %>% predict(images_X)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"predictionList = apply(predictions, 1, RETRIEVE_PREDICTIONS)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submissionFrame = data.frame('id'=testingImages$id, 'attribute_ids'=as.character((predictionList)), stringsAsFactors=FALSE)\nsubmissionFrame$id = str_replace_all(submissionFrame$id, TEST_IMG_DIR, \"\")\nsubmissionFrame$id = str_replace_all(submissionFrame$id, \".png\", \"\")\n\nsubmissionFrame$attribute_ids = str_replace_all(submissionFrame$attribute_ids,\"c[(]\", \"\")\nsubmissionFrame$attribute_ids = str_replace_all(submissionFrame$attribute_ids,\"[)]\", \"\")\nsubmissionFrame$attribute_ids = str_replace_all(submissionFrame$attribute_ids,\",\", \"\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"head(submissionFrame)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"write.csv(submissionFrame, OUTPUT_SUBMISSION_FILENAME, row.names=FALSE)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# References\n- “R Interface to 'Keras'.” R Interface to 'Keras' • Keras, keras.rstudio.com/index.html [Accessed 2 May 2020].\n- Verma, Shiva. “Multi-Label Image Classification with Neural Network: Keras.” Medium, Towards Data Science, 23 Apr. 2020, towardsdatascience.com/multi-label-image-classification-with-neural-network-keras-ddc1ab1afede. [Accessed 29 April 2020].\n- J, Vijayabhaskar. “Multi-Label Image Classification Tutorial with Keras ImageDataGenerator.” Medium, Medium, 1 Mar. 2020, medium.com/@vijayabhaskar96/multi-label-image-classification-tutorial-with-keras-imagedatagenerator-cd541f8eaf24. [Accessed 20 April 2020].\n- Python, B., 2020. Build Your First Multi-Label Image Classification Model In Python. [online] Analytics Vidhya. Available at: <https://www.analyticsvidhya.com/blog/2019/04/build-first-multi-label-image-classification-model-python/> [Accessed 6 May 2020].\n- Brownlee, J., 2020. How To Develop A Deep CNN For Multi-Label Classification Of Photos. [online] Machine Learning Mastery. Available at: <https://machinelearningmastery.com/how-to-develop-a-convolutional-neural-network-to-classify-satellite-photos-of-the-amazon-rainforest/> [Accessed 6 May 2020].\n- knowledge Transfer. 2020. Multi-Label Image Classification Model In Keras - Knowledge Transfer. [online] Available at: <https://androidkt.com/multi-label-image-classification-model-in-keras/> [Accessed 6 May 2020].\n"}],"metadata":{"kernelspec":{"display_name":"R","language":"R","name":"ir"},"language_info":{"mimetype":"text/x-r-source","name":"R","pygments_lexer":"r","version":"3.4.2","file_extension":".r","codemirror_mode":"r"}},"nbformat":4,"nbformat_minor":4}