{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Classification of Bengali Handwritten Graphemes with Google AutoML"},{"metadata":{"trusted":true},"cell_type":"markdown","source":"This Notebook is intended as a starting point to use Google AutoML in image classification. Its creation is motivated by [this post](https://www.kaggle.com/c/bengaliai-cv19/discussion/122924).\n\nDisclaimer: Googel AutoML is not for free and actually costs quit a bit if you train big models. As of January 2020, new users get a $300 credit to try the google cloud platform including AutoML"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_labels=pd.read_csv('../input/bengaliai-cv19/train.csv')\ntest_labels=pd.read_csv('../input/bengaliai-cv19/test.csv')\nclass_map=pd.read_csv('../input/bengaliai-cv19/class_map.csv')\nsample_submission=pd.read_csv('../input/bengaliai-cv19/sample_submission.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class_map.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_labels.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Setting up the Google cloud"},{"metadata":{},"cell_type":"markdown","source":"For settingup the google.cloud.storage class look [here](https://cloud.google.com/storage/docs/reference/libraries#client-libraries-install-python) and for more details consult the [documentation](https://googleapis.dev/python/storage/latest/client.html). Note that in google cloud storage everything is in stored in buckets. Unlike folders in an os they are not intended to be stacked.\n\nDocumentation for google AutoMl can be found [here](https://googleapis.dev/python/automl/latest/index.html) and a detailed Tutorial (for AutoMLVision) in multiple Programming Languages can be found [here](https://cloud.google.com/vision/automl/docs/tutorial). See also [this notebook](https://www.kaggle.com/devvret/automl-tables-tutorial-notebook) containing a AutoML tabels tutorial."},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"# Set your own project id here\nPROJECT_ID = 'noble-return-265322'\nBUCKET_REGION = 'us-central1'#europe-west3 is frankfurt but other \n                             #regions than the first are not supp'd  \n\nfrom google.cloud import storage\n\nstorage_client = storage.Client(project=PROJECT_ID)\n\n# The name for the new bucket\nBUCKET_NAME = 'bengaliai'\n\n# Creates the new bucket\nbucket=storage.Bucket(storage_client,name=BUCKET_NAME)\nif not bucket.exists():\n    bucket.create(location=BUCKET_REGION)\n\nprint(\"Bucket {} created.\".format(BUCKET_NAME))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#to upload data (datapoint=blob) to a bucket\n\ndef upload_blob(bucket_name, source_file_name, destination_blob_name,printyes=False):\n    \"\"\"Uploads a file to the bucket.\"\"\"\n    # bucket_name = \"your-bucket-name\"\n    # source_file_name = \"local/path/to/file\"\n    # destination_blob_name = \"storage-object-name\"\n    \n    bucket = storage_client.get_bucket(bucket_name)\n    blob = bucket.blob(destination_blob_name)\n\n    blob.upload_from_filename(source_file_name)\n    \n    if printyes:\n        print(\n            \"File {} uploaded to {}.\".format(\n                source_file_name, destination_blob_name\n                )\n            )\n        \ndef download_to_kaggle(bucket_name,destination_directory,file_name,prefix=None):\n    \"\"\"\n    Takes the data from your GCS Bucket and puts it\n    into the working directory of your Kaggle notebook\n    \"\"\"\n    os.makedirs(destination_directory, exist_ok = True)\n    full_file_path = os.path.join(destination_directory, file_name)\n    blobs = storage_client.list_blobs(bucket_name,prefix=prefix)\n    for blob in blobs:\n        blob.download_to_filename(full_file_path)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We are now ready to import our data into the bucket. For that we first transform the data (i.e. the graphemes) into actual images. The data is given as rows in a csv each row being the concatenates pixels of a 137x236 grayscale images."},{"metadata":{"trusted":true},"cell_type":"code","source":"HEIGHT = 137\nWIDTH = 236\nN_CHANNELS=1","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"i=0 \nname=f'train_image_data_{i}.parquet'\ntrain_img = pd.read_parquet('../input/bengaliai-cv19/'+name)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_img.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_img.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Visualize few samples of current training dataset\nfrom matplotlib import pyplot as plt\n\nfig, ax = plt.subplots(nrows=3, ncols=4, figsize=(16, 8))\ncount=0\nfor row in ax:\n    for col in row:\n        col.imshow(train_img.iloc[[count]].drop(['image_id'],axis=1).to_numpy(dtype=np.float32).reshape(HEIGHT, WIDTH).astype(np.float64),cmap='binary')\n        count += 1\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We use the 'image_id' attribute as the URI for our cloud storage bucket and copy the images (saved as PNG).\n\nThe follwoing code does this for one image in the dataset. If this was succesfull you can also view the file in the google [cloud storage browser](https://console.cloud.google.com/storage/browser)."},{"metadata":{"trusted":true},"cell_type":"code","source":"from matplotlib.image import imsave\nimg=train_img.iloc[1]\nimage_path=img.image_id+'.png'\nimsave(image_path,img.drop(['image_id']).to_numpy(dtype=np.float64).reshape(HEIGHT, WIDTH).astype(np.float64))\n\nupload_blob(BUCKET_NAME,image_path,image_path,printyes=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We do this now for all the images in the training set and afterwards also for the images in the test set. In order not to clutter our workingdirectory we delete every image after completing the upload."},{"metadata":{"trusted":true},"cell_type":"code","source":"from matplotlib.image import imsave\nfrom os import remove\n\ndef upload_bengaliai_data(path):\n    train_img = pd.read_parquet(path)\n    num=0\n    shape=train_img.shape\n    for i in range(shape[0]):\n        img=train_img.iloc[i]\n        image_path=img.image_id+'.png'    \n        #create a .png out of the columns and save under theimage_id\n        imsave(image_path,img.drop(['image_id']).to_numpy(dtype=np.float64).reshape(HEIGHT, WIDTH).astype(np.float64))\n        \n        #upload to the bucket\n        if num%1000==0:\n            upload_blob(BUCKET_NAME,image_path,image_path,printyes=True)\n        else:\n            upload_blob(BUCKET_NAME,image_path,image_path)\n        \n        #delete the file in the working directory\n        remove(image_path)\n        \n        num+=1            ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Uploading takes a while so set the following to True only if you havent done this step before. However, if the command was run a second time the images will be overwritten."},{"metadata":{"trusted":true},"cell_type":"code","source":"download_data=False\nif download_data:\n    for i in range(4):\n        print(f'staring with file{i}...')\n        upload_bengaliai_data(f'/kaggle/input/bengaliai-cv19/train_image_data_{i}.parquet')\n        print(f'file {i} finished.')\n    \n    for i in range(4):\n        print(f'staring with file{i}...')\n        upload_bengaliai_data(f'/kaggle/input/bengaliai-cv19/test_image_data_{i}.parquet')\n        print(f'file {i} finished.')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Pepraring training data"},{"metadata":{},"cell_type":"markdown","source":"In order for AutoML to work we need to prepare our data according to [this guideline](https://cloud.google.com/vision/automl/docs/prepare). Therefore, we create a new csv from the train.csv and the test.csv containing the following columns for each datapoint\n\n1. 'TRAIN', 'VALIDATION' or 'TEST' determining if the datapoint belong to the training the validation or test set. If this is not set (i.e you start the line with the URI) AutoML will create a test and validation set on its own.\n2. The google Cloud storage URI\n3. A comma separated List of the labels that identify how the image is categorized "},{"metadata":{"trusted":true},"cell_type":"code","source":"import csv\n\nBUCKET_LINK='gs://'+BUCKET_NAME\n\nBUCKET_NAME='bengaliai'\nBUCKET_LINK='gs://'+BUCKET_NAME\ntrain_labels['uri']=[BUCKET_LINK+'/'+image_id+'.png' for image_id in train_labels['image_id']]\ntrain_labels['g']=['g'+str(num) for num in train_labels['grapheme_root']]\ntrain_labels['v']=['v'+str(num) for num in train_labels['vowel_diacritic']]\ntrain_labels['c']=['c'+str(num) for num in train_labels['consonant_diacritic']]\nlabels=train_labels.drop(['image_id','grapheme_root','vowel_diacritic','consonant_diacritic','grapheme'],axis=1)\nlabels.to_csv('all_data.csv',header=False,index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"all_data=pd.read_csv('all_data.csv')\nall_data.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"all_data.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We chose to modify the labels since we have two choices to classify the images\n1. MultiLabel: Multiple labels are allowed for one example.\n2. MultiClass: At most one label is allowed per example. \n\nIn our case we want to tell if and what grapheme_root,vowel_diacritic and consonant_diacritic\nare present in the image. For each of these three components there can be different values \n(168 different grapheme roots, 11 vowel diacritics and 7 consonant_diacritics). In our model \nwe will view each of these  186 symbols as a class and then predict the presence of the classes \nin the image with the 'Multilabel' classifier. \n\nThis might not be the optimal way to express the problem since in every image there can only be \none of each component and hence our model will need more examples to learn this apriori known \nfact (please correct me if I'm wrong with any of my thoughts).\n\nTo sum up our new classes are\n* g0,...,g168, for the grapheme roots\n* v0,...,v11, the vowel diacritics\n* c0,...,c7, corresponding to the consonant diacritics\n\nIt would be better to split the classification task into three part and train on each class of labels seperately."},{"metadata":{},"cell_type":"markdown","source":"## Create and populate dataset"},{"metadata":{},"cell_type":"markdown","source":"We need to create an empty dataset. (again we can consult the [AutoML Tutorial](https://cloud.google.com/vision/automl/docs/tutorial) for the following steps)."},{"metadata":{"trusted":true},"cell_type":"code","source":"#set up the AutoMl client\n\nfrom google.cloud import automl_v1beta1 as automl\n#automl_client = automl.AutoMlClient() #not working at the moment\n\nfrom google.api_core.gapic_v1.client_info import ClientInfo\nautoml_client = automl.AutoMlClient(client_info=ClientInfo())\n\ndisplay_name='bengaliai_dataset'\n\n# A resource that represents Google Cloud Platform location.\nproject_location = automl_client.location_path(PROJECT_ID, BUCKET_REGION)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"list the available datasets"},{"metadata":{"trusted":true},"cell_type":"code","source":"dataset_names=[]\nfor dataset in automl_client.list_datasets(project_location):\n    dataset_names.append(dataset.name)\n    print(dataset.name)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Choose the dataset and try to load or create a new one:"},{"metadata":{"trusted":true},"cell_type":"code","source":"new_dataset=False\ntry:\n    response = automl_client.get_dataset(name=dataset_names[0])\n    print('loading successfull.')\nexcept:\n    print('couldn\\'t get Dataset. Creating new Dataset')\n    new_dataset = True\n    #Specify the classification type\n    #Types:\n    #MultiLabel: Multiple labels are allowed for one example.\n    #MultiClass: At most one label is allowed per example.\n    metadata = automl.types.ImageClassificationDatasetMetadata(classification_type=automl.enums.ClassificationType.MULTILABEL)\n    dataset = automl.types.Dataset(display_name=display_name,image_classification_dataset_metadata=metadata)\n    response = automl_client.create_dataset(project_location, dataset)\n    \n    # Create a dataset with the dataset metadata in the region.\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Dataset name: {}\".format(response.name))\nprint(\"Dataset id: {}\".format(response.name.split(\"/\")[-1]))\nprint(\"Dataset display name: {}\".format(response.display_name))\nprint(\"Image classification dataset metadata:\")\nprint(\"\\t{}\".format(dataset.image_classification_dataset_metadata))\nprint(\"Dataset example count: {}\".format(response.example_count))\nprint(\"Dataset create time:\")\nprint(\"\\tseconds: {}\".format(response.create_time.seconds))\nprint(\"\\tnanos: {}\".format(response.create_time.nanos))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We need the dataset id of the just created empty dataset."},{"metadata":{"trusted":true},"cell_type":"code","source":"DATASET_ID=response.name.split(\"/\")[-1]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Get the full path of the dataset.=response.name\ndataset_full_id = automl_client.dataset_path(PROJECT_ID, 'us-central1', DATASET_ID)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The import_data method need the dataset.name and the uri to the 'all_data.csv' that we created and uploaded above."},{"metadata":{"trusted":true},"cell_type":"code","source":"all_data_path = 'gs://' + BUCKET_NAME + '/all_data.csv'\n\ninput_uris = all_data_path.split(\",\")\ninput_config = {\"gcs_source\": {\"input_uris\": input_uris}}\n\n\nimport_data=False #set to true if you havent imported\nif import_data:\n    response=automl_client.import_data(name=response.name,input_config=input_config)\n\n    print(\"Processing import...\")\n    # synchronous check of operation status.\n    print(\"Data imported. {}\".format(response.result()))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Training"},{"metadata":{},"cell_type":"markdown","source":"We can finally create a model and start training in AutoML Vision."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Set model name and model metadata for the image dataset.\nTRAIN_BUDGET = 1 # (specified in hours, from 1-100 (int))\nMODEL_NAME='bengaliai'","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"List the already available models for your dataset"},{"metadata":{"trusted":true},"cell_type":"code","source":"models=automl_client.list_models(project_location)\nmodel_names=[]\nfor md in models:\n    model_names.append(md.name)\n    print(md.name)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The first model in this list is the newest."},{"metadata":{"trusted":true},"cell_type":"code","source":"model=None\n\ntry:\n    model=automl_client.get_model(model_names[0])\n    print('loaded the model {}'.format(model.name))\nexcept:\n    model_params= {\n    \"display_name\": MODEL_NAME,\n    \"dataset_id\": DATASET_ID,\n    \"image_classification_model_metadata\": {\"train_budget\": TRAIN_BUDGET}\n    if TRAIN_BUDGET\n    else {},}\n    print('loading model unscucessfull.')\n    print('creating new model')\n    response=automl_client.create_model(project_location,model_params)\n    print(\"Training operation name: {}\".format(response.operation.name))\n    print(\"Training started...\")\n    \n    #wait till training is done\n    model=response.result()\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Model name: {}'.format(model.name))\nprint(print(\"Model id: {}\".format(model.name.split(\"/\")[-1])))\n\n#save the model_id for further use as with the dataset\nMODEL_FULL_ID=model.name\nMODEL_ID=model.name.split(\"/\")[-1]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Evaluation"},{"metadata":{},"cell_type":"markdown","source":"Auto ML provides some scores to evaluate the model. In the [Google Cloud Platform console](https://console.cloud.google.com/vision/dashboard) you also have the option to view model evaluation (precision recall curves etc.) in the browser."},{"metadata":{"trusted":true},"cell_type":"code","source":"print('List of model evaluations:')\nnum=0#\nfor evaluation in automl_client.list_model_evaluations(MODEL_FULL_ID, ''):\n    if num<=1:\n        #take this evaluation and show some metric within        \n        response = automl_client.get_model_evaluation(evaluation.name)\n\n        print(u'Model evaluation name: {}'.format(response.name))\n        print(u'Model annotation spec id: {}'.format(response.annotation_spec_id))\n        print('Create Time:')\n        print(u'\\tseconds: {}'.format(response.create_time.seconds))\n        print(u'\\tnanos: {}'.format(response.create_time.nanos / 1e9))\n        print(u'Evaluation example count: {}'.format(\n            response.evaluated_example_count))\n        print('Classification model evaluation metrics: {}'.format(\n            response.classification_evaluation_metrics))\n        num=num+1\n    \n    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"automl_client.list_model_evaluations(MODEL_FULL_ID, '')\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Make predictions"},{"metadata":{},"cell_type":"markdown","source":"Lastly we look at some predictions of the model (on the test set). To do that we need to deploy the model(this takes a couple of minutes)."},{"metadata":{"trusted":true},"cell_type":"code","source":"#response=automl_client.deploy_model(model.name) #uncomment if not deployed before","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#we need to set up a prediction client \nprediction_client = automl.PredictionServiceClient(client_info=ClientInfo())\n\n#read the file to be predicted\nimport io\ndef bengali_make_predict(images):\n    \"\"\"\n    Returns a prediction of the grapheme components of the images in 'images'\n    -images=pd.dataframe containing the image as rows with first column being the image_id\n    \"\"\"\n    for i in range(images.shape[0]):\n        #convert the rows to the correct size np.array\n        img=images.iloc[i].drop(['image_id']).to_numpy(dtype=np.float64).reshape(HEIGHT, WIDTH).astype(np.float64)\n        \n        #create a stream as to not save the image in the workspace and pass directly\n        imageBytearray=io.BytesIO()\n        imsave(imageBytearray,img,format='png')\n        \n        image=automl.types.Image(image_bytes=imageBytearray.getvalue())\n        payload=automl.types.ExamplePayload(image=image)\n        \n        #define some parameters of the model\n        params={'score_threshold': '0.8'}\n        \n        response=prediction_client.predict(MODEL_FULL_ID,payload, params)\n        print('Prediction results:')\n        for result in response.payload:\n            print(u'Predicted class name: {}'.format(result.display_name))\n            print(u'Predicted class score: {}'.format(result.classification.score))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_images=pd.read_parquet('../input/bengaliai-cv19/test_image_data_0.parquet')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_images.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"bengali_make_predict(test_images)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The predictions still need to be put into the right form for submission to the competition."},{"metadata":{},"cell_type":"markdown","source":"## The End"},{"metadata":{},"cell_type":"markdown","source":"This notebook was written to demonstrate Auto ML from Google. There are many things that could be done more efficiently and the model that was trained above is not very good. Keep this in mind. I still appreciate your comments especially to improve presentation."}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}