{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Set Parameters\n\nIn GCP, work is organized into projects.\n\n**gcs_path** is the path that images and the training csv will be uploaded to. AutoML Vision requires the images to be stored in a directory with the root directory being gs://{project_id}-vcm. \n\n**train_filename** is the name of the csv file that's uploaded to GCS. Doesn't matter what you choose here. \n\n**gcp_service_account_json** is the path to the service account key. Service accounts allow you to authenticate with GCP using a JSON key (rather than typing in a password). I uploaded my service account key as a private dataset. Read more about setting one up at [https://cloud.google.com/iam/docs/understanding-service-accounts](https://cloud.google.com/iam/docs/understanding-service-accounts)\n\n**train_budget** is the number of node hours. 1 is min. 24 is maximum. I believe AutoML Vision Classifaction costs $20 per node hour.\n\n**dataset_name** is the name of the dataset that's loaded into AutoML. Doesn't matter what you choose here. \n\n**model_name** is the name of the model in AutoML. Doesn't matter what you choose here. I use the convention of dataset_name *underscore* train_budget\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"gcp_project_id = 'kaggle-playground-170215'\ngcs_path = \"gs://{}-vcm/recursion-cellular-image-classification/RGB224/\".format(gcp_project_id)\ntrain_filename = \"automl_train.csv\"\ngcp_service_account_json = '/kaggle/input/gcloudserviceaccountkey/kaggle-playground-170215-4ece6a076f22.json'\ngcp_compute_region = 'us-central1' #for now, AutoML is only available in this region\ntrain_budget = 24\ndataset_name = 'recursion_224px_wo_controls'\nmodel_name = \"{}_{}\".format(dataset_name,train_budget) \n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Install Google Cloud SDK and AutoML Package\nFollowed the instructions at [https://cloud.google.com/sdk/install](https://cloud.google.com/sdk/install) with some slight modifications for this environment. Need the Google Cloud SDK to use gsutil, which is the fast way to transfer the training data to Google Cloud Storage (GCS). \n\nAlso need to install the AutoML Python Package. "},{"metadata":{"trusted":true},"cell_type":"code","source":"#google cloud SDK\n!echo \"deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main\" | tee -a /etc/apt/sources.list.d/google-cloud-sdk.list\n!apt-get install apt-transport-https ca-certificates\n!curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key --keyring /usr/share/keyrings/cloud.google.gpg add -\n!apt-get update && apt-get install --yes --allow-unauthenticated google-cloud-sdk\n\n#AutoML package\n!pip install google-cloud-automl\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#authenticate the Google Cloud SDK\n!gcloud config set project $gcp_project_id\n!gcloud auth activate-service-account --key-file=$gcp_service_account_json\n\n#uncomment if you don't already have this gcs bucket setup\n#!gsutil mb -p $gcp_project_id -c regional -l $gcp_compute_region gs://$gcp_project_id-vcm/\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Import libraries\nImporting libraries after all packages have been installed"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport zipfile\nimport os\nfrom google.cloud import automl_v1beta1 as automl\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Create the training CSV and upload to Google Cloud Storage (GCS)\nHere are the docs for what your data should look like for AutoML vision"},{"metadata":{},"cell_type":"markdown","source":"## Split data by experiment\nBy default, splitting betweeen train, validation and test is optional for AutoML (by default it splits randomly). But it's important for this use case. "},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train = pd.read_csv('../input/recursion-2019-load-resize-and-save-images/new_train.csv')\n\nvalidation = ['HEPG2-07','HUVEC-16','RPE-07','U2OS-02']\ntest = ['HEPG2-03','HUVEC-04','RPE-05','U2OS-01']\n\n\ndf_train['split'] = 'TRAIN' \n#put experiments in the lists aboe in Validation and TEST\ndf_train.loc[df_train['experiment'].isin(validation),'split'] = 'VALIDATION' \ndf_train.loc[df_train['experiment'].isin(test),'split'] = 'TEST'\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Upload training CSV to GCS"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"\n#add gcs path\ndf_train['gcspath'] = gcs_path + 'train/' + df_train['filename']\n\n#AutoML requires the label to be an int not a float\ndf_train['sirna'] = df_train['sirna'].astype(int)\n\ndf_train[['split','gcspath','sirna']].to_csv(train_filename,index=False,header=False)\n!gsutil cp $train_filename $gcs_path$train_filename #upload csv file to GCS\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Extract images and upload to GCS"},{"metadata":{"trusted":true},"cell_type":"code","source":"with zipfile.ZipFile('../input/recursion-2019-load-resize-and-save-images/train.zip', 'r') as zip_ref:\n    zip_ref.extractall('./train/')\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!gsutil -q -m cp -r ./train/* $gcs_path/ #upload images to gcs\n!rm -r ./train/ #need to do this because otherwise you get a \"too many output files error\"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Kick off an AutoML training job\n\nRequires a three step process:\n1. Setup an AutoML data object\n2. Load data into the object\n3. Train the model\n\nThis is mostly boilerplate copied from:\n[https://cloud.google.com/vision/automl/docs/tutorial](http://https://cloud.google.com/vision/automl/docs/tutorial)"},{"metadata":{"trusted":true},"cell_type":"code","source":"#1. Setup AutoML Data Object\nclient = automl.AutoMlClient.from_service_account_json(gcp_service_account_json)\nproject_location = client.location_path(gcp_project_id, gcp_compute_region)\n\nmy_dataset = {\n    \"display_name\": dataset_name,\n    \"image_classification_dataset_metadata\": {\"classification_type\": \"MULTICLASS\"},\n}\n\n# Create a dataset with the dataset metadata in the region\ndataset = client.create_dataset(project_location, my_dataset)\ndataset_id = (dataset.name.split(\"/\")[-1])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#2 Load data into the object\ndataset_full_id = client.dataset_path(\n    gcp_project_id, gcp_compute_region, dataset_id\n)\n\ninput_uris = ('{}{}'.format(gcs_path ,train_filename)).split(\",\")\ninput_config = {\"gcs_source\": {\"input_uris\": input_uris}}\n\nresponse = client.import_data(dataset_full_id, input_config)\n\nprint(\"Processing import...\")\nprint(\"Data imported. {}\".format(response.result()))\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#3. Train the model\nmy_model = {\n    \"display_name\": model_name,\n    \"dataset_id\": dataset_id,\n    \"image_classification_model_metadata\": {\"train_budget\": train_budget}\n    if train_budget\n    else {},\n}\n\nresponse = client.create_model(project_location, my_model)\n\nprint(\"Training operation name: {}\".format(response.operation.name))\nprint(\"Training started...\")","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}