{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport os\nfrom fastai.vision.all import *\nfrom sklearn.model_selection import StratifiedKFold, StratifiedShuffleSplit\nimport albumentations as A","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Introduction\nIn this notebook, we hope to accomplish two primary things:\n1) establish a system for drawing a *representative* sample of the dataset to facilitate rapid testing, possibly in combination with reductions to the size of the images; and\n2) track experiments using [Weights and Biases](wandb.ai), particularly hyperparameters used between runs, in order to compare hyperparamter choices to accuracy across a large number of runs.\n\n## Notes and Caveats\nFor (2) to work, we need to run this as an *online* notebook. We will endeavor to make this function *offline* as well as possible using conditionals throughout the notebook whenever `wandb` commands are invoked.\n\n# Setting up Weights and Biases\n1. Make a [free wandb account](wandb.ai).\n2. Create a new project on wandb. After you create a new project, select `fast.ai` as your framework and you will be presented with instructions for getting started.\n3. import `wandb`\n4. `from fastai.callback.wandb import WandbCallback` (This allows fastai to communicate with/send metrics to `wandb`).\n5. `from kaggle_secrets import UserSecretsClient` -- this will allow us to store our `wandb` API key in a way that is invisible to any other users.\n6. Obtain your API key from the following link: https://wandb.ai/authorize. Copy it to the clipboard.\n7. Go to `Add-ons` -> `Secrets` -> `Add a new secret`. Under `Label` put `WANDB_API_KEY`. Under `Value` paste the API key you copied to your clipboard. Make sure to check `Attach to Notebook`."},{"metadata":{"trusted":true},"cell_type":"code","source":"import wandb\nfrom fastai.callback.wandb import WandbCallback\nfrom kaggle_secrets import UserSecretsClient","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"8\\. Save your secret to the environment:"},{"metadata":{"trusted":true},"cell_type":"code","source":"user_secrets = UserSecretsClient()\nsecret_value_0 = user_secrets.get_secret(\"WANDB_API_KEY\")\nos.environ[\"WANDB_API_KEY\"] = secret_value_0\nos.environ[\"WANDB_RUN_GROUP\"] = \"WANDB-EXAMPLE\" # + wandb.util.generate_id() if you want a new random id for each run","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can, in this step, also define a \"group name.\" This is useful for grouping all runs of a given notebook together in the `wandb` interface. See the cell above for details. You might consider putting this a bit later and conditially changing it whether you are doing a full run, subset, or test/debug run."},{"metadata":{},"cell_type":"markdown","source":"9\\. Confirm that this worked by running `wandb.login()`. You can comment this out afterward. If it worked, you will see a message saying `wandb: Currently logged in as: {username} (use 'wandb login --relogin' to force relogin)`. It will output `False` if unsuccessful."},{"metadata":{"trusted":true},"cell_type":"code","source":"#wandb.login() #only first time and not in commit","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"10\\. Define the hyperparameters you want to track. There are various ways to do this. In this case, we will be defining all of the hyperparameters of interest in a dictionary and passing that to the wandb config.\n    \nWe are defining two config dictionaries: one which we will use when running the 'full' dataset and one for the smaller dataset which we will use for rapid iteration"},{"metadata":{"trusted":true},"cell_type":"code","source":"TEST = True\nRAPID = True\n\nhyper_rapid = dict(\n    TEST = TEST,\n    FROZEN_EPOCHS = 1 if TEST else 5,\n    UNFROZEN_EPOCHS = 1 if TEST else 12,\n    BATCH_SIZE = 10 if TEST else 32,\n    TEST_BATCH = 128,\n    PROP_DATA = 0.15, # percentage of data to use\n    PRESIZE = 512,\n    MAX_ZOOM = 3.0\n)\n    \nhyper_full = dict(\n    TEST = TEST,\n    FROZEN_EPOCHS = 2 if TEST else 5,\n    UNFROZEN_EPOCHS = 2 if TEST else 5,\n    BATCH_SIZE = 10 if TEST else 64,\n    TEST_BATCH = 128,\n    PRESIZE = 512,\n    MAX_ZOOM = 3.0\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"11\\. initialize the weights and biases session with the name of your project (chosen earlier) using `wandb.init(project=\"project_name\")`. Note: you might want to save this step until later, e.g. inside of a cross validation loop, in some situations. But to keep the \"core\" weights and biases content together, I'm putting it at the front. Upon initializing, a new entry will app"},{"metadata":{"trusted":true},"cell_type":"code","source":"wandb.init(project=\"cassava_classification\", config = hyper_rapid if RAPID else hyper_full)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Data Setup\n\nNow `wandb` is up and running and will track many, many metrics about the model. You can view them at the project page on `wandb.ai`.\n\nOur next step is setting up the data. This will mostly follow the typical approach, except that we will optionally use `StratifiedShuffleSplit` to pull a random sample of the desired size from the dataset."},{"metadata":{"trusted":true},"cell_type":"code","source":"path = Path('../input/cassava-leaf-disease-classification')\ntrain_df = pd.read_csv(path/'train.csv')\ntrain_df['image_id'] = train_df['image_id'].apply(lambda x: f'train_images/{x}')\nif TEST and RAPID: train_df = train_df[0:900]\nif TEST and not(RAPID): train_df = train_df[0:120]\ntrain_df","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Smaller Representative Sample\n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Make a Small Sample\n# Work from here https://stackoverflow.com/questions/48425201/sample-x-examples-from-each-class-label\n\nif RAPID:\n    tmp = train_df\n    sub = {}\n    sp = StratifiedShuffleSplit(n_splits=1, test_size=1-wandb.config.PROP_DATA, random_state=97)\n    for i,(tr_idx,val_idx) in enumerate(sp.split(tmp, tmp['label'])):\n        sub[i] = [tr_idx, val_idx]\n    \ntrain_df = tmp.loc[sub[0][0]]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# DataLoaders"},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_x(row): return path/row['image_id']\ndef get_y(row): return row['label']\nlabels = train_df.to_dict()['label']\n\n\ndef get_data(labels,bs=32, presize=wandb.config.PRESIZE, max_zoom = wandb.config.MAX_ZOOM):  # removed resize=384\n    db = DataBlock(blocks=(ImageBlock, CategoryBlock),\n         get_x = get_x,\n         get_y = get_y,\n         splitter=RandomSplitter(valid_pct=0.2),\n         item_tfms=[Resize(presize)],\n         batch_tfms=[Zoom(1.1, max_zoom, draw_x=0.5, draw_y=0.5, p=1),\n                     Rotate(max_deg=360, batch=True), Normalize.from_stats(*imagenet_stats)])\n    return db.dataloaders(train_df, bs=bs,num_workers=8)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if TEST:\n  test = get_data(labels, bs=wandb.config.BATCH_SIZE)\n  test.show_batch()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if not os.path.exists('/root/.cache/torch/hub/checkpoints/'):\n        os.makedirs('/root/.cache/torch/hub/checkpoints/')\n!cp '../input/resnet18/resnet18-5c106cde.pth' '/root/.cache/torch/hub/checkpoints/resnet18-5c106cde.pth'","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Note the `WandbCallback` invoked below. For details, read documentation [here](https://docs.fast.ai/callback.wandb.html)."},{"metadata":{"trusted":true},"cell_type":"code","source":"dls = get_data(labels,bs=wandb.config.BATCH_SIZE)\nif torch.cuda.device_count() > 0:\n    learn = cnn_learner(dls, resnet18, metrics=[accuracy],\n                       cbs=[WandbCallback(log_dataset=False,\n                                          log_model=True,\n                                          n_preds = 9)]).to_fp16() #SaveModelCallback\nelse:\n    learn = cnn_learner(dls, resnet18, metrics=[accuracy],\n                        cbs=[WandbCallback(log_dataset=False,\n                                          log_model=True, n_preds = 9)]) #SaveModelCallback\nlearn.freeze()\nlearn.fit_one_cycle(wandb.config.FROZEN_EPOCHS) #, cbs=[MixUp()])\nlearn.unfreeze()\nlearn.fit_one_cycle(wandb.config.UNFROZEN_EPOCHS) #, cbs=[MixUp()])\n#learn.fine_tune(10)\n#learn.export(f'resnet18-full-fold_{split}')\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As a sample of something slightly \"fancier\" we can do with `wandb`, let's export a confusion matrix.\n\nThis was more challenging than I expected (because of `fastai` and `matplotlib`, not wandb). The `wandb` part is dead simple: `wandb.log({\"name\" : object})`."},{"metadata":{"trusted":true},"cell_type":"code","source":"interp = ClassificationInterpretation.from_learner(learn)\n\ncm = interp.confusion_matrix()\nfig = plt.figure(figsize=(10,10))\nplt.imshow(cm, interpolation='nearest', cmap=\"Blues\")\nplt.title(\"Confusion Matrix\")\nplt.colorbar()\nplt.ylabel('True label')\nplt.xlabel('Predicted label')\nthresh = cm.max() / 2\nfor i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n                plt.text(j, i, \"{:,}\".format(cm[i, j]),\n                horizontalalignment=\"center\",\n                color=\"white\" if cm[i, j] > thresh else \"black\")\n\n\nwandb.log({\"conf_mat\" : fig})\n\nwandb.finish()","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}