{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":11848,"databundleVersionId":862157,"sourceType":"competition"}],"dockerImageVersionId":30636,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Histopathologic Cancer Detection\n\nThis submission, woefully late for the actual competition, is to satisfy Week 3 coursework for the Introduction to Deep Learning class, a portion of the Data Science Masters Degree at University of Colorado, Boulder.\n\nIntention, I will build CNN classifiers from scratch for image detection of cancer cells. The competition scoring is based on finding a single cancerous cell in the centermost 32x32 square of each image. This sublte little point is what made this competition very difficult. Since we are not competing, these classifiers will score the whole image as malignant or benign. This will result in lower scores but easier to train (converge) CNN models. That these classifiers are built to this altered specification (different than the competition) is a crucial subpoint...\n\nThis notebook will include several iterations of CNNs to try things:\n1. The first CNN will be basic, and try a myriad of possibilities until I have a working classifier.\n2. The second will be more complex than the first, and iterate on the success.\n3. The third will be even more complex than the first two, and iterate once more.\n4. The 4th through 9th models scored are transfer learning examples using Densenet 121, 169, & 201. \n\nThe goal here is to iterate on what works three times to try new things, illustrate a transfer learning implementation, and then finally an ensemble implementation. Once we have the first one working, I'll try different activation functions, adding and subtracting layers, and dialing in the hyperparameters for the models.\n\nI will use the kaggle dataset located here: (https://www.kaggle.com/competitions/histopathologic-cancer-detection/data). This dataset has been curated and differs from the original PCAM dataset because duplicate images were remove. Kaggle is hosting this curated dataset for the machine learning community to use for fun and practice. This dataset was provided by Bas Veeling, with additional input from Babak Ehteshami Bejnordi, Geert Litjens, and Jeroen van der Laak. You may view and download the official Pcam dataset from GitHub (https://github.com/basveeling/pcam). The data is provided under the CC0 License, following the license of Camelyon16. \n\n### TLDR Results","metadata":{}},{"cell_type":"code","source":"import pandas as pd\n#TLDR Results\nimport pandas as pd\nfin = pd.DataFrame({\"Model\":['DenseNet 201 (2 training epochs)','DenseNet 169 (12 training epochs)','DenseNet 121 (12 training epochs)','DenseNet 201 (12 training epochs)','Densenet169 (2 training epochs)','Densenet 121 (2 training Epochs)','Model 2 CNN','Model 3 CNN'],\n\"Private Score\":[0.815,0.8199,0.815,0.7865,0.8133,0.817,0.773,0.7477],\n'Public Score':[0.8396,0.8452,0.8425,0.8307,0.8253,0.8325,0.8336,0.801]})\nfin.sort_values(by='Private Score',ascending=False)","metadata":{"execution":{"iopub.status.busy":"2024-01-14T15:03:51.120915Z","iopub.execute_input":"2024-01-14T15:03:51.121261Z","iopub.status.idle":"2024-01-14T15:03:51.136219Z","shell.execute_reply.started":"2024-01-14T15:03:51.121237Z","shell.execute_reply":"2024-01-14T15:03:51.135205Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Work to do.\n1. EDA\n2. Instantiate GPU/TPU Strategy\n3. Data Preprocessing\n4. Build Appropriate Model(s)\n5. Train and Test Models \n6. Iterate \n7. Transfer Learning Example\n8. Submit Each Iteration for Scoring \n9. Ensemble Results (second notebook here: https://www.kaggle.com/toddgardiner/binary-cancer-classifier-ensembler/ )\n10. Conclusion\n\nLet's begin by loading the libraries we'll need...","metadata":{}},{"cell_type":"code","source":"import numpy as np \nimport pandas as pd \nfrom PIL import Image\nimport matplotlib.pyplot as plt\nimport tensorflow as tf \nimport tensorflow_io as tfio\nimport keras\nfrom keras.models import Sequential\nfrom keras.layers import AvgPool2D,BatchNormalization, Conv2D, Dense, Flatten, Input, GlobalAveragePooling2D, Dropout \nfrom keras.layers import MaxPool2D, MaxPooling2D, ReLU, concatenate\nimport math, gc, copy\nAUTOTUNE = tf.data.AUTOTUNE\nimport warnings\nwarnings.filterwarnings('ignore')\nimport os\n# for dirname, _, filenames in os.walk('/kaggle/input'):\n#     for filename in filenames:\n#         print(os.path.join(dirname, filename))\nprint(\"Tensorflow Version In Use: \", tf.__version__ , \" \\nNotebook was built using tf version 2.13.0\")","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2024-01-14T10:13:00.806066Z","iopub.execute_input":"2024-01-14T10:13:00.806970Z","iopub.status.idle":"2024-01-14T10:13:13.352913Z","shell.execute_reply.started":"2024-01-14T10:13:00.806933Z","shell.execute_reply":"2024-01-14T10:13:13.351836Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Next, we detect our hardware and light up GPUs or TPUs if we have them.","metadata":{}},{"cell_type":"code","source":"# Detect hardware and light up the GPUs/TPUs\ntry:\n     # detect and init the TPU\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n\n    # instantiate a distribution strategy\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    tpu_strategy = tf.distribute.TPUStrategy(tpu)\n \n    # tell us what happened\n    print('Running on TPU ', tpu.cluster_spec().as_dict())\n\nexcept ValueError: # If TPU not found\n    tpu = None\n    tpu_strategy = tf.distribute.get_strategy() # Default strategy that works on CPU and single GPU\n    print('Running on CPU instead')\n\nprint(\"Number of accelerators: \", tpu_strategy.num_replicas_in_sync)\nprint(\"TPU: \", tpu)\n","metadata":{"execution":{"iopub.status.busy":"2024-01-14T10:13:13.354779Z","iopub.execute_input":"2024-01-14T10:13:13.355303Z","iopub.status.idle":"2024-01-14T10:13:13.366078Z","shell.execute_reply.started":"2024-01-14T10:13:13.355275Z","shell.execute_reply":"2024-01-14T10:13:13.364375Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# EDA: Exploratory Data Analysis\n\nWith our libraries and hardware set up, we begin the exploratory data analysis or EDA. This process will involve several steps including.\n1. Load the data\n2. Characterize the data\n3. Visualize the data\n4. Develop (Initial) Strategy for Modeling\n","metadata":{}},{"cell_type":"code","source":"# Load the data into a dataframe (df)\n\ndf = pd.read_csv('/kaggle/input/histopathologic-cancer-detection/train_labels.csv')\n\n# get basic df information\nprint(df.info())\nprint('')\n\n# get basic df statistical information\nprint(df.describe())\nprint('')\n\n#see the raw data\ndf.head()\n","metadata":{"execution":{"iopub.status.busy":"2024-01-14T10:13:13.367333Z","iopub.execute_input":"2024-01-14T10:13:13.367917Z","iopub.status.idle":"2024-01-14T10:13:13.945852Z","shell.execute_reply.started":"2024-01-14T10:13:13.367874Z","shell.execute_reply":"2024-01-14T10:13:13.944827Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"From those lines of code we know that there are 220 thousand rows of data, 40% of the observations have a cancer, the labels in the output are binary (1,0), there are no nulls values in either column, and we can be fairly certain that this maintained and curated dataset needs no cleaning. \n\nThere is only one thing left to check, are there 220,025 images in the training folder to match? Let's load them into a list and compare the lengths to find out.","metadata":{}},{"cell_type":"code","source":"imagelist = os.listdir('/kaggle/input/histopathologic-cancer-detection/train')\nprint(\"Length of Image List:\", len(imagelist))","metadata":{"execution":{"iopub.status.busy":"2024-01-14T10:13:13.948012Z","iopub.execute_input":"2024-01-14T10:13:13.948302Z","iopub.status.idle":"2024-01-14T10:13:20.819260Z","shell.execute_reply.started":"2024-01-14T10:13:13.948277Z","shell.execute_reply":"2024-01-14T10:13:20.818346Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The number of rows in the csv match the number of images in the training folder. Preliminarily, the training data looks good. Let's move on to the test data folder and repeat the process. Since there is no training.csv, we will see if submission.csv matches the image list first.","metadata":{}},{"cell_type":"code","source":"# Load the data into a dataframe (dfva from testing)\n\ndfva = pd.read_csv('/kaggle/input/histopathologic-cancer-detection/sample_submission.csv')\n\n# get basic df information\nprint(dfva.info())\nprint('')\n\n# get basic df statistical information\nprint(dfva.describe())\nprint('')\n\n# get the length of files in the testing folder\nimagevalist = os.listdir('/kaggle/input/histopathologic-cancer-detection/test/')\nprint(\"Length of Validation Image List:\", len(imagevalist))\n\n#see the raw data\ndfva.head()","metadata":{"execution":{"iopub.status.busy":"2024-01-14T10:13:20.820682Z","iopub.execute_input":"2024-01-14T10:13:20.820952Z","iopub.status.idle":"2024-01-14T10:13:22.818517Z","shell.execute_reply.started":"2024-01-14T10:13:20.820928Z","shell.execute_reply":"2024-01-14T10:13:22.817573Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This data is indeed a validation set. The data has no labels, but otherwise conforms to a clean set of data (no nulls, images match csv rows, etc.). This data will serve as the validation set in our model, scored after the fact for testing. \n\nWhat we learned:\n1. We will need to do a train/test split on the training data to build the model effectively.\n2. We have a lot of data to work with (220,000+ observations).\n3. The dataset isn't perfectly balanced (60/40) but it's not bad.\n4. There are only two classes (0,1) so we are building a binary classifier.\n5. The data in csv doesn't contain path information to the files (we'll have to add that).\n\nWhat we still need to know:\n1. What do the images look like? (format, channel construction, size)\n2. Are the images uniform?\n3. What are the implications on RAM, CPU, and GPU/TPU moving forward?\n\nLet's look at some images and find out...","metadata":{}},{"cell_type":"code","source":"fig, axs = plt.subplots(3,3) \nh = 0\nv = 0\nfor i in range(9):\n    imid = df.id.sample(1).values[0]\n    #print(imid)\n    image = Image.open('/kaggle/input/histopathologic-cancer-detection/train/'+imid+'.tif')\n    axs[h,v].imshow(image)\n    if h == 2:\n        v +=1\n        h = 0\n    else:\n        h +=1\nprint(\"Last Image Specifications: Shape\",image.size,\"\\nFormat:\",image.info )\nprint(os.stat('/kaggle/input/histopathologic-cancer-detection/train/'+imid+'.tif').st_size, \"bytes on disk\")","metadata":{"execution":{"iopub.status.busy":"2024-01-14T10:13:22.819961Z","iopub.execute_input":"2024-01-14T10:13:22.820734Z","iopub.status.idle":"2024-01-14T10:13:24.352519Z","shell.execute_reply.started":"2024-01-14T10:13:22.820696Z","shell.execute_reply":"2024-01-14T10:13:24.351488Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I run that cell multiple times to get a representative sample. These are consistently 96x96 tif images in the RAW format weighing 27935 bytes (27kb). Let's see if the validation set is the same size.\n","metadata":{}},{"cell_type":"code","source":"fig, axs = plt.subplots(3,3) \nh = 0\nv = 0\nfor i in range(9):\n    imid = dfva.id.sample(1).values[0]\n    #print(imid)\n    image = Image.open('/kaggle/input/histopathologic-cancer-detection/test/'+imid+'.tif')\n    axs[h,v].imshow(image)\n    if h == 2:\n        v +=1\n        h = 0\n    else:\n        h +=1\nprint(\"Last Image Specifications: Shape\",image.size,\"\\nFormat:\",image.info )\nprint(os.stat('/kaggle/input/histopathologic-cancer-detection/test/'+imid+'.tif').st_size, \"bytes on disk\")","metadata":{"execution":{"iopub.status.busy":"2024-01-14T10:13:24.353877Z","iopub.execute_input":"2024-01-14T10:13:24.354186Z","iopub.status.idle":"2024-01-14T10:13:25.529650Z","shell.execute_reply.started":"2024-01-14T10:13:24.354162Z","shell.execute_reply":"2024-01-14T10:13:25.528749Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Great, let's check with the command line to confirm our intutions.","metadata":{}},{"cell_type":"code","source":"# !ls -U -hal /kaggle/input/histopathologic-cancer-detection/train/ | head -10\n\n# total 5.9G\n# drwxr-xr-x 2 nobody nogroup   0 Feb 13  2023 .\n# drwxr-xr-x 4 nobody nogroup   0 Feb 13  2023 ..\n# -rw-r--r-- 1 nobody nogroup 28K Feb 13  2023 d43c081bafa286f9c1f7e921883f26ceafebc912.tif\n# -rw-r--r-- 1 nobody nogroup 28K Feb 13  2023 092d0eedebce504847715ee046b6ad74b57599b4.tif\n# -rw-r--r-- 1 nobody nogroup 28K Feb 13  2023 b0d2582c6218a8764323fc940b41312282b99bf4.tif\n# -rw-r--r-- 1 nobody nogroup 28K Feb 13  2023 187c99df762f13f99818e5593d4bab4c6577e7e3.tif\n# -rw-r--r-- 1 nobody nogroup 28K Feb 13  2023 7c5270c83837de5a5cbb2dca511559dc39d19d53.tif\n# -rw-r--r-- 1 nobody nogroup 28K Feb 13  2023 5a32933e093185f5fc91d30fc83ad571c6818d25.tif\n# -rw-r--r-- 1 nobody nogroup 28K Feb 13  2023 42e77d193e73811e0bb65a0cbd9b01c5c27900fa.tif","metadata":{"execution":{"iopub.status.busy":"2024-01-14T10:13:25.531019Z","iopub.execute_input":"2024-01-14T10:13:25.531385Z","iopub.status.idle":"2024-01-14T10:13:25.536683Z","shell.execute_reply.started":"2024-01-14T10:13:25.531353Z","shell.execute_reply":"2024-01-14T10:13:25.535724Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# !ls -U -hal /kaggle/input/histopathologic-cancer-detection/test/ | head -10\n\n# total 1.6G\n# drwxr-xr-x 2 nobody nogroup   0 Feb 13  2023 .\n# drwxr-xr-x 4 nobody nogroup   0 Feb 13  2023 ..\n# -rw-r--r-- 1 nobody nogroup 28K Feb 13  2023 a7ea26360815d8492433b14cd8318607bcf99d9e.tif\n# -rw-r--r-- 1 nobody nogroup 28K Feb 13  2023 59d21133c845dff1ebc7a0c7cf40c145ea9e9664.tif\n# -rw-r--r-- 1 nobody nogroup 28K Feb 13  2023 5fde41ce8c6048a5c2f38eca12d6528fa312cdbb.tif\n# -rw-r--r-- 1 nobody nogroup 28K Feb 13  2023 bd953a3b1db1f7041ee95ff482594c4f46c73ed0.tif\n# -rw-r--r-- 1 nobody nogroup 28K Feb 13  2023 523fc2efd7aba53e597ab0f69cc2cbded7a6ce62.tif\n# -rw-r--r-- 1 nobody nogroup 28K Feb 13  2023 d23c66547f4a00555a174d2fcb860ae399b66edc.tif\n# -rw-r--r-- 1 nobody nogroup 28K Feb 13  2023 fabf2fca23f71655974767e29eda86a9b2c97a72.tif\n","metadata":{"execution":{"iopub.status.busy":"2024-01-14T10:13:25.538013Z","iopub.execute_input":"2024-01-14T10:13:25.538327Z","iopub.status.idle":"2024-01-14T10:13:25.565320Z","shell.execute_reply.started":"2024-01-14T10:13:25.538300Z","shell.execute_reply":"2024-01-14T10:13:25.564313Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Everything with the files seems good. Each file is 28K, as we computed above. Let's move on to making a pipeline for our data.\n\n# Data Preprocessing\n\nWe found that there are 5.9GB of photo data in the last step. We could load all the photos into memory (RAM) and build a model, but we'd be limiting the models ability to use RAM in training. Loading batches of images will be better, and we'll let the system define the optimizations with tf.data.AUTOTUNE turned on.\n\nWe also know that we need to add paths to the data, balance the classes, and select random training and test sets. We'll handle all that in this first step before we make a tensorflow dataset.","metadata":{}},{"cell_type":"code","source":"randomseed = 12\n\n# Balance the classes as we pull out the training set.\nprint(\"Hom Many Class 1 Observations:\",df.label.sum())\npull80 = int(df.label.sum() * .8)\npull20 = df.label.sum()-pull80\nprint(\"Define 80% of these Observations:\",pull80)\nprint(\"Define 20% of these Observations:\",pull20)\n\n# Make dataframes of index positions for cancer and benign\ncancerlist = pd.DataFrame(df.index[df['label'] ==1].tolist(),columns=['id'])\nbenignlist = pd.DataFrame(df.index[df['label'] ==0].tolist(),columns=['id'])\n\n# Random sample the 71294 for each list\ntrainlistc = cancerlist['id'].sample(pull80,replace=False,random_state = randomseed)\ntrainlistb = benignlist['id'].sample(pull80,replace=False,random_state = randomseed)\n\nprint(\"Length of Cancer Training List:\",len(trainlistc))\nprint(\"Length of Benign Training List:\",len(trainlistb))\n\n# Add Columns To Account For Training and Testing\ncancerlist['train'] = 0\nbenignlist['train'] = 0\ncancerlist['test'] = 0\nbenignlist['test'] = 0\n\n# Populate Column\nfor i in range(len(trainlistc)):\n    cancerlist['train'].loc[cancerlist['id'] == trainlistc.iat[i]] = 1\n    benignlist['train'].loc[benignlist['id'] == trainlistb.iat[i]] = 1\n\n# Sample out test set from remainder\ntestlistc = cancerlist['id'].loc[cancerlist['train']==0].sample(pull20,\\\n                                    replace=False,random_state=randomseed)\ntestlistb = benignlist['id'].loc[benignlist['train']==0].sample(pull20,\\\n                                    replace=False,random_state=randomseed)\n\n# Populate Column\nfor i in range(len(testlistc)):\n    cancerlist['test'].loc[cancerlist['id'] == testlistc.iat[i]] ==1\n    benignlist['test'].loc[benignlist['id'] == testlistb.iat[i]] ==1\n\n# Share Output Status\nprint(\"Length of Cancer Testing List:\",len(testlistc))\nprint(\"Length of Benign Testing List:\",len(testlistb))\n\n# Ensure we didn't get rows in both\nprint(\"Number of Cancer rows in both Testing and Training Set\", \\\n      cancerlist.id.loc[(cancerlist['train']==1)&(cancerlist['test']==1)].count())\nprint(\"Number of Benign rows in both Testing and Training Set\", \\\n      benignlist.id.loc[(benignlist['train']==1)&(benignlist['test']==1)].count())\n","metadata":{"execution":{"iopub.status.busy":"2024-01-14T10:13:25.569308Z","iopub.execute_input":"2024-01-14T10:13:25.569621Z","iopub.status.idle":"2024-01-14T10:14:56.116713Z","shell.execute_reply.started":"2024-01-14T10:13:25.569582Z","shell.execute_reply":"2024-01-14T10:14:56.115708Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we have index numbers to randomly select train/test sets with balanced classes.\nWe will build the dataframes for the train and test set, complete with image paths.","metadata":{}},{"cell_type":"code","source":"def addimginfo(id):\n    return f\"/kaggle/input/histopathologic-cancer-detection/train/{id}.tif\"\n    \n#Build Training Dataframe and View\ndftrc = df.loc[df.index[trainlistc.tolist()]]\ndftrb = df.loc[df.index[trainlistb.tolist()]]\ndftr = pd.concat([dftrc,dftrb]).sample(frac=1).sample(frac=1,random_state=randomseed).reset_index(drop=True)\ndftr['path'] = dftr.id.apply(addimginfo)\nprint(\"Length:\",len(dftr.id),\" Number of Cancer Obs:\",dftr.label.sum())\ndftr.head()","metadata":{"execution":{"iopub.status.busy":"2024-01-14T10:14:56.118064Z","iopub.execute_input":"2024-01-14T10:14:56.118457Z","iopub.status.idle":"2024-01-14T10:14:56.289906Z","shell.execute_reply.started":"2024-01-14T10:14:56.118421Z","shell.execute_reply":"2024-01-14T10:14:56.288959Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Build Testing Dataframe and View\ndftec = df.loc[df.index[testlistc.tolist()]]\ndfteb = df.loc[df.index[testlistb.tolist()]]\ndfte = pd.concat([dftec,dfteb]).sample(frac=1,random_state=randomseed).sample(frac=1).reset_index(drop=True)\ndfte['path'] = dfte.id.apply(addimginfo)\nprint(\"Length:\",len(dfte.id),\" Number of Cancer Obs:\",dfte.label.sum())\ndfte.head()","metadata":{"execution":{"iopub.status.busy":"2024-01-14T10:14:56.291380Z","iopub.execute_input":"2024-01-14T10:14:56.292200Z","iopub.status.idle":"2024-01-14T10:14:56.345351Z","shell.execute_reply.started":"2024-01-14T10:14:56.292163Z","shell.execute_reply":"2024-01-14T10:14:56.344378Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Collect the garbage before moving on\ndel dftec\ndel dfteb\ndel dftrc\ndel dftrb\ndel imagelist, imagevalist\ndel pull20, pull80, fig, axs\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2024-01-14T10:14:56.346704Z","iopub.execute_input":"2024-01-14T10:14:56.347065Z","iopub.status.idle":"2024-01-14T10:14:56.568617Z","shell.execute_reply.started":"2024-01-14T10:14:56.347033Z","shell.execute_reply":"2024-01-14T10:14:56.567730Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Next we turn these dataframes into tensorflow Datasets. We do so by making numpy arrays of the data, applying the dataset function, and then apply a map function to download the binary image data on the fly. Notice that we have also set them to batch through at 64 images per iteration, and give tensorflow the ability to optimize the prefecthing of batches for speed (AUTOTUNE).","metadata":{}},{"cell_type":"code","source":"# define function to open file, decode, convert to float, and normalize to 0-1\n\n@tf.function\ndef grab_images(path):\n    file = tf.io.read_file(path)\n    img = tfio.experimental.image.decode_tiff(file, index=0)\n    img = tf.image.random_flip_left_right(img, seed=None)\n    img = tf.image.random_flip_up_down(img, seed=None)\n    img =img[:,:,0:-1]\n    img = img/255\n    img = tf.image.convert_image_dtype(img,dtype=tf.float32)\n    return img\n\n# test the function\ntester = grab_images('/kaggle/input/histopathologic-cancer-detection/train/0001a2bc5d4aa55989f014bfad74a95ac3dfff54.tif')\nplt.imshow(tester)\nplt.show()\n# ensure we are normalized between 0-1\nprint(tester[0:5,0:5,:])\n\n# ensure we have the right shape for RGBA (4th channel is pixel intensity of 1)\ntester.shape\n","metadata":{"execution":{"iopub.status.busy":"2024-01-14T10:23:57.134468Z","iopub.execute_input":"2024-01-14T10:23:57.135340Z","iopub.status.idle":"2024-01-14T10:23:57.569746Z","shell.execute_reply.started":"2024-01-14T10:23:57.135304Z","shell.execute_reply":"2024-01-14T10:23:57.568763Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# make both label datasets\ntrlabs = tf.data.Dataset.from_tensor_slices(np.array([np.array([0,1]) if i ==1 else np.array([1,0]) for i in dftr.label.values ]))\ntelabs = tf.data.Dataset.from_tensor_slices(np.array([np.array([0,1]) if i ==1 else np.array([1,0]) for i in dfte.label.values ]))\n\n# make both path datasets\ntrpaths = tf.data.Dataset.from_tensor_slices(np.array([path for path in dftr.path.values]))\ntepaths = tf.data.Dataset.from_tensor_slices(np.array([path for path in dfte.path.values]))\n\n# create image datasets on the fly\ntrimgs = trpaths.map(grab_images)\nteimgs = tepaths.map(grab_images)\n\n# zip them together\ntrset = tf.data.Dataset.zip((trimgs,trlabs)).batch(64).prefetch(AUTOTUNE)\nteset = tf.data.Dataset.zip((teimgs,telabs)).batch(64).prefetch(AUTOTUNE)","metadata":{"execution":{"iopub.status.busy":"2024-01-14T10:24:35.571280Z","iopub.execute_input":"2024-01-14T10:24:35.571671Z","iopub.status.idle":"2024-01-14T10:24:36.302361Z","shell.execute_reply.started":"2024-01-14T10:24:35.571640Z","shell.execute_reply":"2024-01-14T10:24:36.301553Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# # Ensure the datasets are working...\n# for i in trset.take(1):\n#     for element in i:\n#         print(element)","metadata":{"execution":{"iopub.status.busy":"2024-01-12T15:50:23.485271Z","iopub.execute_input":"2024-01-12T15:50:23.486111Z","iopub.status.idle":"2024-01-12T15:50:23.489919Z","shell.execute_reply.started":"2024-01-12T15:50:23.486078Z","shell.execute_reply":"2024-01-12T15:50:23.488932Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"checkpoint_filepath =''\n#define the callbacks for upcoming models\nearlyst = tf.keras.callbacks.EarlyStopping(monitor=\"binary_crossentropy\", \n                                           patience = 5)\nrlrop = tf.keras.callbacks.ReduceLROnPlateau(monitor=\"binary_crossentropy\", \n                                             factor=.1,\n                                             patience = 2,\n                                             min_lr = 0)","metadata":{"execution":{"iopub.status.busy":"2024-01-14T10:27:06.868248Z","iopub.status.idle":"2024-01-14T10:27:06.868632Z","shell.execute_reply.started":"2024-01-14T10:27:06.868420Z","shell.execute_reply":"2024-01-14T10:27:06.868435Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"With collated tf datasets in hand, it's time to do some modeling. \n\n# Build Models and Experiment Until Success\n\nFirst, several things didn't work at all. What you'll see below is my 12th or 15th attempt at a working model. Each of these things were tried but led to an exploding gradient.\n- adam optimizer\n- nadam optimizer \n- categorical cross-entropy loss function\n- momentum between .2 and .5\n- learning rates above .00025\n- softmax activation on final layer\n- sigmoid activation on final layer\n\n\n# Model 1: Basic CNN\nThe first thing I tried that worked is shown below. A fairly simple convolutional neural network built in Keras Sequential. I used 5 layers of a convolution and an average pooling layer, followed by 3 dense layers of neurons with final activation of tanh and a Binary Cross Entropy Loss Function to make the model choose from our two classes. You'll notice I programmed the model inside the tpu_strategy scope so the accelerators all function seamlessly. The JIT compiler is also turned on for speed.","metadata":{}},{"cell_type":"code","source":" \nwith tpu_strategy.scope():\n    model = Sequential([\n    Input(shape=(96, 96, 3)),  \n   \n    Conv2D(32, 3, padding='same', activation = 'relu'),\n    AvgPool2D(pool_size=2, padding='same'),      \n    \n    Conv2D(32, 3, padding='same', activation = 'relu'),\n    AvgPool2D(pool_size=2, padding='same'),\n \n    Conv2D(64, 3, padding='same', activation = 'relu'),\n    AvgPool2D(pool_size=2, padding='same'),\n    \n    Conv2D(64, 3, padding='same', activation = 'relu'),\n    AvgPool2D(pool_size=2, padding='same'),\n \n    Conv2D(32, 3, padding='same', activation = 'relu'),\n    AvgPool2D(pool_size=2, padding='same'),\n \n    # Transition to Neural Network\n    Flatten(),\n    Dense(288, activation='relu'),\n    Dense(128, activation='relu'),\n    Dense(2, activation='tanh')\n    ])\n    \n    model.compile(\n         \n#         optimizer = tf.keras.optimizers.Adam(\n#                         learning_rate=0.0025,\n#                     # USE L1 Regularization?\n#                       beta_1=0.9,\n#                         # USE L2 Regularization?\n#                         beta_2=0.2,\n#                                 epsilon=1e-07,\n#                                 amsgrad=False,\n#                                 weight_decay=None,\n#                                 clipnorm=None,\n#                                 clipvalue=None,\n#                                 global_clipnorm=None,\n#                                 use_ema=False,\n#                                 ema_momentum=0.97,\n#                                 ema_overwrite_frequency=None,\n#                                 jit_compile=True,\n#                         ),\n     optimizer =    tf.keras.optimizers.experimental.RMSprop(\n            learning_rate=0.00025,\n#             rho=0.9,\n            momentum=0.15,\n#             epsilon=1e-07,\n#             centered=False,\n#             weight_decay=None,\n#             clipnorm=None,\n#             clipvalue=None,\n#             global_clipnorm=None,\n#             use_ema=False,\n#             ema_momentum=0.99,\n#             ema_overwrite_frequency=100,\n            jit_compile=True,\n#             name='RMSprop',\n#             **kwargs\n        ),\n        loss= 'BinaryCrossentropy',\n        metrics=[ 'BinaryCrossentropy', 'accuracy']\n    )\n\nmodel.summary()\n","metadata":{"execution":{"iopub.status.busy":"2024-01-14T11:08:33.642623Z","iopub.execute_input":"2024-01-14T11:08:33.643680Z","iopub.status.idle":"2024-01-14T11:08:33.826413Z","shell.execute_reply.started":"2024-01-14T11:08:33.643632Z","shell.execute_reply":"2024-01-14T11:08:33.825526Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# define model save location\ncheckpoint_filepath = '/kaggle/working/model1/'\n!mkdir {checkpoint_filepath}\n                                           \ncheckpoints = model_checkpoint_callback = tf.keras.callbacks.ModelCheckpoint(\n    filepath=checkpoint_filepath,\n    save_weights_only=False,\n    monitor='val_accuracy',\n    mode='max',\n    save_best_only=True)\n\nhistory = model.fit(\n                    trset,\n                    epochs=30,\n                    callbacks=[rlrop,earlyst,checkpoints],\n                    validation_data = teset\n                    )\n\n# summarize history for accuracy\nplt.plot(history.history['accuracy'])\nplt.plot(history.history['val_accuracy'])\nplt.title('model accuracy')\nplt.ylabel('accuracy')\nplt.xlabel('epoch')\nplt.legend(['train', 'test'], loc='upper left')\nplt.show()\n# summarize history for loss\nplt.plot(history.history['loss'])\nplt.plot(history.history['val_loss'])\nplt.title('model loss')\nplt.ylabel('loss')\nplt.xlabel('epoch')\nplt.legend(['train', 'test'], loc='upper left')\nplt.show()\n\nprint(\"Make Some Predictions\")\nx = model.predict(teset.take(1)) #batch of 64\n\nprint(\"Binary Decision Logits:\\n\",x[0:10])\n\nprint(\"Predictions:\\n\",[np.argmax(x) for x in x[0:30]],'\\nTruth:\\n',[x for x in dfte.label.values[0:30]])","metadata":{"execution":{"iopub.status.busy":"2024-01-14T11:08:35.093970Z","iopub.execute_input":"2024-01-14T11:08:35.094846Z","iopub.status.idle":"2024-01-14T11:21:08.426656Z","shell.execute_reply.started":"2024-01-14T11:08:35.094813Z","shell.execute_reply":"2024-01-14T11:21:08.425275Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see the logits for predictions are not all of the same class, manual inspection of predictions yields roughly the same result as the tensorflow output, and the graphs populate. This is a working model.\n\nThis model is pretty good, with accuracy above 80%, but it is nowhere near optimal. There is a zero slope in the graphs for accuracy and loss, telling us the model is fully trained. The learning rate reduction on plateu callback works really well, as does the callback for early stopping. This model was not submitted for scoring.\n\nThe problem is that our accuracy is already receding in epoch 4 and the learning rate dives by an order of magnitude. This tells me that I have too high an initial momentum, learning rate, and that we can improve.\n\n# Model 2: CNN Iteration\nV2 was scored at 83.36%% on the public leaderboard and 77.3% on the private leaderboard.\n\nNext I tried adding in batch normalization, reducing the learning rate, and lowering the momentum too. I also halved the learning rate reduction on plateaus to see if we were getting too small too fast. This model is an improvement on the first successful model. ","metadata":{"execution":{"iopub.status.busy":"2024-01-12T16:37:53.452133Z","iopub.execute_input":"2024-01-12T16:37:53.453029Z","iopub.status.idle":"2024-01-12T16:37:54.016064Z","shell.execute_reply.started":"2024-01-12T16:37:53.452989Z","shell.execute_reply":"2024-01-12T16:37:54.015184Z"}}},{"cell_type":"code","source":"dropout_conv = 0.2\nwith tpu_strategy.scope():\n    model2 = Sequential([\n    Input(shape=(96, 96, 3)), \n    # Layer 1\n    Conv2D(32, 3, padding='same', activation = 'relu'),\n    AvgPool2D(pool_size=2, padding='same'),      \n    Conv2D(32, 3, padding='same', activation = 'relu'),\n    AvgPool2D(pool_size=2, padding='same'),\n    BatchNormalization(), \n        \n    # Layer 2\n    Conv2D(64, 3, padding='same', activation = 'relu'),\n    AvgPool2D(pool_size=2, padding='same'),\n    Conv2D(64, 3, padding='same', activation = 'relu'),\n    AvgPool2D(pool_size=2, padding='same'), \n    BatchNormalization(), \n        \n    # Layer 3        \n    Conv2D(32, 3, padding='same', activation = 'relu'),\n    AvgPool2D(pool_size=2, padding='same'),\n    BatchNormalization(),\n\n    # Transition to Neural Network\n    Flatten(),\n    Dense(288, activation='relu'),\n    Dense(128, activation='relu'),\n    Dense(2, activation='tanh')\n    ])\n    \n    model2.compile(\n\n     optimizer =    tf.keras.optimizers.experimental.RMSprop(\n            learning_rate=0.000025,\n#             rho=0.9,\n            momentum=0.025,\n#             epsilon=1e-07,\n#             centered=False,\n#             weight_decay=None,\n#             clipnorm=None,\n#             clipvalue=None,\n#             global_clipnorm=None,\n#             use_ema=False,\n#             ema_momentum=0.99,\n#             ema_overwrite_frequency=100,\n             jit_compile=True,\n#             name='RMSprop',\n#             **kwargs\n        ),\n        loss= 'BinaryCrossentropy',\n        metrics=[ 'BinaryCrossentropy', 'accuracy']\n    )\n\nmodel2.summary()\n","metadata":{"execution":{"iopub.status.busy":"2024-01-12T18:42:56.912606Z","iopub.execute_input":"2024-01-12T18:42:56.913063Z","iopub.status.idle":"2024-01-12T18:42:57.113730Z","shell.execute_reply.started":"2024-01-12T18:42:56.913029Z","shell.execute_reply":"2024-01-12T18:42:57.112855Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# modify callbacks\nearlyst = tf.keras.callbacks.EarlyStopping(monitor=\"binary_crossentropy\", \n                                           patience = 5)\nrlrop = tf.keras.callbacks.ReduceLROnPlateau(monitor=\"binary_crossentropy\", \n                                             factor=.5,\n                                             patience = 2,\n                                             min_lr = 0)\ncheckpoint_filepath = '/kaggle/working/model1/'\n!mkdir {checkpoint_filepath}\ncheckpoints = model_checkpoint_callback = tf.keras.callbacks.ModelCheckpoint(\n    filepath=checkpoint_filepath,\n    save_weights_only=False,\n    monitor='val_accuracy',\n    mode='max',\n    save_best_only=True)\n\nhistory2 = model2.fit(\n                    trset,\n                    epochs=100,\n                    callbacks=[rlrop,earlyst,checkpoints],\n                    validation_data = teset\n                    )\n\n# summarize history for accuracy\nplt.plot(history2.history['accuracy'])\nplt.plot(history2.history['val_accuracy'])\nplt.title('model accuracy')\nplt.ylabel('accuracy')\nplt.xlabel('epoch')\nplt.legend(['train', 'test'], loc='upper left')\nplt.show()\n# summarize history for loss\nplt.plot(history2.history['loss'])\nplt.plot(history2.history['val_loss'])\nplt.title('model loss')\nplt.ylabel('loss')\nplt.xlabel('epoch')\nplt.legend(['train', 'test'], loc='upper left')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-01-12T18:43:18.006940Z","iopub.execute_input":"2024-01-12T18:43:18.007326Z","iopub.status.idle":"2024-01-12T19:55:01.549800Z","shell.execute_reply.started":"2024-01-12T18:43:18.007296Z","shell.execute_reply":"2024-01-12T19:55:01.548821Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Again, we have a slope of zero at the end of the training, logits that are depicting real predictions (not shown), and manual inspection confirms the model results. This model trained more evenly (consistent improvement through 20 epochs before a lr reduction, denoting a consistent learning rate), trained far longer, and is more accurate. The final epochs show me that this model is fairly topped out though. To improve further I'll need a change of architecture.\n\n# Model 3: CNN w New Layout\nScored 80.1% on the Public Leaderboard and 74.77% on the private Leaderboard\n\nThe nature of this problem is cell structure recognition. For a final interation I wanted to try to layer the convolutions up so that they could work together in each layer. So, I added a few convolutional layers, and tried out the JIT compiler. Here I tried another callback, checkpoints, to save the best model automatically. Finally, I tried sigmoid but went back to the tanh as a final activation. Although this model did better in training, the model didn't generalize as well to the scoring system. This tells us that we in fact overfit the data here.","metadata":{}},{"cell_type":"code","source":"with tpu_strategy.scope():\n    model3 = Sequential([\n    Input(shape=(96, 96, 3)), \n    # Layer 1\n    Conv2D(16, 3, padding='same', activation = 'relu'),\n    Conv2D(16, 3, padding='same', activation = 'relu'),\n    Conv2D(16, 3, padding='same', activation = 'relu'),\n    AvgPool2D(pool_size=2, padding='same'),  \n    BatchNormalization(), \n    \n    # Layer 2\n    Conv2D(32, 3, padding='same', activation = 'relu'),\n    Conv2D(32, 3, padding='same', activation = 'relu'),\n    Conv2D(32, 3, padding='same', activation = 'relu'),\n    AvgPool2D(pool_size=2, padding='same'),\n    BatchNormalization(), \n    \n    # Layer 3\n    Conv2D(64, 3, padding='same', activation = 'relu'),\n    Conv2D(64, 3, padding='same', activation = 'relu'),\n    Conv2D(64, 3, padding='same', activation = 'relu'),\n    AvgPool2D(pool_size=2, padding='same'),\n    BatchNormalization(),    \n    \n    # Layer 4\n    Conv2D(128, 3, padding='same', activation = 'relu'),\n    Conv2D(128, 3, padding='same', activation = 'relu'),\n    Conv2D(128, 3, padding='same', activation = 'relu'),\n    AvgPool2D(pool_size=2, padding='same'), \n    BatchNormalization(), \n        \n    # Layer 5        \n    Conv2D(64, 3, padding='same', activation = 'relu'),\n    AvgPool2D(pool_size=2, padding='same'),\n    \n\n    # Transition to Neural Network\n    Flatten(),\n    Dense(576, activation='relu'),\n    Dense(128, activation='relu'),\n    Dense(2, activation='tanh')\n    ])\n    \n    model3.compile(\n\n     optimizer =    tf.keras.optimizers.experimental.RMSprop(\n            learning_rate=0.000025,\n#             rho=0.9,\n            momentum=0.025,\n#             epsilon=1e-07,\n#             centered=False,\n#             weight_decay=None,\n#             clipnorm=None,\n#             clipvalue=None,\n#             global_clipnorm=None,\n#             use_ema=False,\n#             ema_momentum=0.99,\n#             ema_overwrite_frequency=100,\n            jit_compile=True,\n#             name='RMSprop',\n#             **kwargs\n        ),\n        loss= 'BinaryCrossentropy',\n        metrics=[ 'BinaryCrossentropy', 'accuracy']\n    )\n\nmodel3.summary()\n","metadata":{"execution":{"iopub.status.busy":"2024-01-12T20:46:12.894199Z","iopub.execute_input":"2024-01-12T20:46:12.894933Z","iopub.status.idle":"2024-01-12T20:46:13.218614Z","shell.execute_reply.started":"2024-01-12T20:46:12.894897Z","shell.execute_reply":"2024-01-12T20:46:13.217691Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# !rm -rf {checkpoint_filepath}","metadata":{"execution":{"iopub.status.busy":"2024-01-12T20:46:14.595182Z","iopub.execute_input":"2024-01-12T20:46:14.595812Z","iopub.status.idle":"2024-01-12T20:46:15.662250Z","shell.execute_reply.started":"2024-01-12T20:46:14.595779Z","shell.execute_reply":"2024-01-12T20:46:15.661068Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"checkpoint_filepath = '/kaggle/working/model3/'\n!mkdir {checkpoint_filepath}\n\ncheckpoints = model_checkpoint_callback = tf.keras.callbacks.ModelCheckpoint(\n    filepath=checkpoint_filepath,\n    save_weights_only=False,\n    monitor='val_accuracy',\n    mode='max',\n    save_best_only=True)\n\nhistory3 = model3.fit(\n                    trset,\n                    epochs=100,\n                    callbacks=[rlrop,earlyst,checkpoints],\n                    validation_data = teset\n                    )\n\nprint(\"Make Some Predictions\")\nx = model3.predict(teset.take(1)) #batch of 64\n\nprint(\"Binary Decision Logits:\\n\",x[0:10])\n\nprint(\"Predictions:\\n\",[np.argmax(x) for x in x[0:30]],'\\nTruth:\\n',[x for x in dfte.label.values[0:30]])\n# summarize history for accuracy\nplt.plot(history3.history['accuracy'])\nplt.plot(history3.history['val_accuracy'])\nplt.title('model accuracy')\nplt.ylabel('accuracy')\nplt.xlabel('epoch')\nplt.legend(['train', 'test'], loc='upper left')\nplt.show()\n# summarize history for loss\nplt.plot(history3.history['loss'])\nplt.plot(history3.history['val_loss'])\nplt.title('model loss')\nplt.ylabel('loss')\nplt.xlabel('epoch')\nplt.legend(['train', 'test'], loc='upper left')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-01-12T20:46:17.275822Z","iopub.execute_input":"2024-01-12T20:46:17.276224Z","iopub.status.idle":"2024-01-12T22:36:44.143610Z","shell.execute_reply.started":"2024-01-12T20:46:17.276176Z","shell.execute_reply":"2024-01-12T22:36:44.142358Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This was by far the best model built from scratch, and was used to build and submit our predictions for V1 (scores above). However, this model was overfit. The final results were not as good as Model 2.\n\n# Model 4 -5 : Densenet 121 \n* (Densenet 121 - 2 Training Epochs) Private Leaderboard 81.7 Public Leaderboard 83.25\n* (Densenet 121 - 12 Training Epochs) Private Leaderboard 81.5 Public Leaderboard 84.25\n\nHere I transitioned away from models I built from scratch into models that are pretrained. I tried the densenet 121, 169, and 201 models. The results above are marginally better than the self constructed models, but only marginally so. We'd expect, the densenet 201 model to score the best. But that's not what happened\n\n","metadata":{}},{"cell_type":"code","source":"# Import Densenet\n# Versions tried DenseNet121, DenseNet169, DenseNet201\n\ndensemodel = tf.keras.applications.densenet.DenseNet121(weights='imagenet', input_shape = (96,96,3), include_top=False)\n# densemodel = tf.keras.applications.densenet.DenseNet169(weights='imagenet', input_shape = (96,96,3), include_top=False)\n\nfor layer in densemodel.layers:\n    layer.trainable=False","metadata":{"execution":{"iopub.status.busy":"2024-01-14T10:34:42.060495Z","iopub.execute_input":"2024-01-14T10:34:42.060916Z","iopub.status.idle":"2024-01-14T10:34:47.491155Z","shell.execute_reply.started":"2024-01-14T10:34:42.060881Z","shell.execute_reply":"2024-01-14T10:34:47.490216Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with tpu_strategy.scope():\n    model4 = Sequential([\n        densemodel,\n        AvgPool2D(pool_size=2, padding='same'),\n        \n        # Transition to Neural Network\n        Flatten(),\n        Dense(1920, activation='relu'),\n        Dense(128, activation='relu'),\n        Dense(2, activation='tanh')\n        ])\n    \n    model4.compile(\n\n     optimizer =    tf.keras.optimizers.experimental.RMSprop(\n            learning_rate=0.000025,\n#             rho=0.9,\n            momentum=0.025,\n#             epsilon=1e-07,\n#             centered=False,\n#             weight_decay=None,\n#             clipnorm=None,\n#             clipvalue=None,\n#             global_clipnorm=None,\n#             use_ema=False,\n#             ema_momentum=0.99,\n#             ema_overwrite_frequency=100,\n            jit_compile=True,\n#             name='RMSprop',\n#             **kwargs\n        ),\n        loss= 'BinaryCrossentropy',\n        metrics=[ 'BinaryCrossentropy', 'accuracy']\n    )\n\nmodel4.summary()\n","metadata":{"execution":{"iopub.status.busy":"2024-01-14T11:40:17.520296Z","iopub.execute_input":"2024-01-14T11:40:17.520733Z","iopub.status.idle":"2024-01-14T11:40:18.771551Z","shell.execute_reply.started":"2024-01-14T11:40:17.520696Z","shell.execute_reply":"2024-01-14T11:40:18.770576Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"checkpoint_filepath = '/kaggle/working/model4/'\n!mkdir {checkpoint_filepath}\ncheckpoints = model_checkpoint_callback = tf.keras.callbacks.ModelCheckpoint(\n    filepath=checkpoint_filepath,\n    save_weights_only=False,\n    monitor='val_accuracy',\n    mode='max',\n    save_best_only=True)\n\n\nhistory4 = model4.fit(\n                    trset,\n                    epochs=12,\n                    callbacks=[rlrop,earlyst,checkpoints],\n                    validation_data = teset\n                    )\n\nprint(\"Make Some Predictions\")\nx = model4.predict(teset.take(1)) #batch of 64\n\nprint(\"Binary Decision Logits:\\n\",x[0:10])\n\nprint(\"Predictions:\\n\",[np.argmax(x) for x in x[0:30]],'\\nTruth:\\n',[x for x in dfte.label.values[0:30]])\n# summarize history for accuracy\nplt.plot(history4.history['accuracy'])\nplt.plot(history4.history['val_accuracy'])\nplt.title('model accuracy')\nplt.ylabel('accuracy')\nplt.xlabel('epoch')\nplt.legend(['train', 'test'], loc='upper left')\nplt.show()\n# summarize history for loss\nplt.plot(history4.history['loss'])\nplt.plot(history4.history['val_loss'])\nplt.title('model loss')\nplt.ylabel('loss')\nplt.xlabel('epoch')\nplt.legend(['train', 'test'], loc='upper left')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-01-14T11:40:19.248619Z","iopub.execute_input":"2024-01-14T11:40:19.249375Z","iopub.status.idle":"2024-01-14T12:23:56.610807Z","shell.execute_reply.started":"2024-01-14T11:40:19.249342Z","shell.execute_reply":"2024-01-14T12:23:56.609810Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Models 6-7: Densenet 169\n* (Densenet 169 - 2 Training Epochs) Private Leaderboard 81.33 Public Leaderboard 84.52\n* (Densenet 169 - 2 Training Epochs) Private Leaderboard 81.99 Public Leaderboard 82.53\n","metadata":{}},{"cell_type":"code","source":"densemodel = tf.keras.applications.densenet.DenseNet169(weights='imagenet', input_shape = (96,96,3), include_top=False)\n\nfor layer in densemodel.layers:\n    layer.trainable=False\n\nwith tpu_strategy.scope():\n    model5 = Sequential([\n        densemodel,\n        AvgPool2D(pool_size=2, padding='same'),\n        \n        # Transition to Neural Network\n        Flatten(),\n        Dense(1920, activation='relu'),\n        Dense(128, activation='relu'),\n        Dense(2, activation='tanh')\n        ])\n    \n    model5.compile(\n\n     optimizer =    tf.keras.optimizers.experimental.RMSprop(\n            learning_rate=0.000025, \n            momentum=0.025, \n            jit_compile=True\n        ),\n        loss= 'BinaryCrossentropy',\n        metrics=[ 'BinaryCrossentropy', 'accuracy']\n    )\n\nmodel5.summary()\n","metadata":{"execution":{"iopub.status.busy":"2024-01-14T13:41:44.334181Z","iopub.execute_input":"2024-01-14T13:41:44.335096Z","iopub.status.idle":"2024-01-14T13:41:50.551372Z","shell.execute_reply.started":"2024-01-14T13:41:44.335066Z","shell.execute_reply":"2024-01-14T13:41:50.550408Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"checkpoint_filepath = '/kaggle/working/model5/'\n!mkdir {checkpoint_filepath}\ncheckpoints = model_checkpoint_callback = tf.keras.callbacks.ModelCheckpoint(\n    filepath=checkpoint_filepath,\n    save_weights_only=False,\n    monitor='val_accuracy',\n    mode='max',\n    save_best_only=True)\n\n\nhistory5 = model5.fit(\n                    trset,\n                    epochs=12,\n                    callbacks=[rlrop,earlyst,checkpoints],\n                    validation_data = teset\n                    )\n\nprint(\"Make Some Predictions\")\nx = model5.predict(teset.take(1)) #batch of 64\n\nprint(\"Binary Decision Logits:\\n\",x[0:10])\n\nprint(\"Predictions:\\n\",[np.argmax(x) for x in x[0:30]],'\\nTruth:\\n',[x for x in dfte.label.values[0:30]])\n# summarize history for accuracy\nplt.plot(history5.history['accuracy'])\nplt.plot(history5.history['val_accuracy'])\nplt.title('model accuracy')\nplt.ylabel('accuracy')\nplt.xlabel('epoch')\nplt.legend(['train', 'test'], loc='upper left')\nplt.show()\n# summarize history for loss\nplt.plot(history5.history['loss'])\nplt.plot(history5.history['val_loss'])\nplt.title('model loss')\nplt.ylabel('loss')\nplt.xlabel('epoch')\nplt.legend(['train', 'test'], loc='upper left')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-01-14T13:41:50.552994Z","iopub.execute_input":"2024-01-14T13:41:50.553297Z","iopub.status.idle":"2024-01-14T14:31:12.868155Z","shell.execute_reply.started":"2024-01-14T13:41:50.553271Z","shell.execute_reply":"2024-01-14T14:31:12.867095Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Models 8-9: DenseNet 201\n\n* (Densenet 201 - 2 Training Epochs) Private Leaderboard 81.5 Public Leaderboard 83.96\n* (Densenet 201 - 2 Training Epochs) Private Leaderboard 78.65 Public Leaderboard 83.07","metadata":{}},{"cell_type":"code","source":"densemodel = tf.keras.applications.densenet.DenseNet201(weights='imagenet', input_shape = (96,96,3), include_top=False)\n\nfor layer in densemodel.layers:\n    layer.trainable=False\n\nwith tpu_strategy.scope():\n    model6 = Sequential([\n        densemodel,\n        AvgPool2D(pool_size=2, padding='same'),\n        \n        # Transition to Neural Network\n        Flatten(),\n        Dense(1920, activation='relu'),\n        Dense(128, activation='relu'),\n        Dense(2, activation='tanh')\n        ])\n    \n    model6.compile(\n\n     optimizer =    tf.keras.optimizers.experimental.RMSprop(\n            learning_rate=0.000025, \n            momentum=0.025,\n            jit_compile=True\n        ),\n        loss= 'BinaryCrossentropy',\n        metrics=[ 'BinaryCrossentropy', 'accuracy']\n    )\n\nmodel6.summary()\n","metadata":{"execution":{"iopub.status.busy":"2024-01-14T14:31:12.869977Z","iopub.execute_input":"2024-01-14T14:31:12.870283Z","iopub.status.idle":"2024-01-14T14:31:20.353995Z","shell.execute_reply.started":"2024-01-14T14:31:12.870255Z","shell.execute_reply":"2024-01-14T14:31:20.353049Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"checkpoint_filepath = '/kaggle/working/model6/'\n!mkdir {checkpoint_filepath}\ncheckpoints = model_checkpoint_callback = tf.keras.callbacks.ModelCheckpoint(\n    filepath=checkpoint_filepath,\n    save_weights_only=False,\n    monitor='val_accuracy',\n    mode='max',\n    save_best_only=True)\n\n\nhistory6 = model6.fit(\n                    trset,\n                    epochs=12,\n                    callbacks=[rlrop,earlyst,checkpoints],\n                    validation_data = teset\n                    )\n\nprint(\"Make Some Predictions\")\nx = model6.predict(teset.take(1)) #batch of 64\n\nprint(\"Binary Decision Logits:\\n\",x[0:10])\n\nprint(\"Predictions:\\n\",[np.argmax(x) for x in x[0:30]],'\\nTruth:\\n',[x for x in dfte.label.values[0:30]])\n# summarize history for accuracy\nplt.plot(history6.history['accuracy'])\nplt.plot(history6.history['val_accuracy'])\nplt.title('model accuracy')\nplt.ylabel('accuracy')\nplt.xlabel('epoch')\nplt.legend(['train', 'test'], loc='upper left')\nplt.show()\n# summarize history for loss\nplt.plot(history6.history['loss'])\nplt.plot(history6.history['val_loss'])\nplt.title('model loss')\nplt.ylabel('loss')\nplt.xlabel('epoch')\nplt.legend(['train', 'test'], loc='upper left')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-01-14T14:31:20.355264Z","iopub.execute_input":"2024-01-14T14:31:20.355580Z","iopub.status.idle":"2024-01-14T14:40:24.592852Z","shell.execute_reply.started":"2024-01-14T14:31:20.355554Z","shell.execute_reply":"2024-01-14T14:40:24.591736Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The above results show the training for the Densenet architectures. I trained the nets for 2 and 12 epochs each and scored them. The results are impressive, the training short, and the amount of code to implement small. That this only took 4 lines of code to implement hundreds of layers is cool. The only drawbacks are the amount of time needed to train each epoch (10m on DenseNet121 up to 40m on DenseNet201), and scoring slowly. The results are under the titles above. \n\nThe big takeaways are is that this is easy, but not significantly better than building a custom CNN. And that I need to try everything. The big surprise is that Densenet 169 won based on private leaderboard scoring. Both DenseNet 121 and 201 scored best on 2 training epochs, not more. Very interesting results.\n\n# Create Submission(s)\n\nTo create a submission, we need to make a dataframe from the csv, load the id column (complete with path references to the files) into a dataset, run the dataset through the get image function, then through the model of choice (after loading), get the argument max for each row of predictions, repopulate the labels column in the dataframe, and finally overwrite the CSV file with the new data.","metadata":{}},{"cell_type":"code","source":"# add path data to ids, switch to np.array, then make tf dataset\nval = np.array(['/kaggle/input/histopathologic-cancer-detection/test/'+i+'.tif' for i in dfva.id.values])        \nvapaths = tf.data.Dataset.from_tensor_slices(val)\n\n# create image datasets on the fly\nvaimgs = vapaths.map(grab_images).batch(64).prefetch(AUTOTUNE)\n\n# load best model from iteration chosen model1, model2, model3, model4, model5, or model6\nwith tpu_strategy.scope():\n    modelp = tf.keras.models.load_model('/kaggle/working/model5/')\n\n# make predictions\npredsraw = modelp.predict(vaimgs)\n\n# get classifications from logits\npreds = [np.argmax(x) for x in predsraw]\n\n# post preds to dataframe\ndfva.label = preds\n\n# save the df to csv for submission\ndfva.to_csv('submission.csv',index=False)\n                \n# Technical note: If you save a submission file to your computer, \n# you can upload the submission directly to the competition page and get a score.  \n# This method was used on all transfer learning iterations to reduce GPU usage.","metadata":{"execution":{"iopub.status.busy":"2024-01-14T14:50:31.233563Z","iopub.execute_input":"2024-01-14T14:50:31.234404Z","iopub.status.idle":"2024-01-14T14:52:12.105936Z","shell.execute_reply.started":"2024-01-14T14:50:31.234372Z","shell.execute_reply":"2024-01-14T14:52:12.104931Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!head submission.csv ","metadata":{"execution":{"iopub.status.busy":"2024-01-14T13:31:03.898151Z","iopub.execute_input":"2024-01-14T13:31:03.898416Z","iopub.status.idle":"2024-01-14T13:31:04.959734Z","shell.execute_reply.started":"2024-01-14T13:31:03.898392Z","shell.execute_reply":"2024-01-14T13:31:04.958438Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Table of Results","metadata":{}},{"cell_type":"code","source":" gridset = pd.DataFrame({\n 'Architecture':['Model 1 Basic CNN','Model 1 Basic CNN','Model 1 Basic CNN','Model 1 Basic CNN',\n                 'Model 1 Basic CNN','Model 1 Basic CNN','Model 1Basic CNN', 'Model 2 Basic CNN',\n                 'Model 3 Basic CNN','DenseNet 121 (2 epoch training)','DenseNet 121 (12 epoch training)','DenseNet 169 (2 epoch training)',\n                 'DenseNet 169 (12 epoch training)','DenseNet 201 (2 epoch training)','DenseNet 201 (12 epoch training)'],\n 'Learning Rate':[.001,.001,.001,.0025,\n                  .0025,.00025,.00025,.00025,\n                  .000025,.000025,.000025,.000025,\n                  .000025,.000025,.000025],\n 'Momentum':[.3,.25,.25,.2,\n             .15,.15,.025,.025,\n             .025,.025,.025,.025,\n             .025,.025,.025],\n 'Optimizer':['Adam','Nadam','Adam','Adam',\n              'RMS Prop','RMS Prop','RMS Prop','RMS Prop',\n              'RMS Prop','RMS Prop','RMS Prop','RMS Prop',\n              'RMS Prop','RMS Prop','RMS Prop'],\n 'C Activation':['Relu','Relu','Relu','Relu',\n                 'Relu','Relu','Relu','Relu',\n                 'Relu','Relu','Relu','Relu',\n                 'Relu','Relu','Relu'],\n 'NN Activation':['Sigmoid','Sigmoid','Tanh','Softmax',\n                  'Sigmoid','Sigmoid','Tanh','Tanh',\n                  'Tanh','Tanh','Tanh','Tanh',\n                  'Tanh','Tanh','Tanh'],\n \n \"Private LB\":['Gradient Runaway','Gradient Runaway','Gradient Runaway','Gradient Runaway',\n                'Gradient Runaway','Not Scored', 'Not Scored', 77.3, \n                74.77 , 81.7, 81.5 , 81.33 ,\n                81.99 , 81.5 , 78.65 \n          ],\n\"Public LB\":['Gradient Runaway','Gradient Runaway','Gradient Runaway','Gradient Runaway',\n             'Gradient Runaway','Not Scored','Not Scored',  83.36,\n             80.1, 83.25, 84.25, 82.53,\n             84.52, 83.96, 83.07\n          ],\n \"Rank\":[None,None,None,None,\n         None,None,None,7,\n         8,2,3,5,\n         1,3,6]\n\n })\n\ngridset","metadata":{"execution":{"iopub.status.busy":"2024-01-14T15:33:07.676744Z","iopub.execute_input":"2024-01-14T15:33:07.677136Z","iopub.status.idle":"2024-01-14T15:33:07.706790Z","shell.execute_reply.started":"2024-01-14T15:33:07.677104Z","shell.execute_reply":"2024-01-14T15:33:07.705784Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Conclusion:\n\nThis project was to detect cancer cells in images. We found out that a CNN is very capable of such a task. Things that are incredibly important to the constuctionof useful CNN classifiers are optimizers, loss functions, and learning rates. Get these wrong and even a basic classifier is incapable of running. Finding out that NADAM and ADAM optimizers were not capable was helpful. SGD and RMSProp work very well, but SGD is considerably slower to converge than RMSProp. Settling on an adequate learning rate was equally as important, without a proper learning rate the classifier was inoperable. Once I got past the runaway gradients problem, I could use the callbacks function to really dial inn the learning rates and make changes to the basic architecture. The learning rate changes, to what was appropriate, yielded significant gains in the basic classifier too. The final major learning point is the difference between tanh and sigmoid activation functions on the neural network. Tanh, in my opinion is the superior choice.\n\nWhen it comes to the model architecture, more isn't always better. Not only because of overfitting, but considering how long things take to run. The Densenet 169 won based on the private leaderboard, beating out the more complex 201 model. The Densenet 201 took longer to train too. The smaller Densenet 121 took less time per epoch to train and scored second with only 2 training epochs. None of the models was remarkably better, within a few percentage points of accuracy from hand constructed models. The same held true for the basic CNNs I built. Complexity doesn't help a CNN avoid overfitting. I was lucky to have over 140,000 training examples to help mitigate the overfitting issue, and added random flips (horizontal and vertical) to the training. The more complex model outperformed the basic model and is the basis for the best submission from a CNN I built from scratch. Were I to try this again, I'd explore the transfer learning opportunities with efficientnet, inceptionV3, Xception, and ConvNext. I think that doing so would expand upon what I've learned here. \n\nTensorflow is a very capable framework once you get the Datasets operating on the fly. The program keeps the RAM load down, and facilitate focus on the modeling. I was impressed with the JIT compiler, which sped up training considerably, too. \n\nThanks for taking the moments to look at my workbook.","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}