\n \n | DEVICE | \n is GPU or TPU. | \n
\n \n | SEED | \n a different seed produces a different triple stratified kfold split. | \n
\n \n | FOLDS | \n number of folds. Best set to 3, 5, or 15 but can be any number between 2 and 15. | \n
\n \n | IMG_SIZES | \n is a Python list of length FOLDS. These are the image sizes to use each fold. | \n
\n \n | INC2019 | \n This includes the new half of the 2019 competition data. | \n
\n \n | INC2018 | \n This includes the second half of the 2019 competition data. | \n
\n \n | BATCH_SIZES | \n is a list of length FOLDS. These are batch sizes for each fold. | \n
\n \n | EPOCHS | \n is a list of length FOLDS. These are maximum epochs. | \n
\n \n | EFF_NETS | \n is a list of length FOLDS. These are the ConvNeXt to use each fold. | \n
\n \n | EFF_NETS_NUM | \n is a list of length FOLDS. These are the index for each item of EFF_NETS. | \n
\n \n | WGTS | \n this should be `1/FOLDS` for each fold. | \n
\n \n | TTA | \n test time augmentation. | \n
\n
\n","metadata":{}},{"cell_type":"markdown","source":"### New Variables\n\n \n | EXPERIMENTS | \n number of experiments to perform | \n
\n \n | FNUMBER | \n which of the KFolds to repeatedly perform experiments on. | \n
\n \n | M1 | \n is a list for each experiment, choose 0, 1, 2, 3, etc. Determines how many additional copies of malignant images from this year's comp data to add | \n
\n \n | M2 | \n is a list. Adds copies of malignant images from ISIC archive that are not in 2020, 2019, 2018, 2017 comp data | \n
\n \n | M3 | \n is a list. Adds copies of malignant images from 2019 comp data. They have been filtered to include the ones that look like 2020 data | \n
\n \n | M4 | \n is a list. Adds copies of malignant images from 2018 2017 data. | \n
\n \n | DROP_FREQ | \n a list of floats between 0 and 1. Determines the proportion of train images to apply coarse dropout to | \n
\n \n | DROP_CT | \n a list of ints. How many squares to remove from train images when applying dropout | \n
\n \n | DROP_SIZE | \n a list of floats between 0 and 1. The size of square side equals IMG_SIZE * DROP_SIZE | \n
\n \n | INFER_TEST | \n whether to predict test images each experiment | \n
\n
\n","metadata":{}},{"cell_type":"code","source":"DEVICE = \"GPU\"\n\n# USE DIFFERENT SEED FOR DIFFERENT STRATIFIED KFOLD\nSEED = 42\n\n# NUMBER OF FOLDS. USE 3, 5, OR 15 \nFOLDS = 5\n\n# WHICH FOLD TO PERFORM EXPERIMENTS ON\nFNUMBER = 1\nEXPERIMENTS = 1\n\n# WHICH IMAGE SIZES TO LOAD EACH FOLD: 128, 192, 256, 384, 512, 768 \nIMG_SIZES = [256]*EXPERIMENTS\n\n# INCLUDE OLD COMP DATA? YES=1 NO=0\nINC2019 = [0]*EXPERIMENTS\nINC2018 = [0]*EXPERIMENTS\n\n# UPSAMPLE MALIGNANT COUNT TIMES\nM1 = [0] #2020 malig\nM2 = [0] #ISIC malig\nM3 = [0] #2019 good malig\nM4 = [0] #2018 2017 malig\n\n# COARSE DROPOUT\nDROP_FREQ = [0] # between 0 and 1\nDROP_CT = [0] # may slow training if CT>16\nDROP_SIZE = [0] # between 0 and 1\n\n# BATCH SIZE AND EPOCHS\nBATCH_SIZES = [16]*EXPERIMENTS\nEPOCHS = [25]*EXPERIMENTS\n\n# WHICH EFFICIENTNET B? TO USE\nEFF_NETS = ['ResNet50','ResNet50','ResNet50','ResNet50','ResNet50']\nEFF_NETS_NUMS = [0,1,2,4,5]\n\n# WEIGHTS FOR FOLD MODELS WHEN PREDICTING TEST\nWGTS = [1/EXPERIMENTS]*EXPERIMENTS\n\n# TEST TIME AUGMENTATION STEPS\nTTA = 11\nINFER_TEST = True","metadata":{"execution":{"iopub.status.busy":"2024-03-24T22:44:21.150352Z","iopub.execute_input":"2024-03-24T22:44:21.150744Z","iopub.status.idle":"2024-03-24T22:44:21.160550Z","shell.execute_reply.started":"2024-03-24T22:44:21.150706Z","shell.execute_reply":"2024-03-24T22:44:21.159583Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if DEVICE == \"TPU\":\n print(\"connecting to TPU...\")\n try:\n tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n print('Running on TPU ', tpu.master())\n except ValueError:\n print(\"Could not connect to TPU\")\n tpu = None\n\n if tpu:\n try:\n print(\"initializing TPU ...\")\n tf.config.experimental_connect_to_cluster(tpu)\n tf.tpu.experimental.initialize_tpu_system(tpu)\n strategy = tf.distribute.experimental.TPUStrategy(tpu)\n print(\"TPU initialized\")\n except _:\n print(\"failed to initialize TPU\")\n else:\n DEVICE = \"GPU\"\n\nif DEVICE != \"TPU\":\n print(\"Using default strategy for CPU and single GPU\")\n strategy = tf.distribute.get_strategy()\n\nif DEVICE == \"GPU\":\n print(\"Num GPUs Available: \", len(tf.config.experimental.list_physical_devices('GPU')))\n \n\nAUTO = tf.data.experimental.AUTOTUNE\nREPLICAS = strategy.num_replicas_in_sync\nprint(f'REPLICAS: {REPLICAS}')","metadata":{"execution":{"iopub.status.busy":"2024-03-24T22:44:21.161685Z","iopub.execute_input":"2024-03-24T22:44:21.162029Z","iopub.status.idle":"2024-03-24T22:44:21.218693Z","shell.execute_reply.started":"2024-03-24T22:44:21.162006Z","shell.execute_reply":"2024-03-24T22:44:21.217762Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Step 1: Preprocess Data**\nPreprocess has already been done and saved to TFRecords. Here we choose which size to load. We can use either 128x128, 192x192, 256x256, 384x384, 512x512, 768x768 by changing the `IMG_SIZES` variable in the preceeding code section. These TFRecords are discussed [here][1]. The advantage of using different input sizes is discussed [here][2]\n\n[1]: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/155579\n[2]: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/160147","metadata":{}},{"cell_type":"code","source":"GCS_PATH = [None]*FOLDS\nGCS_PATH2 = [None]*FOLDS\nGCS_PATH3 = [None]*FOLDS\n\nfor i,k in enumerate(IMG_SIZES[:FOLDS]):\n GCS_PATH[i] = KaggleDatasets().get_gcs_path('melanoma-%ix%i'%(k,k))\n GCS_PATH2[i] = KaggleDatasets().get_gcs_path('isic2019-%ix%i'%(k,k))\n GCS_PATH3[i] = KaggleDatasets().get_gcs_path('malignant-v2-%ix%i'%(k,k))\nfiles_train = np.sort(np.array(tf.io.gfile.glob(GCS_PATH[0] + '/train*.tfrec')))\nfiles_test = np.sort(np.array(tf.io.gfile.glob(GCS_PATH[0] + '/test*.tfrec')))","metadata":{"execution":{"iopub.status.busy":"2024-03-24T22:44:21.219788Z","iopub.execute_input":"2024-03-24T22:44:21.220057Z","iopub.status.idle":"2024-03-24T22:44:23.065148Z","shell.execute_reply.started":"2024-03-24T22:44:21.220036Z","shell.execute_reply":"2024-03-24T22:44:23.064140Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Step 2: Data Augmentation**\nThis notebook uses rotation, sheer, zoom, shift augmentation first shown in this notebook [here][1] and successfully used in Melanoma comp by AgentAuers [here][2]. This notebook also uses horizontal flip, hue, saturation, contrast, brightness augmentation similar to last years winner and also similar to AgentAuers' notebook.\n\nAdditionally we can decide to use external data by changing the variables `INC2019` and `INC2018` in the preceeding code section. These variables respectively indicate whether to load last year 2019 data and/or year 2018 + 2017 data. These datasets are discussed [here][3]\n\nConsider experimenting with different augmenation and/or external data. The code to load TFRecords is taken from AgentAuers' notebook [here][2]. Thank you AgentAuers, this is great work.\n\n[1]: https://www.kaggle.com/cdeotte/rotation-augmentation-gpu-tpu-0-96\n[2]: https://www.kaggle.com/agentauers/incredible-tpus-finetune-effnetb0-b6-at-once\n[3]: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/164910","metadata":{}},{"cell_type":"code","source":"ROT_ = 180.0\nSHR_ = 2.0\nHZOOM_ = 8.0\nWZOOM_ = 8.0\nHSHIFT_ = 8.0\nWSHIFT_ = 8.0","metadata":{"execution":{"iopub.status.busy":"2024-03-24T22:44:23.066988Z","iopub.execute_input":"2024-03-24T22:44:23.067366Z","iopub.status.idle":"2024-03-24T22:44:23.072218Z","shell.execute_reply.started":"2024-03-24T22:44:23.067330Z","shell.execute_reply":"2024-03-24T22:44:23.071121Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_mat(rotation, shear, height_zoom, width_zoom, height_shift, width_shift):\n # returns 3x3 transformmatrix which transforms indicies\n \n # CONVERT DEGREES TO RADIANS\n rotation = math.pi * rotation / 180.\n shear = math.pi * shear / 180.\n\n def get_3x3_mat(lst):\n return tf.reshape(tf.concat([lst],axis=0), [3,3])\n \n # ROTATION MATRIX\n c1 = tf.math.cos(rotation)\n s1 = tf.math.sin(rotation)\n one = tf.constant([1],dtype='float32')\n zero = tf.constant([0],dtype='float32')\n \n rotation_matrix = get_3x3_mat([c1, s1, zero, \n -s1, c1, zero, \n zero, zero, one]) \n # SHEAR MATRIX\n c2 = tf.math.cos(shear)\n s2 = tf.math.sin(shear) \n \n shear_matrix = get_3x3_mat([one, s2, zero, \n zero, c2, zero, \n zero, zero, one]) \n # ZOOM MATRIX\n zoom_matrix = get_3x3_mat([one/height_zoom, zero, zero, \n zero, one/width_zoom, zero, \n zero, zero, one]) \n # SHIFT MATRIX\n shift_matrix = get_3x3_mat([one, zero, height_shift, \n zero, one, width_shift, \n zero, zero, one])\n \n return K.dot(K.dot(rotation_matrix, shear_matrix), \n K.dot(zoom_matrix, shift_matrix))\n\n\ndef transform(image, DIM=256): \n # input image - is one image of size [dim,dim,3] not a batch of [b,dim,dim,3]\n # output - image randomly rotated, sheared, zoomed, and shifted\n XDIM = DIM%2 #fix for size 331\n \n rot = ROT_ * tf.random.normal([1], dtype='float32')\n shr = SHR_ * tf.random.normal([1], dtype='float32') \n h_zoom = 1.0 + tf.random.normal([1], dtype='float32') / HZOOM_\n w_zoom = 1.0 + tf.random.normal([1], dtype='float32') / WZOOM_\n h_shift = HSHIFT_ * tf.random.normal([1], dtype='float32') \n w_shift = WSHIFT_ * tf.random.normal([1], dtype='float32') \n\n # GET TRANSFORMATION MATRIX\n m = get_mat(rot,shr,h_zoom,w_zoom,h_shift,w_shift) \n\n # LIST DESTINATION PIXEL INDICES\n x = tf.repeat(tf.range(DIM//2, -DIM//2,-1), DIM)\n y = tf.tile(tf.range(-DIM//2, DIM//2), [DIM])\n z = tf.ones([DIM*DIM], dtype='int32')\n idx = tf.stack( [x,y,z] )\n \n # ROTATE DESTINATION PIXELS ONTO ORIGIN PIXELS\n idx2 = K.dot(m, tf.cast(idx, dtype='float32'))\n idx2 = K.cast(idx2, dtype='int32')\n idx2 = K.clip(idx2, -DIM//2+XDIM+1, DIM//2)\n \n # FIND ORIGIN PIXEL VALUES \n idx3 = tf.stack([DIM//2-idx2[0,], DIM//2-1+idx2[1,]])\n d = tf.gather_nd(image, tf.transpose(idx3))\n \n return tf.reshape(d,[DIM, DIM,3])","metadata":{"execution":{"iopub.status.busy":"2024-03-24T22:44:23.073587Z","iopub.execute_input":"2024-03-24T22:44:23.073936Z","iopub.status.idle":"2024-03-24T22:44:23.096142Z","shell.execute_reply.started":"2024-03-24T22:44:23.073905Z","shell.execute_reply":"2024-03-24T22:44:23.095244Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def dropout(image, DIM=256, PROBABILITY = 0.75, CT = 8, SZ = 0.2):\n # input image - is one image of size [dim,dim,3] not a batch of [b,dim,dim,3]\n # output - image with CT squares of side size SZ*DIM removed\n \n # DO DROPOUT WITH PROBABILITY DEFINED ABOVE\n P = tf.cast( tf.random.uniform([],0,1)