{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"
\n SIIM-ISIC Melanoma Classification \n
\n\n\n
\n Identify melanoma in lesion images \n
","metadata":{}},{"cell_type":"markdown","source":"\n\nSkin cancer is the most prevalent type of cancer. **Melanoma**, specifically, is responsible for **75%** of skin cancer deaths, despite being the least common skin cancer. The American Cancer Society estimates over 100,000 new melanoma cases will be diagnosed in 2020. It's also expected that almost 7,000 people will die from the disease. As with other cancers, early and accurate detection—potentially aided by data science—can make treatment more effective.\n\nCurrently, dermatologists evaluate every one of a patient's moles to identify outlier lesions or “ugly ducklings” that are most likely to be melanoma. Existing AI approaches have not adequately considered this clinical frame of reference. Dermatologists could enhance their diagnostic accuracy if detection algorithms take into account “contextual” images within the same patient to determine which images represent a melanoma. If successful, classifiers would be more accurate and could better support dermatological clinic work.\n\nAs the leading healthcare organization for informatics in medical imaging, the [Society for Imaging Informatics in Medicine (SIIM)](https://siim.org/)'s mission is to advance medical imaging informatics through education, research, and innovation in a multi-disciplinary community. SIIM is joined by the [International Skin Imaging Collaboration (ISIC)](https://www.isic-archive.com/), an international effort to improve melanoma diagnosis. The ISIC Archive contains the largest publicly available collection of quality-controlled dermoscopic images of skin lesions.\n\nIn this competition, you’ll identify melanoma in images of skin lesions. In particular, you’ll use images within the same patient and determine which are likely to represent a melanoma. Using patient-level contextual information may help the development of image analysis tools, which could better support clinical dermatologists.\n\nMelanoma is a deadly disease, but if caught early, most melanomas can be cured with minor surgery. Image analysis tools that automate the diagnosis of melanoma will improve dermatologists' diagnostic accuracy. Better detection of melanoma has the opportunity to positively impact millions of people.","metadata":{}},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"markdown","source":"

Table of Contents

\n\n#### 1. EfficientNet\n#### 2. Load Required Libraries\n#### 3. Configs\n#### 4. Device\n#### 5. Paths\n#### 6. Augmentation\n#### 7. Get dataset\n#### 8. EDA\n#### 9. Monitor model performance\n#### 10. Build model\n#### 11. kFold\n#### 12. Evaluate on test\n#### 13. Submit predictions\n#### References\n","metadata":{}},{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\n\nif False:\n for dirname, _, filenames in os.walk('/kaggle/input'):\n for filename in filenames:\n print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"execution":{"iopub.status.busy":"2021-06-05T23:38:22.43987Z","iopub.execute_input":"2021-06-05T23:38:22.440305Z","iopub.status.idle":"2021-06-05T23:38:22.451352Z","shell.execute_reply.started":"2021-06-05T23:38:22.440201Z","shell.execute_reply":"2021-06-05T23:38:22.450427Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

1. EfficientNet

","metadata":{}},{"cell_type":"markdown","source":"
\n Model Size vs. ImageNet Accuracy. \n
\n\nEfficientNets significantly outperform other Convoluational Neural Networks [Tan, M. and Le, Q.V.. 2019]. \n\n ","metadata":{}},{"cell_type":"code","source":"# EfficientNet\n!pip install -q efficientnet >> /dev/null","metadata":{"execution":{"iopub.status.busy":"2021-05-27T17:08:18.331172Z","iopub.execute_input":"2021-05-27T17:08:18.331646Z","iopub.status.idle":"2021-05-27T17:08:26.822172Z","shell.execute_reply.started":"2021-05-27T17:08:18.331617Z","shell.execute_reply":"2021-05-27T17:08:26.821167Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n EfficientNetB0 \n
\n\n\n\nImage: [T. A. Putra et al 2020]\n","metadata":{}},{"cell_type":"code","source":"# Select EfficientNetBx to use\n# EfficientNetB0\n# EfficientNetB1\n# EfficientNetB2\n# EfficientNetB3\n# EfficientNetB4\n# EfficientNetB5\n# EfficientNetB6\n# EfficientNetB7\n\nMODEL = 5","metadata":{"execution":{"iopub.status.busy":"2021-05-27T17:08:26.823538Z","iopub.execute_input":"2021-05-27T17:08:26.823833Z","iopub.status.idle":"2021-05-27T17:08:26.829211Z","shell.execute_reply.started":"2021-05-27T17:08:26.823802Z","shell.execute_reply":"2021-05-27T17:08:26.828248Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

2. Load Required Libraries

","metadata":{}},{"cell_type":"code","source":"import re\nimport time\nimport math\n\nimport random\nrandom.seed(a=42)\n\nfrom tqdm import tqdm\n\nimport PIL\n\nfrom sklearn.model_selection import KFold\nfrom sklearn.metrics import roc_auc_score\n\nimport tensorflow as tf\nimport tensorflow.keras.backend as K\nimport efficientnet.tfkeras as efn\n\nfrom kaggle_datasets import KaggleDatasets","metadata":{"execution":{"iopub.status.busy":"2021-05-27T17:08:26.831437Z","iopub.execute_input":"2021-05-27T17:08:26.83183Z","iopub.status.idle":"2021-05-27T17:08:27.182128Z","shell.execute_reply.started":"2021-05-27T17:08:26.831792Z","shell.execute_reply":"2021-05-27T17:08:27.181138Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

3. Configs

","metadata":{}},{"cell_type":"code","source":"CFG = dict(\n net_count = 7, \n batch_size = 8, # 8; 16; 32; 64; bigger batch size => moemry allocation issue\n epochs = 10, # 5; 10; 20;\n verbose = 1, # 0; 1\n \n optimizer = 'adam',\n \n # kFold\n NSPLITS = 5, \n RANDOM_STATE = 42, \n \n # LR\n LR_START = 0.000005, # 5e-6\n LR_MAX = 0.000010, # 1e-5; 2e-5; 3e-5\n LR_MIN = 0.000001, # 1e-6\n LR_RAMPUP_EPOCHS = 5,\n LR_SUSTAIN_EPOCHS = 0,\n LR_EXP_DECAY = 0.8,\n \n # Images sizes\n read_size = 256, # 256 (a); 384 (b); 512 (c); 768 (d);\n crop_size = 250, # 250 (a); 370 (b); 500 (c); 750 (d);\n net_size = 240, # 240 (a); 352 (b); 480 (c); 730 (d);\n \n # Images augs\n ROTATION = 180.0,\n SHEAR = 2.0,\n HZOOM = 8.0,\n WZOOM = 8.0,\n HSHIFT = 8.0,\n WSHIFT = 8.0,\n\n # Postprocessing\n label_smooth_fac = 0, # 0.01; 0.05; 0.1; 0.2;\n tta_steps = 5 # 5; 10; 15; 25; 30;\n)","metadata":{"execution":{"iopub.status.busy":"2021-05-27T17:08:27.183988Z","iopub.execute_input":"2021-05-27T17:08:27.184392Z","iopub.status.idle":"2021-05-27T17:08:27.192652Z","shell.execute_reply.started":"2021-05-27T17:08:27.184348Z","shell.execute_reply":"2021-05-27T17:08:27.191071Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

4. Device

","metadata":{}},{"cell_type":"code","source":"DEVICE = \"TPU\"\n#DEVICE = \"GPU\"","metadata":{"execution":{"iopub.status.busy":"2021-05-27T17:08:27.194049Z","iopub.execute_input":"2021-05-27T17:08:27.194372Z","iopub.status.idle":"2021-05-27T17:08:27.210842Z","shell.execute_reply.started":"2021-05-27T17:08:27.194342Z","shell.execute_reply":"2021-05-27T17:08:27.209889Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if DEVICE == \"TPU\":\n print(\"connecting to TPU...\")\n try:\n tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n print('Running on TPU ', tpu.master())\n except ValueError:\n print(\"Could not connect to TPU\")\n tpu = None\n\n if tpu:\n try:\n print(\"initializing TPU ...\")\n tf.config.experimental_connect_to_cluster(tpu)\n tf.tpu.experimental.initialize_tpu_system(tpu)\n strategy = tf.distribute.experimental.TPUStrategy(tpu)\n print(\"TPU initialized\")\n except _:\n print(\"failed to initialize TPU\")\n else:\n DEVICE = \"GPU\"\n\nif DEVICE != \"TPU\":\n print(\"Using default strategy for CPU and single GPU\")\n strategy = tf.distribute.get_strategy()\n\nif DEVICE == \"GPU\":\n print(\"Num GPUs Available: \", len(tf.config.experimental.list_physical_devices('GPU')))\n \n\nAUTO = tf.data.experimental.AUTOTUNE\nREPLICAS = strategy.num_replicas_in_sync\nprint(f'REPLICAS: {REPLICAS}')","metadata":{"execution":{"iopub.status.busy":"2021-05-27T17:08:27.212196Z","iopub.execute_input":"2021-05-27T17:08:27.212493Z","iopub.status.idle":"2021-05-27T17:08:32.722647Z","shell.execute_reply.started":"2021-05-27T17:08:27.212465Z","shell.execute_reply":"2021-05-27T17:08:32.720253Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!nvidia-smi","metadata":{"execution":{"iopub.status.busy":"2021-05-27T17:08:32.723642Z","iopub.execute_input":"2021-05-27T17:08:32.723922Z","iopub.status.idle":"2021-05-27T17:08:33.45509Z","shell.execute_reply.started":"2021-05-27T17:08:32.723897Z","shell.execute_reply":"2021-05-27T17:08:33.454035Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

5. Paths

","metadata":{}},{"cell_type":"code","source":"BASEPATH = \"../input/siim-isic-melanoma-classification\"\ndf_train = pd.read_csv(os.path.join(BASEPATH, 'train.csv'))\ndf_test = pd.read_csv(os.path.join(BASEPATH, 'test.csv'))\ndf_sub = pd.read_csv(os.path.join(BASEPATH, 'sample_submission.csv'))","metadata":{"execution":{"iopub.status.busy":"2021-05-27T17:08:33.45986Z","iopub.execute_input":"2021-05-27T17:08:33.460211Z","iopub.status.idle":"2021-05-27T17:08:33.60202Z","shell.execute_reply.started":"2021-05-27T17:08:33.460176Z","shell.execute_reply":"2021-05-27T17:08:33.600928Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We have images in TFRecords of different sizes:\n- 256x256\n- 384x384\n- 512x512\n- 768x768","metadata":{}},{"cell_type":"code","source":"!ls ../input/melanoma*","metadata":{"execution":{"iopub.status.busy":"2021-05-27T17:08:33.603476Z","iopub.execute_input":"2021-05-27T17:08:33.603791Z","iopub.status.idle":"2021-05-27T17:08:34.364755Z","shell.execute_reply.started":"2021-05-27T17:08:33.603741Z","shell.execute_reply":"2021-05-27T17:08:34.36365Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Select image size of ionterest\nSIZE = 256","metadata":{"execution":{"iopub.status.busy":"2021-05-27T17:08:34.366267Z","iopub.execute_input":"2021-05-27T17:08:34.366588Z","iopub.status.idle":"2021-05-27T17:08:34.371049Z","shell.execute_reply.started":"2021-05-27T17:08:34.366557Z","shell.execute_reply":"2021-05-27T17:08:34.369869Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"GCS_PATH = KaggleDatasets().get_gcs_path('melanoma-%ix%i' % (SIZE,SIZE))\n#GCS_PATH = KaggleDatasets().get_gcs_path('melanoma-256x256')\n#GCS_PATH = KaggleDatasets().get_gcs_path('melanoma-384x384')\n#GCS_PATH = KaggleDatasets().get_gcs_path('melanoma-512x512')\n#GCS_PATH = KaggleDatasets().get_gcs_path('melanoma-768x768')\nfiles_train = np.sort(np.array(tf.io.gfile.glob(GCS_PATH + '/train*.tfrec')))\nfiles_test = np.sort(np.array(tf.io.gfile.glob(GCS_PATH + '/test*.tfrec')))","metadata":{"execution":{"iopub.status.busy":"2021-05-27T17:08:34.372625Z","iopub.execute_input":"2021-05-27T17:08:34.373068Z","iopub.status.idle":"2021-05-27T17:08:35.001901Z","shell.execute_reply.started":"2021-05-27T17:08:34.373037Z","shell.execute_reply":"2021-05-27T17:08:35.000874Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(len(df_train),len(df_train[df_train.target==0]),len(df_train[df_train.target==1]))","metadata":{"execution":{"iopub.status.busy":"2021-05-27T17:08:35.003216Z","iopub.execute_input":"2021-05-27T17:08:35.003526Z","iopub.status.idle":"2021-05-27T17:08:35.045903Z","shell.execute_reply.started":"2021-05-27T17:08:35.003496Z","shell.execute_reply":"2021-05-27T17:08:35.044559Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

6. Augmentation

","metadata":{}},{"cell_type":"code","source":"# by Chris Deotte\n\ndef get_mat(rotation, shear, height_zoom, width_zoom, height_shift, width_shift):\n # returns 3x3 transformmatrix which transforms indicies\n \n # CONVERT DEGREES TO RADIANS\n rotation = math.pi * rotation / 180.\n shear = math.pi * shear / 180.\n\n def get_3x3_mat(lst):\n return tf.reshape(tf.concat([lst],axis=0), [3,3])\n \n \n # ROTATION MATRIX\n c1 = tf.math.cos(rotation)\n s1 = tf.math.sin(rotation)\n one = tf.constant([1],dtype='float32')\n zero = tf.constant([0],dtype='float32')\n \n rotation_matrix = get_3x3_mat([c1, s1, zero, \n -s1, c1, zero, \n zero, zero, one]) \n # SHEAR MATRIX\n c2 = tf.math.cos(shear)\n s2 = tf.math.sin(shear)\n shear_matrix = get_3x3_mat([one, s2, zero, \n zero, c2, zero, \n zero, zero, one]) \n \n # ZOOM MATRIX\n zoom_matrix = get_3x3_mat([one/height_zoom, zero, zero, \n zero, one/width_zoom, zero, \n zero, zero, one]) \n # SHIFT MATRIX\n shift_matrix = get_3x3_mat([one, zero, height_shift, \n zero, one, width_shift, \n zero, zero, one])\n \n return K.dot(K.dot(rotation_matrix, shear_matrix), \n K.dot(zoom_matrix, shift_matrix))\n\n\ndef transform(image, cfg): \n # input image - is one image of size [dim,dim,3] not a batch of [b,dim,dim,3]\n # output - image randomly rotated, sheared, zoomed, and shifted\n DIM = cfg[\"read_size\"]\n XDIM = DIM%2 #fix for size 331\n \n rot = cfg['ROTATION'] * tf.random.normal([1], dtype='float32')\n shr = cfg['SHEAR'] * tf.random.normal([1], dtype='float32') \n h_zoom = 1.0 + tf.random.normal([1], dtype='float32') / cfg['HZOOM']\n w_zoom = 1.0 + tf.random.normal([1], dtype='float32') / cfg['WZOOM']\n h_shift = cfg['HSHIFT'] * tf.random.normal([1], dtype='float32') \n w_shift = cfg['WSHIFT'] * tf.random.normal([1], dtype='float32') \n\n # GET TRANSFORMATION MATRIX\n m = get_mat(rot,shr,h_zoom,w_zoom,h_shift,w_shift) \n\n # LIST DESTINATION PIXEL INDICES\n x = tf.repeat(tf.range(DIM//2, -DIM//2,-1), DIM)\n y = tf.tile(tf.range(-DIM//2, DIM//2), [DIM])\n z = tf.ones([DIM*DIM], dtype='int32')\n idx = tf.stack( [x,y,z] )\n \n # ROTATE DESTINATION PIXELS ONTO ORIGIN PIXELS\n idx2 = K.dot(m, tf.cast(idx, dtype='float32'))\n idx2 = K.cast(idx2, dtype='int32')\n idx2 = K.clip(idx2, -DIM//2+XDIM+1, DIM//2)\n \n # FIND ORIGIN PIXEL VALUES \n idx3 = tf.stack([DIM//2-idx2[0,], DIM//2-1+idx2[1,]])\n d = tf.gather_nd(image, tf.transpose(idx3))\n \n return tf.reshape(d,[DIM, DIM,3])","metadata":{"execution":{"iopub.status.busy":"2021-05-27T17:08:35.047397Z","iopub.execute_input":"2021-05-27T17:08:35.04774Z","iopub.status.idle":"2021-05-27T17:08:35.267036Z","shell.execute_reply.started":"2021-05-27T17:08:35.047707Z","shell.execute_reply":"2021-05-27T17:08:35.265844Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def read_labeled_tfrecord(example):\n tfrec_format = {\n 'image' : tf.io.FixedLenFeature([], tf.string),\n 'image_name' : tf.io.FixedLenFeature([], tf.string),\n 'patient_id' : tf.io.FixedLenFeature([], tf.int64),\n 'sex' : tf.io.FixedLenFeature([], tf.int64),\n 'age_approx' : tf.io.FixedLenFeature([], tf.int64),\n 'anatom_site_general_challenge': tf.io.FixedLenFeature([], tf.int64),\n 'diagnosis' : tf.io.FixedLenFeature([], tf.int64),\n 'target' : tf.io.FixedLenFeature([], tf.int64)\n } \n example = tf.io.parse_single_example(example, tfrec_format)\n return example['image'], example['target']\n\n\ndef read_unlabeled_tfrecord(example, return_image_name):\n tfrec_format = {\n 'image' : tf.io.FixedLenFeature([], tf.string),\n 'image_name' : tf.io.FixedLenFeature([], tf.string),\n }\n example = tf.io.parse_single_example(example, tfrec_format)\n return example['image'], example['image_name'] if return_image_name else 0\n\n \ndef prepare_image(img, cfg=None, augment=True): \n img = tf.image.decode_jpeg(img, channels=3)\n img = tf.image.resize(img, [cfg['read_size'], cfg['read_size']])\n img = tf.cast(img, tf.float32) / 255.0\n \n if augment:\n img = transform(img, cfg)\n img = tf.image.random_crop(img, [cfg['crop_size'], cfg['crop_size'], 3])\n img = tf.image.random_flip_left_right(img)\n img = tf.image.random_hue(img, 0.01) # 0.01; 0.015; 0.02\n img = tf.image.random_saturation(img, 0.75, 1.25) # 0.5,1.5; 0.7,1.3; 0.8, 1.2; 0.9,1.1;\n img = tf.image.random_contrast(img, 0.8, 1.2) # 0.9,1.1; 0.8,1.2;\n img = tf.image.random_brightness(img, 0.1) # 0.10; 0.15; 0.20;\n\n else:\n img = tf.image.central_crop(img, cfg['crop_size'] / cfg['read_size'])\n \n img = tf.image.resize(img, [cfg['net_size'], cfg['net_size']])\n img = tf.reshape(img, [cfg['net_size'], cfg['net_size'], 3])\n return img\n\ndef count_data_items(filenames):\n n = [int(re.compile(r\"-([0-9]*)\\.\").search(filename).group(1)) \n for filename in filenames]\n return np.sum(n)","metadata":{"execution":{"iopub.status.busy":"2021-05-27T17:08:35.268584Z","iopub.execute_input":"2021-05-27T17:08:35.269173Z","iopub.status.idle":"2021-05-27T17:08:35.286573Z","shell.execute_reply.started":"2021-05-27T17:08:35.269125Z","shell.execute_reply":"2021-05-27T17:08:35.285394Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

7. Get dataset

","metadata":{}},{"cell_type":"code","source":"def get_dataset(files, cfg, \n augment = False, \n shuffle = False, \n repeat = False, \n labeled=True, \n return_image_names=True\n ):\n \n ds = tf.data.TFRecordDataset(files, num_parallel_reads=AUTO)\n ds = ds.cache()\n \n if repeat:\n ds = ds.repeat()\n \n if shuffle: \n ds = ds.shuffle(1024*8)\n opt = tf.data.Options()\n opt.experimental_deterministic = False\n ds = ds.with_options(opt)\n \n if labeled: \n ds = ds.map(read_labeled_tfrecord, num_parallel_calls=AUTO)\n else:\n ds = ds.map(lambda example: read_unlabeled_tfrecord(example, return_image_names), num_parallel_calls=AUTO) \n \n ds = ds.map(lambda img, imgname_or_label: (prepare_image(img, augment=augment, cfg=cfg), imgname_or_label), num_parallel_calls=AUTO)\n \n ds = ds.batch(cfg['batch_size'] * REPLICAS)\n ds = ds.prefetch(AUTO)\n return ds","metadata":{"execution":{"iopub.status.busy":"2021-05-27T17:08:35.288363Z","iopub.execute_input":"2021-05-27T17:08:35.28892Z","iopub.status.idle":"2021-05-27T17:08:35.302952Z","shell.execute_reply.started":"2021-05-27T17:08:35.288857Z","shell.execute_reply":"2021-05-27T17:08:35.30186Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

8. EDA

","metadata":{}},{"cell_type":"code","source":"def show_dataset(thumb_size, cols, rows, ds):\n mosaic = PIL.Image.new(mode='RGB', size=(thumb_size*cols + (cols-1), thumb_size*rows + (rows-1)))\n \n for idx, data in enumerate(iter(ds)):\n img, target_or_imgid = data\n ix = idx % cols\n iy = idx // cols\n img = np.clip(img.numpy() * 255, 0, 255).astype(np.uint8)\n img = PIL.Image.fromarray(img)\n img = img.resize((thumb_size, thumb_size), resample=PIL.Image.BILINEAR)\n mosaic.paste(img, (ix*thumb_size + ix, iy*thumb_size + iy))\n\n display(mosaic)","metadata":{"execution":{"iopub.status.busy":"2021-05-27T17:08:35.305261Z","iopub.execute_input":"2021-05-27T17:08:35.305702Z","iopub.status.idle":"2021-05-27T17:08:35.319721Z","shell.execute_reply.started":"2021-05-27T17:08:35.305659Z","shell.execute_reply":"2021-05-27T17:08:35.318753Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"nrows = 6\nncols = 12\nds = get_dataset(files_train, CFG).unbatch().take(nrows*ncols) ","metadata":{"execution":{"iopub.status.busy":"2021-05-27T17:08:35.321982Z","iopub.execute_input":"2021-05-27T17:08:35.322415Z","iopub.status.idle":"2021-05-27T17:08:35.701259Z","shell.execute_reply.started":"2021-05-27T17:08:35.322371Z","shell.execute_reply":"2021-05-27T17:08:35.697917Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ds = tf.data.TFRecordDataset(files_train, num_parallel_reads=AUTO)\nds = ds.take(1).cache().repeat()\nds = ds.map(read_labeled_tfrecord, num_parallel_calls=AUTO)\nds = ds.map(lambda img, target: (prepare_image(img, cfg=CFG, augment=True), target), num_parallel_calls=AUTO)\nds = ds.take(ncols*nrows)\nds = ds.prefetch(AUTO)","metadata":{"execution":{"iopub.status.busy":"2021-05-27T17:09:17.670936Z","iopub.execute_input":"2021-05-27T17:09:17.671285Z","iopub.status.idle":"2021-05-27T17:09:17.845923Z","shell.execute_reply.started":"2021-05-27T17:09:17.671257Z","shell.execute_reply":"2021-05-27T17:09:17.844896Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"show_dataset(ncols*(nrows+1), ncols, nrows, ds)","metadata":{"execution":{"iopub.status.busy":"2021-05-27T17:11:50.385083Z","iopub.execute_input":"2021-05-27T17:11:50.38563Z","iopub.status.idle":"2021-05-27T17:11:51.082742Z","shell.execute_reply.started":"2021-05-27T17:11:50.385582Z","shell.execute_reply":"2021-05-27T17:11:51.081871Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

9. Monitor model performance

","metadata":{}},{"cell_type":"code","source":"def get_lr_callback(cfg):\n lr_start = cfg['LR_START']\n lr_max = cfg['LR_MAX'] * strategy.num_replicas_in_sync # can be set dynamic cfg['LR_MAX']*cfg['batch_size'] * replicas\n lr_min = cfg['LR_MIN']\n lr_ramp_ep = cfg['LR_RAMPUP_EPOCHS']\n lr_sus_ep = cfg['LR_SUSTAIN_EPOCHS']\n lr_decay = cfg['LR_EXP_DECAY']\n \n def lrfn(epoch):\n if epoch < lr_ramp_ep:\n lr = (lr_max - lr_start) / lr_ramp_ep * epoch + lr_start\n \n elif epoch < lr_ramp_ep + lr_sus_ep:\n lr = lr_max\n \n else:\n lr = (lr_max - lr_min) * lr_decay**(epoch - lr_ramp_ep - lr_sus_ep) + lr_min\n \n return lr\n\n #lr_callback = tf.keras.callbacks.LearningRateScheduler(lrfn, verbose=False)\n lr_callback = tf.keras.callbacks.ReduceLROnPlateau(monitor='val_loss', \n factor=0.25, \n patience=2, \n verbose=0, \n mode='auto'\n )\n return lr_callback","metadata":{"execution":{"iopub.status.busy":"2021-05-26T12:46:45.070401Z","iopub.execute_input":"2021-05-26T12:46:45.070791Z","iopub.status.idle":"2021-05-26T12:46:45.079467Z","shell.execute_reply.started":"2021-05-26T12:46:45.07075Z","shell.execute_reply":"2021-05-26T12:46:45.078318Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

10. Build model

","metadata":{}},{"cell_type":"code","source":"def get_model(cfg):\n model_input = tf.keras.Input(shape=(cfg['net_size'], cfg['net_size'], 3), name='imgIn')\n\n dummy = tf.keras.layers.Lambda(lambda x:x)(model_input)\n\n #constructor = getattr(efn, f'EfficientNetB1')\n constructor = getattr(efn, 'EfficientNetB{}'.format(MODEL)) # 1,3,5,6,7\n x = constructor(include_top=False, weights='imagenet', \n input_shape=(cfg['net_size'], cfg['net_size'], 3), \n pooling='avg')(dummy)\n \n x = tf.keras.layers.Dense(1, activation='sigmoid')(x)\n outputs = [x]\n \n model = tf.keras.Model(model_input, outputs, name='aNetwork')\n model.summary()\n return model\n\ndef compile_new_model(cfg): \n with strategy.scope():\n model = get_model(cfg)\n \n #losses = tf.keras.losses.BinaryCrossentropy() # default from_logits=False\n #losses = [tf.keras.losses.BinaryCrossentropy(label_smoothing = cfg['label_smooth_fac'])]\n #losses = [tf.keras.losses.BinaryCrossentropy(label_smoothing = cfg['label_smooth_fac']) for i in range(cfg['net_count'])]\n losses = [tf.keras.losses.BinaryCrossentropy(label_smoothing = cfg['label_smooth_fac']) if cfg['label_smooth_fac'] > 0 else tf.keras.losses.BinaryCrossentropy()]\n \n model.compile(\n optimizer = cfg['optimizer'],\n loss = losses,\n metrics = [tf.keras.metrics.AUC(name='auc')])\n \n return model","metadata":{"execution":{"iopub.status.busy":"2021-05-26T12:46:45.081129Z","iopub.execute_input":"2021-05-26T12:46:45.081565Z","iopub.status.idle":"2021-05-26T12:46:45.103921Z","shell.execute_reply.started":"2021-05-26T12:46:45.081522Z","shell.execute_reply":"2021-05-26T12:46:45.102779Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"`label_smoothing`\n- Float in [0, 1]. \n - When 0, no smoothing occurs. \n - When > 0, we compute the loss between the predicted labels and a smoothed version of the true labels, where the smoothing squeezes the labels towards 0.5. Larger values of `label_smoothing` correspond to heavier smoothing.","metadata":{}},{"cell_type":"markdown","source":"

11. kFold

","metadata":{}},{"cell_type":"code","source":"folds = KFold(n_splits=CFG['NSPLITS'], \n shuffle = True, \n random_state = CFG['RANDOM_STATE'])","metadata":{"execution":{"iopub.status.busy":"2021-05-26T12:46:45.105788Z","iopub.execute_input":"2021-05-26T12:46:45.106227Z","iopub.status.idle":"2021-05-26T12:46:45.119936Z","shell.execute_reply.started":"2021-05-26T12:46:45.106193Z","shell.execute_reply":"2021-05-26T12:46:45.118939Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# OPTIONS\n\n# TRAIN\noptAug_tr = True\noptShuffle_tr = True \noptRepeat_tr = True\n\n# TRAIN AUG\noptAug_trAug = True\noptRepeat_trAug = True\noptLabeled_trAug = False\noptReturnImgNames_trAug = False\n\n\n# VALIDATION\noptAug_va = False # no Aug in validation!\noptRepeat_va = False\noptLabeled_va = False\noptReturnImgNames_va = True\n\n# TEST AUG\noptAug_tsAug = True\noptRepeat_tsAug = True\noptLabeled_tsAug = False\noptReturnImgNames_tsAug = False\n\n\n# TEST\noptAug_ts = False \noptRepeat_ts = False\noptLabeled_ts = False\noptReturnImgNames_ts = True","metadata":{"execution":{"iopub.status.busy":"2021-05-26T12:46:45.121194Z","iopub.execute_input":"2021-05-26T12:46:45.121483Z","iopub.status.idle":"2021-05-26T12:46:45.131825Z","shell.execute_reply.started":"2021-05-26T12:46:45.121457Z","shell.execute_reply":"2021-05-26T12:46:45.131076Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pred_tr = pd.DataFrame()\ncnt = 0\n\nprint('\\nKeep patience, takes long ...')\n\nfor fold_idx, (tr_idx,va_idx) in enumerate(folds.split(files_train)):\n \n print('\\nProcessing FOLD {}'.format(fold_idx))\n \n files_train_tr = files_train[tr_idx]\n files_train_va = files_train[va_idx]\n \n ds_train = get_dataset(files_train_tr, \n CFG, \n augment=optAug_tr, \n shuffle=optShuffle_tr, \n repeat=optRepeat_tr\n )\n \n #ds_train = ds_train.map(lambda img, label: (img, tuple([label] * CFG['net_count'])))\n ds_train = ds_train.map(lambda img, label: (img, tuple([label])))\n \n steps_train = count_data_items(files_train_tr) / (CFG['batch_size'] * REPLICAS)\n\n \n # COMPILE MODEL\n model = compile_new_model(CFG)\n \n # SAVE BEST MODEL EACH FOLD\n path_to_best_model = 'fold-%i.h5'%fold_idx\n saveBestModel = tf.keras.callbacks.ModelCheckpoint(filepath=path_to_best_model, \n monitor='val_loss', \n verbose=CFG['verbose'], \n save_best_only=True,\n save_weights_only=True, \n mode='min', \n save_freq='epoch'\n )\n print('\\nTraining ...')\n history = model.fit(ds_train, \n epochs = CFG['epochs'],\n steps_per_epoch = steps_train,\n verbose = 1,\n callbacks = [get_lr_callback(CFG), saveBestModel]\n )\n \n #print('\\nLoading best model...')\n #model.load_weights(path_to_best_model)\n \n # MAKE TRAIN PREDICTION\n CFG['batch_size'] = 256 # increase batch size\n\n cnt_train = count_data_items(files_train_va)\n steps = cnt_train / (CFG['batch_size'] * REPLICAS) * CFG['tta_steps']\n \n ds_trainAug = get_dataset(files_train_va, \n CFG, \n augment=optAug_trAug,\n repeat=optRepeat_trAug,\n labeled=optLabeled_trAug, \n return_image_names=optReturnImgNames_trAug\n )\n # Proba\n probs = model.predict(ds_trainAug, verbose=1, steps=steps)\n probs = probs[:cnt_train * CFG['tta_steps'],:]\n probs = np.stack(np.split(probs, CFG['tta_steps'], axis=0), axis=0)\n \n ds = get_dataset(files_train_va, \n CFG, \n augment=optAug_va, \n repeat=optRepeat_va, \n labeled=optLabeled_va, \n return_image_names=optReturnImgNames_va\n )\n \n image_names = np.array([img_name.numpy().decode(\"utf-8\") for img, img_name in iter(ds.unbatch())])\n \n pred = pd.DataFrame(dict(image_name = image_names,\n target = np.mean(probs[:,:,0], axis=0))\n )\n \n pred_tr = pd.concat([pred_tr, pred], axis=0)\n \n # MAKE SUBMISSION DATA\n cnt_test = count_data_items(files_test)\n steps = cnt_test / (CFG['batch_size'] * REPLICAS) * CFG['tta_steps']\n \n ds_testAug = get_dataset(files_test, \n CFG, \n augment=optAug_tsAug, \n repeat=optRepeat_tsAug, \n labeled=optLabeled_tsAug, \n return_image_names=optReturnImgNames_tsAug\n )\n\n probs = model.predict(ds_testAug, verbose=1, steps=steps)\n probs = probs[:cnt_test * CFG['tta_steps'],:]\n probs = np.stack(np.split(probs, CFG['tta_steps'], axis=0), axis=0)\n \n if cnt == 0:\n probs_sub = probs/CFG['NSPLITS']\n cnt = 1\n else:\n probs_sub += probs/CFG['NSPLITS']","metadata":{"execution":{"iopub.status.busy":"2021-05-26T12:46:45.13289Z","iopub.execute_input":"2021-05-26T12:46:45.13332Z","iopub.status.idle":"2021-05-26T12:57:20.895102Z","shell.execute_reply.started":"2021-05-26T12:46:45.133291Z","shell.execute_reply":"2021-05-26T12:57:20.894101Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pred_tr = pred_tr.sort_values('image_name') \npred_tr.to_csv('pred_tr.csv', index=False)\npred_tr.head(10)","metadata":{"execution":{"iopub.status.busy":"2021-05-26T12:57:20.896364Z","iopub.execute_input":"2021-05-26T12:57:20.896636Z","iopub.status.idle":"2021-05-26T12:57:21.049666Z","shell.execute_reply.started":"2021-05-26T12:57:20.89661Z","shell.execute_reply":"2021-05-26T12:57:21.048557Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# RESULTS\npred_tr = pred_tr.merge(df_train, on = [\"image_name\"], how = \"left\")\nprint(roc_auc_score(pred_tr[\"target_y\"], pred_tr[\"target_x\"]))","metadata":{"execution":{"iopub.status.busy":"2021-05-26T12:57:21.055734Z","iopub.execute_input":"2021-05-26T12:57:21.056078Z","iopub.status.idle":"2021-05-26T12:57:21.107915Z","shell.execute_reply.started":"2021-05-26T12:57:21.056049Z","shell.execute_reply":"2021-05-26T12:57:21.106885Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

12. Evaluate on test

","metadata":{}},{"cell_type":"code","source":"ds = get_dataset(files_test, \n CFG, \n augment=optAug_ts, \n repeat=optRepeat_ts, \n labeled=optLabeled_ts, \n return_image_names=optReturnImgNames_ts\n )\n\nimage_names = np.array([img_name.numpy().decode(\"utf-8\") for img, img_name in iter(ds.unbatch())])","metadata":{"execution":{"iopub.status.busy":"2021-05-26T12:57:21.109338Z","iopub.execute_input":"2021-05-26T12:57:21.109662Z","iopub.status.idle":"2021-05-26T12:57:40.717428Z","shell.execute_reply.started":"2021-05-26T12:57:21.109631Z","shell.execute_reply":"2021-05-26T12:57:40.716437Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

13. Submit predictions

","metadata":{}},{"cell_type":"code","source":"submission = pd.DataFrame(dict(image_name = image_names,\n target = np.mean(probs_sub[:,:,0], axis=0))\n #target = np.mean(probs, axis = 1))\n )\n \n\nsubmission = submission.sort_values('image_name') \nsubmission.to_csv('submission.csv', index=False)\nsubmission.head(10)","metadata":{"execution":{"iopub.status.busy":"2021-05-26T12:57:40.718623Z","iopub.execute_input":"2021-05-26T12:57:40.718893Z","iopub.status.idle":"2021-05-26T12:57:40.784582Z","shell.execute_reply.started":"2021-05-26T12:57:40.718868Z","shell.execute_reply":"2021-05-26T12:57:40.783434Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

References

","metadata":{}},{"cell_type":"markdown","source":"- [Notebook] [Incredible TPUs - finetune EffNetB0-B6 at once](https://www.kaggle.com/agentauers/incredible-tpus-finetune-effnetb0-b6-at-once) \n- [Notebook] [Triple Stratified KFold with TFRecords](https://www.kaggle.com/cdeotte/triple-stratified-kfold-with-tfrecords) \n- [Notebook] [Rotation Augmentation GPU/TPU](https://www.kaggle.com/cdeotte/rotation-augmentation-gpu-tpu-0-96) \n- [Notebook] [SIIM-ISIC EfficientNet B6 Single Model](https://www.kaggle.com/roydatascience/siim-isic-efficientnet-b6-single-model-lb-0-9475) \n- [MSThesis] [Medical images analyses using neural networks](https://ela.kpi.ua/bitstream/123456789/38254/1/Doms_magistr.pdf) \n- [TF/Keras] [Keras Model](https://www.tensorflow.org/api_docs/python/tf/keras/Model) \n- [TF/Keras] [ModelCheckpoint](https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/ModelCheckpoint), [see also](https://keras.io/api/callbacks/model_checkpoint/) \n- [TF/Keras] [BinaryCrossentropy](https://www.tensorflow.org/api_docs/python/tf/keras/losses/BinaryCrossentropy) \n- [EfficientNet] [EfficientNet](https://github.com/qubvel/efficientnet) \n- [Article] Tan, M. and Le, Q.V. (2019). Efficientnet: rethinking model scaling for convolutional neural networks. arXiv preprint [arXiv:1905.11946](https://arxiv.org/pdf/1905.11946.pdf)\n- [Article] T. A. Putra, S. I. Rufaida and J. Leu, \"Enhanced Skin Condition Prediction Through Machine Learning Using Dynamic Training and Testing Augmentation,\" in IEEE Access, vol. 8, pp. 40536-40546, 2020, [doi: 10.1109/ACCESS.2020.2976045](https://ieeexplore.ieee.org/document/9007729)","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}