![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1723677%2F8db206436734b5f0c7b2179c2e152eb0%2FScreen%20Shot%202020-07-22%20at%201.28.49%20PM.png?generation=1595632223386869&alt=media\")
When using TensorFlow's tf.data.Dataset() we cannot use Albumenations to do our augmentation. Instead we must write our own functions in TensorFlow. Previously I posted how to perform (1) Rotation, Sheer, Zoom, Shift for GPU/TPU here and here (2) Cutmix and Mixup for GPU/TPU here
Coarse Dropout and Cutout augmentation are techniques to prevent overfitting and encourage generalization. They randomly remove rectangles from training images. By removing portions of the images, we challenge our models to pay attention to the entire image because it never knows what part of the image will be present. (This is similar and different to dropout layer within a CNN).
\n\nBy changing the parameters below, we can have either coarse dropout or cutout. (For cutout, you'll need to add tf.random.uniform for random size. I leave this as an exercise for the reader).
I've posted a starter notebook here showing how to do coarse dropout with TFRecords and tf.data.Dataset. The starter notebook also demonstrates upsampling (oversampling) the minority class.
The reason that this code looks overly complicated just to turn a portion of an array to zeros is because you cannot do image[ya:yb,xa:xb,:] = 0 in TensorFlow. TF does not allow index assignment. Instead we break an image into five pieces and then concatenate the five pieces while replacing the middle piece with all zeros.
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1723677%2F476c96c48b8b31969e00b55b26cef2f3%2Fcut.jpg?generation=1595632770269562&alt=media\")
def dropout(image, DIM=256, PROBABILITY = 0.75, CT = 8, SZ = 0.2):\n # input - one image of size [dim,dim,3] not a batch of [b,dim,dim,3]\n # output - image with CT squares of side size SZ*DIM removed\n\n # DO DROPOUT WITH PROBABILITY DEFINED ABOVE\n P = tf.cast( tf.random.uniform([],0,1) < PROBABILITY, tf.int32)\n if (P == 0)|(CT == 0)|(SZ == 0): return image\n\n for k in range( CT ):\n # CHOOSE RANDOM LOCATION\n x = tf.cast( tf.random.uniform([],0,DIM),tf.int32)\n y = tf.cast( tf.random.uniform([],0,DIM),tf.int32)\n # COMPUTE SQUARE \n WIDTH = tf.cast( SZ*DIM,tf.int32) * P\n ya = tf.math.maximum(0,y-WIDTH//2)\n yb = tf.math.minimum(DIM,y+WIDTH//2)\n xa = tf.math.maximum(0,x-WIDTH//2)\n xb = tf.math.minimum(DIM,x+WIDTH//2)\n # DROPOUT IMAGE\n one = image[ya:yb,0:xa,:]\n two = tf.zeros([yb-ya,xb-xa,3]) \n three = image[ya:yb,xb:DIM,:]\n middle = tf.concat([one,two,three],axis=1)\n image = tf.concat([image[0:ya,:,:],middle,image[yb:DIM,:,:]],axis=0)\n\n # RESHAPE HACK SO TPU COMPILER KNOWS SHAPE OF OUTPUT TENSOR \n image = tf.reshape(image,[DIM,DIM,3])\n return image\nWhen using TensorFlow's tf.data.Dataset() we cannot use Albumenations to do our augmentation. Instead we must write our own functions in TensorFlow. Previously I posted how to perform (1) Rotation, Sheer, Zoom, Shift for GPU/TPU here and here (2) Cutmix and Mixup for GPU/TPU here
Coarse Dropout and Cutout augmentation are techniques to prevent overfitting and encourage generalization. They randomly remove rectangles from training images. By removing portions of the images, we challenge our models to pay attention to the entire image because it never knows what part of the image will be present. (This is similar and different to dropout layer within a CNN).
\n\nBy changing the parameters below, we can have either coarse dropout or cutout. (For cutout, you'll need to add tf.random.uniform for random size. I leave this as an exercise for the reader).
I've posted a starter notebook here showing how to do coarse dropout with TFRecords and tf.data.Dataset. The starter notebook also demonstrates upsampling (oversampling) the minority class.
The reason that this code looks overly complicated just to turn a portion of an array to zeros is because you cannot do image[ya:yb,xa:xb,:] = 0 in TensorFlow. TF does not allow index assignment. Instead we break an image into five pieces and then concatenate the five pieces while replacing the middle piece with all zeros.
def dropout(image, DIM=256, PROBABILITY = 0.75, CT = 8, SZ = 0.2):\n # input - one image of size [dim,dim,3] not a batch of [b,dim,dim,3]\n # output - image with CT squares of side size SZ*DIM removed\n\n # DO DROPOUT WITH PROBABILITY DEFINED ABOVE\n P = tf.cast( tf.random.uniform([],0,1) < PROBABILITY, tf.int32)\n if (P == 0)|(CT == 0)|(SZ == 0): return image\n\n for k in range( CT ):\n # CHOOSE RANDOM LOCATION\n x = tf.cast( tf.random.uniform([],0,DIM),tf.int32)\n y = tf.cast( tf.random.uniform([],0,DIM),tf.int32)\n # COMPUTE SQUARE \n WIDTH = tf.cast( SZ*DIM,tf.int32) * P\n ya = tf.math.maximum(0,y-WIDTH//2)\n yb = tf.math.minimum(DIM,y+WIDTH//2)\n xa = tf.math.maximum(0,x-WIDTH//2)\n xb = tf.math.minimum(DIM,x+WIDTH//2)\n # DROPOUT IMAGE\n one = image[ya:yb,0:xa,:]\n two = tf.zeros([yb-ya,xb-xa,3]) \n three = image[ya:yb,xb:DIM,:]\n middle = tf.concat([one,two,three],axis=1)\n image = tf.concat([image[0:ya,:,:],middle,image[yb:DIM,:,:]],axis=0)\n\n # RESHAPE HACK SO TPU COMPILER KNOWS SHAPE OF OUTPUT TENSOR \n image = tf.reshape(image,[DIM,DIM,3])\n return image\nI've created a version that accepts batches of images, for anyone who is interested. I've tested it on a GPU setup, so I can't be sure this works on a TPU yet.
\n\nedit: I see that the prob arg isn't actually utilized yet, I'll fix this tommorow.\nedit 2: probabilities now included :)\nedit 3: replaced a type in tf.scatter_nd where the image sizes were fixed to 128,128 -> now adjusted for variable sizes
\n\n```python\ndef coarse_dropout(images, square_size, num_squares, prob):\n ''' Coarse image dropout \n Coarse dropout masks are created by using tf.scatter_nd\n Which can update matrices according to coordinates specified\n by the user.
\n\nParameters\n----------\nimages : tf Tensor\n A batch of images of size [BS, H, W, C] \nsquare_size : float\n A float between [0,1], a percentage of the image size\nnum_squares : int > 0\n The number of dropout squares per single image, must be >0\nprob : float\n A float between [0,1], probability that dropout is used for this batch\n\nYields\n------\nTYPE\n DESCRIPTION.\n\n'''\n # random dropout\nprob = tf.cast( tf.random.uniform([],0,1) < prob, tf.int32)\nif (prob == 0): return images\n\nimg_shape = images.shape\n_, h, w, c = img_shape[0], img_shape[1], img_shape[2], img_shape[3]\n\n# For some odd reason, the batch size is lost in the processing pipeline\n# seriously, how can it get lost ._., it literally says ds.batch()...\n# tensorflow shenanigans...\nbs = tf.cast(tf.reduce_sum(tf.ones_like(images)) / (images.shape[1] * images.shape[2] * images.shape[3]), tf.int32)\n\n\n# size of square in pixels (ssp), ASSUMING SQUARE IMAGES!\nssp = tf.cast(tf.math.ceil(h * square_size), tf.int32)\n\n# Create random start x coordinates\ncoords_x = tf.random.uniform([bs, num_squares], \n minval=0, \n maxval= (h - ssp), \n dtype=tf.int32) \n\n# Create ranges from start to the end of the line\n# [ssp, bs, num_squares]\ncoords_x = tf.linspace(coords_x, coords_x + ssp - 1, ssp)\n\n# [num_squares, bs, ssp]\ncoords_x = tf.cast(tf.transpose(coords_x), tf.int32)\n\n# Create random start y coordinates\ncoords_y = tf.random.uniform([bs, num_squares], \n minval=0, \n maxval= (h - ssp), \n dtype=tf.int32) \n\n# Create ranges from start to the end of the line\n# [ssp, bs, num_squares]\ncoords_y = tf.linspace(coords_y, coords_y + ssp - 1, ssp)\n\n# [num_squares, bs, ssp]\ncoords_y = tf.cast(tf.transpose(coords_y), tf.int32)\n\n\n# Create coordinate range combinations \n# and reshape to [bs, num_squares, 1, ssp * ssp]\ngrid_y = tf.reshape(tf.tile(coords_y, [1,1,ssp]), \n (bs, num_squares, 1, ssp * ssp))\n\n# and reshape to [bs, num_squares, ssp, ssp], transpose the inner matrices\ngrid_y = tf.transpose(tf.reshape(grid_y, \n (bs, num_squares, ssp, ssp)), \n (0, 1, 3, 2))\n\n# Repeat for x coordinates\ngrid_x = tf.reshape(tf.tile(coords_x, [1,1,ssp]), \n (bs, num_squares, 1, ssp * ssp))\n\ngrid_x = tf.reshape(grid_x, (bs, num_squares, ssp, ssp))\n\n# Stack the grids into a single matrix\n# grid is [2, bs, num_squares, ssp, ssp]\ngrid = tf.stack([grid_y, grid_x], axis=0)\n\n# Transpose and reshape [ bs, ssp * ssp * num_squares, 2] \n# Creates an array of 2D coordinates ([[x1,y1], [x2, y2], ..., [xn, yn]])\n# over all squares (num_squares), for each combination (ssp*ssp) of coordinates \n# and each batch (bs)\ngrid = tf.reshape(tf.transpose(grid, (1, 4, 3, 2, 0)), \n (bs, ssp * ssp * num_squares, 2))\n\n# [bs, sz*sz*num_squares, 2]\n#grid = tf.reshape(grid, (bs, ssp*ssp*num_squares, 2))\n\n# create batch indices [0,..., bs] and reshape to [ssp * ssp * num_squares, bs]\nbatch_indices = tf.reshape(tf.tile(tf.range(0, bs), \n [ssp * ssp * num_squares]), \n (ssp * ssp * num_squares, bs))\n\n# Transpose to get the right order, and reshape to match grid shape\nbatch_indices = tf.reshape(tf.transpose(batch_indices), \n (bs, ssp * ssp * num_squares, 1))\n\n# concatenate batch indices with the grid\n# these yield 3D coordinates like e.g.\n# [[bs0, x1, y1], [bs0, x2, y1], ..., [bsn, xn, yn]]\ngrid = tf.concat([batch_indices, grid], axis=2)\n\n# create a matrix of zeros, and update the matrix with the grid indices\n# this essentially creates a mask with coarse dropout squares\nmasks = tf.scatter_nd(grid[tf.newaxis,...], \n tf.ones([1,bs,ssp*ssp*num_squares]) * -1, \n shape=(bs, h, w)) +1\n\n# Due to overlap of squares, some get coordinates get updated twice\n# and result in values < -1, clip these values\nmasks = tf.clip_by_value(masks, 0, 1)\n\nreturn images * masks[..., tf.newaxis] \n```
", "rawMarkdown": "I've created a version that accepts batches of images, for anyone who is interested. I've tested it on a GPU setup, so I can't be sure this works on a TPU yet.\n\nedit: I see that the prob arg isn't actually utilized yet, I'll fix this tommorow.\nedit 2: probabilities now included :)\nedit 3: replaced a type in tf.scatter_nd where the image sizes were fixed to 128,128 -> now adjusted for variable sizes\n\n```python\ndef coarse_dropout(images, square_size, num_squares, prob):\n ''' Coarse image dropout \n Coarse dropout masks are created by using tf.scatter_nd\n Which can update matrices according to coordinates specified\n by the user. \n\n Parameters\n ----------\n images : tf Tensor\n A batch of images of size [BS, H, W, C] \n square_size : float\n A float between [0,1], a percentage of the image size\n num_squares : int > 0\n The number of dropout squares per single image, must be >0\n prob : float\n A float between [0,1], probability that dropout is used for this batch\n\n Yields\n ------\n TYPE\n DESCRIPTION.\n\n '''\n # random dropout\n prob = tf.cast( tf.random.uniform([],0,1) < prob, tf.int32)\n if (prob == 0): return images\n\n img_shape = images.shape\n _, h, w, c = img_shape[0], img_shape[1], img_shape[2], img_shape[3]\n \n # For some odd reason, the batch size is lost in the processing pipeline\n # seriously, how can it get lost ._., it literally says ds.batch()...\n # tensorflow shenanigans...\n bs = tf.cast(tf.reduce_sum(tf.ones_like(images)) / (images.shape[1] * images.shape[2] * images.shape[3]), tf.int32)\n\n\n # size of square in pixels (ssp), ASSUMING SQUARE IMAGES!\n ssp = tf.cast(tf.math.ceil(h * square_size), tf.int32)\n\n # Create random start x coordinates\n coords_x = tf.random.uniform([bs, num_squares], \n minval=0, \n maxval= (h - ssp), \n dtype=tf.int32) \n\n # Create ranges from start to the end of the line\n # [ssp, bs, num_squares]\n coords_x = tf.linspace(coords_x, coords_x + ssp - 1, ssp)\n\n # [num_squares, bs, ssp]\n coords_x = tf.cast(tf.transpose(coords_x), tf.int32)\n\n # Create random start y coordinates\n coords_y = tf.random.uniform([bs, num_squares], \n minval=0, \n maxval= (h - ssp), \n dtype=tf.int32) \n\n # Create ranges from start to the end of the line\n # [ssp, bs, num_squares]\n coords_y = tf.linspace(coords_y, coords_y + ssp - 1, ssp)\n\n # [num_squares, bs, ssp]\n coords_y = tf.cast(tf.transpose(coords_y), tf.int32)\n\n\n # Create coordinate range combinations \n # and reshape to [bs, num_squares, 1, ssp * ssp]\n grid_y = tf.reshape(tf.tile(coords_y, [1,1,ssp]), \n (bs, num_squares, 1, ssp * ssp))\n\n # and reshape to [bs, num_squares, ssp, ssp], transpose the inner matrices\n grid_y = tf.transpose(tf.reshape(grid_y, \n (bs, num_squares, ssp, ssp)), \n (0, 1, 3, 2))\n\n # Repeat for x coordinates\n grid_x = tf.reshape(tf.tile(coords_x, [1,1,ssp]), \n (bs, num_squares, 1, ssp * ssp))\n\n grid_x = tf.reshape(grid_x, (bs, num_squares, ssp, ssp))\n\n # Stack the grids into a single matrix\n # grid is [2, bs, num_squares, ssp, ssp]\n grid = tf.stack([grid_y, grid_x], axis=0)\n\n # Transpose and reshape [ bs, ssp * ssp * num_squares, 2] \n # Creates an array of 2D coordinates ([[x1,y1], [x2, y2], ..., [xn, yn]])\n # over all squares (num_squares), for each combination (ssp*ssp) of coordinates \n # and each batch (bs)\n grid = tf.reshape(tf.transpose(grid, (1, 4, 3, 2, 0)), \n (bs, ssp * ssp * num_squares, 2))\n\n # [bs, sz*sz*num_squares, 2]\n #grid = tf.reshape(grid, (bs, ssp*ssp*num_squares, 2))\n\n # create batch indices [0,..., bs] and reshape to [ssp * ssp * num_squares, bs]\n batch_indices = tf.reshape(tf.tile(tf.range(0, bs), \n [ssp * ssp * num_squares]), \n (ssp * ssp * num_squares, bs))\n\n # Transpose to get the right order, and reshape to match grid shape\n batch_indices = tf.reshape(tf.transpose(batch_indices), \n (bs, ssp * ssp * num_squares, 1))\n\n # concatenate batch indices with the grid\n # these yield 3D coordinates like e.g.\n # [[bs0, x1, y1], [bs0, x2, y1], ..., [bsn, xn, yn]]\n grid = tf.concat([batch_indices, grid], axis=2)\n\n # create a matrix of zeros, and update the matrix with the grid indices\n # this essentially creates a mask with coarse dropout squares\n masks = tf.scatter_nd(grid[tf.newaxis,...], \n tf.ones([1,bs,ssp*ssp*num_squares]) * -1, \n shape=(bs, h, w)) +1\n \n # Due to overlap of squares, some get coordinates get updated twice\n # and result in values < -1, clip these values\n masks = tf.clip_by_value(masks, 0, 1)\n\n return images * masks[..., tf.newaxis] \n```", "votes": 7, "replies": [ { "id": 954270, "postDate": "2020-08-01T15:24:34.653Z", "content": "Thanks @stephanovich ! when i get time i will play with this. I'm curious to benchmark the speed and see if this is faster than \"per image\".
\n\nI've noticed when using \"per image\", if we cutout more than 16 squares it will begin to slow down training, perhaps your \"per batch\" code can do more than 16 and not slow down training.
", "rawMarkdown": "Thanks @stephanovich ! when i get time i will play with this. I'm curious to benchmark the speed and see if this is faster than \"per image\".\n\nI've noticed when using \"per image\", if we cutout more than 16 squares it will begin to slow down training, perhaps your \"per batch\" code can do more than 16 and not slow down training.", "votes": 1 }, { "id": 955084, "postDate": "2020-08-02T10:17:17.990Z", "content": "Edit: you might want to try the augmentation from this discussion and notebook, it seems to be a more generalized version of coarse dropout, and does not use for loops either (i.e. vectorized).
\n\nSo I ran a small benchmark myself on a cpu with 32 threads, here's the results:\nThe experiments are run on 128*128 images with a variable number of squares each and a square size of 0.1 * the size of the image. The probability is set to one in both cases. I parsed through 60000 examples, which in my opinion was already enough to at least see a difference.\nI've included 3 runs: \n1. one without any sort of augmentation, which acts as the lower bound (or golden standard) for time benchmarking (it can never go faster than this).\npython\nCPU times: user 11min 27s, sys: 30.9 s, total: 11min 58s\nWall time: 29.2 s\n
2 . The new batch version. This version first prepares the image, then batches the images, and then augments them. Otherwise, we would not see any improvements.\n``` python
\n\nCPU times: user 28min 6s, sys: 1min 38s, total: 29min 44s\nWall time: 1min 12s
\n\nCPU times: user 33min, sys: 1min 36s, total: 34min 36s\nWall time: 1min 17s
\n\n```
\n\n3 . Your version. In the processing pipeline, the images are first prepared through prepare_image, then are augmented, and finally batched.\n```python
\n\nCPU times: user 50min 36s, sys: 2min 37s, total: 53min 14s\nWall time: 2min 4s
\n\nCPU times: user 1h 15min 50s, sys: 2min 41s, total: 1h 18min 32s\nWall time: 2min 48s\n```
\n\nIt seems the batch version does run faster, at the cost of some higher memory footprint (due to vectorization)
\n\nOf course, the times can decrease/increase depending on the image sizes, number of squares which you already mentioned above, and possibly other factors.
", "rawMarkdown": "Edit: you might want to try the augmentation from this [discussion](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/171778) and [notebook](https://www.kaggle.com/benboren/tfrecord-progressive-sprinkles), it seems to be a more generalized version of coarse dropout, and does not use for loops either (i.e. vectorized).\n\nSo I ran a small benchmark myself on a cpu with 32 threads, here's the results:\nThe experiments are run on 128*128 images with a variable number of squares each and a square size of 0.1 * the size of the image. The probability is set to one in both cases. I parsed through 60000 examples, which in my opinion was already enough to at least see a difference.\nI've included 3 runs: \n1. one without any sort of augmentation, which acts as the lower bound (or golden standard) for time benchmarking (it can never go faster than this).\n```python\nCPU times: user 11min 27s, sys: 30.9 s, total: 11min 58s\nWall time: 29.2 s\n```\n\n2 . The new batch version. This version first prepares the image, then batches the images, and then augments them. Otherwise, we would not see any improvements.\n``` python\n\n# Using 8 squares\nCPU times: user 28min 6s, sys: 1min 38s, total: 29min 44s\nWall time: 1min 12s\n\n# Using 16 squares\nCPU times: user 33min, sys: 1min 36s, total: 34min 36s\nWall time: 1min 17s\n\n```\n\n3 . Your version. In the processing pipeline, the images are first prepared through prepare_image, then are augmented, and finally batched.\n```python\n\n# Using 8 squares\nCPU times: user 50min 36s, sys: 2min 37s, total: 53min 14s\nWall time: 2min 4s\n\n# Using 16 squares\nCPU times: user 1h 15min 50s, sys: 2min 41s, total: 1h 18min 32s\nWall time: 2min 48s\n```\n\n\nIt seems the batch version does run faster, at the cost of some higher memory footprint (due to vectorization)\n\nOf course, the times can decrease/increase depending on the image sizes, number of squares which you already mentioned above, and possibly other factors.", "votes": 4 } ] }, { "id": 945580, "postDate": "2020-07-26T01:30:08.590Z", "content": "Notice how experiment 1 without dropout has a larger gap between train AUC and validation AUC. While experiment 2 with dropout has a smaller gap. In the below example, dropout has helped the CNN generalize thus increasing validation AUC while preventing training AUC from overfitting (i.e. reducing training AUC). Notebook here
\n\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1723677%2F731ced48b76769659f1358d253f137af%2FScreen%20Shot%202020-07-25%20at%206.20.23%20PM.png?generation=1595726783638841&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1723677%2Ff8547390e3c0c420be6336ed92f44dae%2FScreen%20Shot%202020-07-25%20at%206.20.58%20PM.png?generation=1595726813782566&alt=media\")
This is really useful for tensorflow users in this competition @cdeotte :D
", "rawMarkdown": "This is really useful for tensorflow users in this competition @cdeotte :D ", "votes": 3 }, { "id": 944692, "postDate": "2020-07-25T09:00:35.247Z", "content": "UPDATE: If your forked version 15 or earlier of the starter notebook, note that it does not apply dropout. (It only applies upsample). During training, in the call to get_dataset(), I forgot to pass the dropout parameters (so they use the default of zero). In the 3 lines below, the last line was missing in notebook version 15 and earlier. I just committed notebook now. Versions 16 onward will have this line added and will apply dropout.
get_dataset(files_train, augment=True, shuffle=True, repeat=True,\n dim=IMG_SIZES[fold], batch_size = BATCH_SIZES[fold],\n droprate = DROP_FREQ[fold], dropct = DROP_CT[fold], dropsize = DROP_SIZE[fold])\nWe also need to add this last line to validation TTA prediction and test TTA prediction.
", "rawMarkdown": "UPDATE: If your forked version 15 or earlier of the starter notebook, note that it does **not** apply **dropout**. (It only applies **upsample**). During training, in the call to `get_dataset()`, I forgot to pass the dropout parameters (so they use the default of zero). In the 3 lines below, the last line was missing in notebook version 15 and earlier. I just committed notebook now. Versions 16 onward will have this line added and **will apply** dropout.\n\n get_dataset(files_train, augment=True, shuffle=True, repeat=True,\n dim=IMG_SIZES[fold], batch_size = BATCH_SIZES[fold],\n droprate = DROP_FREQ[fold], dropct = DROP_CT[fold], dropsize = DROP_SIZE[fold])\n\nWe also need to add this last line to validation TTA prediction and test TTA prediction.", "votes": 3 }, { "id": 954351, "postDate": "2020-08-01T16:46:06.593Z", "content": "@cdeotte Thank you for your great work! Please allow me one question.\nHow do you think about the difference between Coarse Dropout and GridMask ?
", "rawMarkdown": "@cdeotte Thank you for your great work! Please allow me one question.\nHow do you think about the difference between Coarse Dropout and GridMask ?", "votes": 1, "replies": [ { "id": 954356, "postDate": "2020-08-01T16:52:39.810Z", "content": "I have never used GridMask before. There is an implementation of GridMask for TensorFlow here by Xie29.
\n\n(Note: I have not tested Xie29's implementation. If you use it, i would compare training with and without it to make sure the code is efficient and doesn't slow down training).
", "rawMarkdown": "I have never used GridMask before. There is an implementation of GridMask for TensorFlow [here][1] by Xie29. \n\n(Note: I have not tested Xie29's implementation. If you use it, i would compare training **with** and **without** it to make sure the code is efficient and doesn't slow down training).\n\n[1]: https://www.kaggle.com/c/flower-classification-with-tpus/discussion/132986", "votes": 1 }, { "id": 954362, "postDate": "2020-08-01T17:05:21.667Z", "content": "Thank you for your quick reply and suggestion. I plan to try both of them.\nAgain, I appreciate your nice works!
", "rawMarkdown": "Thank you for your quick reply and suggestion. I plan to try both of them.\nAgain, I appreciate your nice works!", "votes": 1 } ] }, { "id": 952933, "postDate": "2020-07-31T10:34:42.307Z", "content": "@cdeotte Hi Chris! I really would like to try to find ensembling weights by using a machine learning algorithm and cv oof auc scores. Did you ever try this? Do you have an url or tip how to do this?\nBest Roman
", "rawMarkdown": "@cdeotte Hi Chris! I really would like to try to find ensembling weights by using a machine learning algorithm and cv oof auc scores. Did you ever try this? Do you have an url or tip how to do this?\nBest Roman", "votes": 1, "replies": [ { "id": 953123, "postDate": "2020-07-31T14:47:45.470Z", "content": "This is what all the top competitors do in every competition. This is the correct way to ensemble. (The incorrect way is trial and error using feedback from public LB because public test set contains less data than train set).
\n\nYou train multiple models using the same KFold folds and save all your OOF. Then you find weights with the OOF. For example if you have 3 models, then you find w1, w2, w3 with w1+w2+w3 = 1 where you maximize AUC below
oof = w1*oof1 + w2*oof2 + w3*oof3\nAUC = roc_auc_score(TRUE, oof)\nLastly you use these weights for your Kaggle submission as in
\n\npred = w1*sub1 + w2*sub2 + w3*sub3\nsub = pd.DataFrame(dict(image_name=NAMES, target=PRED))\nsub.to_csv('submission.csv',index=False)\nThe simplest thing is to (1) find these weights by hand or grid search (i.e. with nested for-loops). Or (2) you can find these weights by fitting a linear or logistic regression model. (Since linear and logistic regression are simple models, you don't need holdout set, just fit_transform all the oof against true). Or (3) if you want to use a more complex stage 2 model like XGB, you need to set up a holdout set.
Hello, @cdeotte thanks for the above pointer. I was trying to set weight values based on the CV score rather than LB feedback with different trails and errors. However, I totally agree with @romanweilguny that we should use ML algorithm to find an optimal weight for the ensemble.
\nLater I've tried to use the following method to find optimal weight, though few things are yet not clear to me, working on it. Do you have any suggestions for that? Basically we've kinda followed the above procedure that you've mentioned, the aim was to get max(roc_auc_score(TRUE, oof)) after some iteration.
Thank you -)
", "rawMarkdown": "Hello, @cdeotte thanks for the above pointer. I was trying to set weight values based on the CV score rather than LB feedback with different trails and errors. However, I totally agree with @romanweilguny that we should use ML algorithm to find an optimal weight for the ensemble. \n\nLater I've tried to use [the following method](https://www.kaggle.com/ipythonx/efficientnet-b6-oof-weights-finder-seed-42) to find optimal weight, though few things are yet not clear to me, working on it. Do you have any suggestions for that? Basically we've kinda followed the above procedure that you've mentioned, the aim was to get `max(roc_auc_score(TRUE, oof))` after some iteration. \n\nThank you -)" }, { "id": 967258, "postDate": "2020-08-12T05:53:37.050Z", "content": "I think an ML approach would be interesting. Not sure how fast the compute is you have access to, but say you had 4 models you wanted to ensemble, you could brute force the weights with 10,000 iterations, which can totally be parallelized across cores, if you used values of .00, .10, .20, etc through .90.
", "rawMarkdown": "I think an ML approach would be interesting. Not sure how fast the compute is you have access to, but say you had 4 models you wanted to ensemble, you could brute force the weights with 10,000 iterations, which can totally be parallelized across cores, if you used values of .00, .10, .20, etc through .90." } ] }, { "id": 953320, "postDate": "2020-07-31T17:36:33.090Z", "content": "Thx - thats what I have been looking for!
", "rawMarkdown": "Thx - thats what I have been looking for!", "votes": 1 }, { "id": 954193, "postDate": "2020-08-01T13:33:51.620Z", "content": "I have noticed in your notebooks you select the best model in a fold based on the lowest loss.Would it be wrong to select the best model using max AUC, even if there is 0.3-0.5 difference between the lowest loss and the max AUC epoch's loss? Or is it important to select a model based on loss
", "rawMarkdown": "I have noticed in your notebooks you select the best model in a fold based on the lowest loss.Would it be wrong to select the best model using max AUC, even if there is 0.3-0.5 difference between the lowest loss and the max AUC epoch's loss? Or is it important to select a model based on loss" }, { "id": 954268, "postDate": "2020-08-01T15:20:30.477Z", "content": "There are basically 3 options.\n* select lowest val loss\n* select highest val auc\n* select after fixed number of epochs, like 12th each time
\n\nThere are pros and cons for each. The resultant LB AUC will be some random number added or subtracted to each Val AUC, so any of these 3 choices could produce the best LB AUC. We cannot know which is the best choice. Some people will even take all 3 and ensemble the 3 models (but that's not necessarily better either).
\n\nIn conclusion, i'm saying experiment will all, and do what you prefer. All will work well.
", "rawMarkdown": "There are basically 3 options.\n* select lowest val loss\n* select highest val auc\n* select after fixed number of epochs, like 12th each time\n\nThere are pros and cons for each. The resultant LB AUC will be some random number added or subtracted to each Val AUC, so any of these 3 choices could produce the best LB AUC. We cannot know which is the best choice. Some people will even take all 3 and ensemble the 3 models (but that's not necessarily better either).\n\nIn conclusion, i'm saying experiment will all, and do what you prefer. All will work well.", "votes": 1 }, { "id": 959717, "postDate": "2020-08-05T20:55:43.773Z", "content": "@cdeotte Hi Chris! Do you use OOF with TTA or without TTA, selecting weights?
\n\n\n", "rawMarkdown": "@cdeotte Hi Chris! Do you use OOF with TTA or without TTA, selecting weights? \n\n> You train multiple models using the same KFold folds and save all your OOF. Then you find weights with the OOF. For example if you have 3 models, then you find w1, w2, w3 with w1+w2+w3 = 1 where you maximize AUC below" }, { "id": 967089, "postDate": "2020-08-11T23:12:53.580Z", "content": "You train multiple models using the same KFold folds and save all your OOF. Then you find weights with the OOF. For example if you have 3 models, then you find w1, w2, w3 with w1+w2+w3 = 1 where you maximize AUC below
\n
You should the same thing that was applied to your submission.csv. If your submission.csv has TTA applied then use OOF with TTA applied.
@cdeotte a simple question: Do you have any ratio between image size and rest of the droprate, dropct and dropsize or you keep them same for all? Thanks for your great sources for this competition again...
", "rawMarkdown": "@cdeotte a simple question: Do you have any ratio between image size and rest of the droprate, dropct and dropsize or you keep them same for all? Thanks for your great sources for this competition again...", "votes": 1, "replies": [ { "id": 952231, "postDate": "2020-07-30T18:12:41.463Z", "content": "Good question. DROP_SIZE is a ratio between 0 and 1. So it scales your squares with your image size. (Square side to cutout out is IMAGE_SIZE * DROP_SIZE). Likewise DROP_RATE and DROP_CT naturally scale. Therefore once you find good numbers, you can use the same numbers for all image sizes.
Here's an important note. The variable DROP_CT removes CT squares using a for-loop. I recommend using 16 or less and preferably 8 or less. If you want to remove more image, increase DROP_RATE and/or DROP_SIZE.
If you turn this augmentation on and off, you will find that if DROP_CT>=16 then it starts to slightly slow down your training time per epoch because the CPU can't augment (execute for-loop) fast enough (when doing all the other common stuff like rotation, sheer, zoom, etc etc).
So for example if you set DROP_FREQ = 1.0, DROP_CT = 8, and DROP_SIZE = 0.2 then regardless of image size, it probably removes about 20% of every image. (If the squares did not overlap it would be 25%).
Thank you for the answer! I wanted to play with your approach today, implemented it on my model and some feedback here maybe you find them useful:
\n\nIn my case I didn't feel slowing down up to 16 CT's maybe 5% longer per epoch or something at high numbers, didn't go over 16 though.
\n\nBut it definitely slowed down my models learning process with increased randomness, had to increase epochs like 30% to reach dropout off levels of train auc.
\n\nHaven't played enough to comment about cv, lb yet... But anyways thank you again! @cdeotte
", "rawMarkdown": "Thank you for the answer! I wanted to play with your approach today, implemented it on my model and some feedback here maybe you find them useful:\n\nIn my case I didn't feel slowing down up to 16 CT's maybe 5% longer per epoch or something at high numbers, didn't go over 16 though.\n\nBut it definitely slowed down my models learning process with increased randomness, had to increase epochs like 30% to reach dropout off levels of train auc.\n\nHaven't played enough to comment about cv, lb yet... But anyways thank you again! @cdeotte ", "votes": 1 }, { "id": 952284, "postDate": "2020-07-30T19:23:31.897Z", "content": "Yes using image dropout or layer dropout means that you have to train for more epochs. The idea is that your model is learning slower but it will achieve a higher level of intelligence.
", "rawMarkdown": "Yes using image dropout or layer dropout means that you have to train for more epochs. The idea is that your model is learning slower but it will achieve a higher level of intelligence.", "votes": 1 }, { "id": 952613, "postDate": "2020-07-31T04:55:47.803Z", "content": "Is there any method to speed up the for loop process. It seems to waste the incredible capabilities of TPU while doing for loop.
", "rawMarkdown": "Is there any method to speed up the for loop process. It seems to waste the incredible capabilities of TPU while doing for loop. ", "votes": 1 }, { "id": 952631, "postDate": "2020-07-31T05:17:12.977Z", "content": "It does not waste TPU time. All augmentation of tf.data.Dataset happens on the TPU's CPU in parallel. So while the TPU is training one batch, the CPU is preparing the next batch. As long as DROP_CT is low enough it doesn't change train time at all (because the CPU batch time is less than the TPU batch time).
Good post Chris!
", "rawMarkdown": "Good post Chris!", "votes": 1 }, { "id": 945624, "postDate": "2020-07-26T03:19:34.997Z", "content": "In my understanding, we should not use dropout or cutout when predicting validation data or test data. Am I right?
", "rawMarkdown": "In my understanding, we should not use dropout or cutout when predicting validation data or test data. Am I right?", "votes": 1, "replies": [ { "id": 945695, "postDate": "2020-07-26T04:53:30.620Z", "content": "The best answer is try both and see what produces higher CV LB. Since we're using TTA in both our validation and test, I think it may be best to add dropout and/or cutout to the TTA.
\n\nI will point out that simply removing dropout or cutout from predicting test or validation may not work. When you remove a dropout layer inside a CNN, you need to rescale the activations to adjust for the missing dropout. I'm not sure how you would \"adjust\" the validation or test predictions if we train with coarse dropout and then don't use coarse dropout for prediction.
", "rawMarkdown": "The best answer is try both and see what produces higher CV LB. Since we're using TTA in both our validation and test, I think it may be best to add dropout and/or cutout to the TTA.\n\nI will point out that simply removing dropout or cutout from predicting test or validation may not work. When you remove a dropout layer inside a CNN, you need to rescale the activations to adjust for the missing dropout. I'm not sure how you would \"adjust\" the validation or test predictions if we train with coarse dropout and then don't use coarse dropout for prediction.", "votes": 2 }, { "id": 946158, "postDate": "2020-07-26T11:59:09.063Z", "content": "Thanks for your reply! I did a few experiments (I know it is not sufficient.) and found that as you said, removing dropout or cutout from prediction is worse. I'll report more reliable results asap!
", "rawMarkdown": "Thanks for your reply! I did a few experiments (I know it is not sufficient.) and found that as you said, removing dropout or cutout from prediction is worse. I'll report more reliable results asap! ", "votes": 1 }, { "id": 946176, "postDate": "2020-07-26T12:18:17.910Z", "content": "Though I can understand the importance of TTA, I can't understand why using dropout works with validation or test data. My concern is that using these augmentations may lose important information.
", "rawMarkdown": "Though I can understand the importance of TTA, I can't understand why using dropout works with validation or test data. My concern is that using these augmentations may lose important information. ", "votes": 2 }, { "id": 946373, "postDate": "2020-07-26T14:41:27.497Z", "content": "Keep in mind that the CNN is being trained with dropout. (And is smart enough to find malignant even with rectangles of images removed). If during TTA prediction there is no dropout, then the CNN could get confused.
\n\nThat's like teaching a tennis player to use a solid wood racket for months and during the tournament giving the tennis player a real racket with strings. You would think it would help, but the tennis player never learned how to use the real racket with strings.
", "rawMarkdown": "Keep in mind that the CNN is being trained with dropout. (And is smart enough to find malignant even with rectangles of images removed). If during TTA prediction there is no dropout, then the CNN could get confused.\n\nThat's like teaching a tennis player to use a solid wood racket for months and during the tournament giving the tennis player a real racket with strings. You would think it would help, but the tennis player never learned how to use the real racket with strings.", "votes": 3 }, { "id": 946457, "postDate": "2020-07-26T15:34:31.377Z", "content": "@cdeotte This is the first time I see someone using random cutout in TTA to be honest. I will try it, interesting idea. Usually it works well to train with cutout and don't do cutout in inference, it behaves very similar to dropout which you also dont use in inference.
\n\nWhat it definitely does it that it adds randomness, but having randomness in inference can be risky. Random TTA is already something I dont feel comfortable with.
", "rawMarkdown": "@cdeotte This is the first time I see someone using random cutout in TTA to be honest. I will try it, interesting idea. Usually it works well to train with cutout and don't do cutout in inference, it behaves very similar to dropout which you also dont use in inference. \n\nWhat it definitely does it that it adds randomness, but having randomness in inference can be risky. Random TTA is already something I dont feel comfortable with." }, { "id": 946461, "postDate": "2020-07-26T15:38:32.620Z", "content": "For TTA (test time augmentation), isn't it best to use the augmentation that your network was trained with?
\n\nIf we remove image dropout from TTA, should we adjust predictions? Because if you add a dropout layer in your CNN like x = Dropout(0.2)(x). Then when the network makes predictions it removes the layer dropout and it reduces the activations by 20%. (i.e. during inference Keras replaces layer dropout with x = Lambda(x: 0.8*x)(x)) What is the equivalent of \"reducing activations by 20%\" when using image dropout?
Doesn't the scaling happen during training and prediction is identity (is similar to what you say)? But you are right, haven't thought much about that when doing cutout. In practise I havent seen any issues with not doing cutout in inference, but also have never tried to keep it.
", "rawMarkdown": "Doesn't the scaling happen during training and prediction is identity (is similar to what you say)? But you are right, haven't thought much about that when doing cutout. In practise I havent seen any issues with not doing cutout in inference, but also have never tried to keep it." }, { "id": 946467, "postDate": "2020-07-26T15:45:35.240Z", "content": "\n\n\n\n\nRandom TTA is already something I dont feel comfortable with.
\n
This competition takes TTA to a whole new level. People are using 25+ TTA steps. At that point, randomness gets averaged away.
", "rawMarkdown": "@philippsinger \n> Random TTA is already something I dont feel comfortable with.\n\nThis competition takes TTA to a whole new level. People are using 25+ TTA steps. At that point, randomness gets averaged away." }, { "id": 946474, "postDate": "2020-07-26T15:50:05.400Z", "content": "@cdeotte Havent tested tta yet, but if tta outside of standard flips works this can be an approach indeed. Inference is fast.
\n\nDo you just use train augs multiple times, or limit them?
", "rawMarkdown": "@cdeotte Havent tested tta yet, but if tta outside of standard flips works this can be an approach indeed. Inference is fast.\n\nDo you just use train augs multiple times, or limit them?" }, { "id": 946481, "postDate": "2020-07-26T15:51:27.990Z", "content": "\n\n\nDoesn't the scaling happen during training and prediction is identity
\n
Oh yes, i think you're right. A layer of x = dropout(0.2)(x) simultaneously applies 20% dropout and increases activations by 25% internally with something like x = Lambda(x: 1.25*x)(x). Then during inference it just removes the dropout layer and all is well.
For TTA, i use all the training augmentations! I use rotation, sheer, scale, shift, contrast, saturation, brightness, flip, dropout, etc, etc. Using TTA 25+ is the powerhouse in this comp. It increases CV LB by something like 0.01+ or 0.02+!
\n\n print('Predicting Test with TTA...')\n ds_test = get_dataset(files_test,labeled=False,return_image_names=False,augment=True,\n repeat=True,shuffle=False,dim=IMG_SIZES[fold],batch_size=BATCH_SIZE)\n ct_test = count_data_items(files_test); STEPS = TTA * ct_test/BATCH_SIZE\n pred = model.predict(ds_test,steps=STEPS,verbose=VERBOSE)[:TTA*ct_test,] \n preds += np.mean(pred.reshape((ct_test,TTA),order='F'),axis=1) / skf.n_splits\nAnd get_dataset(augment=True) is the same pipeline that train uses.
The power of TTA was first shown by AgentAuers' in his notebook here
", "rawMarkdown": "For TTA, i use all the training augmentations! I use rotation, sheer, scale, shift, contrast, saturation, brightness, flip, dropout, etc, etc. Using TTA 25+ is the powerhouse in this comp. It increases CV LB by something like 0.01+ or 0.02+!\n\n print('Predicting Test with TTA...')\n ds_test = get_dataset(files_test,labeled=False,return_image_names=False,augment=True,\n repeat=True,shuffle=False,dim=IMG_SIZES[fold],batch_size=BATCH_SIZE)\n ct_test = count_data_items(files_test); STEPS = TTA * ct_test/BATCH_SIZE\n pred = model.predict(ds_test,steps=STEPS,verbose=VERBOSE)[:TTA*ct_test,] \n preds += np.mean(pred.reshape((ct_test,TTA),order='F'),axis=1) / skf.n_splits\n\nAnd `get_dataset(augment=True)` is the same pipeline that train uses.\n\nThe power of TTA was first shown by AgentAuers' in his notebook [here][1]\n\n[1]: https://www.kaggle.com/agentauers/incredible-tpus-finetune-effnetb0-b6-at-once", "votes": 1 }, { "id": 946501, "postDate": "2020-07-26T16:00:57.423Z", "content": "Thank you, interesting, will try it.
", "rawMarkdown": "Thank you, interesting, will try it.", "votes": 1 }, { "id": 946509, "postDate": "2020-07-26T16:09:43.877Z", "content": "I removed Cutout during my TTA.
\n\nBut I will try it to see if it improves.
", "rawMarkdown": "I removed Cutout during my TTA. \n\nBut I will try it to see if it improves. ", "votes": 1 }, { "id": 946531, "postDate": "2020-07-26T16:21:54.737Z", "content": "Yes @serigne , let us know. I have only recently added cutout to my augmentations. So I haven't done enough experiments myself to know whether it is best to include cutout or not during TTA inference.
", "rawMarkdown": "Yes @serigne , let us know. I have only recently added cutout to my augmentations. So I haven't done enough experiments myself to know whether it is best to include cutout or not during TTA inference.", "votes": 1 }, { "id": 946549, "postDate": "2020-07-26T16:36:20.617Z", "content": "@cdeotte Cutout definitely improved my training model. But, as @philippsinger , I was much reluctant to use Random Augmentation (specially those cutting the input images) during TTA.
\n\nBut, after @optimo suggestion , I gave a try to +15 TTA (with all my training augs except Cutout) and saw relatively huge improvement on LB. Next step is to add Cutout and let you know.
", "rawMarkdown": "@cdeotte Cutout definitely improved my training model. But, as @philippsinger , I was much reluctant to use Random Augmentation (specially those cutting the input images) during TTA. \n\nBut, after @optimo suggestion , I gave a try to +15 TTA (with all my training augs except Cutout) and saw relatively huge improvement on LB. Next step is to add Cutout and let you know. ", "votes": 1 }, { "id": 946560, "postDate": "2020-07-26T16:41:15.190Z", "content": "@serigne Does it also improve CV?
", "rawMarkdown": "@serigne Does it also improve CV?" }, { "id": 946602, "postDate": "2020-07-26T17:10:28.570Z", "content": "@philippsinger for me it improves CV by about 0.01.
\n\nTo be honest I started with a lazy implementation, I was using coarse dropout and I left it as is in my TTA, but I was planning to remove it at some point. I agree that I would expect better results without masking part of my input during inference (I guess everyone would agree not to do mixcut during inference for example). It might also depend on the size of your dropout patches, if they are too big, you may be able to train your model but using it for inference would hurt your results.
\n\nBut I feel very comfortable with doing 20x TTA or even 100x TTA (even though I have better things to do haha), for me it would just converge to the best possible prediction for my model, it might be useless at some point but can't harm much.
", "rawMarkdown": "@philippsinger for me it improves CV by about 0.01.\n\nTo be honest I started with a lazy implementation, I was using coarse dropout and I left it as is in my TTA, but I was planning to remove it at some point. I agree that I would expect better results without masking part of my input during inference (I guess everyone would agree not to do mixcut during inference for example). It might also depend on the size of your dropout patches, if they are too big, you may be able to train your model but using it for inference would hurt your results. \n\nBut I feel very comfortable with doing 20x TTA or even 100x TTA (even though I have better things to do haha), for me it would just converge to the best possible prediction for my model, it might be useless at some point but can't harm much. ", "votes": 1 }, { "id": 946604, "postDate": "2020-07-26T17:12:03.947Z", "content": "Oups I was eager to try it on test data but I didn't even check on validation data ( I used to try Random Augs only on training data so far)
\n\nBut you're right. These augmentations need to be checked on validation data first. I will try to do it more systematically from now.
", "rawMarkdown": "Oups I was eager to try it on test data but I didn't even check on validation data ( I used to try Random Augs only on training data so far)\n\nBut you're right. These augmentations need to be checked on validation data first. I will try to do it more systematically from now. ", "votes": 2 }, { "id": 947551, "postDate": "2020-07-27T10:40:42.523Z", "content": "@cdeotte You talk about using TTA in both your validation as well as test. I have used TTA in test, but not tried it in validation and will give it a try. So do you end up doing the same augmentations you do in train, on both validation and test? Or do you do some on one and not on others?
", "rawMarkdown": "@cdeotte You talk about using TTA in both your validation as well as test. I have used TTA in test, but not tried it in validation and will give it a try. So do you end up doing the same augmentations you do in train, on both validation and test? Or do you do some on one and not on others?" }, { "id": 947753, "postDate": "2020-07-27T13:14:11.913Z", "content": "I use TTA in both test and validation. That way I can choose the TTA parameters that maximize validation score. Currently, I use all augmentation techniques for train, validation, and test. (I have not experimented with using less augmentation on validation and test compared with train. Perhaps using less would increase CV score more).
", "rawMarkdown": "I use TTA in both test and validation. That way I can choose the TTA parameters that maximize validation score. Currently, I use all augmentation techniques for train, validation, and test. (I have not experimented with using less augmentation on validation and test compared with train. Perhaps using less would increase CV score more)." }, { "id": 950170, "postDate": "2020-07-29T08:19:29.713Z", "content": "@cdeotte Cutout on TTA improved further my LB score .
\n\nI have run an experiment for fold0 on 256x256.
\n10 TTA random Augment w/o Cutout : LB 0.9357\n10 TTA random Augment w/ Cutout : LB 0.9392
However I haven't check yet any random augment on validation data.
\n\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F974295%2F5b02ef384e321357eabe730a8e38a4ce%2FCapre.PNG?generation=1596010735794067&alt=media\")
@cdeotte Thanks (you replied 18 days ago but just thinking again about this). In your Triple Stratified notebook, if I am reading it right (I run PyTorch so I have not run that notebook), your validation dataset looks like it has augment=False or something along the lines of that, which is why I thought maybe in your approach you didn't do it in validation. But I am possibly just reading your notebook wrong.
Can we also extend it for images with 4 channels? And also why compiler needs to know image = tf.reshape(image,[DIM,DIM,3])?
TensorFlow needs to determine the output shape of each function that operates on tf.data.Dataset(). If you remove image = tf.reshape(image,[DIM,DIM,3]), then i think TensorFlow complains that it can't determine the shape. (I'm not sure why it can't figure it out).
If you have images with 4 channels, just change the 3 to a 4 in two places\n* two = tf.zeros([yb-ya,xb-xa,3])\n* image = tf.reshape(image,[DIM,DIM,3])
Or replace both 3 with image.shape[-1]. That should work and then it can handle whatever number of channels comes its way.
Thanks. I'm bothering you too much these days I guess...
", "rawMarkdown": "Thanks. I'm bothering you too much these days I guess..." } ] }, { "id": 944488, "postDate": "2020-07-25T06:42:59.263Z", "content": "Can I use this function for JPEG image augmentation ?
", "rawMarkdown": "Can I use this function for JPEG image augmentation ?", "votes": 1, "replies": [ { "id": 944575, "postDate": "2020-07-25T07:36:13.063Z", "content": "For JPEG, you can use Albumentations' albumentations.augmentations.transforms.Cutout and albumentations.augmentations.transforms.CoarseDropout, the API is here
Thank you so much.This is very helpful.
", "rawMarkdown": "Thank you so much.This is very helpful.", "votes": 1 } ] }, { "id": 944404, "postDate": "2020-07-25T04:54:44.833Z", "content": "Nice work!
", "rawMarkdown": "Nice work! ", "votes": 1 }, { "id": 944383, "postDate": "2020-07-25T04:26:19.457Z", "content": "Nice and precise implementations @cdeotte! I was curious if AutoAugment can also be implemented in TF as it is proved to improve the model's performance. I found the pytorch implementation of AutoAugmentation here.
Sure. Anything can be coded in TensorFlow. This would be interesting to try out.
", "rawMarkdown": "Sure. Anything can be coded in TensorFlow. This would be interesting to try out." } ] }, { "id": 944232, "postDate": "2020-07-25T00:15:18.477Z", "content": "Why makes it suitable for TPU and GPU?
", "rawMarkdown": "Why makes it suitable for TPU and GPU? ", "votes": 1, "replies": [ { "id": 944236, "postDate": "2020-07-25T00:20:18.633Z", "content": "This code can be executed directly on the GPU/TPU if you desire.
\n\nTensorFlow runs their tf.data.Dataset pipeline on CPU in parallel with GPU/TPU training but once you have TensorFlow code, you can push it into a tf.keras.layer and make it part of your model. Then it will execute on the GPU/TPU inside your model. Sometimes this is faster than CPU parallel sometimes it is not.
However, i don't suggest pushing it into a tf.keras.layer. I suggest just using it with tf.data.Dataset if you're using TensorFlow GPU/TPU.
It seems that we can't use albumentations in tf.data.Dataet of if we can somehow convert all the operations in tensorflow then they'll work fine?
", "rawMarkdown": "It seems that we can't use albumentations in tf.data.Dataet of if we can somehow convert all the operations in tensorflow then they'll work fine?", "votes": 1 }, { "id": 944331, "postDate": "2020-07-25T03:01:14.850Z", "content": "Yeah, unfortunately we cannot use Albumentations with tf.data.Dataset nor use tf.contrib libraries nor any libraries. All augmentation needs to be written from scratch using TensorFlow language.
I think we now have all the code to do the main augmentations with tf.data.Dataset. Rotation, Sheer, Zoom, Shift here, Cutmix and mixup here, and now Cutout and sparse dropout in this discussion. And TensorFlow already has functions for color manipulations such as contrast, brightness, hue, sharpness, etc.
Thanks again for this Chris. I've been resisting the temptation to reduce the number and types of augmentations. I know that in the long run, I'll be better off. As you mentioned it's not as simple as one would hope to do custom augmentations in TF. I'm keen to try this out.
", "rawMarkdown": "Thanks again for this Chris. I've been resisting the temptation to reduce the number and types of augmentations. I know that in the long run, I'll be better off. As you mentioned it's not as simple as one would hope to do custom augmentations in TF. I'm keen to try this out. ", "votes": 1 }, { "id": 947848, "postDate": "2020-07-27T14:10:36.407Z", "content": "Technically, you can use Albumentations through the tf dataset pipeline using tf.numpy_function. The downside is that TPU's do not support this feature (yet). On a GPU however, this should work.
", "rawMarkdown": "Technically, you can use Albumentations through the tf dataset pipeline using tf.numpy_function. The downside is that TPU's do not support this feature (yet). On a GPU however, this should work.", "votes": 2, "replies": [ { "id": 947855, "postDate": "2020-07-27T14:15:53.297Z", "content": "Thanks for letting me know that.
", "rawMarkdown": "Thanks for letting me know that.", "votes": 1 } ] }, { "id": 945900, "postDate": "2020-07-26T07:49:01.987Z", "content": "You don't seem to be using hair_aug(), do you think we should use hair augmentation?
", "rawMarkdown": "You don't seem to be using hair_aug(), do you think we should use hair augmentation?", "votes": 2, "replies": [ { "id": 946375, "postDate": "2020-07-26T14:42:52.723Z", "content": "I have not tried hair augmentation yet. I would guess that it is helpful. Has anyone posted a TensorFlow tf.data.Dataset implementation? Please post the link here.
Here is my notebook implementing Roman's advance hair augmentation in TF:
\n\nSIIM: Data Augmentation in TF: Hair + Batch Affine
\n\nI have not converted it to the batch form, so it does slow down training.
", "rawMarkdown": "Here is my notebook implementing Roman's advance hair augmentation in TF:\n\n[SIIM: Data Augmentation in TF: Hair + Batch Affine](https://www.kaggle.com/graf10a/siim-data-augmentation-in-tf-hair-batch-affine)\n\nI have not converted it to the batch form, so it does slow down training. ", "votes": 4 }, { "id": 946943, "postDate": "2020-07-27T01:00:18.797Z", "content": "Great thanks @graf10a , i will play with it.
", "rawMarkdown": "Great thanks @graf10a , i will play with it.", "votes": 2 } ] }, { "id": 1182038, "postDate": "2021-02-02T09:15:43.760Z", "content": "This is really very useful
", "rawMarkdown": "This is really very useful" }, { "id": 1182039, "postDate": "2021-02-02T09:15:43.760Z", "content": "This is really very useful
", "rawMarkdown": "This is really very useful" }, { "id": 972499, "postDate": "2020-08-16T15:31:32.547Z", "content": "is it same the cutmix ?
\ndef cutmix_aug(image, label=None, DIM=256, BATCH_SIZE=4, PROBABILITY=0.4):
\n # input image - is a batch of images of size [n,dim,dim,3] not a single image of [dim,dim,3]
\n # output - a batch of images with cutmix applied
imgs = []; labs = []\n\nfor j in range(BATCH_SIZE):\n\n #random_uniform( shape, minval=0, maxval=None) \n # DO CUTMIX WITH PROBABILITY DEFINED ABOVE\n P = tf.cast(tf.random.uniform([], 0, 1) <= PROBABILITY, tf.int32)\n\n # CHOOSE RANDOM IMAGE TO CUTMIX WITH\n k = tf.cast(tf.random.uniform([], 0, BATCH_SIZE), tf.int32)\n\n # CHOOSE RANDOM LOCATION\n x = tf.cast(tf.random.uniform([], 0, DIM), tf.int32)\n y = tf.cast(tf.random.uniform([], 0, DIM), tf.int32)\n\n # Beta(1, 1)\n b = tf.random.uniform([], 0, 1) # this is beta dist with alpha=1.0\n\n\n WIDTH = tf.cast(DIM * tf.math.sqrt(1-b),tf.int32) * P\n ya = tf.math.maximum(0,y-WIDTH//2)\n yb = tf.math.minimum(DIM,y+WIDTH//2)\n xa = tf.math.maximum(0,x-WIDTH//2)\n xb = tf.math.minimum(DIM,x+WIDTH//2)\n\n # MAKE CUTMIX IMAGE\n one = image[j,ya:yb,0:xa,:]\n two = image[k,ya:yb,xa:xb,:]\n three = image[j,ya:yb,xb:DIM,:] \n #ya:yb\n middle = tf.concat([one,two,three],axis=1)\n\n img = tf.concat([image[j,0:ya,:,:],middle,image[j,yb:DIM,:,:]],axis=0)\n imgs.append(img)\n\n # MAKE CUTMIX LABEL\n a = tf.cast(WIDTH*WIDTH/DIM/DIM,tf.float32)\n lab1 = label[j,]\n lab2 = label[k,]\n labs.append((1-a)*lab1 + a*lab2)\n\nimage2 = tf.reshape(tf.stack(imgs),(BATCH_SIZE, DIM, DIM, 3))\nlabel2 = tf.reshape(tf.stack(labs),(BATCH_SIZE, 1))\nreturn image2, label2\nthis script handle the label too, cast label to float32
", "rawMarkdown": "is it same the cutmix ?\n\ndef cutmix_aug(image, label=None, DIM=256, BATCH_SIZE=4, PROBABILITY=0.4):\n # input image - is a batch of images of size [n,dim,dim,3] not a single image of [dim,dim,3]\n # output - a batch of images with cutmix applied\n\n imgs = []; labs = []\n \n for j in range(BATCH_SIZE):\n \n #random_uniform( shape, minval=0, maxval=None) \n # DO CUTMIX WITH PROBABILITY DEFINED ABOVE\n P = tf.cast(tf.random.uniform([], 0, 1) <= PROBABILITY, tf.int32)\n \n # CHOOSE RANDOM IMAGE TO CUTMIX WITH\n k = tf.cast(tf.random.uniform([], 0, BATCH_SIZE), tf.int32)\n \n # CHOOSE RANDOM LOCATION\n x = tf.cast(tf.random.uniform([], 0, DIM), tf.int32)\n y = tf.cast(tf.random.uniform([], 0, DIM), tf.int32)\n \n # Beta(1, 1)\n b = tf.random.uniform([], 0, 1) # this is beta dist with alpha=1.0\n \n\n WIDTH = tf.cast(DIM * tf.math.sqrt(1-b),tf.int32) * P\n ya = tf.math.maximum(0,y-WIDTH//2)\n yb = tf.math.minimum(DIM,y+WIDTH//2)\n xa = tf.math.maximum(0,x-WIDTH//2)\n xb = tf.math.minimum(DIM,x+WIDTH//2)\n \n # MAKE CUTMIX IMAGE\n one = image[j,ya:yb,0:xa,:]\n two = image[k,ya:yb,xa:xb,:]\n three = image[j,ya:yb,xb:DIM,:] \n #ya:yb\n middle = tf.concat([one,two,three],axis=1)\n\n img = tf.concat([image[j,0:ya,:,:],middle,image[j,yb:DIM,:,:]],axis=0)\n imgs.append(img)\n \n # MAKE CUTMIX LABEL\n a = tf.cast(WIDTH*WIDTH/DIM/DIM,tf.float32)\n lab1 = label[j,]\n lab2 = label[k,]\n labs.append((1-a)*lab1 + a*lab2)\n\n image2 = tf.reshape(tf.stack(imgs),(BATCH_SIZE, DIM, DIM, 3))\n label2 = tf.reshape(tf.stack(labs),(BATCH_SIZE, 1))\n return image2, label2\n\nthis script handle the label too, cast label to float32", "replies": [ { "id": 972924, "postDate": "2020-08-17T01:25:30.137Z", "content": "I published a CutMix notebook here. For coarse dropout, I used that CutMix notebook and just replace the second image with all zeros (to perform the dropout).
", "rawMarkdown": "I published a CutMix notebook [here][1]. For coarse dropout, I used that CutMix notebook and just replace the second image with all zeros (to perform the dropout).\n\n[1]: https://www.kaggle.com/cdeotte/cutmix-and-mixup-on-gpu-tpu" }, { "id": 972971, "postDate": "2020-08-17T02:22:35.937Z", "content": "ah ok, so label still keep 👍
", "rawMarkdown": "ah ok, so label still keep 👍", "votes": 1 } ] }, { "id": 949312, "postDate": "2020-07-28T14:51:35.453Z", "content": "This method is cool .\nBut I have a doubt\nas the dataset is imbalanced , does by adding small black boxes to the image increases its performance because it is learning from the presence of small black boxes to predict the image as non malignant and as the number of non malignant class are more it may cause a delusion that the model is working better with the dropout where as it is learning wrong thing which will affect its performance on the test data
", "rawMarkdown": "This method is cool .\nBut I have a doubt\nas the dataset is imbalanced , does by adding small black boxes to the image increases its performance because it is learning from the presence of small black boxes to predict the image as non malignant and as the number of non malignant class are more it may cause a delusion that the model is working better with the dropout where as it is learning wrong thing which will affect its performance on the test data \n\n", "replies": [ { "id": 949342, "postDate": "2020-07-28T15:13:19.070Z", "content": "As long as the black boxes are added to both benign and malignant images, all is good.
", "rawMarkdown": "As long as the black boxes are added to both benign and malignant images, all is good.", "votes": 2 }, { "id": 949400, "postDate": "2020-07-28T16:04:50.390Z", "content": "Thank you for your time and response .\nI also wanted to know for this competition up sampling the malignant data is better than using weights or do you advise on using both at the same time
", "rawMarkdown": "Thank you for your time and response .\nI also wanted to know for this competition up sampling the malignant data is better than using weights or do you advise on using both at the same time " }, { "id": 949433, "postDate": "2020-07-28T16:25:01.307Z", "content": "Upsampling or using weights are very similar. For example you can double all the malignant images or you can use weights {0:1, 1:2} (malignant is 2x). One advantage of weights is less train data and faster training epochs. One advantage of upsample is that each batch now contains some malignant and the malignant learning process is more incremental.
Additionally since we're using data augmentation and external data malignant images, then doing upsample adds more variety to the malignant class and hopefully encourages more malignant generalization.
", "rawMarkdown": "Upsampling or using weights are very similar. For example you can double all the malignant images or you can use weights `{0:1, 1:2}` (malignant is 2x). One advantage of weights is less train data and faster training epochs. One advantage of upsample is that each batch now contains some malignant and the malignant learning process is more incremental.\n\nAdditionally since we're using data augmentation and external data malignant images, then doing upsample adds more variety to the malignant class and hopefully encourages more malignant generalization." } ] }, { "id": 945544, "postDate": "2020-07-26T00:02:59.827Z", "content": "I tried Dropout in another Dataset and I got this error while training...\n\nCompilation failure: Dynamic Spatial Convolution is not supported\n
hmm... i've seen that error before. It relates to TensorFlow knowing the size of stuff. Can your post your code that adds the dropout to your tf.data.Dataset? Make sure you give dropout the correct image size. And make sure that you add dropout before you batch, i.e. ds = ds.map(dropout) before ds = ds.batch(BATCH_SIZE)
dropout(image, DIM=???, PROBABILITY = 0.75, CT = 8, SZ = 0.2)\nreplace ??? with correct size
", "rawMarkdown": "hmm... i've seen that error before. It relates to TensorFlow knowing the size of stuff. Can your post your code that adds the dropout to your `tf.data.Dataset`? Make sure you give `dropout` the correct image size. And make sure that you add dropout **before** you batch, i.e. `ds = ds.map(dropout)` **before** `ds = ds.batch(BATCH_SIZE)`\n\n dropout(image, DIM=???, PROBABILITY = 0.75, CT = 8, SZ = 0.2)\n\nreplace ??? with correct size" }, { "id": 946183, "postDate": "2020-07-26T12:23:43.453Z", "rawMarkdown": "", "isDeleted": true }, { "id": 946188, "postDate": "2020-07-26T12:26:32.670Z", "content": "Same code runs for GPU and CPU but shows error in TPU..\n```\ndef dropout_img(image,DIM=256, PROBABILITY = 0.5, CT = 50, SZ = 0.06):\n # input image - is one image of size [dim,dim,3] not a batch of [b,dim,dim,3]\n # output - image with CT squares of side size SZ*DIM removed
\n\n# DO DROPOUT WITH PROBABILITY DEFINED ABOVE\nP = tf.cast( tf.random.uniform([],0,1)<PROBABILITY, tf.int32)\nif (P==0)|(CT==0)|(SZ==0): \n return image, mask\n\nfor k in range(CT):\n # CHOOSE RANDOM LOCATION\n x = tf.cast( tf.random.uniform([],0,DIM),tf.int32)\n y = tf.cast( tf.random.uniform([],0,DIM),tf.int32)\n # COMPUTE SQUARE \n WIDTH = tf.cast( SZ*DIM,tf.int32) * P\n ya = tf.math.maximum(0,y-WIDTH//2)\n yb = tf.math.minimum(DIM,y+WIDTH//2)\n xa = tf.math.maximum(0,x-WIDTH//2)\n xb = tf.math.minimum(DIM,x+WIDTH//2)\n # DROPOUT IMAGE\n one = image[ya:yb,0:xa,:]\n two = tf.zeros([yb-ya,xb-xa,4], dtype = image.dtype) \n three = image[ya:yb,xb:DIM,:]\n middle = tf.concat([one,two,three],axis=1)\n image = tf.concat([image[0:ya,:,:],middle,image[yb:DIM,:,:]],axis=0)\n\nimage = tf.reshape(image,[DIM,DIM,-1])\n\nreturn image\n\n\ndef augmentation_img(img, dim=256):
seed = np.int(np.round(np.random.uniform()))\n\nimg = tf.image.random_flip_left_right(img, seed = seed)\n\nimg = transform_img(img, DIM =dim)\nimg = dropout_img(img, DIM=dim, PROBABILITY = 0.5, CT = 50, SZ = 0.06)\n\n\nimg = tf.image.random_contrast(img, 0.8, 1.2)\nimg = tf.image.random_brightness(img, 0.1)\n\nimg = tf.reshape(img, [dim, dim, -1])\n\nreturn img\n\n\n ds = tf.data.TFRecordDataset(files, num_parallel_reads=AUTO)\n ds = ds.cache()
if repeat:\n ds = ds.repeat()\n\nif shuffle: \n ds = ds.shuffle(1024*8)\n opt = tf.data.Options()\n opt.experimental_deterministic = False\n ds = ds.with_options(opt)\n\n ds = ds.map(lambda example: read_unlabeled_tfrecord(example, return_image_names),num_parallel_calls=AUTO) \n\n if augment:\n ds = ds.map(lambda img, name: (augmentation_img(img, dim=dim), name), num_parallel_calls=AUTO)\n\nds = ds.batch(batch_size * REPLICAS)\nds = ds.prefetch(AUTO)\nreturn ds\n```
", "rawMarkdown": "Same code runs for GPU and CPU but shows error in TPU..\n```\ndef dropout_img(image,DIM=256, PROBABILITY = 0.5, CT = 50, SZ = 0.06):\n # input image - is one image of size [dim,dim,3] not a batch of [b,dim,dim,3]\n # output - image with CT squares of side size SZ*DIM removed\n \n # DO DROPOUT WITH PROBABILITY DEFINED ABOVE\n P = tf.cast( tf.random.uniform([],0,1)" } ] }, { "id": 967894, "postDate": "2020-08-12T15:10:31.593Z", "rawMarkdown": "", "isDeleted": true }, { "id": 953250, "postDate": "2020-07-31T16:14:52.437Z", "content": "Thank you !!
", "rawMarkdown": "Thank you !!", "votes": 1 }, { "id": 951260, "postDate": "2020-07-30T02:49:07.167Z", "content": "Thanks for Sharing!
", "rawMarkdown": "Thanks for Sharing!", "votes": 1 }, { "id": 948203, "postDate": "2020-07-27T18:14:56.500Z", "content": "Nice work, thanks for sharing!
", "rawMarkdown": "Nice work, thanks for sharing!", "votes": 1 } ], "comments": [ { "id": 953624, "author_name": "Stephan", "author_url": "", "post_date": "2020-07-31T23:34:25.127000", "content": "I've created a version that accepts batches of images, for anyone who is interested. I've tested it on a GPU setup, so I can't be sure this works on a TPU yet.
\n\nedit: I see that the prob arg isn't actually utilized yet, I'll fix this tommorow.\nedit 2: probabilities now included :)\nedit 3: replaced a type in tf.scatter_nd where the image sizes were fixed to 128,128 -> now adjusted for variable sizes
\n\n```python\ndef coarse_dropout(images, square_size, num_squares, prob):\n ''' Coarse image dropout \n Coarse dropout masks are created by using tf.scatter_nd\n Which can update matrices according to coordinates specified\n by the user.
\n\nParameters\n----------\nimages : tf Tensor\n A batch of images of size [BS, H, W, C] \nsquare_size : float\n A float between [0,1], a percentage of the image size\nnum_squares : int > 0\n The number of dropout squares per single image, must be >0\nprob : float\n A float between [0,1], probability that dropout is used for this batch\n\nYields\n------\nTYPE\n DESCRIPTION.\n\n'''\n # random dropout\nprob = tf.cast( tf.random.uniform([],0,1) < prob, tf.int32)\nif (prob == 0): return images\n\nimg_shape = images.shape\n_, h, w, c = img_shape[0], img_shape[1], img_shape[2], img_shape[3]\n\n# For some odd reason, the batch size is lost in the processing pipeline\n# seriously, how can it get lost ._., it literally says ds.batch()...\n# tensorflow shenanigans...\nbs = tf.cast(tf.reduce_sum(tf.ones_like(images)) / (images.shape[1] * images.shape[2] * images.shape[3]), tf.int32)\n\n\n# size of square in pixels (ssp), ASSUMING SQUARE IMAGES!\nssp = tf.cast(tf.math.ceil(h * square_size), tf.int32)\n\n# Create random start x coordinates\ncoords_x = tf.random.uniform([bs, num_squares], \n minval=0, \n maxval= (h - ssp), \n dtype=tf.int32) \n\n# Create ranges from start to the end of the line\n# [ssp, bs, num_squares]\ncoords_x = tf.linspace(coords_x, coords_x + ssp - 1, ssp)\n\n# [num_squares, bs, ssp]\ncoords_x = tf.cast(tf.transpose(coords_x), tf.int32)\n\n# Create random start y coordinates\ncoords_y = tf.random.uniform([bs, num_squares], \n minval=0, \n maxval= (h - ssp), \n dtype=tf.int32) \n\n# Create ranges from start to the end of the line\n# [ssp, bs, num_squares]\ncoords_y = tf.linspace(coords_y, coords_y + ssp - 1, ssp)\n\n# [num_squares, bs, ssp]\ncoords_y = tf.cast(tf.transpose(coords_y), tf.int32)\n\n\n# Create coordinate range combinations \n# and reshape to [bs, num_squares, 1, ssp * ssp]\ngrid_y = tf.reshape(tf.tile(coords_y, [1,1,ssp]), \n (bs, num_squares, 1, ssp * ssp))\n\n# and reshape to [bs, num_squares, ssp, ssp], transpose the inner matrices\ngrid_y = tf.transpose(tf.reshape(grid_y, \n (bs, num_squares, ssp, ssp)), \n (0, 1, 3, 2))\n\n# Repeat for x coordinates\ngrid_x = tf.reshape(tf.tile(coords_x, [1,1,ssp]), \n (bs, num_squares, 1, ssp * ssp))\n\ngrid_x = tf.reshape(grid_x, (bs, num_squares, ssp, ssp))\n\n# Stack the grids into a single matrix\n# grid is [2, bs, num_squares, ssp, ssp]\ngrid = tf.stack([grid_y, grid_x], axis=0)\n\n# Transpose and reshape [ bs, ssp * ssp * num_squares, 2] \n# Creates an array of 2D coordinates ([[x1,y1], [x2, y2], ..., [xn, yn]])\n# over all squares (num_squares), for each combination (ssp*ssp) of coordinates \n# and each batch (bs)\ngrid = tf.reshape(tf.transpose(grid, (1, 4, 3, 2, 0)), \n (bs, ssp * ssp * num_squares, 2))\n\n# [bs, sz*sz*num_squares, 2]\n#grid = tf.reshape(grid, (bs, ssp*ssp*num_squares, 2))\n\n# create batch indices [0,..., bs] and reshape to [ssp * ssp * num_squares, bs]\nbatch_indices = tf.reshape(tf.tile(tf.range(0, bs), \n [ssp * ssp * num_squares]), \n (ssp * ssp * num_squares, bs))\n\n# Transpose to get the right order, and reshape to match grid shape\nbatch_indices = tf.reshape(tf.transpose(batch_indices), \n (bs, ssp * ssp * num_squares, 1))\n\n# concatenate batch indices with the grid\n# these yield 3D coordinates like e.g.\n# [[bs0, x1, y1], [bs0, x2, y1], ..., [bsn, xn, yn]]\ngrid = tf.concat([batch_indices, grid], axis=2)\n\n# create a matrix of zeros, and update the matrix with the grid indices\n# this essentially creates a mask with coarse dropout squares\nmasks = tf.scatter_nd(grid[tf.newaxis,...], \n tf.ones([1,bs,ssp*ssp*num_squares]) * -1, \n shape=(bs, h, w)) +1\n\n# Due to overlap of squares, some get coordinates get updated twice\n# and result in values < -1, clip these values\nmasks = tf.clip_by_value(masks, 0, 1)\n\nreturn images * masks[..., tf.newaxis] \n```
", "votes": 7, "replies": [ { "id": 954270, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-01T15:24:34.653000", "content": "Thanks @stephanovich ! when i get time i will play with this. I'm curious to benchmark the speed and see if this is faster than \"per image\".
\n\nI've noticed when using \"per image\", if we cutout more than 16 squares it will begin to slow down training, perhaps your \"per batch\" code can do more than 16 and not slow down training.
", "votes": 1, "replies": [] }, { "id": 955084, "author_name": "Stephan", "author_url": "", "post_date": "2020-08-02T10:17:17.990000", "content": "Edit: you might want to try the augmentation from this discussion and notebook, it seems to be a more generalized version of coarse dropout, and does not use for loops either (i.e. vectorized).
\n\nSo I ran a small benchmark myself on a cpu with 32 threads, here's the results:\nThe experiments are run on 128*128 images with a variable number of squares each and a square size of 0.1 * the size of the image. The probability is set to one in both cases. I parsed through 60000 examples, which in my opinion was already enough to at least see a difference.\nI've included 3 runs: \n1. one without any sort of augmentation, which acts as the lower bound (or golden standard) for time benchmarking (it can never go faster than this).\npython\nCPU times: user 11min 27s, sys: 30.9 s, total: 11min 58s\nWall time: 29.2 s\n
2 . The new batch version. This version first prepares the image, then batches the images, and then augments them. Otherwise, we would not see any improvements.\n``` python
\n\nCPU times: user 28min 6s, sys: 1min 38s, total: 29min 44s\nWall time: 1min 12s
\n\nCPU times: user 33min, sys: 1min 36s, total: 34min 36s\nWall time: 1min 17s
\n\n```
\n\n3 . Your version. In the processing pipeline, the images are first prepared through prepare_image, then are augmented, and finally batched.\n```python
\n\nCPU times: user 50min 36s, sys: 2min 37s, total: 53min 14s\nWall time: 2min 4s
\n\nCPU times: user 1h 15min 50s, sys: 2min 41s, total: 1h 18min 32s\nWall time: 2min 48s\n```
\n\nIt seems the batch version does run faster, at the cost of some higher memory footprint (due to vectorization)
\n\nOf course, the times can decrease/increase depending on the image sizes, number of squares which you already mentioned above, and possibly other factors.
", "votes": 4, "replies": [] } ] }, { "id": 945580, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-07-26T01:30:08.590000", "content": "Notice how experiment 1 without dropout has a larger gap between train AUC and validation AUC. While experiment 2 with dropout has a smaller gap. In the below example, dropout has helped the CNN generalize thus increasing validation AUC while preventing training AUC from overfitting (i.e. reducing training AUC). Notebook here
\n\n
This is really useful for tensorflow users in this competition @cdeotte :D
", "votes": 3, "replies": [] }, { "id": 944692, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-07-25T09:00:35.247000", "content": "UPDATE: If your forked version 15 or earlier of the starter notebook, note that it does not apply dropout. (It only applies upsample). During training, in the call to get_dataset(), I forgot to pass the dropout parameters (so they use the default of zero). In the 3 lines below, the last line was missing in notebook version 15 and earlier. I just committed notebook now. Versions 16 onward will have this line added and will apply dropout.
get_dataset(files_train, augment=True, shuffle=True, repeat=True,\n dim=IMG_SIZES[fold], batch_size = BATCH_SIZES[fold],\n droprate = DROP_FREQ[fold], dropct = DROP_CT[fold], dropsize = DROP_SIZE[fold])\nWe also need to add this last line to validation TTA prediction and test TTA prediction.
", "votes": 3, "replies": [] }, { "id": 954351, "author_name": "cocoinit23", "author_url": "", "post_date": "2020-08-01T16:46:06.593000", "content": "@cdeotte Thank you for your great work! Please allow me one question.\nHow do you think about the difference between Coarse Dropout and GridMask ?
", "votes": 1, "replies": [ { "id": 954356, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-01T16:52:39.810000", "content": "I have never used GridMask before. There is an implementation of GridMask for TensorFlow here by Xie29.
\n\n(Note: I have not tested Xie29's implementation. If you use it, i would compare training with and without it to make sure the code is efficient and doesn't slow down training).
", "votes": 1, "replies": [] }, { "id": 954362, "author_name": "cocoinit23", "author_url": "", "post_date": "2020-08-01T17:05:21.667000", "content": "Thank you for your quick reply and suggestion. I plan to try both of them.\nAgain, I appreciate your nice works!
", "votes": 1, "replies": [] } ] }, { "id": 952933, "author_name": "Roman Weilguny", "author_url": "", "post_date": "2020-07-31T10:34:42.307000", "content": "@cdeotte Hi Chris! I really would like to try to find ensembling weights by using a machine learning algorithm and cv oof auc scores. Did you ever try this? Do you have an url or tip how to do this?\nBest Roman
", "votes": 1, "replies": [ { "id": 953123, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-07-31T14:47:45.470000", "content": "This is what all the top competitors do in every competition. This is the correct way to ensemble. (The incorrect way is trial and error using feedback from public LB because public test set contains less data than train set).
\n\nYou train multiple models using the same KFold folds and save all your OOF. Then you find weights with the OOF. For example if you have 3 models, then you find w1, w2, w3 with w1+w2+w3 = 1 where you maximize AUC below
oof = w1*oof1 + w2*oof2 + w3*oof3\nAUC = roc_auc_score(TRUE, oof)\nLastly you use these weights for your Kaggle submission as in
\n\npred = w1*sub1 + w2*sub2 + w3*sub3\nsub = pd.DataFrame(dict(image_name=NAMES, target=PRED))\nsub.to_csv('submission.csv',index=False)\nThe simplest thing is to (1) find these weights by hand or grid search (i.e. with nested for-loops). Or (2) you can find these weights by fitting a linear or logistic regression model. (Since linear and logistic regression are simple models, you don't need holdout set, just fit_transform all the oof against true). Or (3) if you want to use a more complex stage 2 model like XGB, you need to set up a holdout set.
Hello, @cdeotte thanks for the above pointer. I was trying to set weight values based on the CV score rather than LB feedback with different trails and errors. However, I totally agree with @romanweilguny that we should use ML algorithm to find an optimal weight for the ensemble.
\nLater I've tried to use the following method to find optimal weight, though few things are yet not clear to me, working on it. Do you have any suggestions for that? Basically we've kinda followed the above procedure that you've mentioned, the aim was to get max(roc_auc_score(TRUE, oof)) after some iteration.
Thank you -)
", "votes": 0, "replies": [] }, { "id": 967258, "author_name": "Signal", "author_url": "", "post_date": "2020-08-12T05:53:37.050000", "content": "I think an ML approach would be interesting. Not sure how fast the compute is you have access to, but say you had 4 models you wanted to ensemble, you could brute force the weights with 10,000 iterations, which can totally be parallelized across cores, if you used values of .00, .10, .20, etc through .90.
", "votes": 0, "replies": [] } ] }, { "id": 953320, "author_name": "Roman Weilguny", "author_url": "", "post_date": "2020-07-31T17:36:33.090000", "content": "Thx - thats what I have been looking for!
", "votes": 1, "replies": [] }, { "id": 954193, "author_name": "Rishiacharya", "author_url": "", "post_date": "2020-08-01T13:33:51.620000", "content": "I have noticed in your notebooks you select the best model in a fold based on the lowest loss.Would it be wrong to select the best model using max AUC, even if there is 0.3-0.5 difference between the lowest loss and the max AUC epoch's loss? Or is it important to select a model based on loss
", "votes": 0, "replies": [] }, { "id": 954268, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-01T15:20:30.477000", "content": "There are basically 3 options.\n* select lowest val loss\n* select highest val auc\n* select after fixed number of epochs, like 12th each time
\n\nThere are pros and cons for each. The resultant LB AUC will be some random number added or subtracted to each Val AUC, so any of these 3 choices could produce the best LB AUC. We cannot know which is the best choice. Some people will even take all 3 and ensemble the 3 models (but that's not necessarily better either).
\n\nIn conclusion, i'm saying experiment will all, and do what you prefer. All will work well.
", "votes": 1, "replies": [] }, { "id": 959717, "author_name": "ELEVEN", "author_url": "", "post_date": "2020-08-05T20:55:43.773000", "content": "@cdeotte Hi Chris! Do you use OOF with TTA or without TTA, selecting weights?
\n\n\n", "votes": 0, "replies": [] }, { "id": 967089, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-11T23:12:53.580000", "content": "You train multiple models using the same KFold folds and save all your OOF. Then you find weights with the OOF. For example if you have 3 models, then you find w1, w2, w3 with w1+w2+w3 = 1 where you maximize AUC below
\n
You should the same thing that was applied to your submission.csv. If your submission.csv has TTA applied then use OOF with TTA applied.
@cdeotte a simple question: Do you have any ratio between image size and rest of the droprate, dropct and dropsize or you keep them same for all? Thanks for your great sources for this competition again...
", "votes": 1, "replies": [ { "id": 952231, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-07-30T18:12:41.463000", "content": "Good question. DROP_SIZE is a ratio between 0 and 1. So it scales your squares with your image size. (Square side to cutout out is IMAGE_SIZE * DROP_SIZE). Likewise DROP_RATE and DROP_CT naturally scale. Therefore once you find good numbers, you can use the same numbers for all image sizes.
Here's an important note. The variable DROP_CT removes CT squares using a for-loop. I recommend using 16 or less and preferably 8 or less. If you want to remove more image, increase DROP_RATE and/or DROP_SIZE.
If you turn this augmentation on and off, you will find that if DROP_CT>=16 then it starts to slightly slow down your training time per epoch because the CPU can't augment (execute for-loop) fast enough (when doing all the other common stuff like rotation, sheer, zoom, etc etc).
So for example if you set DROP_FREQ = 1.0, DROP_CT = 8, and DROP_SIZE = 0.2 then regardless of image size, it probably removes about 20% of every image. (If the squares did not overlap it would be 25%).
Thank you for the answer! I wanted to play with your approach today, implemented it on my model and some feedback here maybe you find them useful:
\n\nIn my case I didn't feel slowing down up to 16 CT's maybe 5% longer per epoch or something at high numbers, didn't go over 16 though.
\n\nBut it definitely slowed down my models learning process with increased randomness, had to increase epochs like 30% to reach dropout off levels of train auc.
\n\nHaven't played enough to comment about cv, lb yet... But anyways thank you again! @cdeotte
", "votes": 1, "replies": [] }, { "id": 952284, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-07-30T19:23:31.897000", "content": "Yes using image dropout or layer dropout means that you have to train for more epochs. The idea is that your model is learning slower but it will achieve a higher level of intelligence.
", "votes": 1, "replies": [] }, { "id": 952613, "author_name": "adigamer970", "author_url": "", "post_date": "2020-07-31T04:55:47.803000", "content": "Is there any method to speed up the for loop process. It seems to waste the incredible capabilities of TPU while doing for loop.
", "votes": 1, "replies": [] }, { "id": 952631, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-07-31T05:17:12.977000", "content": "It does not waste TPU time. All augmentation of tf.data.Dataset happens on the TPU's CPU in parallel. So while the TPU is training one batch, the CPU is preparing the next batch. As long as DROP_CT is low enough it doesn't change train time at all (because the CPU batch time is less than the TPU batch time).
Good post Chris!
", "votes": 1, "replies": [] }, { "id": 945624, "author_name": "Hideaki Takahashi", "author_url": "", "post_date": "2020-07-26T03:19:34.997000", "content": "In my understanding, we should not use dropout or cutout when predicting validation data or test data. Am I right?
", "votes": 1, "replies": [ { "id": 945695, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-07-26T04:53:30.620000", "content": "The best answer is try both and see what produces higher CV LB. Since we're using TTA in both our validation and test, I think it may be best to add dropout and/or cutout to the TTA.
\n\nI will point out that simply removing dropout or cutout from predicting test or validation may not work. When you remove a dropout layer inside a CNN, you need to rescale the activations to adjust for the missing dropout. I'm not sure how you would \"adjust\" the validation or test predictions if we train with coarse dropout and then don't use coarse dropout for prediction.
", "votes": 2, "replies": [] }, { "id": 946158, "author_name": "Hideaki Takahashi", "author_url": "", "post_date": "2020-07-26T11:59:09.063000", "content": "Thanks for your reply! I did a few experiments (I know it is not sufficient.) and found that as you said, removing dropout or cutout from prediction is worse. I'll report more reliable results asap!
", "votes": 1, "replies": [] }, { "id": 946176, "author_name": "Hideaki Takahashi", "author_url": "", "post_date": "2020-07-26T12:18:17.910000", "content": "Though I can understand the importance of TTA, I can't understand why using dropout works with validation or test data. My concern is that using these augmentations may lose important information.
", "votes": 2, "replies": [] }, { "id": 946373, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-07-26T14:41:27.497000", "content": "Keep in mind that the CNN is being trained with dropout. (And is smart enough to find malignant even with rectangles of images removed). If during TTA prediction there is no dropout, then the CNN could get confused.
\n\nThat's like teaching a tennis player to use a solid wood racket for months and during the tournament giving the tennis player a real racket with strings. You would think it would help, but the tennis player never learned how to use the real racket with strings.
", "votes": 3, "replies": [] }, { "id": 946457, "author_name": "Psi", "author_url": "", "post_date": "2020-07-26T15:34:31.377000", "content": "@cdeotte This is the first time I see someone using random cutout in TTA to be honest. I will try it, interesting idea. Usually it works well to train with cutout and don't do cutout in inference, it behaves very similar to dropout which you also dont use in inference.
\n\nWhat it definitely does it that it adds randomness, but having randomness in inference can be risky. Random TTA is already something I dont feel comfortable with.
", "votes": 0, "replies": [] }, { "id": 946461, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-07-26T15:38:32.620000", "content": "For TTA (test time augmentation), isn't it best to use the augmentation that your network was trained with?
\n\nIf we remove image dropout from TTA, should we adjust predictions? Because if you add a dropout layer in your CNN like x = Dropout(0.2)(x). Then when the network makes predictions it removes the layer dropout and it reduces the activations by 20%. (i.e. during inference Keras replaces layer dropout with x = Lambda(x: 0.8*x)(x)) What is the equivalent of \"reducing activations by 20%\" when using image dropout?
Doesn't the scaling happen during training and prediction is identity (is similar to what you say)? But you are right, haven't thought much about that when doing cutout. In practise I havent seen any issues with not doing cutout in inference, but also have never tried to keep it.
", "votes": 0, "replies": [] }, { "id": 946467, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-07-26T15:45:35.240000", "content": "\n\n\n\n\nRandom TTA is already something I dont feel comfortable with.
\n
This competition takes TTA to a whole new level. People are using 25+ TTA steps. At that point, randomness gets averaged away.
", "votes": 0, "replies": [] }, { "id": 946474, "author_name": "Psi", "author_url": "", "post_date": "2020-07-26T15:50:05.400000", "content": "@cdeotte Havent tested tta yet, but if tta outside of standard flips works this can be an approach indeed. Inference is fast.
\n\nDo you just use train augs multiple times, or limit them?
", "votes": 0, "replies": [] }, { "id": 946481, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-07-26T15:51:27.990000", "content": "\n\n\nDoesn't the scaling happen during training and prediction is identity
\n
Oh yes, i think you're right. A layer of x = dropout(0.2)(x) simultaneously applies 20% dropout and increases activations by 25% internally with something like x = Lambda(x: 1.25*x)(x). Then during inference it just removes the dropout layer and all is well.
For TTA, i use all the training augmentations! I use rotation, sheer, scale, shift, contrast, saturation, brightness, flip, dropout, etc, etc. Using TTA 25+ is the powerhouse in this comp. It increases CV LB by something like 0.01+ or 0.02+!
\n\n print('Predicting Test with TTA...')\n ds_test = get_dataset(files_test,labeled=False,return_image_names=False,augment=True,\n repeat=True,shuffle=False,dim=IMG_SIZES[fold],batch_size=BATCH_SIZE)\n ct_test = count_data_items(files_test); STEPS = TTA * ct_test/BATCH_SIZE\n pred = model.predict(ds_test,steps=STEPS,verbose=VERBOSE)[:TTA*ct_test,] \n preds += np.mean(pred.reshape((ct_test,TTA),order='F'),axis=1) / skf.n_splits\nAnd get_dataset(augment=True) is the same pipeline that train uses.
The power of TTA was first shown by AgentAuers' in his notebook here
", "votes": 1, "replies": [] }, { "id": 946501, "author_name": "Psi", "author_url": "", "post_date": "2020-07-26T16:00:57.423000", "content": "Thank you, interesting, will try it.
", "votes": 1, "replies": [] }, { "id": 946509, "author_name": "Serigne ", "author_url": "", "post_date": "2020-07-26T16:09:43.877000", "content": "I removed Cutout during my TTA.
\n\nBut I will try it to see if it improves.
", "votes": 1, "replies": [] }, { "id": 946531, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-07-26T16:21:54.737000", "content": "Yes @serigne , let us know. I have only recently added cutout to my augmentations. So I haven't done enough experiments myself to know whether it is best to include cutout or not during TTA inference.
", "votes": 1, "replies": [] }, { "id": 946549, "author_name": "Serigne ", "author_url": "", "post_date": "2020-07-26T16:36:20.617000", "content": "@cdeotte Cutout definitely improved my training model. But, as @philippsinger , I was much reluctant to use Random Augmentation (specially those cutting the input images) during TTA.
\n\nBut, after @optimo suggestion , I gave a try to +15 TTA (with all my training augs except Cutout) and saw relatively huge improvement on LB. Next step is to add Cutout and let you know.
", "votes": 1, "replies": [] }, { "id": 946560, "author_name": "Psi", "author_url": "", "post_date": "2020-07-26T16:41:15.190000", "content": "@serigne Does it also improve CV?
", "votes": 0, "replies": [] }, { "id": 946602, "author_name": "Optimo", "author_url": "", "post_date": "2020-07-26T17:10:28.570000", "content": "@philippsinger for me it improves CV by about 0.01.
\n\nTo be honest I started with a lazy implementation, I was using coarse dropout and I left it as is in my TTA, but I was planning to remove it at some point. I agree that I would expect better results without masking part of my input during inference (I guess everyone would agree not to do mixcut during inference for example). It might also depend on the size of your dropout patches, if they are too big, you may be able to train your model but using it for inference would hurt your results.
\n\nBut I feel very comfortable with doing 20x TTA or even 100x TTA (even though I have better things to do haha), for me it would just converge to the best possible prediction for my model, it might be useless at some point but can't harm much.
", "votes": 1, "replies": [] }, { "id": 946604, "author_name": "Serigne ", "author_url": "", "post_date": "2020-07-26T17:12:03.947000", "content": "Oups I was eager to try it on test data but I didn't even check on validation data ( I used to try Random Augs only on training data so far)
\n\nBut you're right. These augmentations need to be checked on validation data first. I will try to do it more systematically from now.
", "votes": 2, "replies": [] }, { "id": 947551, "author_name": "Signal", "author_url": "", "post_date": "2020-07-27T10:40:42.523000", "content": "@cdeotte You talk about using TTA in both your validation as well as test. I have used TTA in test, but not tried it in validation and will give it a try. So do you end up doing the same augmentations you do in train, on both validation and test? Or do you do some on one and not on others?
", "votes": 0, "replies": [] }, { "id": 947753, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-07-27T13:14:11.913000", "content": "I use TTA in both test and validation. That way I can choose the TTA parameters that maximize validation score. Currently, I use all augmentation techniques for train, validation, and test. (I have not experimented with using less augmentation on validation and test compared with train. Perhaps using less would increase CV score more).
", "votes": 0, "replies": [] }, { "id": 950170, "author_name": "Serigne ", "author_url": "", "post_date": "2020-07-29T08:19:29.713000", "content": "@cdeotte Cutout on TTA improved further my LB score .
\n\nI have run an experiment for fold0 on 256x256.
\n10 TTA random Augment w/o Cutout : LB 0.9357\n10 TTA random Augment w/ Cutout : LB 0.9392
However I haven't check yet any random augment on validation data.
\n\n
@cdeotte Thanks (you replied 18 days ago but just thinking again about this). In your Triple Stratified notebook, if I am reading it right (I run PyTorch so I have not run that notebook), your validation dataset looks like it has augment=False or something along the lines of that, which is why I thought maybe in your approach you didn't do it in validation. But I am possibly just reading your notebook wrong.
Can we also extend it for images with 4 channels? And also why compiler needs to know image = tf.reshape(image,[DIM,DIM,3])?
TensorFlow needs to determine the output shape of each function that operates on tf.data.Dataset(). If you remove image = tf.reshape(image,[DIM,DIM,3]), then i think TensorFlow complains that it can't determine the shape. (I'm not sure why it can't figure it out).
If you have images with 4 channels, just change the 3 to a 4 in two places\n* two = tf.zeros([yb-ya,xb-xa,3])\n* image = tf.reshape(image,[DIM,DIM,3])
Or replace both 3 with image.shape[-1]. That should work and then it can handle whatever number of channels comes its way.
Thanks. I'm bothering you too much these days I guess...
", "votes": 0, "replies": [] } ] }, { "id": 944488, "author_name": "Khan Fashee Monowar (Sawrup)", "author_url": "", "post_date": "2020-07-25T06:42:59.263000", "content": "Can I use this function for JPEG image augmentation ?
", "votes": 1, "replies": [ { "id": 944575, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-07-25T07:36:13.063000", "content": "For JPEG, you can use Albumentations' albumentations.augmentations.transforms.Cutout and albumentations.augmentations.transforms.CoarseDropout, the API is here
Thank you so much.This is very helpful.
", "votes": 1, "replies": [] } ] }, { "id": 944404, "author_name": "Luck", "author_url": "", "post_date": "2020-07-25T04:54:44.833000", "content": "Nice work!
", "votes": 1, "replies": [] }, { "id": 944383, "author_name": "Sarthak khandelwal", "author_url": "", "post_date": "2020-07-25T04:26:19.457000", "content": "Nice and precise implementations @cdeotte! I was curious if AutoAugment can also be implemented in TF as it is proved to improve the model's performance. I found the pytorch implementation of AutoAugmentation here.
Sure. Anything can be coded in TensorFlow. This would be interesting to try out.
", "votes": 0, "replies": [] } ] }, { "id": 944232, "author_name": "Awsaf", "author_url": "", "post_date": "2020-07-25T00:15:18.477000", "content": "Why makes it suitable for TPU and GPU?
", "votes": 1, "replies": [ { "id": 944236, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-07-25T00:20:18.633000", "content": "This code can be executed directly on the GPU/TPU if you desire.
\n\nTensorFlow runs their tf.data.Dataset pipeline on CPU in parallel with GPU/TPU training but once you have TensorFlow code, you can push it into a tf.keras.layer and make it part of your model. Then it will execute on the GPU/TPU inside your model. Sometimes this is faster than CPU parallel sometimes it is not.
However, i don't suggest pushing it into a tf.keras.layer. I suggest just using it with tf.data.Dataset if you're using TensorFlow GPU/TPU.
It seems that we can't use albumentations in tf.data.Dataet of if we can somehow convert all the operations in tensorflow then they'll work fine?
", "votes": 1, "replies": [] }, { "id": 944331, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-07-25T03:01:14.850000", "content": "Yeah, unfortunately we cannot use Albumentations with tf.data.Dataset nor use tf.contrib libraries nor any libraries. All augmentation needs to be written from scratch using TensorFlow language.
I think we now have all the code to do the main augmentations with tf.data.Dataset. Rotation, Sheer, Zoom, Shift here, Cutmix and mixup here, and now Cutout and sparse dropout in this discussion. And TensorFlow already has functions for color manipulations such as contrast, brightness, hue, sharpness, etc.
Thanks again for this Chris. I've been resisting the temptation to reduce the number and types of augmentations. I know that in the long run, I'll be better off. As you mentioned it's not as simple as one would hope to do custom augmentations in TF. I'm keen to try this out.
", "votes": 1, "replies": [] }, { "id": 947848, "author_name": "Stephan", "author_url": "", "post_date": "2020-07-27T14:10:36.407000", "content": "Technically, you can use Albumentations through the tf dataset pipeline using tf.numpy_function. The downside is that TPU's do not support this feature (yet). On a GPU however, this should work.
", "votes": 2, "replies": [ { "id": 947855, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-07-27T14:15:53.297000", "content": "Thanks for letting me know that.
", "votes": 1, "replies": [] } ] }, { "id": 945900, "author_name": "YujiAriyasu", "author_url": "", "post_date": "2020-07-26T07:49:01.987000", "content": "You don't seem to be using hair_aug(), do you think we should use hair augmentation?
", "votes": 2, "replies": [ { "id": 946375, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-07-26T14:42:52.723000", "content": "I have not tried hair augmentation yet. I would guess that it is helpful. Has anyone posted a TensorFlow tf.data.Dataset implementation? Please post the link here.
Here is my notebook implementing Roman's advance hair augmentation in TF:
\n\nSIIM: Data Augmentation in TF: Hair + Batch Affine
\n\nI have not converted it to the batch form, so it does slow down training.
", "votes": 4, "replies": [] }, { "id": 946943, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-07-27T01:00:18.797000", "content": "Great thanks @graf10a , i will play with it.
", "votes": 2, "replies": [] } ] }, { "id": 1182038, "author_name": "Shruti", "author_url": "", "post_date": "2021-02-02T09:15:43.760000", "content": "This is really very useful
", "votes": 0, "replies": [] }, { "id": 1182039, "author_name": "Shruti", "author_url": "", "post_date": "2021-02-02T09:15:43.760000", "content": "This is really very useful
", "votes": 0, "replies": [] }, { "id": 972499, "author_name": "( ͡° ͜ʖ ͡°)", "author_url": "", "post_date": "2020-08-16T15:31:32.547000", "content": "is it same the cutmix ?
\ndef cutmix_aug(image, label=None, DIM=256, BATCH_SIZE=4, PROBABILITY=0.4):
\n # input image - is a batch of images of size [n,dim,dim,3] not a single image of [dim,dim,3]
\n # output - a batch of images with cutmix applied
imgs = []; labs = []\n\nfor j in range(BATCH_SIZE):\n\n #random_uniform( shape, minval=0, maxval=None) \n # DO CUTMIX WITH PROBABILITY DEFINED ABOVE\n P = tf.cast(tf.random.uniform([], 0, 1) <= PROBABILITY, tf.int32)\n\n # CHOOSE RANDOM IMAGE TO CUTMIX WITH\n k = tf.cast(tf.random.uniform([], 0, BATCH_SIZE), tf.int32)\n\n # CHOOSE RANDOM LOCATION\n x = tf.cast(tf.random.uniform([], 0, DIM), tf.int32)\n y = tf.cast(tf.random.uniform([], 0, DIM), tf.int32)\n\n # Beta(1, 1)\n b = tf.random.uniform([], 0, 1) # this is beta dist with alpha=1.0\n\n\n WIDTH = tf.cast(DIM * tf.math.sqrt(1-b),tf.int32) * P\n ya = tf.math.maximum(0,y-WIDTH//2)\n yb = tf.math.minimum(DIM,y+WIDTH//2)\n xa = tf.math.maximum(0,x-WIDTH//2)\n xb = tf.math.minimum(DIM,x+WIDTH//2)\n\n # MAKE CUTMIX IMAGE\n one = image[j,ya:yb,0:xa,:]\n two = image[k,ya:yb,xa:xb,:]\n three = image[j,ya:yb,xb:DIM,:] \n #ya:yb\n middle = tf.concat([one,two,three],axis=1)\n\n img = tf.concat([image[j,0:ya,:,:],middle,image[j,yb:DIM,:,:]],axis=0)\n imgs.append(img)\n\n # MAKE CUTMIX LABEL\n a = tf.cast(WIDTH*WIDTH/DIM/DIM,tf.float32)\n lab1 = label[j,]\n lab2 = label[k,]\n labs.append((1-a)*lab1 + a*lab2)\n\nimage2 = tf.reshape(tf.stack(imgs),(BATCH_SIZE, DIM, DIM, 3))\nlabel2 = tf.reshape(tf.stack(labs),(BATCH_SIZE, 1))\nreturn image2, label2\nthis script handle the label too, cast label to float32
", "votes": 0, "replies": [ { "id": 972924, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-17T01:25:30.137000", "content": "I published a CutMix notebook here. For coarse dropout, I used that CutMix notebook and just replace the second image with all zeros (to perform the dropout).
", "votes": 0, "replies": [] }, { "id": 972971, "author_name": "( ͡° ͜ʖ ͡°)", "author_url": "", "post_date": "2020-08-17T02:22:35.937000", "content": "ah ok, so label still keep 👍
", "votes": 1, "replies": [] } ] }, { "id": 949312, "author_name": "Sagar", "author_url": "", "post_date": "2020-07-28T14:51:35.453000", "content": "This method is cool .\nBut I have a doubt\nas the dataset is imbalanced , does by adding small black boxes to the image increases its performance because it is learning from the presence of small black boxes to predict the image as non malignant and as the number of non malignant class are more it may cause a delusion that the model is working better with the dropout where as it is learning wrong thing which will affect its performance on the test data
", "votes": 0, "replies": [ { "id": 949342, "author_name": "Gilles Vandewiele", "author_url": "", "post_date": "2020-07-28T15:13:19.070000", "content": "As long as the black boxes are added to both benign and malignant images, all is good.
", "votes": 2, "replies": [] }, { "id": 949400, "author_name": "Sagar", "author_url": "", "post_date": "2020-07-28T16:04:50.390000", "content": "Thank you for your time and response .\nI also wanted to know for this competition up sampling the malignant data is better than using weights or do you advise on using both at the same time
", "votes": 0, "replies": [] }, { "id": 949433, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-07-28T16:25:01.307000", "content": "Upsampling or using weights are very similar. For example you can double all the malignant images or you can use weights {0:1, 1:2} (malignant is 2x). One advantage of weights is less train data and faster training epochs. One advantage of upsample is that each batch now contains some malignant and the malignant learning process is more incremental.
Additionally since we're using data augmentation and external data malignant images, then doing upsample adds more variety to the malignant class and hopefully encourages more malignant generalization.
", "votes": 0, "replies": [] } ] }, { "id": 945544, "author_name": "Awsaf", "author_url": "", "post_date": "2020-07-26T00:02:59.827000", "content": "I tried Dropout in another Dataset and I got this error while training...\n\nCompilation failure: Dynamic Spatial Convolution is not supported\n
hmm... i've seen that error before. It relates to TensorFlow knowing the size of stuff. Can your post your code that adds the dropout to your tf.data.Dataset? Make sure you give dropout the correct image size. And make sure that you add dropout before you batch, i.e. ds = ds.map(dropout) before ds = ds.batch(BATCH_SIZE)
dropout(image, DIM=???, PROBABILITY = 0.75, CT = 8, SZ = 0.2)\nreplace ??? with correct size
", "votes": 0, "replies": [] }, { "id": 946183, "author_name": "", "author_url": "", "post_date": "2020-07-26T12:23:43.453000", "content": "", "votes": 0, "replies": [] }, { "id": 946188, "author_name": "Awsaf", "author_url": "", "post_date": "2020-07-26T12:26:32.670000", "content": "Same code runs for GPU and CPU but shows error in TPU..\n```\ndef dropout_img(image,DIM=256, PROBABILITY = 0.5, CT = 50, SZ = 0.06):\n # input image - is one image of size [dim,dim,3] not a batch of [b,dim,dim,3]\n # output - image with CT squares of side size SZ*DIM removed
\n\n# DO DROPOUT WITH PROBABILITY DEFINED ABOVE\nP = tf.cast( tf.random.uniform([],0,1)<PROBABILITY, tf.int32)\nif (P==0)|(CT==0)|(SZ==0): \n return image, mask\n\nfor k in range(CT):\n # CHOOSE RANDOM LOCATION\n x = tf.cast( tf.random.uniform([],0,DIM),tf.int32)\n y = tf.cast( tf.random.uniform([],0,DIM),tf.int32)\n # COMPUTE SQUARE \n WIDTH = tf.cast( SZ*DIM,tf.int32) * P\n ya = tf.math.maximum(0,y-WIDTH//2)\n yb = tf.math.minimum(DIM,y+WIDTH//2)\n xa = tf.math.maximum(0,x-WIDTH//2)\n xb = tf.math.minimum(DIM,x+WIDTH//2)\n # DROPOUT IMAGE\n one = image[ya:yb,0:xa,:]\n two = tf.zeros([yb-ya,xb-xa,4], dtype = image.dtype) \n three = image[ya:yb,xb:DIM,:]\n middle = tf.concat([one,two,three],axis=1)\n image = tf.concat([image[0:ya,:,:],middle,image[yb:DIM,:,:]],axis=0)\n\nimage = tf.reshape(image,[DIM,DIM,-1])\n\nreturn image\n\n\ndef augmentation_img(img, dim=256):
seed = np.int(np.round(np.random.uniform()))\n\nimg = tf.image.random_flip_left_right(img, seed = seed)\n\nimg = transform_img(img, DIM =dim)\nimg = dropout_img(img, DIM=dim, PROBABILITY = 0.5, CT = 50, SZ = 0.06)\n\n\nimg = tf.image.random_contrast(img, 0.8, 1.2)\nimg = tf.image.random_brightness(img, 0.1)\n\nimg = tf.reshape(img, [dim, dim, -1])\n\nreturn img\n\n\n ds = tf.data.TFRecordDataset(files, num_parallel_reads=AUTO)\n ds = ds.cache()
if repeat:\n ds = ds.repeat()\n\nif shuffle: \n ds = ds.shuffle(1024*8)\n opt = tf.data.Options()\n opt.experimental_deterministic = False\n ds = ds.with_options(opt)\n\n ds = ds.map(lambda example: read_unlabeled_tfrecord(example, return_image_names),num_parallel_calls=AUTO) \n\n if augment:\n ds = ds.map(lambda img, name: (augmentation_img(img, dim=dim), name), num_parallel_calls=AUTO)\n\nds = ds.batch(batch_size * REPLICAS)\nds = ds.prefetch(AUTO)\nreturn ds\n```
", "votes": 0, "replies": [] } ] }, { "id": 967894, "author_name": "", "author_url": "", "post_date": "2020-08-12T15:10:31.593000", "content": "", "votes": 0, "replies": [] }, { "id": 953250, "author_name": "Aditya Baurai", "author_url": "", "post_date": "2020-07-31T16:14:52.437000", "content": "Thank you !!
", "votes": 1, "replies": [] }, { "id": 951260, "author_name": "Shravan Raikar", "author_url": "", "post_date": "2020-07-30T02:49:07.167000", "content": "Thanks for Sharing!
", "votes": 1, "replies": [] }, { "id": 948203, "author_name": "Sanjay K", "author_url": "", "post_date": "2020-07-27T18:14:56.500000", "content": "Nice work, thanks for sharing!
", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "944208": "When using TensorFlow's `tf.data.Dataset()` we cannot use Albumenations to do our augmentation. Instead we must write our own functions in TensorFlow. Previously I posted how to perform (1) Rotation, Sheer, Zoom, Shift for GPU/TPU [here][1] and [here][3] (2) Cutmix and Mixup for GPU/TPU [here][2]\n\n# Coarse Dropout and Cutout Augmentation for GPU/TPU\nCoarse Dropout and Cutout augmentation are techniques to prevent overfitting and encourage generalization. They randomly remove rectangles from training images. By removing portions of the images, we challenge our models to pay attention to the entire image because it never knows what part of the image will be present. (This is similar and different to dropout layer within a CNN).\n\n* Cutout is the technique of removing 1 large rectangle of random size \n* Coarse dropout is the technique of removing many small rectanges of similar size. \n\nBy changing the parameters below, we can have either coarse dropout or cutout. (For cutout, you'll need to add `tf.random.uniform` for random size. I leave this as an exercise for the reader).\n\n\n\n\n# Starter Notebook\nI've posted a starter notebook [here][4] showing how to do coarse dropout with TFRecords and `tf.data.Dataset`. The starter notebook also demonstrates upsampling (oversampling) the minority class.\n\n# Code\nThe reason that this code looks overly complicated just to turn a portion of an array to zeros is because you cannot do `image[ya:yb,xa:xb,:] = 0` in TensorFlow. TF does not allow index assignment. Instead we break an image into five pieces and then concatenate the five pieces while replacing the middle piece with all zeros.\n\n\n\n def dropout(image, DIM=256, PROBABILITY = 0.75, CT = 8, SZ = 0.2):\n # input - one image of size [dim,dim,3] not a batch of [b,dim,dim,3]\n # output - image with CT squares of side size SZ*DIM removed\n \n # DO DROPOUT WITH PROBABILITY DEFINED ABOVE\n P = tf.cast( tf.random.uniform([],0,1) < PROBABILITY, tf.int32)\n if (P == 0)|(CT == 0)|(SZ == 0): return image\n \n for k in range( CT ):\n # CHOOSE RANDOM LOCATION\n x = tf.cast( tf.random.uniform([],0,DIM),tf.int32)\n y = tf.cast( tf.random.uniform([],0,DIM),tf.int32)\n # COMPUTE SQUARE \n WIDTH = tf.cast( SZ*DIM,tf.int32) * P\n ya = tf.math.maximum(0,y-WIDTH//2)\n yb = tf.math.minimum(DIM,y+WIDTH//2)\n xa = tf.math.maximum(0,x-WIDTH//2)\n xb = tf.math.minimum(DIM,x+WIDTH//2)\n # DROPOUT IMAGE\n one = image[ya:yb,0:xa,:]\n two = tf.zeros([yb-ya,xb-xa,3]) \n three = image[ya:yb,xb:DIM,:]\n middle = tf.concat([one,two,three],axis=1)\n image = tf.concat([image[0:ya,:,:],middle,image[yb:DIM,:,:]],axis=0)\n \n # RESHAPE HACK SO TPU COMPILER KNOWS SHAPE OF OUTPUT TENSOR \n image = tf.reshape(image,[DIM,DIM,3])\n return image\n\n[1]: https://www.kaggle.com/cdeotte/rotation-augmentation-gpu-tpu-0-96\n[2]: https://www.kaggle.com/cdeotte/cutmix-and-mixup-on-gpu-tpu\n[3]: https://www.kaggle.com/cdeotte/triple-stratified-kfold-with-tfrecords\n[4]: https://www.kaggle.com/cdeotte/tfrecord-experiments-upsample-and-coarse-dropout", "953624": "I've created a version that accepts batches of images, for anyone who is interested. I've tested it on a GPU setup, so I can't be sure this works on a TPU yet.\n\nedit: I see that the prob arg isn't actually utilized yet, I'll fix this tommorow.\nedit 2: probabilities now included :)\nedit 3: replaced a type in tf.scatter_nd where the image sizes were fixed to 128,128 -> now adjusted for variable sizes\n\n```python\ndef coarse_dropout(images, square_size, num_squares, prob):\n ''' Coarse image dropout \n Coarse dropout masks are created by using tf.scatter_nd\n Which can update matrices according to coordinates specified\n by the user. \n\n Parameters\n ----------\n images : tf Tensor\n A batch of images of size [BS, H, W, C] \n square_size : float\n A float between [0,1], a percentage of the image size\n num_squares : int > 0\n The number of dropout squares per single image, must be >0\n prob : float\n A float between [0,1], probability that dropout is used for this batch\n\n Yields\n ------\n TYPE\n DESCRIPTION.\n\n '''\n # random dropout\n prob = tf.cast( tf.random.uniform([],0,1) < prob, tf.int32)\n if (prob == 0): return images\n\n img_shape = images.shape\n _, h, w, c = img_shape[0], img_shape[1], img_shape[2], img_shape[3]\n \n # For some odd reason, the batch size is lost in the processing pipeline\n # seriously, how can it get lost ._., it literally says ds.batch()...\n # tensorflow shenanigans...\n bs = tf.cast(tf.reduce_sum(tf.ones_like(images)) / (images.shape[1] * images.shape[2] * images.shape[3]), tf.int32)\n\n\n # size of square in pixels (ssp), ASSUMING SQUARE IMAGES!\n ssp = tf.cast(tf.math.ceil(h * square_size), tf.int32)\n\n # Create random start x coordinates\n coords_x = tf.random.uniform([bs, num_squares], \n minval=0, \n maxval= (h - ssp), \n dtype=tf.int32) \n\n # Create ranges from start to the end of the line\n # [ssp, bs, num_squares]\n coords_x = tf.linspace(coords_x, coords_x + ssp - 1, ssp)\n\n # [num_squares, bs, ssp]\n coords_x = tf.cast(tf.transpose(coords_x), tf.int32)\n\n # Create random start y coordinates\n coords_y = tf.random.uniform([bs, num_squares], \n minval=0, \n maxval= (h - ssp), \n dtype=tf.int32) \n\n # Create ranges from start to the end of the line\n # [ssp, bs, num_squares]\n coords_y = tf.linspace(coords_y, coords_y + ssp - 1, ssp)\n\n # [num_squares, bs, ssp]\n coords_y = tf.cast(tf.transpose(coords_y), tf.int32)\n\n\n # Create coordinate range combinations \n # and reshape to [bs, num_squares, 1, ssp * ssp]\n grid_y = tf.reshape(tf.tile(coords_y, [1,1,ssp]), \n (bs, num_squares, 1, ssp * ssp))\n\n # and reshape to [bs, num_squares, ssp, ssp], transpose the inner matrices\n grid_y = tf.transpose(tf.reshape(grid_y, \n (bs, num_squares, ssp, ssp)), \n (0, 1, 3, 2))\n\n # Repeat for x coordinates\n grid_x = tf.reshape(tf.tile(coords_x, [1,1,ssp]), \n (bs, num_squares, 1, ssp * ssp))\n\n grid_x = tf.reshape(grid_x, (bs, num_squares, ssp, ssp))\n\n # Stack the grids into a single matrix\n # grid is [2, bs, num_squares, ssp, ssp]\n grid = tf.stack([grid_y, grid_x], axis=0)\n\n # Transpose and reshape [ bs, ssp * ssp * num_squares, 2] \n # Creates an array of 2D coordinates ([[x1,y1], [x2, y2], ..., [xn, yn]])\n # over all squares (num_squares), for each combination (ssp*ssp) of coordinates \n # and each batch (bs)\n grid = tf.reshape(tf.transpose(grid, (1, 4, 3, 2, 0)), \n (bs, ssp * ssp * num_squares, 2))\n\n # [bs, sz*sz*num_squares, 2]\n #grid = tf.reshape(grid, (bs, ssp*ssp*num_squares, 2))\n\n # create batch indices [0,..., bs] and reshape to [ssp * ssp * num_squares, bs]\n batch_indices = tf.reshape(tf.tile(tf.range(0, bs), \n [ssp * ssp * num_squares]), \n (ssp * ssp * num_squares, bs))\n\n # Transpose to get the right order, and reshape to match grid shape\n batch_indices = tf.reshape(tf.transpose(batch_indices), \n (bs, ssp * ssp * num_squares, 1))\n\n # concatenate batch indices with the grid\n # these yield 3D coordinates like e.g.\n # [[bs0, x1, y1], [bs0, x2, y1], ..., [bsn, xn, yn]]\n grid = tf.concat([batch_indices, grid], axis=2)\n\n # create a matrix of zeros, and update the matrix with the grid indices\n # this essentially creates a mask with coarse dropout squares\n masks = tf.scatter_nd(grid[tf.newaxis,...], \n tf.ones([1,bs,ssp*ssp*num_squares]) * -1, \n shape=(bs, h, w)) +1\n \n # Due to overlap of squares, some get coordinates get updated twice\n # and result in values < -1, clip these values\n masks = tf.clip_by_value(masks, 0, 1)\n\n return images * masks[..., tf.newaxis] \n```", "945580": "Notice how experiment 1 **without** dropout has a larger gap between train AUC and validation AUC. While experiment 2 **with** dropout has a smaller gap. In the below example, dropout has helped the CNN generalize thus **increasing validation** AUC while preventing training AUC from overfitting (i.e. **reducing training** AUC). Notebook [here][1]\n## WITHOUT coarse dropout\n\n## WITH coarse dropout\n\n[1]: https://www.kaggle.com/cdeotte/tfrecord-experiments-upsample-and-coarse-dropout\n\n", "944950": "This is really useful for tensorflow users in this competition @cdeotte :D ", "944692": "UPDATE: If your forked version 15 or earlier of the starter notebook, note that it does **not** apply **dropout**. (It only applies **upsample**). During training, in the call to `get_dataset()`, I forgot to pass the dropout parameters (so they use the default of zero). In the 3 lines below, the last line was missing in notebook version 15 and earlier. I just committed notebook now. Versions 16 onward will have this line added and **will apply** dropout.\n\n get_dataset(files_train, augment=True, shuffle=True, repeat=True,\n dim=IMG_SIZES[fold], batch_size = BATCH_SIZES[fold],\n droprate = DROP_FREQ[fold], dropct = DROP_CT[fold], dropsize = DROP_SIZE[fold])\n\nWe also need to add this last line to validation TTA prediction and test TTA prediction.", "954351": "@cdeotte Thank you for your great work! Please allow me one question.\nHow do you think about the difference between Coarse Dropout and GridMask ?", "952933": "@cdeotte Hi Chris! I really would like to try to find ensembling weights by using a machine learning algorithm and cv oof auc scores. Did you ever try this? Do you have an url or tip how to do this?\nBest Roman", "952157": "@cdeotte a simple question: Do you have any ratio between image size and rest of the droprate, dropct and dropsize or you keep them same for all? Thanks for your great sources for this competition again...", "947100": "Good post Chris!", "945624": "In my understanding, we should not use dropout or cutout when predicting validation data or test data. Am I right?", "944899": "Can we also extend it for images with 4 channels? And also why compiler needs to know `image = tf.reshape(image,[DIM,DIM,3])`? ", "944488": "Can I use this function for JPEG image augmentation ?", "944404": "Nice work! ", "944383": "Nice and precise implementations @cdeotte! I was curious if `AutoAugment` can also be implemented in TF as it is proved to improve the model's performance. I found the pytorch implementation of `AutoAugmentation` [here](https://github.com/DeepVoltaire/AutoAugment). ", "944232": "Why makes it suitable for TPU and GPU? ", "944227": "Thanks again for this Chris. I've been resisting the temptation to reduce the number and types of augmentations. I know that in the long run, I'll be better off. As you mentioned it's not as simple as one would hope to do custom augmentations in TF. I'm keen to try this out. ", "947848": "Technically, you can use Albumentations through the tf dataset pipeline using tf.numpy_function. The downside is that TPU's do not support this feature (yet). On a GPU however, this should work.", "945900": "You don't seem to be using hair_aug(), do you think we should use hair augmentation?", "1182038": "This is really very useful", "1182039": "This is really very useful", "972499": "is it same the cutmix ?\n\ndef cutmix_aug(image, label=None, DIM=256, BATCH_SIZE=4, PROBABILITY=0.4):\n # input image - is a batch of images of size [n,dim,dim,3] not a single image of [dim,dim,3]\n # output - a batch of images with cutmix applied\n\n imgs = []; labs = []\n \n for j in range(BATCH_SIZE):\n \n #random_uniform( shape, minval=0, maxval=None) \n # DO CUTMIX WITH PROBABILITY DEFINED ABOVE\n P = tf.cast(tf.random.uniform([], 0, 1) <= PROBABILITY, tf.int32)\n \n # CHOOSE RANDOM IMAGE TO CUTMIX WITH\n k = tf.cast(tf.random.uniform([], 0, BATCH_SIZE), tf.int32)\n \n # CHOOSE RANDOM LOCATION\n x = tf.cast(tf.random.uniform([], 0, DIM), tf.int32)\n y = tf.cast(tf.random.uniform([], 0, DIM), tf.int32)\n \n # Beta(1, 1)\n b = tf.random.uniform([], 0, 1) # this is beta dist with alpha=1.0\n \n\n WIDTH = tf.cast(DIM * tf.math.sqrt(1-b),tf.int32) * P\n ya = tf.math.maximum(0,y-WIDTH//2)\n yb = tf.math.minimum(DIM,y+WIDTH//2)\n xa = tf.math.maximum(0,x-WIDTH//2)\n xb = tf.math.minimum(DIM,x+WIDTH//2)\n \n # MAKE CUTMIX IMAGE\n one = image[j,ya:yb,0:xa,:]\n two = image[k,ya:yb,xa:xb,:]\n three = image[j,ya:yb,xb:DIM,:] \n #ya:yb\n middle = tf.concat([one,two,three],axis=1)\n\n img = tf.concat([image[j,0:ya,:,:],middle,image[j,yb:DIM,:,:]],axis=0)\n imgs.append(img)\n \n # MAKE CUTMIX LABEL\n a = tf.cast(WIDTH*WIDTH/DIM/DIM,tf.float32)\n lab1 = label[j,]\n lab2 = label[k,]\n labs.append((1-a)*lab1 + a*lab2)\n\n image2 = tf.reshape(tf.stack(imgs),(BATCH_SIZE, DIM, DIM, 3))\n label2 = tf.reshape(tf.stack(labs),(BATCH_SIZE, 1))\n return image2, label2\n\nthis script handle the label too, cast label to float32", "949312": "This method is cool .\nBut I have a doubt\nas the dataset is imbalanced , does by adding small black boxes to the image increases its performance because it is learning from the presence of small black boxes to predict the image as non malignant and as the number of non malignant class are more it may cause a delusion that the model is working better with the dropout where as it is learning wrong thing which will affect its performance on the test data \n\n", "945544": "I tried Dropout in another Dataset and I got this error while training...\n```\nCompilation failure: Dynamic Spatial Convolution is not supported\n```", "967894": "", "953250": "Thank you !!", "951260": "Thanks for Sharing!", "948203": "Nice work, thanks for sharing!" } }