![\"\"](\"https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Aug-2021/ex1b.png\")
\n
![\"\"](\"https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Aug-2021/ex2b.png\")
\n
![\"\"](\"https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Aug-2021/ex3b.png\")
\n
![\"\"](\"https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Aug-2021/ex4b.png\")
\n
![\"\"](\"https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Aug-2021/ex5b.png\")
\n
![\"\"](\"https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Aug-2021/ex6b.png\")
\n
![\"\"](\"https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Aug-2021/ex7b.png\")
\n
![\"\"](\"https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Aug-2021/ex8b.png\")
What an interesting competition and difficult puzzle! Thank you Kaggle and SETI for hosting. I started this competition last Wednesday and worked hard every day. In this time, I closed CV LB gap from 0.125 to 0.104. I would have liked to have more time and continue the puzzle of closing the CV LB gap. Below is my journey over the past 7 days.
\nLast Wednesday, Thursday, and Friday, I built dozens of my own models without reading public discussions nor notebooks. I was excited when my best model achieved CV 0.890 but surprised and disappointed when I submitted and received LB 0.765 🙁This was a 0.125 gap between CV and LB. Something is going on!
\nMy best model used only \"on\" cadence in a spatial stack resized to 768x768 and fed into a EfficientNetB4 backbone. The only augmentation was horizontal flip and vertical flip. It used cosine learning schedule and 15 epochs.
\nAt this point, I thought the gap was caused by new pattens in test data. So, on Friday and Saturday, i starting building more creative models that used more than just classify train patterns.
\ntarget=0. Increased CV 0.003 but decreased LB 0.010target=0. Did not affect CV but decreased LB like 0.050!test image as target=1 and train image as target=0. The hope was that this model would find any new patterns in test data because that would be the difference between train and test. To my surprise the model achieved CV 0.99 but LB was terrible 🙁img[::2] versus \"off\" cadence img[1::2]. So discard target column and use cadence on as target=1 and cadence off as target=0. The advantage here is that we can train directly on test data without train data!My favorite model was the last which achieved LB 0.550 and did not use any train data. It simply compared crops of test image \"on\" cadence with crops of test image \"off\" cadence and searched for dissimilarity.
\nBy Saturday, i was frustrated that I couldn't reach Bronze zone. How was everyone doing it?
\nI began reading discussions and notebooks. I found @tuckerarrants amazing \"SETI - Learned Image Resizing\" notebook here. What impressed me most was that his GAP was only 0.102! His CV was 0.846 and his LB was 0.744. This was a smaller gap than my model.
\nWithout understanding why, I quickly ran his notebook locally with larger models and larger image sizes to secure bronze position. The following table are results for fold 0 only.
\nNote that Tucker's notebook has a resize module before the EfficientNet, so we put image_in into the resize module and image_out comes out. We then feed image_out into the EfficientNet backbone:
| Backbone | \nImage In | \nImage Out | \nCV - fold 0 | \nLB - fold 0 | \n
|---|---|---|---|---|
| EB3 | \n512 | \n256 | \n0.859 | \n0.764 | \n
| EB3 | \n640 | \n320 | \n0.870 | \n0.768 | \n
| EB3 | \n768 | \n384 | \n0.870 | \n\n |
| EB3 | \norig | \n448 | \n0.872 | \n\n |
| EB4 | \n640 | \n320 | \n0.873 | \n0.770 | \n
| EB4 | \n768 | \n384 | \n0.872 | \n\n |
| EB4 | \norig | \n448 | \n0.869 | \n\n |
| EB6 | \n640 | \n320 | \n0.872 | \n0.770 | \n
| Ensemble | \n\n | \n | 0.883 | \n0.777 | \n
Then 5-Fold was LB 0.778 and post process got LB 0.779
\nFor the past few days, Nvidia V100 GPUs were running constantly to train the models in the table above. I am learning PyTorch and discovered a great trick here. Using Tucker's notebook, you can add 1 line of code to train PyTorch with multiple GPUs (for large backbones and batch sizes). Just replace the one line model.to(device) with the three lines:
import torch.nn as nn\nmodel = nn.DataParallel(model)\nmodel.to(device)\nThat's it! With 1 line of code, we can train larger models and larger batch sizes by using multiple Nvidia GPU.
\nMany people (including @dragonzhang) were confused why they could not train efficientnet B4 with Tucker's notebook. That is because timm does not contain an imagenet pretrained efficientnet B4. (So if you use model_name='efficientnet_b4' you train with a not-pretrained effnet)
Timm only contains pretrains of EB4, 5, 6, 7 of the TF converted weights, tf_efficientnet_b4, tf_efficientnet_b4_ns, tf_efficientnet_b4_ap. The first is original efficient net, the next two are pretrained with noisy student and adv prop. To view the full list of available timm pretrained backbones, type
import timm\nfrom pprint import pprint\nmodel_names = timm.list_models(pretrained=True)\npprint(model_names)\nAfter securing bronze medal position, I began to investigate the CV LB gap again. Four hours before competition deadline, I plotted the difference between my model and public notebook model. I sorted by test images where the difference between my model prediction and public model prediction were greatest. To my surprise, I found a mysterious \"S\" shaped test pattern!
\nThe image on the left is original comp test data \"on\" cadence vertical stack. With time on x axis and frequency on y axis. The second image is pixel histogram. The third image is \"on\" cadence with emboss, blur, and adaptive histogram equalization. (Filter code shown below) The fourth image (right image) is the \"off\" cadence.
\nimport cv2\nclahe = cv2.createCLAHE(clipLimit=16.0, tileGridSize=(8,8))\n# FILTERS FOR BETTER IMAGE DISPLAY\nimg = np.vstack(cadence[::2,])\nimg = img[1:,1:] - img[:-1,:-1] #emboss\nimg -= np.min(img)\nimg /= np.max(img)\nimg = (img*255).astype('uint8')\nimg = cv2.GaussianBlur(img,(5,5),0)\nimg = clahe.apply(img)\nAbove the left image we see the public notebook prediction value. And above the third image we see my model's prediction value.
\n
\n
\n
\n
\n
\n
\n
\n
With only 4 hours remaining, I trained a model on the images that my model and public notebook model disagreed most on. I then inferred the entire test data. For all images with p>0.6, I increased their submission prediction. This post processed closed the CV LB gap by 0.001 and increased LB by 0.001. Thus there is something special about these \"S\" patterns!
After the comp ended, i have been continuing with experiments to determine why my original model's CV LB is large. It appears that mixup is very important to teach your model to generalize to test data which is different than the train data. It both closes the CV LB gap and boosts CV LB.
\nUsing 1 fold of EB4 and image size 768x768 with only Hflip, Vflip, and Mixup (alpha=3, max target). One can achieve Silver medal with public LB 0.786 and private LB 0.781. Notebook here. (Also important is large image size and large backbone).
\n![\"\"](\"https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Aug-2021/mixup3.png\")
To climb from Silver medal to Gold medal, we need to detect \"s-shapes\" and remove noise. The first place winning solution here explains how. Additionally, repeatedly using test pseudo labels also helps climb to top of Gold Medals! (and using old data before reset can help too).
\nI posted a discussion here about grad cam and a notebook here using grad cam. Using grad cam can help us understand what makes a particular image a target=1. For example, our model can make a test prediction and then circle what it thinks is causing a target=1 prediction. The image below is 100% generated by code and no human labeling!
\n![\"\"](\"https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Aug-2021/grad_cam_3.png\")
What an interesting competition and difficult puzzle! Thank you Kaggle and SETI for hosting. I started this competition last Wednesday and worked hard every day. In this time, I closed CV LB gap from 0.125 to 0.104. I would have liked to have more time and continue the puzzle of closing the CV LB gap. Below is my journey over the past 7 days.
\nLast Wednesday, Thursday, and Friday, I built dozens of my own models without reading public discussions nor notebooks. I was excited when my best model achieved CV 0.890 but surprised and disappointed when I submitted and received LB 0.765 🙁This was a 0.125 gap between CV and LB. Something is going on!
\nMy best model used only \"on\" cadence in a spatial stack resized to 768x768 and fed into a EfficientNetB4 backbone. The only augmentation was horizontal flip and vertical flip. It used cosine learning schedule and 15 epochs.
\nAt this point, I thought the gap was caused by new pattens in test data. So, on Friday and Saturday, i starting building more creative models that used more than just classify train patterns.
\ntarget=0. Increased CV 0.003 but decreased LB 0.010target=0. Did not affect CV but decreased LB like 0.050!test image as target=1 and train image as target=0. The hope was that this model would find any new patterns in test data because that would be the difference between train and test. To my surprise the model achieved CV 0.99 but LB was terrible 🙁img[::2] versus \"off\" cadence img[1::2]. So discard target column and use cadence on as target=1 and cadence off as target=0. The advantage here is that we can train directly on test data without train data!My favorite model was the last which achieved LB 0.550 and did not use any train data. It simply compared crops of test image \"on\" cadence with crops of test image \"off\" cadence and searched for dissimilarity.
\nBy Saturday, i was frustrated that I couldn't reach Bronze zone. How was everyone doing it?
\nI began reading discussions and notebooks. I found @tuckerarrants amazing \"SETI - Learned Image Resizing\" notebook here. What impressed me most was that his GAP was only 0.102! His CV was 0.846 and his LB was 0.744. This was a smaller gap than my model.
\nWithout understanding why, I quickly ran his notebook locally with larger models and larger image sizes to secure bronze position. The following table are results for fold 0 only.
\nNote that Tucker's notebook has a resize module before the EfficientNet, so we put image_in into the resize module and image_out comes out. We then feed image_out into the EfficientNet backbone:
| Backbone | \nImage In | \nImage Out | \nCV - fold 0 | \nLB - fold 0 | \n
|---|---|---|---|---|
| EB3 | \n512 | \n256 | \n0.859 | \n0.764 | \n
| EB3 | \n640 | \n320 | \n0.870 | \n0.768 | \n
| EB3 | \n768 | \n384 | \n0.870 | \n\n |
| EB3 | \norig | \n448 | \n0.872 | \n\n |
| EB4 | \n640 | \n320 | \n0.873 | \n0.770 | \n
| EB4 | \n768 | \n384 | \n0.872 | \n\n |
| EB4 | \norig | \n448 | \n0.869 | \n\n |
| EB6 | \n640 | \n320 | \n0.872 | \n0.770 | \n
| Ensemble | \n\n | \n | 0.883 | \n0.777 | \n
Then 5-Fold was LB 0.778 and post process got LB 0.779
\nFor the past few days, Nvidia V100 GPUs were running constantly to train the models in the table above. I am learning PyTorch and discovered a great trick here. Using Tucker's notebook, you can add 1 line of code to train PyTorch with multiple GPUs (for large backbones and batch sizes). Just replace the one line model.to(device) with the three lines:
import torch.nn as nn\nmodel = nn.DataParallel(model)\nmodel.to(device)\nThat's it! With 1 line of code, we can train larger models and larger batch sizes by using multiple Nvidia GPU.
\nMany people (including @dragonzhang) were confused why they could not train efficientnet B4 with Tucker's notebook. That is because timm does not contain an imagenet pretrained efficientnet B4. (So if you use model_name='efficientnet_b4' you train with a not-pretrained effnet)
Timm only contains pretrains of EB4, 5, 6, 7 of the TF converted weights, tf_efficientnet_b4, tf_efficientnet_b4_ns, tf_efficientnet_b4_ap. The first is original efficient net, the next two are pretrained with noisy student and adv prop. To view the full list of available timm pretrained backbones, type
import timm\nfrom pprint import pprint\nmodel_names = timm.list_models(pretrained=True)\npprint(model_names)\nAfter securing bronze medal position, I began to investigate the CV LB gap again. Four hours before competition deadline, I plotted the difference between my model and public notebook model. I sorted by test images where the difference between my model prediction and public model prediction were greatest. To my surprise, I found a mysterious \"S\" shaped test pattern!
\nThe image on the left is original comp test data \"on\" cadence vertical stack. With time on x axis and frequency on y axis. The second image is pixel histogram. The third image is \"on\" cadence with emboss, blur, and adaptive histogram equalization. (Filter code shown below) The fourth image (right image) is the \"off\" cadence.
\nimport cv2\nclahe = cv2.createCLAHE(clipLimit=16.0, tileGridSize=(8,8))\n# FILTERS FOR BETTER IMAGE DISPLAY\nimg = np.vstack(cadence[::2,])\nimg = img[1:,1:] - img[:-1,:-1] #emboss\nimg -= np.min(img)\nimg /= np.max(img)\nimg = (img*255).astype('uint8')\nimg = cv2.GaussianBlur(img,(5,5),0)\nimg = clahe.apply(img)\nAbove the left image we see the public notebook prediction value. And above the third image we see my model's prediction value.
\n
\n
\n
\n
\n
\n
\n
\n
With only 4 hours remaining, I trained a model on the images that my model and public notebook model disagreed most on. I then inferred the entire test data. For all images with p>0.6, I increased their submission prediction. This post processed closed the CV LB gap by 0.001 and increased LB by 0.001. Thus there is something special about these \"S\" patterns!
After the comp ended, i have been continuing with experiments to determine why my original model's CV LB is large. It appears that mixup is very important to teach your model to generalize to test data which is different than the train data. It both closes the CV LB gap and boosts CV LB.
\nUsing 1 fold of EB4 and image size 768x768 with only Hflip, Vflip, and Mixup (alpha=3, max target). One can achieve Silver medal with public LB 0.786 and private LB 0.781. Notebook here. (Also important is large image size and large backbone).
\n
To climb from Silver medal to Gold medal, we need to detect \"s-shapes\" and remove noise. The first place winning solution here explains how. Additionally, repeatedly using test pseudo labels also helps climb to top of Gold Medals! (and using old data before reset can help too).
\nI posted a discussion here about grad cam and a notebook here using grad cam. Using grad cam can help us understand what makes a particular image a target=1. For example, our model can make a test prediction and then circle what it thinks is causing a target=1 prediction. The image below is 100% generated by code and no human labeling!
\n
Great write up Chris and congratulations on getting bronze. I'm glad my notebook was useful during your 7 day journey!
", "rawMarkdown": "Great write up Chris and congratulations on getting bronze. I'm glad my notebook was useful during your 7 day journey!", "votes": 3 }, { "id": 1482912, "postDate": "2021-08-20T10:50:09.637Z", "content": "Looks like we lost another one to PyTorch. 😄
\nThanks for sharing Chris. A pleasure as always.
Thank you for your detailed and clear explanation of your own thinking process, thank you for your open source code, and all kinds of thoughtful notes in the article. These are very friendly to novices like me, and I can always learn a lot from you.😄
", "rawMarkdown": "Thank you for your detailed and clear explanation of your own thinking process, thank you for your open source code, and all kinds of thoughtful notes in the article. These are very friendly to novices like me, and I can always learn a lot from you.😄", "votes": 1 }, { "id": 1488383, "postDate": "2021-08-24T08:51:33.210Z", "content": "Wow! It's a grand post. Brilliantly done and congratulations @cdeotte
", "rawMarkdown": "Wow! It's a grand post. Brilliantly done and congratulations @cdeotte ", "votes": 1 }, { "id": 1484548, "postDate": "2021-08-21T12:07:13.687Z", "content": "Hi Chris, congratulations and thanks for this writeup, this really helps me to learn. Just two quick questions:
\nHi Markus. My model is more than just an EfficientNet. It has a special module beforehand which learns to resize the image by itself. Here's a diagram of just the module. (The efficientnet is not in the picture):
\n![\"\"](\"https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Aug-2021/resize.png\")
\nSo first i load the Numpy array and resize it to image_in with img = np.vstack( img[::2] ) and img = cv2.resize(img,(image_in,image_in)).
Then it gets fed into the module. Then the module learns to resize it to image_out. Finally that result is fed into the EfficientNet. I used Tucker's public notebook here
Note that i used this public notebook because i had a few days left and could not figure out how to close the CV LB gap with my own model. I am now investigating why that notebook has a smaller CV LB gap than my model. I have discovered that this special module is not needed. I believe the secret of that notebook is that it uses mixup augmentation whereas my model did not.
", "rawMarkdown": "Hi Markus. My model is more than just an EfficientNet. It has a special module beforehand which learns to resize the image by itself. Here's a diagram of just the module. (The efficientnet is not in the picture):\n\nSo first i load the Numpy array and resize it to `image_in` with `img = np.vstack( img[::2] )` and `img = cv2.resize(img,(image_in,image_in))`. \n\nThen it gets fed into the module. Then the module learns to resize it to `image_out`. Finally that result is fed into the EfficientNet. I used Tucker's public notebook [here][1]\n\nNote that i used this public notebook because i had a few days left and could not figure out how to close the CV LB gap with my own model. I am now investigating why that notebook has a smaller CV LB gap than my model. I have discovered that this special module is not needed. I believe the secret of that notebook is that it uses mixup augmentation whereas my model did not.\n\n[1]: https://www.kaggle.com/tuckerarrants/seti-learned-image-resizing", "votes": 2 }, { "id": 1484572, "postDate": "2021-08-21T12:42:40.013Z", "content": "@friedchips There are two efficientnet's you can use for TensorFlow. You can use the one in https://keras.io/api/applications/ which was pretrained with image pixels of 0 thru 255. Or you can use pip install efficientnet here which was pretrained with image pixels normalized with mean 0 and std 1.
If you use the PyTorch efficientnet's. They are pretrained with pixels mean 0 and std 1.
\nIt will help your efficientnet converge faster to use the same pixel range that the pretraining used. However if it you get it wrong, the model should still converge and learn to adjust to what you are feeding it.
", "rawMarkdown": "@friedchips There are two efficientnet's you can use for TensorFlow. You can use the one in `https://keras.io/api/applications/` which was pretrained with image pixels of 0 thru 255. Or you can use `pip install efficientnet` [here][1] which was pretrained with image pixels normalized with mean 0 and std 1. \n\nIf you use the PyTorch efficientnet's. They are pretrained with pixels mean 0 and std 1.\n\nIt will help your efficientnet converge faster to use the same pixel range that the pretraining used. However if it you get it wrong, the model should still converge and learn to adjust to what you are feeding it.\n\n[1]: https://pypi.org/project/efficientnet/", "votes": 2 }, { "id": 1484743, "postDate": "2021-08-21T14:48:05.130Z", "content": "Thanks! I had overlooked your image resizing stage. I just rescaled the original input images with a Recaling layer before feeding them into my EffNetB2 and also saw that the bigger the better, finally going with 768x768 as input to the EffNet.
\nYes, Mixup was extremely helpful, especially with alpha very large (5.0). It just never occurred to me to not mix the labels…
\nThanks for the other efficientnet implementation. My Keras efficientnet converged much worse (CV 0.82 instead of 0.87) when I fed it with 0-1 data before realizing that it expects 0-255.
", "rawMarkdown": "Thanks! I had overlooked your image resizing stage. I just rescaled the original input images with a Recaling layer before feeding them into my EffNetB2 and also saw that the bigger the better, finally going with 768x768 as input to the EffNet.\n\nYes, Mixup was extremely helpful, especially with alpha very large (5.0). It just never occurred to me to not mix the labels...\n\nThanks for the other efficientnet implementation. My Keras efficientnet converged much worse (CV 0.82 instead of 0.87) when I fed it with 0-1 data before realizing that it expects 0-255.", "votes": 2 } ] }, { "id": 1487265, "postDate": "2021-08-23T14:06:19.443Z", "content": "Update. I have been conducting more experiments. I have updated my post to identify the Silver Medal Magic. And the Gold Medal Magic!
", "rawMarkdown": "Update. I have been conducting more experiments. I have updated my post to identify the Silver Medal Magic. And the Gold Medal Magic!", "votes": 2, "replies": [ { "id": 1491516, "postDate": "2021-08-26T12:50:15.690Z", "content": "Thanks @cdeotte great insights as usual.
\nI realise many contestants plot pixel differences like you did in image comps. May I know if you can share the code you used to plot the above diagrams so I can compare if I’m doing it right
", "rawMarkdown": "Thanks @cdeotte great insights as usual. \n\nI realise many contestants plot pixel differences like you did in image comps. May I know if you can share the code you used to plot the above diagrams so I can compare if I’m doing it right ", "votes": 2 }, { "id": 1491537, "postDate": "2021-08-26T12:59:34.140Z", "content": "The code is posted above. For \"on\" cadence, i would read NumPy like
\nfile = numpy .load(path+name[0]+'/'+name+'.npy').astype('float32')\nON_CADENCE = file[::2,]\nAnd for the same \"off\" cadence, use the same name and load
file = numpy .load(path+name[0]+'/'+name+'.npy').astype('float32')\nOFF_CADENCE = file[1::2,]\nNext, i apply the following filter to make the spectrograms easier to see with human eye
\nimport cv2\nclahe = cv2.createCLAHE(clipLimit=16.0, tileGridSize=(8,8))\nimg = np.vstack(CADENCE)\nimg = img[1:,1:] - img[:-1,:-1] #emboss\nimg -= np.min(img)\nimg /= np.max(img)\nimg = (img*255).astype('uint8')\nimg = cv2.GaussianBlur(img,(5,5),0)\nimg = clahe.apply(img)\nTo plot 4 tiles in a row, you use subplot like
plt.figure(figsize=(20,5)\nplt.subplot(1,4,1)\nplt.imshow(img1)\nplt.subplot(1,4,2)\nplt.imshow(img2)\nplt.subplot(1,4,3)\nplt.imshow(img3)\nplt.subplot(1,4,4)\nplt.imshow(img4)\nplt.show()\nThanks! Got it.
", "rawMarkdown": "Thanks! Got it. ", "votes": 2 } ] }, { "id": 1483185, "postDate": "2021-08-20T13:47:22.503Z", "content": "Nice write up as always Chris, thanks for sharing your findings. I'm excited to see your future pytorch experiments too, you did pretty good with pytorch written scripts in our last competition, I'm sure you'll get great at that in no time too!
", "rawMarkdown": "Nice write up as always Chris, thanks for sharing your findings. I'm excited to see your future pytorch experiments too, you did pretty good with pytorch written scripts in our last competition, I'm sure you'll get great at that in no time too!", "votes": 2, "replies": [ { "id": 1483218, "postDate": "2021-08-20T14:07:47.837Z", "content": "Thanks @datafan07 Congrats earning Silver medal. I was watching your team. At one point, i saw you jump from LB 775 to LB 783. What discovery did you make at that point?
\nI'd love to hear about your solution. What was your strongest single model? And what preprocessing and/or post processing techniques did you use?
", "rawMarkdown": "Thanks @datafan07 Congrats earning Silver medal. I was watching your team. At one point, i saw you jump from LB 775 to LB 783. What discovery did you make at that point?\n\nI'd love to hear about your solution. What was your strongest single model? And what preprocessing and/or post processing techniques did you use?", "votes": 1 }, { "id": 1483268, "postDate": "2021-08-20T14:40:06.850Z", "content": "Ah I remember that jump, funny enough it was from single fold model predictions. We were building different models, some team members building tf tpu models and I was working with pytorch and that single model comes from pytorch trial. I remember adding heavy augmentations with mixup + cutmix and changing backbone to 'EfficientNetV2-M' helped me a lot. It was our best single model with almost 0.90cv.
\nAfter that I didn't have enough resources and time to make trials on rest of the folds so we went for ensembling for last days. I tried test same setup on TPU's with effnetv2 etc. to get faster results but didn't get same score as pytorch one. cv was decent but lb failed with that one…
", "rawMarkdown": "Ah I remember that jump, funny enough it was from single fold model predictions. We were building different models, some team members building tf tpu models and I was working with pytorch and that single model comes from pytorch trial. I remember adding heavy augmentations with mixup + cutmix and changing backbone to 'EfficientNetV2-M' helped me a lot. It was our best single model with almost 0.90cv.\n\nAfter that I didn't have enough resources and time to make trials on rest of the folds so we went for ensembling for last days. I tried test same setup on TPU's with effnetv2 etc. to get faster results but didn't get same score as pytorch one. cv was decent but lb failed with that one...", "votes": 2 }, { "id": 1483304, "postDate": "2021-08-20T15:07:46.733Z", "content": "Wow, CV 0.90, that's great. I would to see code that scores over LB 0.780. To me, achieving over 0.780 is still magic. No team posted code that achieves over LB 0.780 yet. (Second place posted code but it throws an error if you run it).
", "rawMarkdown": "Wow, CV 0.90, that's great. I would to see code that scores over LB 0.780. To me, achieving over 0.780 is still magic. No team posted code that achieves over LB 0.780 yet. (Second place posted code but it throws an error if you run it).", "votes": 3 }, { "id": 1483350, "postDate": "2021-08-20T15:35:37.697Z", "content": "Like I did in commonlit I actually shared baseline version of my training long time ago here. I just added some more updates based on that pipeline, like cutmix, different backbones etc. and more epochs. I didn't update the public code after leak reset but main part still stands same…(There might be some errors in this one too since some packages got important updates, like checkpoint code)
", "rawMarkdown": "Like I did in commonlit I actually shared baseline version of my training [long time ago here](https://www.kaggle.com/datafan07/pytorch-lightning-single-fold-training-lb-0-97). I just added some more updates based on that pipeline, like cutmix, different backbones etc. and more epochs. I didn't update the public code after leak reset but main part still stands same...(There might be some errors in this one too since some packages got important updates, like checkpoint code)", "votes": 1 } ] }, { "id": 1482418, "postDate": "2021-08-20T04:47:34.590Z", "content": "Thanks for sharing @cdeotte … the evolution of thought and how you proceed with different alternatives is helpful on how to think through the steps and try out multiple approaches!
", "rawMarkdown": "Thanks for sharing @cdeotte ... the evolution of thought and how you proceed with different alternatives is helpful on how to think through the steps and try out multiple approaches! ", "votes": 2 }, { "id": 1481643, "postDate": "2021-08-19T16:03:21.090Z", "content": "Thanks as always for your kind explanation. :) @cdeotte
", "rawMarkdown": "Thanks as always for your kind explanation. :) @cdeotte ", "votes": 2 } ], "comments": [ { "id": 1481535, "author_name": "Tucker Arrants", "author_url": "", "post_date": "2021-08-19T15:07:49.567000", "content": "Great write up Chris and congratulations on getting bronze. I'm glad my notebook was useful during your 7 day journey!
", "votes": 3, "replies": [] }, { "id": 1482912, "author_name": "Jared Savage", "author_url": "", "post_date": "2021-08-20T10:50:09.637000", "content": "Looks like we lost another one to PyTorch. 😄
\nThanks for sharing Chris. A pleasure as always.
Thank you for your detailed and clear explanation of your own thinking process, thank you for your open source code, and all kinds of thoughtful notes in the article. These are very friendly to novices like me, and I can always learn a lot from you.😄
", "votes": 1, "replies": [] }, { "id": 1488383, "author_name": "Kalilur Rahman", "author_url": "", "post_date": "2021-08-24T08:51:33.210000", "content": "Wow! It's a grand post. Brilliantly done and congratulations @cdeotte
", "votes": 1, "replies": [] }, { "id": 1484548, "author_name": "Markus Frank", "author_url": "", "post_date": "2021-08-21T12:07:13.687000", "content": "Hi Chris, congratulations and thanks for this writeup, this really helps me to learn. Just two quick questions:
\nHi Markus. My model is more than just an EfficientNet. It has a special module beforehand which learns to resize the image by itself. Here's a diagram of just the module. (The efficientnet is not in the picture):
\n
\nSo first i load the Numpy array and resize it to image_in with img = np.vstack( img[::2] ) and img = cv2.resize(img,(image_in,image_in)).
Then it gets fed into the module. Then the module learns to resize it to image_out. Finally that result is fed into the EfficientNet. I used Tucker's public notebook here
Note that i used this public notebook because i had a few days left and could not figure out how to close the CV LB gap with my own model. I am now investigating why that notebook has a smaller CV LB gap than my model. I have discovered that this special module is not needed. I believe the secret of that notebook is that it uses mixup augmentation whereas my model did not.
", "votes": 2, "replies": [] }, { "id": 1484572, "author_name": "Chris Deotte", "author_url": "", "post_date": "2021-08-21T12:42:40.013000", "content": "@friedchips There are two efficientnet's you can use for TensorFlow. You can use the one in https://keras.io/api/applications/ which was pretrained with image pixels of 0 thru 255. Or you can use pip install efficientnet here which was pretrained with image pixels normalized with mean 0 and std 1.
If you use the PyTorch efficientnet's. They are pretrained with pixels mean 0 and std 1.
\nIt will help your efficientnet converge faster to use the same pixel range that the pretraining used. However if it you get it wrong, the model should still converge and learn to adjust to what you are feeding it.
", "votes": 2, "replies": [] }, { "id": 1484743, "author_name": "Markus Frank", "author_url": "", "post_date": "2021-08-21T14:48:05.130000", "content": "Thanks! I had overlooked your image resizing stage. I just rescaled the original input images with a Recaling layer before feeding them into my EffNetB2 and also saw that the bigger the better, finally going with 768x768 as input to the EffNet.
\nYes, Mixup was extremely helpful, especially with alpha very large (5.0). It just never occurred to me to not mix the labels…
\nThanks for the other efficientnet implementation. My Keras efficientnet converged much worse (CV 0.82 instead of 0.87) when I fed it with 0-1 data before realizing that it expects 0-255.
", "votes": 2, "replies": [] } ] }, { "id": 1487265, "author_name": "Chris Deotte", "author_url": "", "post_date": "2021-08-23T14:06:19.443000", "content": "Update. I have been conducting more experiments. I have updated my post to identify the Silver Medal Magic. And the Gold Medal Magic!
", "votes": 2, "replies": [ { "id": 1491516, "author_name": "gao-hongnan", "author_url": "", "post_date": "2021-08-26T12:50:15.690000", "content": "Thanks @cdeotte great insights as usual.
\nI realise many contestants plot pixel differences like you did in image comps. May I know if you can share the code you used to plot the above diagrams so I can compare if I’m doing it right
", "votes": 2, "replies": [] }, { "id": 1491537, "author_name": "Chris Deotte", "author_url": "", "post_date": "2021-08-26T12:59:34.140000", "content": "The code is posted above. For \"on\" cadence, i would read NumPy like
\nfile = numpy .load(path+name[0]+'/'+name+'.npy').astype('float32')\nON_CADENCE = file[::2,]\nAnd for the same \"off\" cadence, use the same name and load
file = numpy .load(path+name[0]+'/'+name+'.npy').astype('float32')\nOFF_CADENCE = file[1::2,]\nNext, i apply the following filter to make the spectrograms easier to see with human eye
\nimport cv2\nclahe = cv2.createCLAHE(clipLimit=16.0, tileGridSize=(8,8))\nimg = np.vstack(CADENCE)\nimg = img[1:,1:] - img[:-1,:-1] #emboss\nimg -= np.min(img)\nimg /= np.max(img)\nimg = (img*255).astype('uint8')\nimg = cv2.GaussianBlur(img,(5,5),0)\nimg = clahe.apply(img)\nTo plot 4 tiles in a row, you use subplot like
plt.figure(figsize=(20,5)\nplt.subplot(1,4,1)\nplt.imshow(img1)\nplt.subplot(1,4,2)\nplt.imshow(img2)\nplt.subplot(1,4,3)\nplt.imshow(img3)\nplt.subplot(1,4,4)\nplt.imshow(img4)\nplt.show()\nThanks! Got it.
", "votes": 2, "replies": [] } ] }, { "id": 1483185, "author_name": "Ertuğrul Demir", "author_url": "", "post_date": "2021-08-20T13:47:22.503000", "content": "Nice write up as always Chris, thanks for sharing your findings. I'm excited to see your future pytorch experiments too, you did pretty good with pytorch written scripts in our last competition, I'm sure you'll get great at that in no time too!
", "votes": 2, "replies": [ { "id": 1483218, "author_name": "Chris Deotte", "author_url": "", "post_date": "2021-08-20T14:07:47.837000", "content": "Thanks @datafan07 Congrats earning Silver medal. I was watching your team. At one point, i saw you jump from LB 775 to LB 783. What discovery did you make at that point?
\nI'd love to hear about your solution. What was your strongest single model? And what preprocessing and/or post processing techniques did you use?
", "votes": 1, "replies": [] }, { "id": 1483268, "author_name": "Ertuğrul Demir", "author_url": "", "post_date": "2021-08-20T14:40:06.850000", "content": "Ah I remember that jump, funny enough it was from single fold model predictions. We were building different models, some team members building tf tpu models and I was working with pytorch and that single model comes from pytorch trial. I remember adding heavy augmentations with mixup + cutmix and changing backbone to 'EfficientNetV2-M' helped me a lot. It was our best single model with almost 0.90cv.
\nAfter that I didn't have enough resources and time to make trials on rest of the folds so we went for ensembling for last days. I tried test same setup on TPU's with effnetv2 etc. to get faster results but didn't get same score as pytorch one. cv was decent but lb failed with that one…
", "votes": 2, "replies": [] }, { "id": 1483304, "author_name": "Chris Deotte", "author_url": "", "post_date": "2021-08-20T15:07:46.733000", "content": "Wow, CV 0.90, that's great. I would to see code that scores over LB 0.780. To me, achieving over 0.780 is still magic. No team posted code that achieves over LB 0.780 yet. (Second place posted code but it throws an error if you run it).
", "votes": 3, "replies": [] }, { "id": 1483350, "author_name": "Ertuğrul Demir", "author_url": "", "post_date": "2021-08-20T15:35:37.697000", "content": "Like I did in commonlit I actually shared baseline version of my training long time ago here. I just added some more updates based on that pipeline, like cutmix, different backbones etc. and more epochs. I didn't update the public code after leak reset but main part still stands same…(There might be some errors in this one too since some packages got important updates, like checkpoint code)
", "votes": 1, "replies": [] } ] }, { "id": 1482418, "author_name": "Kamal Das", "author_url": "", "post_date": "2021-08-20T04:47:34.590000", "content": "Thanks for sharing @cdeotte … the evolution of thought and how you proceed with different alternatives is helpful on how to think through the steps and try out multiple approaches!
", "votes": 2, "replies": [] }, { "id": 1481643, "author_name": "Heroseo", "author_url": "", "post_date": "2021-08-19T16:03:21.090000", "content": "Thanks as always for your kind explanation. :) @cdeotte
", "votes": 2, "replies": [] } ], "raw_markdown_by_id": { "1481348": "# Exciting Competition!\nWhat an interesting competition and difficult puzzle! Thank you Kaggle and SETI for hosting. I started this competition last Wednesday and worked hard every day. In this time, I closed CV LB gap from 0.125 to 0.104. I would have liked to have more time and continue the puzzle of closing the CV LB gap. Below is my journey over the past 7 days.\n\n# My Model - CV 0.890 - LB 0.765 - GAP = 0.125 (Days 1-3)\nLast Wednesday, Thursday, and Friday, I built dozens of my own models without reading public discussions nor notebooks. I was excited when my best model achieved CV 0.890 but surprised and disappointed when I submitted and received LB 0.765 🙁This was a 0.125 gap between CV and LB. Something is going on!\n\nMy best model used only \"on\" cadence in a spatial stack resized to 768x768 and fed into a EfficientNetB4 backbone. The only augmentation was horizontal flip and vertical flip. It used cosine learning schedule and 15 epochs.\n\n# Alternative Ideas (Days 3-4)\nAt this point, I thought the gap was caused by new pattens in test data. So, on Friday and Saturday, i starting building more creative models that used more than just classify train patterns.\n* Use \"off\" cadence train images as more `target=0`. Increased CV 0.003 but decreased LB 0.010\n* Use \"off\" cadence test images as more `target=0`. Did not affect CV but decreased LB like 0.050!\n* Use test pseudo labels. Decrease LB like 0.030!\n* Train model to classify \"train\" image versus \"test\" image. So discard target column and use `test image as target=1` and `train image as target=0`. The hope was that this model would find any new patterns in test data because that would be the difference between train and test. To my surprise the model achieved CV 0.99 but LB was terrible 🙁\n* Train model to classify \"on\" cadence `img[::2]` versus \"off\" cadence `img[1::2]`. So discard target column and use `cadence on as target=1` and `cadence off as target=0`. The advantage here is that we can train directly on test data without train data!\n* Sliding window over \"on\" cadence and compute cosine similarity with sliding window over \"off\" cadence. Take minimum cosine similarity value over all crops of \"off\" cadence. Then take maximum (of these minimums) over all crops of \"on\" cadence. One advantage here is that we can apply this to test data without using train data!\n\nMy favorite model was the last which achieved LB 0.550 and did not use any train data. It simply compared crops of test image \"on\" cadence with crops of test image \"off\" cadence and searched for dissimilarity.\n\n# Public Notebook - CV 884 - LB 0.779 - GAP = 0.105 (Days 5-6)\nBy Saturday, i was frustrated that I couldn't reach Bronze zone. How was everyone doing it? \n\nI began reading discussions and notebooks. I found @tuckerarrants amazing \"SETI - Learned Image Resizing\" notebook [here][1]. What impressed me most was that his **GAP was only 0.102**! His CV was 0.846 and his LB was 0.744. This was a smaller gap than my model. \n\nWithout understanding why, I quickly ran his notebook locally with larger models and larger image sizes to secure bronze position. The following table are results for fold 0 only. \n\nNote that Tucker's notebook has a resize module before the EfficientNet, so we put `image_in` into the resize module and `image_out` comes out. We then feed `image_out` into the EfficientNet backbone:\n| Backbone | Image In | Image Out |CV - fold 0 | LB - fold 0 |\n| --- | --- | --- | --- | --- |\n| EB3 | 512 | 256 | 0.859 | 0.764 |\n| EB3 | 640 | 320 | 0.870 | 0.768 |\n| EB3 | 768 | 384 | 0.870 | |\n| EB3 | orig | 448 | 0.872 | |\n| EB4 | 640 | 320 | 0.873 | 0.770 |\n| EB4 | 768 | 384 | 0.872 | |\n| EB4 | orig | 448 | 0.869 | |\n| EB6 | 640 | 320 | 0.872 | 0.770 |\n| Ensemble| | | 0.883 | 0.777 |\n\nThen 5-Fold was LB 0.778 and post process got LB 0.779\n\n# Nvidia 4xV100 GPUs\nFor the past few days, Nvidia V100 GPUs were running constantly to train the models in the table above. I am learning PyTorch and discovered a great trick [here][2]. Using Tucker's notebook, you can add 1 line of code to train PyTorch with multiple GPUs (for large backbones and batch sizes). Just replace the one line `model.to(device)` with the three lines:\n\n import torch.nn as nn\n model = nn.DataParallel(model)\n model.to(device)\n\nThat's it! With 1 line of code, we can train larger models and larger batch sizes by using multiple Nvidia GPU. \n\n# The Secret of Timm Backbones\nMany people (including @dragonzhang) were confused why they could not train efficientnet B4 with Tucker's notebook. That is because timm does not contain an imagenet pretrained efficientnet B4. (So if you use `model_name='efficientnet_b4'` you train with a not-pretrained effnet) \n\nTimm only contains pretrains of EB4, 5, 6, 7 of the TF converted weights, `tf_efficientnet_b4`, `tf_efficientnet_b4_ns`, `tf_efficientnet_b4_ap`. The first is original efficient net, the next two are pretrained with noisy student and adv prop. To view the full list of available timm pretrained backbones, type\n\n import timm\n from pprint import pprint\n model_names = timm.list_models(pretrained=True)\n pprint(model_names)\n\n# Model Difference - Mysterious \"S\" Shape! (Day 7)\nAfter securing bronze medal position, I began to investigate the CV LB gap again. Four hours before competition deadline, I plotted the difference between my model and public notebook model. I sorted by test images where the difference between my model prediction and public model prediction were greatest. To my surprise, I found a mysterious \"S\" shaped test pattern!\n\nThe image on the left is original comp test data \"on\" cadence vertical stack. With time on x axis and frequency on y axis. The second image is pixel histogram. The third image is \"on\" cadence with emboss, blur, and adaptive histogram equalization. (Filter code shown below) The fourth image (right image) is the \"off\" cadence.\n \n import cv2\n clahe = cv2.createCLAHE(clipLimit=16.0, tileGridSize=(8,8))\n # FILTERS FOR BETTER IMAGE DISPLAY\n img = np.vstack(cadence[::2,])\n img = img[1:,1:] - img[:-1,:-1] #emboss\n img -= np.min(img)\n img /= np.max(img)\n img = (img*255).astype('uint8')\n img = cv2.GaussianBlur(img,(5,5),0)\n img = clahe.apply(img)\n\nAbove the left image we see the public notebook prediction value. And above the third image we see my model's prediction value.\n\n\n\n\n\n\n\n\n\n\n# Post Process - LB +0.001 - GAP = 0.104 (Last 4 hours!)\nWith only 4 hours remaining, I trained a model on the images that my model and public notebook model disagreed most on. I then inferred the entire test data. For all images with `p>0.6`, I increased their submission prediction. This post processed closed the CV LB gap by `0.001` and increased LB by `0.001`. Thus there is something special about these \"S\" patterns!\n\n# UPDATE - Mixup is the Silver Medal Magic!\nAfter the comp ended, i have been continuing with experiments to determine why my original model's CV LB is large. It appears that mixup is very important to teach your model to generalize to test data which is different than the train data. It both closes the CV LB gap and boosts CV LB. \n\nUsing 1 fold of EB4 and image size 768x768 with only Hflip, Vflip, and Mixup (alpha=3, max target). One can achieve Silver medal with public LB 0.786 and private LB 0.781. Notebook [here][5]. (Also important is large image size and large backbone).\n\n\n# UPDATE - \"S-shape\" and \"Remove Noise\" is Gold Medal Magic!\nTo climb from Silver medal to Gold medal, we need to detect \"s-shapes\" and remove noise. The first place winning solution [here][3] explains how. Additionally, repeatedly using test pseudo labels also helps climb to top of Gold Medals! (and using old data before reset can help too).\n\n# UPDATE - Grad Cam\nI posted a discussion [here][6] about grad cam and a notebook [here][5] using grad cam. Using grad cam can help us understand what makes a particular image a `target=1`. For example, our model can make a test prediction and then circle what it thinks is causing a `target=1` prediction. The image below is 100% generated by code and no human labeling!\n\n\n[1]: https://www.kaggle.com/tuckerarrants/seti-learned-image-resizing?scriptVersionId=68644223\n[2]: https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html#create-model-and-dataparallel\n[3]: https://www.kaggle.com/c/seti-breakthrough-listen/discussion/266385\n[4]: https://www.kaggle.com/c/seti-breakthrough-listen/discussion/266513#1483010\n[5]: https://www.kaggle.com/cdeotte/silver-medal-with-grad-cam-lb-0-780\n[6]: https://www.kaggle.com/c/seti-breakthrough-listen/discussion/268314", "1481535": "Great write up Chris and congratulations on getting bronze. I'm glad my notebook was useful during your 7 day journey!", "1482912": "Looks like we lost another one to PyTorch. 😄\nThanks for sharing Chris. A pleasure as always.", "1493974": "Thank you for your detailed and clear explanation of your own thinking process, thank you for your open source code, and all kinds of thoughtful notes in the article. These are very friendly to novices like me, and I can always learn a lot from you.😄", "1488383": "Wow! It's a grand post. Brilliantly done and congratulations @cdeotte ", "1484548": "Hi Chris, congratulations and thanks for this writeup, this really helps me to learn. Just two quick questions:\n- What exactly do the \"Image in\" and \"Image out\" resolutions mean?\n- I used Keras EfficientNets which expect inputs in the 0-255 range. Do you also prepare your data to be in that range or is there a way to feed EffNets with ~mean 0, std 1 data?\nThanks!", "1487265": "Update. I have been conducting more experiments. I have updated my post to identify the Silver Medal Magic. And the Gold Medal Magic!", "1483185": "Nice write up as always Chris, thanks for sharing your findings. I'm excited to see your future pytorch experiments too, you did pretty good with pytorch written scripts in our last competition, I'm sure you'll get great at that in no time too!", "1482418": "Thanks for sharing @cdeotte ... the evolution of thought and how you proceed with different alternatives is helpful on how to think through the steps and try out multiple approaches! ", "1481643": "Thanks as always for your kind explanation. :) @cdeotte " } }