First of all, thanks a lot to the hosts for organizing such an interesting competition! Also, congrats to all the participants, especially to the winners.
\nThanks to @shonenkov with his fantastic kernel here, I (and maybe many others) had a great approach to the problem. I mainly used his kernel as baseline of my solution, with some modification. Here are details.
\nFrom my understandings, when applying DCT changes, the errors will be spread along all the pixels. We use DL model to catch those changes in the pixel domain. So, any augmentation without interpolation (by interpolation I mean some image processing like resize, rotate non-90, etc) can be used (discussion).
\nI've tried quite a few backbone for this competition:
\nI used 2 kinds of input: RGB and YCrCb. Their performance are toe to toe to each other, but together they boost my score.
\nOptimizer: RAdam along with Lookahead
\n------------------------------------\nEfficientNet-B3 RGB, YCrCb, fold 0\nEfficientNet-B3 RGB, whole dataset\nEfficientNet-B4 RGB, YCrCb, fold 4 \nEfficientNet-B5 RGB, YCrCb, fold 2\n------------------------------------\nI will update this post if I remember something interesting to share. Thank you all again!
", "messages": [ { "id": "938476", "postDate": "07/21/2020 15:02:57", "content": "First of all, thanks a lot to the hosts for organizing such an interesting competition! Also, congrats to all the participants, especially to the winners.
\nThanks to @shonenkov with his fantastic kernel here, I (and maybe many others) had a great approach to the problem. I mainly used his kernel as baseline of my solution, with some modification. Here are details.
\nFrom my understandings, when applying DCT changes, the errors will be spread along all the pixels. We use DL model to catch those changes in the pixel domain. So, any augmentation without interpolation (by interpolation I mean some image processing like resize, rotate non-90, etc) can be used (discussion).
\nI've tried quite a few backbone for this competition:
\nI used 2 kinds of input: RGB and YCrCb. Their performance are toe to toe to each other, but together they boost my score.
\nOptimizer: RAdam along with Lookahead
\n------------------------------------\nEfficientNet-B3 RGB, YCrCb, fold 0\nEfficientNet-B3 RGB, whole dataset\nEfficientNet-B4 RGB, YCrCb, fold 4 \nEfficientNet-B5 RGB, YCrCb, fold 2\n------------------------------------\nI will update this post if I remember something interesting to share. Thank you all again!
", "rawMarkdown": "First of all, thanks a lot to the hosts for organizing such an interesting competition! Also, congrats to all the participants, especially to the winners.\n\nThanks to @shonenkov with his fantastic kernel [here](https://www.kaggle.com/shonenkov/train-inference-gpu-baseline), I (and maybe many others) had a great approach to the problem. I mainly used his kernel as baseline of my solution, with some modification. Here are details.\n\n# Augmentation\n- Horizontal + vertical flip\n- Rotate 90\n- Cutout\n- Crop + padding\n\nFrom my understandings, when applying DCT changes, the errors will be spread along all the pixels. We use DL model to catch those changes in the pixel domain. So, any augmentation without interpolation (by interpolation I mean some image processing like resize, rotate non-90, etc) can be used (discussion).\n\n# Modeling\nI've tried quite a few backbone for this competition:\n- Se-ResNeXt, DenseNet, NASNet, etc. They might work as well, but converge very slowly in comparison to EfficientNet\n- EfficientNet B3, B4, B5 (B6, B7 or B8 seem forever for me :((... ). They worked quite well!\n\nI used 2 kinds of input: RGB and YCrCb. Their performance are toe to toe to each other, but together they boost my score.\n\nOptimizer: RAdam along with Lookahead\n\n# Inference\n- TTAx8: flip + transpose\n- Weight ensemble: 7 models, comprise of B3, B4, B5 with RGB, YCrCb input:\n```\n------------------------------------\nEfficientNet-B3 RGB, YCrCb, fold 0\nEfficientNet-B3 RGB, whole dataset\nEfficientNet-B4 RGB, YCrCb, fold 4 \nEfficientNet-B5 RGB, YCrCb, fold 2\n------------------------------------\n```\n\n# Result\n- My best single model score: 0.92\n- My best ensemble score: 0.924\n\nI will update this post if I remember something interesting to share. Thank you all again!", "votes": null }, { "id": "938515", "postDate": "07/21/2020 15:25:49", "content": "Congrats @nejicool96
\nHow will cutout improve the learning? The part which had the hidden message will also be removed during random cutouts, right ?
", "rawMarkdown": "Congrats @nejicool96 \n\nHow will cutout improve the learning? The part which had the hidden message will also be removed during random cutouts, right ?", "votes": null }, { "id": "938527", "postDate": "07/21/2020 15:31:44", "content": "Congrats on result! Thanks for sharing your solution @nejicool96
", "rawMarkdown": "Congrats on result! Thanks for sharing your solution @nejicool96", "votes": null }, { "id": "938540", "postDate": "07/21/2020 15:43:54", "content": "Thank you @Vishnu.
\nAfter watching this video, I knew that DCT manipulation will lead to changes in high frequency signal (he talks about frequency in 2:40). I think those high frequencies appear all over the image, not only some parts of the image.
Thank you :D
", "rawMarkdown": "Thank you :D", "votes": null }, { "id": "938556", "postDate": "07/21/2020 15:51:27", "content": "Keep working hard for the next competitions :D
", "rawMarkdown": "Keep working hard for the next competitions :D", "votes": null }, { "id": "938568", "postDate": "07/21/2020 15:55:21", "content": "Yes, I will try my best. Glad to see a competitor from HUST too :)).
", "rawMarkdown": "Yes, I will try my best. Glad to see a competitor from HUST too :)).", "votes": null }, { "id": "938710", "postDate": "07/21/2020 17:38:00", "content": "Congrats @nejicool96!!
\n\nHow about other optimizer?\nI tested AdamW, AdamP, SGD, SGDP, etc. and I couldn't feel a huge change.\nPersonally, AdamW seems good.
\n\np.s. I won a bronze medal because I didn't use the results applying TTA. 😂 \nThe final submission was worse than my single model.\n(And I haven't even been able to personally submit a higher score on CV.😹)
", "rawMarkdown": "Congrats @nejicool96!! \n\nHow about other optimizer?\nI tested AdamW, AdamP, SGD, SGDP, etc. and I couldn't feel a huge change.\nPersonally, AdamW seems good.\n\np.s. I won a bronze medal because I didn't use the results applying TTA. 😂 \nThe final submission was worse than my single model.\n(And I haven't even been able to personally submit a higher score on CV.😹)", "votes": null }, { "id": "938724", "postDate": "07/21/2020 17:50:26", "content": "Thank you :D.
\nHonestly, RAdam is my favorite optimizer. If it's gonna work, I won't try others and this is the case :)). It is great that you could try those optimizers.
\nAlso, congrats on your bronze medal :D
RAdam + Lookahead is your favorite optimizer? 😊
\n\nThanks for replying!
", "rawMarkdown": "RAdam + Lookahead is your favorite optimizer? 😊 \n\nThanks for replying!", "votes": null }, { "id": "938730", "postDate": "07/21/2020 17:56:10", "content": "Yes, they are. You are welcome 🤣
", "rawMarkdown": "Yes, they are. You are welcome 🤣", "votes": null }, { "id": "939374", "postDate": "07/22/2020 07:38:48", "content": "Great to hear from you about your approach .. I kind new to CV, Is it possible for you to share your notebook ..
", "rawMarkdown": "Great to hear from you about your approach .. I kind new to CV, Is it possible for you to share your notebook ..", "votes": null }, { "id": "939410", "postDate": "07/22/2020 08:19:38", "content": "Actually, I don't code in notebook. At present my code is a bit 'dirty' too so I don't want to make it public at least now. You can follow Shonenkov's notebook I mentioned and try some of my modifications (they are not hard to implement).
", "rawMarkdown": "Actually, I don't code in notebook. At present my code is a bit 'dirty' too so I don't want to make it public at least now. You can follow Shonenkov's notebook I mentioned and try some of my modifications (they are not hard to implement).", "votes": null }, { "id": "939682", "postDate": "07/22/2020 11:32:00", "content": "Thanks @nejicool96 sure will check it.. Hope you manage all your work and will public your notebook soon..
", "rawMarkdown": "Thanks @nejicool96 sure will check it.. Hope you manage all your work and will public your notebook soon..", "votes": null }, { "id": "940014", "postDate": "07/22/2020 16:13:10", "content": "Thank you for sharing. That's in impressive set of model to ensemble !\nHow did you decide when to stop training since you are doing so many of them ?
", "rawMarkdown": "Thank you for sharing. That's in impressive set of model to ensemble !\nHow did you decide when to stop training since you are doing so many of them ?", "votes": null }, { "id": "940412", "postDate": "07/22/2020 23:49:34", "content": "For B3 ~ 100 epochs, I trained until val loss and roc_auc couldn't be improved (smt like 'manual' early stopping).\nFor B4 (~50e) and B5 (~40e), I stopped simply because the competition had ended :)). They hadn't converged yet.
", "rawMarkdown": "For B3 ~ 100 epochs, I trained until val loss and roc_auc couldn't be improved (smt like 'manual' early stopping).\nFor B4 (~50e) and B5 (~40e), I stopped simply because the competition had ended :)). They hadn't converged yet.", "votes": null }, { "id": "1037634", "postDate": "10/05/2020 07:32:29", "content": "Incredible !
", "rawMarkdown": "Incredible !", "votes": null } ], "comments": [ { "id": 938515, "author_name": "vishnurapps", "author_url": "", "post_date": "07/21/2020 15:25:49", "content": "Congrats @nejicool96
\nHow will cutout improve the learning? The part which had the hidden message will also be removed during random cutouts, right ?
", "votes": null, "replies": [ { "id": 938540, "author_name": "nejicool96", "author_url": "", "post_date": "07/21/2020 15:43:54", "content": "Thank you @Vishnu.
\nAfter watching this video, I knew that DCT manipulation will lead to changes in high frequency signal (he talks about frequency in 2:40). I think those high frequencies appear all over the image, not only some parts of the image.
Congrats on result! Thanks for sharing your solution @nejicool96
", "votes": null, "replies": [ { "id": 938541, "author_name": "nejicool96", "author_url": "", "post_date": "07/21/2020 15:44:09", "content": "Thank you :D
", "votes": null, "replies": [] }, { "id": 938556, "author_name": "duykhanh99", "author_url": "", "post_date": "07/21/2020 15:51:27", "content": "Keep working hard for the next competitions :D
", "votes": null, "replies": [] }, { "id": 938568, "author_name": "nejicool96", "author_url": "", "post_date": "07/21/2020 15:55:21", "content": "Yes, I will try my best. Glad to see a competitor from HUST too :)).
", "votes": null, "replies": [] } ] }, { "id": 1037634, "author_name": "daonguyenduong", "author_url": "", "post_date": "10/05/2020 07:32:29", "content": "Incredible !
", "votes": null, "replies": [] }, { "id": 938710, "author_name": "piantic", "author_url": "", "post_date": "07/21/2020 17:38:00", "content": "Congrats @nejicool96!!
\n\nHow about other optimizer?\nI tested AdamW, AdamP, SGD, SGDP, etc. and I couldn't feel a huge change.\nPersonally, AdamW seems good.
\n\np.s. I won a bronze medal because I didn't use the results applying TTA. 😂 \nThe final submission was worse than my single model.\n(And I haven't even been able to personally submit a higher score on CV.😹)
", "votes": null, "replies": [ { "id": 938724, "author_name": "nejicool96", "author_url": "", "post_date": "07/21/2020 17:50:26", "content": "Thank you :D.
\nHonestly, RAdam is my favorite optimizer. If it's gonna work, I won't try others and this is the case :)). It is great that you could try those optimizers.
\nAlso, congrats on your bronze medal :D
RAdam + Lookahead is your favorite optimizer? 😊
\n\nThanks for replying!
", "votes": null, "replies": [] }, { "id": 938730, "author_name": "nejicool96", "author_url": "", "post_date": "07/21/2020 17:56:10", "content": "Yes, they are. You are welcome 🤣
", "votes": null, "replies": [] } ] }, { "id": 939374, "author_name": "vin1234", "author_url": "", "post_date": "07/22/2020 07:38:48", "content": "Great to hear from you about your approach .. I kind new to CV, Is it possible for you to share your notebook ..
", "votes": null, "replies": [ { "id": 939410, "author_name": "nejicool96", "author_url": "", "post_date": "07/22/2020 08:19:38", "content": "Actually, I don't code in notebook. At present my code is a bit 'dirty' too so I don't want to make it public at least now. You can follow Shonenkov's notebook I mentioned and try some of my modifications (they are not hard to implement).
", "votes": null, "replies": [] }, { "id": 939682, "author_name": "vin1234", "author_url": "", "post_date": "07/22/2020 11:32:00", "content": "Thanks @nejicool96 sure will check it.. Hope you manage all your work and will public your notebook soon..
", "votes": null, "replies": [] } ] }, { "id": 940014, "author_name": "nyleve", "author_url": "", "post_date": "07/22/2020 16:13:10", "content": "Thank you for sharing. That's in impressive set of model to ensemble !\nHow did you decide when to stop training since you are doing so many of them ?
", "votes": null, "replies": [ { "id": 940412, "author_name": "nejicool96", "author_url": "", "post_date": "07/22/2020 23:49:34", "content": "For B3 ~ 100 epochs, I trained until val loss and roc_auc couldn't be improved (smt like 'manual' early stopping).\nFor B4 (~50e) and B5 (~40e), I stopped simply because the competition had ended :)). They hadn't converged yet.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "938476": "First of all, thanks a lot to the hosts for organizing such an interesting competition! Also, congrats to all the participants, especially to the winners.\n\nThanks to @shonenkov with his fantastic kernel [here](https://www.kaggle.com/shonenkov/train-inference-gpu-baseline), I (and maybe many others) had a great approach to the problem. I mainly used his kernel as baseline of my solution, with some modification. Here are details.\n\n# Augmentation\n- Horizontal + vertical flip\n- Rotate 90\n- Cutout\n- Crop + padding\n\nFrom my understandings, when applying DCT changes, the errors will be spread along all the pixels. We use DL model to catch those changes in the pixel domain. So, any augmentation without interpolation (by interpolation I mean some image processing like resize, rotate non-90, etc) can be used (discussion).\n\n# Modeling\nI've tried quite a few backbone for this competition:\n- Se-ResNeXt, DenseNet, NASNet, etc. They might work as well, but converge very slowly in comparison to EfficientNet\n- EfficientNet B3, B4, B5 (B6, B7 or B8 seem forever for me :((... ). They worked quite well!\n\nI used 2 kinds of input: RGB and YCrCb. Their performance are toe to toe to each other, but together they boost my score.\n\nOptimizer: RAdam along with Lookahead\n\n# Inference\n- TTAx8: flip + transpose\n- Weight ensemble: 7 models, comprise of B3, B4, B5 with RGB, YCrCb input:\n```\n------------------------------------\nEfficientNet-B3 RGB, YCrCb, fold 0\nEfficientNet-B3 RGB, whole dataset\nEfficientNet-B4 RGB, YCrCb, fold 4 \nEfficientNet-B5 RGB, YCrCb, fold 2\n------------------------------------\n```\n\n# Result\n- My best single model score: 0.92\n- My best ensemble score: 0.924\n\nI will update this post if I remember something interesting to share. Thank you all again!", "938515": "Congrats @nejicool96 \n\nHow will cutout improve the learning? The part which had the hidden message will also be removed during random cutouts, right ?", "938527": "Congrats on result! Thanks for sharing your solution @nejicool96", "938540": "Thank you @Vishnu.\nAfter watching this [video](https://www.youtube.com/watch?v=Q2aEzeMDHMA&list=PLzH6n4zXuckoAod3z31QEST1ZaizBuNHh&index=3), I knew that DCT manipulation will lead to changes in high frequency signal (he talks about frequency in 2:40). I think those high frequencies appear all over the image, not only some parts of the image.", "938541": "Thank you :D", "938556": "Keep working hard for the next competitions :D", "938568": "Yes, I will try my best. Glad to see a competitor from HUST too :)).", "938710": "Congrats @nejicool96!! \n\nHow about other optimizer?\nI tested AdamW, AdamP, SGD, SGDP, etc. and I couldn't feel a huge change.\nPersonally, AdamW seems good.\n\np.s. I won a bronze medal because I didn't use the results applying TTA. 😂 \nThe final submission was worse than my single model.\n(And I haven't even been able to personally submit a higher score on CV.😹)", "938724": "Thank you :D.\nHonestly, RAdam is my favorite optimizer. If it's gonna work, I won't try others and this is the case :)). It is great that you could try those optimizers.\nAlso, congrats on your bronze medal :D", "938726": "RAdam + Lookahead is your favorite optimizer? 😊 \n\nThanks for replying!", "938730": "Yes, they are. You are welcome 🤣", "939374": "Great to hear from you about your approach .. I kind new to CV, Is it possible for you to share your notebook ..", "939410": "Actually, I don't code in notebook. At present my code is a bit 'dirty' too so I don't want to make it public at least now. You can follow Shonenkov's notebook I mentioned and try some of my modifications (they are not hard to implement).", "939682": "Thanks @nejicool96 sure will check it.. Hope you manage all your work and will public your notebook soon..", "940014": "Thank you for sharing. That's in impressive set of model to ensemble !\nHow did you decide when to stop training since you are doing so many of them ?", "940412": "For B3 ~ 100 epochs, I trained until val loss and roc_auc couldn't be improved (smt like 'manual' early stopping).\nFor B4 (~50e) and B5 (~40e), I stopped simply because the competition had ended :)). They hadn't converged yet.", "1037634": "Incredible !" }, "source": "meta" }