{ "id": 168510, "title": "Private LB 13th Solution - so near yet so far :)", "url": "/competitions/alaska2-image-steganalysis/writeups/all-data-are-ext-private-lb-13th-solution-so-near-", "author_name": "", "post_date": "2020-07-21T01:07:33.023Z", "votes": 41, "comment_count": 7, "views": 0, "content": "

First of all, congratulation to all, and especially to the prize winner and gold medalists.
\nSecondly, ouch, narrowly missing out on gold.

\n

So here are some essential points of our solutions:

\n\n

We were particularly careful about overfitting the LB so throughout the competition we didn't use it for feedback. But in general, we observed that for the same fold of data, better CV usually (not always) yields better LB

\n

Best Single Model - B5 with upsampling covers, added convolution heads, drop blocks -> Private LB 0.927, Public LB 0.928

\n

Congratulation to my deal teammates @garybios @haqishen @strideradu and @yl1202 for the effort, I really had a great time in this short and yet intensive competition! :)

", "messages": [ { "id": "937359", "postDate": "07/21/2020 00:55:50", "content": "

First of all, congratulation to all, and especially to the prize winner and gold medalists.
\nSecondly, ouch, narrowly missing out on gold.

\n

So here are some essential points of our solutions:

\n\n

We were particularly careful about overfitting the LB so throughout the competition we didn't use it for feedback. But in general, we observed that for the same fold of data, better CV usually (not always) yields better LB

\n

Best Single Model - B5 with upsampling covers, added convolution heads, drop blocks -> Private LB 0.927, Public LB 0.928

\n

Congratulation to my deal teammates @garybios @haqishen @strideradu and @yl1202 for the effort, I really had a great time in this short and yet intensive competition! :)

", "rawMarkdown": "First of all, congratulation to all, and especially to the prize winner and gold medalists.\nSecondly, ouch, narrowly missing out on gold.\n\nSo here are some essential points of our solutions:\n\n- DL framework: Pytorch\n\n- Used Architecture: Effnet(timm/geffnet) B0/B2/B3/B4/B5, [Rexnet](https://github.com/clovaai/rexnet) 1.3/1.5/2.0 \n\n- Model customization: @haqishen and @garybios did some amazing work in adding convolution head and SE head to our chosen architecture, and dropping the last two blocks from before the final FC output also helps - I will let them expand on these. \n\n- Upsampling Cover: We find that upsampling cover image to match amount to Stegos helps to improve both CV and LB\n\n- Method of ensemble: we have trained about 80+ models, we have taken the top models with top 6CV in each fold, and did unweighted gmean \n\nWe were particularly careful about overfitting the LB so throughout the competition we didn't use it for feedback. But in general, we observed that for the same fold of data, better CV usually (not always) yields better LB\n\nBest Single Model - B5 with upsampling covers, added convolution heads, drop blocks -> Private LB 0.927, Public LB 0.928\n\nCongratulation to my deal teammates @garybios @haqishen @strideradu and @yl1202 for the effort, I really had a great time in this short and yet intensive competition! :)", "votes": null }, { "id": "937428", "postDate": "07/21/2020 02:41:09", "content": "

Hi @yifanxie ,

\n

Congratulations for the brilliant work. Training 80+ model says a lot. Where exactly did you train the model (Colab/Kaggle GPU/TPU) ? How long did it take for each model to train ?

", "rawMarkdown": "Hi @yifanxie ,\n\nCongratulations for the brilliant work. Training 80+ model says a lot. Where exactly did you train the model (Colab/Kaggle GPU/TPU) ? How long did it take for each model to train ?", "votes": null }, { "id": "937687", "postDate": "07/21/2020 05:58:47", "content": "

Congrats @yifanxie and team

", "rawMarkdown": "Congrats @yifanxie and team", "votes": null }, { "id": "938213", "postDate": "07/21/2020 12:05:44", "content": "
\n

Hi @yifanxie ,

\n

Congratulations for the brilliant work. Training 80+ model says a lot. Where exactly did you train the model (Colab/Kaggle GPU/TPU) ? How long did it take for each model to train ?

\n
\n

Just to clarify - when I say 80+ models, I am talking about during the whole competition that we trained about 80 models, this includes the very first baseline type of models that is shared on the kernel.
\nAs I said our 1st solution has 30 models - 6 models for each fold.

\n

We used a combination of personal own GPUs and cloud GPUs via Vast.ai. While I agree hardware is an import factor in this competition, I would say it is not the major factor - as shown in both 1st and 2nd ranked solution there are domain-specific information that we could use

\n

As mentioned by @haqishen here even with B0 we can achieve quite good CV/LB scores, and there is quite a lot of skilful work in there by my teammates :)

", "rawMarkdown": "> Hi @yifanxie ,\n> \n> Congratulations for the brilliant work. Training 80+ model says a lot. Where exactly did you train the model (Colab/Kaggle GPU/TPU) ? How long did it take for each model to train ?\n\nJust to clarify - when I say 80+ models, I am talking about during the whole competition that we trained about 80 models, this includes the very first baseline type of models that is shared on the kernel.\nAs I said our 1st solution has 30 models - 6 models for each fold. \n\nWe used a combination of personal own GPUs and cloud GPUs via Vast.ai. While I agree hardware is an import factor in this competition, I would say it is not the major factor - as shown in both 1st and 2nd ranked solution there are domain-specific information that we could use \n\nAs mentioned by @haqishen [here](https://www.kaggle.com/c/alaska2-image-steganalysis/discussion/168542) even with B0 we can achieve quite good CV/LB scores, and there is quite a lot of skilful work in there by my teammates :)", "votes": null }, { "id": "938301", "postDate": "07/21/2020 12:49:29", "content": "

congrats @yifanxie and team! You guys have a solid solution.

", "rawMarkdown": "congrats @yifanxie and team! You guys have a solid solution.", "votes": null }, { "id": "938315", "postDate": "07/21/2020 12:59:21", "content": "

@yifanxie congratulations =)

", "rawMarkdown": "yifanxie congratulations =)", "votes": null }, { "id": "938544", "postDate": "07/21/2020 15:45:02", "content": "

Congrats @yifanxie . Great work. Could you elaborate on 'B5 with upsampling covers, added covolution head, drop blocks'.

", "rawMarkdown": "Congrats @yifanxie . Great work. Could you elaborate on 'B5 with upsampling covers, added covolution head, drop blocks'.", "votes": null }, { "id": "945483", "postDate": "07/25/2020 21:15:50", "content": "
\n

Kalyan Kumar Pichuka wrote:

\n \n

Congrats @yifanxie . Great work. Could you elaborate on 'B5 with upsampling covers, added convolution head, drop blocks'.

\n
\n\n

thanks. For the convolution head and drop blocks stuff, refer to @haqishen 's post here

\n\n

For upsampling, we were training on 12 classes, but we kept the original binary class, and use that to make sure at each epoch the amount of Cover and Stego images involved are balanced. We found this bring out better CV although with longer training cycle.

", "rawMarkdown": "> **Kalyan Kumar Pichuka wrote:**\n> \n> Congrats @yifanxie . Great work. Could you elaborate on 'B5 with upsampling covers, added convolution head, drop blocks'.\n\nthanks. For the convolution head and drop blocks stuff, refer to @haqishen 's post [here](https://www.kaggle.com/c/alaska2-image-steganalysis/discussion/168542) \n\nFor upsampling, we were training on 12 classes, but we kept the original binary class, and use that to make sure at each epoch the amount of Cover and Stego images involved are balanced. We found this bring out better CV although with longer training cycle.", "votes": null } ], "comments": [ { "id": 937428, "author_name": "shwetank3", "author_url": "", "post_date": "07/21/2020 02:41:09", "content": "

Hi @yifanxie ,

\n

Congratulations for the brilliant work. Training 80+ model says a lot. Where exactly did you train the model (Colab/Kaggle GPU/TPU) ? How long did it take for each model to train ?

", "votes": null, "replies": [ { "id": 938213, "author_name": "yifanxie", "author_url": "", "post_date": "07/21/2020 12:05:44", "content": "
\n

Hi @yifanxie ,

\n

Congratulations for the brilliant work. Training 80+ model says a lot. Where exactly did you train the model (Colab/Kaggle GPU/TPU) ? How long did it take for each model to train ?

\n
\n

Just to clarify - when I say 80+ models, I am talking about during the whole competition that we trained about 80 models, this includes the very first baseline type of models that is shared on the kernel.
\nAs I said our 1st solution has 30 models - 6 models for each fold.

\n

We used a combination of personal own GPUs and cloud GPUs via Vast.ai. While I agree hardware is an import factor in this competition, I would say it is not the major factor - as shown in both 1st and 2nd ranked solution there are domain-specific information that we could use

\n

As mentioned by @haqishen here even with B0 we can achieve quite good CV/LB scores, and there is quite a lot of skilful work in there by my teammates :)

", "votes": null, "replies": [] } ] }, { "id": 937687, "author_name": "vishnurapps", "author_url": "", "post_date": "07/21/2020 05:58:47", "content": "

Congrats @yifanxie and team

", "votes": null, "replies": [] }, { "id": 938315, "author_name": "ipythonx", "author_url": "", "post_date": "07/21/2020 12:59:21", "content": "

@yifanxie congratulations =)

", "votes": null, "replies": [] }, { "id": 938301, "author_name": "titericz", "author_url": "", "post_date": "07/21/2020 12:49:29", "content": "

congrats @yifanxie and team! You guys have a solid solution.

", "votes": null, "replies": [] }, { "id": 938544, "author_name": "kalyanpichuka", "author_url": "", "post_date": "07/21/2020 15:45:02", "content": "

Congrats @yifanxie . Great work. Could you elaborate on 'B5 with upsampling covers, added covolution head, drop blocks'.

", "votes": null, "replies": [ { "id": 945483, "author_name": "yifanxie", "author_url": "", "post_date": "07/25/2020 21:15:50", "content": "
\n

Kalyan Kumar Pichuka wrote:

\n \n

Congrats @yifanxie . Great work. Could you elaborate on 'B5 with upsampling covers, added convolution head, drop blocks'.

\n
\n\n

thanks. For the convolution head and drop blocks stuff, refer to @haqishen 's post here

\n\n

For upsampling, we were training on 12 classes, but we kept the original binary class, and use that to make sure at each epoch the amount of Cover and Stego images involved are balanced. We found this bring out better CV although with longer training cycle.

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "937359": "First of all, congratulation to all, and especially to the prize winner and gold medalists.\nSecondly, ouch, narrowly missing out on gold.\n\nSo here are some essential points of our solutions:\n\n- DL framework: Pytorch\n\n- Used Architecture: Effnet(timm/geffnet) B0/B2/B3/B4/B5, [Rexnet](https://github.com/clovaai/rexnet) 1.3/1.5/2.0 \n\n- Model customization: @haqishen and @garybios did some amazing work in adding convolution head and SE head to our chosen architecture, and dropping the last two blocks from before the final FC output also helps - I will let them expand on these. \n\n- Upsampling Cover: We find that upsampling cover image to match amount to Stegos helps to improve both CV and LB\n\n- Method of ensemble: we have trained about 80+ models, we have taken the top models with top 6CV in each fold, and did unweighted gmean \n\nWe were particularly careful about overfitting the LB so throughout the competition we didn't use it for feedback. But in general, we observed that for the same fold of data, better CV usually (not always) yields better LB\n\nBest Single Model - B5 with upsampling covers, added convolution heads, drop blocks -> Private LB 0.927, Public LB 0.928\n\nCongratulation to my deal teammates @garybios @haqishen @strideradu and @yl1202 for the effort, I really had a great time in this short and yet intensive competition! :)", "937428": "Hi @yifanxie ,\n\nCongratulations for the brilliant work. Training 80+ model says a lot. Where exactly did you train the model (Colab/Kaggle GPU/TPU) ? How long did it take for each model to train ?", "937687": "Congrats @yifanxie and team", "938213": "> Hi @yifanxie ,\n> \n> Congratulations for the brilliant work. Training 80+ model says a lot. Where exactly did you train the model (Colab/Kaggle GPU/TPU) ? How long did it take for each model to train ?\n\nJust to clarify - when I say 80+ models, I am talking about during the whole competition that we trained about 80 models, this includes the very first baseline type of models that is shared on the kernel.\nAs I said our 1st solution has 30 models - 6 models for each fold. \n\nWe used a combination of personal own GPUs and cloud GPUs via Vast.ai. While I agree hardware is an import factor in this competition, I would say it is not the major factor - as shown in both 1st and 2nd ranked solution there are domain-specific information that we could use \n\nAs mentioned by @haqishen [here](https://www.kaggle.com/c/alaska2-image-steganalysis/discussion/168542) even with B0 we can achieve quite good CV/LB scores, and there is quite a lot of skilful work in there by my teammates :)", "938301": "congrats @yifanxie and team! You guys have a solid solution.", "938315": "yifanxie congratulations =)", "938544": "Congrats @yifanxie . Great work. Could you elaborate on 'B5 with upsampling covers, added covolution head, drop blocks'.", "945483": "> **Kalyan Kumar Pichuka wrote:**\n> \n> Congrats @yifanxie . Great work. Could you elaborate on 'B5 with upsampling covers, added convolution head, drop blocks'.\n\nthanks. For the convolution head and drop blocks stuff, refer to @haqishen 's post [here](https://www.kaggle.com/c/alaska2-image-steganalysis/discussion/168542) \n\nFor upsampling, we were training on 12 classes, but we kept the original binary class, and use that to make sure at each epoch the amount of Cover and Stego images involved are balanced. We found this bring out better CV although with longer training cycle." }, "source": "meta" }