{ "id": 168548, "title": "1st place solution", "url": "/competitions/alaska2-image-steganalysis/discussion/168548", "author_name": "Guanshuo Xu", "post_date": "2020-07-21T04:29:29.382000", "votes": 175, "comment_count": 82, "views": 0, "content": "

Methods:\n I combined a model trained in the spatial domain (YCbCr) and a model trained in the DCT domain with a second-level model trained using their bottleneck features.\n The spatial domain model was seresnet18 initialized with imagenet weights. To make training work I removed the stride in the first conv layer and the max-pooling layer. It's important for the model to stay at higher resolutions longer to capture the weak stego signals. Channel-attention with the se-block also gives significant improvement when comparing the results of resnet18 and seresnet18.\n To model in the DCT domain, I transformed the 8x8=64 DCT components as input \"channels\", so the original 3x512x512 becomes (64x3=192)x64x64. The 192 raw DCT values was one-hot encoded before entering the CNN following the idea of this paper (http://www.ws.binghamton.edu/fridrich/Research/OneHot_Revised.pdf). The CNN has six 3x3 conv layers with residual connections and se-blocks, spatial size is constant 64x64 until the last GAP layer.\n For augmentations, besides rotate90 and flips, cutmix worked pretty well. Each time a stego image was met during training, I also randomly re-assigned +1 and -1 to the DCT values in the modified positions. \n The 2nd-level models was a regular fully-connected net.

\n\n

Results:\n I made totally four 65000x4/10000x4 (train/validation) splits. On each split, I trained one set of the models described above. The final submission was a simple average of the probabilities.\n Except split4 spatial domain model, which was trained with mixup and 10 classes, all the other models were trained with cutmix and 12 classes.\n The validation results are shown below. For this competition, we should not use public LB for model selection.\n| |YCC| DCT|combined|\n| --- | --- |\n|split3|0.9395|0.8706|0.9439|\n|split4|0.9381|0.8749|0.9434|\n|split6 | 0.9409 | 0.8674 | 0.9452|\n|split7 | 0.9418 | 0.8719 | 0.9453|

", "messages": [ { "id": 937538, "postDate": "2020-07-21T04:29:29.383Z", "content": "

Methods:\n I combined a model trained in the spatial domain (YCbCr) and a model trained in the DCT domain with a second-level model trained using their bottleneck features.\n The spatial domain model was seresnet18 initialized with imagenet weights. To make training work I removed the stride in the first conv layer and the max-pooling layer. It's important for the model to stay at higher resolutions longer to capture the weak stego signals. Channel-attention with the se-block also gives significant improvement when comparing the results of resnet18 and seresnet18.\n To model in the DCT domain, I transformed the 8x8=64 DCT components as input \"channels\", so the original 3x512x512 becomes (64x3=192)x64x64. The 192 raw DCT values was one-hot encoded before entering the CNN following the idea of this paper (http://www.ws.binghamton.edu/fridrich/Research/OneHot_Revised.pdf). The CNN has six 3x3 conv layers with residual connections and se-blocks, spatial size is constant 64x64 until the last GAP layer.\n For augmentations, besides rotate90 and flips, cutmix worked pretty well. Each time a stego image was met during training, I also randomly re-assigned +1 and -1 to the DCT values in the modified positions. \n The 2nd-level models was a regular fully-connected net.

\n\n

Results:\n I made totally four 65000x4/10000x4 (train/validation) splits. On each split, I trained one set of the models described above. The final submission was a simple average of the probabilities.\n Except split4 spatial domain model, which was trained with mixup and 10 classes, all the other models were trained with cutmix and 12 classes.\n The validation results are shown below. For this competition, we should not use public LB for model selection.\n| |YCC| DCT|combined|\n| --- | --- |\n|split3|0.9395|0.8706|0.9439|\n|split4|0.9381|0.8749|0.9434|\n|split6 | 0.9409 | 0.8674 | 0.9452|\n|split7 | 0.9418 | 0.8719 | 0.9453|

", "rawMarkdown": "**Methods:**\n I combined a model trained in the spatial domain (YCbCr) and a model trained in the DCT domain with a second-level model trained using their bottleneck features.\n The spatial domain model was seresnet18 initialized with imagenet weights. To make training work I removed the stride in the first conv layer and the max-pooling layer. It's important for the model to stay at higher resolutions longer to capture the weak stego signals. Channel-attention with the se-block also gives significant improvement when comparing the results of resnet18 and seresnet18.\n To model in the DCT domain, I transformed the 8x8=64 DCT components as input \"channels\", so the original 3x512x512 becomes (64x3=192)x64x64. The 192 raw DCT values was one-hot encoded before entering the CNN following the idea of this paper (http://www.ws.binghamton.edu/fridrich/Research/OneHot_Revised.pdf). The CNN has six 3x3 conv layers with residual connections and se-blocks, spatial size is constant 64x64 until the last GAP layer.\n For augmentations, besides rotate90 and flips, cutmix worked pretty well. Each time a stego image was met during training, I also randomly re-assigned +1 and -1 to the DCT values in the modified positions. \n The 2nd-level models was a regular fully-connected net.\n\n**Results:**\n I made totally four 65000x4/10000x4 (train/validation) splits. On each split, I trained one set of the models described above. The final submission was a simple average of the probabilities.\n Except split4 spatial domain model, which was trained with mixup and 10 classes, all the other models were trained with cutmix and 12 classes.\n The validation results are shown below. For this competition, we should not use public LB for model selection.\n| |YCC| DCT|combined|\n| --- | --- |\n|split3|0.9395|0.8706|0.9439|\n|split4|0.9381|0.8749|0.9434|\n|split6 | 0.9409 | 0.8674 | 0.9452|\n|split7 | 0.9418 | 0.8719 | 0.9453|\n\n", "votes": 175 }, { "id": 938321, "postDate": "2020-07-21T13:04:46.627Z", "content": "

Congrats @wowfattie , nice solution. Interesting that I also tried to model DCT coefficients directly reshaping it to 64x64x192, but it scored pretty bad to me (<0.90) and I abandoned the idea. Now I see that even with the low score (~0.87) it adds diversity to the blend. Nice job.

", "rawMarkdown": "Congrats @wowfattie , nice solution. Interesting that I also tried to model DCT coefficients directly reshaping it to 64x64x192, but it scored pretty bad to me (<0.90) and I abandoned the idea. Now I see that even with the low score (~0.87) it adds diversity to the blend. Nice job. ", "votes": 8, "replies": [ { "id": 938418, "postDate": "2020-07-21T14:11:13.027Z", "content": "

To me, 0.9 in the DCT is very good score.

", "rawMarkdown": "To me, 0.9 in the DCT is very good score. ", "votes": 2 } ] }, { "id": 1495832, "postDate": "2021-08-29T20:32:56.373Z", "content": "

Great work! I learn a lot of from your idea !

", "rawMarkdown": "Great work! I learn a lot of from your idea !", "votes": 1 }, { "id": 937557, "postDate": "2020-07-21T04:42:04.857Z", "content": "
\n

For this competition, we should not use public LB for model selection.\n For this competition, we should not use public LB for model selection.\n For this competition, we should not use public LB for model selection.

\n
\n\n

!!!

", "rawMarkdown": "> For this competition, we should not use public LB for model selection.\n> For this competition, we should not use public LB for model selection.\n> For this competition, we should not use public LB for model selection.\n\n!!!", "votes": 5, "replies": [ { "id": 937581, "postDate": "2020-07-21T04:57:43.667Z", "content": "

Yes… This is it!!! I would like to highlight the sentence.

", "rawMarkdown": "Yes... This is it!!! I would like to highlight the sentence.", "votes": 1 }, { "id": 937586, "postDate": "2020-07-21T05:03:22.220Z", "content": "

And here is the provement.\n\"\"

", "rawMarkdown": "And here is the provement.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2006644%2F172ef16743de92b0613731164fca3fd9%2Fcv.png?generation=1595307788634314&alt=media)\n", "votes": 1 }, { "id": 937696, "postDate": "2020-07-21T06:07:25.867Z", "content": "

It is funny that you say
\n\" For this competition, we should not use public LB for model selection.\"
\nwhile you were the guy who make this beautiful analysis that definitively proves it. Probably one the most useful post IMHO by the way !
\nI watch this post with glee since I could access the private LB that totally support you analysis)
\nBeware, however that this analysis also holds true for the the final ranking / the private LB.
\nIf you only pick the top scorer from private LB, they can only improve as compared to the public LB and the opposite is also true; not only showing overfiting, but also a straight consequence of randomized selection.
\nTo conclude on the overfiting, I would rather look at the absolute difference between score from Public LB / Private LB for the very same user ; in you case it sadly show that it overfit a bit too much (which really break my heart given the tons of advise and analysis you made in the discussion !)

\n

PS : the 2nd team were to the same conclusion \"trust you CV\" not the public LB (the former being made of 300.000 image vs 1.000 for latter, no wonder why)

", "rawMarkdown": "It is funny that you say\n*\" For this competition, we should not use public LB for model selection.\"*\nwhile you were the guy [who make this beautiful analysis that definitively proves it](https://www.kaggle.com/c/alaska2-image-steganalysis/discussion/162936). Probably one the most useful post IMHO by the way !\nI watch this post with glee since I could access the private LB that totally support you analysis)\nBeware, however that this analysis also holds true for the the final ranking / the private LB.\nIf you only pick the top scorer from private LB, they can only improve as compared to the public LB and the opposite is also true; not only showing overfiting, but also a straight consequence of randomized selection.\nTo conclude on the overfiting, I would rather look at the absolute difference between score from Public LB / Private LB for the very same user ; in you case it sadly show that it overfit a bit too much (which really break my heart given the tons of advise and analysis you made in the discussion !)\n\nPS : the 2nd team were to the same conclusion \"[trust you CV](https://www.kaggle.com/c/alaska2-image-steganalysis/discussion/168546)\" not the public LB (the former being made of 300.000 image vs 1.000 for latter, no wonder why)", "votes": 4 }, { "id": 937730, "postDate": "2020-07-21T06:22:48.077Z", "content": "

@remicogranne Thank you for this great competition. I learned a lot from it.

", "rawMarkdown": "@remicogranne Thank you for this great competition. I learned a lot from it." }, { "id": 937818, "postDate": "2020-07-21T07:16:47.650Z", "content": "

@remicogranne

\n\n
\n

It is funny that you say\n \" For this competition, we should not use public LB for model selection.\"

\n
\n\n

This is NOT my words, The winner said it. I think it is the most important thing I learned from this competition. So I repeated it three times 😄

", "rawMarkdown": "@remicogranne \n> It is funny that you say\n\" For this competition, we should not use public LB for model selection.\"\n\n\nThis is NOT my words, The winner said it. I think it is the most important thing I learned from this competition. So I repeated it three times 😄 " }, { "id": 937868, "postDate": "2020-07-21T08:00:44.387Z", "content": "

what's your best private LB in your submission?

", "rawMarkdown": "what's your best private LB in your submission?" }, { "id": 937906, "postDate": "2020-07-21T08:28:57.907Z", "content": "

\"\"

", "rawMarkdown": "![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2006644%2Fca60c3aecbc2ca2e271c91e4657410bc%2F2020-07-21%2017.27.43.png?generation=1595320115753277&alt=media)\n" } ] }, { "id": 937706, "postDate": "2020-07-21T06:11:17.293Z", "content": "

Congratulations! I am so glad to see a steganalyst winning this competition. The one-hot encoded net was something I dismissed early on as not being very effective against this type of stego methods despite Yassine's questions \"if we should give it a shot\". Well, sometimes the advisor is wrong and the student is right. This seems to have given you the edge. It reminds me of Gul's features from the first steganalysis competition BOSS (2010). One does not need the detector to be strong as long it is at least little bit complementary. I wonder where the boost from the DCT domain came from, which stego algorithm? BTW, your modified seresnet itself is extremely impressive. I hope you will find time to write the details for WIFS. Once more, great job and enjoy the spotlight! (will this make you a no. 1 kaggler?)

", "rawMarkdown": "Congratulations! I am so glad to see a steganalyst winning this competition. The one-hot encoded net was something I dismissed early on as not being very effective against this type of stego methods despite Yassine's questions \"if we should give it a shot\". Well, sometimes the advisor is wrong and the student is right. This seems to have given you the edge. It reminds me of Gul's features from the first steganalysis competition BOSS (2010). One does not need the detector to be strong as long it is at least little bit complementary. I wonder where the boost from the DCT domain came from, which stego algorithm? BTW, your modified seresnet itself is extremely impressive. I hope you will find time to write the details for WIFS. Once more, great job and enjoy the spotlight! (will this make you a no. 1 kaggler?)", "votes": 6, "replies": [ { "id": 937723, "postDate": "2020-07-21T06:20:09.433Z", "content": "

I am also more than pleased that eventually :
\n(1) two steganalyst get the two first place (and that H. Jang also did great) ; that was not sure at all when looking at ranking during the whole competition
\n(2) the two winning team are both located in NY state. Will certainly motivate them to submit a paper to IEEE WIFS, help in NYC, and will allow to save much of the travel grant !! 😄😄😄😄😄😄

", "rawMarkdown": "I am also more than pleased that eventually :\n(1) two steganalyst get the two first place (and that H. Jang also did great) ; that was not sure at all when looking at ranking during the whole competition\n(2) the two winning team are both located in NY state. Will certainly motivate them to submit a paper to IEEE WIFS, help in NYC, and will allow to save much of the travel grant !! 😄😄😄😄😄😄", "votes": 2 }, { "id": 937769, "postDate": "2020-07-21T06:50:13.530Z", "content": "

@remicogranne isn't IEEE WIFS going to be online?

", "rawMarkdown": "@remicogranne isn't IEEE WIFS going to be online?" }, { "id": 937807, "postDate": "2020-07-21T07:06:24.687Z", "content": "

Oh you are right, did not notice.
\nGood point is that we will save even more on the travel grants :)
\nBad point is that I don't know what to do with the money from IEEE now :c
\nOn the minus sign, I wish you could have a drink to the competition, which we definitively not be the same online :c

", "rawMarkdown": "Oh you are right, did not notice. \nGood point is that we will save even more on the travel grants :)\nBad point is that I don't know what to do with the money from IEEE now :c\nOn the minus sign, I wish you could have a drink to the competition, which we definitively not be the same online :c", "votes": 1 }, { "id": 937809, "postDate": "2020-07-21T07:07:07.720Z", "content": "

👀

", "rawMarkdown": "👀 " }, { "id": 938123, "postDate": "2020-07-21T11:04:52.637Z", "content": "

Remi, do what I do with leftover grant money. Buy a helicopter!

", "rawMarkdown": "Remi, do what I do with leftover grant money. Buy a helicopter!", "votes": 4 } ] }, { "id": 938132, "postDate": "2020-07-21T11:12:47.880Z", "content": "

This is brilliantly mind blowing!!! Does DL flow in your vein? \nMy (gpu)warm congratulations to you @wowfattie :)

", "rawMarkdown": "This is brilliantly mind blowing!!! Does DL flow in your vein? \nMy (gpu)warm congratulations to you @wowfattie :) ", "votes": 3 }, { "id": 937583, "postDate": "2020-07-21T04:58:20.797Z", "content": "

Thanks for sharing your solution. Full of novel ideas and those worked with such a small backbone! Exciting!

", "rawMarkdown": "Thanks for sharing your solution. Full of novel ideas and those worked with such a small backbone! Exciting! ", "votes": 3 }, { "id": 941922, "postDate": "2020-07-23T13:53:11Z", "content": "

Congratulation for the 1st place and for the very innovative proposal (while most users used EfficientNet and only in special domain, you did address the problem in a very different way and it clearly paid off eventually !!);\nA quick question @wowfattie however.\nI noted that you did made a fantastic finish by posting a very last submissions (about an hour before the very deadline) that made you jumped from 0.932 to 0.936.\nI am curious, what did you update / changed for this last submission ?\nIn other words (look at the final board) what definitively secured your win ?

", "rawMarkdown": "Congratulation for the 1st place and for the very innovative proposal (while most users used EfficientNet and only in special domain, you did address the problem in a very different way and it clearly paid off eventually !!);\nA quick question @wowfattie however.\nI noted that you did made a fantastic finish by posting a very last submissions (about an hour before the very deadline) that made you jumped from 0.932 to 0.936.\nI am curious, what did you update / changed for this last submission ?\nIn other words (look at the final board) what definitively secured your win ?", "votes": 4, "replies": [ { "id": 942046, "postDate": "2020-07-23T14:56:54.227Z", "content": "

The 0.936 is the 4-fold ensemble. The 0.932 is a single fold. I only submitted the ensemble in the last day because my last model finished training in the last day.

", "rawMarkdown": "The 0.936 is the 4-fold ensemble. The 0.932 is a single fold. I only submitted the ensemble in the last day because my last model finished training in the last day. ", "votes": 6 }, { "id": 942102, "postDate": "2020-07-23T15:26:05.900Z", "content": "

Thanks for the quick answer. It was really thrilling to watch the private LB over the very last hours : ABBA team has also been improving over the last few days until you slam this last submission that made me go to sleep convinced that \"it is all over now\" ;)

\n\n

Another remark, however, I don't know SE-block, hence SEResNet. It is claimed in this post that \n\"Se-ResNeXt, DenseNet, NASNet, etc. They might work as well, but converge very slowly in comparison to EfficientNet\"

\n\n

Did you observe a (very) long time until convergence from SEResNet ? especially over EfficientNet-bx ?

", "rawMarkdown": "Thanks for the quick answer. It was really thrilling to watch the private LB over the very last hours : ABBA team has also been improving over the last few days until you slam this last submission that made me go to sleep convinced that \"it is all over now\" ;)\n\nAnother remark, however, I don't know SE-block, hence SEResNet. It is claimed in [this post](https://www.kaggle.com/c/alaska2-image-steganalysis/discussion/168702) that \n\"Se-ResNeXt, DenseNet, NASNet, etc. They might work as well, but converge very slowly in comparison to EfficientNet\"\n\nDid you observe a (very) long time until convergence from SEResNet ? especially over EfficientNet-bx ?", "votes": 2 }, { "id": 942192, "postDate": "2020-07-23T16:23:17.800Z", "content": "

The original Se-ResNeXt or DenseNet have slow convergence because they only start 'serious' modelling from 1/4 resolution (two consecutive downsampling at the beginning). For steganalysis, it's important to model the high resolutions. Efficientnets start modeling on the 1/2 resolution, so they can converge fast. My seresnet18 converges fast too because I removed the first two 'downsampling', so it starts modeling from the original resolution (just like YeNet and SRNet).

", "rawMarkdown": "The original Se-ResNeXt or DenseNet have slow convergence because they only start 'serious' modelling from 1/4 resolution (two consecutive downsampling at the beginning). For steganalysis, it's important to model the high resolutions. Efficientnets start modeling on the 1/2 resolution, so they can converge fast. My seresnet18 converges fast too because I removed the first two 'downsampling', so it starts modeling from the original resolution (just like YeNet and SRNet). ", "votes": 5 } ] }, { "id": 937548, "postDate": "2020-07-21T04:35:28.290Z", "content": "

Wow. Just Brilliant. \nCould you please elaborate a little on using bottleneck features. Are you referring to features after global average pooling? or something else?

\n\n

We also tried DCT alone and embeddings for stacking alone, but not DCT embeddings in stacking. Should have been though about it.

\n\n

Well deserved. Congratz!

", "rawMarkdown": "Wow. Just Brilliant. \nCould you please elaborate a little on using bottleneck features. Are you referring to features after global average pooling? or something else? \n\nWe also tried DCT alone and embeddings for stacking alone, but not DCT embeddings in stacking. Should have been though about it.\n\nWell deserved. Congratz!", "votes": 2, "replies": [ { "id": 938282, "postDate": "2020-07-21T12:36:35.700Z", "content": "

Thanks. Yes, the output of the GAP layer. (512-D for seresnet18)

", "rawMarkdown": "Thanks. Yes, the output of the GAP layer. (512-D for seresnet18)", "votes": 1 } ] }, { "id": 938944, "postDate": "2020-07-21T22:07:44.180Z", "content": "

lalala

", "rawMarkdown": "lalala", "votes": -1 }, { "id": 1310860, "postDate": "2021-05-17T03:10:02.590Z", "content": "

Congratulations… Can you please share your notebook

", "rawMarkdown": "Congratulations... Can you please share your notebook" }, { "id": 959584, "postDate": "2020-08-05T18:13:00.783Z", "content": "

Congrats!

", "rawMarkdown": "Congrats!" }, { "id": 946694, "postDate": "2020-07-26T18:40:14.790Z", "content": "

Congratulations.

", "rawMarkdown": "Congratulations." }, { "id": 946502, "postDate": "2020-07-26T16:03:40.877Z", "content": "

Very interesting approach with removing the stride in the first conv layer and the max-pooling layer. Hope that this can help me in real tasks. How much score boost it gives? And how much gives fold's ensembling?

", "rawMarkdown": "Very interesting approach with removing the stride in the first conv layer and the max-pooling layer. Hope that this can help me in real tasks. How much score boost it gives? And how much gives fold's ensembling?" }, { "id": 944694, "postDate": "2020-07-25T09:02:54.793Z", "content": "

Congrats!

", "rawMarkdown": "Congrats!" }, { "id": 944382, "postDate": "2020-07-25T04:25:59.157Z", "content": "

Congrats!

", "rawMarkdown": "Congrats!" }, { "id": 944367, "postDate": "2020-07-25T04:06:36.690Z", "content": "

Nice work！

", "rawMarkdown": "Nice work！" }, { "id": 944128, "postDate": "2020-07-24T21:13:48.797Z", "content": "

Congratulations.

", "rawMarkdown": "Congratulations." }, { "id": 944086, "postDate": "2020-07-24T20:04:31.343Z", "content": "

Congrats on first place!

", "rawMarkdown": "Congrats on first place! " }, { "id": 943528, "postDate": "2020-07-24T12:16:10.310Z", "content": "

Well done.Nice solution

", "rawMarkdown": "Well done.Nice solution" }, { "id": 942629, "postDate": "2020-07-23T22:15:30.697Z", "content": "

Congrats

", "rawMarkdown": "Congrats\n" }, { "id": 942321, "postDate": "2020-07-23T17:40:35.667Z", "content": "

Congrats

", "rawMarkdown": "Congrats" }, { "id": 942281, "postDate": "2020-07-23T17:14:56.603Z", "content": "

Thanks @wowfattie for your outstanding work and share with us

", "rawMarkdown": "Thanks @wowfattie for your outstanding work and share with us" }, { "id": 941475, "postDate": "2020-07-23T08:34:00.797Z", "content": "

Wow, that's pretty nice

", "rawMarkdown": "Wow, that's pretty nice" }, { "id": 940815, "postDate": "2020-07-23T05:52:57.223Z", "content": "

congrats

", "rawMarkdown": "congrats" }, { "id": 940810, "postDate": "2020-07-23T05:48:39.800Z", "content": "

Congratulations, very successful

", "rawMarkdown": "Congratulations, very successful\n" }, { "id": 940055, "postDate": "2020-07-22T16:42:45.660Z", "content": "

This is quite a nice solution! Cheers!

", "rawMarkdown": "This is quite a nice solution! Cheers!" }, { "id": 939995, "postDate": "2020-07-22T16:01:34.140Z", "content": "

Congrats and thanks for sharing the info. Its always helpful to learn the methods for beginners like me

", "rawMarkdown": "Congrats and thanks for sharing the info. Its always helpful to learn the methods for beginners like me" }, { "id": 939967, "postDate": "2020-07-22T15:45:56.067Z", "content": "

congrats, always learn something new from your solutions

", "rawMarkdown": "congrats, always learn something new from your solutions" }, { "id": 939919, "postDate": "2020-07-22T15:07:59.490Z", "content": "

this is great

", "rawMarkdown": "this is great" }, { "id": 939881, "postDate": "2020-07-22T14:49:14.617Z", "content": "

Nice! Very instructive for some of the things with image processing

", "rawMarkdown": "Nice! Very instructive for some of the things with image processing" }, { "id": 938820, "postDate": "2020-07-21T19:18:26.153Z", "content": "

Wowww, very nice!!

", "rawMarkdown": "Wowww, very nice!!" }, { "id": 938685, "postDate": "2020-07-21T17:19:46.700Z", "content": "

Congrats for your 1st place !

", "rawMarkdown": "Congrats for your 1st place ! " }, { "id": 938682, "postDate": "2020-07-21T17:17:26.983Z", "content": "

Nice catch! Congratulations!

", "rawMarkdown": "Nice catch! Congratulations!" }, { "id": 938656, "postDate": "2020-07-21T16:51:09.577Z", "content": "

Congratulations!

", "rawMarkdown": "Congratulations!" }, { "id": 938609, "postDate": "2020-07-21T16:20:53.010Z", "content": "

Congrats @wowfattie

\n\n

thank you for sharing the solution steps!

\n\n

While , we all understand we should not use public LB for model selection.\nhow did you decide on when you think it starts to overfit? any suggestions and inputs! thanks!

", "rawMarkdown": "Congrats @wowfattie \n\nthank you for sharing the solution steps!\n\nWhile , we all understand we should not use public LB for model selection.\nhow did you decide on when you think it starts to overfit? any suggestions and inputs! thanks!" }, { "id": 938517, "postDate": "2020-07-21T15:27:05.103Z", "content": "

nice idea inspire my mind

", "rawMarkdown": "nice idea inspire my mind" }, { "id": 938487, "postDate": "2020-07-21T15:11:14.617Z", "content": "

Congratulations.

", "rawMarkdown": "Congratulations." }, { "id": 938340, "postDate": "2020-07-21T13:17:30.447Z", "content": "

Congrats:D

", "rawMarkdown": "Congrats:D" }, { "id": 938296, "postDate": "2020-07-21T12:44:11.350Z", "content": "

Nice job!

", "rawMarkdown": "Nice job!" }, { "id": 938287, "postDate": "2020-07-21T12:39:21.360Z", "content": "

@wowfattie thanks a lot for the great write up and congratulations! Is your solution approach have similarity in the principle of Harmonic Convolution block? https://paperswithcode.com/method/harmonic-block, https://github.com/matej-ulicny/harmonic-networks

", "rawMarkdown": "@wowfattie thanks a lot for the great write up and congratulations! Is your solution approach have similarity in the principle of Harmonic Convolution block? https://paperswithcode.com/method/harmonic-block, https://github.com/matej-ulicny/harmonic-networks ", "replies": [ { "id": 938420, "postDate": "2020-07-21T14:12:39.867Z", "content": "

I'm not sure. I don't know this harmonic block

", "rawMarkdown": "I'm not sure. I don't know this harmonic block\n" } ] }, { "id": 938199, "postDate": "2020-07-21T11:53:08.883Z", "content": "

@wowfattie

\n\n

thanks for the writeup. i am now implementing your methods.

\n\n

It is mentioned: \"The spatial domain model was seresnet18 initialized with imagenet weights. To make training work I removed the stride in the first conv layer and the max-pooling layer\"

\n\n

with stride and max pooling removed, it takes a long time to train. May i know how long does it takes to train a epoch and what is the batch size?

", "rawMarkdown": "@wowfattie \n\nthanks for the writeup. i am now implementing your methods. \n\nIt is mentioned: \"The spatial domain model was seresnet18 initialized with imagenet weights. To make training work I removed the stride in the first conv layer and the max-pooling layer\"\n\nwith stride and max pooling removed, it takes a long time to train. May i know how long does it takes to train a epoch and what is the batch size?", "replies": [ { "id": 938423, "postDate": "2020-07-21T14:15:44.707Z", "content": "

Yes, removing the strides and pooling makes the CNN 16 times more complex. It took 48-50 minutes/epoch . The computation complexity should be similar to the efficientnet b5 in pytorch.

", "rawMarkdown": "Yes, removing the strides and pooling makes the CNN 16 times more complex. It took 48-50 minutes/epoch . The computation complexity should be similar to the efficientnet b5 in pytorch.", "votes": 3 } ] }, { "id": 938018, "postDate": "2020-07-21T09:54:29.783Z", "content": "

Congrats mate

", "rawMarkdown": "Congrats mate" }, { "id": 937942, "postDate": "2020-07-21T08:50:26.643Z", "content": "

Congrats @wowfattie and thank you for the write-up!

", "rawMarkdown": "Congrats @wowfattie and thank you for the write-up!" }, { "id": 937820, "postDate": "2020-07-21T07:17:27.283Z", "content": "

Thanks for sharing! It is very nice work without large off-the-shelf models.
\nCan I know how much did cutmix help?

", "rawMarkdown": "Thanks for sharing! It is very nice work without large off-the-shelf models.\nCan I know how much did cutmix help?", "replies": [ { "id": 938427, "postDate": "2020-07-21T14:17:33.203Z", "content": "

I don't have a solid comparison. My estimate would be 0.005-0.01 improvement.

", "rawMarkdown": "I don't have a solid comparison. My estimate would be 0.005-0.01 improvement.", "votes": 1 } ] }, { "id": 937814, "postDate": "2020-07-21T07:12:47.227Z", "content": "

Nice job! Could you please share hardware details?

", "rawMarkdown": "Nice job! Could you please share hardware details? " }, { "id": 937660, "postDate": "2020-07-21T05:45:42.957Z", "content": "

Congrats @wowfattie, glad you were inspired by our OneHotConv work! \nBTW your DCT model performs around 2% better in wAUC compared to the rich models we used (JRM/DCTR) which is also compatible with our findings in the paper you cited!

", "rawMarkdown": "Congrats @wowfattie, glad you were inspired by our OneHotConv work! \nBTW your DCT model performs around 2% better in wAUC compared to the rich models we used (JRM/DCTR) which is also compatible with our findings in the paper you cited!\n", "replies": [ { "id": 937691, "postDate": "2020-07-21T06:04:37.643Z", "content": "

I am impressed that this paper (published two months ago) was already a source of inspiration !
\nNew papers spread almost as quickly as the coronavirus !

", "rawMarkdown": "I am impressed that this paper (published two months ago) was already a source of inspiration !\nNew papers spread almost as quickly as the coronavirus !\n", "votes": 2 }, { "id": 938310, "postDate": "2020-07-21T12:56:08.817Z", "content": "

Thanks. Your paper is a rare masterpiece for steganalysis in the DCT domain.

", "rawMarkdown": "Thanks. Your paper is a rare masterpiece for steganalysis in the DCT domain.", "votes": 1 } ] }, { "id": 937614, "postDate": "2020-07-21T05:27:02.593Z", "content": "

Great Job!\nAnd congratulations 1st place!

", "rawMarkdown": "Great Job!\nAnd congratulations 1st place! " }, { "id": 937584, "postDate": "2020-07-21T05:02:12.653Z", "content": "

A brilliant idea that combines the model based on the spatial domain and the model based on the DCT domain. I also tried to train the model based on the DCT domain(OneHotNet) and did not achieve good prediction.

\n

Congratulations!!!

", "rawMarkdown": "A brilliant idea that combines the model based on the spatial domain and the model based on the DCT domain. I also tried to train the model based on the DCT domain(OneHotNet) and did not achieve good prediction.\n\nCongratulations!!!", "replies": [ { "id": 937743, "postDate": "2020-07-21T06:30:36.080Z", "content": "

Yup it seems that DCT domain (oneHotNet or other) do perform worse yet it help when combined with spatial domain analysis. And this 0.5% ~ 1% could have pushed to the first place

", "rawMarkdown": "Yup it seems that DCT domain (oneHotNet or other) do perform worse yet it help when combined with spatial domain analysis. And this 0.5% ~ 1% could have pushed to the first place", "votes": 1 } ] }, { "id": 937552, "postDate": "2020-07-21T04:38:02.443Z", "content": "

Thanks for sharing. And congratulations 1st rank for both this competition and all competitions. :)

", "rawMarkdown": "Thanks for sharing. And congratulations 1st rank for both this competition and all competitions. :)" }, { "id": 943540, "postDate": "2020-07-24T12:28:42.800Z", "rawMarkdown": "", "isDeleted": true }, { "id": 942067, "postDate": "2020-07-23T15:06:11.443Z", "rawMarkdown": "", "isDeleted": true }, { "id": 941360, "postDate": "2020-07-23T07:25:09.867Z", "rawMarkdown": "", "isDeleted": true, "replies": [ { "id": 941876, "postDate": "2020-07-23T13:18:08.817Z", "content": "

Set the stride parameter of the first convolutional layer (torch.nn.Conv2d) to 1. Most of the main stream CNNs have stride=2 in their first conv layer.

", "rawMarkdown": "Set the stride parameter of the first convolutional layer (torch.nn.Conv2d) to 1. Most of the main stream CNNs have stride=2 in their first conv layer.", "votes": 1 } ] }, { "id": 939620, "postDate": "2020-07-22T10:39:14.253Z", "rawMarkdown": "", "isDeleted": true }, { "id": 938906, "postDate": "2020-07-21T21:08:38.240Z", "rawMarkdown": "", "isDeleted": true }, { "id": 937803, "postDate": "2020-07-21T07:05:02.813Z", "rawMarkdown": "", "isDeleted": true }, { "id": 946542, "postDate": "2020-07-26T16:31:27.727Z", "content": "

Thanks for sharing your solution.

", "rawMarkdown": "Thanks for sharing your solution." }, { "id": 944906, "postDate": "2020-07-25T12:49:13.347Z", "content": "

Thank you.

", "rawMarkdown": "Thank you." }, { "id": 938494, "postDate": "2020-07-21T15:16:46.297Z", "content": "

Awesome, very informative. Thanks

", "rawMarkdown": "Awesome, very informative. Thanks" } ], "comments": [ { "id": 938321, "author_name": "Giba", "author_url": "", "post_date": "2020-07-21T13:04:46.627000", "content": "

Congrats @wowfattie , nice solution. Interesting that I also tried to model DCT coefficients directly reshaping it to 64x64x192, but it scored pretty bad to me (<0.90) and I abandoned the idea. Now I see that even with the low score (~0.87) it adds diversity to the blend. Nice job.

", "votes": 8, "replies": [ { "id": 938418, "author_name": "Guanshuo Xu", "author_url": "", "post_date": "2020-07-21T14:11:13.027000", "content": "

To me, 0.9 in the DCT is very good score.

", "votes": 2, "replies": [] } ] }, { "id": 1495832, "author_name": "Fuco", "author_url": "", "post_date": "2021-08-29T20:32:56.373000", "content": "

Great work! I learn a lot of from your idea !

", "votes": 1, "replies": [] }, { "id": 937557, "author_name": "Johnny Lee", "author_url": "", "post_date": "2020-07-21T04:42:04.857000", "content": "
\n

For this competition, we should not use public LB for model selection.\n For this competition, we should not use public LB for model selection.\n For this competition, we should not use public LB for model selection.

\n
\n\n

!!!

", "votes": 5, "replies": [ { "id": 937581, "author_name": "Haneol Jang", "author_url": "", "post_date": "2020-07-21T04:57:43.667000", "content": "

Yes… This is it!!! I would like to highlight the sentence.

", "votes": 1, "replies": [] }, { "id": 937586, "author_name": "Johnny Lee", "author_url": "", "post_date": "2020-07-21T05:03:22.220000", "content": "

And here is the provement.\n\"\"

", "votes": 1, "replies": [] }, { "id": 937696, "author_name": "Rémi Cogranne", "author_url": "", "post_date": "2020-07-21T06:07:25.867000", "content": "

It is funny that you say
\n\" For this competition, we should not use public LB for model selection.\"
\nwhile you were the guy who make this beautiful analysis that definitively proves it. Probably one the most useful post IMHO by the way !
\nI watch this post with glee since I could access the private LB that totally support you analysis)
\nBeware, however that this analysis also holds true for the the final ranking / the private LB.
\nIf you only pick the top scorer from private LB, they can only improve as compared to the public LB and the opposite is also true; not only showing overfiting, but also a straight consequence of randomized selection.
\nTo conclude on the overfiting, I would rather look at the absolute difference between score from Public LB / Private LB for the very same user ; in you case it sadly show that it overfit a bit too much (which really break my heart given the tons of advise and analysis you made in the discussion !)

\n

PS : the 2nd team were to the same conclusion \"trust you CV\" not the public LB (the former being made of 300.000 image vs 1.000 for latter, no wonder why)

", "votes": 4, "replies": [] }, { "id": 937730, "author_name": "Johnny Lee", "author_url": "", "post_date": "2020-07-21T06:22:48.077000", "content": "

@remicogranne Thank you for this great competition. I learned a lot from it.

", "votes": 0, "replies": [] }, { "id": 937818, "author_name": "Johnny Lee", "author_url": "", "post_date": "2020-07-21T07:16:47.650000", "content": "

@remicogranne

\n\n
\n

It is funny that you say\n \" For this competition, we should not use public LB for model selection.\"

\n
\n\n

This is NOT my words, The winner said it. I think it is the most important thing I learned from this competition. So I repeated it three times 😄

", "votes": 0, "replies": [] }, { "id": 937868, "author_name": "BokingChen", "author_url": "", "post_date": "2020-07-21T08:00:44.387000", "content": "

what's your best private LB in your submission?

", "votes": 0, "replies": [] }, { "id": 937906, "author_name": "Johnny Lee", "author_url": "", "post_date": "2020-07-21T08:28:57.907000", "content": "

\"\"

", "votes": 0, "replies": [] } ] }, { "id": 937706, "author_name": "Jessica Fridrich", "author_url": "", "post_date": "2020-07-21T06:11:17.293000", "content": "

Congratulations! I am so glad to see a steganalyst winning this competition. The one-hot encoded net was something I dismissed early on as not being very effective against this type of stego methods despite Yassine's questions \"if we should give it a shot\". Well, sometimes the advisor is wrong and the student is right. This seems to have given you the edge. It reminds me of Gul's features from the first steganalysis competition BOSS (2010). One does not need the detector to be strong as long it is at least little bit complementary. I wonder where the boost from the DCT domain came from, which stego algorithm? BTW, your modified seresnet itself is extremely impressive. I hope you will find time to write the details for WIFS. Once more, great job and enjoy the spotlight! (will this make you a no. 1 kaggler?)

", "votes": 6, "replies": [ { "id": 937723, "author_name": "Rémi Cogranne", "author_url": "", "post_date": "2020-07-21T06:20:09.433000", "content": "

I am also more than pleased that eventually :
\n(1) two steganalyst get the two first place (and that H. Jang also did great) ; that was not sure at all when looking at ranking during the whole competition
\n(2) the two winning team are both located in NY state. Will certainly motivate them to submit a paper to IEEE WIFS, help in NYC, and will allow to save much of the travel grant !! 😄😄😄😄😄😄

", "votes": 2, "replies": [] }, { "id": 937769, "author_name": "Yassine Yousfi", "author_url": "", "post_date": "2020-07-21T06:50:13.530000", "content": "

@remicogranne isn't IEEE WIFS going to be online?

", "votes": 0, "replies": [] }, { "id": 937807, "author_name": "Rémi Cogranne", "author_url": "", "post_date": "2020-07-21T07:06:24.687000", "content": "

Oh you are right, did not notice.
\nGood point is that we will save even more on the travel grants :)
\nBad point is that I don't know what to do with the money from IEEE now :c
\nOn the minus sign, I wish you could have a drink to the competition, which we definitively not be the same online :c

", "votes": 1, "replies": [] }, { "id": 937809, "author_name": "Yassine Yousfi", "author_url": "", "post_date": "2020-07-21T07:07:07.720000", "content": "

👀

", "votes": 0, "replies": [] }, { "id": 938123, "author_name": "Jessica Fridrich", "author_url": "", "post_date": "2020-07-21T11:04:52.637000", "content": "

Remi, do what I do with leftover grant money. Buy a helicopter!

", "votes": 4, "replies": [] } ] }, { "id": 938132, "author_name": "olivier", "author_url": "", "post_date": "2020-07-21T11:12:47.880000", "content": "

This is brilliantly mind blowing!!! Does DL flow in your vein? \nMy (gpu)warm congratulations to you @wowfattie :)

", "votes": 3, "replies": [] }, { "id": 937583, "author_name": "Shai", "author_url": "", "post_date": "2020-07-21T04:58:20.797000", "content": "

Thanks for sharing your solution. Full of novel ideas and those worked with such a small backbone! Exciting!

", "votes": 3, "replies": [] }, { "id": 941922, "author_name": "Rémi Cogranne", "author_url": "", "post_date": "2020-07-23T13:53:11", "content": "

Congratulation for the 1st place and for the very innovative proposal (while most users used EfficientNet and only in special domain, you did address the problem in a very different way and it clearly paid off eventually !!);\nA quick question @wowfattie however.\nI noted that you did made a fantastic finish by posting a very last submissions (about an hour before the very deadline) that made you jumped from 0.932 to 0.936.\nI am curious, what did you update / changed for this last submission ?\nIn other words (look at the final board) what definitively secured your win ?

", "votes": 4, "replies": [ { "id": 942046, "author_name": "Guanshuo Xu", "author_url": "", "post_date": "2020-07-23T14:56:54.227000", "content": "

The 0.936 is the 4-fold ensemble. The 0.932 is a single fold. I only submitted the ensemble in the last day because my last model finished training in the last day.

", "votes": 6, "replies": [] }, { "id": 942102, "author_name": "Rémi Cogranne", "author_url": "", "post_date": "2020-07-23T15:26:05.900000", "content": "

Thanks for the quick answer. It was really thrilling to watch the private LB over the very last hours : ABBA team has also been improving over the last few days until you slam this last submission that made me go to sleep convinced that \"it is all over now\" ;)

\n\n

Another remark, however, I don't know SE-block, hence SEResNet. It is claimed in this post that \n\"Se-ResNeXt, DenseNet, NASNet, etc. They might work as well, but converge very slowly in comparison to EfficientNet\"

\n\n

Did you observe a (very) long time until convergence from SEResNet ? especially over EfficientNet-bx ?

", "votes": 2, "replies": [] }, { "id": 942192, "author_name": "Guanshuo Xu", "author_url": "", "post_date": "2020-07-23T16:23:17.800000", "content": "

The original Se-ResNeXt or DenseNet have slow convergence because they only start 'serious' modelling from 1/4 resolution (two consecutive downsampling at the beginning). For steganalysis, it's important to model the high resolutions. Efficientnets start modeling on the 1/2 resolution, so they can converge fast. My seresnet18 converges fast too because I removed the first two 'downsampling', so it starts modeling from the original resolution (just like YeNet and SRNet).

", "votes": 5, "replies": [] } ] }, { "id": 937548, "author_name": "Eugene Khvedchenya", "author_url": "", "post_date": "2020-07-21T04:35:28.290000", "content": "

Wow. Just Brilliant. \nCould you please elaborate a little on using bottleneck features. Are you referring to features after global average pooling? or something else?

\n\n

We also tried DCT alone and embeddings for stacking alone, but not DCT embeddings in stacking. Should have been though about it.

\n\n

Well deserved. Congratz!

", "votes": 2, "replies": [ { "id": 938282, "author_name": "Guanshuo Xu", "author_url": "", "post_date": "2020-07-21T12:36:35.700000", "content": "

Thanks. Yes, the output of the GAP layer. (512-D for seresnet18)

", "votes": 1, "replies": [] } ] }, { "id": 938944, "author_name": "Deborah Dias Okasaki", "author_url": "", "post_date": "2020-07-21T22:07:44.180000", "content": "

lalala

", "votes": -1, "replies": [] }, { "id": 1310860, "author_name": "Hemanth Harikrishnan", "author_url": "", "post_date": "2021-05-17T03:10:02.590000", "content": "

Congratulations… Can you please share your notebook

", "votes": 0, "replies": [] }, { "id": 959584, "author_name": "Jack", "author_url": "", "post_date": "2020-08-05T18:13:00.783000", "content": "

Congrats!

", "votes": 0, "replies": [] }, { "id": 946694, "author_name": "Md Shafiul Islam", "author_url": "", "post_date": "2020-07-26T18:40:14.790000", "content": "

Congratulations.

", "votes": 0, "replies": [] }, { "id": 946502, "author_name": "ZavodRobotov", "author_url": "", "post_date": "2020-07-26T16:03:40.877000", "content": "

Very interesting approach with removing the stride in the first conv layer and the max-pooling layer. Hope that this can help me in real tasks. How much score boost it gives? And how much gives fold's ensembling?

", "votes": 0, "replies": [] }, { "id": 944694, "author_name": "Parmveer Singh", "author_url": "", "post_date": "2020-07-25T09:02:54.793000", "content": "

Congrats!

", "votes": 0, "replies": [] }, { "id": 944382, "author_name": "pheasheiwdb", "author_url": "", "post_date": "2020-07-25T04:25:59.157000", "content": "

Congrats!

", "votes": 0, "replies": [] }, { "id": 944367, "author_name": "shettornvxwa", "author_url": "", "post_date": "2020-07-25T04:06:36.690000", "content": "

Nice work！

", "votes": 0, "replies": [] }, { "id": 944128, "author_name": "kuntsevich", "author_url": "", "post_date": "2020-07-24T21:13:48.797000", "content": "

Congratulations.

", "votes": 0, "replies": [] }, { "id": 944086, "author_name": "Sheridan Green", "author_url": "", "post_date": "2020-07-24T20:04:31.343000", "content": "

Congrats on first place!

", "votes": 0, "replies": [] }, { "id": 943528, "author_name": "H.Yiğit Cem Altıntaş", "author_url": "", "post_date": "2020-07-24T12:16:10.310000", "content": "

Well done.Nice solution

", "votes": 0, "replies": [] }, { "id": 942629, "author_name": "Aman Kumar Mallik", "author_url": "", "post_date": "2020-07-23T22:15:30.697000", "content": "

Congrats

", "votes": 0, "replies": [] }, { "id": 942321, "author_name": "Rodrigo Martins", "author_url": "", "post_date": "2020-07-23T17:40:35.667000", "content": "

Congrats

", "votes": 0, "replies": [] }, { "id": 942281, "author_name": "Mahmud Hasan", "author_url": "", "post_date": "2020-07-23T17:14:56.603000", "content": "

Thanks @wowfattie for your outstanding work and share with us

", "votes": 0, "replies": [] }, { "id": 941475, "author_name": "Dmytro Hlushenkov", "author_url": "", "post_date": "2020-07-23T08:34:00.797000", "content": "

Wow, that's pretty nice

", "votes": 0, "replies": [] }, { "id": 940815, "author_name": "Syed Md Danish E Azam", "author_url": "", "post_date": "2020-07-23T05:52:57.223000", "content": "

congrats

", "votes": 0, "replies": [] }, { "id": 940810, "author_name": "Hamza Tanç", "author_url": "", "post_date": "2020-07-23T05:48:39.800000", "content": "

Congratulations, very successful

", "votes": 0, "replies": [] }, { "id": 940055, "author_name": "Prasanna Rahavendra A", "author_url": "", "post_date": "2020-07-22T16:42:45.660000", "content": "

This is quite a nice solution! Cheers!

", "votes": 0, "replies": [] }, { "id": 939995, "author_name": "Anil", "author_url": "", "post_date": "2020-07-22T16:01:34.140000", "content": "

Congrats and thanks for sharing the info. Its always helpful to learn the methods for beginners like me

", "votes": 0, "replies": [] }, { "id": 939967, "author_name": "steelrose", "author_url": "", "post_date": "2020-07-22T15:45:56.067000", "content": "

congrats, always learn something new from your solutions

", "votes": 0, "replies": [] }, { "id": 939919, "author_name": "SAI KOUSHIK MUPPARAAPU", "author_url": "", "post_date": "2020-07-22T15:07:59.490000", "content": "

this is great

", "votes": 0, "replies": [] }, { "id": 939881, "author_name": "", "author_url": "", "post_date": "2020-07-22T14:49:14.617000", "content": "", "votes": 0, "replies": [] }, { "id": 938820, "author_name": "", "author_url": "", "post_date": "2020-07-21T19:18:26.153000", "content": "", "votes": 0, "replies": [] }, { "id": 938685, "author_name": "", "author_url": "", "post_date": "2020-07-21T17:19:46.700000", "content": "", "votes": 0, "replies": [] }, { "id": 938682, "author_name": "", "author_url": "", "post_date": "2020-07-21T17:17:26.983000", "content": "", "votes": 0, "replies": [] }, { "id": 938656, "author_name": "", "author_url": "", "post_date": "2020-07-21T16:51:09.577000", "content": "", "votes": 0, "replies": [] }, { "id": 938609, "author_name": "", "author_url": "", "post_date": "2020-07-21T16:20:53.010000", "content": "", "votes": 0, "replies": [] }, { "id": 938517, "author_name": "", "author_url": "", "post_date": "2020-07-21T15:27:05.103000", "content": "", "votes": 0, "replies": [] }, { "id": 938487, "author_name": "", "author_url": "", "post_date": "2020-07-21T15:11:14.617000", "content": "", "votes": 0, "replies": [] }, { "id": 938340, "author_name": "", "author_url": "", "post_date": "2020-07-21T13:17:30.447000", "content": "", "votes": 0, "replies": [] }, { "id": 938296, "author_name": "", "author_url": "", "post_date": "2020-07-21T12:44:11.350000", "content": "", "votes": 0, "replies": [] }, { "id": 938287, "author_name": "", "author_url": "", "post_date": "2020-07-21T12:39:21.360000", "content": "", "votes": 0, "replies": [ { "id": 938420, "author_name": "", "author_url": "", "post_date": "2020-07-21T14:12:39.867000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 938199, "author_name": "", "author_url": "", "post_date": "2020-07-21T11:53:08.883000", "content": "", "votes": 0, "replies": [ { "id": 938423, "author_name": "", "author_url": "", "post_date": "2020-07-21T14:15:44.707000", "content": "", "votes": 3, "replies": [] } ] }, { "id": 938018, "author_name": "", "author_url": "", "post_date": "2020-07-21T09:54:29.783000", "content": "", "votes": 0, "replies": [] }, { "id": 937942, "author_name": "", "author_url": "", "post_date": "2020-07-21T08:50:26.643000", "content": "", "votes": 0, "replies": [] }, { "id": 937820, "author_name": "", "author_url": "", "post_date": "2020-07-21T07:17:27.283000", "content": "", "votes": 0, "replies": [ { "id": 938427, "author_name": "", "author_url": "", "post_date": "2020-07-21T14:17:33.203000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 937814, "author_name": "", "author_url": "", "post_date": "2020-07-21T07:12:47.227000", "content": "", "votes": 0, "replies": [] }, { "id": 937660, "author_name": "", "author_url": "", "post_date": "2020-07-21T05:45:42.957000", "content": "", "votes": 0, "replies": [ { "id": 937691, "author_name": "", "author_url": "", "post_date": "2020-07-21T06:04:37.643000", "content": "", "votes": 2, "replies": [] }, { "id": 938310, "author_name": "", "author_url": "", "post_date": "2020-07-21T12:56:08.817000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 937614, "author_name": "", "author_url": "", "post_date": "2020-07-21T05:27:02.593000", "content": "", "votes": 0, "replies": [] }, { "id": 937584, "author_name": "", "author_url": "", "post_date": "2020-07-21T05:02:12.653000", "content": "", "votes": 0, "replies": [ { "id": 937743, "author_name": "", "author_url": "", "post_date": "2020-07-21T06:30:36.080000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 937552, "author_name": "", "author_url": "", "post_date": "2020-07-21T04:38:02.443000", "content": "", "votes": 0, "replies": [] }, { "id": 943540, "author_name": "", "author_url": "", "post_date": "2020-07-24T12:28:42.800000", "content": "", "votes": 0, "replies": [] }, { "id": 942067, "author_name": "", "author_url": "", "post_date": "2020-07-23T15:06:11.443000", "content": "", "votes": 0, "replies": [] }, { "id": 941360, "author_name": "", "author_url": "", "post_date": "2020-07-23T07:25:09.867000", "content": "", "votes": 0, "replies": [ { "id": 941876, "author_name": "", "author_url": "", "post_date": "2020-07-23T13:18:08.817000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 939620, "author_name": "", "author_url": "", "post_date": "2020-07-22T10:39:14.253000", "content": "", "votes": 0, "replies": [] }, { "id": 938906, "author_name": "", "author_url": "", "post_date": "2020-07-21T21:08:38.240000", "content": "", "votes": 0, "replies": [] }, { "id": 937803, "author_name": "", "author_url": "", "post_date": "2020-07-21T07:05:02.813000", "content": "", "votes": 0, "replies": [] }, { "id": 946542, "author_name": "", "author_url": "", "post_date": "2020-07-26T16:31:27.727000", "content": "", "votes": 0, "replies": [] }, { "id": 944906, "author_name": "", "author_url": "", "post_date": "2020-07-25T12:49:13.347000", "content": "", "votes": 0, "replies": [] }, { "id": 938494, "author_name": "", "author_url": "", "post_date": "2020-07-21T15:16:46.297000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "937538": "**Methods:**\n I combined a model trained in the spatial domain (YCbCr) and a model trained in the DCT domain with a second-level model trained using their bottleneck features.\n The spatial domain model was seresnet18 initialized with imagenet weights. To make training work I removed the stride in the first conv layer and the max-pooling layer. It's important for the model to stay at higher resolutions longer to capture the weak stego signals. Channel-attention with the se-block also gives significant improvement when comparing the results of resnet18 and seresnet18.\n To model in the DCT domain, I transformed the 8x8=64 DCT components as input \"channels\", so the original 3x512x512 becomes (64x3=192)x64x64. The 192 raw DCT values was one-hot encoded before entering the CNN following the idea of this paper (http://www.ws.binghamton.edu/fridrich/Research/OneHot_Revised.pdf). The CNN has six 3x3 conv layers with residual connections and se-blocks, spatial size is constant 64x64 until the last GAP layer.\n For augmentations, besides rotate90 and flips, cutmix worked pretty well. Each time a stego image was met during training, I also randomly re-assigned +1 and -1 to the DCT values in the modified positions. \n The 2nd-level models was a regular fully-connected net.\n\n**Results:**\n I made totally four 65000x4/10000x4 (train/validation) splits. On each split, I trained one set of the models described above. The final submission was a simple average of the probabilities.\n Except split4 spatial domain model, which was trained with mixup and 10 classes, all the other models were trained with cutmix and 12 classes.\n The validation results are shown below. For this competition, we should not use public LB for model selection.\n| |YCC| DCT|combined|\n| --- | --- |\n|split3|0.9395|0.8706|0.9439|\n|split4|0.9381|0.8749|0.9434|\n|split6 | 0.9409 | 0.8674 | 0.9452|\n|split7 | 0.9418 | 0.8719 | 0.9453|\n\n", "938321": "Congrats @wowfattie , nice solution. Interesting that I also tried to model DCT coefficients directly reshaping it to 64x64x192, but it scored pretty bad to me (<0.90) and I abandoned the idea. Now I see that even with the low score (~0.87) it adds diversity to the blend. Nice job. ", "1495832": "Great work! I learn a lot of from your idea !", "937557": "> For this competition, we should not use public LB for model selection.\n> For this competition, we should not use public LB for model selection.\n> For this competition, we should not use public LB for model selection.\n\n!!!", "937706": "Congratulations! I am so glad to see a steganalyst winning this competition. The one-hot encoded net was something I dismissed early on as not being very effective against this type of stego methods despite Yassine's questions \"if we should give it a shot\". Well, sometimes the advisor is wrong and the student is right. This seems to have given you the edge. It reminds me of Gul's features from the first steganalysis competition BOSS (2010). One does not need the detector to be strong as long it is at least little bit complementary. I wonder where the boost from the DCT domain came from, which stego algorithm? BTW, your modified seresnet itself is extremely impressive. I hope you will find time to write the details for WIFS. Once more, great job and enjoy the spotlight! (will this make you a no. 1 kaggler?)", "938132": "This is brilliantly mind blowing!!! Does DL flow in your vein? \nMy (gpu)warm congratulations to you @wowfattie :) ", "937583": "Thanks for sharing your solution. Full of novel ideas and those worked with such a small backbone! Exciting! ", "941922": "Congratulation for the 1st place and for the very innovative proposal (while most users used EfficientNet and only in special domain, you did address the problem in a very different way and it clearly paid off eventually !!);\nA quick question @wowfattie however.\nI noted that you did made a fantastic finish by posting a very last submissions (about an hour before the very deadline) that made you jumped from 0.932 to 0.936.\nI am curious, what did you update / changed for this last submission ?\nIn other words (look at the final board) what definitively secured your win ?", "937548": "Wow. Just Brilliant. \nCould you please elaborate a little on using bottleneck features. Are you referring to features after global average pooling? or something else? \n\nWe also tried DCT alone and embeddings for stacking alone, but not DCT embeddings in stacking. Should have been though about it.\n\nWell deserved. Congratz!", "938944": "lalala", "1310860": "Congratulations... Can you please share your notebook", "959584": "Congrats!", "946694": "Congratulations.", "946502": "Very interesting approach with removing the stride in the first conv layer and the max-pooling layer. Hope that this can help me in real tasks. How much score boost it gives? And how much gives fold's ensembling?", "944694": "Congrats!", "944382": "Congrats!", "944367": "Nice work！", "944128": "Congratulations.", "944086": "Congrats on first place! ", "943528": "Well done.Nice solution", "942629": "Congrats\n", "942321": "Congrats", "942281": "Thanks @wowfattie for your outstanding work and share with us", "941475": "Wow, that's pretty nice", "940815": "congrats", "940810": "Congratulations, very successful\n", "940055": "This is quite a nice solution! Cheers!", "939995": "Congrats and thanks for sharing the info. Its always helpful to learn the methods for beginners like me", "939967": "congrats, always learn something new from your solutions", "939919": "this is great", "939881": "Nice! Very instructive for some of the things with image processing", "938820": "Wowww, very nice!!", "938685": "Congrats for your 1st place ! ", "938682": "Nice catch! Congratulations!", "938656": "Congratulations!", "938609": "Congrats @wowfattie \n\nthank you for sharing the solution steps!\n\nWhile , we all understand we should not use public LB for model selection.\nhow did you decide on when you think it starts to overfit? any suggestions and inputs! thanks!", "938517": "nice idea inspire my mind", "938487": "Congratulations.", "938340": "Congrats:D", "938296": "Nice job!", "938287": "@wowfattie thanks a lot for the great write up and congratulations! Is your solution approach have similarity in the principle of Harmonic Convolution block? https://paperswithcode.com/method/harmonic-block, https://github.com/matej-ulicny/harmonic-networks ", "938199": "@wowfattie \n\nthanks for the writeup. i am now implementing your methods. \n\nIt is mentioned: \"The spatial domain model was seresnet18 initialized with imagenet weights. To make training work I removed the stride in the first conv layer and the max-pooling layer\"\n\nwith stride and max pooling removed, it takes a long time to train. May i know how long does it takes to train a epoch and what is the batch size?", "938018": "Congrats mate", "937942": "Congrats @wowfattie and thank you for the write-up!", "937820": "Thanks for sharing! It is very nice work without large off-the-shelf models.\nCan I know how much did cutmix help?", "937814": "Nice job! Could you please share hardware details? ", "937660": "Congrats @wowfattie, glad you were inspired by our OneHotConv work! \nBTW your DCT model performs around 2% better in wAUC compared to the rich models we used (JRM/DCTR) which is also compatible with our findings in the paper you cited!\n", "937614": "Great Job!\nAnd congratulations 1st place! ", "937584": "A brilliant idea that combines the model based on the spatial domain and the model based on the DCT domain. I also tried to train the model based on the DCT domain(OneHotNet) and did not achieve good prediction.\n\nCongratulations!!!", "937552": "Thanks for sharing. And congratulations 1st rank for both this competition and all competitions. :)", "943540": "", "942067": "", "941360": "", "939620": "", "938906": "", "937803": "", "946542": "Thanks for sharing your solution.", "944906": "Thank you.", "938494": "Awesome, very informative. Thanks" } }