![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2006644%2F54f9121359e651430b61358453513f69%2Fdownload.png?generation=1591160254445015&alt=media\")
Here is some information of my model. It seems that JUNIWARD is the most difficult one to detect.
\n\nAUC of all: 0.9401968754566749\nAUC of JMiPOD: 0.955721745383109\nAUC of JUNIWARD: 0.8982780573071116\nAUC of UERD: 0.9664128910156301
\n\n
Is it the same as yours? Any suggestions would be appreciated.
", "messages": [ { "id": "872335", "postDate": "06/03/2020 05:15:52", "content": "Here is some information of my model. It seems that JUNIWARD is the most difficult one to detect.
\n\nAUC of all: 0.9401968754566749\nAUC of JMiPOD: 0.955721745383109\nAUC of JUNIWARD: 0.8982780573071116\nAUC of UERD: 0.9664128910156301
\n\n
Is it the same as yours? Any suggestions would be appreciated.
", "rawMarkdown": "Here is some information of my model. It seems that JUNIWARD is the most difficult one to detect. \n\nAUC of **all**: 0.9401968754566749\nAUC of **JMiPOD**: 0.955721745383109\nAUC of **JUNIWARD**: 0.8982780573071116\nAUC of **UERD**: 0.9664128910156301\n\n\n\nIs it the same as yours? Any suggestions would be appreciated.", "votes": null }, { "id": "872443", "postDate": "06/03/2020 07:36:10", "content": "Hello there,
\n\nThat is, from my point of view, very interesting and this is typically the kind of feedback I have been expecting :)\nIt is quite acknowledged in the \"steganography research\" community that J-UNIWARD is the current state of the art.\nHowever, those are usually evaluated over grayscale image. And J-MiPOD is an algorithm I have designed specifically for this challenge such that it is not (fully) publicly available yet. \nTherefore, for this new algorithm and for color images and with respect to current art in DL, we did not know exactly .... even though it is not very surprising that J-UNIWARD is the most secure.
\n\nDid you train your model your model for each stego algorithm (and then merging the results for testing using a multiclass approach) or did you train your model as a binary classifier blending all three algorithms ?
\n\nRémi
", "rawMarkdown": "Hello there,\n\nThat is, from my point of view, very interesting and this is typically the kind of feedback I have been expecting :)\nIt is quite acknowledged in the \"steganography research\" community that J-UNIWARD is the current state of the art.\nHowever, those are usually evaluated over grayscale image. And J-MiPOD is an algorithm I have designed specifically for this challenge such that it is not (fully) publicly available yet. \nTherefore, for this new algorithm and for color images and with respect to current art in DL, we did not know exactly .... even though it is not very surprising that J-UNIWARD is the most secure.\n\nDid you train your model your model for each stego algorithm (and then merging the results for testing using a multiclass approach) or did you train your model as a binary classifier blending all three algorithms ?\n\nRémi", "votes": null }, { "id": "872455", "postDate": "06/03/2020 07:45:29", "content": "To answer your question, it seems that with other models similar observations holds true : \nsee this post
\n\nThe analysis of the results for each JPEG quality factor is also interesting? In the above results did you merge them all ? \nDid you train a model for each JPEG quality factor ?
", "rawMarkdown": "To answer your question, it seems that with other models similar observations holds true : \n[see this post ](https://www.kaggle.com/c/alaska2-image-steganalysis/discussion/155821)\n\nThe analysis of the results for each JPEG quality factor is also interesting? In the above results did you merge them all ? \nDid you train a model for each JPEG quality factor ?", "votes": null }, { "id": "872492", "postDate": "06/03/2020 08:31:59", "content": "I didn't train model for each stego algorithm or each quality factor yet. The chart came from one multiclass model. \nThank you for your advice. I'll try it later.
", "rawMarkdown": "I didn't train model for each stego algorithm or each quality factor yet. The chart came from one multiclass model. \nThank you for your advice. I'll try it later.", "votes": null }, { "id": "872745", "postDate": "06/03/2020 13:21:30", "content": "wow, do you really have overall weighted AUC with one model 0.940???\nbut anyway, yes - I have made the same observation, here's my weigted AUC on my val. set: 978, 923, 970 (JMiPOD, JUNI, UERD)\nwhat is \"weird\" though, my overall AUC for this one is 917 (which somehow doesn't make sense ... I was expecting average of the algorithms, but I assumed it's actually ok because I couldn't see any error so far in my code)
", "rawMarkdown": "wow, do you really have overall weighted AUC with one model 0.940???\nbut anyway, yes - I have made the same observation, here's my weigted AUC on my val. set: 978, 923, 970 (JMiPOD, JUNI, UERD)\nwhat is \"weird\" though, my overall AUC for this one is 917 (which somehow doesn't make sense ... I was expecting average of the algorithms, but I assumed it's actually ok because I couldn't see any error so far in my code)", "votes": null }, { "id": "872806", "postDate": "06/03/2020 14:17:28", "content": "Yes, it's one fold of one model. (LB is only 0.933)\nIf you use multiple models for each stego algorithm, maybe it's difficult to merge it (I haven't tried it yet). Because the metric is AUC, the important thing is the order.
", "rawMarkdown": "Yes, it's one fold of one model. (LB is only 0.933)\nIf you use multiple models for each stego algorithm, maybe it's difficult to merge it (I haven't tried it yet). Because the metric is AUC, the important thing is the order.", "votes": null }, { "id": "872948", "postDate": "06/03/2020 16:12:41", "content": "in my case it's also just one model with multiclass, guess I have to check where's my \"error\"
\n\nedit: btw. how do u calculate the AUC per class?\nI take my full validation set, and consider all with the specific class as \"1\" and cover + 2 other classes as \"0\" and calculate the AUC over the whole set\nI guess that if you calculate the AUC only over the specific algorithm as \"1\" and cover as \"0\" (excluding the other classes from AUC calculation), you could have the different numbers ... have to try it
", "rawMarkdown": "in my case it's also just one model with multiclass, guess I have to check where's my \"error\"\n\nedit: btw. how do u calculate the AUC per class?\nI take my full validation set, and consider all with the specific class as \"1\" and cover + 2 other classes as \"0\" and calculate the AUC over the whole set\nI guess that if you calculate the AUC only over the specific algorithm as \"1\" and cover as \"0\" (excluding the other classes from AUC calculation), you could have the different numbers ... have to try it", "votes": null }, { "id": "873150", "postDate": "06/03/2020 21:10:55", "content": "I am having the same issue. Here is the confusion matrix for a model trained against all Q factors by stego approach. The most troubling of these is the confusion between Cover and the JUNIWARD stegos.
\n\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1083130%2F00f97e992cf38b765cfa80143249b36f%2FUntitled%208.jpeg?generation=1591218403562294&alt=media\")
The only paper I have seen that focused on this stego type uses grayscale images with SRNET. I didn't think that would work, but gave it a shot. I did not see good results.
\n\nI also have trained independent 4 class models by Jpeg Q factor, results were not good, but perhaps I have a problem in my pipeline for those. I am going to take a look.
", "rawMarkdown": "I am having the same issue. Here is the confusion matrix for a model trained against all Q factors by stego approach. The most troubling of these is the confusion between Cover and the JUNIWARD stegos. \n\n\n\nThe only paper I have seen that focused on this stego type uses grayscale images with SRNET. I didn't think that would work, but gave it a shot. I did not see good results.\n\nI also have trained independent 4 class models by Jpeg Q factor, results were not good, but perhaps I have a problem in my pipeline for those. I am going to take a look.", "votes": null }, { "id": "873226", "postDate": "06/03/2020 23:46:43", "content": "\n\n\nI guess that if you calculate the AUC only over the specific algorithm as \"1\" and cover as \"0\" (excluding the other classes from AUC calculation), you could have the different numbers … have to try it
\n
Yes, I calculate the AUC in this way.
", "rawMarkdown": "> I guess that if you calculate the AUC only over the specific algorithm as \"1\" and cover as \"0\" (excluding the other classes from AUC calculation), you could have the different numbers … have to try it\n\nYes, I calculate the AUC in this way.", "votes": null }, { "id": "873254", "postDate": "06/04/2020 01:04:04", "content": "Nice confusion matrix.\nMay I ask why didn't you separate cover into three Q factors? \nI also tried SRNET. The result was not so good too.
", "rawMarkdown": "Nice confusion matrix.\nMay I ask why didn't you separate cover into three Q factors? \nI also tried SRNET. The result was not so good too.", "votes": null }, { "id": "873405", "postDate": "06/04/2020 06:01:03", "content": "thanks
", "rawMarkdown": "thanks", "votes": null }, { "id": "873969", "postDate": "06/04/2020 14:46:30", "content": "for me this is not true:) for now uerd is the worst one) Overall Validation metric is around 0.9. Juniward is the best one, lol. But i have a different approach to this, not like the ones described until now. But i am surprised that the accuracy on the most difficult class is much better, than for others. I would love to share results bit later, after i plot everyting accurately and try my approach on lb.
", "rawMarkdown": "for me this is not true:) for now uerd is the worst one) Overall Validation metric is around 0.9. Juniward is the best one, lol. But i have a different approach to this, not like the ones described until now. But i am surprised that the accuracy on the most difficult class is much better, than for others. I would love to share results bit later, after i plot everyting accurately and try my approach on lb.", "votes": null }, { "id": "874272", "postDate": "06/04/2020 19:16:48", "content": "I did one set of experiments comparing 12 classes vs 10 classes. Holding all things equal other than the number of classes, 12 classes outperformed 10 on CV, but 10 classes outperformed 12 on LB. On 10 class models, my LB and CV are nearly identical. My hypothesis was the 12 cat model is really learning the QFactor not the stego. After these experiments, I abandoned the 12 class model.
\n\nThat said, I am now doing 3 separate models per Qfactor followed by ensemble. I am not done yet so I have no useful info to add on this topic as of yet.
\n\nPS - Congrats on your LB position.
", "rawMarkdown": "I did one set of experiments comparing 12 classes vs 10 classes. Holding all things equal other than the number of classes, 12 classes outperformed 10 on CV, but 10 classes outperformed 12 on LB. On 10 class models, my LB and CV are nearly identical. My hypothesis was the 12 cat model is really learning the QFactor not the stego. After these experiments, I abandoned the 12 class model. \n\nThat said, I am now doing 3 separate models per Qfactor followed by ensemble. I am not done yet so I have no useful info to add on this topic as of yet.\n\nPS - Congrats on your LB position.", "votes": null }, { "id": "874394", "postDate": "06/05/2020 00:01:04", "content": "Thank you. I just took part in this competition earlier. And encounter the bottleneck for now.
", "rawMarkdown": "Thank you. I just took part in this competition earlier. And encounter the bottleneck for now.", "votes": null }, { "id": "881087", "postDate": "06/10/2020 18:44:40", "content": "Great work!👍
", "rawMarkdown": "Great work!👍", "votes": null }, { "id": "881637", "postDate": "06/11/2020 08:17:25", "content": "Is J-MiPOD a variation of MiPOD (Minimizing the Power of Optimal Detector) as in 'Content-adaptive steganography by minimizing statistical detectability” ?
", "rawMarkdown": "Is J-MiPOD a variation of MiPOD (Minimizing the Power of Optimal Detector) as in 'Content-adaptive steganography by minimizing statistical detectability” ?", "votes": null }, { "id": "917422", "postDate": "07/06/2020 13:44:10", "content": "did you guys get the same results? what is the reason for the mysterious cluster in the middle???
\n\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F113660%2Ffe6b663ec796826bb7e0f5513c537f8e%2FSelection_202.png?generation=1594043048368894&alt=media\")
I also observed this every time.
", "rawMarkdown": "I also observed this every time.", "votes": null }, { "id": "917601", "postDate": "07/06/2020 16:20:13", "content": "It's the point of maximum confusion of the model. It occurs most frequently when there are low # of bit changes between cover and stego, and represents ~equal prediction probability between the four classes which is why it shows up at 0.75 for 1-prob metric.\nMore interesting to me is the bimodal distribution of # of bits changed for each stego type (both including and excluding image quality). There's a clear break at the 50th percentile of each distribution. Maybe two embedding schemes? Haven't been able to exploit it yet.
", "rawMarkdown": "It's the point of maximum confusion of the model. It occurs most frequently when there are low # of bit changes between cover and stego, and represents ~equal prediction probability between the four classes which is why it shows up at 0.75 for 1-prob metric.\nMore interesting to me is the bimodal distribution of # of bits changed for each stego type (both including and excluding image quality). There's a clear break at the 50th percentile of each distribution. Maybe two embedding schemes? Haven't been able to exploit it yet.", "votes": null }, { "id": "917748", "postDate": "07/06/2020 18:20:33", "content": "\n\nyou are right! ![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F113660%2Fa611ec74ee6526ce14b6b72accaef258%2FSelection_208.png?generation=1594059628569785&alt=media\")
\n see more plots:
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F113660%2Fae0fcf77e4875ad35d6e3837da327064%2FSelection_209.png?generation=1594059730205427&alt=media\")
Hello there,
\n\nThat is, from my point of view, very interesting and this is typically the kind of feedback I have been expecting :)\nIt is quite acknowledged in the \"steganography research\" community that J-UNIWARD is the current state of the art.\nHowever, those are usually evaluated over grayscale image. And J-MiPOD is an algorithm I have designed specifically for this challenge such that it is not (fully) publicly available yet. \nTherefore, for this new algorithm and for color images and with respect to current art in DL, we did not know exactly .... even though it is not very surprising that J-UNIWARD is the most secure.
\n\nDid you train your model your model for each stego algorithm (and then merging the results for testing using a multiclass approach) or did you train your model as a binary classifier blending all three algorithms ?
\n\nRémi
", "votes": null, "replies": [ { "id": 881637, "author_name": "something4kag", "author_url": "", "post_date": "06/11/2020 08:17:25", "content": "Is J-MiPOD a variation of MiPOD (Minimizing the Power of Optimal Detector) as in 'Content-adaptive steganography by minimizing statistical detectability” ?
", "votes": null, "replies": [] } ] }, { "id": 872455, "author_name": "remicogranne", "author_url": "", "post_date": "06/03/2020 07:45:29", "content": "To answer your question, it seems that with other models similar observations holds true : \nsee this post
\n\nThe analysis of the results for each JPEG quality factor is also interesting? In the above results did you merge them all ? \nDid you train a model for each JPEG quality factor ?
", "votes": null, "replies": [ { "id": 872492, "author_name": "wuliaokaola", "author_url": "", "post_date": "06/03/2020 08:31:59", "content": "I didn't train model for each stego algorithm or each quality factor yet. The chart came from one multiclass model. \nThank you for your advice. I'll try it later.
", "votes": null, "replies": [] } ] }, { "id": 872745, "author_name": "steelrose", "author_url": "", "post_date": "06/03/2020 13:21:30", "content": "wow, do you really have overall weighted AUC with one model 0.940???\nbut anyway, yes - I have made the same observation, here's my weigted AUC on my val. set: 978, 923, 970 (JMiPOD, JUNI, UERD)\nwhat is \"weird\" though, my overall AUC for this one is 917 (which somehow doesn't make sense ... I was expecting average of the algorithms, but I assumed it's actually ok because I couldn't see any error so far in my code)
", "votes": null, "replies": [ { "id": 872806, "author_name": "wuliaokaola", "author_url": "", "post_date": "06/03/2020 14:17:28", "content": "Yes, it's one fold of one model. (LB is only 0.933)\nIf you use multiple models for each stego algorithm, maybe it's difficult to merge it (I haven't tried it yet). Because the metric is AUC, the important thing is the order.
", "votes": null, "replies": [] }, { "id": 872948, "author_name": "steelrose", "author_url": "", "post_date": "06/03/2020 16:12:41", "content": "in my case it's also just one model with multiclass, guess I have to check where's my \"error\"
\n\nedit: btw. how do u calculate the AUC per class?\nI take my full validation set, and consider all with the specific class as \"1\" and cover + 2 other classes as \"0\" and calculate the AUC over the whole set\nI guess that if you calculate the AUC only over the specific algorithm as \"1\" and cover as \"0\" (excluding the other classes from AUC calculation), you could have the different numbers ... have to try it
", "votes": null, "replies": [] }, { "id": 873226, "author_name": "wuliaokaola", "author_url": "", "post_date": "06/03/2020 23:46:43", "content": "\n\n\nI guess that if you calculate the AUC only over the specific algorithm as \"1\" and cover as \"0\" (excluding the other classes from AUC calculation), you could have the different numbers … have to try it
\n
Yes, I calculate the AUC in this way.
", "votes": null, "replies": [] }, { "id": 873405, "author_name": "steelrose", "author_url": "", "post_date": "06/04/2020 06:01:03", "content": "thanks
", "votes": null, "replies": [] } ] }, { "id": 873150, "author_name": "jbfarrar", "author_url": "", "post_date": "06/03/2020 21:10:55", "content": "I am having the same issue. Here is the confusion matrix for a model trained against all Q factors by stego approach. The most troubling of these is the confusion between Cover and the JUNIWARD stegos.
\n\n
The only paper I have seen that focused on this stego type uses grayscale images with SRNET. I didn't think that would work, but gave it a shot. I did not see good results.
\n\nI also have trained independent 4 class models by Jpeg Q factor, results were not good, but perhaps I have a problem in my pipeline for those. I am going to take a look.
", "votes": null, "replies": [ { "id": 873254, "author_name": "wuliaokaola", "author_url": "", "post_date": "06/04/2020 01:04:04", "content": "Nice confusion matrix.\nMay I ask why didn't you separate cover into three Q factors? \nI also tried SRNET. The result was not so good too.
", "votes": null, "replies": [] }, { "id": 874272, "author_name": "jbfarrar", "author_url": "", "post_date": "06/04/2020 19:16:48", "content": "I did one set of experiments comparing 12 classes vs 10 classes. Holding all things equal other than the number of classes, 12 classes outperformed 10 on CV, but 10 classes outperformed 12 on LB. On 10 class models, my LB and CV are nearly identical. My hypothesis was the 12 cat model is really learning the QFactor not the stego. After these experiments, I abandoned the 12 class model.
\n\nThat said, I am now doing 3 separate models per Qfactor followed by ensemble. I am not done yet so I have no useful info to add on this topic as of yet.
\n\nPS - Congrats on your LB position.
", "votes": null, "replies": [] }, { "id": 874394, "author_name": "wuliaokaola", "author_url": "", "post_date": "06/05/2020 00:01:04", "content": "Thank you. I just took part in this competition earlier. And encounter the bottleneck for now.
", "votes": null, "replies": [] } ] }, { "id": 873969, "author_name": "vovanf98", "author_url": "", "post_date": "06/04/2020 14:46:30", "content": "for me this is not true:) for now uerd is the worst one) Overall Validation metric is around 0.9. Juniward is the best one, lol. But i have a different approach to this, not like the ones described until now. But i am surprised that the accuracy on the most difficult class is much better, than for others. I would love to share results bit later, after i plot everyting accurately and try my approach on lb.
", "votes": null, "replies": [] }, { "id": 881087, "author_name": "pratikbarua", "author_url": "", "post_date": "06/10/2020 18:44:40", "content": "Great work!👍
", "votes": null, "replies": [] }, { "id": 917422, "author_name": "hengck23", "author_url": "", "post_date": "07/06/2020 13:44:10", "content": "did you guys get the same results? what is the reason for the mysterious cluster in the middle???
\n\n
I also observed this every time.
", "votes": null, "replies": [] }, { "id": 917601, "author_name": "tivfrvqhs5", "author_url": "", "post_date": "07/06/2020 16:20:13", "content": "It's the point of maximum confusion of the model. It occurs most frequently when there are low # of bit changes between cover and stego, and represents ~equal prediction probability between the four classes which is why it shows up at 0.75 for 1-prob metric.\nMore interesting to me is the bimodal distribution of # of bits changed for each stego type (both including and excluding image quality). There's a clear break at the 50th percentile of each distribution. Maybe two embedding schemes? Haven't been able to exploit it yet.
", "votes": null, "replies": [] }, { "id": 917748, "author_name": "hengck23", "author_url": "", "post_date": "07/06/2020 18:20:33", "content": "\n\nyou are right! \n see more plots: