I've two submissions from two different models, one of them scored 0.910 another 0.911. I've tried to blend these two but got worse results, ~80. I'm wondering if anyone is facing this? As the target images are sensitive, I'm wondering do we need to consider some specialty here.
However, the ensemble with the same architectural models sometimes gave a reasonable score but less. Also in my case, TTA improves a single model but worse in an ensemble.
\n\nSolved
", "messages": [ { "id": 934202, "postDate": "2020-07-18T09:59:53.220Z", "content": "\"I've two submissions from two different models, one of them scored 0.910 another 0.911. I've tried to blend these two but got worse results, ~80.\"
\n\nyou can break the problems into subproblems.
\n\nthe test image has equal number of qf75,90,95 images.\nit is mentioned that the type of steganography happens with equal probability.
\n\nwe already know that validation accuracy on different subgroups (qf, steganography) are different.\nyou can do experiments to see if TTA will improves different subgroups or not.
\n\nThen you can decide if you want to TTA for the test samples subgroups or not since you know the qf (and approximate steganography type) of the test samples
\n\nthere is one more information available to both test/train, but i haven't investigated yet: num of non-zero ac coefficients.
", "rawMarkdown": "\"I've two submissions from two different models, one of them scored 0.910 another 0.911. I've tried to blend these two but got worse results, ~80.\"\n\n\nyou can break the problems into subproblems.\n\nthe test image has equal number of qf75,90,95 images.\nit is mentioned that the type of steganography happens with equal probability.\n\nwe already know that validation accuracy on different subgroups (qf, steganography) are different.\nyou can do experiments to see if TTA will improves different subgroups or not.\n\nThen you can decide if you want to TTA for the test samples subgroups or not since you know the qf (and approximate steganography type) of the test samples \n\n\n\n---\n\nthere is one more information available to both test/train, but i haven't investigated yet: num of non-zero ac coefficients.\n", "votes": 1 }, { "id": 926210, "postDate": "2020-07-12T15:20:12.907Z", "content": "It is for sure a bug. While blending always make sure to join 2 submissions on ids.
", "rawMarkdown": "It is for sure a bug. While blending always make sure to join 2 submissions on ids.", "votes": 1, "replies": [ { "id": 926281, "postDate": "2020-07-12T16:08:22.817Z", "content": "Yes, thank you. I've checked the id though :(
", "rawMarkdown": "Yes, thank you. I've checked the id though :(" } ] }, { "id": 926361, "postDate": "2020-07-12T16:59:31.477Z", "content": "also, try rank averaging
", "rawMarkdown": "also, try rank averaging ", "votes": 2, "replies": [ { "id": 926647, "postDate": "2020-07-12T20:37:00.017Z", "content": "thanks for the tips :)
", "rawMarkdown": "thanks for the tips :)" }, { "id": 927910, "postDate": "2020-07-13T16:24:56.547Z", "content": "Could you explain a little more? :)\n@yifanxie
", "rawMarkdown": "Could you explain a little more? :)\n@yifanxie " }, { "id": 928047, "postDate": "2020-07-13T18:00:43.273Z", "content": "\n\n\nHeroseo wrote:
\n \nCould you explain a little more? :)\n @yifanxie
\n
For AUC related metric, only the rank of the probability matters - so you can convert probability predictions to ranking using functions like Panda's DataFrame.rank, and then merge the ranks of predictions from different models.
\n\nSome models tend to generate probability values of very different magnitude even within the same [0,1] interval, and as a result, there are situations in which the average prediction get dominated by probability output of larger magnitute. In such situations, doing rank averaging operation remove the side effect of merging by raw probability values
", "rawMarkdown": "> **Heroseo wrote:**\n> \n> Could you explain a little more? :)\n> @yifanxie \n\nFor AUC related metric, only the rank of the probability matters - so you can convert probability predictions to ranking using functions like Panda's [DataFrame.rank](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.rank.html), and then merge the ranks of predictions from different models.\n\nSome models tend to generate probability values of very different magnitude even within the same [0,1] interval, and as a result, there are situations in which the average prediction get dominated by probability output of larger magnitute. In such situations, doing rank averaging operation remove the side effect of merging by raw probability values \n", "votes": 4 }, { "id": 928105, "postDate": "2020-07-13T18:53:18.897Z", "content": "Thanks a lot! :)\n@yifanxie
", "rawMarkdown": "Thanks a lot! :)\n@yifanxie " } ] }, { "id": 926135, "postDate": "2020-07-12T14:19:43.283Z", "content": "I've two submissions from two different models, one of them scored 0.910 another 0.911. I've tried to blend these two but got worse results, ~80. I'm wondering if anyone is facing this? As the target images are sensitive, I'm wondering do we need to consider some specialty here.
However, the ensemble with the same architectural models sometimes gave a reasonable score but less. Also in my case, TTA improves a single model but worse in an ensemble.
\n\nSolved
", "rawMarkdown": "I've two submissions from two different models, one of them scored `0.910` another `0.911`. I've tried to blend these two but got worse results, ~80. I'm wondering if anyone is facing this? As the target images are sensitive, I'm wondering do we need to consider some specialty here. \n\nHowever, the ensemble with the same architectural models sometimes gave a reasonable score but less. Also in my case, TTA improves a single model but worse in an ensemble. \n\n### Update\n**Solved**", "votes": 2 }, { "id": 928045, "postDate": "2020-07-13T17:56:48.140Z", "content": "Just you must reset the index and make the ids sorted then you can do your blending easily !
", "rawMarkdown": "Just you must reset the index and make the ids sorted then you can do your blending easily !", "replies": [ { "id": 932127, "postDate": "2020-07-16T18:17:39.773Z", "content": "Yes, I'm aware of it. Thank you.
", "rawMarkdown": "Yes, I'm aware of it. Thank you." } ] }, { "id": 935264, "postDate": "2020-07-19T07:58:41.587Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 934202, "author_name": "hengck23", "author_url": "", "post_date": "2020-07-18T09:59:53.220000", "content": "\"I've two submissions from two different models, one of them scored 0.910 another 0.911. I've tried to blend these two but got worse results, ~80.\"
\n\nyou can break the problems into subproblems.
\n\nthe test image has equal number of qf75,90,95 images.\nit is mentioned that the type of steganography happens with equal probability.
\n\nwe already know that validation accuracy on different subgroups (qf, steganography) are different.\nyou can do experiments to see if TTA will improves different subgroups or not.
\n\nThen you can decide if you want to TTA for the test samples subgroups or not since you know the qf (and approximate steganography type) of the test samples
\n\nthere is one more information available to both test/train, but i haven't investigated yet: num of non-zero ac coefficients.
", "votes": 1, "replies": [] }, { "id": 926210, "author_name": "Ahmet Erdem", "author_url": "", "post_date": "2020-07-12T15:20:12.907000", "content": "It is for sure a bug. While blending always make sure to join 2 submissions on ids.
", "votes": 1, "replies": [ { "id": 926281, "author_name": "Innat", "author_url": "", "post_date": "2020-07-12T16:08:22.817000", "content": "Yes, thank you. I've checked the id though :(
", "votes": 0, "replies": [] } ] }, { "id": 926361, "author_name": "Yifan Xie", "author_url": "", "post_date": "2020-07-12T16:59:31.477000", "content": "also, try rank averaging
", "votes": 2, "replies": [ { "id": 926647, "author_name": "Innat", "author_url": "", "post_date": "2020-07-12T20:37:00.017000", "content": "thanks for the tips :)
", "votes": 0, "replies": [] }, { "id": 927910, "author_name": "Heroseo", "author_url": "", "post_date": "2020-07-13T16:24:56.547000", "content": "Could you explain a little more? :)\n@yifanxie
", "votes": 0, "replies": [] }, { "id": 928047, "author_name": "Yifan Xie", "author_url": "", "post_date": "2020-07-13T18:00:43.273000", "content": "\n\n\nHeroseo wrote:
\n \nCould you explain a little more? :)\n @yifanxie
\n
For AUC related metric, only the rank of the probability matters - so you can convert probability predictions to ranking using functions like Panda's DataFrame.rank, and then merge the ranks of predictions from different models.
\n\nSome models tend to generate probability values of very different magnitude even within the same [0,1] interval, and as a result, there are situations in which the average prediction get dominated by probability output of larger magnitute. In such situations, doing rank averaging operation remove the side effect of merging by raw probability values
", "votes": 4, "replies": [] }, { "id": 928105, "author_name": "Heroseo", "author_url": "", "post_date": "2020-07-13T18:53:18.897000", "content": "Thanks a lot! :)\n@yifanxie
", "votes": 0, "replies": [] } ] }, { "id": 928045, "author_name": "Ashraf Mahdhi", "author_url": "", "post_date": "2020-07-13T17:56:48.140000", "content": "Just you must reset the index and make the ids sorted then you can do your blending easily !
", "votes": 0, "replies": [ { "id": 932127, "author_name": "Innat", "author_url": "", "post_date": "2020-07-16T18:17:39.773000", "content": "Yes, I'm aware of it. Thank you.
", "votes": 0, "replies": [] } ] }, { "id": 935264, "author_name": "", "author_url": "", "post_date": "2020-07-19T07:58:41.587000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "934202": "\"I've two submissions from two different models, one of them scored 0.910 another 0.911. I've tried to blend these two but got worse results, ~80.\"\n\n\nyou can break the problems into subproblems.\n\nthe test image has equal number of qf75,90,95 images.\nit is mentioned that the type of steganography happens with equal probability.\n\nwe already know that validation accuracy on different subgroups (qf, steganography) are different.\nyou can do experiments to see if TTA will improves different subgroups or not.\n\nThen you can decide if you want to TTA for the test samples subgroups or not since you know the qf (and approximate steganography type) of the test samples \n\n\n\n---\n\nthere is one more information available to both test/train, but i haven't investigated yet: num of non-zero ac coefficients.\n", "926210": "It is for sure a bug. While blending always make sure to join 2 submissions on ids.", "926361": "also, try rank averaging ", "926135": "I've two submissions from two different models, one of them scored `0.910` another `0.911`. I've tried to blend these two but got worse results, ~80. I'm wondering if anyone is facing this? As the target images are sensitive, I'm wondering do we need to consider some specialty here. \n\nHowever, the ensemble with the same architectural models sometimes gave a reasonable score but less. Also in my case, TTA improves a single model but worse in an ensemble. \n\n### Update\n**Solved**", "928045": "Just you must reset the index and make the ids sorted then you can do your blending easily !", "935264": "" } }