1) I used an ensemble of 5 se_resnet50 models. Each of these models were trained exactly the same, just on different subsets of training examples. More complicated ensembles (with a bigger LB score) performed worse, so I guess they were overfitting to the LB.
\n\n2) Splitting by WSI helped, judging by submissions before splitting with WSI and after.
\n\n3) More intensive TTA helped. I used 16-TTA, which performed better than 4-TTA.
\n\n4) Obviously resizing to 196x196 helped, but I'm not sure that 196x196 is the best size.
\n\n5) I also used ReduceLROnPlateau (2 epocs), but have no idea whether it helped or not
\n\nP.S. In the end I selected 2 submissions: one crazy weighted ensemble with 0.9841 Public LB and one simple ensemble described above with 0.9798 Public LB. Had a feeling that the crazy ensemble really overfits the Public LB. Turns out, I was right: they have 0.9788 and 0.9810 Private LB respectively
\n\nP.S.S. Screw those who used the 1.0 submission for Private LB. What is wrong with you?
", "messages": [ { "id": "504245", "postDate": "03/31/2019 08:31:53", "content": "1) I used an ensemble of 5 se_resnet50 models. Each of these models were trained exactly the same, just on different subsets of training examples. More complicated ensembles (with a bigger LB score) performed worse, so I guess they were overfitting to the LB.
\n\n2) Splitting by WSI helped, judging by submissions before splitting with WSI and after.
\n\n3) More intensive TTA helped. I used 16-TTA, which performed better than 4-TTA.
\n\n4) Obviously resizing to 196x196 helped, but I'm not sure that 196x196 is the best size.
\n\n5) I also used ReduceLROnPlateau (2 epocs), but have no idea whether it helped or not
\n\nP.S. In the end I selected 2 submissions: one crazy weighted ensemble with 0.9841 Public LB and one simple ensemble described above with 0.9798 Public LB. Had a feeling that the crazy ensemble really overfits the Public LB. Turns out, I was right: they have 0.9788 and 0.9810 Private LB respectively
\n\nP.S.S. Screw those who used the 1.0 submission for Private LB. What is wrong with you?
", "rawMarkdown": "1) I used an ensemble of 5 se_resnet50 models. Each of these models were trained exactly the same, just on different subsets of training examples. More complicated ensembles (with a bigger LB score) performed worse, so I guess they were overfitting to the LB.\n\n2) Splitting by WSI helped, judging by submissions before splitting with WSI and after. \n\n3) More intensive TTA helped. I used 16-TTA, which performed better than 4-TTA.\n\n4) Obviously resizing to 196x196 helped, but I'm not sure that 196x196 is the best size. \n\n5) I also used ReduceLROnPlateau (2 epocs), but have no idea whether it helped or not\n\nP.S. In the end I selected 2 submissions: one crazy weighted ensemble with 0.9841 Public LB and one simple ensemble described above with 0.9798 Public LB. Had a feeling that the crazy ensemble really overfits the Public LB. Turns out, I was right: they have 0.9788 and 0.9810 Private LB respectively \n\nP.S.S. Screw those who used the 1.0 submission for Private LB. What is wrong with you?", "votes": null }, { "id": "504274", "postDate": "03/31/2019 10:03:13", "content": "Great job!
", "rawMarkdown": "Great job!", "votes": null }, { "id": "504314", "postDate": "03/31/2019 11:39:50", "content": "How much gives resizing to LB?
", "rawMarkdown": "How much gives resizing to LB?", "votes": null }, { "id": "504318", "postDate": "03/31/2019 11:48:27", "content": "That depends on what sizes we're talking about. If we compare 32x32 and 196x196 then the difference in LB score is just huge. But the difference with 196x196 and 224x224 is not that significant
", "rawMarkdown": "That depends on what sizes we're talking about. If we compare 32x32 and 196x196 then the difference in LB score is just huge. But the difference with 196x196 and 224x224 is not that significant", "votes": null }, { "id": "504355", "postDate": "03/31/2019 13:17:40", "content": "Nice work! You are actually really lucky. All my Public LB weighted ensemble submissions turned out to be less than 0.975. (although I did not use them)
", "rawMarkdown": "Nice work! You are actually really lucky. All my Public LB weighted ensemble submissions turned out to be less than 0.975. (although I did not use them)", "votes": null }, { "id": "504366", "postDate": "03/31/2019 13:39:22", "content": "Thanks! Well, if I based weights on Public LB score, then it worked poorly, since there was a clear overfitting to the Public LB. Just using the average (equal weights) worked better.
", "rawMarkdown": "Thanks! Well, if I based weights on Public LB score, then it worked poorly, since there was a clear overfitting to the Public LB. Just using the average (equal weights) worked better.", "votes": null }, { "id": "515216", "postDate": "04/12/2019 10:31:18", "content": "Which optimizer did u use?\nDid u use One cycle policy?
", "rawMarkdown": "Which optimizer did u use?\nDid u use One cycle policy?", "votes": null }, { "id": "515242", "postDate": "04/12/2019 10:59:23", "content": "I usually use Adam. So I used it here as well.
\n\nNo, during this competition a didn't us cycle learning, in particular, one cycle policy. But it would have been definitely a good idea to try.
", "rawMarkdown": "I usually use Adam. So I used it here as well. \n\nNo, during this competition a didn't us cycle learning, in particular, one cycle policy. But it would have been definitely a good idea to try.", "votes": null }, { "id": "515377", "postDate": "04/12/2019 14:17:35", "content": "Did u use 3e-4 as lr?...lol!!
", "rawMarkdown": "Did u use 3e-4 as lr?...lol!!", "votes": null }, { "id": "515386", "postDate": "04/12/2019 14:36:27", "content": "I don't remember the exact learning rate, but yeah, it was fixed. But you should also keep in mind that Adam uses a unique learning rate for every parameter and it adjusts them accordingly. So it's still powerful.
", "rawMarkdown": "I don't remember the exact learning rate, but yeah, it was fixed. But you should also keep in mind that Adam uses a unique learning rate for every parameter and it adjusts them accordingly. So it's still powerful.", "votes": null } ], "comments": [ { "id": 504274, "author_name": "robotdreams", "author_url": "", "post_date": "03/31/2019 10:03:13", "content": "Great job!
", "votes": null, "replies": [] }, { "id": 504314, "author_name": "ajalnine", "author_url": "", "post_date": "03/31/2019 11:39:50", "content": "How much gives resizing to LB?
", "votes": null, "replies": [ { "id": 504318, "author_name": "ivanpan", "author_url": "", "post_date": "03/31/2019 11:48:27", "content": "That depends on what sizes we're talking about. If we compare 32x32 and 196x196 then the difference in LB score is just huge. But the difference with 196x196 and 224x224 is not that significant
", "votes": null, "replies": [] } ] }, { "id": 504355, "author_name": "kokecacao", "author_url": "", "post_date": "03/31/2019 13:17:40", "content": "Nice work! You are actually really lucky. All my Public LB weighted ensemble submissions turned out to be less than 0.975. (although I did not use them)
", "votes": null, "replies": [ { "id": 504366, "author_name": "ivanpan", "author_url": "", "post_date": "03/31/2019 13:39:22", "content": "Thanks! Well, if I based weights on Public LB score, then it worked poorly, since there was a clear overfitting to the Public LB. Just using the average (equal weights) worked better.
", "votes": null, "replies": [] } ] }, { "id": 515216, "author_name": "aaryapatel98", "author_url": "", "post_date": "04/12/2019 10:31:18", "content": "Which optimizer did u use?\nDid u use One cycle policy?
", "votes": null, "replies": [ { "id": 515242, "author_name": "ivanpan", "author_url": "", "post_date": "04/12/2019 10:59:23", "content": "I usually use Adam. So I used it here as well.
\n\nNo, during this competition a didn't us cycle learning, in particular, one cycle policy. But it would have been definitely a good idea to try.
", "votes": null, "replies": [] }, { "id": 515377, "author_name": "aaryapatel98", "author_url": "", "post_date": "04/12/2019 14:17:35", "content": "Did u use 3e-4 as lr?...lol!!
", "votes": null, "replies": [] }, { "id": 515386, "author_name": "ivanpan", "author_url": "", "post_date": "04/12/2019 14:36:27", "content": "I don't remember the exact learning rate, but yeah, it was fixed. But you should also keep in mind that Adam uses a unique learning rate for every parameter and it adjusts them accordingly. So it's still powerful.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "504245": "1) I used an ensemble of 5 se_resnet50 models. Each of these models were trained exactly the same, just on different subsets of training examples. More complicated ensembles (with a bigger LB score) performed worse, so I guess they were overfitting to the LB.\n\n2) Splitting by WSI helped, judging by submissions before splitting with WSI and after. \n\n3) More intensive TTA helped. I used 16-TTA, which performed better than 4-TTA.\n\n4) Obviously resizing to 196x196 helped, but I'm not sure that 196x196 is the best size. \n\n5) I also used ReduceLROnPlateau (2 epocs), but have no idea whether it helped or not\n\nP.S. In the end I selected 2 submissions: one crazy weighted ensemble with 0.9841 Public LB and one simple ensemble described above with 0.9798 Public LB. Had a feeling that the crazy ensemble really overfits the Public LB. Turns out, I was right: they have 0.9788 and 0.9810 Private LB respectively \n\nP.S.S. Screw those who used the 1.0 submission for Private LB. What is wrong with you?", "504274": "Great job!", "504314": "How much gives resizing to LB?", "504318": "That depends on what sizes we're talking about. If we compare 32x32 and 196x196 then the difference in LB score is just huge. But the difference with 196x196 and 224x224 is not that significant", "504355": "Nice work! You are actually really lucky. All my Public LB weighted ensemble submissions turned out to be less than 0.975. (although I did not use them)", "504366": "Thanks! Well, if I based weights on Public LB score, then it worked poorly, since there was a clear overfitting to the Public LB. Just using the average (equal weights) worked better.", "515216": "Which optimizer did u use?\nDid u use One cycle policy?", "515242": "I usually use Adam. So I used it here as well. \n\nNo, during this competition a didn't us cycle learning, in particular, one cycle policy. But it would have been definitely a good idea to try.", "515377": "Did u use 3e-4 as lr?...lol!!", "515386": "I don't remember the exact learning rate, but yeah, it was fixed. But you should also keep in mind that Adam uses a unique learning rate for every parameter and it adjusts them accordingly. So it's still powerful." }, "source": "meta" }