In this competence, I tried:\n1. Nvidia apex.\nI have been training different imagenet models with apex for a long time. And I and noticed only a minor drop in accuracy. So, I trained with O2 opt level even at 1080Ti. The speed did not boost, but the batch was higher, and therefore the BN statistics are more pleasant.\nIn readme of apex repo one promised that you need to add only 3 lines of code. In reality, of course, it is not. But if just apex converts the code into spaghetti, then adding distributed compatibility means throwing an additional pack of noodles.\nAt the same time, I haven't figured out how to scale the LR when training for 4-8 GPUs distributed. Fine tuning of scheduling goes away and SyncBN does not help.
\n\nFocalLoss\nweighted by crop and tag. Tags assigned more weight than culture.
Mixup. \nI used implementation where the batch is mixed with its own shuffle version. As a result, at first it seemed that the nets were less overfitted, but this was due to the fact that the metric on the train was wrong. When I added additional validation train part without augmentations and compared it, it turned out that the nets just learn longer.
Batch accumulation. \nWith accum = 20.
Fine tuning of scheduling. \nNets overfitted a lot in this competition. So I tried to minimize the number of passes through the dataset. During grid search of right LR for different stages, it turned out that this is about how to raise the initial LR on one step and put ReduceLrOnPlatoe with Patience 0.
All experiments were performed using the se-resnext50. I still consider it as the best architecture in speed-accuracy trade-off. Most accurate was se-resnext101 and pnasnet-large. SENet154 I didn’t learn well for a long time with the explosion of gradients, and as a result, it didn’t perform very well. effnet-b3, se-densenet161, densenet161, didn’t work.
Label smoothing. \nTried with fixed epsilon and when it is selected individually by class depending on the frequency of occurrence of this dataset. As a result, the optimum threshold was slightly different. But these models helped in the ensemble.
Cleaning of data helps a lot. I've done: Clean up less than 20 labels and low per class AUC. Add labels by the confidence of OOF predictions. Clean up bad markup by confidence.
Augmentations, I also tried a lot of options. \nMostly from the article about tricks on Imagenet. CutOut did not boost but did not interfere. Large rotations and scale worsened score. As a result, I pad up to 320 along the narrow side with BorderReplicate, because it seemed to me that there were exhibits with a frame and I wanted the net to learn this pattern. Cut long sausages-style image and resize to net input. I end with:\n`
\n\nalb.PadIfNeeded(scale_size, scale_size, border_mode=cv2.BORDER_REPLICATE),\nalb.ShiftScaleRotate(\n shift_limit=0.1,\n scale_limit=(-0.0, 0.0),\n rotate_limit=5,\n p=0.5,\n interpolation=cv2.INTER_LINEAR,\n border_mode=cv2.BORDER_REPLICATE,\n),\nalb.Cutout(p=0.2),\nPowerFistRandomSizedCrop(\n min_max_height=min_max_height,\n height=scale_size,\n width=scale_size,\n w2h_ratio=w2h_ratio,\n),\nalb.HorizontalFlip(p=0.5),\nalb.RandomBrightnessContrast(p=0.3),\nalb.RandomGamma(gamma_limit=(95, 105), p=0.3),\npost_transform(),\n`
", "messages": [ { "id": 549666, "postDate": "2019-06-10T23:06:52.117Z", "content": "Don’t you want to say sth about your public LB?
\n\nIMHO, you should prove your label smoothing and data cleaning don’t use your external data.
", "rawMarkdown": "Don’t you want to say sth about your public LB? \n\nIMHO, you should prove your label smoothing and data cleaning don’t use your external data. ", "votes": 22 }, { "id": 549682, "postDate": "2019-06-10T23:50:51.627Z", "content": "The PB story must be explained, it's easy to get such score just tuning your model with the external data (no need to train on it ).
", "rawMarkdown": "The PB story must be explained, it's easy to get such score just tuning your model with the external data (no need to train on it ). ", "votes": 19 }, { "id": 549720, "postDate": "2019-06-11T00:56:57.797Z", "content": "Please explain about your Public LB Score jump.
", "rawMarkdown": "Please explain about your Public LB Score jump.", "votes": 20 }, { "id": 549729, "postDate": "2019-06-11T01:23:50.820Z", "content": "Crawler grandmaster~ Never mind the scandal and libel!
", "rawMarkdown": "Crawler grandmaster~ Never mind the scandal and libel!", "votes": 18 }, { "id": 552404, "postDate": "2019-06-13T22:25:54.667Z", "content": "That celebration!\nhttps://www.linkedin.com/feed/update/urn:li:activity:6545036462444818434
", "rawMarkdown": "That celebration!\nhttps://www.linkedin.com/feed/update/urn:li:activity:6545036462444818434", "votes": 6 }, { "id": 549637, "postDate": "2019-06-10T22:25:13.787Z", "content": "Thanks! Could you please share your code too?
", "rawMarkdown": "Thanks! Could you please share your code too?", "votes": 5, "replies": [ { "id": 549662, "postDate": "2019-06-10T23:02:07.930Z", "content": "It is very messy. I will try to find time to clean and share.
", "rawMarkdown": "It is very messy. I will try to find time to clean and share.", "votes": -23 }, { "id": 549804, "postDate": "2019-06-11T03:54:18.050Z", "content": "If you want to clear doubts, I think it would be better to share your code ASAP, even if it is messy ...
", "rawMarkdown": "If you want to clear doubts, I think it would be better to share your code ASAP, even if it is messy ...", "votes": 1 } ] }, { "id": 549792, "postDate": "2019-06-11T03:31:54.390Z", "content": "Why you don't explain Public LB jump? I'm so frustrated to yours, top 5 public LB team. Why? Can you speak English? If not, you should explain it in Russian.
", "rawMarkdown": "Why you don't explain Public LB jump? I'm so frustrated to yours, top 5 public LB team. Why? Can you speak English? If not, you should explain it in Russian.", "replies": [ { "id": 549839, "postDate": "2019-06-11T04:44:28.630Z", "rawMarkdown": "", "votes": -7, "isDeleted": true }, { "id": 549994, "postDate": "2019-06-11T07:50:00.077Z", "content": "This is actually so toxic, no matter of were rules violated or not.
", "rawMarkdown": "This is actually so toxic, no matter of were rules violated or not. ", "votes": 21 }, { "id": 550062, "postDate": "2019-06-11T09:08:04.257Z", "content": "it seems somebody need sedative
", "rawMarkdown": "it seems somebody need sedative", "votes": 3 }, { "id": 550114, "postDate": "2019-06-11T10:06:49.313Z", "content": "I was so emotional. Sorry.
", "rawMarkdown": "I was so emotional. Sorry." } ] }, { "id": 550093, "postDate": "2019-06-11T09:47:56.683Z", "content": "All doubts can be dispelled if admins ( @wcukierski ) confirm that the selected submissions for stage2 coincide with those that gave the top5 result on the public LB.
", "rawMarkdown": "All doubts can be dispelled if admins ( @wcukierski ) confirm that the selected submissions for stage2 coincide with those that gave the top5 result on the public LB.", "votes": 3 }, { "id": 549639, "postDate": "2019-06-10T22:29:29.810Z", "content": "Hey, thanks for sharing! Wondering if you have plan to go to CVPR?
", "rawMarkdown": "Hey, thanks for sharing! Wondering if you have plan to go to CVPR?", "votes": 1, "replies": [ { "id": 549647, "postDate": "2019-06-10T22:40:05.750Z", "content": "With this generic approach people can go to CVPR?
", "rawMarkdown": "With this generic approach people can go to CVPR? ", "votes": 10 }, { "id": 549657, "postDate": "2019-06-10T22:57:02.617Z", "content": "They can tell their LB top 5 story, that would be much more interesting than this version.
", "rawMarkdown": "They can tell their LB top 5 story, that would be much more interesting than this version. ", "votes": 9 }, { "id": 549665, "postDate": "2019-06-10T23:06:03.330Z", "content": "I do not have a visa. And I hardly get it quickly. Can I delegate my speech to another person? We did not work together on this particular task, but he can perform very well. This is Vladimir Iglovikov @iglovikov .
", "rawMarkdown": "I do not have a visa. And I hardly get it quickly. Can I delegate my speech to another person? We did not work together on this particular task, but he can perform very well. This is Vladimir Iglovikov @iglovikov .", "votes": -18 }, { "id": 549997, "postDate": "2019-06-11T07:51:27.903Z", "content": "Что ты мне сделаешь, я в другом городе
", "rawMarkdown": "Что ты мне сделаешь, я в другом городе " } ] }, { "id": 549892, "postDate": "2019-06-11T05:51:36.200Z", "content": "Why you ignore us? What are you afraid of?
", "rawMarkdown": "Why you ignore us? What are you afraid of?", "votes": 1 }, { "id": 549630, "postDate": "2019-06-10T22:22:09.310Z", "content": "In this competence, I tried:\n1. Nvidia apex.\nI have been training different imagenet models with apex for a long time. And I and noticed only a minor drop in accuracy. So, I trained with O2 opt level even at 1080Ti. The speed did not boost, but the batch was higher, and therefore the BN statistics are more pleasant.\nIn readme of apex repo one promised that you need to add only 3 lines of code. In reality, of course, it is not. But if just apex converts the code into spaghetti, then adding distributed compatibility means throwing an additional pack of noodles.\nAt the same time, I haven't figured out how to scale the LR when training for 4-8 GPUs distributed. Fine tuning of scheduling goes away and SyncBN does not help.
\n\nFocalLoss\nweighted by crop and tag. Tags assigned more weight than culture.
Mixup. \nI used implementation where the batch is mixed with its own shuffle version. As a result, at first it seemed that the nets were less overfitted, but this was due to the fact that the metric on the train was wrong. When I added additional validation train part without augmentations and compared it, it turned out that the nets just learn longer.
Batch accumulation. \nWith accum = 20.
Fine tuning of scheduling. \nNets overfitted a lot in this competition. So I tried to minimize the number of passes through the dataset. During grid search of right LR for different stages, it turned out that this is about how to raise the initial LR on one step and put ReduceLrOnPlatoe with Patience 0.
All experiments were performed using the se-resnext50. I still consider it as the best architecture in speed-accuracy trade-off. Most accurate was se-resnext101 and pnasnet-large. SENet154 I didn’t learn well for a long time with the explosion of gradients, and as a result, it didn’t perform very well. effnet-b3, se-densenet161, densenet161, didn’t work.
Label smoothing. \nTried with fixed epsilon and when it is selected individually by class depending on the frequency of occurrence of this dataset. As a result, the optimum threshold was slightly different. But these models helped in the ensemble.
Cleaning of data helps a lot. I've done: Clean up less than 20 labels and low per class AUC. Add labels by the confidence of OOF predictions. Clean up bad markup by confidence.
Augmentations, I also tried a lot of options. \nMostly from the article about tricks on Imagenet. CutOut did not boost but did not interfere. Large rotations and scale worsened score. As a result, I pad up to 320 along the narrow side with BorderReplicate, because it seemed to me that there were exhibits with a frame and I wanted the net to learn this pattern. Cut long sausages-style image and resize to net input. I end with:\n`
\n\nalb.PadIfNeeded(scale_size, scale_size, border_mode=cv2.BORDER_REPLICATE),\nalb.ShiftScaleRotate(\n shift_limit=0.1,\n scale_limit=(-0.0, 0.0),\n rotate_limit=5,\n p=0.5,\n interpolation=cv2.INTER_LINEAR,\n border_mode=cv2.BORDER_REPLICATE,\n),\nalb.Cutout(p=0.2),\nPowerFistRandomSizedCrop(\n min_max_height=min_max_height,\n height=scale_size,\n width=scale_size,\n w2h_ratio=w2h_ratio,\n),\nalb.HorizontalFlip(p=0.5),\nalb.RandomBrightnessContrast(p=0.3),\nalb.RandomGamma(gamma_limit=(95, 105), p=0.3),\npost_transform(),\n`
", "rawMarkdown": "In this competence, I tried:\n1. Nvidia apex.\nI have been training different imagenet models with apex for a long time. And I and noticed only a minor drop in accuracy. So, I trained with O2 opt level even at 1080Ti. The speed did not boost, but the batch was higher, and therefore the BN statistics are more pleasant.\nIn readme of apex repo one promised that you need to add only 3 lines of code. In reality, of course, it is not. But if just apex converts the code into spaghetti, then adding distributed compatibility means throwing an additional pack of noodles.\nAt the same time, I haven't figured out how to scale the LR when training for 4-8 GPUs distributed. Fine tuning of scheduling goes away and SyncBN does not help.\n\n2. FocalLoss\nweighted by crop and tag. Tags assigned more weight than culture. \n\n3. Mixup. \nI used implementation where the batch is mixed with its own shuffle version. As a result, at first it seemed that the nets were less overfitted, but this was due to the fact that the metric on the train was wrong. When I added additional validation train part without augmentations and compared it, it turned out that the nets just learn longer.\n\n4. Batch accumulation. \nWith accum = 20.\n\n5. Fine tuning of scheduling. \nNets overfitted a lot in this competition. So I tried to minimize the number of passes through the dataset. During grid search of right LR for different stages, it turned out that this is about how to raise the initial LR on one step and put ReduceLrOnPlatoe with Patience 0.\n\n\n7. All experiments were performed using the se-resnext50. I still consider it as the best architecture in speed-accuracy trade-off. Most accurate was se-resnext101 and pnasnet-large. SENet154 I didn’t learn well for a long time with the explosion of gradients, and as a result, it didn’t perform very well. effnet-b3, se-densenet161, densenet161, didn’t work.\n\n8. Label smoothing. \nTried with fixed epsilon and when it is selected individually by class depending on the frequency of occurrence of this dataset. As a result, the optimum threshold was slightly different. But these models helped in the ensemble.\n\n9. Cleaning of data helps a lot. I've done: Clean up less than 20 labels and low per class AUC. Add labels by the confidence of OOF predictions. Clean up bad markup by confidence. \n\n6. Augmentations, I also tried a lot of options. \nMostly from the article about tricks on Imagenet. CutOut did not boost but did not interfere. Large rotations and scale worsened score. As a result, I pad up to 320 along the narrow side with BorderReplicate, because it seemed to me that there were exhibits with a frame and I wanted the net to learn this pattern. Cut long sausages-style image and resize to net input. I end with:\n`\n\n\n alb.PadIfNeeded(scale_size, scale_size, border_mode=cv2.BORDER_REPLICATE),\n alb.ShiftScaleRotate(\n shift_limit=0.1,\n scale_limit=(-0.0, 0.0),\n rotate_limit=5,\n p=0.5,\n interpolation=cv2.INTER_LINEAR,\n border_mode=cv2.BORDER_REPLICATE,\n ),\n alb.Cutout(p=0.2),\n PowerFistRandomSizedCrop(\n min_max_height=min_max_height,\n height=scale_size,\n width=scale_size,\n w2h_ratio=w2h_ratio,\n ),\n alb.HorizontalFlip(p=0.5),\n alb.RandomBrightnessContrast(p=0.3),\n alb.RandomGamma(gamma_limit=(95, 105), p=0.3),\n post_transform(),\n\n`", "votes": -51 }, { "id": 549911, "postDate": "2019-06-11T06:17:31.290Z", "content": "I found that many people think you violated the rules and used external data sets. Could you clarify that?
", "rawMarkdown": "I found that many people think you violated the rules and used external data sets. Could you clarify that?", "votes": 2 }, { "id": 549825, "postDate": "2019-06-11T04:13:33.180Z", "content": "Many thanks for your shared solutions. Is it possible for you to share source code solutions for us to recreate this? Congratulations! :-)
", "rawMarkdown": "Many thanks for your shared solutions. Is it possible for you to share source code solutions for us to recreate this? Congratulations! :-)" }, { "id": 550031, "postDate": "2019-06-11T08:32:47.870Z", "content": "All your comments look very emotional and not very constructive. I do not want to waste time on this kind of discussion.\nAll I can say about public results: seems that I overfitted. I have done this many times and not always successfully. TGS Salt: 11th (in Gold) Public -> 40th Private; Airbus: 2nd (!) Public -> 36th Private.\nThis time, more fortunate. And in the iMaterialist Fashion competition I unintentionally acted without local validation at all and the public LB correlated very well with the private LB.
", "rawMarkdown": "All your comments look very emotional and not very constructive. I do not want to waste time on this kind of discussion.\nAll I can say about public results: seems that I overfitted. I have done this many times and not always successfully. TGS Salt: 11th (in Gold) Public -> 40th Private; Airbus: 2nd (!) Public -> 36th Private.\nThis time, more fortunate. And in the iMaterialist Fashion competition I unintentionally acted without local validation at all and the public LB correlated very well with the private LB.", "votes": -28, "replies": [ { "id": 550039, "postDate": "2019-06-11T08:38:33.283Z", "content": "I think cheating or private sharing are preferable over thinking that all the Kagglers are stupid and can believe in such thing. You did something and Kaggle admins are ok with it. You make it worse every time you speak. A constructive advice: just don't make this situation even uglier.
", "rawMarkdown": "I think cheating or private sharing are preferable over thinking that all the Kagglers are stupid and can believe in such thing. You did something and Kaggle admins are ok with it. You make it worse every time you speak. A constructive advice: just don't make this situation even uglier.", "votes": 7 }, { "id": 550046, "postDate": "2019-06-11T08:48:58.877Z", "content": "Actually, u are wasting our time.
", "rawMarkdown": "Actually, u are wasting our time.", "votes": 11 }, { "id": 550065, "postDate": "2019-06-11T09:09:37.133Z", "content": "It is strange that X5 overfitted.
", "rawMarkdown": "It is strange that X5 overfitted.\n", "votes": 9 }, { "id": 550066, "postDate": "2019-06-11T09:09:55.177Z", "content": "I agree. The best would be to just not participate in second stage and admit your mistake. Don't you have any pride in your work to claim gold for this cheating crap?
\n\n\n\n\nI unintentionally acted without local validation at all and the public LB correlated very well with the private LB.
\n
LoL, we all know what you did.
", "rawMarkdown": "I agree. The best would be to just not participate in second stage and admit your mistake. Don't you have any pride in your work to claim gold for this cheating crap?\n\n> I unintentionally acted without local validation at all and the public LB correlated very well with the private LB.\n\nLoL, we all know what you did.", "votes": 6 }, { "id": 550068, "postDate": "2019-06-11T09:11:24.537Z", "content": "talk is cheap,show me your code
", "rawMarkdown": "talk is cheap,show me your code", "votes": 10 }, { "id": 550118, "postDate": "2019-06-11T10:07:49.480Z", "content": "Considering data and evaluation metric, it is difficult to overfit public LB. What do you think about this?
", "rawMarkdown": "Considering data and evaluation metric, it is difficult to overfit public LB. What do you think about this?", "votes": 8 }, { "id": 595070, "postDate": "2019-08-08T19:58:56.630Z", "content": "@seesee could I ask you be more specific of \"LoL, we all know what you did.\"\nJust wanna know what's done really. Thanks ))))
", "rawMarkdown": "@seesee could I ask you be more specific of \"LoL, we all know what you did.\"\nJust wanna know what's done really. Thanks ))))" } ] }, { "id": 550279, "postDate": "2019-06-11T13:11:35.993Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 549666, "author_name": "Yiheng Wang", "author_url": "", "post_date": "2019-06-10T23:06:52.117000", "content": "Don’t you want to say sth about your public LB?
\n\nIMHO, you should prove your label smoothing and data cleaning don’t use your external data.
", "votes": 22, "replies": [] }, { "id": 549682, "author_name": "SeuTao", "author_url": "", "post_date": "2019-06-10T23:50:51.627000", "content": "The PB story must be explained, it's easy to get such score just tuning your model with the external data (no need to train on it ).
", "votes": 19, "replies": [] }, { "id": 549720, "author_name": "owruby", "author_url": "", "post_date": "2019-06-11T00:56:57.797000", "content": "Please explain about your Public LB Score jump.
", "votes": 20, "replies": [] }, { "id": 549729, "author_name": "earhian", "author_url": "", "post_date": "2019-06-11T01:23:50.820000", "content": "Crawler grandmaster~ Never mind the scandal and libel!
", "votes": 18, "replies": [] }, { "id": 552404, "author_name": "Kulbear", "author_url": "", "post_date": "2019-06-13T22:25:54.667000", "content": "That celebration!\nhttps://www.linkedin.com/feed/update/urn:li:activity:6545036462444818434
", "votes": 6, "replies": [] }, { "id": 549637, "author_name": "Abhishek Thakur", "author_url": "", "post_date": "2019-06-10T22:25:13.787000", "content": "Thanks! Could you please share your code too?
", "votes": 5, "replies": [ { "id": 549662, "author_name": "n01z3", "author_url": "", "post_date": "2019-06-10T23:02:07.930000", "content": "It is very messy. I will try to find time to clean and share.
", "votes": -23, "replies": [] }, { "id": 549804, "author_name": "nekoder", "author_url": "", "post_date": "2019-06-11T03:54:18.050000", "content": "If you want to clear doubts, I think it would be better to share your code ASAP, even if it is messy ...
", "votes": 1, "replies": [] } ] }, { "id": 549792, "author_name": "phalanx", "author_url": "", "post_date": "2019-06-11T03:31:54.390000", "content": "Why you don't explain Public LB jump? I'm so frustrated to yours, top 5 public LB team. Why? Can you speak English? If not, you should explain it in Russian.
", "votes": 0, "replies": [ { "id": 549839, "author_name": "", "author_url": "", "post_date": "2019-06-11T04:44:28.630000", "content": "", "votes": -7, "replies": [] }, { "id": 549994, "author_name": "", "author_url": "", "post_date": "2019-06-11T07:50:00.077000", "content": "This is actually so toxic, no matter of were rules violated or not.
", "votes": 21, "replies": [] }, { "id": 550062, "author_name": "nomad", "author_url": "", "post_date": "2019-06-11T09:08:04.257000", "content": "it seems somebody need sedative
", "votes": 3, "replies": [] }, { "id": 550114, "author_name": "phalanx", "author_url": "", "post_date": "2019-06-11T10:06:49.313000", "content": "I was so emotional. Sorry.
", "votes": 0, "replies": [] } ] }, { "id": 550093, "author_name": "Taras Baranyuk", "author_url": "", "post_date": "2019-06-11T09:47:56.683000", "content": "All doubts can be dispelled if admins ( @wcukierski ) confirm that the selected submissions for stage2 coincide with those that gave the top5 result on the public LB.
", "votes": 3, "replies": [] }, { "id": 549639, "author_name": "Chenyang Zhang", "author_url": "", "post_date": "2019-06-10T22:29:29.810000", "content": "Hey, thanks for sharing! Wondering if you have plan to go to CVPR?
", "votes": 1, "replies": [ { "id": 549647, "author_name": "Abhishek Thakur", "author_url": "", "post_date": "2019-06-10T22:40:05.750000", "content": "With this generic approach people can go to CVPR?
", "votes": 10, "replies": [] }, { "id": 549657, "author_name": "Xuan Cao", "author_url": "", "post_date": "2019-06-10T22:57:02.617000", "content": "They can tell their LB top 5 story, that would be much more interesting than this version.
", "votes": 9, "replies": [] }, { "id": 549665, "author_name": "n01z3", "author_url": "", "post_date": "2019-06-10T23:06:03.330000", "content": "I do not have a visa. And I hardly get it quickly. Can I delegate my speech to another person? We did not work together on this particular task, but he can perform very well. This is Vladimir Iglovikov @iglovikov .
", "votes": -18, "replies": [] }, { "id": 549997, "author_name": "", "author_url": "", "post_date": "2019-06-11T07:51:27.903000", "content": "Что ты мне сделаешь, я в другом городе
", "votes": 0, "replies": [] } ] }, { "id": 549892, "author_name": "phalanx", "author_url": "", "post_date": "2019-06-11T05:51:36.200000", "content": "Why you ignore us? What are you afraid of?
", "votes": 1, "replies": [] }, { "id": 549911, "author_name": "seefun", "author_url": "", "post_date": "2019-06-11T06:17:31.290000", "content": "I found that many people think you violated the rules and used external data sets. Could you clarify that?
", "votes": 2, "replies": [] }, { "id": 549825, "author_name": "FGPC", "author_url": "", "post_date": "2019-06-11T04:13:33.180000", "content": "Many thanks for your shared solutions. Is it possible for you to share source code solutions for us to recreate this? Congratulations! :-)
", "votes": 0, "replies": [] }, { "id": 550031, "author_name": "n01z3", "author_url": "", "post_date": "2019-06-11T08:32:47.870000", "content": "All your comments look very emotional and not very constructive. I do not want to waste time on this kind of discussion.\nAll I can say about public results: seems that I overfitted. I have done this many times and not always successfully. TGS Salt: 11th (in Gold) Public -> 40th Private; Airbus: 2nd (!) Public -> 36th Private.\nThis time, more fortunate. And in the iMaterialist Fashion competition I unintentionally acted without local validation at all and the public LB correlated very well with the private LB.
", "votes": -28, "replies": [ { "id": 550039, "author_name": "Ahmet Erdem", "author_url": "", "post_date": "2019-06-11T08:38:33.283000", "content": "I think cheating or private sharing are preferable over thinking that all the Kagglers are stupid and can believe in such thing. You did something and Kaggle admins are ok with it. You make it worse every time you speak. A constructive advice: just don't make this situation even uglier.
", "votes": 7, "replies": [] }, { "id": 550046, "author_name": "SeuTao", "author_url": "", "post_date": "2019-06-11T08:48:58.877000", "content": "Actually, u are wasting our time.
", "votes": 11, "replies": [] }, { "id": 550065, "author_name": "Gary", "author_url": "", "post_date": "2019-06-11T09:09:37.133000", "content": "It is strange that X5 overfitted.
", "votes": 9, "replies": [] }, { "id": 550066, "author_name": "See--", "author_url": "", "post_date": "2019-06-11T09:09:55.177000", "content": "I agree. The best would be to just not participate in second stage and admit your mistake. Don't you have any pride in your work to claim gold for this cheating crap?
\n\n\n\n\nI unintentionally acted without local validation at all and the public LB correlated very well with the private LB.
\n
LoL, we all know what you did.
", "votes": 6, "replies": [] }, { "id": 550068, "author_name": "qrfaction", "author_url": "", "post_date": "2019-06-11T09:11:24.537000", "content": "talk is cheap,show me your code
", "votes": 10, "replies": [] }, { "id": 550118, "author_name": "phalanx", "author_url": "", "post_date": "2019-06-11T10:07:49.480000", "content": "Considering data and evaluation metric, it is difficult to overfit public LB. What do you think about this?
", "votes": 8, "replies": [] }, { "id": 595070, "author_name": "Mukharbek Organokov", "author_url": "", "post_date": "2019-08-08T19:58:56.630000", "content": "@seesee could I ask you be more specific of \"LoL, we all know what you did.\"\nJust wanna know what's done really. Thanks ))))
", "votes": 0, "replies": [] } ] }, { "id": 550279, "author_name": "", "author_url": "", "post_date": "2019-06-11T13:11:35.993000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "549666": "Don’t you want to say sth about your public LB? \n\nIMHO, you should prove your label smoothing and data cleaning don’t use your external data. ", "549682": "The PB story must be explained, it's easy to get such score just tuning your model with the external data (no need to train on it ). ", "549720": "Please explain about your Public LB Score jump.", "549729": "Crawler grandmaster~ Never mind the scandal and libel!", "552404": "That celebration!\nhttps://www.linkedin.com/feed/update/urn:li:activity:6545036462444818434", "549637": "Thanks! Could you please share your code too?", "549792": "Why you don't explain Public LB jump? I'm so frustrated to yours, top 5 public LB team. Why? Can you speak English? If not, you should explain it in Russian.", "550093": "All doubts can be dispelled if admins ( @wcukierski ) confirm that the selected submissions for stage2 coincide with those that gave the top5 result on the public LB.", "549639": "Hey, thanks for sharing! Wondering if you have plan to go to CVPR?", "549892": "Why you ignore us? What are you afraid of?", "549630": "In this competence, I tried:\n1. Nvidia apex.\nI have been training different imagenet models with apex for a long time. And I and noticed only a minor drop in accuracy. So, I trained with O2 opt level even at 1080Ti. The speed did not boost, but the batch was higher, and therefore the BN statistics are more pleasant.\nIn readme of apex repo one promised that you need to add only 3 lines of code. In reality, of course, it is not. But if just apex converts the code into spaghetti, then adding distributed compatibility means throwing an additional pack of noodles.\nAt the same time, I haven't figured out how to scale the LR when training for 4-8 GPUs distributed. Fine tuning of scheduling goes away and SyncBN does not help.\n\n2. FocalLoss\nweighted by crop and tag. Tags assigned more weight than culture. \n\n3. Mixup. \nI used implementation where the batch is mixed with its own shuffle version. As a result, at first it seemed that the nets were less overfitted, but this was due to the fact that the metric on the train was wrong. When I added additional validation train part without augmentations and compared it, it turned out that the nets just learn longer.\n\n4. Batch accumulation. \nWith accum = 20.\n\n5. Fine tuning of scheduling. \nNets overfitted a lot in this competition. So I tried to minimize the number of passes through the dataset. During grid search of right LR for different stages, it turned out that this is about how to raise the initial LR on one step and put ReduceLrOnPlatoe with Patience 0.\n\n\n7. All experiments were performed using the se-resnext50. I still consider it as the best architecture in speed-accuracy trade-off. Most accurate was se-resnext101 and pnasnet-large. SENet154 I didn’t learn well for a long time with the explosion of gradients, and as a result, it didn’t perform very well. effnet-b3, se-densenet161, densenet161, didn’t work.\n\n8. Label smoothing. \nTried with fixed epsilon and when it is selected individually by class depending on the frequency of occurrence of this dataset. As a result, the optimum threshold was slightly different. But these models helped in the ensemble.\n\n9. Cleaning of data helps a lot. I've done: Clean up less than 20 labels and low per class AUC. Add labels by the confidence of OOF predictions. Clean up bad markup by confidence. \n\n6. Augmentations, I also tried a lot of options. \nMostly from the article about tricks on Imagenet. CutOut did not boost but did not interfere. Large rotations and scale worsened score. As a result, I pad up to 320 along the narrow side with BorderReplicate, because it seemed to me that there were exhibits with a frame and I wanted the net to learn this pattern. Cut long sausages-style image and resize to net input. I end with:\n`\n\n\n alb.PadIfNeeded(scale_size, scale_size, border_mode=cv2.BORDER_REPLICATE),\n alb.ShiftScaleRotate(\n shift_limit=0.1,\n scale_limit=(-0.0, 0.0),\n rotate_limit=5,\n p=0.5,\n interpolation=cv2.INTER_LINEAR,\n border_mode=cv2.BORDER_REPLICATE,\n ),\n alb.Cutout(p=0.2),\n PowerFistRandomSizedCrop(\n min_max_height=min_max_height,\n height=scale_size,\n width=scale_size,\n w2h_ratio=w2h_ratio,\n ),\n alb.HorizontalFlip(p=0.5),\n alb.RandomBrightnessContrast(p=0.3),\n alb.RandomGamma(gamma_limit=(95, 105), p=0.3),\n post_transform(),\n\n`", "549911": "I found that many people think you violated the rules and used external data sets. Could you clarify that?", "549825": "Many thanks for your shared solutions. Is it possible for you to share source code solutions for us to recreate this? Congratulations! :-)", "550031": "All your comments look very emotional and not very constructive. I do not want to waste time on this kind of discussion.\nAll I can say about public results: seems that I overfitted. I have done this many times and not always successfully. TGS Salt: 11th (in Gold) Public -> 40th Private; Airbus: 2nd (!) Public -> 36th Private.\nThis time, more fortunate. And in the iMaterialist Fashion competition I unintentionally acted without local validation at all and the public LB correlated very well with the private LB.", "550279": "" } }