It turns out there was a pretty significant bug in the previous version of \"Mean Teachers Find More Birds 🐦\". In the public version that scored 0.844 (v21), I had the consistency_rampup parameter set to 1000 which means that the consistency weight wasn't really being applied and it was just training a ~baseline model with useless extra steps.
I'd previously set it to 1000 to do a baseline run and then failed to switch it back. The corrected version (consistency_rampup = 6 # epochs) does a good bit better at 0.857.
I don't mean to spam my notebook, but I figured I should post something since a handful of people forked a bad version. Sorry about that!
", "messages": [ { "id": "1198126", "postDate": "02/12/2021 18:37:13", "content": "It turns out there was a pretty significant bug in the previous version of \"Mean Teachers Find More Birds 🐦\". In the public version that scored 0.844 (v21), I had the consistency_rampup parameter set to 1000 which means that the consistency weight wasn't really being applied and it was just training a ~baseline model with useless extra steps.
I'd previously set it to 1000 to do a baseline run and then failed to switch it back. The corrected version (consistency_rampup = 6 # epochs) does a good bit better at 0.857.
I don't mean to spam my notebook, but I figured I should post something since a handful of people forked a bad version. Sorry about that!
", "rawMarkdown": "It turns out there was a pretty significant bug in the previous version of \"[Mean Teachers Find More Birds 🐦](https://www.kaggle.com/reppic/mean-teachers-find-more-birds)\". In the public version that scored 0.844 (v21), I had the `consistency_rampup` parameter set to `1000` which means that the consistency weight wasn't really being applied and it was just training a ~baseline model with useless extra steps. \n\nI'd previously set it to 1000 to do a baseline run and then failed to switch it back. The corrected version (`consistency_rampup = 6 # epochs`) does a good bit better at 0.857. \n\nI don't mean to spam my notebook, but I figured I should post something since a handful of people forked a bad version. Sorry about that!", "votes": null }, { "id": "1198252", "postDate": "02/12/2021 20:31:05", "content": "Hello! Just thought to write what changes you made because I did not notice the difference in parameters or in the model between the versions of your notebook:-). Thank you!
", "rawMarkdown": "Hello! Just thought to write what changes you made because I did not notice the difference in parameters or in the model between the versions of your notebook:-). Thank you!", "votes": null }, { "id": "1198552", "postDate": "02/13/2021 06:25:53", "content": "Thanks for bringing this up. It seems to me like the primary value is coming from looking at the unlabeled regions of the data. I wonder if maybe using them as soft 0's might work as well without the additional overhead of a mean teacher model setup
", "rawMarkdown": "Thanks for bringing this up. It seems to me like the primary value is coming from looking at the unlabeled regions of the data. I wonder if maybe using them as soft 0's might work as well without the additional overhead of a mean teacher model setup", "votes": null }, { "id": "1198565", "postDate": "02/13/2021 06:37:23", "content": "Yea, I think that's possible. I'd tried randomly sampling unlabeled segments with 0's, but it didn't seem to do quite as well as mean teacher. Although I didn't do a rigorous comparison, and I think it was close, so who knows.
", "rawMarkdown": "Yea, I think that's possible. I'd tried randomly sampling unlabeled segments with 0's, but it didn't seem to do *quite* as well as mean teacher. Although I didn't do a rigorous comparison, and I think it was close, so who knows.", "votes": null }, { "id": "1200014", "postDate": "02/14/2021 10:39:36", "content": "I actually didn't see your notebook, it's a great piece of work.
\nChanges a lot from the baselines, blends and EDAs we're all used to. :)
I didn't read your notebook either but this was on my todo list. Thanks for sharing your code.
", "rawMarkdown": "I didn't read your notebook either but this was on my todo list. Thanks for sharing your code.", "votes": null }, { "id": "1200341", "postDate": "02/14/2021 15:34:50", "content": "@reppic - 1 position away from your first gold..Go for it buddy…It is always exciting (and motivating) to see new faces at the top…
", "rawMarkdown": "reppic - 1 position away from your first gold..Go for it buddy...It is always exciting (and motivating) to see new faces at the top...", "votes": null }, { "id": "1200664", "postDate": "02/14/2021 20:51:18", "content": "Thanks @allohvk ! I never figured out a good validation strategy, so I've just been relying on the public lb. I'm a little nervous about overfitting after I dropped 758 places in the private lb for the Global Wheat contest. 😬
", "rawMarkdown": "Thanks @allohvk ! I never figured out a good validation strategy, so I've just been relying on the public lb. I'm a little nervous about overfitting after I dropped 758 places in the private lb for the Global Wheat contest. 😬", "votes": null }, { "id": "1200665", "postDate": "02/14/2021 20:51:50", "content": "Thanks Theo! I really appreciate that. 😁
", "rawMarkdown": "Thanks Theo! I really appreciate that. 😁", "votes": null }, { "id": "1200685", "postDate": "02/14/2021 21:33:59", "content": "Good luck to you! And the right choice of model-)!
", "rawMarkdown": "Good luck to you! And the right choice of model-)!", "votes": null }, { "id": "1200773", "postDate": "02/15/2021 02:02:20", "content": "I didn't think Ryan is new as I have seen him doing great and giving insights (thanks Ryan!) in many competitions since at least RSNA last year.
", "rawMarkdown": "I didn't think Ryan is new as I have seen him doing great and giving insights (thanks Ryan!) in many competitions since at least RSNA last year.", "votes": null } ], "comments": [ { "id": 1198252, "author_name": "aikhmelnytskyy", "author_url": "", "post_date": "02/12/2021 20:31:05", "content": "Hello! Just thought to write what changes you made because I did not notice the difference in parameters or in the model between the versions of your notebook:-). Thank you!
", "votes": null, "replies": [] }, { "id": 1198552, "author_name": "ryches", "author_url": "", "post_date": "02/13/2021 06:25:53", "content": "Thanks for bringing this up. It seems to me like the primary value is coming from looking at the unlabeled regions of the data. I wonder if maybe using them as soft 0's might work as well without the additional overhead of a mean teacher model setup
", "votes": null, "replies": [ { "id": 1198565, "author_name": "reppic", "author_url": "", "post_date": "02/13/2021 06:37:23", "content": "Yea, I think that's possible. I'd tried randomly sampling unlabeled segments with 0's, but it didn't seem to do quite as well as mean teacher. Although I didn't do a rigorous comparison, and I think it was close, so who knows.
", "votes": null, "replies": [] } ] }, { "id": 1200014, "author_name": "theoviel", "author_url": "", "post_date": "02/14/2021 10:39:36", "content": "I actually didn't see your notebook, it's a great piece of work.
\nChanges a lot from the baselines, blends and EDAs we're all used to. :)
Thanks Theo! I really appreciate that. 😁
", "votes": null, "replies": [] } ] }, { "id": 1200114, "author_name": "cpmpml", "author_url": "", "post_date": "02/14/2021 12:18:25", "content": "I didn't read your notebook either but this was on my todo list. Thanks for sharing your code.
", "votes": null, "replies": [] }, { "id": 1200341, "author_name": "allohvk", "author_url": "", "post_date": "02/14/2021 15:34:50", "content": "@reppic - 1 position away from your first gold..Go for it buddy…It is always exciting (and motivating) to see new faces at the top…
", "votes": null, "replies": [ { "id": 1200664, "author_name": "reppic", "author_url": "", "post_date": "02/14/2021 20:51:18", "content": "Thanks @allohvk ! I never figured out a good validation strategy, so I've just been relying on the public lb. I'm a little nervous about overfitting after I dropped 758 places in the private lb for the Global Wheat contest. 😬
", "votes": null, "replies": [] }, { "id": 1200685, "author_name": "aikhmelnytskyy", "author_url": "", "post_date": "02/14/2021 21:33:59", "content": "Good luck to you! And the right choice of model-)!
", "votes": null, "replies": [] }, { "id": 1200773, "author_name": "ratthachat", "author_url": "", "post_date": "02/15/2021 02:02:20", "content": "I didn't think Ryan is new as I have seen him doing great and giving insights (thanks Ryan!) in many competitions since at least RSNA last year.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1198126": "It turns out there was a pretty significant bug in the previous version of \"[Mean Teachers Find More Birds 🐦](https://www.kaggle.com/reppic/mean-teachers-find-more-birds)\". In the public version that scored 0.844 (v21), I had the `consistency_rampup` parameter set to `1000` which means that the consistency weight wasn't really being applied and it was just training a ~baseline model with useless extra steps. \n\nI'd previously set it to 1000 to do a baseline run and then failed to switch it back. The corrected version (`consistency_rampup = 6 # epochs`) does a good bit better at 0.857. \n\nI don't mean to spam my notebook, but I figured I should post something since a handful of people forked a bad version. Sorry about that!", "1198252": "Hello! Just thought to write what changes you made because I did not notice the difference in parameters or in the model between the versions of your notebook:-). Thank you!", "1198552": "Thanks for bringing this up. It seems to me like the primary value is coming from looking at the unlabeled regions of the data. I wonder if maybe using them as soft 0's might work as well without the additional overhead of a mean teacher model setup", "1198565": "Yea, I think that's possible. I'd tried randomly sampling unlabeled segments with 0's, but it didn't seem to do *quite* as well as mean teacher. Although I didn't do a rigorous comparison, and I think it was close, so who knows.", "1200014": "I actually didn't see your notebook, it's a great piece of work. \nChanges a lot from the baselines, blends and EDAs we're all used to. :)", "1200114": "I didn't read your notebook either but this was on my todo list. Thanks for sharing your code.", "1200341": "reppic - 1 position away from your first gold..Go for it buddy...It is always exciting (and motivating) to see new faces at the top...", "1200664": "Thanks @allohvk ! I never figured out a good validation strategy, so I've just been relying on the public lb. I'm a little nervous about overfitting after I dropped 758 places in the private lb for the Global Wheat contest. 😬", "1200665": "Thanks Theo! I really appreciate that. 😁", "1200685": "Good luck to you! And the right choice of model-)!", "1200773": "I didn't think Ryan is new as I have seen him doing great and giving insights (thanks Ryan!) in many competitions since at least RSNA last year." }, "source": "meta" }