You copy someone else and got a zero. Who should be responsible for that???ππ
", "messages": [ { "id": "1303135", "postDate": "05/11/2021 23:36:00", "content": "You copy someone else and got a zero. Who should be responsible for that???ππ
", "rawMarkdown": "You copy someone else and got a zero. Who should be responsible for that???ππ", "votes": null }, { "id": "1303259", "postDate": "05/12/2021 01:33:37", "content": "angtkππππππ
", "rawMarkdown": "angtkππππππ", "votes": null }, { "id": "1303301", "postDate": "05/12/2021 02:21:15", "content": "Xiaren Zhuxin
", "rawMarkdown": "Xiaren Zhuxin", "votes": null }, { "id": "1303343", "postDate": "05/12/2021 03:07:47", "content": "Respectfully, an inference pipeline is hardly what this competition was about - every person in the competition leveraged modules made by somebody else. Did you use pytorch or tensorflow? If so I guess your guilty too.
\nAs an example of my frustration and perhaps others in this competition, a model made > 6 weeks ago used rasterio (also open source guess I'm a cheater π€) as the pipeline and ended up with a 94 on the private LB - I decided to change my pipeline to deepflash because the sampling using PDFs I though was pretty cool and might help improve my models further - as a result of this change all the work I've done over the most recent 6 weeks is null and void just because I changed my pipelineβ¦hours and hours of my life gone.
\nI think many people would agree that a competition with over 60% of people with a score of 0 indicates an issue with the competition processing and not the competitors. I understand that you and others benefited but what's the point of competing to see who is the best if over half the competition gets dq'ed for something completely out of their control in terms of planning and implementation?
\nβ¦.When I was in undergrad one of my professors would say that a well designed test would have 50% of people scoring in the 70% range (thereby getting a C) - this would allow the test to separate those that knew the material from those that didn't. I'd say that scoring in this competition hardly followed a normal distribution and did not serve as a well designed test of skill since the majority of competitors ended up with an F instead of a C.
", "rawMarkdown": "Respectfully, an inference pipeline is hardly what this competition was about - every person in the competition leveraged modules made by somebody else. Did you use pytorch or tensorflow? If so I guess your guilty too.\n\nAs an example of my frustration and perhaps others in this competition, a model made > 6 weeks ago used rasterio (also open source guess I'm a cheater π€) as the pipeline and ended up with a 94 on the private LB - I decided to change my pipeline to deepflash because the sampling using PDFs I though was pretty cool and might help improve my models further - as a result of this change all the work I've done over the most recent 6 weeks is null and void just because I changed my pipeline...hours and hours of my life gone.\n\nI think many people would agree that a competition with over 60% of people with a score of 0 indicates an issue with the competition processing and not the competitors. I understand that you and others benefited but what's the point of competing to see who is the best if over half the competition gets dq'ed for something completely out of their control in terms of planning and implementation?\n\n....When I was in undergrad one of my professors would say that a well designed test would have 50% of people scoring in the 70% range (thereby getting a C) - this would allow the test to separate those that knew the material from those that didn't. I'd say that scoring in this competition hardly followed a normal distribution and did not serve as a well designed test of skill since the majority of competitors ended up with an F instead of a C.", "votes": null }, { "id": "1303357", "postDate": "05/12/2021 03:17:17", "content": "pandaβs bamboo shoots belong to youπ
", "rawMarkdown": "pandaβs bamboo shoots belong to youπ", "votes": null }, { "id": "1303553", "postDate": "05/12/2021 06:13:12", "content": "One of the issues with this particular notebook is that is was changed 3-4 days prior to competition ending to use rasterio. As a public notebook there is a banner that says not to post high scoring ones near the competition end. This is accepted practice by most/nearly all in Kaggle. Obviously the owner put a lot of work in and if they found issues they needed to change that is fine - changing your own work in the last week is no problem, keep it private. But putting a new version out there for others to fork at the last minute while tanking hundreds of others is kind of low.
\nThe comments has some discussion between a few people who would have been in the know about the changes, it is arguably public but not nearly so as what people look at in Discussion posts. Just because someone forks a notebook they are not notified of all comments and do not believe you can \"follow\" a notebook. It is pedantic to say now it was \"public\". People often look for a suitable inference that works and once they are using it seldom revisit the original. As you say the work is on the model(s).
\nHowever, this was not the only cause of 0 scores for others here as they did not use this notebook or pipeline at all. There is at least an issue here in having a public test set that is not representative of private test. Clearly if there is an image that is so large notebooks will fail silently that is also kind of low. I question this Submission selection reminder \"Note: many teams have submissions that have a private score of 0. Make sure, when selecting your final submissions, that you do not select those!\" posted hours before the competition ended. So it was known there were issues of private score of 0. However no one would know private score not to select until after the end. What was the point of that topic post? A tip off to some?
\nWhat is unfortunate is the impression this can leave on Kaggle novices, newcomers to Kaggle or these type of competitions. Not all are like this and all you can do is learn from the experience and be resilient.
", "rawMarkdown": "One of the issues with this particular notebook is that is was changed 3-4 days prior to competition ending to use rasterio. As a public notebook there is a banner that says not to post high scoring ones near the competition end. This is accepted practice by most/nearly all in Kaggle. Obviously the owner put a lot of work in and if they found issues they needed to change that is fine - changing your own work in the last week is no problem, keep it private. But putting a new version out there for others to fork at the last minute while tanking hundreds of others is kind of low. \n\nThe comments has some discussion between a few people who would have been in the know about the changes, it is arguably public but not nearly so as what people look at in Discussion posts. Just because someone forks a notebook they are not notified of all comments and do not believe you can \"follow\" a notebook. It is pedantic to say now it was \"public\". People often look for a suitable inference that works and once they are using it seldom revisit the original. As you say the work is on the model(s). \n\nHowever, this was not the only cause of 0 scores for others here as they did not use this notebook or pipeline at all. There is at least an issue here in having a public test set that is not representative of private test. Clearly if there is an image that is so large notebooks will fail silently that is also kind of low. I question this [Submission selection reminder](https://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/237904) \"Note: many teams have submissions that have a private score of 0. Make sure, when selecting your final submissions, that you do not select those!\" posted hours before the competition ended. So it was known there were issues of private score of 0. However no one would know private score not to select until after the end. What was the point of that topic post? A tip off to some? \n\nWhat is unfortunate is the impression this can leave on Kaggle novices, newcomers to Kaggle or these type of competitions. Not all are like this and all you can do is learn from the experience and be resilient.", "votes": null }, { "id": "1303822", "postDate": "05/12/2021 09:06:59", "content": "If my adversarial instincts were acting they would probably suggest this could have been done as sabotage π
\nI thought about this before when I found mistakes in public baselines or solutions that had honest mistakes in them. You could spend days before realizing you were going in the wrong direction. But that's ok, it's part of the process.
\nBut, in principle, a bad actor could intentionally create a baseline that has hidden or subtle mistakes or false leads in it that could result, in the worst case, in the clusterfunk that happened here. But the bad actor doesn't need to aim for that, if the solution they shared throws enough people off track for a while and leads them to waste precious time on a dead.
", "rawMarkdown": "If my adversarial instincts were acting they would probably suggest this could have been done as sabotage π \n\nI thought about this before when I found mistakes in public baselines or solutions that had honest mistakes in them. You could spend days before realizing you were going in the wrong direction. But that's ok, it's part of the process.\n\nBut, in principle, a bad actor could intentionally create a baseline that has hidden or subtle mistakes or false leads in it that could result, in the worst case, in the clusterfunk that happened here. But the bad actor doesn't need to aim for that, if the solution they shared throws enough people off track for a while and leads them to waste precious time on a dead.", "votes": null } ], "comments": [ { "id": 1303259, "author_name": "jasonhuangcn", "author_url": "", "post_date": "05/12/2021 01:33:37", "content": "angtkππππππ
", "votes": null, "replies": [] }, { "id": 1303301, "author_name": "yienngxiong", "author_url": "", "post_date": "05/12/2021 02:21:15", "content": "Xiaren Zhuxin
", "votes": null, "replies": [] }, { "id": 1303343, "author_name": "goalieperson", "author_url": "", "post_date": "05/12/2021 03:07:47", "content": "Respectfully, an inference pipeline is hardly what this competition was about - every person in the competition leveraged modules made by somebody else. Did you use pytorch or tensorflow? If so I guess your guilty too.
\nAs an example of my frustration and perhaps others in this competition, a model made > 6 weeks ago used rasterio (also open source guess I'm a cheater π€) as the pipeline and ended up with a 94 on the private LB - I decided to change my pipeline to deepflash because the sampling using PDFs I though was pretty cool and might help improve my models further - as a result of this change all the work I've done over the most recent 6 weeks is null and void just because I changed my pipelineβ¦hours and hours of my life gone.
\nI think many people would agree that a competition with over 60% of people with a score of 0 indicates an issue with the competition processing and not the competitors. I understand that you and others benefited but what's the point of competing to see who is the best if over half the competition gets dq'ed for something completely out of their control in terms of planning and implementation?
\nβ¦.When I was in undergrad one of my professors would say that a well designed test would have 50% of people scoring in the 70% range (thereby getting a C) - this would allow the test to separate those that knew the material from those that didn't. I'd say that scoring in this competition hardly followed a normal distribution and did not serve as a well designed test of skill since the majority of competitors ended up with an F instead of a C.
", "votes": null, "replies": [ { "id": 1303553, "author_name": "something4kag", "author_url": "", "post_date": "05/12/2021 06:13:12", "content": "One of the issues with this particular notebook is that is was changed 3-4 days prior to competition ending to use rasterio. As a public notebook there is a banner that says not to post high scoring ones near the competition end. This is accepted practice by most/nearly all in Kaggle. Obviously the owner put a lot of work in and if they found issues they needed to change that is fine - changing your own work in the last week is no problem, keep it private. But putting a new version out there for others to fork at the last minute while tanking hundreds of others is kind of low.
\nThe comments has some discussion between a few people who would have been in the know about the changes, it is arguably public but not nearly so as what people look at in Discussion posts. Just because someone forks a notebook they are not notified of all comments and do not believe you can \"follow\" a notebook. It is pedantic to say now it was \"public\". People often look for a suitable inference that works and once they are using it seldom revisit the original. As you say the work is on the model(s).
\nHowever, this was not the only cause of 0 scores for others here as they did not use this notebook or pipeline at all. There is at least an issue here in having a public test set that is not representative of private test. Clearly if there is an image that is so large notebooks will fail silently that is also kind of low. I question this Submission selection reminder \"Note: many teams have submissions that have a private score of 0. Make sure, when selecting your final submissions, that you do not select those!\" posted hours before the competition ended. So it was known there were issues of private score of 0. However no one would know private score not to select until after the end. What was the point of that topic post? A tip off to some?
\nWhat is unfortunate is the impression this can leave on Kaggle novices, newcomers to Kaggle or these type of competitions. Not all are like this and all you can do is learn from the experience and be resilient.
", "votes": null, "replies": [] } ] }, { "id": 1303357, "author_name": "fatliuyun", "author_url": "", "post_date": "05/12/2021 03:17:17", "content": "pandaβs bamboo shoots belong to youπ
", "votes": null, "replies": [] }, { "id": 1303822, "author_name": "rosuluc", "author_url": "", "post_date": "05/12/2021 09:06:59", "content": "If my adversarial instincts were acting they would probably suggest this could have been done as sabotage π
\nI thought about this before when I found mistakes in public baselines or solutions that had honest mistakes in them. You could spend days before realizing you were going in the wrong direction. But that's ok, it's part of the process.
\nBut, in principle, a bad actor could intentionally create a baseline that has hidden or subtle mistakes or false leads in it that could result, in the worst case, in the clusterfunk that happened here. But the bad actor doesn't need to aim for that, if the solution they shared throws enough people off track for a while and leads them to waste precious time on a dead.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1303135": "You copy someone else and got a zero. Who should be responsible for that???ππ", "1303259": "angtkππππππ", "1303301": "Xiaren Zhuxin", "1303343": "Respectfully, an inference pipeline is hardly what this competition was about - every person in the competition leveraged modules made by somebody else. Did you use pytorch or tensorflow? If so I guess your guilty too.\n\nAs an example of my frustration and perhaps others in this competition, a model made > 6 weeks ago used rasterio (also open source guess I'm a cheater π€) as the pipeline and ended up with a 94 on the private LB - I decided to change my pipeline to deepflash because the sampling using PDFs I though was pretty cool and might help improve my models further - as a result of this change all the work I've done over the most recent 6 weeks is null and void just because I changed my pipeline...hours and hours of my life gone.\n\nI think many people would agree that a competition with over 60% of people with a score of 0 indicates an issue with the competition processing and not the competitors. I understand that you and others benefited but what's the point of competing to see who is the best if over half the competition gets dq'ed for something completely out of their control in terms of planning and implementation?\n\n....When I was in undergrad one of my professors would say that a well designed test would have 50% of people scoring in the 70% range (thereby getting a C) - this would allow the test to separate those that knew the material from those that didn't. I'd say that scoring in this competition hardly followed a normal distribution and did not serve as a well designed test of skill since the majority of competitors ended up with an F instead of a C.", "1303357": "pandaβs bamboo shoots belong to youπ", "1303553": "One of the issues with this particular notebook is that is was changed 3-4 days prior to competition ending to use rasterio. As a public notebook there is a banner that says not to post high scoring ones near the competition end. This is accepted practice by most/nearly all in Kaggle. Obviously the owner put a lot of work in and if they found issues they needed to change that is fine - changing your own work in the last week is no problem, keep it private. But putting a new version out there for others to fork at the last minute while tanking hundreds of others is kind of low. \n\nThe comments has some discussion between a few people who would have been in the know about the changes, it is arguably public but not nearly so as what people look at in Discussion posts. Just because someone forks a notebook they are not notified of all comments and do not believe you can \"follow\" a notebook. It is pedantic to say now it was \"public\". People often look for a suitable inference that works and once they are using it seldom revisit the original. As you say the work is on the model(s). \n\nHowever, this was not the only cause of 0 scores for others here as they did not use this notebook or pipeline at all. There is at least an issue here in having a public test set that is not representative of private test. Clearly if there is an image that is so large notebooks will fail silently that is also kind of low. I question this [Submission selection reminder](https://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/237904) \"Note: many teams have submissions that have a private score of 0. Make sure, when selecting your final submissions, that you do not select those!\" posted hours before the competition ended. So it was known there were issues of private score of 0. However no one would know private score not to select until after the end. What was the point of that topic post? A tip off to some? \n\nWhat is unfortunate is the impression this can leave on Kaggle novices, newcomers to Kaggle or these type of competitions. Not all are like this and all you can do is learn from the experience and be resilient.", "1303822": "If my adversarial instincts were acting they would probably suggest this could have been done as sabotage π \n\nI thought about this before when I found mistakes in public baselines or solutions that had honest mistakes in them. You could spend days before realizing you were going in the wrong direction. But that's ok, it's part of the process.\n\nBut, in principle, a bad actor could intentionally create a baseline that has hidden or subtle mistakes or false leads in it that could result, in the worst case, in the clusterfunk that happened here. But the bad actor doesn't need to aim for that, if the solution they shared throws enough people off track for a while and leads them to waste precious time on a dead." }, "source": "meta" }