I scored in the top 5% last year. This year I score .55 to .59 no matter what I do. So I partnered with someone and made a team. He submitted a notebook / model and scored .62. I submitted the identical notebook / model and scored .55. @tomdenton could you follow up? There is a very strange bug in the submission process.
", "messages": [ { "id": "2789713", "postDate": "05/02/2024 19:29:03", "content": "I scored in the top 5% last year. This year I score .55 to .59 no matter what I do. So I partnered with someone and made a team. He submitted a notebook / model and scored .62. I submitted the identical notebook / model and scored .55. @tomdenton could you follow up? There is a very strange bug in the submission process.
", "rawMarkdown": "I scored in the top 5% last year. This year I score .55 to .59 no matter what I do. So I partnered with someone and made a team. He submitted a notebook / model and scored .62. I submitted the identical notebook / model and scored .55. @tomdenton could you follow up? There is a very strange bug in the submission process.", "votes": null }, { "id": "2789715", "postDate": "05/02/2024 19:34:31", "content": "Hi, Jan! Is the notebook deteministic? It may be worth checking that you get the same outputs on the unlabeled data in consecutive runs.
\nIf you've got some clues as to which submissions these are, we can double check that the submitted CSVs are identical.
", "rawMarkdown": "Hi, Jan! Is the notebook deteministic? It may be worth checking that you get the same outputs on the unlabeled data in consecutive runs.\n\nIf you've got some clues as to which submissions these are, we can double check that the submitted CSVs are identical.", "votes": null }, { "id": "2789724", "postDate": "05/02/2024 19:40:01", "content": "Hi Tom. I've been working on this for weeks and my scores really make no sense. I've tried lots of notebooks and models. Training a sensible model for 3 epochs gives me the same score as training it for 20 epochs. Please believe me that there is a weird bug here.
", "rawMarkdown": "Hi Tom. I've been working on this for weeks and my scores really make no sense. I've tried lots of notebooks and models. Training a sensible model for 3 epochs gives me the same score as training it for 20 epochs. Please believe me that there is a weird bug here.", "votes": null }, { "id": "2789734", "postDate": "05/02/2024 19:43:07", "content": "Can we take this offline? I have your email address - how about I email you? Then we can step through a couple of experiments to compare my submissions with my teammates.
", "rawMarkdown": "Can we take this offline? I have your email address - how about I email you? Then we can step through a couple of experiments to compare my submissions with my teammates.", "votes": null }, { "id": "2789750", "postDate": "05/02/2024 19:52:28", "content": "Sounds good - fire away.
", "rawMarkdown": "Sounds good - fire away.", "votes": null }, { "id": "2791167", "postDate": "05/03/2024 14:05:09", "content": "Apologies - we tried a different notebook and got the same score. I'll keep digging.
", "rawMarkdown": "Apologies - we tried a different notebook and got the same score. I'll keep digging.", "votes": null }, { "id": "2791299", "postDate": "05/03/2024 15:32:04", "content": "BTW, I took some time yesterday and confirmed that the scoring function is behaving correctly and that there's clear signal of model quality in the LB.
", "rawMarkdown": "BTW, I took some time yesterday and confirmed that the scoring function is behaving correctly and that there's clear signal of model quality in the LB.", "votes": null }, { "id": "2791331", "postDate": "05/03/2024 15:54:40", "content": "Thanks again Tom. Hopefully I'll figure this out soon!
", "rawMarkdown": "Thanks again Tom. Hopefully I'll figure this out soon!", "votes": null } ], "comments": [ { "id": 2789715, "author_name": "tomdenton", "author_url": "", "post_date": "05/02/2024 19:34:31", "content": "Hi, Jan! Is the notebook deteministic? It may be worth checking that you get the same outputs on the unlabeled data in consecutive runs.
\nIf you've got some clues as to which submissions these are, we can double check that the submitted CSVs are identical.
", "votes": null, "replies": [ { "id": 2789724, "author_name": "janhuus", "author_url": "", "post_date": "05/02/2024 19:40:01", "content": "Hi Tom. I've been working on this for weeks and my scores really make no sense. I've tried lots of notebooks and models. Training a sensible model for 3 epochs gives me the same score as training it for 20 epochs. Please believe me that there is a weird bug here.
", "votes": null, "replies": [ { "id": 2789734, "author_name": "janhuus", "author_url": "", "post_date": "05/02/2024 19:43:07", "content": "Can we take this offline? I have your email address - how about I email you? Then we can step through a couple of experiments to compare my submissions with my teammates.
", "votes": null, "replies": [ { "id": 2789750, "author_name": "tomdenton", "author_url": "", "post_date": "05/02/2024 19:52:28", "content": "Sounds good - fire away.
", "votes": null, "replies": [] } ] } ] }, { "id": 2791167, "author_name": "janhuus", "author_url": "", "post_date": "05/03/2024 14:05:09", "content": "Apologies - we tried a different notebook and got the same score. I'll keep digging.
", "votes": null, "replies": [] } ] }, { "id": 2791299, "author_name": "tomdenton", "author_url": "", "post_date": "05/03/2024 15:32:04", "content": "BTW, I took some time yesterday and confirmed that the scoring function is behaving correctly and that there's clear signal of model quality in the LB.
", "votes": null, "replies": [ { "id": 2791331, "author_name": "janhuus", "author_url": "", "post_date": "05/03/2024 15:54:40", "content": "Thanks again Tom. Hopefully I'll figure this out soon!
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2789713": "I scored in the top 5% last year. This year I score .55 to .59 no matter what I do. So I partnered with someone and made a team. He submitted a notebook / model and scored .62. I submitted the identical notebook / model and scored .55. @tomdenton could you follow up? There is a very strange bug in the submission process.", "2789715": "Hi, Jan! Is the notebook deteministic? It may be worth checking that you get the same outputs on the unlabeled data in consecutive runs.\n\nIf you've got some clues as to which submissions these are, we can double check that the submitted CSVs are identical.", "2789724": "Hi Tom. I've been working on this for weeks and my scores really make no sense. I've tried lots of notebooks and models. Training a sensible model for 3 epochs gives me the same score as training it for 20 epochs. Please believe me that there is a weird bug here.", "2789734": "Can we take this offline? I have your email address - how about I email you? Then we can step through a couple of experiments to compare my submissions with my teammates.", "2789750": "Sounds good - fire away.", "2791167": "Apologies - we tried a different notebook and got the same score. I'll keep digging.", "2791299": "BTW, I took some time yesterday and confirmed that the scoring function is behaving correctly and that there's clear signal of model quality in the LB.", "2791331": "Thanks again Tom. Hopefully I'll figure this out soon!" }, "source": "meta" }