We had some hick-ups on our rescoring backend. I've worked with eng to clear these up and am working on a new rescore. I'll reply here when it's completed.
", "messages": [ { "id": "2817413", "postDate": "05/16/2024 21:57:19", "content": "We had some hick-ups on our rescoring backend. I've worked with eng to clear these up and am working on a new rescore. I'll reply here when it's completed.
", "rawMarkdown": "We had some hick-ups on our rescoring backend. I've worked with eng to clear these up and am working on a new rescore. I'll reply here when it's completed.", "votes": null }, { "id": "2817517", "postDate": "05/17/2024 00:59:55", "content": "The scoring is completed. If you still see errors you don't believe are warranted, please let me know.
", "rawMarkdown": "The scoring is completed. If you still see errors you don't believe are warranted, please let me know.", "votes": null }, { "id": "2817523", "postDate": "05/17/2024 01:11:45", "content": "thanks!
\nall more problems are solved except for one.
one submission is become pending after 2nd round of update.
\nthis submission is made after 1st round of update and it is ok and showed a score.
\nit \"suddenly\" becomes pending
For me, it's working. All the submissions got rescored, and submitted successfully. No pending submissions.
", "rawMarkdown": "For me, it's working. All the submissions got rescored, and submitted successfully. No pending submissions.", "votes": null }, { "id": "2817670", "postDate": "05/17/2024 04:11:47", "content": "Hi! same here - two submissions sent after the rescoring were scored but now are pending
", "rawMarkdown": "Hi! same here - two submissions sent after the rescoring were scored but now are pending", "votes": null }, { "id": "2818171", "postDate": "05/17/2024 10:20:40", "content": "Same again here. Yesterday's last two post-update submissions scored correctly at first, but have now gone into pending purgatory
", "rawMarkdown": "Same again here. Yesterday's last two post-update submissions scored correctly at first, but have now gone into pending purgatory", "votes": null }, { "id": "2818339", "postDate": "05/17/2024 12:59:20", "content": "Can we have a implementation metric calculation notebook?
", "rawMarkdown": "Can we have a implementation metric calculation notebook?", "votes": null }, { "id": "2819644", "postDate": "05/17/2024 16:02:00", "content": "Those have been fixed now. Thanks for reporting!
", "rawMarkdown": "Those have been fixed now. Thanks for reporting!", "votes": null }, { "id": "2819645", "postDate": "05/17/2024 16:02:05", "content": "Here's the code
", "rawMarkdown": "Here's the [code](https://www.kaggle.com/code/metric/leash-average-map)", "votes": null }, { "id": "2820755", "postDate": "05/17/2024 17:30:03", "content": "I have two recent submission with Internal error. Can you have a look if it is expected?
", "rawMarkdown": "I have two recent submission with Internal error. Can you have a look if it is expected?", "votes": null }, { "id": "2821083", "postDate": "05/17/2024 22:47:24", "content": "Thanks for posting the metric code, but its documentation seems a bit sparse. Could you clarify the definitions of the \"solution\" and \"submission\" arguments to the score function?
", "rawMarkdown": "Thanks for posting the metric code, but its documentation seems a bit sparse. Could you clarify the definitions of the \"solution\" and \"submission\" arguments to the score function?", "votes": null }, { "id": "2821200", "postDate": "05/18/2024 02:00:58", "content": "it is not possible for \"full working code\" of evaluation metric to be given. that will disclose the \"secret public and private splitting\" of the test data.
\nOnly high level code for illustrative and clarification purposes can be given.
\nin summary, this is how the evaluation process is done:
\nthere are 6 groups for public score and 9 groups for private one.
\nmore details are given in the post
\nhttps://www.kaggle.com/competitions/leash-BELKA/discussion/503232
solution the ground truth csv and submission is your submitted csv.
\nthe code shows:
\nAssuming that the sample submission file is the correct order ??
", "rawMarkdown": "Assuming that the sample submission file is the correct order ??", "votes": null }, { "id": "2823802", "postDate": "05/19/2024 12:29:31", "content": "there could be some cleaning function before the this metric evaluation function is called. we don't know. but if you just consider the metric function itself, it expects id in the correct order.
", "rawMarkdown": "there could be some cleaning function before the this metric evaluation function is called. we don't know. but if you just consider the metric function itself, it expects id in the correct order.", "votes": null } ], "comments": [ { "id": 2817517, "author_name": "inversion", "author_url": "", "post_date": "05/17/2024 00:59:55", "content": "The scoring is completed. If you still see errors you don't believe are warranted, please let me know.
", "votes": null, "replies": [ { "id": 2817523, "author_name": "hengck23", "author_url": "", "post_date": "05/17/2024 01:11:45", "content": "thanks!
\nall more problems are solved except for one.
one submission is become pending after 2nd round of update.
\nthis submission is made after 1st round of update and it is ok and showed a score.
\nit \"suddenly\" becomes pending
Hi! same here - two submissions sent after the rescoring were scored but now are pending
", "votes": null, "replies": [ { "id": 2818171, "author_name": "jbomitchell", "author_url": "", "post_date": "05/17/2024 10:20:40", "content": "Same again here. Yesterday's last two post-update submissions scored correctly at first, but have now gone into pending purgatory
", "votes": null, "replies": [ { "id": 2819644, "author_name": "inversion", "author_url": "", "post_date": "05/17/2024 16:02:00", "content": "Those have been fixed now. Thanks for reporting!
", "votes": null, "replies": [ { "id": 2820755, "author_name": "w5833946", "author_url": "", "post_date": "05/17/2024 17:30:03", "content": "I have two recent submission with Internal error. Can you have a look if it is expected?
", "votes": null, "replies": [] } ] } ] } ] } ] }, { "id": 2817534, "author_name": "ahsuna123", "author_url": "", "post_date": "05/17/2024 01:31:27", "content": "For me, it's working. All the submissions got rescored, and submitted successfully. No pending submissions.
", "votes": null, "replies": [] }, { "id": 2818339, "author_name": "yuanzhezhou", "author_url": "", "post_date": "05/17/2024 12:59:20", "content": "Can we have a implementation metric calculation notebook?
", "votes": null, "replies": [ { "id": 2819645, "author_name": "inversion", "author_url": "", "post_date": "05/17/2024 16:02:05", "content": "Here's the code
", "votes": null, "replies": [ { "id": 2821083, "author_name": "dslate", "author_url": "", "post_date": "05/17/2024 22:47:24", "content": "Thanks for posting the metric code, but its documentation seems a bit sparse. Could you clarify the definitions of the \"solution\" and \"submission\" arguments to the score function?
", "votes": null, "replies": [ { "id": 2821200, "author_name": "hengck23", "author_url": "", "post_date": "05/18/2024 02:00:58", "content": "it is not possible for \"full working code\" of evaluation metric to be given. that will disclose the \"secret public and private splitting\" of the test data.
\nOnly high level code for illustrative and clarification purposes can be given.
\nin summary, this is how the evaluation process is done:
\nthere are 6 groups for public score and 9 groups for private one.
\nmore details are given in the post
\nhttps://www.kaggle.com/competitions/leash-BELKA/discussion/503232
solution the ground truth csv and submission is your submitted csv.
\nthe code shows:
\nAssuming that the sample submission file is the correct order ??
", "votes": null, "replies": [ { "id": 2823802, "author_name": "hengck23", "author_url": "", "post_date": "05/19/2024 12:29:31", "content": "there could be some cleaning function before the this metric evaluation function is called. we don't know. but if you just consider the metric function itself, it expects id in the correct order.
", "votes": null, "replies": [] } ] } ] } ] } ] } ] } ], "raw_markdown_by_id": { "2817413": "We had some hick-ups on our rescoring backend. I've worked with eng to clear these up and am working on a new rescore. I'll reply here when it's completed.", "2817517": "The scoring is completed. If you still see errors you don't believe are warranted, please let me know.", "2817523": "thanks!\nall more problems are solved except for one.\n\none submission is become pending after 2nd round of update.\nthis submission is made after 1st round of update and it is ok and showed a score.\nit \"suddenly\" becomes pending", "2817534": "For me, it's working. All the submissions got rescored, and submitted successfully. No pending submissions.", "2817670": "Hi! same here - two submissions sent after the rescoring were scored but now are pending", "2818171": "Same again here. Yesterday's last two post-update submissions scored correctly at first, but have now gone into pending purgatory", "2818339": "Can we have a implementation metric calculation notebook?", "2819644": "Those have been fixed now. Thanks for reporting!", "2819645": "Here's the [code](https://www.kaggle.com/code/metric/leash-average-map)", "2820755": "I have two recent submission with Internal error. Can you have a look if it is expected?", "2821083": "Thanks for posting the metric code, but its documentation seems a bit sparse. Could you clarify the definitions of the \"solution\" and \"submission\" arguments to the score function?", "2821200": "it is not possible for \"full working code\" of evaluation metric to be given. that will disclose the \"secret public and private splitting\" of the test data.\n\nOnly high level code for illustrative and clarification purposes can be given. \n\nin summary, this is how the evaluation process is done:\n1. the test SMILES in the kaggle test.csv are split into different groups (aka \"protein_nam\"x \"split_group\")\n2. apply average_precision_score() to each group. If there are 9 groups, you would have 9 group scores.\n3. apply average to the group scores to get final score on the public board.\n\nthere are 6 groups for public score and 9 groups for private one.\nmore details are given in the post\nhttps://www.kaggle.com/competitions/leash-BELKA/discussion/503232\n\nsolution the ground truth csv and submission is your submitted csv.\n\n---\n\nthe code shows:\n1. there is no check for the order of 'id' : del solution[row_id_column_name]\n(so you must submit in the correct order)\n2. raise ParticipantVisibleError: submission values must be between [0,1], numeric, no NANs, etc", "2822394": "Assuming that the sample submission file is the correct order ??", "2823802": "there could be some cleaning function before the this metric evaluation function is called. we don't know. but if you just consider the metric function itself, it expects id in the correct order." }, "source": "meta" }