{ "id": 408000, "title": "Same notebook but different result between 2run ??", "url": "/competitions/tlvmc-parkinsons-freezing-gait-prediction/discussion/408000", "author_name": "", "post_date": "2023-05-09T00:58:11.674808100Z", "votes": 2, "comment_count": 9, "views": 0, "content": "

is wried, one of version of my notebook have a score 2.17. But when i copied it, then rerun the model, I only got 1.87.

\n

since i was copied the version, and did not made change, why they will have the different score??

\n

Ifound that 2 models have different feature importance

\n

original:
\n0 Time 0.465456
\n3 AccAP 0.346994
\n1 AccV 0.074586
\n4 s_kmeans_1 0.050755
\n5 s_kmeans_3 0.040626
\n2 AccML 0.021584

\n

coppied:
\n0 Time 0.483317
\n3 AccAP 0.347417
\n5 s_kmeans_3 0.100661
\n1 AccV 0.039379
\n2 AccML 0.017303
\n4 s_kmeans_1 0.011923

\n

but i still can't unstand, since i did not make any change on the data or model parameter.

", "messages": [ { "id": "2250985", "postDate": "05/09/2023 00:58:11", "content": "

is wried, one of version of my notebook have a score 2.17. But when i copied it, then rerun the model, I only got 1.87.

\n

since i was copied the version, and did not made change, why they will have the different score??

\n

Ifound that 2 models have different feature importance

\n

original:
\n0 Time 0.465456
\n3 AccAP 0.346994
\n1 AccV 0.074586
\n4 s_kmeans_1 0.050755
\n5 s_kmeans_3 0.040626
\n2 AccML 0.021584

\n

coppied:
\n0 Time 0.483317
\n3 AccAP 0.347417
\n5 s_kmeans_3 0.100661
\n1 AccV 0.039379
\n2 AccML 0.017303
\n4 s_kmeans_1 0.011923

\n

but i still can't unstand, since i did not make any change on the data or model parameter.

", "rawMarkdown": "is wried, one of version of my notebook have a score 2.17. But when i copied it, then rerun the model, I only got 1.87. \n\nsince i was copied the version, and did not made change, why they will have the different score??\n\nIfound that 2 models have different feature importance\n\noriginal:\n0 Time 0.465456\n3 AccAP 0.346994\n1 AccV 0.074586\n4 s_kmeans_1 0.050755\n5 s_kmeans_3 0.040626\n2 AccML 0.021584\n\n\ncoppied:\n0 Time 0.483317\n3 AccAP 0.347417\n5 s_kmeans_3 0.100661\n1 AccV 0.039379\n2 AccML 0.017303\n4 s_kmeans_1 0.011923\n\n\n\nbut i still can't unstand, since i did not make any change on the data or model parameter.", "votes": null }, { "id": "2251149", "postDate": "05/09/2023 05:52:30", "content": "

Did you set a random state for reproducibility?

\n
 random\nrandom.seed() \n
", "rawMarkdown": "Did you set a random state for reproducibility?\n\n```python\nimport random\nrandom.seed(42) # for example\n```", "votes": null }, { "id": "2253137", "postDate": "05/10/2023 00:52:32", "content": "

Not really ，it‘s totally a counterpart.

\n

i think it's different between on data upadte and before by the staff, i just knew a few hours before.

\n

But what it show Is interesting, that the old version data bring a better model, does it menan our trainng set not so correlate with the test data?

\n

Still thanks for the replay

", "rawMarkdown": "Not really ，it‘s totally a counterpart.\n\ni think it's different between on data upadte and before by the staff, i just knew a few hours before.\n\nBut what it show Is interesting, that the old version data bring a better model, does it menan our trainng set not so correlate with the test data?\n\nStill thanks for the replay", "votes": null }, { "id": "2254422", "postDate": "05/11/2023 00:59:49", "content": "

Same here, I am facing the same issue. And the score difference is huge: 3.10 -> 2.62

", "rawMarkdown": "Same here, I am facing the same issue. And the score difference is huge: 3.10 -> 2.62", "votes": null }, { "id": "2254479", "postDate": "05/11/2023 03:13:24", "content": "

There is a guess, the test data is not update, so the test data is related to the old train data. lt is why the score is decrease, and not idea the test will be update on the end of the deadline

", "rawMarkdown": "There is a guess, the test data is not update, so the test data is related to the old train data. lt is why the score is decrease, and not idea the test will be update on the end of the deadline", "votes": null }, { "id": "2254554", "postDate": "05/11/2023 04:40:41", "content": "

I am facing the same issue. Checked multiple times if there is some difference in the two notebooks. Nada and yet one is giving 0.286 and the other 0.302.

", "rawMarkdown": "I am facing the same issue. Checked multiple times if there is some difference in the two notebooks. Nada and yet one is giving 0.286 and the other 0.302.", "votes": null }, { "id": "2254668", "postDate": "05/11/2023 06:43:44", "content": "

What is weird is that when they updated the data they re-ran the notebooks and my notebook lost only 0.06 but when I re-run it manually it's getting less than 3.

", "rawMarkdown": "What is weird is that when they updated the data they re-ran the notebooks and my notebook lost only 0.06 but when I re-run it manually it's getting less than 3.", "votes": null }, { "id": "2254670", "postDate": "05/11/2023 06:44:29", "content": "

@ryanholbrook do you have an explanation for this please? Thank you in advance

", "rawMarkdown": "ryanholbrook do you have an explanation for this please? Thank you in advance", "votes": null }, { "id": "2254968", "postDate": "05/11/2023 11:42:29", "content": "

Hi @bechirjamousi,

\n

In the update only the target labels in the test data changed, not the test data itself that your model uses as inputs. After the update I rescored all of the submissions against the new targets. However, I did not rerun all of the notebooks from scratch to produce a new submission. If your submission notebook only made use of the test data, there should be no change from the new score after submitting it again. I'd recommend retraining the model on the new data and resubmitting.

", "rawMarkdown": "Hi @bechirjamousi,\n\nIn the update only the target labels in the test data changed, not the test data itself that your model uses as inputs. After the update I rescored all of the submissions against the new targets. However, I did not rerun all of the notebooks from scratch to produce a new submission. If your submission notebook only made use of the test data, there should be no change from the new score after submitting it again. I'd recommend retraining the model on the new data and resubmitting.", "votes": null }, { "id": "2255023", "postDate": "05/11/2023 12:54:23", "content": "

Thank you for the fast reply. It’s weird, that’s what I did but my score is still down.

", "rawMarkdown": "Thank you for the fast reply. It’s weird, that’s what I did but my score is still down.", "votes": null } ], "comments": [ { "id": 2251149, "author_name": "avivlevi815", "author_url": "", "post_date": "05/09/2023 05:52:30", "content": "

Did you set a random state for reproducibility?

\n
 random\nrandom.seed() \n
", "votes": null, "replies": [ { "id": 2253137, "author_name": "xianzwaikato", "author_url": "", "post_date": "05/10/2023 00:52:32", "content": "

Not really ，it‘s totally a counterpart.

\n

i think it's different between on data upadte and before by the staff, i just knew a few hours before.

\n

But what it show Is interesting, that the old version data bring a better model, does it menan our trainng set not so correlate with the test data?

\n

Still thanks for the replay

", "votes": null, "replies": [] } ] }, { "id": 2254422, "author_name": "bechirjamousi", "author_url": "", "post_date": "05/11/2023 00:59:49", "content": "

Same here, I am facing the same issue. And the score difference is huge: 3.10 -> 2.62

", "votes": null, "replies": [ { "id": 2254479, "author_name": "xianzwaikato", "author_url": "", "post_date": "05/11/2023 03:13:24", "content": "

There is a guess, the test data is not update, so the test data is related to the old train data. lt is why the score is decrease, and not idea the test will be update on the end of the deadline

", "votes": null, "replies": [ { "id": 2254668, "author_name": "bechirjamousi", "author_url": "", "post_date": "05/11/2023 06:43:44", "content": "

What is weird is that when they updated the data they re-ran the notebooks and my notebook lost only 0.06 but when I re-run it manually it's getting less than 3.

", "votes": null, "replies": [] } ] } ] }, { "id": 2254554, "author_name": "ahsuna123", "author_url": "", "post_date": "05/11/2023 04:40:41", "content": "

I am facing the same issue. Checked multiple times if there is some difference in the two notebooks. Nada and yet one is giving 0.286 and the other 0.302.

", "votes": null, "replies": [] }, { "id": 2254670, "author_name": "bechirjamousi", "author_url": "", "post_date": "05/11/2023 06:44:29", "content": "

@ryanholbrook do you have an explanation for this please? Thank you in advance

", "votes": null, "replies": [ { "id": 2254968, "author_name": "ryanholbrook", "author_url": "", "post_date": "05/11/2023 11:42:29", "content": "

Hi @bechirjamousi,

\n

In the update only the target labels in the test data changed, not the test data itself that your model uses as inputs. After the update I rescored all of the submissions against the new targets. However, I did not rerun all of the notebooks from scratch to produce a new submission. If your submission notebook only made use of the test data, there should be no change from the new score after submitting it again. I'd recommend retraining the model on the new data and resubmitting.

", "votes": null, "replies": [ { "id": 2255023, "author_name": "bechirjamousi", "author_url": "", "post_date": "05/11/2023 12:54:23", "content": "

Thank you for the fast reply. It’s weird, that’s what I did but my score is still down.

", "votes": null, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "2250985": "is wried, one of version of my notebook have a score 2.17. But when i copied it, then rerun the model, I only got 1.87. \n\nsince i was copied the version, and did not made change, why they will have the different score??\n\nIfound that 2 models have different feature importance\n\noriginal:\n0 Time 0.465456\n3 AccAP 0.346994\n1 AccV 0.074586\n4 s_kmeans_1 0.050755\n5 s_kmeans_3 0.040626\n2 AccML 0.021584\n\n\ncoppied:\n0 Time 0.483317\n3 AccAP 0.347417\n5 s_kmeans_3 0.100661\n1 AccV 0.039379\n2 AccML 0.017303\n4 s_kmeans_1 0.011923\n\n\n\nbut i still can't unstand, since i did not make any change on the data or model parameter.", "2251149": "Did you set a random state for reproducibility?\n\n```python\nimport random\nrandom.seed(42) # for example\n```", "2253137": "Not really ，it‘s totally a counterpart.\n\ni think it's different between on data upadte and before by the staff, i just knew a few hours before.\n\nBut what it show Is interesting, that the old version data bring a better model, does it menan our trainng set not so correlate with the test data?\n\nStill thanks for the replay", "2254422": "Same here, I am facing the same issue. And the score difference is huge: 3.10 -> 2.62", "2254479": "There is a guess, the test data is not update, so the test data is related to the old train data. lt is why the score is decrease, and not idea the test will be update on the end of the deadline", "2254554": "I am facing the same issue. Checked multiple times if there is some difference in the two notebooks. Nada and yet one is giving 0.286 and the other 0.302.", "2254668": "What is weird is that when they updated the data they re-ran the notebooks and my notebook lost only 0.06 but when I re-run it manually it's getting less than 3.", "2254670": "ryanholbrook do you have an explanation for this please? Thank you in advance", "2254968": "Hi @bechirjamousi,\n\nIn the update only the target labels in the test data changed, not the test data itself that your model uses as inputs. After the update I rescored all of the submissions against the new targets. However, I did not rerun all of the notebooks from scratch to produce a new submission. If your submission notebook only made use of the test data, there should be no change from the new score after submitting it again. I'd recommend retraining the model on the new data and resubmitting.", "2255023": "Thank you for the fast reply. It’s weird, that’s what I did but my score is still down." }, "source": "meta" }