![\"nlp](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F10671485%2Fccc914dcad5f07cf3c93a38fe4510537%2F_20231117165331.png?generation=1700268827943621&alt=media\")
Has anyone else encountered this situation? Using the prediction results of random as the default results can greatly improve both lb and pb. However, if the model is used to directly predict the default, the effect will be much worse. Something wrong with the test data? What is the reason for this situation?😮😮😮
For the below one, nlp random was replaced by nlp default, but got higher score on both public leaderboard and private leaderboard.
Has anyone else encountered this situation? Using the prediction results of random as the default results can greatly improve both lb and pb. However, if the model is used to directly predict the default, the effect will be much worse. Something wrong with the test data? What is the reason for this situation?😮😮😮
For the below one, nlp random was replaced by nlp default, but got higher score on both public leaderboard and private leaderboard.
Yes, my CV and LB were also better when I copied the test predictions for the random sets to the default sets.
\nMaybe the random search strategy leads to more diverse configurations and is consequently better for training?
", "rawMarkdown": "Yes, my CV and LB were also better when I copied the test predictions for the random sets to the default sets.\n\nMaybe the random search strategy leads to more diverse configurations and is consequently better for training?", "votes": null }, { "id": "2529134", "postDate": "11/18/2023 01:21:25", "content": "Maybe, but the final result can be overfitting.😂
", "rawMarkdown": "Maybe, but the final result can be overfitting.😂", "votes": null }, { "id": "2529136", "postDate": "11/18/2023 01:22:54", "content": "i identify the common subset of config and node from xla:random and xla:default in the train set.
\nalthough the input is the same, the ground truth runtime is different. (the runtime ranking is also different)
Thanks for sharing. We didn't notice it before your sharing. I used prediction(pb: 0.710) results of random as the default results to make a late submission.The improve is amazing. And I have tried to use random model to predict default, but cv didn't improve. ![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1069377%2Fe29b89ec1f335d4aeae5fe097fd9fcad%2F_20231118091949.png?generation=1700270405776237&alt=media\")
Congratulations on winning the gold medal！
", "rawMarkdown": "Congratulations on winning the gold medal！", "votes": null }, { "id": "2529139", "postDate": "11/18/2023 01:28:00", "content": "oops! 0.794 is very good!
\n(is the same trend observed in the validation data?)
Yes, that might be the reason. Congratulations on winning the gold medal！
", "rawMarkdown": "Yes, that might be the reason. Congratulations on winning the gold medal！", "votes": null }, { "id": "2529143", "postDate": "11/18/2023 01:35:31", "content": "I tried it too (just copied the nlp/random result directly to the nlp/default result)
\nI guess that the order of correct answers is the same in random and default.
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3026173%2Fb0e581fcdd32428cac58eba953e0d737%2F2023-11-18%20103214.png?generation=1700271327130903&alt=media\")
i think i read in some paper that says the sample configuration from random is derived from default set using genetic algorithm.
\nhence random set can be seen as an augmentation from default set (in the input space)?
random set spans larger congfiguration space and should be more stable?
\nbut there is one issue when we consider the output space, i think the runtime is measured and subjected to measurement noise.
\n(i.e. if you run the graph many times the measured runtime is different. i am not sure if the provided runtime ground truth has been averaged over many runs or just a single run results)
Amazing😂!
", "rawMarkdown": "Amazing😂!", "votes": null }, { "id": "2529162", "postDate": "11/18/2023 02:09:30", "content": "My result 😂
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F9088007%2F73b51b2f637353fd60b751758c220bdc%2FF_Lns4dbcAAvDSn.jpeg?generation=1700273347843016&alt=media\")
We also only found out about this now. Probably some data preparation issue like other people suggested.
\nJust tried here and our score also improves a lot by doing it. Crazy that is unlikely that someone found this before the deadline, otherwise we probably would have seen scores above 0.8.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1648129%2Fff5732646fb6d386575d19989f0458e6%2Fss_score.png?generation=1700274112296160&alt=media\")
I confirm this observation, as I am looking at my code that generated the final dataset files. Indeed, the indices that sort the runtime of any nlp/random/test/<filename>.npz are the same indices that sort its corresponding nlp/default/test/<filename>.npz -- this was by (mal)design -- we took 1000 configurations, sorted them, then shuffled them by initializing a random seed that is based on the os.path.basename(<filename>) :D
Thank you for noticing it (we should've caught it ourselves) -- we should've shuffled on the entire relative path rather than only basename.
For my own curiosity, I can check if any of the prize winners submitted the indices for random and for default.
", "rawMarkdown": "I confirm this observation, as I am looking at my code that generated the final dataset files. Indeed, the **indices** that sort the runtime of any `nlp/random/test/Yes, you may check if any of the teams that won a prize used this method. This might indicate that their models are not necessarily robust enough, as it concerns the issue of prize money, which indeed requires some caution.
", "rawMarkdown": "Yes, you may check if any of the teams that won a prize used this method. This might indicate that their models are not necessarily robust enough, as it concerns the issue of prize money, which indeed requires some caution.", "votes": null }, { "id": "2529219", "postDate": "11/18/2023 03:55:26", "content": "There should also have been instances of data leakage in the history of Kaggle, but it seems that exploiting this is considered permissible.
", "rawMarkdown": "There should also have been instances of data leakage in the history of Kaggle, but it seems that exploiting this is considered permissible.", "votes": null }, { "id": "2529783", "postDate": "11/18/2023 14:38:12", "content": "Noted 👍. Thanks for sharing
", "rawMarkdown": "Noted 👍. Thanks for sharing", "votes": null }, { "id": "2530054", "postDate": "11/18/2023 18:38:31", "content": "Google is fast but not as human, because google created by humans
", "rawMarkdown": "Google is fast but not as human, because google created by humans", "votes": null }, { "id": "2530078", "postDate": "11/18/2023 18:53:16", "content": "Olala, that is quite something! Didn't know about it either and would raise my score quite a bit as well
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F10842317%2Fe0f32fff6f939f8437ac32c134b38e1d%2Fnew_score.png?generation=1700333516259810&alt=media\")
Great work! Congratulations on winning the gold medal！
\nActually I think most people didn't find this interesting thing. 🤔
Yes, my CV and LB were also better when I copied the test predictions for the random sets to the default sets.
\nMaybe the random search strategy leads to more diverse configurations and is consequently better for training?
", "votes": null, "replies": [ { "id": 2529134, "author_name": "lizhecheng", "author_url": "", "post_date": "11/18/2023 01:21:25", "content": "Maybe, but the final result can be overfitting.😂
", "votes": null, "replies": [] } ] }, { "id": 2529136, "author_name": "hengck23", "author_url": "", "post_date": "11/18/2023 01:22:54", "content": "i identify the common subset of config and node from xla:random and xla:default in the train set.
\nalthough the input is the same, the ground truth runtime is different. (the runtime ranking is also different)
Yes, that might be the reason. Congratulations on winning the gold medal！
", "votes": null, "replies": [] } ] }, { "id": 2529137, "author_name": "chenxin1991", "author_url": "", "post_date": "11/18/2023 01:23:39", "content": "Thanks for sharing. We didn't notice it before your sharing. I used prediction(pb: 0.710) results of random as the default results to make a late submission.The improve is amazing. And I have tried to use random model to predict default, but cv didn't improve.
Congratulations on winning the gold medal！
", "votes": null, "replies": [] }, { "id": 2529139, "author_name": "hengck23", "author_url": "", "post_date": "11/18/2023 01:28:00", "content": "oops! 0.794 is very good!
\n(is the same trend observed in the validation data?)
I tried it too (just copied the nlp/random result directly to the nlp/default result)
\nI guess that the order of correct answers is the same in random and default.
Amazing😂!
", "votes": null, "replies": [] } ] }, { "id": 2529149, "author_name": "hengck23", "author_url": "", "post_date": "11/18/2023 01:45:36", "content": "i think i read in some paper that says the sample configuration from random is derived from default set using genetic algorithm.
\nhence random set can be seen as an augmentation from default set (in the input space)?
random set spans larger congfiguration space and should be more stable?
\nbut there is one issue when we consider the output space, i think the runtime is measured and subjected to measurement noise.
\n(i.e. if you run the graph many times the measured runtime is different. i am not sure if the provided runtime ground truth has been averaged over many runs or just a single run results)
My result 😂
\n
We also only found out about this now. Probably some data preparation issue like other people suggested.
\nJust tried here and our score also improves a lot by doing it. Crazy that is unlikely that someone found this before the deadline, otherwise we probably would have seen scores above 0.8.
\n
I confirm this observation, as I am looking at my code that generated the final dataset files. Indeed, the indices that sort the runtime of any nlp/random/test/<filename>.npz are the same indices that sort its corresponding nlp/default/test/<filename>.npz -- this was by (mal)design -- we took 1000 configurations, sorted them, then shuffled them by initializing a random seed that is based on the os.path.basename(<filename>) :D
Thank you for noticing it (we should've caught it ourselves) -- we should've shuffled on the entire relative path rather than only basename.
For my own curiosity, I can check if any of the prize winners submitted the indices for random and for default.
", "votes": null, "replies": [ { "id": 2529212, "author_name": "lizhecheng", "author_url": "", "post_date": "11/18/2023 03:50:04", "content": "Yes, you may check if any of the teams that won a prize used this method. This might indicate that their models are not necessarily robust enough, as it concerns the issue of prize money, which indeed requires some caution.
", "votes": null, "replies": [] }, { "id": 2529219, "author_name": "lizhecheng", "author_url": "", "post_date": "11/18/2023 03:55:26", "content": "There should also have been instances of data leakage in the history of Kaggle, but it seems that exploiting this is considered permissible.
", "votes": null, "replies": [] } ] }, { "id": 2529783, "author_name": "ankitanain", "author_url": "", "post_date": "11/18/2023 14:38:12", "content": "Noted 👍. Thanks for sharing
", "votes": null, "replies": [] }, { "id": 2530054, "author_name": "vikrampython", "author_url": "", "post_date": "11/18/2023 18:38:31", "content": "Google is fast but not as human, because google created by humans
", "votes": null, "replies": [] }, { "id": 2530078, "author_name": "janisfluri", "author_url": "", "post_date": "11/18/2023 18:53:16", "content": "Olala, that is quite something! Didn't know about it either and would raise my score quite a bit as well
\n
Great work! Congratulations on winning the gold medal！
\nActually I think most people didn't find this interesting thing. 🤔