![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F20631422%2F383d0c0855da48c2f5c323d1094a0118%2F2024-12-18%20125651.png?generation=1734494270734023&alt=media\")
I tried to find out why the LB score changes dramatically when we change the random seeds.
\nIf the reason is just luck, even if a model gets great score on one test dataset, it may get bad score on another test dataset. On the other hand, if the fitting ability of the model actually depend on the random seeds, when it get great score on one test dataset, it is likely to get good score on another test dataset too.
\nI used Yu Yang Chang's notebook: https://www.kaggle.com/code/cchangyyy/0-494-notebook to research.
I divided the train data into four groups, train set, 1st validation set, 2nd validation set, 3rd validation set. The train set is two thirds of full train data, and the size of each validation set is one ninth of full data. And I did the division in three ways. The following picture shows the division ways.
\n
I trained the model with train set, using 41 different seeds, and did validation with the three validation sets separately. And calculated the correlation coefficients of the scores came from each set.
\nHere's the result.
Division 1
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F20631422%2Fbbb616fb58cffc192611e1f102649b75%2F2024-12-18%20124034.png?generation=1734494132836606&alt=media\")
Division 2
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F20631422%2F477a70ee60d6d101868a27358b568fda%2F2024-12-18%20124045.png?generation=1734494678508784&alt=media\")
Division 3
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F20631422%2F38acd6058fbac78c6bf3beb4bb2644fd%2F2024-12-18%20124057.png?generation=1734494691998420&alt=media\")
The result in division 3 shows that there's actually a strong correlation between the scores. It means that some random seeds didn't let the model fit the train data well, and some random seeds did.
\nHowever, the results in division 1 and 2 didn't show such a correlation. This is weird. I'd appreciate if you could do some research about this topic.
", "messages": [ { "id": "3074779", "postDate": "12/18/2024 04:22:00", "content": "I tried to find out why the LB score changes dramatically when we change the random seeds.
\nIf the reason is just luck, even if a model gets great score on one test dataset, it may get bad score on another test dataset. On the other hand, if the fitting ability of the model actually depend on the random seeds, when it get great score on one test dataset, it is likely to get good score on another test dataset too.
\nI used Yu Yang Chang's notebook: https://www.kaggle.com/code/cchangyyy/0-494-notebook to research.
I divided the train data into four groups, train set, 1st validation set, 2nd validation set, 3rd validation set. The train set is two thirds of full train data, and the size of each validation set is one ninth of full data. And I did the division in three ways. The following picture shows the division ways.
\n
I trained the model with train set, using 41 different seeds, and did validation with the three validation sets separately. And calculated the correlation coefficients of the scores came from each set.
\nHere's the result.
Division 1
\n
Division 2
\n
Division 3
\n
The result in division 3 shows that there's actually a strong correlation between the scores. It means that some random seeds didn't let the model fit the train data well, and some random seeds did.
\nHowever, the results in division 1 and 2 didn't show such a correlation. This is weird. I'd appreciate if you could do some research about this topic.
", "rawMarkdown": "I tried to find out why the LB score changes dramatically when we change the random seeds.\nIf the reason is just luck, even if a model gets great score on one test dataset, it may get bad score on another test dataset. On the other hand, if the fitting ability of the model actually depend on the random seeds, when it get great score on one test dataset, it is likely to get good score on another test dataset too.\nI used Yu Yang Chang's notebook: https://www.kaggle.com/code/cchangyyy/0-494-notebook to research.\n\nI divided the train data into four groups, train set, 1st validation set, 2nd validation set, 3rd validation set. The train set is two thirds of full train data, and the size of each validation set is one ninth of full data. And I did the division in three ways. The following picture shows the division ways.\n\n\n\nI trained the model with train set, using 41 different seeds, and did validation with the three validation sets separately. And calculated the correlation coefficients of the scores came from each set.\nHere's the result.\n\n**Division 1**\n\n\n**Division 2**\n\n\n**Division 3**\n\n\nThe result in division 3 shows that there's actually a strong correlation between the scores. It means that some random seeds didn't let the model fit the train data well, and some random seeds did.\n\nHowever, the results in division 1 and 2 didn't show such a correlation. This is weird. I'd appreciate if you could do some research about this topic.", "votes": null }, { "id": "3074790", "postDate": "12/18/2024 04:33:08", "content": "Additional information: the raw data
\nThe rows are random seeds and the columns are the numbers of the validation sets.
Division 1
\n[[0.44655614, 0.40265133, 0.42577484],
\n [0.43938948, 0.39924151, 0.42652542],
\n [0.45226249, 0.38764518, 0.43750977],
\n [0.45466004, 0.39070161, 0.40012332],
\n [0.45945535, 0.3965368, 0.41594502],
\n [0.45559757, 0.40886301, 0.44428786],
\n [0.42760569, 0.39108826, 0.42363855],
\n [0.46460268, 0.38165833, 0.44117555],
\n [0.40376182, 0.38550401, 0.38650026],
\n [0.44364263, 0.37873754, 0.41930189],
\n [0.43282491, 0.35580857, 0.41956185],
\n [0.40572587, 0.40886948, 0.41880023],
\n [0.45915958, 0.40478224, 0.43184945],
\n [0.44462038, 0.41059931, 0.4326625],
\n [0.46549327, 0.40177563, 0.42513733],
\n [0.46730515, 0.40192771, 0.42674966],
\n [0.4518205, 0.38344784, 0.4194876],
\n [0.43910115, 0.39979533, 0.42634428],
\n [0.44322529, 0.37268325, 0.41139453],
\n [0.45592021, 0.39275237, 0.39878105],
\n [0.438342, 0.34271193, 0.39135484],
\n [0.43638604, 0.4161556, 0.38873761],
\n [0.45109522, 0.39643278, 0.40573066],
\n [0.40066256, 0.39637898, 0.42508019],
\n [0.43858364, 0.4048974, 0.43254909],
\n [0.43474474, 0.39632773, 0.38541107],
\n [0.44073707, 0.36464941, 0.42266437],
\n [0.43799172, 0.38346667, 0.41167445],
\n [0.44213753, 0.38049615, 0.39974758],
\n [0.44594678, 0.38220174, 0.41647975],
\n [0.44353609, 0.39624556, 0.41855811],
\n [0.44808116, 0.39938556, 0.41368951],
\n [0.44451008, 0.3957412, 0.4276843],
\n [0.45147831, 0.37237237, 0.43121018],
\n [0.43392841, 0.3863993, 0.4113189],
\n [0.44716545, 0.38756374, 0.4320298],
\n [0.40869314, 0.3760323, 0.41031149],
\n [0.4465637, 0.37137778, 0.39937243],
\n [0.46814871, 0.38963925, 0.42529409],
\n [0.44569854, 0.37848285, 0.42182732],
\n [0.44396916, 0.39192196, 0.43574327]]\"
Division 2
\n[[0.40040989, 0.49925732, 0.38330228],
\n [0.37633394, 0.49847809, 0.39579222],
\n [0.38506787, 0.5018225, 0.38720848],
\n [0.40154607, 0.49991163, 0.39686087],
\n [0.38292322, 0.52568353, 0.40175034],
\n [0.44633017, 0.48310715, 0.41309434],
\n [0.39398637, 0.49441101, 0.40808516],
\n [0.36049099, 0.48295879, 0.31996852],
\n [0.37852131, 0.50400048, 0.39819889],
\n [0.41654233, 0.50896356, 0.3980169],
\n [0.41510856, 0.48390848, 0.41832921],
\n [0.39874684, 0.51037231, 0.41766399],
\n [0.38899721, 0.48704442, 0.43390842],
\n [0.45308505, 0.49138825, 0.40626807],
\n [0.41259421, 0.52156695, 0.40181404],
\n [0.41110062, 0.49195904, 0.42621214],
\n [0.39791179, 0.48692042, 0.39050084],
\n [0.39149505, 0.51530791, 0.39819059],
\n [0.39580068, 0.49220237, 0.40661956],
\n [0.39359841, 0.48103932, 0.41313332],
\n [0.40311557, 0.49424357, 0.41775115],
\n [0.36975667, 0.47190946, 0.41279624],
\n [0.35433853, 0.50804511, 0.39345885],
\n [0.40646949, 0.50594994, 0.41786064],
\n [0.41116549, 0.512501, 0.42812967],
\n [0.43335476, 0.53170198, 0.39035332],
\n [0.38889713, 0.48213029, 0.40387337],
\n [0.41435859, 0.49318215, 0.41692188],
\n [0.41721343, 0.52061582, 0.37624205],
\n [0.41088231, 0.52450559, 0.37538198],
\n [0.40171268, 0.4925826, 0.40621754],
\n [0.36923077, 0.46000302, 0.40660204],
\n [0.41260373, 0.47474788, 0.40954342],
\n [0.41187926, 0.50019624, 0.41471762],
\n [0.44867166, 0.50674327, 0.39213572],
\n [0.38540712, 0.5080551, 0.41146048],
\n [0.43368149, 0.50655738, 0.39171812],
\n [0.35579116, 0.50152791, 0.41412312],
\n [0.38881469, 0.50217316, 0.38358321],
\n [0.40160953, 0.48664955, 0.39574332],
\n [0.37298326, 0.48364424, 0.41507152]]
Division 3
\n[[0.48998871, 0.49598749, 0.41278398],
\n [0.50162859, 0.49851213, 0.42115839],
\n [0.46918678, 0.48017966, 0.41520173],
\n [0.48634788, 0.4920295, 0.41542151],
\n [0.48452486, 0.50279194, 0.45175377],
\n [0.48213131, 0.51733087, 0.44822876],
\n [0.43733943, 0.46430416, 0.41616259],
\n [0.4474766, 0.44603517, 0.42345305],
\n [0.45066799, 0.5060551, 0.44472907],
\n [0.42870687, 0.42610055, 0.40946432],
\n [0.46513369, 0.47507156, 0.42581648],
\n [0.44971788, 0.42317725, 0.36416119],
\n [0.49745207, 0.47060647, 0.42579329],
\n [0.48774613, 0.51158454, 0.42222291],
\n [0.48984925, 0.51056278, 0.41378772],
\n [0.4718397, 0.51271206, 0.43054518],
\n [0.46051461, 0.48807974, 0.44487494],
\n [0.49455965, 0.47638171, 0.44593292],
\n [0.461136, 0.48854409, 0.44426224],
\n [0.3950901, 0.43703443, 0.38812547],
\n [0.42348264, 0.4453751, 0.40410056],
\n [0.48196429, 0.46762088, 0.4145032],
\n [0.47692193, 0.5047723, 0.42736193],
\n [0.47141453, 0.5108349, 0.41767486],
\n [0.4753405, 0.52101924, 0.43601137],
\n [0.4635814, 0.53200413, 0.45168274],
\n [0.42596339, 0.42842064, 0.40358988],
\n [0.39545601, 0.44723646, 0.3944218],
\n [0.47037247, 0.45736229, 0.44722485],
\n [0.45128245, 0.48811942, 0.43283986],
\n [0.42596161, 0.43567523, 0.40789264],
\n [0.47948745, 0.47030033, 0.46440856],
\n [0.4878356, 0.46670863, 0.46240204],
\n [0.40756693, 0.43440888, 0.386949],
\n [0.45131151, 0.48830397, 0.42550901],
\n [0.41703242, 0.43761524, 0.39944729],
\n [0.47237268, 0.48543404, 0.45598015],
\n [0.48765681, 0.49973248, 0.43358857],
\n [0.46967769, 0.52006981, 0.40397092],
\n [0.48037833, 0.51954978, 0.45338543],
\n [0.3770562, 0.44456417, 0.37270666]]
There is no guarantee of good seeds will work on private test set. That's why we use multiple seeds and take the average of predictions. This stabilizes the score and reduces the variance.
", "rawMarkdown": "There is no guarantee of good seeds will work on private test set. That's why we use multiple seeds and take the average of predictions. This stabilizes the score and reduces the variance.", "votes": null }, { "id": "3074816", "postDate": "12/18/2024 05:27:15", "content": "Such cherry picking may not generalize @ykawakita
\nIt is better advised to discuss the impact of your strategy in 2 days when the private LB is disclosed.
Thank you for your advice! I'll wait until 2 days later before starting to think about how well I did in this competition.
", "rawMarkdown": "Thank you for your advice! I'll wait until 2 days later before starting to think about how well I did in this competition.", "votes": null }, { "id": "3074972", "postDate": "12/18/2024 09:28:41", "content": "I tried the approach of multiple seeds. It actually stabilized the score. However, the average score of the notebooks with multiple seeds was almost the same as the average score of the notebooks with one seed. I think in this case, we just lose the chance of accidentally getting a score much higher than the model's actual performance. And we probably get a lower private score than the competitors who used only one seed but were lucky. This is the first time I joined in such a competition where the LB score is unstable, so I'm not very confident in this opinion. What do you think about it?
", "rawMarkdown": "I tried the approach of multiple seeds. It actually stabilized the score. However, the average score of the notebooks with multiple seeds was almost the same as the average score of the notebooks with one seed. I think in this case, we just lose the chance of accidentally getting a score much higher than the model's actual performance. And we probably get a lower private score than the competitors who used only one seed but were lucky. This is the first time I joined in such a competition where the LB score is unstable, so I'm not very confident in this opinion. What do you think about it?", "votes": null } ], "comments": [ { "id": 3074790, "author_name": "ykawakita", "author_url": "", "post_date": "12/18/2024 04:33:08", "content": "Additional information: the raw data
\nThe rows are random seeds and the columns are the numbers of the validation sets.
Division 1
\n[[0.44655614, 0.40265133, 0.42577484],
\n [0.43938948, 0.39924151, 0.42652542],
\n [0.45226249, 0.38764518, 0.43750977],
\n [0.45466004, 0.39070161, 0.40012332],
\n [0.45945535, 0.3965368, 0.41594502],
\n [0.45559757, 0.40886301, 0.44428786],
\n [0.42760569, 0.39108826, 0.42363855],
\n [0.46460268, 0.38165833, 0.44117555],
\n [0.40376182, 0.38550401, 0.38650026],
\n [0.44364263, 0.37873754, 0.41930189],
\n [0.43282491, 0.35580857, 0.41956185],
\n [0.40572587, 0.40886948, 0.41880023],
\n [0.45915958, 0.40478224, 0.43184945],
\n [0.44462038, 0.41059931, 0.4326625],
\n [0.46549327, 0.40177563, 0.42513733],
\n [0.46730515, 0.40192771, 0.42674966],
\n [0.4518205, 0.38344784, 0.4194876],
\n [0.43910115, 0.39979533, 0.42634428],
\n [0.44322529, 0.37268325, 0.41139453],
\n [0.45592021, 0.39275237, 0.39878105],
\n [0.438342, 0.34271193, 0.39135484],
\n [0.43638604, 0.4161556, 0.38873761],
\n [0.45109522, 0.39643278, 0.40573066],
\n [0.40066256, 0.39637898, 0.42508019],
\n [0.43858364, 0.4048974, 0.43254909],
\n [0.43474474, 0.39632773, 0.38541107],
\n [0.44073707, 0.36464941, 0.42266437],
\n [0.43799172, 0.38346667, 0.41167445],
\n [0.44213753, 0.38049615, 0.39974758],
\n [0.44594678, 0.38220174, 0.41647975],
\n [0.44353609, 0.39624556, 0.41855811],
\n [0.44808116, 0.39938556, 0.41368951],
\n [0.44451008, 0.3957412, 0.4276843],
\n [0.45147831, 0.37237237, 0.43121018],
\n [0.43392841, 0.3863993, 0.4113189],
\n [0.44716545, 0.38756374, 0.4320298],
\n [0.40869314, 0.3760323, 0.41031149],
\n [0.4465637, 0.37137778, 0.39937243],
\n [0.46814871, 0.38963925, 0.42529409],
\n [0.44569854, 0.37848285, 0.42182732],
\n [0.44396916, 0.39192196, 0.43574327]]\"
Division 2
\n[[0.40040989, 0.49925732, 0.38330228],
\n [0.37633394, 0.49847809, 0.39579222],
\n [0.38506787, 0.5018225, 0.38720848],
\n [0.40154607, 0.49991163, 0.39686087],
\n [0.38292322, 0.52568353, 0.40175034],
\n [0.44633017, 0.48310715, 0.41309434],
\n [0.39398637, 0.49441101, 0.40808516],
\n [0.36049099, 0.48295879, 0.31996852],
\n [0.37852131, 0.50400048, 0.39819889],
\n [0.41654233, 0.50896356, 0.3980169],
\n [0.41510856, 0.48390848, 0.41832921],
\n [0.39874684, 0.51037231, 0.41766399],
\n [0.38899721, 0.48704442, 0.43390842],
\n [0.45308505, 0.49138825, 0.40626807],
\n [0.41259421, 0.52156695, 0.40181404],
\n [0.41110062, 0.49195904, 0.42621214],
\n [0.39791179, 0.48692042, 0.39050084],
\n [0.39149505, 0.51530791, 0.39819059],
\n [0.39580068, 0.49220237, 0.40661956],
\n [0.39359841, 0.48103932, 0.41313332],
\n [0.40311557, 0.49424357, 0.41775115],
\n [0.36975667, 0.47190946, 0.41279624],
\n [0.35433853, 0.50804511, 0.39345885],
\n [0.40646949, 0.50594994, 0.41786064],
\n [0.41116549, 0.512501, 0.42812967],
\n [0.43335476, 0.53170198, 0.39035332],
\n [0.38889713, 0.48213029, 0.40387337],
\n [0.41435859, 0.49318215, 0.41692188],
\n [0.41721343, 0.52061582, 0.37624205],
\n [0.41088231, 0.52450559, 0.37538198],
\n [0.40171268, 0.4925826, 0.40621754],
\n [0.36923077, 0.46000302, 0.40660204],
\n [0.41260373, 0.47474788, 0.40954342],
\n [0.41187926, 0.50019624, 0.41471762],
\n [0.44867166, 0.50674327, 0.39213572],
\n [0.38540712, 0.5080551, 0.41146048],
\n [0.43368149, 0.50655738, 0.39171812],
\n [0.35579116, 0.50152791, 0.41412312],
\n [0.38881469, 0.50217316, 0.38358321],
\n [0.40160953, 0.48664955, 0.39574332],
\n [0.37298326, 0.48364424, 0.41507152]]
Division 3
\n[[0.48998871, 0.49598749, 0.41278398],
\n [0.50162859, 0.49851213, 0.42115839],
\n [0.46918678, 0.48017966, 0.41520173],
\n [0.48634788, 0.4920295, 0.41542151],
\n [0.48452486, 0.50279194, 0.45175377],
\n [0.48213131, 0.51733087, 0.44822876],
\n [0.43733943, 0.46430416, 0.41616259],
\n [0.4474766, 0.44603517, 0.42345305],
\n [0.45066799, 0.5060551, 0.44472907],
\n [0.42870687, 0.42610055, 0.40946432],
\n [0.46513369, 0.47507156, 0.42581648],
\n [0.44971788, 0.42317725, 0.36416119],
\n [0.49745207, 0.47060647, 0.42579329],
\n [0.48774613, 0.51158454, 0.42222291],
\n [0.48984925, 0.51056278, 0.41378772],
\n [0.4718397, 0.51271206, 0.43054518],
\n [0.46051461, 0.48807974, 0.44487494],
\n [0.49455965, 0.47638171, 0.44593292],
\n [0.461136, 0.48854409, 0.44426224],
\n [0.3950901, 0.43703443, 0.38812547],
\n [0.42348264, 0.4453751, 0.40410056],
\n [0.48196429, 0.46762088, 0.4145032],
\n [0.47692193, 0.5047723, 0.42736193],
\n [0.47141453, 0.5108349, 0.41767486],
\n [0.4753405, 0.52101924, 0.43601137],
\n [0.4635814, 0.53200413, 0.45168274],
\n [0.42596339, 0.42842064, 0.40358988],
\n [0.39545601, 0.44723646, 0.3944218],
\n [0.47037247, 0.45736229, 0.44722485],
\n [0.45128245, 0.48811942, 0.43283986],
\n [0.42596161, 0.43567523, 0.40789264],
\n [0.47948745, 0.47030033, 0.46440856],
\n [0.4878356, 0.46670863, 0.46240204],
\n [0.40756693, 0.43440888, 0.386949],
\n [0.45131151, 0.48830397, 0.42550901],
\n [0.41703242, 0.43761524, 0.39944729],
\n [0.47237268, 0.48543404, 0.45598015],
\n [0.48765681, 0.49973248, 0.43358857],
\n [0.46967769, 0.52006981, 0.40397092],
\n [0.48037833, 0.51954978, 0.45338543],
\n [0.3770562, 0.44456417, 0.37270666]]
Such cherry picking may not generalize @ykawakita
\nIt is better advised to discuss the impact of your strategy in 2 days when the private LB is disclosed.
Thank you for your advice! I'll wait until 2 days later before starting to think about how well I did in this competition.
", "votes": null, "replies": [] } ] } ] }, { "id": 3074800, "author_name": "gunesevitan", "author_url": "", "post_date": "12/18/2024 04:55:31", "content": "There is no guarantee of good seeds will work on private test set. That's why we use multiple seeds and take the average of predictions. This stabilizes the score and reduces the variance.
", "votes": null, "replies": [ { "id": 3074972, "author_name": "ykawakita", "author_url": "", "post_date": "12/18/2024 09:28:41", "content": "I tried the approach of multiple seeds. It actually stabilized the score. However, the average score of the notebooks with multiple seeds was almost the same as the average score of the notebooks with one seed. I think in this case, we just lose the chance of accidentally getting a score much higher than the model's actual performance. And we probably get a lower private score than the competitors who used only one seed but were lucky. This is the first time I joined in such a competition where the LB score is unstable, so I'm not very confident in this opinion. What do you think about it?
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "3074779": "I tried to find out why the LB score changes dramatically when we change the random seeds.\nIf the reason is just luck, even if a model gets great score on one test dataset, it may get bad score on another test dataset. On the other hand, if the fitting ability of the model actually depend on the random seeds, when it get great score on one test dataset, it is likely to get good score on another test dataset too.\nI used Yu Yang Chang's notebook: https://www.kaggle.com/code/cchangyyy/0-494-notebook to research.\n\nI divided the train data into four groups, train set, 1st validation set, 2nd validation set, 3rd validation set. The train set is two thirds of full train data, and the size of each validation set is one ninth of full data. And I did the division in three ways. The following picture shows the division ways.\n\n\n\nI trained the model with train set, using 41 different seeds, and did validation with the three validation sets separately. And calculated the correlation coefficients of the scores came from each set.\nHere's the result.\n\n**Division 1**\n\n\n**Division 2**\n\n\n**Division 3**\n\n\nThe result in division 3 shows that there's actually a strong correlation between the scores. It means that some random seeds didn't let the model fit the train data well, and some random seeds did.\n\nHowever, the results in division 1 and 2 didn't show such a correlation. This is weird. I'd appreciate if you could do some research about this topic.", "3074790": "Additional information: the raw data\nThe rows are random seeds and the columns are the numbers of the validation sets.\n\n**Division 1**\n[[0.44655614, 0.40265133, 0.42577484],\n [0.43938948, 0.39924151, 0.42652542],\n [0.45226249, 0.38764518, 0.43750977],\n [0.45466004, 0.39070161, 0.40012332],\n [0.45945535, 0.3965368, 0.41594502],\n [0.45559757, 0.40886301, 0.44428786],\n [0.42760569, 0.39108826, 0.42363855],\n [0.46460268, 0.38165833, 0.44117555],\n [0.40376182, 0.38550401, 0.38650026],\n [0.44364263, 0.37873754, 0.41930189],\n [0.43282491, 0.35580857, 0.41956185],\n [0.40572587, 0.40886948, 0.41880023],\n [0.45915958, 0.40478224, 0.43184945],\n [0.44462038, 0.41059931, 0.4326625],\n [0.46549327, 0.40177563, 0.42513733],\n [0.46730515, 0.40192771, 0.42674966],\n [0.4518205, 0.38344784, 0.4194876],\n [0.43910115, 0.39979533, 0.42634428],\n [0.44322529, 0.37268325, 0.41139453],\n [0.45592021, 0.39275237, 0.39878105],\n [0.438342, 0.34271193, 0.39135484],\n [0.43638604, 0.4161556, 0.38873761],\n [0.45109522, 0.39643278, 0.40573066],\n [0.40066256, 0.39637898, 0.42508019],\n [0.43858364, 0.4048974, 0.43254909],\n [0.43474474, 0.39632773, 0.38541107],\n [0.44073707, 0.36464941, 0.42266437],\n [0.43799172, 0.38346667, 0.41167445],\n [0.44213753, 0.38049615, 0.39974758],\n [0.44594678, 0.38220174, 0.41647975],\n [0.44353609, 0.39624556, 0.41855811],\n [0.44808116, 0.39938556, 0.41368951],\n [0.44451008, 0.3957412, 0.4276843],\n [0.45147831, 0.37237237, 0.43121018],\n [0.43392841, 0.3863993, 0.4113189],\n [0.44716545, 0.38756374, 0.4320298],\n [0.40869314, 0.3760323, 0.41031149],\n [0.4465637, 0.37137778, 0.39937243],\n [0.46814871, 0.38963925, 0.42529409],\n [0.44569854, 0.37848285, 0.42182732],\n [0.44396916, 0.39192196, 0.43574327]]\"\n\n\n**Division 2**\n[[0.40040989, 0.49925732, 0.38330228],\n [0.37633394, 0.49847809, 0.39579222],\n [0.38506787, 0.5018225, 0.38720848],\n [0.40154607, 0.49991163, 0.39686087],\n [0.38292322, 0.52568353, 0.40175034],\n [0.44633017, 0.48310715, 0.41309434],\n [0.39398637, 0.49441101, 0.40808516],\n [0.36049099, 0.48295879, 0.31996852],\n [0.37852131, 0.50400048, 0.39819889],\n [0.41654233, 0.50896356, 0.3980169],\n [0.41510856, 0.48390848, 0.41832921],\n [0.39874684, 0.51037231, 0.41766399],\n [0.38899721, 0.48704442, 0.43390842],\n [0.45308505, 0.49138825, 0.40626807],\n [0.41259421, 0.52156695, 0.40181404],\n [0.41110062, 0.49195904, 0.42621214],\n [0.39791179, 0.48692042, 0.39050084],\n [0.39149505, 0.51530791, 0.39819059],\n [0.39580068, 0.49220237, 0.40661956],\n [0.39359841, 0.48103932, 0.41313332],\n [0.40311557, 0.49424357, 0.41775115],\n [0.36975667, 0.47190946, 0.41279624],\n [0.35433853, 0.50804511, 0.39345885],\n [0.40646949, 0.50594994, 0.41786064],\n [0.41116549, 0.512501, 0.42812967],\n [0.43335476, 0.53170198, 0.39035332],\n [0.38889713, 0.48213029, 0.40387337],\n [0.41435859, 0.49318215, 0.41692188],\n [0.41721343, 0.52061582, 0.37624205],\n [0.41088231, 0.52450559, 0.37538198],\n [0.40171268, 0.4925826, 0.40621754],\n [0.36923077, 0.46000302, 0.40660204],\n [0.41260373, 0.47474788, 0.40954342],\n [0.41187926, 0.50019624, 0.41471762],\n [0.44867166, 0.50674327, 0.39213572],\n [0.38540712, 0.5080551, 0.41146048],\n [0.43368149, 0.50655738, 0.39171812],\n [0.35579116, 0.50152791, 0.41412312],\n [0.38881469, 0.50217316, 0.38358321],\n [0.40160953, 0.48664955, 0.39574332],\n [0.37298326, 0.48364424, 0.41507152]]\n\n\n**Division 3**\n[[0.48998871, 0.49598749, 0.41278398],\n [0.50162859, 0.49851213, 0.42115839],\n [0.46918678, 0.48017966, 0.41520173],\n [0.48634788, 0.4920295, 0.41542151],\n [0.48452486, 0.50279194, 0.45175377],\n [0.48213131, 0.51733087, 0.44822876],\n [0.43733943, 0.46430416, 0.41616259],\n [0.4474766, 0.44603517, 0.42345305],\n [0.45066799, 0.5060551, 0.44472907],\n [0.42870687, 0.42610055, 0.40946432],\n [0.46513369, 0.47507156, 0.42581648],\n [0.44971788, 0.42317725, 0.36416119],\n [0.49745207, 0.47060647, 0.42579329],\n [0.48774613, 0.51158454, 0.42222291],\n [0.48984925, 0.51056278, 0.41378772],\n [0.4718397, 0.51271206, 0.43054518],\n [0.46051461, 0.48807974, 0.44487494],\n [0.49455965, 0.47638171, 0.44593292],\n [0.461136, 0.48854409, 0.44426224],\n [0.3950901, 0.43703443, 0.38812547],\n [0.42348264, 0.4453751, 0.40410056],\n [0.48196429, 0.46762088, 0.4145032],\n [0.47692193, 0.5047723, 0.42736193],\n [0.47141453, 0.5108349, 0.41767486],\n [0.4753405, 0.52101924, 0.43601137],\n [0.4635814, 0.53200413, 0.45168274],\n [0.42596339, 0.42842064, 0.40358988],\n [0.39545601, 0.44723646, 0.3944218],\n [0.47037247, 0.45736229, 0.44722485],\n [0.45128245, 0.48811942, 0.43283986],\n [0.42596161, 0.43567523, 0.40789264],\n [0.47948745, 0.47030033, 0.46440856],\n [0.4878356, 0.46670863, 0.46240204],\n [0.40756693, 0.43440888, 0.386949],\n [0.45131151, 0.48830397, 0.42550901],\n [0.41703242, 0.43761524, 0.39944729],\n [0.47237268, 0.48543404, 0.45598015],\n [0.48765681, 0.49973248, 0.43358857],\n [0.46967769, 0.52006981, 0.40397092],\n [0.48037833, 0.51954978, 0.45338543],\n [0.3770562, 0.44456417, 0.37270666]]", "3074800": "There is no guarantee of good seeds will work on private test set. That's why we use multiple seeds and take the average of predictions. This stabilizes the score and reduces the variance.", "3074816": "Such cherry picking may not generalize @ykawakita \nIt is better advised to discuss the impact of your strategy in 2 days when the private LB is disclosed.", "3074956": "Thank you for your advice! I'll wait until 2 days later before starting to think about how well I did in this competition.", "3074972": "I tried the approach of multiple seeds. It actually stabilized the score. However, the average score of the notebooks with multiple seeds was almost the same as the average score of the notebooks with one seed. I think in this case, we just lose the chance of accidentally getting a score much higher than the model's actual performance. And we probably get a lower private score than the competitors who used only one seed but were lucky. This is the first time I joined in such a competition where the LB score is unstable, so I'm not very confident in this opinion. What do you think about it?" }, "source": "meta" }