when i revise the public model i relize that the some highest scores Notebooks are using some architecture that build derivadede from autoencoder :
\nhttps://www.kaggle.com/code/honganzhu/cmi-piu-competition?scriptVersionId=201912528 #0.492
\nmy notebook: https://www.kaggle.com/code/saidkoussi/vae-pytorch-lightgbm-wih-cuda-df #0.490
what do you guess about the winning solutions?
", "messages": [ { "id": "3036508", "postDate": "11/04/2024 16:15:42", "content": "when i revise the public model i relize that the some highest scores Notebooks are using some architecture that build derivadede from autoencoder :
\nhttps://www.kaggle.com/code/honganzhu/cmi-piu-competition?scriptVersionId=201912528 #0.492
\nmy notebook: https://www.kaggle.com/code/saidkoussi/vae-pytorch-lightgbm-wih-cuda-df #0.490
what do you guess about the winning solutions?
", "rawMarkdown": "when i revise the public model i relize that the some highest scores Notebooks are using some architecture that build derivadede from autoencoder :\nhttps://www.kaggle.com/code/honganzhu/cmi-piu-competition?scriptVersionId=201912528 #0.492\nmy notebook: https://www.kaggle.com/code/saidkoussi/vae-pytorch-lightgbm-wih-cuda-df #0.490\n\nwhat do you guess about the winning solutions?", "votes": null }, { "id": "3036732", "postDate": "11/04/2024 21:33:42", "content": "Good question. It seems your final submission is a voting of submission 1,2,3, and only submission1 is using an autoencoder + KNN impute + feature engineering, and only time-series is autoencoded.
\nHave you tried using only submission1 and what's the public score?
\nYou can't tell unless you try doing the above mentioned one at a time.
\nI'm interested to know as well.
\nGood luck.
@tomyuen i tried to ensemble various tree-based models with various autoencoder architecture , but the scores are unstable, so we must be carefull about data leckage, focusing on just increasing leadboard can be disapoints in the private leadboard , i think we must build robust model that has high cv , leadboard score
", "rawMarkdown": "tomyuen i tried to ensemble various tree-based models with various autoencoder architecture , but the scores are unstable, so we must be carefull about data leckage, focusing on just increasing leadboard can be disapoints in the private leadboard , i think we must build robust model that has high cv , leadboard score", "votes": null }, { "id": "3037280", "postDate": "11/05/2024 14:58:18", "content": "I can tell you that the autoencoder is not correctly applied in these notebooks because it comes from the same copy-paste source where the original author made a mistake in applying it on data - so take it like a hint for debugging.
", "rawMarkdown": "I can tell you that the autoencoder is not correctly applied in these notebooks because it comes from the same copy-paste source where the original author made a mistake in applying it on data - so take it like a hint for debugging.", "votes": null }, { "id": "3037289", "postDate": "11/05/2024 15:09:03", "content": "I also used a voting strategy and tried to submit each submission separately.
\nSubmitting separately leads to worse results than using a voting strategy for the final submission. Maybe because voting between submissions reduces model bias which leads to better generalization, single models might vary widely in predictions for certain challenging cases.
\nI saw great improvements of my LB scores since I started voting between submissions.
\nWhen using voting strategies we also need to consider that voting can mask the underlying variability of each model’s performance on different data samples.
\nIn other words, the ensemble might look consistent on the public set, but individual models may vary widely on certain data patterns, indicating that the model ensemble’s true generalization ability may be weaker than it appears.
\nThis can have an impact in our case : Kaggle test splits (public vs. private) are typically designed to represent the distribution of the entire test set. However, there can still be subtle distributional shifts between the public and private splits due to random sampling.
\nIf the ensemble voting strategy unintentionally overfits to specific quirks in the public test data, it may generalize poorly on the private test set. The final score on the private leaderboard could then be lower than expected.
\nTo briefly conclude on this, voting currently leads to better leaderboard scores, but it doesn’t mean those scores will translate onto the final test set and leaderboard.
", "rawMarkdown": "I also used a voting strategy and tried to submit each submission separately. \n\nSubmitting separately leads to worse results than using a voting strategy for the final submission. Maybe because voting between submissions reduces model bias which leads to better generalization, single models might vary widely in predictions for certain challenging cases.\n\nI saw great improvements of my LB scores since I started voting between submissions.\n\nWhen using voting strategies we also need to consider that voting can mask the underlying variability of each model’s performance on different data samples.\n\nIn other words, the ensemble might look consistent on the public set, but individual models may vary widely on certain data patterns, indicating that the model ensemble’s true generalization ability may be weaker than it appears.\n\nThis can have an impact in our case : Kaggle test splits (public vs. private) are typically designed to represent the distribution of the entire test set. However, there can still be subtle distributional shifts between the public and private splits due to random sampling.\n\nIf the ensemble voting strategy unintentionally overfits to specific quirks in the public test data, it may generalize poorly on the private test set. The final score on the private leaderboard could then be lower than expected.\n\nTo briefly conclude on this, voting currently leads to better leaderboard scores, but it doesn’t mean those scores will translate onto the final test set and leaderboard.", "votes": null }, { "id": "3037309", "postDate": "11/05/2024 15:24:08", "content": "VotingRegressor doesn't mask anything - it returns the average of the results, it's just a fancy np.mean(your_results).
\nHere is the official description - \"A voting regressor is an ensemble meta-estimator that fits several base regressors, each on the whole dataset. Then it averages the individual predictions to form a final prediction.\"
You’re absolutely right that VotingRegressor performs a straightforward averaging of predictions and doesn’t inherently mask variability.
\nHowever, in this case, we’re not talking about VotingRegressors. Voting between submissions uses majority voting, which is different from averaging.
\nBy using majority voting rather than averaging, the aim is to reduce the impact of outlier predictions from individual models. This approach is particularly helpful for discrete ordinal predictions, where taking the mode can reduce the influence of individual model biases, as it discards predictions that diverge too much from the majority. This is less likely to happen with simple averaging.
\nI wish you good luck in this competition.
", "rawMarkdown": "You’re absolutely right that VotingRegressor performs a straightforward averaging of predictions and doesn’t inherently mask variability. \n\nHowever, in this case, we’re not talking about VotingRegressors. Voting between submissions uses majority voting, which is different from averaging.\n\nBy using majority voting rather than averaging, the aim is to reduce the impact of outlier predictions from individual models. This approach is particularly helpful for discrete ordinal predictions, where taking the mode can reduce the influence of individual model biases, as it discards predictions that diverge too much from the majority. This is less likely to happen with simple averaging. \n\nI wish you good luck in this competition.", "votes": null }, { "id": "3037335", "postDate": "11/05/2024 15:48:03", "content": "I think not always the autocoder can be the winner model.They might be the \"winner\" for tasks that rely on dimensionality reduction, feature extraction, or anomaly detection, especially when data is unlabeled.
", "rawMarkdown": "I think not always the autocoder can be the winner model.They might be the \"winner\" for tasks that rely on dimensionality reduction, feature extraction, or anomaly detection, especially when data is unlabeled.", "votes": null }, { "id": "3037591", "postDate": "11/05/2024 20:57:10", "content": "I frequently observed the GaussianNoise layer receiving input data at the start of the AutoEncoder architectures, but here it's skipped, perhaps for some reason. On the step of scaling, I also noticed that the Gauss Rank Scaler was used rather than other scalers. Another addition is a Swap Noise. These three things I associate with Denoising AutoEncoder primarily. Here, as I can follow, AutoEncoder does feature extraction, so noise is not necessary, although it may be useful as a form of regularization to learn a more robust representation and improve generalization since there is a lack of data, but the issue of unidentified randomness still remains, thus how to test anything is a question.
@sumit08 yes agree bout some portion of the data are unabled , so autoencoder may have the upper hand
", "rawMarkdown": "sumit08 yes agree bout some portion of the data are unabled , so autoencoder may have the upper hand", "votes": null }, { "id": "3041461", "postDate": "11/10/2024 11:47:24", "content": "Can you explain what is the mistake? thank you.
", "rawMarkdown": "Can you explain what is the mistake? thank you.", "votes": null } ], "comments": [ { "id": 3036732, "author_name": "tomyuen", "author_url": "", "post_date": "11/04/2024 21:33:42", "content": "Good question. It seems your final submission is a voting of submission 1,2,3, and only submission1 is using an autoencoder + KNN impute + feature engineering, and only time-series is autoencoded.
\nHave you tried using only submission1 and what's the public score?
\nYou can't tell unless you try doing the above mentioned one at a time.
\nI'm interested to know as well.
\nGood luck.
@tomyuen i tried to ensemble various tree-based models with various autoencoder architecture , but the scores are unstable, so we must be carefull about data leckage, focusing on just increasing leadboard can be disapoints in the private leadboard , i think we must build robust model that has high cv , leadboard score
", "votes": null, "replies": [] }, { "id": 3037289, "author_name": "alixnguyen", "author_url": "", "post_date": "11/05/2024 15:09:03", "content": "I also used a voting strategy and tried to submit each submission separately.
\nSubmitting separately leads to worse results than using a voting strategy for the final submission. Maybe because voting between submissions reduces model bias which leads to better generalization, single models might vary widely in predictions for certain challenging cases.
\nI saw great improvements of my LB scores since I started voting between submissions.
\nWhen using voting strategies we also need to consider that voting can mask the underlying variability of each model’s performance on different data samples.
\nIn other words, the ensemble might look consistent on the public set, but individual models may vary widely on certain data patterns, indicating that the model ensemble’s true generalization ability may be weaker than it appears.
\nThis can have an impact in our case : Kaggle test splits (public vs. private) are typically designed to represent the distribution of the entire test set. However, there can still be subtle distributional shifts between the public and private splits due to random sampling.
\nIf the ensemble voting strategy unintentionally overfits to specific quirks in the public test data, it may generalize poorly on the private test set. The final score on the private leaderboard could then be lower than expected.
\nTo briefly conclude on this, voting currently leads to better leaderboard scores, but it doesn’t mean those scores will translate onto the final test set and leaderboard.
", "votes": null, "replies": [ { "id": 3037309, "author_name": "eu1234", "author_url": "", "post_date": "11/05/2024 15:24:08", "content": "VotingRegressor doesn't mask anything - it returns the average of the results, it's just a fancy np.mean(your_results).
\nHere is the official description - \"A voting regressor is an ensemble meta-estimator that fits several base regressors, each on the whole dataset. Then it averages the individual predictions to form a final prediction.\"
You’re absolutely right that VotingRegressor performs a straightforward averaging of predictions and doesn’t inherently mask variability.
\nHowever, in this case, we’re not talking about VotingRegressors. Voting between submissions uses majority voting, which is different from averaging.
\nBy using majority voting rather than averaging, the aim is to reduce the impact of outlier predictions from individual models. This approach is particularly helpful for discrete ordinal predictions, where taking the mode can reduce the influence of individual model biases, as it discards predictions that diverge too much from the majority. This is less likely to happen with simple averaging.
\nI wish you good luck in this competition.
", "votes": null, "replies": [] } ] } ] } ] }, { "id": 3037280, "author_name": "eu1234", "author_url": "", "post_date": "11/05/2024 14:58:18", "content": "I can tell you that the autoencoder is not correctly applied in these notebooks because it comes from the same copy-paste source where the original author made a mistake in applying it on data - so take it like a hint for debugging.
", "votes": null, "replies": [ { "id": 3041461, "author_name": "taimour", "author_url": "", "post_date": "11/10/2024 11:47:24", "content": "Can you explain what is the mistake? thank you.
", "votes": null, "replies": [] } ] }, { "id": 3037335, "author_name": "sumit08", "author_url": "", "post_date": "11/05/2024 15:48:03", "content": "I think not always the autocoder can be the winner model.They might be the \"winner\" for tasks that rely on dimensionality reduction, feature extraction, or anomaly detection, especially when data is unlabeled.
", "votes": null, "replies": [ { "id": 3037690, "author_name": "saidkoussi", "author_url": "", "post_date": "11/06/2024 02:27:03", "content": "@sumit08 yes agree bout some portion of the data are unabled , so autoencoder may have the upper hand
", "votes": null, "replies": [] } ] }, { "id": 3037591, "author_name": "yekenot", "author_url": "", "post_date": "11/05/2024 20:57:10", "content": "I frequently observed the GaussianNoise layer receiving input data at the start of the AutoEncoder architectures, but here it's skipped, perhaps for some reason. On the step of scaling, I also noticed that the Gauss Rank Scaler was used rather than other scalers. Another addition is a Swap Noise. These three things I associate with Denoising AutoEncoder primarily. Here, as I can follow, AutoEncoder does feature extraction, so noise is not necessary, although it may be useful as a form of regularization to learn a more robust representation and improve generalization since there is a lack of data, but the issue of unidentified randomness still remains, thus how to test anything is a question.