Aside from how to do a better metric hacking, I would like to share some techniques that have helped me achieve better stability.
\nThe first one is adversarial training. I didn't train models offline, so during training I have access to the real test data. For categorical variables, I drop values that have very different distribution between train and test. For numeric variables, I clip train values based on the test data. After that I do adversrial validation. I gradually drop important features, since adversarial validation model tends to focus on very few features. This process is done multiple times. The importance of features is evaluated using permutation importance. Once the auc of the adversarial model reaches a certain threshold (I use multiple threholds), I start training the main model. So the final model is an ensemble of models trained at different thresholds.
\nThe second technique is pseudo lableing. I belive this helps the model learn test distribution, which is very different from train. Although its effect on public LB is not very significant, it shows a much better effect on private LB.
\nOne area I spent a lot time but didn't work out was training a neural network model that creates an indistinguishable feature space between train and test sets to classify defualt customers. This concept is from the domain of domain adaptation.
\nFinally, I want to thank the hosts for bringing stability into the picture. This is a very intertesing topic. In my humble opinion, I think a better metirc would be just the auc of data after one or two years. The cuurent metric penalizes the downtrend of Gini so much that the auc of the recent test data doesn't even matter.
", "messages": [ { "id": 2840968, "postDate": "2024-05-28T10:33:41.830Z", "content": "Aside from how to do a better metric hacking, I would like to share some techniques that have helped me achieve better stability.
\nThe first one is adversarial training. I didn't train models offline, so during training I have access to the real test data. For categorical variables, I drop values that have very different distribution between train and test. For numeric variables, I clip train values based on the test data. After that I do adversrial validation. I gradually drop important features, since adversarial validation model tends to focus on very few features. This process is done multiple times. The importance of features is evaluated using permutation importance. Once the auc of the adversarial model reaches a certain threshold (I use multiple threholds), I start training the main model. So the final model is an ensemble of models trained at different thresholds.
\nThe second technique is pseudo lableing. I belive this helps the model learn test distribution, which is very different from train. Although its effect on public LB is not very significant, it shows a much better effect on private LB.
\nOne area I spent a lot time but didn't work out was training a neural network model that creates an indistinguishable feature space between train and test sets to classify defualt customers. This concept is from the domain of domain adaptation.
\nFinally, I want to thank the hosts for bringing stability into the picture. This is a very intertesing topic. In my humble opinion, I think a better metirc would be just the auc of data after one or two years. The cuurent metric penalizes the downtrend of Gini so much that the auc of the recent test data doesn't even matter.
", "rawMarkdown": "Aside from how to do a better metric hacking, I would like to share some techniques that have helped me achieve better stability.\n\nThe first one is adversarial training. I didn't train models offline, so during training I have access to the real test data. For categorical variables, I drop values that have very different distribution between train and test. For numeric variables, I clip train values based on the test data. After that I do adversrial validation. I gradually drop important features, since adversarial validation model tends to focus on very few features. This process is done multiple times. The importance of features is evaluated using permutation importance. Once the auc of the adversarial model reaches a certain threshold (I use multiple threholds), I start training the main model. So the final model is an ensemble of models trained at different thresholds.\n\nThe second technique is pseudo lableing. I belive this helps the model learn test distribution, which is very different from train. Although its effect on public LB is not very significant, it shows a much better effect on private LB.\n\nOne area I spent a lot time but didn't work out was training a neural network model that creates an indistinguishable feature space between train and test sets to classify defualt customers. This concept is from the domain of domain adaptation.\n\nFinally, I want to thank the hosts for bringing stability into the picture. This is a very intertesing topic. In my humble opinion, I think a better metirc would be just the auc of data after one or two years. The cuurent metric penalizes the downtrend of Gini so much that the auc of the recent test data doesn't even matter.", "votes": 16 }, { "id": 2849009, "postDate": "2024-06-01T10:31:38.157Z", "content": "\n\nI didn't train models offline, so during training I have access to the real test data.
\n
Can you elaborate how you get access to real test data?
", "rawMarkdown": "> I didn't train models offline, so during training I have access to the real test data.\n\nCan you elaborate how you get access to real test data?", "votes": 1, "replies": [ { "id": 2851896, "postDate": "2024-06-03T02:31:08.273Z", "content": "When you click the submission button, the sample test files would be replaced by real test data. I train my models after I click the submit button, so my model can get access to real test data.
", "rawMarkdown": "When you click the submission button, the sample test files would be replaced by real test data. I train my models after I click the submit button, so my model can get access to real test data. " } ] }, { "id": 2841929, "postDate": "2024-05-28T18:55:23.777Z", "content": "\nCould you elaborate on the concept of adversarial training? From a quick search, I understood that it's a process of manipulating a model to make worse predictions. How does this help to achieve stability on future data?
I think what he said is this.https://www.kaggle.com/code/carlmcbrideellis/what-is-adversarial-validation
", "rawMarkdown": "I think what he said is this.https://www.kaggle.com/code/carlmcbrideellis/what-is-adversarial-validation", "votes": 2, "replies": [ { "id": 2875530, "postDate": "2024-06-17T08:07:10.363Z", "content": "Can I just say, it's nice seeing you here :D still following the competition, I thought you had disappeared after the linear regression notebook. This is my first Kaggle competition, while I can't complete on time since I can't get the submission right, your first few notebooks had been of great help so I distinctively remember you alot.
", "rawMarkdown": "Can I just say, it's nice seeing you here :D still following the competition, I thought you had disappeared after the linear regression notebook. This is my first Kaggle competition, while I can't complete on time since I can't get the submission right, your first few notebooks had been of great help so I distinctively remember you alot." } ] }, { "id": 2842261, "postDate": "2024-05-29T02:05:16.313Z", "content": "The point is to improve generalization of model by decresing the distribution difference between train and test. The auc the adversarial validation between train and real test is above 0.9 or even higher, so there are some variables like refresh_date that are very important features in adversarial validatioin model. By removing these variables, you could decrease the distribution difference between tran and test.
", "rawMarkdown": "The point is to improve generalization of model by decresing the distribution difference between train and test. The auc the adversarial validation between train and real test is above 0.9 or even higher, so there are some variables like refresh_date that are very important features in adversarial validatioin model. By removing these variables, you could decrease the distribution difference between tran and test.", "votes": 2 } ] }, { "id": 2841568, "postDate": "2024-05-28T16:07:05.683Z", "content": "Very interesting, thank you for sharing that. If you want to share it with more details you could create some example notebook.
", "rawMarkdown": "Very interesting, thank you for sharing that. If you want to share it with more details you could create some example notebook." }, { "id": 2843670, "postDate": "2024-05-29T17:00:42.423Z", "content": "Thank you for this post Yan. I have learned a lot from this particular thread of discussion. I will eventually try out these with my code as well so I learn these two techniques. It is very interesting. I have been trying to look up some articles and references to better understand how it works. This is for reference for others who may want to know about the intuition behind these.
\nAdversarial validation (same as shared by yunsuxiaozi ): https://www.kaggle.com/code/carlmcbrideellis/what-is-adversarial-validation
\nPsuedo-labeling: https://www.kaggle.com/code/cdeotte/pseudo-labeling-qda-0-969/notebook
\nThank you once again for the code snippets as well. Helps a lot! 🙏
", "rawMarkdown": "Thank you for this post Yan. I have learned a lot from this particular thread of discussion. I will eventually try out these with my code as well so I learn these two techniques. It is very interesting. I have been trying to look up some articles and references to better understand how it works. This is for reference for others who may want to know about the intuition behind these.\n\nAdversarial validation (same as shared by yunsuxiaozi ): https://www.kaggle.com/code/carlmcbrideellis/what-is-adversarial-validation\n\nPsuedo-labeling: https://www.kaggle.com/code/cdeotte/pseudo-labeling-qda-0-969/notebook\n\nThank you once again for the code snippets as well. Helps a lot! 🙏" }, { "id": 2841028, "postDate": "2024-05-28T11:15:27.810Z", "content": "Please tell me about the following questions
\nWhat a metric did you use?
\n2-1. \"The second technique is pseudo lableing.\"
\nWhat models did you use? Because of the limitation of running time, we cannot use this technique. (e.g. catboost)
\n2-2. \"I belive this helps the model learn test distribution, which is very different from train. \"
\n However you remove some columns for the reason, \"very different distribution between train and test.\". please tell me how you decided the standard whether drop a column or not.
Hi Gita,
\nThanks for your interest.
\ncol_dist = df.loc[df[] == , col].value_counts(normalize=, dropna=).to_frame()\ncol_dist[] = df.loc[df[] == , col].value_counts(normalize=, dropna=)\ncol_dist[] = col_dist[].fillna() / (col_dist[].fillna() + )\ncond0 = (col_dist[] < ) | (col_dist[] < )\ncond1 = (col_dist[] < / ) | (col_dist[] > )\n2-1. You are correct. I only use lightgbm.
\n2-2. Removing values that are different and adding pseudo labled test data into train is not contradictory. The goal is to decrese the distribution difference between train and test.
\nLastly, I build upon the public engineering methods. The main changes I made are 1) OHE categorical variables and aggreate into the mean of non-null values(excluding a55475b1); 2) combining tax data source into one; 3) extract client info from person data. Num_group1 means client.
", "rawMarkdown": "Hi Gita, \n\nThanks for your interest. \n\n1. In interms of removing categorical values, I remove values that are 4 times more frequent or 1/4 lest frequent in train than test. And also values appear less than 5% frequent in train or test\n\n```python\ncol_dist = df.loc[df[\"is_test\"] == 0, col].value_counts(normalize=True, dropna=False).to_frame(\"train_dist\")\ncol_dist[\"test_dist\"] = df.loc[df[\"is_test\"] == 1, col].value_counts(normalize=True, dropna=False)\ncol_dist[\"train_test_ratio\"] = col_dist[\"train_dist\"].fillna(0) / (col_dist[\"test_dist\"].fillna(0) + 1e-5)\ncond0 = (col_dist[\"train_dist\"] < 0.05) | (col_dist[\"test_dist\"] < 0.05)\ncond1 = (col_dist[\"train_test_ratio\"] < 1 / 4) | (col_dist[\"train_test_ratio\"] > 4)\n\n```\n\n2-1. You are correct. I only use lightgbm.\n\n2-2. Removing values that are different and adding pseudo labled test data into train is not contradictory. The goal is to decrese the distribution difference between train and test.\n\nLastly, I build upon the public engineering methods. The main changes I made are 1) OHE categorical variables and aggreate into the mean of non-null values(excluding a55475b1); 2) combining tax data source into one; 3) extract client info from person data. Num_group1 means client. ", "votes": 1, "replies": [ { "id": 2844233, "postDate": "2024-05-30T02:35:07.443Z", "content": "Hello Yan,
\nThank you for answering my question! It was very helpful!
", "rawMarkdown": "Hello Yan,\n\nThank you for answering my question! It was very helpful!" } ] } ] }, { "id": 2840972, "postDate": "2024-05-28T10:35:24.167Z", "rawMarkdown": "", "isDeleted": true, "replies": [ { "id": 2841237, "postDate": "2024-05-28T13:24:56.847Z", "content": "May be get model refit with the pseudo label on test data？
", "rawMarkdown": "May be get model refit with the pseudo label on test data？", "isDeleted": true, "replies": [ { "id": 2841267, "postDate": "2024-05-28T13:46:52.237Z", "content": "Yes, the important thing is get model to learn new distribution.
", "rawMarkdown": "Yes, the important thing is get model to learn new distribution." }, { "id": 2841834, "postDate": "2024-05-28T17:52:26.743Z", "content": "Would really appreciate if you could give more details about how to achieve this. If you could please share snippets of code if not the whole notebook that helps us understand how this can be done, it will be great for a beginner like me to learn.
", "rawMarkdown": "Would really appreciate if you could give more details about how to achieve this. If you could please share snippets of code if not the whole notebook that helps us understand how this can be done, it will be great for a beginner like me to learn." }, { "id": 2842819, "postDate": "2024-05-29T09:18:56.237Z", "content": "This code snipt should be able to explain how to do pseudo labeling. Just transform the most confident predition on test data into target column and add it to train data.
\nbad_cut = df_test[].quantile()\ngood_cut = df_test[].quantile()\ndf_test = df_test[(df_test[] > bad_cut) | (df_test[] < good_cut)]\ndf_test[] = (df_test[] > ).astype()\ndf_test.drop(, axis=, inplace=)\n\ndf_train = pd.read_parquet()\ndf_train = pd.concat([df_train, df_test[df_train.columns]], ignore_index=, copy=).reset_index(drop=)\n\n\nI didn't train models offline, so during training I have access to the real test data.
\n
Can you elaborate how you get access to real test data?
", "votes": 1, "replies": [ { "id": 2851896, "author_name": "Evan", "author_url": "", "post_date": "2024-06-03T02:31:08.273000", "content": "When you click the submission button, the sample test files would be replaced by real test data. I train my models after I click the submit button, so my model can get access to real test data.
", "votes": 0, "replies": [] } ] }, { "id": 2841929, "author_name": "Andreas Bisiadis", "author_url": "", "post_date": "2024-05-28T18:55:23.777000", "content": "\nCould you elaborate on the concept of adversarial training? From a quick search, I understood that it's a process of manipulating a model to make worse predictions. How does this help to achieve stability on future data?
I think what he said is this.https://www.kaggle.com/code/carlmcbrideellis/what-is-adversarial-validation
", "votes": 2, "replies": [ { "id": 2875530, "author_name": "FlyingCamel", "author_url": "", "post_date": "2024-06-17T08:07:10.363000", "content": "Can I just say, it's nice seeing you here :D still following the competition, I thought you had disappeared after the linear regression notebook. This is my first Kaggle competition, while I can't complete on time since I can't get the submission right, your first few notebooks had been of great help so I distinctively remember you alot.
", "votes": 0, "replies": [] } ] }, { "id": 2842261, "author_name": "Evan", "author_url": "", "post_date": "2024-05-29T02:05:16.313000", "content": "The point is to improve generalization of model by decresing the distribution difference between train and test. The auc the adversarial validation between train and real test is above 0.9 or even higher, so there are some variables like refresh_date that are very important features in adversarial validatioin model. By removing these variables, you could decrease the distribution difference between tran and test.
", "votes": 2, "replies": [] } ] }, { "id": 2841568, "author_name": "Daniel Herman", "author_url": "", "post_date": "2024-05-28T16:07:05.683000", "content": "Very interesting, thank you for sharing that. If you want to share it with more details you could create some example notebook.
", "votes": 0, "replies": [] }, { "id": 2843670, "author_name": "Varuni Rao", "author_url": "", "post_date": "2024-05-29T17:00:42.423000", "content": "Thank you for this post Yan. I have learned a lot from this particular thread of discussion. I will eventually try out these with my code as well so I learn these two techniques. It is very interesting. I have been trying to look up some articles and references to better understand how it works. This is for reference for others who may want to know about the intuition behind these.
\nAdversarial validation (same as shared by yunsuxiaozi ): https://www.kaggle.com/code/carlmcbrideellis/what-is-adversarial-validation
\nPsuedo-labeling: https://www.kaggle.com/code/cdeotte/pseudo-labeling-qda-0-969/notebook
\nThank you once again for the code snippets as well. Helps a lot! 🙏
", "votes": 0, "replies": [] }, { "id": 2841028, "author_name": "Aria", "author_url": "", "post_date": "2024-05-28T11:15:27.810000", "content": "Please tell me about the following questions
\nWhat a metric did you use?
\n2-1. \"The second technique is pseudo lableing.\"
\nWhat models did you use? Because of the limitation of running time, we cannot use this technique. (e.g. catboost)
\n2-2. \"I belive this helps the model learn test distribution, which is very different from train. \"
\n However you remove some columns for the reason, \"very different distribution between train and test.\". please tell me how you decided the standard whether drop a column or not.
Hi Gita,
\nThanks for your interest.
\ncol_dist = df.loc[df[] == , col].value_counts(normalize=, dropna=).to_frame()\ncol_dist[] = df.loc[df[] == , col].value_counts(normalize=, dropna=)\ncol_dist[] = col_dist[].fillna() / (col_dist[].fillna() + )\ncond0 = (col_dist[] < ) | (col_dist[] < )\ncond1 = (col_dist[] < / ) | (col_dist[] > )\n2-1. You are correct. I only use lightgbm.
\n2-2. Removing values that are different and adding pseudo labled test data into train is not contradictory. The goal is to decrese the distribution difference between train and test.
\nLastly, I build upon the public engineering methods. The main changes I made are 1) OHE categorical variables and aggreate into the mean of non-null values(excluding a55475b1); 2) combining tax data source into one; 3) extract client info from person data. Num_group1 means client.
", "votes": 1, "replies": [ { "id": 2844233, "author_name": "Aria", "author_url": "", "post_date": "2024-05-30T02:35:07.443000", "content": "Hello Yan,
\nThank you for answering my question! It was very helpful!
", "votes": 0, "replies": [] } ] } ] }, { "id": 2840972, "author_name": "", "author_url": "", "post_date": "2024-05-28T10:35:24.167000", "content": "", "votes": 0, "replies": [ { "id": 2841237, "author_name": "", "author_url": "", "post_date": "2024-05-28T13:24:56.847000", "content": "May be get model refit with the pseudo label on test data？
", "votes": 0, "replies": [ { "id": 2841267, "author_name": "Evan", "author_url": "", "post_date": "2024-05-28T13:46:52.237000", "content": "Yes, the important thing is get model to learn new distribution.
", "votes": 0, "replies": [] }, { "id": 2841834, "author_name": "Varuni Rao", "author_url": "", "post_date": "2024-05-28T17:52:26.743000", "content": "Would really appreciate if you could give more details about how to achieve this. If you could please share snippets of code if not the whole notebook that helps us understand how this can be done, it will be great for a beginner like me to learn.
", "votes": 0, "replies": [] }, { "id": 2842819, "author_name": "Evan", "author_url": "", "post_date": "2024-05-29T09:18:56.237000", "content": "This code snipt should be able to explain how to do pseudo labeling. Just transform the most confident predition on test data into target column and add it to train data.
\nbad_cut = df_test[].quantile()\ngood_cut = df_test[].quantile()\ndf_test = df_test[(df_test[] > bad_cut) | (df_test[] < good_cut)]\ndf_test[] = (df_test[] > ).astype()\ndf_test.drop(, axis=, inplace=)\n\ndf_train = pd.read_parquet()\ndf_train = pd.concat([df_train, df_test[df_train.columns]], ignore_index=, copy=).reset_index(drop=)\n