During the competition, I often read someone mention the importance of feature engineering (and ensembling). This is true, of course! But I felt at the time, and now see a lot of evidence that we can be at least a little more specific. Below is my take on the high-level keys of the competition, and how the top models showed concrete implementations to accomplish this in practice.
\nKey #1: [especially for tree models]: Extracting all signal from each column's time series of 13 statements.
\nExamples:
Key #2: [key #1 for NN models]:
\n2.A: Leveraging the unlabeled test data
2.B: Compensating for differences between train and public test vs private test.
\nOther keys, without examples or much discussion on them:
\nKey #3: memory resources / memory management. I think the winners mostly just had sufficient resources? shrug It was important in this competition, due to solutions to key #1 tending to be memory heavy. Feature selection, in my view, is a memory resource technique not an optimization technique, at least in this competition.
Key #4: Diverse ensembles. I guess I can't think of anything more specific to add to that, lol. Not my focus here, and I'm still a beginner in this area. I have a little more insight - I hope - to the feature engineering side of things.
\nClosing thoughts: If doing a similar competition in the future, or refining this one, I would look at, first, checking EVERY box above, not just some of them. More than that, at ways to further \"ensemble\" the Key #1 itself. Not only do meta-features like predict target with a single time series column, but multiple different models predicting target, multiple models predicting 'next in sequence', and non-model ways of predicting next in sequence (ARIMA, least squares line, and/or whatever).
", "messages": [ { "id": "1916642", "postDate": "08/28/2022 03:52:50", "content": "During the competition, I often read someone mention the importance of feature engineering (and ensembling). This is true, of course! But I felt at the time, and now see a lot of evidence that we can be at least a little more specific. Below is my take on the high-level keys of the competition, and how the top models showed concrete implementations to accomplish this in practice.
\nKey #1: [especially for tree models]: Extracting all signal from each column's time series of 13 statements.
\nExamples:
Key #2: [key #1 for NN models]:
\n2.A: Leveraging the unlabeled test data
2.B: Compensating for differences between train and public test vs private test.
\nOther keys, without examples or much discussion on them:
\nKey #3: memory resources / memory management. I think the winners mostly just had sufficient resources? shrug It was important in this competition, due to solutions to key #1 tending to be memory heavy. Feature selection, in my view, is a memory resource technique not an optimization technique, at least in this competition.
Key #4: Diverse ensembles. I guess I can't think of anything more specific to add to that, lol. Not my focus here, and I'm still a beginner in this area. I have a little more insight - I hope - to the feature engineering side of things.
\nClosing thoughts: If doing a similar competition in the future, or refining this one, I would look at, first, checking EVERY box above, not just some of them. More than that, at ways to further \"ensemble\" the Key #1 itself. Not only do meta-features like predict target with a single time series column, but multiple different models predicting target, multiple models predicting 'next in sequence', and non-model ways of predicting next in sequence (ARIMA, least squares line, and/or whatever).
", "rawMarkdown": "During the competition, I often read someone mention the importance of feature engineering (and ensembling). This is true, of course! But I felt at the time, and now see a lot of evidence that we can be at least a little more specific. Below is my take on the high-level keys of the competition, and how the top models showed concrete implementations to accomplish this in practice. \n\nKey #1: [especially for tree models]: Extracting all signal from each column's time series of 13 statements.\nExamples:\n* Vast quantity of aggregated features, not only on full set, but winners often did on last 3, last 6, etc. Number of features generated per numeric base feature: [First place: 39](https://www.kaggle.com/competitions/amex-default-prediction/discussion/348111), [Second place: 35+?](https://www.kaggle.com/competitions/amex-default-prediction/discussion/347637), [12th place: ~25?](https://www.kaggle.com/competitions/amex-default-prediction/discussion/347786), [13th place: 15-25??](https://www.kaggle.com/competitions/amex-default-prediction/discussion/348058), [14th place: public features: 15-20?](https://www.kaggle.com/competitions/amex-default-prediction/discussion/348014), [18th place: 20+?](https://www.kaggle.com/competitions/amex-default-prediction/discussion/347850), [25th place: 16++?](https://www.kaggle.com/competitions/amex-default-prediction/discussion/347880), [46th place: 16](https://www.kaggle.com/competitions/amex-default-prediction/discussion/347966)\n* Meta-features:\n * Predicting target with all data for a each single month [First place, that's his 'LGB series oof', right?](https://www.kaggle.com/competitions/amex-default-prediction/discussion/348111), [5th place](https://www.kaggle.com/competitions/amex-default-prediction/discussion/348097), [12th place](https://www.kaggle.com/competitions/amex-default-prediction/discussion/347786), [13th place](https://www.kaggle.com/competitions/amex-default-prediction/discussion/348058), [16th place](https://www.kaggle.com/competitions/amex-default-prediction/discussion/347908). [46th place* (predicted 'missed payment', not target)](https://www.kaggle.com/competitions/amex-default-prediction/discussion/347966)\n * Predicting target with each single feature time series. [20th place: Mini-LSTM](https://www.kaggle.com/competitions/amex-default-prediction/discussion/347651), [25th place](https://www.kaggle.com/competitions/amex-default-prediction/discussion/347880)\n * Predicting next in sequence [11th place](https://www.kaggle.com/competitions/amex-default-prediction/discussion/347668)\n\n\nKey #2: [key #1 for NN models]:\n2.A: Leveraging the unlabeled test data\n* Predict something with high correlation to the target. I predicted 'missed payments': [46th place](https://www.kaggle.com/competitions/amex-default-prediction/discussion/347966)\n* Pre-training NN on all data: with features as outputs: [5th place](https://www.kaggle.com/competitions/amex-default-prediction/discussion/348118). With soft labels: [14th place? \"knowledge distillation\"](https://www.kaggle.com/competitions/amex-default-prediction/discussion/348014), [15th place](https://www.kaggle.com/competitions/amex-default-prediction/discussion/347641).\n* Pseudo-labeling, to an extent it should help here. [14th place](https://www.kaggle.com/competitions/amex-default-prediction/discussion/348014), [18th place](https://www.kaggle.com/competitions/amex-default-prediction/discussion/347850), [20th place: NN](https://www.kaggle.com/competitions/amex-default-prediction/discussion/347651)\n\n2.B: Compensating for differences between train and public test vs private test.\n* 2.A can inherently help with this, but other options include:\n* Remove features via adversarial validation or other inspection technique. [12th place: R_1, D59, S_11, B_29](https://www.kaggle.com/competitions/amex-default-prediction/discussion/347786), [14th place: some models: B_29, R_1, D_59](https://www.kaggle.com/competitions/amex-default-prediction/discussion/348014), [20th place: NN: D_59, D_86](https://www.kaggle.com/competitions/amex-default-prediction/discussion/347651), [46th place: B_29](https://www.kaggle.com/competitions/amex-default-prediction/discussion/347966)\n* Normalize features. [2nd place](https://www.kaggle.com/competitions/amex-default-prediction/discussion/347637)\n* Special mention of 'month rank' and 'user rank' which is hard to evaluate as a normalization technique alongside non-normalized data, but may have been a great way to cover 2.B. [First place](https://www.kaggle.com/competitions/amex-default-prediction/discussion/348111) \n\nOther keys, without examples or much discussion on them:\nKey #3: memory resources / memory management. I think the winners mostly just had sufficient resources? *shrug* It was important in this competition, due to solutions to key #1 tending to be memory heavy. Feature selection, in my view, is a memory resource technique not an optimization technique, at least in this competition.\n\nKey #4: Diverse ensembles. I guess I can't think of anything more specific to add to that, lol. Not my focus here, and I'm still a beginner in this area. I have a little more insight - I hope - to the feature engineering side of things.\n\nClosing thoughts: If doing a similar competition in the future, or refining this one, I would look at, first, checking EVERY box above, not just some of them. More than that, at ways to further \"ensemble\" the Key #1 itself. Not only do meta-features like predict target with a single time series column, but multiple different models predicting target, multiple models predicting 'next in sequence', and non-model ways of predicting next in sequence (ARIMA, least squares line, and/or whatever).", "votes": null }, { "id": "1916672", "postDate": "08/28/2022 04:50:01", "content": "This is a great summary. Thank you!
\nI'd add another key component here for making a good NN (especially in this competition) is how top teams deal with missing values.
I didn't put much there because I have zero real experience with NN! That's good to know
", "rawMarkdown": "I didn't put much there because I have zero real experience with NN! That's good to know", "votes": null } ], "comments": [ { "id": 1916672, "author_name": "raphael1123", "author_url": "", "post_date": "08/28/2022 04:50:01", "content": "This is a great summary. Thank you!
\nI'd add another key component here for making a good NN (especially in this competition) is how top teams deal with missing values.
I didn't put much there because I have zero real experience with NN! That's good to know
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1916642": "During the competition, I often read someone mention the importance of feature engineering (and ensembling). This is true, of course! But I felt at the time, and now see a lot of evidence that we can be at least a little more specific. Below is my take on the high-level keys of the competition, and how the top models showed concrete implementations to accomplish this in practice. \n\nKey #1: [especially for tree models]: Extracting all signal from each column's time series of 13 statements.\nExamples:\n* Vast quantity of aggregated features, not only on full set, but winners often did on last 3, last 6, etc. Number of features generated per numeric base feature: [First place: 39](https://www.kaggle.com/competitions/amex-default-prediction/discussion/348111), [Second place: 35+?](https://www.kaggle.com/competitions/amex-default-prediction/discussion/347637), [12th place: ~25?](https://www.kaggle.com/competitions/amex-default-prediction/discussion/347786), [13th place: 15-25??](https://www.kaggle.com/competitions/amex-default-prediction/discussion/348058), [14th place: public features: 15-20?](https://www.kaggle.com/competitions/amex-default-prediction/discussion/348014), [18th place: 20+?](https://www.kaggle.com/competitions/amex-default-prediction/discussion/347850), [25th place: 16++?](https://www.kaggle.com/competitions/amex-default-prediction/discussion/347880), [46th place: 16](https://www.kaggle.com/competitions/amex-default-prediction/discussion/347966)\n* Meta-features:\n * Predicting target with all data for a each single month [First place, that's his 'LGB series oof', right?](https://www.kaggle.com/competitions/amex-default-prediction/discussion/348111), [5th place](https://www.kaggle.com/competitions/amex-default-prediction/discussion/348097), [12th place](https://www.kaggle.com/competitions/amex-default-prediction/discussion/347786), [13th place](https://www.kaggle.com/competitions/amex-default-prediction/discussion/348058), [16th place](https://www.kaggle.com/competitions/amex-default-prediction/discussion/347908). [46th place* (predicted 'missed payment', not target)](https://www.kaggle.com/competitions/amex-default-prediction/discussion/347966)\n * Predicting target with each single feature time series. [20th place: Mini-LSTM](https://www.kaggle.com/competitions/amex-default-prediction/discussion/347651), [25th place](https://www.kaggle.com/competitions/amex-default-prediction/discussion/347880)\n * Predicting next in sequence [11th place](https://www.kaggle.com/competitions/amex-default-prediction/discussion/347668)\n\n\nKey #2: [key #1 for NN models]:\n2.A: Leveraging the unlabeled test data\n* Predict something with high correlation to the target. I predicted 'missed payments': [46th place](https://www.kaggle.com/competitions/amex-default-prediction/discussion/347966)\n* Pre-training NN on all data: with features as outputs: [5th place](https://www.kaggle.com/competitions/amex-default-prediction/discussion/348118). With soft labels: [14th place? \"knowledge distillation\"](https://www.kaggle.com/competitions/amex-default-prediction/discussion/348014), [15th place](https://www.kaggle.com/competitions/amex-default-prediction/discussion/347641).\n* Pseudo-labeling, to an extent it should help here. [14th place](https://www.kaggle.com/competitions/amex-default-prediction/discussion/348014), [18th place](https://www.kaggle.com/competitions/amex-default-prediction/discussion/347850), [20th place: NN](https://www.kaggle.com/competitions/amex-default-prediction/discussion/347651)\n\n2.B: Compensating for differences between train and public test vs private test.\n* 2.A can inherently help with this, but other options include:\n* Remove features via adversarial validation or other inspection technique. [12th place: R_1, D59, S_11, B_29](https://www.kaggle.com/competitions/amex-default-prediction/discussion/347786), [14th place: some models: B_29, R_1, D_59](https://www.kaggle.com/competitions/amex-default-prediction/discussion/348014), [20th place: NN: D_59, D_86](https://www.kaggle.com/competitions/amex-default-prediction/discussion/347651), [46th place: B_29](https://www.kaggle.com/competitions/amex-default-prediction/discussion/347966)\n* Normalize features. [2nd place](https://www.kaggle.com/competitions/amex-default-prediction/discussion/347637)\n* Special mention of 'month rank' and 'user rank' which is hard to evaluate as a normalization technique alongside non-normalized data, but may have been a great way to cover 2.B. [First place](https://www.kaggle.com/competitions/amex-default-prediction/discussion/348111) \n\nOther keys, without examples or much discussion on them:\nKey #3: memory resources / memory management. I think the winners mostly just had sufficient resources? *shrug* It was important in this competition, due to solutions to key #1 tending to be memory heavy. Feature selection, in my view, is a memory resource technique not an optimization technique, at least in this competition.\n\nKey #4: Diverse ensembles. I guess I can't think of anything more specific to add to that, lol. Not my focus here, and I'm still a beginner in this area. I have a little more insight - I hope - to the feature engineering side of things.\n\nClosing thoughts: If doing a similar competition in the future, or refining this one, I would look at, first, checking EVERY box above, not just some of them. More than that, at ways to further \"ensemble\" the Key #1 itself. Not only do meta-features like predict target with a single time series column, but multiple different models predicting target, multiple models predicting 'next in sequence', and non-model ways of predicting next in sequence (ARIMA, least squares line, and/or whatever).", "1916672": "This is a great summary. Thank you! \nI'd add another key component here for making a good NN (especially in this competition) is how top teams deal with missing values.", "1916711": "I didn't put much there because I have zero real experience with NN! That's good to know" }, "source": "meta" }