Thank to AMEX for this competition and thank to @raddar for his wonedful data compression notebook.
\nMy model is the enesemble of DART and Neural networks.
Neural Network models: Tabular data(same used for Tree learners), Gru, Transformer and Stack Model(Gru+Tabular Model).
\nFollowing are the scenarios where neural nets struggled compared to the tree based learners.
\n1 . Outlier numbers in some features.(lightgbm process it as quantiles, hence roboust)
\n2 . Understanding if the value is missing; even if we fix it as -1, still will be difficult to process.
\n3 . Dominant easy examples.(Gradients are not getting propogated to hard/varied examples as most examples are easy to classify).
1. Preprocessing:
\nClip the values of the features to 95th or 99th percentile of the feature. Tabular Data Clip values, Sequential Data Clip values
2. Add Embeddings to the Missing values
\nFor both FFN (tabular) and sequential data(GRU, transformer), for each feature given more information that the values are missing or not. Adding this information helps the model to converge faster and better generalization.
Tabular Model: here
\nGru Model: here
\nStack Model: here
\nTransformer Model: here
Thank to AMEX for this competition and thank to @raddar for his wonedful data compression notebook.
\nMy model is the enesemble of DART and Neural networks.
Neural Network models: Tabular data(same used for Tree learners), Gru, Transformer and Stack Model(Gru+Tabular Model).
\nFollowing are the scenarios where neural nets struggled compared to the tree based learners.
\n1 . Outlier numbers in some features.(lightgbm process it as quantiles, hence roboust)
\n2 . Understanding if the value is missing; even if we fix it as -1, still will be difficult to process.
\n3 . Dominant easy examples.(Gradients are not getting propogated to hard/varied examples as most examples are easy to classify).
1. Preprocessing:
\nClip the values of the features to 95th or 99th percentile of the feature. Tabular Data Clip values, Sequential Data Clip values
2. Add Embeddings to the Missing values
\nFor both FFN (tabular) and sequential data(GRU, transformer), for each feature given more information that the values are missing or not. Adding this information helps the model to converge faster and better generalization.
Tabular Model: here
\nGru Model: here
\nStack Model: here
\nTransformer Model: here
Congratulations @narendra . You're right, the meaningful features is crucial for neural nets. So far, I still use pretrain technology to fillna or deal with outline value.
", "rawMarkdown": "Congratulations @narendra . You're right, the meaningful features is crucial for neural nets. So far, I still use pretrain technology to fillna or deal with outline value.", "votes": null }, { "id": "1936753", "postDate": "09/13/2022 03:26:20", "content": "thanks @jacksonyou
", "rawMarkdown": "thanks @jacksonyou", "votes": null } ], "comments": [ { "id": 1929563, "author_name": "jacksonyou", "author_url": "", "post_date": "09/07/2022 07:10:25", "content": "Congratulations @narendra . You're right, the meaningful features is crucial for neural nets. So far, I still use pretrain technology to fillna or deal with outline value.
", "votes": null, "replies": [ { "id": 1936753, "author_name": "narendra", "author_url": "", "post_date": "09/13/2022 03:26:20", "content": "thanks @jacksonyou
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1924100": "Thank to AMEX for this competition and thank to @raddar for his wonedful data compression notebook.\nMy model is the enesemble of DART and Neural networks.\n\nNeural Network models: Tabular data(same used for Tree learners), Gru, Transformer and Stack Model(Gru+Tabular Model).\n\nFollowing are the scenarios where neural nets struggled compared to the tree based learners.\n1 . Outlier numbers in some features.(lightgbm process it as quantiles, hence roboust)\n2 . Understanding if the value is missing; even if we fix it as -1, still will be difficult to process.\n3 . Dominant easy examples.(Gradients are not getting propogated to hard/varied examples as most examples are easy to classify).\n\n**1. Preprocessing:**\nClip the values of the features to 95th or 99th percentile of the feature. [Tabular Data Clip values](https://www.kaggle.com/code/narendra/amex-tabular-nn-data-clip-map), [Sequential Data Clip values](https://www.kaggle.com/code/narendra/amex-sequence-data-clip-map) \n\n\n**2. Add Embeddings to the Missing values**\nFor both FFN (tabular) and sequential data(GRU, transformer), for each feature given more information that the values are missing or not. Adding this information helps the model to converge faster and better generalization.\n\nTabular Model: [here](https://www.kaggle.com/code/narendra/amex-nn-ranking-train-1024#model)\nGru Model: [here](https://www.kaggle.com/code/narendra/amex-gru-with-tab-embedds-train-v3#Sequence-Model)\nStack Model: [here](https://www.kaggle.com/code/narendra/amex-model-stack)\nTransformer Model: [here](https://www.kaggle.com/code/narendra/amex-transformer-train-v3)", "1929563": "Congratulations @narendra . You're right, the meaningful features is crucial for neural nets. So far, I still use pretrain technology to fillna or deal with outline value.", "1936753": "thanks @jacksonyou" }, "source": "meta" }