![\"\"](\"https://raw.githubusercontent.com/Carl-McBride-Ellis/images_for_kaggle/main/Notebook_restarted.png\")
Aghhhh; …again!….
\n
Why does this competition have so much test data?
\nWe have a lot of training data (ostensibly 16.39 GB) but maybe that is just an aspect of the challenge; exploring formats other than CSV, downcasting word lengths, reading in chunks, sub-sampling, garbage collection etc.
\nBut why do we need so much test data? (33.82 GB, almost twice as much as there is train data). Is not the purpose of the test data simply to provide a reliable leaderboard score to three significant figures; and could that not be achieved with a tenth (or indeed much less) of the test data we have been provided with? (and if not, perhaps a more \"stable\" evaluation metric?)
\nAny feature engineering on the training data has to also be applied to the test data, and it is this process that is causing me even more red memory banners than the training phase. At least when training we can down-sample, but we need all of the customer_ID to create a valid submission file.
In a deployment situation we could evaluate the model on the test data incrementally in batches whose size is commensurate with our resources, but here we have to get the job done in a 16GB CPU notebook all at once (or 13GB if one is using GPU).
\nI am very much in favor of the recent suggestion by @jamesmcguigan of 32GB RAM Notebooks, either that, or reducing the test dataset to a necessary and sufficient number of customer_ID.
Anyway, thankfully other competitors have very kindly provided some excellent approaches and observations to this 'data engineering' problem in the comments below.
\nAll the best,
\ncarl
Aghhhh; …again!….
\n
Why does this competition have so much test data?
\nWe have a lot of training data (ostensibly 16.39 GB) but maybe that is just an aspect of the challenge; exploring formats other than CSV, downcasting word lengths, reading in chunks, sub-sampling, garbage collection etc.
\nBut why do we need so much test data? (33.82 GB, almost twice as much as there is train data). Is not the purpose of the test data simply to provide a reliable leaderboard score to three significant figures; and could that not be achieved with a tenth (or indeed much less) of the test data we have been provided with? (and if not, perhaps a more \"stable\" evaluation metric?)
\nAny feature engineering on the training data has to also be applied to the test data, and it is this process that is causing me even more red memory banners than the training phase. At least when training we can down-sample, but we need all of the customer_ID to create a valid submission file.
In a deployment situation we could evaluate the model on the test data incrementally in batches whose size is commensurate with our resources, but here we have to get the job done in a 16GB CPU notebook all at once (or 13GB if one is using GPU).
\nI am very much in favor of the recent suggestion by @jamesmcguigan of 32GB RAM Notebooks, either that, or reducing the test dataset to a necessary and sufficient number of customer_ID.
Anyway, thankfully other competitors have very kindly provided some excellent approaches and observations to this 'data engineering' problem in the comments below.
\nAll the best,
\ncarl
The competition needs so much test data because recall-based metrics on unbalanced data are inherently noisy.
\n![\"noisy_metric\"](\"https://i.imgur.com/mEuAhVd.png\")
The noisiness of the metric can be shown in an experiment. I split the training data into three parts: A training set (80 %) and two validation sets (10 %, i.e. 45891 customers each). Then I trained my gradient booster for 2000 iterations and validated it on both validation sets. The diagram shows that iteration 700 gets a 0.002 higher score with validation set B than with validation set A. Iteration 1600 gets a 0.002 lower score with validation set B than with validation set A.
\nIf you take my two validation sets for public and private leaderboard, you'll get a huge shakeup - you could be at the top of the public leaderboard and end up 0.004 behind on the private leaderboard. The competition would amount to a lottery. Even with ten times more test data, we should be prepared for a shakeup.
\nWhy is the metric so noisy? Recall (default rate captured at 4 %) is defined as true positives divided by real positives. My validation sets have 11883 real positives each (10 % of 118828 positive customers with a stratified split). For both validation sets together, the model produces 16000 true positives. The true positives are assigned to the validation sets at random (i.e. not stratified). It is easily possible that one validation set gets 8050 true positives and the other one 7950. This distribution gives recalls of 8050/11883 = 0.677 and 7950/11883 = 0.669, respectively.
\nWith such a noisy metric, we need a lot of test data to distinguish good models from bad ones and to guarantee a fair competition.
\nBy the way, the noisiness of the metric explains why Kaggle shows only three digits of the score in the leaderboard: Any additional digit would be noise.
", "rawMarkdown": "The competition needs so much test data **because recall-based metrics on unbalanced data are inherently noisy**.\n\n\n\nThe noisiness of the metric can be shown in an experiment. I split the training data into three parts: A training set (80 %) and two validation sets (10 %, i.e. 45891 customers each). Then I trained my gradient booster for 2000 iterations and validated it on both validation sets. The diagram shows that iteration 700 gets a 0.002 higher score with validation set B than with validation set A. Iteration 1600 gets a 0.002 lower score with validation set B than with validation set A.\n\nIf you take my two validation sets for public and private leaderboard, you'll get a huge shakeup - you could be at the top of the public leaderboard and end up 0.004 behind on the private leaderboard. The competition would amount to a lottery. Even with ten times more test data, we should be prepared for a shakeup.\n\nWhy is the metric so noisy? Recall *(default rate captured at 4 %)* is defined as true positives divided by real positives. My validation sets have 11883 real positives each (10 % of 118828 positive customers with a stratified split). For both validation sets together, the model produces 16000 true positives. The true positives are assigned to the validation sets at random (i.e. not stratified). It is easily possible that one validation set gets 8050 true positives and the other one 7950. This distribution gives recalls of 8050/11883 = 0.677 and 7950/11883 = 0.669, respectively.\n\nWith such a noisy metric, we need a lot of test data to distinguish good models from bad ones and to guarantee a fair competition.\n\nBy the way, the noisiness of the metric explains why Kaggle shows only three digits of the score in the leaderboard: Any additional digit would be noise.", "votes": 26, "replies": [ { "id": 1811804, "postDate": "2022-06-05T06:49:28.180Z", "content": "Dear @ambrosm
\nThank you for your magnificent insights, your work is always thought provoking, especially when read in conjunction with your topic \"Graphical explanation of the competition metric\".
\nIf I understand correctly if it takes so much test data to obtain a stable LB score to three significant figures, what implication does this have for the reliability of our cross-validation scores and hold-out score, evidently obtained using substantially less data?
\nAll the best and many thanks,
\ncarl
Dear @ambrosm
\nPS: It is perhaps also worth mentioning that, as spotted both by Raddar, the data has random uniform noise added and also by Chris Deotte in Strange Histograms, so maybe to some extent the evaluation metric is not totally to blame. This uniform noise has a range of ± 0.005.
\nThis noise can be fairly easily extracted. However, it is perhaps also worth noting that having just a little added noise (or \"jitter\") is not necessarily a bad thing, and is sometimes it is even deliberately added to features as a mechanism to reduce overfitting.
\nAll the best,
\ncarl
When inferring test data in a Kaggle notebook, we can read a few million rows at a time and therefore read, process, and infer the test data in parts.
\nall_preds = []\nfor parts in range(NUM_PARTS):\n test_part = pd.read_csv('test', nrows = ROWS, skiprows = SKIP)\n test_part = process(test_part)\n all_preds.append( infer(test_part) )\nI provide an example in my XGB starter notebook here where I read test in 4 parts. When creating 3D data from the 2D provided CSV, then i infer test data in 20 chunks. I provide an example in my GRU starter notebook here
", "rawMarkdown": "When inferring test data in a Kaggle notebook, we can read a few million rows at a time and therefore read, process, and infer the test data in parts. \n\n all_preds = []\n for parts in range(NUM_PARTS):\n test_part = pd.read_csv('test', nrows = ROWS, skiprows = SKIP)\n test_part = process(test_part)\n all_preds.append( infer(test_part) )\n\nI provide an example in my XGB starter notebook [here][1] where I read test in 4 parts. When creating 3D data from the 2D provided CSV, then i infer test data in 20 chunks. I provide an example in my GRU starter notebook [here][2]\n\n[1]: https://www.kaggle.com/code/cdeotte/xgboost-starter-0-793\n[2]: https://www.kaggle.com/code/cdeotte/tensorflow-gru-starter-0-790", "votes": 10, "replies": [ { "id": 1811562, "postDate": "2022-06-04T20:28:57.170Z", "content": "Dear @cdeotte
\nExcellent suggestion. Worth being careful not to split any customer_ID when creating the chunks, otherwise it could create two predictions for the same customer_ID and shift all the subsequent predictions incorrectly…
All the best,
\ncarl
Yes correct. In my notebook first i read the entire column of customer_ID. Then i find which row positions we can split the list without splitting customers into two different parts. Then i read those rows from the file on disk.
I think the host has given a clear answer to this here, emphasis mine.
\n\n\nWe're excited to launch this competition and publish an industrial scale data for the community to stress test your [modelling] ingenuity
\n
I think the constraints make this competition very interesting, there are a couple of stand-out datasets made by kagglers that can be used where precision and size is balanced allowing competitors to choose if they wish to go for less-is-more or brute-force feature engineering.
\nStarter notebooks achieving very high LB scores also back this statement up! Splitting notebooks into training and inference is also the easiest way to expand on those already excellent notebooks.
", "rawMarkdown": "I think the host has given a clear answer to this [here](https://www.kaggle.com/competitions/amex-default-prediction/discussion/327189), emphasis mine.\n> We're excited to launch this competition and publish an *industrial scale* data for the community to stress test your [modelling] *ingenuity*\n\nI think the constraints make this competition very interesting, there are a couple of stand-out datasets made by kagglers that can be used where precision and size is balanced allowing competitors to choose if they wish to go for less-is-more or brute-force feature engineering. \n\nStarter notebooks achieving very high LB scores also back this statement up! Splitting notebooks into training and inference is also the easiest way to expand on those already excellent notebooks.", "votes": 3 }, { "id": 1811535, "postDate": "2022-06-04T19:31:30.230Z", "content": "Hi @carlmcbrideellis, on top of the usual memory tricks (float32) there is some unusual noise in the data changing some features from categorical to float (process is questionnable at best). @raddar has posted a notebook to remove that noise (and an associated dataset). I use that dataset. After that, yes, the solution is to process data in chunks, for feature engineering and prediction.
", "rawMarkdown": "Hi @carlmcbrideellis, on top of the usual memory tricks (float32) there is some unusual noise in the data changing some features from categorical to float (process is questionnable at best). @raddar has posted a notebook to remove that noise (and an associated dataset). I use that dataset. After that, yes, the solution is to process data in chunks, for feature engineering and prediction. ", "votes": 3 }, { "id": 1811781, "postDate": "2022-06-05T06:07:47.423Z", "content": "@carlmcbrideellis Here are my notes on doing CPU / RAM optimization for large parquet files, including a pandas generator function for batch reading a large dataset (Bengali AI dataset was large but AMEX large). Unsure what needs to be changed to make it work for feather files rather than parquet.
\nAlso consider being explicit about your column dtypes, as there is a 4x difference between float64 and float16 when it comes to RAM optimization
https://www.kaggle.com/code/jamesmcguigan/reading-parquet-files-ram-cpu-optimization/
", "rawMarkdown": "@carlmcbrideellis Here are my notes on doing CPU / RAM optimization for large parquet files, including a pandas generator function for batch reading a large dataset (Bengali AI dataset was large but AMEX large). Unsure what needs to be changed to make it work for feather files rather than parquet.\n\nAlso consider being explicit about your column dtypes, as there is a 4x difference between `float64` and `float16` when it comes to RAM optimization\n\nhttps://www.kaggle.com/code/jamesmcguigan/reading-parquet-files-ram-cpu-optimization/", "votes": 2 }, { "id": 1811756, "postDate": "2022-06-05T05:34:34.120Z", "content": "I agree. I mean .csv???
\nI wonder if they are already implementing the techniques fished out of the comp so far at AMEX.
\nCamera pans to data engineers at AMEX reading through the comp discussions and notebooks wide eyed at these new \"parquet\" and \"feather\" techniques…
\nNot to mention the lack of constraints on value types for many variables.
\nI hope this data was not just scraped from their \"actual\" warehouse.
", "rawMarkdown": "I agree. I mean .csv??? \n\nI wonder if they are already implementing the techniques fished out of the comp so far at AMEX. \n\nCamera pans to data engineers at AMEX reading through the comp discussions and notebooks wide eyed at these new \"parquet\" and \"feather\" techniques...\n\nNot to mention the lack of constraints on value types for many variables.\n\nI hope this data was not just scraped from their \"actual\" warehouse.", "votes": 2, "replies": [ { "id": 1811798, "postDate": "2022-06-05T06:37:59.713Z", "content": "Dear @krist0phersmith
\nCSV: Actually my very first notebook for this competition I set out to trying PySpark with SQL queries (having lazy evaluation) by loading the CSV into a table. It is worth mentioning that as of Spark 3.2 one can use pandas via import pyspark.pandas as ps without the need for UDFs. If kaggle wants to add some data engineering flavor to competitions perhaps they could start moving away from monolithic CSV files and provide the competition data via a database? I am sure many people would enjoy trying out their SQL skills on kaggle!
All the best,
\ncarl
The competition needs so much test data because recall-based metrics on unbalanced data are inherently noisy.
\n
The noisiness of the metric can be shown in an experiment. I split the training data into three parts: A training set (80 %) and two validation sets (10 %, i.e. 45891 customers each). Then I trained my gradient booster for 2000 iterations and validated it on both validation sets. The diagram shows that iteration 700 gets a 0.002 higher score with validation set B than with validation set A. Iteration 1600 gets a 0.002 lower score with validation set B than with validation set A.
\nIf you take my two validation sets for public and private leaderboard, you'll get a huge shakeup - you could be at the top of the public leaderboard and end up 0.004 behind on the private leaderboard. The competition would amount to a lottery. Even with ten times more test data, we should be prepared for a shakeup.
\nWhy is the metric so noisy? Recall (default rate captured at 4 %) is defined as true positives divided by real positives. My validation sets have 11883 real positives each (10 % of 118828 positive customers with a stratified split). For both validation sets together, the model produces 16000 true positives. The true positives are assigned to the validation sets at random (i.e. not stratified). It is easily possible that one validation set gets 8050 true positives and the other one 7950. This distribution gives recalls of 8050/11883 = 0.677 and 7950/11883 = 0.669, respectively.
\nWith such a noisy metric, we need a lot of test data to distinguish good models from bad ones and to guarantee a fair competition.
\nBy the way, the noisiness of the metric explains why Kaggle shows only three digits of the score in the leaderboard: Any additional digit would be noise.
", "votes": 26, "replies": [ { "id": 1811804, "author_name": "Carl McBride Ellis", "author_url": "", "post_date": "2022-06-05T06:49:28.180000", "content": "Dear @ambrosm
\nThank you for your magnificent insights, your work is always thought provoking, especially when read in conjunction with your topic \"Graphical explanation of the competition metric\".
\nIf I understand correctly if it takes so much test data to obtain a stable LB score to three significant figures, what implication does this have for the reliability of our cross-validation scores and hold-out score, evidently obtained using substantially less data?
\nAll the best and many thanks,
\ncarl
Dear @ambrosm
\nPS: It is perhaps also worth mentioning that, as spotted both by Raddar, the data has random uniform noise added and also by Chris Deotte in Strange Histograms, so maybe to some extent the evaluation metric is not totally to blame. This uniform noise has a range of ± 0.005.
\nThis noise can be fairly easily extracted. However, it is perhaps also worth noting that having just a little added noise (or \"jitter\") is not necessarily a bad thing, and is sometimes it is even deliberately added to features as a mechanism to reduce overfitting.
\nAll the best,
\ncarl
When inferring test data in a Kaggle notebook, we can read a few million rows at a time and therefore read, process, and infer the test data in parts.
\nall_preds = []\nfor parts in range(NUM_PARTS):\n test_part = pd.read_csv('test', nrows = ROWS, skiprows = SKIP)\n test_part = process(test_part)\n all_preds.append( infer(test_part) )\nI provide an example in my XGB starter notebook here where I read test in 4 parts. When creating 3D data from the 2D provided CSV, then i infer test data in 20 chunks. I provide an example in my GRU starter notebook here
", "votes": 10, "replies": [ { "id": 1811562, "author_name": "Carl McBride Ellis", "author_url": "", "post_date": "2022-06-04T20:28:57.170000", "content": "Dear @cdeotte
\nExcellent suggestion. Worth being careful not to split any customer_ID when creating the chunks, otherwise it could create two predictions for the same customer_ID and shift all the subsequent predictions incorrectly…
All the best,
\ncarl
Yes correct. In my notebook first i read the entire column of customer_ID. Then i find which row positions we can split the list without splitting customers into two different parts. Then i read those rows from the file on disk.
I think the host has given a clear answer to this here, emphasis mine.
\n\n\nWe're excited to launch this competition and publish an industrial scale data for the community to stress test your [modelling] ingenuity
\n
I think the constraints make this competition very interesting, there are a couple of stand-out datasets made by kagglers that can be used where precision and size is balanced allowing competitors to choose if they wish to go for less-is-more or brute-force feature engineering.
\nStarter notebooks achieving very high LB scores also back this statement up! Splitting notebooks into training and inference is also the easiest way to expand on those already excellent notebooks.
", "votes": 3, "replies": [] }, { "id": 1811535, "author_name": "Lucas Morin", "author_url": "", "post_date": "2022-06-04T19:31:30.230000", "content": "Hi @carlmcbrideellis, on top of the usual memory tricks (float32) there is some unusual noise in the data changing some features from categorical to float (process is questionnable at best). @raddar has posted a notebook to remove that noise (and an associated dataset). I use that dataset. After that, yes, the solution is to process data in chunks, for feature engineering and prediction.
", "votes": 3, "replies": [] }, { "id": 1811781, "author_name": "James McGuigan", "author_url": "", "post_date": "2022-06-05T06:07:47.423000", "content": "@carlmcbrideellis Here are my notes on doing CPU / RAM optimization for large parquet files, including a pandas generator function for batch reading a large dataset (Bengali AI dataset was large but AMEX large). Unsure what needs to be changed to make it work for feather files rather than parquet.
\nAlso consider being explicit about your column dtypes, as there is a 4x difference between float64 and float16 when it comes to RAM optimization
https://www.kaggle.com/code/jamesmcguigan/reading-parquet-files-ram-cpu-optimization/
", "votes": 2, "replies": [] }, { "id": 1811756, "author_name": "Kris Smith", "author_url": "", "post_date": "2022-06-05T05:34:34.120000", "content": "I agree. I mean .csv???
\nI wonder if they are already implementing the techniques fished out of the comp so far at AMEX.
\nCamera pans to data engineers at AMEX reading through the comp discussions and notebooks wide eyed at these new \"parquet\" and \"feather\" techniques…
\nNot to mention the lack of constraints on value types for many variables.
\nI hope this data was not just scraped from their \"actual\" warehouse.
", "votes": 2, "replies": [ { "id": 1811798, "author_name": "Carl McBride Ellis", "author_url": "", "post_date": "2022-06-05T06:37:59.713000", "content": "Dear @krist0phersmith
\nCSV: Actually my very first notebook for this competition I set out to trying PySpark with SQL queries (having lazy evaluation) by loading the CSV into a table. It is worth mentioning that as of Spark 3.2 one can use pandas via import pyspark.pandas as ps without the need for UDFs. If kaggle wants to add some data engineering flavor to competitions perhaps they could start moving away from monolithic CSV files and provide the competition data via a database? I am sure many people would enjoy trying out their SQL skills on kaggle!
All the best,
\ncarl