It went roughly like this:
\nIn order to spare others the same path (except for the first step perhaps, since you are already here ;-) here's a dataset compressed towards something manageable:
\nhttps://www.kaggle.com/datasets/konradb/pickled-shrunken-training-data
\nGood luck.
", "messages": [ { "id": 1801308, "postDate": "2022-05-25T15:54:25.100Z", "content": "It went roughly like this:
\nIn order to spare others the same path (except for the first step perhaps, since you are already here ;-) here's a dataset compressed towards something manageable:
\nhttps://www.kaggle.com/datasets/konradb/pickled-shrunken-training-data
\nGood luck.
", "rawMarkdown": "It went roughly like this:\n- tweet from @inversion about this comp\n- ecstasy\n- start a new notebook, load entire data\n- !@#%^)_* kind of reaction\n\nIn order to spare others the same path (except for the first step perhaps, since you are already here ;-) here's a dataset compressed towards something manageable:\n\n- all numerical columns cast to float16\n- pickle \n\nhttps://www.kaggle.com/datasets/konradb/pickled-shrunken-training-data\n\nGood luck.\n", "votes": 35 }, { "id": 1801470, "postDate": "2022-05-25T19:33:29.467Z", "content": "Thanks for the pickle @konradb
\nYou can reduce the size even more (~400 Mb) if you convert the categorical columns.
\ncategories = ['B_30', 'B_38', 'D_114', 'D_116', 'D_117', 'D_120', 'D_126', 'D_63', 'D_64', 'D_66', 'D_68']\n\ndf[categories] = df[categories].astype(\"category\")\n@ruchi798 has created a dataset with these column transformations.
", "rawMarkdown": "@ruchi798 has created a dataset [with these column transformations](https://www.kaggle.com/competitions/amex-default-prediction/discussion/327143).\n", "votes": 1 }, { "id": 1802002, "postDate": "2022-05-26T10:39:13.747Z", "content": "@yashlab No, I haven't. Probably @konradb will update his dataset.
", "rawMarkdown": "@yashlab No, I haven't. Probably @konradb will update his dataset." } ] }, { "id": 1801722, "postDate": "2022-05-26T05:00:10.667Z", "content": "Hi,
\nI have created pickled dataset (train & test) through the notebook.
\nhttps://www.kaggle.com/code/aninda/creating-smaller-train-test-data/notebook
", "rawMarkdown": "Hi,\n\nI have created pickled dataset (train & test) through the notebook.\n\nhttps://www.kaggle.com/code/aninda/creating-smaller-train-test-data/notebook\n", "votes": 1 }, { "id": 1900117, "postDate": "2022-08-15T17:59:44.433Z", "content": "@konradb thanks a lot for the file! A laggard speaking here: does this compression to float16 affect in any way the quality of the dataset? I assume the original data was 64 bits float.
", "rawMarkdown": "@konradb thanks a lot for the file! A laggard speaking here: does this compression to float16 affect in any way the quality of the dataset? I assume the original data was 64 bits float.", "replies": [ { "id": 1903365, "postDate": "2022-08-17T11:06:07.950Z", "content": "Either that or 'object', yes.
", "rawMarkdown": "Either that or 'object', yes.", "votes": 1 } ] } ], "comments": [ { "id": 1801470, "author_name": "Peter", "author_url": "", "post_date": "2022-05-25T19:33:29.467000", "content": "Thanks for the pickle @konradb
\nYou can reduce the size even more (~400 Mb) if you convert the categorical columns.
\ncategories = ['B_30', 'B_38', 'D_114', 'D_116', 'D_117', 'D_120', 'D_126', 'D_63', 'D_64', 'D_66', 'D_68']\n\ndf[categories] = df[categories].astype(\"category\")\n@ruchi798 has created a dataset with these column transformations.
", "votes": 1, "replies": [] }, { "id": 1802002, "author_name": "Peter", "author_url": "", "post_date": "2022-05-26T10:39:13.747000", "content": "@yashlab No, I haven't. Probably @konradb will update his dataset.
", "votes": 0, "replies": [] } ] }, { "id": 1801722, "author_name": "Aninda Goswamy", "author_url": "", "post_date": "2022-05-26T05:00:10.667000", "content": "Hi,
\nI have created pickled dataset (train & test) through the notebook.
\nhttps://www.kaggle.com/code/aninda/creating-smaller-train-test-data/notebook
", "votes": 1, "replies": [] }, { "id": 1900117, "author_name": "Antonio Intini", "author_url": "", "post_date": "2022-08-15T17:59:44.433000", "content": "@konradb thanks a lot for the file! A laggard speaking here: does this compression to float16 affect in any way the quality of the dataset? I assume the original data was 64 bits float.
", "votes": 0, "replies": [ { "id": 1903365, "author_name": "Konrad Banachewicz", "author_url": "", "post_date": "2022-08-17T11:06:07.950000", "content": "Either that or 'object', yes.
", "votes": 1, "replies": [] } ] } ], "raw_markdown_by_id": { "1801308": "It went roughly like this:\n- tweet from @inversion about this comp\n- ecstasy\n- start a new notebook, load entire data\n- !@#%^)_* kind of reaction\n\nIn order to spare others the same path (except for the first step perhaps, since you are already here ;-) here's a dataset compressed towards something manageable:\n\n- all numerical columns cast to float16\n- pickle \n\nhttps://www.kaggle.com/datasets/konradb/pickled-shrunken-training-data\n\nGood luck.\n", "1801470": "Thanks for the pickle @konradb \n\nYou can reduce the size even more (~400 Mb) if you convert the categorical columns.\n\n```\ncategories = ['B_30', 'B_38', 'D_114', 'D_116', 'D_117', 'D_120', 'D_126', 'D_63', 'D_64', 'D_66', 'D_68']\n\ndf[categories] = df[categories].astype(\"category\")\n```", "1801722": "Hi,\n\nI have created pickled dataset (train & test) through the notebook.\n\nhttps://www.kaggle.com/code/aninda/creating-smaller-train-test-data/notebook\n", "1900117": "@konradb thanks a lot for the file! A laggard speaking here: does this compression to float16 affect in any way the quality of the dataset? I assume the original data was 64 bits float." } }