{ "id": 307848, "title": "What should be my approach?", "url": "/competitions/h-and-m-personalized-fashion-recommendations/discussion/307848", "author_name": "", "post_date": "2022-02-15T21:45:11.781865300Z", "votes": 8, "comment_count": 6, "views": 0, "content": "

How should I approach competition with such a large dataset?
\nI mainly use the Kaggle notebook to perform EDA and Model building how should I approach this competition with 16GB ram when the dataset is 34.5GB?

\n

Thank You!

", "messages": [ { "id": "1692169", "postDate": "02/15/2022 21:45:11", "content": "

How should I approach competition with such a large dataset?
\nI mainly use the Kaggle notebook to perform EDA and Model building how should I approach this competition with 16GB ram when the dataset is 34.5GB?

\n

Thank You!

", "rawMarkdown": "How should I approach competition with such a large dataset?\nI mainly use the Kaggle notebook to perform EDA and Model building how should I approach this competition with 16GB ram when the dataset is 34.5GB?\n\nThank You!", "votes": null }, { "id": "1692261", "postDate": "02/15/2022 23:40:08", "content": "

Most of the dataset size is images, so you can start with tabular data without any problem.

\n

Even for images, you don't load them all into memory in one shot - you create a PyTorch DataLoader or Tensorflow Dataset object that loads the images in batches and feeds them to a model.

", "rawMarkdown": "Most of the dataset size is images, so you can start with tabular data without any problem.\n\nEven for images, you don't load them all into memory in one shot - you create a PyTorch DataLoader or Tensorflow Dataset object that loads the images in batches and feeds them to a model.", "votes": null }, { "id": "1692272", "postDate": "02/16/2022 00:00:55", "content": "

Thank You for Heads up

", "rawMarkdown": "Thank You for Heads up", "votes": null }, { "id": "1692612", "postDate": "02/16/2022 06:25:01", "content": "

As @jacob34 kindly suggested-start with a simple baseline (just using tabular data) and then add interesting things.

", "rawMarkdown": "As @jacob34 kindly suggested-start with a simple baseline (just using tabular data) and then add interesting things.", "votes": null }, { "id": "1692672", "postDate": "02/16/2022 07:05:05", "content": "

I too had the similar doubt, thanks for asking!!

", "rawMarkdown": "I too had the similar doubt, thanks for asking!!", "votes": null }, { "id": "1693052", "postDate": "02/16/2022 12:12:08", "content": "

Thanks for Asking

", "rawMarkdown": "Thanks for Asking", "votes": null }, { "id": "1693242", "postDate": "02/16/2022 14:43:39", "content": "

You can use the samples I created here to play around with data https://www.kaggle.com/paweljankiewicz/hm-create-dataset-samples. But eventually you will need much bigger machine to win this competition :).

", "rawMarkdown": "You can use the samples I created here to play around with data https://www.kaggle.com/paweljankiewicz/hm-create-dataset-samples. But eventually you will need much bigger machine to win this competition :).", "votes": null } ], "comments": [ { "id": 1692261, "author_name": "jacob34", "author_url": "", "post_date": "02/15/2022 23:40:08", "content": "

Most of the dataset size is images, so you can start with tabular data without any problem.

\n

Even for images, you don't load them all into memory in one shot - you create a PyTorch DataLoader or Tensorflow Dataset object that loads the images in batches and feeds them to a model.

", "votes": null, "replies": [ { "id": 1692272, "author_name": "yekahaaagayeham", "author_url": "", "post_date": "02/16/2022 00:00:55", "content": "

Thank You for Heads up

", "votes": null, "replies": [] }, { "id": 1692612, "author_name": "init27", "author_url": "", "post_date": "02/16/2022 06:25:01", "content": "

As @jacob34 kindly suggested-start with a simple baseline (just using tabular data) and then add interesting things.

", "votes": null, "replies": [] } ] }, { "id": 1692672, "author_name": "lallucycle", "author_url": "", "post_date": "02/16/2022 07:05:05", "content": "

I too had the similar doubt, thanks for asking!!

", "votes": null, "replies": [] }, { "id": 1693052, "author_name": "agyatneuron", "author_url": "", "post_date": "02/16/2022 12:12:08", "content": "

Thanks for Asking

", "votes": null, "replies": [] }, { "id": 1693242, "author_name": "paweljankiewicz", "author_url": "", "post_date": "02/16/2022 14:43:39", "content": "

You can use the samples I created here to play around with data https://www.kaggle.com/paweljankiewicz/hm-create-dataset-samples. But eventually you will need much bigger machine to win this competition :).

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1692169": "How should I approach competition with such a large dataset?\nI mainly use the Kaggle notebook to perform EDA and Model building how should I approach this competition with 16GB ram when the dataset is 34.5GB?\n\nThank You!", "1692261": "Most of the dataset size is images, so you can start with tabular data without any problem.\n\nEven for images, you don't load them all into memory in one shot - you create a PyTorch DataLoader or Tensorflow Dataset object that loads the images in batches and feeds them to a model.", "1692272": "Thank You for Heads up", "1692612": "As @jacob34 kindly suggested-start with a simple baseline (just using tabular data) and then add interesting things.", "1692672": "I too had the similar doubt, thanks for asking!!", "1693052": "Thanks for Asking", "1693242": "You can use the samples I created here to play around with data https://www.kaggle.com/paweljankiewicz/hm-create-dataset-samples. But eventually you will need much bigger machine to win this competition :)." }, "source": "meta" }