Hi.
\nAs Written in tittle, I would like to ask you how to learn if datasets size is very huge.
\nIf datasets size is huge, it takes a lot of time to train.
\nWhen I want to check a few effects to change parameter a bit, I feel inconvenienced because of taking much time to learn.
\nUsually, split datasets??
Hi.
\nAs Written in tittle, I would like to ask you how to learn if datasets size is very huge.
\nIf datasets size is huge, it takes a lot of time to train.
\nWhen I want to check a few effects to change parameter a bit, I feel inconvenienced because of taking much time to learn.
\nUsually, split datasets??
Hello @haruki741
\nUsually we do Big data while dealing with large datasets with Hadoop or Spark etc.
\nBut I assume you're doing with Python for now? In such case, split the datasets - Be caution, do it randomly and don't split on horizontal axis (as that can give you diff results due to various well known possible factors)
\nObviously split it into two or if it still doesn't hold up good, make it into three….and your work will be done for sure quite smoothly.
Happy Kaggling ^_^
", "rawMarkdown": "Hello @haruki741 \n\nUsually we do Big data while dealing with large datasets with Hadoop or Spark etc. \nBut I assume you're doing with Python for now? In such case, split the datasets - Be caution, do it randomly and don't split on horizontal axis (as that can give you diff results due to various well known possible factors) \nObviously split it into two or if it still doesn't hold up good, make it into three....and your work will be done for sure quite smoothly. \n\nHappy Kaggling ^_^", "votes": null }, { "id": "1713191", "postDate": "03/05/2022 18:25:15", "content": "I would suggest you to take a subset of the original data (a subset which resembles the original data) and do the experimentations on it. When you find that a particular experiment work well on the subset of the data, then try it on the full dataset.
", "rawMarkdown": "I would suggest you to take a subset of the original data (a subset which resembles the original data) and do the experimentations on it. When you find that a particular experiment work well on the subset of the data, then try it on the full dataset.", "votes": null }, { "id": "1713606", "postDate": "03/06/2022 07:33:50", "content": "Thanks for the advise!
\nso, you mean that I should use the subset of the original data, which is not cropped and resized data in case of this competition?
I think you should start on smaller images to begin with. If you really want some performance at start take a subset of the resized data.
", "rawMarkdown": "I think you should start on smaller images to begin with. If you really want some performance at start take a subset of the resized data.", "votes": null }, { "id": "1714451", "postDate": "03/07/2022 03:00:13", "content": "A good GPU is the most important thing , so here is the key to deeplearing ------money , money and money :)
", "rawMarkdown": "A good GPU is the most important thing , so here is the key to deeplearing ------money , money and money :)", "votes": null }, { "id": "1718574", "postDate": "03/11/2022 01:13:14", "content": "Not even a good GPU cuts it these days. It's either a better model/algorithm or better/bigger datacenters
", "rawMarkdown": "Not even a good GPU cuts it these days. It's either a better model/algorithm or better/bigger datacenters", "votes": null } ], "comments": [ { "id": 1712589, "author_name": "surajjha101", "author_url": "", "post_date": "03/05/2022 04:49:40", "content": "Hello @haruki741
\nUsually we do Big data while dealing with large datasets with Hadoop or Spark etc.
\nBut I assume you're doing with Python for now? In such case, split the datasets - Be caution, do it randomly and don't split on horizontal axis (as that can give you diff results due to various well known possible factors)
\nObviously split it into two or if it still doesn't hold up good, make it into three….and your work will be done for sure quite smoothly.
Happy Kaggling ^_^
", "votes": null, "replies": [] }, { "id": 1713191, "author_name": "sadivamadaan", "author_url": "", "post_date": "03/05/2022 18:25:15", "content": "I would suggest you to take a subset of the original data (a subset which resembles the original data) and do the experimentations on it. When you find that a particular experiment work well on the subset of the data, then try it on the full dataset.
", "votes": null, "replies": [ { "id": 1713606, "author_name": "haruki741", "author_url": "", "post_date": "03/06/2022 07:33:50", "content": "Thanks for the advise!
\nso, you mean that I should use the subset of the original data, which is not cropped and resized data in case of this competition?
I think you should start on smaller images to begin with. If you really want some performance at start take a subset of the resized data.
", "votes": null, "replies": [] } ] }, { "id": 1714451, "author_name": "xujingzhao", "author_url": "", "post_date": "03/07/2022 03:00:13", "content": "A good GPU is the most important thing , so here is the key to deeplearing ------money , money and money :)
", "votes": null, "replies": [ { "id": 1718574, "author_name": "dentistdad", "author_url": "", "post_date": "03/11/2022 01:13:14", "content": "Not even a good GPU cuts it these days. It's either a better model/algorithm or better/bigger datacenters
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1712574": "Hi.\nAs Written in tittle, I would like to ask you how to learn if datasets size is very huge.\nIf datasets size is huge, it takes a lot of time to train.\nWhen I want to check a few effects to change parameter a bit, I feel inconvenienced because of taking much time to learn.\nUsually, split datasets??", "1712589": "Hello @haruki741 \n\nUsually we do Big data while dealing with large datasets with Hadoop or Spark etc. \nBut I assume you're doing with Python for now? In such case, split the datasets - Be caution, do it randomly and don't split on horizontal axis (as that can give you diff results due to various well known possible factors) \nObviously split it into two or if it still doesn't hold up good, make it into three....and your work will be done for sure quite smoothly. \n\nHappy Kaggling ^_^", "1713191": "I would suggest you to take a subset of the original data (a subset which resembles the original data) and do the experimentations on it. When you find that a particular experiment work well on the subset of the data, then try it on the full dataset.", "1713606": "Thanks for the advise!\nso, you mean that I should use the subset of the original data, which is not cropped and resized data in case of this competition?", "1714358": "I think you should start on smaller images to begin with. If you really want some performance at start take a subset of the resized data.", "1714451": "A good GPU is the most important thing , so here is the key to deeplearing ------money , money and money :)", "1718574": "Not even a good GPU cuts it these days. It's either a better model/algorithm or better/bigger datacenters" }, "source": "meta" }