When i read the TnC of this competition ,very first thing came in mind how would many kagglers who rely on kaggle resources be able to participate into this competition . As there is no way for them to store huge amount of data . They can only train model using fraction of data so at disadvantage compared to those you can afford paying for big VM instances to store the data.\nI think kaggle should have made some better arrangements to enable more participation from many other kagglers who cant afford training model with huge data.
", "messages": [ { "id": "728007", "postDate": "01/24/2020 10:28:27", "content": "When i read the TnC of this competition ,very first thing came in mind how would many kagglers who rely on kaggle resources be able to participate into this competition . As there is no way for them to store huge amount of data . They can only train model using fraction of data so at disadvantage compared to those you can afford paying for big VM instances to store the data.\nI think kaggle should have made some better arrangements to enable more participation from many other kagglers who cant afford training model with huge data.
", "rawMarkdown": "When i read the TnC of this competition ,very first thing came in mind how would many kagglers who rely on kaggle resources be able to participate into this competition . As there is no way for them to store huge amount of data . They can only train model using fraction of data so at disadvantage compared to those you can afford paying for big VM instances to store the data.\nI think kaggle should have made some better arrangements to enable more participation from many other kagglers who cant afford training model with huge data.", "votes": null }, { "id": "728010", "postDate": "01/24/2020 10:34:02", "content": "Please upvote if u like this suggestion in this post\nhttps://www.kaggle.com/general/127505
", "rawMarkdown": "Please upvote if u like this suggestion in this post\nhttps://www.kaggle.com/general/127505", "votes": null }, { "id": "728012", "postDate": "01/24/2020 10:34:22", "content": "Please see the discussion here: https://www.kaggle.com/c/deepfake-detection-challenge/discussion/121279
", "rawMarkdown": "Please see the discussion here: https://www.kaggle.com/c/deepfake-detection-challenge/discussion/121279", "votes": null }, { "id": "728023", "postDate": "01/24/2020 10:42:23", "content": "well, I think google colab is enough for competing.
", "rawMarkdown": "well, I think google colab is enough for competing.", "votes": null }, { "id": "728125", "postDate": "01/24/2020 12:28:04", "content": "That's why they offer free AWS / TPU credits. ;)
", "rawMarkdown": "That's why they offer free AWS / TPU credits. ;)", "votes": null }, { "id": "728131", "postDate": "01/24/2020 12:35:27", "content": "you cant download more than 40 gb there unlike before when it used to be 370 gb limit of storage. Every one is tightning the hands
", "rawMarkdown": "you cant download more than 40 gb there unlike before when it used to be 370 gb limit of storage. Every one is tightning the hands", "votes": null }, { "id": "728134", "postDate": "01/24/2020 12:36:31", "content": "M not sure about the processing time of TPU compared to GPU but it cost even more than V100 GPU /hour.
\n\nSo it should vanish soon in less than 15-20 days if used heavily for experiments
", "rawMarkdown": "M not sure about the processing time of TPU compared to GPU but it cost even more than V100 GPU /hour. \n\nSo it should vanish soon in less than 15-20 days if used heavily for experiments", "votes": null }, { "id": "728265", "postDate": "01/24/2020 14:49:08", "content": "I don't think Google Colab would be enough!
", "rawMarkdown": "I don't think Google Colab would be enough!", "votes": null }, { "id": "728294", "postDate": "01/24/2020 15:15:23", "content": "I'm using google colab and we can iterate over the dataset (download->train->delete->repeat). And I don't think we need all of the data at a time. Downloading and unzipping takes almost 7 and half minutes. so it's not that much of problem. One more thing, If you create videos with only cropped face of size 224*224 then each dataset part will be of almost 800 mb. and It'll reduce downloading and unzipping time to be less then 1 minute.
", "rawMarkdown": "I'm using google colab and we can iterate over the dataset (download->train->delete->repeat). And I don't think we need all of the data at a time. Downloading and unzipping takes almost 7 and half minutes. so it's not that much of problem. One more thing, If you create videos with only cropped face of size 224*224 then each dataset part will be of almost 800 mb. and It'll reduce downloading and unzipping time to be less then 1 minute.", "votes": null }, { "id": "728298", "postDate": "01/24/2020 15:20:33", "content": "I think it is.
", "rawMarkdown": "I think it is.", "votes": null }, { "id": "728505", "postDate": "01/24/2020 20:09:02", "content": "\n\n\nJaideep wrote:
\n \nM not sure about the processing time of TPU compared to GPU but it cost even more than V100 GPU /hour.
\n \nSo it should vanish soon in less than 15-20 days if used heavily for experiments
\n
I agree with you that those of us have more compute resource has an advantage on this competition, up till this point. however, Kaggle/Google/AWS are doing things to level the playing field.
\n\nThe free TPU credit I have just received give unlimited 60 days of 5 TPU V3-8 on-demand, and 10 preemptible. I would say this free resource would go a very long way of coping with this dataset.
", "rawMarkdown": "> **Jaideep wrote:**\n> \n> M not sure about the processing time of TPU compared to GPU but it cost even more than V100 GPU /hour. \n> \n> So it should vanish soon in less than 15-20 days if used heavily for experiments\n> \n\nI agree with you that those of us have more compute resource has an advantage on this competition, up till this point. however, Kaggle/Google/AWS are doing things to level the playing field.\n\nThe free TPU credit I have just received give unlimited 60 days of 5 TPU V3-8 on-demand, and 10 preemptible. I would say this free resource would go a very long way of coping with this dataset.", "votes": null }, { "id": "729269", "postDate": "01/26/2020 00:46:06", "content": "It seems like Kaggle has done a decent job over the past year to keep data set sizes in a range that most can handle. While a large data set - that does take a bit of time to download - I am not sure that I would call this a \"huge amount of data\". Not with 2TB hard drives selling for $50. But the size does force us to use local PC or a cloud service as only a fractional part of the files can be used directly on Kaggle.
\n\nAt the same time they seem to have done a bad job of letting us share their resources. I speak of course, about the restriction of GPU time to 30 hours per week, the fractional amount of data available on kaggle, etc.
\n\nAs an additional nice touch they are limiting submissions to 2 per day - with some hidden limits (number of files used) that a lack of good error feedback take a while (3 weeks for me) to figure out.
\n\nSo - I think the clear answer to your question is Yes - this competition is best suited to those rich in resources.
", "rawMarkdown": "It seems like Kaggle has done a decent job over the past year to keep data set sizes in a range that most can handle. While a large data set - that does take a bit of time to download - I am not sure that I would call this a \"huge amount of data\". Not with 2TB hard drives selling for $50. But the size does force us to use local PC or a cloud service as only a fractional part of the files can be used directly on Kaggle.\n\nAt the same time they seem to have done a bad job of letting us share their resources. I speak of course, about the restriction of GPU time to 30 hours per week, the fractional amount of data available on kaggle, etc.\n\nAs an additional nice touch they are limiting submissions to 2 per day - with some hidden limits (number of files used) that a lack of good error feedback take a while (3 weeks for me) to figure out.\n\nSo - I think the clear answer to your question is Yes - this competition is best suited to those rich in resources.", "votes": null }, { "id": "732653", "postDate": "01/30/2020 03:34:48", "content": "Hello, how to get data using google colab from kaggle directly? very thanks
", "rawMarkdown": "Hello, how to get data using google colab from kaggle directly? very thanks", "votes": null }, { "id": "732732", "postDate": "01/30/2020 07:05:07", "content": "This might be a bit of an unpopular opinion, but hear me out on this one:
\n\nThis competition isn't really about being fair to everyone. Google, Amazon, Microsoft and all the others that are trying to tackle the issues of deep fakes don't really care IMHO if people without access to resources can compete, because what they are mostly interested in is getting a good solution to this problem. Do not get me wrong, I don't have anything but a low spec laptop from a couple of years ago, so the more accessible compute resources are, the better it also is for me, I'm all for making data science more accessible! But the reality also is, not everything is fair, and from a business point of view, I'm not sure it's worth the investment for those companies to give compute resources to everyone. The prize money is already huge in comparison to other Kaggle competitions, but I would also think that the people / teams capable of producing the best ranking models also have quite some experience and have invested in local hardware for previous competitions / personnal projects.
\n\nOnce again, I'm just saying I think it is better for everyone if competitions are accessible, but these are competitions that have a real world implication after all. The sponsors want the best model possible, and that makes sense from that point of view. But I also am totally on the side of making resources more accessible, trying to get more people on board, and not only the ones that can afford high end hardware.
\n\nThis competition is an very interesting source of discussions and conversations, as well as ideas when looking at the kernels / topics. There's a lot to learn from those even without trying to win, IMHO.
", "rawMarkdown": "This might be a bit of an unpopular opinion, but hear me out on this one:\n\nThis competition isn't really about being fair to everyone. Google, Amazon, Microsoft and all the others that are trying to tackle the issues of deep fakes don't really care IMHO if people without access to resources can compete, because what they are mostly interested in is getting a good solution to this problem. Do not get me wrong, I don't have anything but a low spec laptop from a couple of years ago, so the more accessible compute resources are, the better it also is for me, I'm all for making data science more accessible! But the reality also is, not everything is fair, and from a business point of view, I'm not sure it's worth the investment for those companies to give compute resources to everyone. The prize money is already huge in comparison to other Kaggle competitions, but I would also think that the people / teams capable of producing the best ranking models also have quite some experience and have invested in local hardware for previous competitions / personnal projects.\n\nOnce again, I'm just saying I think it is better for everyone if competitions are accessible, but these are competitions that have a real world implication after all. The sponsors want the best model possible, and that makes sense from that point of view. But I also am totally on the side of making resources more accessible, trying to get more people on board, and not only the ones that can afford high end hardware.\n\nThis competition is an very interesting source of discussions and conversations, as well as ideas when looking at the kernels / topics. There's a lot to learn from those even without trying to win, IMHO.", "votes": null }, { "id": "732772", "postDate": "01/30/2020 08:49:31", "content": "No, it is not. Amazon and google are giving free credits so everyone can participate. Even if you don't have huge resources, get a 2tb hd, pre process images from the videos, upload to kaggle or colab and train there. You might not get first place, but people have reached very good LB placing using very low resources. \nThis competition is actually one of the fairest I have seen. Being code only competition, with blind private dataset, limits the participants from creating ensembles of 20 models - it just wont run in the given time.\nI suggest to think positively, find ways of achieving your goals instead of thinking of what you might have done. Look at kernels and discussions. @Humananalog method could get you into top 50 last week using simple preprocessing on your computer and training on kaggle. I am not talking about just submitting his kernel, I am talking about his training method that could be achived on kaggle if you extract the images on your computer, build a dataset and upload it.
", "rawMarkdown": "No, it is not. Amazon and google are giving free credits so everyone can participate. Even if you don't have huge resources, get a 2tb hd, pre process images from the videos, upload to kaggle or colab and train there. You might not get first place, but people have reached very good LB placing using very low resources. \nThis competition is actually one of the fairest I have seen. Being code only competition, with blind private dataset, limits the participants from creating ensembles of 20 models - it just wont run in the given time.\nI suggest to think positively, find ways of achieving your goals instead of thinking of what you might have done. Look at kernels and discussions. @Humananalog method could get you into top 50 last week using simple preprocessing on your computer and training on kaggle. I am not talking about just submitting his kernel, I am talking about his training method that could be achived on kaggle if you extract the images on your computer, build a dataset and upload it.", "votes": null } ], "comments": [ { "id": 728010, "author_name": "jaideepvalani", "author_url": "", "post_date": "01/24/2020 10:34:02", "content": "Please upvote if u like this suggestion in this post\nhttps://www.kaggle.com/general/127505
", "votes": null, "replies": [] }, { "id": 728012, "author_name": "init27", "author_url": "", "post_date": "01/24/2020 10:34:22", "content": "Please see the discussion here: https://www.kaggle.com/c/deepfake-detection-challenge/discussion/121279
", "votes": null, "replies": [] }, { "id": 728023, "author_name": "ankitsainiankit", "author_url": "", "post_date": "01/24/2020 10:42:23", "content": "well, I think google colab is enough for competing.
", "votes": null, "replies": [ { "id": 728131, "author_name": "jaideepvalani", "author_url": "", "post_date": "01/24/2020 12:35:27", "content": "you cant download more than 40 gb there unlike before when it used to be 370 gb limit of storage. Every one is tightning the hands
", "votes": null, "replies": [] }, { "id": 728294, "author_name": "ankitsainiankit", "author_url": "", "post_date": "01/24/2020 15:15:23", "content": "I'm using google colab and we can iterate over the dataset (download->train->delete->repeat). And I don't think we need all of the data at a time. Downloading and unzipping takes almost 7 and half minutes. so it's not that much of problem. One more thing, If you create videos with only cropped face of size 224*224 then each dataset part will be of almost 800 mb. and It'll reduce downloading and unzipping time to be less then 1 minute.
", "votes": null, "replies": [] }, { "id": 732653, "author_name": "beeaware", "author_url": "", "post_date": "01/30/2020 03:34:48", "content": "Hello, how to get data using google colab from kaggle directly? very thanks
", "votes": null, "replies": [] } ] }, { "id": 728125, "author_name": "dagnelies", "author_url": "", "post_date": "01/24/2020 12:28:04", "content": "That's why they offer free AWS / TPU credits. ;)
", "votes": null, "replies": [ { "id": 728134, "author_name": "jaideepvalani", "author_url": "", "post_date": "01/24/2020 12:36:31", "content": "M not sure about the processing time of TPU compared to GPU but it cost even more than V100 GPU /hour.
\n\nSo it should vanish soon in less than 15-20 days if used heavily for experiments
", "votes": null, "replies": [] }, { "id": 728505, "author_name": "yifanxie", "author_url": "", "post_date": "01/24/2020 20:09:02", "content": "\n\n\nJaideep wrote:
\n \nM not sure about the processing time of TPU compared to GPU but it cost even more than V100 GPU /hour.
\n \nSo it should vanish soon in less than 15-20 days if used heavily for experiments
\n
I agree with you that those of us have more compute resource has an advantage on this competition, up till this point. however, Kaggle/Google/AWS are doing things to level the playing field.
\n\nThe free TPU credit I have just received give unlimited 60 days of 5 TPU V3-8 on-demand, and 10 preemptible. I would say this free resource would go a very long way of coping with this dataset.
", "votes": null, "replies": [] } ] }, { "id": 728265, "author_name": "ashwin88", "author_url": "", "post_date": "01/24/2020 14:49:08", "content": "I don't think Google Colab would be enough!
", "votes": null, "replies": [ { "id": 728298, "author_name": "ankitsainiankit", "author_url": "", "post_date": "01/24/2020 15:20:33", "content": "I think it is.
", "votes": null, "replies": [] } ] }, { "id": 729269, "author_name": "pcjimmmy", "author_url": "", "post_date": "01/26/2020 00:46:06", "content": "It seems like Kaggle has done a decent job over the past year to keep data set sizes in a range that most can handle. While a large data set - that does take a bit of time to download - I am not sure that I would call this a \"huge amount of data\". Not with 2TB hard drives selling for $50. But the size does force us to use local PC or a cloud service as only a fractional part of the files can be used directly on Kaggle.
\n\nAt the same time they seem to have done a bad job of letting us share their resources. I speak of course, about the restriction of GPU time to 30 hours per week, the fractional amount of data available on kaggle, etc.
\n\nAs an additional nice touch they are limiting submissions to 2 per day - with some hidden limits (number of files used) that a lack of good error feedback take a while (3 weeks for me) to figure out.
\n\nSo - I think the clear answer to your question is Yes - this competition is best suited to those rich in resources.
", "votes": null, "replies": [] }, { "id": 732732, "author_name": "maxlenormand", "author_url": "", "post_date": "01/30/2020 07:05:07", "content": "This might be a bit of an unpopular opinion, but hear me out on this one:
\n\nThis competition isn't really about being fair to everyone. Google, Amazon, Microsoft and all the others that are trying to tackle the issues of deep fakes don't really care IMHO if people without access to resources can compete, because what they are mostly interested in is getting a good solution to this problem. Do not get me wrong, I don't have anything but a low spec laptop from a couple of years ago, so the more accessible compute resources are, the better it also is for me, I'm all for making data science more accessible! But the reality also is, not everything is fair, and from a business point of view, I'm not sure it's worth the investment for those companies to give compute resources to everyone. The prize money is already huge in comparison to other Kaggle competitions, but I would also think that the people / teams capable of producing the best ranking models also have quite some experience and have invested in local hardware for previous competitions / personnal projects.
\n\nOnce again, I'm just saying I think it is better for everyone if competitions are accessible, but these are competitions that have a real world implication after all. The sponsors want the best model possible, and that makes sense from that point of view. But I also am totally on the side of making resources more accessible, trying to get more people on board, and not only the ones that can afford high end hardware.
\n\nThis competition is an very interesting source of discussions and conversations, as well as ideas when looking at the kernels / topics. There's a lot to learn from those even without trying to win, IMHO.
", "votes": null, "replies": [] }, { "id": 732772, "author_name": "moshel", "author_url": "", "post_date": "01/30/2020 08:49:31", "content": "No, it is not. Amazon and google are giving free credits so everyone can participate. Even if you don't have huge resources, get a 2tb hd, pre process images from the videos, upload to kaggle or colab and train there. You might not get first place, but people have reached very good LB placing using very low resources. \nThis competition is actually one of the fairest I have seen. Being code only competition, with blind private dataset, limits the participants from creating ensembles of 20 models - it just wont run in the given time.\nI suggest to think positively, find ways of achieving your goals instead of thinking of what you might have done. Look at kernels and discussions. @Humananalog method could get you into top 50 last week using simple preprocessing on your computer and training on kaggle. I am not talking about just submitting his kernel, I am talking about his training method that could be achived on kaggle if you extract the images on your computer, build a dataset and upload it.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "728007": "When i read the TnC of this competition ,very first thing came in mind how would many kagglers who rely on kaggle resources be able to participate into this competition . As there is no way for them to store huge amount of data . They can only train model using fraction of data so at disadvantage compared to those you can afford paying for big VM instances to store the data.\nI think kaggle should have made some better arrangements to enable more participation from many other kagglers who cant afford training model with huge data.", "728010": "Please upvote if u like this suggestion in this post\nhttps://www.kaggle.com/general/127505", "728012": "Please see the discussion here: https://www.kaggle.com/c/deepfake-detection-challenge/discussion/121279", "728023": "well, I think google colab is enough for competing.", "728125": "That's why they offer free AWS / TPU credits. ;)", "728131": "you cant download more than 40 gb there unlike before when it used to be 370 gb limit of storage. Every one is tightning the hands", "728134": "M not sure about the processing time of TPU compared to GPU but it cost even more than V100 GPU /hour. \n\nSo it should vanish soon in less than 15-20 days if used heavily for experiments", "728265": "I don't think Google Colab would be enough!", "728294": "I'm using google colab and we can iterate over the dataset (download->train->delete->repeat). And I don't think we need all of the data at a time. Downloading and unzipping takes almost 7 and half minutes. so it's not that much of problem. One more thing, If you create videos with only cropped face of size 224*224 then each dataset part will be of almost 800 mb. and It'll reduce downloading and unzipping time to be less then 1 minute.", "728298": "I think it is.", "728505": "> **Jaideep wrote:**\n> \n> M not sure about the processing time of TPU compared to GPU but it cost even more than V100 GPU /hour. \n> \n> So it should vanish soon in less than 15-20 days if used heavily for experiments\n> \n\nI agree with you that those of us have more compute resource has an advantage on this competition, up till this point. however, Kaggle/Google/AWS are doing things to level the playing field.\n\nThe free TPU credit I have just received give unlimited 60 days of 5 TPU V3-8 on-demand, and 10 preemptible. I would say this free resource would go a very long way of coping with this dataset.", "729269": "It seems like Kaggle has done a decent job over the past year to keep data set sizes in a range that most can handle. While a large data set - that does take a bit of time to download - I am not sure that I would call this a \"huge amount of data\". Not with 2TB hard drives selling for $50. But the size does force us to use local PC or a cloud service as only a fractional part of the files can be used directly on Kaggle.\n\nAt the same time they seem to have done a bad job of letting us share their resources. I speak of course, about the restriction of GPU time to 30 hours per week, the fractional amount of data available on kaggle, etc.\n\nAs an additional nice touch they are limiting submissions to 2 per day - with some hidden limits (number of files used) that a lack of good error feedback take a while (3 weeks for me) to figure out.\n\nSo - I think the clear answer to your question is Yes - this competition is best suited to those rich in resources.", "732653": "Hello, how to get data using google colab from kaggle directly? very thanks", "732732": "This might be a bit of an unpopular opinion, but hear me out on this one:\n\nThis competition isn't really about being fair to everyone. Google, Amazon, Microsoft and all the others that are trying to tackle the issues of deep fakes don't really care IMHO if people without access to resources can compete, because what they are mostly interested in is getting a good solution to this problem. Do not get me wrong, I don't have anything but a low spec laptop from a couple of years ago, so the more accessible compute resources are, the better it also is for me, I'm all for making data science more accessible! But the reality also is, not everything is fair, and from a business point of view, I'm not sure it's worth the investment for those companies to give compute resources to everyone. The prize money is already huge in comparison to other Kaggle competitions, but I would also think that the people / teams capable of producing the best ranking models also have quite some experience and have invested in local hardware for previous competitions / personnal projects.\n\nOnce again, I'm just saying I think it is better for everyone if competitions are accessible, but these are competitions that have a real world implication after all. The sponsors want the best model possible, and that makes sense from that point of view. But I also am totally on the side of making resources more accessible, trying to get more people on board, and not only the ones that can afford high end hardware.\n\nThis competition is an very interesting source of discussions and conversations, as well as ideas when looking at the kernels / topics. There's a lot to learn from those even without trying to win, IMHO.", "732772": "No, it is not. Amazon and google are giving free credits so everyone can participate. Even if you don't have huge resources, get a 2tb hd, pre process images from the videos, upload to kaggle or colab and train there. You might not get first place, but people have reached very good LB placing using very low resources. \nThis competition is actually one of the fairest I have seen. Being code only competition, with blind private dataset, limits the participants from creating ensembles of 20 models - it just wont run in the given time.\nI suggest to think positively, find ways of achieving your goals instead of thinking of what you might have done. Look at kernels and discussions. @Humananalog method could get you into top 50 last week using simple preprocessing on your computer and training on kaggle. I am not talking about just submitting his kernel, I am talking about his training method that could be achived on kaggle if you extract the images on your computer, build a dataset and upload it." }, "source": "meta" }