This is almost 17th time in the last 5 days. \nWhat's the matter with this competition. Why am I unable to load the dataset by any means?
\n\nThe kernel dies or runs out of resources every time. I'm going crazy.
\n\nI don't have monster machine and Kaggle or Colab are my only hope.
\n\nSomeone advice me some efficient technique to load the dataset without crashing the kernel.
", "messages": [ { "id": "573684", "postDate": "07/12/2019 15:53:26", "content": "This is almost 17th time in the last 5 days. \nWhat's the matter with this competition. Why am I unable to load the dataset by any means?
\n\nThe kernel dies or runs out of resources every time. I'm going crazy.
\n\nI don't have monster machine and Kaggle or Colab are my only hope.
\n\nSomeone advice me some efficient technique to load the dataset without crashing the kernel.
", "rawMarkdown": "This is almost 17th time in the last 5 days. \nWhat's the matter with this competition. Why am I unable to load the dataset by any means?\n\nThe kernel dies or runs out of resources every time. I'm going **crazy.**\n\nI don't have monster machine and Kaggle or Colab are my only hope. \n\nSomeone advice me some efficient technique to load the dataset without crashing the kernel.", "votes": null }, { "id": "573689", "postDate": "07/12/2019 15:57:22", "content": "Can you give a little bit more details about what are you trying to upload?
", "rawMarkdown": "Can you give a little bit more details about what are you trying to upload?", "votes": null }, { "id": "573711", "postDate": "07/12/2019 16:40:54", "content": "what are the image size and batch size of the dataset that you are trying to load?
", "rawMarkdown": "what are the image size and batch size of the dataset that you are trying to load?", "votes": null }, { "id": "573808", "postDate": "07/12/2019 19:29:08", "content": "Just for a simple investigation: \ncan you fork a simple kernel from others and run it as well as submit?\nIf no, you can report the issue to Kaggle!\nI hope...
", "rawMarkdown": "Just for a simple investigation: \ncan you fork a simple kernel from others and run it as well as submit?\nIf no, you can report the issue to Kaggle!\nI hope...", "votes": null }, { "id": "573926", "postDate": "07/13/2019 00:53:08", "content": "Don't worry, our kaggle avenger team now just arrived! Please give more details as my fellows stated in other posts.
", "rawMarkdown": "Don't worry, our kaggle avenger team now just arrived! Please give more details as my fellows stated in other posts.", "votes": null }, { "id": "573936", "postDate": "07/13/2019 01:40:11", "content": "Are you uploading the data or using the AddDataset option cause in the latter option there wont likely be an issue,uploading the dataset does cause some trouble at times.
", "rawMarkdown": "Are you uploading the data or using the AddDataset option cause in the latter option there wont likely be an issue,uploading the dataset does cause some trouble at times.", "votes": null }, { "id": "574069", "postDate": "07/13/2019 07:24:51", "content": "I am just trying to read in the dataset on a kaggle kernel itself and soon as I am done with reading the dataset the kernel crashes. \nI have tried many different codes for the same. Be it \"Image Data Bunch\" from Fastai , \"Flow from directory\" by Keras , Dataloader by \"Pytorch\" or in simple python and opencv code making use of list comprehensions and lambda expression. No matter what I do I end up failing to load the dataset.
", "rawMarkdown": "I am just trying to read in the dataset on a kaggle kernel itself and soon as I am done with reading the dataset the kernel crashes. \nI have tried many different codes for the same. Be it \"Image Data Bunch\" from Fastai , \"Flow from directory\" by Keras , Dataloader by \"Pytorch\" or in simple python and opencv code making use of list comprehensions and lambda expression. No matter what I do I end up failing to load the dataset.", "votes": null }, { "id": "574071", "postDate": "07/13/2019 07:28:26", "content": "Used batch_size and image size of 16, 224 * 224 * 3 for Pytorch Dataloder, same goes for Fastai and 32, 224 * 224 * 3 for Keras flow from directory. \nDon't really think that's the main issue. \nThough would be glad if you can give me some directions. Thanks in advance!
", "rawMarkdown": "Used batch_size and image size of 16, 224 * 224 * 3 for Pytorch Dataloder, same goes for Fastai and 32, 224 * 224 * 3 for Keras flow from directory. \nDon't really think that's the main issue. \nThough would be glad if you can give me some directions. Thanks in advance!", "votes": null }, { "id": "574072", "postDate": "07/13/2019 07:29:48", "content": "Not sure if I understand correctly, but do you mean you wanted to load all of them into memory (RAM)? if yes, you have to resize them into a much smaller one (e.g. 224x224)
\n\nEDIT: I replied too fast, I just saw you are currently replying to other people ... I’ll wait for the full information.
", "rawMarkdown": "Not sure if I understand correctly, but do you mean you wanted to load all of them into memory (RAM)? if yes, you have to resize them into a much smaller one (e.g. 224x224)\n\nEDIT: I replied too fast, I just saw you are currently replying to other people ... I’ll wait for the full information.", "votes": null }, { "id": "574073", "postDate": "07/13/2019 07:30:56", "content": "😂 Forking a kernel is what I had planned for today. Thanks for the advice and yup if that doesn't works out I'm taking a flight to Kaggle HQ.
", "rawMarkdown": "😂 Forking a kernel is what I had planned for today. Thanks for the advice and yup if that doesn't works out I'm taking a flight to Kaggle HQ.", "votes": null }, { "id": "574075", "postDate": "07/13/2019 07:33:04", "content": "Can you give some idea on how am I supposed to do that. I am confused.
", "rawMarkdown": "Can you give some idea on how am I supposed to do that. I am confused.", "votes": null }, { "id": "574080", "postDate": "07/13/2019 07:46:18", "content": "Thanks buddy, don't know how but its working now.
", "rawMarkdown": "Thanks buddy, don't know how but its working now.", "votes": null }, { "id": "574518", "postDate": "07/14/2019 02:31:07", "content": "There is a lot of public kernels could run a valid submit with keras|pytorch|fastai , why don't you use them directly.
", "rawMarkdown": "There is a lot of public kernels could run a valid submit with keras|pytorch|fastai , why don't you use them directly.", "votes": null }, { "id": "574639", "postDate": "07/14/2019 08:25:09", "content": "That's what helped me get through this. Thank you :)
", "rawMarkdown": "That's what helped me get through this. Thank you :)", "votes": null }, { "id": "574682", "postDate": "07/14/2019 10:18:58", "content": "I don't know which framework do you use, but there should be a pytorch kernel with score 0.75 and a fastai kernel with score 0.71 , you can improve your score if possible
", "rawMarkdown": "I don't know which framework do you use, but there should be a pytorch kernel with score 0.75 and a fastai kernel with score 0.71 , you can improve your score if possible", "votes": null }, { "id": "575247", "postDate": "07/15/2019 07:59:28", "content": "If you don't have a monster machine, I think that Kaggle Kernels is a supercool solution for you. They have powerful GPUs, and you can run 4 at the same time. So it's basically like you have a server with 4 GPUs :D More to say, Kernels even made possible to solve such a competition with enormous dataset as https://www.kaggle.com/c/landmark-recognition-2019/discussion/95077#latest-551166 . So get familiar and you won't regret :)
\n\nAnd of course you can use some public kernel as a baseline and start to build your pipeline from it
", "rawMarkdown": "If you don't have a monster machine, I think that Kaggle Kernels is a supercool solution for you. They have powerful GPUs, and you can run **4 at the same time**. So it's basically like you have a server with 4 GPUs :D More to say, Kernels even made possible to solve such a competition with enormous dataset as https://www.kaggle.com/c/landmark-recognition-2019/discussion/95077#latest-551166 . So get familiar and you won't regret :)\n\nAnd of course you can use some public kernel as a baseline and start to build your pipeline from it", "votes": null }, { "id": "578101", "postDate": "07/17/2019 10:51:46", "content": "If it hadn't been this guy I would've never believed that something like download-train-discard methodology can ever work just right. I have kinda tried to take this approach before but didn't work. At that time committing the kernel failed, but now I know why that happened. While committing we need the whole data thus we need to do the inference somewhere else. \nYet kaggle kernels have been running out resources or breaking down frequently recently. Yet I'll try to do this with the open images challenges once more. Thanks @blackitten13 for this.
", "rawMarkdown": "If it hadn't been this guy I would've never believed that something like **download-train-discard methodology** can ever work just right. I have kinda tried to take this approach before but didn't work. At that time committing the kernel failed, but now I know why that happened. While committing we need the whole data thus we need to do the inference somewhere else. \nYet kaggle kernels have been running out resources or breaking down frequently recently. Yet I'll try to do this with the open images challenges once more. Thanks @blackitten13 for this.", "votes": null } ], "comments": [ { "id": 573689, "author_name": "drhabib", "author_url": "", "post_date": "07/12/2019 15:57:22", "content": "Can you give a little bit more details about what are you trying to upload?
", "votes": null, "replies": [ { "id": 574069, "author_name": "rhtsingh", "author_url": "", "post_date": "07/13/2019 07:24:51", "content": "I am just trying to read in the dataset on a kaggle kernel itself and soon as I am done with reading the dataset the kernel crashes. \nI have tried many different codes for the same. Be it \"Image Data Bunch\" from Fastai , \"Flow from directory\" by Keras , Dataloader by \"Pytorch\" or in simple python and opencv code making use of list comprehensions and lambda expression. No matter what I do I end up failing to load the dataset.
", "votes": null, "replies": [] }, { "id": 574072, "author_name": "ratthachat", "author_url": "", "post_date": "07/13/2019 07:29:48", "content": "Not sure if I understand correctly, but do you mean you wanted to load all of them into memory (RAM)? if yes, you have to resize them into a much smaller one (e.g. 224x224)
\n\nEDIT: I replied too fast, I just saw you are currently replying to other people ... I’ll wait for the full information.
", "votes": null, "replies": [] }, { "id": 574075, "author_name": "rhtsingh", "author_url": "", "post_date": "07/13/2019 07:33:04", "content": "Can you give some idea on how am I supposed to do that. I am confused.
", "votes": null, "replies": [] }, { "id": 574080, "author_name": "rhtsingh", "author_url": "", "post_date": "07/13/2019 07:46:18", "content": "Thanks buddy, don't know how but its working now.
", "votes": null, "replies": [] } ] }, { "id": 573711, "author_name": "projdev", "author_url": "", "post_date": "07/12/2019 16:40:54", "content": "what are the image size and batch size of the dataset that you are trying to load?
", "votes": null, "replies": [ { "id": 574071, "author_name": "rhtsingh", "author_url": "", "post_date": "07/13/2019 07:28:26", "content": "Used batch_size and image size of 16, 224 * 224 * 3 for Pytorch Dataloder, same goes for Fastai and 32, 224 * 224 * 3 for Keras flow from directory. \nDon't really think that's the main issue. \nThough would be glad if you can give me some directions. Thanks in advance!
", "votes": null, "replies": [] } ] }, { "id": 573808, "author_name": "esmaeil391", "author_url": "", "post_date": "07/12/2019 19:29:08", "content": "Just for a simple investigation: \ncan you fork a simple kernel from others and run it as well as submit?\nIf no, you can report the issue to Kaggle!\nI hope...
", "votes": null, "replies": [ { "id": 574073, "author_name": "rhtsingh", "author_url": "", "post_date": "07/13/2019 07:30:56", "content": "😂 Forking a kernel is what I had planned for today. Thanks for the advice and yup if that doesn't works out I'm taking a flight to Kaggle HQ.
", "votes": null, "replies": [] } ] }, { "id": 573926, "author_name": "ratthachat", "author_url": "", "post_date": "07/13/2019 00:53:08", "content": "Don't worry, our kaggle avenger team now just arrived! Please give more details as my fellows stated in other posts.
", "votes": null, "replies": [] }, { "id": 573936, "author_name": "amardeepganguly", "author_url": "", "post_date": "07/13/2019 01:40:11", "content": "Are you uploading the data or using the AddDataset option cause in the latter option there wont likely be an issue,uploading the dataset does cause some trouble at times.
", "votes": null, "replies": [] }, { "id": 574518, "author_name": "xujingzhao", "author_url": "", "post_date": "07/14/2019 02:31:07", "content": "There is a lot of public kernels could run a valid submit with keras|pytorch|fastai , why don't you use them directly.
", "votes": null, "replies": [ { "id": 574639, "author_name": "rhtsingh", "author_url": "", "post_date": "07/14/2019 08:25:09", "content": "That's what helped me get through this. Thank you :)
", "votes": null, "replies": [] }, { "id": 574682, "author_name": "xujingzhao", "author_url": "", "post_date": "07/14/2019 10:18:58", "content": "I don't know which framework do you use, but there should be a pytorch kernel with score 0.75 and a fastai kernel with score 0.71 , you can improve your score if possible
", "votes": null, "replies": [] } ] }, { "id": 575247, "author_name": "blackitten13", "author_url": "", "post_date": "07/15/2019 07:59:28", "content": "If you don't have a monster machine, I think that Kaggle Kernels is a supercool solution for you. They have powerful GPUs, and you can run 4 at the same time. So it's basically like you have a server with 4 GPUs :D More to say, Kernels even made possible to solve such a competition with enormous dataset as https://www.kaggle.com/c/landmark-recognition-2019/discussion/95077#latest-551166 . So get familiar and you won't regret :)
\n\nAnd of course you can use some public kernel as a baseline and start to build your pipeline from it
", "votes": null, "replies": [ { "id": 578101, "author_name": "rhtsingh", "author_url": "", "post_date": "07/17/2019 10:51:46", "content": "If it hadn't been this guy I would've never believed that something like download-train-discard methodology can ever work just right. I have kinda tried to take this approach before but didn't work. At that time committing the kernel failed, but now I know why that happened. While committing we need the whole data thus we need to do the inference somewhere else. \nYet kaggle kernels have been running out resources or breaking down frequently recently. Yet I'll try to do this with the open images challenges once more. Thanks @blackitten13 for this.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "573684": "This is almost 17th time in the last 5 days. \nWhat's the matter with this competition. Why am I unable to load the dataset by any means?\n\nThe kernel dies or runs out of resources every time. I'm going **crazy.**\n\nI don't have monster machine and Kaggle or Colab are my only hope. \n\nSomeone advice me some efficient technique to load the dataset without crashing the kernel.", "573689": "Can you give a little bit more details about what are you trying to upload?", "573711": "what are the image size and batch size of the dataset that you are trying to load?", "573808": "Just for a simple investigation: \ncan you fork a simple kernel from others and run it as well as submit?\nIf no, you can report the issue to Kaggle!\nI hope...", "573926": "Don't worry, our kaggle avenger team now just arrived! Please give more details as my fellows stated in other posts.", "573936": "Are you uploading the data or using the AddDataset option cause in the latter option there wont likely be an issue,uploading the dataset does cause some trouble at times.", "574069": "I am just trying to read in the dataset on a kaggle kernel itself and soon as I am done with reading the dataset the kernel crashes. \nI have tried many different codes for the same. Be it \"Image Data Bunch\" from Fastai , \"Flow from directory\" by Keras , Dataloader by \"Pytorch\" or in simple python and opencv code making use of list comprehensions and lambda expression. No matter what I do I end up failing to load the dataset.", "574071": "Used batch_size and image size of 16, 224 * 224 * 3 for Pytorch Dataloder, same goes for Fastai and 32, 224 * 224 * 3 for Keras flow from directory. \nDon't really think that's the main issue. \nThough would be glad if you can give me some directions. Thanks in advance!", "574072": "Not sure if I understand correctly, but do you mean you wanted to load all of them into memory (RAM)? if yes, you have to resize them into a much smaller one (e.g. 224x224)\n\nEDIT: I replied too fast, I just saw you are currently replying to other people ... I’ll wait for the full information.", "574073": "😂 Forking a kernel is what I had planned for today. Thanks for the advice and yup if that doesn't works out I'm taking a flight to Kaggle HQ.", "574075": "Can you give some idea on how am I supposed to do that. I am confused.", "574080": "Thanks buddy, don't know how but its working now.", "574518": "There is a lot of public kernels could run a valid submit with keras|pytorch|fastai , why don't you use them directly.", "574639": "That's what helped me get through this. Thank you :)", "574682": "I don't know which framework do you use, but there should be a pytorch kernel with score 0.75 and a fastai kernel with score 0.71 , you can improve your score if possible", "575247": "If you don't have a monster machine, I think that Kaggle Kernels is a supercool solution for you. They have powerful GPUs, and you can run **4 at the same time**. So it's basically like you have a server with 4 GPUs :D More to say, Kernels even made possible to solve such a competition with enormous dataset as https://www.kaggle.com/c/landmark-recognition-2019/discussion/95077#latest-551166 . So get familiar and you won't regret :)\n\nAnd of course you can use some public kernel as a baseline and start to build your pipeline from it", "578101": "If it hadn't been this guy I would've never believed that something like **download-train-discard methodology** can ever work just right. I have kinda tried to take this approach before but didn't work. At that time committing the kernel failed, but now I know why that happened. While committing we need the whole data thus we need to do the inference somewhere else. \nYet kaggle kernels have been running out resources or breaking down frequently recently. Yet I'll try to do this with the open images challenges once more. Thanks @blackitten13 for this." }, "source": "meta" }