Hi everybody, here is a link to my notebook:\nhttps://www.kaggle.com/linainversez/bengali-fast-ai-inference
\n\nI keep getting \"Submission Scoring Error\" on all of my kernels, and I have tried pretty much everything, so I am opening the floor up to any of your kind suggestions.
\n\nThe notebook above uses CPU to compute the predictions. Although it's really slow, I tried it on the training set and it should take ~4 hrs to complete, which is less time than the 9 hour window. However, the submission mysteriously craps out at around ~2 hrs.
\n\nI have also tried using GPU, using the lafoss notebook / others as a guide. However, I am having a lot of trouble applying the same transformations to the data that fastai does. I've done a lot of digging into the source code, and it seems that fastai really wants people to load datasets from files, which is not the case in this competition. Also, since I am trying a single-output CNN, I haven't really found a lot of code that I can copy.
\n\nI'll probably start over with a new pipeline if this doesn't work out, but I was wondering if anybody had suggestions before I scrapped the thing.
\n\nThanks,\nBryan
", "messages": [ { "id": "725332", "postDate": "01/22/2020 02:05:45", "content": "Hi everybody, here is a link to my notebook:\nhttps://www.kaggle.com/linainversez/bengali-fast-ai-inference
\n\nI keep getting \"Submission Scoring Error\" on all of my kernels, and I have tried pretty much everything, so I am opening the floor up to any of your kind suggestions.
\n\nThe notebook above uses CPU to compute the predictions. Although it's really slow, I tried it on the training set and it should take ~4 hrs to complete, which is less time than the 9 hour window. However, the submission mysteriously craps out at around ~2 hrs.
\n\nI have also tried using GPU, using the lafoss notebook / others as a guide. However, I am having a lot of trouble applying the same transformations to the data that fastai does. I've done a lot of digging into the source code, and it seems that fastai really wants people to load datasets from files, which is not the case in this competition. Also, since I am trying a single-output CNN, I haven't really found a lot of code that I can copy.
\n\nI'll probably start over with a new pipeline if this doesn't work out, but I was wondering if anybody had suggestions before I scrapped the thing.
\n\nThanks,\nBryan
", "rawMarkdown": "Hi everybody, here is a link to my notebook:\nhttps://www.kaggle.com/linainversez/bengali-fast-ai-inference\n\nI keep getting \"Submission Scoring Error\" on all of my kernels, and I have tried pretty much everything, so I am opening the floor up to any of your kind suggestions. \n\nThe notebook above uses CPU to compute the predictions. Although it's really slow, I tried it on the training set and it should take ~4 hrs to complete, which is less time than the 9 hour window. However, the submission mysteriously craps out at around ~2 hrs.\n\nI have also tried using GPU, using the lafoss notebook / others as a guide. However, I am having a lot of trouble applying the same transformations to the data that fastai does. I've done a lot of digging into the source code, and it seems that fastai really wants people to load datasets from files, which is not the case in this competition. Also, since I am trying a single-output CNN, I haven't really found a lot of code that I can copy.\n\nI'll probably start over with a new pipeline if this doesn't work out, but I was wondering if anybody had suggestions before I scrapped the thing.\n\nThanks,\nBryan", "votes": null }, { "id": "725366", "postDate": "01/22/2020 03:05:16", "content": "\ntest = pd.read_parquet('/kaggle/input/bengaliai-cv19/test_image_data_{}.parquet'.format(i))\ndata = test.iloc[:, 1:].values.reshape(-1, HEIGHT, WIDTH).astype(np.uint8)\ndel test\ngc.collect()\n
Try this?
", "rawMarkdown": "```\ntest = pd.read_parquet('/kaggle/input/bengaliai-cv19/test_image_data_{}.parquet'.format(i))\ndata = test.iloc[:, 1:].values.reshape(-1, HEIGHT, WIDTH).astype(np.uint8)\ndel test\ngc.collect()\n```\n\nTry this?", "votes": null }, { "id": "725369", "postDate": "01/22/2020 03:09:59", "content": "Hi Qishen,
\n\nI tried this in a previous version but it did not work :(. Thanks for the suggestion though!
", "rawMarkdown": "Hi Qishen,\n\nI tried this in a previous version but it did not work :(. Thanks for the suggestion though!", "votes": null }, { "id": "725395", "postDate": "01/22/2020 03:41:07", "content": "maybe this work
\n\n```
\n\ntest = pd.read_parquet('/kaggle/input/bengaliai-cv19/test_image_data_{}.parquet'.format(i))\nfor b in ...\n input= test.iloc[b:b+batch_size, 1:].values.reshape(-1, HEIGHT, WIDTH).astype(np.uint8)\n output = net(input)
\n\n```
", "rawMarkdown": "maybe this work\n\n```\n\ntest = pd.read_parquet('/kaggle/input/bengaliai-cv19/test_image_data_{}.parquet'.format(i))\nfor b in ...\n input= test.iloc[b:b+batch_size, 1:].values.reshape(-1, HEIGHT, WIDTH).astype(np.uint8)\n output = net(input)\n\n\n```", "votes": null }, { "id": "725418", "postDate": "01/22/2020 04:35:18", "content": "will try this, thanks!
", "rawMarkdown": "will try this, thanks!", "votes": null }, { "id": "725452", "postDate": "01/22/2020 05:44:59", "content": "In my case, I kept getting these errors if I used any image size greater than 128x128. Sometimes I got \"Notebook Exceeded Allowed Compute\" or \"Submission CSV Not Found\". Maybe you too can try reducing the image size.
\n\nAlso, please double check that you're writing values for all the private set entries in your submission.csv
\n\nCheers!
", "rawMarkdown": "In my case, I kept getting these errors if I used any image size greater than 128x128. Sometimes I got \"Notebook Exceeded Allowed Compute\" or \"Submission CSV Not Found\". Maybe you too can try reducing the image size.\n\nAlso, please double check that you're writing values for all the private set entries in your submission.csv\n\nCheers!", "votes": null }, { "id": "725935", "postDate": "01/22/2020 16:23:54", "content": "thanks for the tips mighty!
", "rawMarkdown": "thanks for the tips mighty!", "votes": null } ], "comments": [ { "id": 725366, "author_name": "haqishen", "author_url": "", "post_date": "01/22/2020 03:05:16", "content": "\ntest = pd.read_parquet('/kaggle/input/bengaliai-cv19/test_image_data_{}.parquet'.format(i))\ndata = test.iloc[:, 1:].values.reshape(-1, HEIGHT, WIDTH).astype(np.uint8)\ndel test\ngc.collect()\n
Try this?
", "votes": null, "replies": [ { "id": 725369, "author_name": "linainversez", "author_url": "", "post_date": "01/22/2020 03:09:59", "content": "Hi Qishen,
\n\nI tried this in a previous version but it did not work :(. Thanks for the suggestion though!
", "votes": null, "replies": [] }, { "id": 725395, "author_name": "hengck23", "author_url": "", "post_date": "01/22/2020 03:41:07", "content": "maybe this work
\n\n```
\n\ntest = pd.read_parquet('/kaggle/input/bengaliai-cv19/test_image_data_{}.parquet'.format(i))\nfor b in ...\n input= test.iloc[b:b+batch_size, 1:].values.reshape(-1, HEIGHT, WIDTH).astype(np.uint8)\n output = net(input)
\n\n```
", "votes": null, "replies": [] }, { "id": 725418, "author_name": "linainversez", "author_url": "", "post_date": "01/22/2020 04:35:18", "content": "will try this, thanks!
", "votes": null, "replies": [] } ] }, { "id": 725452, "author_name": "mightyrains", "author_url": "", "post_date": "01/22/2020 05:44:59", "content": "In my case, I kept getting these errors if I used any image size greater than 128x128. Sometimes I got \"Notebook Exceeded Allowed Compute\" or \"Submission CSV Not Found\". Maybe you too can try reducing the image size.
\n\nAlso, please double check that you're writing values for all the private set entries in your submission.csv
\n\nCheers!
", "votes": null, "replies": [ { "id": 725935, "author_name": "linainversez", "author_url": "", "post_date": "01/22/2020 16:23:54", "content": "thanks for the tips mighty!
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "725332": "Hi everybody, here is a link to my notebook:\nhttps://www.kaggle.com/linainversez/bengali-fast-ai-inference\n\nI keep getting \"Submission Scoring Error\" on all of my kernels, and I have tried pretty much everything, so I am opening the floor up to any of your kind suggestions. \n\nThe notebook above uses CPU to compute the predictions. Although it's really slow, I tried it on the training set and it should take ~4 hrs to complete, which is less time than the 9 hour window. However, the submission mysteriously craps out at around ~2 hrs.\n\nI have also tried using GPU, using the lafoss notebook / others as a guide. However, I am having a lot of trouble applying the same transformations to the data that fastai does. I've done a lot of digging into the source code, and it seems that fastai really wants people to load datasets from files, which is not the case in this competition. Also, since I am trying a single-output CNN, I haven't really found a lot of code that I can copy.\n\nI'll probably start over with a new pipeline if this doesn't work out, but I was wondering if anybody had suggestions before I scrapped the thing.\n\nThanks,\nBryan", "725366": "```\ntest = pd.read_parquet('/kaggle/input/bengaliai-cv19/test_image_data_{}.parquet'.format(i))\ndata = test.iloc[:, 1:].values.reshape(-1, HEIGHT, WIDTH).astype(np.uint8)\ndel test\ngc.collect()\n```\n\nTry this?", "725369": "Hi Qishen,\n\nI tried this in a previous version but it did not work :(. Thanks for the suggestion though!", "725395": "maybe this work\n\n```\n\ntest = pd.read_parquet('/kaggle/input/bengaliai-cv19/test_image_data_{}.parquet'.format(i))\nfor b in ...\n input= test.iloc[b:b+batch_size, 1:].values.reshape(-1, HEIGHT, WIDTH).astype(np.uint8)\n output = net(input)\n\n\n```", "725418": "will try this, thanks!", "725452": "In my case, I kept getting these errors if I used any image size greater than 128x128. Sometimes I got \"Notebook Exceeded Allowed Compute\" or \"Submission CSV Not Found\". Maybe you too can try reducing the image size.\n\nAlso, please double check that you're writing values for all the private set entries in your submission.csv\n\nCheers!", "725935": "thanks for the tips mighty!" }, "source": "meta" }