I've taken a look at this previous competition: https://www.kaggle.com/c/diabetic-retinopathy-detection/data
\n\nApart from the 1000 sample images, It seems that most of the data (~20,000 images) is unusable. Although the zip file was broken down into multiple parts, I don't seem to be able to recombine them when I load them inside a kernel.
\n\nHas anyone been able to use it?
", "messages": [ { "id": 565134, "postDate": "2019-06-30T13:38:37.853Z", "content": "I uploaded 896x896 resized dataset with combined training data and the labels from the previous one.
\n\nI tried the 1024x1024 but it was about 23GB so a bit large for the 20GB size limit. Also I made a kernel with a few visualizations of it. Maybe a different image format or some other processing could keep it a bit larger but it lets me play with it.
", "rawMarkdown": "I uploaded 896x896 resized [dataset](https://www.kaggle.com/donkeys/retinopathy-train-2015) with combined training data and the labels from the previous one.\n\nI tried the 1024x1024 but it was about 23GB so a bit large for the 20GB size limit. Also I made a [kernel](https://www.kaggle.com/donkeys/looking-in-the-eyes-of-past-and-present) with a few visualizations of it. Maybe a different image format or some other processing could keep it a bit larger but it lets me play with it. ", "votes": 10, "replies": [ { "id": 565475, "postDate": "2019-07-01T02:02:54.240Z", "content": "Thank you for your good work! I will downsample them to 224 for imagenet-like models.
\n\nYour public quota is 20 GB; are you worried about not being able to upload datasets in the future?
", "rawMarkdown": "Thank you for your good work! I will downsample them to 224 for imagenet-like models.\n\n Your public quota is 20 GB; are you worried about not being able to upload datasets in the future?", "votes": 1 }, { "id": 565745, "postDate": "2019-07-01T10:24:37.503Z", "content": "Glad you like it!
\n\nAs far as I understand the private quota is 20GB but there is no quota for public datasets. So I hope it has no impact :)
", "rawMarkdown": "Glad you like it! \n\nAs far as I understand the private quota is 20GB but there is no quota for public datasets. So I hope it has no impact :)", "votes": 1 }, { "id": 565886, "postDate": "2019-07-01T13:50:48.293Z", "content": "You are totally right! I forgot about that!
", "rawMarkdown": "You are totally right! I forgot about that!" } ] }, { "id": 565498, "postDate": "2019-07-01T03:04:47.110Z", "content": "I made the dataset available here:\nhttps://www.kaggle.com/c/aptos2019-blindness-detection/discussion/98059
\n\nI have two versions. One that is simply resized, and another that is cropped first by finding the mask of the fundus. This cropping will hopefully improve the score, as I have noticed it has helped in my own experiments with the previous competition dataset.
\n\nSorry it was later than I promised. Unfortunately, I also didn't have time to work on training a model for this competition using the previous competition data. But that should be coming out in the next couple of days.
", "rawMarkdown": "I made the dataset available here:\nhttps://www.kaggle.com/c/aptos2019-blindness-detection/discussion/98059\n\nI have two versions. One that is simply resized, and another that is cropped first by finding the mask of the fundus. This cropping will hopefully improve the score, as I have noticed it has helped in my own experiments with the previous competition dataset.\n\nSorry it was later than I promised. Unfortunately, I also didn't have time to work on training a model for this competition using the previous competition data. But that should be coming out in the next couple of days.", "votes": 8 }, { "id": 563939, "postDate": "2019-06-28T19:43:23.663Z", "content": "I've taken a look at this previous competition: https://www.kaggle.com/c/diabetic-retinopathy-detection/data
\n\nApart from the 1000 sample images, It seems that most of the data (~20,000 images) is unusable. Although the zip file was broken down into multiple parts, I don't seem to be able to recombine them when I load them inside a kernel.
\n\nHas anyone been able to use it?
", "rawMarkdown": "I've taken a look at this previous competition: https://www.kaggle.com/c/diabetic-retinopathy-detection/data\n\nApart from the 1000 sample images, It seems that most of the data (~20,000 images) is unusable. Although the zip file was broken down into multiple parts, I don't seem to be able to recombine them when I load them inside a kernel.\n\nHas anyone been able to use it?", "votes": 6 }, { "id": 564049, "postDate": "2019-06-28T23:38:44.467Z", "content": "I plan to make preprocessed versions of the previous competition data by tomorrow.
", "rawMarkdown": "I plan to make preprocessed versions of the previous competition data by tomorrow.", "votes": 5, "replies": [ { "id": 564264, "postDate": "2019-06-29T08:00:13.910Z", "content": "Thank you !
", "rawMarkdown": "Thank you ! " }, { "id": 564265, "postDate": "2019-06-29T08:00:33.710Z", "content": "@tanlikesmath thanks for that. Can you please make it public? \nThis competition data will hit plateau as because of small size. Then we need to use that dataset.
", "rawMarkdown": "@tanlikesmath thanks for that. Can you please make it public? \nThis competition data will hit plateau as because of small size. Then we need to use that dataset." }, { "id": 564392, "postDate": "2019-06-29T11:04:16.943Z", "content": "Yes I will i will first make my submission with it and write a kernel on how to use it and then make it public. I have the dataset already created from previous work and is currently private but once I get it to work with this competition I will make it public! 😃
", "rawMarkdown": "Yes I will i will first make my submission with it and write a kernel on how to use it and then make it public. I have the dataset already created from previous work and is currently private but once I get it to work with this competition I will make it public! 😃", "votes": 1 }, { "id": 564716, "postDate": "2019-06-29T21:16:27.137Z", "content": "@tanlikesmath thank you for your good work. What size will it be (e.g. 224x224)?
", "rawMarkdown": "@tanlikesmath thank you for your good work. What size will it be (e.g. 224x224)?" }, { "id": 564740, "postDate": "2019-06-29T22:27:35.317Z", "content": "@xhlulu it is 1024x1024. You can further resize the pictures later if you want.
", "rawMarkdown": "@xhlulu it is 1024x1024. You can further resize the pictures later if you want. ", "votes": 2 } ] }, { "id": 564883, "postDate": "2019-06-30T05:52:47.657Z", "content": "I can't use it in my kernel, but I can use it in local.\nhttps://www.kaggle.com/c/aptos2019-blindness-detection/discussion/97947#latest-564872
\n\nIn this competition we don't need to train in kernel. So it is better not to load this data in your kernel because it takes a lot of time.
", "rawMarkdown": "I can't use it in my kernel, but I can use it in local.\nhttps://www.kaggle.com/c/aptos2019-blindness-detection/discussion/97947#latest-564872\n\nIn this competition we don't need to train in kernel. So it is better not to load this data in your kernel because it takes a lot of time.", "votes": 2, "replies": [ { "id": 564890, "postDate": "2019-06-30T06:08:04.900Z", "content": "Well worked ! \nThank you !
", "rawMarkdown": "Well worked ! \nThank you !", "votes": 1 }, { "id": 564900, "postDate": "2019-06-30T06:21:27.490Z", "content": "@bluexleoxgreen a fix for this coming soon ;)
", "rawMarkdown": "@bluexleoxgreen a fix for this coming soon ;)", "votes": 1 }, { "id": 565215, "postDate": "2019-06-30T15:40:32.913Z", "content": "> a fix for this coming soon ;)
\n\n@tanlikesmath waiting for that.. \nHope we can do all the things here on kernels. Many people don't have GPUs\nOnly labeled training data (no test images) also would be fine
", "rawMarkdown": "> a fix for this coming soon ;)\n\n@tanlikesmath waiting for that.. \nHope we can do all the things here on kernels. Many people don't have GPUs\nOnly labeled training data (no test images) also would be fine", "votes": 2 }, { "id": 565278, "postDate": "2019-06-30T17:37:48.073Z", "content": "@prashantkikani \nI think the labels of test data are also published in the previous competition.
\nRetionopathy_solution csv file includes both labels.
\nFor more detail, see my above post.
Updated:
\nAs @bluexleoxgreen adviced me, these files are just divided files.
\nNow I became to extract files.
\nThanks @bluexleoxgreen !
I deleted my post in external dataset thread not to confuse.
\n\n@xhlulu \nThank you for positing this issue. I have also faced this problem(I could not unzip).
\n\nJust to be sure, I have reported this problem in external dataset thread.\nhttps://www.kaggle.com/c/aptos2019-blindness-detection/discussion/97605#latest-564847
", "rawMarkdown": "Updated: \nAs @bluexleoxgreen adviced me, these files are just divided files. \nNow I became to extract files. \nThanks @bluexleoxgreen !\n \n**I deleted my post in external dataset thread not to confuse.**\n\n---\n\n@xhlulu \nThank you for positing this issue. I have also faced this problem(I could not unzip). \n\nJust to be sure, I have reported this problem in external dataset thread.\nhttps://www.kaggle.com/c/aptos2019-blindness-detection/discussion/97605#latest-564847", "votes": 2 }, { "id": 566171, "postDate": "2019-07-01T21:34:33.643Z", "content": "You'll need 7zip or Keka (if you're on a Mac) to be able to extract all the volumes in the old dataset. Due to size limitations, the data is broken into parts/volumes named train.zip.001 - train.zip.005.
", "rawMarkdown": "You'll need 7zip or Keka (if you're on a Mac) to be able to extract all the volumes in the old dataset. Due to size limitations, the data is broken into parts/volumes named train.zip.001 - train.zip.005." }, { "id": 564874, "postDate": "2019-06-30T05:32:19.743Z", "content": "You need 7-Zip to unzip those files.
", "rawMarkdown": "You need 7-Zip to unzip those files. ", "replies": [ { "id": 564886, "postDate": "2019-06-30T05:55:38.500Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 564735, "postDate": "2019-06-29T22:09:28.593Z", "content": "What do you mean by unusable? Also, are you doing an comparison of images from the two diff datasets? If so, how are you doing it?
", "rawMarkdown": "What do you mean by unusable? Also, are you doing an comparison of images from the two diff datasets? If so, how are you doing it?", "replies": [ { "id": 565473, "postDate": "2019-07-01T02:00:24.427Z", "content": "I'm not sure what you mean by comparison of images. They are unusable because when you open a kernel with the previous competition's dataset, you are unable to even locate the files using os.listdir.
I uploaded 896x896 resized dataset with combined training data and the labels from the previous one.
\n\nI tried the 1024x1024 but it was about 23GB so a bit large for the 20GB size limit. Also I made a kernel with a few visualizations of it. Maybe a different image format or some other processing could keep it a bit larger but it lets me play with it.
", "votes": 10, "replies": [ { "id": 565475, "author_name": "xhlulu", "author_url": "", "post_date": "2019-07-01T02:02:54.240000", "content": "Thank you for your good work! I will downsample them to 224 for imagenet-like models.
\n\nYour public quota is 20 GB; are you worried about not being able to upload datasets in the future?
", "votes": 1, "replies": [] }, { "id": 565745, "author_name": "averagemn", "author_url": "", "post_date": "2019-07-01T10:24:37.503000", "content": "Glad you like it!
\n\nAs far as I understand the private quota is 20GB but there is no quota for public datasets. So I hope it has no impact :)
", "votes": 1, "replies": [] }, { "id": 565886, "author_name": "xhlulu", "author_url": "", "post_date": "2019-07-01T13:50:48.293000", "content": "You are totally right! I forgot about that!
", "votes": 0, "replies": [] } ] }, { "id": 565498, "author_name": "ilovescience", "author_url": "", "post_date": "2019-07-01T03:04:47.110000", "content": "I made the dataset available here:\nhttps://www.kaggle.com/c/aptos2019-blindness-detection/discussion/98059
\n\nI have two versions. One that is simply resized, and another that is cropped first by finding the mask of the fundus. This cropping will hopefully improve the score, as I have noticed it has helped in my own experiments with the previous competition dataset.
\n\nSorry it was later than I promised. Unfortunately, I also didn't have time to work on training a model for this competition using the previous competition data. But that should be coming out in the next couple of days.
", "votes": 8, "replies": [] }, { "id": 564049, "author_name": "ilovescience", "author_url": "", "post_date": "2019-06-28T23:38:44.467000", "content": "I plan to make preprocessed versions of the previous competition data by tomorrow.
", "votes": 5, "replies": [ { "id": 564264, "author_name": "CVxTz", "author_url": "", "post_date": "2019-06-29T08:00:13.910000", "content": "Thank you !
", "votes": 0, "replies": [] }, { "id": 564265, "author_name": "Prashant Kikani", "author_url": "", "post_date": "2019-06-29T08:00:33.710000", "content": "@tanlikesmath thanks for that. Can you please make it public? \nThis competition data will hit plateau as because of small size. Then we need to use that dataset.
", "votes": 0, "replies": [] }, { "id": 564392, "author_name": "ilovescience", "author_url": "", "post_date": "2019-06-29T11:04:16.943000", "content": "Yes I will i will first make my submission with it and write a kernel on how to use it and then make it public. I have the dataset already created from previous work and is currently private but once I get it to work with this competition I will make it public! 😃
", "votes": 1, "replies": [] }, { "id": 564716, "author_name": "xhlulu", "author_url": "", "post_date": "2019-06-29T21:16:27.137000", "content": "@tanlikesmath thank you for your good work. What size will it be (e.g. 224x224)?
", "votes": 0, "replies": [] }, { "id": 564740, "author_name": "ilovescience", "author_url": "", "post_date": "2019-06-29T22:27:35.317000", "content": "@xhlulu it is 1024x1024. You can further resize the pictures later if you want.
", "votes": 2, "replies": [] } ] }, { "id": 564883, "author_name": "leo", "author_url": "", "post_date": "2019-06-30T05:52:47.657000", "content": "I can't use it in my kernel, but I can use it in local.\nhttps://www.kaggle.com/c/aptos2019-blindness-detection/discussion/97947#latest-564872
\n\nIn this competition we don't need to train in kernel. So it is better not to load this data in your kernel because it takes a lot of time.
", "votes": 2, "replies": [ { "id": 564890, "author_name": "Maxwell", "author_url": "", "post_date": "2019-06-30T06:08:04.900000", "content": "Well worked ! \nThank you !
", "votes": 1, "replies": [] }, { "id": 564900, "author_name": "ilovescience", "author_url": "", "post_date": "2019-06-30T06:21:27.490000", "content": "@bluexleoxgreen a fix for this coming soon ;)
", "votes": 1, "replies": [] }, { "id": 565215, "author_name": "Prashant Kikani", "author_url": "", "post_date": "2019-06-30T15:40:32.913000", "content": "> a fix for this coming soon ;)
\n\n@tanlikesmath waiting for that.. \nHope we can do all the things here on kernels. Many people don't have GPUs\nOnly labeled training data (no test images) also would be fine
", "votes": 2, "replies": [] }, { "id": 565278, "author_name": "Maxwell", "author_url": "", "post_date": "2019-06-30T17:37:48.073000", "content": "@prashantkikani \nI think the labels of test data are also published in the previous competition.
\nRetionopathy_solution csv file includes both labels.
\nFor more detail, see my above post.
Updated:
\nAs @bluexleoxgreen adviced me, these files are just divided files.
\nNow I became to extract files.
\nThanks @bluexleoxgreen !
I deleted my post in external dataset thread not to confuse.
\n\n@xhlulu \nThank you for positing this issue. I have also faced this problem(I could not unzip).
\n\nJust to be sure, I have reported this problem in external dataset thread.\nhttps://www.kaggle.com/c/aptos2019-blindness-detection/discussion/97605#latest-564847
", "votes": 2, "replies": [] }, { "id": 566171, "author_name": "EdwardPie", "author_url": "", "post_date": "2019-07-01T21:34:33.643000", "content": "You'll need 7zip or Keka (if you're on a Mac) to be able to extract all the volumes in the old dataset. Due to size limitations, the data is broken into parts/volumes named train.zip.001 - train.zip.005.
", "votes": 0, "replies": [] }, { "id": 564874, "author_name": "Xuan Cao", "author_url": "", "post_date": "2019-06-30T05:32:19.743000", "content": "You need 7-Zip to unzip those files.
", "votes": 0, "replies": [ { "id": 564886, "author_name": "", "author_url": "", "post_date": "2019-06-30T05:55:38.500000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 564735, "author_name": "Tim Yee", "author_url": "", "post_date": "2019-06-29T22:09:28.593000", "content": "What do you mean by unusable? Also, are you doing an comparison of images from the two diff datasets? If so, how are you doing it?
", "votes": 0, "replies": [ { "id": 565473, "author_name": "xhlulu", "author_url": "", "post_date": "2019-07-01T02:00:24.427000", "content": "I'm not sure what you mean by comparison of images. They are unusable because when you open a kernel with the previous competition's dataset, you are unable to even locate the files using os.listdir.