Hi,
\nin order to make effective use of TPU one approach is to pre-process and pack images and labels in TFRecord files.
\nI'm going to explore this approach in this competition, and I'll publish a set of datasets, with different resolutions.
\nI have started publishing an initial dataset: HERB2021-256. Now the dataset contains only 500k images of the total train set, but I hope soon to publish a new, complete version.
\nI'll publish, as soon as it works, a Notebook explaining how to read TFRecord, with a classification model based on EfficientNet.
If anyone decides to try with this dataset, I'm greatly interested to hear comments and results.
", "messages": [ { "id": 1242384, "postDate": "2021-03-17T15:24:34.473Z", "content": "Hi,
\nin order to make effective use of TPU one approach is to pre-process and pack images and labels in TFRecord files.
\nI'm going to explore this approach in this competition, and I'll publish a set of datasets, with different resolutions.
\nI have started publishing an initial dataset: HERB2021-256. Now the dataset contains only 500k images of the total train set, but I hope soon to publish a new, complete version.
\nI'll publish, as soon as it works, a Notebook explaining how to read TFRecord, with a classification model based on EfficientNet.
If anyone decides to try with this dataset, I'm greatly interested to hear comments and results.
", "rawMarkdown": "Hi,\n\nin order to make effective use of TPU one approach is to pre-process and pack images and labels in **TFRecord** files.\nI'm going to explore this approach in this competition, and I'll publish a set of datasets, with different resolutions.\nI have started publishing an initial dataset: [HERB2021-256](https://www.kaggle.com/luigisaetta/herb2021-256). Now the dataset contains only **500k** images of the total train set, but I hope soon to publish a new, complete version.\nI'll publish, as soon as it works, a Notebook explaining how to read TFRecord, with a classification model based on EfficientNet.\n\nIf anyone decides to try with this dataset, I'm greatly interested to hear comments and results.\n", "votes": 11 }, { "id": 1243732, "postDate": "2021-03-18T13:04:24.703Z", "content": "Thanks, @luigisaetta, looking forward to the complete dataset. Ran out of GPU for this week 😅. Having TFRecords would be helpful.
", "rawMarkdown": "Thanks, @luigisaetta, looking forward to the complete dataset. Ran out of GPU for this week 😅. Having TFRecords would be helpful.", "votes": 1, "replies": [ { "id": 1245338, "postDate": "2021-03-19T17:37:39.417Z", "content": "why don't you try with TPU? It is much faster
", "rawMarkdown": "why don't you try with TPU? It is much faster" }, { "id": 1245795, "postDate": "2021-03-20T07:00:46.950Z", "content": "AFAIK, Keras just plays well with TPU. I haven't worked with TFRecords much. I tried making them on my own just couldn't. Hopefully can do some stuff once there's a public TFRecords dataset available.
\nAlso, was saving TPU for Plant Pathology 😅
", "rawMarkdown": "AFAIK, Keras just plays well with TPU. I haven't worked with TFRecords much. I tried making them on my own just couldn't. Hopefully can do some stuff once there's a public TFRecords dataset available.\n\nAlso, was saving TPU for Plant Pathology 😅", "votes": 1 } ] }, { "id": 1243323, "postDate": "2021-03-18T06:30:51.600Z", "content": "@luigisaetta Thanks for the TF Records dataset Luigi
", "rawMarkdown": "@luigisaetta Thanks for the TF Records dataset Luigi", "votes": 1 }, { "id": 1242799, "postDate": "2021-03-17T20:13:58.047Z", "content": "Training on the dataset (500K images) achieve 50% accuracy. Good start.
", "rawMarkdown": "Training on the dataset (500K images) achieve 50% accuracy. Good start.", "votes": 1 }, { "id": 1242796, "postDate": "2021-03-17T20:11:56.720Z", "content": "I have also added a second dataset with all test images: Test dataset
", "rawMarkdown": "I have also added a second dataset with **all test images**: [Test dataset](https://www.kaggle.com/luigisaetta/herb2021-test-256)", "votes": 1 }, { "id": 1548209, "postDate": "2021-10-18T04:44:04.120Z", "content": "Hey @luigisaetta, thanks so much for your tfrecords. I finally was able to train using them. It does ok in terms of performance 0.5 test F1 score. I also found your 384 tfrecords and I am training on those. I wonder if you have available your code for tfrecords creation? Will let you know how the 384 goes. Thanks so much!
", "rawMarkdown": "Hey @luigisaetta, thanks so much for your tfrecords. I finally was able to train using them. It does ok in terms of performance 0.5 test F1 score. I also found your 384 tfrecords and I am training on those. I wonder if you have available your code for tfrecords creation? Will let you know how the 384 goes. Thanks so much! " }, { "id": 1278220, "postDate": "2021-04-19T16:56:32.643Z", "content": "I have updated the dataset. Now it contains all the train images.
", "rawMarkdown": "I have updated the dataset. Now it contains all the train images." }, { "id": 1255699, "postDate": "2021-03-29T04:51:12.113Z", "content": "Hey @luigisaetta, I was wondering how did you manage to create a feature_map for parsing the TFRecords ?
Thanks for the initial dataset.
", "rawMarkdown": "Thanks for the initial dataset." } ], "comments": [ { "id": 1243732, "author_name": "Saurav Maheshkar ☕️", "author_url": "", "post_date": "2021-03-18T13:04:24.703000", "content": "Thanks, @luigisaetta, looking forward to the complete dataset. Ran out of GPU for this week 😅. Having TFRecords would be helpful.
", "votes": 1, "replies": [ { "id": 1245338, "author_name": "Luigi Saetta", "author_url": "", "post_date": "2021-03-19T17:37:39.417000", "content": "why don't you try with TPU? It is much faster
", "votes": 0, "replies": [] }, { "id": 1245795, "author_name": "Saurav Maheshkar ☕️", "author_url": "", "post_date": "2021-03-20T07:00:46.950000", "content": "AFAIK, Keras just plays well with TPU. I haven't worked with TFRecords much. I tried making them on my own just couldn't. Hopefully can do some stuff once there's a public TFRecords dataset available.
\nAlso, was saving TPU for Plant Pathology 😅
", "votes": 1, "replies": [] } ] }, { "id": 1243323, "author_name": "Tensor Girl", "author_url": "", "post_date": "2021-03-18T06:30:51.600000", "content": "@luigisaetta Thanks for the TF Records dataset Luigi
", "votes": 1, "replies": [] }, { "id": 1242799, "author_name": "Luigi Saetta", "author_url": "", "post_date": "2021-03-17T20:13:58.047000", "content": "Training on the dataset (500K images) achieve 50% accuracy. Good start.
", "votes": 1, "replies": [] }, { "id": 1242796, "author_name": "Luigi Saetta", "author_url": "", "post_date": "2021-03-17T20:11:56.720000", "content": "I have also added a second dataset with all test images: Test dataset
", "votes": 1, "replies": [] }, { "id": 1548209, "author_name": "Ivan Felipe Rodriguez", "author_url": "", "post_date": "2021-10-18T04:44:04.120000", "content": "Hey @luigisaetta, thanks so much for your tfrecords. I finally was able to train using them. It does ok in terms of performance 0.5 test F1 score. I also found your 384 tfrecords and I am training on those. I wonder if you have available your code for tfrecords creation? Will let you know how the 384 goes. Thanks so much!
", "votes": 0, "replies": [] }, { "id": 1278220, "author_name": "Luigi Saetta", "author_url": "", "post_date": "2021-04-19T16:56:32.643000", "content": "I have updated the dataset. Now it contains all the train images.
", "votes": 0, "replies": [] }, { "id": 1255699, "author_name": "Saurav Maheshkar ☕️", "author_url": "", "post_date": "2021-03-29T04:51:12.113000", "content": "Hey @luigisaetta, I was wondering how did you manage to create a feature_map for parsing the TFRecords ?
Thanks for the initial dataset.
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1242384": "Hi,\n\nin order to make effective use of TPU one approach is to pre-process and pack images and labels in **TFRecord** files.\nI'm going to explore this approach in this competition, and I'll publish a set of datasets, with different resolutions.\nI have started publishing an initial dataset: [HERB2021-256](https://www.kaggle.com/luigisaetta/herb2021-256). Now the dataset contains only **500k** images of the total train set, but I hope soon to publish a new, complete version.\nI'll publish, as soon as it works, a Notebook explaining how to read TFRecord, with a classification model based on EfficientNet.\n\nIf anyone decides to try with this dataset, I'm greatly interested to hear comments and results.\n", "1243732": "Thanks, @luigisaetta, looking forward to the complete dataset. Ran out of GPU for this week 😅. Having TFRecords would be helpful.", "1243323": "@luigisaetta Thanks for the TF Records dataset Luigi", "1242799": "Training on the dataset (500K images) achieve 50% accuracy. Good start.", "1242796": "I have also added a second dataset with **all test images**: [Test dataset](https://www.kaggle.com/luigisaetta/herb2021-test-256)", "1548209": "Hey @luigisaetta, thanks so much for your tfrecords. I finally was able to train using them. It does ok in terms of performance 0.5 test F1 score. I also found your 384 tfrecords and I am training on those. I wonder if you have available your code for tfrecords creation? Will let you know how the 384 goes. Thanks so much! ", "1278220": "I have updated the dataset. Now it contains all the train images.", "1255699": "Hey @luigisaetta, I was wondering how did you manage to create a `feature_map` for parsing the TFRecords ?", "1261355": "Thanks for the initial dataset." } }