{
  "id": 198744,
  "title": "Stratified TFRecords datasets",
  "url": "/competitions/cassava-leaf-disease-classification/discussion/198744",
  "author_name": "",
  "post_date": "2020-11-22T20:32:24.032311700Z",
  "votes": 56,
  "comment_count": 21,
  "views": 0,
  "content": "<p>As reported by other members I also got a much lower score using the TFRecords provided by the competition, so I recreated the TFRecords from the JPG images and made them public.</p>\n<p>I have created two versions of the TFRecords, one that resized the original 600x800 images and another one that first center cropped the images to 600x600 and then resized them.</p>\n<table>\n<thead>\n<tr>\n<th>Resolution</th>\n<th>Resized</th>\n<th>Center croped</th>\n<th>External Center croped</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>128x128</td>\n<td><a href=\"https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tfrecords-128x128\" target=\"_blank\">15 TFRecords</a></td>\n<td><a href=\"https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tfrecords-center-128x128\" target=\"_blank\">15 TFRecords</a></td>\n<td><a href=\"https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tfrecords-external-128x128\" target=\"_blank\">15 TFRecords</a></td>\n</tr>\n<tr>\n<td>256x256</td>\n<td><a href=\"https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tfrecords-256x256\" target=\"_blank\">15 TFRecords</a></td>\n<td><a href=\"https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tfrecords-center-256x256\" target=\"_blank\">15 TFRecords</a></td>\n<td><a href=\"https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tfrecords-external-256x256\" target=\"_blank\">15 TFRecords</a></td>\n</tr>\n<tr>\n<td>384x384</td>\n<td><a href=\"https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tfrecords-384x384\" target=\"_blank\">15 TFRecords</a></td>\n<td><a href=\"https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tfrecords-center-384x384\" target=\"_blank\">15 TFRecords</a></td>\n<td><a href=\"https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tfrecords-external-384x384\" target=\"_blank\">15 TFRecords</a></td>\n</tr>\n<tr>\n<td>512x512</td>\n<td><a href=\"https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tfrecords-512x512\" target=\"_blank\">15 TFRecords</a></td>\n<td><a href=\"https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tfrecords-center-512x512\" target=\"_blank\">15 TFRecords</a> - <a href=\"https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-50-tfrecords-center-512x512\" target=\"_blank\">50 TFRecords</a></td>\n<td><a href=\"https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tfrecords-external-512x512\" target=\"_blank\">15 TFRecords</a> - <a href=\"https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-50-tfrecords-external-512x512\" target=\"_blank\">50 TFRecords</a></td>\n</tr>\n</tbody>\n</table>\n<hr>\n<h4>Divided by classes</h4>\n<table>\n<thead>\n<tr>\n<th>Resolution</th>\n<th>Center croped</th>\n<th>External Center croped</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>512x512</td>\n<td><a href=\"https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tfrecords-classes-512x512\" target=\"_blank\">15 TFRecords</a> - <a href=\"https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-50-tfrecords-classes-512x512\" target=\"_blank\">50 TFRecords</a></td>\n<td><a href=\"https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tfrecords-classes-ext-512x512\" target=\"_blank\">15 TFRecords</a> - <a href=\"https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-ext-50-tfrec-classes-512x512\" target=\"_blank\">50 TFRecords</a></td>\n</tr>\n</tbody>\n</table>\n<p>I have added this last dataset that is divided by classes, it might be useful if you want to do some kind of class sampling or train GANs.</p>\n<p>Here you can find the <a href=\"https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-stratified-tfrecords-256x256\" target=\"_blank\">notebook to create the TFRecords</a>, you can modify the resolution parameters to change the output image resolution.</p>\n<h4>The files were divided into 15 TFRecods files to make it easier to do K-Fold splits, they were stratified by the label.</h4>\n<pre><code>File: 1 has 1427 samples\nFile: 2 has 1427 samples\nFile: 3 has 1427 samples\nFile: 4 has 1427 samples\nFile: 5 has 1427 samples\nFile: 6 has 1427 samples\nFile: 7 has 1427 samples\nFile: 8 has 1426 samples\nFile: 9 has 1426 samples\nFile: 10 has 1426 samples\nFile: 11 has 1426 samples\nFile: 12 has 1426 samples\nFile: 13 has 1426 samples\nFile: 14 has 1426 samples\nFile: 15 has 1426 samples\n</code></pre>\n<h4>Example of the label's distribution comparing the complete set to one of the TFRecord files</h4>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1182060%2Fe384961ab4fecada33571cf08bcfa972%2FScreenshot%20from%202020-11-22%2017-28-39.png?generation=1606076946320113&amp;alt=media\" alt=\"\"></p>\n<h4>Image samples from the dataset</h4>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1182060%2F7ca3f7c21c83e6157a1074ca96a5fdc8%2FScreenshot%20from%202020-11-23%2008-12-29.png?generation=1606129985784142&amp;alt=media\" alt=\"\"></p>",
  "messages": [
    {
      "id": "1087545",
      "postDate": "11/22/2020 20:32:24",
      "content": "<p>As reported by other members I also got a much lower score using the TFRecords provided by the competition, so I recreated the TFRecords from the JPG images and made them public.</p>\n<p>I have created two versions of the TFRecords, one that resized the original 600x800 images and another one that first center cropped the images to 600x600 and then resized them.</p>\n<table>\n<thead>\n<tr>\n<th>Resolution</th>\n<th>Resized</th>\n<th>Center croped</th>\n<th>External Center croped</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>128x128</td>\n<td><a href=\"https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tfrecords-128x128\" target=\"_blank\">15 TFRecords</a></td>\n<td><a href=\"https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tfrecords-center-128x128\" target=\"_blank\">15 TFRecords</a></td>\n<td><a href=\"https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tfrecords-external-128x128\" target=\"_blank\">15 TFRecords</a></td>\n</tr>\n<tr>\n<td>256x256</td>\n<td><a href=\"https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tfrecords-256x256\" target=\"_blank\">15 TFRecords</a></td>\n<td><a href=\"https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tfrecords-center-256x256\" target=\"_blank\">15 TFRecords</a></td>\n<td><a href=\"https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tfrecords-external-256x256\" target=\"_blank\">15 TFRecords</a></td>\n</tr>\n<tr>\n<td>384x384</td>\n<td><a href=\"https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tfrecords-384x384\" target=\"_blank\">15 TFRecords</a></td>\n<td><a href=\"https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tfrecords-center-384x384\" target=\"_blank\">15 TFRecords</a></td>\n<td><a href=\"https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tfrecords-external-384x384\" target=\"_blank\">15 TFRecords</a></td>\n</tr>\n<tr>\n<td>512x512</td>\n<td><a href=\"https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tfrecords-512x512\" target=\"_blank\">15 TFRecords</a></td>\n<td><a href=\"https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tfrecords-center-512x512\" target=\"_blank\">15 TFRecords</a> - <a href=\"https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-50-tfrecords-center-512x512\" target=\"_blank\">50 TFRecords</a></td>\n<td><a href=\"https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tfrecords-external-512x512\" target=\"_blank\">15 TFRecords</a> - <a href=\"https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-50-tfrecords-external-512x512\" target=\"_blank\">50 TFRecords</a></td>\n</tr>\n</tbody>\n</table>\n<hr>\n<h4>Divided by classes</h4>\n<table>\n<thead>\n<tr>\n<th>Resolution</th>\n<th>Center croped</th>\n<th>External Center croped</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>512x512</td>\n<td><a href=\"https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tfrecords-classes-512x512\" target=\"_blank\">15 TFRecords</a> - <a href=\"https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-50-tfrecords-classes-512x512\" target=\"_blank\">50 TFRecords</a></td>\n<td><a href=\"https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tfrecords-classes-ext-512x512\" target=\"_blank\">15 TFRecords</a> - <a href=\"https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-ext-50-tfrec-classes-512x512\" target=\"_blank\">50 TFRecords</a></td>\n</tr>\n</tbody>\n</table>\n<p>I have added this last dataset that is divided by classes, it might be useful if you want to do some kind of class sampling or train GANs.</p>\n<p>Here you can find the <a href=\"https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-stratified-tfrecords-256x256\" target=\"_blank\">notebook to create the TFRecords</a>, you can modify the resolution parameters to change the output image resolution.</p>\n<h4>The files were divided into 15 TFRecods files to make it easier to do K-Fold splits, they were stratified by the label.</h4>\n<pre><code>File: 1 has 1427 samples\nFile: 2 has 1427 samples\nFile: 3 has 1427 samples\nFile: 4 has 1427 samples\nFile: 5 has 1427 samples\nFile: 6 has 1427 samples\nFile: 7 has 1427 samples\nFile: 8 has 1426 samples\nFile: 9 has 1426 samples\nFile: 10 has 1426 samples\nFile: 11 has 1426 samples\nFile: 12 has 1426 samples\nFile: 13 has 1426 samples\nFile: 14 has 1426 samples\nFile: 15 has 1426 samples\n</code></pre>\n<h4>Example of the label's distribution comparing the complete set to one of the TFRecord files</h4>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1182060%2Fe384961ab4fecada33571cf08bcfa972%2FScreenshot%20from%202020-11-22%2017-28-39.png?generation=1606076946320113&amp;alt=media\" alt=\"\"></p>\n<h4>Image samples from the dataset</h4>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1182060%2F7ca3f7c21c83e6157a1074ca96a5fdc8%2FScreenshot%20from%202020-11-23%2008-12-29.png?generation=1606129985784142&amp;alt=media\" alt=\"\"></p>",
      "rawMarkdown": "As reported by other members I also got a much lower score using the TFRecords provided by the competition, so I recreated the TFRecords from the JPG images and made them public.\n\nI have created two versions of the TFRecords, one that resized the original 600x800 images and another one that first center cropped the images to 600x600 and then resized them.\n\n\n\n\n| Resolution | Resized | Center croped | External Center croped |\n|------------|---------|----------------|----------|\n|   128x128  | [15 TFRecords](https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tfrecords-128x128) | [15 TFRecords](https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tfrecords-center-128x128) | [15 TFRecords](https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tfrecords-external-128x128) |\n|   256x256 | [15 TFRecords](https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tfrecords-256x256) | [15 TFRecords](https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tfrecords-center-256x256) | [15 TFRecords](https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tfrecords-external-256x256) |\n|   384x384| [15 TFRecords](https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tfrecords-384x384) | [15 TFRecords](https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tfrecords-center-384x384) | [15 TFRecords](https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tfrecords-external-384x384) |\n|   512x512  | [15 TFRecords](https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tfrecords-512x512) | [15 TFRecords](https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tfrecords-center-512x512) - [50 TFRecords](https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-50-tfrecords-center-512x512) | [15 TFRecords](https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tfrecords-external-512x512) - [50 TFRecords](https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-50-tfrecords-external-512x512) |\n\n___\n\n#### Divided by classes\n\n| Resolution | Center croped | External Center croped |\n|------------|----------------|-------------------------|\n| 512x512 | [15 TFRecords](https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tfrecords-classes-512x512) - [50 TFRecords](https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-50-tfrecords-classes-512x512) | [15 TFRecords](https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tfrecords-classes-ext-512x512) - [50 TFRecords](https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-ext-50-tfrec-classes-512x512) |\n\n\nI have added this last dataset that is divided by classes, it might be useful if you want to do some kind of class sampling or train GANs.\n\nHere you can find the [notebook to create the TFRecords](https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-stratified-tfrecords-256x256), you can modify the resolution parameters to change the output image resolution.\n\n#### The files were divided into 15 TFRecods files to make it easier to do K-Fold splits, they were stratified by the label.\n```\nFile: 1 has 1427 samples\nFile: 2 has 1427 samples\nFile: 3 has 1427 samples\nFile: 4 has 1427 samples\nFile: 5 has 1427 samples\nFile: 6 has 1427 samples\nFile: 7 has 1427 samples\nFile: 8 has 1426 samples\nFile: 9 has 1426 samples\nFile: 10 has 1426 samples\nFile: 11 has 1426 samples\nFile: 12 has 1426 samples\nFile: 13 has 1426 samples\nFile: 14 has 1426 samples\nFile: 15 has 1426 samples\n```\n#### Example of the label's distribution comparing the complete set to one of the TFRecord files\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1182060%2Fe384961ab4fecada33571cf08bcfa972%2FScreenshot%20from%202020-11-22%2017-28-39.png?generation=1606076946320113&alt=media)\n\n#### Image samples from the dataset\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1182060%2F7ca3f7c21c83e6157a1074ca96a5fdc8%2FScreenshot%20from%202020-11-23%2008-12-29.png?generation=1606129985784142&alt=media)",
      "votes": null
    },
    {
      "id": "1092568",
      "postDate": "11/27/2020 02:05:06",
      "content": "<p>Updates to people that are using these datasets:</p>\n<ul>\n<li>I have changed the encoding quality to 100%, now the resulting images should have better quality.</li>\n<li>Now you can use two different datasets for each resolution, one resized the original images, and the other first center crops them to a square resolution (600x600) and then resizes the images.</li>\n<li>The datasets also have the <code>training.csv</code> file with a <code>file</code> column, this column has the TFRecord file that each row belongs to.<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1182060%2F6a570377b5e6630607dfefcd6738c2e0%2FScreenshot%20from%202020-11-26%2023-01-55.png?generation=1606442531542450&amp;alt=media\" alt=\"\"></li>\n</ul>",
      "rawMarkdown": "Updates to people that are using these datasets:\n- I have changed the encoding quality to 100%, now the resulting images should have better quality.\n- Now you can use two different datasets for each resolution, one resized the original images, and the other first center crops them to a square resolution (600x600) and then resizes the images.\n- The datasets also have the `training.csv` file with a `file` column, this column has the TFRecord file that each row belongs to.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1182060%2F6a570377b5e6630607dfefcd6738c2e0%2FScreenshot%20from%202020-11-26%2023-01-55.png?generation=1606442531542450&alt=media)",
      "votes": null
    },
    {
      "id": "1094242",
      "postDate": "11/28/2020 12:39:06",
      "content": "<p>Just added a 512x512 dataset that is TFRecords divided by classes, it might be useful if you want to do some kind of class sampling or train GANs.</p>",
      "rawMarkdown": "Just added a 512x512 dataset that is TFRecords divided by classes, it might be useful if you want to do some kind of class sampling or train GANs.",
      "votes": null
    },
    {
      "id": "1095807",
      "postDate": "11/30/2020 01:15:29",
      "content": "<p>Another update, I have added datasets with the external data (from previous competition), this data was also stratified split into 15 TFRecords files, this should make it easier to concatenate with the original data.</p>\n<p>I have some concern about the quality of the external data, the images do not have all the same resolution, and the overall quality seems a little lower, here is a sample:</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1182060%2F7c9d379d6ed1596a89b26de25cc712f4%2FScreenshot%20from%202020-11-29%2022-15-00.png?generation=1606698926594994&amp;alt=media\" alt=\"\"></p>",
      "rawMarkdown": "Another update, I have added datasets with the external data (from previous competition), this data was also stratified split into 15 TFRecords files, this should make it easier to concatenate with the original data.\n\nI have some concern about the quality of the external data, the images do not have all the same resolution, and the overall quality seems a little lower, here is a sample:\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1182060%2F7c9d379d6ed1596a89b26de25cc712f4%2FScreenshot%20from%202020-11-29%2022-15-00.png?generation=1606698926594994&alt=media)",
      "votes": null
    },
    {
      "id": "1103257",
      "postDate": "12/05/2020 19:06:20",
      "content": "<p>Small update, I have removed the <code>2</code> duplicated files from all datasets</p>",
      "rawMarkdown": "Small update, I have removed the `2` duplicated files from all datasets",
      "votes": null
    },
    {
      "id": "1103820",
      "postDate": "12/06/2020 10:22:53",
      "content": "<p>Hey thanks for the data, are you considering creating tf records containing 2019 + 2020 data ? It would be extremely helpful ! </p>",
      "rawMarkdown": "Hey thanks for the data, are you considering creating tf records containing 2019 + 2020 data ? It would be extremely helpful !",
      "votes": null
    },
    {
      "id": "1104085",
      "postDate": "12/06/2020 15:42:40",
      "content": "<p>Would be really great if you create tf records with this data : <a href=\"https://www.kaggle.com/ammarali32/cassava-datasetv2\" target=\"_blank\">https://www.kaggle.com/ammarali32/cassava-datasetv2</a></p>",
      "rawMarkdown": "Would be really great if you create tf records with this data : https://www.kaggle.com/ammarali32/cassava-datasetv2",
      "votes": null
    },
    {
      "id": "1104108",
      "postDate": "12/06/2020 16:15:58",
      "content": "<p>Hey <a href=\"https://www.kaggle.com/aziz69\" target=\"_blank\">@aziz69</a> I have created the TFRecords for the 2019 data, it is the external data. I left them in different  datasets because this way you have more control on how you wish to use it</p>",
      "rawMarkdown": "Hey @aziz69 I have created the TFRecords for the 2019 data, it is the external data. I left them in different  datasets because this way you have more control on how you wish to use it",
      "votes": null
    },
    {
      "id": "1104111",
      "postDate": "12/06/2020 16:20:13",
      "content": "<p>Is this the dataser that has cropped images from the leaves?</p>",
      "rawMarkdown": "Is this the dataser that has cropped images from the leaves?",
      "votes": null
    },
    {
      "id": "1104155",
      "postDate": "12/06/2020 17:03:30",
      "content": "<p>yes, this is the discussion for that data <a href=\"https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/200722\" target=\"_blank\">https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/200722</a></p>",
      "rawMarkdown": "yes, this is the discussion for that data https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/200722",
      "votes": null
    },
    {
      "id": "1104163",
      "postDate": "12/06/2020 17:06:21",
      "content": "<p>Thanks amazing work.</p>",
      "rawMarkdown": "Thanks amazing work.",
      "votes": null
    },
    {
      "id": "1104418",
      "postDate": "12/06/2020 23:09:00",
      "content": "<p>Thanks <a href=\"https://www.kaggle.com/aziz69\" target=\"_blank\">@aziz69</a> , I will try to do some experiments with it to see if they look promissing.</p>",
      "rawMarkdown": "Thanks @aziz69 , I will try to do some experiments with it to see if they look promissing.",
      "votes": null
    },
    {
      "id": "1192078",
      "postDate": "02/08/2021 22:43:22",
      "content": "<p>Hi <a href=\"https://www.kaggle.com/dimitreoliveira\" target=\"_blank\">@dimitreoliveira</a> <br>\nI can't leave one thing unattended<br>\nUsing your dataset I get higher OOF accuracy<br>\nPlease tell me or am I wrong or do you know what is the reason?<br>\nI I'm using tf.image.resize(default bilinear) + dataset.cache <br>\nYou are using cv2.resize(default bilinear)</p>",
      "rawMarkdown": "Hi @dimitreoliveira \nI can't leave one thing unattended\nUsing your dataset I get higher OOF accuracy\nPlease tell me or am I wrong or do you know what is the reason?\nI I'm using tf.image.resize(default bilinear) + dataset.cache \nYou are using cv2.resize(default bilinear)",
      "votes": null
    },
    {
      "id": "1193120",
      "postDate": "02/09/2021 13:21:54",
      "content": "<p>In case of training on TPU, I got an error: : AttributeError: 'tuple' object has no attribute '_from_compatible_tensor_list'<br>\nI think the TPU doesn't support the local TF records. If I'm wrong, please give a correct comment.</p>",
      "rawMarkdown": "In case of training on TPU, I got an error: : AttributeError: 'tuple' object has no attribute '_from_compatible_tensor_list'\nI think the TPU doesn't support the local TF records. If I'm wrong, please give a correct comment.",
      "votes": null
    },
    {
      "id": "1193179",
      "postDate": "02/09/2021 13:52:12",
      "content": "<p>You are right -- when working with TPU you must get your tfrecords from  a Google Cloud Storage bucket, not from your local file system.</p>",
      "rawMarkdown": "You are right -- when working with TPU you must get your tfrecords from  a Google Cloud Storage bucket, not from your local file system.",
      "votes": null
    },
    {
      "id": "1193510",
      "postDate": "02/09/2021 17:09:20",
      "content": "<p>Hi <a href=\"https://www.kaggle.com/theodotus\" target=\"_blank\">@theodotus</a> , I am glad my dataset is giving good results to you.</p>\n<p>About the reason, I am not sure may it is because some of my datasets use center crop before resizing?</p>",
      "rawMarkdown": "Hi @theodotus , I am glad my dataset is giving good results to you.\n\nAbout the reason, I am not sure may it is because some of my datasets use center crop before resizing?",
      "votes": null
    },
    {
      "id": "1193515",
      "postDate": "02/09/2021 17:11:41",
      "content": "<p>Yes <a href=\"https://www.kaggle.com/sahini\" target=\"_blank\">@sahini</a> as Alexey mentioned you must fetch the TFRecords from the GCS bucket, <a href=\"https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tpu-tensorflow-training/notebook\" target=\"_blank\">here is an example</a>.</p>",
      "rawMarkdown": "Yes @sahini as Alexey mentioned you must fetch the TFRecords from the GCS bucket, [here is an example](https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tpu-tensorflow-training/notebook).",
      "votes": null
    },
    {
      "id": "1193674",
      "postDate": "02/09/2021 19:31:25",
      "content": "<p>Thanks <a href=\"https://www.kaggle.com/DimitreOliveria\" target=\"_blank\">@DimitreOliveria</a>, you have provided the GCS path. <br>\nGCS_PATH = KaggleDatasets().get_gcs_path(f'cassava-leaf-disease-tfrecords-center-{HEIGHT}x{WIDTH}'</p>\n<p>I will try it. Thanks for your valuable contrubitions.</p>",
      "rawMarkdown": "Thanks @DimitreOliveria, you have provided the GCS path. \nGCS_PATH = KaggleDatasets().get_gcs_path(f'cassava-leaf-disease-tfrecords-center-{HEIGHT}x{WIDTH}'\n\nI will try it. Thanks for your valuable contrubitions.",
      "votes": null
    },
    {
      "id": "1195357",
      "postDate": "02/10/2021 18:15:11",
      "content": "<p>I achieved 0.893 LB score with centered-512x512 tf records after 5-fold cv. I think that the tf records with original size (600x800) is better than using 512x512 size. Because, applying some augmentations on orginal size (600x800)  and resizing to 512x512 increases the LB score. However I dont know any link that is sharing the original size of tf records. Anyway, thanks again for your efforts.</p>",
      "rawMarkdown": "I achieved 0.893 LB score with centered-512x512 tf records after 5-fold cv. I think that the tf records with original size (600x800) is better than using 512x512 size. Because, applying some augmentations on orginal size (600x800)  and resizing to 512x512 increases the LB score. However I dont know any link that is sharing the original size of tf records. Anyway, thanks again for your efforts.",
      "votes": null
    },
    {
      "id": "1206545",
      "postDate": "02/17/2021 12:28:25",
      "content": "<p><a href=\"https://www.kaggle.com/dimitreoliveira\" target=\"_blank\">@dimitreoliveira</a> thanks for Your answer<br>\ncenter crop gives better results</p>\n<p>But even if I use just resize, Your dataset gives better result than my<br>\n As an answer to this question, I can <a href=\"https://github.com/mlcommons/inference/issues/201\" target=\"_blank\">suggest this issue </a><br>\nThere is mentioned that the resize method greatly affects the accuracy of the neural network.<br>\nAnd many people have noticed that cv2.resize is better than tf.image.resize when training.</p>",
      "rawMarkdown": "dimitreoliveira thanks for Your answer\ncenter crop gives better results\n\nBut even if I use just resize, Your dataset gives better result than my\n As an answer to this question, I can [suggest this issue ](https://github.com/mlcommons/inference/issues/201)\nThere is mentioned that the resize method greatly affects the accuracy of the neural network.\nAnd many people have noticed that cv2.resize is better than tf.image.resize when training.",
      "votes": null
    },
    {
      "id": "1209039",
      "postDate": "02/18/2021 16:41:52",
      "content": "<p>Hi <a href=\"https://www.kaggle.com/theodotus\" target=\"_blank\">@theodotus</a> , this is a nice catch, but are you also stratifying your files?</p>",
      "rawMarkdown": "Hi @theodotus , this is a nice catch, but are you also stratifying your files?",
      "votes": null
    },
    {
      "id": "1209041",
      "postDate": "02/18/2021 16:42:51",
      "content": "<p>You're welcome <a href=\"https://www.kaggle.com/sahini\" target=\"_blank\">@sahini</a> ,</p>\n<p>You can use the original TFRecords with that resolution, but note that they are not stratifyed.</p>",
      "rawMarkdown": "You're welcome @sahini ,\n\nYou can use the original TFRecords with that resolution, but note that they are not stratifyed.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1092568,
      "author_name": "dimitreoliveira",
      "author_url": "",
      "post_date": "11/27/2020 02:05:06",
      "content": "<p>Updates to people that are using these datasets:</p>\n<ul>\n<li>I have changed the encoding quality to 100%, now the resulting images should have better quality.</li>\n<li>Now you can use two different datasets for each resolution, one resized the original images, and the other first center crops them to a square resolution (600x600) and then resizes the images.</li>\n<li>The datasets also have the <code>training.csv</code> file with a <code>file</code> column, this column has the TFRecord file that each row belongs to.<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1182060%2F6a570377b5e6630607dfefcd6738c2e0%2FScreenshot%20from%202020-11-26%2023-01-55.png?generation=1606442531542450&amp;alt=media\" alt=\"\"></li>\n</ul>",
      "votes": null,
      "replies": [
        {
          "id": 1094242,
          "author_name": "dimitreoliveira",
          "author_url": "",
          "post_date": "11/28/2020 12:39:06",
          "content": "<p>Just added a 512x512 dataset that is TFRecords divided by classes, it might be useful if you want to do some kind of class sampling or train GANs.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1103257,
          "author_name": "dimitreoliveira",
          "author_url": "",
          "post_date": "12/05/2020 19:06:20",
          "content": "<p>Small update, I have removed the <code>2</code> duplicated files from all datasets</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1095807,
      "author_name": "dimitreoliveira",
      "author_url": "",
      "post_date": "11/30/2020 01:15:29",
      "content": "<p>Another update, I have added datasets with the external data (from previous competition), this data was also stratified split into 15 TFRecords files, this should make it easier to concatenate with the original data.</p>\n<p>I have some concern about the quality of the external data, the images do not have all the same resolution, and the overall quality seems a little lower, here is a sample:</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1182060%2F7c9d379d6ed1596a89b26de25cc712f4%2FScreenshot%20from%202020-11-29%2022-15-00.png?generation=1606698926594994&amp;alt=media\" alt=\"\"></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1103820,
      "author_name": "aziz69",
      "author_url": "",
      "post_date": "12/06/2020 10:22:53",
      "content": "<p>Hey thanks for the data, are you considering creating tf records containing 2019 + 2020 data ? It would be extremely helpful ! </p>",
      "votes": null,
      "replies": [
        {
          "id": 1104108,
          "author_name": "dimitreoliveira",
          "author_url": "",
          "post_date": "12/06/2020 16:15:58",
          "content": "<p>Hey <a href=\"https://www.kaggle.com/aziz69\" target=\"_blank\">@aziz69</a> I have created the TFRecords for the 2019 data, it is the external data. I left them in different  datasets because this way you have more control on how you wish to use it</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1104163,
          "author_name": "aziz69",
          "author_url": "",
          "post_date": "12/06/2020 17:06:21",
          "content": "<p>Thanks amazing work.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1104085,
      "author_name": "aziz69",
      "author_url": "",
      "post_date": "12/06/2020 15:42:40",
      "content": "<p>Would be really great if you create tf records with this data : <a href=\"https://www.kaggle.com/ammarali32/cassava-datasetv2\" target=\"_blank\">https://www.kaggle.com/ammarali32/cassava-datasetv2</a></p>",
      "votes": null,
      "replies": [
        {
          "id": 1104111,
          "author_name": "dimitreoliveira",
          "author_url": "",
          "post_date": "12/06/2020 16:20:13",
          "content": "<p>Is this the dataser that has cropped images from the leaves?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1104155,
          "author_name": "aziz69",
          "author_url": "",
          "post_date": "12/06/2020 17:03:30",
          "content": "<p>yes, this is the discussion for that data <a href=\"https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/200722\" target=\"_blank\">https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/200722</a></p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1104418,
          "author_name": "dimitreoliveira",
          "author_url": "",
          "post_date": "12/06/2020 23:09:00",
          "content": "<p>Thanks <a href=\"https://www.kaggle.com/aziz69\" target=\"_blank\">@aziz69</a> , I will try to do some experiments with it to see if they look promissing.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1192078,
      "author_name": "theodotus",
      "author_url": "",
      "post_date": "02/08/2021 22:43:22",
      "content": "<p>Hi <a href=\"https://www.kaggle.com/dimitreoliveira\" target=\"_blank\">@dimitreoliveira</a> <br>\nI can't leave one thing unattended<br>\nUsing your dataset I get higher OOF accuracy<br>\nPlease tell me or am I wrong or do you know what is the reason?<br>\nI I'm using tf.image.resize(default bilinear) + dataset.cache <br>\nYou are using cv2.resize(default bilinear)</p>",
      "votes": null,
      "replies": [
        {
          "id": 1193510,
          "author_name": "dimitreoliveira",
          "author_url": "",
          "post_date": "02/09/2021 17:09:20",
          "content": "<p>Hi <a href=\"https://www.kaggle.com/theodotus\" target=\"_blank\">@theodotus</a> , I am glad my dataset is giving good results to you.</p>\n<p>About the reason, I am not sure may it is because some of my datasets use center crop before resizing?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1206545,
          "author_name": "theodotus",
          "author_url": "",
          "post_date": "02/17/2021 12:28:25",
          "content": "<p><a href=\"https://www.kaggle.com/dimitreoliveira\" target=\"_blank\">@dimitreoliveira</a> thanks for Your answer<br>\ncenter crop gives better results</p>\n<p>But even if I use just resize, Your dataset gives better result than my<br>\n As an answer to this question, I can <a href=\"https://github.com/mlcommons/inference/issues/201\" target=\"_blank\">suggest this issue </a><br>\nThere is mentioned that the resize method greatly affects the accuracy of the neural network.<br>\nAnd many people have noticed that cv2.resize is better than tf.image.resize when training.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1209039,
          "author_name": "dimitreoliveira",
          "author_url": "",
          "post_date": "02/18/2021 16:41:52",
          "content": "<p>Hi <a href=\"https://www.kaggle.com/theodotus\" target=\"_blank\">@theodotus</a> , this is a nice catch, but are you also stratifying your files?</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1193120,
      "author_name": "sahini",
      "author_url": "",
      "post_date": "02/09/2021 13:21:54",
      "content": "<p>In case of training on TPU, I got an error: : AttributeError: 'tuple' object has no attribute '_from_compatible_tensor_list'<br>\nI think the TPU doesn't support the local TF records. If I'm wrong, please give a correct comment.</p>",
      "votes": null,
      "replies": [
        {
          "id": 1193179,
          "author_name": "graf10a",
          "author_url": "",
          "post_date": "02/09/2021 13:52:12",
          "content": "<p>You are right -- when working with TPU you must get your tfrecords from  a Google Cloud Storage bucket, not from your local file system.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1193515,
          "author_name": "dimitreoliveira",
          "author_url": "",
          "post_date": "02/09/2021 17:11:41",
          "content": "<p>Yes <a href=\"https://www.kaggle.com/sahini\" target=\"_blank\">@sahini</a> as Alexey mentioned you must fetch the TFRecords from the GCS bucket, <a href=\"https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tpu-tensorflow-training/notebook\" target=\"_blank\">here is an example</a>.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1193674,
          "author_name": "sahini",
          "author_url": "",
          "post_date": "02/09/2021 19:31:25",
          "content": "<p>Thanks <a href=\"https://www.kaggle.com/DimitreOliveria\" target=\"_blank\">@DimitreOliveria</a>, you have provided the GCS path. <br>\nGCS_PATH = KaggleDatasets().get_gcs_path(f'cassava-leaf-disease-tfrecords-center-{HEIGHT}x{WIDTH}'</p>\n<p>I will try it. Thanks for your valuable contrubitions.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1195357,
          "author_name": "sahini",
          "author_url": "",
          "post_date": "02/10/2021 18:15:11",
          "content": "<p>I achieved 0.893 LB score with centered-512x512 tf records after 5-fold cv. I think that the tf records with original size (600x800) is better than using 512x512 size. Because, applying some augmentations on orginal size (600x800)  and resizing to 512x512 increases the LB score. However I dont know any link that is sharing the original size of tf records. Anyway, thanks again for your efforts.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1209041,
          "author_name": "dimitreoliveira",
          "author_url": "",
          "post_date": "02/18/2021 16:42:51",
          "content": "<p>You're welcome <a href=\"https://www.kaggle.com/sahini\" target=\"_blank\">@sahini</a> ,</p>\n<p>You can use the original TFRecords with that resolution, but note that they are not stratifyed.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1087545": "As reported by other members I also got a much lower score using the TFRecords provided by the competition, so I recreated the TFRecords from the JPG images and made them public.\n\nI have created two versions of the TFRecords, one that resized the original 600x800 images and another one that first center cropped the images to 600x600 and then resized them.\n\n\n\n\n| Resolution | Resized | Center croped | External Center croped |\n|------------|---------|----------------|----------|\n|   128x128  | [15 TFRecords](https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tfrecords-128x128) | [15 TFRecords](https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tfrecords-center-128x128) | [15 TFRecords](https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tfrecords-external-128x128) |\n|   256x256 | [15 TFRecords](https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tfrecords-256x256) | [15 TFRecords](https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tfrecords-center-256x256) | [15 TFRecords](https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tfrecords-external-256x256) |\n|   384x384| [15 TFRecords](https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tfrecords-384x384) | [15 TFRecords](https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tfrecords-center-384x384) | [15 TFRecords](https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tfrecords-external-384x384) |\n|   512x512  | [15 TFRecords](https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tfrecords-512x512) | [15 TFRecords](https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tfrecords-center-512x512) - [50 TFRecords](https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-50-tfrecords-center-512x512) | [15 TFRecords](https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tfrecords-external-512x512) - [50 TFRecords](https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-50-tfrecords-external-512x512) |\n\n___\n\n#### Divided by classes\n\n| Resolution | Center croped | External Center croped |\n|------------|----------------|-------------------------|\n| 512x512 | [15 TFRecords](https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tfrecords-classes-512x512) - [50 TFRecords](https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-50-tfrecords-classes-512x512) | [15 TFRecords](https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tfrecords-classes-ext-512x512) - [50 TFRecords](https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-ext-50-tfrec-classes-512x512) |\n\n\nI have added this last dataset that is divided by classes, it might be useful if you want to do some kind of class sampling or train GANs.\n\nHere you can find the [notebook to create the TFRecords](https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-stratified-tfrecords-256x256), you can modify the resolution parameters to change the output image resolution.\n\n#### The files were divided into 15 TFRecods files to make it easier to do K-Fold splits, they were stratified by the label.\n```\nFile: 1 has 1427 samples\nFile: 2 has 1427 samples\nFile: 3 has 1427 samples\nFile: 4 has 1427 samples\nFile: 5 has 1427 samples\nFile: 6 has 1427 samples\nFile: 7 has 1427 samples\nFile: 8 has 1426 samples\nFile: 9 has 1426 samples\nFile: 10 has 1426 samples\nFile: 11 has 1426 samples\nFile: 12 has 1426 samples\nFile: 13 has 1426 samples\nFile: 14 has 1426 samples\nFile: 15 has 1426 samples\n```\n#### Example of the label's distribution comparing the complete set to one of the TFRecord files\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1182060%2Fe384961ab4fecada33571cf08bcfa972%2FScreenshot%20from%202020-11-22%2017-28-39.png?generation=1606076946320113&alt=media)\n\n#### Image samples from the dataset\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1182060%2F7ca3f7c21c83e6157a1074ca96a5fdc8%2FScreenshot%20from%202020-11-23%2008-12-29.png?generation=1606129985784142&alt=media)",
    "1092568": "Updates to people that are using these datasets:\n- I have changed the encoding quality to 100%, now the resulting images should have better quality.\n- Now you can use two different datasets for each resolution, one resized the original images, and the other first center crops them to a square resolution (600x600) and then resizes the images.\n- The datasets also have the `training.csv` file with a `file` column, this column has the TFRecord file that each row belongs to.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1182060%2F6a570377b5e6630607dfefcd6738c2e0%2FScreenshot%20from%202020-11-26%2023-01-55.png?generation=1606442531542450&alt=media)",
    "1094242": "Just added a 512x512 dataset that is TFRecords divided by classes, it might be useful if you want to do some kind of class sampling or train GANs.",
    "1095807": "Another update, I have added datasets with the external data (from previous competition), this data was also stratified split into 15 TFRecords files, this should make it easier to concatenate with the original data.\n\nI have some concern about the quality of the external data, the images do not have all the same resolution, and the overall quality seems a little lower, here is a sample:\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1182060%2F7c9d379d6ed1596a89b26de25cc712f4%2FScreenshot%20from%202020-11-29%2022-15-00.png?generation=1606698926594994&alt=media)",
    "1103257": "Small update, I have removed the `2` duplicated files from all datasets",
    "1103820": "Hey thanks for the data, are you considering creating tf records containing 2019 + 2020 data ? It would be extremely helpful !",
    "1104085": "Would be really great if you create tf records with this data : https://www.kaggle.com/ammarali32/cassava-datasetv2",
    "1104108": "Hey @aziz69 I have created the TFRecords for the 2019 data, it is the external data. I left them in different  datasets because this way you have more control on how you wish to use it",
    "1104111": "Is this the dataser that has cropped images from the leaves?",
    "1104155": "yes, this is the discussion for that data https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/200722",
    "1104163": "Thanks amazing work.",
    "1104418": "Thanks @aziz69 , I will try to do some experiments with it to see if they look promissing.",
    "1192078": "Hi @dimitreoliveira \nI can't leave one thing unattended\nUsing your dataset I get higher OOF accuracy\nPlease tell me or am I wrong or do you know what is the reason?\nI I'm using tf.image.resize(default bilinear) + dataset.cache \nYou are using cv2.resize(default bilinear)",
    "1193120": "In case of training on TPU, I got an error: : AttributeError: 'tuple' object has no attribute '_from_compatible_tensor_list'\nI think the TPU doesn't support the local TF records. If I'm wrong, please give a correct comment.",
    "1193179": "You are right -- when working with TPU you must get your tfrecords from  a Google Cloud Storage bucket, not from your local file system.",
    "1193510": "Hi @theodotus , I am glad my dataset is giving good results to you.\n\nAbout the reason, I am not sure may it is because some of my datasets use center crop before resizing?",
    "1193515": "Yes @sahini as Alexey mentioned you must fetch the TFRecords from the GCS bucket, [here is an example](https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tpu-tensorflow-training/notebook).",
    "1193674": "Thanks @DimitreOliveria, you have provided the GCS path. \nGCS_PATH = KaggleDatasets().get_gcs_path(f'cassava-leaf-disease-tfrecords-center-{HEIGHT}x{WIDTH}'\n\nI will try it. Thanks for your valuable contrubitions.",
    "1195357": "I achieved 0.893 LB score with centered-512x512 tf records after 5-fold cv. I think that the tf records with original size (600x800) is better than using 512x512 size. Because, applying some augmentations on orginal size (600x800)  and resizing to 512x512 increases the LB score. However I dont know any link that is sharing the original size of tf records. Anyway, thanks again for your efforts.",
    "1206545": "dimitreoliveira thanks for Your answer\ncenter crop gives better results\n\nBut even if I use just resize, Your dataset gives better result than my\n As an answer to this question, I can [suggest this issue ](https://github.com/mlcommons/inference/issues/201)\nThere is mentioned that the resize method greatly affects the accuracy of the neural network.\nAnd many people have noticed that cv2.resize is better than tf.image.resize when training.",
    "1209039": "Hi @theodotus , this is a nice catch, but are you also stratifying your files?",
    "1209041": "You're welcome @sahini ,\n\nYou can use the original TFRecords with that resolution, but note that they are not stratifyed."
  },
  "source": "meta"
}