{
  "id": 148329,
  "title": "How to Score Over LB 0.970+",
  "url": "/competitions/flower-classification-with-tpus/discussion/148329",
  "author_name": "Chris Deotte",
  "post_date": "2020-05-04T00:34:54.523000",
  "votes": 37,
  "comment_count": 47,
  "views": 0,
  "content": "<h1>===========================</h1>\n\n<h1>IMPORTANT UPDATE May 6, 2020:</h1>\n\n<p>Kaggle has disallowed the use of images that overlap with test data within the following 5 external datasets:  ImageNet, Oxford 102 Category Flowers, TF Flowers, Open Images, and iNaturalist. Kaggle has added the following new rule:</p>\n\n<p>&gt;Training on any examples included in the test set will result in disqualification.</p>\n\n<p>Therefore the post below is for information purposes only. Do not select a final submission that trains on test images within these 5 datasets.</p>\n\n<h1>===========================</h1>\n\n<h1>How Is LB 0.970+ Possible?</h1>\n\n<p>You may have struggled to approach LB 0.970, so you may be wondering how so many people have LB over 0.970. To score over LB 0.970, you need to use external datasets.</p>\n\n<h1>Disclaimer: Using Flower External Data May Not Create Better Model</h1>\n\n<p>Normally more data means better model. However in Flower Comp, the reason that external data is so effective is because it contains many of this competition's answers. That means that although your LB score increases, your model isn't necessarily better at classifying flowers if you used your model offline or at home with your personal photos. (The LB is artificially higher because it has seen many of the answers).</p>\n\n<h1>Is It Allowed?</h1>\n\n<p>It would be more interesting if external datasets were disallowed, but Kaggle has officially said <a href=\"https://www.kaggle.com/c/flower-classification-with-tpus/discussion/134701#769297\">here</a> that we are allowed to use these external datasets. Furthermore to finish in the top LB and/or win a prize, you will need to use external datasets because everyone else is. This is sad, but it is what it is.</p>\n\n<h1>How To Include External Datasets</h1>\n\n<p>If you wish to train your TPU/GPU models with external datasets, below are instructions. In Martin's starter code <a href=\"https://www.kaggle.com/mgornergoogle/getting-started-with-100-flowers-on-tpu\">here</a>, he begins with the line</p>\n\n<pre><code>GCS_DS_PATH = KaggleDatasets().get_gcs_path()\n</code></pre>\n\n<h3>Step 1</h3>\n\n<p>When using additional datasets, you now need to explicitly declare the directories. So the above line must be changed to </p>\n\n<pre><code>GCS_DS_PATH = KaggleDatasets().get_gcs_path('flower-classification-with-tpus')\n</code></pre>\n\n<p>Next, you need to attach the additional Kaggle datasets to your notebook. User Kirill Blinov has created TFRecords <a href=\"https://www.kaggle.com/kirillblinov/tf-flower-photo-tfrec\">here</a> of Heng CherKeng's scrapped external datasets (explained <a href=\"https://www.kaggle.com/c/flower-classification-with-tpus/discussion/140866\">here</a>). I have verified (using pseudo label check) that Kirill and Heng have converted all the labels to match this competition's 104 labels. That means, you just need to add these datasets and start training!</p>\n\n<h3>Step 2</h3>\n\n<p>After attaching Kirill's datasets <a href=\"https://www.kaggle.com/kirillblinov/tf-flower-photo-tfrec\">here</a> to your notebook, add this line</p>\n\n<pre><code>PATH2 = KaggleDatasets().get_gcs_path('tf-flower-photo-tfrec')\n</code></pre>\n\n<p>And finally after creating the variable <code>TRAINING_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/train/*.tfrec')</code>, then add the following 5 lines. In place of <code>224</code> below, use either 192, 224, 331, or 512 depending on your desired resolution.</p>\n\n<pre><code>TRAINING_FILENAMES += tf.io.gfile.glob(PATH2 + '/imagenet/tfrecords-jpeg-224x224/*.tfrec')\nTRAINING_FILENAMES += tf.io.gfile.glob(PATH2 + '/inaturalist_1/tfrecords-jpeg-224x224/*.tfrec')\nTRAINING_FILENAMES += tf.io.gfile.glob(PATH2 + '/openimage/tfrecords-jpeg-224x224/*.tfrec')\nTRAINING_FILENAMES += tf.io.gfile.glob(PATH2 + '/oxford_102/tfrecords-jpeg-224x224/*.tfrec')\nTRAINING_FILENAMES += tf.io.gfile.glob(PATH2 + '/tf_flowers/tfrecords-jpeg-224x224/*.tfrec')\n</code></pre>\n\n<h3>Step 3</h3>\n\n<p>That's it. Now your <code>TRAINING_FILENAMES</code> contain 66758 training images! And you can continue your notebook the same as before. Training EfficientNetB7 with 224x224 takes 3 minutes per epoch and DenseNet201 takes 2 minutes. If you wish to train longer than 3 hours and/or ensemble models. Then after you train for a while, just </p>\n\n<pre><code>model.save_weights('my_weights.h5')\n</code></pre>\n\n<p>Add those weights to a Kaggle dataset, and start a new notebook, attach the Kaggle dataset, and before training or inferring, run</p>\n\n<pre><code>model.load_weights('my_weights.h5')\n</code></pre>\n\n<h3>Step 4</h3>\n\n<p>Good luck. Have fun! </p>\n\n<p>An example notebook has been published by Haveri <a href=\"https://www.kaggle.com/haveri/efficientnet-with-all-5-imagesets-s1\">here</a> (version 1 scores LB 0.972).</p>",
  "messages": [
    {
      "id": 832197,
      "postDate": "2020-05-04T00:34:54.523Z",
      "content": "<h1>===========================</h1>\n\n<h1>IMPORTANT UPDATE May 6, 2020:</h1>\n\n<p>Kaggle has disallowed the use of images that overlap with test data within the following 5 external datasets:  ImageNet, Oxford 102 Category Flowers, TF Flowers, Open Images, and iNaturalist. Kaggle has added the following new rule:</p>\n\n<p>&gt;Training on any examples included in the test set will result in disqualification.</p>\n\n<p>Therefore the post below is for information purposes only. Do not select a final submission that trains on test images within these 5 datasets.</p>\n\n<h1>===========================</h1>\n\n<h1>How Is LB 0.970+ Possible?</h1>\n\n<p>You may have struggled to approach LB 0.970, so you may be wondering how so many people have LB over 0.970. To score over LB 0.970, you need to use external datasets.</p>\n\n<h1>Disclaimer: Using Flower External Data May Not Create Better Model</h1>\n\n<p>Normally more data means better model. However in Flower Comp, the reason that external data is so effective is because it contains many of this competition's answers. That means that although your LB score increases, your model isn't necessarily better at classifying flowers if you used your model offline or at home with your personal photos. (The LB is artificially higher because it has seen many of the answers).</p>\n\n<h1>Is It Allowed?</h1>\n\n<p>It would be more interesting if external datasets were disallowed, but Kaggle has officially said <a href=\"https://www.kaggle.com/c/flower-classification-with-tpus/discussion/134701#769297\">here</a> that we are allowed to use these external datasets. Furthermore to finish in the top LB and/or win a prize, you will need to use external datasets because everyone else is. This is sad, but it is what it is.</p>\n\n<h1>How To Include External Datasets</h1>\n\n<p>If you wish to train your TPU/GPU models with external datasets, below are instructions. In Martin's starter code <a href=\"https://www.kaggle.com/mgornergoogle/getting-started-with-100-flowers-on-tpu\">here</a>, he begins with the line</p>\n\n<pre><code>GCS_DS_PATH = KaggleDatasets().get_gcs_path()\n</code></pre>\n\n<h3>Step 1</h3>\n\n<p>When using additional datasets, you now need to explicitly declare the directories. So the above line must be changed to </p>\n\n<pre><code>GCS_DS_PATH = KaggleDatasets().get_gcs_path('flower-classification-with-tpus')\n</code></pre>\n\n<p>Next, you need to attach the additional Kaggle datasets to your notebook. User Kirill Blinov has created TFRecords <a href=\"https://www.kaggle.com/kirillblinov/tf-flower-photo-tfrec\">here</a> of Heng CherKeng's scrapped external datasets (explained <a href=\"https://www.kaggle.com/c/flower-classification-with-tpus/discussion/140866\">here</a>). I have verified (using pseudo label check) that Kirill and Heng have converted all the labels to match this competition's 104 labels. That means, you just need to add these datasets and start training!</p>\n\n<h3>Step 2</h3>\n\n<p>After attaching Kirill's datasets <a href=\"https://www.kaggle.com/kirillblinov/tf-flower-photo-tfrec\">here</a> to your notebook, add this line</p>\n\n<pre><code>PATH2 = KaggleDatasets().get_gcs_path('tf-flower-photo-tfrec')\n</code></pre>\n\n<p>And finally after creating the variable <code>TRAINING_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/train/*.tfrec')</code>, then add the following 5 lines. In place of <code>224</code> below, use either 192, 224, 331, or 512 depending on your desired resolution.</p>\n\n<pre><code>TRAINING_FILENAMES += tf.io.gfile.glob(PATH2 + '/imagenet/tfrecords-jpeg-224x224/*.tfrec')\nTRAINING_FILENAMES += tf.io.gfile.glob(PATH2 + '/inaturalist_1/tfrecords-jpeg-224x224/*.tfrec')\nTRAINING_FILENAMES += tf.io.gfile.glob(PATH2 + '/openimage/tfrecords-jpeg-224x224/*.tfrec')\nTRAINING_FILENAMES += tf.io.gfile.glob(PATH2 + '/oxford_102/tfrecords-jpeg-224x224/*.tfrec')\nTRAINING_FILENAMES += tf.io.gfile.glob(PATH2 + '/tf_flowers/tfrecords-jpeg-224x224/*.tfrec')\n</code></pre>\n\n<h3>Step 3</h3>\n\n<p>That's it. Now your <code>TRAINING_FILENAMES</code> contain 66758 training images! And you can continue your notebook the same as before. Training EfficientNetB7 with 224x224 takes 3 minutes per epoch and DenseNet201 takes 2 minutes. If you wish to train longer than 3 hours and/or ensemble models. Then after you train for a while, just </p>\n\n<pre><code>model.save_weights('my_weights.h5')\n</code></pre>\n\n<p>Add those weights to a Kaggle dataset, and start a new notebook, attach the Kaggle dataset, and before training or inferring, run</p>\n\n<pre><code>model.load_weights('my_weights.h5')\n</code></pre>\n\n<h3>Step 4</h3>\n\n<p>Good luck. Have fun! </p>\n\n<p>An example notebook has been published by Haveri <a href=\"https://www.kaggle.com/haveri/efficientnet-with-all-5-imagesets-s1\">here</a> (version 1 scores LB 0.972).</p>",
      "rawMarkdown": "# ===========================\n# IMPORTANT UPDATE May 6, 2020:\nKaggle has disallowed the use of images that overlap with test data within the following 5 external datasets:  ImageNet, Oxford 102 Category Flowers, TF Flowers, Open Images, and iNaturalist. Kaggle has added the following new rule:\n\n&gt;Training on any examples included in the test set will result in disqualification.\n\nTherefore the post below is for information purposes only. Do not select a final submission that trains on test images within these 5 datasets.\n# ===========================\n\n# How Is LB 0.970+ Possible?\nYou may have struggled to approach LB 0.970, so you may be wondering how so many people have LB over 0.970. To score over LB 0.970, you need to use external datasets.\n\n# Disclaimer: Using Flower External Data May Not Create Better Model\nNormally more data means better model. However in Flower Comp, the reason that external data is so effective is because it contains many of this competition's answers. That means that although your LB score increases, your model isn't necessarily better at classifying flowers if you used your model offline or at home with your personal photos. (The LB is artificially higher because it has seen many of the answers).\n\n# Is It Allowed?\nIt would be more interesting if external datasets were disallowed, but Kaggle has officially said [here][1] that we are allowed to use these external datasets. Furthermore to finish in the top LB and/or win a prize, you will need to use external datasets because everyone else is. This is sad, but it is what it is.\n\n# How To Include External Datasets\nIf you wish to train your TPU/GPU models with external datasets, below are instructions. In Martin's starter code [here][2], he begins with the line\n\n    GCS_DS_PATH = KaggleDatasets().get_gcs_path()\n\n### Step 1\nWhen using additional datasets, you now need to explicitly declare the directories. So the above line must be changed to \n\n    GCS_DS_PATH = KaggleDatasets().get_gcs_path('flower-classification-with-tpus')\n\nNext, you need to attach the additional Kaggle datasets to your notebook. User Kirill Blinov has created TFRecords [here][4] of Heng CherKeng's scrapped external datasets (explained [here][3]). I have verified (using pseudo label check) that Kirill and Heng have converted all the labels to match this competition's 104 labels. That means, you just need to add these datasets and start training!\n\n### Step 2\nAfter attaching Kirill's datasets [here][4] to your notebook, add this line\n\n    PATH2 = KaggleDatasets().get_gcs_path('tf-flower-photo-tfrec')\n\nAnd finally after creating the variable `TRAINING_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/train/*.tfrec')`, then add the following 5 lines. In place of `224` below, use either 192, 224, 331, or 512 depending on your desired resolution.\n\n    TRAINING_FILENAMES += tf.io.gfile.glob(PATH2 + '/imagenet/tfrecords-jpeg-224x224/*.tfrec')\n    TRAINING_FILENAMES += tf.io.gfile.glob(PATH2 + '/inaturalist_1/tfrecords-jpeg-224x224/*.tfrec')\n    TRAINING_FILENAMES += tf.io.gfile.glob(PATH2 + '/openimage/tfrecords-jpeg-224x224/*.tfrec')\n    TRAINING_FILENAMES += tf.io.gfile.glob(PATH2 + '/oxford_102/tfrecords-jpeg-224x224/*.tfrec')\n    TRAINING_FILENAMES += tf.io.gfile.glob(PATH2 + '/tf_flowers/tfrecords-jpeg-224x224/*.tfrec')\n\n### Step 3\nThat's it. Now your `TRAINING_FILENAMES` contain 66758 training images! And you can continue your notebook the same as before. Training EfficientNetB7 with 224x224 takes 3 minutes per epoch and DenseNet201 takes 2 minutes. If you wish to train longer than 3 hours and/or ensemble models. Then after you train for a while, just \n\n    model.save_weights('my_weights.h5')\n\nAdd those weights to a Kaggle dataset, and start a new notebook, attach the Kaggle dataset, and before training or inferring, run\n\n    model.load_weights('my_weights.h5')\n\n### Step 4\nGood luck. Have fun! \n\nAn example notebook has been published by Haveri [here][5] (version 1 scores LB 0.972).\n\n[1]: https://www.kaggle.com/c/flower-classification-with-tpus/discussion/134701#769297\n[2]: https://www.kaggle.com/mgornergoogle/getting-started-with-100-flowers-on-tpu\n[3]: https://www.kaggle.com/c/flower-classification-with-tpus/discussion/140866\n[4]: https://www.kaggle.com/kirillblinov/tf-flower-photo-tfrec\n[5]: https://www.kaggle.com/haveri/efficientnet-with-all-5-imagesets-s1\n[6]: https://www.kaggle.com/c/flower-classification-with-tpus/discussion/148329#836394",
      "votes": 37
    },
    {
      "id": 834764,
      "postDate": "2020-05-05T18:50:10.113Z",
      "content": "<p>This is a playground competition. The purpose is educational. So yes, training on the test data will get you higher scores but it is not very educational. As for the rules, yes, in a playground competition no one will prevent you from doing so, because it is a playground and you do more or less what you want.</p>\n\n<p><a href=\"/cdeotte\">@cdeotte</a> For the purpose of people's education, could you please add a comment to your post that training on the test data artificially inflates your accuracy score but does not result in better real-life performance, i.e. if you were to try this classifier on pictures you took yourself in nature.</p>",
      "rawMarkdown": "This is a playground competition. The purpose is educational. So yes, training on the test data will get you higher scores but it is not very educational. As for the rules, yes, in a playground competition no one will prevent you from doing so, because it is a playground and you do more or less what you want.\n\n@cdeotte For the purpose of people's education, could you please add a comment to your post that training on the test data artificially inflates your accuracy score but does not result in better real-life performance, i.e. if you were to try this classifier on pictures you took yourself in nature.",
      "votes": 6,
      "replies": [
        {
          "id": 834823,
          "postDate": "2020-05-05T19:49:01.320Z",
          "content": "<p>I agree it may not be educational and i would prefer that Kaggle disallowed it. But currently only participants who train with external data (which includes test data) will win a prize, so it is what it is...</p>\n\n<p>&gt; TPU Leaderboard Prizes: Awarded to the solutions using TPUs with highest leaderboard rankings.\n1st Place: Coral USB edge TPU accelerator + Raspberry Pi 4, 4GB RAM + Rapberry Pi Camera V2\n2nd Place: Coral USB edge TPU accelerator\n3rd Place: Coral USB edge TPU accelerator</p>",
          "rawMarkdown": "I agree it may not be educational and i would prefer that Kaggle disallowed it. But currently only participants who train with external data (which includes test data) will win a prize, so it is what it is...\n\n&gt; TPU Leaderboard Prizes: Awarded to the solutions using TPUs with highest leaderboard rankings.\n1st Place: Coral USB edge TPU accelerator + Raspberry Pi 4, 4GB RAM + Rapberry Pi Camera V2\n2nd Place: Coral USB edge TPU accelerator\n3rd Place: Coral USB edge TPU accelerator\n\n[1]: https://www.kaggle.com/c/flower-classification-with-tpus/discussion/134701",
          "votes": 3
        },
        {
          "id": 834838,
          "postDate": "2020-05-05T20:07:55.933Z",
          "content": "<p>I added a prominent disclaimer to my post so participants realize what's what.</p>",
          "rawMarkdown": "I added a prominent disclaimer to my post so participants realize what's what."
        },
        {
          "id": 834993,
          "postDate": "2020-05-06T00:58:51.240Z",
          "content": "<p>OK, thanks</p>",
          "rawMarkdown": "OK, thanks"
        }
      ]
    },
    {
      "id": 832563,
      "postDate": "2020-05-04T08:36:37.357Z",
      "content": "<p>I try using these dataset to train, but get nan? especially using imagenet, openimage and xxxxxlist</p>",
      "rawMarkdown": "I try using these dataset to train, but get nan? especially using imagenet, openimage and xxxxxlist",
      "votes": 1,
      "replies": [
        {
          "id": 832968,
          "postDate": "2020-05-04T14:26:58.207Z",
          "content": "<p>That's weird. If you only train with the original dataset, do you still get nan? Are you using <code>sparse_categorical_crossentropy</code> loss and leaving your <code>y_train</code> as integers? </p>",
          "rawMarkdown": "That's weird. If you only train with the original dataset, do you still get nan? Are you using `sparse_categorical_crossentropy` loss and leaving your `y_train` as integers? "
        },
        {
          "id": 833623,
          "postDate": "2020-05-05T00:23:15.333Z",
          "content": "<p>I noticed the same problem. </p>",
          "rawMarkdown": "I noticed the same problem. "
        },
        {
          "id": 833635,
          "postDate": "2020-05-05T00:45:47.113Z",
          "content": "<p>Using official dataset and oxford ext dataset, it works as usual with sparse categorical ce, not nan...\nI think these dataset I said above maybe noisy or wrong labels...</p>",
          "rawMarkdown": "Using official dataset and oxford ext dataset, it works as usual with sparse categorical ce, not nan...\nI think these dataset I said above maybe noisy or wrong labels..."
        },
        {
          "id": 833873,
          "postDate": "2020-05-05T05:48:16.520Z",
          "content": "<p>I don't have this problem with the original dataset. \nI copy-pasted your instructions. \nI can display images of the dataset.\nI looked into the whole label set. Everything looks fine:</p>\n\n<p>Class 73 has the highest number of images: 7315\nClass 15 has the lowest number of images: 36</p>\n\n<p>But I get a loss = NaN with both efficientNet and DenseNet when fitting. I am puzzled.</p>",
          "rawMarkdown": "I don't have this problem with the original dataset. \nI copy-pasted your instructions. \nI can display images of the dataset.\nI looked into the whole label set. Everything looks fine:\n\nClass 73 has the highest number of images: 7315\nClass 15 has the lowest number of images: 36\n\nBut I get a loss = NaN with both efficientNet and DenseNet when fitting. I am puzzled."
        },
        {
          "id": 833878,
          "postDate": "2020-05-05T05:59:12.040Z",
          "content": "<p>I am not 100% sure but it seems that some data augmentation operations are the cause of this problem. I disabled data augmentation and it seems working now.</p>",
          "rawMarkdown": "I am not 100% sure but it seems that some data augmentation operations are the cause of this problem. I disabled data augmentation and it seems working now.",
          "votes": 1
        },
        {
          "id": 835069,
          "postDate": "2020-05-06T03:04:47.313Z",
          "content": "<p><a href=\"/cdeotte\">@cdeotte</a> \nI am now ranked 24th using the external dataset you suggested. Many Thx. Unfortunately, I almost exhausted my TPU quota and cannot improve </p>",
          "rawMarkdown": "@cdeotte \nI am now ranked 24th using the external dataset you suggested. Many Thx. Unfortunately, I almost exhausted my TPU quota and cannot improve ",
          "votes": 2
        },
        {
          "id": 835084,
          "postDate": "2020-05-06T03:23:38.183Z",
          "content": "<p>Nice job Alban</p>",
          "rawMarkdown": "Nice job Alban"
        },
        {
          "id": 835232,
          "postDate": "2020-05-06T06:09:16.587Z",
          "content": "<p>You are right! I disabled augmentation and it works again! But it is still puzzled, why augmenting these dataset can lead to nan?</p>",
          "rawMarkdown": "You are right! I disabled augmentation and it works again! But it is still puzzled, why augmenting these dataset can lead to nan?"
        },
        {
          "id": 836120,
          "postDate": "2020-05-06T18:23:43.280Z",
          "content": "<p>I do not know.  I do not understand why it works with the original dataset and not with the external dataset.</p>",
          "rawMarkdown": "I do not know.  I do not understand why it works with the original dataset and not with the external dataset."
        }
      ]
    },
    {
      "id": 836678,
      "postDate": "2020-05-07T06:33:04.220Z",
      "content": "<p>Thanks Chris for pursuing this and the important update above. I will copy them on to the kernel/notebooks I have shared. Already many have forked the code and may not know about the new rule. I was lucky to notice this discussion topic while searching for some thing else. Will need to spread this information and hope all realize it.</p>",
      "rawMarkdown": "Thanks Chris for pursuing this and the important update above. I will copy them on to the kernel/notebooks I have shared. Already many have forked the code and may not know about the new rule. I was lucky to notice this discussion topic while searching for some thing else. Will need to spread this information and hope all realize it.",
      "votes": 2
    },
    {
      "id": 836349,
      "postDate": "2020-05-06T22:50:48.610Z",
      "content": "<p>Hi all, thanks for raising the concern about external datasets that mirror the same images as the test set. I want to reiterate that the rules prohibit training on test set images: “Submissions may not use or incorporate information from hand labeling or human prediction of the validation dataset or test data records.” So, while you are permitted to use the 5 datasets that we used to source the training/test sets as <a href=\"https://www.kaggle.com/c/flower-classification-with-tpus/discussion/134701#769297\">I stated previously here</a>, <strong>it is prohibited to train your models using any mirrors/overlaps/duplication of the competition's test data within other flower datasets, as this conflicts with the above-stated rule.</strong></p>\n\n<p>All potential winners will be required to hand over their full code, including specifying the exact external datasets they've used. We will be cross-checking training images with those contained in the test set to confirm no overlaps as a condition to awarding prizes. If you've already committed this violation, you should ensure you do not select any submissions making use of prohibited data for your final 2 submissions that will be evaluated on the final Private Leaderboard.</p>",
      "rawMarkdown": "Hi all, thanks for raising the concern about external datasets that mirror the same images as the test set. I want to reiterate that the rules prohibit training on test set images: “Submissions may not use or incorporate information from hand labeling or human prediction of the validation dataset or test data records.” So, while you are permitted to use the 5 datasets that we used to source the training/test sets as [I stated previously here](https://www.kaggle.com/c/flower-classification-with-tpus/discussion/134701#769297), **it is prohibited to train your models using any mirrors/overlaps/duplication of the competition's test data within other flower datasets, as this conflicts with the above-stated rule.**\n\nAll potential winners will be required to hand over their full code, including specifying the exact external datasets they've used. We will be cross-checking training images with those contained in the test set to confirm no overlaps as a condition to awarding prizes. If you've already committed this violation, you should ensure you do not select any submissions making use of prohibited data for your final 2 submissions that will be evaluated on the final Private Leaderboard.",
      "replies": [
        {
          "id": 836369,
          "postDate": "2020-05-06T23:43:24.293Z",
          "content": "<p>Hi, I read your previous comment to relate only to hand-labeling the test set or pre-labeled versions of the test set, not external datasets that may include examples included in the test set. </p>\n\n<p>&gt; However, the rules do specify that hand-labeling of the test set and incorporating hand-labeled or pre-labeled versions of the test set in your training is prohibited.</p>\n\n<p>Could you confirm that training on any examples included in the test set, even if they were not hand labeled, will result in disqualification? </p>",
          "rawMarkdown": "Hi, I read your previous comment to relate only to hand-labeling the test set or pre-labeled versions of the test set, not external datasets that may include examples included in the test set. \n\n&gt; However, the rules do specify that hand-labeling of the test set and incorporating hand-labeled or pre-labeled versions of the test set in your training is prohibited.\n\nCould you confirm that training on any examples included in the test set, even if they were not hand labeled, will result in disqualification? "
        },
        {
          "id": 836394,
          "postDate": "2020-05-07T00:16:12.490Z",
          "content": "<p><a href=\"/calebeverett\">@calebeverett</a> That's right. Training on any examples included in the test set will result in disqualification. The hand/pre-labeled specification is intended to cover the entirety of the use of such test set images, whether that be someone taking the unlabeled test data and hand-labeling it or using some pre-labeled version of that data. So yes, for complete avoidance of doubt, don't use images that are or are the same as the test set's images.</p>",
          "rawMarkdown": "@calebeverett That's right. Training on any examples included in the test set will result in disqualification. The hand/pre-labeled specification is intended to cover the entirety of the use of such test set images, whether that be someone taking the unlabeled test data and hand-labeling it or using some pre-labeled version of that data. So yes, for complete avoidance of doubt, don't use images that are or are the same as the test set's images."
        },
        {
          "id": 836401,
          "postDate": "2020-05-07T00:19:04.887Z",
          "content": "<p>omg, did I train on images including the test set?  i.e does the external data set at <a href=\"https://www.kaggle.com/kirillblinov/tf-flower-photo-tfrec\">https://www.kaggle.com/kirillblinov/tf-flower-photo-tfrec</a> include the test set? I even do not know it....</p>",
          "rawMarkdown": "omg, did I train on images including the test set?  i.e does the external data set at https://www.kaggle.com/kirillblinov/tf-flower-photo-tfrec include the test set? I even do not know it...."
        },
        {
          "id": 836405,
          "postDate": "2020-05-07T00:23:05.047Z",
          "content": "<p>Are we allowed to use pseudo labeled external data? (i.e. download external datasets then use a model trained on competition training data to predict the labels on external images?) </p>",
          "rawMarkdown": "Are we allowed to use pseudo labeled external data? (i.e. download external datasets then use a model trained on competition training data to predict the labels on external images?) ",
          "votes": 1
        },
        {
          "id": 836410,
          "postDate": "2020-05-07T00:31:01.627Z",
          "content": "<p><a href=\"/juliaelliott\">@juliaelliott</a> I would like to point out that the following clause does not prevent the use of external images that overlap the test dataset.\n&gt; However, the rules do specify that hand-labeling of the test set and incorporating hand-labeled or pre-labeled versions of the test set in your training is prohibited.</p>\n\n<p>For example, we can download a folder of external images without labels. Then we use a model trained on competition training data to predict the labels in the folder. Lastly we set all labels in the folder equal to the most common predicted label.</p>\n\n<p>None-the-less, I understand your intention is to disallow the 5 external datasets used to create the test data, so I have updated my discussion post above.</p>",
          "rawMarkdown": "@juliaelliott I would like to point out that the following clause does not prevent the use of external images that overlap the test dataset.\n&gt; However, the rules do specify that hand-labeling of the test set and incorporating hand-labeled or pre-labeled versions of the test set in your training is prohibited.\n\nFor example, we can download a folder of external images without labels. Then we use a model trained on competition training data to predict the labels in the folder. Lastly we set all labels in the folder equal to the most common predicted label.\n\nNone-the-less, I understand your intention is to disallow the 5 external datasets used to create the test data, so I have updated my discussion post above."
        },
        {
          "id": 836465,
          "postDate": "2020-05-07T02:28:04.720Z",
          "content": "<p>on a related side note, one can find  pretrained model  train on external data with the external labels. For example you can find pretrained models of inaturalist, imagenet-21K or open image JFT-300m. Will that be allowed? one might use external models for:\n- fine tuning \n- using extracted image features from external model</p>",
          "rawMarkdown": "on a related side note, one can find  pretrained model  train on external data with the external labels. For example you can find pretrained models of inaturalist, imagenet-21K or open image JFT-300m. Will that be allowed? one might use external models for:\n- fine tuning \n- using extracted image features from external model\n\n"
        },
        {
          "id": 836473,
          "postDate": "2020-05-07T02:32:55.467Z",
          "content": "<p>Oh right lol. Kaggle's starter notebook <a href=\"https://www.kaggle.com/mgornergoogle/getting-started-with-100-flowers-on-tpu\">here</a> is illegal. Kaggle uses pretrained weights from <code>ImageNet</code> which includes the test images. Here is their model</p>\n\n<pre><code>    pretrained_model = tf.keras.applications.VGG16(weights='imagenet', include_top=False ,input_shape=[*IMAGE_SIZE, 3])\n</code></pre>",
          "rawMarkdown": "Oh right lol. Kaggle's starter notebook [here][1] is illegal. Kaggle uses pretrained weights from `ImageNet` which includes the test images. Here is their model\n\n        pretrained_model = tf.keras.applications.VGG16(weights='imagenet', include_top=False ,input_shape=[*IMAGE_SIZE, 3])\n\n[1]: https://www.kaggle.com/mgornergoogle/getting-started-with-100-flowers-on-tpu",
          "votes": 1
        },
        {
          "id": 836474,
          "postDate": "2020-05-07T02:35:09.293Z",
          "content": "<p>Also note that Kaggle's new rule</p>\n\n<blockquote>\n  <p>Training on any examples included in the test set will result in disqualification</p>\n</blockquote>\n\n<p>disallows pseudo labeling of the test data.</p>",
          "rawMarkdown": "Also note that Kaggle's new rule\n\n&gt; Training on any examples included in the test set will result in disqualification\n\n disallows pseudo labeling of the test data.",
          "votes": 1
        },
        {
          "id": 836606,
          "postDate": "2020-05-07T05:26:07.517Z",
          "rawMarkdown": "",
          "isDeleted": true
        },
        {
          "id": 836701,
          "postDate": "2020-05-07T06:56:36.947Z",
          "content": "<p><a href=\"/cdeotte\">@cdeotte</a> \n\"Oh right lol. Kaggle's starter notebook here is illegal. Kaggle uses pretrained weights from ImageNet which includes the test images. Here is their model\"</p>\n\n<p>Note that there are imagenet(21K class) and ILSVRC-imagenet (1000 class). ILSVRC-imagenet-1000 is subset of imagenet-21k. pretarin models like vgg,resnet are based on 1000 class and i think there no overlap with the kaggle test set.</p>\n\n<p>i am actually refer to imagenet 21K, like:\n<a href=\"https://github.com/dmlc/mxnet-model-gallery/blob/master/imagenet-21k-inception.md\">https://github.com/dmlc/mxnet-model-gallery/blob/master/imagenet-21k-inception.md</a>\n<a href=\"https://github.com/pertusa/InceptionBN-21K-for-Caffe\">https://github.com/pertusa/InceptionBN-21K-for-Caffe</a></p>\n\n<p>jft 300-m:\n<a href=\"https://github.com/Tencent/tencent-ml-images\">https://github.com/Tencent/tencent-ml-images</a>\nefficientnet noisy student? : <a href=\"https://paperswithcode.com/sota/image-classification-on-imagenet\">https://paperswithcode.com/sota/image-classification-on-imagenet</a></p>\n\n<p>inaturalist\n<a href=\"https://coral.ai/models/\">https://coral.ai/models/</a></p>\n\n<hr>\n\n<p>efficientnet for noisy student(pretrained with jft-300m could be a big issue here) ?</p>",
          "rawMarkdown": "@cdeotte \n\"Oh right lol. Kaggle's starter notebook here is illegal. Kaggle uses pretrained weights from ImageNet which includes the test images. Here is their model\"\n\nNote that there are imagenet(21K class) and ILSVRC-imagenet (1000 class). ILSVRC-imagenet-1000 is subset of imagenet-21k. pretarin models like vgg,resnet are based on 1000 class and i think there no overlap with the kaggle test set.\n\ni am actually refer to imagenet 21K, like:\nhttps://github.com/dmlc/mxnet-model-gallery/blob/master/imagenet-21k-inception.md\nhttps://github.com/pertusa/InceptionBN-21K-for-Caffe\n\njft 300-m:\nhttps://github.com/Tencent/tencent-ml-images\nefficientnet noisy student? : https://paperswithcode.com/sota/image-classification-on-imagenet\n\ninaturalist\nhttps://coral.ai/models/\n\n----\nefficientnet for noisy student(pretrained with jft-300m could be a big issue here) ?",
          "votes": 2
        },
        {
          "id": 836817,
          "postDate": "2020-05-07T09:05:23.620Z",
          "content": "<p><a href=\"/juliaelliott\">@juliaelliott</a> Kaggle should clarify this situation fast</p>\n\n<p>1) using imagenet, noisy student weights in pretrained models</p>\n\n<p>2) pseudo labeling of the test data</p>\n\n<p>3) using/loading own pre-trained models, like <a href=\"/cdeotte\">@cdeotte</a> mentioned above (did such a model have had to exist before 5.5. in a public dataset???) ( I hope I got the tenses correct :) ) - so I mean, is it allowed to pretrain 3 models and load them for the competition entry and ensemble them? btw no one would be able to check with which data those models are trained.</p>\n\n<p>How can I know if the published external dataset (<a href=\"https://www.kaggle.com/kirillblinov/tf-flower-photo-tfrec\">https://www.kaggle.com/kirillblinov/tf-flower-photo-tfrec</a>) has 1 image in it, that is included in the test dataset?</p>\n\n<p>Can you commit that the test dataset contains only transformed variations of the original images in the  external dataset?</p>\n\n<p>btw tf2.2 is released</p>",
          "rawMarkdown": "@juliaelliott Kaggle should clarify this situation fast\n\n1) using imagenet, noisy student weights in pretrained models\n\n2) pseudo labeling of the test data\n\n3) using/loading own pre-trained models, like @cdeotte mentioned above (did such a model have had to exist before 5.5. in a public dataset???) ( I hope I got the tenses correct :) ) - so I mean, is it allowed to pretrain 3 models and load them for the competition entry and ensemble them? btw no one would be able to check with which data those models are trained.\n\nHow can I know if the published external dataset (https://www.kaggle.com/kirillblinov/tf-flower-photo-tfrec) has 1 image in it, that is included in the test dataset?\n\nCan you commit that the test dataset contains only transformed variations of the original images in the  external dataset?\n\nbtw tf2.2 is released\n"
        },
        {
          "id": 837427,
          "postDate": "2020-05-07T19:03:16.677Z",
          "content": "<p>All great clarification questions.\n- Pseudolabeling is permitted unless you're pseudolabeling using images from or duplicative of images in the test set.\n- Using publicly-available pre-trained models is allowed, including those with components of the test set underlying the pre-trained models. To be super clear, this means you're not permitted to be \"pre-training\" models using the test set and then calling that \"pre-trained,\" but you can use the host of ImageNet or other canned publicly-available pre-trained models mentioned in preceding posts.\n- The currently published datasets can be used (including the 5 datasets from which the training/test sets were sourced), but images which overlap with or are the same as the test set cannot be used (should be removed). Correct, no new external datasets can be announced.</p>\n\n<p>Lastly, a reiteration that this is a <strong>playground competition</strong>. As with all playgrounds, it was created in the spirit of learning (specifically for the launch of our TPU integration), more than the pursuit of winning. Hence, it is not awarding points/medals, and prizes are non-monetary &amp; centered around TPU use. We launched this with the knowledge there would be vulnerabilities and more grey area than we'd prefer in a featured competition, hence sending it to the playground. I hope sound judgment is used by participants that aligns with this competition's spirit, so that we can continue to offer opportunities like this.</p>",
          "rawMarkdown": "All great clarification questions.\n- Pseudolabeling is permitted unless you're pseudolabeling using images from or duplicative of images in the test set.\n- Using publicly-available pre-trained models is allowed, including those with components of the test set underlying the pre-trained models. To be super clear, this means you're not permitted to be \"pre-training\" models using the test set and then calling that \"pre-trained,\" but you can use the host of ImageNet or other canned publicly-available pre-trained models mentioned in preceding posts.\n- The currently published datasets can be used (including the 5 datasets from which the training/test sets were sourced), but images which overlap with or are the same as the test set cannot be used (should be removed). Correct, no new external datasets can be announced.\n\nLastly, a reiteration that this is a **playground competition**. As with all playgrounds, it was created in the spirit of learning (specifically for the launch of our TPU integration), more than the pursuit of winning. Hence, it is not awarding points/medals, and prizes are non-monetary &amp; centered around TPU use. We launched this with the knowledge there would be vulnerabilities and more grey area than we'd prefer in a featured competition, hence sending it to the playground. I hope sound judgment is used by participants that aligns with this competition's spirit, so that we can continue to offer opportunities like this.",
          "votes": 3
        },
        {
          "id": 837629,
          "postDate": "2020-05-07T23:06:44.833Z",
          "content": "<p>Thank you for the opportunity to work on the excellent v3 TPUs and on\nthis problem.  I have learnt a lot about TPUs, tensorflow, deep\nnetworks and machine learning in general.</p>\n\n<p>Unfortunately, it is now almost impossible to tell how the public\nleaderboard would look like without the tainted submissions.  It seems\nthat most of top listings in the public leaderboard would have used\nthese external datasets.</p>\n\n<p>Have the rules been modified?  I do not see a rule that \"Training on\nany examples included in the test set will result in disqualification\"\nin the <a href=\"https://www.kaggle.com/c/flower-classification-with-tpus/rules\">Rules</a>.  While this rule has been mentioned in a comment in this\nthread, it seems that it should be included in the Rules page to make it\nofficial.</p>\n\n<p>Is it possible to mark the entries on the leaderboard that are tainted\nor better yet, perhaps create an alternate leaderboard that has\nsubmissions that do not include the disallowed datasets?</p>\n\n<p>Why not disallow all the shared datasets thus far rather than saying\nthat they can be used after removing overlapped images?  IMHO, it may\nnot be trivial to remove the duplicate images and that could lead to\ncontroversies of their own.</p>\n\n<p>To spread more the rules modification (assuming that it is indeed\nmodified), perhaps a prominent note can be added to the Leaderboard\npage with this new information.  Had I not come to this page by\naccident, I may have never known!</p>\n\n<p>Also, I ended up including these datasets after seeing this post only\nlike a couple days ago and after seeing another Kaggle Team member's\nresponse acknowledging the disclaimer put up by this topic's author,\nbut did not say that including the datasets would result in a\ndisqualification.  I wouldn't have had I known.  I feel like I have\nwasted a significant part of my TPU quota (and because of it I have\nalmost run out of quota), not to mention time, etc.</p>\n\n<p>There are two aspects to this playground competition--(1) it is a\nplayground for learning, and (2) it is a competition (even if it is\nfor bragging rights and nothing else).  Also, I do understand the\nspirit of all this--including the test data in the training set\ndefeats the purpose of building a model that can recognize samples in\ngeneral.  But it is a competition nonetheless and I feel that the\nplaying field has been mucked up (I think unintentionally).  It would\nbe great to make it level again for the spirit of the competition.\nFrom a competition perspective, I don't think it is level the way\nthings are right now.</p>\n\n<p>That said, I am extremely extremely grateful for the opportunity to\nuse v3 TPUs with eight compute cores (for free) and the opportunity to\nwork on this problem.  I have learnt a lot about TPUs and tensorflow\nand not to mention a hundred and four types of flowers 😊.  So the\nplayground aspect of it has been more than satisfied.  And if we're\nnot able to solve the competition aspect of it, I guess that's totally\nO.K. as well.</p>\n\n<p>Thank you!</p>",
          "rawMarkdown": "Thank you for the opportunity to work on the excellent v3 TPUs and on\nthis problem.  I have learnt a lot about TPUs, tensorflow, deep\nnetworks and machine learning in general.\n\nUnfortunately, it is now almost impossible to tell how the public\nleaderboard would look like without the tainted submissions.  It seems\nthat most of top listings in the public leaderboard would have used\nthese external datasets.\n\nHave the rules been modified?  I do not see a rule that \"Training on\nany examples included in the test set will result in disqualification\"\nin the [Rules](https://www.kaggle.com/c/flower-classification-with-tpus/rules).  While this rule has been mentioned in a comment in this\nthread, it seems that it should be included in the Rules page to make it\nofficial.\n\nIs it possible to mark the entries on the leaderboard that are tainted\nor better yet, perhaps create an alternate leaderboard that has\nsubmissions that do not include the disallowed datasets?\n\nWhy not disallow all the shared datasets thus far rather than saying\nthat they can be used after removing overlapped images?  IMHO, it may\nnot be trivial to remove the duplicate images and that could lead to\ncontroversies of their own.\n\nTo spread more the rules modification (assuming that it is indeed\nmodified), perhaps a prominent note can be added to the Leaderboard\npage with this new information.  Had I not come to this page by\naccident, I may have never known!\n\nAlso, I ended up including these datasets after seeing this post only\nlike a couple days ago and after seeing another Kaggle Team member's\nresponse acknowledging the disclaimer put up by this topic's author,\nbut did not say that including the datasets would result in a\ndisqualification.  I wouldn't have had I known.  I feel like I have\nwasted a significant part of my TPU quota (and because of it I have\nalmost run out of quota), not to mention time, etc.\n\nThere are two aspects to this playground competition--(1) it is a\nplayground for learning, and (2) it is a competition (even if it is\nfor bragging rights and nothing else).  Also, I do understand the\nspirit of all this--including the test data in the training set\ndefeats the purpose of building a model that can recognize samples in\ngeneral.  But it is a competition nonetheless and I feel that the\nplaying field has been mucked up (I think unintentionally).  It would\nbe great to make it level again for the spirit of the competition.\nFrom a competition perspective, I don't think it is level the way\nthings are right now.\n\nThat said, I am extremely extremely grateful for the opportunity to\nuse v3 TPUs with eight compute cores (for free) and the opportunity to\nwork on this problem.  I have learnt a lot about TPUs and tensorflow\nand not to mention a hundred and four types of flowers 😊.  So the\nplayground aspect of it has been more than satisfied.  And if we're\nnot able to solve the competition aspect of it, I guess that's totally\nO.K. as well.\n\nThank you!",
          "votes": 5
        },
        {
          "id": 837989,
          "postDate": "2020-05-08T07:51:43.253Z",
          "content": "<p>Thanks for the clarification. But some things are not completely clear.</p>\n\n<p>First, how do I know that the published external datasets that I might use do not contain any images from the test set? Is there a way to verify it? Was the test set published already that I'm ignorant of?</p>\n\n<p>Second, can I have a clarity on the 'publicly available pre-trained models' that I can use? In the sense, if I make my model (pre-trained on publicly available datasets) public, will I be able to use it without violating the rules? Please comment upon this.</p>\n\n<p>Thanks!</p>",
          "rawMarkdown": "Thanks for the clarification. But some things are not completely clear.\n\nFirst, how do I know that the published external datasets that I might use do not contain any images from the test set? Is there a way to verify it? Was the test set published already that I'm ignorant of?\n\nSecond, can I have a clarity on the 'publicly available pre-trained models' that I can use? In the sense, if I make my model (pre-trained on publicly available datasets) public, will I be able to use it without violating the rules? Please comment upon this.\n\nThanks!"
        },
        {
          "id": 838028,
          "postDate": "2020-05-08T08:26:58.080Z",
          "content": "<p>Akash G. Published external datasets do contain images from the test set. At least 12 images are an exact match. There are many more that come very close to the test images. <a href=\"https://www.kaggle.com/haveri/how-similar-are-the-test-and-training-set\">.Some experiments done and results published can be found here.</a></p>\n\n<p>A possibly faster way to compare the images exists in <a href=\"https://www.kaggle.com/kirillblinov/prepare-dataset\">code released by Kirill here</a>.</p>\n\n<p>At this time it is best to quit using the external dataset and just use what was provided in the competition.</p>",
          "rawMarkdown": "Akash G. Published external datasets do contain images from the test set. At least 12 images are an exact match. There are many more that come very close to the test images. [.Some experiments done and results published can be found here.](https://www.kaggle.com/haveri/how-similar-are-the-test-and-training-set)\n\nA possibly faster way to compare the images exists in [code released by Kirill here](https://www.kaggle.com/kirillblinov/prepare-dataset).\n\nAt this time it is best to quit using the external dataset and just use what was provided in the competition.",
          "votes": 2
        },
        {
          "id": 849618,
          "postDate": "2020-05-15T23:35:07.437Z",
          "content": "<p><a href=\"/juliaelliott\">@juliaelliott</a> what you state in the above is stronger than the rules, especially:</p>\n\n<blockquote>\n  <p>Pseudolabeling is permitted unless you're pseudolabeling using images from or duplicative of images in the test set.</p>\n</blockquote>\n\n<p>I am not challenging the change in rules you set for the TPU Flower competition, but I am challenging if this becomes a general policy where you can change rules in the middle of a competition like this.  </p>\n\n<p>If pseudo labeling of test data is prohibited then it should be written in competition rules.  To be crystal clear, pseudo labeling is not hand labeling nor scrapping labels from an external source. It is not forbidden by current rules.</p>",
          "rawMarkdown": "@juliaelliott what you state in the above is stronger than the rules, especially:\n\n&gt; Pseudolabeling is permitted unless you're pseudolabeling using images from or duplicative of images in the test set.\n\nI am not challenging the change in rules you set for the TPU Flower competition, but I am challenging if this becomes a general policy where you can change rules in the middle of a competition like this.  \n\nIf pseudo labeling of test data is prohibited then it should be written in competition rules.  To be crystal clear, pseudo labeling is not hand labeling nor scrapping labels from an external source. It is not forbidden by current rules.",
          "votes": 2
        },
        {
          "id": 850413,
          "postDate": "2020-05-16T15:46:37.173Z",
          "content": "<p><a href=\"/juliaelliott\">@juliaelliott</a> This is the first time I ever hear that pseudo labeling of test data is not allowed. This has been part of many, many competition solutions on kaggle. Is this a general new rule? Can you please clarify.</p>",
          "rawMarkdown": "@juliaelliott This is the first time I ever hear that pseudo labeling of test data is not allowed. This has been part of many, many competition solutions on kaggle. Is this a general new rule? Can you please clarify.",
          "votes": 1
        },
        {
          "id": 852699,
          "postDate": "2020-05-18T16:08:01.080Z",
          "content": "<p>Hi all! Thanks for your feedback.</p>\n\n<p>Pseudo-labeling when most of a dataset's labels are public is of course nuanced. The rules mentioned here are not \"general new rules\". They apply to this competition, which has a public dataset that includes labels that are directly in use by the competition.</p>\n\n<p>Pseudo-labeling of test data is allowed. If you somehow have the test labels, and you're using pseudo-labeling to incorporate them into your model as a backdoor to avoid using the test labels directly - it's prohibited by the rules. If we can't tell the difference, we may err on the side of caution and disallow pseudo-labeling on the test data.</p>\n\n<p>The general principle is that, whatever is done must work on an unseen set of data without human intervention.</p>\n\n<p>We recognize the competition’s rules become challenging to interpret when a test set’s labels are public. In the future, we’ll do our best to make any nuanced clarifications upfront.</p>",
          "rawMarkdown": "Hi all! Thanks for your feedback.\n\nPseudo-labeling when most of a dataset's labels are public is of course nuanced. The rules mentioned here are not \"general new rules\". They apply to this competition, which has a public dataset that includes labels that are directly in use by the competition.\n\nPseudo-labeling of test data is allowed. If you somehow have the test labels, and you're using pseudo-labeling to incorporate them into your model as a backdoor to avoid using the test labels directly - it's prohibited by the rules. If we can't tell the difference, we may err on the side of caution and disallow pseudo-labeling on the test data.\n\nThe general principle is that, whatever is done must work on an unseen set of data without human intervention.\n\nWe recognize the competition’s rules become challenging to interpret when a test set’s labels are public. In the future, we’ll do our best to make any nuanced clarifications upfront.",
          "votes": 1
        },
        {
          "id": 852952,
          "postDate": "2020-05-18T20:33:25.193Z",
          "content": "<p>Thanks for the response.  If I summarize: when test labels are available online then using test data when training models is prohibited.</p>",
          "rawMarkdown": "Thanks for the response.  If I summarize: when test labels are available online then using test data when training models is prohibited.\n\n",
          "votes": 1
        }
      ]
    },
    {
      "id": 1249543,
      "postDate": "2021-03-23T11:08:54.743Z",
      "content": "<p>Thank you for sharing this . It's quite helpful. <a href=\"https://www.kaggle.com/cdeotte\" target=\"_blank\">@cdeotte</a> Do you have any other references that can be explored regarding this? Is the use of external sources going to be part of the submission Or just to build confidence?</p>",
      "rawMarkdown": "Thank you for sharing this . It's quite helpful. @cdeotte Do you have any other references that can be explored regarding this? Is the use of external sources going to be part of the submission Or just to build confidence?"
    },
    {
      "id": 839118,
      "postDate": "2020-05-09T05:04:22.347Z",
      "content": "<p><a href=\"/juliaelliott\">@juliaelliott</a> \n        Hello, thank you for giving us an opportunity to learn and use TPU. But by the end of the game, many on the lb list had used five additional public data sets to score high. Although I also feel that the overlap between the open data set and the test set is unreasonable. But I learned the news too late.\n         We can ensure that no additional data sets are used. But what you said, please don't use the image that overlaps with the test set and you can continue to use the original 5 public data sets. These two points make me feel a little ambiguous. I hope you can give us some advice at the last moment, so that we can have time to retrain, and at the same time guarantee the fairness.\n         Thank you very much!</p>\n\n<pre><code>     At the same time, thanks for Kirill and Heng‘ sharing. I have to admit that the original code in the latest data set you provided really cannot get the current score. I enjoyed it, but I wanted it to be fair. Hopefully you can all use the same data set.\n</code></pre>",
      "rawMarkdown": "@juliaelliott \n        Hello, thank you for giving us an opportunity to learn and use TPU. But by the end of the game, many on the lb list had used five additional public data sets to score high. Although I also feel that the overlap between the open data set and the test set is unreasonable. But I learned the news too late.\n         We can ensure that no additional data sets are used. But what you said, please don't use the image that overlaps with the test set and you can continue to use the original 5 public data sets. These two points make me feel a little ambiguous. I hope you can give us some advice at the last moment, so that we can have time to retrain, and at the same time guarantee the fairness.\n         Thank you very much!\n\n         At the same time, thanks for Kirill and Heng‘ sharing. I have to admit that the original code in the latest data set you provided really cannot get the current score. I enjoyed it, but I wanted it to be fair. Hopefully you can all use the same data set.\n\n"
    },
    {
      "id": 839031,
      "postDate": "2020-05-09T02:40:32.103Z",
      "content": "<p>How do we know the real leaderboard now? It is unlikely that we will be able to get better results and beat the current leaderboard ... </p>",
      "rawMarkdown": "How do we know the real leaderboard now? It is unlikely that we will be able to get better results and beat the current leaderboard ... "
    },
    {
      "id": 834942,
      "postDate": "2020-05-05T23:33:21.297Z",
      "content": "<p>Can we get LB 0.97+ with GPU?</p>",
      "rawMarkdown": "Can we get LB 0.97+ with GPU?",
      "replies": [
        {
          "id": 834971,
          "postDate": "2020-05-06T00:17:48.973Z",
          "content": "<p>Certainly. Follow the instructions above to add external data and then train with GPU. </p>\n\n<p>(If you run online with Kaggle's P100 GPU, I suggest using resolution 192x192. If you run offline with a faster GPU then use any resolution).</p>",
          "rawMarkdown": "Certainly. Follow the instructions above to add external data and then train with GPU. \n\n(If you run online with Kaggle's P100 GPU, I suggest using resolution 192x192. If you run offline with a faster GPU then use any resolution)."
        }
      ]
    },
    {
      "id": 832432,
      "postDate": "2020-05-04T06:03:18.887Z",
      "content": "<p>Hi Chris,</p>\n\n<p>\"Next, you need to attach the additional Kaggle datasets to your notebook.\" </p>\n\n<p>How to do this ? do I have to download the dataset first ?</p>\n\n<p>thanks in advance</p>",
      "rawMarkdown": "Hi Chris,\n\n\"Next, you need to attach the additional Kaggle datasets to your notebook.\" \n\nHow to do this ? do I have to download the dataset first ?\n\nthanks in advance",
      "replies": [
        {
          "id": 832445,
          "postDate": "2020-05-04T06:17:52.460Z",
          "content": "<p>ok sorry I figured it out.</p>\n\n<p>but I have more training images than 68094:</p>\n\n<p>Dataset: 70470 training images, 3712 validation images, 7382 unlabeled test images</p>",
          "rawMarkdown": "ok sorry I figured it out.\n\nbut I have more training images than 68094:\n\nDataset: 70470 training images, 3712 validation images, 7382 unlabeled test images\n"
        },
        {
          "id": 832972,
          "postDate": "2020-05-04T14:27:57.043Z",
          "content": "<p>You have included the validation images also. 66758 train + 3712 valid = 70470 images.</p>",
          "rawMarkdown": "You have included the validation images also. 66758 train + 3712 valid = 70470 images.",
          "votes": 1
        },
        {
          "id": 833485,
          "postDate": "2020-05-04T21:04:30.507Z",
          "content": "<p>oups you are right. My Validation boolean was set to false</p>",
          "rawMarkdown": "oups you are right. My Validation boolean was set to false"
        }
      ]
    },
    {
      "id": 833674,
      "postDate": "2020-05-05T01:50:39.707Z",
      "rawMarkdown": "",
      "isDeleted": true,
      "replies": [
        {
          "id": 833688,
          "postDate": "2020-05-05T02:07:23.430Z",
          "content": "<p><code>STEPS_PER_EPOCH = NUM_TRAINING_IMAGES // BATCH_SIZE</code> which is <code>66758 // 128 = 521</code>. </p>\n\n<p>To train faster, you can reduce resolution. Using 224x224 and ensembling, you will have no trouble training within 3 hours and you can beat LB 0.98+. Or if you want higher resolutions, you can also try training smaller models like EfficientNetB6, B5, B4 or DenseNet201. They are all quicker to train than EfficientNetB7.</p>",
          "rawMarkdown": "`STEPS_PER_EPOCH = NUM_TRAINING_IMAGES // BATCH_SIZE` which is `66758 // 128 = 521`. \n\nTo train faster, you can reduce resolution. Using 224x224 and ensembling, you will have no trouble training within 3 hours and you can beat LB 0.98+. Or if you want higher resolutions, you can also try training smaller models like EfficientNetB6, B5, B4 or DenseNet201. They are all quicker to train than EfficientNetB7.",
          "votes": 2
        },
        {
          "id": 836790,
          "postDate": "2020-05-07T08:40:46.153Z",
          "rawMarkdown": "",
          "isDeleted": true
        }
      ]
    }
  ],
  "comments": [
    {
      "id": 834764,
      "author_name": "Martin Görner",
      "author_url": "",
      "post_date": "2020-05-05T18:50:10.113000",
      "content": "<p>This is a playground competition. The purpose is educational. So yes, training on the test data will get you higher scores but it is not very educational. As for the rules, yes, in a playground competition no one will prevent you from doing so, because it is a playground and you do more or less what you want.</p>\n\n<p><a href=\"/cdeotte\">@cdeotte</a> For the purpose of people's education, could you please add a comment to your post that training on the test data artificially inflates your accuracy score but does not result in better real-life performance, i.e. if you were to try this classifier on pictures you took yourself in nature.</p>",
      "votes": 6,
      "replies": [
        {
          "id": 834823,
          "author_name": "Chris Deotte",
          "author_url": "",
          "post_date": "2020-05-05T19:49:01.320000",
          "content": "<p>I agree it may not be educational and i would prefer that Kaggle disallowed it. But currently only participants who train with external data (which includes test data) will win a prize, so it is what it is...</p>\n\n<p>&gt; TPU Leaderboard Prizes: Awarded to the solutions using TPUs with highest leaderboard rankings.\n1st Place: Coral USB edge TPU accelerator + Raspberry Pi 4, 4GB RAM + Rapberry Pi Camera V2\n2nd Place: Coral USB edge TPU accelerator\n3rd Place: Coral USB edge TPU accelerator</p>",
          "votes": 3,
          "replies": []
        },
        {
          "id": 834838,
          "author_name": "Chris Deotte",
          "author_url": "",
          "post_date": "2020-05-05T20:07:55.933000",
          "content": "<p>I added a prominent disclaimer to my post so participants realize what's what.</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 834993,
          "author_name": "Martin Görner",
          "author_url": "",
          "post_date": "2020-05-06T00:58:51.240000",
          "content": "<p>OK, thanks</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 832563,
      "author_name": "astzls",
      "author_url": "",
      "post_date": "2020-05-04T08:36:37.357000",
      "content": "<p>I try using these dataset to train, but get nan? especially using imagenet, openimage and xxxxxlist</p>",
      "votes": 1,
      "replies": [
        {
          "id": 832968,
          "author_name": "Chris Deotte",
          "author_url": "",
          "post_date": "2020-05-04T14:26:58.207000",
          "content": "<p>That's weird. If you only train with the original dataset, do you still get nan? Are you using <code>sparse_categorical_crossentropy</code> loss and leaving your <code>y_train</code> as integers? </p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 833623,
          "author_name": "Alban Gabillon",
          "author_url": "",
          "post_date": "2020-05-05T00:23:15.333000",
          "content": "<p>I noticed the same problem. </p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 833635,
          "author_name": "astzls",
          "author_url": "",
          "post_date": "2020-05-05T00:45:47.113000",
          "content": "<p>Using official dataset and oxford ext dataset, it works as usual with sparse categorical ce, not nan...\nI think these dataset I said above maybe noisy or wrong labels...</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 833873,
          "author_name": "Alban Gabillon",
          "author_url": "",
          "post_date": "2020-05-05T05:48:16.520000",
          "content": "<p>I don't have this problem with the original dataset. \nI copy-pasted your instructions. \nI can display images of the dataset.\nI looked into the whole label set. Everything looks fine:</p>\n\n<p>Class 73 has the highest number of images: 7315\nClass 15 has the lowest number of images: 36</p>\n\n<p>But I get a loss = NaN with both efficientNet and DenseNet when fitting. I am puzzled.</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 833878,
          "author_name": "Alban Gabillon",
          "author_url": "",
          "post_date": "2020-05-05T05:59:12.040000",
          "content": "<p>I am not 100% sure but it seems that some data augmentation operations are the cause of this problem. I disabled data augmentation and it seems working now.</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 835069,
          "author_name": "Alban Gabillon",
          "author_url": "",
          "post_date": "2020-05-06T03:04:47.313000",
          "content": "<p><a href=\"/cdeotte\">@cdeotte</a> \nI am now ranked 24th using the external dataset you suggested. Many Thx. Unfortunately, I almost exhausted my TPU quota and cannot improve </p>",
          "votes": 2,
          "replies": []
        },
        {
          "id": 835084,
          "author_name": "Chris Deotte",
          "author_url": "",
          "post_date": "2020-05-06T03:23:38.183000",
          "content": "<p>Nice job Alban</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 835232,
          "author_name": "astzls",
          "author_url": "",
          "post_date": "2020-05-06T06:09:16.587000",
          "content": "<p>You are right! I disabled augmentation and it works again! But it is still puzzled, why augmenting these dataset can lead to nan?</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 836120,
          "author_name": "Alban Gabillon",
          "author_url": "",
          "post_date": "2020-05-06T18:23:43.280000",
          "content": "<p>I do not know.  I do not understand why it works with the original dataset and not with the external dataset.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 836678,
      "author_name": "haveri",
      "author_url": "",
      "post_date": "2020-05-07T06:33:04.220000",
      "content": "<p>Thanks Chris for pursuing this and the important update above. I will copy them on to the kernel/notebooks I have shared. Already many have forked the code and may not know about the new rule. I was lucky to notice this discussion topic while searching for some thing else. Will need to spread this information and hope all realize it.</p>",
      "votes": 2,
      "replies": []
    },
    {
      "id": 836349,
      "author_name": "Julia Elliott",
      "author_url": "",
      "post_date": "2020-05-06T22:50:48.610000",
      "content": "<p>Hi all, thanks for raising the concern about external datasets that mirror the same images as the test set. I want to reiterate that the rules prohibit training on test set images: “Submissions may not use or incorporate information from hand labeling or human prediction of the validation dataset or test data records.” So, while you are permitted to use the 5 datasets that we used to source the training/test sets as <a href=\"https://www.kaggle.com/c/flower-classification-with-tpus/discussion/134701#769297\">I stated previously here</a>, <strong>it is prohibited to train your models using any mirrors/overlaps/duplication of the competition's test data within other flower datasets, as this conflicts with the above-stated rule.</strong></p>\n\n<p>All potential winners will be required to hand over their full code, including specifying the exact external datasets they've used. We will be cross-checking training images with those contained in the test set to confirm no overlaps as a condition to awarding prizes. If you've already committed this violation, you should ensure you do not select any submissions making use of prohibited data for your final 2 submissions that will be evaluated on the final Private Leaderboard.</p>",
      "votes": 0,
      "replies": [
        {
          "id": 836369,
          "author_name": "Caleb",
          "author_url": "",
          "post_date": "2020-05-06T23:43:24.293000",
          "content": "<p>Hi, I read your previous comment to relate only to hand-labeling the test set or pre-labeled versions of the test set, not external datasets that may include examples included in the test set. </p>\n\n<p>&gt; However, the rules do specify that hand-labeling of the test set and incorporating hand-labeled or pre-labeled versions of the test set in your training is prohibited.</p>\n\n<p>Could you confirm that training on any examples included in the test set, even if they were not hand labeled, will result in disqualification? </p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 836394,
          "author_name": "Julia Elliott",
          "author_url": "",
          "post_date": "2020-05-07T00:16:12.490000",
          "content": "<p><a href=\"/calebeverett\">@calebeverett</a> That's right. Training on any examples included in the test set will result in disqualification. The hand/pre-labeled specification is intended to cover the entirety of the use of such test set images, whether that be someone taking the unlabeled test data and hand-labeling it or using some pre-labeled version of that data. So yes, for complete avoidance of doubt, don't use images that are or are the same as the test set's images.</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 836401,
          "author_name": "Alban Gabillon",
          "author_url": "",
          "post_date": "2020-05-07T00:19:04.887000",
          "content": "<p>omg, did I train on images including the test set?  i.e does the external data set at <a href=\"https://www.kaggle.com/kirillblinov/tf-flower-photo-tfrec\">https://www.kaggle.com/kirillblinov/tf-flower-photo-tfrec</a> include the test set? I even do not know it....</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 836405,
          "author_name": "Chris Deotte",
          "author_url": "",
          "post_date": "2020-05-07T00:23:05.047000",
          "content": "<p>Are we allowed to use pseudo labeled external data? (i.e. download external datasets then use a model trained on competition training data to predict the labels on external images?) </p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 836410,
          "author_name": "Chris Deotte",
          "author_url": "",
          "post_date": "2020-05-07T00:31:01.627000",
          "content": "<p><a href=\"/juliaelliott\">@juliaelliott</a> I would like to point out that the following clause does not prevent the use of external images that overlap the test dataset.\n&gt; However, the rules do specify that hand-labeling of the test set and incorporating hand-labeled or pre-labeled versions of the test set in your training is prohibited.</p>\n\n<p>For example, we can download a folder of external images without labels. Then we use a model trained on competition training data to predict the labels in the folder. Lastly we set all labels in the folder equal to the most common predicted label.</p>\n\n<p>None-the-less, I understand your intention is to disallow the 5 external datasets used to create the test data, so I have updated my discussion post above.</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 836465,
          "author_name": "hengck23",
          "author_url": "",
          "post_date": "2020-05-07T02:28:04.720000",
          "content": "<p>on a related side note, one can find  pretrained model  train on external data with the external labels. For example you can find pretrained models of inaturalist, imagenet-21K or open image JFT-300m. Will that be allowed? one might use external models for:\n- fine tuning \n- using extracted image features from external model</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 836473,
          "author_name": "Chris Deotte",
          "author_url": "",
          "post_date": "2020-05-07T02:32:55.467000",
          "content": "<p>Oh right lol. Kaggle's starter notebook <a href=\"https://www.kaggle.com/mgornergoogle/getting-started-with-100-flowers-on-tpu\">here</a> is illegal. Kaggle uses pretrained weights from <code>ImageNet</code> which includes the test images. Here is their model</p>\n\n<pre><code>    pretrained_model = tf.keras.applications.VGG16(weights='imagenet', include_top=False ,input_shape=[*IMAGE_SIZE, 3])\n</code></pre>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 836474,
          "author_name": "Chris Deotte",
          "author_url": "",
          "post_date": "2020-05-07T02:35:09.293000",
          "content": "<p>Also note that Kaggle's new rule</p>\n\n<blockquote>\n  <p>Training on any examples included in the test set will result in disqualification</p>\n</blockquote>\n\n<p>disallows pseudo labeling of the test data.</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 836606,
          "author_name": "",
          "author_url": "",
          "post_date": "2020-05-07T05:26:07.517000",
          "content": "",
          "votes": 0,
          "replies": []
        },
        {
          "id": 836701,
          "author_name": "hengck23",
          "author_url": "",
          "post_date": "2020-05-07T06:56:36.947000",
          "content": "<p><a href=\"/cdeotte\">@cdeotte</a> \n\"Oh right lol. Kaggle's starter notebook here is illegal. Kaggle uses pretrained weights from ImageNet which includes the test images. Here is their model\"</p>\n\n<p>Note that there are imagenet(21K class) and ILSVRC-imagenet (1000 class). ILSVRC-imagenet-1000 is subset of imagenet-21k. pretarin models like vgg,resnet are based on 1000 class and i think there no overlap with the kaggle test set.</p>\n\n<p>i am actually refer to imagenet 21K, like:\n<a href=\"https://github.com/dmlc/mxnet-model-gallery/blob/master/imagenet-21k-inception.md\">https://github.com/dmlc/mxnet-model-gallery/blob/master/imagenet-21k-inception.md</a>\n<a href=\"https://github.com/pertusa/InceptionBN-21K-for-Caffe\">https://github.com/pertusa/InceptionBN-21K-for-Caffe</a></p>\n\n<p>jft 300-m:\n<a href=\"https://github.com/Tencent/tencent-ml-images\">https://github.com/Tencent/tencent-ml-images</a>\nefficientnet noisy student? : <a href=\"https://paperswithcode.com/sota/image-classification-on-imagenet\">https://paperswithcode.com/sota/image-classification-on-imagenet</a></p>\n\n<p>inaturalist\n<a href=\"https://coral.ai/models/\">https://coral.ai/models/</a></p>\n\n<hr>\n\n<p>efficientnet for noisy student(pretrained with jft-300m could be a big issue here) ?</p>",
          "votes": 2,
          "replies": []
        },
        {
          "id": 836817,
          "author_name": "Roman Weilguny",
          "author_url": "",
          "post_date": "2020-05-07T09:05:23.620000",
          "content": "<p><a href=\"/juliaelliott\">@juliaelliott</a> Kaggle should clarify this situation fast</p>\n\n<p>1) using imagenet, noisy student weights in pretrained models</p>\n\n<p>2) pseudo labeling of the test data</p>\n\n<p>3) using/loading own pre-trained models, like <a href=\"/cdeotte\">@cdeotte</a> mentioned above (did such a model have had to exist before 5.5. in a public dataset???) ( I hope I got the tenses correct :) ) - so I mean, is it allowed to pretrain 3 models and load them for the competition entry and ensemble them? btw no one would be able to check with which data those models are trained.</p>\n\n<p>How can I know if the published external dataset (<a href=\"https://www.kaggle.com/kirillblinov/tf-flower-photo-tfrec\">https://www.kaggle.com/kirillblinov/tf-flower-photo-tfrec</a>) has 1 image in it, that is included in the test dataset?</p>\n\n<p>Can you commit that the test dataset contains only transformed variations of the original images in the  external dataset?</p>\n\n<p>btw tf2.2 is released</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 837427,
          "author_name": "Julia Elliott",
          "author_url": "",
          "post_date": "2020-05-07T19:03:16.677000",
          "content": "<p>All great clarification questions.\n- Pseudolabeling is permitted unless you're pseudolabeling using images from or duplicative of images in the test set.\n- Using publicly-available pre-trained models is allowed, including those with components of the test set underlying the pre-trained models. To be super clear, this means you're not permitted to be \"pre-training\" models using the test set and then calling that \"pre-trained,\" but you can use the host of ImageNet or other canned publicly-available pre-trained models mentioned in preceding posts.\n- The currently published datasets can be used (including the 5 datasets from which the training/test sets were sourced), but images which overlap with or are the same as the test set cannot be used (should be removed). Correct, no new external datasets can be announced.</p>\n\n<p>Lastly, a reiteration that this is a <strong>playground competition</strong>. As with all playgrounds, it was created in the spirit of learning (specifically for the launch of our TPU integration), more than the pursuit of winning. Hence, it is not awarding points/medals, and prizes are non-monetary &amp; centered around TPU use. We launched this with the knowledge there would be vulnerabilities and more grey area than we'd prefer in a featured competition, hence sending it to the playground. I hope sound judgment is used by participants that aligns with this competition's spirit, so that we can continue to offer opportunities like this.</p>",
          "votes": 3,
          "replies": []
        },
        {
          "id": 837629,
          "author_name": "cabuckeye",
          "author_url": "",
          "post_date": "2020-05-07T23:06:44.833000",
          "content": "<p>Thank you for the opportunity to work on the excellent v3 TPUs and on\nthis problem.  I have learnt a lot about TPUs, tensorflow, deep\nnetworks and machine learning in general.</p>\n\n<p>Unfortunately, it is now almost impossible to tell how the public\nleaderboard would look like without the tainted submissions.  It seems\nthat most of top listings in the public leaderboard would have used\nthese external datasets.</p>\n\n<p>Have the rules been modified?  I do not see a rule that \"Training on\nany examples included in the test set will result in disqualification\"\nin the <a href=\"https://www.kaggle.com/c/flower-classification-with-tpus/rules\">Rules</a>.  While this rule has been mentioned in a comment in this\nthread, it seems that it should be included in the Rules page to make it\nofficial.</p>\n\n<p>Is it possible to mark the entries on the leaderboard that are tainted\nor better yet, perhaps create an alternate leaderboard that has\nsubmissions that do not include the disallowed datasets?</p>\n\n<p>Why not disallow all the shared datasets thus far rather than saying\nthat they can be used after removing overlapped images?  IMHO, it may\nnot be trivial to remove the duplicate images and that could lead to\ncontroversies of their own.</p>\n\n<p>To spread more the rules modification (assuming that it is indeed\nmodified), perhaps a prominent note can be added to the Leaderboard\npage with this new information.  Had I not come to this page by\naccident, I may have never known!</p>\n\n<p>Also, I ended up including these datasets after seeing this post only\nlike a couple days ago and after seeing another Kaggle Team member's\nresponse acknowledging the disclaimer put up by this topic's author,\nbut did not say that including the datasets would result in a\ndisqualification.  I wouldn't have had I known.  I feel like I have\nwasted a significant part of my TPU quota (and because of it I have\nalmost run out of quota), not to mention time, etc.</p>\n\n<p>There are two aspects to this playground competition--(1) it is a\nplayground for learning, and (2) it is a competition (even if it is\nfor bragging rights and nothing else).  Also, I do understand the\nspirit of all this--including the test data in the training set\ndefeats the purpose of building a model that can recognize samples in\ngeneral.  But it is a competition nonetheless and I feel that the\nplaying field has been mucked up (I think unintentionally).  It would\nbe great to make it level again for the spirit of the competition.\nFrom a competition perspective, I don't think it is level the way\nthings are right now.</p>\n\n<p>That said, I am extremely extremely grateful for the opportunity to\nuse v3 TPUs with eight compute cores (for free) and the opportunity to\nwork on this problem.  I have learnt a lot about TPUs and tensorflow\nand not to mention a hundred and four types of flowers 😊.  So the\nplayground aspect of it has been more than satisfied.  And if we're\nnot able to solve the competition aspect of it, I guess that's totally\nO.K. as well.</p>\n\n<p>Thank you!</p>",
          "votes": 5,
          "replies": []
        },
        {
          "id": 837989,
          "author_name": "Akash G",
          "author_url": "",
          "post_date": "2020-05-08T07:51:43.253000",
          "content": "<p>Thanks for the clarification. But some things are not completely clear.</p>\n\n<p>First, how do I know that the published external datasets that I might use do not contain any images from the test set? Is there a way to verify it? Was the test set published already that I'm ignorant of?</p>\n\n<p>Second, can I have a clarity on the 'publicly available pre-trained models' that I can use? In the sense, if I make my model (pre-trained on publicly available datasets) public, will I be able to use it without violating the rules? Please comment upon this.</p>\n\n<p>Thanks!</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 838028,
          "author_name": "haveri",
          "author_url": "",
          "post_date": "2020-05-08T08:26:58.080000",
          "content": "<p>Akash G. Published external datasets do contain images from the test set. At least 12 images are an exact match. There are many more that come very close to the test images. <a href=\"https://www.kaggle.com/haveri/how-similar-are-the-test-and-training-set\">.Some experiments done and results published can be found here.</a></p>\n\n<p>A possibly faster way to compare the images exists in <a href=\"https://www.kaggle.com/kirillblinov/prepare-dataset\">code released by Kirill here</a>.</p>\n\n<p>At this time it is best to quit using the external dataset and just use what was provided in the competition.</p>",
          "votes": 2,
          "replies": []
        },
        {
          "id": 849618,
          "author_name": "CPMP",
          "author_url": "",
          "post_date": "2020-05-15T23:35:07.437000",
          "content": "<p><a href=\"/juliaelliott\">@juliaelliott</a> what you state in the above is stronger than the rules, especially:</p>\n\n<blockquote>\n  <p>Pseudolabeling is permitted unless you're pseudolabeling using images from or duplicative of images in the test set.</p>\n</blockquote>\n\n<p>I am not challenging the change in rules you set for the TPU Flower competition, but I am challenging if this becomes a general policy where you can change rules in the middle of a competition like this.  </p>\n\n<p>If pseudo labeling of test data is prohibited then it should be written in competition rules.  To be crystal clear, pseudo labeling is not hand labeling nor scrapping labels from an external source. It is not forbidden by current rules.</p>",
          "votes": 2,
          "replies": []
        },
        {
          "id": 850413,
          "author_name": "Psi",
          "author_url": "",
          "post_date": "2020-05-16T15:46:37.173000",
          "content": "<p><a href=\"/juliaelliott\">@juliaelliott</a> This is the first time I ever hear that pseudo labeling of test data is not allowed. This has been part of many, many competition solutions on kaggle. Is this a general new rule? Can you please clarify.</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 852699,
          "author_name": "Phil Culliton",
          "author_url": "",
          "post_date": "2020-05-18T16:08:01.080000",
          "content": "<p>Hi all! Thanks for your feedback.</p>\n\n<p>Pseudo-labeling when most of a dataset's labels are public is of course nuanced. The rules mentioned here are not \"general new rules\". They apply to this competition, which has a public dataset that includes labels that are directly in use by the competition.</p>\n\n<p>Pseudo-labeling of test data is allowed. If you somehow have the test labels, and you're using pseudo-labeling to incorporate them into your model as a backdoor to avoid using the test labels directly - it's prohibited by the rules. If we can't tell the difference, we may err on the side of caution and disallow pseudo-labeling on the test data.</p>\n\n<p>The general principle is that, whatever is done must work on an unseen set of data without human intervention.</p>\n\n<p>We recognize the competition’s rules become challenging to interpret when a test set’s labels are public. In the future, we’ll do our best to make any nuanced clarifications upfront.</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 852952,
          "author_name": "CPMP",
          "author_url": "",
          "post_date": "2020-05-18T20:33:25.193000",
          "content": "<p>Thanks for the response.  If I summarize: when test labels are available online then using test data when training models is prohibited.</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 1249543,
      "author_name": "Olusesi Adebisi",
      "author_url": "",
      "post_date": "2021-03-23T11:08:54.743000",
      "content": "<p>Thank you for sharing this . It's quite helpful. <a href=\"https://www.kaggle.com/cdeotte\" target=\"_blank\">@cdeotte</a> Do you have any other references that can be explored regarding this? Is the use of external sources going to be part of the submission Or just to build confidence?</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 839118,
      "author_name": "Crown.W",
      "author_url": "",
      "post_date": "2020-05-09T05:04:22.347000",
      "content": "<p><a href=\"/juliaelliott\">@juliaelliott</a> \n        Hello, thank you for giving us an opportunity to learn and use TPU. But by the end of the game, many on the lb list had used five additional public data sets to score high. Although I also feel that the overlap between the open data set and the test set is unreasonable. But I learned the news too late.\n         We can ensure that no additional data sets are used. But what you said, please don't use the image that overlaps with the test set and you can continue to use the original 5 public data sets. These two points make me feel a little ambiguous. I hope you can give us some advice at the last moment, so that we can have time to retrain, and at the same time guarantee the fairness.\n         Thank you very much!</p>\n\n<pre><code>     At the same time, thanks for Kirill and Heng‘ sharing. I have to admit that the original code in the latest data set you provided really cannot get the current score. I enjoyed it, but I wanted it to be fair. Hopefully you can all use the same data set.\n</code></pre>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 839031,
      "author_name": "Fabrizio Milo",
      "author_url": "",
      "post_date": "2020-05-09T02:40:32.103000",
      "content": "<p>How do we know the real leaderboard now? It is unlikely that we will be able to get better results and beat the current leaderboard ... </p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 834942,
      "author_name": "Talha Anwar",
      "author_url": "",
      "post_date": "2020-05-05T23:33:21.297000",
      "content": "<p>Can we get LB 0.97+ with GPU?</p>",
      "votes": 0,
      "replies": [
        {
          "id": 834971,
          "author_name": "Chris Deotte",
          "author_url": "",
          "post_date": "2020-05-06T00:17:48.973000",
          "content": "<p>Certainly. Follow the instructions above to add external data and then train with GPU. </p>\n\n<p>(If you run online with Kaggle's P100 GPU, I suggest using resolution 192x192. If you run offline with a faster GPU then use any resolution).</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 832432,
      "author_name": "Alban Gabillon",
      "author_url": "",
      "post_date": "2020-05-04T06:03:18.887000",
      "content": "<p>Hi Chris,</p>\n\n<p>\"Next, you need to attach the additional Kaggle datasets to your notebook.\" </p>\n\n<p>How to do this ? do I have to download the dataset first ?</p>\n\n<p>thanks in advance</p>",
      "votes": 0,
      "replies": [
        {
          "id": 832445,
          "author_name": "Alban Gabillon",
          "author_url": "",
          "post_date": "2020-05-04T06:17:52.460000",
          "content": "<p>ok sorry I figured it out.</p>\n\n<p>but I have more training images than 68094:</p>\n\n<p>Dataset: 70470 training images, 3712 validation images, 7382 unlabeled test images</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 832972,
          "author_name": "Chris Deotte",
          "author_url": "",
          "post_date": "2020-05-04T14:27:57.043000",
          "content": "<p>You have included the validation images also. 66758 train + 3712 valid = 70470 images.</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 833485,
          "author_name": "Alban Gabillon",
          "author_url": "",
          "post_date": "2020-05-04T21:04:30.507000",
          "content": "<p>oups you are right. My Validation boolean was set to false</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 833674,
      "author_name": "",
      "author_url": "",
      "post_date": "2020-05-05T01:50:39.707000",
      "content": "",
      "votes": 0,
      "replies": [
        {
          "id": 833688,
          "author_name": "Chris Deotte",
          "author_url": "",
          "post_date": "2020-05-05T02:07:23.430000",
          "content": "<p><code>STEPS_PER_EPOCH = NUM_TRAINING_IMAGES // BATCH_SIZE</code> which is <code>66758 // 128 = 521</code>. </p>\n\n<p>To train faster, you can reduce resolution. Using 224x224 and ensembling, you will have no trouble training within 3 hours and you can beat LB 0.98+. Or if you want higher resolutions, you can also try training smaller models like EfficientNetB6, B5, B4 or DenseNet201. They are all quicker to train than EfficientNetB7.</p>",
          "votes": 2,
          "replies": []
        },
        {
          "id": 836790,
          "author_name": "",
          "author_url": "",
          "post_date": "2020-05-07T08:40:46.153000",
          "content": "",
          "votes": 0,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "832197": "# ===========================\n# IMPORTANT UPDATE May 6, 2020:\nKaggle has disallowed the use of images that overlap with test data within the following 5 external datasets:  ImageNet, Oxford 102 Category Flowers, TF Flowers, Open Images, and iNaturalist. Kaggle has added the following new rule:\n\n&gt;Training on any examples included in the test set will result in disqualification.\n\nTherefore the post below is for information purposes only. Do not select a final submission that trains on test images within these 5 datasets.\n# ===========================\n\n# How Is LB 0.970+ Possible?\nYou may have struggled to approach LB 0.970, so you may be wondering how so many people have LB over 0.970. To score over LB 0.970, you need to use external datasets.\n\n# Disclaimer: Using Flower External Data May Not Create Better Model\nNormally more data means better model. However in Flower Comp, the reason that external data is so effective is because it contains many of this competition's answers. That means that although your LB score increases, your model isn't necessarily better at classifying flowers if you used your model offline or at home with your personal photos. (The LB is artificially higher because it has seen many of the answers).\n\n# Is It Allowed?\nIt would be more interesting if external datasets were disallowed, but Kaggle has officially said [here][1] that we are allowed to use these external datasets. Furthermore to finish in the top LB and/or win a prize, you will need to use external datasets because everyone else is. This is sad, but it is what it is.\n\n# How To Include External Datasets\nIf you wish to train your TPU/GPU models with external datasets, below are instructions. In Martin's starter code [here][2], he begins with the line\n\n    GCS_DS_PATH = KaggleDatasets().get_gcs_path()\n\n### Step 1\nWhen using additional datasets, you now need to explicitly declare the directories. So the above line must be changed to \n\n    GCS_DS_PATH = KaggleDatasets().get_gcs_path('flower-classification-with-tpus')\n\nNext, you need to attach the additional Kaggle datasets to your notebook. User Kirill Blinov has created TFRecords [here][4] of Heng CherKeng's scrapped external datasets (explained [here][3]). I have verified (using pseudo label check) that Kirill and Heng have converted all the labels to match this competition's 104 labels. That means, you just need to add these datasets and start training!\n\n### Step 2\nAfter attaching Kirill's datasets [here][4] to your notebook, add this line\n\n    PATH2 = KaggleDatasets().get_gcs_path('tf-flower-photo-tfrec')\n\nAnd finally after creating the variable `TRAINING_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/train/*.tfrec')`, then add the following 5 lines. In place of `224` below, use either 192, 224, 331, or 512 depending on your desired resolution.\n\n    TRAINING_FILENAMES += tf.io.gfile.glob(PATH2 + '/imagenet/tfrecords-jpeg-224x224/*.tfrec')\n    TRAINING_FILENAMES += tf.io.gfile.glob(PATH2 + '/inaturalist_1/tfrecords-jpeg-224x224/*.tfrec')\n    TRAINING_FILENAMES += tf.io.gfile.glob(PATH2 + '/openimage/tfrecords-jpeg-224x224/*.tfrec')\n    TRAINING_FILENAMES += tf.io.gfile.glob(PATH2 + '/oxford_102/tfrecords-jpeg-224x224/*.tfrec')\n    TRAINING_FILENAMES += tf.io.gfile.glob(PATH2 + '/tf_flowers/tfrecords-jpeg-224x224/*.tfrec')\n\n### Step 3\nThat's it. Now your `TRAINING_FILENAMES` contain 66758 training images! And you can continue your notebook the same as before. Training EfficientNetB7 with 224x224 takes 3 minutes per epoch and DenseNet201 takes 2 minutes. If you wish to train longer than 3 hours and/or ensemble models. Then after you train for a while, just \n\n    model.save_weights('my_weights.h5')\n\nAdd those weights to a Kaggle dataset, and start a new notebook, attach the Kaggle dataset, and before training or inferring, run\n\n    model.load_weights('my_weights.h5')\n\n### Step 4\nGood luck. Have fun! \n\nAn example notebook has been published by Haveri [here][5] (version 1 scores LB 0.972).\n\n[1]: https://www.kaggle.com/c/flower-classification-with-tpus/discussion/134701#769297\n[2]: https://www.kaggle.com/mgornergoogle/getting-started-with-100-flowers-on-tpu\n[3]: https://www.kaggle.com/c/flower-classification-with-tpus/discussion/140866\n[4]: https://www.kaggle.com/kirillblinov/tf-flower-photo-tfrec\n[5]: https://www.kaggle.com/haveri/efficientnet-with-all-5-imagesets-s1\n[6]: https://www.kaggle.com/c/flower-classification-with-tpus/discussion/148329#836394",
    "834764": "This is a playground competition. The purpose is educational. So yes, training on the test data will get you higher scores but it is not very educational. As for the rules, yes, in a playground competition no one will prevent you from doing so, because it is a playground and you do more or less what you want.\n\n@cdeotte For the purpose of people's education, could you please add a comment to your post that training on the test data artificially inflates your accuracy score but does not result in better real-life performance, i.e. if you were to try this classifier on pictures you took yourself in nature.",
    "832563": "I try using these dataset to train, but get nan? especially using imagenet, openimage and xxxxxlist",
    "836678": "Thanks Chris for pursuing this and the important update above. I will copy them on to the kernel/notebooks I have shared. Already many have forked the code and may not know about the new rule. I was lucky to notice this discussion topic while searching for some thing else. Will need to spread this information and hope all realize it.",
    "836349": "Hi all, thanks for raising the concern about external datasets that mirror the same images as the test set. I want to reiterate that the rules prohibit training on test set images: “Submissions may not use or incorporate information from hand labeling or human prediction of the validation dataset or test data records.” So, while you are permitted to use the 5 datasets that we used to source the training/test sets as [I stated previously here](https://www.kaggle.com/c/flower-classification-with-tpus/discussion/134701#769297), **it is prohibited to train your models using any mirrors/overlaps/duplication of the competition's test data within other flower datasets, as this conflicts with the above-stated rule.**\n\nAll potential winners will be required to hand over their full code, including specifying the exact external datasets they've used. We will be cross-checking training images with those contained in the test set to confirm no overlaps as a condition to awarding prizes. If you've already committed this violation, you should ensure you do not select any submissions making use of prohibited data for your final 2 submissions that will be evaluated on the final Private Leaderboard.",
    "1249543": "Thank you for sharing this . It's quite helpful. @cdeotte Do you have any other references that can be explored regarding this? Is the use of external sources going to be part of the submission Or just to build confidence?",
    "839118": "@juliaelliott \n        Hello, thank you for giving us an opportunity to learn and use TPU. But by the end of the game, many on the lb list had used five additional public data sets to score high. Although I also feel that the overlap between the open data set and the test set is unreasonable. But I learned the news too late.\n         We can ensure that no additional data sets are used. But what you said, please don't use the image that overlaps with the test set and you can continue to use the original 5 public data sets. These two points make me feel a little ambiguous. I hope you can give us some advice at the last moment, so that we can have time to retrain, and at the same time guarantee the fairness.\n         Thank you very much!\n\n         At the same time, thanks for Kirill and Heng‘ sharing. I have to admit that the original code in the latest data set you provided really cannot get the current score. I enjoyed it, but I wanted it to be fair. Hopefully you can all use the same data set.\n\n",
    "839031": "How do we know the real leaderboard now? It is unlikely that we will be able to get better results and beat the current leaderboard ... ",
    "834942": "Can we get LB 0.97+ with GPU?",
    "832432": "Hi Chris,\n\n\"Next, you need to attach the additional Kaggle datasets to your notebook.\" \n\nHow to do this ? do I have to download the dataset first ?\n\nthanks in advance",
    "833674": ""
  }
}