{
  "id": 214402,
  "title": "5 TTA increases local CV but decreases LB!",
  "url": "/competitions/cassava-leaf-disease-classification/discussion/214402",
  "author_name": "",
  "post_date": "2021-01-26T14:35:00.930861200Z",
  "votes": null,
  "comment_count": 5,
  "views": 0,
  "content": "<p>I have been doing quite a few experiments with TTA and saw that 5 TTA has higher CV in general vs lower TTAs. However, the LB trend is the opposite for most cases. My current best LB is from lower TTA ensembles, however, I am not sure if that is trustable. </p>\n<p>Anybody else seeing the same thing? </p>\n<p>Trusting CV is hard!</p>",
  "messages": [
    {
      "id": "1170931",
      "postDate": "01/26/2021 14:35:00",
      "content": "<p>I have been doing quite a few experiments with TTA and saw that 5 TTA has higher CV in general vs lower TTAs. However, the LB trend is the opposite for most cases. My current best LB is from lower TTA ensembles, however, I am not sure if that is trustable. </p>\n<p>Anybody else seeing the same thing? </p>\n<p>Trusting CV is hard!</p>",
      "rawMarkdown": "I have been doing quite a few experiments with TTA and saw that 5 TTA has higher CV in general vs lower TTAs. However, the LB trend is the opposite for most cases. My current best LB is from lower TTA ensembles, however, I am not sure if that is trustable. \n\nAnybody else seeing the same thing? \n\nTrusting CV is hard!",
      "votes": null
    },
    {
      "id": "1170998",
      "postDate": "01/26/2021 15:17:54",
      "content": "<p>Trust CV might be hard, but could hardly be wrong. </p>\n<p>My best LB submission had a quite low cv too. I have stopped seeking for a higher public LB rank.</p>\n<p>Even if the private LB is a lottery, it would not be a free gift for everyone in the zone. </p>\n<p>The shake-up is always the pain in the ass.</p>",
      "rawMarkdown": "Trust CV might be hard, but could hardly be wrong. \n\nMy best LB submission had a quite low cv too. I have stopped seeking for a higher public LB rank.\n\nEven if the private LB is a lottery, it would not be a free gift for everyone in the zone. \n\nThe shake-up is always the pain in the ass.",
      "votes": null
    },
    {
      "id": "1171027",
      "postDate": "01/26/2021 15:35:44",
      "content": "<p>whats the cv on your best submissions?</p>",
      "rawMarkdown": "whats the cv on your best submissions?",
      "votes": null
    },
    {
      "id": "1171039",
      "postDate": "01/26/2021 15:41:46",
      "content": "<p>Yeah I agree; I guess its a good thing that the trend is atleast consistent. I am gonna trust my CV and see how that goes. </p>",
      "rawMarkdown": "Yeah I agree; I guess its a good thing that the trend is atleast consistent. I am gonna trust my CV and see how that goes.",
      "votes": null
    },
    {
      "id": "1171048",
      "postDate": "01/26/2021 15:49:22",
      "content": "<p>My model also did not increase LB with TTA or emsemble😂</p>",
      "rawMarkdown": "My model also did not increase LB with TTA or emsemble😂",
      "votes": null
    },
    {
      "id": "1171371",
      "postDate": "01/26/2021 19:40:08",
      "content": "<p>Give us a bit more information so we can give you a better answer.</p>\n<p>Size of the difference in CV?<br>\nSize of the difference on LB?<br>\nDoing \"heavy\" augmentation or \"light\"?</p>\n<p>I always like to trust the basic principals of life.  The <strong>central limit theory</strong> is one of those basics for statistics.  I have a local PC running a 16 model TTA right now - would love to use TTA of 50.  Sadly at age of 74 I might die before my local machine could complete the task, so I settled for 3.</p>\n<p>When doing TTA on Kaggle I always select the number based on the kaggle time limit of 9 hours.  So - I do blind trust in the central limit theory.</p>\n<p>I also like to trust my mothers basic rules of life - her <strong>central stupid limit theory</strong> often applies to me.  Doing the wrong thing more times does not improve the answer.  Are you confident you have the code right?</p>\n<p>So is your code giving you a plot of cv vs TTA number that makes sense (more than 3 and 5 needed).  My plot when I did a 1 thru 30 experiment indicated lots of variability in the cv until around 10.  (0.874 to 0.884 variation)  </p>\n<p>The TTA number until reasonably stable levels of CV will depend on your model and your augmentation.</p>",
      "rawMarkdown": "Give us a bit more information so we can give you a better answer.\n\nSize of the difference in CV?\nSize of the difference on LB?\nDoing \"heavy\" augmentation or \"light\"?\n\nI always like to trust the basic principals of life.  The **central limit theory** is one of those basics for statistics.  I have a local PC running a 16 model TTA right now - would love to use TTA of 50.  Sadly at age of 74 I might die before my local machine could complete the task, so I settled for 3.\n\nWhen doing TTA on Kaggle I always select the number based on the kaggle time limit of 9 hours.  So - I do blind trust in the central limit theory.\n\nI also like to trust my mothers basic rules of life - her **central stupid limit theory** often applies to me.  Doing the wrong thing more times does not improve the answer.  Are you confident you have the code right?\n\nSo is your code giving you a plot of cv vs TTA number that makes sense (more than 3 and 5 needed).  My plot when I did a 1 thru 30 experiment indicated lots of variability in the cv until around 10.  (0.874 to 0.884 variation)  \n\nThe TTA number until reasonably stable levels of CV will depend on your model and your augmentation.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1170998,
      "author_name": "zlanan",
      "author_url": "",
      "post_date": "01/26/2021 15:17:54",
      "content": "<p>Trust CV might be hard, but could hardly be wrong. </p>\n<p>My best LB submission had a quite low cv too. I have stopped seeking for a higher public LB rank.</p>\n<p>Even if the private LB is a lottery, it would not be a free gift for everyone in the zone. </p>\n<p>The shake-up is always the pain in the ass.</p>",
      "votes": null,
      "replies": [
        {
          "id": 1171027,
          "author_name": "proletheus",
          "author_url": "",
          "post_date": "01/26/2021 15:35:44",
          "content": "<p>whats the cv on your best submissions?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1171039,
          "author_name": "pheadrus",
          "author_url": "",
          "post_date": "01/26/2021 15:41:46",
          "content": "<p>Yeah I agree; I guess its a good thing that the trend is atleast consistent. I am gonna trust my CV and see how that goes. </p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1171048,
      "author_name": "kani23",
      "author_url": "",
      "post_date": "01/26/2021 15:49:22",
      "content": "<p>My model also did not increase LB with TTA or emsemble😂</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1171371,
      "author_name": "pcjimmmy",
      "author_url": "",
      "post_date": "01/26/2021 19:40:08",
      "content": "<p>Give us a bit more information so we can give you a better answer.</p>\n<p>Size of the difference in CV?<br>\nSize of the difference on LB?<br>\nDoing \"heavy\" augmentation or \"light\"?</p>\n<p>I always like to trust the basic principals of life.  The <strong>central limit theory</strong> is one of those basics for statistics.  I have a local PC running a 16 model TTA right now - would love to use TTA of 50.  Sadly at age of 74 I might die before my local machine could complete the task, so I settled for 3.</p>\n<p>When doing TTA on Kaggle I always select the number based on the kaggle time limit of 9 hours.  So - I do blind trust in the central limit theory.</p>\n<p>I also like to trust my mothers basic rules of life - her <strong>central stupid limit theory</strong> often applies to me.  Doing the wrong thing more times does not improve the answer.  Are you confident you have the code right?</p>\n<p>So is your code giving you a plot of cv vs TTA number that makes sense (more than 3 and 5 needed).  My plot when I did a 1 thru 30 experiment indicated lots of variability in the cv until around 10.  (0.874 to 0.884 variation)  </p>\n<p>The TTA number until reasonably stable levels of CV will depend on your model and your augmentation.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1170931": "I have been doing quite a few experiments with TTA and saw that 5 TTA has higher CV in general vs lower TTAs. However, the LB trend is the opposite for most cases. My current best LB is from lower TTA ensembles, however, I am not sure if that is trustable. \n\nAnybody else seeing the same thing? \n\nTrusting CV is hard!",
    "1170998": "Trust CV might be hard, but could hardly be wrong. \n\nMy best LB submission had a quite low cv too. I have stopped seeking for a higher public LB rank.\n\nEven if the private LB is a lottery, it would not be a free gift for everyone in the zone. \n\nThe shake-up is always the pain in the ass.",
    "1171027": "whats the cv on your best submissions?",
    "1171039": "Yeah I agree; I guess its a good thing that the trend is atleast consistent. I am gonna trust my CV and see how that goes.",
    "1171048": "My model also did not increase LB with TTA or emsemble😂",
    "1171371": "Give us a bit more information so we can give you a better answer.\n\nSize of the difference in CV?\nSize of the difference on LB?\nDoing \"heavy\" augmentation or \"light\"?\n\nI always like to trust the basic principals of life.  The **central limit theory** is one of those basics for statistics.  I have a local PC running a 16 model TTA right now - would love to use TTA of 50.  Sadly at age of 74 I might die before my local machine could complete the task, so I settled for 3.\n\nWhen doing TTA on Kaggle I always select the number based on the kaggle time limit of 9 hours.  So - I do blind trust in the central limit theory.\n\nI also like to trust my mothers basic rules of life - her **central stupid limit theory** often applies to me.  Doing the wrong thing more times does not improve the answer.  Are you confident you have the code right?\n\nSo is your code giving you a plot of cv vs TTA number that makes sense (more than 3 and 5 needed).  My plot when I did a 1 thru 30 experiment indicated lots of variability in the cv until around 10.  (0.874 to 0.884 variation)  \n\nThe TTA number until reasonably stable levels of CV will depend on your model and your augmentation."
  },
  "source": "meta"
}