{
  "id": 312227,
  "title": "Upper Bound on Expected Accuracy",
  "url": "/competitions/ultra-mnist/discussion/312227",
  "author_name": "",
  "post_date": "2022-03-11T01:19:21.634544500Z",
  "votes": 18,
  "comment_count": 10,
  "views": 0,
  "content": "<p>Hi there, I wanted to query/comment on the potential upper limit to accuracy in this competition.</p>\n<p><br></p>\n<hr>\n<p><br></p>\n<p>In the original MNIST competition, the upper limit on accuracy (w/o crazy stuff) seems to be around… </p>\n<ul>\n<li><strong>99.8%</strong></li>\n</ul>\n<p><br></p>\n<p>If we assume an average of 3 digits (I have to confirm this and will update this accordingly) then the accuracy figure degrades exponentially with that term <em>(i.e. any wrong guess in an image with 3 numbers will result in an incorrect prediction… so it is three times 'harder'… this is a bit handwavey as the numerology behind it might indicate something more complex… but whatever)</em>:</p>\n<ul>\n<li><strong>0.998^3 = 0.99401</strong> or …</li>\n<li><strong>99.41%</strong></li>\n</ul>\n<p><br></p>\n<p>If we assume some rate (0.1%) intrinsic dataset error caused through the generation of the dataset (edge cases where it may be impossible for the model to detect the correct digit/sum) as 1 in every 1000 images that yields…</p>\n<ul>\n<li><strong>0.999*(0.998^3)=0.9931</strong> or…</li>\n<li><strong>99.31%</strong></li>\n</ul>\n<p><br></p>\n<p>If we further assume that the resizing of the digits compromises the ability of the original MNIST model by some rate (let's say 1% for horseshoes and hand grenades) then we are left with a final possible accuracy of…</p>\n<ul>\n<li><strong>0.999*(0.988^3)=0.96347</strong> or… </li>\n<li><strong>96.347%</strong></li>\n</ul>\n<p><br></p>\n<p>As you can see, most of the detriment comes from the fact that any loss in the capability of the original model is exponentially exacerbated by the average number of digits found in an image.</p>\n<p><br></p>\n<hr>\n<p><br></p>\n<p>For an average of <strong>2 digits per image/sum</strong></p>\n<ul>\n<li>Assuming single digit accuracy of <strong>98.8</strong> --&gt; <strong>97.517%</strong></li>\n<li>Assuming single digit accuracy of <strong>97.9</strong> --&gt; <strong>95.748%</strong></li>\n<li>Assuming single digit accuracy of <strong>97.0</strong> --&gt; <strong>93.996%</strong></li>\n<li>Assuming single digit accuracy of <strong>96.0</strong> --&gt; <strong>92.068%</strong></li>\n<li>Assuming single digit accuracy of <strong>95.0</strong> --&gt; <strong>90.160%</strong></li>\n</ul>\n<p><br></p>\n<hr>\n<p><br></p>\n<p>For an average of <strong>3 digits per image/sum</strong></p>\n<ul>\n<li>Assuming single digit accuracy of <strong>98.8</strong> --&gt; <strong>96.347%</strong></li>\n<li>Assuming single digit accuracy of <strong>97.9</strong> --&gt; <strong>93.738%</strong></li>\n<li>Assuming single digit accuracy of <strong>97.0</strong> --&gt; <strong>91.176%</strong></li>\n<li>Assuming single digit accuracy of <strong>96.0</strong> --&gt; <strong>88.385%</strong></li>\n<li>Assuming single digit accuracy of <strong>95.0</strong> --&gt; <strong>85.652%</strong></li>\n</ul>\n<p><br></p>\n<hr>\n<p>For an average of <strong>4 digits per image/sum</strong></p>\n<ul>\n<li>Assuming single digit accuracy of <strong>98.8</strong> --&gt; <strong>95.190%</strong></li>\n<li>Assuming single digit accuracy of <strong>97.9</strong> --&gt; <strong>91.769%</strong></li>\n<li>Assuming single digit accuracy of <strong>97.0</strong> --&gt; <strong>88.441%</strong></li>\n<li>Assuming single digit accuracy of <strong>96.0</strong> --&gt; <strong>84.850%</strong></li>\n<li>Assuming single digit accuracy of <strong>95.0</strong> --&gt; <strong>81.370%</strong></li>\n</ul>\n<p><br></p>\n<hr>\n<p><br></p>\n<p>As it stands, I'm not 100% certain about how many digits there are on average per image… I think somewhere around 3. </p>\n<p>Knowing this, and assuming that the model only suffer a little due to the domain shift, <strong>my forecast for the accuracy upper limit in this competition is ~95-96%</strong></p>\n<p><br></p>\n<hr>\n<p><br></p>\n<p>This was just on my mind so I figured I'd write it down and share it. Cheers!</p>",
  "messages": [
    {
      "id": "1718576",
      "postDate": "03/11/2022 01:19:21",
      "content": "<p>Hi there, I wanted to query/comment on the potential upper limit to accuracy in this competition.</p>\n<p><br></p>\n<hr>\n<p><br></p>\n<p>In the original MNIST competition, the upper limit on accuracy (w/o crazy stuff) seems to be around… </p>\n<ul>\n<li><strong>99.8%</strong></li>\n</ul>\n<p><br></p>\n<p>If we assume an average of 3 digits (I have to confirm this and will update this accordingly) then the accuracy figure degrades exponentially with that term <em>(i.e. any wrong guess in an image with 3 numbers will result in an incorrect prediction… so it is three times 'harder'… this is a bit handwavey as the numerology behind it might indicate something more complex… but whatever)</em>:</p>\n<ul>\n<li><strong>0.998^3 = 0.99401</strong> or …</li>\n<li><strong>99.41%</strong></li>\n</ul>\n<p><br></p>\n<p>If we assume some rate (0.1%) intrinsic dataset error caused through the generation of the dataset (edge cases where it may be impossible for the model to detect the correct digit/sum) as 1 in every 1000 images that yields…</p>\n<ul>\n<li><strong>0.999*(0.998^3)=0.9931</strong> or…</li>\n<li><strong>99.31%</strong></li>\n</ul>\n<p><br></p>\n<p>If we further assume that the resizing of the digits compromises the ability of the original MNIST model by some rate (let's say 1% for horseshoes and hand grenades) then we are left with a final possible accuracy of…</p>\n<ul>\n<li><strong>0.999*(0.988^3)=0.96347</strong> or… </li>\n<li><strong>96.347%</strong></li>\n</ul>\n<p><br></p>\n<p>As you can see, most of the detriment comes from the fact that any loss in the capability of the original model is exponentially exacerbated by the average number of digits found in an image.</p>\n<p><br></p>\n<hr>\n<p><br></p>\n<p>For an average of <strong>2 digits per image/sum</strong></p>\n<ul>\n<li>Assuming single digit accuracy of <strong>98.8</strong> --&gt; <strong>97.517%</strong></li>\n<li>Assuming single digit accuracy of <strong>97.9</strong> --&gt; <strong>95.748%</strong></li>\n<li>Assuming single digit accuracy of <strong>97.0</strong> --&gt; <strong>93.996%</strong></li>\n<li>Assuming single digit accuracy of <strong>96.0</strong> --&gt; <strong>92.068%</strong></li>\n<li>Assuming single digit accuracy of <strong>95.0</strong> --&gt; <strong>90.160%</strong></li>\n</ul>\n<p><br></p>\n<hr>\n<p><br></p>\n<p>For an average of <strong>3 digits per image/sum</strong></p>\n<ul>\n<li>Assuming single digit accuracy of <strong>98.8</strong> --&gt; <strong>96.347%</strong></li>\n<li>Assuming single digit accuracy of <strong>97.9</strong> --&gt; <strong>93.738%</strong></li>\n<li>Assuming single digit accuracy of <strong>97.0</strong> --&gt; <strong>91.176%</strong></li>\n<li>Assuming single digit accuracy of <strong>96.0</strong> --&gt; <strong>88.385%</strong></li>\n<li>Assuming single digit accuracy of <strong>95.0</strong> --&gt; <strong>85.652%</strong></li>\n</ul>\n<p><br></p>\n<hr>\n<p>For an average of <strong>4 digits per image/sum</strong></p>\n<ul>\n<li>Assuming single digit accuracy of <strong>98.8</strong> --&gt; <strong>95.190%</strong></li>\n<li>Assuming single digit accuracy of <strong>97.9</strong> --&gt; <strong>91.769%</strong></li>\n<li>Assuming single digit accuracy of <strong>97.0</strong> --&gt; <strong>88.441%</strong></li>\n<li>Assuming single digit accuracy of <strong>96.0</strong> --&gt; <strong>84.850%</strong></li>\n<li>Assuming single digit accuracy of <strong>95.0</strong> --&gt; <strong>81.370%</strong></li>\n</ul>\n<p><br></p>\n<hr>\n<p><br></p>\n<p>As it stands, I'm not 100% certain about how many digits there are on average per image… I think somewhere around 3. </p>\n<p>Knowing this, and assuming that the model only suffer a little due to the domain shift, <strong>my forecast for the accuracy upper limit in this competition is ~95-96%</strong></p>\n<p><br></p>\n<hr>\n<p><br></p>\n<p>This was just on my mind so I figured I'd write it down and share it. Cheers!</p>",
      "rawMarkdown": "Hi there, I wanted to query/comment on the potential upper limit to accuracy in this competition.\n\n<br>\n\n---\n\n<br>\n\nIn the original MNIST competition, the upper limit on accuracy (w/o crazy stuff) seems to be around... \n* **99.8%**\n\n<br>\n\nIf we assume an average of 3 digits (I have to confirm this and will update this accordingly) then the accuracy figure degrades exponentially with that term *(i.e. any wrong guess in an image with 3 numbers will result in an incorrect prediction... so it is three times 'harder'... this is a bit handwavey as the numerology behind it might indicate something more complex... but whatever)*:\n* **0.998^3 = 0.99401** or ...\n* **99.41%**\n\n<br>\n\nIf we assume some rate (0.1%) intrinsic dataset error caused through the generation of the dataset (edge cases where it may be impossible for the model to detect the correct digit/sum) as 1 in every 1000 images that yields...\n* **0.999*(0.998^3)=0.9931** or...\n* **99.31%**\n\n<br>\n\nIf we further assume that the resizing of the digits compromises the ability of the original MNIST model by some rate (let's say 1% for horseshoes and hand grenades) then we are left with a final possible accuracy of...\n* **0.999*(0.988^3)=0.96347** or... \n* **96.347%**\n\n<br>\n\nAs you can see, most of the detriment comes from the fact that any loss in the capability of the original model is exponentially exacerbated by the average number of digits found in an image.\n\n<br>\n\n---\n\n<br>\n\nFor an average of **2 digits per image/sum**\n* Assuming single digit accuracy of **98.8** --> **97.517%**\n* Assuming single digit accuracy of **97.9** --> **95.748%**\n* Assuming single digit accuracy of **97.0** --> **93.996%**\n* Assuming single digit accuracy of **96.0** --> **92.068%**\n* Assuming single digit accuracy of **95.0** --> **90.160%**\n\n<br>\n\n---\n\n<br>\n\nFor an average of **3 digits per image/sum**\n* Assuming single digit accuracy of **98.8** --> **96.347%**\n* Assuming single digit accuracy of **97.9** --> **93.738%**\n* Assuming single digit accuracy of **97.0** --> **91.176%**\n* Assuming single digit accuracy of **96.0** --> **88.385%**\n* Assuming single digit accuracy of **95.0** --> **85.652%**\n\n<br>\n\n---\n\nFor an average of **4 digits per image/sum**\n* Assuming single digit accuracy of **98.8** --> **95.190%**\n* Assuming single digit accuracy of **97.9** --> **91.769%**\n* Assuming single digit accuracy of **97.0** --> **88.441%**\n* Assuming single digit accuracy of **96.0** --> **84.850%**\n* Assuming single digit accuracy of **95.0** --> **81.370%**\n\n<br>\n\n---\n\n<br>\n\nAs it stands, I'm not 100% certain about how many digits there are on average per image... I think somewhere around 3. \n\nKnowing this, and assuming that the model only suffer a little due to the domain shift, **my forecast for the accuracy upper limit in this competition is ~95-96%**\n\n<br>\n\n---\n\n<br>\n\nThis was just on my mind so I figured I'd write it down and share it. Cheers!",
      "votes": null
    },
    {
      "id": "1718591",
      "postDate": "03/11/2022 01:46:54",
      "content": "<p>Great post!</p>\n<p>I did some similar calculations yesterday and they are similar to your numbers.</p>\n<p>According to the data section:</p>\n<blockquote>\n  <p>UltraMNIST dataset comprises very large-scale images, each of 4000x4000 pixels with 3-5 digits per image</p>\n</blockquote>\n<p>I think we can assume that there will be on average 4 digits per image. There is a lot of work to be done in data cleaning and I think we can probably say that there will be a higher error when extracting digits. Models will need to be very robust. </p>",
      "rawMarkdown": "Great post!\n\nI did some similar calculations yesterday and they are similar to your numbers.\n\nAccording to the data section:\n> UltraMNIST dataset comprises very large-scale images, each of 4000x4000 pixels with 3-5 digits per image\n\nI think we can assume that there will be on average 4 digits per image. There is a lot of work to be done in data cleaning and I think we can probably say that there will be a higher error when extracting digits. Models will need to be very robust.",
      "votes": null
    },
    {
      "id": "1718627",
      "postDate": "03/11/2022 02:56:57",
      "content": "<p>Thanks! I hadn’t seen that! </p>\n<p>That makes sense then. That would probably indicate a slightly lower top?</p>\n<p>Maybe 93-95?</p>\n<p>I guess we shall see! 😅</p>",
      "rawMarkdown": "Thanks! I hadn’t seen that! \n\nThat makes sense then. That would probably indicate a slightly lower top?\n\nMaybe 93-95?\n\nI guess we shall see! 😅",
      "votes": null
    },
    {
      "id": "1718632",
      "postDate": "03/11/2022 03:06:06",
      "content": "<p>Yeah, I think around 93~95 is a good upper bound. I think a more realistic accuracy we can expect to see is in the high 80s or very low 90s like 91 or 90%.  </p>",
      "rawMarkdown": "Yeah, I think around 93~95 is a good upper bound. I think a more realistic accuracy we can expect to see is in the high 80s or very low 90s like 91 or 90%.",
      "votes": null
    },
    {
      "id": "1719679",
      "postDate": "03/12/2022 01:39:12",
      "content": "<p>You blew my predictions out of the water. Great job on the LB score!</p>",
      "rawMarkdown": "You blew my predictions out of the water. Great job on the LB score!",
      "votes": null
    },
    {
      "id": "1719697",
      "postDate": "03/12/2022 02:38:55",
      "content": "<p>Thank you for another awesome writeup, Darien! </p>\n<p>I think the real limit might be a bit lower too-incase we have some (intentional/unintentional) label errors. We'll find out soon via the LB however :)</p>",
      "rawMarkdown": "Thank you for another awesome writeup, Darien! \n\nI think the real limit might be a bit lower too-incase we have some (intentional/unintentional) label errors. We'll find out soon via the LB however :)",
      "votes": null
    },
    {
      "id": "1720250",
      "postDate": "03/12/2022 15:52:28",
      "content": "<p>Thanks! I think the rules forbid what I have done (using pre-trained MNIST models) so it's not really that great, but I think my predictions were fairly aligned with reality… so that's cool!</p>\n<p>w.r.t. the new rules/clarifications, I think it will be impossible to empirically determine accuracy as I have done so above.</p>",
      "rawMarkdown": "Thanks! I think the rules forbid what I have done (using pre-trained MNIST models) so it's not really that great, but I think my predictions were fairly aligned with reality... so that's cool!\n\nw.r.t. the new rules/clarifications, I think it will be impossible to empirically determine accuracy as I have done so above.",
      "votes": null
    },
    {
      "id": "1720251",
      "postDate": "03/12/2022 15:52:49",
      "content": "<p>Thanks <a href=\"https://www.kaggle.com/init27\" target=\"_blank\">@init27</a> ! </p>",
      "rawMarkdown": "Thanks @init27 !",
      "votes": null
    },
    {
      "id": "1722188",
      "postDate": "03/14/2022 09:27:34",
      "content": "<p>Great calculations! But I think upper bound about 98%. Resizing doesn't strong affect on result (in my opinion).<br>\nI got about 97.5% accuracy. And I notice that the main problem is extracting numbers.<br>\nBut without original MNIST I think it will be really hard to achieve even 90% accuracy.</p>",
      "rawMarkdown": "Great calculations! But I think upper bound about 98%. Resizing doesn't strong affect on result (in my opinion).\nI got about 97.5% accuracy. And I notice that the main problem is extracting numbers.\nBut without original MNIST I think it will be really hard to achieve even 90% accuracy.",
      "votes": null
    },
    {
      "id": "1722455",
      "postDate": "03/14/2022 14:53:23",
      "content": "<p>Thanks and agreed!</p>",
      "rawMarkdown": "Thanks and agreed!",
      "votes": null
    },
    {
      "id": "1723366",
      "postDate": "03/15/2022 11:15:00",
      "content": "<p>Great post</p>",
      "rawMarkdown": "Great post",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1718591,
      "author_name": "outwrest",
      "author_url": "",
      "post_date": "03/11/2022 01:46:54",
      "content": "<p>Great post!</p>\n<p>I did some similar calculations yesterday and they are similar to your numbers.</p>\n<p>According to the data section:</p>\n<blockquote>\n  <p>UltraMNIST dataset comprises very large-scale images, each of 4000x4000 pixels with 3-5 digits per image</p>\n</blockquote>\n<p>I think we can assume that there will be on average 4 digits per image. There is a lot of work to be done in data cleaning and I think we can probably say that there will be a higher error when extracting digits. Models will need to be very robust. </p>",
      "votes": null,
      "replies": [
        {
          "id": 1718627,
          "author_name": "dschettler8845",
          "author_url": "",
          "post_date": "03/11/2022 02:56:57",
          "content": "<p>Thanks! I hadn’t seen that! </p>\n<p>That makes sense then. That would probably indicate a slightly lower top?</p>\n<p>Maybe 93-95?</p>\n<p>I guess we shall see! 😅</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1718632,
          "author_name": "outwrest",
          "author_url": "",
          "post_date": "03/11/2022 03:06:06",
          "content": "<p>Yeah, I think around 93~95 is a good upper bound. I think a more realistic accuracy we can expect to see is in the high 80s or very low 90s like 91 or 90%.  </p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1719679,
      "author_name": "outwrest",
      "author_url": "",
      "post_date": "03/12/2022 01:39:12",
      "content": "<p>You blew my predictions out of the water. Great job on the LB score!</p>",
      "votes": null,
      "replies": [
        {
          "id": 1720250,
          "author_name": "dschettler8845",
          "author_url": "",
          "post_date": "03/12/2022 15:52:28",
          "content": "<p>Thanks! I think the rules forbid what I have done (using pre-trained MNIST models) so it's not really that great, but I think my predictions were fairly aligned with reality… so that's cool!</p>\n<p>w.r.t. the new rules/clarifications, I think it will be impossible to empirically determine accuracy as I have done so above.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1719697,
      "author_name": "init27",
      "author_url": "",
      "post_date": "03/12/2022 02:38:55",
      "content": "<p>Thank you for another awesome writeup, Darien! </p>\n<p>I think the real limit might be a bit lower too-incase we have some (intentional/unintentional) label errors. We'll find out soon via the LB however :)</p>",
      "votes": null,
      "replies": [
        {
          "id": 1720251,
          "author_name": "dschettler8845",
          "author_url": "",
          "post_date": "03/12/2022 15:52:49",
          "content": "<p>Thanks <a href=\"https://www.kaggle.com/init27\" target=\"_blank\">@init27</a> ! </p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1722188,
      "author_name": "dmitrylessy",
      "author_url": "",
      "post_date": "03/14/2022 09:27:34",
      "content": "<p>Great calculations! But I think upper bound about 98%. Resizing doesn't strong affect on result (in my opinion).<br>\nI got about 97.5% accuracy. And I notice that the main problem is extracting numbers.<br>\nBut without original MNIST I think it will be really hard to achieve even 90% accuracy.</p>",
      "votes": null,
      "replies": [
        {
          "id": 1722455,
          "author_name": "dschettler8845",
          "author_url": "",
          "post_date": "03/14/2022 14:53:23",
          "content": "<p>Thanks and agreed!</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1723366,
      "author_name": "ishanmehta115",
      "author_url": "",
      "post_date": "03/15/2022 11:15:00",
      "content": "<p>Great post</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1718576": "Hi there, I wanted to query/comment on the potential upper limit to accuracy in this competition.\n\n<br>\n\n---\n\n<br>\n\nIn the original MNIST competition, the upper limit on accuracy (w/o crazy stuff) seems to be around... \n* **99.8%**\n\n<br>\n\nIf we assume an average of 3 digits (I have to confirm this and will update this accordingly) then the accuracy figure degrades exponentially with that term *(i.e. any wrong guess in an image with 3 numbers will result in an incorrect prediction... so it is three times 'harder'... this is a bit handwavey as the numerology behind it might indicate something more complex... but whatever)*:\n* **0.998^3 = 0.99401** or ...\n* **99.41%**\n\n<br>\n\nIf we assume some rate (0.1%) intrinsic dataset error caused through the generation of the dataset (edge cases where it may be impossible for the model to detect the correct digit/sum) as 1 in every 1000 images that yields...\n* **0.999*(0.998^3)=0.9931** or...\n* **99.31%**\n\n<br>\n\nIf we further assume that the resizing of the digits compromises the ability of the original MNIST model by some rate (let's say 1% for horseshoes and hand grenades) then we are left with a final possible accuracy of...\n* **0.999*(0.988^3)=0.96347** or... \n* **96.347%**\n\n<br>\n\nAs you can see, most of the detriment comes from the fact that any loss in the capability of the original model is exponentially exacerbated by the average number of digits found in an image.\n\n<br>\n\n---\n\n<br>\n\nFor an average of **2 digits per image/sum**\n* Assuming single digit accuracy of **98.8** --> **97.517%**\n* Assuming single digit accuracy of **97.9** --> **95.748%**\n* Assuming single digit accuracy of **97.0** --> **93.996%**\n* Assuming single digit accuracy of **96.0** --> **92.068%**\n* Assuming single digit accuracy of **95.0** --> **90.160%**\n\n<br>\n\n---\n\n<br>\n\nFor an average of **3 digits per image/sum**\n* Assuming single digit accuracy of **98.8** --> **96.347%**\n* Assuming single digit accuracy of **97.9** --> **93.738%**\n* Assuming single digit accuracy of **97.0** --> **91.176%**\n* Assuming single digit accuracy of **96.0** --> **88.385%**\n* Assuming single digit accuracy of **95.0** --> **85.652%**\n\n<br>\n\n---\n\nFor an average of **4 digits per image/sum**\n* Assuming single digit accuracy of **98.8** --> **95.190%**\n* Assuming single digit accuracy of **97.9** --> **91.769%**\n* Assuming single digit accuracy of **97.0** --> **88.441%**\n* Assuming single digit accuracy of **96.0** --> **84.850%**\n* Assuming single digit accuracy of **95.0** --> **81.370%**\n\n<br>\n\n---\n\n<br>\n\nAs it stands, I'm not 100% certain about how many digits there are on average per image... I think somewhere around 3. \n\nKnowing this, and assuming that the model only suffer a little due to the domain shift, **my forecast for the accuracy upper limit in this competition is ~95-96%**\n\n<br>\n\n---\n\n<br>\n\nThis was just on my mind so I figured I'd write it down and share it. Cheers!",
    "1718591": "Great post!\n\nI did some similar calculations yesterday and they are similar to your numbers.\n\nAccording to the data section:\n> UltraMNIST dataset comprises very large-scale images, each of 4000x4000 pixels with 3-5 digits per image\n\nI think we can assume that there will be on average 4 digits per image. There is a lot of work to be done in data cleaning and I think we can probably say that there will be a higher error when extracting digits. Models will need to be very robust.",
    "1718627": "Thanks! I hadn’t seen that! \n\nThat makes sense then. That would probably indicate a slightly lower top?\n\nMaybe 93-95?\n\nI guess we shall see! 😅",
    "1718632": "Yeah, I think around 93~95 is a good upper bound. I think a more realistic accuracy we can expect to see is in the high 80s or very low 90s like 91 or 90%.",
    "1719679": "You blew my predictions out of the water. Great job on the LB score!",
    "1719697": "Thank you for another awesome writeup, Darien! \n\nI think the real limit might be a bit lower too-incase we have some (intentional/unintentional) label errors. We'll find out soon via the LB however :)",
    "1720250": "Thanks! I think the rules forbid what I have done (using pre-trained MNIST models) so it's not really that great, but I think my predictions were fairly aligned with reality... so that's cool!\n\nw.r.t. the new rules/clarifications, I think it will be impossible to empirically determine accuracy as I have done so above.",
    "1720251": "Thanks @init27 !",
    "1722188": "Great calculations! But I think upper bound about 98%. Resizing doesn't strong affect on result (in my opinion).\nI got about 97.5% accuracy. And I notice that the main problem is extracting numbers.\nBut without original MNIST I think it will be really hard to achieve even 90% accuracy.",
    "1722455": "Thanks and agreed!",
    "1723366": "Great post"
  },
  "source": "meta"
}