{
  "id": 266237,
  "title": "Is the public score almost random ",
  "url": "/competitions/rsna-miccai-brain-tumor-radiogenomic-classification/discussion/266237",
  "author_name": "",
  "post_date": "2021-08-18T11:19:47.424823900Z",
  "votes": 3,
  "comment_count": 4,
  "views": 0,
  "content": "<p>Recently i was training some 3d cnns for classification they were fast and computationally efficient . So i created bunch of them on different mri types . I also did the same to my 2d CNNS . Then i ensembled all of them together , here are the submission records :-</p>\n<p>However i also did one submission which included only one 3d cnn but all the 2d cnns ensembled :-<br>\nThe 2d cnns and flair 3d cnn Score :- 0.636</p>\n<p>But then i used all the 3d cnns , however i did a minor mistake in my notebook i gave flair image type to all the 3d cnns ever to the ones who were trained on different types . Score :- 0.662</p>\n<p>I fixed everything and submitted again . Score :- 0.623</p>\n<p>is anyone facing or has solved this problem</p>",
  "messages": [
    {
      "id": "1479259",
      "postDate": "08/18/2021 11:19:47",
      "content": "<p>Recently i was training some 3d cnns for classification they were fast and computationally efficient . So i created bunch of them on different mri types . I also did the same to my 2d CNNS . Then i ensembled all of them together , here are the submission records :-</p>\n<p>However i also did one submission which included only one 3d cnn but all the 2d cnns ensembled :-<br>\nThe 2d cnns and flair 3d cnn Score :- 0.636</p>\n<p>But then i used all the 3d cnns , however i did a minor mistake in my notebook i gave flair image type to all the 3d cnns ever to the ones who were trained on different types . Score :- 0.662</p>\n<p>I fixed everything and submitted again . Score :- 0.623</p>\n<p>is anyone facing or has solved this problem</p>",
      "rawMarkdown": "Recently i was training some 3d cnns for classification they were fast and computationally efficient . So i created bunch of them on different mri types . I also did the same to my 2d CNNS . Then i ensembled all of them together , here are the submission records :-\n\nHowever i also did one submission which included only one 3d cnn but all the 2d cnns ensembled :-\nThe 2d cnns and flair 3d cnn Score :- 0.636\n\nBut then i used all the 3d cnns , however i did a minor mistake in my notebook i gave flair image type to all the 3d cnns ever to the ones who were trained on different types . Score :- 0.662\n\nI fixed everything and submitted again . Score :- 0.623\n\nis anyone facing or has solved this problem",
      "votes": null
    },
    {
      "id": "1483214",
      "postDate": "08/20/2021 14:05:14",
      "content": "<p>I guess all the submissions are evaluated against the sample submission files where all the values are 0.5 :-)</p>",
      "rawMarkdown": "I guess all the submissions are evaluated against the sample submission files where all the values are 0.5 :-)",
      "votes": null
    },
    {
      "id": "1483233",
      "postDate": "08/20/2021 14:19:01",
      "content": "<p>but how can you measure area under roc curve against sample submission . And if it was then sample submission file would have been rank 1 ?</p>",
      "rawMarkdown": "but how can you measure area under roc curve against sample submission . And if it was then sample submission file would have been rank 1 ?",
      "votes": null
    },
    {
      "id": "1485077",
      "postDate": "08/21/2021 18:54:47",
      "content": "<p>The only thing that makes sense is that the public LB is based on the actual AUC scores of your predictions on the 87 \"validation\" records vs the hidden ground truth for those records. The public LB scores are not random but, due to the very small number of records, are quite statistically noisy. I have made only 1 submission so far, and it scored 0.581, which is not very good but roughly matched the CV it was based on.</p>",
      "rawMarkdown": "The only thing that makes sense is that the public LB is based on the actual AUC scores of your predictions on the 87 \"validation\" records vs the hidden ground truth for those records. The public LB scores are not random but, due to the very small number of records, are quite statistically noisy. I have made only 1 submission so far, and it scored 0.581, which is not very good but roughly matched the CV it was based on.",
      "votes": null
    },
    {
      "id": "1485411",
      "postDate": "08/22/2021 04:35:23",
      "content": "<p>Yes i think you are right i think i will select the submissions based on my cv </p>",
      "rawMarkdown": "Yes i think you are right i think i will select the submissions based on my cv",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1483214,
      "author_name": "jhasanov",
      "author_url": "",
      "post_date": "08/20/2021 14:05:14",
      "content": "<p>I guess all the submissions are evaluated against the sample submission files where all the values are 0.5 :-)</p>",
      "votes": null,
      "replies": [
        {
          "id": 1483233,
          "author_name": "swaralipibose",
          "author_url": "",
          "post_date": "08/20/2021 14:19:01",
          "content": "<p>but how can you measure area under roc curve against sample submission . And if it was then sample submission file would have been rank 1 ?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1485077,
          "author_name": "dslate",
          "author_url": "",
          "post_date": "08/21/2021 18:54:47",
          "content": "<p>The only thing that makes sense is that the public LB is based on the actual AUC scores of your predictions on the 87 \"validation\" records vs the hidden ground truth for those records. The public LB scores are not random but, due to the very small number of records, are quite statistically noisy. I have made only 1 submission so far, and it scored 0.581, which is not very good but roughly matched the CV it was based on.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1485411,
          "author_name": "swaralipibose",
          "author_url": "",
          "post_date": "08/22/2021 04:35:23",
          "content": "<p>Yes i think you are right i think i will select the submissions based on my cv </p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1479259": "Recently i was training some 3d cnns for classification they were fast and computationally efficient . So i created bunch of them on different mri types . I also did the same to my 2d CNNS . Then i ensembled all of them together , here are the submission records :-\n\nHowever i also did one submission which included only one 3d cnn but all the 2d cnns ensembled :-\nThe 2d cnns and flair 3d cnn Score :- 0.636\n\nBut then i used all the 3d cnns , however i did a minor mistake in my notebook i gave flair image type to all the 3d cnns ever to the ones who were trained on different types . Score :- 0.662\n\nI fixed everything and submitted again . Score :- 0.623\n\nis anyone facing or has solved this problem",
    "1483214": "I guess all the submissions are evaluated against the sample submission files where all the values are 0.5 :-)",
    "1483233": "but how can you measure area under roc curve against sample submission . And if it was then sample submission file would have been rank 1 ?",
    "1485077": "The only thing that makes sense is that the public LB is based on the actual AUC scores of your predictions on the 87 \"validation\" records vs the hidden ground truth for those records. The public LB scores are not random but, due to the very small number of records, are quite statistically noisy. I have made only 1 submission so far, and it scored 0.581, which is not very good but roughly matched the CV it was based on.",
    "1485411": "Yes i think you are right i think i will select the submissions based on my cv"
  },
  "source": "meta"
}