{
  "id": 349583,
  "title": "Why does public CV score don't align public leaderboard?",
  "url": "/competitions/open-problems-multimodal/discussion/349583",
  "author_name": "",
  "post_date": "2022-09-01T20:52:29.304786700Z",
  "votes": 9,
  "comment_count": 5,
  "views": 0,
  "content": "<p>If you check public CV score for Multiome the pearson correlation is near 0.65-0.66. <br>\nIf you check public CV score for CITEseq the pearson correlation is near 0.86-0.88.</p>\n<p>We have far more Multiome data in our test predictions regarding the 30% row and 15% column filter (applied on predictions).</p>\n<p>Why public leaderboard scores 0.84+?</p>\n<p>Could it be because public leaderboard uses days 2, 3, and 7 for Multiome meaning that day 4 performance is really bad?</p>\n<p>Any thoughts?</p>",
  "messages": [
    {
      "id": "1922934",
      "postDate": "09/01/2022 20:52:29",
      "content": "<p>If you check public CV score for Multiome the pearson correlation is near 0.65-0.66. <br>\nIf you check public CV score for CITEseq the pearson correlation is near 0.86-0.88.</p>\n<p>We have far more Multiome data in our test predictions regarding the 30% row and 15% column filter (applied on predictions).</p>\n<p>Why public leaderboard scores 0.84+?</p>\n<p>Could it be because public leaderboard uses days 2, 3, and 7 for Multiome meaning that day 4 performance is really bad?</p>\n<p>Any thoughts?</p>",
      "rawMarkdown": "If you check public CV score for Multiome the pearson correlation is near 0.65-0.66. \nIf you check public CV score for CITEseq the pearson correlation is near 0.86-0.88.\n\nWe have far more Multiome data in our test predictions regarding the 30% row and 15% column filter (applied on predictions).\n\nWhy public leaderboard scores 0.84+?\n\nCould it be because public leaderboard uses days 2, 3, and 7 for Multiome meaning that day 4 performance is really bad?\n\nAny thoughts?",
      "votes": null
    },
    {
      "id": "1922936",
      "postDate": "09/01/2022 20:55:28",
      "content": "<p>What do you mean by public CV ? What folds do you use ? Or I am missing something ?</p>",
      "rawMarkdown": "What do you mean by public CV ? What folds do you use ? Or I am missing something ?",
      "votes": null
    },
    {
      "id": "1922957",
      "postDate": "09/01/2022 21:22:47",
      "content": "<p>There are fewer multiome cells in the test set, multiome just has more targets per cell. The metric is the average correlation per cell, so multiome has a smaller weight in the test set.</p>",
      "rawMarkdown": "There are fewer multiome cells in the test set, multiome just has more targets per cell. The metric is the average correlation per cell, so multiome has a smaller weight in the test set.",
      "votes": null
    },
    {
      "id": "1922982",
      "postDate": "09/01/2022 22:03:54",
      "content": "<p>Ahhhhh lol my bad, you are correct. So Cite has 48663 test rows, and multi has 55935 rows (rows = amount of cells). </p>\n<p>Multi just use 30% of the rows, meaning it actually has 16780 cells.</p>\n<p>Therefore the weight for cite will be 0.743 and multi 0.257. It make sense now. Thanks</p>",
      "rawMarkdown": "Ahhhhh lol my bad, you are correct. So Cite has 48663 test rows, and multi has 55935 rows (rows = amount of cells). \n\nMulti just use 30% of the rows, meaning it actually has 16780 cells.\n\nTherefore the weight for cite will be 0.743 and multi 0.257. It make sense now. Thanks",
      "votes": null
    },
    {
      "id": "1923009",
      "postDate": "09/01/2022 23:02:58",
      "content": "<p>If you check a public notebook you will see that the cross validation score is around what I said</p>",
      "rawMarkdown": "If you check a public notebook you will see that the cross validation score is around what I said",
      "votes": null
    },
    {
      "id": "1928766",
      "postDate": "09/06/2022 15:49:49",
      "content": "<p>Hello. I got a stupid question. Why for Cite, rows = amounts of cells? </p>",
      "rawMarkdown": "Hello. I got a stupid question. Why for Cite, rows = amounts of cells?",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1922936,
      "author_name": "alexandervc",
      "author_url": "",
      "post_date": "09/01/2022 20:55:28",
      "content": "<p>What do you mean by public CV ? What folds do you use ? Or I am missing something ?</p>",
      "votes": null,
      "replies": [
        {
          "id": 1923009,
          "author_name": "ragnar123",
          "author_url": "",
          "post_date": "09/01/2022 23:02:58",
          "content": "<p>If you check a public notebook you will see that the cross validation score is around what I said</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1922957,
      "author_name": "jacobyjaeger",
      "author_url": "",
      "post_date": "09/01/2022 21:22:47",
      "content": "<p>There are fewer multiome cells in the test set, multiome just has more targets per cell. The metric is the average correlation per cell, so multiome has a smaller weight in the test set.</p>",
      "votes": null,
      "replies": [
        {
          "id": 1922982,
          "author_name": "ragnar123",
          "author_url": "",
          "post_date": "09/01/2022 22:03:54",
          "content": "<p>Ahhhhh lol my bad, you are correct. So Cite has 48663 test rows, and multi has 55935 rows (rows = amount of cells). </p>\n<p>Multi just use 30% of the rows, meaning it actually has 16780 cells.</p>\n<p>Therefore the weight for cite will be 0.743 and multi 0.257. It make sense now. Thanks</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1928766,
          "author_name": "chouchouchen",
          "author_url": "",
          "post_date": "09/06/2022 15:49:49",
          "content": "<p>Hello. I got a stupid question. Why for Cite, rows = amounts of cells? </p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1922934": "If you check public CV score for Multiome the pearson correlation is near 0.65-0.66. \nIf you check public CV score for CITEseq the pearson correlation is near 0.86-0.88.\n\nWe have far more Multiome data in our test predictions regarding the 30% row and 15% column filter (applied on predictions).\n\nWhy public leaderboard scores 0.84+?\n\nCould it be because public leaderboard uses days 2, 3, and 7 for Multiome meaning that day 4 performance is really bad?\n\nAny thoughts?",
    "1922936": "What do you mean by public CV ? What folds do you use ? Or I am missing something ?",
    "1922957": "There are fewer multiome cells in the test set, multiome just has more targets per cell. The metric is the average correlation per cell, so multiome has a smaller weight in the test set.",
    "1922982": "Ahhhhh lol my bad, you are correct. So Cite has 48663 test rows, and multi has 55935 rows (rows = amount of cells). \n\nMulti just use 30% of the rows, meaning it actually has 16780 cells.\n\nTherefore the weight for cite will be 0.743 and multi 0.257. It make sense now. Thanks",
    "1923009": "If you check a public notebook you will see that the cross validation score is around what I said",
    "1928766": "Hello. I got a stupid question. Why for Cite, rows = amounts of cells?"
  },
  "source": "meta"
}