{
  "id": 353713,
  "title": "Mental note to myself: keep in mind that this is a 2 in 1 competition",
  "url": "/competitions/open-problems-multimodal/discussion/353713",
  "author_name": "",
  "post_date": "2022-09-19T17:11:15.790679700Z",
  "votes": 12,
  "comment_count": 1,
  "views": 0,
  "content": "<p>Mental note to myself: keep in mind that this is a 2 in 1 competition. I mean, we are looking for two models (CITE model and MULTIOME model) and these models must be evaluated in isolation.</p>\n<h2>How?</h2>\n<p>According the competition evaluation:</p>\n<p>$$LB = \\frac{\\sum \\rho}{T}$$</p>\n<p>$$LB = \\frac{\\sum \\rho_C + \\sum \\rho_M}{T}$$</p>\n<p>$$LB = \\frac{C}{T} LB_C+ \\frac{M}{T} LB_M$$</p>\n<ul>\n<li>T = Total cells</li>\n<li>C = Total CITE cells</li>\n<li>M = Total MULTIOME cells</li>\n</ul>\n<p>As a result of this we can evaluate a MULTIOME model setting to 0 the CITE model predictions because then the CITE model LB must be -1:</p>\n<p>$$LB_M = \\frac{LB + C/T}{M/T}$$</p>\n<h2>Practical example</h2>\n<p>See <a href=\"https://www.kaggle.com/code/josecarmona/msci22-keep-in-mind-2in1\" target=\"_blank\">this notebook</a>.</p>\n<p>The \"<a href=\"https://www.kaggle.com/code/swimmy/blending-of-five-submission\" target=\"_blank\">Blending of five submission by </a><a href=\"https://www.kaggle.com/swimmy\" target=\"_blank\">@swimmy</a>\" notebook submission score 0.810 (top 90 in LB today). On the other hand, \"<a href=\"https://www.kaggle.com/code/vslaykovsky/lb-0-811-normalized-ensembles-for-pearson-s-r/notebook?scriptVersionId=105258966\" target=\"_blank\">Normalized Ensembles for Pearson's r by </a><a href=\"https://www.kaggle.com/vslaykovsky\" target=\"_blank\">@vslaykovsky</a>\" score 0.811 (top 58 in LB today). Then the second ensemble model is better than the first blending model. </p>\n<p>But, evaluating the MULTIOME model in isolation, the first model score LB=-0.438 and the second model LB=-0.439: the firt one is better. If we mix the two submission selecting the first MULTIOME model and the second CITE model then <a href=\"https://www.kaggle.com/code/josecarmona/msci22-keep-in-mind-2in1\" target=\"_blank\">score 0.811 and top 48 in LB</a>.</p>",
  "messages": [
    {
      "id": "1946230",
      "postDate": "09/19/2022 17:11:15",
      "content": "<p>Mental note to myself: keep in mind that this is a 2 in 1 competition. I mean, we are looking for two models (CITE model and MULTIOME model) and these models must be evaluated in isolation.</p>\n<h2>How?</h2>\n<p>According the competition evaluation:</p>\n<p>$$LB = \\frac{\\sum \\rho}{T}$$</p>\n<p>$$LB = \\frac{\\sum \\rho_C + \\sum \\rho_M}{T}$$</p>\n<p>$$LB = \\frac{C}{T} LB_C+ \\frac{M}{T} LB_M$$</p>\n<ul>\n<li>T = Total cells</li>\n<li>C = Total CITE cells</li>\n<li>M = Total MULTIOME cells</li>\n</ul>\n<p>As a result of this we can evaluate a MULTIOME model setting to 0 the CITE model predictions because then the CITE model LB must be -1:</p>\n<p>$$LB_M = \\frac{LB + C/T}{M/T}$$</p>\n<h2>Practical example</h2>\n<p>See <a href=\"https://www.kaggle.com/code/josecarmona/msci22-keep-in-mind-2in1\" target=\"_blank\">this notebook</a>.</p>\n<p>The \"<a href=\"https://www.kaggle.com/code/swimmy/blending-of-five-submission\" target=\"_blank\">Blending of five submission by </a><a href=\"https://www.kaggle.com/swimmy\" target=\"_blank\">@swimmy</a>\" notebook submission score 0.810 (top 90 in LB today). On the other hand, \"<a href=\"https://www.kaggle.com/code/vslaykovsky/lb-0-811-normalized-ensembles-for-pearson-s-r/notebook?scriptVersionId=105258966\" target=\"_blank\">Normalized Ensembles for Pearson's r by </a><a href=\"https://www.kaggle.com/vslaykovsky\" target=\"_blank\">@vslaykovsky</a>\" score 0.811 (top 58 in LB today). Then the second ensemble model is better than the first blending model. </p>\n<p>But, evaluating the MULTIOME model in isolation, the first model score LB=-0.438 and the second model LB=-0.439: the firt one is better. If we mix the two submission selecting the first MULTIOME model and the second CITE model then <a href=\"https://www.kaggle.com/code/josecarmona/msci22-keep-in-mind-2in1\" target=\"_blank\">score 0.811 and top 48 in LB</a>.</p>",
      "rawMarkdown": "Mental note to myself: keep in mind that this is a 2 in 1 competition. I mean, we are looking for two models (CITE model and MULTIOME model) and these models must be evaluated in isolation.\n\n## How?\n\nAccording the competition evaluation:\n\n\n$$LB = \\frac{\\sum \\rho}{T}$$\n\n\n$$LB = \\frac{\\sum \\rho_C + \\sum \\rho_M}{T}$$\n\n$$LB = \\frac{C}{T} LB_C+ \\frac{M}{T} LB_M$$\n\n* T = Total cells\n* C = Total CITE cells\n* M = Total MULTIOME cells\n\nAs a result of this we can evaluate a MULTIOME model setting to 0 the CITE model predictions because then the CITE model LB must be -1:\n\n$$LB_M = \\frac{LB + C/T}{M/T}$$\n\n\n\n\n## Practical example\n\nSee [this notebook](https://www.kaggle.com/code/josecarmona/msci22-keep-in-mind-2in1).\n\nThe \"[Blending of five submission by @swimmy](https://www.kaggle.com/code/swimmy/blending-of-five-submission)\" notebook submission score 0.810 (top 90 in LB today). On the other hand, \"[Normalized Ensembles for Pearson's r by @vslaykovsky](https://www.kaggle.com/code/vslaykovsky/lb-0-811-normalized-ensembles-for-pearson-s-r/notebook?scriptVersionId=105258966)\" score 0.811 (top 58 in LB today). Then the second ensemble model is better than the first blending model. \n\nBut, evaluating the MULTIOME model in isolation, the first model score LB=-0.438 and the second model LB=-0.439: the firt one is better. If we mix the two submission selecting the first MULTIOME model and the second CITE model then [score 0.811 and top 48 in LB](https://www.kaggle.com/code/josecarmona/msci22-keep-in-mind-2in1).",
      "votes": null
    },
    {
      "id": "1952862",
      "postDate": "09/24/2022 02:51:52",
      "content": "<p>Your comments were very helpful. I got the -.438 score when I looked at the MULTIOME model in isolation and was baffled by it until I realized that I needed mix it with CITE data to get valid results.<br>\nThanks, Kickback</p>",
      "rawMarkdown": "Your comments were very helpful. I got the -.438 score when I looked at the MULTIOME model in isolation and was baffled by it until I realized that I needed mix it with CITE data to get valid results.\nThanks, Kickback",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1952862,
      "author_name": "bkosar1640",
      "author_url": "",
      "post_date": "09/24/2022 02:51:52",
      "content": "<p>Your comments were very helpful. I got the -.438 score when I looked at the MULTIOME model in isolation and was baffled by it until I realized that I needed mix it with CITE data to get valid results.<br>\nThanks, Kickback</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1946230": "Mental note to myself: keep in mind that this is a 2 in 1 competition. I mean, we are looking for two models (CITE model and MULTIOME model) and these models must be evaluated in isolation.\n\n## How?\n\nAccording the competition evaluation:\n\n\n$$LB = \\frac{\\sum \\rho}{T}$$\n\n\n$$LB = \\frac{\\sum \\rho_C + \\sum \\rho_M}{T}$$\n\n$$LB = \\frac{C}{T} LB_C+ \\frac{M}{T} LB_M$$\n\n* T = Total cells\n* C = Total CITE cells\n* M = Total MULTIOME cells\n\nAs a result of this we can evaluate a MULTIOME model setting to 0 the CITE model predictions because then the CITE model LB must be -1:\n\n$$LB_M = \\frac{LB + C/T}{M/T}$$\n\n\n\n\n## Practical example\n\nSee [this notebook](https://www.kaggle.com/code/josecarmona/msci22-keep-in-mind-2in1).\n\nThe \"[Blending of five submission by @swimmy](https://www.kaggle.com/code/swimmy/blending-of-five-submission)\" notebook submission score 0.810 (top 90 in LB today). On the other hand, \"[Normalized Ensembles for Pearson's r by @vslaykovsky](https://www.kaggle.com/code/vslaykovsky/lb-0-811-normalized-ensembles-for-pearson-s-r/notebook?scriptVersionId=105258966)\" score 0.811 (top 58 in LB today). Then the second ensemble model is better than the first blending model. \n\nBut, evaluating the MULTIOME model in isolation, the first model score LB=-0.438 and the second model LB=-0.439: the firt one is better. If we mix the two submission selecting the first MULTIOME model and the second CITE model then [score 0.811 and top 48 in LB](https://www.kaggle.com/code/josecarmona/msci22-keep-in-mind-2in1).",
    "1952862": "Your comments were very helpful. I got the -.438 score when I looked at the MULTIOME model in isolation and was baffled by it until I realized that I needed mix it with CITE data to get valid results.\nThanks, Kickback"
  },
  "source": "meta"
}