{
  "id": 451958,
  "title": "CV / LB Thread",
  "url": "/competitions/predict-ai-model-runtime/discussion/451958",
  "author_name": "Nischay Dhankhar",
  "post_date": "2023-10-31T08:25:58.962000",
  "votes": 14,
  "comment_count": 7,
  "views": 0,
  "content": "<p>Couldn't find a common thread in this competition where best validation and leaderboard scores are shared, So starting one with my results. You can volunteer to share your splits/ OPA scores/ Kendall tau scores.</p>\n<p>Validation strategy: same train/val provided by host.</p>\n<h4>Layout single models result [CV is calculated on entire sequences, no sampling]</h4>\n<table>\n<thead>\n<tr>\n<th>Type</th>\n<th>Subtype</th>\n<th>CV(Mean Kendall Tau)</th>\n<th></th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>NLP</td>\n<td>Default</td>\n<td>0.375</td>\n<td></td>\n</tr>\n<tr>\n<td>NLP</td>\n<td>Random</td>\n<td>0.697</td>\n<td></td>\n</tr>\n<tr>\n<td>XLA</td>\n<td>Default</td>\n<td>0.112</td>\n<td></td>\n</tr>\n<tr>\n<td>XLA</td>\n<td>Random</td>\n<td>0.186</td>\n<td></td>\n</tr>\n</tbody>\n</table>\n<h4>Tile data: still using public notebook (lb: 0.18x)</h4>\n<p>With current CV, I achieved 0.498 on leaderboard. </p>",
  "messages": [
    {
      "id": 2512249,
      "postDate": "2023-11-04T09:26:47.990Z",
      "content": "<p>Single model<br>\nCV: 0.740 (average of 5 subtypes, calculated with same train/valid provided by host)<br>\nLB: 0.720</p>\n<p>I don't want to show the CV for each subtype because it contains some clues.</p>",
      "rawMarkdown": "Single model\nCV: 0.740 (average of 5 subtypes, calculated with same train/valid provided by host)\nLB: 0.720\n\nI don't want to show the CV for each subtype because it contains some clues.",
      "votes": 11,
      "replies": [
        {
          "id": 2512532,
          "postDate": "2023-11-04T15:51:50.133Z",
          "content": "<p>Thank you for sharing.<br>\nHow are you calculating the validation score? Are you sampling a couple of times from each file? Or are you sampling a big chunk of configurations? If so, how many configurations are you sampling per file?</p>",
          "rawMarkdown": "Thank you for sharing.\nHow are you calculating the validation score? Are you sampling a couple of times from each file? Or are you sampling a big chunk of configurations? If so, how many configurations are you sampling per file?"
        },
        {
          "id": 2517717,
          "postDate": "2023-11-08T17:37:59.563Z",
          "content": "<p>Wow, your CV LB relation is amazing! And the score, too. </p>",
          "rawMarkdown": "Wow, your CV LB relation is amazing! And the score, too. "
        }
      ]
    },
    {
      "id": 2506341,
      "postDate": "2023-10-31T08:25:58.963Z",
      "content": "<p>Couldn't find a common thread in this competition where best validation and leaderboard scores are shared, So starting one with my results. You can volunteer to share your splits/ OPA scores/ Kendall tau scores.</p>\n<p>Validation strategy: same train/val provided by host.</p>\n<h4>Layout single models result [CV is calculated on entire sequences, no sampling]</h4>\n<table>\n<thead>\n<tr>\n<th>Type</th>\n<th>Subtype</th>\n<th>CV(Mean Kendall Tau)</th>\n<th></th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>NLP</td>\n<td>Default</td>\n<td>0.375</td>\n<td></td>\n</tr>\n<tr>\n<td>NLP</td>\n<td>Random</td>\n<td>0.697</td>\n<td></td>\n</tr>\n<tr>\n<td>XLA</td>\n<td>Default</td>\n<td>0.112</td>\n<td></td>\n</tr>\n<tr>\n<td>XLA</td>\n<td>Random</td>\n<td>0.186</td>\n<td></td>\n</tr>\n</tbody>\n</table>\n<h4>Tile data: still using public notebook (lb: 0.18x)</h4>\n<p>With current CV, I achieved 0.498 on leaderboard. </p>",
      "rawMarkdown": "Couldn't find a common thread in this competition where best validation and leaderboard scores are shared, So starting one with my results. You can volunteer to share your splits/ OPA scores/ Kendall tau scores.\n\nValidation strategy: same train/val provided by host.\n\n\n#### Layout single models result [CV is calculated on entire sequences, no sampling] \n\n| Type | Subtype | CV(Mean Kendall Tau) |   |\n|------|---------|----------------------|---|\n| NLP  | Default | 0.375                |   |\n| NLP  | Random  | 0.697                |   |\n| XLA  | Default | 0.112                |   |\n| XLA  | Random  | 0.186                |   |\n\n#### Tile data: still using public notebook (lb: 0.18x)\n\nWith current CV, I achieved 0.498 on leaderboard. \n\n",
      "votes": 13
    },
    {
      "id": 2518390,
      "postDate": "2023-11-09T08:57:27.470Z",
      "content": "<p>I have added my result in the sample submission CSV, so it also includes default values for other configurations.</p>\n<p>Layout result [Segment and sampling is used to get CV and Inference result] </p>\n<table>\n<thead>\n<tr>\n<th>Type</th>\n<th>CV(Mean Kendall Tau)</th>\n<th>LB</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>XLA</td>\n<td>0.407</td>\n<td>0.272</td>\n</tr>\n<tr>\n<td>NLP</td>\n<td>0.63</td>\n<td>0.353</td>\n</tr>\n</tbody>\n</table>\n<p><strong>Tile result: using public best (LB: 0.194)</strong></p>\n<p>With these, I reached 0.58 on LB.</p>",
      "rawMarkdown": "I have added my result in the sample submission CSV, so it also includes default values for other configurations.\n\nLayout result [Segment and sampling is used to get CV and Inference result] \n|Type|CV(Mean Kendall Tau)|LB|\n| --- | --- |\n| XLA | 0.407 | 0.272 |\n| NLP | 0.63 | 0.353 |\n\n**Tile result: using public best (LB: 0.194)**\n\nWith these, I reached 0.58 on LB.",
      "votes": 1,
      "replies": [
        {
          "id": 2518604,
          "postDate": "2023-11-09T13:03:53.170Z",
          "content": "<p>It seems very interesting that your NLP results are lower than XLA. Mine are similar, but flipped. </p>",
          "rawMarkdown": "It seems very interesting that your NLP results are lower than XLA. Mine are similar, but flipped. ",
          "replies": [
            {
              "id": 2518615,
              "postDate": "2023-11-09T13:06:58.670Z",
              "content": "<p>Sorry about the confusion I wrote that by mistake. Updated it with correct.</p>",
              "rawMarkdown": "Sorry about the confusion I wrote that by mistake. Updated it with correct.\n"
            }
          ]
        }
      ]
    },
    {
      "id": 2506390,
      "postDate": "2023-10-31T09:16:18.913Z",
      "rawMarkdown": "",
      "votes": -1,
      "isDeleted": true
    }
  ],
  "comments": [
    {
      "id": 2512249,
      "author_name": "Shun_PI",
      "author_url": "",
      "post_date": "2023-11-04T09:26:47.990000",
      "content": "<p>Single model<br>\nCV: 0.740 (average of 5 subtypes, calculated with same train/valid provided by host)<br>\nLB: 0.720</p>\n<p>I don't want to show the CV for each subtype because it contains some clues.</p>",
      "votes": 11,
      "replies": [
        {
          "id": 2512532,
          "author_name": "Amit Aharoni",
          "author_url": "",
          "post_date": "2023-11-04T15:51:50.133000",
          "content": "<p>Thank you for sharing.<br>\nHow are you calculating the validation score? Are you sampling a couple of times from each file? Or are you sampling a big chunk of configurations? If so, how many configurations are you sampling per file?</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 2517717,
          "author_name": "Abdur Rahim",
          "author_url": "",
          "post_date": "2023-11-08T17:37:59.563000",
          "content": "<p>Wow, your CV LB relation is amazing! And the score, too. </p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 2518390,
      "author_name": "Yash Goel",
      "author_url": "",
      "post_date": "2023-11-09T08:57:27.470000",
      "content": "<p>I have added my result in the sample submission CSV, so it also includes default values for other configurations.</p>\n<p>Layout result [Segment and sampling is used to get CV and Inference result] </p>\n<table>\n<thead>\n<tr>\n<th>Type</th>\n<th>CV(Mean Kendall Tau)</th>\n<th>LB</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>XLA</td>\n<td>0.407</td>\n<td>0.272</td>\n</tr>\n<tr>\n<td>NLP</td>\n<td>0.63</td>\n<td>0.353</td>\n</tr>\n</tbody>\n</table>\n<p><strong>Tile result: using public best (LB: 0.194)</strong></p>\n<p>With these, I reached 0.58 on LB.</p>",
      "votes": 1,
      "replies": [
        {
          "id": 2518604,
          "author_name": "Rob Freeman",
          "author_url": "",
          "post_date": "2023-11-09T13:03:53.170000",
          "content": "<p>It seems very interesting that your NLP results are lower than XLA. Mine are similar, but flipped. </p>",
          "votes": 0,
          "replies": [
            {
              "id": 2518615,
              "author_name": "Yash Goel",
              "author_url": "",
              "post_date": "2023-11-09T13:06:58.670000",
              "content": "<p>Sorry about the confusion I wrote that by mistake. Updated it with correct.</p>",
              "votes": 0,
              "replies": []
            }
          ]
        }
      ]
    },
    {
      "id": 2506390,
      "author_name": "",
      "author_url": "",
      "post_date": "2023-10-31T09:16:18.913000",
      "content": "",
      "votes": -1,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2512249": "Single model\nCV: 0.740 (average of 5 subtypes, calculated with same train/valid provided by host)\nLB: 0.720\n\nI don't want to show the CV for each subtype because it contains some clues.",
    "2506341": "Couldn't find a common thread in this competition where best validation and leaderboard scores are shared, So starting one with my results. You can volunteer to share your splits/ OPA scores/ Kendall tau scores.\n\nValidation strategy: same train/val provided by host.\n\n\n#### Layout single models result [CV is calculated on entire sequences, no sampling] \n\n| Type | Subtype | CV(Mean Kendall Tau) |   |\n|------|---------|----------------------|---|\n| NLP  | Default | 0.375                |   |\n| NLP  | Random  | 0.697                |   |\n| XLA  | Default | 0.112                |   |\n| XLA  | Random  | 0.186                |   |\n\n#### Tile data: still using public notebook (lb: 0.18x)\n\nWith current CV, I achieved 0.498 on leaderboard. \n\n",
    "2518390": "I have added my result in the sample submission CSV, so it also includes default values for other configurations.\n\nLayout result [Segment and sampling is used to get CV and Inference result] \n|Type|CV(Mean Kendall Tau)|LB|\n| --- | --- |\n| XLA | 0.407 | 0.272 |\n| NLP | 0.63 | 0.353 |\n\n**Tile result: using public best (LB: 0.194)**\n\nWith these, I reached 0.58 on LB.",
    "2506390": ""
  }
}