{
  "id": 444722,
  "title": "Any success with Ensembles ?",
  "url": "/competitions/bengaliai-speech/discussion/444722",
  "author_name": "",
  "post_date": "2023-10-03T09:10:05.500805600Z",
  "votes": null,
  "comment_count": 13,
  "views": 0,
  "content": "<p>When I ensemble my best two performing models (same design and trained different fold), my performance deteriorates. Did anyone succeed in setting up Ensemble ?</p>",
  "messages": [
    {
      "id": "2465762",
      "postDate": "10/03/2023 09:10:05",
      "content": "<p>When I ensemble my best two performing models (same design and trained different fold), my performance deteriorates. Did anyone succeed in setting up Ensemble ?</p>",
      "rawMarkdown": "When I ensemble my best two performing models (same design and trained different fold), my performance deteriorates. Did anyone succeed in setting up Ensemble ?",
      "votes": null
    },
    {
      "id": "2465799",
      "postDate": "10/03/2023 09:50:32",
      "content": "<p>You can't ensemble CTC models because the predictions are not guarenteed to be aligned.</p>",
      "rawMarkdown": "You can't ensemble CTC models because the predictions are not guarenteed to be aligned.",
      "votes": null
    },
    {
      "id": "2465803",
      "postDate": "10/03/2023 10:00:06",
      "content": "<p>It may not make sense, but I get 0.02 boost on LB by blindly weight averaging the logits </p>",
      "rawMarkdown": "It may not make sense, but I get 0.02 boost on LB by blindly weight averaging the logits",
      "votes": null
    },
    {
      "id": "2466135",
      "postDate": "10/03/2023 15:23:17",
      "content": "<p>maybe we can try dynamic time warping to align them ? </p>",
      "rawMarkdown": "maybe we can try dynamic time warping to align them ?",
      "votes": null
    },
    {
      "id": "2466578",
      "postDate": "10/04/2023 00:34:51",
      "content": "<p>Do two models have the same vocab.json? Welll, even if the corresponding vocab doesn't align, you still can manually adjust it to fit them together right…</p>",
      "rawMarkdown": "Do two models have the same vocab.json? Welll, even if the corresponding vocab doesn't align, you still can manually adjust it to fit them together right...",
      "votes": null
    },
    {
      "id": "2466613",
      "postDate": "10/04/2023 01:35:23",
      "content": "<p>Yea, for ease of averaging the two has exactly same vocab.</p>",
      "rawMarkdown": "Yea, for ease of averaging the two has exactly same vocab.",
      "votes": null
    },
    {
      "id": "2466995",
      "postDate": "10/04/2023 08:13:52",
      "content": "<p>Are the two models you ensemble from different seed training or from different epochs of the same training?</p>",
      "rawMarkdown": "Are the two models you ensemble from different seed training or from different epochs of the same training?",
      "votes": null
    },
    {
      "id": "2467004",
      "postDate": "10/04/2023 08:25:26",
      "content": "<p>I guess it's more effective if it's based on distinct datasets. There are actually many wav2vec2 finetuned on Bengali online, though they may not be effective.</p>",
      "rawMarkdown": "I guess it's more effective if it's based on distinct datasets. There are actually many wav2vec2 finetuned on Bengali online, though they may not be effective.",
      "votes": null
    },
    {
      "id": "2467381",
      "postDate": "10/04/2023 14:00:38",
      "content": "<p>Yep, you are right they are trained on different dataset.  </p>",
      "rawMarkdown": "Yep, you are right they are trained on different dataset.",
      "votes": null
    },
    {
      "id": "2468990",
      "postDate": "10/06/2023 03:51:49",
      "content": "<p>I did the similar way you did. Two models’s probabilities were averaged before converting into letters. With 3 test files it ran ok but got “Can’t find CSV error” when scoring. Anyone experienced this? Any help would be appreciated! (also len==0 post processing couldn’t help)</p>",
      "rawMarkdown": "I did the similar way you did. Two models’s probabilities were averaged before converting into letters. With 3 test files it ran ok but got “Can’t find CSV error” when scoring. Anyone experienced this? Any help would be appreciated! (also len==0 post processing couldn’t help)",
      "votes": null
    },
    {
      "id": "2469065",
      "postDate": "10/06/2023 06:06:32",
      "content": "<p>Most likely something went wrong on the test set that is being calculated error, unfortunately error debuging is terrible….</p>",
      "rawMarkdown": "Most likely something went wrong on the test set that is being calculated error, unfortunately error debuging is terrible....",
      "votes": null
    },
    {
      "id": "2469086",
      "postDate": "10/06/2023 06:23:36",
      "content": "<p>I have tried ensemble by taking the mean, though the score gets worse. I didn't encounter the bug you have described, sorry. Maybe underflow or overflow? But the logits should be already normalized</p>",
      "rawMarkdown": "I have tried ensemble by taking the mean, though the score gets worse. I didn't encounter the bug you have described, sorry. Maybe underflow or overflow? But the logits should be already normalized",
      "votes": null
    },
    {
      "id": "2483217",
      "postDate": "10/15/2023 14:57:43",
      "content": "<p><a href=\"https://www.kaggle.com/code/jjleesunny/how-to-ensemble-ctc-models\" target=\"_blank\">https://www.kaggle.com/code/jjleesunny/how-to-ensemble-ctc-models</a></p>",
      "rawMarkdown": "https://www.kaggle.com/code/jjleesunny/how-to-ensemble-ctc-models",
      "votes": null
    },
    {
      "id": "2484274",
      "postDate": "10/16/2023 11:01:41",
      "content": "<p><a href=\"https://www.kaggle.com/enddl22\" target=\"_blank\">@enddl22</a> have you able to resolved it . i'm facing the same issue if you could please help me </p>",
      "rawMarkdown": "enddl22 have you able to resolved it . i'm facing the same issue if you could please help me",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2465799,
      "author_name": "tugstugi",
      "author_url": "",
      "post_date": "10/03/2023 09:50:32",
      "content": "<p>You can't ensemble CTC models because the predictions are not guarenteed to be aligned.</p>",
      "votes": null,
      "replies": [
        {
          "id": 2466135,
          "author_name": "nyleve",
          "author_url": "",
          "post_date": "10/03/2023 15:23:17",
          "content": "<p>maybe we can try dynamic time warping to align them ? </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 2483217,
          "author_name": "hypnotu",
          "author_url": "",
          "post_date": "10/15/2023 14:57:43",
          "content": "<p><a href=\"https://www.kaggle.com/code/jjleesunny/how-to-ensemble-ctc-models\" target=\"_blank\">https://www.kaggle.com/code/jjleesunny/how-to-ensemble-ctc-models</a></p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 2465803,
      "author_name": "nyleve",
      "author_url": "",
      "post_date": "10/03/2023 10:00:06",
      "content": "<p>It may not make sense, but I get 0.02 boost on LB by blindly weight averaging the logits </p>",
      "votes": null,
      "replies": [
        {
          "id": 2466578,
          "author_name": "renyiwei",
          "author_url": "",
          "post_date": "10/04/2023 00:34:51",
          "content": "<p>Do two models have the same vocab.json? Welll, even if the corresponding vocab doesn't align, you still can manually adjust it to fit them together right…</p>",
          "votes": null,
          "replies": [
            {
              "id": 2466613,
              "author_name": "nyleve",
              "author_url": "",
              "post_date": "10/04/2023 01:35:23",
              "content": "<p>Yea, for ease of averaging the two has exactly same vocab.</p>",
              "votes": null,
              "replies": []
            }
          ]
        },
        {
          "id": 2466995,
          "author_name": "baohaoliao",
          "author_url": "",
          "post_date": "10/04/2023 08:13:52",
          "content": "<p>Are the two models you ensemble from different seed training or from different epochs of the same training?</p>",
          "votes": null,
          "replies": [
            {
              "id": 2467004,
              "author_name": "renyiwei",
              "author_url": "",
              "post_date": "10/04/2023 08:25:26",
              "content": "<p>I guess it's more effective if it's based on distinct datasets. There are actually many wav2vec2 finetuned on Bengali online, though they may not be effective.</p>",
              "votes": null,
              "replies": [
                {
                  "id": 2467381,
                  "author_name": "nyleve",
                  "author_url": "",
                  "post_date": "10/04/2023 14:00:38",
                  "content": "<p>Yep, you are right they are trained on different dataset.  </p>",
                  "votes": null,
                  "replies": []
                }
              ]
            }
          ]
        },
        {
          "id": 2468990,
          "author_name": "enddl22",
          "author_url": "",
          "post_date": "10/06/2023 03:51:49",
          "content": "<p>I did the similar way you did. Two models’s probabilities were averaged before converting into letters. With 3 test files it ran ok but got “Can’t find CSV error” when scoring. Anyone experienced this? Any help would be appreciated! (also len==0 post processing couldn’t help)</p>",
          "votes": null,
          "replies": [
            {
              "id": 2469065,
              "author_name": "hubert101",
              "author_url": "",
              "post_date": "10/06/2023 06:06:32",
              "content": "<p>Most likely something went wrong on the test set that is being calculated error, unfortunately error debuging is terrible….</p>",
              "votes": null,
              "replies": []
            },
            {
              "id": 2469086,
              "author_name": "renyiwei",
              "author_url": "",
              "post_date": "10/06/2023 06:23:36",
              "content": "<p>I have tried ensemble by taking the mean, though the score gets worse. I didn't encounter the bug you have described, sorry. Maybe underflow or overflow? But the logits should be already normalized</p>",
              "votes": null,
              "replies": []
            },
            {
              "id": 2484274,
              "author_name": "shanusaumya",
              "author_url": "",
              "post_date": "10/16/2023 11:01:41",
              "content": "<p><a href=\"https://www.kaggle.com/enddl22\" target=\"_blank\">@enddl22</a> have you able to resolved it . i'm facing the same issue if you could please help me </p>",
              "votes": null,
              "replies": []
            }
          ]
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "2465762": "When I ensemble my best two performing models (same design and trained different fold), my performance deteriorates. Did anyone succeed in setting up Ensemble ?",
    "2465799": "You can't ensemble CTC models because the predictions are not guarenteed to be aligned.",
    "2465803": "It may not make sense, but I get 0.02 boost on LB by blindly weight averaging the logits",
    "2466135": "maybe we can try dynamic time warping to align them ?",
    "2466578": "Do two models have the same vocab.json? Welll, even if the corresponding vocab doesn't align, you still can manually adjust it to fit them together right...",
    "2466613": "Yea, for ease of averaging the two has exactly same vocab.",
    "2466995": "Are the two models you ensemble from different seed training or from different epochs of the same training?",
    "2467004": "I guess it's more effective if it's based on distinct datasets. There are actually many wav2vec2 finetuned on Bengali online, though they may not be effective.",
    "2467381": "Yep, you are right they are trained on different dataset.",
    "2468990": "I did the similar way you did. Two models’s probabilities were averaged before converting into letters. With 3 test files it ran ok but got “Can’t find CSV error” when scoring. Anyone experienced this? Any help would be appreciated! (also len==0 post processing couldn’t help)",
    "2469065": "Most likely something went wrong on the test set that is being calculated error, unfortunately error debuging is terrible....",
    "2469086": "I have tried ensemble by taking the mean, though the score gets worse. I didn't encounter the bug you have described, sorry. Maybe underflow or overflow? But the logits should be already normalized",
    "2483217": "https://www.kaggle.com/code/jjleesunny/how-to-ensemble-ctc-models",
    "2484274": "enddl22 have you able to resolved it . i'm facing the same issue if you could please help me"
  },
  "source": "meta"
}