{
  "id": 220071,
  "title": "CV calculation for different models",
  "url": "/competitions/cassava-leaf-disease-classification/discussion/220071",
  "author_name": "Mykola Lavreniuk",
  "post_date": "2021-02-17T09:18:11.270000",
  "votes": 7,
  "comment_count": 0,
  "views": 0,
  "content": "<p>I am curious about how to calculate the CV in the correct way. For example, I have some model like Eff4 and trained it for 5 folds, after that I have changed some parametrs (like loss function, augmentation, etc.) and trained the same model with new strategy for the same 5 folds (with the same seed). Consequently, I received some scores for each fold and each strategy: sc_fold1_1, sc_fold2_1, sc_fold3_1, sc_fold4_1, sc_fold5_1, and sc_fold1_2, sc_fold2_2, sc_fold3_2, sc_fold4_2, sc_fold5_2. Than the CV for model with strategy 1 calculates as mean(sc_fold1_1, sc_fold2_1, sc_fold3_1, sc_fold4_1, sc_fold5_1).<br>\nFor example, my sc_fold1_1&gt;sc_fold1_2, sc_fold2_1&gt;sc_fold2_2, sc_fold3_1&gt;sc_fold3_2. Hovewer, the sc_fold4_1&lt;sc_fold4_2, sc_fold5_1&lt;sc_fold5_2.</p>\n<p>The question is the next: is it correct to mix the models folds from different strategy for increasing the CV (and is such way of calc CV is right)?<br>\nnew_CV = mean(sc_fold1_1, sc_fold2_1, sc_fold3_1, sc_fold4_2, sc_fold5_2).<br>\nOr CV is the score for the entire model and strategy, and we could not calculate new CV in such way and should to pick the best model with all the folds without mixing the folds?</p>",
  "messages": [
    {
      "id": 1206389,
      "postDate": "2021-02-17T09:18:11.270Z",
      "content": "<p>I am curious about how to calculate the CV in the correct way. For example, I have some model like Eff4 and trained it for 5 folds, after that I have changed some parametrs (like loss function, augmentation, etc.) and trained the same model with new strategy for the same 5 folds (with the same seed). Consequently, I received some scores for each fold and each strategy: sc_fold1_1, sc_fold2_1, sc_fold3_1, sc_fold4_1, sc_fold5_1, and sc_fold1_2, sc_fold2_2, sc_fold3_2, sc_fold4_2, sc_fold5_2. Than the CV for model with strategy 1 calculates as mean(sc_fold1_1, sc_fold2_1, sc_fold3_1, sc_fold4_1, sc_fold5_1).<br>\nFor example, my sc_fold1_1&gt;sc_fold1_2, sc_fold2_1&gt;sc_fold2_2, sc_fold3_1&gt;sc_fold3_2. Hovewer, the sc_fold4_1&lt;sc_fold4_2, sc_fold5_1&lt;sc_fold5_2.</p>\n<p>The question is the next: is it correct to mix the models folds from different strategy for increasing the CV (and is such way of calc CV is right)?<br>\nnew_CV = mean(sc_fold1_1, sc_fold2_1, sc_fold3_1, sc_fold4_2, sc_fold5_2).<br>\nOr CV is the score for the entire model and strategy, and we could not calculate new CV in such way and should to pick the best model with all the folds without mixing the folds?</p>",
      "rawMarkdown": "I am curious about how to calculate the CV in the correct way. For example, I have some model like Eff4 and trained it for 5 folds, after that I have changed some parametrs (like loss function, augmentation, etc.) and trained the same model with new strategy for the same 5 folds (with the same seed). Consequently, I received some scores for each fold and each strategy: sc_fold1_1, sc_fold2_1, sc_fold3_1, sc_fold4_1, sc_fold5_1, and sc_fold1_2, sc_fold2_2, sc_fold3_2, sc_fold4_2, sc_fold5_2. Than the CV for model with strategy 1 calculates as mean(sc_fold1_1, sc_fold2_1, sc_fold3_1, sc_fold4_1, sc_fold5_1).\nFor example, my sc_fold1_1>sc_fold1_2, sc_fold2_1>sc_fold2_2, sc_fold3_1>sc_fold3_2. Hovewer, the sc_fold4_1<sc_fold4_2, sc_fold5_1<sc_fold5_2.\n\nThe question is the next: is it correct to mix the models folds from different strategy for increasing the CV (and is such way of calc CV is right)?\nnew_CV = mean(sc_fold1_1, sc_fold2_1, sc_fold3_1, sc_fold4_2, sc_fold5_2).\nOr CV is the score for the entire model and strategy, and we could not calculate new CV in such way and should to pick the best model with all the folds without mixing the folds?",
      "votes": 7
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "1206389": "I am curious about how to calculate the CV in the correct way. For example, I have some model like Eff4 and trained it for 5 folds, after that I have changed some parametrs (like loss function, augmentation, etc.) and trained the same model with new strategy for the same 5 folds (with the same seed). Consequently, I received some scores for each fold and each strategy: sc_fold1_1, sc_fold2_1, sc_fold3_1, sc_fold4_1, sc_fold5_1, and sc_fold1_2, sc_fold2_2, sc_fold3_2, sc_fold4_2, sc_fold5_2. Than the CV for model with strategy 1 calculates as mean(sc_fold1_1, sc_fold2_1, sc_fold3_1, sc_fold4_1, sc_fold5_1).\nFor example, my sc_fold1_1>sc_fold1_2, sc_fold2_1>sc_fold2_2, sc_fold3_1>sc_fold3_2. Hovewer, the sc_fold4_1<sc_fold4_2, sc_fold5_1<sc_fold5_2.\n\nThe question is the next: is it correct to mix the models folds from different strategy for increasing the CV (and is such way of calc CV is right)?\nnew_CV = mean(sc_fold1_1, sc_fold2_1, sc_fold3_1, sc_fold4_2, sc_fold5_2).\nOr CV is the score for the entire model and strategy, and we could not calculate new CV in such way and should to pick the best model with all the folds without mixing the folds?"
  }
}