{
  "id": 271786,
  "title": "Writing my CV / LB observations",
  "url": "/competitions/rsna-miccai-brain-tumor-radiogenomic-classification/discussion/271786",
  "author_name": "CoreyJamesLevinson",
  "post_date": "2021-09-12T15:16:27.335000",
  "votes": 18,
  "comment_count": 0,
  "views": 0,
  "content": "<p>Stratified KFold (5 folds)</p>\n<p>avg of 6 models on different types. CV Logloss was: 0.6536, 0.6913, 0.6726, 0.6930, 0.6X, 0.6909.  The LB is: 0.605</p>\n<p>try method 2<br>\nCV Logloss / CV AUC / LB<br>\n<strong>0.67726 / 0.60602 / 0.635 (trained only on T2w)</strong><br>\n0.67882 / 0.61851 / 0.614 (averaged across all 4 types)<br>\n0.67725 / 0.61812 / 0.609 (averaged across all 4 types, less importance on T1wCE)<br>\n0.64681 / 0.69543 / 0.587 (using all 4 types and stacking their predictions instead of averaging)<br>\n0.65006 / 0.68779 / 0.597 (using all 4 types, stacking using only GBDT)<br>\n0.66310 / 0.64212 / 0.608 (using all 4 types, stacking using only Ridge)<br>\n0.62903 / 0.70655 / 0.554 (using all 4 types, stacking using LogReg, looks super overfit)<br>\n0.66017 / 0.66162 / 0.603 (using all 4 types, stacking using RF)</p>\n<p>new method:<br>\nCV Logloss / CV AUC / LB<br>\n0.6776 / 0.6385 / 0.595 😭</p>\n<p>Important to look at correlations between submissions. Submission A and Submission B were -12% rank correlated. Submission A scored LB 0.594 and Submission B scored LB 0.565</p>\n<p>What can I say in conclusion? Probably avoid stacking; 500 rows is not enough to use. 4 channels, despite being slow to train, is probably better model because the correlation between individual mri type models I saw wasn't very strong</p>\n<p>30 days left in this competition, let's keep it up</p>",
  "messages": [
    {
      "id": 1510620,
      "postDate": "2021-09-12T15:16:27.337Z",
      "content": "<p>Stratified KFold (5 folds)</p>\n<p>avg of 6 models on different types. CV Logloss was: 0.6536, 0.6913, 0.6726, 0.6930, 0.6X, 0.6909.  The LB is: 0.605</p>\n<p>try method 2<br>\nCV Logloss / CV AUC / LB<br>\n<strong>0.67726 / 0.60602 / 0.635 (trained only on T2w)</strong><br>\n0.67882 / 0.61851 / 0.614 (averaged across all 4 types)<br>\n0.67725 / 0.61812 / 0.609 (averaged across all 4 types, less importance on T1wCE)<br>\n0.64681 / 0.69543 / 0.587 (using all 4 types and stacking their predictions instead of averaging)<br>\n0.65006 / 0.68779 / 0.597 (using all 4 types, stacking using only GBDT)<br>\n0.66310 / 0.64212 / 0.608 (using all 4 types, stacking using only Ridge)<br>\n0.62903 / 0.70655 / 0.554 (using all 4 types, stacking using LogReg, looks super overfit)<br>\n0.66017 / 0.66162 / 0.603 (using all 4 types, stacking using RF)</p>\n<p>new method:<br>\nCV Logloss / CV AUC / LB<br>\n0.6776 / 0.6385 / 0.595 😭</p>\n<p>Important to look at correlations between submissions. Submission A and Submission B were -12% rank correlated. Submission A scored LB 0.594 and Submission B scored LB 0.565</p>\n<p>What can I say in conclusion? Probably avoid stacking; 500 rows is not enough to use. 4 channels, despite being slow to train, is probably better model because the correlation between individual mri type models I saw wasn't very strong</p>\n<p>30 days left in this competition, let's keep it up</p>",
      "rawMarkdown": "Stratified KFold (5 folds)\n\navg of 6 models on different types. CV Logloss was: 0.6536, 0.6913, 0.6726, 0.6930, 0.6X, 0.6909.  The LB is: 0.605\n\ntry method 2\nCV Logloss / CV AUC / LB\n**0.67726 / 0.60602 / 0.635 (trained only on T2w)**\n0.67882 / 0.61851 / 0.614 (averaged across all 4 types)\n0.67725 / 0.61812 / 0.609 (averaged across all 4 types, less importance on T1wCE)\n0.64681 / 0.69543 / 0.587 (using all 4 types and stacking their predictions instead of averaging)\n0.65006 / 0.68779 / 0.597 (using all 4 types, stacking using only GBDT)\n0.66310 / 0.64212 / 0.608 (using all 4 types, stacking using only Ridge)\n0.62903 / 0.70655 / 0.554 (using all 4 types, stacking using LogReg, looks super overfit)\n0.66017 / 0.66162 / 0.603 (using all 4 types, stacking using RF)\n\nnew method:\nCV Logloss / CV AUC / LB\n0.6776 / 0.6385 / 0.595 😭\n\nImportant to look at correlations between submissions. Submission A and Submission B were -12% rank correlated. Submission A scored LB 0.594 and Submission B scored LB 0.565\n\nWhat can I say in conclusion? Probably avoid stacking; 500 rows is not enough to use. 4 channels, despite being slow to train, is probably better model because the correlation between individual mri type models I saw wasn't very strong\n\n30 days left in this competition, let's keep it up",
      "votes": 18
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "1510620": "Stratified KFold (5 folds)\n\navg of 6 models on different types. CV Logloss was: 0.6536, 0.6913, 0.6726, 0.6930, 0.6X, 0.6909.  The LB is: 0.605\n\ntry method 2\nCV Logloss / CV AUC / LB\n**0.67726 / 0.60602 / 0.635 (trained only on T2w)**\n0.67882 / 0.61851 / 0.614 (averaged across all 4 types)\n0.67725 / 0.61812 / 0.609 (averaged across all 4 types, less importance on T1wCE)\n0.64681 / 0.69543 / 0.587 (using all 4 types and stacking their predictions instead of averaging)\n0.65006 / 0.68779 / 0.597 (using all 4 types, stacking using only GBDT)\n0.66310 / 0.64212 / 0.608 (using all 4 types, stacking using only Ridge)\n0.62903 / 0.70655 / 0.554 (using all 4 types, stacking using LogReg, looks super overfit)\n0.66017 / 0.66162 / 0.603 (using all 4 types, stacking using RF)\n\nnew method:\nCV Logloss / CV AUC / LB\n0.6776 / 0.6385 / 0.595 😭\n\nImportant to look at correlations between submissions. Submission A and Submission B were -12% rank correlated. Submission A scored LB 0.594 and Submission B scored LB 0.565\n\nWhat can I say in conclusion? Probably avoid stacking; 500 rows is not enough to use. 4 channels, despite being slow to train, is probably better model because the correlation between individual mri type models I saw wasn't very strong\n\n30 days left in this competition, let's keep it up"
  }
}