{ "id": 175330, "title": "105 place solution (0.9413 on LB). ", "url": "/competitions/siim-isic-melanoma-classification/discussion/175330", "author_name": "Mehul Sampat", "post_date": "2020-08-18T00:36:19.870000", "votes": 1, "comment_count": 0, "views": 0, "content": "

Summary

\n
    \n
  1. Ensemble of 6 models gave a Private LB of 0.9413; The ensemble code in here
    2. \n
  2. The models are just simple modifications of @cdeotte baseline
    3. \n
  3. We will share these later
    4. \n
\n

Details:

\n
    \n

  1. We used @cdeotte baseline as a starter.

    2. \n

  2. Then we train various different models. In the end we had 12 different models. (lost some models because i forgot to save oof.csv :( ).

    3. \n

  3. then for ensemble it seems liked number of models combined was important.

    4. \n

  4. So we ranked the 12 models by their OOF_AUC.

    5. \n

  5. The ensemble was done as per the starter kernel provided by @steubk

    6. \n

  6. This was modified for 4,5,6,8,9,12 models.

    7. \n

  7. This is version with 6 models which gave us the Private LB score of 0.9413.

    8. \n

  8. Then we found the OOF_AUC with Bayesian Optimization looked promising.

    9. \n

  9. the difference between OOF_AUC_Bayesian_Optimization and LB was not very large.

    10. \n

  10. For ensemble of 6 models we should have run 12111098*7 combinations;

    11. \n

  11. but i did not have time to run this. so I picked top 6 models; and a few different combinations of top 6 models.

    12. \n

  12. The summary is in this spreadsheet

    13. \n

  13. Heuristics used to select models:

    14. \n

  14. We have 4 measures of ensemble AUC:
    \n14a OOF_avg_auc, OOF_rank_auc, OOF_pow_auc,OOF_bo_auc

    15. \n

  15. I took the average of these 4 models. **If a model is good, them my heuristic is that all 4 OOF_AUC measures should be high. **

    16. \n

  16. I also prefer, the highest OOF_bo_AUC or highest OOF average **with lowest number of models. **

    17. \n

  17. We did not finish full extensive analysis but ensemble of 6 models scored very high;
    \n18 Lastly, the difference between OOF_bo_AUC and Public LB is quite low for this combination.

    18. \n
", "messages": [ { "id": 974467, "postDate": "2020-08-18T00:36:19.870Z", "content": "

Summary

\n
    \n
  1. Ensemble of 6 models gave a Private LB of 0.9413; The ensemble code in here
    2. \n
  2. The models are just simple modifications of @cdeotte baseline
    3. \n
  3. We will share these later
    4. \n
\n

Details:

\n
    \n

  1. We used @cdeotte baseline as a starter.

    2. \n

  2. Then we train various different models. In the end we had 12 different models. (lost some models because i forgot to save oof.csv :( ).

    3. \n

  3. then for ensemble it seems liked number of models combined was important.

    4. \n

  4. So we ranked the 12 models by their OOF_AUC.

    5. \n

  5. The ensemble was done as per the starter kernel provided by @steubk

    6. \n

  6. This was modified for 4,5,6,8,9,12 models.

    7. \n

  7. This is version with 6 models which gave us the Private LB score of 0.9413.

    8. \n

  8. Then we found the OOF_AUC with Bayesian Optimization looked promising.

    9. \n

  9. the difference between OOF_AUC_Bayesian_Optimization and LB was not very large.

    10. \n

  10. For ensemble of 6 models we should have run 12111098*7 combinations;

    11. \n

  11. but i did not have time to run this. so I picked top 6 models; and a few different combinations of top 6 models.

    12. \n

  12. The summary is in this spreadsheet

    13. \n

  13. Heuristics used to select models:

    14. \n

  14. We have 4 measures of ensemble AUC:
    \n14a OOF_avg_auc, OOF_rank_auc, OOF_pow_auc,OOF_bo_auc

    15. \n

  15. I took the average of these 4 models. **If a model is good, them my heuristic is that all 4 OOF_AUC measures should be high. **

    16. \n

  16. I also prefer, the highest OOF_bo_AUC or highest OOF average **with lowest number of models. **

    17. \n

  17. We did not finish full extensive analysis but ensemble of 6 models scored very high;
    \n18 Lastly, the difference between OOF_bo_AUC and Public LB is quite low for this combination.

    18. \n
", "rawMarkdown": "## Summary\n1. Ensemble of 6 models gave a Private LB of 0.9413; The ensemble code in [here](https://www.kaggle.com/mpsampat/final-simple-oof-ensembling-methods-6-models)\n2. The models are just simple modifications of @cdeotte [baseline]( https://www.kaggle.com/cdeotte/triple-stratified-kfold-with-tfrecords)\n3. We will share these later\n\n## Details: \n1. We used @cdeotte [baseline]( https://www.kaggle.com/cdeotte/triple-stratified-kfold-with-tfrecords) as a starter. \n2. Then we train various different models. In the end we had 12 different models. (lost some models because i forgot to save oof.csv :( ). \n\n3. then for ensemble it seems liked number of models combined was important. \n4. So we ranked the 12 models by their OOF_AUC. \n\n5. The ensemble was done as per the [starter](https://www.kaggle.com/steubk/simple-oof-ensembling-methods-for-classification) kernel provided by @steubk\n\n6. This was modified for 4,5,6,8,9,12 models. \n7. This is version with [6 models](https://www.kaggle.com/mpsampat/final-simple-oof-ensembling-methods-6-models) which gave us the Private LB score of 0.9413. \n\n\n8. Then we found the OOF_AUC with Bayesian Optimization looked promising. \n9. the difference between OOF_AUC_Bayesian_Optimization and LB was not very large. \n10. For ensemble of 6 models we should have run 12*11*10*9*8*7 combinations; \n11. but i did not have time to run this. so I picked top 6 models; and a few different combinations of top 6 models. \n12. The summary is in this [spreadsheet](https://docs.google.com/spreadsheets/d/1Gw7xJwkxxp-61YHHOEkS2DImxD4jkQ5HYRi-0j72QJM/edit?usp=sharing)\n13. Heuristics used to select models: \n14. We have 4 measures of ensemble AUC: \n14a OOF_avg_auc, OOF_rank_auc, OOF_pow_auc,OOF_bo_auc\n15. I took the average of these 4 models. **If a model is good, them my heuristic is that all 4 OOF_AUC measures should be high. **\n16. I also prefer, the highest OOF_bo_AUC or highest OOF average **with lowest number of models. **\n17. We did not finish full extensive analysis but ensemble of 6 models scored very high; \n18 Lastly, the difference between OOF_bo_AUC and Public LB is quite low for this combination.\n\n", "votes": 1 } ], "comments": [], "raw_markdown_by_id": { "974467": "## Summary\n1. Ensemble of 6 models gave a Private LB of 0.9413; The ensemble code in [here](https://www.kaggle.com/mpsampat/final-simple-oof-ensembling-methods-6-models)\n2. The models are just simple modifications of @cdeotte [baseline]( https://www.kaggle.com/cdeotte/triple-stratified-kfold-with-tfrecords)\n3. We will share these later\n\n## Details: \n1. We used @cdeotte [baseline]( https://www.kaggle.com/cdeotte/triple-stratified-kfold-with-tfrecords) as a starter. \n2. Then we train various different models. In the end we had 12 different models. (lost some models because i forgot to save oof.csv :( ). \n\n3. then for ensemble it seems liked number of models combined was important. \n4. So we ranked the 12 models by their OOF_AUC. \n\n5. The ensemble was done as per the [starter](https://www.kaggle.com/steubk/simple-oof-ensembling-methods-for-classification) kernel provided by @steubk\n\n6. This was modified for 4,5,6,8,9,12 models. \n7. This is version with [6 models](https://www.kaggle.com/mpsampat/final-simple-oof-ensembling-methods-6-models) which gave us the Private LB score of 0.9413. \n\n\n8. Then we found the OOF_AUC with Bayesian Optimization looked promising. \n9. the difference between OOF_AUC_Bayesian_Optimization and LB was not very large. \n10. For ensemble of 6 models we should have run 12*11*10*9*8*7 combinations; \n11. but i did not have time to run this. so I picked top 6 models; and a few different combinations of top 6 models. \n12. The summary is in this [spreadsheet](https://docs.google.com/spreadsheets/d/1Gw7xJwkxxp-61YHHOEkS2DImxD4jkQ5HYRi-0j72QJM/edit?usp=sharing)\n13. Heuristics used to select models: \n14. We have 4 measures of ensemble AUC: \n14a OOF_avg_auc, OOF_rank_auc, OOF_pow_auc,OOF_bo_auc\n15. I took the average of these 4 models. **If a model is good, them my heuristic is that all 4 OOF_AUC measures should be high. **\n16. I also prefer, the highest OOF_bo_AUC or highest OOF average **with lowest number of models. **\n17. We did not finish full extensive analysis but ensemble of 6 models scored very high; \n18 Lastly, the difference between OOF_bo_AUC and Public LB is quite low for this combination.\n\n" } }