My team is truly struggling to get higher auc score just on a single model as well as ensemble. But we are not successful till now. We are bounded by 0.95 auc. Please give some advice to make our effort fruitful.
", "messages": [ { "id": "962939", "postDate": "08/08/2020 14:59:47", "content": "My team is truly struggling to get higher auc score just on a single model as well as ensemble. But we are not successful till now. We are bounded by 0.95 auc. Please give some advice to make our effort fruitful.
", "rawMarkdown": "My team is truly struggling to get higher auc score just on a single model as well as ensemble. But we are not successful till now. We are bounded by 0.95 auc. Please give some advice to make our effort fruitful.", "votes": null }, { "id": "963020", "postDate": "08/08/2020 15:47:41", "content": "Scoring over LB 0.950 with a single model is difficult and may just be overfitting LB. I haven't heard of any team scoring over CV 0.950 with a single model. Below are some ideas to help.
\nMany high scoring public notebook are ensembles, but here is a single model public notebook (version 4) without meta that scores over LB 0.950. And here is a single model public notebook (version 3) with meta that scores at LB 0.950 with 3 Fold (and conversion to 5 Fold scores over LB 0.950). There is also a discussion post here discussioning solutions over LB 0.950.
", "rawMarkdown": "Scoring over LB 0.950 with a single model is difficult and may just be overfitting LB. I haven't heard of any team scoring over CV 0.950 with a single model. Below are some ideas to help.\n\nMany high scoring public notebook are ensembles, but here is a single model public [notebook][1] (version 4) without meta that scores over LB 0.950. And here is a single model public [notebook][2] (version 3) with meta that scores at LB 0.950 with 3 Fold (and conversion to 5 Fold scores over LB 0.950). There is also a discussion post [here][3] discussioning solutions over LB 0.950. \n\n[1]: https://www.kaggle.com/ajaykumar7778/efficientnet-cv?scriptVersionId=40043341\n[2]: https://www.kaggle.com/rajnishe/rc-fork-siim-isic-melanoma-384x384?scriptVersionId=39612412\n[3]: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/172003", "votes": null }, { "id": "963026", "postDate": "08/08/2020 15:54:04", "content": "@cdeotte It's possible to get + 0.956 by modifying one of your kernels :)
\nAnd I am sure it's possible to push them further and even some on LB have already managed to do it.
\nSadly I don't have time to dive deeper on them as I mainly work with Pytorch so far.
Nice. I have pushed my notebooks very far too :-) But it may just be overfitting LB
\nIs PyTorch giving lower CV LB than TensorFlow?
", "rawMarkdown": "Nice. I have pushed my notebooks very far too :-) But it may just be overfitting LB\n\nIs PyTorch giving lower CV LB than TensorFlow?", "votes": null }, { "id": "963044", "postDate": "08/08/2020 16:12:45", "content": "CV 0.9087
\nLB 0.950
the dream
", "rawMarkdown": "CV 0.9087\nLB 0.950\n\nthe dream", "votes": null }, { "id": "963046", "postDate": "08/08/2020 16:13:13", "content": "I found it really hard to iterate with Pytorch because you are rapidly capped by your GPU's memory/performance (at least to me). The best CV I could pull off was ~0.92.
", "rawMarkdown": "I found it really hard to iterate with Pytorch because you are rapidly capped by your GPU's memory/performance (at least to me). The best CV I could pull off was ~0.92.", "votes": null }, { "id": "963051", "postDate": "08/08/2020 16:20:57", "content": "Thanks @cdeotte for this helpful suggestion.
", "rawMarkdown": "Thanks @cdeotte for this helpful suggestion.", "votes": null }, { "id": "963058", "postDate": "08/08/2020 16:24:08", "content": "@cdeotte My best Pytorch model got LB 0.952 with CV 0.936 (OOF)
", "rawMarkdown": "cdeotte My best Pytorch model got LB 0.952 with CV 0.936 (OOF)", "votes": null }, { "id": "963083", "postDate": "08/08/2020 16:46:03", "content": "@cdeotte If it's not a secret, what is your best CV after ensemble?
", "rawMarkdown": "cdeotte If it's not a secret, what is your best CV after ensemble?", "votes": null }, { "id": "963502", "postDate": "08/09/2020 04:45:51", "content": "Seems model is not generalize. Dont use too many layer after backbone.
\nFor same model B4 , try different LR , exponential decay , warmup scheduler .
\nAnd then ensemble.
I will try
", "rawMarkdown": "I will try", "votes": null }, { "id": "966015", "postDate": "08/11/2020 03:53:04", "content": "Hi Ritacheta,
\n\nDifficult to suggest without knowing what your team has tried so far. Are you aware of the duplicates in the data? Are you one-hot encoding? or train several models individually based on where the cancer is located? For example:
\n\n\n\n\ntrain.groupby('anatom_site_general_challenge')['image_name'].count()\n anatom_site_general_challenge\n head/neck 1855\n lower extremity 8417\n oral/genital 124\n palms/soles 375\n torso 16845\n upper extremity 4983\n Name: image_name, dtype: int64
\n \ntest.groupby('anatom_site_general_challenge')['image_name'].count()\n anatom_site_general_challenge\n head/neck 576\n lower extremity 2501\n oral/genital 26\n palms/soles 108\n torso 5847\n upper extremity 1573\n Name: image_name, dtype: int64
\n
Maybe combine these 6 categories into 3 and build 3 different models. Then create 3 submissions and combine them to submit.
\n\nRegards,\nSrikanth
", "rawMarkdown": "Hi Ritacheta, \n\nDifficult to suggest without knowing what your team has tried so far. Are you aware of the duplicates in the data? Are you one-hot encoding? or train several models individually based on where the cancer is located? For example: \n\n> train.groupby('anatom_site_general_challenge')['image_name'].count()\nanatom_site_general_challenge\nhead/neck 1855\nlower extremity 8417\noral/genital 124\npalms/soles 375\ntorso 16845\nupper extremity 4983\nName: image_name, dtype: int64\n\n> test.groupby('anatom_site_general_challenge')['image_name'].count()\nanatom_site_general_challenge\nhead/neck 576\nlower extremity 2501\noral/genital 26\npalms/soles 108\ntorso 5847\nupper extremity 1573\nName: image_name, dtype: int64\n\nMaybe combine these 6 categories into 3 and build 3 different models. Then create 3 submissions and combine them to submit.\n\nRegards,\nSrikanth", "votes": null } ], "comments": [ { "id": 963020, "author_name": "cdeotte", "author_url": "", "post_date": "08/08/2020 15:47:41", "content": "Scoring over LB 0.950 with a single model is difficult and may just be overfitting LB. I haven't heard of any team scoring over CV 0.950 with a single model. Below are some ideas to help.
\nMany high scoring public notebook are ensembles, but here is a single model public notebook (version 4) without meta that scores over LB 0.950. And here is a single model public notebook (version 3) with meta that scores at LB 0.950 with 3 Fold (and conversion to 5 Fold scores over LB 0.950). There is also a discussion post here discussioning solutions over LB 0.950.
", "votes": null, "replies": [ { "id": 963026, "author_name": "serigne", "author_url": "", "post_date": "08/08/2020 15:54:04", "content": "@cdeotte It's possible to get + 0.956 by modifying one of your kernels :)
\nAnd I am sure it's possible to push them further and even some on LB have already managed to do it.
\nSadly I don't have time to dive deeper on them as I mainly work with Pytorch so far.
Nice. I have pushed my notebooks very far too :-) But it may just be overfitting LB
\nIs PyTorch giving lower CV LB than TensorFlow?
", "votes": null, "replies": [] }, { "id": 963044, "author_name": "philippsinger", "author_url": "", "post_date": "08/08/2020 16:12:45", "content": "CV 0.9087
\nLB 0.950
the dream
", "votes": null, "replies": [] }, { "id": 963046, "author_name": "romainfabre", "author_url": "", "post_date": "08/08/2020 16:13:13", "content": "I found it really hard to iterate with Pytorch because you are rapidly capped by your GPU's memory/performance (at least to me). The best CV I could pull off was ~0.92.
", "votes": null, "replies": [] }, { "id": 963051, "author_name": "ritachetadas", "author_url": "", "post_date": "08/08/2020 16:20:57", "content": "Thanks @cdeotte for this helpful suggestion.
", "votes": null, "replies": [] }, { "id": 963058, "author_name": "serigne", "author_url": "", "post_date": "08/08/2020 16:24:08", "content": "@cdeotte My best Pytorch model got LB 0.952 with CV 0.936 (OOF)
", "votes": null, "replies": [] }, { "id": 963083, "author_name": "ks2019", "author_url": "", "post_date": "08/08/2020 16:46:03", "content": "@cdeotte If it's not a secret, what is your best CV after ensemble?
", "votes": null, "replies": [] } ] }, { "id": 963502, "author_name": "rajnishe", "author_url": "", "post_date": "08/09/2020 04:45:51", "content": "Seems model is not generalize. Dont use too many layer after backbone.
\nFor same model B4 , try different LR , exponential decay , warmup scheduler .
\nAnd then ensemble.
I will try
", "votes": null, "replies": [] } ] }, { "id": 966015, "author_name": "srikanthpotukuchi", "author_url": "", "post_date": "08/11/2020 03:53:04", "content": "Hi Ritacheta,
\n\nDifficult to suggest without knowing what your team has tried so far. Are you aware of the duplicates in the data? Are you one-hot encoding? or train several models individually based on where the cancer is located? For example:
\n\n\n\n\ntrain.groupby('anatom_site_general_challenge')['image_name'].count()\n anatom_site_general_challenge\n head/neck 1855\n lower extremity 8417\n oral/genital 124\n palms/soles 375\n torso 16845\n upper extremity 4983\n Name: image_name, dtype: int64
\n \ntest.groupby('anatom_site_general_challenge')['image_name'].count()\n anatom_site_general_challenge\n head/neck 576\n lower extremity 2501\n oral/genital 26\n palms/soles 108\n torso 5847\n upper extremity 1573\n Name: image_name, dtype: int64
\n
Maybe combine these 6 categories into 3 and build 3 different models. Then create 3 submissions and combine them to submit.
\n\nRegards,\nSrikanth
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "962939": "My team is truly struggling to get higher auc score just on a single model as well as ensemble. But we are not successful till now. We are bounded by 0.95 auc. Please give some advice to make our effort fruitful.", "963020": "Scoring over LB 0.950 with a single model is difficult and may just be overfitting LB. I haven't heard of any team scoring over CV 0.950 with a single model. Below are some ideas to help.\n\nMany high scoring public notebook are ensembles, but here is a single model public [notebook][1] (version 4) without meta that scores over LB 0.950. And here is a single model public [notebook][2] (version 3) with meta that scores at LB 0.950 with 3 Fold (and conversion to 5 Fold scores over LB 0.950). There is also a discussion post [here][3] discussioning solutions over LB 0.950. \n\n[1]: https://www.kaggle.com/ajaykumar7778/efficientnet-cv?scriptVersionId=40043341\n[2]: https://www.kaggle.com/rajnishe/rc-fork-siim-isic-melanoma-384x384?scriptVersionId=39612412\n[3]: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/172003", "963026": "cdeotte It's possible to get + 0.956 by modifying one of your kernels :)\nAnd I am sure it's possible to push them further and even some on LB have already managed to do it. \nSadly I don't have time to dive deeper on them as I mainly work with Pytorch so far.", "963031": "Nice. I have pushed my notebooks very far too :-) But it may just be overfitting LB\n\nIs PyTorch giving lower CV LB than TensorFlow?", "963044": "CV 0.9087\nLB 0.950\n\nthe dream", "963046": "I found it really hard to iterate with Pytorch because you are rapidly capped by your GPU's memory/performance (at least to me). The best CV I could pull off was ~0.92.", "963051": "Thanks @cdeotte for this helpful suggestion.", "963058": "cdeotte My best Pytorch model got LB 0.952 with CV 0.936 (OOF)", "963083": "cdeotte If it's not a secret, what is your best CV after ensemble?", "963502": "Seems model is not generalize. Dont use too many layer after backbone.\nFor same model B4 , try different LR , exponential decay , warmup scheduler .\nAnd then ensemble.", "963510": "I will try", "966015": "Hi Ritacheta, \n\nDifficult to suggest without knowing what your team has tried so far. Are you aware of the duplicates in the data? Are you one-hot encoding? or train several models individually based on where the cancer is located? For example: \n\n> train.groupby('anatom_site_general_challenge')['image_name'].count()\nanatom_site_general_challenge\nhead/neck 1855\nlower extremity 8417\noral/genital 124\npalms/soles 375\ntorso 16845\nupper extremity 4983\nName: image_name, dtype: int64\n\n> test.groupby('anatom_site_general_challenge')['image_name'].count()\nanatom_site_general_challenge\nhead/neck 576\nlower extremity 2501\noral/genital 26\npalms/soles 108\ntorso 5847\nupper extremity 1573\nName: image_name, dtype: int64\n\nMaybe combine these 6 categories into 3 and build 3 different models. Then create 3 submissions and combine them to submit.\n\nRegards,\nSrikanth" }, "source": "meta" }