{
  "id": 175466,
  "title": "For what it's worth: 10 public / 101 private (silver)",
  "url": "/competitions/siim-isic-melanoma-classification/discussion/175466",
  "author_name": "Gilles Vandewiele",
  "post_date": "2020-08-18T09:07:00.743000",
  "votes": 39,
  "comment_count": 19,
  "views": 0,
  "content": "<p>Hi everyone,</p>\n<p>The competition has ended and it seemed that we didn't suffer too much from the shake-up.  We are quite happy about this result, because our best (non-selected) submission doesn't even come close to the gold zone. Clearly we missed some crucial tricks that the top competitors used to break 0.945+: </p>\n<ul>\n<li>using the diagnosis as (auxiliary) target</li>\n<li>pseudo-labeling</li>\n<li>using all data (including external taking into account the many duplicates) in your cross-validation for more stable results.</li>\n<li>…</li>\n</ul>\n<p>For what it's worth, I will provide a rather brief overview of our solution, which is a blend of different models. So nothing really fancy at all! We ensured diversity by modifying (i) the input data, (ii) training scheme and (iii) model architecture:</p>\n<ul>\n<li>We used <a href=\"https://www.kaggle.com/chriscareaga\" target=\"_blank\">@chriscareaga</a> his <a href=\"https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/171745\" target=\"_blank\">cropping technique</a> to create cropped versions of our images. There were some mistakes in the cropping and it didn't work very well for harder images (with lots of moles), but the results were quite OK and rather uncorrelated to our previous results. We also tried training on both the cropped and original version and then combining the image embeddings for the final linear layer.</li>\n<li>Gradual training: first train with 256 images for a few epochs, then use those learned weights as initialization for a next round of training with 384 images, and so on. This upscaling was done until images of 768x768.</li>\n<li>EfficientNetB7, EfficientNetB6 and ResNeSt worked quite well (in that order). EfficientNetB7 with images of 512x512 turned out to be our best model with CV 0.933 / private 0.9373 / public 0.9570. For EfficientNet, we also had some more luck with initializing with AdvProp weights instead of the default imagenet ones.</li>\n</ul>\n<p>We then used scipy.minimize to find a weighted average for the ranks for each of these submissions. In the end, we also blended with a submission made by <a href=\"https://www.kaggle.com/ruozha001\" target=\"_blank\">@ruozha001</a> to get quite a significant boost in public LB (and slightly on private).</p>\n<p>Our 0.9729 submission was a blending mess, and we never expected it to do well. Looks like our expectations were right as all of those blends scores &lt; 0.935 on private. </p>\n<p>I would like to thank my teammates <a href=\"https://www.kaggle.com/chriscareaga\" target=\"_blank\">@chriscareaga</a>, <a href=\"https://www.kaggle.com/bsteenwi\" target=\"_blank\">@bsteenwi</a> and <a href=\"https://www.kaggle.com/khahuras\" target=\"_blank\">@khahuras</a> for this fun competition! And congrats to <a href=\"https://www.kaggle.com/bsteenwi\" target=\"_blank\">@bsteenwi</a> on probably becoming Competition Expert! On to the next!</p>\n<p>PS:<br>\nI already briefly spoke about  <a href=\"https://www.kaggle.com/ruozha001\" target=\"_blank\">@ruozha001</a> 's behaviour in <a href=\"https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/175156\" target=\"_blank\">my topic</a> yesterday, but I'd like to further elaborate upon this issue. Yesterday, he/she came online for the first time in 2 weeks (after seeing the post) and we discussed with him/her how he/she actually obtained the solution. We also mailed some of her former teammates. <strong>I do not believe or have any conclusive evidence he/she is involved in any private sharing.</strong> Rather, it seems that this person's tactic is to quickly climb the public LB by some sketchy blending and then join a team to completely ghost them for the remainder of the competition. So let this be a warning for others (as I've learned the lesson myself now): is to <strong>always be very critical before teaming up and make sure to ask enough questions so that you are convinced that this person will actually have a contribution to the team.</strong></p>",
  "messages": [
    {
      "id": 975369,
      "postDate": "2020-08-18T09:07:00.743Z",
      "content": "<p>Hi everyone,</p>\n<p>The competition has ended and it seemed that we didn't suffer too much from the shake-up.  We are quite happy about this result, because our best (non-selected) submission doesn't even come close to the gold zone. Clearly we missed some crucial tricks that the top competitors used to break 0.945+: </p>\n<ul>\n<li>using the diagnosis as (auxiliary) target</li>\n<li>pseudo-labeling</li>\n<li>using all data (including external taking into account the many duplicates) in your cross-validation for more stable results.</li>\n<li>…</li>\n</ul>\n<p>For what it's worth, I will provide a rather brief overview of our solution, which is a blend of different models. So nothing really fancy at all! We ensured diversity by modifying (i) the input data, (ii) training scheme and (iii) model architecture:</p>\n<ul>\n<li>We used <a href=\"https://www.kaggle.com/chriscareaga\" target=\"_blank\">@chriscareaga</a> his <a href=\"https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/171745\" target=\"_blank\">cropping technique</a> to create cropped versions of our images. There were some mistakes in the cropping and it didn't work very well for harder images (with lots of moles), but the results were quite OK and rather uncorrelated to our previous results. We also tried training on both the cropped and original version and then combining the image embeddings for the final linear layer.</li>\n<li>Gradual training: first train with 256 images for a few epochs, then use those learned weights as initialization for a next round of training with 384 images, and so on. This upscaling was done until images of 768x768.</li>\n<li>EfficientNetB7, EfficientNetB6 and ResNeSt worked quite well (in that order). EfficientNetB7 with images of 512x512 turned out to be our best model with CV 0.933 / private 0.9373 / public 0.9570. For EfficientNet, we also had some more luck with initializing with AdvProp weights instead of the default imagenet ones.</li>\n</ul>\n<p>We then used scipy.minimize to find a weighted average for the ranks for each of these submissions. In the end, we also blended with a submission made by <a href=\"https://www.kaggle.com/ruozha001\" target=\"_blank\">@ruozha001</a> to get quite a significant boost in public LB (and slightly on private).</p>\n<p>Our 0.9729 submission was a blending mess, and we never expected it to do well. Looks like our expectations were right as all of those blends scores &lt; 0.935 on private. </p>\n<p>I would like to thank my teammates <a href=\"https://www.kaggle.com/chriscareaga\" target=\"_blank\">@chriscareaga</a>, <a href=\"https://www.kaggle.com/bsteenwi\" target=\"_blank\">@bsteenwi</a> and <a href=\"https://www.kaggle.com/khahuras\" target=\"_blank\">@khahuras</a> for this fun competition! And congrats to <a href=\"https://www.kaggle.com/bsteenwi\" target=\"_blank\">@bsteenwi</a> on probably becoming Competition Expert! On to the next!</p>\n<p>PS:<br>\nI already briefly spoke about  <a href=\"https://www.kaggle.com/ruozha001\" target=\"_blank\">@ruozha001</a> 's behaviour in <a href=\"https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/175156\" target=\"_blank\">my topic</a> yesterday, but I'd like to further elaborate upon this issue. Yesterday, he/she came online for the first time in 2 weeks (after seeing the post) and we discussed with him/her how he/she actually obtained the solution. We also mailed some of her former teammates. <strong>I do not believe or have any conclusive evidence he/she is involved in any private sharing.</strong> Rather, it seems that this person's tactic is to quickly climb the public LB by some sketchy blending and then join a team to completely ghost them for the remainder of the competition. So let this be a warning for others (as I've learned the lesson myself now): is to <strong>always be very critical before teaming up and make sure to ask enough questions so that you are convinced that this person will actually have a contribution to the team.</strong></p>",
      "rawMarkdown": "Hi everyone,\n\nThe competition has ended and it seemed that we didn't suffer too much from the shake-up.  We are quite happy about this result, because our best (non-selected) submission doesn't even come close to the gold zone. Clearly we missed some crucial tricks that the top competitors used to break 0.945+: \n* using the diagnosis as (auxiliary) target\n* pseudo-labeling\n* using all data (including external taking into account the many duplicates) in your cross-validation for more stable results.\n* ...\n\nFor what it's worth, I will provide a rather brief overview of our solution, which is a blend of different models. So nothing really fancy at all! We ensured diversity by modifying (i) the input data, (ii) training scheme and (iii) model architecture:\n* We used @chriscareaga his [cropping technique](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/171745) to create cropped versions of our images. There were some mistakes in the cropping and it didn't work very well for harder images (with lots of moles), but the results were quite OK and rather uncorrelated to our previous results. We also tried training on both the cropped and original version and then combining the image embeddings for the final linear layer.\n* Gradual training: first train with 256 images for a few epochs, then use those learned weights as initialization for a next round of training with 384 images, and so on. This upscaling was done until images of 768x768.\n* EfficientNetB7, EfficientNetB6 and ResNeSt worked quite well (in that order). EfficientNetB7 with images of 512x512 turned out to be our best model with CV 0.933 / private 0.9373 / public 0.9570. For EfficientNet, we also had some more luck with initializing with AdvProp weights instead of the default imagenet ones.\n\nWe then used scipy.minimize to find a weighted average for the ranks for each of these submissions. In the end, we also blended with a submission made by @ruozha001 to get quite a significant boost in public LB (and slightly on private).\n\nOur 0.9729 submission was a blending mess, and we never expected it to do well. Looks like our expectations were right as all of those blends scores < 0.935 on private. \n\nI would like to thank my teammates @chriscareaga, @bsteenwi and @khahuras for this fun competition! And congrats to @bsteenwi on probably becoming Competition Expert! On to the next!\n\n\nPS:\nI already briefly spoke about  @ruozha001 's behaviour in [my topic](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/175156) yesterday, but I'd like to further elaborate upon this issue. Yesterday, he/she came online for the first time in 2 weeks (after seeing the post) and we discussed with him/her how he/she actually obtained the solution. We also mailed some of her former teammates. **I do not believe or have any conclusive evidence he/she is involved in any private sharing.** Rather, it seems that this person's tactic is to quickly climb the public LB by some sketchy blending and then join a team to completely ghost them for the remainder of the competition. So let this be a warning for others (as I've learned the lesson myself now): is to **always be very critical before teaming up and make sure to ask enough questions so that you are convinced that this person will actually have a contribution to the team.**",
      "votes": 39
    },
    {
      "id": 975495,
      "postDate": "2020-08-18T10:17:11.890Z",
      "content": "<p>Thanks for your approach, <a href=\"https://www.kaggle.com/group16\" target=\"_blank\">@group16</a>.</p>\n<h3>Our Story on Teammate issue</h3>\n<p>I would also like to mention about this <strong>teammate issue</strong> that happened with me in this competition.<br>\nWe have a teammate <strong>(XIAOWANG)</strong> who had a <strong>high baseline score</strong> when he approached me to team up &amp; just because of I wanted to learn and discuss new ideas and to try them, I agreed to get him in my team. After that, we discussed what we have done so far for a few days and what we are planning to do. But every time when I and my other two teammates asked him about the experiments and it's results, he would come up with some argument to divert the topic.</p>\n<p>During the past few days before the merger deadline, We asked him about his experiments and results. The answer we received was that he had <strong>deleted all the results</strong> as they all failed 🤯.  So we asked him about his then <strong>best model</strong>, he said it's <strong>the baseline he had before</strong>. We all were pretty upset so, we asked him if he could at least try Ensembling model using different image-sizes and heavy augmentations. He agreed and before the merger deadline we let him in our team.</p>\n<p><strong>He stabbed us in the back.</strong> He got <strong>unresponsive</strong> everywhere on social media and he did not respond to our daily messages. I also messaged one of his teammates in other ongoing competition but did not get anything about him. </p>\n<p>Maybe, he just wanted to get in a team to get a medal easily by doing nothing but training a baseline.</p>\n<p><strong>Our current rank and the silver medal is because of my and my other two teammates hard work. There is not even a little contribution from the forth teammate.</strong></p>\n<h4>Things that lured me to take him in our team</h4>\n<ul>\n<li>His baseline score was higher than my then best score</li>\n<li>He had a bronze medal in one previous competition</li>\n<li>And he said he has access to an Unlimited GPU resource</li>\n</ul>\n<h4>This was the biggest mistake I made during this competition and I will try to avoid such mistakes in future 😅</h4>\n<p>Special thanks, to my teammates, <a href=\"https://www.kaggle.com/vatsalparsaniya\" target=\"_blank\">@vatsalparsaniya</a> &amp; <a href=\"https://www.kaggle.com/geektoday\" target=\"_blank\">@geektoday</a> 🔥✌🏻</p>\n<p>Peace ✌🏻</p>",
      "rawMarkdown": "Thanks for your approach, @group16.\n\n### Our Story on Teammate issue\n\nI would also like to mention about this **teammate issue** that happened with me in this competition.\nWe have a teammate **(XIAOWANG)** who had a **high baseline score** when he approached me to team up & just because of I wanted to learn and discuss new ideas and to try them, I agreed to get him in my team. After that, we discussed what we have done so far for a few days and what we are planning to do. But every time when I and my other two teammates asked him about the experiments and it's results, he would come up with some argument to divert the topic.\n\nDuring the past few days before the merger deadline, We asked him about his experiments and results. The answer we received was that he had **deleted all the results** as they all failed 🤯.  So we asked him about his then **best model**, he said it's **the baseline he had before**. We all were pretty upset so, we asked him if he could at least try Ensembling model using different image-sizes and heavy augmentations. He agreed and before the merger deadline we let him in our team.\n\n**He stabbed us in the back.** He got **unresponsive** everywhere on social media and he did not respond to our daily messages. I also messaged one of his teammates in other ongoing competition but did not get anything about him. \n\nMaybe, he just wanted to get in a team to get a medal easily by doing nothing but training a baseline.\n\n**Our current rank and the silver medal is because of my and my other two teammates hard work. There is not even a little contribution from the forth teammate.**\n\n#### Things that lured me to take him in our team\n* His baseline score was higher than my then best score\n* He had a bronze medal in one previous competition\n* And he said he has access to an Unlimited GPU resource\n\n#### This was the biggest mistake I made during this competition and I will try to avoid such mistakes in future 😅\n\nSpecial thanks, to my teammates, @vatsalparsaniya & @geektoday 🔥✌🏻\n\nPeace ✌🏻",
      "votes": 7,
      "replies": [
        {
          "id": 975693,
          "postDate": "2020-08-18T12:27:27.443Z",
          "content": "<p>I learnt not to trust people on what they say or promise you so easily. now I'll never repeat my mistake again. \"<strong>Never team up with someone you don't know</strong>\".<br>\nappreciate your hard hard work <a href=\"https://www.kaggle.com/meemr5\" target=\"_blank\">@meemr5</a> &amp; <a href=\"https://www.kaggle.com/geektoday\" target=\"_blank\">@geektoday</a> in our first team competition.</p>",
          "rawMarkdown": "\nI learnt not to trust people on what they say or promise you so easily. now I'll never repeat my mistake again. \"**Never team up with someone you don't know**\".\n\nappreciate your hard hard work @meemr5 & @geektoday in our first team competition.",
          "votes": 2
        },
        {
          "id": 977041,
          "postDate": "2020-08-19T08:53:31.203Z",
          "content": "<p><a href=\"https://www.kaggle.com/meemr5\" target=\"_blank\">@meemr5</a> <a href=\"https://www.kaggle.com/vatsalparsaniya\" target=\"_blank\">@vatsalparsaniya</a> - Congrats on your Silver Medal… I would say its always better to verify twice or thrice before merging with any team member.. Do check if the particular person has a LinkedIn profile/Github profile/ decent activity in Kaggle before merging. With experience you guys would start choosing right members who will help you to reach higher stages. Teamwork make or break competition efforts.  All the best for your future competitions.. </p>",
          "rawMarkdown": "@meemr5 @vatsalparsaniya - Congrats on your Silver Medal... I would say its always better to verify twice or thrice before merging with any team member.. Do check if the particular person has a LinkedIn profile/Github profile/ decent activity in Kaggle before merging. With experience you guys would start choosing right members who will help you to reach higher stages. Teamwork make or break competition efforts.  All the best for your future competitions.. ",
          "votes": 2
        },
        {
          "id": 978293,
          "postDate": "2020-08-20T04:58:45.493Z",
          "content": "<p>Yeah, you are right, <a href=\"https://www.kaggle.com/manojprabhaakr\" target=\"_blank\">@manojprabhaakr</a>, I would keep that in mind! <br>\nAll the best to you too 👍 for your future competitions. <br>\nThanks ✌🏻</p>",
          "rawMarkdown": "Yeah, you are right, @manojprabhaakr, I would keep that in mind! \nAll the best to you too 👍 for your future competitions. \nThanks ✌🏻"
        }
      ]
    },
    {
      "id": 976858,
      "postDate": "2020-08-19T06:25:57.627Z",
      "content": "<p>Congrats for the silver and extra congrats for the \"gold in finding the virus\"!</p>",
      "rawMarkdown": "Congrats for the silver and extra congrats for the \"gold in finding the virus\"!",
      "votes": 3
    },
    {
      "id": 975601,
      "postDate": "2020-08-18T11:24:54.347Z",
      "content": "<p>We should all be very careful, there're some people who want to team up and then totally disappeared after getting into the team. I do believe some of them just take your high scoring submissions (or code) to share privately with some other teams or sell them. (I even saw competitors posting this: <a href=\"https://www.kaggle.com/c/global-wheat-detection/discussion/161296\" target=\"_blank\">https://www.kaggle.com/c/global-wheat-detection/discussion/161296</a>)</p>",
      "rawMarkdown": "We should all be very careful, there're some people who want to team up and then totally disappeared after getting into the team. I do believe some of them just take your high scoring submissions (or code) to share privately with some other teams or sell them. (I even saw competitors posting this: https://www.kaggle.com/c/global-wheat-detection/discussion/161296)",
      "votes": 3
    },
    {
      "id": 975415,
      "postDate": "2020-08-18T09:36:25.573Z",
      "content": "<blockquote>\n  <p><em>\"… quickly climb the public LB by some sketchy blending and then join a team to completely ghost them for the remainder of the competition\"</em></p>\n</blockquote>\n<p>I am new to the Featured competitions section of kaggle and I must say that I am surprised, and disillusioned,  by the underhand tactics used by some individuals to obtain medals. Your excellent posts over the last few days, as well as the multitude of insightful comments on them,  have been a real eye-opener. </p>\n<p>On a more positive note, congratulations for your teams well deserved silver medal, and thank you for posting indications regarding how it was achieved.</p>\n<p>All the best, <br>\ncarl</p>",
      "rawMarkdown": "> *\"... quickly climb the public LB by some sketchy blending and then join a team to completely ghost them for the remainder of the competition\"*\n\nI am new to the Featured competitions section of kaggle and I must say that I am surprised, and disillusioned,  by the underhand tactics used by some individuals to obtain medals. Your excellent posts over the last few days, as well as the multitude of insightful comments on them,  have been a real eye-opener. \n\nOn a more positive note, congratulations for your teams well deserved silver medal, and thank you for posting indications regarding how it was achieved.\n\nAll the best, \ncarl",
      "votes": 4,
      "replies": [
        {
          "id": 975443,
          "postDate": "2020-08-18T09:52:59.697Z",
          "content": "<p>Thank you Carl! </p>",
          "rawMarkdown": "Thank you Carl! "
        }
      ]
    },
    {
      "id": 976560,
      "postDate": "2020-08-19T00:27:11.813Z",
      "content": "<p>Congrats Gilles and team. Our final models look similar. My strategy was also</p>\n<blockquote>\n  <p>We ensured diversity by modifying (i) the input data, (ii) training scheme and (iii) model architecture:</p>\n</blockquote>",
      "rawMarkdown": "Congrats Gilles and team. Our final models look similar. My strategy was also\n\n> We ensured diversity by modifying (i) the input data, (ii) training scheme and (iii) model architecture:\n\n\n",
      "votes": 1,
      "replies": [
        {
          "id": 976813,
          "postDate": "2020-08-19T05:37:02.177Z",
          "content": "<p>Thanks <a href=\"https://www.kaggle.com/cdeotte\" target=\"_blank\">@cdeotte</a>! And thanks for creating many useful datasets during this competition :)</p>",
          "rawMarkdown": "Thanks @cdeotte! And thanks for creating many useful datasets during this competition :)",
          "votes": 1
        }
      ]
    },
    {
      "id": 975668,
      "postDate": "2020-08-18T12:10:43.453Z",
      "content": "<p>Nice finish <a href=\"https://www.kaggle.com/group16\" target=\"_blank\">@group16</a> , and thanks for your contributions to this competition.</p>\n<p>That ghost team member is very sad, this kind of thing is one of the reasons I am very skeptical of joining on teams, in my case I would only join at the first half of the competition or in case that I am out of ideas by the end, we can never know if the other people will work as hard as us, or if they will work at all.</p>",
      "rawMarkdown": "Nice finish @group16 , and thanks for your contributions to this competition.\n\nThat ghost team member is very sad, this kind of thing is one of the reasons I am very skeptical of joining on teams, in my case I would only join at the first half of the competition or in case that I am out of ideas by the end, we can never know if the other people will work as hard as us, or if they will work at all.",
      "votes": 2
    },
    {
      "id": 977915,
      "postDate": "2020-08-19T19:43:26.257Z",
      "content": "<p>congrats <a href=\"https://www.kaggle.com/group16\" target=\"_blank\">@group16</a> and your team !<br>\nwhat does \" Pseudo Labeling \" mean?<br>\ndo you mean \" Augmentation\" by \" modifying Input Data \" ?<br>\nand if not so what is it ??</p>",
      "rawMarkdown": "congrats @group16 and your team !\nwhat does \" Pseudo Labeling \" mean?\ndo you mean \" Augmentation\" by \" modifying Input Data \" ?\nand if not so what is it ??",
      "replies": [
        {
          "id": 978410,
          "postDate": "2020-08-20T06:50:36.183Z",
          "content": "<p>Hi Pawan.</p>\n<p>About the modifying input data: I was mostly hinting at using cropped versions of the images. Pseudo-labeling is when you make predictions with your model on the Kaggle test-data and then take the samples of which your predictions are very \"confident\" (i.e. probabilities both close to 0 and 1) and add these to your training data.</p>",
          "rawMarkdown": "Hi Pawan.\n\nAbout the modifying input data: I was mostly hinting at using cropped versions of the images. Pseudo-labeling is when you make predictions with your model on the Kaggle test-data and then take the samples of which your predictions are very \"confident\" (i.e. probabilities both close to 0 and 1) and add these to your training data.",
          "votes": 1
        },
        {
          "id": 978488,
          "postDate": "2020-08-20T08:06:34.753Z",
          "content": "<p>thanks for that clear explanation👍</p>",
          "rawMarkdown": "thanks for that clear explanation👍"
        }
      ]
    },
    {
      "id": 975969,
      "postDate": "2020-08-18T14:54:02.863Z",
      "rawMarkdown": "",
      "isDeleted": true,
      "replies": [
        {
          "id": 975976,
          "postDate": "2020-08-18T14:57:37.093Z",
          "content": "<p>Sorry, but something about your account is fishy. I may be wrong but it doesn't add up.</p>",
          "rawMarkdown": "Sorry, but something about your account is fishy. I may be wrong but it doesn't add up.",
          "votes": 1,
          "replies": [
            {
              "id": 976151,
              "postDate": "2020-08-18T17:06:16.563Z",
              "content": "<p><a href=\"https://www.kaggle.com/meemr5\" target=\"_blank\">@meemr5</a> your whole practice is against the rule . At the max ,you might have tried to go solo and told one of your friend that if nothing works out at the end we might team up . Thats it . There can't be exchange of ideas , discussions ,progress checking   etc in private channel without teaming up . </p>",
              "rawMarkdown": "@meemr5 your whole practice is against the rule . At the max ,you might have tried to go solo and told one of your friend that if nothing works out at the end we might team up . Thats it . There can't be exchange of ideas , discussions ,progress checking   etc in private channel without teaming up . ",
              "votes": 1
            },
            {
              "id": 976176,
              "postDate": "2020-08-18T17:29:23.300Z",
              "content": "<p>Thanks for drawing my attention to this, <br>\nI think 💭 private sharing of executable codes and submissions is not allowed. We can still discuss ideas with our teammates like we do in this discussion forum. </p>\n<p>That is why I still added him in my team in the end because he was a part of the team from the beginning.</p>\n<p>Thanks ✌🏻</p>",
              "rawMarkdown": "Thanks for drawing my attention to this, \nI think 💭 private sharing of executable codes and submissions is not allowed. We can still discuss ideas with our teammates like we do in this discussion forum. \n\nThat is why I still added him in my team in the end because he was a part of the team from the beginning.\n\nThanks ✌🏻"
            },
            {
              "id": 976290,
              "postDate": "2020-08-18T19:12:40.717Z",
              "content": "<p>It feels a bit of risky to do that with unknown people . And also I think somewhere I have read in kaggle rules or from organizers that discussions like this before merging is not the right practice . But  I cant really confirm on that , so you could be right as well .  Essentially by sharing the ideas and not merging you are able to do probably more experiment before merging , which might not be true for all the teams . </p>",
              "rawMarkdown": "It feels a bit of risky to do that with unknown people . And also I think somewhere I have read in kaggle rules or from organizers that discussions like this before merging is not the right practice . But  I cant really confirm on that , so you could be right as well .  Essentially by sharing the ideas and not merging you are able to do probably more experiment before merging , which might not be true for all the teams . ",
              "votes": 2
            }
          ]
        }
      ]
    }
  ],
  "comments": [
    {
      "id": 975495,
      "author_name": "Meet Ranoliya",
      "author_url": "",
      "post_date": "2020-08-18T10:17:11.890000",
      "content": "<p>Thanks for your approach, <a href=\"https://www.kaggle.com/group16\" target=\"_blank\">@group16</a>.</p>\n<h3>Our Story on Teammate issue</h3>\n<p>I would also like to mention about this <strong>teammate issue</strong> that happened with me in this competition.<br>\nWe have a teammate <strong>(XIAOWANG)</strong> who had a <strong>high baseline score</strong> when he approached me to team up &amp; just because of I wanted to learn and discuss new ideas and to try them, I agreed to get him in my team. After that, we discussed what we have done so far for a few days and what we are planning to do. But every time when I and my other two teammates asked him about the experiments and it's results, he would come up with some argument to divert the topic.</p>\n<p>During the past few days before the merger deadline, We asked him about his experiments and results. The answer we received was that he had <strong>deleted all the results</strong> as they all failed 🤯.  So we asked him about his then <strong>best model</strong>, he said it's <strong>the baseline he had before</strong>. We all were pretty upset so, we asked him if he could at least try Ensembling model using different image-sizes and heavy augmentations. He agreed and before the merger deadline we let him in our team.</p>\n<p><strong>He stabbed us in the back.</strong> He got <strong>unresponsive</strong> everywhere on social media and he did not respond to our daily messages. I also messaged one of his teammates in other ongoing competition but did not get anything about him. </p>\n<p>Maybe, he just wanted to get in a team to get a medal easily by doing nothing but training a baseline.</p>\n<p><strong>Our current rank and the silver medal is because of my and my other two teammates hard work. There is not even a little contribution from the forth teammate.</strong></p>\n<h4>Things that lured me to take him in our team</h4>\n<ul>\n<li>His baseline score was higher than my then best score</li>\n<li>He had a bronze medal in one previous competition</li>\n<li>And he said he has access to an Unlimited GPU resource</li>\n</ul>\n<h4>This was the biggest mistake I made during this competition and I will try to avoid such mistakes in future 😅</h4>\n<p>Special thanks, to my teammates, <a href=\"https://www.kaggle.com/vatsalparsaniya\" target=\"_blank\">@vatsalparsaniya</a> &amp; <a href=\"https://www.kaggle.com/geektoday\" target=\"_blank\">@geektoday</a> 🔥✌🏻</p>\n<p>Peace ✌🏻</p>",
      "votes": 7,
      "replies": [
        {
          "id": 975693,
          "author_name": "Vatsal Parsaniya",
          "author_url": "",
          "post_date": "2020-08-18T12:27:27.443000",
          "content": "<p>I learnt not to trust people on what they say or promise you so easily. now I'll never repeat my mistake again. \"<strong>Never team up with someone you don't know</strong>\".<br>\nappreciate your hard hard work <a href=\"https://www.kaggle.com/meemr5\" target=\"_blank\">@meemr5</a> &amp; <a href=\"https://www.kaggle.com/geektoday\" target=\"_blank\">@geektoday</a> in our first team competition.</p>",
          "votes": 2,
          "replies": []
        },
        {
          "id": 977041,
          "author_name": "Manoj Prabhakar",
          "author_url": "",
          "post_date": "2020-08-19T08:53:31.203000",
          "content": "<p><a href=\"https://www.kaggle.com/meemr5\" target=\"_blank\">@meemr5</a> <a href=\"https://www.kaggle.com/vatsalparsaniya\" target=\"_blank\">@vatsalparsaniya</a> - Congrats on your Silver Medal… I would say its always better to verify twice or thrice before merging with any team member.. Do check if the particular person has a LinkedIn profile/Github profile/ decent activity in Kaggle before merging. With experience you guys would start choosing right members who will help you to reach higher stages. Teamwork make or break competition efforts.  All the best for your future competitions.. </p>",
          "votes": 2,
          "replies": []
        },
        {
          "id": 978293,
          "author_name": "Meet Ranoliya",
          "author_url": "",
          "post_date": "2020-08-20T04:58:45.493000",
          "content": "<p>Yeah, you are right, <a href=\"https://www.kaggle.com/manojprabhaakr\" target=\"_blank\">@manojprabhaakr</a>, I would keep that in mind! <br>\nAll the best to you too 👍 for your future competitions. <br>\nThanks ✌🏻</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 976858,
      "author_name": "",
      "author_url": "",
      "post_date": "2020-08-19T06:25:57.627000",
      "content": "<p>Congrats for the silver and extra congrats for the \"gold in finding the virus\"!</p>",
      "votes": 3,
      "replies": []
    },
    {
      "id": 975601,
      "author_name": "Jie Lu",
      "author_url": "",
      "post_date": "2020-08-18T11:24:54.347000",
      "content": "<p>We should all be very careful, there're some people who want to team up and then totally disappeared after getting into the team. I do believe some of them just take your high scoring submissions (or code) to share privately with some other teams or sell them. (I even saw competitors posting this: <a href=\"https://www.kaggle.com/c/global-wheat-detection/discussion/161296\" target=\"_blank\">https://www.kaggle.com/c/global-wheat-detection/discussion/161296</a>)</p>",
      "votes": 3,
      "replies": []
    },
    {
      "id": 975415,
      "author_name": "Carl McBride Ellis",
      "author_url": "",
      "post_date": "2020-08-18T09:36:25.573000",
      "content": "<blockquote>\n  <p><em>\"… quickly climb the public LB by some sketchy blending and then join a team to completely ghost them for the remainder of the competition\"</em></p>\n</blockquote>\n<p>I am new to the Featured competitions section of kaggle and I must say that I am surprised, and disillusioned,  by the underhand tactics used by some individuals to obtain medals. Your excellent posts over the last few days, as well as the multitude of insightful comments on them,  have been a real eye-opener. </p>\n<p>On a more positive note, congratulations for your teams well deserved silver medal, and thank you for posting indications regarding how it was achieved.</p>\n<p>All the best, <br>\ncarl</p>",
      "votes": 4,
      "replies": [
        {
          "id": 975443,
          "author_name": "Gilles Vandewiele",
          "author_url": "",
          "post_date": "2020-08-18T09:52:59.697000",
          "content": "<p>Thank you Carl! </p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 976560,
      "author_name": "Chris Deotte",
      "author_url": "",
      "post_date": "2020-08-19T00:27:11.813000",
      "content": "<p>Congrats Gilles and team. Our final models look similar. My strategy was also</p>\n<blockquote>\n  <p>We ensured diversity by modifying (i) the input data, (ii) training scheme and (iii) model architecture:</p>\n</blockquote>",
      "votes": 1,
      "replies": [
        {
          "id": 976813,
          "author_name": "Gilles Vandewiele",
          "author_url": "",
          "post_date": "2020-08-19T05:37:02.177000",
          "content": "<p>Thanks <a href=\"https://www.kaggle.com/cdeotte\" target=\"_blank\">@cdeotte</a>! And thanks for creating many useful datasets during this competition :)</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 975668,
      "author_name": "DimitreOliveira",
      "author_url": "",
      "post_date": "2020-08-18T12:10:43.453000",
      "content": "<p>Nice finish <a href=\"https://www.kaggle.com/group16\" target=\"_blank\">@group16</a> , and thanks for your contributions to this competition.</p>\n<p>That ghost team member is very sad, this kind of thing is one of the reasons I am very skeptical of joining on teams, in my case I would only join at the first half of the competition or in case that I am out of ideas by the end, we can never know if the other people will work as hard as us, or if they will work at all.</p>",
      "votes": 2,
      "replies": []
    },
    {
      "id": 977915,
      "author_name": "Pawan KS",
      "author_url": "",
      "post_date": "2020-08-19T19:43:26.257000",
      "content": "<p>congrats <a href=\"https://www.kaggle.com/group16\" target=\"_blank\">@group16</a> and your team !<br>\nwhat does \" Pseudo Labeling \" mean?<br>\ndo you mean \" Augmentation\" by \" modifying Input Data \" ?<br>\nand if not so what is it ??</p>",
      "votes": 0,
      "replies": [
        {
          "id": 978410,
          "author_name": "Gilles Vandewiele",
          "author_url": "",
          "post_date": "2020-08-20T06:50:36.183000",
          "content": "<p>Hi Pawan.</p>\n<p>About the modifying input data: I was mostly hinting at using cropped versions of the images. Pseudo-labeling is when you make predictions with your model on the Kaggle test-data and then take the samples of which your predictions are very \"confident\" (i.e. probabilities both close to 0 and 1) and add these to your training data.</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 978488,
          "author_name": "Pawan KS",
          "author_url": "",
          "post_date": "2020-08-20T08:06:34.753000",
          "content": "<p>thanks for that clear explanation👍</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 975969,
      "author_name": "",
      "author_url": "",
      "post_date": "2020-08-18T14:54:02.863000",
      "content": "",
      "votes": 0,
      "replies": [
        {
          "id": 975976,
          "author_name": "FChmiel",
          "author_url": "",
          "post_date": "2020-08-18T14:57:37.093000",
          "content": "<p>Sorry, but something about your account is fishy. I may be wrong but it doesn't add up.</p>",
          "votes": 1,
          "replies": [
            {
              "id": 976151,
              "author_name": "Nirjhar Roy",
              "author_url": "",
              "post_date": "2020-08-18T17:06:16.563000",
              "content": "<p><a href=\"https://www.kaggle.com/meemr5\" target=\"_blank\">@meemr5</a> your whole practice is against the rule . At the max ,you might have tried to go solo and told one of your friend that if nothing works out at the end we might team up . Thats it . There can't be exchange of ideas , discussions ,progress checking   etc in private channel without teaming up . </p>",
              "votes": 1,
              "replies": []
            },
            {
              "id": 976176,
              "author_name": "Meet Ranoliya",
              "author_url": "",
              "post_date": "2020-08-18T17:29:23.300000",
              "content": "<p>Thanks for drawing my attention to this, <br>\nI think 💭 private sharing of executable codes and submissions is not allowed. We can still discuss ideas with our teammates like we do in this discussion forum. </p>\n<p>That is why I still added him in my team in the end because he was a part of the team from the beginning.</p>\n<p>Thanks ✌🏻</p>",
              "votes": 0,
              "replies": []
            },
            {
              "id": 976290,
              "author_name": "Nirjhar Roy",
              "author_url": "",
              "post_date": "2020-08-18T19:12:40.717000",
              "content": "<p>It feels a bit of risky to do that with unknown people . And also I think somewhere I have read in kaggle rules or from organizers that discussions like this before merging is not the right practice . But  I cant really confirm on that , so you could be right as well .  Essentially by sharing the ideas and not merging you are able to do probably more experiment before merging , which might not be true for all the teams . </p>",
              "votes": 2,
              "replies": []
            }
          ]
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "975369": "Hi everyone,\n\nThe competition has ended and it seemed that we didn't suffer too much from the shake-up.  We are quite happy about this result, because our best (non-selected) submission doesn't even come close to the gold zone. Clearly we missed some crucial tricks that the top competitors used to break 0.945+: \n* using the diagnosis as (auxiliary) target\n* pseudo-labeling\n* using all data (including external taking into account the many duplicates) in your cross-validation for more stable results.\n* ...\n\nFor what it's worth, I will provide a rather brief overview of our solution, which is a blend of different models. So nothing really fancy at all! We ensured diversity by modifying (i) the input data, (ii) training scheme and (iii) model architecture:\n* We used @chriscareaga his [cropping technique](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/171745) to create cropped versions of our images. There were some mistakes in the cropping and it didn't work very well for harder images (with lots of moles), but the results were quite OK and rather uncorrelated to our previous results. We also tried training on both the cropped and original version and then combining the image embeddings for the final linear layer.\n* Gradual training: first train with 256 images for a few epochs, then use those learned weights as initialization for a next round of training with 384 images, and so on. This upscaling was done until images of 768x768.\n* EfficientNetB7, EfficientNetB6 and ResNeSt worked quite well (in that order). EfficientNetB7 with images of 512x512 turned out to be our best model with CV 0.933 / private 0.9373 / public 0.9570. For EfficientNet, we also had some more luck with initializing with AdvProp weights instead of the default imagenet ones.\n\nWe then used scipy.minimize to find a weighted average for the ranks for each of these submissions. In the end, we also blended with a submission made by @ruozha001 to get quite a significant boost in public LB (and slightly on private).\n\nOur 0.9729 submission was a blending mess, and we never expected it to do well. Looks like our expectations were right as all of those blends scores < 0.935 on private. \n\nI would like to thank my teammates @chriscareaga, @bsteenwi and @khahuras for this fun competition! And congrats to @bsteenwi on probably becoming Competition Expert! On to the next!\n\n\nPS:\nI already briefly spoke about  @ruozha001 's behaviour in [my topic](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/175156) yesterday, but I'd like to further elaborate upon this issue. Yesterday, he/she came online for the first time in 2 weeks (after seeing the post) and we discussed with him/her how he/she actually obtained the solution. We also mailed some of her former teammates. **I do not believe or have any conclusive evidence he/she is involved in any private sharing.** Rather, it seems that this person's tactic is to quickly climb the public LB by some sketchy blending and then join a team to completely ghost them for the remainder of the competition. So let this be a warning for others (as I've learned the lesson myself now): is to **always be very critical before teaming up and make sure to ask enough questions so that you are convinced that this person will actually have a contribution to the team.**",
    "975495": "Thanks for your approach, @group16.\n\n### Our Story on Teammate issue\n\nI would also like to mention about this **teammate issue** that happened with me in this competition.\nWe have a teammate **(XIAOWANG)** who had a **high baseline score** when he approached me to team up & just because of I wanted to learn and discuss new ideas and to try them, I agreed to get him in my team. After that, we discussed what we have done so far for a few days and what we are planning to do. But every time when I and my other two teammates asked him about the experiments and it's results, he would come up with some argument to divert the topic.\n\nDuring the past few days before the merger deadline, We asked him about his experiments and results. The answer we received was that he had **deleted all the results** as they all failed 🤯.  So we asked him about his then **best model**, he said it's **the baseline he had before**. We all were pretty upset so, we asked him if he could at least try Ensembling model using different image-sizes and heavy augmentations. He agreed and before the merger deadline we let him in our team.\n\n**He stabbed us in the back.** He got **unresponsive** everywhere on social media and he did not respond to our daily messages. I also messaged one of his teammates in other ongoing competition but did not get anything about him. \n\nMaybe, he just wanted to get in a team to get a medal easily by doing nothing but training a baseline.\n\n**Our current rank and the silver medal is because of my and my other two teammates hard work. There is not even a little contribution from the forth teammate.**\n\n#### Things that lured me to take him in our team\n* His baseline score was higher than my then best score\n* He had a bronze medal in one previous competition\n* And he said he has access to an Unlimited GPU resource\n\n#### This was the biggest mistake I made during this competition and I will try to avoid such mistakes in future 😅\n\nSpecial thanks, to my teammates, @vatsalparsaniya & @geektoday 🔥✌🏻\n\nPeace ✌🏻",
    "976858": "Congrats for the silver and extra congrats for the \"gold in finding the virus\"!",
    "975601": "We should all be very careful, there're some people who want to team up and then totally disappeared after getting into the team. I do believe some of them just take your high scoring submissions (or code) to share privately with some other teams or sell them. (I even saw competitors posting this: https://www.kaggle.com/c/global-wheat-detection/discussion/161296)",
    "975415": "> *\"... quickly climb the public LB by some sketchy blending and then join a team to completely ghost them for the remainder of the competition\"*\n\nI am new to the Featured competitions section of kaggle and I must say that I am surprised, and disillusioned,  by the underhand tactics used by some individuals to obtain medals. Your excellent posts over the last few days, as well as the multitude of insightful comments on them,  have been a real eye-opener. \n\nOn a more positive note, congratulations for your teams well deserved silver medal, and thank you for posting indications regarding how it was achieved.\n\nAll the best, \ncarl",
    "976560": "Congrats Gilles and team. Our final models look similar. My strategy was also\n\n> We ensured diversity by modifying (i) the input data, (ii) training scheme and (iii) model architecture:\n\n\n",
    "975668": "Nice finish @group16 , and thanks for your contributions to this competition.\n\nThat ghost team member is very sad, this kind of thing is one of the reasons I am very skeptical of joining on teams, in my case I would only join at the first half of the competition or in case that I am out of ideas by the end, we can never know if the other people will work as hard as us, or if they will work at all.",
    "977915": "congrats @group16 and your team !\nwhat does \" Pseudo Labeling \" mean?\ndo you mean \" Augmentation\" by \" modifying Input Data \" ?\nand if not so what is it ??",
    "975969": ""
  }
}