First of all, thanks to the Kaggle community and the organizers.
\nI learned a lot through the competition.
Some people suspect me to be a bot, so I will provide 4th place solution.
\nCertainly, I can't hide my surprise at this result.
\nBut I'm a little disappointed with the skepticism of the competition itself.
\nI hope the contest is held in our good faith.
I could not submit it many times, of course, because the participation period was not long.
\nIn addition, the model was selected based on the CV results.
\nCV was done with multiple resolutions, and 384*384 was the best.
\nAlso, When ensemble with the best scoring model,
\nI was able to efficiently raise the score by performing the ensemble at the rate that the difference from the result after the ensemble became the minimum.
\nBy using this method, we were able to exclude the ones with very bad differences.
\nHowever, this method also carries the risk of overfitting public data.
\nI think there is also a risk in ignoring CV results and increasing the number of applications and raising the score.
\nIn fact, the 4th private score I got wasn't the best model in public.
And I think the main factors to win are:
\nI have found that some of the prediction results of the CV model predict a high positive rate, whereas many kernels predict a low positive rate.
\nI thought this was a false negative in many published models.
I finally submitted the following 3 models.
\n(1) Best score
\n(2) Ensemble with best score and model considered in CV
\n(3) Due to risk of overfitting, ensemble with model other than the best score and model considered in CV
As a result, Model (3) was the 4th place result. and, (1) was the worst.
\nFrom this result, the following can be said.
\nLike many kernel predictive models, the best-scoring model are also models with low sensitivity and is more likely to predict false negatives.
\nThe highest scoring model fits public data too much.
I think the following are valid:
\nTrust CV as many claims show
\nWe also observe trends by comparing with the results of many models that have a low correlation with the results examined in CV.
Of course, it is very lucky that the model (3) got the 4th place result.
\nI am aware of my lack of skills. However, I've found the fun of kaggle.
\nI will continue to deepen my learning through other contests.
Best Regards
", "messages": [ { "id": "977593", "postDate": "08/19/2020 15:29:43", "content": "First of all, thanks to the Kaggle community and the organizers.
\nI learned a lot through the competition.
Some people suspect me to be a bot, so I will provide 4th place solution.
\nCertainly, I can't hide my surprise at this result.
\nBut I'm a little disappointed with the skepticism of the competition itself.
\nI hope the contest is held in our good faith.
I could not submit it many times, of course, because the participation period was not long.
\nIn addition, the model was selected based on the CV results.
\nCV was done with multiple resolutions, and 384*384 was the best.
\nAlso, When ensemble with the best scoring model,
\nI was able to efficiently raise the score by performing the ensemble at the rate that the difference from the result after the ensemble became the minimum.
\nBy using this method, we were able to exclude the ones with very bad differences.
\nHowever, this method also carries the risk of overfitting public data.
\nI think there is also a risk in ignoring CV results and increasing the number of applications and raising the score.
\nIn fact, the 4th private score I got wasn't the best model in public.
And I think the main factors to win are:
\nI have found that some of the prediction results of the CV model predict a high positive rate, whereas many kernels predict a low positive rate.
\nI thought this was a false negative in many published models.
I finally submitted the following 3 models.
\n(1) Best score
\n(2) Ensemble with best score and model considered in CV
\n(3) Due to risk of overfitting, ensemble with model other than the best score and model considered in CV
As a result, Model (3) was the 4th place result. and, (1) was the worst.
\nFrom this result, the following can be said.
\nLike many kernel predictive models, the best-scoring model are also models with low sensitivity and is more likely to predict false negatives.
\nThe highest scoring model fits public data too much.
I think the following are valid:
\nTrust CV as many claims show
\nWe also observe trends by comparing with the results of many models that have a low correlation with the results examined in CV.
Of course, it is very lucky that the model (3) got the 4th place result.
\nI am aware of my lack of skills. However, I've found the fun of kaggle.
\nI will continue to deepen my learning through other contests.
Best Regards
", "rawMarkdown": "First of all, thanks to the Kaggle community and the organizers.\nI learned a lot through the competition.\n\nSome people suspect me to be a bot, so I will provide 4th place solution.\nCertainly, I can't hide my surprise at this result.\nBut I'm a little disappointed with the skepticism of the competition itself.\nI hope the contest is held in our good faith.\n\nI could not submit it many times, of course, because the participation period was not long.\nIn addition, the model was selected based on the CV results.\nCV was done with multiple resolutions, and 384*384 was the best.\nAlso, When ensemble with the best scoring model,\nI was able to efficiently raise the score by performing the ensemble at the rate that the difference from the result after the ensemble became the minimum.\nBy using this method, we were able to exclude the ones with very bad differences.\nHowever, this method also carries the risk of overfitting public data.\nI think there is also a risk in ignoring CV results and increasing the number of applications and raising the score.\nIn fact, the 4th private score I got wasn't the best model in public.\n\nAnd I think the main factors to win are:\nI have found that some of the prediction results of the CV model predict a high positive rate, whereas many kernels predict a low positive rate.\nI thought this was a false negative in many published models.\n\nI finally submitted the following 3 models.\n(1) Best score\n(2) Ensemble with best score and model considered in CV\n(3) Due to risk of overfitting, ensemble with model other than the best score and model considered in CV \n\nAs a result, Model (3) was the 4th place result. and, (1) was the worst.\nFrom this result, the following can be said.\nLike many kernel predictive models, the best-scoring model are also models with low sensitivity and is more likely to predict false negatives.\nThe highest scoring model fits public data too much.\n\nI think the following are valid:\nTrust CV as many claims show\nWe also observe trends by comparing with the results of many models that have a low correlation with the results examined in CV.\n\nOf course, it is very lucky that the model (3) got the 4th place result.\n\nI am aware of my lack of skills. However, I've found the fun of kaggle.\nI will continue to deepen my learning through other contests.\n\nBest Regards", "votes": null }, { "id": "977623", "postDate": "08/19/2020 15:46:59", "content": "Honestly, you did not convince me not to be a bot! 😂
\nJust kidding, did not get your solution but congrats on your gold medal anyway!
", "rawMarkdown": "Honestly, you did not convince me not to be a bot! 😂\n\nJust kidding, did not get your solution but congrats on your gold medal anyway!", "votes": null }, { "id": "977641", "postDate": "08/19/2020 15:57:47", "content": "Omedetou Yuya-san, keep kaggling
", "rawMarkdown": "Omedetou Yuya-san, keep kaggling", "votes": null }, { "id": "977674", "postDate": "08/19/2020 16:29:23", "content": "congrats @atagiyuya and I hope to see you other competitions.
", "rawMarkdown": "congrats @atagiyuya and I hope to see you other competitions.", "votes": null }, { "id": "977714", "postDate": "08/19/2020 16:59:45", "content": "Congrats, man. I don't think you're a bot anymore. Next time, no one will think you're a bot. Good luck!
", "rawMarkdown": "Congrats, man. I don't think you're a bot anymore. Next time, no one will think you're a bot. Good luck!", "votes": null }, { "id": "977825", "postDate": "08/19/2020 18:16:05", "content": "Congrats! Very impressive with so few submissions! May I ask a question, please?
\nAlso, When ensemble with the best scoring model, I was able to efficiently raise the score by performing the ensemble at the rate that the difference from the result after the ensemble became the minimum. By using this method, we were able to exclude the ones with very bad differences.
Would you mind explained a little bit about this part? How do you measure differences? By variance? Could do provide any formulations or codes? Thanks!
", "rawMarkdown": "Congrats! Very impressive with so few submissions! May I ask a question, please?\n\n`Also, When ensemble with the best scoring model, I was able to efficiently raise the score by performing the ensemble at the rate that the difference from the result after the ensemble became the minimum. By using this method, we were able to exclude the ones with very bad differences.`\n\nWould you mind explained a little bit about this part? How do you measure differences? By variance? Could do provide any formulations or codes? Thanks!", "votes": null }, { "id": "978092", "postDate": "08/19/2020 23:45:35", "content": "Congratulations with the gold!
", "rawMarkdown": "Congratulations with the gold!", "votes": null }, { "id": "978788", "postDate": "08/20/2020 11:42:59", "content": "Thank you everyone for celebrating me.
\nTo Helen-san
\nThank you for your comment.
\nI'm sorry to explain a very simple method.
\nTo use this method, you first need to find one prediction that contributed to best score model by the ensemble.
\nThat prediction is y.Then try to run the following code, where x is the prediction of the model you want to ensemble.
\nThat's how the models that helped to best score model are improved and help to score further.
`import pandas as pd
\nimport numpy as np
\nfrom statistics import mean
x=pd.read_csv(\"submission_x.csv\")[\"target\"]
\ny=pd.read_csv(\"submission_y.csv\")[\"target\"]
\nz=pd.read_csv(\"submission_best.csv\")[\"target\"]
df=pd.read_csv(\"sample_submission.csv\")
\nbest=len(x)
\nfor a in np.arange(0, 1, 0.1):
\n b=1-a
\n c=(ax+by)
\n d=abs(z-c)/z
\n res=mean(d)
\n if res <best:
\n best=res
\n beat_a=a
\n beat_b=b
\n best_c=c
if best>0.5:
\n print(\"Do not recommend ensemble\")
\nelse:
\n best_pre=z0.9+best_c0.1
\n df[\"target\"]=best_pre
\n df.to_csv(\"submission.csv\", index=False)
\nprint(best)
\nprint(f'a: {beat_a}')
\nprint(f'b: {beat_b}')`
# I think I understand what's going on here.\nx = New Ensemble candidate\ny = Strongest single model's predictions\nz = Highest Scoring Current CV (or LB) Ensemble\nIs that accurate?
", "rawMarkdown": "```\n# I think I understand what's going on here.\nx = New Ensemble candidate\ny = Strongest single model's predictions\nz = Highest Scoring Current CV (or LB) Ensemble\n```\n\nIs that accurate?", "votes": null }, { "id": "978845", "postDate": "08/20/2020 12:53:13", "content": "Hi, you don't give much details. What data do you use? Images only? meta data? Do you use any external data? What image sizes?
\nYou don't describe what models you use. Efficientnets? What model sizes? What loss function?
\nEtc.
\nFrankly, I don't know what one can learn from your writeup.
", "rawMarkdown": "Hi, you don't give much details. What data do you use? Images only? meta data? Do you use any external data? What image sizes?\n\nYou don't describe what models you use. Efficientnets? What model sizes? What loss function?\n\nEtc.\n\nFrankly, I don't know what one can learn from your writeup.", "votes": null }, { "id": "979163", "postDate": "08/20/2020 17:02:13", "content": "You made me notice a lot. Thank you very much.
\nEach parameter I used is:
\nDataSet:2017-2018-2019 + 2020 TFrecords (thanks to @cdeotte)
\nImage_size:384*384
\nModel:efficientnet-b6
\nLoss:binary_crossentropy
\nOptimizer:Adam
\nSeed:42
\nTTA:15
\nModel building:
\nWith reference to *https://www.kaggle.com/ajaykumar7778/efficientnet-cv (thanks to @ajaykumar7778)
\n x = base(keras.layers.Input(shape=(384,384,3)))
\n x = keras.layers.GlobalAveragePooling2D()(x)
\n x = keras.layers.Dropout(0.3)(x)
\n x = keras.layers.Dense(1024)(x)
\n x = keras.layers.Dropout(0.2)(x)
\n x = keras.layers.Dense(512)(x)
\n x = keras.layers.Dropout(0.4)(x)
\n x = keras.layers.Dense(256)(x)
\n x = keras.layers.Dropout(0.3)(x)
\n x = keras.layers.Dense(128)(x)
\n x = keras.layers.Dense(1)(x)
\n x = keras.layers.Activation('sigmoid', dtype='float32')(x)
By blending with *https://www.kaggle.com/paklau9/minmax-highest-public-lb-9619, Scored 0.9596 public LB and 0.9476 private LB. (thanks to@Pak Lau9)
\nI believe that one of the reason is probably that the R2 score is less than 0.6 between the ensemble models, the correlation is not large, and the generalization performance is high.
Almost the same interpretation
\ny does not have to be a single model, but it is not recommended if the R2 score between y and z is high (R2> 0.7).
\nOn the other hand, choosing y with a higher R2 may increase the public score, but at the risk of overfitting the public score.
\nIf you find a model where R2 between y and z is between 0.5 and 0.7 and can contribute to the public score, your private score will be guaranteed too.
thanks, this is much better.
", "rawMarkdown": "thanks, this is much better.", "votes": null }, { "id": "982710", "postDate": "08/23/2020 15:46:57", "content": "@atagiyuya Thanks for your explanation. Sorry for the late reply, i missed it. Now I understand, it's really a good idea. Learned a lot! 👍
", "rawMarkdown": "atagiyuya Thanks for your explanation. Sorry for the late reply, i missed it. Now I understand, it's really a good idea. Learned a lot! 👍", "votes": null }, { "id": "2917812", "postDate": "07/11/2024 19:49:58", "content": "Hello Atagi Yuya :)
\nI figured given your excellent performance in the last (2020 SIIM-ISIC) Melanoma challenge, I'd just point out we are running our next challenge with Kaggle:
\nhttps://www.kaggle.com/competitions/isic-2024-challenge/overview
\nWould be fun to see you join!
", "rawMarkdown": "Hello Atagi Yuya :)\n\nI figured given your excellent performance in the last (2020 SIIM-ISIC) Melanoma challenge, I'd just point out we are running our next challenge with Kaggle:\n\nhttps://www.kaggle.com/competitions/isic-2024-challenge/overview\n\nWould be fun to see you join!", "votes": null } ], "comments": [ { "id": 977623, "author_name": "optimo", "author_url": "", "post_date": "08/19/2020 15:46:59", "content": "Honestly, you did not convince me not to be a bot! 😂
\nJust kidding, did not get your solution but congrats on your gold medal anyway!
", "votes": null, "replies": [] }, { "id": 977641, "author_name": "authman", "author_url": "", "post_date": "08/19/2020 15:57:47", "content": "Omedetou Yuya-san, keep kaggling
", "votes": null, "replies": [] }, { "id": 977674, "author_name": "piantic", "author_url": "", "post_date": "08/19/2020 16:29:23", "content": "congrats @atagiyuya and I hope to see you other competitions.
", "votes": null, "replies": [] }, { "id": 977714, "author_name": "vadimtimakin", "author_url": "", "post_date": "08/19/2020 16:59:45", "content": "Congrats, man. I don't think you're a bot anymore. Next time, no one will think you're a bot. Good luck!
", "votes": null, "replies": [] }, { "id": 977825, "author_name": "yuanlin08", "author_url": "", "post_date": "08/19/2020 18:16:05", "content": "Congrats! Very impressive with so few submissions! May I ask a question, please?
\nAlso, When ensemble with the best scoring model, I was able to efficiently raise the score by performing the ensemble at the rate that the difference from the result after the ensemble became the minimum. By using this method, we were able to exclude the ones with very bad differences.
Would you mind explained a little bit about this part? How do you measure differences? By variance? Could do provide any formulations or codes? Thanks!
", "votes": null, "replies": [] }, { "id": 978092, "author_name": "antaresnyc", "author_url": "", "post_date": "08/19/2020 23:45:35", "content": "Congratulations with the gold!
", "votes": null, "replies": [] }, { "id": 978788, "author_name": "atagiyuya", "author_url": "", "post_date": "08/20/2020 11:42:59", "content": "Thank you everyone for celebrating me.
\nTo Helen-san
\nThank you for your comment.
\nI'm sorry to explain a very simple method.
\nTo use this method, you first need to find one prediction that contributed to best score model by the ensemble.
\nThat prediction is y.Then try to run the following code, where x is the prediction of the model you want to ensemble.
\nThat's how the models that helped to best score model are improved and help to score further.
`import pandas as pd
\nimport numpy as np
\nfrom statistics import mean
x=pd.read_csv(\"submission_x.csv\")[\"target\"]
\ny=pd.read_csv(\"submission_y.csv\")[\"target\"]
\nz=pd.read_csv(\"submission_best.csv\")[\"target\"]
df=pd.read_csv(\"sample_submission.csv\")
\nbest=len(x)
\nfor a in np.arange(0, 1, 0.1):
\n b=1-a
\n c=(ax+by)
\n d=abs(z-c)/z
\n res=mean(d)
\n if res <best:
\n best=res
\n beat_a=a
\n beat_b=b
\n best_c=c
if best>0.5:
\n print(\"Do not recommend ensemble\")
\nelse:
\n best_pre=z0.9+best_c0.1
\n df[\"target\"]=best_pre
\n df.to_csv(\"submission.csv\", index=False)
\nprint(best)
\nprint(f'a: {beat_a}')
\nprint(f'b: {beat_b}')`
# I think I understand what's going on here.\nx = New Ensemble candidate\ny = Strongest single model's predictions\nz = Highest Scoring Current CV (or LB) Ensemble\nIs that accurate?
", "votes": null, "replies": [] }, { "id": 979171, "author_name": "atagiyuya", "author_url": "", "post_date": "08/20/2020 17:06:59", "content": "Almost the same interpretation
\ny does not have to be a single model, but it is not recommended if the R2 score between y and z is high (R2> 0.7).
\nOn the other hand, choosing y with a higher R2 may increase the public score, but at the risk of overfitting the public score.
\nIf you find a model where R2 between y and z is between 0.5 and 0.7 and can contribute to the public score, your private score will be guaranteed too.
@atagiyuya Thanks for your explanation. Sorry for the late reply, i missed it. Now I understand, it's really a good idea. Learned a lot! 👍
", "votes": null, "replies": [] } ] }, { "id": 978845, "author_name": "cpmpml", "author_url": "", "post_date": "08/20/2020 12:53:13", "content": "Hi, you don't give much details. What data do you use? Images only? meta data? Do you use any external data? What image sizes?
\nYou don't describe what models you use. Efficientnets? What model sizes? What loss function?
\nEtc.
\nFrankly, I don't know what one can learn from your writeup.
", "votes": null, "replies": [ { "id": 979163, "author_name": "atagiyuya", "author_url": "", "post_date": "08/20/2020 17:02:13", "content": "You made me notice a lot. Thank you very much.
\nEach parameter I used is:
\nDataSet:2017-2018-2019 + 2020 TFrecords (thanks to @cdeotte)
\nImage_size:384*384
\nModel:efficientnet-b6
\nLoss:binary_crossentropy
\nOptimizer:Adam
\nSeed:42
\nTTA:15
\nModel building:
\nWith reference to *https://www.kaggle.com/ajaykumar7778/efficientnet-cv (thanks to @ajaykumar7778)
\n x = base(keras.layers.Input(shape=(384,384,3)))
\n x = keras.layers.GlobalAveragePooling2D()(x)
\n x = keras.layers.Dropout(0.3)(x)
\n x = keras.layers.Dense(1024)(x)
\n x = keras.layers.Dropout(0.2)(x)
\n x = keras.layers.Dense(512)(x)
\n x = keras.layers.Dropout(0.4)(x)
\n x = keras.layers.Dense(256)(x)
\n x = keras.layers.Dropout(0.3)(x)
\n x = keras.layers.Dense(128)(x)
\n x = keras.layers.Dense(1)(x)
\n x = keras.layers.Activation('sigmoid', dtype='float32')(x)
By blending with *https://www.kaggle.com/paklau9/minmax-highest-public-lb-9619, Scored 0.9596 public LB and 0.9476 private LB. (thanks to@Pak Lau9)
\nI believe that one of the reason is probably that the R2 score is less than 0.6 between the ensemble models, the correlation is not large, and the generalization performance is high.
thanks, this is much better.
", "votes": null, "replies": [] } ] }, { "id": 2917812, "author_name": "jwebermsk", "author_url": "", "post_date": "07/11/2024 19:49:58", "content": "Hello Atagi Yuya :)
\nI figured given your excellent performance in the last (2020 SIIM-ISIC) Melanoma challenge, I'd just point out we are running our next challenge with Kaggle:
\nhttps://www.kaggle.com/competitions/isic-2024-challenge/overview
\nWould be fun to see you join!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "977593": "First of all, thanks to the Kaggle community and the organizers.\nI learned a lot through the competition.\n\nSome people suspect me to be a bot, so I will provide 4th place solution.\nCertainly, I can't hide my surprise at this result.\nBut I'm a little disappointed with the skepticism of the competition itself.\nI hope the contest is held in our good faith.\n\nI could not submit it many times, of course, because the participation period was not long.\nIn addition, the model was selected based on the CV results.\nCV was done with multiple resolutions, and 384*384 was the best.\nAlso, When ensemble with the best scoring model,\nI was able to efficiently raise the score by performing the ensemble at the rate that the difference from the result after the ensemble became the minimum.\nBy using this method, we were able to exclude the ones with very bad differences.\nHowever, this method also carries the risk of overfitting public data.\nI think there is also a risk in ignoring CV results and increasing the number of applications and raising the score.\nIn fact, the 4th private score I got wasn't the best model in public.\n\nAnd I think the main factors to win are:\nI have found that some of the prediction results of the CV model predict a high positive rate, whereas many kernels predict a low positive rate.\nI thought this was a false negative in many published models.\n\nI finally submitted the following 3 models.\n(1) Best score\n(2) Ensemble with best score and model considered in CV\n(3) Due to risk of overfitting, ensemble with model other than the best score and model considered in CV \n\nAs a result, Model (3) was the 4th place result. and, (1) was the worst.\nFrom this result, the following can be said.\nLike many kernel predictive models, the best-scoring model are also models with low sensitivity and is more likely to predict false negatives.\nThe highest scoring model fits public data too much.\n\nI think the following are valid:\nTrust CV as many claims show\nWe also observe trends by comparing with the results of many models that have a low correlation with the results examined in CV.\n\nOf course, it is very lucky that the model (3) got the 4th place result.\n\nI am aware of my lack of skills. However, I've found the fun of kaggle.\nI will continue to deepen my learning through other contests.\n\nBest Regards", "977623": "Honestly, you did not convince me not to be a bot! 😂\n\nJust kidding, did not get your solution but congrats on your gold medal anyway!", "977641": "Omedetou Yuya-san, keep kaggling", "977674": "congrats @atagiyuya and I hope to see you other competitions.", "977714": "Congrats, man. I don't think you're a bot anymore. Next time, no one will think you're a bot. Good luck!", "977825": "Congrats! Very impressive with so few submissions! May I ask a question, please?\n\n`Also, When ensemble with the best scoring model, I was able to efficiently raise the score by performing the ensemble at the rate that the difference from the result after the ensemble became the minimum. By using this method, we were able to exclude the ones with very bad differences.`\n\nWould you mind explained a little bit about this part? How do you measure differences? By variance? Could do provide any formulations or codes? Thanks!", "978092": "Congratulations with the gold!", "978788": "Thank you everyone for celebrating me.\n\nTo Helen-san\nThank you for your comment.\nI'm sorry to explain a very simple method.\nTo use this method, you first need to find one prediction that contributed to best score model by the ensemble.\nThat prediction is y.Then try to run the following code, where x is the prediction of the model you want to ensemble.\nThat's how the models that helped to best score model are improved and help to score further.\n\n`import pandas as pd\nimport numpy as np\nfrom statistics import mean\n\nx=pd.read_csv(\"submission_x.csv\")[\"target\"]\ny=pd.read_csv(\"submission_y.csv\")[\"target\"]\nz=pd.read_csv(\"submission_best.csv\")[\"target\"]\n#x:Prediction you want to ensemble\n#y:Prediction that contributed to the best prediction\n#z:Best prediction\ndf=pd.read_csv(\"sample_submission.csv\")\n\nbest=len(x)\nfor a in np.arange(0, 1, 0.1):\n b=1-a\n c=(a*x+b*y)\n d=abs(z-c)/z\n res=mean(d)\n if res