That's the joy of joining late. You can sit on shoulders of giants, and you can make progress much faster. But...
\n\nList of things I haven't tried yet.
\n\nWhen I see this list I'm tempted to move to my next competition... At the same time I welcome additional suggestions. I must be a masochist.
\n\nWhat pushes me is that many of you are in same situation probably, with too many things to try given the time left.
", "messages": [ { "id": 965703, "postDate": "2020-08-10T19:30:25.800Z", "content": "That's the joy of joining late. You can sit on shoulders of giants, and you can make progress much faster. But...
\n\nList of things I haven't tried yet.
\n\nWhen I see this list I'm tempted to move to my next competition... At the same time I welcome additional suggestions. I must be a masochist.
\n\nWhat pushes me is that many of you are in same situation probably, with too many things to try given the time left.
", "rawMarkdown": "That's the joy of joining late. You can sit on shoulders of giants, and you can make progress much faster. But...\n\nList of things I haven't tried yet. \n\n- image size other than 256x256\n- hair augmentation\n- upsampling ( I tried but don't get any benefit yet)\n- other model than effnetb0 from efficientnet-pytorch\n- ensembling other than average\n- use of tabular data per sample\n- use of patient context across samples\n- look at public notebooks\n\nWhen I see this list I'm tempted to move to my next competition... At the same time I welcome additional suggestions. I must be a masochist. \n\nWhat pushes me is that many of you are in same situation probably, with too many things to try given the time left.", "votes": 55 }, { "id": 965719, "postDate": "2020-08-10T19:43:59.067Z", "content": "I've tried something new everyday for the past 60 days! And I've read the results from everyone else's notebooks and discussions. I still have a list that I will not finish before the comp ends! And I just read an article yesterday that gave me 3 new promising ideas! There is never enough time!
", "rawMarkdown": "I've tried something new everyday for the past 60 days! And I've read the results from everyone else's notebooks and discussions. I still have a list that I will not finish before the comp ends! And I just read an article yesterday that gave me 3 new promising ideas! There is never enough time!", "votes": 22, "replies": [ { "id": 965721, "postDate": "2020-08-10T19:46:33.017Z", "content": "Indeed!
", "rawMarkdown": "Indeed!", "votes": 4 }, { "id": 965797, "postDate": "2020-08-10T21:28:15.217Z", "content": "Any plans to share some recipes from the list after the competition ends? 😍
", "rawMarkdown": "Any plans to share some recipes from the list after the competition ends? 😍" }, { "id": 965822, "postDate": "2020-08-10T21:58:58.387Z", "content": "I always share after competition end if my result is not too bad.
\nEdit. Maybe the questions was for Chris actually. I must adjust to times where I am no longer the most popular guy ;)
", "rawMarkdown": "I always share after competition end if my result is not too bad.\n\nEdit. Maybe the questions was for Chris actually. I must adjust to times where I am no longer the most popular guy ;)", "votes": 17 } ] }, { "id": 965840, "postDate": "2020-08-10T22:37:29.003Z", "content": "I now have 2 serial downvoters. I feel for them.
\nActually I don't. But I'll stay polite.
", "rawMarkdown": "I now have 2 serial downvoters. I feel for them.\n\nActually I don't. But I'll stay polite.", "votes": 6, "replies": [ { "id": 966019, "postDate": "2020-08-11T04:00:47.047Z", "content": "Seems like you guys are having a great relationship 👀, that guy is never going to leave your back! 😁
", "rawMarkdown": "Seems like you guys are having a great relationship 👀, that guy is never going to leave your back! 😁" }, { "id": 966294, "postDate": "2020-08-11T10:08:29.583Z", "content": "There is more than one now.
", "rawMarkdown": "There is more than one now.", "votes": 1 }, { "id": 966306, "postDate": "2020-08-11T10:13:58.620Z", "content": "Seems like your negative fan base is increasing!
\nAlthough, on a serious note, I guess its time for Kaggle team to peek into who is this person and ask what he did not like in each post of yours which everyone else is appreciating, although this person in my intuition do not deserve this much attention too.
", "rawMarkdown": "Seems like your negative fan base is increasing!\n\nAlthough, on a serious note, I guess its time for Kaggle team to peek into who is this person and ask what he did not like in each post of yours which everyone else is appreciating, although this person in my intuition do not deserve this much attention too.", "votes": 1 }, { "id": 967026, "postDate": "2020-08-11T21:14:19.493Z", "content": "You're so funny! 😄
", "rawMarkdown": "You're so funny! 😄", "votes": 2 }, { "id": 967027, "postDate": "2020-08-11T21:14:20.977Z", "rawMarkdown": "", "isDeleted": true }, { "id": 967109, "postDate": "2020-08-12T00:23:29.497Z", "content": "Honestly, I really think it's lame that it doesn't show who upvotes/downvotes things. What's the point of anonymity? I get emails when people upvote, but apparently that doesn't happen for downvotes. I don't think that's a good policy and its ripe for abuse.
", "rawMarkdown": "Honestly, I really think it's lame that it doesn't show who upvotes/downvotes things. What's the point of anonymity? I get emails when people upvote, but apparently that doesn't happen for downvotes. I don't think that's a good policy and its ripe for abuse." }, { "id": 967716, "postDate": "2020-08-12T13:02:31.447Z", "content": "\n\nYou're so funny! 😄
\n
Who is funny, me? Why?
", "rawMarkdown": "> You're so funny! 😄\n\nWho is funny, me? Why?" }, { "id": 968095, "postDate": "2020-08-12T17:40:04.310Z", "content": "@cpmpml Yes, you said you had two 2 serial downvoters. It reminds me of serial killers.. Maybe I've seen too many police movies these days. 😄
", "rawMarkdown": "@cpmpml Yes, you said you had two 2 serial downvoters. It reminds me of serial killers.. Maybe I've seen too many police movies these days. 😄 ", "votes": 1 }, { "id": 968672, "postDate": "2020-08-13T07:04:43.020Z", "content": "@cpmpml in some cultures, \"you are funny\" means \"you have a great sense of humor\", i think that's what @yuanlin08 meant…
", "rawMarkdown": "@cpmpml in some cultures, \"you are funny\" means \"you have a great sense of humor\", i think that's what @yuanlin08 meant... ", "votes": 1, "replies": [ { "id": 969516, "postDate": "2020-08-13T18:34:52.537Z", "content": "That's how I understood it as well. I was not sure she was speaking of me. hence I asked.
", "rawMarkdown": "That's how I understood it as well. I was not sure she was speaking of me. hence I asked.", "votes": 2 } ] }, { "id": 969489, "postDate": "2020-08-13T18:02:58.140Z", "content": "Yes, i simply wanted to say the expression made me laugh, sorry for the confusion~
", "rawMarkdown": "Yes, i simply wanted to say the expression made me laugh, sorry for the confusion~ ", "votes": 1 } ] }, { "id": 965778, "postDate": "2020-08-10T21:03:23.117Z", "content": "@cpmpml I've also really started late in the competition (last 17 days). I budgeted and planned everything. Did all my trials like some of yours during last week kept all the things that increase local CV and now I have 1 week to go down the list of model I want to train for later ensemble. Using google colab to do the bigger ones. The last 2 days will be focused on ensembling all of this.
\nUsing pytorch.
\nhere is what I tried:
\nOrder of tests:
\n Base B2 256
\n Color Consistency +
\n Dense head -
\n Coarse + Hair augmentation ++
\n Include 2018 data +
\n Include extra_malignant data +
\n Concat Pooling -
\n Resnest50 (backbone) ++
\n 20 epochs (with resnest50) -
\n Include 2019 data =
\n Posweight = 10 =
\n Pseudo Labelling Failed, no time to fix
\n add Meta data -- no time to find what's going wrong
Thanks a lot. I'll share my list with evals like yours tomorrow. But I confirm coarse is a killer.
", "rawMarkdown": "Thanks a lot. I'll share my list with evals like yours tomorrow. But I confirm coarse is a killer.", "votes": 3 }, { "id": 966311, "postDate": "2020-08-11T10:18:13.780Z", "content": "As promised, list of things I tried (I certainly forget some and will add them as I remember them later). I am using pytorch too.
\nBase B0 256
\nHair removal -
\nColor Consistency +
\nMixup +
\nOther loss than BCE -
\nFp16 +
\nOptimizer+scheduler tuning + 20 epochs +
\nAlbumentations ++
\nCrop -
\nGem pooling -
\nTta+
\nUpscaling -
\nPosweight increase -
\nLabel smoothing -
\nCoarse ++
\nAll 2019 data ++
\nOnly 2019 positive --
\nCustom head ++
Currently trying hair augmentation, but first try is disappointing. I guess I need less coarse dropout.
", "rawMarkdown": "As promised, list of things I tried (I certainly forget some and will add them as I remember them later). I am using pytorch too.\n\nBase B0 256\nHair removal -\nColor Consistency +\nMixup +\nOther loss than BCE -\nFp16 +\nOptimizer+scheduler tuning + 20 epochs +\nAlbumentations ++\nCrop -\nGem pooling -\nTta+\nUpscaling -\nPosweight increase -\nLabel smoothing -\nCoarse ++\nAll 2019 data ++\nOnly 2019 positive --\nCustom head ++\n\nCurrently trying hair augmentation, but first try is disappointing. I guess I need less coarse dropout.", "votes": 10, "replies": [ { "id": 966345, "postDate": "2020-08-11T10:53:55.657Z", "content": "I've found GAP is more promising than GeM pooling (most of my model training). TTA helps a lot. Label smoothing (in my case) with BCE kinda shows better than Focal loss. Color consistency, not sure but didn't show any sign of improvement. I've many custom head, + followed the previous winning solution but in my case, simple approach was enough. MixUp/CutMix, couldn't make it work, I guess, need some solid tricks. However, I'm using tensorflow though.
", "rawMarkdown": "I've found GAP is more promising than GeM pooling (most of my model training). TTA helps a lot. Label smoothing (in my case) with BCE kinda shows better than Focal loss. Color consistency, not sure but didn't show any sign of improvement. I've many custom head, + followed the previous winning solution but in my case, simple approach was enough. MixUp/CutMix, couldn't make it work, I guess, need some solid tricks. However, I'm using tensorflow though. " }, { "id": 970419, "postDate": "2020-08-14T12:51:18.753Z", "content": "we have unbalance data and again you want to try more data of un-balanced 2019 ++ , my qns was deep learning models we know more the data better the model perform but how about more un balanced data ??
", "rawMarkdown": "we have unbalance data and again you want to try more data of un-balanced 2019 ++ , my qns was deep learning models we know more the data better the model perform but how about more un balanced data ??" } ] }, { "id": 966335, "postDate": "2020-08-11T10:37:01.740Z", "content": "why are you sharing this now? How about sharing when comp is over
", "rawMarkdown": "why are you sharing this now? How about sharing when comp is over", "votes": 3 }, { "id": 966352, "postDate": "2020-08-11T10:59:28.400Z", "content": "I won't share more, but I don't think I shared anything new either. And my current results is such that my list isn't very valuable.
", "rawMarkdown": "I won't share more, but I don't think I shared anything new either. And my current results is such that my list isn't very valuable.", "votes": 5 }, { "id": 966663, "postDate": "2020-08-11T15:44:26.887Z", "content": "Could you explain what is color consistence here?
", "rawMarkdown": "Could you explain what is color consistence here?", "replies": [ { "id": 966782, "postDate": "2020-08-11T16:55:32.320Z", "content": "@waylongo
\nhttps://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/154876#867412
@cpmpml Hair Augmentation worked for me, B1/256 did better than B0/256 and WeightedFocalLoss works better than BCE BUT understand its designed for imbalanced datasets so if you balance your dataset don't expect FocalLoss to do better. alpha of .75-.80, gamma of 2
", "rawMarkdown": "@cpmpml Hair Augmentation worked for me, B1/256 did better than B0/256 and WeightedFocalLoss works better than BCE BUT understand its designed for imbalanced datasets so if you balance your dataset don't expect FocalLoss to do better. alpha of .75-.80, gamma of 2", "votes": 1 }, { "id": 967147, "postDate": "2020-08-12T02:48:52.637Z", "content": "@arroqc posweight 10 you mean for class 1? and then just using a weight of 1 for class 0? I have tried various weights, and even using scikits weight estimator. I think it helps, but wasn't a huge deal. I assume you are using BCE with logits loss, since that takes pos weights. I have also found focal loss to be a decent.
", "rawMarkdown": "@arroqc posweight 10 you mean for class 1? and then just using a weight of 1 for class 0? I have tried various weights, and even using scikits weight estimator. I think it helps, but wasn't a huge deal. I assume you are using BCE with logits loss, since that takes pos weights. I have also found focal loss to be a decent." }, { "id": 967527, "postDate": "2020-08-12T10:06:09.717Z", "content": "weird including 2019 data reduces the score somehow.
", "rawMarkdown": "weird including 2019 data reduces the score somehow.", "votes": 1 }, { "id": 968098, "postDate": "2020-08-12T17:40:53.570Z", "content": "\nIn Pytorch :
\nB0 384
\nColor Consistency -Didnt try
\nDense head -
\nCoarse + Hair augmentation ++
\nInclude 2018 data +
\nInclude extra_malignant data =
\nConcat Pooling -
\nGeM Pooling =
\nInclude 2019 data =
\nPosweight = 3/10/15 =
\nPseudo Labelling Failed, no time to fix
\nadd Meta data --
How to apply color consistency,Mixups ? Couldnt find them in Albumentation
\nAnd what is FP16 ?
FP16 is half precision, faster, less memory, not stable on all GPU's…..stable on those that have Tensor Cores.
", "rawMarkdown": "FP16 is half precision, faster, less memory, not stable on all GPU's.....stable on those that have Tensor Cores.", "votes": 1 }, { "id": 969116, "postDate": "2020-08-13T13:16:06.407Z", "content": "Cool , How to apply Custom Head/Mixups in albumentation framework ?You guys have Any code link for that ?
", "rawMarkdown": "Cool , How to apply Custom Head/Mixups in albumentation framework ?You guys have Any code link for that ?", "replies": [ { "id": 969226, "postDate": "2020-08-13T14:57:13.760Z", "content": "custom head means a change in the NN architecture.
\nmixup is combining two images. You cannot do that in albumentation given it works single image per image. If you google it then you'll find code for mixup. Or if you look at past competition forums, or even this competition forum probably.
", "rawMarkdown": "custom head means a change in the NN architecture.\n\nmixup is combining two images. You cannot do that in albumentation given it works single image per image. If you google it then you'll find code for mixup. Or if you look at past competition forums, or even this competition forum probably.", "votes": 2 }, { "id": 969269, "postDate": "2020-08-13T15:22:56.553Z", "content": "Okay ,Thanks :D I will try :D
", "rawMarkdown": "Okay ,Thanks :D I will try :D " } ] } ] }, { "id": 968662, "postDate": "2020-08-13T06:58:15.953Z", "content": "I might update this time to time when I remember what I tried or try something new:
\nGoing to try concat meta and image inputs in network soon, again was planning to try shades of gray but I think it's kinda late for it to creating preprocessed dataset with it…
", "rawMarkdown": "I might update this time to time when I remember what I tried or try something new:\n\n- Progressive coarse dropout helped on cv not on lb(if you care about it),\n- Again same with upsampling,\n- Custom heads didn't work for me,\n- Weighted binary crossentropy didn't work again\n\nGoing to try concat meta and image inputs in network soon, again was planning to try shades of gray but I think it's kinda late for it to creating preprocessed dataset with it...", "votes": 3, "replies": [ { "id": 968867, "postDate": "2020-08-13T09:43:35.620Z", "content": "I have tried meta and image dual input for CNN from 384 to 768 size, it doesn't work for LB(but it surely boosted CV)
\nIf you notice any improvements in concatenating image and meta, sharing it here would be great :D
I haven't used any meta-data so far… that might be risky for me :)
", "rawMarkdown": "I haven't used any meta-data so far... that might be risky for me :)", "votes": 2 }, { "id": 969348, "postDate": "2020-08-13T16:15:57.743Z", "content": "Maybe that's why you made progress quickly rather ;)
", "rawMarkdown": "Maybe that's why you made progress quickly rather ;)", "votes": 1 }, { "id": 969683, "postDate": "2020-08-13T21:05:56.907Z", "content": "@deepkim tried basic version today but only on 256x because of timing problems got 93+ lb and 94+ cv, sadly out of TPU time on kaggle now, have to test it later on colab… But it was promising for me
", "rawMarkdown": "@deepkim tried basic version today but only on 256x because of timing problems got 93+ lb and 94+ cv, sadly out of TPU time on kaggle now, have to test it later on colab... But it was promising for me", "votes": 1 } ] }, { "id": 965748, "postDate": "2020-08-10T20:15:46.843Z", "content": "I tried many things, most of them didn't work some did, then tried to merge what worked before, again some worked some didn't. Still have many ideas to try but not enough time or computing power left.
\nThere is another list of ideas where in theory and beyond my skill so that's an another list for things to research and learn after the competition. I'm kinda new to competitions but feels like I'll end up with list of lists 😀
", "rawMarkdown": "I tried many things, most of them didn't work some did, then tried to merge what worked before, again some worked some didn't. Still have many ideas to try but not enough time or computing power left.\n\nThere is another list of ideas where in theory and beyond my skill so that's an another list for things to research and learn after the competition. I'm kinda new to competitions but feels like I'll end up with list of lists 😀", "votes": 3 }, { "id": 965852, "postDate": "2020-08-10T22:53:26.070Z", "content": "I used this competition to gain experience in using TPU with Pytorch. I also learned some new things to me, like OneCycleLR and improved my image processing in Python. I hope I can use these skills in next image competition, however looks like Pytorch + TPU is still much worse combination than Tensorflow.
", "rawMarkdown": "I used this competition to gain experience in using TPU with Pytorch. I also learned some new things to me, like OneCycleLR and improved my image processing in Python. I hope I can use these skills in next image competition, however looks like Pytorch + TPU is still much worse combination than Tensorflow.", "votes": 4 }, { "id": 972188, "postDate": "2020-08-16T10:08:14.967Z", "content": "My experiences:
\nI stopped working on this competition at all few days ago, but I am still here to see the huge shakeup and surprised faces :)
", "rawMarkdown": "I stopped working on this competition at all few days ago, but I am still here to see the huge shakeup and surprised faces :)", "votes": 1, "replies": [ { "id": 970188, "postDate": "2020-08-14T09:15:15.883Z", "content": "@jacekpoplawski I hope someone makes a visualization, maybe using Parallel Coordinates, with scores on the left Public LB and scores on the right Private LB, I can imagine many of them clustered and moving the same. I hope I am one of them, but moving up!
", "rawMarkdown": "@jacekpoplawski I hope someone makes a visualization, maybe using Parallel Coordinates, with scores on the left Public LB and scores on the right Private LB, I can imagine many of them clustered and moving the same. I hope I am one of them, but moving up!", "votes": 1 } ] }, { "id": 968854, "postDate": "2020-08-13T09:30:56.307Z", "content": "I want to try predicting diagnosis too together with label. I read in some discussions that it certainly is giving better results. If anyone tries this out, do share your results! :)
", "rawMarkdown": "I want to try predicting diagnosis too together with label. I read in some discussions that it certainly is giving better results. If anyone tries this out, do share your results! :)", "votes": 1 }, { "id": 967422, "postDate": "2020-08-12T08:38:25.223Z", "content": "i try hair augmentation，but i dont know why it become lower
", "rawMarkdown": "i try hair augmentation，but i dont know why it become lower", "votes": 1, "replies": [ { "id": 967655, "postDate": "2020-08-12T12:07:19.367Z", "content": "I tried as I wrote somewhere in this thread.
", "rawMarkdown": "I tried as I wrote somewhere in this thread." } ] }, { "id": 966242, "postDate": "2020-08-11T09:10:46.843Z", "content": "And computers are never fast enough…
", "rawMarkdown": "And computers are never fast enough... ", "votes": 1, "replies": [ { "id": 966293, "postDate": "2020-08-11T10:07:52.723Z", "content": "Actually, not having too much compute resource is beneficial: it forces you to think more before running experiments.
", "rawMarkdown": "Actually, not having too much compute resource is beneficial: it forces you to think more before running experiments.", "votes": 26 }, { "id": 966357, "postDate": "2020-08-11T11:10:18.613Z", "rawMarkdown": "", "isDeleted": true }, { "id": 967519, "postDate": "2020-08-12T10:02:27.663Z", "content": "On the other hand, some part of data science competitions is about how fast you can iterate your ideas.
", "rawMarkdown": "On the other hand, some part of data science competitions is about how fast you can iterate your ideas.", "votes": 2 }, { "id": 969273, "postDate": "2020-08-13T15:23:24.403Z", "content": "Having lost of resources near the end of comp to train at scale is indeed useful too. But having too much resources early isn't great IMHO.
", "rawMarkdown": "Having lost of resources near the end of comp to train at scale is indeed useful too. But having too much resources early isn't great IMHO.", "votes": 1 } ] }, { "id": 965844, "postDate": "2020-08-10T22:45:34.707Z", "content": "@cpmpml
\nvery true, many things to try yet but no enough time :(
\nanyway, I was thinking, it would be nice if we can see who downvote because in that case, the downvoter will have the reason why he/she do this. 😄
for me it's \"join late and only got a single 1080ti\"...
\n\ni have a list similar to yours, and I just added \"testing weight decay while ignoring BN layers\" to it.
", "rawMarkdown": "for me it's \"join late and only got a single 1080ti\"... \n\ni have a list similar to yours, and I just added \"testing weight decay while ignoring BN layers\" to it. \n\n", "votes": 1, "replies": [ { "id": 965720, "postDate": "2020-08-10T19:45:49.833Z", "content": "I went through this and a number of things fortunately, including optimizer, scheduler, fp16, augmentations, 2019 data, etc. But the more I do, the more I see left to do...
\n\nGood luck anyway. But given you only have one GPU I would consider using Kaggle kernels with both TPU and GPU in addition to your machine.
", "rawMarkdown": "I went through this and a number of things fortunately, including optimizer, scheduler, fp16, augmentations, 2019 data, etc. But the more I do, the more I see left to do...\n\nGood luck anyway. But given you only have one GPU I would consider using Kaggle kernels with both TPU and GPU in addition to your machine.", "votes": 2 } ] }, { "id": 968596, "postDate": "2020-08-13T06:11:45.770Z", "content": "Hair Augmentation doesn't give a major increase in the lb score we had tried it out although the cv score increases.
", "rawMarkdown": "Hair Augmentation doesn't give a major increase in the lb score we had tried it out although the cv score increases.", "votes": 2, "replies": [ { "id": 968608, "postDate": "2020-08-13T06:21:59.303Z", "content": "Thanks for the update. I've tried hundreds of things but I haven't tried hair augmentation yet.
", "rawMarkdown": "Thanks for the update. I've tried hundreds of things but I haven't tried hair augmentation yet.", "votes": 1 }, { "id": 969014, "postDate": "2020-08-13T12:04:54.230Z", "content": "I didn't see any benefit from hair augmentation on cv so far.
", "rawMarkdown": "I didn't see any benefit from hair augmentation on cv so far.", "votes": 2 }, { "id": 969532, "postDate": "2020-08-13T18:51:33.503Z", "content": "Chris, have you tried any post processing method so far. Just curious.
", "rawMarkdown": "Chris, have you tried any post processing method so far. Just curious." } ] }, { "id": 967522, "postDate": "2020-08-12T10:03:07.867Z", "content": "let me answer some that I tested :
\nImage size increases the score for sure , tricky part is too fit within the 3-hour TPU limit
\nhaven't tried hair augment yet
\nupsampling is kinda insignificant
\neffnet b6 is giving me best results, got better than b7
\nmean average got me good results
\nthe tabular ensemble with image prediction gives a significant boost to public LB
\nThe public notebook is screwing the LB, to be honest, but helped me too in the single model case.
B6 is giving better results in my case too!
", "rawMarkdown": "B6 is giving better results in my case too!", "votes": 1 }, { "id": 967653, "postDate": "2020-08-12T12:06:55.817Z", "content": "@yash612 thanks.
\n\n\nImage size increases the score for sure , tricky part is too fit within the 3-hour TPU limit
\n
I am not using TPU nor am I using @cdeotte code. Therefore some of the conclusions one can get out of Chris code may not apply to other ways of doing things. Anyway, thanks for the feedback on the rest of your list. I am kind of reassured you also don't see much benefit form upsampling as this is the case for me. I haven't tried b6 yet, will do at a point for sure. Working on meta data is on my todo list.
", "rawMarkdown": "@yash612 thanks.\n\n> Image size increases the score for sure , tricky part is too fit within the 3-hour TPU limit\n\nI am not using TPU nor am I using @cdeotte code. Therefore some of the conclusions one can get out of Chris code may not apply to other ways of doing things. Anyway, thanks for the feedback on the rest of your list. I am kind of reassured you also don't see much benefit form upsampling as this is the case for me. I haven't tried b6 yet, will do at a point for sure. Working on meta data is on my todo list.", "votes": 1 } ] }, { "id": 967207, "postDate": "2020-08-12T05:03:59.277Z", "content": "@cpmpml I also tried upsampling (per batch upsampling) but it did not work. I changed this to per epoch downsampling where I trained on 65% random selection of the negative samples + all positive per epoch and it worked well.
\nI also feel that the problem in scores is not about TF or torch, I think it is about the datasets. I am using Tensorflow with the JPEG dataset and I got best single model score 0.939 with basic TTA.
\nall public kernels scoring 0.95+ are using the tfrecord datasets.
Thanks. Your downsampling is interesting.
", "rawMarkdown": "Thanks. Your downsampling is interesting." }, { "id": 968423, "postDate": "2020-08-13T02:12:06.813Z", "content": "@cdeotte Could the TFRecords values in your datasets be before JPEG compression ?
\nEdit: Nevermind saw your code and you encode at JPEG94. I imagine you do that also for the .jpg files.
I wish we saw the code used for generating all datasets.
", "rawMarkdown": "I wish we saw the code used for generating all datasets. ", "votes": 1 }, { "id": 968429, "postDate": "2020-08-13T02:21:41.960Z", "content": "@arroqc I don't understand your question. Both my TFRecords and JPEG dataset come from the same NumPy image array. After reading in original Kaggle jpeg images, they are max square center crop, then resize with cv2.resize(img,(DIM,DIM),interpolation = cv2.INTER_AREA), then the image gets saved into a NumPy array of shape (len(train), DIM, DIM, 3)
The TFRecords are made by compressing with cv2.imencode('.jpg', img)[1].tostring() and the JPEG on disk are made with cv2.imwrite(PATH+files[k],img)
Yes I'm starting to wonder if the differences mentioned by @asabry79400 are not due to some different processes between what is in a record and what is in a .jpeg.
", "rawMarkdown": "Yes I'm starting to wonder if the differences mentioned by @asabry79400 are not due to some different processes between what is in a record and what is in a .jpeg.", "votes": 1 }, { "id": 968432, "postDate": "2020-08-13T02:27:13.570Z", "content": "\n\nEdit: Nevermind saw your code and you encode at JPEG94. I imagine you do that also for the .jpg files.
\n
I only encode that notebook at JPEG94 which makes my https://www.kaggle.com/cdeotte/512x512-melanoma-tfrecords-70k-images dataset
\nAll my other TFRecords datasets use the cv2.imencode('.jpg', img)[1].tostring() which i believe has default CV_IMWRITE_JPEG_QUALITY = 95. And JPEG datasets use cv2.imwrite(PATH+files[k],img) which also has default CV_IMWRITE_JPEG_QUALITY = 95
@cdeotte Well you seem to have perfectly understood the question thanks for the answer. I was wondering if the method of saving images are different. I would expect both the default for imencode and imwrite to be the same though.
", "rawMarkdown": "@cdeotte Well you seem to have perfectly understood the question thanks for the answer. I was wondering if the method of saving images are different. I would expect both the default for imencode and imwrite to be the same though." }, { "id": 968434, "postDate": "2020-08-13T02:30:56.500Z", "content": "\n\nI wish we saw the code used for generating all datasets.
\n
Here is the full code to generate my JPEG and TFRecord datasets:
\nDIM = 256\nimages = np.zeros((33126,DIM,DIM,3),dtype='uint8')\n\nfor k in range(len(files)):\n img = cv2.imread(PATH2+files[k])\n w = img.shape[1]\n h = img.shape[0]\n s = min(w,h)\n w2 = (w-s)//2\n h2 = (h-s)//2\n img = img[h2:h-h2,w2:w-w2,:]\n img = cv2.resize(img,(DIM,DIM),interpolation = cv2.INTER_AREA)\n cv2.imwrite(PATH2+files[k],img)\n img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n images[k,] = img\nThe above code both writes the JPEG dataset and saves the images into a NumPy array. In the same offline Jupyter notebook, the TFRecords are made later with
\n img = cv2.cvtColor(images[row.i.values[0],], cv2.COLOR_RGB2BGR)\n img = cv2.imencode('.jpg', img)[1].tostring()\n example = serialize_example(\n img, str.encode(name),\n row.patient_id.values[0],\n row.sex.values[0],\n row.age_approx.values[0], \n row.anatom_site_general_challenge.values[0],\n row.diagnosis.values[0],\n row.target.values[0],\n row.width.values[0],\n row.height.values[0])\n writer.write(example)\nThanks as a quick try I tried the following:
\nimg = cv2.imread('H:/SIIM/768/train/ISIC_0000070.jpg')\n\nstring = cv2.imencode('.jpg', img)[1].tostring()\ncv2.imwrite('test.jpg',img)\nnparr = np.frombuffer(string, np.uint8)\nnew_img_records = cv2.imdecode(nparr, cv2.IMREAD_COLOR)\n\nnew_img_jpeg = cv2.imread('test.jpg')\nprint(np.all(new_img_jpeg == new_img_records))\n\nfrom PIL import Image\nnew_img_pil = cv2.cvtColor(np.array(Image.open('test.jpg')), cv2.COLOR_RGB2BGR)\nprint(np.all(new_img_pil == new_img_records))\nprint(np.mean(np.abs(new_img_pil - new_img_records)))\nFirst is True, second is False and average difference is ~5. Note that visually I can't see the difference and this proves nothing. Interesting though.
", "rawMarkdown": "Thanks as a quick try I tried the following:\n```\nimg = cv2.imread('H:/SIIM/768/train/ISIC_0000070.jpg')\n\nstring = cv2.imencode('.jpg', img)[1].tostring()\ncv2.imwrite('test.jpg',img)\nnparr = np.frombuffer(string, np.uint8)\nnew_img_records = cv2.imdecode(nparr, cv2.IMREAD_COLOR)\n\nnew_img_jpeg = cv2.imread('test.jpg')\nprint(np.all(new_img_jpeg == new_img_records))\n\nfrom PIL import Image\nnew_img_pil = cv2.cvtColor(np.array(Image.open('test.jpg')), cv2.COLOR_RGB2BGR)\nprint(np.all(new_img_pil == new_img_records))\nprint(np.mean(np.abs(new_img_pil - new_img_records)))\n```\n\nFirst is True, second is False and average difference is ~5. Note that visually I can't see the difference and this proves nothing. Interesting though.", "votes": 1 }, { "id": 968451, "postDate": "2020-08-13T03:18:53.203Z", "content": "@cdeotte if you want to see some serious speed increase in running that, just install pythons Parallel module (from pip or wherever). The more cores the better, crazy fast. Then:
num_cores = 12 \n\nDIM = 256\nimages = np.zeros((33126,DIM,DIM,3),dtype='uint8')\n\ndef resize(k):\n img = cv2.imread(PATH2+files[k])\n w = img.shape[1]\n h = img.shape[0]\n s = min(w,h)\n w2 = (w-s)//2\n h2 = (h-s)//2\n img = img[h2:h-h2,w2:w-w2,:]\n img = cv2.resize(img,(DIM,DIM),interpolation = cv2.INTER_AREA)\n cv2.imwrite(PATH2+files[k],img)\n img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n images[k,] = img\n\nParallel(n_jobs=num_cores, verbose=10)(delayed(resize)(k) for k in range(len(files)))\nNote: Edited this to match
", "rawMarkdown": "@cdeotte if you want to see some serious speed increase in running that, just install pythons `Parallel` module (from `pip` or wherever). The more cores the better, crazy fast. Then:\n\n```\nnum_cores = 12 \n\nDIM = 256\nimages = np.zeros((33126,DIM,DIM,3),dtype='uint8')\n\ndef resize(k):\n img = cv2.imread(PATH2+files[k])\n w = img.shape[1]\n h = img.shape[0]\n s = min(w,h)\n w2 = (w-s)//2\n h2 = (h-s)//2\n img = img[h2:h-h2,w2:w-w2,:]\n img = cv2.resize(img,(DIM,DIM),interpolation = cv2.INTER_AREA)\n cv2.imwrite(PATH2+files[k],img)\n img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n images[k,] = img\n\nParallel(n_jobs=num_cores, verbose=10)(delayed(resize)(k) for k in range(len(files)))\n```\n\nNote: Edited this to match", "votes": 2 }, { "id": 968452, "postDate": "2020-08-13T03:21:21.620Z", "content": "Wow, awesome, thanks @brianfeeny I'll try that.
", "rawMarkdown": "Wow, awesome, thanks @brianfeeny I'll try that.", "votes": 1 }, { "id": 969020, "postDate": "2020-08-13T12:09:09.530Z", "content": "The theory I mentioned previously was indeed that difference in data explains difference in model performance.
", "rawMarkdown": "The theory I mentioned previously was indeed that difference in data explains difference in model performance. \n", "votes": 1 }, { "id": 969520, "postDate": "2020-08-13T18:37:43.103Z", "content": "@cpmpml Is there difference? From the analysis it does not seem so.
", "rawMarkdown": "@cpmpml Is there difference? From the analysis it does not seem so.", "replies": [ { "id": 969673, "postDate": "2020-08-13T20:59:08.127Z", "content": "I saw a difference in my first try, but not in the second. I guess it is noise.
", "rawMarkdown": "I saw a difference in my first try, but not in the second. I guess it is noise." } ] }, { "id": 969657, "postDate": "2020-08-13T20:42:13.683Z", "content": "What I find really suspicious is that while some of these public kernels get good LB like 0.95, their CV is really mediocre like 0.91 or 0.92. I have some models in the 0.93 region that use the exact same splits but \"just\" give 0.94 LB. I'm going to trust CV more but I wonder if there is more to the story.
", "rawMarkdown": "What I find really suspicious is that while some of these public kernels get good LB like 0.95, their CV is really mediocre like 0.91 or 0.92. I have some models in the 0.93 region that use the exact same splits but \"just\" give 0.94 LB. I'm going to trust CV more but I wonder if there is more to the story.", "votes": 2 }, { "id": 969675, "postDate": "2020-08-13T21:00:57.247Z", "content": "@arroqc is that really remarkable in this comp though? I mean I have models that are .89 and .90 in CV and do .93 on LB. It's frustrating for sure, that things aren't tighter with the scoring, which apparently is due to the few amount of positives in the dataset.
", "rawMarkdown": "@arroqc is that really remarkable in this comp though? I mean I have models that are .89 and .90 in CV and do .93 on LB. It's frustrating for sure, that things aren't tighter with the scoring, which apparently is due to the few amount of positives in the dataset." }, { "id": 969693, "postDate": "2020-08-13T21:16:45.740Z", "content": "@brianfeeny No I'm well aware of that. My point was more about I wouldn't put that much trust in the score of the public kernel but also wondering if there is indeed more to it than \"LB is noisy\". That said when running some of them I had variation up to 0.01 in score (the ones involving training).
", "rawMarkdown": "@brianfeeny No I'm well aware of that. My point was more about I wouldn't put that much trust in the score of the public kernel but also wondering if there is indeed more to it than \"LB is noisy\". That said when running some of them I had variation up to 0.01 in score (the ones involving training)." }, { "id": 969866, "postDate": "2020-08-14T02:51:19.650Z", "content": "@philippsinger I noticed you recently improved your score, did you use the TF data?
", "rawMarkdown": "@philippsinger I noticed you recently improved your score, did you use the TF data?" }, { "id": 970648, "postDate": "2020-08-14T16:27:06.497Z", "content": "no, I am too stubborn to use tensorflow :D
", "rawMarkdown": "no, I am too stubborn to use tensorflow :D", "votes": 5 }, { "id": 970690, "postDate": "2020-08-14T17:13:19.857Z", "content": "The golden rule: \"Do not put each foot in a different boat\".
", "rawMarkdown": "The golden rule: \"Do not put each foot in a different boat\"." } ] }, { "id": 965743, "postDate": "2020-08-10T20:09:39.897Z", "content": "I would advice you to try TF instead of pytorch. I did not succeed to have as good as results in pytorch for this competition :/ model with 384 and 512 give very good results. I can have 0.95X with them on public LB
", "rawMarkdown": "I would advice you to try TF instead of pytorch. I did not succeed to have as good as results in pytorch for this competition :/ model with 384 and 512 give very good results. I can have 0.95X with them on public LB", "votes": 2, "replies": [ { "id": 965759, "postDate": "2020-08-10T20:37:00.397Z", "content": "This makes me sad. Any idea why this is the case?
\nthere is no single high scoring LB pytorch kernel this time
", "rawMarkdown": "This makes me sad. Any idea why this is the case?\n\nthere is no single high scoring LB pytorch kernel this time", "votes": 6, "replies": [ { "id": 965791, "postDate": "2020-08-10T21:24:54.260Z", "content": "Don't mean to mock but you've answered the question to this here: https://www.kaggle.com/c/tweet-sentiment-extraction/discussion/143869
\nPeople are using ensemble of EfficientNet's as one single model plus some additional techniques as well but when I compared this seemed as the main ingredient. When I tried to implement the same with torch or torch_xla there were oom issues most of the time.
\nI really admire you btw, just wanted to let you know.
", "rawMarkdown": "Don't mean to mock but you've answered the question to this here: https://www.kaggle.com/c/tweet-sentiment-extraction/discussion/143869\n\nPeople are using ensemble of EfficientNet's as one single model plus some additional techniques as well but when I compared this seemed as the main ingredient. When I tried to implement the same with torch or torch_xla there were oom issues most of the time. \n\nI really admire you btw, just wanted to let you know. " } ] }, { "id": 965766, "postDate": "2020-08-10T20:50:25.070Z", "content": "Unfortunately, I am not sure. It is the first time I was using efficientNet (on both TF/pytorch). My experiment was using TPU, and I have already seen some little drop of performance with pytorch tpu on nlp task. But for this image task , I was wondering if it was not the library. I will probably try to do some experiment on cifar or similar dataset to compare the TF VS torch for efficientNet on gpu. It really bugs me.
", "rawMarkdown": "Unfortunately, I am not sure. It is the first time I was using efficientNet (on both TF/pytorch). My experiment was using TPU, and I have already seen some little drop of performance with pytorch tpu on nlp task. But for this image task , I was wondering if it was not the library. I will probably try to do some experiment on cifar or similar dataset to compare the TF VS torch for efficientNet on gpu. It really bugs me." }, { "id": 965821, "postDate": "2020-08-10T21:58:11.283Z", "content": "I have a theory about why Pytorch works worse than TF here. But I need to experiment to validate it. Will report one way or the other when done.
", "rawMarkdown": "I have a theory about why Pytorch works worse than TF here. But I need to experiment to validate it. Will report one way or the other when done.\n", "votes": 4 }, { "id": 965828, "postDate": "2020-08-10T22:13:27.800Z", "content": "\n\nI would advice you to try TF instead of pytorch.
\n
i will run some public TF kernels for sure to add diversity, but I won't invest time in developing something new. Once you have tried pytorch it is hard to go back.
", "rawMarkdown": "> I would advice you to try TF instead of pytorch. \n\ni will run some public TF kernels for sure to add diversity, but I won't invest time in developing something new. Once you have tried pytorch it is hard to go back.", "votes": 8 }, { "id": 966055, "postDate": "2020-08-11T05:21:14.727Z", "content": "Whats the theory?
", "rawMarkdown": "Whats the theory?" }, { "id": 966192, "postDate": "2020-08-11T08:18:41.310Z", "content": "I thought because of Tensorflow's better compatibility with TPU people, are using it instead of Pytorch. Still, I am new to this, am I missing something?
", "rawMarkdown": "I thought because of Tensorflow's better compatibility with TPU people, are using it instead of Pytorch. Still, I am new to this, am I missing something?" }, { "id": 966361, "postDate": "2020-08-11T11:12:46.527Z", "content": "TPU are designed to run Tensorflow, no wonder it works well. But this would explain why training TF on TPU is fast. It does not explain why Pytorch models seem to have lower quality here.
", "rawMarkdown": "TPU are designed to run Tensorflow, no wonder it works well. But this would explain why training TF on TPU is fast. It does not explain why Pytorch models seem to have lower quality here.", "votes": 2 }, { "id": 966363, "postDate": "2020-08-11T11:14:22.180Z", "content": "\n\nWhats the theory?
\n
I won't share now as I have not tested it yet. It can be one of the many promising ideas that miserably fail when implemented…
\nAnd given we are approaching competition end, as my colleague @christofhenkel rightfully reminded me, it is better to wait till competition end.
", "rawMarkdown": "> Whats the theory?\n\nI won't share now as I have not tested it yet. It can be one of the many promising ideas that miserably fail when implemented...\n\n And given we are approaching competition end, as my colleague @christofhenkel rightfully reminded me, it is better to wait till competition end.", "votes": 2 }, { "id": 966480, "postDate": "2020-08-11T12:59:40.993Z", "content": "I think not because Pytorch is not good but TPU+TF dominate.
", "rawMarkdown": "I think not because Pytorch is not good but TPU+TF dominate.", "votes": 1 }, { "id": 966490, "postDate": "2020-08-11T13:07:49.280Z", "content": "I think one reason is that @cdeotte baseline is extremely strong, and most participants built on top of it.
\nIn competitions where someone shared early enough a strong pytorch baseline, for instance Abhishek Thakur baseline in Tweet Sentiment, then most participants use pytorch.
", "rawMarkdown": "I think one reason is that @cdeotte baseline is extremely strong, and most participants built on top of it.\n\nIn competitions where someone shared early enough a strong pytorch baseline, for instance Abhishek Thakur baseline in Tweet Sentiment, then most participants use pytorch.", "votes": 5 }, { "id": 966531, "postDate": "2020-08-11T13:56:29.867Z", "content": "Yeah to call it \"simple baseline\" is a bad joke imho. Its a heavy efficientnet with triple stratification cv, label smoothing, external data, highly tuned augmentation and tta.
", "rawMarkdown": "Yeah to call it \"simple baseline\" is a bad joke imho. Its a heavy efficientnet with triple stratification cv, label smoothing, external data, highly tuned augmentation and tta. ", "votes": 11 }, { "id": 966581, "postDate": "2020-08-11T14:41:38.453Z", "content": "oh, Chris has label smoothing too? That's why people asked for pytorch code. Interesting. I should definitely look at his code now.
", "rawMarkdown": "oh, Chris has label smoothing too? That's why people asked for pytorch code. Interesting. I should definitely look at his code now.", "votes": 1 }, { "id": 966738, "postDate": "2020-08-11T16:25:44.810Z", "content": "There may be some high scoring PyTorch, we will see. I thought Serigne was using PyTorch and he is top 100
", "rawMarkdown": "There may be some high scoring PyTorch, we will see. I thought Serigne was using PyTorch and he is top 100" }, { "id": 966740, "postDate": "2020-08-11T16:29:02.190Z", "content": "You are right, I think many built on top of Chris's excellent work, and there is nothing wrong with that, it raises the bar, makes everything more challenging and overall will produce a better result. However, there was supposedly some notebook shared, that I don't know anything about, that apparently several hundred people are using or whatever, and that caused the leaderboard to go crazy. At least someone should reveal what was in that notebook that gave them so much advantage, at least then we could level the playing field a little. The notebook was taken down after a few hours and people have been tight-lipped about it………meanwhile, so many others continue to share helpful tips, I just wish that info in that public notebook wasn't squandered.
", "rawMarkdown": "You are right, I think many built on top of Chris's excellent work, and there is nothing wrong with that, it raises the bar, makes everything more challenging and overall will produce a better result. However, there was supposedly some notebook shared, that I don't know anything about, that apparently several hundred people are using or whatever, and that caused the leaderboard to go crazy. At least someone should reveal what was in that notebook that gave them so much advantage, at least then we could level the playing field a little. The notebook was taken down after a few hours and people have been tight-lipped about it.........meanwhile, so many others continue to share helpful tips, I just wish that info in that public notebook wasn't squandered.", "votes": 1 }, { "id": 966827, "postDate": "2020-08-11T17:21:06.040Z", "content": "You can go to Notebook tab and sort by best score. Current is at 0.9619 https://www.kaggle.com/paklau9/minmax-highest-public-lb-9619
", "rawMarkdown": "You can go to Notebook tab and sort by best score. Current is at 0.9619 https://www.kaggle.com/paklau9/minmax-highest-public-lb-9619", "votes": 1 }, { "id": 966954, "postDate": "2020-08-11T19:40:45.130Z", "content": "yeah, that's not even a real kernel though. I mean, I guess Kaggle allows you to include your submission files/scores too? So you have maybe 100+ people submitting a .9600………..and they all have the same values, out to like 9 significant digits, for every prediction……..how is that legal? Or, in the end will they be cleaved? I don't know much about how Kaggle deals with all that. Personally I don't think you should be able to include scores/predictions in kernels that are nothing more than blenders/stackers/ensembles. Even with regular kernels, why share predictions.
", "rawMarkdown": "yeah, that's not even a real kernel though. I mean, I guess Kaggle allows you to include your submission files/scores too? So you have maybe 100+ people submitting a .9600...........and they all have the same values, out to like 9 significant digits, for every prediction........how is that legal? Or, in the end will they be cleaved? I don't know much about how Kaggle deals with all that. Personally I don't think you should be able to include scores/predictions in kernels that are nothing more than blenders/stackers/ensembles. Even with regular kernels, why share predictions." }, { "id": 968958, "postDate": "2020-08-13T11:21:23.873Z", "content": "@brianfeeny My best Pytorch model won't definitely fit on kernel (weak RAM and low CPU cores on kernels) .
\nI think @yuval6967 has the best Pytorch model ( LB 0.960 reported much earlier in this comp , thus I think he found way to improved since then)
", "rawMarkdown": "@brianfeeny My best Pytorch model won't definitely fit on kernel (weak RAM and low CPU cores on kernels) . \n\nI think @yuval6967 has the best Pytorch model ( LB 0.960 reported much earlier in this comp , thus I think he found way to improved since then)", "votes": 1 }, { "id": 970360, "postDate": "2020-08-14T11:37:44.837Z", "content": "Join the game late, and i use pytorch.
\nIt is easy to achieve 0.94+ with effnetb0, 256x256 input.
But with higher resolution or bigger model, the score basicly the same. And that make me feel disappointed.
", "rawMarkdown": "Join the game late, and i use pytorch.\nIt is easy to achieve 0.94+ with effnetb0, 256x256 input. \n\nBut with higher resolution or bigger model, the score basicly the same. And that make me feel disappointed.\n" }, { "id": 970393, "postDate": "2020-08-14T12:11:12.007Z", "content": "@serigne My model is not the \"best Pytorch model\" - it might be the reported single model with the best public LB. But this doesn't say anything. I don't believe in the public LB, the number of true values is too small. - \"Shakeup is Coming …\"
", "rawMarkdown": "@serigne My model is not the \"best Pytorch model\" - it might be the reported single model with the best public LB. But this doesn't say anything. I don't believe in the public LB, the number of true values is too small. - \"Shakeup is Coming ...\" ", "votes": 2 }, { "id": 970499, "postDate": "2020-08-14T14:06:41.203Z", "content": "You're right ! I should say best public LB
", "rawMarkdown": "You're right ! I should say best public LB " } ] }, { "id": 971447, "postDate": "2020-08-15T14:31:14.770Z", "content": "The things I want to try before it is over:
\n2 Days to go ;)
", "rawMarkdown": "The things I want to try before it is over:\n- Use bigger models (EF B5, B6 .. ) \n- improve my metadata classifier\n- try different losses (Focal Loss, Binary Focal Loss, ...)\n- use TTA\n\n2 Days to go ;)" }, { "id": 969551, "postDate": "2020-08-13T19:07:34.213Z", "content": "I don't get it, some top kagglers are in 1000+ range!!!! Are all those core PyTorch users or the LB is really really really overfitted??
", "rawMarkdown": "I don't get it, some top kagglers are in 1000+ range!!!! Are all those core PyTorch users or the LB is really really really overfitted?? ", "replies": [ { "id": 969604, "postDate": "2020-08-13T19:56:40.237Z", "content": "There are people in the 100's, and certain < 500 ranking that are brand new and asking questions you would find in the first few chapters of a good book on machine learning. Literally, newbies, who have no idea what they are doing. The solutions don't have to be overfitting, they could be valid, just shared, I guess time will tell.
", "rawMarkdown": "There are people in the 100's, and certain < 500 ranking that are brand new and asking questions you would find in the first few chapters of a good book on machine learning. Literally, newbies, who have no idea what they are doing. The solutions don't have to be overfitting, they could be valid, just shared, I guess time will tell.", "votes": 2, "replies": [ { "id": 969680, "postDate": "2020-08-13T21:05:04.763Z", "content": "That's the issue of strong public baselines: lots of monkey testing happens. And some will get lucky.
", "rawMarkdown": "That's the issue of strong public baselines: lots of monkey testing happens. And some will get lucky. ", "votes": 2 }, { "id": 970199, "postDate": "2020-08-14T09:23:38.347Z", "content": "Half a dozen very strong public baseline and 'starter' models has raised the game generally and has allowed competitors to focus on experimentation and discovery rather than hours of coding and debugging. This is very good in the context of the type of competition this is, medical research to benefit cancer victims. Better to spend the time on innovation than hacking.
", "rawMarkdown": "Half a dozen very strong public baseline and 'starter' models has raised the game generally and has allowed competitors to focus on experimentation and discovery rather than hours of coding and debugging. This is very good in the context of the type of competition this is, medical research to benefit cancer victims. Better to spend the time on innovation than hacking. " }, { "id": 970241, "postDate": "2020-08-14T09:40:20.400Z", "content": "A lot of learning opportunities were lost by avoiding all the hurdles around data preparation, cv setting, scheduler tuning, etc. It is better to learn people how to fish than giving them a fish.
\nWell, that's my way of thinking, and I get that other ways are more popular.
\nThe argument that the final solution will be better isn't valid IMHO. Top teams don't need a public strong baseline.
", "rawMarkdown": "A lot of learning opportunities were lost by avoiding all the hurdles around data preparation, cv setting, scheduler tuning, etc. It is better to learn people how to fish than giving them a fish.\n\nWell, that's my way of thinking, and I get that other ways are more popular.\n\nThe argument that the final solution will be better isn't valid IMHO. Top teams don't need a public strong baseline. ", "votes": 1 }, { "id": 970265, "postDate": "2020-08-14T10:00:09.817Z", "content": "I am also a believer in obtaining a strong foundation in the first principles of any specialism in which I participate.
\nFrom my personal experience (as a newbie), the strong starter code and baselines have helped hugely in the learning process, including experimenting with the data prep, learning rates, batch sizes, augmentation, upsampling, the list goes on. Don't get me wrong, I won't use any code unless I understand fully what its doing. Often I will also do external research around these areas when experimenting. But having starter code will have increased my opportunity time for this external research and experimentation by maybe 2-3x.
\nI get your point that the top data scientists who will likely populate the top of the leaderboard won't have benefited from the baseline and starter models as they will likely already have these or can easily create their preferred workflows from their own precedents. But for the legions who are up and coming and some day may challenge these teams, they likely will have :)
\nIf my journey in ML and participating in this competition has taught me anything, its that we all stand on the shoulders of giants!!
", "rawMarkdown": "I am also a believer in obtaining a strong foundation in the first principles of any specialism in which I participate.\n\nFrom my personal experience (as a newbie), the strong starter code and baselines have helped hugely in the learning process, including experimenting with the data prep, learning rates, batch sizes, augmentation, upsampling, the list goes on. Don't get me wrong, I won't use any code unless I understand fully what its doing. Often I will also do external research around these areas when experimenting. But having starter code will have increased my opportunity time for this external research and experimentation by maybe 2-3x.\n\nI get your point that the top data scientists who will likely populate the top of the leaderboard won't have benefited from the baseline and starter models as they will likely already have these or can easily create their preferred workflows from their own precedents. But for the legions who are up and coming and some day may challenge these teams, they likely will have :)\n\nIf my journey in ML and participating in this competition has taught me anything, its that we all stand on the shoulders of giants!!", "votes": 3 }, { "id": 970297, "postDate": "2020-08-14T10:40:00.080Z", "content": "OK, you're not a monkey tester.
", "rawMarkdown": "OK, you're not a monkey tester. " }, { "id": 970299, "postDate": "2020-08-14T10:43:57.023Z", "content": "No I never test monkeys.
", "rawMarkdown": "No I never test monkeys.", "votes": 3 } ] }, { "id": 969679, "postDate": "2020-08-13T21:03:32.473Z", "content": "\n\nI don't get it, some top kagglers are in 1000+ range!!!!
\n
This is public LB. With 77 or 78 positives. One positive more and the score moves by up to 0.006. I would not draw conclusions before competition end. I would not be surprised if some top kagglers jump close to the top then.
", "rawMarkdown": "> I don't get it, some top kagglers are in 1000+ range!!!!\n\nThis is public LB. With 77 or 78 positives. One positive more and the score moves by up to 0.006. I would not draw conclusions before competition end. I would not be surprised if some top kagglers jump close to the top then.", "votes": 1 } ] }, { "id": 966567, "postDate": "2020-08-11T14:27:54.833Z", "content": "Yeah, indeed. I have lots of things wanna try out, but the Google Colab Pro seems not cooperating. It keeps disconnecting if I set the \"num_workers\" a high value. Basically I gave up on this competition a week ago, since I need to focus on my thesis for now. Good luck to all of you!
", "rawMarkdown": "Yeah, indeed. I have lots of things wanna try out, but the Google Colab Pro seems not cooperating. It keeps disconnecting if I set the \"num_workers\" a high value. Basically I gave up on this competition a week ago, since I need to focus on my thesis for now. Good luck to all of you!", "replies": [ { "id": 966578, "postDate": "2020-08-11T14:39:30.907Z", "content": "You probably hit memory limit with too many workers. Each time you increase the number of workers by 1 you add all the memory needed to store your worker data. UNless you use clever multi process data sharing as described in the forum.
", "rawMarkdown": "You probably hit memory limit with too many workers. Each time you increase the number of workers by 1 you add all the memory needed to store your worker data. UNless you use clever multi process data sharing as described in the forum.", "votes": 1 }, { "id": 968322, "postDate": "2020-08-12T21:37:06.097Z", "content": "I don't think so. Because sometimes it can run smoothly, sometimes it just stopped, with the same code, nothing changed. The reason might be, occasionally, when google cloud is not so busy, they allow me to use extra cpus, but if it's getting crowd, they kick me out :).
", "rawMarkdown": "I don't think so. Because sometimes it can run smoothly, sometimes it just stopped, with the same code, nothing changed. The reason might be, occasionally, when google cloud is not so busy, they allow me to use extra cpus, but if it's getting crowd, they kick me out :)." }, { "id": 968428, "postDate": "2020-08-13T02:21:34.260Z", "content": "Same code doe snot mean same execution path, especially if you use multi processing.
", "rawMarkdown": "Same code doe snot mean same execution path, especially if you use multi processing." } ] }, { "id": 967521, "postDate": "2020-08-12T10:03:07.830Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 965719, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-10T19:43:59.067000", "content": "I've tried something new everyday for the past 60 days! And I've read the results from everyone else's notebooks and discussions. I still have a list that I will not finish before the comp ends! And I just read an article yesterday that gave me 3 new promising ideas! There is never enough time!
", "votes": 22, "replies": [ { "id": 965721, "author_name": "CPMP", "author_url": "", "post_date": "2020-08-10T19:46:33.017000", "content": "Indeed!
", "votes": 4, "replies": [] }, { "id": 965797, "author_name": "torch", "author_url": "", "post_date": "2020-08-10T21:28:15.217000", "content": "Any plans to share some recipes from the list after the competition ends? 😍
", "votes": 0, "replies": [] }, { "id": 965822, "author_name": "CPMP", "author_url": "", "post_date": "2020-08-10T21:58:58.387000", "content": "I always share after competition end if my result is not too bad.
\nEdit. Maybe the questions was for Chris actually. I must adjust to times where I am no longer the most popular guy ;)
", "votes": 17, "replies": [] } ] }, { "id": 965840, "author_name": "CPMP", "author_url": "", "post_date": "2020-08-10T22:37:29.003000", "content": "I now have 2 serial downvoters. I feel for them.
\nActually I don't. But I'll stay polite.
", "votes": 6, "replies": [ { "id": 966019, "author_name": "Gajendra Saraswat", "author_url": "", "post_date": "2020-08-11T04:00:47.047000", "content": "Seems like you guys are having a great relationship 👀, that guy is never going to leave your back! 😁
", "votes": 0, "replies": [] }, { "id": 966294, "author_name": "CPMP", "author_url": "", "post_date": "2020-08-11T10:08:29.583000", "content": "There is more than one now.
", "votes": 1, "replies": [] }, { "id": 966306, "author_name": "Gajendra Saraswat", "author_url": "", "post_date": "2020-08-11T10:13:58.620000", "content": "Seems like your negative fan base is increasing!
\nAlthough, on a serious note, I guess its time for Kaggle team to peek into who is this person and ask what he did not like in each post of yours which everyone else is appreciating, although this person in my intuition do not deserve this much attention too.
", "votes": 1, "replies": [] }, { "id": 967026, "author_name": "Helen", "author_url": "", "post_date": "2020-08-11T21:14:19.493000", "content": "You're so funny! 😄
", "votes": 2, "replies": [] }, { "id": 967027, "author_name": "", "author_url": "", "post_date": "2020-08-11T21:14:20.977000", "content": "", "votes": 0, "replies": [] }, { "id": 967109, "author_name": "Signal", "author_url": "", "post_date": "2020-08-12T00:23:29.497000", "content": "Honestly, I really think it's lame that it doesn't show who upvotes/downvotes things. What's the point of anonymity? I get emails when people upvote, but apparently that doesn't happen for downvotes. I don't think that's a good policy and its ripe for abuse.
", "votes": 0, "replies": [] }, { "id": 967716, "author_name": "CPMP", "author_url": "", "post_date": "2020-08-12T13:02:31.447000", "content": "\n\nYou're so funny! 😄
\n
Who is funny, me? Why?
", "votes": 0, "replies": [] }, { "id": 968095, "author_name": "Helen", "author_url": "", "post_date": "2020-08-12T17:40:04.310000", "content": "@cpmpml Yes, you said you had two 2 serial downvoters. It reminds me of serial killers.. Maybe I've seen too many police movies these days. 😄
", "votes": 1, "replies": [] }, { "id": 968672, "author_name": "yimacs", "author_url": "", "post_date": "2020-08-13T07:04:43.020000", "content": "@cpmpml in some cultures, \"you are funny\" means \"you have a great sense of humor\", i think that's what @yuanlin08 meant…
", "votes": 1, "replies": [ { "id": 969516, "author_name": "CPMP", "author_url": "", "post_date": "2020-08-13T18:34:52.537000", "content": "That's how I understood it as well. I was not sure she was speaking of me. hence I asked.
", "votes": 2, "replies": [] } ] }, { "id": 969489, "author_name": "Helen", "author_url": "", "post_date": "2020-08-13T18:02:58.140000", "content": "Yes, i simply wanted to say the expression made me laugh, sorry for the confusion~
", "votes": 1, "replies": [] } ] }, { "id": 965778, "author_name": "Arnaud Roussel", "author_url": "", "post_date": "2020-08-10T21:03:23.117000", "content": "@cpmpml I've also really started late in the competition (last 17 days). I budgeted and planned everything. Did all my trials like some of yours during last week kept all the things that increase local CV and now I have 1 week to go down the list of model I want to train for later ensemble. Using google colab to do the bigger ones. The last 2 days will be focused on ensembling all of this.
\nUsing pytorch.
\nhere is what I tried:
\nOrder of tests:
\n Base B2 256
\n Color Consistency +
\n Dense head -
\n Coarse + Hair augmentation ++
\n Include 2018 data +
\n Include extra_malignant data +
\n Concat Pooling -
\n Resnest50 (backbone) ++
\n 20 epochs (with resnest50) -
\n Include 2019 data =
\n Posweight = 10 =
\n Pseudo Labelling Failed, no time to fix
\n add Meta data -- no time to find what's going wrong
Thanks a lot. I'll share my list with evals like yours tomorrow. But I confirm coarse is a killer.
", "votes": 3, "replies": [] }, { "id": 966311, "author_name": "CPMP", "author_url": "", "post_date": "2020-08-11T10:18:13.780000", "content": "As promised, list of things I tried (I certainly forget some and will add them as I remember them later). I am using pytorch too.
\nBase B0 256
\nHair removal -
\nColor Consistency +
\nMixup +
\nOther loss than BCE -
\nFp16 +
\nOptimizer+scheduler tuning + 20 epochs +
\nAlbumentations ++
\nCrop -
\nGem pooling -
\nTta+
\nUpscaling -
\nPosweight increase -
\nLabel smoothing -
\nCoarse ++
\nAll 2019 data ++
\nOnly 2019 positive --
\nCustom head ++
Currently trying hair augmentation, but first try is disappointing. I guess I need less coarse dropout.
", "votes": 10, "replies": [ { "id": 966345, "author_name": "Innat", "author_url": "", "post_date": "2020-08-11T10:53:55.657000", "content": "I've found GAP is more promising than GeM pooling (most of my model training). TTA helps a lot. Label smoothing (in my case) with BCE kinda shows better than Focal loss. Color consistency, not sure but didn't show any sign of improvement. I've many custom head, + followed the previous winning solution but in my case, simple approach was enough. MixUp/CutMix, couldn't make it work, I guess, need some solid tricks. However, I'm using tensorflow though.
", "votes": 0, "replies": [] }, { "id": 970419, "author_name": "Anil Kumar Reddy", "author_url": "", "post_date": "2020-08-14T12:51:18.753000", "content": "we have unbalance data and again you want to try more data of un-balanced 2019 ++ , my qns was deep learning models we know more the data better the model perform but how about more un balanced data ??
", "votes": 0, "replies": [] } ] }, { "id": 966335, "author_name": "Dieter", "author_url": "", "post_date": "2020-08-11T10:37:01.740000", "content": "why are you sharing this now? How about sharing when comp is over
", "votes": 3, "replies": [] }, { "id": 966352, "author_name": "CPMP", "author_url": "", "post_date": "2020-08-11T10:59:28.400000", "content": "I won't share more, but I don't think I shared anything new either. And my current results is such that my list isn't very valuable.
", "votes": 5, "replies": [] }, { "id": 966663, "author_name": "Waylon Wu", "author_url": "", "post_date": "2020-08-11T15:44:26.887000", "content": "Could you explain what is color consistence here?
", "votes": 0, "replies": [ { "id": 966782, "author_name": "Innat", "author_url": "", "post_date": "2020-08-11T16:55:32.320000", "content": "@waylongo
\nhttps://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/154876#867412
@cpmpml Hair Augmentation worked for me, B1/256 did better than B0/256 and WeightedFocalLoss works better than BCE BUT understand its designed for imbalanced datasets so if you balance your dataset don't expect FocalLoss to do better. alpha of .75-.80, gamma of 2
", "votes": 1, "replies": [] }, { "id": 967147, "author_name": "Signal", "author_url": "", "post_date": "2020-08-12T02:48:52.637000", "content": "@arroqc posweight 10 you mean for class 1? and then just using a weight of 1 for class 0? I have tried various weights, and even using scikits weight estimator. I think it helps, but wasn't a huge deal. I assume you are using BCE with logits loss, since that takes pos weights. I have also found focal loss to be a decent.
", "votes": 0, "replies": [] }, { "id": 967527, "author_name": "yash chaudhary", "author_url": "", "post_date": "2020-08-12T10:06:09.717000", "content": "weird including 2019 data reduces the score somehow.
", "votes": 1, "replies": [] }, { "id": 968098, "author_name": "Nirjhar Roy", "author_url": "", "post_date": "2020-08-12T17:40:53.570000", "content": "\nIn Pytorch :
\nB0 384
\nColor Consistency -Didnt try
\nDense head -
\nCoarse + Hair augmentation ++
\nInclude 2018 data +
\nInclude extra_malignant data =
\nConcat Pooling -
\nGeM Pooling =
\nInclude 2019 data =
\nPosweight = 3/10/15 =
\nPseudo Labelling Failed, no time to fix
\nadd Meta data --
How to apply color consistency,Mixups ? Couldnt find them in Albumentation
\nAnd what is FP16 ?
FP16 is half precision, faster, less memory, not stable on all GPU's…..stable on those that have Tensor Cores.
", "votes": 1, "replies": [] }, { "id": 969116, "author_name": "Haider Ali Shuvo", "author_url": "", "post_date": "2020-08-13T13:16:06.407000", "content": "Cool , How to apply Custom Head/Mixups in albumentation framework ?You guys have Any code link for that ?
", "votes": 0, "replies": [ { "id": 969226, "author_name": "CPMP", "author_url": "", "post_date": "2020-08-13T14:57:13.760000", "content": "custom head means a change in the NN architecture.
\nmixup is combining two images. You cannot do that in albumentation given it works single image per image. If you google it then you'll find code for mixup. Or if you look at past competition forums, or even this competition forum probably.
", "votes": 2, "replies": [] }, { "id": 969269, "author_name": "Haider Ali Shuvo", "author_url": "", "post_date": "2020-08-13T15:22:56.553000", "content": "Okay ,Thanks :D I will try :D
", "votes": 0, "replies": [] } ] } ] }, { "id": 968662, "author_name": "Ertuğrul Demir", "author_url": "", "post_date": "2020-08-13T06:58:15.953000", "content": "I might update this time to time when I remember what I tried or try something new:
\nGoing to try concat meta and image inputs in network soon, again was planning to try shades of gray but I think it's kinda late for it to creating preprocessed dataset with it…
", "votes": 3, "replies": [ { "id": 968867, "author_name": "kaggler", "author_url": "", "post_date": "2020-08-13T09:43:35.620000", "content": "I have tried meta and image dual input for CNN from 384 to 768 size, it doesn't work for LB(but it surely boosted CV)
\nIf you notice any improvements in concatenating image and meta, sharing it here would be great :D
I haven't used any meta-data so far… that might be risky for me :)
", "votes": 2, "replies": [] }, { "id": 969348, "author_name": "CPMP", "author_url": "", "post_date": "2020-08-13T16:15:57.743000", "content": "Maybe that's why you made progress quickly rather ;)
", "votes": 1, "replies": [] }, { "id": 969683, "author_name": "Ertuğrul Demir", "author_url": "", "post_date": "2020-08-13T21:05:56.907000", "content": "@deepkim tried basic version today but only on 256x because of timing problems got 93+ lb and 94+ cv, sadly out of TPU time on kaggle now, have to test it later on colab… But it was promising for me
", "votes": 1, "replies": [] } ] }, { "id": 965748, "author_name": "Ertuğrul Demir", "author_url": "", "post_date": "2020-08-10T20:15:46.843000", "content": "I tried many things, most of them didn't work some did, then tried to merge what worked before, again some worked some didn't. Still have many ideas to try but not enough time or computing power left.
\nThere is another list of ideas where in theory and beyond my skill so that's an another list for things to research and learn after the competition. I'm kinda new to competitions but feels like I'll end up with list of lists 😀
", "votes": 3, "replies": [] }, { "id": 965852, "author_name": "Jacek Poplawski", "author_url": "", "post_date": "2020-08-10T22:53:26.070000", "content": "I used this competition to gain experience in using TPU with Pytorch. I also learned some new things to me, like OneCycleLR and improved my image processing in Python. I hope I can use these skills in next image competition, however looks like Pytorch + TPU is still much worse combination than Tensorflow.
", "votes": 4, "replies": [] }, { "id": 972188, "author_name": "Matrix", "author_url": "", "post_date": "2020-08-16T10:08:14.967000", "content": "My experiences:
\nI stopped working on this competition at all few days ago, but I am still here to see the huge shakeup and surprised faces :)
", "votes": 1, "replies": [ { "id": 970188, "author_name": "Signal", "author_url": "", "post_date": "2020-08-14T09:15:15.883000", "content": "@jacekpoplawski I hope someone makes a visualization, maybe using Parallel Coordinates, with scores on the left Public LB and scores on the right Private LB, I can imagine many of them clustered and moving the same. I hope I am one of them, but moving up!
", "votes": 1, "replies": [] } ] }, { "id": 968854, "author_name": "Gajendra Saraswat", "author_url": "", "post_date": "2020-08-13T09:30:56.307000", "content": "I want to try predicting diagnosis too together with label. I read in some discussions that it certainly is giving better results. If anyone tries this out, do share your results! :)
", "votes": 1, "replies": [] }, { "id": 967422, "author_name": "", "author_url": "", "post_date": "2020-08-12T08:38:25.223000", "content": "i try hair augmentation，but i dont know why it become lower
", "votes": 1, "replies": [ { "id": 967655, "author_name": "CPMP", "author_url": "", "post_date": "2020-08-12T12:07:19.367000", "content": "I tried as I wrote somewhere in this thread.
", "votes": 0, "replies": [] } ] }, { "id": 966242, "author_name": "Tord Malmgren", "author_url": "", "post_date": "2020-08-11T09:10:46.843000", "content": "And computers are never fast enough…
", "votes": 1, "replies": [ { "id": 966293, "author_name": "CPMP", "author_url": "", "post_date": "2020-08-11T10:07:52.723000", "content": "Actually, not having too much compute resource is beneficial: it forces you to think more before running experiments.
", "votes": 26, "replies": [] }, { "id": 966357, "author_name": "", "author_url": "", "post_date": "2020-08-11T11:10:18.613000", "content": "", "votes": 0, "replies": [] }, { "id": 967519, "author_name": "Maksim Zhdanov", "author_url": "", "post_date": "2020-08-12T10:02:27.663000", "content": "On the other hand, some part of data science competitions is about how fast you can iterate your ideas.
", "votes": 2, "replies": [] }, { "id": 969273, "author_name": "CPMP", "author_url": "", "post_date": "2020-08-13T15:23:24.403000", "content": "Having lost of resources near the end of comp to train at scale is indeed useful too. But having too much resources early isn't great IMHO.
", "votes": 1, "replies": [] } ] }, { "id": 965844, "author_name": "Innat", "author_url": "", "post_date": "2020-08-10T22:45:34.707000", "content": "@cpmpml
\nvery true, many things to try yet but no enough time :(
\nanyway, I was thinking, it would be nice if we can see who downvote because in that case, the downvoter will have the reason why he/she do this. 😄
for me it's \"join late and only got a single 1080ti\"...
\n\ni have a list similar to yours, and I just added \"testing weight decay while ignoring BN layers\" to it.
", "votes": 1, "replies": [ { "id": 965720, "author_name": "CPMP", "author_url": "", "post_date": "2020-08-10T19:45:49.833000", "content": "I went through this and a number of things fortunately, including optimizer, scheduler, fp16, augmentations, 2019 data, etc. But the more I do, the more I see left to do...
\n\nGood luck anyway. But given you only have one GPU I would consider using Kaggle kernels with both TPU and GPU in addition to your machine.
", "votes": 2, "replies": [] } ] }, { "id": 968596, "author_name": "Abhishek C. Salian", "author_url": "", "post_date": "2020-08-13T06:11:45.770000", "content": "Hair Augmentation doesn't give a major increase in the lb score we had tried it out although the cv score increases.
", "votes": 2, "replies": [ { "id": 968608, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-13T06:21:59.303000", "content": "Thanks for the update. I've tried hundreds of things but I haven't tried hair augmentation yet.
", "votes": 1, "replies": [] }, { "id": 969014, "author_name": "CPMP", "author_url": "", "post_date": "2020-08-13T12:04:54.230000", "content": "I didn't see any benefit from hair augmentation on cv so far.
", "votes": 2, "replies": [] }, { "id": 969532, "author_name": "Phaedrus", "author_url": "", "post_date": "2020-08-13T18:51:33.503000", "content": "Chris, have you tried any post processing method so far. Just curious.
", "votes": 0, "replies": [] } ] }, { "id": 967522, "author_name": "yash chaudhary", "author_url": "", "post_date": "2020-08-12T10:03:07.867000", "content": "let me answer some that I tested :
\nImage size increases the score for sure , tricky part is too fit within the 3-hour TPU limit
\nhaven't tried hair augment yet
\nupsampling is kinda insignificant
\neffnet b6 is giving me best results, got better than b7
\nmean average got me good results
\nthe tabular ensemble with image prediction gives a significant boost to public LB
\nThe public notebook is screwing the LB, to be honest, but helped me too in the single model case.
B6 is giving better results in my case too!
", "votes": 1, "replies": [] }, { "id": 967653, "author_name": "CPMP", "author_url": "", "post_date": "2020-08-12T12:06:55.817000", "content": "@yash612 thanks.
\n\n\nImage size increases the score for sure , tricky part is too fit within the 3-hour TPU limit
\n
I am not using TPU nor am I using @cdeotte code. Therefore some of the conclusions one can get out of Chris code may not apply to other ways of doing things. Anyway, thanks for the feedback on the rest of your list. I am kind of reassured you also don't see much benefit form upsampling as this is the case for me. I haven't tried b6 yet, will do at a point for sure. Working on meta data is on my todo list.
", "votes": 1, "replies": [] } ] }, { "id": 967207, "author_name": "Ahmed Sabry", "author_url": "", "post_date": "2020-08-12T05:03:59.277000", "content": "@cpmpml I also tried upsampling (per batch upsampling) but it did not work. I changed this to per epoch downsampling where I trained on 65% random selection of the negative samples + all positive per epoch and it worked well.
\nI also feel that the problem in scores is not about TF or torch, I think it is about the datasets. I am using Tensorflow with the JPEG dataset and I got best single model score 0.939 with basic TTA.
\nall public kernels scoring 0.95+ are using the tfrecord datasets.
Thanks. Your downsampling is interesting.
", "votes": 0, "replies": [] }, { "id": 968423, "author_name": "Arnaud Roussel", "author_url": "", "post_date": "2020-08-13T02:12:06.813000", "content": "@cdeotte Could the TFRecords values in your datasets be before JPEG compression ?
\nEdit: Nevermind saw your code and you encode at JPEG94. I imagine you do that also for the .jpg files.
I wish we saw the code used for generating all datasets.
", "votes": 1, "replies": [] }, { "id": 968429, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-13T02:21:41.960000", "content": "@arroqc I don't understand your question. Both my TFRecords and JPEG dataset come from the same NumPy image array. After reading in original Kaggle jpeg images, they are max square center crop, then resize with cv2.resize(img,(DIM,DIM),interpolation = cv2.INTER_AREA), then the image gets saved into a NumPy array of shape (len(train), DIM, DIM, 3)
The TFRecords are made by compressing with cv2.imencode('.jpg', img)[1].tostring() and the JPEG on disk are made with cv2.imwrite(PATH+files[k],img)
Yes I'm starting to wonder if the differences mentioned by @asabry79400 are not due to some different processes between what is in a record and what is in a .jpeg.
", "votes": 1, "replies": [] }, { "id": 968432, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-13T02:27:13.570000", "content": "\n\nEdit: Nevermind saw your code and you encode at JPEG94. I imagine you do that also for the .jpg files.
\n
I only encode that notebook at JPEG94 which makes my https://www.kaggle.com/cdeotte/512x512-melanoma-tfrecords-70k-images dataset
\nAll my other TFRecords datasets use the cv2.imencode('.jpg', img)[1].tostring() which i believe has default CV_IMWRITE_JPEG_QUALITY = 95. And JPEG datasets use cv2.imwrite(PATH+files[k],img) which also has default CV_IMWRITE_JPEG_QUALITY = 95
@cdeotte Well you seem to have perfectly understood the question thanks for the answer. I was wondering if the method of saving images are different. I would expect both the default for imencode and imwrite to be the same though.
", "votes": 0, "replies": [] }, { "id": 968434, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-13T02:30:56.500000", "content": "\n\nI wish we saw the code used for generating all datasets.
\n
Here is the full code to generate my JPEG and TFRecord datasets:
\nDIM = 256\nimages = np.zeros((33126,DIM,DIM,3),dtype='uint8')\n\nfor k in range(len(files)):\n img = cv2.imread(PATH2+files[k])\n w = img.shape[1]\n h = img.shape[0]\n s = min(w,h)\n w2 = (w-s)//2\n h2 = (h-s)//2\n img = img[h2:h-h2,w2:w-w2,:]\n img = cv2.resize(img,(DIM,DIM),interpolation = cv2.INTER_AREA)\n cv2.imwrite(PATH2+files[k],img)\n img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n images[k,] = img\nThe above code both writes the JPEG dataset and saves the images into a NumPy array. In the same offline Jupyter notebook, the TFRecords are made later with
\n img = cv2.cvtColor(images[row.i.values[0],], cv2.COLOR_RGB2BGR)\n img = cv2.imencode('.jpg', img)[1].tostring()\n example = serialize_example(\n img, str.encode(name),\n row.patient_id.values[0],\n row.sex.values[0],\n row.age_approx.values[0], \n row.anatom_site_general_challenge.values[0],\n row.diagnosis.values[0],\n row.target.values[0],\n row.width.values[0],\n row.height.values[0])\n writer.write(example)\nThanks as a quick try I tried the following:
\nimg = cv2.imread('H:/SIIM/768/train/ISIC_0000070.jpg')\n\nstring = cv2.imencode('.jpg', img)[1].tostring()\ncv2.imwrite('test.jpg',img)\nnparr = np.frombuffer(string, np.uint8)\nnew_img_records = cv2.imdecode(nparr, cv2.IMREAD_COLOR)\n\nnew_img_jpeg = cv2.imread('test.jpg')\nprint(np.all(new_img_jpeg == new_img_records))\n\nfrom PIL import Image\nnew_img_pil = cv2.cvtColor(np.array(Image.open('test.jpg')), cv2.COLOR_RGB2BGR)\nprint(np.all(new_img_pil == new_img_records))\nprint(np.mean(np.abs(new_img_pil - new_img_records)))\nFirst is True, second is False and average difference is ~5. Note that visually I can't see the difference and this proves nothing. Interesting though.
", "votes": 1, "replies": [] }, { "id": 968451, "author_name": "Signal", "author_url": "", "post_date": "2020-08-13T03:18:53.203000", "content": "@cdeotte if you want to see some serious speed increase in running that, just install pythons Parallel module (from pip or wherever). The more cores the better, crazy fast. Then:
num_cores = 12 \n\nDIM = 256\nimages = np.zeros((33126,DIM,DIM,3),dtype='uint8')\n\ndef resize(k):\n img = cv2.imread(PATH2+files[k])\n w = img.shape[1]\n h = img.shape[0]\n s = min(w,h)\n w2 = (w-s)//2\n h2 = (h-s)//2\n img = img[h2:h-h2,w2:w-w2,:]\n img = cv2.resize(img,(DIM,DIM),interpolation = cv2.INTER_AREA)\n cv2.imwrite(PATH2+files[k],img)\n img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n images[k,] = img\n\nParallel(n_jobs=num_cores, verbose=10)(delayed(resize)(k) for k in range(len(files)))\nNote: Edited this to match
", "votes": 2, "replies": [] }, { "id": 968452, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-13T03:21:21.620000", "content": "Wow, awesome, thanks @brianfeeny I'll try that.
", "votes": 1, "replies": [] }, { "id": 969020, "author_name": "CPMP", "author_url": "", "post_date": "2020-08-13T12:09:09.530000", "content": "The theory I mentioned previously was indeed that difference in data explains difference in model performance.
", "votes": 1, "replies": [] }, { "id": 969520, "author_name": "Psi", "author_url": "", "post_date": "2020-08-13T18:37:43.103000", "content": "@cpmpml Is there difference? From the analysis it does not seem so.
", "votes": 0, "replies": [ { "id": 969673, "author_name": "CPMP", "author_url": "", "post_date": "2020-08-13T20:59:08.127000", "content": "I saw a difference in my first try, but not in the second. I guess it is noise.
", "votes": 0, "replies": [] } ] }, { "id": 969657, "author_name": "Arnaud Roussel", "author_url": "", "post_date": "2020-08-13T20:42:13.683000", "content": "What I find really suspicious is that while some of these public kernels get good LB like 0.95, their CV is really mediocre like 0.91 or 0.92. I have some models in the 0.93 region that use the exact same splits but \"just\" give 0.94 LB. I'm going to trust CV more but I wonder if there is more to the story.
", "votes": 2, "replies": [] }, { "id": 969675, "author_name": "Signal", "author_url": "", "post_date": "2020-08-13T21:00:57.247000", "content": "@arroqc is that really remarkable in this comp though? I mean I have models that are .89 and .90 in CV and do .93 on LB. It's frustrating for sure, that things aren't tighter with the scoring, which apparently is due to the few amount of positives in the dataset.
", "votes": 0, "replies": [] }, { "id": 969693, "author_name": "Arnaud Roussel", "author_url": "", "post_date": "2020-08-13T21:16:45.740000", "content": "@brianfeeny No I'm well aware of that. My point was more about I wouldn't put that much trust in the score of the public kernel but also wondering if there is indeed more to it than \"LB is noisy\". That said when running some of them I had variation up to 0.01 in score (the ones involving training).
", "votes": 0, "replies": [] }, { "id": 969866, "author_name": "Ahmed Sabry", "author_url": "", "post_date": "2020-08-14T02:51:19.650000", "content": "@philippsinger I noticed you recently improved your score, did you use the TF data?
", "votes": 0, "replies": [] }, { "id": 970648, "author_name": "Psi", "author_url": "", "post_date": "2020-08-14T16:27:06.497000", "content": "no, I am too stubborn to use tensorflow :D
", "votes": 5, "replies": [] }, { "id": 970690, "author_name": "Aptha K S", "author_url": "", "post_date": "2020-08-14T17:13:19.857000", "content": "The golden rule: \"Do not put each foot in a different boat\".
", "votes": 0, "replies": [] } ] }, { "id": 965743, "author_name": "Shiro", "author_url": "", "post_date": "2020-08-10T20:09:39.897000", "content": "I would advice you to try TF instead of pytorch. I did not succeed to have as good as results in pytorch for this competition :/ model with 384 and 512 give very good results. I can have 0.95X with them on public LB
", "votes": 2, "replies": [ { "id": 965759, "author_name": "Psi", "author_url": "", "post_date": "2020-08-10T20:37:00.397000", "content": "This makes me sad. Any idea why this is the case?
\nthere is no single high scoring LB pytorch kernel this time
", "votes": 6, "replies": [ { "id": 965791, "author_name": "torch", "author_url": "", "post_date": "2020-08-10T21:24:54.260000", "content": "Don't mean to mock but you've answered the question to this here: https://www.kaggle.com/c/tweet-sentiment-extraction/discussion/143869
\nPeople are using ensemble of EfficientNet's as one single model plus some additional techniques as well but when I compared this seemed as the main ingredient. When I tried to implement the same with torch or torch_xla there were oom issues most of the time.
\nI really admire you btw, just wanted to let you know.
", "votes": 0, "replies": [] } ] }, { "id": 965766, "author_name": "Shiro", "author_url": "", "post_date": "2020-08-10T20:50:25.070000", "content": "Unfortunately, I am not sure. It is the first time I was using efficientNet (on both TF/pytorch). My experiment was using TPU, and I have already seen some little drop of performance with pytorch tpu on nlp task. But for this image task , I was wondering if it was not the library. I will probably try to do some experiment on cifar or similar dataset to compare the TF VS torch for efficientNet on gpu. It really bugs me.
", "votes": 0, "replies": [] }, { "id": 965821, "author_name": "CPMP", "author_url": "", "post_date": "2020-08-10T21:58:11.283000", "content": "I have a theory about why Pytorch works worse than TF here. But I need to experiment to validate it. Will report one way or the other when done.
", "votes": 4, "replies": [] }, { "id": 965828, "author_name": "CPMP", "author_url": "", "post_date": "2020-08-10T22:13:27.800000", "content": "\n\nI would advice you to try TF instead of pytorch.
\n
i will run some public TF kernels for sure to add diversity, but I won't invest time in developing something new. Once you have tried pytorch it is hard to go back.
", "votes": 8, "replies": [] }, { "id": 966055, "author_name": "Phaedrus", "author_url": "", "post_date": "2020-08-11T05:21:14.727000", "content": "Whats the theory?
", "votes": 0, "replies": [] }, { "id": 966192, "author_name": "Aptha K S", "author_url": "", "post_date": "2020-08-11T08:18:41.310000", "content": "I thought because of Tensorflow's better compatibility with TPU people, are using it instead of Pytorch. Still, I am new to this, am I missing something?
", "votes": 0, "replies": [] }, { "id": 966361, "author_name": "CPMP", "author_url": "", "post_date": "2020-08-11T11:12:46.527000", "content": "TPU are designed to run Tensorflow, no wonder it works well. But this would explain why training TF on TPU is fast. It does not explain why Pytorch models seem to have lower quality here.
", "votes": 2, "replies": [] }, { "id": 966363, "author_name": "CPMP", "author_url": "", "post_date": "2020-08-11T11:14:22.180000", "content": "\n\nWhats the theory?
\n
I won't share now as I have not tested it yet. It can be one of the many promising ideas that miserably fail when implemented…
\nAnd given we are approaching competition end, as my colleague @christofhenkel rightfully reminded me, it is better to wait till competition end.
", "votes": 2, "replies": [] }, { "id": 966480, "author_name": "Kha Vo", "author_url": "", "post_date": "2020-08-11T12:59:40.993000", "content": "I think not because Pytorch is not good but TPU+TF dominate.
", "votes": 1, "replies": [] }, { "id": 966490, "author_name": "CPMP", "author_url": "", "post_date": "2020-08-11T13:07:49.280000", "content": "I think one reason is that @cdeotte baseline is extremely strong, and most participants built on top of it.
\nIn competitions where someone shared early enough a strong pytorch baseline, for instance Abhishek Thakur baseline in Tweet Sentiment, then most participants use pytorch.
", "votes": 5, "replies": [] }, { "id": 966531, "author_name": "Dieter", "author_url": "", "post_date": "2020-08-11T13:56:29.867000", "content": "Yeah to call it \"simple baseline\" is a bad joke imho. Its a heavy efficientnet with triple stratification cv, label smoothing, external data, highly tuned augmentation and tta.
", "votes": 11, "replies": [] }, { "id": 966581, "author_name": "CPMP", "author_url": "", "post_date": "2020-08-11T14:41:38.453000", "content": "oh, Chris has label smoothing too? That's why people asked for pytorch code. Interesting. I should definitely look at his code now.
", "votes": 1, "replies": [] }, { "id": 966738, "author_name": "Signal", "author_url": "", "post_date": "2020-08-11T16:25:44.810000", "content": "There may be some high scoring PyTorch, we will see. I thought Serigne was using PyTorch and he is top 100
", "votes": 0, "replies": [] }, { "id": 966740, "author_name": "Signal", "author_url": "", "post_date": "2020-08-11T16:29:02.190000", "content": "You are right, I think many built on top of Chris's excellent work, and there is nothing wrong with that, it raises the bar, makes everything more challenging and overall will produce a better result. However, there was supposedly some notebook shared, that I don't know anything about, that apparently several hundred people are using or whatever, and that caused the leaderboard to go crazy. At least someone should reveal what was in that notebook that gave them so much advantage, at least then we could level the playing field a little. The notebook was taken down after a few hours and people have been tight-lipped about it………meanwhile, so many others continue to share helpful tips, I just wish that info in that public notebook wasn't squandered.
", "votes": 1, "replies": [] }, { "id": 966827, "author_name": "CPMP", "author_url": "", "post_date": "2020-08-11T17:21:06.040000", "content": "You can go to Notebook tab and sort by best score. Current is at 0.9619 https://www.kaggle.com/paklau9/minmax-highest-public-lb-9619
", "votes": 1, "replies": [] }, { "id": 966954, "author_name": "Signal", "author_url": "", "post_date": "2020-08-11T19:40:45.130000", "content": "yeah, that's not even a real kernel though. I mean, I guess Kaggle allows you to include your submission files/scores too? So you have maybe 100+ people submitting a .9600………..and they all have the same values, out to like 9 significant digits, for every prediction……..how is that legal? Or, in the end will they be cleaved? I don't know much about how Kaggle deals with all that. Personally I don't think you should be able to include scores/predictions in kernels that are nothing more than blenders/stackers/ensembles. Even with regular kernels, why share predictions.
", "votes": 0, "replies": [] }, { "id": 968958, "author_name": "Serigne ", "author_url": "", "post_date": "2020-08-13T11:21:23.873000", "content": "@brianfeeny My best Pytorch model won't definitely fit on kernel (weak RAM and low CPU cores on kernels) .
\nI think @yuval6967 has the best Pytorch model ( LB 0.960 reported much earlier in this comp , thus I think he found way to improved since then)
", "votes": 1, "replies": [] }, { "id": 970360, "author_name": "coolz", "author_url": "", "post_date": "2020-08-14T11:37:44.837000", "content": "Join the game late, and i use pytorch.
\nIt is easy to achieve 0.94+ with effnetb0, 256x256 input.
But with higher resolution or bigger model, the score basicly the same. And that make me feel disappointed.
", "votes": 0, "replies": [] }, { "id": 970393, "author_name": "yuval reina", "author_url": "", "post_date": "2020-08-14T12:11:12.007000", "content": "@serigne My model is not the \"best Pytorch model\" - it might be the reported single model with the best public LB. But this doesn't say anything. I don't believe in the public LB, the number of true values is too small. - \"Shakeup is Coming …\"
", "votes": 2, "replies": [] }, { "id": 970499, "author_name": "Serigne ", "author_url": "", "post_date": "2020-08-14T14:06:41.203000", "content": "You're right ! I should say best public LB
", "votes": 0, "replies": [] } ] }, { "id": 971447, "author_name": "Ali Abdin", "author_url": "", "post_date": "2020-08-15T14:31:14.770000", "content": "The things I want to try before it is over:
\n2 Days to go ;)
", "votes": 0, "replies": [] }, { "id": 969551, "author_name": "Aptha K S", "author_url": "", "post_date": "2020-08-13T19:07:34.213000", "content": "I don't get it, some top kagglers are in 1000+ range!!!! Are all those core PyTorch users or the LB is really really really overfitted??
", "votes": 0, "replies": [ { "id": 969604, "author_name": "Signal", "author_url": "", "post_date": "2020-08-13T19:56:40.237000", "content": "There are people in the 100's, and certain < 500 ranking that are brand new and asking questions you would find in the first few chapters of a good book on machine learning. Literally, newbies, who have no idea what they are doing. The solutions don't have to be overfitting, they could be valid, just shared, I guess time will tell.
", "votes": 2, "replies": [ { "id": 969680, "author_name": "CPMP", "author_url": "", "post_date": "2020-08-13T21:05:04.763000", "content": "That's the issue of strong public baselines: lots of monkey testing happens. And some will get lucky.
", "votes": 2, "replies": [] }, { "id": 970199, "author_name": "jsyphil", "author_url": "", "post_date": "2020-08-14T09:23:38.347000", "content": "Half a dozen very strong public baseline and 'starter' models has raised the game generally and has allowed competitors to focus on experimentation and discovery rather than hours of coding and debugging. This is very good in the context of the type of competition this is, medical research to benefit cancer victims. Better to spend the time on innovation than hacking.
", "votes": 0, "replies": [] }, { "id": 970241, "author_name": "CPMP", "author_url": "", "post_date": "2020-08-14T09:40:20.400000", "content": "A lot of learning opportunities were lost by avoiding all the hurdles around data preparation, cv setting, scheduler tuning, etc. It is better to learn people how to fish than giving them a fish.
\nWell, that's my way of thinking, and I get that other ways are more popular.
\nThe argument that the final solution will be better isn't valid IMHO. Top teams don't need a public strong baseline.
", "votes": 1, "replies": [] }, { "id": 970265, "author_name": "jsyphil", "author_url": "", "post_date": "2020-08-14T10:00:09.817000", "content": "I am also a believer in obtaining a strong foundation in the first principles of any specialism in which I participate.
\nFrom my personal experience (as a newbie), the strong starter code and baselines have helped hugely in the learning process, including experimenting with the data prep, learning rates, batch sizes, augmentation, upsampling, the list goes on. Don't get me wrong, I won't use any code unless I understand fully what its doing. Often I will also do external research around these areas when experimenting. But having starter code will have increased my opportunity time for this external research and experimentation by maybe 2-3x.
\nI get your point that the top data scientists who will likely populate the top of the leaderboard won't have benefited from the baseline and starter models as they will likely already have these or can easily create their preferred workflows from their own precedents. But for the legions who are up and coming and some day may challenge these teams, they likely will have :)
\nIf my journey in ML and participating in this competition has taught me anything, its that we all stand on the shoulders of giants!!
", "votes": 3, "replies": [] }, { "id": 970297, "author_name": "CPMP", "author_url": "", "post_date": "2020-08-14T10:40:00.080000", "content": "OK, you're not a monkey tester.
", "votes": 0, "replies": [] }, { "id": 970299, "author_name": "jsyphil", "author_url": "", "post_date": "2020-08-14T10:43:57.023000", "content": "No I never test monkeys.
", "votes": 3, "replies": [] } ] }, { "id": 969679, "author_name": "CPMP", "author_url": "", "post_date": "2020-08-13T21:03:32.473000", "content": "\n\nI don't get it, some top kagglers are in 1000+ range!!!!
\n
This is public LB. With 77 or 78 positives. One positive more and the score moves by up to 0.006. I would not draw conclusions before competition end. I would not be surprised if some top kagglers jump close to the top then.
", "votes": 1, "replies": [] } ] }, { "id": 966567, "author_name": "godelscat", "author_url": "", "post_date": "2020-08-11T14:27:54.833000", "content": "Yeah, indeed. I have lots of things wanna try out, but the Google Colab Pro seems not cooperating. It keeps disconnecting if I set the \"num_workers\" a high value. Basically I gave up on this competition a week ago, since I need to focus on my thesis for now. Good luck to all of you!
", "votes": 0, "replies": [ { "id": 966578, "author_name": "CPMP", "author_url": "", "post_date": "2020-08-11T14:39:30.907000", "content": "You probably hit memory limit with too many workers. Each time you increase the number of workers by 1 you add all the memory needed to store your worker data. UNless you use clever multi process data sharing as described in the forum.
", "votes": 1, "replies": [] }, { "id": 968322, "author_name": "godelscat", "author_url": "", "post_date": "2020-08-12T21:37:06.097000", "content": "I don't think so. Because sometimes it can run smoothly, sometimes it just stopped, with the same code, nothing changed. The reason might be, occasionally, when google cloud is not so busy, they allow me to use extra cpus, but if it's getting crowd, they kick me out :).
", "votes": 0, "replies": [] }, { "id": 968428, "author_name": "CPMP", "author_url": "", "post_date": "2020-08-13T02:21:34.260000", "content": "Same code doe snot mean same execution path, especially if you use multi processing.
", "votes": 0, "replies": [] } ] }, { "id": 967521, "author_name": "", "author_url": "", "post_date": "2020-08-12T10:03:07.830000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "965703": "That's the joy of joining late. You can sit on shoulders of giants, and you can make progress much faster. But...\n\nList of things I haven't tried yet. \n\n- image size other than 256x256\n- hair augmentation\n- upsampling ( I tried but don't get any benefit yet)\n- other model than effnetb0 from efficientnet-pytorch\n- ensembling other than average\n- use of tabular data per sample\n- use of patient context across samples\n- look at public notebooks\n\nWhen I see this list I'm tempted to move to my next competition... At the same time I welcome additional suggestions. I must be a masochist. \n\nWhat pushes me is that many of you are in same situation probably, with too many things to try given the time left.", "965719": "I've tried something new everyday for the past 60 days! And I've read the results from everyone else's notebooks and discussions. I still have a list that I will not finish before the comp ends! And I just read an article yesterday that gave me 3 new promising ideas! There is never enough time!", "965840": "I now have 2 serial downvoters. I feel for them.\n\nActually I don't. But I'll stay polite.", "965778": "@cpmpml I've also really started late in the competition (last 17 days). I budgeted and planned everything. Did all my trials like some of yours during last week kept all the things that increase local CV and now I have 1 week to go down the list of model I want to train for later ensemble. Using google colab to do the bigger ones. The last 2 days will be focused on ensembling all of this.\n\nUsing pytorch.\n\nhere is what I tried:\n\n Order of tests:\n Base B2 256\n Color Consistency +\n Dense head -\n Coarse + Hair augmentation ++\n Include 2018 data +\n Include extra_malignant data +\n Concat Pooling -\n Resnest50 (backbone) ++\n 20 epochs (with resnest50) -\n Include 2019 data =\n Posweight = 10 =\n Pseudo Labelling Failed, no time to fix\n add Meta data -- no time to find what's going wrong", "968662": "I might update this time to time when I remember what I tried or try something new:\n\n- Progressive coarse dropout helped on cv not on lb(if you care about it),\n- Again same with upsampling,\n- Custom heads didn't work for me,\n- Weighted binary crossentropy didn't work again\n\nGoing to try concat meta and image inputs in network soon, again was planning to try shades of gray but I think it's kinda late for it to creating preprocessed dataset with it...", "965748": "I tried many things, most of them didn't work some did, then tried to merge what worked before, again some worked some didn't. Still have many ideas to try but not enough time or computing power left.\n\nThere is another list of ideas where in theory and beyond my skill so that's an another list for things to research and learn after the competition. I'm kinda new to competitions but feels like I'll end up with list of lists 😀", "965852": "I used this competition to gain experience in using TPU with Pytorch. I also learned some new things to me, like OneCycleLR and improved my image processing in Python. I hope I can use these skills in next image competition, however looks like Pytorch + TPU is still much worse combination than Tensorflow.", "972188": "My experiences:\n1. hair augmentation didn't imporve my LB;\n2. 256x256 size image improve almost 1% LB score over 224x224 size image;\n3. set different probability of hori and vertial flip for pos and neg samples (such as 0.8 and 0.2) improve 2% oof score, which significantly narrow the gap between LB and oof prediction, and decrease the deviation in CV. But it did't change my LB score.", "970175": "I stopped working on this competition at all few days ago, but I am still here to see the huge shakeup and surprised faces :)", "968854": "I want to try predicting diagnosis too together with label. I read in some discussions that it certainly is giving better results. If anyone tries this out, do share your results! :)", "967422": "i try hair augmentation，but i dont know why it become lower", "966242": "And computers are never fast enough... ", "965844": "@cpmpml \nvery true, many things to try yet but no enough time :( \nanyway, I was thinking, it would be nice if we can see who downvote because in that case, the downvoter will have the reason why he/she do this. 😄", "965711": "for me it's \"join late and only got a single 1080ti\"... \n\ni have a list similar to yours, and I just added \"testing weight decay while ignoring BN layers\" to it. \n\n", "968596": "Hair Augmentation doesn't give a major increase in the lb score we had tried it out although the cv score increases.", "967522": "let me answer some that I tested :\nImage size increases the score for sure , tricky part is too fit within the 3-hour TPU limit\nhaven't tried hair augment yet\nupsampling is kinda insignificant\neffnet b6 is giving me best results, got better than b7\nmean average got me good results\nthe tabular ensemble with image prediction gives a significant boost to public LB\nThe public notebook is screwing the LB, to be honest, but helped me too in the single model case.", "967207": "@cpmpml I also tried upsampling (per batch upsampling) but it did not work. I changed this to per epoch downsampling where I trained on 65% random selection of the negative samples + all positive per epoch and it worked well.\nI also feel that the problem in scores is not about TF or torch, I think it is about the datasets. I am using Tensorflow with the JPEG dataset and I got best single model score 0.939 with basic TTA.\nall public kernels scoring 0.95+ are using the tfrecord datasets.", "965743": "I would advice you to try TF instead of pytorch. I did not succeed to have as good as results in pytorch for this competition :/ model with 384 and 512 give very good results. I can have 0.95X with them on public LB", "971447": "The things I want to try before it is over:\n- Use bigger models (EF B5, B6 .. ) \n- improve my metadata classifier\n- try different losses (Focal Loss, Binary Focal Loss, ...)\n- use TTA\n\n2 Days to go ;)", "969551": "I don't get it, some top kagglers are in 1000+ range!!!! Are all those core PyTorch users or the LB is really really really overfitted?? ", "966567": "Yeah, indeed. I have lots of things wanna try out, but the Google Colab Pro seems not cooperating. It keeps disconnecting if I set the \"num_workers\" a high value. Basically I gave up on this competition a week ago, since I need to focus on my thesis for now. Good luck to all of you!", "967521": "" } }