![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F75976%2F059179f975ed612d17b23ec901e5b2b6%2FScreenshot_2020-08-19%20Triple%20Stratified%20KFold%20with%20TFRecords.png?generation=1597862437376532&alt=media\")
Many were puzzled that TF models with a rather low CV had such great public LB. We now know that on private LB Pytorch models were competitive.
\nSo, what's the trick here? It seems that Chris Deotte baseline was overfitting public LB, see for instance the scores of his most popular notebook: https://www.kaggle.com/cdeotte/triple-stratified-kfold-with-tfrecords
\n
The relatively low CV was not the anomaly. The score was the anomaly.
", "messages": [ { "id": 977848, "postDate": "2020-08-19T18:42:40.223Z", "content": "Many were puzzled that TF models with a rather low CV had such great public LB. We now know that on private LB Pytorch models were competitive.
\nSo, what's the trick here? It seems that Chris Deotte baseline was overfitting public LB, see for instance the scores of his most popular notebook: https://www.kaggle.com/cdeotte/triple-stratified-kfold-with-tfrecords
\n
The relatively low CV was not the anomaly. The score was the anomaly.
", "rawMarkdown": "Many were puzzled that TF models with a rather low CV had such great public LB. We now know that on private LB Pytorch models were competitive.\n\nSo, what's the trick here? It seems that Chris Deotte baseline was overfitting public LB, see for instance the scores of his most popular notebook: https://www.kaggle.com/cdeotte/triple-stratified-kfold-with-tfrecords\n\n\n\nThe relatively low CV was not the anomaly. The score was the anomaly.\n", "votes": 28 }, { "id": 977872, "postDate": "2020-08-19T19:06:09.340Z", "content": "I'm not sure if that solves the mystery. Here are the two highest scoring PyTorch notebooks. Both TF and PyTorch notebooks have Private LB score that is 0.02 lower than Public LB score.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1723677%2Faf2f8629b5ce574f5f02af2e1d082a71%2Fc1.png?generation=1597863858079347&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1723677%2F3f566653c07a21e85da1ebbf07bc71e7%2Fc2.png?generation=1597863868755546&alt=media\")
Fai enough,. My point is that these notebooks overfit. Yours is also better overall. People trying to replicate your notebook without overfitting could only conclude that pytorch had an issue compared to TF.
\nFor example, my CV - public gap is almost 0 across all my subs, and my cv - private LB is stable at 0.01. If I wanted to replicate your results then I needed to have a CV 0.02 higher than yours.
\nThat's why I say mystery is solved. To be clear: the mystery is why you got such a high LB compared to the CV of your notebook.
", "rawMarkdown": "Fai enough,. My point is that these notebooks overfit. Yours is also better overall. People trying to replicate your notebook without overfitting could only conclude that pytorch had an issue compared to TF.\n\nFor example, my CV - public gap is almost 0 across all my subs, and my cv - private LB is stable at 0.01. If I wanted to replicate your results then I needed to have a CV 0.02 higher than yours. \n\nThat's why I say mystery is solved. To be clear: the mystery is why you got such a high LB compared to the CV of your notebook." }, { "id": 978294, "postDate": "2020-08-20T04:59:58.353Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 977937, "postDate": "2020-08-19T20:01:23.873Z", "content": "I shared my thoughts here on why I think PyTorch performs better than TF. I'll paraphase here:
\nIn short, I believe PyTorch has better ecosystem/support for augmentations. The reason for a huge gap between Public LB and CV on Chris' notebook was likely due to luck. When I added heavy augmentations to Chris' triple stratified notebook, I increased my CV from 0.91 to 0.9379. The public LB and private LB were 0.9394 and 0.9325 respectively. The public LB and CV gap was a pretty big indicator that something wasn't adding up.
\nI don't think there's a comparable library for TF like Albumenations for PyTorch. I refer to Albumentations because that's what first place used. Second place used RandAugment. But the idea is still the same - better support for augmentation library.
", "rawMarkdown": "I shared my thoughts [here](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/175461) on why I think PyTorch performs better than TF. I'll paraphase here: \n\nIn short, I believe PyTorch has better ecosystem/support for augmentations. The reason for a huge gap between Public LB and CV on Chris' notebook was likely due to luck. When I added [heavy augmentations](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/175372) to Chris' triple stratified notebook, I increased my CV from 0.91 to 0.9379. The public LB and private LB were 0.9394 and 0.9325 respectively. The public LB and CV gap was a pretty big indicator that something wasn't adding up.\n\nI don't think there's a comparable library for TF like [Albumenations](https://www.kaggle.com/c/tgs-salt-identification-challenge/discussion/66643) for PyTorch. I refer to Albumentations because that's what first place used. Second place used RandAugment. But the idea is still the same - better support for augmentation library.", "votes": 15, "replies": [ { "id": 979603, "postDate": "2020-08-21T01:45:23.080Z", "content": "@teeyee314 Would you say that heavy augmentations was the key to improve the CV?
", "rawMarkdown": "@teeyee314 Would you say that heavy augmentations was the key to improve the CV?", "votes": 1 }, { "id": 979658, "postDate": "2020-08-21T03:26:10.663Z", "content": "for this competition, single model - yes. in general, better augmentations should lead to better cv score. when you read people's write ups, you will notice mention to specific augmentations used.
", "rawMarkdown": "for this competition, single model - yes. in general, better augmentations should lead to better cv score. when you read people's write ups, you will notice mention to specific augmentations used.", "votes": 2 }, { "id": 980692, "postDate": "2020-08-21T19:23:48.450Z", "content": "Tim, it's my first competition and I'm still discovering things. Albumentations looks fantastic, and on my simple benchmark, it was about 3x faster. Their documentation says that you can use it with TF too, more here - https://albumentations.ai/docs/api_reference/augmentations/transforms/
\nIs there a reason why you think this library only works with Pytorch?
", "rawMarkdown": "Tim, it's my first competition and I'm still discovering things. Albumentations looks fantastic, and on my simple benchmark, it was about 3x faster. Their documentation says that you can use it with TF too, more here - https://albumentations.ai/docs/api_reference/augmentations/transforms/\n\nIs there a reason why you think this library only works with Pytorch?" }, { "id": 981554, "postDate": "2020-08-22T14:33:47.280Z", "content": "@sudhanshuraheja Regarding external library usage with TPU - see this discussion. Maybe this will change in the future.
", "rawMarkdown": "@sudhanshuraheja Regarding external library usage with TPU - see this [discussion](https://www.kaggle.com/c/flower-classification-with-tpus/discussion/132066). Maybe this will change in the future. ", "votes": 1 }, { "id": 981568, "postDate": "2020-08-22T14:51:27.040Z", "content": "Numpy/opencv preprocessing functions, works with the tf.data.dataset as well, but only on GPU so far, there is still a bug for TPU, see: https://github.com/tensorflow/tensorflow/issues/30818
\n def applyAug(x):\n x = x.numpy()\n return aug(image=x)\n\n def _preprocess_for_train(filename, label):\n image_bytes = tf.io.read_file(filename)\n image = tf.cond(\n tf.image.is_jpeg(image_bytes),\n lambda: tf.image.decode_jpeg(image_bytes, channels=3),\n lambda: tf.image.decode_png(image_bytes, channels=3))\n\n\n image = tf.py_function(func=applyAug, inp=[image], Tout=tf.float32)\n image.set_shape(tf.TensorShape([None, None, None])) \n\n image = tf.image.resize(image, im_shape)\n image = tf.cast(image, dtype=tf.float32) / 255.0\n label = tf.one_hot(label, num_classes)\n label = tf.cast(label, tf.float32)\n return image, label\n\n dataset = tf.data.Dataset.from_tensor_slices((filenames, labels))\n autotune = tf.data.experimental.AUTOTUNE\n dataset = dataset.map(_preprocess_for_train, num_parallel_calls=autotune)\n dataset = dataset.batch(batch_size, drop_remainder=True) \nKeep in mind, using this with GPU will slow down the training a little bit, I can only assume even if they fix it for TPU, it will cause horrible throttling for TPU.
", "rawMarkdown": "Numpy/opencv preprocessing functions, works with the tf.data.dataset as well, but only on GPU so far, there is still a bug for TPU, see: https://github.com/tensorflow/tensorflow/issues/30818\n\n```\n def applyAug(x):\n x = x.numpy()\n return aug(image=x)\n\n def _preprocess_for_train(filename, label):\n image_bytes = tf.io.read_file(filename)\n image = tf.cond(\n tf.image.is_jpeg(image_bytes),\n lambda: tf.image.decode_jpeg(image_bytes, channels=3),\n lambda: tf.image.decode_png(image_bytes, channels=3))\n \n \n image = tf.py_function(func=applyAug, inp=[image], Tout=tf.float32)\n image.set_shape(tf.TensorShape([None, None, None])) \n\n image = tf.image.resize(image, im_shape)\n image = tf.cast(image, dtype=tf.float32) / 255.0\n label = tf.one_hot(label, num_classes)\n label = tf.cast(label, tf.float32)\n return image, label\n\n dataset = tf.data.Dataset.from_tensor_slices((filenames, labels))\n autotune = tf.data.experimental.AUTOTUNE\n dataset = dataset.map(_preprocess_for_train, num_parallel_calls=autotune)\n dataset = dataset.batch(batch_size, drop_remainder=True) \n```\n\nKeep in mind, using this with GPU will slow down the training a little bit, I can only assume even if they fix it for TPU, it will cause horrible throttling for TPU.", "votes": 2 } ] }, { "id": 978477, "postDate": "2020-08-20T07:47:54Z", "content": "There is no difference between the accuracy of the PyTorch vs TensorFlow. A few months ago I had to switch to TensorFlow and replicate my PyTorch results with it. At first, I was baffled how can TF be worse, then I figured out, the difference was my lack of skills to implement the same training process. After dialling it in, it was the exact same flow, and I got almost the same results (+- 0.0001-0.0002 F1) as in PyTorch (non-kaggle dataset).
\nEverybody is saying that PyTorch is better for research, I believe this is only true if you don't know the framework inside and out. After 2 months with TF, I feel the ceiling is so much higher. The transition was very hard especially because I had to learn both tf 1.15 and tf 2.2> at the same time, but it was worth it, I'm not looking back.
", "rawMarkdown": "There is no difference between the accuracy of the PyTorch vs TensorFlow. A few months ago I had to switch to TensorFlow and replicate my PyTorch results with it. At first, I was baffled how can TF be worse, then I figured out, the difference was my lack of skills to implement the same training process. After dialling it in, it was the exact same flow, and I got almost the same results (+- 0.0001-0.0002 F1) as in PyTorch (non-kaggle dataset).\n\nEverybody is saying that PyTorch is better for research, I believe this is only true if you don't know the framework inside and out. After 2 months with TF, I feel the ceiling is so much higher. The transition was very hard especially because I had to learn both tf 1.15 and tf 2.2> at the same time, but it was worth it, I'm not looking back.", "votes": 7, "replies": [ { "id": 978585, "postDate": "2020-08-20T09:14:37.337Z", "content": "Great reply. Just to add something, I would include the following article here about Pytorch vs Tensorflow in 2020
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1984321%2F6afbcc31ee52007e844bdc6d4720f416%2F1.png?generation=1597914771146704&alt=media\")
Except they are not \"exactly the same\" at all.
", "rawMarkdown": "Except they are not \"exactly the same\" at all." }, { "id": 981448, "postDate": "2020-08-22T12:56:22.283Z", "content": "Nope, they're now. 😀
", "rawMarkdown": "Nope, they're now. 😀" }, { "id": 981466, "postDate": "2020-08-22T13:15:32.130Z", "content": "Give Pytorch a try and we'll discuss again. On paper TF2x looks close to Pytorch indeed, but there is a difference between the look and the reality.
", "rawMarkdown": "Give Pytorch a try and we'll discuss again. On paper TF2x looks close to Pytorch indeed, but there is a difference between the look and the reality. " }, { "id": 981520, "postDate": "2020-08-22T14:00:22.670Z", "content": "Obviously the coding workflow is not the same, not talking about that. I'm talking about the results, the math behind the adam optimizer etc.
\nKeep in mind, most of the TF notebooks used TPU s with much higher batch sizes, compared to the pytorch ones with GPUs. Batch size does affect the accuracy, but as I said if you compare the exact training flow, you should have very-very close results.
Suppose if there is a difference in result between 2 same training settings, which factors cause it?
", "rawMarkdown": "Suppose if there is a difference in result between 2 same training settings, which factors cause it? " }, { "id": 981624, "postDate": "2020-08-22T15:37:08.710Z", "content": "The reality is, it takes much less time and headache to code advanced workflow on Pytorch than tensorflow. Otherwise TF 2.x wouldn't try to be as close as possible to Pytorch.
\nOpenAI (now ClosedAI) has switched form TF to Pytorch for this reason.
\nI'm saying this while I don't like Facebook at all. And Google has contributed much more on DL than them.
\nP.S. : I was much more familiar with TF (both 1.X and 2.X ) than Pytorch 9 months ago.
", "rawMarkdown": "The reality is, it takes much less time and headache to code advanced workflow on Pytorch than tensorflow. Otherwise TF 2.x wouldn't try to be as close as possible to Pytorch. \n\n OpenAI (now ClosedAI) has switched form TF to Pytorch for this reason. \n\nI'm saying this while I don't like Facebook at all. And Google has contributed much more on DL than them. \n\nP.S. : I was much more familiar with TF (both 1.X and 2.X ) than Pytorch 9 months ago. " }, { "id": 981647, "postDate": "2020-08-22T15:50:51.707Z", "content": "@cpmpml san, I don't find any offensive words in your response, don't get it why it's been downvoted!
\nHowever, I should say, we don't have exact same comprehensive working experience in both TF and PyTorch. Compare to TF, we've very little experience in PyTorch (until now). PyTorch is awesome, no doubt and the coding experience in TF was much pain. My point only was, the coding experience between these two is almost similar now (AFAIK). @doncalculator san points some great stuff. But I would request you to explain these two more in detail (from an experimental perspective, with the latest TF).
", "rawMarkdown": "@cpmpml san, I don't find any offensive words in your response, don't get it why it's been downvoted! \n\nHowever, I should say, we don't have exact same comprehensive working experience in both TF and PyTorch. Compare to TF, we've very little experience in PyTorch (until now). PyTorch is awesome, no doubt and the coding experience in TF was much pain. My point only was, the coding experience between these two is almost similar now (AFAIK). @doncalculator san points some great stuff. But I would request you to explain these two more in detail (from an experimental perspective, with the latest TF). ", "votes": 1 }, { "id": 982077, "postDate": "2020-08-23T03:41:07.603Z", "content": "Is there a way to do something like Funcional API of Keras with PyTorch?
", "rawMarkdown": "Is there a way to do something like Funcional API of Keras with PyTorch? " }, { "id": 982211, "postDate": "2020-08-23T06:46:12.170Z", "content": "@ipythonx Alright, I created identical training flow for both tensorflow and pytorch, I will post the notebooks today I think. Tbh, I'm not sure about the different Efficientnet implementations qubvel's vs lukemelas, but that's gonna be the only difference. Will see.
", "rawMarkdown": "@ipythonx Alright, I created identical training flow for both tensorflow and pytorch, I will post the notebooks today I think. Tbh, I'm not sure about the different Efficientnet implementations qubvel's vs lukemelas, but that's gonna be the only difference. Will see.", "votes": 1 }, { "id": 982404, "postDate": "2020-08-23T10:50:04.227Z", "content": "Awesome. 👍
", "rawMarkdown": "Awesome. 👍" }, { "id": 982665, "postDate": "2020-08-23T15:07:10.340Z", "content": "I published the two notebooks:
\nhttps://www.kaggle.com/doncalculator/tensorflow-vs-pytorch-part-1-tensorflow
\nhttps://www.kaggle.com/doncalculator/tensorflow-vs-pytorch-part-2-pytorch
Here are the results:
\n-----------CV,--------------------PRIVATE, PUBLIC:
\nPytorch: 0.9820760704110646, 0.9077, 0.9196
\nTensorflow: 0.982470040003284 0.9043 0.9203
Because of the low amount of positive images, I chose to include the external
\nmaligns into the validation set to have a more reliable CV score.
Although these scores are close enough to support my statement, due to the low amount of
\nmalignant images, there is a high variance between runs, so if we want to prove that
\nthe results are there same across the two frameworks, this experiment should be rerun 5-10 times.
\n\nThere is no difference between the accuracy of the PyTorch vs TensorFlow
\n
I agree with that statement in general and I assumed it was true here too. It is why there was a mystery: no one could reproduce Chris high LB score when using Pytorch. Now that we know that the high public Lb score was an overfit.
\n\n\n@cpmpml san, I don't find any offensive words in your response, don't get it why it's been downvoted!
\n
I didn't even notice, LOL. Please meet my serial downvoter ;)
", "rawMarkdown": "> There is no difference between the accuracy of the PyTorch vs TensorFlow\n\nI agree with that statement in general and I assumed it was true here too. It is why there was a mystery: no one could reproduce Chris high LB score when using Pytorch. Now that we know that the high public Lb score was an overfit.\n\n> @cpmpml san, I don't find any offensive words in your response, don't get it why it's been downvoted! \n\nI didn't even notice, LOL. Please meet my serial downvoter ;)", "votes": 1 } ] }, { "id": 977962, "postDate": "2020-08-19T20:20:10Z", "content": "The sub with the high score was definitely lucky. I reran it two times, once it was 0.01 worse than the LB score of the NB, second time I removed the 2018 data and it was higher than my first run, but slightly lower than the LB run.
\nI think early stopping makes it super random, it was picking different epochs based on choosing best val loss, which I personally dont believe to be a good idea here if you dont sharpen your learning routine.
\nI still thought for the longest time there is something in that NB that I need to figure out, and it cost me way too many brain cells because I have such a hard time understanding all the tensorflow augmentation code 😁
\nI just checked the results:
\nThis is interesting, I also think that early stopping is not the best way to do experiments, so your suggestions would be to run a few times and see at which epoch the model converges (let's say epoch 15), then write a schedule to end in 15 epochs to compare models even while changing parameters?
", "rawMarkdown": "This is interesting, I also think that early stopping is not the best way to do experiments, so your suggestions would be to run a few times and see at which epoch the model converges (let's say epoch 15), then write a schedule to end in 15 epochs to compare models even while changing parameters?", "votes": 2 }, { "id": 977977, "postDate": "2020-08-19T20:36:18.703Z", "content": "Yes exactly, if your best epoch is let's say between 13-15 out of 15 epochs it can be fine, but in this kernel it was sometimes 3 or 4 out of 10 and in some other folds 10, which is definitely way more random.
\nThe only way imho is to take fixed epochs and multiple bags. We always fitted for 3 bags 5-fold cv with fixed epochs. Correlation was OK, I have seen worse, albeit not perfect.
", "rawMarkdown": "Yes exactly, if your best epoch is let's say between 13-15 out of 15 epochs it can be fine, but in this kernel it was sometimes 3 or 4 out of 10 and in some other folds 10, which is definitely way more random.\n\nThe only way imho is to take fixed epochs and multiple bags. We always fitted for 3 bags 5-fold cv with fixed epochs. Correlation was OK, I have seen worse, albeit not perfect.", "votes": 1 }, { "id": 977978, "postDate": "2020-08-19T20:38:20.123Z", "content": "\n\nI think early stopping makes it super random, it was picking different epochs based on choosing best val loss, which I personally dont believe to be a good idea here if you dont sharpen your learning routine.
\n
I think multiple checkpoints averaging is the only way to stabilize CV. I did it in all my experiments and stopped worrying about public LB. Given it was not really always correlated with my CV
", "rawMarkdown": "> I think early stopping makes it super random, it was picking different epochs based on choosing best val loss, which I personally dont believe to be a good idea here if you dont sharpen your learning routine.\n\nI think multiple checkpoints averaging is the only way to stabilize CV. I did it in all my experiments and stopped worrying about public LB. Given it was not really always correlated with my CV", "votes": 1 }, { "id": 977980, "postDate": "2020-08-19T20:40:21.813Z", "content": "Agreed, I can definitely see the issue, thanks!
", "rawMarkdown": "Agreed, I can definitely see the issue, thanks!" }, { "id": 978000, "postDate": "2020-08-19T20:59:06.647Z", "content": "@serigne There is a difference between choosing the same epochs for checkpoint ensembling (can make sense) vs. picking random epochs (makes it random).
", "rawMarkdown": "@serigne There is a difference between choosing the same epochs for checkpoint ensembling (can make sense) vs. picking random epochs (makes it random)." }, { "id": 978421, "postDate": "2020-08-20T07:00:26.460Z", "content": "What I did was to run relatively large number of epochs and chose 3 or 4 to average based on multiple criteria. For instance for some folds I could chose Epochs : 12, 25, 30 while on others: Epochs 31, 34, 40 etc.
", "rawMarkdown": "What I did was to run relatively large number of epochs and chose 3 or 4 to average based on multiple criteria. For instance for some folds I could chose Epochs : 12, 25, 30 while on others: Epochs 31, 34, 40 etc.\n\n" } ] }, { "id": 980076, "postDate": "2020-08-21T10:16:20.193Z", "content": "This comp has a very high randomness in score. Unusually high for an NN competition.
", "rawMarkdown": "This comp has a very high randomness in score. Unusually high for an NN competition. ", "votes": 3, "replies": [ { "id": 980095, "postDate": "2020-08-21T10:25:00.800Z", "content": "Indeed, probably because of the very low number of positive samples. The metric is very sensitive to one of them being misclassified.
", "rawMarkdown": "Indeed, probably because of the very low number of positive samples. The metric is very sensitive to one of them being misclassified.", "votes": 1 } ] }, { "id": 979737, "postDate": "2020-08-21T05:11:46.040Z", "content": "My submitted custom EfficientNet-B6 384x384 noisy student (based on Chris popular kernel) + effective Focal Loss parameter computation has CV: 0.942, Public LB: 0.9446, and Private LB: 0.943 though it was hard to recreate the result may be because of reproducibility concern in TPU TensorFlow. The gap is between 0.004 - 0.007
", "rawMarkdown": "My submitted custom EfficientNet-B6 384x384 noisy student (based on Chris popular kernel) + effective Focal Loss parameter computation has CV: 0.942, Public LB: 0.9446, and Private LB: 0.943 though it was hard to recreate the result may be because of reproducibility concern in TPU TensorFlow. The gap is between 0.004 - 0.007", "votes": 3, "replies": [ { "id": 979768, "postDate": "2020-08-21T05:35:54.083Z", "content": "So, looks like the difference comes from using effective loss function! Is it?
", "rawMarkdown": "So, looks like the difference comes from using effective loss function! Is it? " }, { "id": 979798, "postDate": "2020-08-21T05:58:22.203Z", "content": "My model architecture consists of the following: 1. Use base model output as input to convolutional block attention modules (CBAM), 2. Create 2 outputs and average it: 2a. use CBAM output as input to Global Average Pooling (GAP) and create a final dense layer with sigmoid activation, 2b. use CBAM output as input to Attention Weighted Average Pooling and create a final dense layer with sigmoid activation. Some of my models have large gaps in CV and LB (0.008, 0.012) given I used that said loss function. Maybe because of the random generated processed output of attention modules.
", "rawMarkdown": "My model architecture consists of the following: 1. Use base model output as input to convolutional block attention modules (CBAM), 2. Create 2 outputs and average it: 2a. use CBAM output as input to Global Average Pooling (GAP) and create a final dense layer with sigmoid activation, 2b. use CBAM output as input to Attention Weighted Average Pooling and create a final dense layer with sigmoid activation. Some of my models have large gaps in CV and LB (0.008, 0.012) given I used that said loss function. Maybe because of the random generated processed output of attention modules." }, { "id": 979826, "postDate": "2020-08-21T06:22:14.107Z", "content": "Interesting. I've posted my solution here. May I ask to know is the Attention Weighted Avg Pooling that you've used are the same as mine.
", "rawMarkdown": "Interesting. I've posted my solution [here](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/175721). May I ask to know is the Attention Weighted Avg Pooling that you've used are the same as mine. " }, { "id": 979827, "postDate": "2020-08-21T06:22:19.397Z", "content": "Loss Function:
\npos_weight = ((1 / total number of positives per fold)*total number of samples per fold) / 2\nneg_weight = ((1 / total number of negatives per fold)*total number of samples per fold) / 2\n\nBinary Focal Loss where pos_weight=pos_weight and gamma=pos_weight*neg_weight\nhttps://github.com/artemmavrin/focal-loss
", "rawMarkdown": "Loss Function: \n\n\tpos_weight = ((1 / total number of positives per fold)*total number of samples per fold) / 2\n\tneg_weight = ((1 / total number of negatives per fold)*total number of samples per fold) / 2\n\n\tBinary Focal Loss where pos_weight=pos_weight and gamma=pos_weight*neg_weight\n\nhttps://github.com/artemmavrin/focal-loss" }, { "id": 979828, "postDate": "2020-08-21T06:24:15.127Z", "content": "I refer to this link for Attention Weighted Average: https://github.com/bfelbo/DeepMoji/blob/master/deepmoji/attlayer.py
", "rawMarkdown": "I refer to this link for Attention Weighted Average: https://github.com/bfelbo/DeepMoji/blob/master/deepmoji/attlayer.py", "votes": 1 }, { "id": 979856, "postDate": "2020-08-21T06:51:17.617Z", "content": "oh, I see. I've used the Guanshuo Xu modified version of it. As far as I can remember he also used the same repo. -)
", "rawMarkdown": "oh, I see. I've used the Guanshuo Xu modified version of it. As far as I can remember he also used the same repo. -)" } ] }, { "id": 977861, "postDate": "2020-08-19T18:51:54.510Z", "content": "Chris Deotte needs to share notebooks with a Disclaimer 😝
", "rawMarkdown": "Chris Deotte needs to share notebooks with a Disclaimer 😝", "votes": 3 }, { "id": 978400, "postDate": "2020-08-20T06:41:21.240Z", "content": "@cpmpml I would like to learn from a Grandmaster such as yourself about creating a stable and LB correlated CV.
\nFor example, my CV - public gap is almost 0 across all my subs, and my cv - private LB is stable at 0.01.
Even Chris's Triple Stratified Leak-Free KFold CV still has a large gap, but your CV has gap of less than 0.01, so could you please share your techniques on this accomplishment? Do you have different techniques for different types of competitions?
", "rawMarkdown": "@cpmpml I would like to learn from a Grandmaster such as yourself about creating a stable and LB correlated CV.\n\n`For example, my CV - public gap is almost 0 across all my subs, and my cv - private LB is stable at 0.01. `\n\nEven Chris's Triple Stratified Leak-Free KFold CV still has a large gap, but **your CV has gap of less than 0.01**, so could you please share your techniques on this accomplishment? Do you have different techniques for different types of competitions?", "votes": 4, "replies": [ { "id": 978521, "postDate": "2020-08-20T08:25:02.207Z", "content": "Thanks for asking.
\nYou can read my previous writeups as I will not create one here. Here is some info still.
\nI always focus on getting a reliable CV. I used the same approach here as in Tweet Sentiment: bagging several runs to average out CV variance.
\nI was surprised to get a small gap. Having a large CV LB gap isn't an issue in itself if CV and LB are correlated. Here, my CV-LB difference was 0 += 0.002.
\nWhen I manage to get a reliable CV in a competition, which is not always the case, then I keep a change only if it improves both CV and public LB.
\nIn this competition I had few outliers when it comes to CV LB gap. A first one was when I added 2019 positive examples only. CV improved and LB sunk. Second outlier was last day when ensembling. I overfit to public LB. I wish I had more time to fix it.
\nTL;DR I use the same overall approach across competitions: reduce CV variance with bagging if need be. It means runs take longer as they need to be repeated, but progress is steady.
", "rawMarkdown": "Thanks for asking.\n\nYou can read my previous writeups as I will not create one here. Here is some info still.\n\nI always focus on getting a reliable CV. I used the same approach here as in Tweet Sentiment: bagging several runs to average out CV variance. \n\nI was surprised to get a small gap. Having a large CV LB gap isn't an issue in itself if CV and LB are correlated. Here, my CV-LB difference was 0 += 0.002.\n\nWhen I manage to get a reliable CV in a competition, which is not always the case, then I keep a change only if it improves both CV and public LB. \n\nIn this competition I had few outliers when it comes to CV LB gap. A first one was when I added 2019 positive examples only. CV improved and LB sunk. Second outlier was last day when ensembling. I overfit to public LB. I wish I had more time to fix it.\n\nTL;DR I use the same overall approach across competitions: reduce CV variance with bagging if need be. It means runs take longer as they need to be repeated, but progress is steady.\n\n", "votes": 5 }, { "id": 984826, "postDate": "2020-08-25T10:15:40.167Z", "content": "@sirishks Is the above what you asked for?
", "rawMarkdown": "@sirishks Is the above what you asked for?" } ] } ], "comments": [ { "id": 977872, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-19T19:06:09.340000", "content": "I'm not sure if that solves the mystery. Here are the two highest scoring PyTorch notebooks. Both TF and PyTorch notebooks have Private LB score that is 0.02 lower than Public LB score.
\n
Fai enough,. My point is that these notebooks overfit. Yours is also better overall. People trying to replicate your notebook without overfitting could only conclude that pytorch had an issue compared to TF.
\nFor example, my CV - public gap is almost 0 across all my subs, and my cv - private LB is stable at 0.01. If I wanted to replicate your results then I needed to have a CV 0.02 higher than yours.
\nThat's why I say mystery is solved. To be clear: the mystery is why you got such a high LB compared to the CV of your notebook.
", "votes": 0, "replies": [] }, { "id": 978294, "author_name": "", "author_url": "", "post_date": "2020-08-20T04:59:58.353000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 977937, "author_name": "Tim Yee", "author_url": "", "post_date": "2020-08-19T20:01:23.873000", "content": "I shared my thoughts here on why I think PyTorch performs better than TF. I'll paraphase here:
\nIn short, I believe PyTorch has better ecosystem/support for augmentations. The reason for a huge gap between Public LB and CV on Chris' notebook was likely due to luck. When I added heavy augmentations to Chris' triple stratified notebook, I increased my CV from 0.91 to 0.9379. The public LB and private LB were 0.9394 and 0.9325 respectively. The public LB and CV gap was a pretty big indicator that something wasn't adding up.
\nI don't think there's a comparable library for TF like Albumenations for PyTorch. I refer to Albumentations because that's what first place used. Second place used RandAugment. But the idea is still the same - better support for augmentation library.
", "votes": 15, "replies": [ { "id": 979603, "author_name": "Hiram Coria 🧬", "author_url": "", "post_date": "2020-08-21T01:45:23.080000", "content": "@teeyee314 Would you say that heavy augmentations was the key to improve the CV?
", "votes": 1, "replies": [] }, { "id": 979658, "author_name": "Tim Yee", "author_url": "", "post_date": "2020-08-21T03:26:10.663000", "content": "for this competition, single model - yes. in general, better augmentations should lead to better cv score. when you read people's write ups, you will notice mention to specific augmentations used.
", "votes": 2, "replies": [] }, { "id": 980692, "author_name": "Sudhanshu Raheja", "author_url": "", "post_date": "2020-08-21T19:23:48.450000", "content": "Tim, it's my first competition and I'm still discovering things. Albumentations looks fantastic, and on my simple benchmark, it was about 3x faster. Their documentation says that you can use it with TF too, more here - https://albumentations.ai/docs/api_reference/augmentations/transforms/
\nIs there a reason why you think this library only works with Pytorch?
", "votes": 0, "replies": [] }, { "id": 981554, "author_name": "Tim Yee", "author_url": "", "post_date": "2020-08-22T14:33:47.280000", "content": "@sudhanshuraheja Regarding external library usage with TPU - see this discussion. Maybe this will change in the future.
", "votes": 1, "replies": [] }, { "id": 981568, "author_name": "Krisztián Fekete", "author_url": "", "post_date": "2020-08-22T14:51:27.040000", "content": "Numpy/opencv preprocessing functions, works with the tf.data.dataset as well, but only on GPU so far, there is still a bug for TPU, see: https://github.com/tensorflow/tensorflow/issues/30818
\n def applyAug(x):\n x = x.numpy()\n return aug(image=x)\n\n def _preprocess_for_train(filename, label):\n image_bytes = tf.io.read_file(filename)\n image = tf.cond(\n tf.image.is_jpeg(image_bytes),\n lambda: tf.image.decode_jpeg(image_bytes, channels=3),\n lambda: tf.image.decode_png(image_bytes, channels=3))\n\n\n image = tf.py_function(func=applyAug, inp=[image], Tout=tf.float32)\n image.set_shape(tf.TensorShape([None, None, None])) \n\n image = tf.image.resize(image, im_shape)\n image = tf.cast(image, dtype=tf.float32) / 255.0\n label = tf.one_hot(label, num_classes)\n label = tf.cast(label, tf.float32)\n return image, label\n\n dataset = tf.data.Dataset.from_tensor_slices((filenames, labels))\n autotune = tf.data.experimental.AUTOTUNE\n dataset = dataset.map(_preprocess_for_train, num_parallel_calls=autotune)\n dataset = dataset.batch(batch_size, drop_remainder=True) \nKeep in mind, using this with GPU will slow down the training a little bit, I can only assume even if they fix it for TPU, it will cause horrible throttling for TPU.
", "votes": 2, "replies": [] } ] }, { "id": 978477, "author_name": "Krisztián Fekete", "author_url": "", "post_date": "2020-08-20T07:47:54", "content": "There is no difference between the accuracy of the PyTorch vs TensorFlow. A few months ago I had to switch to TensorFlow and replicate my PyTorch results with it. At first, I was baffled how can TF be worse, then I figured out, the difference was my lack of skills to implement the same training process. After dialling it in, it was the exact same flow, and I got almost the same results (+- 0.0001-0.0002 F1) as in PyTorch (non-kaggle dataset).
\nEverybody is saying that PyTorch is better for research, I believe this is only true if you don't know the framework inside and out. After 2 months with TF, I feel the ceiling is so much higher. The transition was very hard especially because I had to learn both tf 1.15 and tf 2.2> at the same time, but it was worth it, I'm not looking back.
", "votes": 7, "replies": [ { "id": 978585, "author_name": "Innat", "author_url": "", "post_date": "2020-08-20T09:14:37.337000", "content": "Great reply. Just to add something, I would include the following article here about Pytorch vs Tensorflow in 2020
\n
Except they are not \"exactly the same\" at all.
", "votes": 0, "replies": [] }, { "id": 981448, "author_name": "Innat", "author_url": "", "post_date": "2020-08-22T12:56:22.283000", "content": "Nope, they're now. 😀
", "votes": 0, "replies": [] }, { "id": 981466, "author_name": "CPMP", "author_url": "", "post_date": "2020-08-22T13:15:32.130000", "content": "Give Pytorch a try and we'll discuss again. On paper TF2x looks close to Pytorch indeed, but there is a difference between the look and the reality.
", "votes": 0, "replies": [] }, { "id": 981520, "author_name": "Krisztián Fekete", "author_url": "", "post_date": "2020-08-22T14:00:22.670000", "content": "Obviously the coding workflow is not the same, not talking about that. I'm talking about the results, the math behind the adam optimizer etc.
\nKeep in mind, most of the TF notebooks used TPU s with much higher batch sizes, compared to the pytorch ones with GPUs. Batch size does affect the accuracy, but as I said if you compare the exact training flow, you should have very-very close results.
Suppose if there is a difference in result between 2 same training settings, which factors cause it?
", "votes": 0, "replies": [] }, { "id": 981624, "author_name": "Serigne ", "author_url": "", "post_date": "2020-08-22T15:37:08.710000", "content": "The reality is, it takes much less time and headache to code advanced workflow on Pytorch than tensorflow. Otherwise TF 2.x wouldn't try to be as close as possible to Pytorch.
\nOpenAI (now ClosedAI) has switched form TF to Pytorch for this reason.
\nI'm saying this while I don't like Facebook at all. And Google has contributed much more on DL than them.
\nP.S. : I was much more familiar with TF (both 1.X and 2.X ) than Pytorch 9 months ago.
", "votes": 0, "replies": [] }, { "id": 981647, "author_name": "Innat", "author_url": "", "post_date": "2020-08-22T15:50:51.707000", "content": "@cpmpml san, I don't find any offensive words in your response, don't get it why it's been downvoted!
\nHowever, I should say, we don't have exact same comprehensive working experience in both TF and PyTorch. Compare to TF, we've very little experience in PyTorch (until now). PyTorch is awesome, no doubt and the coding experience in TF was much pain. My point only was, the coding experience between these two is almost similar now (AFAIK). @doncalculator san points some great stuff. But I would request you to explain these two more in detail (from an experimental perspective, with the latest TF).
", "votes": 1, "replies": [] }, { "id": 982077, "author_name": "Hiram Coria 🧬", "author_url": "", "post_date": "2020-08-23T03:41:07.603000", "content": "Is there a way to do something like Funcional API of Keras with PyTorch?
", "votes": 0, "replies": [] }, { "id": 982211, "author_name": "Krisztián Fekete", "author_url": "", "post_date": "2020-08-23T06:46:12.170000", "content": "@ipythonx Alright, I created identical training flow for both tensorflow and pytorch, I will post the notebooks today I think. Tbh, I'm not sure about the different Efficientnet implementations qubvel's vs lukemelas, but that's gonna be the only difference. Will see.
", "votes": 1, "replies": [] }, { "id": 982404, "author_name": "Innat", "author_url": "", "post_date": "2020-08-23T10:50:04.227000", "content": "Awesome. 👍
", "votes": 0, "replies": [] }, { "id": 982665, "author_name": "Krisztián Fekete", "author_url": "", "post_date": "2020-08-23T15:07:10.340000", "content": "I published the two notebooks:
\nhttps://www.kaggle.com/doncalculator/tensorflow-vs-pytorch-part-1-tensorflow
\nhttps://www.kaggle.com/doncalculator/tensorflow-vs-pytorch-part-2-pytorch
Here are the results:
\n-----------CV,--------------------PRIVATE, PUBLIC:
\nPytorch: 0.9820760704110646, 0.9077, 0.9196
\nTensorflow: 0.982470040003284 0.9043 0.9203
Because of the low amount of positive images, I chose to include the external
\nmaligns into the validation set to have a more reliable CV score.
Although these scores are close enough to support my statement, due to the low amount of
\nmalignant images, there is a high variance between runs, so if we want to prove that
\nthe results are there same across the two frameworks, this experiment should be rerun 5-10 times.
\n\nThere is no difference between the accuracy of the PyTorch vs TensorFlow
\n
I agree with that statement in general and I assumed it was true here too. It is why there was a mystery: no one could reproduce Chris high LB score when using Pytorch. Now that we know that the high public Lb score was an overfit.
\n\n\n@cpmpml san, I don't find any offensive words in your response, don't get it why it's been downvoted!
\n
I didn't even notice, LOL. Please meet my serial downvoter ;)
", "votes": 1, "replies": [] } ] }, { "id": 977962, "author_name": "Psi", "author_url": "", "post_date": "2020-08-19T20:20:10", "content": "The sub with the high score was definitely lucky. I reran it two times, once it was 0.01 worse than the LB score of the NB, second time I removed the 2018 data and it was higher than my first run, but slightly lower than the LB run.
\nI think early stopping makes it super random, it was picking different epochs based on choosing best val loss, which I personally dont believe to be a good idea here if you dont sharpen your learning routine.
\nI still thought for the longest time there is something in that NB that I need to figure out, and it cost me way too many brain cells because I have such a hard time understanding all the tensorflow augmentation code 😁
\nI just checked the results:
\nThis is interesting, I also think that early stopping is not the best way to do experiments, so your suggestions would be to run a few times and see at which epoch the model converges (let's say epoch 15), then write a schedule to end in 15 epochs to compare models even while changing parameters?
", "votes": 2, "replies": [] }, { "id": 977977, "author_name": "Psi", "author_url": "", "post_date": "2020-08-19T20:36:18.703000", "content": "Yes exactly, if your best epoch is let's say between 13-15 out of 15 epochs it can be fine, but in this kernel it was sometimes 3 or 4 out of 10 and in some other folds 10, which is definitely way more random.
\nThe only way imho is to take fixed epochs and multiple bags. We always fitted for 3 bags 5-fold cv with fixed epochs. Correlation was OK, I have seen worse, albeit not perfect.
", "votes": 1, "replies": [] }, { "id": 977978, "author_name": "Serigne ", "author_url": "", "post_date": "2020-08-19T20:38:20.123000", "content": "\n\nI think early stopping makes it super random, it was picking different epochs based on choosing best val loss, which I personally dont believe to be a good idea here if you dont sharpen your learning routine.
\n
I think multiple checkpoints averaging is the only way to stabilize CV. I did it in all my experiments and stopped worrying about public LB. Given it was not really always correlated with my CV
", "votes": 1, "replies": [] }, { "id": 977980, "author_name": "DimitreOliveira", "author_url": "", "post_date": "2020-08-19T20:40:21.813000", "content": "Agreed, I can definitely see the issue, thanks!
", "votes": 0, "replies": [] }, { "id": 978000, "author_name": "Psi", "author_url": "", "post_date": "2020-08-19T20:59:06.647000", "content": "@serigne There is a difference between choosing the same epochs for checkpoint ensembling (can make sense) vs. picking random epochs (makes it random).
", "votes": 0, "replies": [] }, { "id": 978421, "author_name": "Serigne ", "author_url": "", "post_date": "2020-08-20T07:00:26.460000", "content": "What I did was to run relatively large number of epochs and chose 3 or 4 to average based on multiple criteria. For instance for some folds I could chose Epochs : 12, 25, 30 while on others: Epochs 31, 34, 40 etc.
", "votes": 0, "replies": [] } ] }, { "id": 980076, "author_name": "Kha Vo", "author_url": "", "post_date": "2020-08-21T10:16:20.193000", "content": "This comp has a very high randomness in score. Unusually high for an NN competition.
", "votes": 3, "replies": [ { "id": 980095, "author_name": "CPMP", "author_url": "", "post_date": "2020-08-21T10:25:00.800000", "content": "Indeed, probably because of the very low number of positive samples. The metric is very sensitive to one of them being misclassified.
", "votes": 1, "replies": [] } ] }, { "id": 979737, "author_name": "FGPC", "author_url": "", "post_date": "2020-08-21T05:11:46.040000", "content": "My submitted custom EfficientNet-B6 384x384 noisy student (based on Chris popular kernel) + effective Focal Loss parameter computation has CV: 0.942, Public LB: 0.9446, and Private LB: 0.943 though it was hard to recreate the result may be because of reproducibility concern in TPU TensorFlow. The gap is between 0.004 - 0.007
", "votes": 3, "replies": [ { "id": 979768, "author_name": "Innat", "author_url": "", "post_date": "2020-08-21T05:35:54.083000", "content": "So, looks like the difference comes from using effective loss function! Is it?
", "votes": 0, "replies": [] }, { "id": 979798, "author_name": "FGPC", "author_url": "", "post_date": "2020-08-21T05:58:22.203000", "content": "My model architecture consists of the following: 1. Use base model output as input to convolutional block attention modules (CBAM), 2. Create 2 outputs and average it: 2a. use CBAM output as input to Global Average Pooling (GAP) and create a final dense layer with sigmoid activation, 2b. use CBAM output as input to Attention Weighted Average Pooling and create a final dense layer with sigmoid activation. Some of my models have large gaps in CV and LB (0.008, 0.012) given I used that said loss function. Maybe because of the random generated processed output of attention modules.
", "votes": 0, "replies": [] }, { "id": 979826, "author_name": "Innat", "author_url": "", "post_date": "2020-08-21T06:22:14.107000", "content": "Interesting. I've posted my solution here. May I ask to know is the Attention Weighted Avg Pooling that you've used are the same as mine.
", "votes": 0, "replies": [] }, { "id": 979827, "author_name": "FGPC", "author_url": "", "post_date": "2020-08-21T06:22:19.397000", "content": "Loss Function:
\npos_weight = ((1 / total number of positives per fold)*total number of samples per fold) / 2\nneg_weight = ((1 / total number of negatives per fold)*total number of samples per fold) / 2\n\nBinary Focal Loss where pos_weight=pos_weight and gamma=pos_weight*neg_weight\nhttps://github.com/artemmavrin/focal-loss
", "votes": 0, "replies": [] }, { "id": 979828, "author_name": "FGPC", "author_url": "", "post_date": "2020-08-21T06:24:15.127000", "content": "I refer to this link for Attention Weighted Average: https://github.com/bfelbo/DeepMoji/blob/master/deepmoji/attlayer.py
", "votes": 1, "replies": [] }, { "id": 979856, "author_name": "Innat", "author_url": "", "post_date": "2020-08-21T06:51:17.617000", "content": "oh, I see. I've used the Guanshuo Xu modified version of it. As far as I can remember he also used the same repo. -)
", "votes": 0, "replies": [] } ] }, { "id": 977861, "author_name": "Mo Fahad", "author_url": "", "post_date": "2020-08-19T18:51:54.510000", "content": "Chris Deotte needs to share notebooks with a Disclaimer 😝
", "votes": 3, "replies": [] }, { "id": 978400, "author_name": "Sirish Somanchi", "author_url": "", "post_date": "2020-08-20T06:41:21.240000", "content": "@cpmpml I would like to learn from a Grandmaster such as yourself about creating a stable and LB correlated CV.
\nFor example, my CV - public gap is almost 0 across all my subs, and my cv - private LB is stable at 0.01.
Even Chris's Triple Stratified Leak-Free KFold CV still has a large gap, but your CV has gap of less than 0.01, so could you please share your techniques on this accomplishment? Do you have different techniques for different types of competitions?
", "votes": 4, "replies": [ { "id": 978521, "author_name": "CPMP", "author_url": "", "post_date": "2020-08-20T08:25:02.207000", "content": "Thanks for asking.
\nYou can read my previous writeups as I will not create one here. Here is some info still.
\nI always focus on getting a reliable CV. I used the same approach here as in Tweet Sentiment: bagging several runs to average out CV variance.
\nI was surprised to get a small gap. Having a large CV LB gap isn't an issue in itself if CV and LB are correlated. Here, my CV-LB difference was 0 += 0.002.
\nWhen I manage to get a reliable CV in a competition, which is not always the case, then I keep a change only if it improves both CV and public LB.
\nIn this competition I had few outliers when it comes to CV LB gap. A first one was when I added 2019 positive examples only. CV improved and LB sunk. Second outlier was last day when ensembling. I overfit to public LB. I wish I had more time to fix it.
\nTL;DR I use the same overall approach across competitions: reduce CV variance with bagging if need be. It means runs take longer as they need to be repeated, but progress is steady.
", "votes": 5, "replies": [] }, { "id": 984826, "author_name": "CPMP", "author_url": "", "post_date": "2020-08-25T10:15:40.167000", "content": "@sirishks Is the above what you asked for?
", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "977848": "Many were puzzled that TF models with a rather low CV had such great public LB. We now know that on private LB Pytorch models were competitive.\n\nSo, what's the trick here? It seems that Chris Deotte baseline was overfitting public LB, see for instance the scores of his most popular notebook: https://www.kaggle.com/cdeotte/triple-stratified-kfold-with-tfrecords\n\n\n\nThe relatively low CV was not the anomaly. The score was the anomaly.\n", "977872": "I'm not sure if that solves the mystery. Here are the two highest scoring PyTorch notebooks. Both TF and PyTorch notebooks have Private LB score that is 0.02 lower than Public LB score. \n\n\n\n", "977937": "I shared my thoughts [here](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/175461) on why I think PyTorch performs better than TF. I'll paraphase here: \n\nIn short, I believe PyTorch has better ecosystem/support for augmentations. The reason for a huge gap between Public LB and CV on Chris' notebook was likely due to luck. When I added [heavy augmentations](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/175372) to Chris' triple stratified notebook, I increased my CV from 0.91 to 0.9379. The public LB and private LB were 0.9394 and 0.9325 respectively. The public LB and CV gap was a pretty big indicator that something wasn't adding up.\n\nI don't think there's a comparable library for TF like [Albumenations](https://www.kaggle.com/c/tgs-salt-identification-challenge/discussion/66643) for PyTorch. I refer to Albumentations because that's what first place used. Second place used RandAugment. But the idea is still the same - better support for augmentation library.", "978477": "There is no difference between the accuracy of the PyTorch vs TensorFlow. A few months ago I had to switch to TensorFlow and replicate my PyTorch results with it. At first, I was baffled how can TF be worse, then I figured out, the difference was my lack of skills to implement the same training process. After dialling it in, it was the exact same flow, and I got almost the same results (+- 0.0001-0.0002 F1) as in PyTorch (non-kaggle dataset).\n\nEverybody is saying that PyTorch is better for research, I believe this is only true if you don't know the framework inside and out. After 2 months with TF, I feel the ceiling is so much higher. The transition was very hard especially because I had to learn both tf 1.15 and tf 2.2> at the same time, but it was worth it, I'm not looking back.", "977962": "The sub with the high score was definitely lucky. I reran it two times, once it was 0.01 worse than the LB score of the NB, second time I removed the 2018 data and it was higher than my first run, but slightly lower than the LB run.\n\nI think early stopping makes it super random, it was picking different epochs based on choosing best val loss, which I personally dont believe to be a good idea here if you dont sharpen your learning routine.\n\nI still thought for the longest time there is something in that NB that I need to figure out, and it cost me way too many brain cells because I have such a hard time understanding all the tensorflow augmentation code 😁\n\nI just checked the results:\n\n- No extra data on that kernel: 0.9428 public LB, 0.9104 private LB\n- Extra data as provided: 0.9342 public LB, 0.9256 private LB", "980076": "This comp has a very high randomness in score. Unusually high for an NN competition. ", "979737": "My submitted custom EfficientNet-B6 384x384 noisy student (based on Chris popular kernel) + effective Focal Loss parameter computation has CV: 0.942, Public LB: 0.9446, and Private LB: 0.943 though it was hard to recreate the result may be because of reproducibility concern in TPU TensorFlow. The gap is between 0.004 - 0.007", "977861": "Chris Deotte needs to share notebooks with a Disclaimer 😝", "978400": "@cpmpml I would like to learn from a Grandmaster such as yourself about creating a stable and LB correlated CV.\n\n`For example, my CV - public gap is almost 0 across all my subs, and my cv - private LB is stable at 0.01. `\n\nEven Chris's Triple Stratified Leak-Free KFold CV still has a large gap, but **your CV has gap of less than 0.01**, so could you please share your techniques on this accomplishment? Do you have different techniques for different types of competitions?" } }