(In particular, for anyone using TPU for this competition)
\nI have a TensorFlow TPU training / validation pipeline for this competition TPU - Track knowledge states of 1M+ students.
\nAs I mentioned in an thread Does the gap between CV and public LB score need to be very small?, the gap between my CV and LB is large, ranging from 0.007 to 0.01. Therefore I spent some time to look deeper. Nothing wrong being found, but there is an observation.
I actually have 2 validation pipelines:
\none is using validation dataset stored in tensorflow record file, and with this, the validation is done with TPU.
another one is using a pipeline similar to the submission process, so small batches are fed into the model running on GPU.
I observed that the prediction from TPU / GPU have some differences.
\nFor example, with TPU, one prediction is like
\n```\n<tf.Tensor: shape=(128,), dtype=float32, numpy=\narray([0.01927836, 0.01927836, 0.01927836, 0.01927836, 0.01927836,\n ......\n 0.3612144 , 0.9305125 , 0.44552624], dtype=float32)>\n```\nwhile with GPU, it gives
\n<tf.Tensor: shape=(128,), dtype=float32, numpy=\narray([0.0193305 , 0.0193305 , 0.0193305 , 0.0193305 , 0.0193305 ,\n ......\n 0.36048502, 0.93047607, 0.46845183], dtype=float32)>\n```\nFor some validation examples, the gaps between the probabilities from these 2 pipeline can be > 0.01, but the final AUC score doesn't change much.
Anyway, this situation is not the cause of the gap between my CV / LB, however I hope it won't give me a bad surprise for the private LB …
", "messages": [ { "id": "1126674", "postDate": "12/25/2020 19:50:34", "content": "(In particular, for anyone using TPU for this competition)
\nI have a TensorFlow TPU training / validation pipeline for this competition TPU - Track knowledge states of 1M+ students.
\nAs I mentioned in an thread Does the gap between CV and public LB score need to be very small?, the gap between my CV and LB is large, ranging from 0.007 to 0.01. Therefore I spent some time to look deeper. Nothing wrong being found, but there is an observation.
I actually have 2 validation pipelines:
\none is using validation dataset stored in tensorflow record file, and with this, the validation is done with TPU.
another one is using a pipeline similar to the submission process, so small batches are fed into the model running on GPU.
I observed that the prediction from TPU / GPU have some differences.
\nFor example, with TPU, one prediction is like
\n```\n<tf.Tensor: shape=(128,), dtype=float32, numpy=\narray([0.01927836, 0.01927836, 0.01927836, 0.01927836, 0.01927836,\n ......\n 0.3612144 , 0.9305125 , 0.44552624], dtype=float32)>\n```\nwhile with GPU, it gives
\n<tf.Tensor: shape=(128,), dtype=float32, numpy=\narray([0.0193305 , 0.0193305 , 0.0193305 , 0.0193305 , 0.0193305 ,\n ......\n 0.36048502, 0.93047607, 0.46845183], dtype=float32)>\n```\nFor some validation examples, the gaps between the probabilities from these 2 pipeline can be > 0.01, but the final AUC score doesn't change much.
Anyway, this situation is not the cause of the gap between my CV / LB, however I hope it won't give me a bad surprise for the private LB …
", "rawMarkdown": "(In particular, for anyone using TPU for this competition)\n\nI have a TensorFlow TPU training / validation pipeline for this competition [TPU - Track knowledge states of 1M+ students](https://www.kaggle.com/yihdarshieh/tpu-track-knowledge-states-of-1m-students).\n\nAs I mentioned in an thread [Does the gap between CV and public LB score need to be very small?](https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/206488), the gap between my CV and LB is large, ranging from `0.007` to `0.01`. Therefore I spent some time to look deeper. Nothing wrong being found, but there is an observation.\n\nI actually have 2 validation pipelines:\n\n - one is using validation dataset stored in tensorflow record file, and with this, the validation is done with TPU.\n\n - another one is using a pipeline similar to the submission process, so small batches are fed into the model running on GPU.\n\nI observed that the prediction from TPU / GPU have some differences.\n\nFor example, with TPU, one prediction is like\n\n ```\nWell since computations are done using bfloat16 for TPUs, there would be small differences in both model outputs even if the weights of the models are same. (Weights are still saved as float32 in both cases)
\nI think this slight differences would become smaller and smaller as we increase the number of interactions to check the AUC on.
", "rawMarkdown": "Well since computations are done using bfloat16 for TPUs, there would be small differences in both model outputs even if the weights of the models are same. (Weights are still saved as float32 in both cases)\n\nI think this slight differences would become smaller and smaller as we increase the number of interactions to check the AUC on.", "votes": null }, { "id": "1127106", "postDate": "12/26/2020 08:43:42", "content": "Yes, I am aware of bfloat16 for TPUs, but didn't know that the gap (between probabilities) could be higher > 0.005 …
Are you using TPU for submission as well? I'm training on TPU but submitting on GPU since my inference pipeline without model is ~1 hour and built on np arrays so can't shift to TPU now.
\nBtw if you have pipeline setup for TPU inference. Submit using GPU and TPU both on the same trained model and check which one gives higher Public LB.
\nDo let me know the result of this experiment :)
", "rawMarkdown": "Are you using TPU for submission as well? I'm training on TPU but submitting on GPU since my inference pipeline without model is ~1 hour and built on np arrays so can't shift to TPU now.\n\nBtw if you have pipeline setup for TPU inference. Submit using GPU and TPU both on the same trained model and check which one gives higher Public LB. \n\nDo let me know the result of this experiment :)", "votes": null }, { "id": "1127111", "postDate": "12/26/2020 08:52:55", "content": "TPU for fast validation, not for submission. For submission, only GPU - can't use internet connection.
\nI took extra effort to setup TPU for validation, because it is way too much faster (to see the CV) than using GPU.
Alright. Let's hope that the results are better from GPU on LB then :D
", "rawMarkdown": "Alright. Let's hope that the results are better from GPU on LB then :D", "votes": null } ], "comments": [ { "id": 1126933, "author_name": "abdurrafae", "author_url": "", "post_date": "12/26/2020 05:16:23", "content": "Well since computations are done using bfloat16 for TPUs, there would be small differences in both model outputs even if the weights of the models are same. (Weights are still saved as float32 in both cases)
\nI think this slight differences would become smaller and smaller as we increase the number of interactions to check the AUC on.
", "votes": null, "replies": [ { "id": 1127106, "author_name": "yihdarshieh", "author_url": "", "post_date": "12/26/2020 08:43:42", "content": "Yes, I am aware of bfloat16 for TPUs, but didn't know that the gap (between probabilities) could be higher > 0.005 …
Are you using TPU for submission as well? I'm training on TPU but submitting on GPU since my inference pipeline without model is ~1 hour and built on np arrays so can't shift to TPU now.
\nBtw if you have pipeline setup for TPU inference. Submit using GPU and TPU both on the same trained model and check which one gives higher Public LB.
\nDo let me know the result of this experiment :)
", "votes": null, "replies": [] }, { "id": 1127111, "author_name": "yihdarshieh", "author_url": "", "post_date": "12/26/2020 08:52:55", "content": "TPU for fast validation, not for submission. For submission, only GPU - can't use internet connection.
\nI took extra effort to setup TPU for validation, because it is way too much faster (to see the CV) than using GPU.
Alright. Let's hope that the results are better from GPU on LB then :D
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1126674": "(In particular, for anyone using TPU for this competition)\n\nI have a TensorFlow TPU training / validation pipeline for this competition [TPU - Track knowledge states of 1M+ students](https://www.kaggle.com/yihdarshieh/tpu-track-knowledge-states-of-1m-students).\n\nAs I mentioned in an thread [Does the gap between CV and public LB score need to be very small?](https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/206488), the gap between my CV and LB is large, ranging from `0.007` to `0.01`. Therefore I spent some time to look deeper. Nothing wrong being found, but there is an observation.\n\nI actually have 2 validation pipelines:\n\n - one is using validation dataset stored in tensorflow record file, and with this, the validation is done with TPU.\n\n - another one is using a pipeline similar to the submission process, so small batches are fed into the model running on GPU.\n\nI observed that the prediction from TPU / GPU have some differences.\n\nFor example, with TPU, one prediction is like\n\n ```\n