{ "id": 209773, "title": "Bi-Tempered Loss [Tensorflow 2.0]", "url": "/competitions/cassava-leaf-disease-classification/discussion/209773", "author_name": "Diulhio de Oliveira", "post_date": "2021-01-08T14:28:03.389000", "votes": 31, "comment_count": 30, "views": 0, "content": "

For those who want to try Bi Tempered Loss and use Tensorflow 2.0, I made an adaptation from pytorch version (https://github.com/mlpanda/bi-tempered-loss-pytorch) :

\n

https://github.com/Diulhio/bitemperedloss-tf

\n

Any problem let me know.

", "messages": [ { "id": 1144553, "postDate": "2021-01-08T14:28:03.390Z", "content": "

For those who want to try Bi Tempered Loss and use Tensorflow 2.0, I made an adaptation from pytorch version (https://github.com/mlpanda/bi-tempered-loss-pytorch) :

\n

https://github.com/Diulhio/bitemperedloss-tf

\n

Any problem let me know.

", "rawMarkdown": "For those who want to try Bi Tempered Loss and use Tensorflow 2.0, I made an adaptation from pytorch version (https://github.com/mlpanda/bi-tempered-loss-pytorch) :\n\nhttps://github.com/Diulhio/bitemperedloss-tf\n\nAny problem let me know.", "votes": 31 }, { "id": 1147854, "postDate": "2021-01-10T18:44:12.593Z", "content": "

Does anyone know if this works with Tensorflow 2.2.0?

", "rawMarkdown": "Does anyone know if this works with Tensorflow 2.2.0?", "votes": 1, "replies": [ { "id": 1147865, "postDate": "2021-01-10T18:53:33.090Z", "content": "

@ayu055 Sorry I don't know your concern, If you want to use TPU and need an updated Tensorflow version you can update both in your kernel and TPU cluster and it is very simple. You can see this kernel where I updated the TensorFlow version. Sorry, again for this is not exactly your question answer and but It will solve your problem.

", "rawMarkdown": "@ayu055 Sorry I don't know your concern, If you want to use TPU and need an updated Tensorflow version you can update both in your kernel and TPU cluster and it is very simple. You can see [this kernel ](https://www.kaggle.com/durbin164/tpu-bitempered-logistic-loss-keras-tensorflow)where I updated the TensorFlow version. Sorry, again for this is not exactly your question answer and but It will solve your problem." }, { "id": 1147880, "postDate": "2021-01-10T19:19:38.803Z", "content": "

I tried Tensorflow 2.2.0 and it didn't work so I did the update and it worked. Thanks!

", "rawMarkdown": "I tried Tensorflow 2.2.0 and it didn't work so I did the update and it worked. Thanks!", "votes": 1 }, { "id": 1148157, "postDate": "2021-01-11T00:14:46.693Z", "content": "

I've using it with TF 2.2.0. Maybe you got a previous version.
\nI updated the repo this afternoon.

", "rawMarkdown": "I've using it with TF 2.2.0. Maybe you got a previous version. \nI updated the repo this afternoon." } ] }, { "id": 1145186, "postDate": "2021-01-09T00:53:16.707Z", "content": "

Thank you for sharing this user-friendly repo!
\nI have tried using this loss in a fork similar to my notebook using Tensorflow 2.3 and I got the following error:
\nTypeError: Input 'y' of 'Maximum' Op has type float32 that does not match type int32 of argument 'x'.
\nI have spent some time trying to figure out whether some variables were not casted into float32 but without success. Would you have any idea where it could be coming from by any chance?

", "rawMarkdown": "Thank you for sharing this user-friendly repo! \nI have tried using this loss in a fork similar to [my notebook](https://www.kaggle.com/frlemarchand/efficientnet-aug-tf-keras-for-cassava-diseases) using Tensorflow 2.3 and I got the following error:\n`TypeError: Input 'y' of 'Maximum' Op has type float32 that does not match type int32 of argument 'x'.`\nI have spent some time trying to figure out whether some variables were not casted into `float32` but without success. Would you have any idea where it could be coming from by any chance?", "votes": 1, "replies": [ { "id": 1145252, "postDate": "2021-01-09T02:15:43.183Z", "content": "

I am also having the same problem

", "rawMarkdown": "I am also having the same problem " }, { "id": 1145405, "postDate": "2021-01-09T05:48:21.240Z", "content": "

@frlemarchand I think it happens because unlike Python, TF is very picky about data types. In this particular case, TF does not like the return of the exp_t(u, t) function which looks like this: tf.math.maximum(0, 1.0 + (1.0 - t) * u) ** (1.0 / (1.0 - t)) in the source code. TF assigns integer type to the first argument of tf.math.maximum and float type to the second argument. This leads to the type mismatch error. Changing the first argument from 0 to 0.0 fixed the problem for me (I am using Colab, TF 2.4, TPU). I am currently on epoch # 8 and it seems to be working fine.

", "rawMarkdown": "@frlemarchand I think it happens because unlike Python, TF is very picky about data types. In this particular case, TF does not like the return of the `exp_t(u, t)` function which looks like this: `tf.math.maximum(0, 1.0 + (1.0 - t) * u) ** (1.0 / (1.0 - t))` in the source code. TF assigns integer type to the first argument of `tf.math.maximum` and float type to the second argument. This leads to the type mismatch error. Changing the first argument from `0` to `0.0` fixed the problem for me (I am using Colab, TF 2.4, TPU). I am currently on epoch # 8 and it seems to be working fine.", "votes": 4 }, { "id": 1146703, "postDate": "2021-01-10T00:50:47.353Z", "content": "

@graf10a is totally correct, the problem is variable type. Tensorflow is really annoying about that.
\nI will modify the code to cast every input as tf.float32, it will avoid this kind of issue in the future.

", "rawMarkdown": "@graf10a is totally correct, the problem is variable type. Tensorflow is really annoying about that.\nI will modify the code to cast every input as tf.float32, it will avoid this kind of issue in the future.", "votes": 1 } ] }, { "id": 1145504, "postDate": "2021-01-09T07:38:03.387Z", "content": "

Is there an argument for label smoothing ?

", "rawMarkdown": "Is there an argument for label smoothing ?\n", "votes": 2, "replies": [ { "id": 1145548, "postDate": "2021-01-09T08:22:11.493Z", "content": "

I have the same problem. The label can't be set too large or too small, but what's the best value?

", "rawMarkdown": "I have the same problem. The label can't be set too large or too small, but what's the best value?" }, { "id": 1147397, "postDate": "2021-01-10T13:41:30.310Z", "content": "

I removed it to do some tests and I forgot to put it again!
\nTake a look now, I included the label_smoothing argument right now.
\nThanks for the comment!

", "rawMarkdown": "I removed it to do some tests and I forgot to put it again!\nTake a look now, I included the label_smoothing argument right now.\nThanks for the comment!", "votes": 2 }, { "id": 1147559, "postDate": "2021-01-10T15:22:17.920Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 1159680, "postDate": "2021-01-19T11:49:30.087Z", "content": "

This is useful. Thanks for sharing @diulhio .

", "rawMarkdown": "This is useful. Thanks for sharing @diulhio ." }, { "id": 1147342, "postDate": "2021-01-10T12:49:12.390Z", "content": "

Is this a resource extensive loss function ?
\nI implemented the same version in my pipeline with some minor changes. Earlier it used to take around 160 seconds to complete one epoch but after adding this now it takes more than double i.e. around 370 seconds/epoch. Although the model is doing pretty good in terms of finding the local minima.

", "rawMarkdown": "Is this a resource extensive loss function ?\nI implemented the same version in my pipeline with some minor changes. Earlier it used to take around 160 seconds to complete one epoch but after adding this now it takes more than double i.e. around 370 seconds/epoch. Although the model is doing pretty good in terms of finding the local minima.", "replies": [ { "id": 1147404, "postDate": "2021-01-10T13:46:39.020Z", "content": "

I didn't have any problem using it on TPU.
\nCan you share your implementation?

", "rawMarkdown": "I didn't have any problem using it on TPU.\nCan you share your implementation?" }, { "id": 1147565, "postDate": "2021-01-10T15:25:51.200Z", "content": "

I was experimenting with this notebook must have done something bad myself. But now that you have updated the repo. I'm using your custom loss ;) and is working completely fine. Thank you for the efforts.

", "rawMarkdown": "I was experimenting with this [notebook](https://www.kaggle.com/durbin164/tpu-bitempered-logistic-loss-keras-tensorflow) must have done something bad myself. But now that you have updated the repo. I'm using your custom loss ;) and is working completely fine. Thank you for the efforts." } ] }, { "id": 1145462, "postDate": "2021-01-09T06:52:34.627Z", "content": "

Google already has an official version, =>https://github.com/google/bi-tempered-loss/tree/master/tensorflow

", "rawMarkdown": "Google already has an official version, =>[https://github.com/google/bi-tempered-loss/tree/master/tensorflow](url)", "replies": [ { "id": 1145495, "postDate": "2021-01-09T07:27:01.987Z", "content": "

When I tried to implement this on Colab's TPU I got a very uninformative error message: Unavailable: Socket closed. Don't know how to debug it. But it works on GPU (much slower!).

", "rawMarkdown": "When I tried to implement this on Colab's TPU I got a very uninformative error message: `Unavailable: Socket closed`. Don't know how to debug it. But it works on GPU (much slower!)." }, { "id": 1145552, "postDate": "2021-01-09T08:23:07.527Z", "content": "

I have the same problem here tf.py_ Fuction, can't be used in TPU?

", "rawMarkdown": "I have the same problem here tf.py_ Fuction, can't be used in TPU?" }, { "id": 1145934, "postDate": "2021-01-09T12:34:46.740Z", "content": "

@graf10a and @zhangeng you can look at this kernel where I am trying to run google Bi-Tempered logistic loss with little modification for Tensorflow 2.3.1. it will run in both TPU and GPU.

", "rawMarkdown": "@graf10a and @zhangeng you can look at [this kernel ](https://www.kaggle.com/durbin164/tpu-bitempered-logistic-loss-keras-tensorflow)where I am trying to run google Bi-Tempered logistic loss with little modification for Tensorflow 2.3.1. it will run in both TPU and GPU.\n", "votes": 4 }, { "id": 1146704, "postDate": "2021-01-10T00:53:14.877Z", "content": "

Is it working for TF2.0? I tried once and it didn't work. Maybe I did something wrong!

", "rawMarkdown": "Is it working for TF2.0? I tried once and it didn't work. Maybe I did something wrong! ", "votes": 1 }, { "id": 1146709, "postDate": "2021-01-10T01:00:37.530Z", "content": "

@diulhio The code must be updated to make it work. See this notebook -- I tried this apporach on Colab with TF 2.4 and it works (my pipeline is different from the one used in this notebook but it does not matter). The trick is to update the original code. This is what Md. Masud Rana did.

", "rawMarkdown": "@diulhio The code must be updated to make it work. See [this notebook](https://www.kaggle.com/durbin164/tpu-bitempered-logistic-loss-keras-tensorflow) -- I tried this apporach on Colab with TF 2.4 and it works (my pipeline is different from the one used in this notebook but it does not matter). The trick is to update the original code. This is what Md. Masud Rana did. ", "votes": 2 }, { "id": 1147191, "postDate": "2021-01-10T10:52:45.967Z", "content": "

@durbin164 keras is used in the training process, it will get stuck when using TPU. It is strange that there is neither training nor error reporting

", "rawMarkdown": "@durbin164 keras is used in the training process, it will get stuck when using TPU. It is strange that there is neither training nor error reporting" }, { "id": 1147350, "postDate": "2021-01-10T12:58:07.927Z", "content": "

@zhangeng TPU needs some initialization time processing. You can see the kernel that i shared previous replay, it will work in TPU. If it not solved please let me know. Thank you.

", "rawMarkdown": "@zhangeng TPU needs some initialization time processing. You can see the kernel that i shared previous replay, it will work in TPU. If it not solved please let me know. Thank you. ", "votes": 2 }, { "id": 1148300, "postDate": "2021-01-11T04:20:43.380Z", "content": "

@durbin164 you very much! After experimentation, I found that I used at the time, which caused the program to freeze, so when using TPU, sometimes I can’t use !

", "rawMarkdown": "@durbin164 you very much! After experimentation, I found that I used at the time, which caused the program to freeze, so when using TPU, sometimes I can’t use !" } ] }, { "id": 1144606, "postDate": "2021-01-08T15:03:49.133Z", "content": "

Is there a pip package ? or is there anyway I can use this in colab

", "rawMarkdown": "Is there a pip package ? or is there anyway I can use this in colab ", "replies": [ { "id": 1144626, "postDate": "2021-01-08T15:15:49.780Z", "content": "

You can just copy the tf_bi_tempered_loss.py content.

", "rawMarkdown": "You can just copy the tf_bi_tempered_loss.py content." }, { "id": 1144670, "postDate": "2021-01-08T15:49:17.203Z", "content": "

Thanks, But a pip package could make it easier and more popular. But thanks for that implementation.

", "rawMarkdown": "Thanks, But a pip package could make it easier and more popular. But thanks for that implementation.\n", "votes": -2 }, { "id": 1147561, "postDate": "2021-01-10T15:23:52.553Z", "content": "
\n

!git clone https://github.com/Diulhio/bitemperedloss-tf.git
\n import sys
\n sys.path.append('./bitemperedloss-tf')
\n from tf_bi_tempered_loss import BiTemperedLogisticLoss

\n
\n

this would do the job

", "rawMarkdown": "> !git clone https://github.com/Diulhio/bitemperedloss-tf.git\nimport sys\nsys.path.append('./bitemperedloss-tf')\nfrom tf_bi_tempered_loss import BiTemperedLogisticLoss\n\nthis would do the job", "votes": 6 }, { "id": 1148272, "postDate": "2021-01-11T03:37:28.353Z", "content": "

Thanks, That worked for me

", "rawMarkdown": "Thanks, That worked for me " } ] } ], "comments": [ { "id": 1147854, "author_name": "ayu055", "author_url": "", "post_date": "2021-01-10T18:44:12.593000", "content": "

Does anyone know if this works with Tensorflow 2.2.0?

", "votes": 1, "replies": [ { "id": 1147865, "author_name": "Md. Masud Rana", "author_url": "", "post_date": "2021-01-10T18:53:33.090000", "content": "

@ayu055 Sorry I don't know your concern, If you want to use TPU and need an updated Tensorflow version you can update both in your kernel and TPU cluster and it is very simple. You can see this kernel where I updated the TensorFlow version. Sorry, again for this is not exactly your question answer and but It will solve your problem.

", "votes": 0, "replies": [] }, { "id": 1147880, "author_name": "ayu055", "author_url": "", "post_date": "2021-01-10T19:19:38.803000", "content": "

I tried Tensorflow 2.2.0 and it didn't work so I did the update and it worked. Thanks!

", "votes": 1, "replies": [] }, { "id": 1148157, "author_name": "Diulhio de Oliveira", "author_url": "", "post_date": "2021-01-11T00:14:46.693000", "content": "

I've using it with TF 2.2.0. Maybe you got a previous version.
\nI updated the repo this afternoon.

", "votes": 0, "replies": [] } ] }, { "id": 1145186, "author_name": "Francois Lemarchand", "author_url": "", "post_date": "2021-01-09T00:53:16.707000", "content": "

Thank you for sharing this user-friendly repo!
\nI have tried using this loss in a fork similar to my notebook using Tensorflow 2.3 and I got the following error:
\nTypeError: Input 'y' of 'Maximum' Op has type float32 that does not match type int32 of argument 'x'.
\nI have spent some time trying to figure out whether some variables were not casted into float32 but without success. Would you have any idea where it could be coming from by any chance?

", "votes": 1, "replies": [ { "id": 1145252, "author_name": "Mithil Salunkhe", "author_url": "", "post_date": "2021-01-09T02:15:43.183000", "content": "

I am also having the same problem

", "votes": 0, "replies": [] }, { "id": 1145405, "author_name": "Alexey Pronin", "author_url": "", "post_date": "2021-01-09T05:48:21.240000", "content": "

@frlemarchand I think it happens because unlike Python, TF is very picky about data types. In this particular case, TF does not like the return of the exp_t(u, t) function which looks like this: tf.math.maximum(0, 1.0 + (1.0 - t) * u) ** (1.0 / (1.0 - t)) in the source code. TF assigns integer type to the first argument of tf.math.maximum and float type to the second argument. This leads to the type mismatch error. Changing the first argument from 0 to 0.0 fixed the problem for me (I am using Colab, TF 2.4, TPU). I am currently on epoch # 8 and it seems to be working fine.

", "votes": 4, "replies": [] }, { "id": 1146703, "author_name": "Diulhio de Oliveira", "author_url": "", "post_date": "2021-01-10T00:50:47.353000", "content": "

@graf10a is totally correct, the problem is variable type. Tensorflow is really annoying about that.
\nI will modify the code to cast every input as tf.float32, it will avoid this kind of issue in the future.

", "votes": 1, "replies": [] } ] }, { "id": 1145504, "author_name": "ilovepotatoes", "author_url": "", "post_date": "2021-01-09T07:38:03.387000", "content": "

Is there an argument for label smoothing ?

", "votes": 2, "replies": [ { "id": 1145548, "author_name": "zhangeng", "author_url": "", "post_date": "2021-01-09T08:22:11.493000", "content": "

I have the same problem. The label can't be set too large or too small, but what's the best value?

", "votes": 0, "replies": [] }, { "id": 1147397, "author_name": "Diulhio de Oliveira", "author_url": "", "post_date": "2021-01-10T13:41:30.310000", "content": "

I removed it to do some tests and I forgot to put it again!
\nTake a look now, I included the label_smoothing argument right now.
\nThanks for the comment!

", "votes": 2, "replies": [] }, { "id": 1147559, "author_name": "", "author_url": "", "post_date": "2021-01-10T15:22:17.920000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 1159680, "author_name": "Saurabh Shahane", "author_url": "", "post_date": "2021-01-19T11:49:30.087000", "content": "

This is useful. Thanks for sharing @diulhio .

", "votes": 0, "replies": [] }, { "id": 1147342, "author_name": "ilovepotatoes", "author_url": "", "post_date": "2021-01-10T12:49:12.390000", "content": "

Is this a resource extensive loss function ?
\nI implemented the same version in my pipeline with some minor changes. Earlier it used to take around 160 seconds to complete one epoch but after adding this now it takes more than double i.e. around 370 seconds/epoch. Although the model is doing pretty good in terms of finding the local minima.

", "votes": 0, "replies": [ { "id": 1147404, "author_name": "Diulhio de Oliveira", "author_url": "", "post_date": "2021-01-10T13:46:39.020000", "content": "

I didn't have any problem using it on TPU.
\nCan you share your implementation?

", "votes": 0, "replies": [] }, { "id": 1147565, "author_name": "ilovepotatoes", "author_url": "", "post_date": "2021-01-10T15:25:51.200000", "content": "

I was experimenting with this notebook must have done something bad myself. But now that you have updated the repo. I'm using your custom loss ;) and is working completely fine. Thank you for the efforts.

", "votes": 0, "replies": [] } ] }, { "id": 1145462, "author_name": "zhangeng", "author_url": "", "post_date": "2021-01-09T06:52:34.627000", "content": "

Google already has an official version, =>https://github.com/google/bi-tempered-loss/tree/master/tensorflow

", "votes": 0, "replies": [ { "id": 1145495, "author_name": "Alexey Pronin", "author_url": "", "post_date": "2021-01-09T07:27:01.987000", "content": "

When I tried to implement this on Colab's TPU I got a very uninformative error message: Unavailable: Socket closed. Don't know how to debug it. But it works on GPU (much slower!).

", "votes": 0, "replies": [] }, { "id": 1145552, "author_name": "zhangeng", "author_url": "", "post_date": "2021-01-09T08:23:07.527000", "content": "

I have the same problem here tf.py_ Fuction, can't be used in TPU?

", "votes": 0, "replies": [] }, { "id": 1145934, "author_name": "Md. Masud Rana", "author_url": "", "post_date": "2021-01-09T12:34:46.740000", "content": "

@graf10a and @zhangeng you can look at this kernel where I am trying to run google Bi-Tempered logistic loss with little modification for Tensorflow 2.3.1. it will run in both TPU and GPU.

", "votes": 4, "replies": [] }, { "id": 1146704, "author_name": "Diulhio de Oliveira", "author_url": "", "post_date": "2021-01-10T00:53:14.877000", "content": "

Is it working for TF2.0? I tried once and it didn't work. Maybe I did something wrong!

", "votes": 1, "replies": [] }, { "id": 1146709, "author_name": "Alexey Pronin", "author_url": "", "post_date": "2021-01-10T01:00:37.530000", "content": "

@diulhio The code must be updated to make it work. See this notebook -- I tried this apporach on Colab with TF 2.4 and it works (my pipeline is different from the one used in this notebook but it does not matter). The trick is to update the original code. This is what Md. Masud Rana did.

", "votes": 2, "replies": [] }, { "id": 1147191, "author_name": "zhangeng", "author_url": "", "post_date": "2021-01-10T10:52:45.967000", "content": "

@durbin164 keras is used in the training process, it will get stuck when using TPU. It is strange that there is neither training nor error reporting

", "votes": 0, "replies": [] }, { "id": 1147350, "author_name": "Md. Masud Rana", "author_url": "", "post_date": "2021-01-10T12:58:07.927000", "content": "

@zhangeng TPU needs some initialization time processing. You can see the kernel that i shared previous replay, it will work in TPU. If it not solved please let me know. Thank you.

", "votes": 2, "replies": [] }, { "id": 1148300, "author_name": "zhangeng", "author_url": "", "post_date": "2021-01-11T04:20:43.380000", "content": "

@durbin164 you very much! After experimentation, I found that I used at the time, which caused the program to freeze, so when using TPU, sometimes I can’t use !

", "votes": 0, "replies": [] } ] }, { "id": 1144606, "author_name": "Mithil Salunkhe", "author_url": "", "post_date": "2021-01-08T15:03:49.133000", "content": "

Is there a pip package ? or is there anyway I can use this in colab

", "votes": 0, "replies": [ { "id": 1144626, "author_name": "Diulhio de Oliveira", "author_url": "", "post_date": "2021-01-08T15:15:49.780000", "content": "

You can just copy the tf_bi_tempered_loss.py content.

", "votes": 0, "replies": [] }, { "id": 1144670, "author_name": "Mithil Salunkhe", "author_url": "", "post_date": "2021-01-08T15:49:17.203000", "content": "

Thanks, But a pip package could make it easier and more popular. But thanks for that implementation.

", "votes": -2, "replies": [] }, { "id": 1147561, "author_name": "ilovepotatoes", "author_url": "", "post_date": "2021-01-10T15:23:52.553000", "content": "
\n

!git clone https://github.com/Diulhio/bitemperedloss-tf.git
\n import sys
\n sys.path.append('./bitemperedloss-tf')
\n from tf_bi_tempered_loss import BiTemperedLogisticLoss

\n
\n

this would do the job

", "votes": 6, "replies": [] }, { "id": 1148272, "author_name": "Mithil Salunkhe", "author_url": "", "post_date": "2021-01-11T03:37:28.353000", "content": "

Thanks, That worked for me

", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "1144553": "For those who want to try Bi Tempered Loss and use Tensorflow 2.0, I made an adaptation from pytorch version (https://github.com/mlpanda/bi-tempered-loss-pytorch) :\n\nhttps://github.com/Diulhio/bitemperedloss-tf\n\nAny problem let me know.", "1147854": "Does anyone know if this works with Tensorflow 2.2.0?", "1145186": "Thank you for sharing this user-friendly repo! \nI have tried using this loss in a fork similar to [my notebook](https://www.kaggle.com/frlemarchand/efficientnet-aug-tf-keras-for-cassava-diseases) using Tensorflow 2.3 and I got the following error:\n`TypeError: Input 'y' of 'Maximum' Op has type float32 that does not match type int32 of argument 'x'.`\nI have spent some time trying to figure out whether some variables were not casted into `float32` but without success. Would you have any idea where it could be coming from by any chance?", "1145504": "Is there an argument for label smoothing ?\n", "1159680": "This is useful. Thanks for sharing @diulhio .", "1147342": "Is this a resource extensive loss function ?\nI implemented the same version in my pipeline with some minor changes. Earlier it used to take around 160 seconds to complete one epoch but after adding this now it takes more than double i.e. around 370 seconds/epoch. Although the model is doing pretty good in terms of finding the local minima.", "1145462": "Google already has an official version, =>[https://github.com/google/bi-tempered-loss/tree/master/tensorflow](url)", "1144606": "Is there a pip package ? or is there anyway I can use this in colab " } }