I spend several hours to configure class_weight arg in model.fit API usage. In official API reference, and official tutorial, the class_weight should be a dict :
\n\n\nclass_weight: Optional dictionary mapping class indices (integers) to a weight (float) value, used for weighting the loss function (during training only). This can be useful to tell the model to \"pay more attention\" to samples from an under-represented class.
\n
And I use sklearn.utils.class_weight.compute_class_weight, which return an array to me. In discussion on stackexchange, it seems I should convert it to dict by enumerate it. It seems works in GPU, but in TPU, I receive an InvalidArgumentError. But it seems work on TPU to pass an array to it directly.
So I wonder, is this an unexpected bug or a correct way to pass an array to class_weight arg in TPU kernels?
I spend several hours to configure class_weight arg in model.fit API usage. In official API reference, and official tutorial, the class_weight should be a dict :
\n\n\nclass_weight: Optional dictionary mapping class indices (integers) to a weight (float) value, used for weighting the loss function (during training only). This can be useful to tell the model to \"pay more attention\" to samples from an under-represented class.
\n
And I use sklearn.utils.class_weight.compute_class_weight, which return an array to me. In discussion on stackexchange, it seems I should convert it to dict by enumerate it. It seems works in GPU, but in TPU, I receive an InvalidArgumentError. But it seems work on TPU to pass an array to it directly.
So I wonder, is this an unexpected bug or a correct way to pass an array to class_weight arg in TPU kernels?
Interesting find. I get the same behaviour as you.\nAs far as I can tell you need to use an array as per @catadanna's kernel.\nhttps://www.kaggle.com/catadanna/data-balancing-solution-with-tpu
", "rawMarkdown": "Interesting find. I get the same behaviour as you.\nAs far as I can tell you need to use an array as per @catadanna's kernel.\nhttps://www.kaggle.com/catadanna/data-balancing-solution-with-tpu", "votes": null }, { "id": "754194", "postDate": "02/23/2020 07:37:05", "content": "Hallo, as I mentioned in my post related to the kernel, this loss is very time consuming. That means, about 2 hours for one epoch == 747 steps ...
", "rawMarkdown": "Hallo, as I mentioned in my post related to the kernel, this loss is very time consuming. That means, about 2 hours for one epoch == 747 steps ...", "votes": null }, { "id": "754242", "postDate": "02/23/2020 09:16:49", "content": "I have seen no significant change in runtime when using class weights, but I don't use your homemade loss function. Thank you for your kernel, I found it useful.
", "rawMarkdown": "I have seen no significant change in runtime when using class weights, but I don't use your homemade loss function. Thank you for your kernel, I found it useful.", "votes": null }, { "id": "754364", "postDate": "02/23/2020 13:14:58", "content": "@ghostskipper Thanks for your reply. I have read the kernel you mentioned. It used a self implemented weighted loss, which can be a walk-around to my case.
", "rawMarkdown": "ghostskipper Thanks for your reply. I have read the kernel you mentioned. It used a self implemented weighted loss, which can be a walk-around to my case.", "votes": null }, { "id": "754366", "postDate": "02/23/2020 13:17:16", "content": "@catadanna I noticed your implementation is based on keras.backend, which use cpu/gpu by default. I wonder if it will help to move it to with strategy.scop() to use tpu. And tf.function decorator always help to accelerating.
what I mean is if you use the weights from @catadanna (weights = df_weights.to_numpy().T)\nand then use them directly in model.fit(...., class_weight=weights)
\nthis should work with minimal performance hit.
In my code I set the variables in with strategy.scope(), indeed. OK modified the Notebook, too. I was already told about tf.function, did not try that yet. I have another problem, even if I set the loss on a tf.keras loss, the behaviour is still the one of my home made loss, 747 steps and takes a lot of time. As if the loss I created was somehow saved somewhere and set by default. I am still struggling in order to be able to set a different loss to model.compile
Oh, sorry for a bit misunderstanding @ghostskipper . In my opinion, it would be more obvious to use a weighted loss function has class_weight parameter, rather than some how magic like keras class_weight parameter.
\n\n```python\ndef weighted_categorical_crossentropy(weights):\n \"\"\"\n A weighted version of keras.objectives.categorical_crossentropy
\n\nVariables:\n weights: numpy array of shape (C,) where C is the number of classes\n\nUsage:\n weights = np.array([0.5,2,10]) # Class one at 0.5, class 2 twice the normal weights, class 3 10x.\n loss = weighted_categorical_crossentropy(weights)\n model.compile(loss=loss,optimizer='adam')\n\"\"\"\n\ndef loss(y_true, y_pred):\n epsilon = 1e-7\n # scale predictions so that the class probas of each sample sum to 1\n y_pred /= tf.math.reduce_sum(y_pred, axis=-1, keepdims=True)\n # clip to prevent NaN's and Inf's\n y_pred = tf.clip_by_value(y_pred, epsilon, 1 - epsilon)\n # calc\n loss = y_true * tf.math.log(y_pred) * weights\n loss = -tf.math.reduce_sum(loss, -1)\n return loss\n\nreturn loss\n```
\n\nI'd like to share my version of weighted categorical cross entropy loss with you, which works as fast as official one. @catadanna
", "rawMarkdown": "Oh, sorry for a bit misunderstanding @ghostskipper . In my opinion, it would be more obvious to use a weighted loss function has class_weight parameter, rather than some how magic like keras class_weight parameter. \n\n```python\ndef weighted_categorical_crossentropy(weights):\n \"\"\"\n A weighted version of keras.objectives.categorical_crossentropy\n \n Variables:\n weights: numpy array of shape (C,) where C is the number of classes\n \n Usage:\n weights = np.array([0.5,2,10]) # Class one at 0.5, class 2 twice the normal weights, class 3 10x.\n loss = weighted_categorical_crossentropy(weights)\n model.compile(loss=loss,optimizer='adam')\n \"\"\"\n \n def loss(y_true, y_pred):\n epsilon = 1e-7\n # scale predictions so that the class probas of each sample sum to 1\n y_pred /= tf.math.reduce_sum(y_pred, axis=-1, keepdims=True)\n # clip to prevent NaN's and Inf's\n y_pred = tf.clip_by_value(y_pred, epsilon, 1 - epsilon)\n # calc\n loss = y_true * tf.math.log(y_pred) * weights\n loss = -tf.math.reduce_sum(loss, -1)\n return loss\n \n return loss\n```\n\nI'd like to share my version of weighted categorical cross entropy loss with you, which works as fast as official one. @catadanna", "votes": null }, { "id": "755469", "postDate": "02/24/2020 19:53:25", "content": "Thank you for finding the solution. I confirm that using an array of weights works on TPU with the standard sparse_categorical_crossentropy loss. Like this:
\nclass_weights = [1.0 for _ in range(len(CLASSES))]\nhistory = model.fit(get_training_dataset(), steps_per_epoch=STEPS_PER_EPOCH, epochs=EPOCHS,\n validation_data=get_validation_dataset(),\n class_weight=class_weights)\n
But I agree with you that, according to the documentation, the class_weights should be a dictionary. I filed a bug.
", "rawMarkdown": "But I agree with you that, according to the documentation, the class_weights should be a dictionary. I filed a bug.", "votes": null }, { "id": "799469", "postDate": "04/06/2020 13:13:12", "content": "class_weight parameter in model.fit() can be used to assigned different weights to different classes, to deal with the problem of imbalanced data , when you have more simple for one class then another (exemple you have 400 image of class 1 and 20 image of class 2 and 10 image of class3 etc )
\n\non keras you have to pass a dictionary to classweight parameter in model.fit() example classweights ={0:50, 1:0.5, 2:20}\nmodel.fit(…. …. classweight=classweights)
\n\nbut on TPU your have to pass flat list of weights (not a dictionary as written in the Keras docs)\nso you have just to convert your dictionary to a flat list
\n\nclassweights =[item for k in classweights for item in (k, class_weights[k])]
\n\nNo need to define a custom loss.
", "rawMarkdown": "class_weight parameter in model.fit() can be used to assigned different weights to different classes, to deal with the problem of imbalanced data , when you have more simple for one class then another (exemple you have 400 image of class 1 and 20 image of class 2 and 10 image of class3 etc )\n\non keras you have to pass a dictionary to classweight parameter in model.fit() example classweights ={0:50, 1:0.5, 2:20}\nmodel.fit(…. …. classweight=classweights)\n\nbut on TPU your have to pass flat list of weights (not a dictionary as written in the Keras docs)\nso you have just to convert your dictionary to a flat list\n\nclassweights =[item for k in classweights for item in (k, class_weights[k])]\n\nNo need to define a custom loss.", "votes": null } ], "comments": [ { "id": 754018, "author_name": "ghostskipper", "author_url": "", "post_date": "02/22/2020 23:40:58", "content": "Interesting find. I get the same behaviour as you.\nAs far as I can tell you need to use an array as per @catadanna's kernel.\nhttps://www.kaggle.com/catadanna/data-balancing-solution-with-tpu
", "votes": null, "replies": [ { "id": 754194, "author_name": "catadanna", "author_url": "", "post_date": "02/23/2020 07:37:05", "content": "Hallo, as I mentioned in my post related to the kernel, this loss is very time consuming. That means, about 2 hours for one epoch == 747 steps ...
", "votes": null, "replies": [] }, { "id": 754242, "author_name": "ghostskipper", "author_url": "", "post_date": "02/23/2020 09:16:49", "content": "I have seen no significant change in runtime when using class weights, but I don't use your homemade loss function. Thank you for your kernel, I found it useful.
", "votes": null, "replies": [] }, { "id": 754364, "author_name": "swordfaith", "author_url": "", "post_date": "02/23/2020 13:14:58", "content": "@ghostskipper Thanks for your reply. I have read the kernel you mentioned. It used a self implemented weighted loss, which can be a walk-around to my case.
", "votes": null, "replies": [] }, { "id": 754366, "author_name": "swordfaith", "author_url": "", "post_date": "02/23/2020 13:17:16", "content": "@catadanna I noticed your implementation is based on keras.backend, which use cpu/gpu by default. I wonder if it will help to move it to with strategy.scop() to use tpu. And tf.function decorator always help to accelerating.
what I mean is if you use the weights from @catadanna (weights = df_weights.to_numpy().T)\nand then use them directly in model.fit(...., class_weight=weights)
\nthis should work with minimal performance hit.
In my code I set the variables in with strategy.scope(), indeed. OK modified the Notebook, too. I was already told about tf.function, did not try that yet. I have another problem, even if I set the loss on a tf.keras loss, the behaviour is still the one of my home made loss, 747 steps and takes a lot of time. As if the loss I created was somehow saved somewhere and set by default. I am still struggling in order to be able to set a different loss to model.compile
Oh, sorry for a bit misunderstanding @ghostskipper . In my opinion, it would be more obvious to use a weighted loss function has class_weight parameter, rather than some how magic like keras class_weight parameter.
\n\n```python\ndef weighted_categorical_crossentropy(weights):\n \"\"\"\n A weighted version of keras.objectives.categorical_crossentropy
\n\nVariables:\n weights: numpy array of shape (C,) where C is the number of classes\n\nUsage:\n weights = np.array([0.5,2,10]) # Class one at 0.5, class 2 twice the normal weights, class 3 10x.\n loss = weighted_categorical_crossentropy(weights)\n model.compile(loss=loss,optimizer='adam')\n\"\"\"\n\ndef loss(y_true, y_pred):\n epsilon = 1e-7\n # scale predictions so that the class probas of each sample sum to 1\n y_pred /= tf.math.reduce_sum(y_pred, axis=-1, keepdims=True)\n # clip to prevent NaN's and Inf's\n y_pred = tf.clip_by_value(y_pred, epsilon, 1 - epsilon)\n # calc\n loss = y_true * tf.math.log(y_pred) * weights\n loss = -tf.math.reduce_sum(loss, -1)\n return loss\n\nreturn loss\n```
\n\nI'd like to share my version of weighted categorical cross entropy loss with you, which works as fast as official one. @catadanna
", "votes": null, "replies": [] } ] }, { "id": 755469, "author_name": "mgorner", "author_url": "", "post_date": "02/24/2020 19:53:25", "content": "Thank you for finding the solution. I confirm that using an array of weights works on TPU with the standard sparse_categorical_crossentropy loss. Like this:
\nclass_weights = [1.0 for _ in range(len(CLASSES))]\nhistory = model.fit(get_training_dataset(), steps_per_epoch=STEPS_PER_EPOCH, epochs=EPOCHS,\n validation_data=get_validation_dataset(),\n class_weight=class_weights)\n
But I agree with you that, according to the documentation, the class_weights should be a dictionary. I filed a bug.
", "votes": null, "replies": [] } ] }, { "id": 799469, "author_name": "hatemamine", "author_url": "", "post_date": "04/06/2020 13:13:12", "content": "class_weight parameter in model.fit() can be used to assigned different weights to different classes, to deal with the problem of imbalanced data , when you have more simple for one class then another (exemple you have 400 image of class 1 and 20 image of class 2 and 10 image of class3 etc )
\n\non keras you have to pass a dictionary to classweight parameter in model.fit() example classweights ={0:50, 1:0.5, 2:20}\nmodel.fit(…. …. classweight=classweights)
\n\nbut on TPU your have to pass flat list of weights (not a dictionary as written in the Keras docs)\nso you have just to convert your dictionary to a flat list
\n\nclassweights =[item for k in classweights for item in (k, class_weights[k])]
\n\nNo need to define a custom loss.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "753521": "I spend several hours to configure `class_weight` arg in ` model.fit` API usage. In [official API reference](https://www.tensorflow.org/api_docs/python/tf/keras/Model), and [official tutorial](https://www.tensorflow.org/tutorials/structured_data/imbalanced_data), the class_weight should be a `dict` : \n\n>class_weight: Optional dictionary mapping class indices (integers) to a weight (float) value, used for weighting the loss function (during training only). This can be useful to tell the model to \"pay more attention\" to samples from an under-represented class.\n\nAnd I use `sklearn.utils.class_weight.compute_class_weight`, which return an array to me. In [discussion on stackexchange](https://datascience.stackexchange.com/questions/13490/how-to-set-class-weights-for-imbalanced-classes-in-keras), it seems I should convert it to `dict` by enumerate it. It seems works in GPU, but in TPU, I receive an InvalidArgumentError. But it seems work on TPU to pass an array to it directly. \n\nSo I wonder, is this an unexpected bug or a correct way to pass an array to `class_weight` arg in TPU kernels?", "754018": "Interesting find. I get the same behaviour as you.\nAs far as I can tell you need to use an array as per @catadanna's kernel.\nhttps://www.kaggle.com/catadanna/data-balancing-solution-with-tpu", "754194": "Hallo, as I mentioned in my post related to the kernel, this loss is very time consuming. That means, about 2 hours for one epoch == 747 steps ...", "754242": "I have seen no significant change in runtime when using class weights, but I don't use your homemade loss function. Thank you for your kernel, I found it useful.", "754364": "ghostskipper Thanks for your reply. I have read the kernel you mentioned. It used a self implemented weighted loss, which can be a walk-around to my case.", "754366": "catadanna I noticed your implementation is based on keras.backend, which use cpu/gpu by default. I wonder if it will help to move it to `with strategy.scop()` to use tpu. And `tf.function` decorator always help to accelerating.", "754375": "what I mean is if you use the weights from @catadanna (`weights = df_weights.to_numpy().T`)\nand then use them directly in `model.fit(...., class_weight=weights)` \nthis should work with minimal performance hit.", "754378": "In my code I set the variables in `with strategy.scope()`, indeed. OK modified the Notebook, too. I was already told about `tf.function`, did not try that yet. I have another problem, even if I set the loss on a `tf.keras` loss, the behaviour is still the one of my home made loss, 747 steps and takes a lot of time. As if the loss I created was somehow saved somewhere and set by default. I am still struggling in order to be able to set a different loss to `model.compile`", "754880": "Oh, sorry for a bit misunderstanding @ghostskipper . In my opinion, it would be more obvious to use a weighted loss function has class_weight parameter, rather than some how magic like keras class_weight parameter. \n\n```python\ndef weighted_categorical_crossentropy(weights):\n \"\"\"\n A weighted version of keras.objectives.categorical_crossentropy\n \n Variables:\n weights: numpy array of shape (C,) where C is the number of classes\n \n Usage:\n weights = np.array([0.5,2,10]) # Class one at 0.5, class 2 twice the normal weights, class 3 10x.\n loss = weighted_categorical_crossentropy(weights)\n model.compile(loss=loss,optimizer='adam')\n \"\"\"\n \n def loss(y_true, y_pred):\n epsilon = 1e-7\n # scale predictions so that the class probas of each sample sum to 1\n y_pred /= tf.math.reduce_sum(y_pred, axis=-1, keepdims=True)\n # clip to prevent NaN's and Inf's\n y_pred = tf.clip_by_value(y_pred, epsilon, 1 - epsilon)\n # calc\n loss = y_true * tf.math.log(y_pred) * weights\n loss = -tf.math.reduce_sum(loss, -1)\n return loss\n \n return loss\n```\n\nI'd like to share my version of weighted categorical cross entropy loss with you, which works as fast as official one. @catadanna", "755469": "Thank you for finding the solution. I confirm that using an array of weights works on TPU with the standard `sparse_categorical_crossentropy` loss. Like this:\n\n```\nclass_weights = [1.0 for _ in range(len(CLASSES))]\nhistory = model.fit(get_training_dataset(), steps_per_epoch=STEPS_PER_EPOCH, epochs=EPOCHS,\n validation_data=get_validation_dataset(),\n class_weight=class_weights)\n```", "755488": "But I agree with you that, according to the documentation, the class_weights should be a dictionary. I filed a bug.", "799469": "class_weight parameter in model.fit() can be used to assigned different weights to different classes, to deal with the problem of imbalanced data , when you have more simple for one class then another (exemple you have 400 image of class 1 and 20 image of class 2 and 10 image of class3 etc )\n\non keras you have to pass a dictionary to classweight parameter in model.fit() example classweights ={0:50, 1:0.5, 2:20}\nmodel.fit(…. …. classweight=classweights)\n\nbut on TPU your have to pass flat list of weights (not a dictionary as written in the Keras docs)\nso you have just to convert your dictionary to a flat list\n\nclassweights =[item for k in classweights for item in (k, class_weights[k])]\n\nNo need to define a custom loss." }, "source": "meta" }