Hello,
\nSo I am using the efficientnet library to use the noisy-student weights to train my model.
\nlike this:
!pip install efficientnet\nAnd then\nbase_model = efn.EfficientNetB4(weights='noisy-student')\nI am getting proper training and validation accuracies.
\nThe problem is that I save this model and load it in another notebook for inference, The prediction from this model is always [[nan, nan, nan, nan, nan]], and the prediction is going to [0] always. I don't know what I am doing wrong, there are no issues in its GitHub regarding this as well.
Due to mixed precision in training version 14, a certain x * y = > float16 * float32 in the code is illegal, so there is an error. If the performance is not considered, you can choose to comment out .
\nIn the new version 15, I have set it right. its working fine it ended due to kaggle's restraint of 9hrs.
\nAnd the performance increase is good so I would like to keep this.
Just looked at your training kernel - the version now being shared (Version 14) has fatal errors.
\nSince you are using mixed precision you should modify the last bit of the model build for each. This looks like a recent change compared to last successful run on Version 10.
\n**Update: adding dtype = …. fixes the issue discussed by zhangeng **
\nmodel.add(Dense(n_CLASS, activation = 'softmax', dtype = 'float32')))
I will make that change and fork the training kernel. Will run with epochs of 1 to see if that fixes the error.
", "rawMarkdown": "Just looked at your training kernel - the version now being shared (Version 14) has fatal errors.\n\nSince you are using mixed precision you should modify the last bit of the model build for each. This looks like a recent change compared to last successful run on Version 10.\n\n**Update: adding dtype = .... fixes the issue discussed by zhangeng **\n\n`model.add(Dense(n_CLASS, activation = 'softmax', dtype = 'float32')))`\n\nI will make that change and fork the training kernel. Will run with epochs of 1 to see if that fixes the error.", "votes": 1, "replies": [ { "id": 1179881, "postDate": "2021-01-31T23:32:12.647Z", "content": "Training ran ok for 1 epoch with the above changes to your different models.
\nThe output of 0 suggests that your model is not really being loaded - likely the folder address is wrong. I could not actually run the prediction kernel since you made the various whl private.
\nGetting folder addresses correct can get messy in kaggle - I always copy and paste the address of added data - anytime I think I can type it I get it wrong.
", "rawMarkdown": "Training ran ok for 1 epoch with the above changes to your different models.\n\nThe output of 0 suggests that your model is not really being loaded - likely the folder address is wrong. I could not actually run the prediction kernel since you made the various whl private.\n\nGetting folder addresses correct can get messy in kaggle - I always copy and paste the address of added data - anytime I think I can type it I get it wrong. " }, { "id": 1180157, "postDate": "2021-02-01T06:09:28.567Z", "content": "Yes The kernel gave this error and I wrote float32 in the final softmax layer to remove this error. The new version 15 which stopped after 9 hours is running fine it stopped due to kaggle's restraint of 9hrs. It had no fatal errors. I think the error is in the submission notebook somewhere. Maybe I will try to delete the dataset and create it again and check whether it improves my situation(the path is also copied, I never enter it by hand because of the confusion.)
\nThank you for the help by the way.
", "rawMarkdown": "Yes The kernel gave this error and I wrote float32 in the final softmax layer to remove this error. The new version 15 which stopped after 9 hours is running fine it stopped due to kaggle's restraint of 9hrs. It had no fatal errors. I think the error is in the submission notebook somewhere. Maybe I will try to delete the dataset and create it again and check whether it improves my situation(the path is also copied, I never enter it by hand because of the confusion.)\n\nThank you for the help by the way." }, { "id": 1180383, "postDate": "2021-02-01T08:25:25.503Z", "content": "OK - you got me a little confused. Your training version 15 ran for 9 hours - right?
\nI can't look at 15 - but looking at 16 - one obvious thing to change that will improve speed a bunch.
\nChange the batch_size from 8 - try 24 it might work.
\nMinor thing to help speed - you don't need to shuffle the validation set.
", "rawMarkdown": "OK - you got me a little confused. Your training version 15 ran for 9 hours - right?\n\nI can't look at 15 - but looking at 16 - one obvious thing to change that will improve speed a bunch.\n\nChange the batch_size from 8 - try 24 it might work. \n\nMinor thing to help speed - you don't need to shuffle the validation set. ", "votes": 1 }, { "id": 1180492, "postDate": "2021-02-01T09:41:03.513Z", "content": "Ok thank you, The version 16 is me trying out something new which I started doing now.
", "rawMarkdown": "Ok thank you, The version 16 is me trying out something new which I started doing now." } ] }, { "id": 1179442, "postDate": "2021-01-31T15:14:29.873Z", "content": "Hello!
\nAccording to the competition rules, all inference notebooks must have the internet connection turned off. So how do you import the efficientnet library to your inference notebook?
yes, I have downloaded the required files .whl files and stored it in a dataset, am installing them in my inference notebook from there. The imports is fine, just the prediction comes out weird.
", "rawMarkdown": "yes, I have downloaded the required files .whl files and stored it in a dataset, am installing them in my inference notebook from there. The imports is fine, just the prediction comes out weird. " }, { "id": 1179516, "postDate": "2021-01-31T15:49:30.817Z", "content": "That's weird.
\nHowever, if you don't mind sharing, I can take a look at your code.
Forked them both and don't see any fatal mistake: the training notebook seems to work fine (I've called model predict and it produced no NaNs), replacing your model with my model in the submission notebook also works as expected (also no NaNs).
The only idea is that your notebook has exceeded the runtime with all the outputs being saved only by the ModelCheckpoint callback. I'd suggest running your model for just one epoch and saving&loading it into the submission notebook again to make sure there's no mistake. If that works fine (like for me) then the error is due to exceeding the runtime.
P.S. I'd suggest removing the Flatten layer after the GlobalAveragePooling2D layer as it carries no sense, and normalizing the images because lack of normalization would greatly affect the model's performance.
OK i will try to do all these measures, but I have one doubt, the normalization is don't in efficientnet layers itself right, is there any need of explicit normalization?
", "rawMarkdown": "OK i will try to do all these measures, but I have one doubt, the normalization is don't in efficientnet layers itself right, is there any need of explicit normalization?" }, { "id": 1180238, "postDate": "2021-02-01T07:06:20.583Z", "content": "This is only happening with the noisy student weights, imagenet weights seems fine.
", "rawMarkdown": "This is only happening with the noisy student weights, imagenet weights seems fine." }, { "id": 1180250, "postDate": "2021-02-01T07:25:22.500Z", "content": "As far as I'm concerned, there are no normalization layers in EfficientNets (except BatchNormalization, which is not the case). It's also a good practice to always normalize your input. This can be done either by placing a tf.keras.layers.experimental.preprocessing.Rescaling(1 / 255.) layer just after your Input layer, with Sklearn's MinMaxScaler class or simply by hand dividing.
If so, I'd suggest changing your source code - there sure must be a mistake. As for me, I'm importing this dataset to both training and inference notebooks and add sys.path.append('/kaggle/input/efficientnet-keras-dataset/efficientnet_kaggle')
\nline to the top of my notebook (efficientnet.tfkeras can then be imported just as usual)
https://keras.io/examples/vision/image_classification_efficientnet_fine_tuning/
\nHere they say Normalization is inbuilt.
", "rawMarkdown": "https://keras.io/examples/vision/image_classification_efficientnet_fine_tuning/\n\nHere they say Normalization is inbuilt." }, { "id": 1181151, "postDate": "2021-02-01T17:24:17.597Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1181152, "postDate": "2021-02-01T17:24:17.597Z", "content": "There are two major versions of efficientnet's - I believe that the version your downloading does need normalization - but not 100% sure.
\nhmmmm - going to be one of those days - I got two comments posted when I hit the go button - I need more coffee or kaggle needs more coffee.
", "rawMarkdown": "There are two major versions of efficientnet's - I believe that the version your downloading does need normalization - but not 100% sure.\n\n\nhmmmm - going to be one of those days - I got two comments posted when I hit the go button - I need more coffee or kaggle needs more coffee." } ] }, { "id": 1179434, "postDate": "2021-01-31T15:06:08.443Z", "content": "Hello,
\nSo I am using the efficientnet library to use the noisy-student weights to train my model.
\nlike this:
!pip install efficientnet\nAnd then\nbase_model = efn.EfficientNetB4(weights='noisy-student')\nI am getting proper training and validation accuracies.
\nThe problem is that I save this model and load it in another notebook for inference, The prediction from this model is always [[nan, nan, nan, nan, nan]], and the prediction is going to [0] always. I don't know what I am doing wrong, there are no issues in its GitHub regarding this as well.
Due to mixed precision in training version 14, a certain x * y = > float16 * float32 in the code is illegal, so there is an error. If the performance is not considered, you can choose to comment out .
\nIn the new version 15, I have set it right. its working fine it ended due to kaggle's restraint of 9hrs.
\nAnd the performance increase is good so I would like to keep this.
Just looked at your training kernel - the version now being shared (Version 14) has fatal errors.
\nSince you are using mixed precision you should modify the last bit of the model build for each. This looks like a recent change compared to last successful run on Version 10.
\n**Update: adding dtype = …. fixes the issue discussed by zhangeng **
\nmodel.add(Dense(n_CLASS, activation = 'softmax', dtype = 'float32')))
I will make that change and fork the training kernel. Will run with epochs of 1 to see if that fixes the error.
", "votes": 1, "replies": [ { "id": 1179881, "author_name": "PC Jimmmy", "author_url": "", "post_date": "2021-01-31T23:32:12.647000", "content": "Training ran ok for 1 epoch with the above changes to your different models.
\nThe output of 0 suggests that your model is not really being loaded - likely the folder address is wrong. I could not actually run the prediction kernel since you made the various whl private.
\nGetting folder addresses correct can get messy in kaggle - I always copy and paste the address of added data - anytime I think I can type it I get it wrong.
", "votes": 0, "replies": [] }, { "id": 1180157, "author_name": "Mohneesh_Sreegirisetty", "author_url": "", "post_date": "2021-02-01T06:09:28.567000", "content": "Yes The kernel gave this error and I wrote float32 in the final softmax layer to remove this error. The new version 15 which stopped after 9 hours is running fine it stopped due to kaggle's restraint of 9hrs. It had no fatal errors. I think the error is in the submission notebook somewhere. Maybe I will try to delete the dataset and create it again and check whether it improves my situation(the path is also copied, I never enter it by hand because of the confusion.)
\nThank you for the help by the way.
", "votes": 0, "replies": [] }, { "id": 1180383, "author_name": "PC Jimmmy", "author_url": "", "post_date": "2021-02-01T08:25:25.503000", "content": "OK - you got me a little confused. Your training version 15 ran for 9 hours - right?
\nI can't look at 15 - but looking at 16 - one obvious thing to change that will improve speed a bunch.
\nChange the batch_size from 8 - try 24 it might work.
\nMinor thing to help speed - you don't need to shuffle the validation set.
", "votes": 1, "replies": [] }, { "id": 1180492, "author_name": "Mohneesh_Sreegirisetty", "author_url": "", "post_date": "2021-02-01T09:41:03.513000", "content": "Ok thank you, The version 16 is me trying out something new which I started doing now.
", "votes": 0, "replies": [] } ] }, { "id": 1179442, "author_name": "Nikita Kuzmenkov", "author_url": "", "post_date": "2021-01-31T15:14:29.873000", "content": "Hello!
\nAccording to the competition rules, all inference notebooks must have the internet connection turned off. So how do you import the efficientnet library to your inference notebook?
yes, I have downloaded the required files .whl files and stored it in a dataset, am installing them in my inference notebook from there. The imports is fine, just the prediction comes out weird.
", "votes": 0, "replies": [] }, { "id": 1179516, "author_name": "Nikita Kuzmenkov", "author_url": "", "post_date": "2021-01-31T15:49:30.817000", "content": "That's weird.
\nHowever, if you don't mind sharing, I can take a look at your code.
Forked them both and don't see any fatal mistake: the training notebook seems to work fine (I've called model predict and it produced no NaNs), replacing your model with my model in the submission notebook also works as expected (also no NaNs).
The only idea is that your notebook has exceeded the runtime with all the outputs being saved only by the ModelCheckpoint callback. I'd suggest running your model for just one epoch and saving&loading it into the submission notebook again to make sure there's no mistake. If that works fine (like for me) then the error is due to exceeding the runtime.
P.S. I'd suggest removing the Flatten layer after the GlobalAveragePooling2D layer as it carries no sense, and normalizing the images because lack of normalization would greatly affect the model's performance.
OK i will try to do all these measures, but I have one doubt, the normalization is don't in efficientnet layers itself right, is there any need of explicit normalization?
", "votes": 0, "replies": [] }, { "id": 1180238, "author_name": "Mohneesh_Sreegirisetty", "author_url": "", "post_date": "2021-02-01T07:06:20.583000", "content": "This is only happening with the noisy student weights, imagenet weights seems fine.
", "votes": 0, "replies": [] }, { "id": 1180250, "author_name": "Nikita Kuzmenkov", "author_url": "", "post_date": "2021-02-01T07:25:22.500000", "content": "As far as I'm concerned, there are no normalization layers in EfficientNets (except BatchNormalization, which is not the case). It's also a good practice to always normalize your input. This can be done either by placing a tf.keras.layers.experimental.preprocessing.Rescaling(1 / 255.) layer just after your Input layer, with Sklearn's MinMaxScaler class or simply by hand dividing.
If so, I'd suggest changing your source code - there sure must be a mistake. As for me, I'm importing this dataset to both training and inference notebooks and add sys.path.append('/kaggle/input/efficientnet-keras-dataset/efficientnet_kaggle')
\nline to the top of my notebook (efficientnet.tfkeras can then be imported just as usual)
https://keras.io/examples/vision/image_classification_efficientnet_fine_tuning/
\nHere they say Normalization is inbuilt.
", "votes": 0, "replies": [] }, { "id": 1181151, "author_name": "", "author_url": "", "post_date": "2021-02-01T17:24:17.597000", "content": "", "votes": 0, "replies": [] }, { "id": 1181152, "author_name": "PC Jimmmy", "author_url": "", "post_date": "2021-02-01T17:24:17.597000", "content": "There are two major versions of efficientnet's - I believe that the version your downloading does need normalization - but not 100% sure.
\nhmmmm - going to be one of those days - I got two comments posted when I hit the go button - I need more coffee or kaggle needs more coffee.
", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "1179946": "Due to mixed precision in training version 14, a certain x * y = > float16 * float32 in the code is illegal, so there is an error. If the performance is not considered, you can choose to comment out .\n# policy = mixed_ precision.Policy ('mixed_ float16')\n#mixed_ precision.set_ policy(policy)", "1179853": "Just looked at your training kernel - the version now being shared (Version 14) has fatal errors.\n\nSince you are using mixed precision you should modify the last bit of the model build for each. This looks like a recent change compared to last successful run on Version 10.\n\n**Update: adding dtype = .... fixes the issue discussed by zhangeng **\n\n`model.add(Dense(n_CLASS, activation = 'softmax', dtype = 'float32')))`\n\nI will make that change and fork the training kernel. Will run with epochs of 1 to see if that fixes the error.", "1179442": "Hello! \n\nAccording to the competition rules, all inference notebooks must have the internet connection turned off. So how do you import the `efficientnet` library to your inference notebook?", "1179434": "Hello,\nSo I am using the efficientnet library to use the noisy-student weights to train my model.\nlike this:\n```\n!pip install efficientnet\nAnd then\nbase_model = efn.EfficientNetB4(weights='noisy-student')\n```\nI am getting proper training and validation accuracies.\nThe problem is that I save this model and load it in another notebook for inference, The prediction from this model is always **[[nan, nan, nan, nan, nan]]**, and the prediction is going to **[0]** always. I don't know what I am doing wrong, there are no issues in its GitHub regarding this as well." } }