I am trying to use the latest environments but i am facing some weird issue, I have tried using a TPU which also had teh same error, the model I am experimenting with is the EfficientNet model from this notebook, the model has 6 output classes, but when predicting it generates wrong number of outputs
\nfor i, batch in enumerate(train_gen):
\n print(f\"Batch {i+1}: Input shape: {batch[0].shape}, Target shape: {batch[1].shape}\")
\n break
with strategy.scope():
\n model = build_model()
\n model.fit( batch[0], batch[1],epochs = 1, verbose=1)
\n oof = model.predict(batch[0], verbose=1)
\n print(batch[0].shape,batch[1].shape)
\n print(model.summary())
(44, 128, 256, 8) (44, 6) are the respectives of batch[0] and batch[1],
\nand the oof.shape is
\n(2,)
\nI nearly wasted 10 hours of GPU previous week trying to saving my notebook versions while experimenting, just to know it was all wasted, I encountered the same is occuring in TPU and when I am passing the datagenerator itself, it runs fine the entire training process and its throwing error in the last split
I would appreciate your help in fixing this
\nThank you.
I am trying to use the latest environments but i am facing some weird issue, I have tried using a TPU which also had teh same error, the model I am experimenting with is the EfficientNet model from this notebook, the model has 6 output classes, but when predicting it generates wrong number of outputs
\nfor i, batch in enumerate(train_gen):
\n print(f\"Batch {i+1}: Input shape: {batch[0].shape}, Target shape: {batch[1].shape}\")
\n break
with strategy.scope():
\n model = build_model()
\n model.fit( batch[0], batch[1],epochs = 1, verbose=1)
\n oof = model.predict(batch[0], verbose=1)
\n print(batch[0].shape,batch[1].shape)
\n print(model.summary())
(44, 128, 256, 8) (44, 6) are the respectives of batch[0] and batch[1],
\nand the oof.shape is
\n(2,)
\nI nearly wasted 10 hours of GPU previous week trying to saving my notebook versions while experimenting, just to know it was all wasted, I encountered the same is occuring in TPU and when I am passing the datagenerator itself, it runs fine the entire training process and its throwing error in the last split
I would appreciate your help in fixing this
\nThank you.
I came across the same issue with TPU, I also tried to move to 2.15.0, so I could switch easily between Colab and Kaggle, but I got alot of headache.
\nI think there is a bug in 2.14 and 2.15 that doesn't register the GPUs properly, you see that error at the very beginning when you try to import tensorflow, that issue could be related, I am not sure.
Thank you for the response, I also tried downgrading tf to 2.13.0 while using newer env, but all the dependency errors that I faced were too much to deal with, I moved to a pytorch based notebook for experimentation, which takes nearly twice as much as time for each epoch when compared with tf (just beginning of the week and already running of resources),
", "rawMarkdown": "Thank you for the response, I also tried downgrading tf to 2.13.0 while using newer env, but all the dependency errors that I faced were too much to deal with, I moved to a pytorch based notebook for experimentation, which takes nearly twice as much as time for each epoch when compared with tf (just beginning of the week and already running of resources),", "votes": null }, { "id": "2679593", "postDate": "03/03/2024 16:09:31", "content": "You could fork this notebook, it uses 2.13.0 by default, so no need to downgrade.
", "rawMarkdown": "You could fork this [notebook](https://www.kaggle.com/code/nartaa/features-head-starter), it uses 2.13.0 by default, so no need to downgrade.", "votes": null }, { "id": "2679624", "postDate": "03/03/2024 16:36:35", "content": "Hi, I already did, it was a really good notebook for you to be able to fit all 4 models in a single notebook, but it can be over whelming when trying to experiment with individual models, so I am focusing on simple notebooks for all the testing and experimentations :')
", "rawMarkdown": "Hi, I already did, it was a really good notebook for you to be able to fit all 4 models in a single notebook, but it can be over whelming when trying to experiment with individual models, so I am focusing on simple notebooks for all the testing and experimentations :')", "votes": null }, { "id": "2679720", "postDate": "03/03/2024 17:34:52", "content": "I hear ya 👍
", "rawMarkdown": "I hear ya 👍", "votes": null }, { "id": "2680011", "postDate": "03/03/2024 22:37:25", "content": "I encountered a similar problem and found a solution. Initially, my code looked like this:
\nwith strategy.scope():\n model = build_model()\nmodel.fit(train_gen, =1, =valid_gen)\noof = model.predict(valid_gen, =1)\nHowever, the shape of oof turned out to be (1, 53), which likely corresponds to (1, number_of_batches).
To address this, I modified my code as follows:
\nwith strategy.scope():\n = build_model()\n.fit(train_gen, verbose=, validation_data=valid_gen)\n\n = build_model() # Rebuild the \n.load_weights(weight_path)\n\noof = .predict(valid_gen, verbose=)\nThis adjustment successfully changed the shape of oof to (3346, 6), which matches the expected (data_length, number_of_outputs). It appears that strategy.scope() can impact the prediction phase.
Thanks for helping out, it fixed the issue :')
", "rawMarkdown": "Thanks for helping out, it fixed the issue :')", "votes": null } ], "comments": [ { "id": 2678585, "author_name": "nartaa", "author_url": "", "post_date": "03/02/2024 22:09:50", "content": "I came across the same issue with TPU, I also tried to move to 2.15.0, so I could switch easily between Colab and Kaggle, but I got alot of headache.
\nI think there is a bug in 2.14 and 2.15 that doesn't register the GPUs properly, you see that error at the very beginning when you try to import tensorflow, that issue could be related, I am not sure.
Thank you for the response, I also tried downgrading tf to 2.13.0 while using newer env, but all the dependency errors that I faced were too much to deal with, I moved to a pytorch based notebook for experimentation, which takes nearly twice as much as time for each epoch when compared with tf (just beginning of the week and already running of resources),
", "votes": null, "replies": [ { "id": 2679593, "author_name": "nartaa", "author_url": "", "post_date": "03/03/2024 16:09:31", "content": "You could fork this notebook, it uses 2.13.0 by default, so no need to downgrade.
", "votes": null, "replies": [ { "id": 2679624, "author_name": "arunsensei", "author_url": "", "post_date": "03/03/2024 16:36:35", "content": "Hi, I already did, it was a really good notebook for you to be able to fit all 4 models in a single notebook, but it can be over whelming when trying to experiment with individual models, so I am focusing on simple notebooks for all the testing and experimentations :')
", "votes": null, "replies": [ { "id": 2679720, "author_name": "nartaa", "author_url": "", "post_date": "03/03/2024 17:34:52", "content": "I hear ya 👍
", "votes": null, "replies": [] } ] } ] } ] } ] }, { "id": 2680011, "author_name": "spicatom0914", "author_url": "", "post_date": "03/03/2024 22:37:25", "content": "I encountered a similar problem and found a solution. Initially, my code looked like this:
\nwith strategy.scope():\n model = build_model()\nmodel.fit(train_gen, =1, =valid_gen)\noof = model.predict(valid_gen, =1)\nHowever, the shape of oof turned out to be (1, 53), which likely corresponds to (1, number_of_batches).
To address this, I modified my code as follows:
\nwith strategy.scope():\n = build_model()\n.fit(train_gen, verbose=, validation_data=valid_gen)\n\n = build_model() # Rebuild the \n.load_weights(weight_path)\n\noof = .predict(valid_gen, verbose=)\nThis adjustment successfully changed the shape of oof to (3346, 6), which matches the expected (data_length, number_of_outputs). It appears that strategy.scope() can impact the prediction phase.
Thanks for helping out, it fixed the issue :')
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2677566": "I am trying to use the latest environments but i am facing some weird issue, I have tried using a TPU which also had teh same error, the model I am experimenting with is the EfficientNet model from this [notebook](https://www.kaggle.com/code/cdeotte/efficientnetb0-starter-lb-0-43), the model has 6 output classes, but when predicting it generates wrong number of outputs\n\nfor i, batch in enumerate(train_gen):\n print(f\"Batch {i+1}: Input shape: {batch[0].shape}, Target shape: {batch[1].shape}\")\n break\n\nwith strategy.scope():\n model = build_model()\n model.fit( batch[0], batch[1],epochs = 1, verbose=1)\n oof = model.predict(batch[0], verbose=1)\n print(batch[0].shape,batch[1].shape)\n print(model.summary())\n\n(44, 128, 256, 8) (44, 6) are the respectives of batch[0] and batch[1], \n\nand the oof.shape is \n(2,)\nI nearly wasted 10 hours of GPU previous week trying to saving my notebook versions while experimenting, just to know it was all wasted, I encountered the same is occuring in TPU and when I am passing the datagenerator itself, it runs fine the entire training process and its throwing error in the last split\n\nI would appreciate your help in fixing this\nThank you.", "2678585": "I came across the same issue with TPU, I also tried to move to 2.15.0, so I could switch easily between Colab and Kaggle, but I got alot of headache.\nI think there is a bug in 2.14 and 2.15 that doesn't register the GPUs properly, you see that error at the very beginning when you try to import tensorflow, that issue could be related, I am not sure.", "2679578": "Thank you for the response, I also tried downgrading tf to 2.13.0 while using newer env, but all the dependency errors that I faced were too much to deal with, I moved to a pytorch based notebook for experimentation, which takes nearly twice as much as time for each epoch when compared with tf (just beginning of the week and already running of resources),", "2679593": "You could fork this [notebook](https://www.kaggle.com/code/nartaa/features-head-starter), it uses 2.13.0 by default, so no need to downgrade.", "2679624": "Hi, I already did, it was a really good notebook for you to be able to fit all 4 models in a single notebook, but it can be over whelming when trying to experiment with individual models, so I am focusing on simple notebooks for all the testing and experimentations :')", "2679720": "I hear ya 👍", "2680011": "I encountered a similar problem and found a solution. Initially, my code looked like this:\n```\nwith strategy.scope():\n model = build_model()\nmodel.fit(train_gen, verbose=1, validation_data=valid_gen)\noof = model.predict(valid_gen, verbose=1)\n```\nHowever, the shape of `oof` turned out to be (1, 53), which likely corresponds to (1, number_of_batches).\n\nTo address this, I modified my code as follows:\n```\nwith strategy.scope():\n model = build_model()\nmodel.fit(train_gen, verbose=1, validation_data=valid_gen)\n\nmodel = build_model() # Rebuild the model\nmodel.load_weights(weight_path)\n\noof = model.predict(valid_gen, verbose=1)\n```\nThis adjustment successfully changed the shape of `oof` to (3346, 6), which matches the expected (data_length, number_of_outputs). It appears that strategy.scope() can impact the prediction phase.", "2680494": "Thanks for helping out, it fixed the issue :')" }, "source": "meta" }