{ "id": 270311, "title": "How to load efficientnet without internet-access", "url": "/competitions/rsna-miccai-brain-tumor-radiogenomic-classification/discussion/270311", "author_name": "", "post_date": "2021-09-04T15:56:47.965379Z", "votes": 2, "comment_count": 9, "views": 0, "content": "

Hi,

\n

I'm not very familiar with kaggle-environment and its restrictions. Hence; what is the best way to be able get e.g. efficientnet - models in spirit of \"Freely & publicly available external data is allowed, including pre-trained models\".

\n

I can do it of course by switching \"internet on\", but the rules say internet should be off to be able to make submissions. Thus - how to get efficientnet to the notebook without internet?

\n

Thank you for kind answer for this \"too simple\" question.

", "messages": [ { "id": "1502751", "postDate": "09/04/2021 15:56:47", "content": "

Hi,

\n

I'm not very familiar with kaggle-environment and its restrictions. Hence; what is the best way to be able get e.g. efficientnet - models in spirit of \"Freely & publicly available external data is allowed, including pre-trained models\".

\n

I can do it of course by switching \"internet on\", but the rules say internet should be off to be able to make submissions. Thus - how to get efficientnet to the notebook without internet?

\n

Thank you for kind answer for this \"too simple\" question.

", "rawMarkdown": "Hi, \n\nI'm not very familiar with kaggle-environment and its restrictions. Hence; what is the best way to be able get e.g. efficientnet - models in spirit of \"Freely & publicly available external data is allowed, including pre-trained models\".\n\nI can do it of course by switching \"internet on\", but the rules say internet should be off to be able to make submissions. Thus - how to get efficientnet to the notebook without internet? \n\nThank you for kind answer for this \"too simple\" question.", "votes": null }, { "id": "1502763", "postDate": "09/04/2021 16:06:31", "content": "

Add it as a dataset to your notebook or any library for that matter. efficientnet3D

\n

import sys\nsys.path.append(*pytorch3dpath*)\nfrom efficientnet_pytorch_3d import EfficientNet3D

", "rawMarkdown": "Add it as a dataset to your notebook or any library for that matter. [efficientnet3D](https://www.kaggle.com/hihunjin/efficientnetpyttorch3d)\n\n`import sys\nsys.path.append(*pytorch3dpath*)\nfrom efficientnet_pytorch_3d import EfficientNet3D`", "votes": null }, { "id": "1502853", "postDate": "09/04/2021 17:39:44", "content": "

Hi, if you are using a regular (2D CNN) TF efficientnet, for instance, this Kaggle dataset, has imagenet weights for all tf.keras.applications.EfficientNetB* models.

\n

Once you add the dataset to your notebook, this would load the pretrained weights: model = EfficientNetB0(weights='/kaggle/input/tfkerasefficientnetimagenetnotop/efficientnetb0_notop.h5', include_top=False)

", "rawMarkdown": "Hi, if you are using a regular (2D CNN) TF efficientnet, for instance, [this Kaggle dataset](https://www.kaggle.com/kaorusasakawa/tfkerasefficientnetimagenetnotop), has imagenet weights for all `tf.keras.applications.EfficientNetB*` models. \n\nOnce you add the dataset to your notebook, this would load the pretrained weights: `model = EfficientNetB0(weights='/kaggle/input/tfkerasefficientnetimagenetnotop/efficientnetb0_notop.h5', include_top=False)`", "votes": null }, { "id": "1502870", "postDate": "09/04/2021 17:59:22", "content": "

Thank you! So whatever weights can be used if found from the \"kaggle dataset\" section! That's clear πŸ‘πŸ‘

", "rawMarkdown": "Thank you! So whatever weights can be used if found from the \"kaggle dataset\" section! That's clear πŸ‘πŸ‘", "votes": null }, { "id": "1502917", "postDate": "09/04/2021 18:46:22", "content": "

And you can create your own, public or private, datasets from e.g. previously trained weights. You can use any Kaggle dataset given it follows the competition rules.

", "rawMarkdown": "And you can create your own, public or private, datasets from e.g. previously trained weights. You can use any Kaggle dataset given it follows the competition rules.", "votes": null }, { "id": "1503151", "postDate": "09/05/2021 04:22:13", "content": "

you can separate training and submission into two notebooks, then switch the internet on in the training notebook. After training, add the training notebook as a dataset in your submission notebook to access your saved model/weight file and do the prediction.

", "rawMarkdown": "you can separate training and submission into two notebooks, then switch the internet on in the training notebook. After training, add the training notebook as a dataset in your submission notebook to access your saved model/weight file and do the prediction.", "votes": null }, { "id": "1504284", "postDate": "09/06/2021 08:36:09", "content": "

You can create 1 notebook for training, and 1 for inference. You can do whatever you need in training notebook, such as accessing internet, run training on TPU and…, then if you get satisfied with the results, save your model model.save(\"model.h5\") , and load it on inference notebook with internet off and make prediction for test set. Than you can submit your inference notebook.

", "rawMarkdown": "You can create 1 notebook for training, and 1 for inference. You can do whatever you need in training notebook, such as accessing internet, run training on TPU and..., then if you get satisfied with the results, save your model `model.save(\"model.h5\")` , and load it on inference notebook with internet off and make prediction for test set. Than you can submit your inference notebook.", "votes": null }, { "id": "1529348", "postDate": "09/30/2021 10:11:20", "content": "

@gdoong thank you for commenting interenet-on-off topic, it was useful. I have a new problem - I would like to use tensorflow version 2.6 instead of the current 2.4.1. version I have. Is there any easy way to do it? I have found something , but is it up-to-date and best way? https://www.kaggle.com/c/severstal-steel-defect-detection/discussion/113195

", "rawMarkdown": "gdoong thank you for commenting interenet-on-off topic, it was useful. I have a new problem - I would like to use tensorflow version 2.6 instead of the current 2.4.1. version I have. Is there any easy way to do it? I have found something , but is it up-to-date and best way? https://www.kaggle.com/c/severstal-steel-defect-detection/discussion/113195", "votes": null }, { "id": "1529354", "postDate": "09/30/2021 10:15:01", "content": "

FYI above for @cdeotte , comments are welcome!

", "rawMarkdown": "FYI above for @cdeotte , comments are welcome!", "votes": null }, { "id": "1530003", "postDate": "09/30/2021 20:56:53", "content": "

Hi, you can update the TF version from pip wheel that is packaged to a dataset. Here is an example notebook.

\n

This dataset has all packages required in tf-gpu 2.6.0 update.

", "rawMarkdown": "Hi, you can update the TF version from pip wheel that is packaged to a dataset. Here is an example [notebook](https://www.kaggle.com/qitvision/tensorflow-gpu-2-6-0-offline).\n\n[This](https://www.kaggle.com/qitvision/tensorflowgpu260) dataset has all packages required in tf-gpu 2.6.0 update.", "votes": null } ], "comments": [ { "id": 1502763, "author_name": "divyanuhan", "author_url": "", "post_date": "09/04/2021 16:06:31", "content": "

Add it as a dataset to your notebook or any library for that matter. efficientnet3D

\n

import sys\nsys.path.append(*pytorch3dpath*)\nfrom efficientnet_pytorch_3d import EfficientNet3D

", "votes": null, "replies": [] }, { "id": 1502853, "author_name": "qitvision", "author_url": "", "post_date": "09/04/2021 17:39:44", "content": "

Hi, if you are using a regular (2D CNN) TF efficientnet, for instance, this Kaggle dataset, has imagenet weights for all tf.keras.applications.EfficientNetB* models.

\n

Once you add the dataset to your notebook, this would load the pretrained weights: model = EfficientNetB0(weights='/kaggle/input/tfkerasefficientnetimagenetnotop/efficientnetb0_notop.h5', include_top=False)

", "votes": null, "replies": [] }, { "id": 1502870, "author_name": "experienceinai", "author_url": "", "post_date": "09/04/2021 17:59:22", "content": "

Thank you! So whatever weights can be used if found from the \"kaggle dataset\" section! That's clear πŸ‘πŸ‘

", "votes": null, "replies": [ { "id": 1502917, "author_name": "qitvision", "author_url": "", "post_date": "09/04/2021 18:46:22", "content": "

And you can create your own, public or private, datasets from e.g. previously trained weights. You can use any Kaggle dataset given it follows the competition rules.

", "votes": null, "replies": [] } ] }, { "id": 1503151, "author_name": "gdoong", "author_url": "", "post_date": "09/05/2021 04:22:13", "content": "

you can separate training and submission into two notebooks, then switch the internet on in the training notebook. After training, add the training notebook as a dataset in your submission notebook to access your saved model/weight file and do the prediction.

", "votes": null, "replies": [] }, { "id": 1504284, "author_name": "kavehshahhosseini", "author_url": "", "post_date": "09/06/2021 08:36:09", "content": "

You can create 1 notebook for training, and 1 for inference. You can do whatever you need in training notebook, such as accessing internet, run training on TPU and…, then if you get satisfied with the results, save your model model.save(\"model.h5\") , and load it on inference notebook with internet off and make prediction for test set. Than you can submit your inference notebook.

", "votes": null, "replies": [] }, { "id": 1529348, "author_name": "experienceinai", "author_url": "", "post_date": "09/30/2021 10:11:20", "content": "

@gdoong thank you for commenting interenet-on-off topic, it was useful. I have a new problem - I would like to use tensorflow version 2.6 instead of the current 2.4.1. version I have. Is there any easy way to do it? I have found something , but is it up-to-date and best way? https://www.kaggle.com/c/severstal-steel-defect-detection/discussion/113195

", "votes": null, "replies": [ { "id": 1530003, "author_name": "qitvision", "author_url": "", "post_date": "09/30/2021 20:56:53", "content": "

Hi, you can update the TF version from pip wheel that is packaged to a dataset. Here is an example notebook.

\n

This dataset has all packages required in tf-gpu 2.6.0 update.

", "votes": null, "replies": [] } ] }, { "id": 1529354, "author_name": "experienceinai", "author_url": "", "post_date": "09/30/2021 10:15:01", "content": "

FYI above for @cdeotte , comments are welcome!

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1502751": "Hi, \n\nI'm not very familiar with kaggle-environment and its restrictions. Hence; what is the best way to be able get e.g. efficientnet - models in spirit of \"Freely & publicly available external data is allowed, including pre-trained models\".\n\nI can do it of course by switching \"internet on\", but the rules say internet should be off to be able to make submissions. Thus - how to get efficientnet to the notebook without internet? \n\nThank you for kind answer for this \"too simple\" question.", "1502763": "Add it as a dataset to your notebook or any library for that matter. [efficientnet3D](https://www.kaggle.com/hihunjin/efficientnetpyttorch3d)\n\n`import sys\nsys.path.append(*pytorch3dpath*)\nfrom efficientnet_pytorch_3d import EfficientNet3D`", "1502853": "Hi, if you are using a regular (2D CNN) TF efficientnet, for instance, [this Kaggle dataset](https://www.kaggle.com/kaorusasakawa/tfkerasefficientnetimagenetnotop), has imagenet weights for all `tf.keras.applications.EfficientNetB*` models. \n\nOnce you add the dataset to your notebook, this would load the pretrained weights: `model = EfficientNetB0(weights='/kaggle/input/tfkerasefficientnetimagenetnotop/efficientnetb0_notop.h5', include_top=False)`", "1502870": "Thank you! So whatever weights can be used if found from the \"kaggle dataset\" section! That's clear πŸ‘πŸ‘", "1502917": "And you can create your own, public or private, datasets from e.g. previously trained weights. You can use any Kaggle dataset given it follows the competition rules.", "1503151": "you can separate training and submission into two notebooks, then switch the internet on in the training notebook. After training, add the training notebook as a dataset in your submission notebook to access your saved model/weight file and do the prediction.", "1504284": "You can create 1 notebook for training, and 1 for inference. You can do whatever you need in training notebook, such as accessing internet, run training on TPU and..., then if you get satisfied with the results, save your model `model.save(\"model.h5\")` , and load it on inference notebook with internet off and make prediction for test set. Than you can submit your inference notebook.", "1529348": "gdoong thank you for commenting interenet-on-off topic, it was useful. I have a new problem - I would like to use tensorflow version 2.6 instead of the current 2.4.1. version I have. Is there any easy way to do it? I have found something , but is it up-to-date and best way? https://www.kaggle.com/c/severstal-steel-defect-detection/discussion/113195", "1529354": "FYI above for @cdeotte , comments are welcome!", "1530003": "Hi, you can update the TF version from pip wheel that is packaged to a dataset. Here is an example [notebook](https://www.kaggle.com/qitvision/tensorflow-gpu-2-6-0-offline).\n\n[This](https://www.kaggle.com/qitvision/tensorflowgpu260) dataset has all packages required in tf-gpu 2.6.0 update." }, "source": "meta" }