{ "id": 553644, "title": "Loading pre-trained models to a notebook", "url": "/competitions/jane-street-real-time-market-data-forecasting/discussion/553644", "author_name": "", "post_date": "2024-12-27T13:37:46.057855100Z", "votes": 1, "comment_count": 7, "views": 0, "content": "

Hi everyone,
\nI am new to Kaggle competitions and I have a basic question. I have a model that is too heavy to be trained and because of the Kaggle kernel memory issues, I had to train the model locally and save it to a pickle file.
\nI want to know if, in the notebook to submit, we can just load the pre-trained model from the inputs or from a shared drive link?
\nPlease if you have any resources/examples for this use case, do not hesitate to share it with me.
\nThanks.

", "messages": [ { "id": "3081972", "postDate": "12/27/2024 13:37:46", "content": "

Hi everyone,
\nI am new to Kaggle competitions and I have a basic question. I have a model that is too heavy to be trained and because of the Kaggle kernel memory issues, I had to train the model locally and save it to a pickle file.
\nI want to know if, in the notebook to submit, we can just load the pre-trained model from the inputs or from a shared drive link?
\nPlease if you have any resources/examples for this use case, do not hesitate to share it with me.
\nThanks.

", "rawMarkdown": "Hi everyone,\nI am new to Kaggle competitions and I have a basic question. I have a model that is too heavy to be trained and because of the Kaggle kernel memory issues, I had to train the model locally and save it to a pickle file.\nI want to know if, in the notebook to submit, we can just load the pre-trained model from the inputs or from a shared drive link?\nPlease if you have any resources/examples for this use case, do not hesitate to share it with me.\nThanks.", "votes": null }, { "id": "3081984", "postDate": "12/27/2024 13:41:09", "content": "

No, because the submission must not have internet access.

\n

Save your model to a dataset and load from there.

", "rawMarkdown": "No, because the submission must not have internet access.\n\nSave your model to a dataset and load from there.", "votes": null }, { "id": "3081991", "postDate": "12/27/2024 13:47:14", "content": "

Thank you for your answer Fernando.
\nSo you mean I put it as a dataset in the inputs of my notebook, right? So basically the submission notebook will have three main parts: Data loading + Model loading + Data preparation (to adapt to the model we are loading) + Prediction function? Am I correct?

", "rawMarkdown": "Thank you for your answer Fernando.\nSo you mean I put it as a dataset in the inputs of my notebook, right? So basically the submission notebook will have three main parts: Data loading + Model loading + Data preparation (to adapt to the model we are loading) + Prediction function? Am I correct?", "votes": null }, { "id": "3081995", "postDate": "12/27/2024 13:49:42", "content": "

This is a preferred method in these competitions - just load the trained model as a joblib file and infer/ submit @ayoubchouikha

\n

You need to use a Kaggle model file/ dataset for this.

", "rawMarkdown": "This is a preferred method in these competitions - just load the trained model as a joblib file and infer/ submit @ayoubchouikha \n\nYou need to use a Kaggle model file/ dataset for this.", "votes": null }, { "id": "3081996", "postDate": "12/27/2024 13:50:51", "content": "

Yes you are right @ayoubchouikha
\nEven your feature engineering script can be loaded from the dataset/ utility script

", "rawMarkdown": "Yes you are right @ayoubchouikha \nEven your feature engineering script can be loaded from the dataset/ utility script", "votes": null }, { "id": "3082001", "postDate": "12/27/2024 14:00:48", "content": "

This is so helpful, thanks a lot Ravi!

", "rawMarkdown": "This is so helpful, thanks a lot Ravi!", "votes": null }, { "id": "3082002", "postDate": "12/27/2024 14:01:15", "content": "

Thank you again Ravi !

", "rawMarkdown": "Thank you again Ravi !", "votes": null }, { "id": "3082177", "postDate": "12/27/2024 18:59:54", "content": "

Most welcome and best wishes @ayoubchouikha

", "rawMarkdown": "Most welcome and best wishes @ayoubchouikha", "votes": null } ], "comments": [ { "id": 3081984, "author_name": "nandodmelo", "author_url": "", "post_date": "12/27/2024 13:41:09", "content": "

No, because the submission must not have internet access.

\n

Save your model to a dataset and load from there.

", "votes": null, "replies": [ { "id": 3081991, "author_name": "ayoubchouikha", "author_url": "", "post_date": "12/27/2024 13:47:14", "content": "

Thank you for your answer Fernando.
\nSo you mean I put it as a dataset in the inputs of my notebook, right? So basically the submission notebook will have three main parts: Data loading + Model loading + Data preparation (to adapt to the model we are loading) + Prediction function? Am I correct?

", "votes": null, "replies": [ { "id": 3081996, "author_name": "ravi20076", "author_url": "", "post_date": "12/27/2024 13:50:51", "content": "

Yes you are right @ayoubchouikha
\nEven your feature engineering script can be loaded from the dataset/ utility script

", "votes": null, "replies": [ { "id": 3082002, "author_name": "ayoubchouikha", "author_url": "", "post_date": "12/27/2024 14:01:15", "content": "

Thank you again Ravi !

", "votes": null, "replies": [ { "id": 3082177, "author_name": "ravi20076", "author_url": "", "post_date": "12/27/2024 18:59:54", "content": "

Most welcome and best wishes @ayoubchouikha

", "votes": null, "replies": [] } ] } ] } ] } ] }, { "id": 3081995, "author_name": "ravi20076", "author_url": "", "post_date": "12/27/2024 13:49:42", "content": "

This is a preferred method in these competitions - just load the trained model as a joblib file and infer/ submit @ayoubchouikha

\n

You need to use a Kaggle model file/ dataset for this.

", "votes": null, "replies": [ { "id": 3082001, "author_name": "ayoubchouikha", "author_url": "", "post_date": "12/27/2024 14:00:48", "content": "

This is so helpful, thanks a lot Ravi!

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "3081972": "Hi everyone,\nI am new to Kaggle competitions and I have a basic question. I have a model that is too heavy to be trained and because of the Kaggle kernel memory issues, I had to train the model locally and save it to a pickle file.\nI want to know if, in the notebook to submit, we can just load the pre-trained model from the inputs or from a shared drive link?\nPlease if you have any resources/examples for this use case, do not hesitate to share it with me.\nThanks.", "3081984": "No, because the submission must not have internet access.\n\nSave your model to a dataset and load from there.", "3081991": "Thank you for your answer Fernando.\nSo you mean I put it as a dataset in the inputs of my notebook, right? So basically the submission notebook will have three main parts: Data loading + Model loading + Data preparation (to adapt to the model we are loading) + Prediction function? Am I correct?", "3081995": "This is a preferred method in these competitions - just load the trained model as a joblib file and infer/ submit @ayoubchouikha \n\nYou need to use a Kaggle model file/ dataset for this.", "3081996": "Yes you are right @ayoubchouikha \nEven your feature engineering script can be loaded from the dataset/ utility script", "3082001": "This is so helpful, thanks a lot Ravi!", "3082002": "Thank you again Ravi !", "3082177": "Most welcome and best wishes @ayoubchouikha" }, "source": "meta" }