I believe many of you are training their models in the cloud since this competition is computationally heavy. I am new to GCP and would like to ask you how do you load your LGBM models trained in GCP?
\nNormally on kaggle, I save the model in the training notebook:
\nmodel.save_model(f\"model.txt\")\njoblib.dump(model.best_iteration, f\"model_best_iteration.pkl.zip\")\nand load it in the inference notebook:
\nmodel = lgb.Booster(model_file=\"../input/riiid-lgbm/model.txt\")\nmodel.best_iteration = joblib.load(\"../input/riiid-lgbm/model_best_iteration.pkl.zip\")\nThis works correctly if the model is trained in a kaggle notebook. However, when I download the model from the AI notebook in the cloud and upload it to a kaggle dataset, I am unable to load it because the notebook restarts session whenever the 'loading model' cell is executed.
\nThe problem is that the models downloaded from the cloud are truncated! Any tips or ideas to overcome this problem? Did anyone have more success loading LGBM models trained in the cloud?
\nYou can find the solution in this discussion topic. You can upload your model directly from your notebook in GCP to your kaggle dataset using Kaggle API
\n#Downgrade your Kaggle API version, the latest version has a bug\npip install kaggle==1.5.4\n\n# Download your API token from your account and upload it to your notebook\n!mkdir ~/.kaggle\n!cp /home/kaggle.json ~/.kaggle/kaggle.json\n\n#Create a metadata file with this code and edit it with your dataset's name\n!kaggle datasets init -p /home/\n\n#After editing the metadata file, create a dataset\n!kaggle datasets create -p /home/\n\n#Put your kaggle username / dataset name as you edited the metadata file\n!kaggle datasets metadata -p . USERNAME/DATASET_NAME #(same as metafile)\n\n#Upload the model to your dataset\n!kaggle datasets version -p . -m \"Updates my model files\"\nI believe many of you are training their models in the cloud since this competition is computationally heavy. I am new to GCP and would like to ask you how do you load your LGBM models trained in GCP?
\nNormally on kaggle, I save the model in the training notebook:
\nmodel.save_model(f\"model.txt\")\njoblib.dump(model.best_iteration, f\"model_best_iteration.pkl.zip\")\nand load it in the inference notebook:
\nmodel = lgb.Booster(model_file=\"../input/riiid-lgbm/model.txt\")\nmodel.best_iteration = joblib.load(\"../input/riiid-lgbm/model_best_iteration.pkl.zip\")\nThis works correctly if the model is trained in a kaggle notebook. However, when I download the model from the AI notebook in the cloud and upload it to a kaggle dataset, I am unable to load it because the notebook restarts session whenever the 'loading model' cell is executed.
\nThe problem is that the models downloaded from the cloud are truncated! Any tips or ideas to overcome this problem? Did anyone have more success loading LGBM models trained in the cloud?
\nYou can find the solution in this discussion topic. You can upload your model directly from your notebook in GCP to your kaggle dataset using Kaggle API
\n#Downgrade your Kaggle API version, the latest version has a bug\npip install kaggle==1.5.4\n\n# Download your API token from your account and upload it to your notebook\n!mkdir ~/.kaggle\n!cp /home/kaggle.json ~/.kaggle/kaggle.json\n\n#Create a metadata file with this code and edit it with your dataset's name\n!kaggle datasets init -p /home/\n\n#After editing the metadata file, create a dataset\n!kaggle datasets create -p /home/\n\n#Put your kaggle username / dataset name as you edited the metadata file\n!kaggle datasets metadata -p . USERNAME/DATASET_NAME #(same as metafile)\n\n#Upload the model to your dataset\n!kaggle datasets version -p . -m \"Updates my model files\"\nI believe for this competition loading from GCP is impossible because the internet in the docker is disabled if you want to submit.
", "rawMarkdown": "I believe for this competition loading from GCP is impossible because the internet in the docker is disabled if you want to submit.", "votes": null }, { "id": "1127894", "postDate": "12/27/2020 02:09:46", "content": "I download the model, upload it to a kaggle dataset and load it from there, so no need to Internet connection. The problem is with the model itself, it can't be loaded for some reason.
", "rawMarkdown": "I download the model, upload it to a kaggle dataset and load it from there, so no need to Internet connection. The problem is with the model itself, it can't be loaded for some reason.", "votes": null }, { "id": "1127903", "postDate": "12/27/2020 02:30:36", "content": "this works for me -
\nfile = 'trained_model.pkl'\npickle.dump(model, open(file, 'wb'))\nand
\nmodel = pickle.load(open(file,'rb'))
Thanks you! I will try it right away
", "rawMarkdown": "Thanks you! I will try it right away", "votes": null }, { "id": "1127981", "postDate": "12/27/2020 04:42:53", "content": "Would someone be nice and donate for me so I can train my models in the cloud lol.
", "rawMarkdown": "Would someone be nice and donate for me so I can train my models in the cloud lol.", "votes": null }, { "id": "1128366", "postDate": "12/27/2020 11:31:00", "content": "If you are student/educator - I would say AWS is quite generous. Many cloud service providers have free tier options also.
", "rawMarkdown": "If you are student/educator - I would say AWS is quite generous. Many cloud service providers have free tier options also.", "votes": null }, { "id": "1128438", "postDate": "12/27/2020 12:47:37", "content": "I save and then load full model with joblib, not best iteration. I use pickle example from https://stackoverflow.com/questions/55208734/save-lgbmregressor-model-from-python-lightgbm-package-to-disc
\nIt works for me, score does not change
Its not solely an issue of money for me, in my country, Tunisia, paypal or any sort of online payment in dollars is yet unavailable.
", "rawMarkdown": "Its not solely an issue of money for me, in my country, Tunisia, paypal or any sort of online payment in dollars is yet unavailable.", "votes": null }, { "id": "1128766", "postDate": "12/27/2020 18:02:10", "content": "Thank you :)
", "rawMarkdown": "Thank you :)", "votes": null }, { "id": "1141147", "postDate": "01/06/2021 14:29:57", "content": "Hi @amiiiney ! Did you eventually solve this? I am having the same problem and none of the above suggestions seem to work for me. Thanks!
", "rawMarkdown": "Hi @amiiiney ! Did you eventually solve this? I am having the same problem and none of the above suggestions seem to work for me. Thanks!", "votes": null }, { "id": "1141176", "postDate": "01/06/2021 14:45:23", "content": "Use the kaggle api to create a dataset -> upload files to your dataset -> import them from your notebook.
\nAfter setting up the kaggle token, updating the database would only take one terminal command.
\nThis worked like charm for me.
@jpcpinto I was planning to update this post with the solution and got distracted with this final leaderboard race :)
\nAs Abdessalem mentioned, you can upload the model directly from your GCP notebook to a kaggle dataset, you can find the solution in this discussion post.
#Downgrade your Kaggle API version, the latest version has a bug\npip install kaggle==1.5.4\n\n# Download your API token from your account and upload it to your notebook\n!mkdir ~/.kaggle\n!cp /home/kaggle.json ~/.kaggle/kaggle.json\n\n#Create a metadata file with this code and edit it with your dataset's name\n!kaggle datasets init -p /home/\n\n#After editing the metafile, create a dataset\n!kaggle datasets create -p /home/\n\n#Put your kaggle username / dataset name as you edited the metafile\n!kaggle datasets metadata -p . USERNAME/DATASET_NAME #(same as metafile)\n\n#Upload the model to your dataset\n!kaggle datasets version -p . -m \"Updates my model files\"\nIf you get some unexpected bugs, let me know :)
", "rawMarkdown": "jpcpinto I was planning to update this post with the solution and got distracted with this final leaderboard race :) \nAs Abdessalem mentioned, you can upload the model directly from your GCP notebook to a kaggle dataset, you can find the solution in [this discussion post](https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/208110#1135728). \n```\n#Downgrade your Kaggle API version, the latest version has a bug\npip install kaggle==1.5.4\n\n# Download your API token from your account and upload it to your notebook\n!mkdir ~/.kaggle\n!cp /home/kaggle.json ~/.kaggle/kaggle.json\n\n#Create a metadata file with this code and edit it with your dataset's name\n!kaggle datasets init -p /home/\n\n#After editing the metafile, create a dataset\n!kaggle datasets create -p /home/\n\n#Put your kaggle username / dataset name as you edited the metafile\n!kaggle datasets metadata -p . USERNAME/DATASET_NAME #(same as metafile)\n\n#Upload the model to your dataset\n!kaggle datasets version -p . -m \"Updates my model files\"\n```\n\nIf you get some unexpected bugs, let me know :)", "votes": null }, { "id": "1141449", "postDate": "01/06/2021 18:07:55", "content": "Super awesome! It worked, thanks! Indeed there seems to be a problem with google cloud as it truncates files if they are \"manually\" downloaded
", "rawMarkdown": "Super awesome! It worked, thanks! Indeed there seems to be a problem with google cloud as it truncates files if they are \"manually\" downloaded", "votes": null }, { "id": "1556813", "postDate": "10/25/2021 06:56:56", "content": "Hi @amiiiney , i want to upload models trained in local service to kaggle dataset, , I try above process in anthor competition, but when i input kaggle datasets init -p /home/ in my local terminal, it always shows
\nValueError: Error: Missing username in configuration.
\nCould you know how to solve this problem? Thank you!
This problem has been solved. Thank you !
\n:)
I believe for this competition loading from GCP is impossible because the internet in the docker is disabled if you want to submit.
", "votes": null, "replies": [ { "id": 1127894, "author_name": "amiiiney", "author_url": "", "post_date": "12/27/2020 02:09:46", "content": "I download the model, upload it to a kaggle dataset and load it from there, so no need to Internet connection. The problem is with the model itself, it can't be loaded for some reason.
", "votes": null, "replies": [] } ] }, { "id": 1127903, "author_name": "rashmibanthia", "author_url": "", "post_date": "12/27/2020 02:30:36", "content": "this works for me -
\nfile = 'trained_model.pkl'\npickle.dump(model, open(file, 'wb'))\nand
\nmodel = pickle.load(open(file,'rb'))
Thanks you! I will try it right away
", "votes": null, "replies": [] } ] }, { "id": 1127981, "author_name": "abdessalemboukil", "author_url": "", "post_date": "12/27/2020 04:42:53", "content": "Would someone be nice and donate for me so I can train my models in the cloud lol.
", "votes": null, "replies": [ { "id": 1128366, "author_name": "rashmibanthia", "author_url": "", "post_date": "12/27/2020 11:31:00", "content": "If you are student/educator - I would say AWS is quite generous. Many cloud service providers have free tier options also.
", "votes": null, "replies": [] }, { "id": 1128465, "author_name": "abdessalemboukil", "author_url": "", "post_date": "12/27/2020 13:08:46", "content": "Its not solely an issue of money for me, in my country, Tunisia, paypal or any sort of online payment in dollars is yet unavailable.
", "votes": null, "replies": [] } ] }, { "id": 1128438, "author_name": "fredegrec", "author_url": "", "post_date": "12/27/2020 12:47:37", "content": "I save and then load full model with joblib, not best iteration. I use pickle example from https://stackoverflow.com/questions/55208734/save-lgbmregressor-model-from-python-lightgbm-package-to-disc
\nIt works for me, score does not change
Thank you :)
", "votes": null, "replies": [] } ] }, { "id": 1141147, "author_name": "jpcpinto", "author_url": "", "post_date": "01/06/2021 14:29:57", "content": "Hi @amiiiney ! Did you eventually solve this? I am having the same problem and none of the above suggestions seem to work for me. Thanks!
", "votes": null, "replies": [ { "id": 1141176, "author_name": "abdessalemboukil", "author_url": "", "post_date": "01/06/2021 14:45:23", "content": "Use the kaggle api to create a dataset -> upload files to your dataset -> import them from your notebook.
\nAfter setting up the kaggle token, updating the database would only take one terminal command.
\nThis worked like charm for me.
@jpcpinto I was planning to update this post with the solution and got distracted with this final leaderboard race :)
\nAs Abdessalem mentioned, you can upload the model directly from your GCP notebook to a kaggle dataset, you can find the solution in this discussion post.
#Downgrade your Kaggle API version, the latest version has a bug\npip install kaggle==1.5.4\n\n# Download your API token from your account and upload it to your notebook\n!mkdir ~/.kaggle\n!cp /home/kaggle.json ~/.kaggle/kaggle.json\n\n#Create a metadata file with this code and edit it with your dataset's name\n!kaggle datasets init -p /home/\n\n#After editing the metafile, create a dataset\n!kaggle datasets create -p /home/\n\n#Put your kaggle username / dataset name as you edited the metafile\n!kaggle datasets metadata -p . USERNAME/DATASET_NAME #(same as metafile)\n\n#Upload the model to your dataset\n!kaggle datasets version -p . -m \"Updates my model files\"\nIf you get some unexpected bugs, let me know :)
", "votes": null, "replies": [] }, { "id": 1141449, "author_name": "jpcpinto", "author_url": "", "post_date": "01/06/2021 18:07:55", "content": "Super awesome! It worked, thanks! Indeed there seems to be a problem with google cloud as it truncates files if they are \"manually\" downloaded
", "votes": null, "replies": [] } ] }, { "id": 1556813, "author_name": "feililan", "author_url": "", "post_date": "10/25/2021 06:56:56", "content": "Hi @amiiiney , i want to upload models trained in local service to kaggle dataset, , I try above process in anthor competition, but when i input kaggle datasets init -p /home/ in my local terminal, it always shows
\nValueError: Error: Missing username in configuration.
\nCould you know how to solve this problem? Thank you!
This problem has been solved. Thank you !
\n:)