\nSolution: Compress the file using bz2. It compressed my 1.2+GB file to only 15MB file (amazing!). Then we can successfully download it. The code is as following:
\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1984447%2F31c6d0a6c7cf9668416c9f6676c7a58a%2Fgcpissue.jpg?generation=1609561873225491&alt=media\")
I uploaded a local lgbm model as my dataset and added it into my kernel. But the kernel automatically restarted when I tried to load this model. Everything goes well when I use kernel output model.
\nAnyone can help? Thanks!
\nUpdated. I found a stupid way to deal with downloading issue.
\nIssue: The size of file I want to download is 1.2+GB, but I can only download up to 200MB from GCP JupyterLab. So, it kills kernel at loading an incomplete file. So, let's reduce the file size.
\nSolution: Compress the file using bz2. It compressed my 1.2+GB file to only 15MB file (amazing!). Then we can successfully download it. The code is as following:
\n
It would be better if I can find a way to connect kaggle dataset with GCP JupyterLab output.
\n\n\nIt would be better if I can find a way to connect kaggle dataset with GCP JupyterLab output.
\n
Update a notebook
", "messages": [ { "id": 1135182, "postDate": "2021-01-02T00:57:04.990Z", "content": "I uploaded a local lgbm model as my dataset and added it into my kernel. But the kernel automatically restarted when I tried to load this model. Everything goes well when I use kernel output model.
\nAnyone can help? Thanks!
\nUpdated. I found a stupid way to deal with downloading issue.
\nIssue: The size of file I want to download is 1.2+GB, but I can only download up to 200MB from GCP JupyterLab. So, it kills kernel at loading an incomplete file. So, let's reduce the file size.
\nSolution: Compress the file using bz2. It compressed my 1.2+GB file to only 15MB file (amazing!). Then we can successfully download it. The code is as following:
\n
It would be better if I can find a way to connect kaggle dataset with GCP JupyterLab output.
\n\n\nIt would be better if I can find a way to connect kaggle dataset with GCP JupyterLab output.
\n
Update a notebook
", "rawMarkdown": "I uploaded a local lgbm model as my dataset and added it into my kernel. But the kernel automatically restarted when I tried to load this model. Everything goes well when I use kernel output model. \n\nAnyone can help? Thanks!\n\n--------------------------------------------------------------\nUpdated. I found a stupid way to deal with downloading issue.\n\n**Issue**: The size of file I want to download is 1.2+GB, but I can only download up to 200MB from GCP JupyterLab. So, it kills kernel at loading an incomplete file. So, let's reduce the file size.\n**Solution**: Compress the file using **bz2**. It compressed my 1.2+GB file to only 15MB file (amazing!). Then we can successfully download it. The code is as following:\n\n\nIt would be better if I can find a way to connect kaggle dataset with GCP JupyterLab output.\n\n--------------------------------------------------------------\n> It would be better if I can find a way to connect kaggle dataset with GCP JupyterLab output.\n\nUpdate a [notebook](https://www.kaggle.com/woshifym/upload-gcp-jupyterlab-data-to-kaggle-dataset?scriptVersionId=52904076)\n", "votes": 2 }, { "id": 1135208, "postDate": "2021-01-02T02:42:17.087Z", "content": "I have the exact same problem, apparently the model files are downloaded/uploaded incomplete and truncated. The problem here is not the way you save or load the model, I tried model.save_model(), pickle and joblib and the problem persists. Something wrong happens when you download the model to your local machine or upload the model to kaggle.
\nAre you using jupyterLab for training and you download the output from there?
I had this problem too, it was my connection apparently, during the download of the model, it failed midway and it got truncated.
", "rawMarkdown": "I had this problem too, it was my connection apparently, during the download of the model, it failed midway and it got truncated.", "votes": 1 }, { "id": 1135213, "postDate": "2021-01-02T02:55:36.900Z", "content": "@abdessalemboukil How did you solve the problem? By changing your internet connection? :)
", "rawMarkdown": "@abdessalemboukil How did you solve the problem? By changing your internet connection? :)" }, { "id": 1135216, "postDate": "2021-01-02T03:12:18.097Z", "content": "I used GCP AI platform. Cannot download a complete model. How can I fix it?
", "rawMarkdown": "I used GCP AI platform. Cannot download a complete model. How can I fix it?" }, { "id": 1135217, "postDate": "2021-01-02T03:20:10.337Z", "content": "Same, I am using GCP. In the AI notebook troubleshooting section they are mentioning some failed download problems, not the same as our case, but they suggest downgrading your notebook package:
\nsudo pip3 install notebook==5.7.5\nsudo service jupyter restart\nI haven't tried it yet, if it works for you, please let me know.
", "rawMarkdown": "Same, I am using GCP. In the [AI notebook troubleshooting section](https://cloud.google.com/ai-platform/notebooks/docs/troubleshooting) they are mentioning some failed download problems, not the same as our case, but they suggest downgrading your notebook package:\n```\nsudo pip3 install notebook==5.7.5\nsudo service jupyter restart\n```\nI haven't tried it yet, if it works for you, please let me know." }, { "id": 1135223, "postDate": "2021-01-02T03:32:47.320Z", "content": "For now, I can only download up to at most 200MB file from GCP notebook, but the size of my model is 1.2GB. It is really stupid. Do you know how to move the file to google cloud or bucket using notebook?
", "rawMarkdown": "For now, I can only download up to at most 200MB file from GCP notebook, but the size of my model is 1.2GB. It is really stupid. Do you know how to move the file to google cloud or bucket using notebook?" }, { "id": 1135234, "postDate": "2021-01-02T03:54:08.060Z", "content": "I am uploading model file in GCP VM instance using kaggle datasets command. This is the fastest :)
", "rawMarkdown": "I am uploading model file in GCP VM instance using kaggle datasets command. This is the fastest :)\n", "votes": 1 }, { "id": 1135237, "postDate": "2021-01-02T03:56:28.603Z", "content": "Assume you have already dataset created.
\nkaggle datasets metadata -p . higepon/your_dataset_namekaggle datasets version -p . -m \"Updates my model files\"@higepon do you store your files in a GS bucket or you run the command and upload directly from the notebook?
", "rawMarkdown": "@higepon do you store your files in a GS bucket or you run the command and upload directly from the notebook?" }, { "id": 1135541, "postDate": "2021-01-02T10:31:14.080Z", "content": "I did it from my notebook.
", "rawMarkdown": "I did it from my notebook." }, { "id": 1135648, "postDate": "2021-01-02T11:58:32.180Z", "content": "Thanks @higepon! I am getting an error while trying to add the metadata file to the created dataset
\n !kaggle datasets metadata -p . amiiiney/riid-model. Any idea how to solve it?
Sure. Could I have error details here? thanks!
", "rawMarkdown": "Sure. Could I have error details here? thanks!", "votes": 1 }, { "id": 1135691, "postDate": "2021-01-02T12:40:40.940Z", "content": "Please note that you have to create your API credential and store it in GCP.
\nhttps://github.com/Kaggle/kaggle-api#api-credentials
I am using my API token that I download from my account
\n!mkdir ~/.kaggle\n!cp /home/kaggle.json ~/.kaggle/kaggle.json\nAnd then I create the metadata file and edit it with my dataset's name
\n!kaggle datasets init -p /home/\n!kaggle datasets create -p /home/\nAnd I get this error when I add the metadata
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3451735%2Fe6f77bc458db25d58c2da3c7c616af90%2FScreen%20Shot%202021-01-02%20at%2013.42.53.png?generation=1609591587042259&alt=media\")
You may want to install another version of kaggle API. See issue here.
\nFIY I tried on my GCP as follows and it worked as expected.
\n-![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2199749%2F95757ed4199de862f36b2ff7e2d86ed8%2F2021-01-02%2022.02.18.png?generation=1609592567487223&alt=media\")
\n-![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2199749%2Facb304afc751880c5339600a86701ae2%2F2021-01-02%2022.02.24.png?generation=1609592585507290&alt=media\")
Thanks a lot @higepon :) It finally worked with kaggle version 1.5.4!
", "rawMarkdown": "Thanks a lot @higepon :) It finally worked with kaggle version 1.5.4!", "votes": 1 } ] }, { "id": 1135227, "postDate": "2021-01-02T03:42:19.123Z", "content": "I suggest trying to load a model locally first see if it works.
", "rawMarkdown": "I suggest trying to load a model locally first see if it works.", "replies": [ { "id": 1135232, "postDate": "2021-01-02T03:48:25.763Z", "content": "Yes. I have found the problem that I cannot download a complete file from GCP JupyterLab. So, I was loading the incomplete file, which was killing the kernel. Have totally no idea how to fix it.
", "rawMarkdown": "Yes. I have found the problem that I cannot download a complete file from GCP JupyterLab. So, I was loading the incomplete file, which was killing the kernel. Have totally no idea how to fix it." } ] }, { "id": 1135184, "postDate": "2021-01-02T01:02:22.870Z", "content": "How do you save and load model?
", "rawMarkdown": "How do you save and load model?", "replies": [ { "id": 1135186, "postDate": "2021-01-02T01:04:39.647Z", "content": "I use joblib and it works for me
\nimport joblib
\njoblib.dump(my_model, 'lgb.pkl')
\nmodel = joblib.load('lgb.pkl')
Thanks @fredegrec
\nI used model.save_model() and lgb.Booster(model_file='…'). I will try your method later. I checked my model.txt,and it seems the file is not completed. Maybe there was something wrong during saving the model.
Can I directly use model.predict after model = joblib.load('lgb.pkl') in your method?
I'm using model.model_to_string() to get model as string and save it with joblib.dump.
\nThen upload the .joblib file to my private dataset, load the .joblib using joblib.loadand lgb.Booster(model_str=model_str) to load the model for submission.
Hope it helps.
", "rawMarkdown": "I'm using ```model.model_to_string()``` to get model as string and save it with ```joblib.dump```.\nThen upload the .joblib file to my private dataset, load the .joblib using ```joblib.load```and ```lgb.Booster(model_str=model_str)``` to load the model for submission.\n\nHope it helps.", "votes": 2 }, { "id": 1135201, "postDate": "2021-01-02T01:48:33.880Z", "content": "@woshifym Yes, you can
", "rawMarkdown": "@woshifym Yes, you can" }, { "id": 1135202, "postDate": "2021-01-02T02:03:39.143Z", "content": "Will try, thanks.
", "rawMarkdown": "Will try, thanks." } ] }, { "id": 1135252, "postDate": "2021-01-02T04:38:17.460Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 1135208, "author_name": "Amin", "author_url": "", "post_date": "2021-01-02T02:42:17.087000", "content": "I have the exact same problem, apparently the model files are downloaded/uploaded incomplete and truncated. The problem here is not the way you save or load the model, I tried model.save_model(), pickle and joblib and the problem persists. Something wrong happens when you download the model to your local machine or upload the model to kaggle.
\nAre you using jupyterLab for training and you download the output from there?
I had this problem too, it was my connection apparently, during the download of the model, it failed midway and it got truncated.
", "votes": 1, "replies": [] }, { "id": 1135213, "author_name": "Amin", "author_url": "", "post_date": "2021-01-02T02:55:36.900000", "content": "@abdessalemboukil How did you solve the problem? By changing your internet connection? :)
", "votes": 0, "replies": [] }, { "id": 1135216, "author_name": "LeoF", "author_url": "", "post_date": "2021-01-02T03:12:18.097000", "content": "I used GCP AI platform. Cannot download a complete model. How can I fix it?
", "votes": 0, "replies": [] }, { "id": 1135217, "author_name": "Amin", "author_url": "", "post_date": "2021-01-02T03:20:10.337000", "content": "Same, I am using GCP. In the AI notebook troubleshooting section they are mentioning some failed download problems, not the same as our case, but they suggest downgrading your notebook package:
\nsudo pip3 install notebook==5.7.5\nsudo service jupyter restart\nI haven't tried it yet, if it works for you, please let me know.
", "votes": 0, "replies": [] }, { "id": 1135223, "author_name": "LeoF", "author_url": "", "post_date": "2021-01-02T03:32:47.320000", "content": "For now, I can only download up to at most 200MB file from GCP notebook, but the size of my model is 1.2GB. It is really stupid. Do you know how to move the file to google cloud or bucket using notebook?
", "votes": 0, "replies": [] }, { "id": 1135234, "author_name": "higepon", "author_url": "", "post_date": "2021-01-02T03:54:08.060000", "content": "I am uploading model file in GCP VM instance using kaggle datasets command. This is the fastest :)
", "votes": 1, "replies": [] }, { "id": 1135237, "author_name": "higepon", "author_url": "", "post_date": "2021-01-02T03:56:28.603000", "content": "Assume you have already dataset created.
\nkaggle datasets metadata -p . higepon/your_dataset_namekaggle datasets version -p . -m \"Updates my model files\"@higepon do you store your files in a GS bucket or you run the command and upload directly from the notebook?
", "votes": 0, "replies": [] }, { "id": 1135541, "author_name": "higepon", "author_url": "", "post_date": "2021-01-02T10:31:14.080000", "content": "I did it from my notebook.
", "votes": 0, "replies": [] }, { "id": 1135648, "author_name": "Amin", "author_url": "", "post_date": "2021-01-02T11:58:32.180000", "content": "Thanks @higepon! I am getting an error while trying to add the metadata file to the created dataset
\n !kaggle datasets metadata -p . amiiiney/riid-model. Any idea how to solve it?
Sure. Could I have error details here? thanks!
", "votes": 1, "replies": [] }, { "id": 1135691, "author_name": "higepon", "author_url": "", "post_date": "2021-01-02T12:40:40.940000", "content": "Please note that you have to create your API credential and store it in GCP.
\nhttps://github.com/Kaggle/kaggle-api#api-credentials
I am using my API token that I download from my account
\n!mkdir ~/.kaggle\n!cp /home/kaggle.json ~/.kaggle/kaggle.json\nAnd then I create the metadata file and edit it with my dataset's name
\n!kaggle datasets init -p /home/\n!kaggle datasets create -p /home/\nAnd I get this error when I add the metadata
\n
You may want to install another version of kaggle API. See issue here.
\nFIY I tried on my GCP as follows and it worked as expected.
\n-
\n-
Thanks a lot @higepon :) It finally worked with kaggle version 1.5.4!
", "votes": 1, "replies": [] } ] }, { "id": 1135227, "author_name": "Shuhao Cao", "author_url": "", "post_date": "2021-01-02T03:42:19.123000", "content": "I suggest trying to load a model locally first see if it works.
", "votes": 0, "replies": [ { "id": 1135232, "author_name": "LeoF", "author_url": "", "post_date": "2021-01-02T03:48:25.763000", "content": "Yes. I have found the problem that I cannot download a complete file from GCP JupyterLab. So, I was loading the incomplete file, which was killing the kernel. Have totally no idea how to fix it.
", "votes": 0, "replies": [] } ] }, { "id": 1135184, "author_name": "Alyona Pasevieva", "author_url": "", "post_date": "2021-01-02T01:02:22.870000", "content": "How do you save and load model?
", "votes": 0, "replies": [ { "id": 1135186, "author_name": "Alyona Pasevieva", "author_url": "", "post_date": "2021-01-02T01:04:39.647000", "content": "I use joblib and it works for me
\nimport joblib
\njoblib.dump(my_model, 'lgb.pkl')
\nmodel = joblib.load('lgb.pkl')
Thanks @fredegrec
\nI used model.save_model() and lgb.Booster(model_file='…'). I will try your method later. I checked my model.txt,and it seems the file is not completed. Maybe there was something wrong during saving the model.
Can I directly use model.predict after model = joblib.load('lgb.pkl') in your method?
I'm using model.model_to_string() to get model as string and save it with joblib.dump.
\nThen upload the .joblib file to my private dataset, load the .joblib using joblib.loadand lgb.Booster(model_str=model_str) to load the model for submission.
Hope it helps.
", "votes": 2, "replies": [] }, { "id": 1135201, "author_name": "Alyona Pasevieva", "author_url": "", "post_date": "2021-01-02T01:48:33.880000", "content": "@woshifym Yes, you can
", "votes": 0, "replies": [] }, { "id": 1135202, "author_name": "LeoF", "author_url": "", "post_date": "2021-01-02T02:03:39.143000", "content": "Will try, thanks.
", "votes": 0, "replies": [] } ] }, { "id": 1135252, "author_name": "", "author_url": "", "post_date": "2021-01-02T04:38:17.460000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1135182": "I uploaded a local lgbm model as my dataset and added it into my kernel. But the kernel automatically restarted when I tried to load this model. Everything goes well when I use kernel output model. \n\nAnyone can help? Thanks!\n\n--------------------------------------------------------------\nUpdated. I found a stupid way to deal with downloading issue.\n\n**Issue**: The size of file I want to download is 1.2+GB, but I can only download up to 200MB from GCP JupyterLab. So, it kills kernel at loading an incomplete file. So, let's reduce the file size.\n**Solution**: Compress the file using **bz2**. It compressed my 1.2+GB file to only 15MB file (amazing!). Then we can successfully download it. The code is as following:\n\n\nIt would be better if I can find a way to connect kaggle dataset with GCP JupyterLab output.\n\n--------------------------------------------------------------\n> It would be better if I can find a way to connect kaggle dataset with GCP JupyterLab output.\n\nUpdate a [notebook](https://www.kaggle.com/woshifym/upload-gcp-jupyterlab-data-to-kaggle-dataset?scriptVersionId=52904076)\n", "1135208": "I have the exact same problem, apparently the model files are downloaded/uploaded incomplete and truncated. The problem here is not the way you save or load the model, I tried model.save_model(), pickle and joblib and the problem persists. Something wrong happens when you download the model to your local machine or upload the model to kaggle.\nAre you using jupyterLab for training and you download the output from there?", "1135227": "I suggest trying to load a model locally first see if it works.", "1135184": "How do you save and load model?", "1135252": "" } }