the submission files always 500MB about, and the uploading is so slow and sometimes hang at a point, how do you deal with it？ Or what about your submission file size ? Or How long did you take to upload a submission?
", "messages": [ { "id": "316651", "postDate": "04/19/2018 15:19:33", "content": "the submission files always 500MB about, and the uploading is so slow and sometimes hang at a point, how do you deal with it？ Or what about your submission file size ? Or How long did you take to upload a submission?
", "rawMarkdown": "the submission files always 500MB about, and the uploading is so slow and sometimes hang at a point, how do you deal with it？ Or what about your submission file size ? Or How long did you take to upload a submission?", "votes": null }, { "id": "316662", "postDate": "04/19/2018 16:04:08", "content": "You can upload in a zip/gz/rar/7z archive.
", "rawMarkdown": "You can upload in a zip/gz/rar/7z archive.", "votes": null }, { "id": "316667", "postDate": "04/19/2018 16:17:00", "content": "I just got started in this competition so I haven't checked if others have already tried this to reduce the file size (in addition to compression):
\n\nsub.to_csv('solution.csv', float_format='%.3e', index=False)
It formats the solution into engineering notation. I tried %.3e and %.2e. %.3e was marginally better so you may use less precision for experiments and use more precision for your final submission.
This is how I've been doing it, thanks to Andy's kernel. You can just submit and score the zipped file in the normal way.
\n\nfinal_sub.to_csv(\"smaller.csv.gz\", index=False, float_format='%.8f'', compression='gzip')\nI'm catching up now. Andy's idea is ingenious. Thanks for pointing to it.
", "rawMarkdown": "I'm catching up now. Andy's idea is ingenious. Thanks for pointing to it.", "votes": null }, { "id": "316799", "postDate": "04/19/2018 23:57:55", "content": "Do you have comparative numbers on the compression that each of zip, gz, rar, 7z achieve on this competition?
", "rawMarkdown": "Do you have comparative numbers on the compression that each of zip, gz, rar, 7z achieve on this competition?", "votes": null }, { "id": "316801", "postDate": "04/20/2018 00:01:23", "content": "I had filed an enhance issue for the unofficial Kaggle CLI:\nsubmit should nag if submission file is not a zip/gz/rar/7z (because upload BW is so low), please vote and/or comment on it so it gets attention from Kaggle.
\n\nI had also commented in that:
\n\n> Also, as to bandwidth, it would be nice to report file size, upload\n> time and BW (Mbps) after upload. Then users could automatically figure\n> out what times of day (UTC-8) are fast/slow. Probably also useful to\n> report which proxy, if any, was used, as that may affect things.
", "rawMarkdown": "I had filed an enhance issue for the unofficial Kaggle CLI:\n[submit should nag if submission file is not a zip/gz/rar/7z (because upload BW is so low)](https://github.com/floydwch/kaggle-cli/issues/73), please vote and/or comment on it so it gets attention from Kaggle.\n\nI had also commented in that:\n\n> Also, as to bandwidth, it would be nice to report file size, upload\n> time and BW (Mbps) after upload. Then users could automatically figure\n> out what times of day (UTC-8) are fast/slow. Probably also useful to\n> report which proxy, if any, was used, as that may affect things.", "votes": null }, { "id": "316863", "postDate": "04/20/2018 05:03:53", "content": "I used only 7z till now. Sub file comes down to around 150MB. I tried Joe's comment gave better result than 7z around 125MB.
", "rawMarkdown": "I used only 7z till now. Sub file comes down to around 150MB. I tried Joe's comment gave better result than 7z around 125MB.", "votes": null }, { "id": "316895", "postDate": "04/20/2018 06:49:08", "content": "thanks very much !
", "rawMarkdown": "thanks very much !", "votes": null }, { "id": "316896", "postDate": "04/20/2018 06:49:32", "content": "thanks for your detailed info
", "rawMarkdown": "thanks for your detailed info", "votes": null }, { "id": "316897", "postDate": "04/20/2018 06:50:18", "content": "But I think this will hurt the auc score, maybe a lot...
", "rawMarkdown": "But I think this will hurt the auc score, maybe a lot...", "votes": null }, { "id": "316900", "postDate": "04/20/2018 06:59:45", "content": "I think the best solution is Andy's above. The engineering notation method has the advantage of preserving probabilities though. I think it should not impact AUC much because engineering notation preserves dynamic range. However, I haven't tested much. You may be right.
", "rawMarkdown": "I think the best solution is [**Andy's**](https://www.kaggle.com/aharless/smallification) above. The engineering notation method has the advantage of preserving probabilities though. I think it should not impact AUC much because engineering notation preserves dynamic range. However, I haven't tested much. You may be right.", "votes": null }, { "id": "316911", "postDate": "04/20/2018 07:11:20", "content": "In my experience, 7z works best on the same csv file.
", "rawMarkdown": "In my experience, 7z works best on the same csv file.", "votes": null }, { "id": "317663", "postDate": "04/22/2018 07:13:22", "content": "You can use my kernel for minimization + 7z, it gives about 81-86mb submission file on my data.
", "rawMarkdown": "You can use my [kernel][1] for minimization + 7z, it gives about 81-86mb submission file on my data.\n\n\n [1]: https://www.kaggle.com/alexfir/minimizing-submission", "votes": null }, { "id": "317741", "postDate": "04/22/2018 12:41:41", "content": "oh, great idea for auc, So In a more straight way, could we just use the 1~n to identify this order ?
", "rawMarkdown": "oh, great idea for auc, So In a more straight way, could we just use the 1~n to identify this order ?", "votes": null }, { "id": "317742", "postDate": "04/22/2018 12:43:04", "content": "7z
", "rawMarkdown": "7z", "votes": null }, { "id": "317749", "postDate": "04/22/2018 13:04:54", "content": "I do not think so, output of classified is interpreted as probability, so valid range should be [0 - 1]. \nMost likely you will get error on submitting for such is_attributed values.\nBut if you have range [1-n], just divide by n.
", "rawMarkdown": "I do not think so, output of classified is interpreted as probability, so valid range should be [0 - 1]. \nMost likely you will get error on submitting for such is_attributed values.\nBut if you have range [1-n], just divide by n.", "votes": null }, { "id": "317755", "postDate": "04/22/2018 13:23:25", "content": "Agree with you
", "rawMarkdown": "Agree with you", "votes": null } ], "comments": [ { "id": 316662, "author_name": "venkatesh8222", "author_url": "", "post_date": "04/19/2018 16:04:08", "content": "You can upload in a zip/gz/rar/7z archive.
", "votes": null, "replies": [ { "id": 316799, "author_name": "smcinerney", "author_url": "", "post_date": "04/19/2018 23:57:55", "content": "Do you have comparative numbers on the compression that each of zip, gz, rar, 7z achieve on this competition?
", "votes": null, "replies": [] }, { "id": 316863, "author_name": "venkatesh8222", "author_url": "", "post_date": "04/20/2018 05:03:53", "content": "I used only 7z till now. Sub file comes down to around 150MB. I tried Joe's comment gave better result than 7z around 125MB.
", "votes": null, "replies": [] }, { "id": 316896, "author_name": "qfzgs1994", "author_url": "", "post_date": "04/20/2018 06:49:32", "content": "thanks for your detailed info
", "votes": null, "replies": [] }, { "id": 316911, "author_name": "enfeizhan", "author_url": "", "post_date": "04/20/2018 07:11:20", "content": "In my experience, 7z works best on the same csv file.
", "votes": null, "replies": [] } ] }, { "id": 316667, "author_name": "pliptor", "author_url": "", "post_date": "04/19/2018 16:17:00", "content": "I just got started in this competition so I haven't checked if others have already tried this to reduce the file size (in addition to compression):
\n\nsub.to_csv('solution.csv', float_format='%.3e', index=False)
It formats the solution into engineering notation. I tried %.3e and %.2e. %.3e was marginally better so you may use less precision for experiments and use more precision for your final submission.
But I think this will hurt the auc score, maybe a lot...
", "votes": null, "replies": [] }, { "id": 316900, "author_name": "pliptor", "author_url": "", "post_date": "04/20/2018 06:59:45", "content": "I think the best solution is Andy's above. The engineering notation method has the advantage of preserving probabilities though. I think it should not impact AUC much because engineering notation preserves dynamic range. However, I haven't tested much. You may be right.
", "votes": null, "replies": [] } ] }, { "id": 316703, "author_name": "aquatic", "author_url": "", "post_date": "04/19/2018 17:45:00", "content": "This is how I've been doing it, thanks to Andy's kernel. You can just submit and score the zipped file in the normal way.
\n\nfinal_sub.to_csv(\"smaller.csv.gz\", index=False, float_format='%.8f'', compression='gzip')\nI'm catching up now. Andy's idea is ingenious. Thanks for pointing to it.
", "votes": null, "replies": [] }, { "id": 316895, "author_name": "qfzgs1994", "author_url": "", "post_date": "04/20/2018 06:49:08", "content": "thanks very much !
", "votes": null, "replies": [] } ] }, { "id": 316801, "author_name": "smcinerney", "author_url": "", "post_date": "04/20/2018 00:01:23", "content": "I had filed an enhance issue for the unofficial Kaggle CLI:\nsubmit should nag if submission file is not a zip/gz/rar/7z (because upload BW is so low), please vote and/or comment on it so it gets attention from Kaggle.
\n\nI had also commented in that:
\n\n> Also, as to bandwidth, it would be nice to report file size, upload\n> time and BW (Mbps) after upload. Then users could automatically figure\n> out what times of day (UTC-8) are fast/slow. Probably also useful to\n> report which proxy, if any, was used, as that may affect things.
", "votes": null, "replies": [] }, { "id": 317663, "author_name": "alexfir", "author_url": "", "post_date": "04/22/2018 07:13:22", "content": "You can use my kernel for minimization + 7z, it gives about 81-86mb submission file on my data.
", "votes": null, "replies": [ { "id": 317741, "author_name": "qfzgs1994", "author_url": "", "post_date": "04/22/2018 12:41:41", "content": "oh, great idea for auc, So In a more straight way, could we just use the 1~n to identify this order ?
", "votes": null, "replies": [] }, { "id": 317749, "author_name": "alexfir", "author_url": "", "post_date": "04/22/2018 13:04:54", "content": "I do not think so, output of classified is interpreted as probability, so valid range should be [0 - 1]. \nMost likely you will get error on submitting for such is_attributed values.\nBut if you have range [1-n], just divide by n.
", "votes": null, "replies": [] }, { "id": 317755, "author_name": "qfzgs1994", "author_url": "", "post_date": "04/22/2018 13:23:25", "content": "Agree with you
", "votes": null, "replies": [] } ] }, { "id": 317742, "author_name": "lizheng167", "author_url": "", "post_date": "04/22/2018 12:43:04", "content": "7z
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "316651": "the submission files always 500MB about, and the uploading is so slow and sometimes hang at a point, how do you deal with it？ Or what about your submission file size ? Or How long did you take to upload a submission?", "316662": "You can upload in a zip/gz/rar/7z archive.", "316667": "I just got started in this competition so I haven't checked if others have already tried this to reduce the file size (in addition to compression):\n\n`sub.to_csv('solution.csv', float_format='%.3e', index=False)`\n\nIt formats the solution into engineering notation. I tried `%.3e` and `%.2e`. `%.3e` was marginally better so you may use less precision for experiments and use more precision for your final submission.", "316703": "This is how I've been doing it, thanks to [Andy's kernel][1]. You can just submit and score the zipped file in the normal way. \n\n final_sub.to_csv(\"smaller.csv.gz\", index=False, float_format='%.8f'', compression='gzip')\n\n\n [1]: https://www.kaggle.com/aharless/smallification", "316730": "I'm catching up now. Andy's idea is ingenious. Thanks for pointing to it.", "316799": "Do you have comparative numbers on the compression that each of zip, gz, rar, 7z achieve on this competition?", "316801": "I had filed an enhance issue for the unofficial Kaggle CLI:\n[submit should nag if submission file is not a zip/gz/rar/7z (because upload BW is so low)](https://github.com/floydwch/kaggle-cli/issues/73), please vote and/or comment on it so it gets attention from Kaggle.\n\nI had also commented in that:\n\n> Also, as to bandwidth, it would be nice to report file size, upload\n> time and BW (Mbps) after upload. Then users could automatically figure\n> out what times of day (UTC-8) are fast/slow. Probably also useful to\n> report which proxy, if any, was used, as that may affect things.", "316863": "I used only 7z till now. Sub file comes down to around 150MB. I tried Joe's comment gave better result than 7z around 125MB.", "316895": "thanks very much !", "316896": "thanks for your detailed info", "316897": "But I think this will hurt the auc score, maybe a lot...", "316900": "I think the best solution is [**Andy's**](https://www.kaggle.com/aharless/smallification) above. The engineering notation method has the advantage of preserving probabilities though. I think it should not impact AUC much because engineering notation preserves dynamic range. However, I haven't tested much. You may be right.", "316911": "In my experience, 7z works best on the same csv file.", "317663": "You can use my [kernel][1] for minimization + 7z, it gives about 81-86mb submission file on my data.\n\n\n [1]: https://www.kaggle.com/alexfir/minimizing-submission", "317741": "oh, great idea for auc, So In a more straight way, could we just use the 1~n to identify this order ?", "317742": "7z", "317749": "I do not think so, output of classified is interpreted as probability, so valid range should be [0 - 1]. \nMost likely you will get error on submitting for such is_attributed values.\nBut if you have range [1-n], just divide by n.", "317755": "Agree with you" }, "source": "meta" }