Hey guys,
\nUnless I am missing something, anything after the 8th decimal point is useless, don't do it :).
We all can save dear bits here and there that could be used for \"better\" stuff.
\nAnd by \"better\" you know what I mean.
Hey guys,
\nUnless I am missing something, anything after the 8th decimal point is useless, don't do it :).
We all can save dear bits here and there that could be used for \"better\" stuff.
\nAnd by \"better\" you know what I mean.
For pandas users reducing bandwidth is as easy as ...
\n\npandasdataframe = pd.read_csv('abc.csv')\npandasdataframe.to_csv('abc.csv.gz', index=False,compression='gzip')\nThanks for the code !
", "rawMarkdown": "Thanks for the code !", "votes": 2 }, { "id": 304792, "postDate": "2018-03-28T02:46:03.897Z", "content": "Is that means we can upload compressed file as submission？
", "rawMarkdown": "Is that means we can upload compressed file as submission？", "votes": 1 }, { "id": 306043, "postDate": "2018-03-29T19:30:00.847Z", "content": "Yes, you can.
", "rawMarkdown": "Yes, you can.", "votes": 1 } ] }, { "id": 301942, "postDate": "2018-03-23T13:55:13.097Z", "content": "Guys you could also use zip or 7zip formats. It also helps reduce the bandwith and upload time ;-)
\n\nUpdate : 320MB submission file becomes 64MB in zip format and 40MB with 7z \ncompression takes about 2 minutes on my PC for 7z
", "rawMarkdown": "Guys you could also use zip or 7zip formats. It also helps reduce the bandwith and upload time ;-)\n\nUpdate : 320MB submission file becomes 64MB in zip format and 40MB with 7z \ncompression takes about 2 minutes on my PC for 7z", "votes": 9, "replies": [ { "id": 302293, "postDate": "2018-03-23T21:41:47.610Z", "content": "Is 7zip better than gzip?
", "rawMarkdown": "Is 7zip better than gzip?", "votes": 1 }, { "id": 302452, "postDate": "2018-03-24T04:34:16.967Z", "content": "Yes.
", "rawMarkdown": "Yes. ", "votes": 1 } ] }, { "id": 302547, "postDate": "2018-03-24T09:33:32.920Z", "content": "library(R.utils)\nlibrary(data.table)\n\n#Write gzip file\ndf <- data.table(var1='Compress me',var2=', please!')\nfwrite(df,'filename.csv',sep=',')\ngzip('filename.csv',destname='filename.csv.gz')`\n\n#Read gzip file\nfread('gzip -dc filename.csv.gz')\nNicked from here
", "rawMarkdown": " library(R.utils)\n library(data.table)\n\n #Write gzip file\n df <- data.table(var1='Compress me',var2=', please!')\n fwrite(df,'filename.csv',sep=',')\n gzip('filename.csv',destname='filename.csv.gz')`\n\n #Read gzip file\n fread('gzip -dc filename.csv.gz')\n\nNicked from [here][1]\n\n\n [1]: https://stackoverflow.com/questions/42788401/is-possible-to-use-fwrite-from-data-table-with-gzfile", "votes": 8 }, { "id": 302317, "postDate": "2018-03-23T22:38:23.890Z", "content": "I made a very short kernel to convert a submission file to an equivalent gzipped ranked version. In the example I used, it saves about 70% of the space. In general, there could be significant digits after the 8th decimal place in the input file, but once the results are expressed in rank form, there is no need for more than 8 digits. I figure, when I remember to do so, I will cut and paste the code from it into anything new I write that creates a submission file.
", "rawMarkdown": "I made [a very short kernel][1] to convert a submission file to an equivalent gzipped ranked version. In the example I used, it saves about 70% of the space. In general, there could be significant digits after the 8th decimal place in the input file, but once the results are expressed in rank form, there is no need for more than 8 digits. I figure, when I remember to do so, I will cut and paste the code from it into anything new I write that creates a submission file.\n\n [1]: https://www.kaggle.com/aharless/smallification", "votes": 3, "replies": [ { "id": 316737, "postDate": "2018-04-19T19:01:26.817Z", "content": "Nice idea. I think it might be compressed even further by sorting by rank prior to compression.
", "rawMarkdown": "Nice idea. I think it might be compressed even further by sorting by rank prior to compression. ", "votes": 1 } ] }, { "id": 301491, "postDate": "2018-03-22T20:22:03.777Z", "content": "Only the sort order of the predictions matters, so how about an alternate submission file format where only the click_id is included, in the order of our predictions? Saves the scorer having to sort, and hopefully confuses the blenders for a while :)
", "rawMarkdown": "Only the sort order of the predictions matters, so how about an alternate submission file format where only the click_id is included, in the order of our predictions? Saves the scorer having to sort, and hopefully confuses the blenders for a while :)", "votes": 4, "replies": [ { "id": 301519, "postDate": "2018-03-22T21:32:46.850Z", "rawMarkdown": "", "votes": 1, "isDeleted": true }, { "id": 301548, "postDate": "2018-03-22T22:38:37.320Z", "content": "The competition evaluation metric is AUC, short for Area Under the ROC curve.
\n\nEssentially, the aim is to order the click_id's by the probability that the is_attributed target is true. A perfect score of 1 would result if your ordering of test set click_id's listed all the is_attributed==0 first, and all the is_attributed==1 afterwards.
\n\nThere's a fuller explanation here, with a faster Python implementation of AUC.
", "rawMarkdown": "The competition [evaluation metric][1] is AUC, short for Area Under the [ROC][2] curve.\n\nEssentially, the aim is to order the click_id's by the probability that the is_attributed target is true. A perfect score of 1 would result if your ordering of test set click_id's listed all the is_attributed==0 first, and all the is_attributed==1 afterwards.\n\nThere's a fuller [explanation here][3], with a faster Python implementation of AUC.\n\n [1]: https://www.kaggle.com/c/talkingdata-adtracking-fraud-detection#evaluation\n [2]: https://en.wikipedia.org/wiki/Receiver_operating_characteristic\n [3]: https://www.ibm.com/developerworks/community/blogs/jfp/entry/Fast_Computation_of_AUC_ROC_score?lang=en\n", "votes": 2 }, { "id": 301554, "postDate": "2018-03-22T22:52:11.893Z", "rawMarkdown": "", "isDeleted": true }, { "id": 301599, "postDate": "2018-03-23T01:39:51.420Z", "content": "Great idea. It took me 2 hours to upload my first sub.
\n\nSure, making twice the same mistake in the file did not help, but still, upload is way too long.
", "rawMarkdown": "Great idea. It took me 2 hours to upload my first sub.\n\nSure, making twice the same mistake in the file did not help, but still, upload is way too long." }, { "id": 301635, "postDate": "2018-03-23T02:44:34.663Z", "content": "And to think: the original test set was about three times the size! That would've meant ~1Gb CSV submissions and a six hour wait...
\n\nThinking about it some more, it would be very easy to support either a one column or two column submission format. If the submission uploaded has only one column the implicit 'is_attributed' could be added.
\n\ni.e. assuming sub is single column sorted click_id
\n\nif 'is_attributed' not in sub.columns:\n sub['is_attributed'] = np.arange(sub.shape[0])\n... which could be sent to the existing scorer. (It'd be more efficient to use a different scoring routine that no longer has to do a sort.) Either format would work so it does not complicate things (for users) at all.
", "rawMarkdown": "And to think: the original test set was about three times the size! That would've meant ~1Gb CSV submissions and a six hour wait...\n\nThinking about it some more, it would be very easy to support either a one column or two column submission format. If the submission uploaded has only one column the implicit 'is_attributed' could be added.\n\ni.e. assuming sub is single column sorted click_id\n\n if 'is_attributed' not in sub.columns:\n sub['is_attributed'] = np.arange(sub.shape[0])\n\n... which could be sent to the existing scorer. (It'd be more efficient to use a different scoring routine that no longer has to do a sort.) Either format would work so it does not complicate things (for users) at all.\n" } ] }, { "id": 316839, "postDate": "2018-04-20T03:50:09.647Z", "content": "I'm using pd.hdf. \ndata.to_hdf('data.h5',key='1')\nIt saves much time.
", "rawMarkdown": "I'm using pd.hdf. \ndata.to_hdf('data.h5',key='1')\nIt saves much time.", "votes": 1, "replies": [ { "id": 316844, "postDate": "2018-04-20T04:02:14.793Z", "content": "It looks like the first letter in key should be alpha but not numeric.
", "rawMarkdown": "It looks like the first letter in key should be alpha but not numeric." }, { "id": 316854, "postDate": "2018-04-20T04:47:52.843Z", "content": "Numeric is ok.I have done it in my code.
", "rawMarkdown": "Numeric is ok.I have done it in my code." } ] }, { "id": 301663, "postDate": "2018-03-23T04:06:10.803Z", "content": "It Took me 2 hours to understand what you mean by 8 digit.
", "rawMarkdown": "It Took me 2 hours to understand what you mean by 8 digit.", "votes": 1 }, { "id": 302154, "postDate": "2018-03-23T18:35:34.980Z", "content": "I have same question
", "rawMarkdown": "I have same question" }, { "id": 301622, "postDate": "2018-03-23T02:12:47.540Z", "content": "Has anyone noticed that the click_id is not in a monotonically increasing order? For example. 9 is before 8. Do we need to sort the submission based on the click_id?
", "rawMarkdown": "Has anyone noticed that the click_id is not in a monotonically increasing order? For example. 9 is before 8. Do we need to sort the submission based on the click_id?", "replies": [ { "id": 301770, "postDate": "2018-03-23T08:05:08.633Z", "content": "I noticed this too. But it seems that Kaggle takes care of sorting by click_id. I tried to submit with different orderings and got the same score both times. So don't worry about it.
For pandas users reducing bandwidth is as easy as ...
\n\npandasdataframe = pd.read_csv('abc.csv')\npandasdataframe.to_csv('abc.csv.gz', index=False,compression='gzip')\nThanks for the code !
", "votes": 2, "replies": [] }, { "id": 304792, "author_name": "Paul Yang", "author_url": "", "post_date": "2018-03-28T02:46:03.897000", "content": "Is that means we can upload compressed file as submission？
", "votes": 1, "replies": [] }, { "id": 306043, "author_name": "Luis Moneda", "author_url": "", "post_date": "2018-03-29T19:30:00.847000", "content": "Yes, you can.
", "votes": 1, "replies": [] } ] }, { "id": 301942, "author_name": "olivier", "author_url": "", "post_date": "2018-03-23T13:55:13.097000", "content": "Guys you could also use zip or 7zip formats. It also helps reduce the bandwith and upload time ;-)
\n\nUpdate : 320MB submission file becomes 64MB in zip format and 40MB with 7z \ncompression takes about 2 minutes on my PC for 7z
", "votes": 9, "replies": [ { "id": 302293, "author_name": "Andy Harless", "author_url": "", "post_date": "2018-03-23T21:41:47.610000", "content": "Is 7zip better than gzip?
", "votes": 1, "replies": [] }, { "id": 302452, "author_name": "CPMP", "author_url": "", "post_date": "2018-03-24T04:34:16.967000", "content": "Yes.
", "votes": 1, "replies": [] } ] }, { "id": 302547, "author_name": "Scirpus", "author_url": "", "post_date": "2018-03-24T09:33:32.920000", "content": "library(R.utils)\nlibrary(data.table)\n\n#Write gzip file\ndf <- data.table(var1='Compress me',var2=', please!')\nfwrite(df,'filename.csv',sep=',')\ngzip('filename.csv',destname='filename.csv.gz')`\n\n#Read gzip file\nfread('gzip -dc filename.csv.gz')\nNicked from here
", "votes": 8, "replies": [] }, { "id": 302317, "author_name": "Andy Harless", "author_url": "", "post_date": "2018-03-23T22:38:23.890000", "content": "I made a very short kernel to convert a submission file to an equivalent gzipped ranked version. In the example I used, it saves about 70% of the space. In general, there could be significant digits after the 8th decimal place in the input file, but once the results are expressed in rank form, there is no need for more than 8 digits. I figure, when I remember to do so, I will cut and paste the code from it into anything new I write that creates a submission file.
", "votes": 3, "replies": [ { "id": 316737, "author_name": "Oscar Takeshita", "author_url": "", "post_date": "2018-04-19T19:01:26.817000", "content": "Nice idea. I think it might be compressed even further by sorting by rank prior to compression.
", "votes": 1, "replies": [] } ] }, { "id": 301491, "author_name": "James Trotman", "author_url": "", "post_date": "2018-03-22T20:22:03.777000", "content": "Only the sort order of the predictions matters, so how about an alternate submission file format where only the click_id is included, in the order of our predictions? Saves the scorer having to sort, and hopefully confuses the blenders for a while :)
", "votes": 4, "replies": [ { "id": 301519, "author_name": "", "author_url": "", "post_date": "2018-03-22T21:32:46.850000", "content": "", "votes": 1, "replies": [] }, { "id": 301548, "author_name": "James Trotman", "author_url": "", "post_date": "2018-03-22T22:38:37.320000", "content": "The competition evaluation metric is AUC, short for Area Under the ROC curve.
\n\nEssentially, the aim is to order the click_id's by the probability that the is_attributed target is true. A perfect score of 1 would result if your ordering of test set click_id's listed all the is_attributed==0 first, and all the is_attributed==1 afterwards.
\n\nThere's a fuller explanation here, with a faster Python implementation of AUC.
", "votes": 2, "replies": [] }, { "id": 301554, "author_name": "", "author_url": "", "post_date": "2018-03-22T22:52:11.893000", "content": "", "votes": 0, "replies": [] }, { "id": 301599, "author_name": "CPMP", "author_url": "", "post_date": "2018-03-23T01:39:51.420000", "content": "Great idea. It took me 2 hours to upload my first sub.
\n\nSure, making twice the same mistake in the file did not help, but still, upload is way too long.
", "votes": 0, "replies": [] }, { "id": 301635, "author_name": "James Trotman", "author_url": "", "post_date": "2018-03-23T02:44:34.663000", "content": "And to think: the original test set was about three times the size! That would've meant ~1Gb CSV submissions and a six hour wait...
\n\nThinking about it some more, it would be very easy to support either a one column or two column submission format. If the submission uploaded has only one column the implicit 'is_attributed' could be added.
\n\ni.e. assuming sub is single column sorted click_id
\n\nif 'is_attributed' not in sub.columns:\n sub['is_attributed'] = np.arange(sub.shape[0])\n... which could be sent to the existing scorer. (It'd be more efficient to use a different scoring routine that no longer has to do a sort.) Either format would work so it does not complicate things (for users) at all.
", "votes": 0, "replies": [] } ] }, { "id": 316839, "author_name": "shark", "author_url": "", "post_date": "2018-04-20T03:50:09.647000", "content": "I'm using pd.hdf. \ndata.to_hdf('data.h5',key='1')\nIt saves much time.
", "votes": 1, "replies": [ { "id": 316844, "author_name": "Oscar Takeshita", "author_url": "", "post_date": "2018-04-20T04:02:14.793000", "content": "It looks like the first letter in key should be alpha but not numeric.
", "votes": 0, "replies": [] }, { "id": 316854, "author_name": "shark", "author_url": "", "post_date": "2018-04-20T04:47:52.843000", "content": "Numeric is ok.I have done it in my code.
", "votes": 0, "replies": [] } ] }, { "id": 301663, "author_name": "Muhammad Alfiansyah", "author_url": "", "post_date": "2018-03-23T04:06:10.803000", "content": "It Took me 2 hours to understand what you mean by 8 digit.
", "votes": 1, "replies": [] }, { "id": 302154, "author_name": "sbihero", "author_url": "", "post_date": "2018-03-23T18:35:34.980000", "content": "I have same question
", "votes": 0, "replies": [] }, { "id": 301622, "author_name": "Tapioca", "author_url": "", "post_date": "2018-03-23T02:12:47.540000", "content": "Has anyone noticed that the click_id is not in a monotonically increasing order? For example. 9 is before 8. Do we need to sort the submission based on the click_id?
", "votes": 0, "replies": [ { "id": 301770, "author_name": "Max Halford", "author_url": "", "post_date": "2018-03-23T08:05:08.633000", "content": "I noticed this too. But it seems that Kaggle takes care of sorting by click_id. I tried to submit with different orderings and got the same score both times. So don't worry about it.