I use R, but this time creating a submission file takes way too long with the processing needed. Here is what I do (and apologies for being lazy to make it a true "kernel" - feel free to do that):
\n\nFirst, I take the test data given (with display_id and ad_id columns for the test.\nI then bind them with my scoring/probabilities (e.g. for each pair there is a score). I write this file as a csv file (it is a big file so it does take a few seconds, but not too many. the expensive part is the processing of the file to the right format which I do outside R.
\n\nI go to my ubuntu/linux command line and type the following:
\n\ncat tmp.csv | sort -t"," -k 1,1n -k 3,3rn | perl -aF, -ne 'chomp $F[1]; print "\\n" x (1 != $.), "$F[0]," if $l ne $F[0]; print " " x ($l eq $F[0]), $F[1]; $l = $F[0] }{ print "\\n"' > output.csv\ntmp.csv is the file wrote from R. the first part (the sort command) takes it and sorts it in the correct way, first grouping it by display_id then by decreasing probabilities within each display id.\nThe cryptic part next is a relatively simple perl script that converts it to the right format really fast, and puts it in the output.csv file.
\n\nuse at your own risk :)
", "messages": [ { "id": 138806, "postDate": "2016-10-10T23:18:29.763Z", "content": "I use R, but this time creating a submission file takes way too long with the processing needed. Here is what I do (and apologies for being lazy to make it a true "kernel" - feel free to do that):
\n\nFirst, I take the test data given (with display_id and ad_id columns for the test.\nI then bind them with my scoring/probabilities (e.g. for each pair there is a score). I write this file as a csv file (it is a big file so it does take a few seconds, but not too many. the expensive part is the processing of the file to the right format which I do outside R.
\n\nI go to my ubuntu/linux command line and type the following:
\n\ncat tmp.csv | sort -t"," -k 1,1n -k 3,3rn | perl -aF, -ne 'chomp $F[1]; print "\\n" x (1 != $.), "$F[0]," if $l ne $F[0]; print " " x ($l eq $F[0]), $F[1]; $l = $F[0] }{ print "\\n"' > output.csv\ntmp.csv is the file wrote from R. the first part (the sort command) takes it and sorts it in the correct way, first grouping it by display_id then by decreasing probabilities within each display id.\nThe cryptic part next is a relatively simple perl script that converts it to the right format really fast, and puts it in the output.csv file.
\n\nuse at your own risk :)
", "rawMarkdown": "I use R, but this time creating a submission file takes way too long with the processing needed. Here is what I do (and apologies for being lazy to make it a true \"kernel\" - feel free to do that):\r\n\r\nFirst, I take the test data given (with display_id and ad_id columns for the test.\r\nI then bind them with my scoring/probabilities (e.g. for each pair there is a score). I write this file as a csv file (it is a big file so it does take a few seconds, but not too many. the expensive part is the processing of the file to the right format which I do outside R.\r\n\r\nI go to my ubuntu/linux command line and type the following:\r\n\r\n cat tmp.csv | sort -t\",\" -k 1,1n -k 3,3rn | perl -aF, -ne 'chomp $F[1]; print \"\\n\" x (1 != $.), \"$F[0],\" if $l ne $F[0]; print \" \" x ($l eq $F[0]), $F[1]; $l = $F[0] }{ print \"\\n\"' > output.csv\r\n\r\ntmp.csv is the file wrote from R. the first part (the sort command) takes it and sorts it in the correct way, first grouping it by display_id then by decreasing probabilities within each display id.\r\nThe cryptic part next is a relatively simple perl script that converts it to the right format really fast, and puts it in the output.csv file.\r\n\r\nuse at your own risk :)\r\n" } ], "comments": [], "raw_markdown_by_id": { "138806": "I use R, but this time creating a submission file takes way too long with the processing needed. Here is what I do (and apologies for being lazy to make it a true \"kernel\" - feel free to do that):\r\n\r\nFirst, I take the test data given (with display_id and ad_id columns for the test.\r\nI then bind them with my scoring/probabilities (e.g. for each pair there is a score). I write this file as a csv file (it is a big file so it does take a few seconds, but not too many. the expensive part is the processing of the file to the right format which I do outside R.\r\n\r\nI go to my ubuntu/linux command line and type the following:\r\n\r\n cat tmp.csv | sort -t\",\" -k 1,1n -k 3,3rn | perl -aF, -ne 'chomp $F[1]; print \"\\n\" x (1 != $.), \"$F[0],\" if $l ne $F[0]; print \" \" x ($l eq $F[0]), $F[1]; $l = $F[0] }{ print \"\\n\"' > output.csv\r\n\r\ntmp.csv is the file wrote from R. the first part (the sort command) takes it and sorts it in the correct way, first grouping it by display_id then by decreasing probabilities within each display id.\r\nThe cryptic part next is a relatively simple perl script that converts it to the right format really fast, and puts it in the output.csv file.\r\n\r\nuse at your own risk :)\r\n" } }