Welcome!
\n\nTo get rolling, we have two baseline scripts attached:
\n\nprocess_html.py: processes the html to extract the core content, links, and urls for images.
classifier.py: is an intentionally modest baseline script which extracts some basic features and changes.
The baseline uses GraphLab Create, the core product our team, at Dato, built for other data geeks like us.
\n\nYou can use any tools for the competition, of course, but we'd love to see how far you can get with GLC. GLC has been used for a number of competitions.
\n\nCheck out our post for how to register and get started w/GLC and then, please
\n\nbeat-the-baseline!
\n\nGame on!
", "messages": [ { "id": "88773", "postDate": "08/07/2015 04:48:56", "content": "Welcome!
\n\nTo get rolling, we have two baseline scripts attached:
\n\nprocess_html.py: processes the html to extract the core content, links, and urls for images.
classifier.py: is an intentionally modest baseline script which extracts some basic features and changes.
The baseline uses GraphLab Create, the core product our team, at Dato, built for other data geeks like us.
\n\nYou can use any tools for the competition, of course, but we'd love to see how far you can get with GLC. GLC has been used for a number of competitions.
\n\nCheck out our post for how to register and get started w/GLC and then, please
\n\nbeat-the-baseline!
\n\nGame on!
", "rawMarkdown": "Welcome!\r\n\r\nTo get rolling, we have two baseline scripts attached:\r\n\r\n - **process_html.py:** processes the html to extract the core content, links, and urls for images.\r\n \r\n - **classifier.py:** is an intentionally modest baseline script which extracts some basic features and changes.\r\n\r\nThe baseline uses GraphLab Create, the core product our team, at Dato, built for other data geeks like us. \r\n\r\nYou can use any tools for the competition, of course, but we'd love to see how far you can get with GLC. GLC has been used for a number of competitions.\r\n\r\nCheck out our [post][1] for how to register and get started w/GLC and then, please\r\n\r\nbeat-the-baseline!\r\n\r\nGame on!\r\n\r\n\r\n [1]: http://blog.dato.com/kaggle-competition-stumbleupon-and-dato-ask-whats-truly-native", "votes": null }, { "id": "88784", "postDate": "08/07/2015 06:19:53", "content": "So, are these files part of the contest as well?
\n\nJust kidding.... because they are full of HTML code inside (i.e., there is an error in your files)
", "rawMarkdown": "So, are these files part of the contest as well?\r\n\r\nJust kidding.... because they are full of HTML code inside (i.e., there is an error in your files)", "votes": null }, { "id": "88785", "postDate": "08/07/2015 06:32:03", "content": "Hi NxGTR,
\n\nThe correct scripts are attached on here. Sorry about the confusion.
\n\nThanks,
\n\nCharlie
", "rawMarkdown": "Hi NxGTR,\r\n\r\nThe correct scripts are attached on here. Sorry about the confusion.\r\n\r\nThanks,\r\n\r\nCharlie", "votes": null }, { "id": "88882", "postDate": "08/08/2015 02:55:48", "content": "Hello! @Cloofa can you explain what the last_bucket means in the process_html.py file? I don't understand why bucket 0 is being processed, but not being saved, as that process seems inefficient!
EDIT: I only downloaded bucket 0, to just get started. I realized that the script will start downloading any bucket other than '0', but after finishing with that bucket, move on to the others like normal.
\n\nTL;DR I would recommmend downloading more than one bucket, even if you are just wanting to play around with a subset of the data.
", "rawMarkdown": "Hello! @Cloofa can you explain what the ```last_bucket``` means in the ```process_html.py``` file? I don't understand why bucket 0 is being processed, but not being saved, as that process seems inefficient!\r\n\r\nEDIT: I only downloaded bucket 0, to just get started. I realized that the script will start downloading any bucket other than '0', but after finishing with that bucket, move on to the others like normal.\r\n\r\nTL;DR I would recommmend downloading more than one bucket, even if you are just wanting to play around with a subset of the data.", "votes": null }, { "id": "88905", "postDate": "08/08/2015 15:27:57", "content": "@Ryan Louie: the script is naive and assumes you have all your files unzipped in the same folder. Since the uncompressed files are large the JSON files may end up being close to 35GB depending on how much you extract. last_bucket is used to track the bucket you are currently parsing, so when the bucket changes, you save the last JSON for that bucket. I did this on purpose, in the case an I/O error happens, you can always continue from your last bucket by modifying a few lines. Feel free to do edit this so that you can save for any given chunk of data.
\n\nAlso, this is a very basic script to get started. If you are familiar with regexp methods, I recommend using that instead of BeautifulSoup4, as it may be faster. We went with this to make things more interpretable for users who are not as familiar with parsing HTML's.
", "rawMarkdown": "Ryan Louie: the script is naive and assumes you have all your files unzipped in the same folder. Since the uncompressed files are large the JSON files may end up being close to 35GB depending on how much you extract. last_bucket is used to track the bucket you are currently parsing, so when the bucket changes, you save the last JSON for that bucket. I did this on purpose, in the case an I/O error happens, you can always continue from your last bucket by modifying a few lines. Feel free to do edit this so that you can save for any given chunk of data. \r\n\r\nAlso, this is a very basic script to get started. If you are familiar with regexp methods, I recommend using that instead of BeautifulSoup4, as it may be faster. We went with this to make things more interpretable for users who are not as familiar with parsing HTML's.", "votes": null }, { "id": "88917", "postDate": "08/08/2015 18:34:45", "content": "So for example you process chunk 0,1,2,3,4\nThen seems you will not save json files for chunk 4. There's no chunk 5, so that no bucket != last_bucket meet for chunk 4?
", "rawMarkdown": "So for example you process chunk 0,1,2,3,4\r\nThen seems you will not save json files for chunk 4. There's no chunk 5, so that no bucket != last_bucket meet for chunk 4?", "votes": null }, { "id": "88921", "postDate": "08/08/2015 18:52:34", "content": "@Jiming Ye: good catch, here is a quick simple fix attached. The other thing you can do is persist to disk a large JSON blob after every X html's are processed, since the buckets themselves are random.
", "rawMarkdown": "Jiming Ye: good catch, here is a quick simple fix attached. The other thing you can do is persist to disk a large JSON blob after every X html's are processed, since the buckets themselves are random.", "votes": null }, { "id": "88935", "postDate": "08/09/2015 01:49:31", "content": "It trained well. But, when predicting, it failed with an error. It looked like the test set was not read right somehow?
", "rawMarkdown": "It trained well. But, when predicting, it failed with an error. It looked like the test set was not read right somehow?", "votes": null }, { "id": "88953", "postDate": "08/09/2015 12:54:45", "content": "@Deep
\n\nremove line:
\n\ntest = test.dropna()\nand you should get proper submission with score: 0.69938
@Dowakin, I removed that line, it still fails with same error as before:
\n\nTraceback (most recent call last):
\n File "classifier.py", line 80, in
\n ypred = model.predict(test)
\n File "/usr/local/lib/python2.7/dist-packages/graphlab/toolkits/classifier/logistic_classifier.py", line 630, in pred\nict
\n missing_value_action = missing_value_action)
\n File "/usr/local/lib/python2.7/dist-packages/graphlab/toolkits/_supervised_learning.py", line 117, in predict
\n target = _graphlab.toolkits._main.run('supervised_learning_predict', options)
\n File "/usr/local/lib/python2.7/dist-packages/graphlab/toolkits/_main.py", line 64, in run
\n (success, message, params) = unity.run_toolkit(toolkit_name, options)
\n File "graphlab/cython/cy_unity.pyx", line 70, in graphlab.cython.cy_unity.UnityGlobalProxy.run_toolkit
\n File "graphlab/cython/cy_unity.pyx", line 74, in graphlab.cython.cy_unity.UnityGlobalProxy.run_toolkit
\nRuntimeError: Runtime Exception. basic_string::resize
@Deep
\n\nOh, I just had error with submission, different number of rows.
\n\nI installed everything in different conda env, just like from their instruction. So if you installed globally, you should try install it locally into new conda env.
", "rawMarkdown": "Deep\r\n\r\nOh, I just had error with submission, different number of rows.\r\n\r\nI installed everything in different conda env, just like from their instruction. So if you installed globally, you should try install it locally into new conda env.", "votes": null }, { "id": "88998", "postDate": "08/10/2015 03:16:30", "content": "@Dowakin, I was using virtualenv, but, I will try conda env. Thanks for the tip!
", "rawMarkdown": "Dowakin, I was using virtualenv, but, I will try conda env. Thanks for the tip!", "votes": null }, { "id": "89840", "postDate": "08/19/2015 19:58:37", "content": "I an new here ...i have this small issue when i ran the code -> submission.save is creating an empty csv file although the sframe submission has data .Anyone else facing this issue ?
", "rawMarkdown": "I an new here ...i have this small issue when i ran the code -> **submission.save is creating an empty csv file although the sframe submission has data .Anyone else facing this issue ?**", "votes": null }, { "id": "90014", "postDate": "08/21/2015 15:56:25", "content": "Can somebody clarify which pass should bu used? \nPATH_TO_JSON = "path/to/data/from/process_html.py"
\n\nI've used path to directory where I've downloaded all archives {0,1,2,3,4}\nbut script fails when parsing compleats.
\n\nProbably, I'm missing something.\nI've got the following error.
\n\nTraceback (most recent call last):\n File "/Users/taras-sereda/PycharmProjects/dato/classifier.py", line 33, in <module>\n sf = sf.unpack('X1',column_name_prefix='')\n File "/usr/local/lib/python2.7/site-packages/graphlab/data_structures/sframe.py", line 4678, in unpack\n new_sf = self[unpack_column].unpack(column_name_prefix, column_types, na_value, limit)\n File "/usr/local/lib/python2.7/site-packages/graphlab/data_structures/sarray.py", line 2692, in unpack\n raise TypeError("Only SArray of dict/list/array type supports unpack")\nTypeError: Only SArray of dict/list/array type supports unpack\n[INFO] Stopping the server connection.\n[quote=taras sereda;90014]
\n\nCan somebody clarify which pass should bu used? \nPATH_TO_JSON = "path/to/data/from/process_html.py"
\n\nI've used path to directory where I've downloaded all archives {0,1,2,3,4}\nbut script fails when parsing compleats.
\n\nProbably, I'm missing something.\nI've got the following error.
\n\nTraceback (most recent call last):\n File "/Users/taras-sereda/PycharmProjects/dato/classifier.py", line 33, in <module>\n sf = sf.unpack('X1',column_name_prefix='')\n File "/usr/local/lib/python2.7/site-packages/graphlab/data_structures/sframe.py", line 4678, in unpack\n new_sf = self[unpack_column].unpack(column_name_prefix, column_types, na_value, limit)\n File "/usr/local/lib/python2.7/site-packages/graphlab/data_structures/sarray.py", line 2692, in unpack\n raise TypeError("Only SArray of dict/list/array type supports unpack")\nTypeError: Only SArray of dict/list/array type supports unpack\n[INFO] Stopping the server connection.\n[/quote]
\n\nMy guess is that you need to run process_html.py first (takes about 8 hours for me) and you will get the json files at the output.\nHowever, I have no ideas what to do after that. In the classifier.py, the code tries to open something as csv file in PATH_TO_JSON, making little sense. Any hints please?
", "rawMarkdown": "[quote=taras sereda;90014]\r\n\r\nCan somebody clarify which pass should bu used? \r\nPATH_TO_JSON = \"path/to/data/from/process_html.py\"\r\n\r\nI've used path to directory where I've downloaded all archives {0,1,2,3,4}\r\nbut script fails when parsing compleats.\r\n\r\nProbably, I'm missing something.\r\nI've got the following error.\r\n\r\n Traceback (most recent call last):\r\n File \"/Users/taras-sereda/PycharmProjects/dato/classifier.py\", line 33, inThanks @FTD2016. I've solved an issue. \nInitially I was expecting a single run of script classifier.py.
\n\nthe problem was that I missunderstood which path to provide.\nPATH_TO_JSON = "path/to/data/from/process_html.py"
\n\nIn classifier.py. csv is used, because each parsed chunk, is not a True json file, actually it's a text file where each line is valid json. So this formulation is also a little bit misleading.
\n\nthe structure of each line in file is a parsed version of raw html in each chunk.\nAs described in parsing script.
\n\ndoc = {\n "id": urlid, \n "text":parse_text(soup),\n "title":parse_title(soup ),\n "links":parse_links(soup),\n "images":parse_images(soup),\n }\nHope, that helped.
", "rawMarkdown": "Thanks @FTD2016. I've solved an issue. \r\nInitially I was expecting a single run of script classifier.py.\r\n\r\nthe problem was that I missunderstood which path to provide.\r\nPATH_TO_JSON = \"path/to/data/from/process_html.py\"\r\n\r\nIn classifier.py. csv is used, because each parsed chunk, is not a True json file, actually it's a text file where each line is valid json. So this formulation is also a little bit misleading.\r\n\r\nthe structure of each line in file is a parsed version of raw html in each chunk.\r\nAs described in parsing script.\r\n\r\n doc = {\r\n \"id\": urlid, \r\n \"text\":parse_text(soup),\r\n \"title\":parse_title(soup ),\r\n \"links\":parse_links(soup),\r\n \"images\":parse_images(soup),\r\n }\r\nHope, that helped.", "votes": null }, { "id": "90167", "postDate": "08/23/2015 18:10:44", "content": "Hi, how did you fix your issue? I ran the process_html.py script on a small subset of files which generated a chunk0.json. I then ran the classifier script. It runs fine till line 34
\n\nsf = gl.SFrame.read_csv(PATH_TO_JSON, header=False)\nand I get "Parsing completed".
\n\nHowever it fails on the next line with a type mismatch.
\n\nI would appreciate any help. Thanks!
", "rawMarkdown": "Hi, how did you fix your issue? I ran the process_html.py script on a small subset of files which generated a chunk0.json. I then ran the classifier script. It runs fine till line 34\r\n\r\n sf = gl.SFrame.read_csv(PATH_TO_JSON, header=False)\r\n\r\nand I get \"Parsing completed\".\r\n\r\nHowever it fails on the next line with a type mismatch.\r\n\r\nI would appreciate any help. Thanks!", "votes": null }, { "id": "91603", "postDate": "09/04/2015 18:53:57", "content": "I want to ask if the code of process_html is bug free or if it needs debugging
", "rawMarkdown": "I want to ask if the code of process_html is bug free or if it needs debugging", "votes": null }, { "id": "91778", "postDate": "09/07/2015 15:41:41", "content": "[quote=cloofa;88905]\nAlso, this is a very basic script to get started. If you are familiar with regexp methods, I recommend using that instead of BeautifulSoup4, as it may be faster. We went with this to make things more interpretable for users who are not as familiar with parsing HTML's.\n[/quote]
\n\nIt's somewhat funny and refreshing to read this as the general opinion about regex being used for html parsing is "you should know better and not do that".
", "rawMarkdown": "[quote=cloofa;88905]\r\nAlso, this is a very basic script to get started. If you are familiar with regexp methods, I recommend using that instead of BeautifulSoup4, as it may be faster. We went with this to make things more interpretable for users who are not as familiar with parsing HTML's.\r\n[/quote]\r\n\r\nIt's somewhat funny and refreshing to read this as the general opinion about regex being used for html parsing is \"you should know better and not do that\".", "votes": null }, { "id": "92230", "postDate": "09/12/2015 04:45:32", "content": "Hi!
\n\nDid anybody get - TypeError: is not JSON serializable when the process_html.py was run?
\n\nThanks
", "rawMarkdown": "Hi!\r\n\r\nDid anybody get - TypeError:Hi,\nThanks for this starter code\nAnother issue\nbow_trn = gl.text_analytics.count_words(train['text_clean'])\ncomplains about count_words needs string SArrays ...\nAny idea ?\nTIA\nRgds\nBruno
", "rawMarkdown": "Hi,\r\nThanks for this starter code\r\nAnother issue\r\nbow_trn = gl.text_analytics.count_words(train['text_clean'])\r\ncomplains about count_words needs string SArrays ...\r\nAny idea ?\r\nTIA\r\nRgds\r\nBruno", "votes": null }, { "id": "93068", "postDate": "09/21/2015 06:01:19", "content": "Hi All,
\n\nAnyone else encounter this error? I am using Python 3.4.3 that comes with Anaconda.
\n\nTraceback (most recent call last):\n File "process_html_new.py", line 158, in \n main(sys.argv)\n File "process_html_new.py", line 139, in main\n json.dump(entry, feedsjson)\n File "/Users/ntogar/anaconda/envs/kaggle/lib/python3.4/json/init.py", line 178, in dump\n for chunk in iterable:\n File "/Users/ntogar/anaconda/envs/kaggle/lib/python3.4/json/encoder.py", line 422, in _iterencode\n yield from _iterencode_dict(o, _current_indent_level)\n File "/Users/ntogar/anaconda/envs/kaggle/lib/python3.4/json/encoder.py", line 396, in _iterencode_dict\n yield from chunks\n File "/Users/ntogar/anaconda/envs/kaggle/lib/python3.4/json/encoder.py", line 429, in _iterencode\n o = _default(o)\n File "/Users/ntogar/anaconda/envs/kaggle/lib/python3.4/json/encoder.py", line 173, in default\n raise TypeError(repr(o) + " is not JSON serializable")\nTypeError: is not JSON serializable
\n\nRegards,\nNaveen
", "rawMarkdown": "Hi All,\r\n\r\nAnyone else encounter this error? I am using Python 3.4.3 that comes with Anaconda.\r\n\r\nTraceback (most recent call last):\r\n File \"process_html_new.py\", line 158, inHi Competition Admin,\nI have had several submissions before the data leak. Recently, due to some personal commitments, I couldn't submit a new entry.
\n\nNow, I just passed the deadline, which I overlooked. Would it be possible to allow previous submitters like myself to submit again?
\n\nVery much appreciated. Thank you.
\n\nBest,\nHarish
", "rawMarkdown": "Hi Competition Admin,\r\nI have had several submissions before the data leak. Recently, due to some personal commitments, I couldn't submit a new entry.\r\n\r\nNow, I just passed the deadline, which I overlooked. Would it be possible to allow previous submitters like myself to submit again?\r\n\r\nVery much appreciated. Thank you.\r\n\r\nBest,\r\nHarish", "votes": null }, { "id": "96126", "postDate": "10/14/2015 17:01:44", "content": "The attached script is an almost automatic workflow for GraphLab Create, except its input relies on the stored feature set from https://www.kaggle.com/c/dato-native/forums/t/16626/beat-the-benchmark-0-90388-with-simple-model/95814#post95814 , which is inspired by @David Shinn's nice work.
", "rawMarkdown": "The attached script is an almost automatic workflow for GraphLab Create, except its input relies on the stored feature set from https://www.kaggle.com/c/dato-native/forums/t/16626/beat-the-benchmark-0-90388-with-simple-model/95814#post95814 , which is inspired by [@David Shinn][1]'s nice work.\r\n\r\n\r\n [1]: https://www.kaggle.com/davidshinn", "votes": null } ], "comments": [ { "id": 88784, "author_name": "carloshuertas", "author_url": "", "post_date": "08/07/2015 06:19:53", "content": "So, are these files part of the contest as well?
\n\nJust kidding.... because they are full of HTML code inside (i.e., there is an error in your files)
", "votes": null, "replies": [] }, { "id": 88785, "author_name": "cloofa", "author_url": "", "post_date": "08/07/2015 06:32:03", "content": "Hi NxGTR,
\n\nThe correct scripts are attached on here. Sorry about the confusion.
\n\nThanks,
\n\nCharlie
", "votes": null, "replies": [] }, { "id": 88882, "author_name": "ryanlouie", "author_url": "", "post_date": "08/08/2015 02:55:48", "content": "Hello! @Cloofa can you explain what the last_bucket means in the process_html.py file? I don't understand why bucket 0 is being processed, but not being saved, as that process seems inefficient!
EDIT: I only downloaded bucket 0, to just get started. I realized that the script will start downloading any bucket other than '0', but after finishing with that bucket, move on to the others like normal.
\n\nTL;DR I would recommmend downloading more than one bucket, even if you are just wanting to play around with a subset of the data.
", "votes": null, "replies": [] }, { "id": 88905, "author_name": "cloofa", "author_url": "", "post_date": "08/08/2015 15:27:57", "content": "@Ryan Louie: the script is naive and assumes you have all your files unzipped in the same folder. Since the uncompressed files are large the JSON files may end up being close to 35GB depending on how much you extract. last_bucket is used to track the bucket you are currently parsing, so when the bucket changes, you save the last JSON for that bucket. I did this on purpose, in the case an I/O error happens, you can always continue from your last bucket by modifying a few lines. Feel free to do edit this so that you can save for any given chunk of data.
\n\nAlso, this is a very basic script to get started. If you are familiar with regexp methods, I recommend using that instead of BeautifulSoup4, as it may be faster. We went with this to make things more interpretable for users who are not as familiar with parsing HTML's.
", "votes": null, "replies": [] }, { "id": 88917, "author_name": "yejiming", "author_url": "", "post_date": "08/08/2015 18:34:45", "content": "So for example you process chunk 0,1,2,3,4\nThen seems you will not save json files for chunk 4. There's no chunk 5, so that no bucket != last_bucket meet for chunk 4?
", "votes": null, "replies": [] }, { "id": 88921, "author_name": "cloofa", "author_url": "", "post_date": "08/08/2015 18:52:34", "content": "@Jiming Ye: good catch, here is a quick simple fix attached. The other thing you can do is persist to disk a large JSON blob after every X html's are processed, since the buckets themselves are random.
", "votes": null, "replies": [] }, { "id": 88935, "author_name": "deepcnn", "author_url": "", "post_date": "08/09/2015 01:49:31", "content": "It trained well. But, when predicting, it failed with an error. It looked like the test set was not read right somehow?
", "votes": null, "replies": [] }, { "id": 88953, "author_name": "dowakin", "author_url": "", "post_date": "08/09/2015 12:54:45", "content": "@Deep
\n\nremove line:
\n\ntest = test.dropna()\nand you should get proper submission with score: 0.69938
@Dowakin, I removed that line, it still fails with same error as before:
\n\nTraceback (most recent call last):
\n File "classifier.py", line 80, in
\n ypred = model.predict(test)
\n File "/usr/local/lib/python2.7/dist-packages/graphlab/toolkits/classifier/logistic_classifier.py", line 630, in pred\nict
\n missing_value_action = missing_value_action)
\n File "/usr/local/lib/python2.7/dist-packages/graphlab/toolkits/_supervised_learning.py", line 117, in predict
\n target = _graphlab.toolkits._main.run('supervised_learning_predict', options)
\n File "/usr/local/lib/python2.7/dist-packages/graphlab/toolkits/_main.py", line 64, in run
\n (success, message, params) = unity.run_toolkit(toolkit_name, options)
\n File "graphlab/cython/cy_unity.pyx", line 70, in graphlab.cython.cy_unity.UnityGlobalProxy.run_toolkit
\n File "graphlab/cython/cy_unity.pyx", line 74, in graphlab.cython.cy_unity.UnityGlobalProxy.run_toolkit
\nRuntimeError: Runtime Exception. basic_string::resize
@Deep
\n\nOh, I just had error with submission, different number of rows.
\n\nI installed everything in different conda env, just like from their instruction. So if you installed globally, you should try install it locally into new conda env.
", "votes": null, "replies": [] }, { "id": 88998, "author_name": "deepcnn", "author_url": "", "post_date": "08/10/2015 03:16:30", "content": "@Dowakin, I was using virtualenv, but, I will try conda env. Thanks for the tip!
", "votes": null, "replies": [] }, { "id": 89840, "author_name": "arnabsarkar", "author_url": "", "post_date": "08/19/2015 19:58:37", "content": "I an new here ...i have this small issue when i ran the code -> submission.save is creating an empty csv file although the sframe submission has data .Anyone else facing this issue ?
", "votes": null, "replies": [] }, { "id": 90014, "author_name": "tsereda", "author_url": "", "post_date": "08/21/2015 15:56:25", "content": "Can somebody clarify which pass should bu used? \nPATH_TO_JSON = "path/to/data/from/process_html.py"
\n\nI've used path to directory where I've downloaded all archives {0,1,2,3,4}\nbut script fails when parsing compleats.
\n\nProbably, I'm missing something.\nI've got the following error.
\n\nTraceback (most recent call last):\n File "/Users/taras-sereda/PycharmProjects/dato/classifier.py", line 33, in <module>\n sf = sf.unpack('X1',column_name_prefix='')\n File "/usr/local/lib/python2.7/site-packages/graphlab/data_structures/sframe.py", line 4678, in unpack\n new_sf = self[unpack_column].unpack(column_name_prefix, column_types, na_value, limit)\n File "/usr/local/lib/python2.7/site-packages/graphlab/data_structures/sarray.py", line 2692, in unpack\n raise TypeError("Only SArray of dict/list/array type supports unpack")\nTypeError: Only SArray of dict/list/array type supports unpack\n[INFO] Stopping the server connection.\n[quote=taras sereda;90014]
\n\nCan somebody clarify which pass should bu used? \nPATH_TO_JSON = "path/to/data/from/process_html.py"
\n\nI've used path to directory where I've downloaded all archives {0,1,2,3,4}\nbut script fails when parsing compleats.
\n\nProbably, I'm missing something.\nI've got the following error.
\n\nTraceback (most recent call last):\n File "/Users/taras-sereda/PycharmProjects/dato/classifier.py", line 33, in <module>\n sf = sf.unpack('X1',column_name_prefix='')\n File "/usr/local/lib/python2.7/site-packages/graphlab/data_structures/sframe.py", line 4678, in unpack\n new_sf = self[unpack_column].unpack(column_name_prefix, column_types, na_value, limit)\n File "/usr/local/lib/python2.7/site-packages/graphlab/data_structures/sarray.py", line 2692, in unpack\n raise TypeError("Only SArray of dict/list/array type supports unpack")\nTypeError: Only SArray of dict/list/array type supports unpack\n[INFO] Stopping the server connection.\n[/quote]
\n\nMy guess is that you need to run process_html.py first (takes about 8 hours for me) and you will get the json files at the output.\nHowever, I have no ideas what to do after that. In the classifier.py, the code tries to open something as csv file in PATH_TO_JSON, making little sense. Any hints please?
", "votes": null, "replies": [] }, { "id": 90089, "author_name": "tsereda", "author_url": "", "post_date": "08/22/2015 07:33:12", "content": "Thanks @FTD2016. I've solved an issue. \nInitially I was expecting a single run of script classifier.py.
\n\nthe problem was that I missunderstood which path to provide.\nPATH_TO_JSON = "path/to/data/from/process_html.py"
\n\nIn classifier.py. csv is used, because each parsed chunk, is not a True json file, actually it's a text file where each line is valid json. So this formulation is also a little bit misleading.
\n\nthe structure of each line in file is a parsed version of raw html in each chunk.\nAs described in parsing script.
\n\ndoc = {\n "id": urlid, \n "text":parse_text(soup),\n "title":parse_title(soup ),\n "links":parse_links(soup),\n "images":parse_images(soup),\n }\nHope, that helped.
", "votes": null, "replies": [] }, { "id": 90167, "author_name": "saurabhsaxena2", "author_url": "", "post_date": "08/23/2015 18:10:44", "content": "Hi, how did you fix your issue? I ran the process_html.py script on a small subset of files which generated a chunk0.json. I then ran the classifier script. It runs fine till line 34
\n\nsf = gl.SFrame.read_csv(PATH_TO_JSON, header=False)\nand I get "Parsing completed".
\n\nHowever it fails on the next line with a type mismatch.
\n\nI would appreciate any help. Thanks!
", "votes": null, "replies": [] }, { "id": 91603, "author_name": "ma2afify", "author_url": "", "post_date": "09/04/2015 18:53:57", "content": "I want to ask if the code of process_html is bug free or if it needs debugging
", "votes": null, "replies": [] }, { "id": 91778, "author_name": "wildwizard", "author_url": "", "post_date": "09/07/2015 15:41:41", "content": "[quote=cloofa;88905]\nAlso, this is a very basic script to get started. If you are familiar with regexp methods, I recommend using that instead of BeautifulSoup4, as it may be faster. We went with this to make things more interpretable for users who are not as familiar with parsing HTML's.\n[/quote]
\n\nIt's somewhat funny and refreshing to read this as the general opinion about regex being used for html parsing is "you should know better and not do that".
", "votes": null, "replies": [] }, { "id": 92230, "author_name": "dataparty", "author_url": "", "post_date": "09/12/2015 04:45:32", "content": "Hi!
\n\nDid anybody get - TypeError: is not JSON serializable when the process_html.py was run?
\n\nThanks
", "votes": null, "replies": [] }, { "id": 93041, "author_name": "bruno16", "author_url": "", "post_date": "09/20/2015 19:24:04", "content": "Hi,\nThanks for this starter code\nAnother issue\nbow_trn = gl.text_analytics.count_words(train['text_clean'])\ncomplains about count_words needs string SArrays ...\nAny idea ?\nTIA\nRgds\nBruno
", "votes": null, "replies": [] }, { "id": 93068, "author_name": "kumarnt", "author_url": "", "post_date": "09/21/2015 06:01:19", "content": "Hi All,
\n\nAnyone else encounter this error? I am using Python 3.4.3 that comes with Anaconda.
\n\nTraceback (most recent call last):\n File "process_html_new.py", line 158, in \n main(sys.argv)\n File "process_html_new.py", line 139, in main\n json.dump(entry, feedsjson)\n File "/Users/ntogar/anaconda/envs/kaggle/lib/python3.4/json/init.py", line 178, in dump\n for chunk in iterable:\n File "/Users/ntogar/anaconda/envs/kaggle/lib/python3.4/json/encoder.py", line 422, in _iterencode\n yield from _iterencode_dict(o, _current_indent_level)\n File "/Users/ntogar/anaconda/envs/kaggle/lib/python3.4/json/encoder.py", line 396, in _iterencode_dict\n yield from chunks\n File "/Users/ntogar/anaconda/envs/kaggle/lib/python3.4/json/encoder.py", line 429, in _iterencode\n o = _default(o)\n File "/Users/ntogar/anaconda/envs/kaggle/lib/python3.4/json/encoder.py", line 173, in default\n raise TypeError(repr(o) + " is not JSON serializable")\nTypeError: is not JSON serializable
\n\nRegards,\nNaveen
", "votes": null, "replies": [] }, { "id": 95533, "author_name": "hvsarma", "author_url": "", "post_date": "10/08/2015 19:40:43", "content": "Hi Competition Admin,\nI have had several submissions before the data leak. Recently, due to some personal commitments, I couldn't submit a new entry.
\n\nNow, I just passed the deadline, which I overlooked. Would it be possible to allow previous submitters like myself to submit again?
\n\nVery much appreciated. Thank you.
\n\nBest,\nHarish
", "votes": null, "replies": [] }, { "id": 96126, "author_name": "tmjiang", "author_url": "", "post_date": "10/14/2015 17:01:44", "content": "The attached script is an almost automatic workflow for GraphLab Create, except its input relies on the stored feature set from https://www.kaggle.com/c/dato-native/forums/t/16626/beat-the-benchmark-0-90388-with-simple-model/95814#post95814 , which is inspired by @David Shinn's nice work.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "88773": "Welcome!\r\n\r\nTo get rolling, we have two baseline scripts attached:\r\n\r\n - **process_html.py:** processes the html to extract the core content, links, and urls for images.\r\n \r\n - **classifier.py:** is an intentionally modest baseline script which extracts some basic features and changes.\r\n\r\nThe baseline uses GraphLab Create, the core product our team, at Dato, built for other data geeks like us. \r\n\r\nYou can use any tools for the competition, of course, but we'd love to see how far you can get with GLC. GLC has been used for a number of competitions.\r\n\r\nCheck out our [post][1] for how to register and get started w/GLC and then, please\r\n\r\nbeat-the-baseline!\r\n\r\nGame on!\r\n\r\n\r\n [1]: http://blog.dato.com/kaggle-competition-stumbleupon-and-dato-ask-whats-truly-native", "88784": "So, are these files part of the contest as well?\r\n\r\nJust kidding.... because they are full of HTML code inside (i.e., there is an error in your files)", "88785": "Hi NxGTR,\r\n\r\nThe correct scripts are attached on here. Sorry about the confusion.\r\n\r\nThanks,\r\n\r\nCharlie", "88882": "Hello! @Cloofa can you explain what the ```last_bucket``` means in the ```process_html.py``` file? I don't understand why bucket 0 is being processed, but not being saved, as that process seems inefficient!\r\n\r\nEDIT: I only downloaded bucket 0, to just get started. I realized that the script will start downloading any bucket other than '0', but after finishing with that bucket, move on to the others like normal.\r\n\r\nTL;DR I would recommmend downloading more than one bucket, even if you are just wanting to play around with a subset of the data.", "88905": "Ryan Louie: the script is naive and assumes you have all your files unzipped in the same folder. Since the uncompressed files are large the JSON files may end up being close to 35GB depending on how much you extract. last_bucket is used to track the bucket you are currently parsing, so when the bucket changes, you save the last JSON for that bucket. I did this on purpose, in the case an I/O error happens, you can always continue from your last bucket by modifying a few lines. Feel free to do edit this so that you can save for any given chunk of data. \r\n\r\nAlso, this is a very basic script to get started. If you are familiar with regexp methods, I recommend using that instead of BeautifulSoup4, as it may be faster. We went with this to make things more interpretable for users who are not as familiar with parsing HTML's.", "88917": "So for example you process chunk 0,1,2,3,4\r\nThen seems you will not save json files for chunk 4. There's no chunk 5, so that no bucket != last_bucket meet for chunk 4?", "88921": "Jiming Ye: good catch, here is a quick simple fix attached. The other thing you can do is persist to disk a large JSON blob after every X html's are processed, since the buckets themselves are random.", "88935": "It trained well. But, when predicting, it failed with an error. It looked like the test set was not read right somehow?", "88953": "Deep\r\n\r\nremove line:\r\n\r\n test = test.dropna()\r\n\r\nand you should get proper submission with score: `0.69938`", "88983": "Dowakin, I removed that line, it still fails with same error as before: \r\n\r\nTraceback (most recent call last): \r\n File \"classifier.py\", line 80, in