{ "id": 21397, "title": "Parsing JSON", "url": "/competitions/avito-duplicate-ads-detection/discussion/21397", "author_name": "", "post_date": "2016-06-02T22:07:08.683Z", "votes": 2, "comment_count": 15, "views": 2354, "content": "

I tried out parsing the russian text from the JSON in R... without luck. Tried packages jsonlite and RJSONIO. Anyone get that working, or is python required ?

", "messages": [ { "id": "122303", "postDate": "06/02/2016 22:07:08", "content": "

I tried out parsing the russian text from the JSON in R... without luck. Tried packages jsonlite and RJSONIO. Anyone get that working, or is python required ?

", "rawMarkdown": "I tried out parsing the russian text from the JSON in R... without luck. Tried packages jsonlite and RJSONIO. Anyone get that working, or is python required ?", "votes": null }, { "id": "122356", "postDate": "06/03/2016 09:42:40", "content": "

Do you have Libre Office installed?

\n\n

If you open a CSV in Libre Office, it shows a window for detecting the encoding, the separator, etc.

\n\n

Anyhow, ItemInfo_train.csv has clearly been encoded in UTF-8. If you look at the attrsJSON column, you can see that it is perfectly valid JSON, it looks like the examples from the RJSONIO vignette.

\n\n

I never used RJOSNIO, but it looks like you are giving the wrong encoding to the encoding parameter.

\n\n

Please provide an example of you using RJOSION or whatever tool you use.

", "rawMarkdown": "Do you have Libre Office installed?\r\n\r\nIf you open a CSV in Libre Office, it shows a window for detecting the encoding, the separator, etc. \r\n\r\nAnyhow, `ItemInfo_train.csv` has clearly been encoded in UTF-8. If you look at the attrsJSON column, you can see that it is perfectly valid JSON, it looks like the examples from the [RJSONIO vignette](https://cran.r-project.org/web/packages/RJSONIO/RJSONIO.pdf).\r\n\r\nI never used RJOSNIO, but it looks like you are giving the wrong encoding to the `encoding` parameter.\r\n\r\nPlease provide an example of you using RJOSION or whatever tool you use.", "votes": null }, { "id": "122360", "postDate": "06/03/2016 10:39:25", "content": "

Try this code:

\n\n
library(readr)\nlibrary(stringi)\nlibrary(jsonlite)\n\n# Read Items_Train\nitemInfoTrain <- read_csv("../input/ItemInfo_train.csv")\n\n# Convert cyrillic to latin (be patient)\nattrsJSONLatin <- stri_trans_general(itemInfoTrain$attrsJSON, "cyrillic-latin")\n\n# Extract JSON into a list\nlistJSON <- sapply(attrsJSONLatin,fromJSON)\n\nprint(listJSON[[1]])\nprint(listJSON[[2]])\n
\n\n

Regards

", "rawMarkdown": "Try this code:\r\n\r\n\r\n\r\n library(readr)\r\n library(stringi)\r\n library(jsonlite)\r\n \r\n # Read Items_Train\r\n itemInfoTrain <- read_csv(\"../input/ItemInfo_train.csv\")\r\n\r\n # Convert cyrillic to latin (be patient)\r\n attrsJSONLatin <- stri_trans_general(itemInfoTrain$attrsJSON, \"cyrillic-latin\")\r\n\r\n # Extract JSON into a list\r\n listJSON <- sapply(attrsJSONLatin,fromJSON)\r\n \r\n print(listJSON[[1]])\r\n print(listJSON[[2]])\r\n\r\n\r\n\r\n\r\nRegards", "votes": null }, { "id": "122378", "postDate": "06/03/2016 14:47:12", "content": "

Thank you very much F.J. (and Ricardo) for the help. This worked really well - I had spent about 3 hours on this before :)

\n\n
> library(readr)\n> library(stringi)\n> library(jsonlite)\n> itemInfoTrain <- read_csv("ItemInfo_train.csv", n_max = 3)\n> attrsJSONLatin <- stri_trans_general(itemInfoTrain$attrsJSON, "cyrillic-latin") \n    > listJSON <- sapply(attrsJSONLatin,fromJSON) \n    > print(listJSON[[1]])\n    $`Vid tehniki`\n[1] "Gruzoviki"\n\n> print(names(listJSON[[1]]))\n[1] "Vid tehniki"\n> \n> print(listJSON)\n$`{"Vid tehniki":"Gruzoviki"}`\n    $`{"Vid tehniki":"Gruzoviki"}`$`Vid tehniki`\n[1] "Gruzoviki"\n\n\n$`{"Vid tehniki":"Motocikly", "Vid motocikla":"Sportivnye"}`\n    $`{"Vid tehniki":"Motocikly", "Vid motocikla":"Sportivnye"}`$`Vid tehniki`\n[1] "Motocikly"\n\n$`{"Vid tehniki":"Motocikly", "Vid motocikla":"Sportivnye"}`$`Vid motocikla`\n[1] "Sportivnye"\n\n\n$`{"Vid telefona":"iPhone"}`\n    $`{"Vid telefona":"iPhone"}`$`Vid telefona`\n[1] "iPhone"\n
", "rawMarkdown": "Thank you very much F.J. (and Ricardo) for the help. This worked really well - I had spent about 3 hours on this before :) \r\n\r\n\r\n > library(readr)\r\n > library(stringi)\r\n > library(jsonlite)\r\n > itemInfoTrain <- read_csv(\"ItemInfo_train.csv\", n_max = 3)\r\n > attrsJSONLatin <- stri_trans_general(itemInfoTrain$attrsJSON, \"cyrillic-latin\") \r\n > listJSON <- sapply(attrsJSONLatin,fromJSON) \r\n > print(listJSON[[1]])\r\n $`Vid tehniki`\r\n [1] \"Gruzoviki\"\r\n \r\n > print(names(listJSON[[1]]))\r\n [1] \"Vid tehniki\"\r\n > \r\n > print(listJSON)\r\n $`{\"Vid tehniki\":\"Gruzoviki\"}`\r\n $`{\"Vid tehniki\":\"Gruzoviki\"}`$`Vid tehniki`\r\n [1] \"Gruzoviki\"\r\n \r\n \r\n $`{\"Vid tehniki\":\"Motocikly\", \"Vid motocikla\":\"Sportivnye\"}`\r\n $`{\"Vid tehniki\":\"Motocikly\", \"Vid motocikla\":\"Sportivnye\"}`$`Vid tehniki`\r\n [1] \"Motocikly\"\r\n \r\n $`{\"Vid tehniki\":\"Motocikly\", \"Vid motocikla\":\"Sportivnye\"}`$`Vid motocikla`\r\n [1] \"Sportivnye\"\r\n \r\n \r\n $`{\"Vid telefona\":\"iPhone\"}`\r\n $`{\"Vid telefona\":\"iPhone\"}`$`Vid telefona`\r\n [1] \"iPhone\"", "votes": null }, { "id": "122389", "postDate": "06/03/2016 16:37:09", "content": "

You're welcome...

", "rawMarkdown": "You're welcome...", "votes": null }, { "id": "122477", "postDate": "06/04/2016 13:14:55", "content": "

Hello Thanks for the question and answers,

\n\n

listJSON <- sapply(attrsJSONLatin,fromJSON) is returning an error Error: parse error: premature EOF

\n\n
                 (right here) ------^\n
\n\n

Is anyone of you facing this problem?

\n\n

Thank you.

", "rawMarkdown": "Hello Thanks for the question and answers,\r\n\r\n listJSON <- sapply(attrsJSONLatin,fromJSON) is returning an error Error: parse error: premature EOF\r\n \r\n (right here) ------^\r\n\r\nIs anyone of you facing this problem?\r\n\r\nThank you.", "votes": null }, { "id": "122507", "postDate": "06/04/2016 17:42:32", "content": "

Are you transforming cyrillic to latin characters before this command? The previous codes should work...

", "rawMarkdown": "Are you transforming cyrillic to latin characters before this command? The previous codes should work...", "votes": null }, { "id": "122509", "postDate": "06/04/2016 18:05:30", "content": "

[quote=F.J.Martinez-de-Pison;122507]

\n\n

Are you transforming cyrillic to latin characters before this command? The previous codes should work...

\n\n

[/quote]

\n\n

yes I am using the full code above, what I have noticed is the error is coming when JSON is empty string "". The first empty string is at 35th position.

\n\n

When I just select 1:34, I am not getting any error.

\n\n

itemInfoTrain <- read_csv("ItemInfo_train.csv")\n attrsJSONLatin <- stri_trans_general(itemInfoTrain$attrsJSON[1:34], "cyrillic-latin") \n listJSON <- sapply(attrsJSONLatin,fromJSON) \n print(listJSON[[1]])

", "rawMarkdown": "[quote=F.J.Martinez-de-Pison;122507]\r\n\r\nAre you transforming cyrillic to latin characters before this command? The previous codes should work...\r\n\r\n[/quote]\r\n\r\nyes I am using the full code above, what I have noticed is the error is coming when JSON is empty string \"\". The first empty string is at 35th position. \r\n\r\nWhen I just select 1:34, I am not getting any error.\r\n\r\nitemInfoTrain <- read_csv(\"ItemInfo_train.csv\")\r\n attrsJSONLatin <- stri_trans_general(itemInfoTrain$attrsJSON[1:34], \"cyrillic-latin\") \r\n listJSON <- sapply(attrsJSONLatin,fromJSON) \r\n print(listJSON[[1]])", "votes": null }, { "id": "122519", "postDate": "06/04/2016 19:18:55", "content": "

Last day, the previous code worked for me, but now I have the same problem.

\n\n

I have just solved the problem changing NAs with "{}":

\n\n
library(readr)\nlibrary(stringi)\nlibrary(jsonlite)\n\n# Read Items_Train\nitemInfoTrain <- read_csv("ItemInfo_train.csv",n_max = 100)\n# Convert cyrillic to latin\nattrsJSONLatin <- stri_trans_general(itemInfoTrain$attrsJSON, "cyrillic-latin")\n\n# Change NAs to "{}"\nprint(attrsJSONLatin[35])\nattrsJSONLatin[is.na(attrsJSONLatin)] <- "{}"\n\n# Convert JSON\nlistJSON <- sapply(attrsJSONLatin,fromJSON)\nprint(listJSON[[35]])\n
\n\n

Regards

", "rawMarkdown": "Last day, the previous code worked for me, but now I have the same problem. \r\n\r\nI have just solved the problem changing NAs with \"{}\":\r\n\r\n\r\n library(readr)\r\n library(stringi)\r\n library(jsonlite)\r\n \r\n # Read Items_Train\r\n itemInfoTrain <- read_csv(\"ItemInfo_train.csv\",n_max = 100)\r\n # Convert cyrillic to latin\r\n attrsJSONLatin <- stri_trans_general(itemInfoTrain$attrsJSON, \"cyrillic-latin\")\r\n \r\n # Change NAs to \"{}\"\r\n print(attrsJSONLatin[35])\r\n attrsJSONLatin[is.na(attrsJSONLatin)] <- \"{}\"\r\n \r\n # Convert JSON\r\n listJSON <- sapply(attrsJSONLatin,fromJSON)\r\n print(listJSON[[35]])\r\n\r\n\r\n\r\nRegards", "votes": null }, { "id": "122553", "postDate": "06/05/2016 03:14:18", "content": "

[quote=F.J.Martinez-de-Pison;122519]

\n\n

Last day, the previous code worked for me, but now I have the same problem.

\n\n

I have just solved the problem changing NAs with "{}":

\n\n
library(readr)\nlibrary(stringi)\nlibrary(jsonlite)\n\n# Read Items_Train\nitemInfoTrain <- read_csv("ItemInfo_train.csv",n_max = 100)\n# Convert cyrillic to latin\nattrsJSONLatin <- stri_trans_general(itemInfoTrain$attrsJSON, "cyrillic-latin")\n\n# Change NAs to "{}"\nprint(attrsJSONLatin[35])\nattrsJSONLatin[is.na(attrsJSONLatin)] <- "{}"\n\n# Convert JSON\nlistJSON <- sapply(attrsJSONLatin,fromJSON)\nprint(listJSON[[35]])\n
\n\n

Regards

\n\n

[/quote]

\n\n

Hello Thanks, when I replaced empty string as below it is working.

\n\n

attrsJSONLatin[attrsJSONLatin == ""] <- "{}"

", "rawMarkdown": "[quote=F.J.Martinez-de-Pison;122519]\r\n\r\nLast day, the previous code worked for me, but now I have the same problem. \r\n\r\nI have just solved the problem changing NAs with \"{}\":\r\n\r\n\r\n library(readr)\r\n library(stringi)\r\n library(jsonlite)\r\n \r\n # Read Items_Train\r\n itemInfoTrain <- read_csv(\"ItemInfo_train.csv\",n_max = 100)\r\n # Convert cyrillic to latin\r\n attrsJSONLatin <- stri_trans_general(itemInfoTrain$attrsJSON, \"cyrillic-latin\")\r\n \r\n # Change NAs to \"{}\"\r\n print(attrsJSONLatin[35])\r\n attrsJSONLatin[is.na(attrsJSONLatin)] <- \"{}\"\r\n \r\n # Convert JSON\r\n listJSON <- sapply(attrsJSONLatin,fromJSON)\r\n print(listJSON[[35]])\r\n\r\n\r\n\r\nRegards\r\n\r\n[/quote]\r\n\r\nHello Thanks, when I replaced empty string as below it is working.\r\n\r\n attrsJSONLatin[attrsJSONLatin == \"\"] <- \"{}\"", "votes": null }, { "id": "122687", "postDate": "06/06/2016 14:06:26", "content": "

If it is not too much to ask, how are you extracting key and value pairs from the list. Thanks

", "rawMarkdown": "If it is not too much to ask, how are you extracting key and value pairs from the list. Thanks", "votes": null }, { "id": "122797", "postDate": "06/07/2016 07:40:50", "content": "

Sorry for the late response...

\n\n

You can paste keys and values using unlist() function:

\n\n
print(listJSON[[100]])\nkeys <- sapply(listJSON,function(x) return(paste0(unlist(names(x)),collapse=" - ")))\nnames(keys) <- NULL\nvalues <- sapply(listJSON,function(x) return(paste0(unlist(x),collapse=" - ")))\nnames(values) <- NULL\nprint(keys[100])\nprint(values[100])\n
\n\n

regards

", "rawMarkdown": "Sorry for the late response...\r\n\r\nYou can paste keys and values using unlist() function:\r\n\r\n print(listJSON[[100]])\r\n keys <- sapply(listJSON,function(x) return(paste0(unlist(names(x)),collapse=\" - \")))\r\n names(keys) <- NULL\r\n values <- sapply(listJSON,function(x) return(paste0(unlist(x),collapse=\" - \")))\r\n names(values) <- NULL\r\n print(keys[100])\r\n print(values[100])\r\n\r\nregards", "votes": null }, { "id": "122805", "postDate": "06/07/2016 11:11:55", "content": "

[quote=F.J.Martinez-de-Pison;122797]

\n\n

Sorry for the late response...

\n\n

You can paste keys and values using unlist() function:

\n\n
print(listJSON[[100]])\nkeys <- sapply(listJSON,function(x) return(paste0(unlist(names(x)),collapse=" - ")))\nnames(keys) <- NULL\nvalues <- sapply(listJSON,function(x) return(paste0(unlist(x),collapse=" - ")))\nnames(values) <- NULL\nprint(keys[100])\nprint(values[100])\n
\n\n

regards

\n\n

[/quote]

\n\n

Thanks.

", "rawMarkdown": "[quote=F.J.Martinez-de-Pison;122797]\r\n\r\nSorry for the late response...\r\n\r\nYou can paste keys and values using unlist() function:\r\n\r\n print(listJSON[[100]])\r\n keys <- sapply(listJSON,function(x) return(paste0(unlist(names(x)),collapse=\" - \")))\r\n names(keys) <- NULL\r\n values <- sapply(listJSON,function(x) return(paste0(unlist(x),collapse=\" - \")))\r\n names(values) <- NULL\r\n print(keys[100])\r\n print(values[100])\r\n\r\nregards\r\n\r\n[/quote]\r\n\r\nThanks.", "votes": null }, { "id": "125212", "postDate": "06/27/2016 17:20:33", "content": "

Anyone had issue parsing attrsJSONLatin[125620]?

\n\n
Error in parse_string(txt, bigint_as_char) : \n  parse error: after key and value, inside map, I expect ',' or '}'\n          娉婚偑瑜?鍐欒姱灞?閳╂湺" 61*"}\n                     (right here) ------^\n\n> attrsJSONLatin[125620]\n[1] "{\\"孝懈锌 芯斜褗褟胁谢械薪懈褟\\":\\"小写邪屑\\", \\"袣芯谢懈褔械褋褌胁芯 泻芯屑薪邪褌\\":\\"1\\", \\"小褉芯泻 邪褉械薪写褘\\":\\"袧邪 写谢懈褌械谢褜薪褘泄 褋褉芯泻\\", \\"袣芯屑懈褋褋懈褟\\":\\"袧械褌\\", \\"袟邪谢芯谐\\":\\"0\\", \\"协褌邪卸\\":\\"7\\", \\"协褌邪卸械泄 胁 写芯屑械\\":\\"9\\", \\"孝懈锌 写芯屑邪\\":\\"袩邪薪械谢褜薪褘泄\\", \\"袩谢芯褖邪写褜\\":\\"40\\", \\"袗写褉械褋\\":\\"褋褌褍写械薪褔械褋泻邪褟 写芯屑 鈩朶\\" 61*\\"}"\n
", "rawMarkdown": "Anyone had issue parsing attrsJSONLatin[125620]?\r\n\r\n Error in parse_string(txt, bigint_as_char) : \r\n parse error: after key and value, inside map, I expect ',' or '}'\r\n 娉婚偑瑜?鍐欒姱灞?閳╂湺\" 61*\"}\r\n (right here) ------^\r\n \r\n > attrsJSONLatin[125620]\r\n [1] \"{\\\"孝懈锌 芯斜褗褟胁谢械薪懈褟\\\":\\\"小写邪屑\\\", \\\"袣芯谢懈褔械褋褌胁芯 泻芯屑薪邪褌\\\":\\\"1\\\", \\\"小褉芯泻 邪褉械薪写褘\\\":\\\"袧邪 写谢懈褌械谢褜薪褘泄 褋褉芯泻\\\", \\\"袣芯屑懈褋褋懈褟\\\":\\\"袧械褌\\\", \\\"袟邪谢芯谐\\\":\\\"0\\\", \\\"协褌邪卸\\\":\\\"7\\\", \\\"协褌邪卸械泄 胁 写芯屑械\\\":\\\"9\\\", \\\"孝懈锌 写芯屑邪\\\":\\\"袩邪薪械谢褜薪褘泄\\\", \\\"袩谢芯褖邪写褜\\\":\\\"40\\\", \\\"袗写褉械褋\\\":\\\"褋褌褍写械薪褔械褋泻邪褟 写芯屑 鈩朶\\\" 61*\\\"}\"", "votes": null }, { "id": "125213", "postDate": "06/27/2016 17:22:19", "content": "

Doesn't look Russian to me

", "rawMarkdown": "Doesn't look Russian to me", "votes": null }, { "id": "125459", "postDate": "06/29/2016 16:12:08", "content": "

Are you reading it as Russian? I had to set , Sys.setlocale('LC_ALL', 'russian'); and encoding="UTF-8" to make R read Russian correctly

\n\n

[quote=yilisg;125212]

\n\n

Anyone had issue parsing attrsJSONLatin[125620]?泄\\", \\"袩谢芯褖邪写褜\\":\\"40\\", \\"袗写褉械褋\\":\\"褋褌褍写械薪褔械褋泻邪褟 写芯屑 鈩朶\\" 61*\\"}"

\n\n

[/quote]

", "rawMarkdown": "Are you reading it as Russian? I had to set , Sys.setlocale('LC_ALL', 'russian'); and encoding=\"UTF-8\" to make R read Russian correctly\r\n\r\n[quote=yilisg;125212]\r\n\r\nAnyone had issue parsing attrsJSONLatin[125620]?泄\\\", \\\"袩谢芯褖邪写褜\\\":\\\"40\\\", \\\"袗写褉械褋\\\":\\\"褋褌褍写械薪褔械褋泻邪褟 写芯屑 鈩朶\\\" 61*\\\"}\"\r\n\r\n[/quote]", "votes": null } ], "comments": [ { "id": 122356, "author_name": "rpmcruz", "author_url": "", "post_date": "06/03/2016 09:42:40", "content": "

Do you have Libre Office installed?

\n\n

If you open a CSV in Libre Office, it shows a window for detecting the encoding, the separator, etc.

\n\n

Anyhow, ItemInfo_train.csv has clearly been encoded in UTF-8. If you look at the attrsJSON column, you can see that it is perfectly valid JSON, it looks like the examples from the RJSONIO vignette.

\n\n

I never used RJOSNIO, but it looks like you are giving the wrong encoding to the encoding parameter.

\n\n

Please provide an example of you using RJOSION or whatever tool you use.

", "votes": null, "replies": [] }, { "id": 122360, "author_name": "jpison", "author_url": "", "post_date": "06/03/2016 10:39:25", "content": "

Try this code:

\n\n
library(readr)\nlibrary(stringi)\nlibrary(jsonlite)\n\n# Read Items_Train\nitemInfoTrain <- read_csv("../input/ItemInfo_train.csv")\n\n# Convert cyrillic to latin (be patient)\nattrsJSONLatin <- stri_trans_general(itemInfoTrain$attrsJSON, "cyrillic-latin")\n\n# Extract JSON into a list\nlistJSON <- sapply(attrsJSONLatin,fromJSON)\n\nprint(listJSON[[1]])\nprint(listJSON[[2]])\n
\n\n

Regards

", "votes": null, "replies": [] }, { "id": 122378, "author_name": "darraghdog", "author_url": "", "post_date": "06/03/2016 14:47:12", "content": "

Thank you very much F.J. (and Ricardo) for the help. This worked really well - I had spent about 3 hours on this before :)

\n\n
> library(readr)\n> library(stringi)\n> library(jsonlite)\n> itemInfoTrain <- read_csv("ItemInfo_train.csv", n_max = 3)\n> attrsJSONLatin <- stri_trans_general(itemInfoTrain$attrsJSON, "cyrillic-latin") \n    > listJSON <- sapply(attrsJSONLatin,fromJSON) \n    > print(listJSON[[1]])\n    $`Vid tehniki`\n[1] "Gruzoviki"\n\n> print(names(listJSON[[1]]))\n[1] "Vid tehniki"\n> \n> print(listJSON)\n$`{"Vid tehniki":"Gruzoviki"}`\n    $`{"Vid tehniki":"Gruzoviki"}`$`Vid tehniki`\n[1] "Gruzoviki"\n\n\n$`{"Vid tehniki":"Motocikly", "Vid motocikla":"Sportivnye"}`\n    $`{"Vid tehniki":"Motocikly", "Vid motocikla":"Sportivnye"}`$`Vid tehniki`\n[1] "Motocikly"\n\n$`{"Vid tehniki":"Motocikly", "Vid motocikla":"Sportivnye"}`$`Vid motocikla`\n[1] "Sportivnye"\n\n\n$`{"Vid telefona":"iPhone"}`\n    $`{"Vid telefona":"iPhone"}`$`Vid telefona`\n[1] "iPhone"\n
", "votes": null, "replies": [] }, { "id": 122389, "author_name": "jpison", "author_url": "", "post_date": "06/03/2016 16:37:09", "content": "

You're welcome...

", "votes": null, "replies": [] }, { "id": 122477, "author_name": "chinta", "author_url": "", "post_date": "06/04/2016 13:14:55", "content": "

Hello Thanks for the question and answers,

\n\n

listJSON <- sapply(attrsJSONLatin,fromJSON) is returning an error Error: parse error: premature EOF

\n\n
                 (right here) ------^\n
\n\n

Is anyone of you facing this problem?

\n\n

Thank you.

", "votes": null, "replies": [] }, { "id": 122507, "author_name": "jpison", "author_url": "", "post_date": "06/04/2016 17:42:32", "content": "

Are you transforming cyrillic to latin characters before this command? The previous codes should work...

", "votes": null, "replies": [] }, { "id": 122509, "author_name": "chinta", "author_url": "", "post_date": "06/04/2016 18:05:30", "content": "

[quote=F.J.Martinez-de-Pison;122507]

\n\n

Are you transforming cyrillic to latin characters before this command? The previous codes should work...

\n\n

[/quote]

\n\n

yes I am using the full code above, what I have noticed is the error is coming when JSON is empty string "". The first empty string is at 35th position.

\n\n

When I just select 1:34, I am not getting any error.

\n\n

itemInfoTrain <- read_csv("ItemInfo_train.csv")\n attrsJSONLatin <- stri_trans_general(itemInfoTrain$attrsJSON[1:34], "cyrillic-latin") \n listJSON <- sapply(attrsJSONLatin,fromJSON) \n print(listJSON[[1]])

", "votes": null, "replies": [] }, { "id": 122519, "author_name": "jpison", "author_url": "", "post_date": "06/04/2016 19:18:55", "content": "

Last day, the previous code worked for me, but now I have the same problem.

\n\n

I have just solved the problem changing NAs with "{}":

\n\n
library(readr)\nlibrary(stringi)\nlibrary(jsonlite)\n\n# Read Items_Train\nitemInfoTrain <- read_csv("ItemInfo_train.csv",n_max = 100)\n# Convert cyrillic to latin\nattrsJSONLatin <- stri_trans_general(itemInfoTrain$attrsJSON, "cyrillic-latin")\n\n# Change NAs to "{}"\nprint(attrsJSONLatin[35])\nattrsJSONLatin[is.na(attrsJSONLatin)] <- "{}"\n\n# Convert JSON\nlistJSON <- sapply(attrsJSONLatin,fromJSON)\nprint(listJSON[[35]])\n
\n\n

Regards

", "votes": null, "replies": [] }, { "id": 122553, "author_name": "chinta", "author_url": "", "post_date": "06/05/2016 03:14:18", "content": "

[quote=F.J.Martinez-de-Pison;122519]

\n\n

Last day, the previous code worked for me, but now I have the same problem.

\n\n

I have just solved the problem changing NAs with "{}":

\n\n
library(readr)\nlibrary(stringi)\nlibrary(jsonlite)\n\n# Read Items_Train\nitemInfoTrain <- read_csv("ItemInfo_train.csv",n_max = 100)\n# Convert cyrillic to latin\nattrsJSONLatin <- stri_trans_general(itemInfoTrain$attrsJSON, "cyrillic-latin")\n\n# Change NAs to "{}"\nprint(attrsJSONLatin[35])\nattrsJSONLatin[is.na(attrsJSONLatin)] <- "{}"\n\n# Convert JSON\nlistJSON <- sapply(attrsJSONLatin,fromJSON)\nprint(listJSON[[35]])\n
\n\n

Regards

\n\n

[/quote]

\n\n

Hello Thanks, when I replaced empty string as below it is working.

\n\n

attrsJSONLatin[attrsJSONLatin == ""] <- "{}"

", "votes": null, "replies": [] }, { "id": 122687, "author_name": "chinta", "author_url": "", "post_date": "06/06/2016 14:06:26", "content": "

If it is not too much to ask, how are you extracting key and value pairs from the list. Thanks

", "votes": null, "replies": [] }, { "id": 122797, "author_name": "jpison", "author_url": "", "post_date": "06/07/2016 07:40:50", "content": "

Sorry for the late response...

\n\n

You can paste keys and values using unlist() function:

\n\n
print(listJSON[[100]])\nkeys <- sapply(listJSON,function(x) return(paste0(unlist(names(x)),collapse=" - ")))\nnames(keys) <- NULL\nvalues <- sapply(listJSON,function(x) return(paste0(unlist(x),collapse=" - ")))\nnames(values) <- NULL\nprint(keys[100])\nprint(values[100])\n
\n\n

regards

", "votes": null, "replies": [] }, { "id": 122805, "author_name": "chinta", "author_url": "", "post_date": "06/07/2016 11:11:55", "content": "

[quote=F.J.Martinez-de-Pison;122797]

\n\n

Sorry for the late response...

\n\n

You can paste keys and values using unlist() function:

\n\n
print(listJSON[[100]])\nkeys <- sapply(listJSON,function(x) return(paste0(unlist(names(x)),collapse=" - ")))\nnames(keys) <- NULL\nvalues <- sapply(listJSON,function(x) return(paste0(unlist(x),collapse=" - ")))\nnames(values) <- NULL\nprint(keys[100])\nprint(values[100])\n
\n\n

regards

\n\n

[/quote]

\n\n

Thanks.

", "votes": null, "replies": [] }, { "id": 125212, "author_name": "yilihome", "author_url": "", "post_date": "06/27/2016 17:20:33", "content": "

Anyone had issue parsing attrsJSONLatin[125620]?

\n\n
Error in parse_string(txt, bigint_as_char) : \n  parse error: after key and value, inside map, I expect ',' or '}'\n          娉婚偑瑜?鍐欒姱灞?閳╂湺" 61*"}\n                     (right here) ------^\n\n> attrsJSONLatin[125620]\n[1] "{\\"孝懈锌 芯斜褗褟胁谢械薪懈褟\\":\\"小写邪屑\\", \\"袣芯谢懈褔械褋褌胁芯 泻芯屑薪邪褌\\":\\"1\\", \\"小褉芯泻 邪褉械薪写褘\\":\\"袧邪 写谢懈褌械谢褜薪褘泄 褋褉芯泻\\", \\"袣芯屑懈褋褋懈褟\\":\\"袧械褌\\", \\"袟邪谢芯谐\\":\\"0\\", \\"协褌邪卸\\":\\"7\\", \\"协褌邪卸械泄 胁 写芯屑械\\":\\"9\\", \\"孝懈锌 写芯屑邪\\":\\"袩邪薪械谢褜薪褘泄\\", \\"袩谢芯褖邪写褜\\":\\"40\\", \\"袗写褉械褋\\":\\"褋褌褍写械薪褔械褋泻邪褟 写芯屑 鈩朶\\" 61*\\"}"\n
", "votes": null, "replies": [] }, { "id": 125213, "author_name": "agrigorev", "author_url": "", "post_date": "06/27/2016 17:22:19", "content": "

Doesn't look Russian to me

", "votes": null, "replies": [] }, { "id": 125459, "author_name": "anwarsadique", "author_url": "", "post_date": "06/29/2016 16:12:08", "content": "

Are you reading it as Russian? I had to set , Sys.setlocale('LC_ALL', 'russian'); and encoding="UTF-8" to make R read Russian correctly

\n\n

[quote=yilisg;125212]

\n\n

Anyone had issue parsing attrsJSONLatin[125620]?泄\\", \\"袩谢芯褖邪写褜\\":\\"40\\", \\"袗写褉械褋\\":\\"褋褌褍写械薪褔械褋泻邪褟 写芯屑 鈩朶\\" 61*\\"}"

\n\n

[/quote]

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "122303": "I tried out parsing the russian text from the JSON in R... without luck. Tried packages jsonlite and RJSONIO. Anyone get that working, or is python required ?", "122356": "Do you have Libre Office installed?\r\n\r\nIf you open a CSV in Libre Office, it shows a window for detecting the encoding, the separator, etc. \r\n\r\nAnyhow, `ItemInfo_train.csv` has clearly been encoded in UTF-8. If you look at the attrsJSON column, you can see that it is perfectly valid JSON, it looks like the examples from the [RJSONIO vignette](https://cran.r-project.org/web/packages/RJSONIO/RJSONIO.pdf).\r\n\r\nI never used RJOSNIO, but it looks like you are giving the wrong encoding to the `encoding` parameter.\r\n\r\nPlease provide an example of you using RJOSION or whatever tool you use.", "122360": "Try this code:\r\n\r\n\r\n\r\n library(readr)\r\n library(stringi)\r\n library(jsonlite)\r\n \r\n # Read Items_Train\r\n itemInfoTrain <- read_csv(\"../input/ItemInfo_train.csv\")\r\n\r\n # Convert cyrillic to latin (be patient)\r\n attrsJSONLatin <- stri_trans_general(itemInfoTrain$attrsJSON, \"cyrillic-latin\")\r\n\r\n # Extract JSON into a list\r\n listJSON <- sapply(attrsJSONLatin,fromJSON)\r\n \r\n print(listJSON[[1]])\r\n print(listJSON[[2]])\r\n\r\n\r\n\r\n\r\nRegards", "122378": "Thank you very much F.J. (and Ricardo) for the help. This worked really well - I had spent about 3 hours on this before :) \r\n\r\n\r\n > library(readr)\r\n > library(stringi)\r\n > library(jsonlite)\r\n > itemInfoTrain <- read_csv(\"ItemInfo_train.csv\", n_max = 3)\r\n > attrsJSONLatin <- stri_trans_general(itemInfoTrain$attrsJSON, \"cyrillic-latin\") \r\n > listJSON <- sapply(attrsJSONLatin,fromJSON) \r\n > print(listJSON[[1]])\r\n $`Vid tehniki`\r\n [1] \"Gruzoviki\"\r\n \r\n > print(names(listJSON[[1]]))\r\n [1] \"Vid tehniki\"\r\n > \r\n > print(listJSON)\r\n $`{\"Vid tehniki\":\"Gruzoviki\"}`\r\n $`{\"Vid tehniki\":\"Gruzoviki\"}`$`Vid tehniki`\r\n [1] \"Gruzoviki\"\r\n \r\n \r\n $`{\"Vid tehniki\":\"Motocikly\", \"Vid motocikla\":\"Sportivnye\"}`\r\n $`{\"Vid tehniki\":\"Motocikly\", \"Vid motocikla\":\"Sportivnye\"}`$`Vid tehniki`\r\n [1] \"Motocikly\"\r\n \r\n $`{\"Vid tehniki\":\"Motocikly\", \"Vid motocikla\":\"Sportivnye\"}`$`Vid motocikla`\r\n [1] \"Sportivnye\"\r\n \r\n \r\n $`{\"Vid telefona\":\"iPhone\"}`\r\n $`{\"Vid telefona\":\"iPhone\"}`$`Vid telefona`\r\n [1] \"iPhone\"", "122389": "You're welcome...", "122477": "Hello Thanks for the question and answers,\r\n\r\n listJSON <- sapply(attrsJSONLatin,fromJSON) is returning an error Error: parse error: premature EOF\r\n \r\n (right here) ------^\r\n\r\nIs anyone of you facing this problem?\r\n\r\nThank you.", "122507": "Are you transforming cyrillic to latin characters before this command? The previous codes should work...", "122509": "[quote=F.J.Martinez-de-Pison;122507]\r\n\r\nAre you transforming cyrillic to latin characters before this command? The previous codes should work...\r\n\r\n[/quote]\r\n\r\nyes I am using the full code above, what I have noticed is the error is coming when JSON is empty string \"\". The first empty string is at 35th position. \r\n\r\nWhen I just select 1:34, I am not getting any error.\r\n\r\nitemInfoTrain <- read_csv(\"ItemInfo_train.csv\")\r\n attrsJSONLatin <- stri_trans_general(itemInfoTrain$attrsJSON[1:34], \"cyrillic-latin\") \r\n listJSON <- sapply(attrsJSONLatin,fromJSON) \r\n print(listJSON[[1]])", "122519": "Last day, the previous code worked for me, but now I have the same problem. \r\n\r\nI have just solved the problem changing NAs with \"{}\":\r\n\r\n\r\n library(readr)\r\n library(stringi)\r\n library(jsonlite)\r\n \r\n # Read Items_Train\r\n itemInfoTrain <- read_csv(\"ItemInfo_train.csv\",n_max = 100)\r\n # Convert cyrillic to latin\r\n attrsJSONLatin <- stri_trans_general(itemInfoTrain$attrsJSON, \"cyrillic-latin\")\r\n \r\n # Change NAs to \"{}\"\r\n print(attrsJSONLatin[35])\r\n attrsJSONLatin[is.na(attrsJSONLatin)] <- \"{}\"\r\n \r\n # Convert JSON\r\n listJSON <- sapply(attrsJSONLatin,fromJSON)\r\n print(listJSON[[35]])\r\n\r\n\r\n\r\nRegards", "122553": "[quote=F.J.Martinez-de-Pison;122519]\r\n\r\nLast day, the previous code worked for me, but now I have the same problem. \r\n\r\nI have just solved the problem changing NAs with \"{}\":\r\n\r\n\r\n library(readr)\r\n library(stringi)\r\n library(jsonlite)\r\n \r\n # Read Items_Train\r\n itemInfoTrain <- read_csv(\"ItemInfo_train.csv\",n_max = 100)\r\n # Convert cyrillic to latin\r\n attrsJSONLatin <- stri_trans_general(itemInfoTrain$attrsJSON, \"cyrillic-latin\")\r\n \r\n # Change NAs to \"{}\"\r\n print(attrsJSONLatin[35])\r\n attrsJSONLatin[is.na(attrsJSONLatin)] <- \"{}\"\r\n \r\n # Convert JSON\r\n listJSON <- sapply(attrsJSONLatin,fromJSON)\r\n print(listJSON[[35]])\r\n\r\n\r\n\r\nRegards\r\n\r\n[/quote]\r\n\r\nHello Thanks, when I replaced empty string as below it is working.\r\n\r\n attrsJSONLatin[attrsJSONLatin == \"\"] <- \"{}\"", "122687": "If it is not too much to ask, how are you extracting key and value pairs from the list. Thanks", "122797": "Sorry for the late response...\r\n\r\nYou can paste keys and values using unlist() function:\r\n\r\n print(listJSON[[100]])\r\n keys <- sapply(listJSON,function(x) return(paste0(unlist(names(x)),collapse=\" - \")))\r\n names(keys) <- NULL\r\n values <- sapply(listJSON,function(x) return(paste0(unlist(x),collapse=\" - \")))\r\n names(values) <- NULL\r\n print(keys[100])\r\n print(values[100])\r\n\r\nregards", "122805": "[quote=F.J.Martinez-de-Pison;122797]\r\n\r\nSorry for the late response...\r\n\r\nYou can paste keys and values using unlist() function:\r\n\r\n print(listJSON[[100]])\r\n keys <- sapply(listJSON,function(x) return(paste0(unlist(names(x)),collapse=\" - \")))\r\n names(keys) <- NULL\r\n values <- sapply(listJSON,function(x) return(paste0(unlist(x),collapse=\" - \")))\r\n names(values) <- NULL\r\n print(keys[100])\r\n print(values[100])\r\n\r\nregards\r\n\r\n[/quote]\r\n\r\nThanks.", "125212": "Anyone had issue parsing attrsJSONLatin[125620]?\r\n\r\n Error in parse_string(txt, bigint_as_char) : \r\n parse error: after key and value, inside map, I expect ',' or '}'\r\n 娉婚偑瑜?鍐欒姱灞?閳╂湺\" 61*\"}\r\n (right here) ------^\r\n \r\n > attrsJSONLatin[125620]\r\n [1] \"{\\\"孝懈锌 芯斜褗褟胁谢械薪懈褟\\\":\\\"小写邪屑\\\", \\\"袣芯谢懈褔械褋褌胁芯 泻芯屑薪邪褌\\\":\\\"1\\\", \\\"小褉芯泻 邪褉械薪写褘\\\":\\\"袧邪 写谢懈褌械谢褜薪褘泄 褋褉芯泻\\\", \\\"袣芯屑懈褋褋懈褟\\\":\\\"袧械褌\\\", \\\"袟邪谢芯谐\\\":\\\"0\\\", \\\"协褌邪卸\\\":\\\"7\\\", \\\"协褌邪卸械泄 胁 写芯屑械\\\":\\\"9\\\", \\\"孝懈锌 写芯屑邪\\\":\\\"袩邪薪械谢褜薪褘泄\\\", \\\"袩谢芯褖邪写褜\\\":\\\"40\\\", \\\"袗写褉械褋\\\":\\\"褋褌褍写械薪褔械褋泻邪褟 写芯屑 鈩朶\\\" 61*\\\"}\"", "125213": "Doesn't look Russian to me", "125459": "Are you reading it as Russian? I had to set , Sys.setlocale('LC_ALL', 'russian'); and encoding=\"UTF-8\" to make R read Russian correctly\r\n\r\n[quote=yilisg;125212]\r\n\r\nAnyone had issue parsing attrsJSONLatin[125620]?泄\\\", \\\"袩谢芯褖邪写褜\\\":\\\"40\\\", \\\"袗写褉械褋\\\":\\\"褋褌褍写械薪褔械褋泻邪褟 写芯屑 鈩朶\\\" 61*\\\"}\"\r\n\r\n[/quote]" }, "source": "meta" }