{ "id": 2720, "title": "Problem parsing train.cvs with basic_benchmark.py ", "url": "/competitions/predict-closed-questions-on-stack-overflow/discussion/2720", "author_name": "", "post_date": "2012-09-20T04:58:29.657Z", "votes": null, "comment_count": 4, "views": 2024, "content": "

I'm having a trouble using the basic_benchmark.py with the trann.cvs file, the problem is that when I parse the cvs file some rows have nan values to the column "BodyMarkdown". 

\r\n

Some one had already this problem? I think this needs to be fixed and posted on github, mostly because new users will use this code.

\r\n

 The problem is that one questions has no body text, but the real question on the site has some text. If you want to 'clean' the file here is my solution:

\r\n

 

\r\n
import csv
import sys
from datetime import datetime
import time

d = {}

def parse_line(PostId, PostCreationDate, OwnerUserId, OwnerCreationDate, ReputationAtPostCreation, OwnerUndeletedAnswerCountAtPostTime, Title, BodyMarkdown, Tag1, Tag2, Tag3, Tag4, Tag5, PostClosedDate, OpenStatus):
    print Title
    print BodyMarkdown
ifile = open('../train.csv', 'rt')
ofile = open('../clean-train.csv', 'wt')
f = csv.reader(ifile, delimiter=',')
w = csv.writer(ofile, delimiter=',', quotechar='"')
for row in f:
    if (len(row) == 15):
        if (len(row[7]) == 0):
            print row
        else:
            w.writerow(row) 
    else:
        print row
ifile.close()
ofile.close()
\r\n

 

\r\n

 

\r\n

 

\r\n

 

\r\n

 

", "messages": [ { "id": "14623", "postDate": "09/20/2012 04:58:29", "content": "

I'm having a trouble using the basic_benchmark.py with the trann.cvs file, the problem is that when I parse the cvs file some rows have nan values to the column "BodyMarkdown". 

\r\n

Some one had already this problem? I think this needs to be fixed and posted on github, mostly because new users will use this code.

\r\n

 The problem is that one questions has no body text, but the real question on the site has some text. If you want to 'clean' the file here is my solution:

\r\n

 

\r\n
import csv
import sys
from datetime import datetime
import time

d = {}

def parse_line(PostId, PostCreationDate, OwnerUserId, OwnerCreationDate, ReputationAtPostCreation, OwnerUndeletedAnswerCountAtPostTime, Title, BodyMarkdown, Tag1, Tag2, Tag3, Tag4, Tag5, PostClosedDate, OpenStatus):
    print Title
    print BodyMarkdown
ifile = open('../train.csv', 'rt')
ofile = open('../clean-train.csv', 'wt')
f = csv.reader(ifile, delimiter=',')
w = csv.writer(ofile, delimiter=',', quotechar='"')
for row in f:
    if (len(row) == 15):
        if (len(row[7]) == 0):
            print row
        else:
            w.writerow(row) 
    else:
        print row
ifile.close()
ofile.close()
\r\n

 

\r\n

 

\r\n

 

\r\n

 

\r\n

 

", "rawMarkdown": "", "votes": null }, { "id": "14735", "postDate": "09/24/2012 03:25:40", "content": "

I tried the script you provided, but still get

\r\n

['2967852', '06/03/2010 16:21:14', '357697', '06/03/2010 16:21:14', '1', '0', 'Working with NSNumberFormatter.', '', 'objective-c', 'nsnumberformatter', '', '', '', '', 'open']

\r\n

    for row in f:
\r\n_csv.Error: newline inside string

", "rawMarkdown": "", "votes": null }, { "id": "14821", "postDate": "09/25/2012 22:15:41", "content": "

The error happens whit my script or when you try to parse the train.cvs in another code? 

", "rawMarkdown": "", "votes": null }, { "id": "14833", "postDate": "09/26/2012 13:39:03", "content": "

I didn't manage to run the basic benchmark because of some Pandas/Numpy problem.

\r\n

However in my own code, csv.reader parses the file without problems.

", "rawMarkdown": "", "votes": null }, { "id": "14843", "postDate": "09/26/2012 18:31:55", "content": "

[quote=Foxtrot;14833]

\r\n

I didn't manage to run the basic benchmark because of some Pandas/Numpy problem.

\r\n

However in my own code, csv.reader parses the file without problems.

\r\n

[/quote]

\r\n

You updated Numpy to install Pandas? In some older Ubuntu versions when you do that you broke some dependencies, I use Uubuntu 11.10 and 12.04 and works like a charm ;D

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 14735, "author_name": "darkoram", "author_url": "", "post_date": "09/24/2012 03:25:40", "content": "

I tried the script you provided, but still get

\r\n

['2967852', '06/03/2010 16:21:14', '357697', '06/03/2010 16:21:14', '1', '0', 'Working with NSNumberFormatter.', '', 'objective-c', 'nsnumberformatter', '', '', '', '', 'open']

\r\n

    for row in f:
\r\n_csv.Error: newline inside string

", "votes": null, "replies": [] }, { "id": 14821, "author_name": "alessandrosena", "author_url": "", "post_date": "09/25/2012 22:15:41", "content": "

The error happens whit my script or when you try to parse the train.cvs in another code? 

", "votes": null, "replies": [] }, { "id": 14833, "author_name": "zygmunt", "author_url": "", "post_date": "09/26/2012 13:39:03", "content": "

I didn't manage to run the basic benchmark because of some Pandas/Numpy problem.

\r\n

However in my own code, csv.reader parses the file without problems.

", "votes": null, "replies": [] }, { "id": 14843, "author_name": "alessandrosena", "author_url": "", "post_date": "09/26/2012 18:31:55", "content": "

[quote=Foxtrot;14833]

\r\n

I didn't manage to run the basic benchmark because of some Pandas/Numpy problem.

\r\n

However in my own code, csv.reader parses the file without problems.

\r\n

[/quote]

\r\n

You updated Numpy to install Pandas? In some older Ubuntu versions when you do that you broke some dependencies, I use Uubuntu 11.10 and 12.04 and works like a charm ;D

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "14623": "", "14735": "", "14821": "", "14833": "", "14843": "" }, "source": "meta" }