{
  "id": 2596,
  "title": "How to load files into SQL Server",
  "url": "/competitions/predict-closed-questions-on-stack-overflow/discussion/2596",
  "author_name": "",
  "post_date": "2012-09-06T08:08:42.480Z",
  "votes": null,
  "comment_count": 2,
  "views": 2245,
  "content": "<p>Since Stackoverflow uses SQL Server, I assumed these files would be in a CSV format that was able to be loaded into a SQL Server DB, but alas, they are not. They use double quotes and commas in a quoted field which BULK INSERT (and the Enterprise Manager\r\n Import Tool) doesn't allow.</p>\r\n<p>How are people getting these files loaded? I can't find a conversion tool that changes the delimeter.</p>",
  "messages": [
    {
      "id": "13990",
      "postDate": "09/06/2012 08:08:42",
      "content": "<p>Since Stackoverflow uses SQL Server, I assumed these files would be in a CSV format that was able to be loaded into a SQL Server DB, but alas, they are not. They use double quotes and commas in a quoted field which BULK INSERT (and the Enterprise Manager\r\n Import Tool) doesn't allow.</p>\r\n<p>How are people getting these files loaded? I can't find a conversion tool that changes the delimeter.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "14017",
      "postDate": "09/06/2012 19:06:56",
      "content": "<p>You'll need to use a csv parsing library in a programming language. I used&nbsp;<a href=\"http://docs.python.org/library/csv.html\">Python's csv module</a>. In your case I might investigate a C# library - see\r\n<a href=\"https://www.kaggle.com/c/predict-closed-questions-on-stack-overflow/forums/t/2455/training-sample-csv-parsing-problems\">\r\nmy earlier thread on parsing the training-sample</a> where a response referenced a C# library that apparently parses the file correctly.&nbsp;</p>\r\n<p>From there you'll need to have that programming language write the data out to SQL Server.&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "14020",
      "postDate": "09/06/2012 20:30:36",
      "content": "<p>What a pain. I tried several libraries and conversion apps and nothing was working. I finally got it to work directly in SQL Server using\r\n<code>OPENROWSET</code>.</p>\r\n<pre>CREATE TABLE [dbo].[trainSample](\r\n\t[postId] [int] NOT NULL,\r\n\t[postCreationDate] [datetime2](7) NOT NULL,\r\n\t[ownerUserId] [int] NOT NULL,\r\n\t[ownerCreationDate] [datetime2](7) NULL,\r\n\t[reputationAtPostCreation] [int] NOT NULL,\r\n\t[ownerUndeletedAnswerCountAtPostTime] [int] NOT NULL,\r\n\t[title] [nvarchar](250) NOT NULL,\r\n\t[bodyMarkdown] [nvarchar](max) NOT NULL,\r\n\t[tag1] [nvarchar](150) NULL,\r\n\t[tag2] [nvarchar](150) NULL,\r\n\t[tag3] [nvarchar](150) NULL,\r\n\t[tag4] [nvarchar](150) NULL,\r\n\t[tag5] [nvarchar](150) NULL,\r\n\t[postCloseDate] [datetime2](7) NULL,\r\n\t[openStatus] [varchar](100) NOT NULL\r\n) ON [PRIMARY]\r\n\r\nGO\r\n\r\nINSERT INTO\r\n\tdbo.trainSample\r\n\t\r\nSELECT \r\n\t* \r\n\t\r\nFROM\r\n\tOPENROWSET(\r\n\t'MSDASQL', \r\n\t'Driver={Microsoft Access Text Driver (*.txt, *.csv)};Extensions=asc,csv,tab,txt;Persist Security Info=False;Dbq=C:\\', \r\n\t'select * from train_sample.csv')\r\n</pre>\r\n<p>Couple of gotchas.</p>\r\n<ul>\r\n<li>If you're on a 64-bit server you must install the 64-bit Access Drivers. </li><li>File names can't have a dash (-) in them, so <code>train-sample.csv</code> had to be renamed\r\n</li><li>You may also need to run this first: </li></ul>\r\n<pre>sp_configure 'show advanced options', 1;\r\nGO\r\nRECONFIGURE;\r\nGO\r\nsp_configure 'Ad Hoc Distributed Queries', 1;\r\nGO\r\nRECONFIGURE;\r\nGO\r\nsp_configure 'show advanced options', 1;\r\nGO\r\nRECONFIGURE;\r\nGO\r\nsp_configure 'Ole Automation Procedures', 1;\r\nGO\r\nRECONFIGURE;\r\nGO\r\n</pre>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 14017,
      "author_name": "orangepips",
      "author_url": "",
      "post_date": "09/06/2012 19:06:56",
      "content": "<p>You'll need to use a csv parsing library in a programming language. I used&nbsp;<a href=\"http://docs.python.org/library/csv.html\">Python's csv module</a>. In your case I might investigate a C# library - see\r\n<a href=\"https://www.kaggle.com/c/predict-closed-questions-on-stack-overflow/forums/t/2455/training-sample-csv-parsing-problems\">\r\nmy earlier thread on parsing the training-sample</a> where a response referenced a C# library that apparently parses the file correctly.&nbsp;</p>\r\n<p>From there you'll need to have that programming language write the data out to SQL Server.&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 14020,
      "author_name": "thinkingstiff",
      "author_url": "",
      "post_date": "09/06/2012 20:30:36",
      "content": "<p>What a pain. I tried several libraries and conversion apps and nothing was working. I finally got it to work directly in SQL Server using\r\n<code>OPENROWSET</code>.</p>\r\n<pre>CREATE TABLE [dbo].[trainSample](\r\n\t[postId] [int] NOT NULL,\r\n\t[postCreationDate] [datetime2](7) NOT NULL,\r\n\t[ownerUserId] [int] NOT NULL,\r\n\t[ownerCreationDate] [datetime2](7) NULL,\r\n\t[reputationAtPostCreation] [int] NOT NULL,\r\n\t[ownerUndeletedAnswerCountAtPostTime] [int] NOT NULL,\r\n\t[title] [nvarchar](250) NOT NULL,\r\n\t[bodyMarkdown] [nvarchar](max) NOT NULL,\r\n\t[tag1] [nvarchar](150) NULL,\r\n\t[tag2] [nvarchar](150) NULL,\r\n\t[tag3] [nvarchar](150) NULL,\r\n\t[tag4] [nvarchar](150) NULL,\r\n\t[tag5] [nvarchar](150) NULL,\r\n\t[postCloseDate] [datetime2](7) NULL,\r\n\t[openStatus] [varchar](100) NOT NULL\r\n) ON [PRIMARY]\r\n\r\nGO\r\n\r\nINSERT INTO\r\n\tdbo.trainSample\r\n\t\r\nSELECT \r\n\t* \r\n\t\r\nFROM\r\n\tOPENROWSET(\r\n\t'MSDASQL', \r\n\t'Driver={Microsoft Access Text Driver (*.txt, *.csv)};Extensions=asc,csv,tab,txt;Persist Security Info=False;Dbq=C:\\', \r\n\t'select * from train_sample.csv')\r\n</pre>\r\n<p>Couple of gotchas.</p>\r\n<ul>\r\n<li>If you're on a 64-bit server you must install the 64-bit Access Drivers. </li><li>File names can't have a dash (-) in them, so <code>train-sample.csv</code> had to be renamed\r\n</li><li>You may also need to run this first: </li></ul>\r\n<pre>sp_configure 'show advanced options', 1;\r\nGO\r\nRECONFIGURE;\r\nGO\r\nsp_configure 'Ad Hoc Distributed Queries', 1;\r\nGO\r\nRECONFIGURE;\r\nGO\r\nsp_configure 'show advanced options', 1;\r\nGO\r\nRECONFIGURE;\r\nGO\r\nsp_configure 'Ole Automation Procedures', 1;\r\nGO\r\nRECONFIGURE;\r\nGO\r\n</pre>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "13990": "",
    "14017": "",
    "14020": ""
  },
  "source": "meta"
}