{
  "id": 115018,
  "title": "Sharing my utility script: Convert line-delimited JSON to pandas",
  "url": "/competitions/tensorflow2-question-answering/discussion/115018",
  "author_name": "",
  "post_date": "2019-10-30T19:02:33.323848900Z",
  "votes": 34,
  "comment_count": 4,
  "views": 0,
  "content": "<p>Hey all! I wanted to share the utility script I created: <a href=\"https://www.kaggle.com/xhlulu/tf-qa-jsonl-to-dataframe\">https://www.kaggle.com/xhlulu/tf-qa-jsonl-to-dataframe</a></p>\n\n<p>Essentially, it lets you use a function called <code>json_to_df(file_path)</code>.</p>\n\n<p>First open your notebook, then click \"File\" &gt; \"Add utility script\", search \"TF QA: jsonl to DataFrame\", then run the following code:</p>\n\n<p>```python\nfrom tf_qa_jsonl_to_dataframe import jsonl_to_df</p>\n\n<p>directory = '/kaggle/input/tensorflow2-question-answering/'\ntrain = jsonl_to_df(directory + 'simplified-nq-train.jsonl', n_rows=10000)\ntest = jsonl_to_df(directory + 'simplified-nq-test.jsonl', load_annotations=False)\n```\nI also included those instructions in the script.</p>\n\n<p>My biggest motivation for creating a separate script was to test out this recently released feature, <a href=\"https://www.kaggle.com/product-feedback/91185\">which was announced here</a>. I am really happy about utility scripts, because I believe it will entice the community to create more reproducible, robust, and well-documented functions. Those functions might even be useful for future competitions.</p>\n\n<p>Let me know your thoughts on that new feature, and feedback about my scripts are highly appreciated :)</p>",
  "messages": [
    {
      "id": "661844",
      "postDate": "10/30/2019 19:02:33",
      "content": "<p>Hey all! I wanted to share the utility script I created: <a href=\"https://www.kaggle.com/xhlulu/tf-qa-jsonl-to-dataframe\">https://www.kaggle.com/xhlulu/tf-qa-jsonl-to-dataframe</a></p>\n\n<p>Essentially, it lets you use a function called <code>json_to_df(file_path)</code>.</p>\n\n<p>First open your notebook, then click \"File\" &gt; \"Add utility script\", search \"TF QA: jsonl to DataFrame\", then run the following code:</p>\n\n<p>```python\nfrom tf_qa_jsonl_to_dataframe import jsonl_to_df</p>\n\n<p>directory = '/kaggle/input/tensorflow2-question-answering/'\ntrain = jsonl_to_df(directory + 'simplified-nq-train.jsonl', n_rows=10000)\ntest = jsonl_to_df(directory + 'simplified-nq-test.jsonl', load_annotations=False)\n```\nI also included those instructions in the script.</p>\n\n<p>My biggest motivation for creating a separate script was to test out this recently released feature, <a href=\"https://www.kaggle.com/product-feedback/91185\">which was announced here</a>. I am really happy about utility scripts, because I believe it will entice the community to create more reproducible, robust, and well-documented functions. Those functions might even be useful for future competitions.</p>\n\n<p>Let me know your thoughts on that new feature, and feedback about my scripts are highly appreciated :)</p>",
      "rawMarkdown": "Hey all! I wanted to share the utility script I created: https://www.kaggle.com/xhlulu/tf-qa-jsonl-to-dataframe\n\nEssentially, it lets you use a function called `json_to_df(file_path)`.\n\nFirst open your notebook, then click \"File\" &gt; \"Add utility script\", search \"TF QA: jsonl to DataFrame\", then run the following code:\n\n```python\nfrom tf_qa_jsonl_to_dataframe import jsonl_to_df\n\ndirectory = '/kaggle/input/tensorflow2-question-answering/'\ntrain = jsonl_to_df(directory + 'simplified-nq-train.jsonl', n_rows=10000)\ntest = jsonl_to_df(directory + 'simplified-nq-test.jsonl', load_annotations=False)\n```\nI also included those instructions in the script.\n\nMy biggest motivation for creating a separate script was to test out this recently released feature, [which was announced here](https://www.kaggle.com/product-feedback/91185). I am really happy about utility scripts, because I believe it will entice the community to create more reproducible, robust, and well-documented functions. Those functions might even be useful for future competitions.\n\nLet me know your thoughts on that new feature, and feedback about my scripts are highly appreciated :)",
      "votes": null
    },
    {
      "id": "661884",
      "postDate": "10/30/2019 20:21:53",
      "content": "<p>This is my second time in a kernel-only competition and the first time I try the new feature (thanks to your hint!). Definitely a very convenient feature, and also a nice script! </p>",
      "rawMarkdown": "This is my second time in a kernel-only competition and the first time I try the new feature (thanks to your hint!). Definitely a very convenient feature, and also a nice script!",
      "votes": null
    },
    {
      "id": "662099",
      "postDate": "10/31/2019 04:40:55",
      "content": "<p>Very Helpful Script\nThanks <a href=\"/xhlulu\">@xhlulu</a> </p>",
      "rawMarkdown": "Very Helpful Script\nThanks @xhlulu",
      "votes": null
    },
    {
      "id": "663488",
      "postDate": "11/02/2019 06:50:20",
      "content": "<p>@xhulu I cannot explain how thankful I am to you! Thank you so much for this awesome script.</p>\n\n<p>I also owe a lot to you for my <a href=\"https://www.kaggle.com/tarunpaparaju/lyft-competition-understanding-the-data\">Lyft Kernel</a>. Your animation code was very important there. So, once again, thank you!</p>",
      "rawMarkdown": "xhulu I cannot explain how thankful I am to you! Thank you so much for this awesome script.\n\nI also owe a lot to you for my [Lyft Kernel](https://www.kaggle.com/tarunpaparaju/lyft-competition-understanding-the-data). Your animation code was very important there. So, once again, thank you!",
      "votes": null
    },
    {
      "id": "698254",
      "postDate": "12/19/2019 01:39:41",
      "content": "<p>Thanks <a href=\"/xhlulu\">@xhlulu</a> !</p>",
      "rawMarkdown": "Thanks @xhlulu !",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 661884,
      "author_name": "roebius",
      "author_url": "",
      "post_date": "10/30/2019 20:21:53",
      "content": "<p>This is my second time in a kernel-only competition and the first time I try the new feature (thanks to your hint!). Definitely a very convenient feature, and also a nice script! </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 662099,
      "author_name": "veeralakrishna",
      "author_url": "",
      "post_date": "10/31/2019 04:40:55",
      "content": "<p>Very Helpful Script\nThanks <a href=\"/xhlulu\">@xhlulu</a> </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 663488,
      "author_name": "tarunpaparaju",
      "author_url": "",
      "post_date": "11/02/2019 06:50:20",
      "content": "<p>@xhulu I cannot explain how thankful I am to you! Thank you so much for this awesome script.</p>\n\n<p>I also owe a lot to you for my <a href=\"https://www.kaggle.com/tarunpaparaju/lyft-competition-understanding-the-data\">Lyft Kernel</a>. Your animation code was very important there. So, once again, thank you!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 698254,
      "author_name": "rohitagarwal",
      "author_url": "",
      "post_date": "12/19/2019 01:39:41",
      "content": "<p>Thanks <a href=\"/xhlulu\">@xhlulu</a> !</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "661844": "Hey all! I wanted to share the utility script I created: https://www.kaggle.com/xhlulu/tf-qa-jsonl-to-dataframe\n\nEssentially, it lets you use a function called `json_to_df(file_path)`.\n\nFirst open your notebook, then click \"File\" &gt; \"Add utility script\", search \"TF QA: jsonl to DataFrame\", then run the following code:\n\n```python\nfrom tf_qa_jsonl_to_dataframe import jsonl_to_df\n\ndirectory = '/kaggle/input/tensorflow2-question-answering/'\ntrain = jsonl_to_df(directory + 'simplified-nq-train.jsonl', n_rows=10000)\ntest = jsonl_to_df(directory + 'simplified-nq-test.jsonl', load_annotations=False)\n```\nI also included those instructions in the script.\n\nMy biggest motivation for creating a separate script was to test out this recently released feature, [which was announced here](https://www.kaggle.com/product-feedback/91185). I am really happy about utility scripts, because I believe it will entice the community to create more reproducible, robust, and well-documented functions. Those functions might even be useful for future competitions.\n\nLet me know your thoughts on that new feature, and feedback about my scripts are highly appreciated :)",
    "661884": "This is my second time in a kernel-only competition and the first time I try the new feature (thanks to your hint!). Definitely a very convenient feature, and also a nice script!",
    "662099": "Very Helpful Script\nThanks @xhlulu",
    "663488": "xhulu I cannot explain how thankful I am to you! Thank you so much for this awesome script.\n\nI also owe a lot to you for my [Lyft Kernel](https://www.kaggle.com/tarunpaparaju/lyft-competition-understanding-the-data). Your animation code was very important there. So, once again, thank you!",
    "698254": "Thanks @xhlulu !"
  },
  "source": "meta"
}