{
  "id": 173015,
  "title": "Has anyone managed to build a dataset using tf.data API?",
  "url": "/competitions/osic-pulmonary-fibrosis-progression/discussion/173015",
  "author_name": "",
  "post_date": "2020-08-07T13:07:33.531465500Z",
  "votes": 1,
  "comment_count": 2,
  "views": 0,
  "content": "<p>In this competition, I am stuck at building an image-label dataset using tf.data API. images themselves I load but then joining them with FVC data by using pandas API is where I'm stuck at. ANy guidance would help. My goal here is to learn working with the tf.data API.</p>",
  "messages": [
    {
      "id": "961730",
      "postDate": "08/07/2020 13:07:33",
      "content": "<p>In this competition, I am stuck at building an image-label dataset using tf.data API. images themselves I load but then joining them with FVC data by using pandas API is where I'm stuck at. ANy guidance would help. My goal here is to learn working with the tf.data API.</p>",
      "rawMarkdown": "In this competition, I am stuck at building an image-label dataset using tf.data API. images themselves I load but then joining them with FVC data by using pandas API is where I'm stuck at. ANy guidance would help. My goal here is to learn working with the tf.data API.",
      "votes": null
    },
    {
      "id": "963966",
      "postDate": "08/09/2020 13:26:41",
      "content": "<p>Generally, the way to handle this is to build one big pd.DataFrame in a preprocessing step, that contains one row for each of your samples, then you create dataset from that, and call a mapping function to turn the contained PatientID into an image tensor (how you go about creating this image is up to you)</p>\n\n<p>Generally, this would look somewhat like this:</p>\n\n<p>```</p>\n\n<h1>Prepare your dataframe with all the data you will need in the tf.data.DataSet</h1>\n\n<p>data = init_big_pandas_dataframe()</p>\n\n<h1>Input columns for additional meta data, aside from the image information</h1>\n\n<p>input_columns = ...</p>\n\n<h1>Your outcome column</h1>\n\n<p>outcome_columns = ...</p>\n\n<p>x = data['Patient]\nx_meta = data[input_columns ]\ny = data[outcome_columns ]</p>\n\n<p>dataset = tf.data.Dataset.from_tensor_slices(((x, x_meta ), y))</p>\n\n<p>def load_image(inputs, y):\n    patient_id = inputs[0]\n    # Get PatientIMG\n   img = create_patient_img(patient_id)</p>\n\n<p>return (img, inputs[1]), y</p>\n\n<p>dataset = dataset.map(load_image, num_parallel_calls=tf.data.experimental.AUTOTUNE)</p>\n\n<h1>More preprocessing (augments, batching, ...)</h1>\n\n<p>...</p>\n\n<p>return dataset\n```</p>",
      "rawMarkdown": "Generally, the way to handle this is to build one big pd.DataFrame in a preprocessing step, that contains one row for each of your samples, then you create dataset from that, and call a mapping function to turn the contained PatientID into an image tensor (how you go about creating this image is up to you)\n\nGenerally, this would look somewhat like this:\n\n```\n# Prepare your dataframe with all the data you will need in the tf.data.DataSet\ndata = init_big_pandas_dataframe()\n# Input columns for additional meta data, aside from the image information\ninput_columns = ...\n# Your outcome column\noutcome_columns = ...\n\nx = data['Patient]\nx_meta = data[input_columns ]\ny = data[outcome_columns ]\n\ndataset = tf.data.Dataset.from_tensor_slices(((x, x_meta ), y))\n\ndef load_image(inputs, y):\n    patient_id = inputs[0]\n    # Get PatientIMG\n   img = create_patient_img(patient_id)\n\n   return (img, inputs[1]), y\n\ndataset = dataset.map(load_image, num_parallel_calls=tf.data.experimental.AUTOTUNE)\n\n# More preprocessing (augments, batching, ...)\n...\n\nreturn dataset\n```",
      "votes": null
    },
    {
      "id": "964432",
      "postDate": "08/09/2020 21:30:12",
      "content": "<p>i moved the other way around. reading in an image first and then failed to index properly into a pd.Dataframe within a function I wrote to map to a dataset. in other words, I tried to map an image to a collection of FVC each observed week. failed to do so because indexing didnt work due to working with tensors at that point as opposed to native Python structures in dataframes.</p>\n<p>your solution should work as expected, thanks 👍</p>",
      "rawMarkdown": "i moved the other way around. reading in an image first and then failed to index properly into a pd.Dataframe within a function I wrote to map to a dataset. in other words, I tried to map an image to a collection of FVC each observed week. failed to do so because indexing didnt work due to working with tensors at that point as opposed to native Python structures in dataframes.\n\nyour solution should work as expected, thanks 👍",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 963966,
      "author_name": "stadlerm",
      "author_url": "",
      "post_date": "08/09/2020 13:26:41",
      "content": "<p>Generally, the way to handle this is to build one big pd.DataFrame in a preprocessing step, that contains one row for each of your samples, then you create dataset from that, and call a mapping function to turn the contained PatientID into an image tensor (how you go about creating this image is up to you)</p>\n\n<p>Generally, this would look somewhat like this:</p>\n\n<p>```</p>\n\n<h1>Prepare your dataframe with all the data you will need in the tf.data.DataSet</h1>\n\n<p>data = init_big_pandas_dataframe()</p>\n\n<h1>Input columns for additional meta data, aside from the image information</h1>\n\n<p>input_columns = ...</p>\n\n<h1>Your outcome column</h1>\n\n<p>outcome_columns = ...</p>\n\n<p>x = data['Patient]\nx_meta = data[input_columns ]\ny = data[outcome_columns ]</p>\n\n<p>dataset = tf.data.Dataset.from_tensor_slices(((x, x_meta ), y))</p>\n\n<p>def load_image(inputs, y):\n    patient_id = inputs[0]\n    # Get PatientIMG\n   img = create_patient_img(patient_id)</p>\n\n<p>return (img, inputs[1]), y</p>\n\n<p>dataset = dataset.map(load_image, num_parallel_calls=tf.data.experimental.AUTOTUNE)</p>\n\n<h1>More preprocessing (augments, batching, ...)</h1>\n\n<p>...</p>\n\n<p>return dataset\n```</p>",
      "votes": null,
      "replies": [
        {
          "id": 964432,
          "author_name": "dronych",
          "author_url": "",
          "post_date": "08/09/2020 21:30:12",
          "content": "<p>i moved the other way around. reading in an image first and then failed to index properly into a pd.Dataframe within a function I wrote to map to a dataset. in other words, I tried to map an image to a collection of FVC each observed week. failed to do so because indexing didnt work due to working with tensors at that point as opposed to native Python structures in dataframes.</p>\n<p>your solution should work as expected, thanks 👍</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "961730": "In this competition, I am stuck at building an image-label dataset using tf.data API. images themselves I load but then joining them with FVC data by using pandas API is where I'm stuck at. ANy guidance would help. My goal here is to learn working with the tf.data API.",
    "963966": "Generally, the way to handle this is to build one big pd.DataFrame in a preprocessing step, that contains one row for each of your samples, then you create dataset from that, and call a mapping function to turn the contained PatientID into an image tensor (how you go about creating this image is up to you)\n\nGenerally, this would look somewhat like this:\n\n```\n# Prepare your dataframe with all the data you will need in the tf.data.DataSet\ndata = init_big_pandas_dataframe()\n# Input columns for additional meta data, aside from the image information\ninput_columns = ...\n# Your outcome column\noutcome_columns = ...\n\nx = data['Patient]\nx_meta = data[input_columns ]\ny = data[outcome_columns ]\n\ndataset = tf.data.Dataset.from_tensor_slices(((x, x_meta ), y))\n\ndef load_image(inputs, y):\n    patient_id = inputs[0]\n    # Get PatientIMG\n   img = create_patient_img(patient_id)\n\n   return (img, inputs[1]), y\n\ndataset = dataset.map(load_image, num_parallel_calls=tf.data.experimental.AUTOTUNE)\n\n# More preprocessing (augments, batching, ...)\n...\n\nreturn dataset\n```",
    "964432": "i moved the other way around. reading in an image first and then failed to index properly into a pd.Dataframe within a function I wrote to map to a dataset. in other words, I tried to map an image to a collection of FVC each observed week. failed to do so because indexing didnt work due to working with tensors at that point as opposed to native Python structures in dataframes.\n\nyour solution should work as expected, thanks 👍"
  },
  "source": "meta"
}