{
  "id": 93639,
  "title": "input kernel cannot use kernel outputs as a data source",
  "url": "/competitions/freesound-audio-tagging-2019/discussion/93639",
  "author_name": "",
  "post_date": "2019-05-28T23:16:50.935760Z",
  "votes": null,
  "comment_count": 3,
  "views": 0,
  "content": "<p>So, I'm doing everything only using kaggle kernels. The pipeline is such:\n1) <code>prepare_dataset</code> - features from .wav and files labels to .h5\n||\nV\n2) <code>train_model</code> - using h5 datasets from previous one this kernel trains a model (runs longer than an hour, so cannot submit from there)\n||\nV\n3) <code>prepare_submission</code> - fork from <code>train_model</code>; gives <code>submission.csv</code> using model from <code>train_model</code> (kernel (2) outputs are added as a dataset) and prepared dataset in .h5 from <code>prepare_dataset</code></p>\n\n<p>but then on step (3) I cannot submit the 'submission.csv' because <code>Your input kernel [train_model] cannot use kernel outputs as a data source for this competition</code></p>\n\n<p>If I instead of adding kernel (1) to (2) will create a dataset of (1) and add to kernel (2), then I still cannot submit 'submission.csv' because \"<code>train_model</code> uses external data source\"</p>\n\n<p>But n days before I had no problem with submitting from step (2), when the whole process of training and inference was shorter than 1 hour.</p>\n\n<p>So, is there a better (and working) way how to deal with that?</p>",
  "messages": [
    {
      "id": "538649",
      "postDate": "05/28/2019 23:16:50",
      "content": "<p>So, I'm doing everything only using kaggle kernels. The pipeline is such:\n1) <code>prepare_dataset</code> - features from .wav and files labels to .h5\n||\nV\n2) <code>train_model</code> - using h5 datasets from previous one this kernel trains a model (runs longer than an hour, so cannot submit from there)\n||\nV\n3) <code>prepare_submission</code> - fork from <code>train_model</code>; gives <code>submission.csv</code> using model from <code>train_model</code> (kernel (2) outputs are added as a dataset) and prepared dataset in .h5 from <code>prepare_dataset</code></p>\n\n<p>but then on step (3) I cannot submit the 'submission.csv' because <code>Your input kernel [train_model] cannot use kernel outputs as a data source for this competition</code></p>\n\n<p>If I instead of adding kernel (1) to (2) will create a dataset of (1) and add to kernel (2), then I still cannot submit 'submission.csv' because \"<code>train_model</code> uses external data source\"</p>\n\n<p>But n days before I had no problem with submitting from step (2), when the whole process of training and inference was shorter than 1 hour.</p>\n\n<p>So, is there a better (and working) way how to deal with that?</p>",
      "rawMarkdown": "So, I'm doing everything only using kaggle kernels. The pipeline is such:\n1) `prepare_dataset` - features from .wav and files labels to .h5\n||\nV\n2) `train_model` - using h5 datasets from previous one this kernel trains a model (runs longer than an hour, so cannot submit from there)\n||\nV\n3) `prepare_submission` - fork from `train_model`; gives `submission.csv` using model from `train_model` (kernel (2) outputs are added as a dataset) and prepared dataset in .h5 from `prepare_dataset`\n\nbut then on step (3) I cannot submit the 'submission.csv' because ```Your input kernel [train_model] cannot use kernel outputs as a data source for this competition```\n\nIf I instead of adding kernel (1) to (2) will create a dataset of (1) and add to kernel (2), then I still cannot submit 'submission.csv' because \"`train_model` uses external data source\"\n\nBut n days before I had no problem with submitting from step (2), when the whole process of training and inference was shorter than 1 hour.\n\nSo, is there a better (and working) way how to deal with that?",
      "votes": null
    },
    {
      "id": "538680",
      "postDate": "05/29/2019 01:11:33",
      "content": "<p>I create a dataset of the model and add that to a separate kernel that does inference only.</p>",
      "rawMarkdown": "I create a dataset of the model and add that to a separate kernel that does inference only.",
      "votes": null
    },
    {
      "id": "538691",
      "postDate": "05/29/2019 01:49:18",
      "content": "<p>Thanks! Now I see that you've already asked this question month ago. \nIt's a pity there's no comfortable way to do that without downloading and then uploading back the weights</p>",
      "rawMarkdown": "Thanks! Now I see that you've already asked this question month ago. \nIt's a pity there's no comfortable way to do that without downloading and then uploading back the weights",
      "votes": null
    },
    {
      "id": "560748",
      "postDate": "06/25/2019 19:11:48",
      "content": "<p>I trained my model offline, upload it as a dataset and use it in kernel. This kernel only does inference. It has been working for weeks, but now it throws out en error:</p>\n\n<p><code>your kernel can not use kernel output as a data source for this competition</code></p>",
      "rawMarkdown": "I trained my model offline, upload it as a dataset and use it in kernel. This kernel only does inference. It has been working for weeks, but now it throws out en error:\n\n`your kernel can not use kernel output as a data source for this competition`",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 538680,
      "author_name": "tanlikesmath",
      "author_url": "",
      "post_date": "05/29/2019 01:11:33",
      "content": "<p>I create a dataset of the model and add that to a separate kernel that does inference only.</p>",
      "votes": null,
      "replies": [
        {
          "id": 538691,
          "author_name": "b0hd4n",
          "author_url": "",
          "post_date": "05/29/2019 01:49:18",
          "content": "<p>Thanks! Now I see that you've already asked this question month ago. \nIt's a pity there's no comfortable way to do that without downloading and then uploading back the weights</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 560748,
          "author_name": "soulmachine",
          "author_url": "",
          "post_date": "06/25/2019 19:11:48",
          "content": "<p>I trained my model offline, upload it as a dataset and use it in kernel. This kernel only does inference. It has been working for weeks, but now it throws out en error:</p>\n\n<p><code>your kernel can not use kernel output as a data source for this competition</code></p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "538649": "So, I'm doing everything only using kaggle kernels. The pipeline is such:\n1) `prepare_dataset` - features from .wav and files labels to .h5\n||\nV\n2) `train_model` - using h5 datasets from previous one this kernel trains a model (runs longer than an hour, so cannot submit from there)\n||\nV\n3) `prepare_submission` - fork from `train_model`; gives `submission.csv` using model from `train_model` (kernel (2) outputs are added as a dataset) and prepared dataset in .h5 from `prepare_dataset`\n\nbut then on step (3) I cannot submit the 'submission.csv' because ```Your input kernel [train_model] cannot use kernel outputs as a data source for this competition```\n\nIf I instead of adding kernel (1) to (2) will create a dataset of (1) and add to kernel (2), then I still cannot submit 'submission.csv' because \"`train_model` uses external data source\"\n\nBut n days before I had no problem with submitting from step (2), when the whole process of training and inference was shorter than 1 hour.\n\nSo, is there a better (and working) way how to deal with that?",
    "538680": "I create a dataset of the model and add that to a separate kernel that does inference only.",
    "538691": "Thanks! Now I see that you've already asked this question month ago. \nIt's a pity there's no comfortable way to do that without downloading and then uploading back the weights",
    "560748": "I trained my model offline, upload it as a dataset and use it in kernel. This kernel only does inference. It has been working for weeks, but now it throws out en error:\n\n`your kernel can not use kernel output as a data source for this competition`"
  },
  "source": "meta"
}