{
  "id": 202104,
  "title": "Could organizers provide a larger public test set ?",
  "url": "/competitions/riiid-test-answer-prediction/discussion/202104",
  "author_name": "",
  "post_date": "2020-12-08T10:37:41.074767400Z",
  "votes": 2,
  "comment_count": 2,
  "views": 0,
  "content": "<p>First of all, I would like to say that I like a lot that idea of the API to simulate a live usecase, its really great ! 😊</p>\n<p>Nethertheless, the very small size of the public test set starts to give me a bit of headhache 🤕:<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1161354%2Faf72bd626b76ece53697a64d06a0e069%2FSans%20titre.png?generation=1607423335772885&amp;alt=media\" alt=\"\"></p>\n<p>I am backtesting my model based on the simulator of Caleb <a href=\"https://www.kaggle.com/calebeverett/riiid-mock-test-iterator\" target=\"_blank\">here</a>, which works fine.</p>\n<p>I also checked that I remove the lectures when submitting the predictions, but still, an error appears after less that 1 min of running with the submission which means that there is a case that I am not handling properly that I cannot see.</p>\n<p>This is really hard to understand where the problem is coming from as everything works fine with the simulator and the small official samples, and we don't have access to the error messages of the submission.</p>\n<p>So it would be really great to have a larger sample of public test (maybe somthing like 50k or 100k rows shall be enough), so we can properly test and debugs our solutions.</p>\n<p>Right now, I am spending most of my time trying to debug and it is a bit frustrating, and I am sure I am not the only one in that case 😔</p>",
  "messages": [
    {
      "id": "1105925",
      "postDate": "12/08/2020 10:37:41",
      "content": "<p>First of all, I would like to say that I like a lot that idea of the API to simulate a live usecase, its really great ! 😊</p>\n<p>Nethertheless, the very small size of the public test set starts to give me a bit of headhache 🤕:<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1161354%2Faf72bd626b76ece53697a64d06a0e069%2FSans%20titre.png?generation=1607423335772885&amp;alt=media\" alt=\"\"></p>\n<p>I am backtesting my model based on the simulator of Caleb <a href=\"https://www.kaggle.com/calebeverett/riiid-mock-test-iterator\" target=\"_blank\">here</a>, which works fine.</p>\n<p>I also checked that I remove the lectures when submitting the predictions, but still, an error appears after less that 1 min of running with the submission which means that there is a case that I am not handling properly that I cannot see.</p>\n<p>This is really hard to understand where the problem is coming from as everything works fine with the simulator and the small official samples, and we don't have access to the error messages of the submission.</p>\n<p>So it would be really great to have a larger sample of public test (maybe somthing like 50k or 100k rows shall be enough), so we can properly test and debugs our solutions.</p>\n<p>Right now, I am spending most of my time trying to debug and it is a bit frustrating, and I am sure I am not the only one in that case 😔</p>",
      "rawMarkdown": "First of all, I would like to say that I like a lot that idea of the API to simulate a live usecase, its really great ! 😊\n\nNethertheless, the very small size of the public test set starts to give me a bit of headhache 🤕:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1161354%2Faf72bd626b76ece53697a64d06a0e069%2FSans%20titre.png?generation=1607423335772885&alt=media)\n\nI am backtesting my model based on the simulator of Caleb [here](https://www.kaggle.com/calebeverett/riiid-mock-test-iterator), which works fine.\n\nI also checked that I remove the lectures when submitting the predictions, but still, an error appears after less that 1 min of running with the submission which means that there is a case that I am not handling properly that I cannot see.\n\nThis is really hard to understand where the problem is coming from as everything works fine with the simulator and the small official samples, and we don't have access to the error messages of the submission.\n\nSo it would be really great to have a larger sample of public test (maybe somthing like 50k or 100k rows shall be enough), so we can properly test and debugs our solutions.\n\n Right now, I am spending most of my time trying to debug and it is a bit frustrating, and I am sure I am not the only one in that case 😔",
      "votes": null
    },
    {
      "id": "1105954",
      "postDate": "12/08/2020 11:12:29",
      "content": "<p>I also have the same problem.<br>\n It's very frustrating.</p>",
      "rawMarkdown": "I also have the same problem.\n It's very frustrating.",
      "votes": null
    },
    {
      "id": "1105958",
      "postDate": "12/08/2020 11:15:10",
      "content": "<p>I actually ending up finding my own issue that was a simple line missing for reading lectures… </p>\n<p>I could have cought that directly with a larger public dataset, to bad!</p>",
      "rawMarkdown": "I actually ending up finding my own issue that was a simple line missing for reading lectures... \n\nI could have cought that directly with a larger public dataset, to bad!",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1105954,
      "author_name": "m10515009",
      "author_url": "",
      "post_date": "12/08/2020 11:12:29",
      "content": "<p>I also have the same problem.<br>\n It's very frustrating.</p>",
      "votes": null,
      "replies": [
        {
          "id": 1105958,
          "author_name": "bowaka",
          "author_url": "",
          "post_date": "12/08/2020 11:15:10",
          "content": "<p>I actually ending up finding my own issue that was a simple line missing for reading lectures… </p>\n<p>I could have cought that directly with a larger public dataset, to bad!</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1105925": "First of all, I would like to say that I like a lot that idea of the API to simulate a live usecase, its really great ! 😊\n\nNethertheless, the very small size of the public test set starts to give me a bit of headhache 🤕:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1161354%2Faf72bd626b76ece53697a64d06a0e069%2FSans%20titre.png?generation=1607423335772885&alt=media)\n\nI am backtesting my model based on the simulator of Caleb [here](https://www.kaggle.com/calebeverett/riiid-mock-test-iterator), which works fine.\n\nI also checked that I remove the lectures when submitting the predictions, but still, an error appears after less that 1 min of running with the submission which means that there is a case that I am not handling properly that I cannot see.\n\nThis is really hard to understand where the problem is coming from as everything works fine with the simulator and the small official samples, and we don't have access to the error messages of the submission.\n\nSo it would be really great to have a larger sample of public test (maybe somthing like 50k or 100k rows shall be enough), so we can properly test and debugs our solutions.\n\n Right now, I am spending most of my time trying to debug and it is a bit frustrating, and I am sure I am not the only one in that case 😔",
    "1105954": "I also have the same problem.\n It's very frustrating.",
    "1105958": "I actually ending up finding my own issue that was a simple line missing for reading lectures... \n\nI could have cought that directly with a larger public dataset, to bad!"
  },
  "source": "meta"
}