{
  "id": 190521,
  "title": "Brainstorm: possible differences between public and private test sets",
  "url": "/competitions/riiid-test-answer-prediction/discussion/190521",
  "author_name": "Alex Bader",
  "post_date": "2020-10-12T06:20:28.931000",
  "votes": 3,
  "comment_count": 6,
  "views": 0,
  "content": "<p>My submissions keep failing even though everything works fine on the public test set.</p>\n<p>I've narrowed it down to one of my functions failing, it is supposed to take the previous test_df together with the list of previous answers and increase some counters in a modified version of questions.csv, basically.</p>\n<p>What difference between public and private test sets can people come up with?<br>\nSo far I got:</p>\n<ul>\n<li>private test may include \"lecture\" rows whereas public test does not</li>\n<li>private test may include new users we haven't seen before (didn't actually check whether public test does too)</li>\n</ul>\n<p>Anything else?</p>",
  "messages": [
    {
      "id": 1046963,
      "postDate": "2020-10-12T06:20:28.930Z",
      "content": "<p>My submissions keep failing even though everything works fine on the public test set.</p>\n<p>I've narrowed it down to one of my functions failing, it is supposed to take the previous test_df together with the list of previous answers and increase some counters in a modified version of questions.csv, basically.</p>\n<p>What difference between public and private test sets can people come up with?<br>\nSo far I got:</p>\n<ul>\n<li>private test may include \"lecture\" rows whereas public test does not</li>\n<li>private test may include new users we haven't seen before (didn't actually check whether public test does too)</li>\n</ul>\n<p>Anything else?</p>",
      "rawMarkdown": "My submissions keep failing even though everything works fine on the public test set.\n\nI've narrowed it down to one of my functions failing, it is supposed to take the previous test_df together with the list of previous answers and increase some counters in a modified version of questions.csv, basically.\n\nWhat difference between public and private test sets can people come up with?\nSo far I got:\n- private test may include \"lecture\" rows whereas public test does not\n- private test may include new users we haven't seen before (didn't actually check whether public test does too)\n\nAnything else?",
      "votes": 3
    },
    {
      "id": 1046992,
      "postDate": "2020-10-12T06:57:55.007Z",
      "content": "<p>If you can open-source, then someone can help for sure!</p>",
      "rawMarkdown": "If you can open-source, then someone can help for sure!",
      "replies": [
        {
          "id": 1047007,
          "postDate": "2020-10-12T07:13:39.343Z",
          "content": "<p>Still working on fixing this kernel:<br>\n<a href=\"https://www.kaggle.com/spacelx/2020-r3id-incremental-learning-pytorch-creme?scriptVersionId=44530623\" target=\"_blank\">https://www.kaggle.com/spacelx/2020-r3id-incremental-learning-pytorch-creme?scriptVersionId=44530623</a><br>\nVersion 13 currently running.</p>\n<p>I had assumed so far that <code>len(test_df.iloc[0]['prior_group_answers_correct'])</code> is equal to the length of the previous <code>test_df</code> <em>after removal of rows including information on lectures</em>. Now I'm thinking, is this maybe where I'm wrong and <code>len(test_df.iloc[0]['prior_group_answers_correct'])</code> is actually the length of <code>test_df</code> in its vanilla state right when you get it off the iterator?</p>",
          "rawMarkdown": "Still working on fixing this kernel:\nhttps://www.kaggle.com/spacelx/2020-r3id-incremental-learning-pytorch-creme?scriptVersionId=44530623\nVersion 13 currently running.\n\nI had assumed so far that <code>len(test_df.iloc[0]['prior_group_answers_correct'])</code> is equal to the length of the previous <code>test_df</code> *after removal of rows including information on lectures*. Now I'm thinking, is this maybe where I'm wrong and <code>len(test_df.iloc[0]['prior_group_answers_correct'])</code> is actually the length of <code>test_df</code> in its vanilla state right when you get it off the iterator?",
          "votes": 1
        },
        {
          "id": 1047053,
          "postDate": "2020-10-12T07:58:22.900Z",
          "content": "<p>it is the full length of the original state , i.e. without removing lectures.</p>",
          "rawMarkdown": "it is the full length of the original state , i.e. without removing lectures.",
          "votes": 2
        },
        {
          "id": 1047076,
          "postDate": "2020-10-12T08:41:05.813Z",
          "content": "<p>This discussion is what I am looking for!<br>\nThanks so much :)</p>",
          "rawMarkdown": "This discussion is what I am looking for!\nThanks so much :)"
        },
        {
          "id": 1047089,
          "postDate": "2020-10-12T08:51:34.550Z",
          "content": "<p>Thanks, then I hope my bug is now fixed… was a long search :)</p>",
          "rawMarkdown": "Thanks, then I hope my bug is now fixed... was a long search :)"
        }
      ]
    },
    {
      "id": 1047009,
      "postDate": "2020-10-12T07:15:12.737Z",
      "rawMarkdown": "",
      "isDeleted": true
    }
  ],
  "comments": [
    {
      "id": 1046992,
      "author_name": "Aditya Soni",
      "author_url": "",
      "post_date": "2020-10-12T06:57:55.007000",
      "content": "<p>If you can open-source, then someone can help for sure!</p>",
      "votes": 0,
      "replies": [
        {
          "id": 1047007,
          "author_name": "Alex Bader",
          "author_url": "",
          "post_date": "2020-10-12T07:13:39.343000",
          "content": "<p>Still working on fixing this kernel:<br>\n<a href=\"https://www.kaggle.com/spacelx/2020-r3id-incremental-learning-pytorch-creme?scriptVersionId=44530623\" target=\"_blank\">https://www.kaggle.com/spacelx/2020-r3id-incremental-learning-pytorch-creme?scriptVersionId=44530623</a><br>\nVersion 13 currently running.</p>\n<p>I had assumed so far that <code>len(test_df.iloc[0]['prior_group_answers_correct'])</code> is equal to the length of the previous <code>test_df</code> <em>after removal of rows including information on lectures</em>. Now I'm thinking, is this maybe where I'm wrong and <code>len(test_df.iloc[0]['prior_group_answers_correct'])</code> is actually the length of <code>test_df</code> in its vanilla state right when you get it off the iterator?</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 1047053,
          "author_name": "Yih-Dar SHIEH",
          "author_url": "",
          "post_date": "2020-10-12T07:58:22.900000",
          "content": "<p>it is the full length of the original state , i.e. without removing lectures.</p>",
          "votes": 2,
          "replies": []
        },
        {
          "id": 1047076,
          "author_name": "Tajimi",
          "author_url": "",
          "post_date": "2020-10-12T08:41:05.813000",
          "content": "<p>This discussion is what I am looking for!<br>\nThanks so much :)</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1047089,
          "author_name": "Alex Bader",
          "author_url": "",
          "post_date": "2020-10-12T08:51:34.550000",
          "content": "<p>Thanks, then I hope my bug is now fixed… was a long search :)</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 1047009,
      "author_name": "",
      "author_url": "",
      "post_date": "2020-10-12T07:15:12.737000",
      "content": "",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1046963": "My submissions keep failing even though everything works fine on the public test set.\n\nI've narrowed it down to one of my functions failing, it is supposed to take the previous test_df together with the list of previous answers and increase some counters in a modified version of questions.csv, basically.\n\nWhat difference between public and private test sets can people come up with?\nSo far I got:\n- private test may include \"lecture\" rows whereas public test does not\n- private test may include new users we haven't seen before (didn't actually check whether public test does too)\n\nAnything else?",
    "1046992": "If you can open-source, then someone can help for sure!",
    "1047009": ""
  }
}