{
  "id": 192124,
  "title": "Submission scoring error - possible reasons collection",
  "url": "/competitions/riiid-test-answer-prediction/discussion/192124",
  "author_name": "alijs",
  "post_date": "2020-10-20T08:10:25.081000",
  "votes": 219,
  "comment_count": 80,
  "views": 0,
  "content": "<p>It looks that questions about \"Submission scoring error\" are raised almost every day. I think it would save a lot of time to collect the known typical reasons for getting this error.</p>\n<p>So here is the checklist of tricky (and less tricky) things I've observed so far, which could cause the error on submission (while potentially being fine when committing):</p>\n<ul>\n<li>There are 104 rows in example test data for commit, but about 2500000 rows in hidden test set when submitting. Check/estimate if your predictions loop is not using too much time/memory on 2500000 rows.</li>\n<li>There are unseen/new users in hidden test data. This may result in unexpected nulls when merging with features. There is however an unseen user in the example test set, too, so this is less likely to be the problem (if you have checked your solution against the example test data).</li>\n<li>Unseen test rows contains not only questions but also lectures. This means:<ul>\n<li>You must make predictions only for questions, not for lectures - make sure you filter lectures out before predicting.</li>\n<li>If you merge test data e.g. with questions.csv, there will be nulls for rows with lectures (if you didn't filter them out before merge).</li></ul></li>\n<li>If you are using <em>prior_group_responses</em> and/or <em>prior_group_answers_correct</em>:<ul>\n<li>These are available only for the first row of each group - the rest are nulls. There can be also an empty list (for the first group).</li>\n<li>If assigning <em>prior_group_responses/prior_group_answers_correct</em> values to the previous group dataframe - check, if you are doing it before or after filtering out the rows with lectures. They contain values also for lectures rows - <em>prior_group_answers_correct</em> is reported to contain \"-1\" for lecture rows.</li></ul></li>\n<li>Check if you are not messing up the original <em>row_id</em> field values in submission file by merging, by excluding lectures or by other manipulations during feature engineering.</li>\n</ul>\n<p>If you have found some other tricky reason for potentially getting \"Submission scoring error\" on submission, please share it in comments.</p>",
  "messages": [
    {
      "id": 1054845,
      "postDate": "2020-10-20T08:10:25.080Z",
      "content": "<p>It looks that questions about \"Submission scoring error\" are raised almost every day. I think it would save a lot of time to collect the known typical reasons for getting this error.</p>\n<p>So here is the checklist of tricky (and less tricky) things I've observed so far, which could cause the error on submission (while potentially being fine when committing):</p>\n<ul>\n<li>There are 104 rows in example test data for commit, but about 2500000 rows in hidden test set when submitting. Check/estimate if your predictions loop is not using too much time/memory on 2500000 rows.</li>\n<li>There are unseen/new users in hidden test data. This may result in unexpected nulls when merging with features. There is however an unseen user in the example test set, too, so this is less likely to be the problem (if you have checked your solution against the example test data).</li>\n<li>Unseen test rows contains not only questions but also lectures. This means:<ul>\n<li>You must make predictions only for questions, not for lectures - make sure you filter lectures out before predicting.</li>\n<li>If you merge test data e.g. with questions.csv, there will be nulls for rows with lectures (if you didn't filter them out before merge).</li></ul></li>\n<li>If you are using <em>prior_group_responses</em> and/or <em>prior_group_answers_correct</em>:<ul>\n<li>These are available only for the first row of each group - the rest are nulls. There can be also an empty list (for the first group).</li>\n<li>If assigning <em>prior_group_responses/prior_group_answers_correct</em> values to the previous group dataframe - check, if you are doing it before or after filtering out the rows with lectures. They contain values also for lectures rows - <em>prior_group_answers_correct</em> is reported to contain \"-1\" for lecture rows.</li></ul></li>\n<li>Check if you are not messing up the original <em>row_id</em> field values in submission file by merging, by excluding lectures or by other manipulations during feature engineering.</li>\n</ul>\n<p>If you have found some other tricky reason for potentially getting \"Submission scoring error\" on submission, please share it in comments.</p>",
      "rawMarkdown": "It looks that questions about \"Submission scoring error\" are raised almost every day. I think it would save a lot of time to collect the known typical reasons for getting this error.\n\nSo here is the checklist of tricky (and less tricky) things I've observed so far, which could cause the error on submission (while potentially being fine when committing):\n* There are 104 rows in example test data for commit, but about 2500000 rows in hidden test set when submitting. Check/estimate if your predictions loop is not using too much time/memory on 2500000 rows.\n* There are unseen/new users in hidden test data. This may result in unexpected nulls when merging with features. There is however an unseen user in the example test set, too, so this is less likely to be the problem (if you have checked your solution against the example test data).\n* Unseen test rows contains not only questions but also lectures. This means:\n  * You must make predictions only for questions, not for lectures - make sure you filter lectures out before predicting.\n  * If you merge test data e.g. with questions.csv, there will be nulls for rows with lectures (if you didn't filter them out before merge).\n* If you are using *prior_group_responses* and/or *prior_group_answers_correct*:\n  * These are available only for the first row of each group - the rest are nulls. There can be also an empty list (for the first group).\n  * If assigning *prior_group_responses/prior_group_answers_correct* values to the previous group dataframe - check, if you are doing it before or after filtering out the rows with lectures. They contain values also for lectures rows - *prior_group_answers_correct* is reported to contain \"-1\" for lecture rows.\n* Check if you are not messing up the original *row_id* field values in submission file by merging, by excluding lectures or by other manipulations during feature engineering.\n\nIf you have found some other tricky reason for potentially getting \"Submission scoring error\" on submission, please share it in comments.\n",
      "votes": 216
    },
    {
      "id": 1068287,
      "postDate": "2020-11-03T08:48:51.543Z",
      "content": "<p>I shared an emurator for iter-test.<br>\nI hope this notebook helps to reduce \"Submission scoring error\"!</p>\n<p><a href=\"https://www.kaggle.com/its7171/iter-test-emulator\" target=\"_blank\">https://www.kaggle.com/its7171/iter-test-emulator</a></p>",
      "rawMarkdown": "I shared an emurator for iter-test.\nI hope this notebook helps to reduce \"Submission scoring error\"!\n\nhttps://www.kaggle.com/its7171/iter-test-emulator",
      "votes": 17,
      "replies": [
        {
          "id": 1087319,
          "postDate": "2020-11-22T15:54:47.297Z",
          "content": "<p>Hi <a href=\"https://www.kaggle.com/its7171\" target=\"_blank\">@its7171</a> . I've modified my submission framework to be faster and I'm encountering submission scoring error after few minutes (I'd say in the first iterations). I used your wonderful API to debug and everything went well. <br>\nDo you think it can remain some differences between your API and kaggle's one ? thanks</p>",
          "rawMarkdown": "Hi @its7171 . I've modified my submission framework to be faster and I'm encountering submission scoring error after few minutes (I'd say in the first iterations). I used your wonderful API to debug and everything went well. \nDo you think it can remain some differences between your API and kaggle's one ? thanks",
          "votes": 1
        },
        {
          "id": 1090067,
          "postDate": "2020-11-25T03:40:39.880Z",
          "content": "<p>same here💔💔💔</p>",
          "rawMarkdown": "same here💔💔💔"
        },
        {
          "id": 1108128,
          "postDate": "2020-12-10T09:56:11.147Z",
          "content": "<p>As far as I know, I have implemented all the features that were officially announced.<br>\nIf you find the reason of the errors which is ok with my scripts, please let me know.</p>",
          "rawMarkdown": "As far as I know, I have implemented all the features that were officially announced.\nIf you find the reason of the errors which is ok with my scripts, please let me know."
        }
      ]
    },
    {
      "id": 1056282,
      "postDate": "2020-10-21T15:06:13.233Z",
      "content": "<p>Good list!<br>\nBy my observation, it also seems like the group sizes we get from the iterator must be pretty small.</p>\n<p>My code is reasonably performant by now given the amount of feature engineering and parameter updating done, and my last submission timed out again. Performance of my code:</p>\n<ul>\n<li>dummy test commit takes ~30 secs, so the total runtime it literally just feature engineering and sticking the result through a model</li>\n<li>predict group of 1e6 rows in ~25 sec: should give a total runtime ~1 min for 2.5e6 private test rows (would be nice but not possible since maximum group size is supposedly 1000 users)</li>\n<li>predict group of 1000 rows in ~0.4 sec: should give a total runtime ~1000 sec or ~20 min if we're generous</li>\n<li>predict group of 100 rows in ~0.2 sec: should give a total runtime ~2 hours</li>\n<li>predict group of 10 rows in ~0.12 sec: should give a total runtime ~8.5 hours, but add some uncertainty and time for the Kaggle API and it will time out (and does so apparently)</li>\n</ul>\n<p>So I guess we have to assume that the size of the groups we get in the dummy test set is representative for the entire private test set, unfortunately, and at that <em>very</em> small. My takeaway therefore is: be especially efficient with feature engineering on small groups and don't worry about inefficient operations on big groups, it doesn't seem to matter.</p>",
      "rawMarkdown": "Good list!\nBy my observation, it also seems like the group sizes we get from the iterator must be pretty small.\n\nMy code is reasonably performant by now given the amount of feature engineering and parameter updating done, and my last submission timed out again. Performance of my code:\n- dummy test commit takes ~30 secs, so the total runtime it literally just feature engineering and sticking the result through a model\n- predict group of 1e6 rows in ~25 sec: should give a total runtime ~1 min for 2.5e6 private test rows (would be nice but not possible since maximum group size is supposedly 1000 users)\n- predict group of 1000 rows in ~0.4 sec: should give a total runtime ~1000 sec or ~20 min if we're generous\n- predict group of 100 rows in ~0.2 sec: should give a total runtime ~2 hours\n- predict group of 10 rows in ~0.12 sec: should give a total runtime ~8.5 hours, but add some uncertainty and time for the Kaggle API and it will time out (and does so apparently)\n\nSo I guess we have to assume that the size of the groups we get in the dummy test set is representative for the entire private test set, unfortunately, and at that *very* small. My takeaway therefore is: be especially efficient with feature engineering on small groups and don't worry about inefficient operations on big groups, it doesn't seem to matter.",
      "votes": 15,
      "replies": [
        {
          "id": 1069505,
          "postDate": "2020-11-04T14:32:51.050Z",
          "content": "<p>Wow! Thank you for the comment. I was getting submission scoring error for 3 days and i saw this comment. I did so many tests but i didnt consider group size of 1. Changed my code according to that and problem solved. So, there are batches even with 1 sample.</p>",
          "rawMarkdown": "Wow! Thank you for the comment. I was getting submission scoring error for 3 days and i saw this comment. I did so many tests but i didnt consider group size of 1. Changed my code according to that and problem solved. So, there are batches even with 1 sample.",
          "votes": 2
        }
      ]
    },
    {
      "id": 1122129,
      "postDate": "2020-12-22T08:00:04.903Z",
      "content": "<p>I'm not sure this is shared before, but I burned more than 20 submissions for this annoying edge case. Those were my first lines in test set iteration loop.</p>\n<pre><code>question_mask = (df_test['content_type_id'] == 0).values.copy()\ndf_test = df_test[question_mask].reset_index(drop=True)\nprior_group_answers_correct = eval(df_test['prior_group_answers_correct'].iloc[0])\n</code></pre>\n<p>The problem is lecture rows are dropped in df_test and then first row of prior_group_answers_correct is passed to eval. In one of the iterations, first row is actually a lecture so it is dropped here, and eval tried to execute NaN. The correct implementation should be:</p>\n<pre><code>prior_group_answers_correct = eval(df_test['prior_group_answers_correct'].iloc[0])\nquestion_mask = (df_test['content_type_id'] == 0).values.copy()\ndf_test = df_test[question_mask].reset_index(drop=True)\n</code></pre>",
      "rawMarkdown": "I'm not sure this is shared before, but I burned more than 20 submissions for this annoying edge case. Those were my first lines in test set iteration loop.\n\n```\nquestion_mask = (df_test['content_type_id'] == 0).values.copy()\ndf_test = df_test[question_mask].reset_index(drop=True)\nprior_group_answers_correct = eval(df_test['prior_group_answers_correct'].iloc[0])\n```\n\nThe problem is lecture rows are dropped in df_test and then first row of prior_group_answers_correct is passed to eval. In one of the iterations, first row is actually a lecture so it is dropped here, and eval tried to execute NaN. The correct implementation should be:\n\n```\nprior_group_answers_correct = eval(df_test['prior_group_answers_correct'].iloc[0])\nquestion_mask = (df_test['content_type_id'] == 0).values.copy()\ndf_test = df_test[question_mask].reset_index(drop=True)\n```",
      "votes": 6,
      "replies": [
        {
          "id": 1122487,
          "postDate": "2020-12-22T13:24:54.557Z",
          "content": "<p>Thanks for sharing. It seems like the same problem for me, but I hid it in the preprocessing function of the test dataset and burned my eyes trying to find it in the inference. </p>",
          "rawMarkdown": "Thanks for sharing. It seems like the same problem for me, but I hid it in the preprocessing function of the test dataset and burned my eyes trying to find it in the inference. "
        },
        {
          "id": 1125261,
          "postDate": "2020-12-24T14:50:48.670Z",
          "content": "<p>Hello all, <br>\nI exhausted all my limits for straight three days. I am trying user, parts and target aggregations. I seem to be handling new user in the given api data but when submit i get the submission scoring error.<br>\nHers the error<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4947407%2F733a868d55ddf84bdca78640dff2acec%2FScreenshot%20(2).png?generation=1608821206491178&amp;alt=media\" alt=\"\"></p>\n<p>Also i have shared my notebook <br>\n<a href=\"https://www.kaggle.com/ptrikp/part-taregt-aggregations\" target=\"_blank\">https://www.kaggle.com/ptrikp/part-taregt-aggregations</a></p>\n<p>What am i doing wrong here ? <br>\nCan any one try this.. greatly appreciated<br>\nThanks</p>",
          "rawMarkdown": "Hello all, \nI exhausted all my limits for straight three days. I am trying user, parts and target aggregations. I seem to be handling new user in the given api data but when submit i get the submission scoring error.\nHers the error\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4947407%2F733a868d55ddf84bdca78640dff2acec%2FScreenshot%20(2).png?generation=1608821206491178&alt=media)\n\nAlso i have shared my notebook \nhttps://www.kaggle.com/ptrikp/part-taregt-aggregations\n\nWhat am i doing wrong here ? \nCan any one try this.. greatly appreciated\nThanks"
        },
        {
          "id": 1126810,
          "postDate": "2020-12-26T01:30:05.917Z",
          "content": "<p>Hi! I am working on a similar problem. If I find a solution, I'll let you know. </p>",
          "rawMarkdown": "Hi! I am working on a similar problem. If I find a solution, I'll let you know. "
        },
        {
          "id": 1126817,
          "postDate": "2020-12-26T01:51:04.103Z",
          "content": "<p>I made it work by taking dictionary part from this notebook and modifying it.<br>\nRefrence : <a href=\"https://www.kaggle.com/markwijkhuizen/riiid-training-and-prediction-using-a-state\" target=\"_blank\">https://www.kaggle.com/markwijkhuizen/riiid-training-and-prediction-using-a-state</a><br>\n Mine notebook is version 6<br>\n<a href=\"https://www.kaggle.com/ptrikp/part-taregt-aggregations\" target=\"_blank\">https://www.kaggle.com/ptrikp/part-taregt-aggregations</a><br>\nMay be i am using future data as someone pointed out </p>",
          "rawMarkdown": "I made it work by taking dictionary part from this notebook and modifying it.\nRefrence : https://www.kaggle.com/markwijkhuizen/riiid-training-and-prediction-using-a-state\n Mine notebook is version 6\nhttps://www.kaggle.com/ptrikp/part-taregt-aggregations\nMay be i am using future data as someone pointed out "
        },
        {
          "id": 1128992,
          "postDate": "2020-12-28T01:07:55.723Z",
          "content": "<p>Thanks, Pratik. I'll review those works and try to fix my submission code.  As <code>bturan19</code> mentioned:</p>\n<blockquote>\n  <p>In my case; I was concatting the dataframes with all records of a user and new records. When there is no old records for any user in the batch, it was crushing because of the 'answered correctly' column does not exist in the new batch. So, I wrote a simple check, if there is no that columns append a column with na values. It worked for me. </p>\n</blockquote>\n<p>I think it might be the cause. </p>",
          "rawMarkdown": "Thanks, Pratik. I'll review those works and try to fix my submission code.  As `bturan19` mentioned:\n> In my case; I was concatting the dataframes with all records of a user and new records. When there is no old records for any user in the batch, it was crushing because of the 'answered correctly' column does not exist in the new batch. So, I wrote a simple check, if there is no that columns append a column with na values. It worked for me. \n\nI think it might be the cause. ",
          "votes": 1
        }
      ]
    },
    {
      "id": 1066649,
      "postDate": "2020-11-01T23:04:13.333Z",
      "content": "<p>Not exactly error but I had a few submissions with a quite lower score than expected. Turned out that I was feeding the model with columns in different order. It's obvious and at the same time it's easy to fail, specially if working with a sequence model when you have to add the previous correct answers (a new column gets added at the end by default). I know this can be off topic but I didn't find any related discussion. </p>",
      "rawMarkdown": "Not exactly error but I had a few submissions with a quite lower score than expected. Turned out that I was feeding the model with columns in different order. It's obvious and at the same time it's easy to fail, specially if working with a sequence model when you have to add the previous correct answers (a new column gets added at the end by default). I know this can be off topic but I didn't find any related discussion. ",
      "votes": 5
    },
    {
      "id": 1136113,
      "postDate": "2021-01-02T18:28:48.767Z",
      "content": "<p>Hello. My latest submission (ver 22) errors out in 6 minutes. Ver 19 timed out. When I run the test it iteration time is 0.2 seconds consistently. No issues in local test for 1M test data. Getting 3.3 iterations/sec on average.  Is there any way to find you why submission fails. I mean would it be hard to create test set more representative (with missing values and etc) of the real test set so that it would be easier troubleshooting issues? The focus of the competition is on creating the best model, right? Otherwise, a lot of participants seem to be spending resources and time on data quality troubleshooting. I don't think it's the most green friendly approach.</p>",
      "rawMarkdown": "Hello. My latest submission (ver 22) errors out in 6 minutes. Ver 19 timed out. When I run the test it iteration time is 0.2 seconds consistently. No issues in local test for 1M test data. Getting 3.3 iterations/sec on average.  Is there any way to find you why submission fails. I mean would it be hard to create test set more representative (with missing values and etc) of the real test set so that it would be easier troubleshooting issues? The focus of the competition is on creating the best model, right? Otherwise, a lot of participants seem to be spending resources and time on data quality troubleshooting. I don't think it's the most green friendly approach.",
      "votes": 3
    },
    {
      "id": 1058396,
      "postDate": "2020-10-23T16:48:08.700Z",
      "content": "<p>Thanks! This is quite helpful.<br>\nIt's one thing to know about the possible issues, it's another thing to identify which one it could be when you get an error 😄</p>\n<p>I'm sure a lot of competitors will need to burn some submissions before getting their pipeline right.</p>",
      "rawMarkdown": "Thanks! This is quite helpful.\nIt's one thing to know about the possible issues, it's another thing to identify which one it could be when you get an error 😄\n\nI'm sure a lot of competitors will need to burn some submissions before getting their pipeline right.",
      "votes": 3
    },
    {
      "id": 1054908,
      "postDate": "2020-10-20T09:37:28.597Z",
      "content": "<p>Nice collection! Pretty much it sums up all the gotchas i also have seen.</p>\n<p>Also, Would just add to the points above that the given test_sample is quite small in nature ~104 rows and I am expecting anything from 1k-4/5k rows at max in each iteration  over test_set generator. Given the volume of the test_data, you will be looping anywhere from ~625 - 2.5k times and that's what takes ~2 hours as you are doing other stuffs (joins/merges/creation of features etc) inside that loop as well, So all in all, they get added up in total…</p>\n<blockquote>\n  <p>These are available only for the first row of each group - the rest are nulls. There can be also an empty list (for the first group).</p>\n</blockquote>\n<p>It will be only for the very first row of the first group as i have successfully accumulated test data <a href=\"https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/191856\" target=\"_blank\">here</a> and make a sub with 1k chunks being accumulated. </p>\n<blockquote>\n  <p>If assigning prior_group_responses/prior_group_answers_correct values to the previous group dataframe - check, if you are doing it before or after filtering out the rows with lectures. I haven't checked which is the correct way, so I'm handling both cases.</p>\n</blockquote>\n<p>The \"prior_group_answers_correct\" has -1 for videos in it.</p>",
      "rawMarkdown": "Nice collection! Pretty much it sums up all the gotchas i also have seen.\n\nAlso, Would just add to the points above that the given test_sample is quite small in nature ~104 rows and I am expecting anything from 1k-4/5k rows at max in each iteration  over test_set generator. Given the volume of the test_data, you will be looping anywhere from ~625 - 2.5k times and that's what takes ~2 hours as you are doing other stuffs (joins/merges/creation of features etc) inside that loop as well, So all in all, they get added up in total...\n\n>These are available only for the first row of each group - the rest are nulls. There can be also an empty list (for the first group).\n\nIt will be only for the very first row of the first group as i have successfully accumulated test data [here](https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/191856) and make a sub with 1k chunks being accumulated. \n\n>If assigning prior_group_responses/prior_group_answers_correct values to the previous group dataframe - check, if you are doing it before or after filtering out the rows with lectures. I haven't checked which is the correct way, so I'm handling both cases.\n\nThe \"prior_group_answers_correct\" has -1 for videos in it.",
      "votes": 3,
      "replies": [
        {
          "id": 1055148,
          "postDate": "2020-10-20T14:14:51.737Z",
          "content": "<p>Thanks for the additional info! Updated original post about prior_group_answers_correct field having -1 for lectures.</p>\n<p>Regarding</p>\n<blockquote>\n  <p>expecting anything from 1k-4/5k rows at max in each iteration over test_set generator</p>\n</blockquote>\n<p>From the Data section there should be between 1 and 1000 rows in each iteration.</p>\n<p>Edit: Data section indeed talks about 1 to 1000 <strong>users</strong>, so max number of rows could be bigger.</p>",
          "rawMarkdown": "Thanks for the additional info! Updated original post about prior_group_answers_correct field having -1 for lectures.\n\nRegarding\n\n> expecting anything from 1k-4/5k rows at max in each iteration over test_set generator\n\nFrom the Data section there should be between 1 and 1000 rows in each iteration.\n\nEdit: Data section indeed talks about 1 to 1000 **users**, so max number of rows could be bigger.",
          "votes": 2
        },
        {
          "id": 1055180,
          "postDate": "2020-10-20T14:42:58.200Z",
          "content": "<p>Ahh, it's a single transaction ID per user and we can have a streak of 3-4 on AVG ques which I have seen in training data. Hence the upper bound.</p>",
          "rawMarkdown": "Ahh, it's a single transaction ID per user and we can have a streak of 3-4 on AVG ques which I have seen in training data. Hence the upper bound.",
          "votes": 4
        }
      ]
    },
    {
      "id": 1055130,
      "postDate": "2020-10-20T13:59:52.673Z",
      "content": "<p>Helpful list, thanks!</p>\n<blockquote>\n  <p>There are unseen/new users in hidden test data. This may result in unexpected nulls when merging with features. There is however an unseen user in the example test set, too, so this is less likely to be the problem (if you have checked your solution against the example test data).</p>\n</blockquote>\n<p>Small thing to add - this may be a bit specific to a bug I encountered (a silly index duplication issue), but handling more than one unseen user can definitely have different behavior than handling only one. I wasted a few hours debugging this -- my suggestion is to simulate multiple unseen users in the sample API data when testing your pipeline, e.g. by seeing what happens if you just drop user data for some of the other users in the sample data before running your API parsing process.  </p>",
      "rawMarkdown": "Helpful list, thanks!\n\n> There are unseen/new users in hidden test data. This may result in unexpected nulls when merging with features. There is however an unseen user in the example test set, too, so this is less likely to be the problem (if you have checked your solution against the example test data).\n\nSmall thing to add - this may be a bit specific to a bug I encountered (a silly index duplication issue), but handling more than one unseen user can definitely have different behavior than handling only one. I wasted a few hours debugging this -- my suggestion is to simulate multiple unseen users in the sample API data when testing your pipeline, e.g. by seeing what happens if you just drop user data for some of the other users in the sample data before running your API parsing process.  ",
      "votes": 4
    },
    {
      "id": 1133026,
      "postDate": "2020-12-30T21:34:01.863Z",
      "content": "<p>I have two submissions running. The one submitted later failed after 10 minutes. The first one has been running for 4 hours. Could it be the reason two were running simultaneously?</p>",
      "rawMarkdown": "I have two submissions running. The one submitted later failed after 10 minutes. The first one has been running for 4 hours. Could it be the reason two were running simultaneously?",
      "votes": 1
    },
    {
      "id": 1077775,
      "postDate": "2020-11-13T23:01:50.477Z",
      "content": "<p>Does the sample submission: <a href=\"https://www.kaggle.com/sohier/quick-sample-submission\" target=\"_blank\">https://www.kaggle.com/sohier/quick-sample-submission</a> cover the filtering of lectures or what exactly do you guys mean by that?</p>",
      "rawMarkdown": "Does the sample submission: https://www.kaggle.com/sohier/quick-sample-submission cover the filtering of lectures or what exactly do you guys mean by that?",
      "votes": 1
    },
    {
      "id": 1060282,
      "postDate": "2020-10-26T02:33:02.420Z",
      "content": "<p>One of the very possible reason for errors is going to be a timeout error. So I feel that doing df.loc in a loop etc is painfully slow. We can try using numpy here to create new features etc, provided we know the index of the columns from  which lets say a particular feature is derived etc. And wrap the whole thing in numba's jit for a speed boost? Plus is it possible to do group by's on numpy arrays? Has anyone tried it? Ty!</p>",
      "rawMarkdown": "One of the very possible reason for errors is going to be a timeout error. So I feel that doing df.loc in a loop etc is painfully slow. We can try using numpy here to create new features etc, provided we know the index of the columns from  which lets say a particular feature is derived etc. And wrap the whole thing in numba's jit for a speed boost? Plus is it possible to do group by's on numpy arrays? Has anyone tried it? Ty!",
      "votes": 1,
      "replies": [
        {
          "id": 1060333,
          "postDate": "2020-10-26T04:19:57.727Z",
          "content": "<p>Avoid loops. It may not be trivial but can give you better speed.<br>\nAll my successful submissions don't use loops and finish scoring within 30 mins.</p>",
          "rawMarkdown": "Avoid loops. It may not be trivial but can give you better speed.\nAll my successful submissions don't use loops and finish scoring within 30 mins.",
          "votes": 2
        }
      ]
    },
    {
      "id": 1119768,
      "postDate": "2020-12-20T11:07:06.340Z",
      "content": "<blockquote>\n  <p>Check if you are not messing up the original row_id field values in submission file by merging, by excluding lectures or by other manipulations during feature engineering.</p>\n</blockquote>\n<p>Thanks, topic starter and all involved! This is really useful information. I have interested in how the <code>row_id</code> would be the cause of raising scoring error - if anyone can explain this?</p>\n<p>In my case, the main reason for the scoring error is the time of execution, even not memory failure. Because of micro-batches in each iteration, the data preprocessing functions do the same with the small test chunks every time, calling the base, merging, etc. So the challenge is NOT to create algorithms for \"Knowledge Tracing,\" BUT to beat the submission API.  </p>\n<p>And it doesn't fit in my head. Because the API doesn't complicate the task, but the task solution delivery process. Hey guys, you'll need to extract features from the 100M-rows base and then merge it 100000 times with the 25 rows)). And you have time/memory restriction… What the value of submission API for the contest problem solving - the rhetorical question.</p>\n<p>But never the less this is one of the most interesting competition and we'll handle it!</p>",
      "rawMarkdown": "> Check if you are not messing up the original row_id field values in submission file by merging, by excluding lectures or by other manipulations during feature engineering.\n\nThanks, topic starter and all involved! This is really useful information. I have interested in how the `row_id` would be the cause of raising scoring error - if anyone can explain this?\n\nIn my case, the main reason for the scoring error is the time of execution, even not memory failure. Because of micro-batches in each iteration, the data preprocessing functions do the same with the small test chunks every time, calling the base, merging, etc. So the challenge is NOT to create algorithms for \"Knowledge Tracing,\" BUT to beat the submission API.  \n\nAnd it doesn't fit in my head. Because the API doesn't complicate the task, but the task solution delivery process. Hey guys, you'll need to extract features from the 100M-rows base and then merge it 100000 times with the 25 rows)). And you have time/memory restriction... What the value of submission API for the contest problem solving - the rhetorical question.\n\nBut never the less this is one of the most interesting competition and we'll handle it!",
      "votes": 2,
      "replies": [
        {
          "id": 1133175,
          "postDate": "2020-12-31T02:17:14.487Z",
          "content": "<p>I'm curious how to know if my test set process break the memory restriction?  I run it sucessfully and get <code>Submission scoring error</code> after submitting.<br>\nThanks very much.</p>",
          "rawMarkdown": "I'm curious how to know if my test set process break the memory restriction?  I run it sucessfully and get `Submission scoring error` after submitting.\nThanks very much."
        },
        {
          "id": 1133604,
          "postDate": "2020-12-31T11:23:03.943Z",
          "content": "<p>Hi! Perhaps this limitation is not by memory but in code execution time. With the providing test sample, it must be near 550 - 600 ms per iteration for successful submission. You can check it using the python time module.</p>",
          "rawMarkdown": "Hi! Perhaps this limitation is not by memory but in code execution time. With the providing test sample, it must be near 550 - 600 ms per iteration for successful submission. You can check it using the python time module.",
          "votes": 1
        },
        {
          "id": 1133621,
          "postDate": "2020-12-31T11:40:30.643Z",
          "content": "<p>Got it!!!<br>\nEvery iterator on my code need a few minute..<br>\nSo are there vecy many many iterations in real test set ?</p>",
          "rawMarkdown": "Got it!!!\nEvery iterator on my code need a few minute..\nSo are there vecy many many iterations in real test set ?"
        },
        {
          "id": 1133648,
          "postDate": "2020-12-31T12:02:42.310Z",
          "content": "<p>Could i get sucess running status if i have execution time problem ? <br>\nI run my code sucessful, May i have the execution time problem? <br>\nThanks very much!</p>",
          "rawMarkdown": "Could i get sucess running status if i have execution time problem ? \nI run my code sucessful, May i have the execution time problem? \nThanks very much!"
        },
        {
          "id": 1133787,
          "postDate": "2020-12-31T14:26:00.540Z",
          "content": "<p>If you have a runtime issue, you obviously won't be able to successfully submit your solution. In a real test suite, each iteration batch has from 1 to 1000 rows, and a total of 2,500,000 rows. Thus, I think the queue length can be at least about 5000. But if we roughly divide 9 hours (total time restriction) by 550 ms (empiric per iter value), we'll reveal that number of chunks can be about 59000. You can see this guess in the notebook <a href=\"url\" target=\"_blank\">https://www.kaggle.com/tomooinubushi/inference-must-be-0-55-sec-iter</a>. </p>",
          "rawMarkdown": "If you have a runtime issue, you obviously won't be able to successfully submit your solution. In a real test suite, each iteration batch has from 1 to 1000 rows, and a total of 2,500,000 rows. Thus, I think the queue length can be at least about 5000. But if we roughly divide 9 hours (total time restriction) by 550 ms (empiric per iter value), we'll reveal that number of chunks can be about 59000. You can see this guess in the notebook [https://www.kaggle.com/tomooinubushi/inference-must-be-0-55-sec-iter](url). ",
          "votes": 1
        }
      ]
    },
    {
      "id": 1106452,
      "postDate": "2020-12-08T21:12:31.017Z",
      "content": "<p>One question if someone could help or know the answer, <a href=\"https://www.kaggle.com/sohier\" target=\"_blank\">@sohier</a> ?<br>\n<em>(Note that we've followed all advice here and we've already some kernels working fine)</em></p>\n<p>On submission, when you get '<strong>Submission Scoring Error</strong>' after <strong>9h</strong>, does it mean the real root cause is <strong>time out</strong> or really a <strong>scoring issue</strong> (like problem with row_id or value not in 0-1.0 range or …). Is it possible that we have a problem in our model inference after 2h but <strong>it waits</strong> 9h to report such error? </p>\n<p>Our previous kernels were working fine but we've such issue on a new kernel, our simulation tests give us 7h runtime (forecast from 250,000 rows tested) with at least 3GB RAM free on the 13GB. We're not able to trouble shoot and we would like to know if it's a real time out.</p>",
      "rawMarkdown": "One question if someone could help or know the answer, @sohier ?\n*(Note that we've followed all advice here and we've already some kernels working fine)*\n\nOn submission, when you get '**Submission Scoring Error**' after **9h**, does it mean the real root cause is **time out** or really a **scoring issue** (like problem with row_id or value not in 0-1.0 range or ...). Is it possible that we have a problem in our model inference after 2h but **it waits** 9h to report such error? \n\nOur previous kernels were working fine but we've such issue on a new kernel, our simulation tests give us 7h runtime (forecast from 250,000 rows tested) with at least 3GB RAM free on the 13GB. We're not able to trouble shoot and we would like to know if it's a real time out.\n",
      "votes": 2,
      "replies": [
        {
          "id": 1106458,
          "postDate": "2020-12-08T21:16:04.903Z",
          "content": "<p>From my knowledge and experience it would throw out the error instantly, meaning that it would NOT wait 9 hours.</p>",
          "rawMarkdown": "From my knowledge and experience it would throw out the error instantly, meaning that it would NOT wait 9 hours.",
          "votes": 1
        },
        {
          "id": 1106510,
          "postDate": "2020-12-08T22:32:41.427Z",
          "content": "<p>If you get GPU memory error it may wait the 9h until the timeout in a frozen-like state. I have seen this happening in live trainings (you get a prompt about the GPU memory but the cell keeps <em>running</em>).</p>",
          "rawMarkdown": "If you get GPU memory error it may wait the 9h until the timeout in a frozen-like state. I have seen this happening in live trainings (you get a prompt about the GPU memory but the cell keeps _running_).",
          "votes": 4
        },
        {
          "id": 1107497,
          "postDate": "2020-12-09T18:29:12.593Z",
          "content": "<p>After 3 days spent on this issue the problem is solved and root cause looks to be in <strong>Kaggle docker image v90</strong>. This version does affect overall performances on inference. The same inference code running under TF2.3.1/DockerImage<strong>v90</strong>: <strong>Timeout</strong>, TF2.3.1/DockerImage<strong>v89</strong>: OK within <strong>6h</strong>.</p>\n<p>Another simple test we've done:</p>\n<ul>\n<li>TF2.3.1/DockerImage<strong>v89</strong>: Model1 inference = 2h30</li>\n<li>TF2.3.1/DockerImage<strong>v90</strong>: Model1 inference = 3h</li>\n</ul>",
          "rawMarkdown": "After 3 days spent on this issue the problem is solved and root cause looks to be in **Kaggle docker image v90**. This version does affect overall performances on inference. The same inference code running under TF2.3.1/DockerImage**v90**: **Timeout**, TF2.3.1/DockerImage**v89**: OK within **6h**.\n\nAnother simple test we've done:\n- TF2.3.1/DockerImage**v89**: Model1 inference = 2h30\n- TF2.3.1/DockerImage**v90**: Model1 inference = 3h",
          "votes": 5
        },
        {
          "id": 1107517,
          "postDate": "2020-12-09T18:43:17.820Z",
          "content": "<p>How to check docker version of a notebook btw?</p>",
          "rawMarkdown": "How to check docker version of a notebook btw?"
        },
        {
          "id": 1107527,
          "postDate": "2020-12-09T18:49:35.970Z",
          "content": "<p>See execution info, then click on docker image link:</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F698363%2Fded9aab37059e1553f1a733ad3e2f6e6%2Fdocker.png?generation=1607539653198072&amp;alt=media\" alt=\"\"></p>\n<p>It will open Google console with the tag/date version:<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F698363%2Feafa9ab0139f2be3596e0216385dcfb6%2Fdocker2.png?generation=1607539752855134&amp;alt=media\" alt=\"\"></p>",
          "rawMarkdown": "See execution info, then click on docker image link:\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F698363%2Fded9aab37059e1553f1a733ad3e2f6e6%2Fdocker.png?generation=1607539653198072&alt=media)\n\nIt will open Google console with the tag/date version:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F698363%2Feafa9ab0139f2be3596e0216385dcfb6%2Fdocker2.png?generation=1607539752855134&alt=media)",
          "votes": 4
        },
        {
          "id": 1107867,
          "postDate": "2020-12-10T02:55:56.673Z",
          "content": "<p>Thank you. It's very informative.<br>\nDo you know or does anybody know how to change/select docker version?</p>",
          "rawMarkdown": "Thank you. It's very informative.\nDo you know or does anybody know how to change/select docker version?",
          "votes": 2
        },
        {
          "id": 1126219,
          "postDate": "2020-12-25T12:43:57.587Z",
          "content": "<p><a href=\"https://www.kaggle.com/mpware\" target=\"_blank\">@mpware</a> Hello, I am facing exactly the same problem, my notebook reach 9h of execution then spits out a submission scoring error rather than timing out. This is confusing as hell! Since you faced the same problem, is it a time out or a bug in my code ?</p>",
          "rawMarkdown": "@mpware Hello, I am facing exactly the same problem, my notebook reach 9h of execution then spits out a submission scoring error rather than timing out. This is confusing as hell! Since you faced the same problem, is it a time out or a bug in my code ?"
        }
      ]
    },
    {
      "id": 1059416,
      "postDate": "2020-10-25T04:42:47.790Z",
      "content": "<p><a href=\"https://www.kaggle.com/alijs1\" target=\"_blank\">@alijs1</a> : Might be worth adding <a href=\"https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/193030\" target=\"_blank\">this</a> to the list.</p>",
      "rawMarkdown": "@alijs1 : Might be worth adding [this](https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/193030) to the list.",
      "votes": 2,
      "replies": [
        {
          "id": 1059673,
          "postDate": "2020-10-25T10:31:42.370Z",
          "content": "<p>Thanks! Updated the list.</p>",
          "rawMarkdown": "Thanks! Updated the list.",
          "votes": 1
        }
      ]
    },
    {
      "id": 1140345,
      "postDate": "2021-01-06T00:18:24.583Z",
      "content": "<p>A few lessons learned. Hope this will help others.</p>\n<p>Yes you can iterate in your code and do well timewise:).</p>\n<p>Iter time: 0.10601687431335449 0-&gt;18 est: 4.10hrs<br>\nIter time: 0.05148053169250488 1-&gt;27 est: 1.33hrs<br>\nIter time: 0.05518960952758789 2-&gt;26 est: 1.48hrs<br>\nIter time: 0.05421710014343262 3-&gt;33 est: 1.15hrs<br>\nProcessed 104 records in 4 batches</p>\n<p>You just have to:<br>\n1) avoid groupby's. They are terribly slow. Use sorting instead.<br>\n2) Pre-cache features whenever possible <br>\n3) avoid lists and impressions.<br>\n4) if using merges use GPU to get more total memory. Try out njit too.</p>\n<p>Other issues that  I was able to get help from others on.</p>\n<p>1) skip group_num column for the submission. Api doesn't report an error but you get a submission error at the end. <br>\n2) keep rows in the same order as they come from api iterator.</p>\n<p>Burned a few submissions on that. Thanks to everyone who helped!</p>",
      "rawMarkdown": "A few lessons learned. Hope this will help others.\n\nYes you can iterate in your code and do well timewise:).\n\nIter time: 0.10601687431335449 0->18 est: 4.10hrs\nIter time: 0.05148053169250488 1->27 est: 1.33hrs\nIter time: 0.05518960952758789 2->26 est: 1.48hrs\nIter time: 0.05421710014343262 3->33 est: 1.15hrs\nProcessed 104 records in 4 batches\n\nYou just have to:\n1) avoid groupby's. They are terribly slow. Use sorting instead.\n2) Pre-cache features whenever possible \n3) avoid lists and impressions.\n4) if using merges use GPU to get more total memory. Try out njit too.\n\n\nOther issues that  I was able to get help from others on.\n\n1) skip group_num column for the submission. Api doesn't report an error but you get a submission error at the end. \n2) keep rows in the same order as they come from api iterator.\n\nBurned a few submissions on that. Thanks to everyone who helped!\n"
    },
    {
      "id": 1137013,
      "postDate": "2021-01-03T15:48:02.550Z",
      "content": "<p>Hello everyone. I would like to share my iteration times for a provided small test set </p>\n<p>Time iter: 0.8021116256713867  (18 records) 0.20076537132263184<em>2500000/(18</em>60*60) ~= 7.74 hrs<br>\nTime iter: 0.22262358665466309 (26 record) -&gt; 0.22262358665466309 *2500000/(26<em>60</em>60) ~=5.946 hrs<br>\nTime iter: 0.23348522186279297 (33 records) --&gt; 0.23348522186279297 <em>2500000/(33</em>60*60) ~= 4.91340955098470054713 hours based on last group with 33 records </p>\n<p>Processed 104 records in 4 batches</p>\n<p>So based on any of the above batches (scratch the first one) the total running time should be between 5 and 8 hours. The larger the groups are the greater the processing throughput of records/sec is.</p>\n<p>Also according to this post<br>\n<a href=\"https://www.kaggle.com/tomooinubushi/inference-must-be-0-55-sec-iter\" target=\"_blank\">https://www.kaggle.com/tomooinubushi/inference-must-be-0-55-sec-iter</a><br>\n0.2 - 0.3 sec per iteration should be sufficient to finish in 9 hours.</p>\n<p>I ran local test with 1M records and it finished in 53m<br>\nThe latest submission has been running for 7 hours. <br>\nI am trying to find any possible way to improve the run time.</p>\n<p>Does anyone else have similar times on the test data and have a successful submission?<br>\nAny help is much appreciated! </p>\n<p>Thank you.</p>\n<p>P.S. The model is dynamic and a bit more involved and thus requires iteration. The earlier version of the code didn't use iterations but execution was not much faster. </p>",
      "rawMarkdown": "Hello everyone. I would like to share my iteration times for a provided small test set \n\nTime iter: 0.8021116256713867 <-- some initialization happen for the first group\nTime iter: 0.20076537132263184 -> (18 records) 0.20076537132263184*2500000/(18*60*60) ~= 7.74 hrs\nTime iter: 0.22262358665466309 (26 record) -> 0.22262358665466309 *2500000/(26*60*60) ~=5.946 hrs\nTime iter: 0.23348522186279297 (33 records) --> 0.23348522186279297 *2500000/(33*60*60) ~= 4.91340955098470054713 hours based on last group with 33 records \n\nProcessed 104 records in 4 batches\n\nSo based on any of the above batches (scratch the first one) the total running time should be between 5 and 8 hours. The larger the groups are the greater the processing throughput of records/sec is.\n\nAlso according to this post\nhttps://www.kaggle.com/tomooinubushi/inference-must-be-0-55-sec-iter\n0.2 - 0.3 sec per iteration should be sufficient to finish in 9 hours.\n\nI ran local test with 1M records and it finished in 53m\nThe latest submission has been running for 7 hours. \nI am trying to find any possible way to improve the run time.\n\nDoes anyone else have similar times on the test data and have a successful submission?\nAny help is much appreciated! \n\nThank you.\n\nP.S. The model is dynamic and a bit more involved and thus requires iteration. The earlier version of the code didn't use iterations but execution was not much faster. ",
      "replies": [
        {
          "id": 1137146,
          "postDate": "2021-01-03T17:59:52.317Z",
          "content": "<p>Update… I applied a few code optimization. Moving to cupa/numpy lowered iteration times:</p>\n<p>load_lkps: 24.306s<br>\nIter time: 0.9248003959655762 0-&gt;18 est: 35.69hrs (high due to initial model initialization)<br>\nIter time: 0.1260981559753418 1-&gt;27 est: 3.25hrs<br>\nIter time: 0.12287306785583496 2-&gt;26 est: 3.29hrs<br>\nIter time: 0.12256026268005371 3-&gt;33 est: 2.58hrs<br>\nProcessed 104 records in 4 batches </p>\n<p>Fingers crossed:).</p>\n<p>P.S. I used the following formula for running time estimate<br>\nprint(f'Iter time: {time.time()-start} {test_input.iloc[0][GRP_NUM]}-&gt;{len(test_input)} est: {(time.time()-start)<em>2_500_000/(len(test_input)</em>60*60):.2f}hrs')</p>",
          "rawMarkdown": "Update... I applied a few code optimization. Moving to cupa/numpy lowered iteration times:\n\nload_lkps: 24.306s\nIter time: 0.9248003959655762 0->18 est: 35.69hrs (high due to initial model initialization)\nIter time: 0.1260981559753418 1->27 est: 3.25hrs\nIter time: 0.12287306785583496 2->26 est: 3.29hrs\nIter time: 0.12256026268005371 3->33 est: 2.58hrs\nProcessed 104 records in 4 batches \n\nFingers crossed:).\n\nP.S. I used the following formula for running time estimate\nprint(f'Iter time: {time.time()-start} {test_input.iloc[0][GRP_NUM]}->{len(test_input)} est: {(time.time()-start)*2_500_000/(len(test_input)*60*60):.2f}hrs')\n"
        }
      ]
    },
    {
      "id": 1135717,
      "postDate": "2021-01-02T13:00:36.440Z",
      "content": "<p>Can we assume that at a user level for each batch there is only a single unique task_container_id?</p>",
      "rawMarkdown": "Can we assume that at a user level for each batch there is only a single unique task_container_id?"
    },
    {
      "id": 1133315,
      "postDate": "2020-12-31T05:30:46.823Z",
      "content": "<p>I got the same error after submit and I have a question…<br>\nCould it prove no memory problem and no feature engineering problem If i run predict of test set and submit sucessful ?</p>\n<pre><code>env = riiideducation.make_env()\niter_test = env.iter_test()\nfor (current_df, sample_prediction_df) in iter_test:\n    current_df = data_process(current_df) # feature engineering\n    current_df['answered_correctly'] = model.predict(current_df[features]) # model predict \n    current_df['answered_correctly'].fillna(0.5, inplace = True)\n    current_df['answered_correctly'][current_df['answered_correctly'] &gt; 1] = 1\n    current_df['answered_correctly'][current_df['answered_correctly'] &lt; 0] = 0\n    env.predict(current_df.loc[:, ['row_id', 'answered_correctly']]) # submit \n</code></pre>\n<p>And I get sucess of Execution Info.<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1811540%2F477c4af79a720ceef31c6888fdc15d78%2Fsubmit-sucess.png?generation=1609382264895415&amp;alt=media\" alt=\"\"><br>\nThanks very much.</p>",
      "rawMarkdown": "I got the same error after submit and I have a question...\nCould it prove no memory problem and no feature engineering problem If i run predict of test set and submit sucessful ?\n```\nenv = riiideducation.make_env()\niter_test = env.iter_test()\nfor (current_df, sample_prediction_df) in iter_test:\n    current_df = data_process(current_df) # feature engineering\n    current_df['answered_correctly'] = model.predict(current_df[features]) # model predict \n    current_df['answered_correctly'].fillna(0.5, inplace = True)\n    current_df['answered_correctly'][current_df['answered_correctly'] > 1] = 1\n    current_df['answered_correctly'][current_df['answered_correctly'] < 0] = 0\n    env.predict(current_df.loc[:, ['row_id', 'answered_correctly']]) # submit \n```\nAnd I get sucess of Execution Info.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1811540%2F477c4af79a720ceef31c6888fdc15d78%2Fsubmit-sucess.png?generation=1609382264895415&alt=media)\nThanks very much.",
      "replies": [
        {
          "id": 1135631,
          "postDate": "2021-01-02T11:44:25.643Z",
          "content": "<p>I got it from <a href=\"https://www.kaggle.com/tomooinubushi/inference-must-be-0-55-sec-iter\" target=\"_blank\">https://www.kaggle.com/tomooinubushi/inference-must-be-0-55-sec-iter</a>.</p>",
          "rawMarkdown": "I got it from https://www.kaggle.com/tomooinubushi/inference-must-be-0-55-sec-iter."
        }
      ]
    },
    {
      "id": 1128934,
      "postDate": "2020-12-27T22:34:23.793Z",
      "content": "<p>Somehow dropping rows requires an index reset, else I will get a \"submission scoring error\". I do not need to reset index when the exact same code was used in offline preprocessing.</p>\n<pre><code>df.reset_index(drop=True, inplace=True)  # only required for submission notebook\ndf['timestamp'].fillna(0, inplace=True)\nmask = (df['content_type_id'] == 1) &amp; (df['timestamp'] &lt; 2000000)\ndf.drop(index=df[mask].index, inplace=True)\n</code></pre>",
      "rawMarkdown": "Somehow dropping rows requires an index reset, else I will get a \"submission scoring error\". I do not need to reset index when the exact same code was used in offline preprocessing.\n\n```\ndf.reset_index(drop=True, inplace=True)  # only required for submission notebook\ndf['timestamp'].fillna(0, inplace=True)\nmask = (df['content_type_id'] == 1) & (df['timestamp'] < 2000000)\ndf.drop(index=df[mask].index, inplace=True)\n```",
      "replies": [
        {
          "id": 1133160,
          "postDate": "2020-12-31T01:54:36.830Z",
          "content": "<p>I know drop <code>df['content_type_id'] == 1</code>, why is conditation <code>df['timestamp'] &lt; 2000000</code>?<br>\nThanks vecy much.</p>",
          "rawMarkdown": "I know drop `df['content_type_id'] == 1`, why is conditation `df['timestamp'] < 2000000`?\nThanks vecy much."
        }
      ]
    },
    {
      "id": 1121131,
      "postDate": "2020-12-21T11:51:54.453Z",
      "content": "<p>I finally figured out why my submission code was reporting an error, I guessed it was the leacture, but I couldn't find the exact error location, thanks!</p>\n<blockquote>\n  <p>If assigning prior_group_responses/prior_group_answers_correct values to the previous group dataframe - check, if you are doing it before or after filtering out the rows with lectures. They contain values also for lectures rows - prior_group_answers_correct is reported to contain \"-1\" for lecture rows.</p>\n</blockquote>",
      "rawMarkdown": "I finally figured out why my submission code was reporting an error, I guessed it was the leacture, but I couldn't find the exact error location, thanks!\n> If assigning prior_group_responses/prior_group_answers_correct values to the previous group dataframe - check, if you are doing it before or after filtering out the rows with lectures. They contain values also for lectures rows - prior_group_answers_correct is reported to contain \"-1\" for lecture rows."
    },
    {
      "id": 1101228,
      "postDate": "2020-12-03T18:22:40.143Z",
      "content": "<p>Not quite related but might be useful to others,</p>\n<p>A good way to use kaggle kernels effectively is to do the below, </p>\n<ul>\n<li>Create independent set of snips that creates your features and caches them as well.</li>\n<li>Call these independent set of scripts one by one. It's likely that you can get them created in 16 gigs when you focus on each one of them separately rather than as a whole in one kernel itself.</li>\n<li>This trick will save you a good deal of RAM.</li>\n</ul>\n<p>Hope it helps! [Just in case you have limited access to compute like me, the above works like a charm]</p>",
      "rawMarkdown": "Not quite related but might be useful to others,\n\nA good way to use kaggle kernels effectively is to do the below, \n\n- Create independent set of snips that creates your features and caches them as well.\n- Call these independent set of scripts one by one. It's likely that you can get them created in 16 gigs when you focus on each one of them separately rather than as a whole in one kernel itself.\n- This trick will save you a good deal of RAM.\n\nHope it helps! [Just in case you have limited access to compute like me, the above works like a charm]",
      "replies": [
        {
          "id": 1105672,
          "postDate": "2020-12-08T04:35:00.727Z",
          "content": "<p>You mean, functionize?</p>",
          "rawMarkdown": "You mean, functionize?"
        }
      ]
    },
    {
      "id": 1099013,
      "postDate": "2020-12-02T02:46:40.540Z",
      "content": "<p>There are 104 rows in example test data for commit, but about 2500000 rows in hidden test set when submitting. Check/estimate if your predictions loop is not using too much time/memory on 2500000 rows.</p>\n<p>Following above-mentioned, how can i check if i get memory error when i submit on 2500000 hidden test set?<br>\nBefore using additional features, i can submit successfully. <br>\nAfter using some new features , i get submission error after 2~3 hours, so i doubt it is memory error, but how can i check it?<br>\nSomeone has any trick to prevent memory error?</p>",
      "rawMarkdown": "There are 104 rows in example test data for commit, but about 2500000 rows in hidden test set when submitting. Check/estimate if your predictions loop is not using too much time/memory on 2500000 rows.\n\nFollowing above-mentioned, how can i check if i get memory error when i submit on 2500000 hidden test set?\nBefore using additional features, i can submit successfully. \nAfter using some new features , i get submission error after 2~3 hours, so i doubt it is memory error, but how can i check it?\nSomeone has any trick to prevent memory error?",
      "replies": [
        {
          "id": 1099116,
          "postDate": "2020-12-02T05:15:21.593Z",
          "content": "<p>enclose your feature engineering step in try and do a dummy prediction in the exception, if your code runs then the error is due to some instability of the feature engineering, otherwise it's from memory.</p>",
          "rawMarkdown": "enclose your feature engineering step in try and do a dummy prediction in the exception, if your code runs then the error is due to some instability of the feature engineering, otherwise it's from memory.",
          "votes": 2
        },
        {
          "id": 1133190,
          "postDate": "2020-12-31T02:36:41.667Z",
          "content": "<p>I have a problem…<br>\nCould it prove no memory problem and no feature engineering problem If i run predict of test set and submit sucessful ?</p>\n<pre><code>env = riiideducation.make_env()\niter_test = env.iter_test()\nfor (current_df, sample_prediction_df) in iter_test:\n    current_df = data_process(current_df) # feature engineering\n    current_df['answered_correctly'] = model.predict(current_df[features]) # model predict \n    current_df['answered_correctly'].fillna(0.5, inplace = True)\n    current_df['answered_correctly'][current_df['answered_correctly'] &gt; 1] = 1\n    current_df['answered_correctly'][current_df['answered_correctly'] &lt; 0] = 0\n    env.predict(current_df.loc[:, ['row_id', 'answered_correctly']]) # submit \n</code></pre>\n<p>And I get sucess of Execution Info.<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1811540%2F477c4af79a720ceef31c6888fdc15d78%2Fsubmit-sucess.png?generation=1609382264895415&amp;alt=media\" alt=\"\"><br>\nThanks very much.</p>",
          "rawMarkdown": "I have a problem...\nCould it prove no memory problem and no feature engineering problem If i run predict of test set and submit sucessful ?\n```\nenv = riiideducation.make_env()\niter_test = env.iter_test()\nfor (current_df, sample_prediction_df) in iter_test:\n    current_df = data_process(current_df) # feature engineering\n    current_df['answered_correctly'] = model.predict(current_df[features]) # model predict \n    current_df['answered_correctly'].fillna(0.5, inplace = True)\n    current_df['answered_correctly'][current_df['answered_correctly'] > 1] = 1\n    current_df['answered_correctly'][current_df['answered_correctly'] < 0] = 0\n    env.predict(current_df.loc[:, ['row_id', 'answered_correctly']]) # submit \n```\nAnd I get sucess of Execution Info.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1811540%2F477c4af79a720ceef31c6888fdc15d78%2Fsubmit-sucess.png?generation=1609382264895415&alt=media)\nThanks very much."
        }
      ]
    },
    {
      "id": 1098677,
      "postDate": "2020-12-01T19:14:03.500Z",
      "content": "<p></p>\n<p>Another situation is that we get multiple task_containers for a single user_id, however only one of the received task_countainers for that user_id would be questions and all others would be lectures. </p>\n<p>Update: This doesn't seem to solve the issue</p>",
      "rawMarkdown": "~~Don't know if someone else faced this issue or not, I think we get a new user_id with first interaction as a lecture. My submission has been failing for a while so I experimented and found this possible reason. ~~\n\nAnother situation is that we get multiple task_containers for a single user_id, however only one of the received task_countainers for that user_id would be questions and all others would be lectures. \n\nUpdate: This doesn't seem to solve the issue"
    },
    {
      "id": 1094880,
      "postDate": "2020-11-29T03:54:06.687Z",
      "content": "<p>Hi everybody! Is there a specific limitation on using cuda? All of my submission failed. I am running locally with 1.5G GPU memory and 2.2RAM stable with ETA for 1M records at about 1hr58 minutes. But I've already been running longer than during submission. I tested unseen users - no problem. I generate content_id randomly - no problem. However, I have not tested NEW content_id. I am stuck. Any help/hints would be very much appreciated. This is my first kaggle competition. Please be gentle:).</p>",
      "rawMarkdown": "Hi everybody! Is there a specific limitation on using cuda? All of my submission failed. I am running locally with 1.5G GPU memory and 2.2RAM stable with ETA for 1M records at about 1hr58 minutes. But I've already been running longer than during submission. I tested unseen users - no problem. I generate content_id randomly - no problem. However, I have not tested NEW content_id. I am stuck. Any help/hints would be very much appreciated. This is my first kaggle competition. Please be gentle:).",
      "replies": [
        {
          "id": 1094885,
          "postDate": "2020-11-29T03:59:28.973Z",
          "content": "<blockquote>\n  <p>Hi everybody! Is there a specific limitation on using cuda?</p>\n</blockquote>\n<p>No, there's none. My sub runs in ~2.15-3 hours. Ensure you are handling new users properly, videos/non-videos, nan rows etc..</p>",
          "rawMarkdown": ">Hi everybody! Is there a specific limitation on using cuda?\n\nNo, there's none. My sub runs in ~2.15-3 hours. Ensure you are handling new users properly, videos/non-videos, nan rows etc.."
        },
        {
          "id": 1094895,
          "postDate": "2020-11-29T04:16:18.203Z",
          "content": "<p>Thank you, Aditya! I am not checking for nas in user_id, conten_id, content_type_id  task_container_id and etc. Are there test records with missing core key values? </p>",
          "rawMarkdown": "Thank you, Aditya! I am not checking for nas in user_id, conten_id, content_type_id  task_container_id and etc. Are there test records with missing core key values? "
        }
      ]
    },
    {
      "id": 1094385,
      "postDate": "2020-11-28T15:17:42.977Z",
      "content": "<p>My submission section takes 700ms in example test and it works fine. But when submit I got error in 5 minutes.</p>",
      "rawMarkdown": "My submission section takes 700ms in example test and it works fine. But when submit I got error in 5 minutes.",
      "replies": [
        {
          "id": 1094599,
          "postDate": "2020-11-28T18:52:14.950Z",
          "content": "<p>Yeah, there are some groups that full of users that we didn't see in train dataset..</p>",
          "rawMarkdown": "Yeah, there are some groups that full of users that we didn't see in train dataset.."
        },
        {
          "id": 1121574,
          "postDate": "2020-12-21T18:43:24.687Z",
          "content": "<p>Hi, <a href=\"https://www.kaggle.com/bturan19\" target=\"_blank\">@bturan19</a>. How did you fix it? I have the same problem with the last submission. Performed 400ms on the test example, all 4 iter pieces went smoothly but got an immediate error. </p>",
          "rawMarkdown": "Hi, @bturan19. How did you fix it? I have the same problem with the last submission. Performed 400ms on the test example, all 4 iter pieces went smoothly but got an immediate error. "
        },
        {
          "id": 1126369,
          "postDate": "2020-12-25T14:53:20.157Z",
          "content": "<p>In my case; I was concatting the dataframes with all records of a user and new records. When there is no old records for any user in the batch, it was crushing because of the 'answered correctly' column does not exist in the new batch. So, I wrote a simple check, if there is no that columns append a column with na values. It worked for me.</p>",
          "rawMarkdown": "In my case; I was concatting the dataframes with all records of a user and new records. When there is no old records for any user in the batch, it was crushing because of the 'answered correctly' column does not exist in the new batch. So, I wrote a simple check, if there is no that columns append a column with na values. It worked for me.",
          "votes": 1
        },
        {
          "id": 1126807,
          "postDate": "2020-12-26T01:26:55.673Z",
          "content": "<p>Thanks a lot. I'll check my old code again 👍 I burned almost 10 submissions chasing the rabbit))</p>",
          "rawMarkdown": "Thanks a lot. I'll check my old code again 👍 I burned almost 10 submissions chasing the rabbit))"
        }
      ]
    },
    {
      "id": 1078269,
      "postDate": "2020-11-14T15:19:44.147Z",
      "content": "<p>The column names of my submission were \"0\" and \"1\". I changed them into \"row_id\" and \"answered_correctly\" to get rid of the submission scoring error.</p>",
      "rawMarkdown": "The column names of my submission were \"0\" and \"1\". I changed them into \"row_id\" and \"answered_correctly\" to get rid of the submission scoring error."
    },
    {
      "id": 1069983,
      "postDate": "2020-11-05T07:29:54.947Z",
      "content": "<p>I have a problem when submitting. I can't select the output file, which shows' no output files found ', but my kernel has \"submission.csv\".</p>",
      "rawMarkdown": "I have a problem when submitting. I can't select the output file, which shows' no output files found ', but my kernel has \"submission.csv\".",
      "replies": [
        {
          "id": 1069990,
          "postDate": "2020-11-05T07:48:28.527Z",
          "content": "<p>You have to save your notebook. If you do \"Quick Save\", then you have to select to save your output in advance settings (in the submission screen). \"Save &amp; Run All\" saves your outputs by default.</p>",
          "rawMarkdown": "You have to save your notebook. If you do \"Quick Save\", then you have to select to save your output in advance settings (in the submission screen). \"Save & Run All\" saves your outputs by default.\n\n",
          "votes": 1
        }
      ]
    },
    {
      "id": 1065337,
      "postDate": "2020-10-31T07:18:28.863Z",
      "content": "<p>Also, the test data will change after the competition right ? </p>\n<ol>\n<li>If yes, then what is the expected number of rows in the new test set after the competition and how many new users are we expected to see ( Well actually in both current test set and new ) ? </li>\n<li>If No, then Wont some people just take all the test set and then store them after commit, since they tell the answer for the previous batch (i.e score around 99% as they wont know the answer only for last batch)?</li>\n</ol>",
      "rawMarkdown": "Also, the test data will change after the competition right ? \n1. If yes, then what is the expected number of rows in the new test set after the competition and how many new users are we expected to see ( Well actually in both current test set and new ) ? \n2. If No, then Wont some people just take all the test set and then store them after commit, since they tell the answer for the previous batch (i.e score around 99% as they wont know the answer only for last batch)?",
      "replies": [
        {
          "id": 1065381,
          "postDate": "2020-10-31T08:15:10.267Z",
          "content": "<p>I strongly suggest, start reading the discussion posts/ data desc / kaggle EDA kernel's etc  first before asking the same thing! </p>\n<p>You don't have access to whole test set at once, you only have access to it via a generator which you can't control, just iterate over, make your press on the required columns and we are done. </p>\n<p>And we cannot commit the log of the test set as we don't have access to it when we commit on interactive mode. The dummy generator (for example_test.csvc) is replaced by the real test set generator in the commit mode which is run by kaggle in the backend in an isolated environment.<br>\nThat's it.</p>",
          "rawMarkdown": "I strongly suggest, start reading the discussion posts/ data desc / kaggle EDA kernel's etc  first before asking the same thing! \n\nYou don't have access to whole test set at once, you only have access to it via a generator which you can't control, just iterate over, make your press on the required columns and we are done. \n\nAnd we cannot commit the log of the test set as we don't have access to it when we commit on interactive mode. The dummy generator (for example_test.csvc) is replaced by the real test set generator in the commit mode which is run by kaggle in the backend in an isolated environment.\nThat's it.",
          "votes": -1
        }
      ]
    },
    {
      "id": 1059116,
      "postDate": "2020-10-24T16:36:18.903Z",
      "content": "<p>wonderful job !<br>\nMay I ask another question: does input files(train.csv、question.csv、etc) change when it rerun my notebook?</p>",
      "rawMarkdown": "wonderful job !\nMay I ask another question: does input files(train.csv、question.csv、etc) change when it rerun my notebook?",
      "replies": [
        {
          "id": 1059674,
          "postDate": "2020-10-25T10:33:45.533Z",
          "content": "<p>No, only test data changes. Everything else (including train.csv, questions.csv) stays the same.</p>",
          "rawMarkdown": "No, only test data changes. Everything else (including train.csv, questions.csv) stays the same."
        }
      ]
    },
    {
      "id": 1058711,
      "postDate": "2020-10-24T06:21:57.110Z",
      "content": "<p>Hello, I want to ask how does kaggle works while predict unseen dataset?</p>\n<ol>\n<li>Just exchange test_sample.csv to unseen dataset(For feature engineering all the same by using of user's code)<br>\n2.after fitted the model with user's code, then predict unseen dataset with official code<br>\n3.others way<br>\nWhich one is more appropriate？<br>\nThanks</li>\n</ol>",
      "rawMarkdown": "Hello, I want to ask how does kaggle works while predict unseen dataset?\n1. Just exchange test_sample.csv to unseen dataset(For feature engineering all the same by using of user's code)\n2.after fitted the model with user's code, then predict unseen dataset with official code\n3.others way\nWhich one is more appropriate？\nThanks",
      "replies": [
        {
          "id": 1059682,
          "postDate": "2020-10-25T10:41:04.987Z",
          "content": "<p>When you iterate through test data using <em>env.iter_test()</em> during commit, the iterator will return  104 example rows. When submitting, this iterator will return about 2500000 unseen test data.</p>\n<p>Don't use <em>example_test.csv</em> file directly in your code, instead use <em>env.iter_test()</em> iterator from <em>riiideducation</em> module when predicting.</p>",
          "rawMarkdown": "When you iterate through test data using *env.iter_test()* during commit, the iterator will return  104 example rows. When submitting, this iterator will return about 2500000 unseen test data.\n\nDon't use *example_test.csv* file directly in your code, instead use *env.iter_test()* iterator from *riiideducation* module when predicting.",
          "votes": 1
        },
        {
          "id": 1064119,
          "postDate": "2020-10-29T18:24:46.347Z",
          "content": "<p>Thanks for this! It's really helpful.</p>",
          "rawMarkdown": "Thanks for this! It's really helpful."
        },
        {
          "id": 1064386,
          "postDate": "2020-10-30T04:18:32.417Z",
          "content": "<p>hi, May I set up the data size of each iteration? How?</p>",
          "rawMarkdown": "hi, May I set up the data size of each iteration? How?"
        },
        {
          "id": 1064414,
          "postDate": "2020-10-30T05:09:09.050Z",
          "content": "<p>For test data we can't, it's preset already. For training data, while reading there's chunksize in read_csv or something equivalent can be used as well.</p>",
          "rawMarkdown": "For test data we can't, it's preset already. For training data, while reading there's chunksize in read_csv or something equivalent can be used as well."
        }
      ]
    },
    {
      "id": 1056021,
      "postDate": "2020-10-21T11:06:05.417Z",
      "content": "<p>Helpful list, especially for a beginner like me. Thanks!</p>",
      "rawMarkdown": "Helpful list, especially for a beginner like me. Thanks!"
    },
    {
      "id": 1125260,
      "postDate": "2020-12-24T14:49:21.870Z",
      "rawMarkdown": "",
      "isDeleted": true
    },
    {
      "id": 1055972,
      "postDate": "2020-10-21T10:11:59.677Z",
      "rawMarkdown": "",
      "isDeleted": true
    },
    {
      "id": 1736634,
      "postDate": "2022-03-27T14:52:38.697Z",
      "content": "<p>thanks<br>\nreally helpful</p>",
      "rawMarkdown": "thanks\nreally helpful"
    },
    {
      "id": 1059121,
      "postDate": "2020-10-24T16:44:56.703Z",
      "content": "<p>Thanks for this  list ! </p>",
      "rawMarkdown": "Thanks for this  list ! "
    },
    {
      "id": 1056916,
      "postDate": "2020-10-22T07:42:49.757Z",
      "content": "<p>Thanks for this! It's really helpful.</p>",
      "rawMarkdown": "Thanks for this! It's really helpful."
    }
  ],
  "comments": [
    {
      "id": 1068287,
      "author_name": "tito",
      "author_url": "",
      "post_date": "2020-11-03T08:48:51.543000",
      "content": "<p>I shared an emurator for iter-test.<br>\nI hope this notebook helps to reduce \"Submission scoring error\"!</p>\n<p><a href=\"https://www.kaggle.com/its7171/iter-test-emulator\" target=\"_blank\">https://www.kaggle.com/its7171/iter-test-emulator</a></p>",
      "votes": 17,
      "replies": [
        {
          "id": 1087319,
          "author_name": "Alex",
          "author_url": "",
          "post_date": "2020-11-22T15:54:47.297000",
          "content": "<p>Hi <a href=\"https://www.kaggle.com/its7171\" target=\"_blank\">@its7171</a> . I've modified my submission framework to be faster and I'm encountering submission scoring error after few minutes (I'd say in the first iterations). I used your wonderful API to debug and everything went well. <br>\nDo you think it can remain some differences between your API and kaggle's one ? thanks</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 1090067,
          "author_name": "Wain Wong",
          "author_url": "",
          "post_date": "2020-11-25T03:40:39.880000",
          "content": "<p>same here💔💔💔</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1108128,
          "author_name": "tito",
          "author_url": "",
          "post_date": "2020-12-10T09:56:11.147000",
          "content": "<p>As far as I know, I have implemented all the features that were officially announced.<br>\nIf you find the reason of the errors which is ok with my scripts, please let me know.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 1056282,
      "author_name": "Alex Bader",
      "author_url": "",
      "post_date": "2020-10-21T15:06:13.233000",
      "content": "<p>Good list!<br>\nBy my observation, it also seems like the group sizes we get from the iterator must be pretty small.</p>\n<p>My code is reasonably performant by now given the amount of feature engineering and parameter updating done, and my last submission timed out again. Performance of my code:</p>\n<ul>\n<li>dummy test commit takes ~30 secs, so the total runtime it literally just feature engineering and sticking the result through a model</li>\n<li>predict group of 1e6 rows in ~25 sec: should give a total runtime ~1 min for 2.5e6 private test rows (would be nice but not possible since maximum group size is supposedly 1000 users)</li>\n<li>predict group of 1000 rows in ~0.4 sec: should give a total runtime ~1000 sec or ~20 min if we're generous</li>\n<li>predict group of 100 rows in ~0.2 sec: should give a total runtime ~2 hours</li>\n<li>predict group of 10 rows in ~0.12 sec: should give a total runtime ~8.5 hours, but add some uncertainty and time for the Kaggle API and it will time out (and does so apparently)</li>\n</ul>\n<p>So I guess we have to assume that the size of the groups we get in the dummy test set is representative for the entire private test set, unfortunately, and at that <em>very</em> small. My takeaway therefore is: be especially efficient with feature engineering on small groups and don't worry about inefficient operations on big groups, it doesn't seem to matter.</p>",
      "votes": 15,
      "replies": [
        {
          "id": 1069505,
          "author_name": "Oguzhan Nefesoglu",
          "author_url": "",
          "post_date": "2020-11-04T14:32:51.050000",
          "content": "<p>Wow! Thank you for the comment. I was getting submission scoring error for 3 days and i saw this comment. I did so many tests but i didnt consider group size of 1. Changed my code according to that and problem solved. So, there are batches even with 1 sample.</p>",
          "votes": 2,
          "replies": []
        }
      ]
    },
    {
      "id": 1122129,
      "author_name": "Gunes Evitan",
      "author_url": "",
      "post_date": "2020-12-22T08:00:04.903000",
      "content": "<p>I'm not sure this is shared before, but I burned more than 20 submissions for this annoying edge case. Those were my first lines in test set iteration loop.</p>\n<pre><code>question_mask = (df_test['content_type_id'] == 0).values.copy()\ndf_test = df_test[question_mask].reset_index(drop=True)\nprior_group_answers_correct = eval(df_test['prior_group_answers_correct'].iloc[0])\n</code></pre>\n<p>The problem is lecture rows are dropped in df_test and then first row of prior_group_answers_correct is passed to eval. In one of the iterations, first row is actually a lecture so it is dropped here, and eval tried to execute NaN. The correct implementation should be:</p>\n<pre><code>prior_group_answers_correct = eval(df_test['prior_group_answers_correct'].iloc[0])\nquestion_mask = (df_test['content_type_id'] == 0).values.copy()\ndf_test = df_test[question_mask].reset_index(drop=True)\n</code></pre>",
      "votes": 6,
      "replies": [
        {
          "id": 1122487,
          "author_name": "Oleksandr Sirenko",
          "author_url": "",
          "post_date": "2020-12-22T13:24:54.557000",
          "content": "<p>Thanks for sharing. It seems like the same problem for me, but I hid it in the preprocessing function of the test dataset and burned my eyes trying to find it in the inference. </p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1125261,
          "author_name": "Pratik Poudel",
          "author_url": "",
          "post_date": "2020-12-24T14:50:48.670000",
          "content": "<p>Hello all, <br>\nI exhausted all my limits for straight three days. I am trying user, parts and target aggregations. I seem to be handling new user in the given api data but when submit i get the submission scoring error.<br>\nHers the error<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4947407%2F733a868d55ddf84bdca78640dff2acec%2FScreenshot%20(2).png?generation=1608821206491178&amp;alt=media\" alt=\"\"></p>\n<p>Also i have shared my notebook <br>\n<a href=\"https://www.kaggle.com/ptrikp/part-taregt-aggregations\" target=\"_blank\">https://www.kaggle.com/ptrikp/part-taregt-aggregations</a></p>\n<p>What am i doing wrong here ? <br>\nCan any one try this.. greatly appreciated<br>\nThanks</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1126810,
          "author_name": "Oleksandr Sirenko",
          "author_url": "",
          "post_date": "2020-12-26T01:30:05.917000",
          "content": "<p>Hi! I am working on a similar problem. If I find a solution, I'll let you know. </p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1126817,
          "author_name": "Pratik Poudel",
          "author_url": "",
          "post_date": "2020-12-26T01:51:04.103000",
          "content": "<p>I made it work by taking dictionary part from this notebook and modifying it.<br>\nRefrence : <a href=\"https://www.kaggle.com/markwijkhuizen/riiid-training-and-prediction-using-a-state\" target=\"_blank\">https://www.kaggle.com/markwijkhuizen/riiid-training-and-prediction-using-a-state</a><br>\n Mine notebook is version 6<br>\n<a href=\"https://www.kaggle.com/ptrikp/part-taregt-aggregations\" target=\"_blank\">https://www.kaggle.com/ptrikp/part-taregt-aggregations</a><br>\nMay be i am using future data as someone pointed out </p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1128992,
          "author_name": "Oleksandr Sirenko",
          "author_url": "",
          "post_date": "2020-12-28T01:07:55.723000",
          "content": "<p>Thanks, Pratik. I'll review those works and try to fix my submission code.  As <code>bturan19</code> mentioned:</p>\n<blockquote>\n  <p>In my case; I was concatting the dataframes with all records of a user and new records. When there is no old records for any user in the batch, it was crushing because of the 'answered correctly' column does not exist in the new batch. So, I wrote a simple check, if there is no that columns append a column with na values. It worked for me. </p>\n</blockquote>\n<p>I think it might be the cause. </p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 1066649,
      "author_name": "Claudio Verdú Ruiz",
      "author_url": "",
      "post_date": "2020-11-01T23:04:13.333000",
      "content": "<p>Not exactly error but I had a few submissions with a quite lower score than expected. Turned out that I was feeding the model with columns in different order. It's obvious and at the same time it's easy to fail, specially if working with a sequence model when you have to add the previous correct answers (a new column gets added at the end by default). I know this can be off topic but I didn't find any related discussion. </p>",
      "votes": 5,
      "replies": []
    },
    {
      "id": 1136113,
      "author_name": "vvm",
      "author_url": "",
      "post_date": "2021-01-02T18:28:48.767000",
      "content": "<p>Hello. My latest submission (ver 22) errors out in 6 minutes. Ver 19 timed out. When I run the test it iteration time is 0.2 seconds consistently. No issues in local test for 1M test data. Getting 3.3 iterations/sec on average.  Is there any way to find you why submission fails. I mean would it be hard to create test set more representative (with missing values and etc) of the real test set so that it would be easier troubleshooting issues? The focus of the competition is on creating the best model, right? Otherwise, a lot of participants seem to be spending resources and time on data quality troubleshooting. I don't think it's the most green friendly approach.</p>",
      "votes": 3,
      "replies": []
    },
    {
      "id": 1058396,
      "author_name": "Vopani",
      "author_url": "",
      "post_date": "2020-10-23T16:48:08.700000",
      "content": "<p>Thanks! This is quite helpful.<br>\nIt's one thing to know about the possible issues, it's another thing to identify which one it could be when you get an error 😄</p>\n<p>I'm sure a lot of competitors will need to burn some submissions before getting their pipeline right.</p>",
      "votes": 3,
      "replies": []
    },
    {
      "id": 1054908,
      "author_name": "Aditya Soni",
      "author_url": "",
      "post_date": "2020-10-20T09:37:28.597000",
      "content": "<p>Nice collection! Pretty much it sums up all the gotchas i also have seen.</p>\n<p>Also, Would just add to the points above that the given test_sample is quite small in nature ~104 rows and I am expecting anything from 1k-4/5k rows at max in each iteration  over test_set generator. Given the volume of the test_data, you will be looping anywhere from ~625 - 2.5k times and that's what takes ~2 hours as you are doing other stuffs (joins/merges/creation of features etc) inside that loop as well, So all in all, they get added up in total…</p>\n<blockquote>\n  <p>These are available only for the first row of each group - the rest are nulls. There can be also an empty list (for the first group).</p>\n</blockquote>\n<p>It will be only for the very first row of the first group as i have successfully accumulated test data <a href=\"https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/191856\" target=\"_blank\">here</a> and make a sub with 1k chunks being accumulated. </p>\n<blockquote>\n  <p>If assigning prior_group_responses/prior_group_answers_correct values to the previous group dataframe - check, if you are doing it before or after filtering out the rows with lectures. I haven't checked which is the correct way, so I'm handling both cases.</p>\n</blockquote>\n<p>The \"prior_group_answers_correct\" has -1 for videos in it.</p>",
      "votes": 3,
      "replies": [
        {
          "id": 1055148,
          "author_name": "alijs",
          "author_url": "",
          "post_date": "2020-10-20T14:14:51.737000",
          "content": "<p>Thanks for the additional info! Updated original post about prior_group_answers_correct field having -1 for lectures.</p>\n<p>Regarding</p>\n<blockquote>\n  <p>expecting anything from 1k-4/5k rows at max in each iteration over test_set generator</p>\n</blockquote>\n<p>From the Data section there should be between 1 and 1000 rows in each iteration.</p>\n<p>Edit: Data section indeed talks about 1 to 1000 <strong>users</strong>, so max number of rows could be bigger.</p>",
          "votes": 2,
          "replies": []
        },
        {
          "id": 1055180,
          "author_name": "Aditya Soni",
          "author_url": "",
          "post_date": "2020-10-20T14:42:58.200000",
          "content": "<p>Ahh, it's a single transaction ID per user and we can have a streak of 3-4 on AVG ques which I have seen in training data. Hence the upper bound.</p>",
          "votes": 4,
          "replies": []
        }
      ]
    },
    {
      "id": 1055130,
      "author_name": "Joe Eddy",
      "author_url": "",
      "post_date": "2020-10-20T13:59:52.673000",
      "content": "<p>Helpful list, thanks!</p>\n<blockquote>\n  <p>There are unseen/new users in hidden test data. This may result in unexpected nulls when merging with features. There is however an unseen user in the example test set, too, so this is less likely to be the problem (if you have checked your solution against the example test data).</p>\n</blockquote>\n<p>Small thing to add - this may be a bit specific to a bug I encountered (a silly index duplication issue), but handling more than one unseen user can definitely have different behavior than handling only one. I wasted a few hours debugging this -- my suggestion is to simulate multiple unseen users in the sample API data when testing your pipeline, e.g. by seeing what happens if you just drop user data for some of the other users in the sample data before running your API parsing process.  </p>",
      "votes": 4,
      "replies": []
    },
    {
      "id": 1133026,
      "author_name": "vvm",
      "author_url": "",
      "post_date": "2020-12-30T21:34:01.863000",
      "content": "<p>I have two submissions running. The one submitted later failed after 10 minutes. The first one has been running for 4 hours. Could it be the reason two were running simultaneously?</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 1077775,
      "author_name": "Ali Abdin",
      "author_url": "",
      "post_date": "2020-11-13T23:01:50.477000",
      "content": "<p>Does the sample submission: <a href=\"https://www.kaggle.com/sohier/quick-sample-submission\" target=\"_blank\">https://www.kaggle.com/sohier/quick-sample-submission</a> cover the filtering of lectures or what exactly do you guys mean by that?</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 1060282,
      "author_name": "Aditya Soni",
      "author_url": "",
      "post_date": "2020-10-26T02:33:02.420000",
      "content": "<p>One of the very possible reason for errors is going to be a timeout error. So I feel that doing df.loc in a loop etc is painfully slow. We can try using numpy here to create new features etc, provided we know the index of the columns from  which lets say a particular feature is derived etc. And wrap the whole thing in numba's jit for a speed boost? Plus is it possible to do group by's on numpy arrays? Has anyone tried it? Ty!</p>",
      "votes": 1,
      "replies": [
        {
          "id": 1060333,
          "author_name": "Vopani",
          "author_url": "",
          "post_date": "2020-10-26T04:19:57.727000",
          "content": "<p>Avoid loops. It may not be trivial but can give you better speed.<br>\nAll my successful submissions don't use loops and finish scoring within 30 mins.</p>",
          "votes": 2,
          "replies": []
        }
      ]
    },
    {
      "id": 1119768,
      "author_name": "Oleksandr Sirenko",
      "author_url": "",
      "post_date": "2020-12-20T11:07:06.340000",
      "content": "<blockquote>\n  <p>Check if you are not messing up the original row_id field values in submission file by merging, by excluding lectures or by other manipulations during feature engineering.</p>\n</blockquote>\n<p>Thanks, topic starter and all involved! This is really useful information. I have interested in how the <code>row_id</code> would be the cause of raising scoring error - if anyone can explain this?</p>\n<p>In my case, the main reason for the scoring error is the time of execution, even not memory failure. Because of micro-batches in each iteration, the data preprocessing functions do the same with the small test chunks every time, calling the base, merging, etc. So the challenge is NOT to create algorithms for \"Knowledge Tracing,\" BUT to beat the submission API.  </p>\n<p>And it doesn't fit in my head. Because the API doesn't complicate the task, but the task solution delivery process. Hey guys, you'll need to extract features from the 100M-rows base and then merge it 100000 times with the 25 rows)). And you have time/memory restriction… What the value of submission API for the contest problem solving - the rhetorical question.</p>\n<p>But never the less this is one of the most interesting competition and we'll handle it!</p>",
      "votes": 2,
      "replies": [
        {
          "id": 1133175,
          "author_name": "wa007",
          "author_url": "",
          "post_date": "2020-12-31T02:17:14.487000",
          "content": "<p>I'm curious how to know if my test set process break the memory restriction?  I run it sucessfully and get <code>Submission scoring error</code> after submitting.<br>\nThanks very much.</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1133604,
          "author_name": "Oleksandr Sirenko",
          "author_url": "",
          "post_date": "2020-12-31T11:23:03.943000",
          "content": "<p>Hi! Perhaps this limitation is not by memory but in code execution time. With the providing test sample, it must be near 550 - 600 ms per iteration for successful submission. You can check it using the python time module.</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 1133621,
          "author_name": "wa007",
          "author_url": "",
          "post_date": "2020-12-31T11:40:30.643000",
          "content": "<p>Got it!!!<br>\nEvery iterator on my code need a few minute..<br>\nSo are there vecy many many iterations in real test set ?</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1133648,
          "author_name": "wa007",
          "author_url": "",
          "post_date": "2020-12-31T12:02:42.310000",
          "content": "<p>Could i get sucess running status if i have execution time problem ? <br>\nI run my code sucessful, May i have the execution time problem? <br>\nThanks very much!</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1133787,
          "author_name": "Oleksandr Sirenko",
          "author_url": "",
          "post_date": "2020-12-31T14:26:00.540000",
          "content": "<p>If you have a runtime issue, you obviously won't be able to successfully submit your solution. In a real test suite, each iteration batch has from 1 to 1000 rows, and a total of 2,500,000 rows. Thus, I think the queue length can be at least about 5000. But if we roughly divide 9 hours (total time restriction) by 550 ms (empiric per iter value), we'll reveal that number of chunks can be about 59000. You can see this guess in the notebook <a href=\"url\" target=\"_blank\">https://www.kaggle.com/tomooinubushi/inference-must-be-0-55-sec-iter</a>. </p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 1106452,
      "author_name": "MPWARE",
      "author_url": "",
      "post_date": "2020-12-08T21:12:31.017000",
      "content": "<p>One question if someone could help or know the answer, <a href=\"https://www.kaggle.com/sohier\" target=\"_blank\">@sohier</a> ?<br>\n<em>(Note that we've followed all advice here and we've already some kernels working fine)</em></p>\n<p>On submission, when you get '<strong>Submission Scoring Error</strong>' after <strong>9h</strong>, does it mean the real root cause is <strong>time out</strong> or really a <strong>scoring issue</strong> (like problem with row_id or value not in 0-1.0 range or …). Is it possible that we have a problem in our model inference after 2h but <strong>it waits</strong> 9h to report such error? </p>\n<p>Our previous kernels were working fine but we've such issue on a new kernel, our simulation tests give us 7h runtime (forecast from 250,000 rows tested) with at least 3GB RAM free on the 13GB. We're not able to trouble shoot and we would like to know if it's a real time out.</p>",
      "votes": 2,
      "replies": [
        {
          "id": 1106458,
          "author_name": "Ali Abdin",
          "author_url": "",
          "post_date": "2020-12-08T21:16:04.903000",
          "content": "<p>From my knowledge and experience it would throw out the error instantly, meaning that it would NOT wait 9 hours.</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 1106510,
          "author_name": "Claudio Verdú Ruiz",
          "author_url": "",
          "post_date": "2020-12-08T22:32:41.427000",
          "content": "<p>If you get GPU memory error it may wait the 9h until the timeout in a frozen-like state. I have seen this happening in live trainings (you get a prompt about the GPU memory but the cell keeps <em>running</em>).</p>",
          "votes": 4,
          "replies": []
        },
        {
          "id": 1107497,
          "author_name": "MPWARE",
          "author_url": "",
          "post_date": "2020-12-09T18:29:12.593000",
          "content": "<p>After 3 days spent on this issue the problem is solved and root cause looks to be in <strong>Kaggle docker image v90</strong>. This version does affect overall performances on inference. The same inference code running under TF2.3.1/DockerImage<strong>v90</strong>: <strong>Timeout</strong>, TF2.3.1/DockerImage<strong>v89</strong>: OK within <strong>6h</strong>.</p>\n<p>Another simple test we've done:</p>\n<ul>\n<li>TF2.3.1/DockerImage<strong>v89</strong>: Model1 inference = 2h30</li>\n<li>TF2.3.1/DockerImage<strong>v90</strong>: Model1 inference = 3h</li>\n</ul>",
          "votes": 5,
          "replies": []
        },
        {
          "id": 1107517,
          "author_name": "AbdurRafae",
          "author_url": "",
          "post_date": "2020-12-09T18:43:17.820000",
          "content": "<p>How to check docker version of a notebook btw?</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1107527,
          "author_name": "MPWARE",
          "author_url": "",
          "post_date": "2020-12-09T18:49:35.970000",
          "content": "<p>See execution info, then click on docker image link:</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F698363%2Fded9aab37059e1553f1a733ad3e2f6e6%2Fdocker.png?generation=1607539653198072&amp;alt=media\" alt=\"\"></p>\n<p>It will open Google console with the tag/date version:<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F698363%2Feafa9ab0139f2be3596e0216385dcfb6%2Fdocker2.png?generation=1607539752855134&amp;alt=media\" alt=\"\"></p>",
          "votes": 4,
          "replies": []
        },
        {
          "id": 1107867,
          "author_name": "tomoo inubushi",
          "author_url": "",
          "post_date": "2020-12-10T02:55:56.673000",
          "content": "<p>Thank you. It's very informative.<br>\nDo you know or does anybody know how to change/select docker version?</p>",
          "votes": 2,
          "replies": []
        },
        {
          "id": 1126219,
          "author_name": "Abdessalem Boukil",
          "author_url": "",
          "post_date": "2020-12-25T12:43:57.587000",
          "content": "<p><a href=\"https://www.kaggle.com/mpware\" target=\"_blank\">@mpware</a> Hello, I am facing exactly the same problem, my notebook reach 9h of execution then spits out a submission scoring error rather than timing out. This is confusing as hell! Since you faced the same problem, is it a time out or a bug in my code ?</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 1059416,
      "author_name": "Vopani",
      "author_url": "",
      "post_date": "2020-10-25T04:42:47.790000",
      "content": "<p><a href=\"https://www.kaggle.com/alijs1\" target=\"_blank\">@alijs1</a> : Might be worth adding <a href=\"https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/193030\" target=\"_blank\">this</a> to the list.</p>",
      "votes": 2,
      "replies": [
        {
          "id": 1059673,
          "author_name": "alijs",
          "author_url": "",
          "post_date": "2020-10-25T10:31:42.370000",
          "content": "<p>Thanks! Updated the list.</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 1140345,
      "author_name": "vvm",
      "author_url": "",
      "post_date": "2021-01-06T00:18:24.583000",
      "content": "<p>A few lessons learned. Hope this will help others.</p>\n<p>Yes you can iterate in your code and do well timewise:).</p>\n<p>Iter time: 0.10601687431335449 0-&gt;18 est: 4.10hrs<br>\nIter time: 0.05148053169250488 1-&gt;27 est: 1.33hrs<br>\nIter time: 0.05518960952758789 2-&gt;26 est: 1.48hrs<br>\nIter time: 0.05421710014343262 3-&gt;33 est: 1.15hrs<br>\nProcessed 104 records in 4 batches</p>\n<p>You just have to:<br>\n1) avoid groupby's. They are terribly slow. Use sorting instead.<br>\n2) Pre-cache features whenever possible <br>\n3) avoid lists and impressions.<br>\n4) if using merges use GPU to get more total memory. Try out njit too.</p>\n<p>Other issues that  I was able to get help from others on.</p>\n<p>1) skip group_num column for the submission. Api doesn't report an error but you get a submission error at the end. <br>\n2) keep rows in the same order as they come from api iterator.</p>\n<p>Burned a few submissions on that. Thanks to everyone who helped!</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 1137013,
      "author_name": "vvm",
      "author_url": "",
      "post_date": "2021-01-03T15:48:02.550000",
      "content": "<p>Hello everyone. I would like to share my iteration times for a provided small test set </p>\n<p>Time iter: 0.8021116256713867  (18 records) 0.20076537132263184<em>2500000/(18</em>60*60) ~= 7.74 hrs<br>\nTime iter: 0.22262358665466309 (26 record) -&gt; 0.22262358665466309 *2500000/(26<em>60</em>60) ~=5.946 hrs<br>\nTime iter: 0.23348522186279297 (33 records) --&gt; 0.23348522186279297 <em>2500000/(33</em>60*60) ~= 4.91340955098470054713 hours based on last group with 33 records </p>\n<p>Processed 104 records in 4 batches</p>\n<p>So based on any of the above batches (scratch the first one) the total running time should be between 5 and 8 hours. The larger the groups are the greater the processing throughput of records/sec is.</p>\n<p>Also according to this post<br>\n<a href=\"https://www.kaggle.com/tomooinubushi/inference-must-be-0-55-sec-iter\" target=\"_blank\">https://www.kaggle.com/tomooinubushi/inference-must-be-0-55-sec-iter</a><br>\n0.2 - 0.3 sec per iteration should be sufficient to finish in 9 hours.</p>\n<p>I ran local test with 1M records and it finished in 53m<br>\nThe latest submission has been running for 7 hours. <br>\nI am trying to find any possible way to improve the run time.</p>\n<p>Does anyone else have similar times on the test data and have a successful submission?<br>\nAny help is much appreciated! </p>\n<p>Thank you.</p>\n<p>P.S. The model is dynamic and a bit more involved and thus requires iteration. The earlier version of the code didn't use iterations but execution was not much faster. </p>",
      "votes": 0,
      "replies": [
        {
          "id": 1137146,
          "author_name": "vvm",
          "author_url": "",
          "post_date": "2021-01-03T17:59:52.317000",
          "content": "<p>Update… I applied a few code optimization. Moving to cupa/numpy lowered iteration times:</p>\n<p>load_lkps: 24.306s<br>\nIter time: 0.9248003959655762 0-&gt;18 est: 35.69hrs (high due to initial model initialization)<br>\nIter time: 0.1260981559753418 1-&gt;27 est: 3.25hrs<br>\nIter time: 0.12287306785583496 2-&gt;26 est: 3.29hrs<br>\nIter time: 0.12256026268005371 3-&gt;33 est: 2.58hrs<br>\nProcessed 104 records in 4 batches </p>\n<p>Fingers crossed:).</p>\n<p>P.S. I used the following formula for running time estimate<br>\nprint(f'Iter time: {time.time()-start} {test_input.iloc[0][GRP_NUM]}-&gt;{len(test_input)} est: {(time.time()-start)<em>2_500_000/(len(test_input)</em>60*60):.2f}hrs')</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 1135717,
      "author_name": "Darren Lahr",
      "author_url": "",
      "post_date": "2021-01-02T13:00:36.440000",
      "content": "<p>Can we assume that at a user level for each batch there is only a single unique task_container_id?</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 1133315,
      "author_name": "wa007",
      "author_url": "",
      "post_date": "2020-12-31T05:30:46.823000",
      "content": "<p>I got the same error after submit and I have a question…<br>\nCould it prove no memory problem and no feature engineering problem If i run predict of test set and submit sucessful ?</p>\n<pre><code>env = riiideducation.make_env()\niter_test = env.iter_test()\nfor (current_df, sample_prediction_df) in iter_test:\n    current_df = data_process(current_df) # feature engineering\n    current_df['answered_correctly'] = model.predict(current_df[features]) # model predict \n    current_df['answered_correctly'].fillna(0.5, inplace = True)\n    current_df['answered_correctly'][current_df['answered_correctly'] &gt; 1] = 1\n    current_df['answered_correctly'][current_df['answered_correctly'] &lt; 0] = 0\n    env.predict(current_df.loc[:, ['row_id', 'answered_correctly']]) # submit \n</code></pre>\n<p>And I get sucess of Execution Info.<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1811540%2F477c4af79a720ceef31c6888fdc15d78%2Fsubmit-sucess.png?generation=1609382264895415&amp;alt=media\" alt=\"\"><br>\nThanks very much.</p>",
      "votes": 0,
      "replies": [
        {
          "id": 1135631,
          "author_name": "wa007",
          "author_url": "",
          "post_date": "2021-01-02T11:44:25.643000",
          "content": "<p>I got it from <a href=\"https://www.kaggle.com/tomooinubushi/inference-must-be-0-55-sec-iter\" target=\"_blank\">https://www.kaggle.com/tomooinubushi/inference-must-be-0-55-sec-iter</a>.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 1128934,
      "author_name": "ruhong",
      "author_url": "",
      "post_date": "2020-12-27T22:34:23.793000",
      "content": "<p>Somehow dropping rows requires an index reset, else I will get a \"submission scoring error\". I do not need to reset index when the exact same code was used in offline preprocessing.</p>\n<pre><code>df.reset_index(drop=True, inplace=True)  # only required for submission notebook\ndf['timestamp'].fillna(0, inplace=True)\nmask = (df['content_type_id'] == 1) &amp; (df['timestamp'] &lt; 2000000)\ndf.drop(index=df[mask].index, inplace=True)\n</code></pre>",
      "votes": 0,
      "replies": [
        {
          "id": 1133160,
          "author_name": "wa007",
          "author_url": "",
          "post_date": "2020-12-31T01:54:36.830000",
          "content": "<p>I know drop <code>df['content_type_id'] == 1</code>, why is conditation <code>df['timestamp'] &lt; 2000000</code>?<br>\nThanks vecy much.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 1121131,
      "author_name": "qiaqia",
      "author_url": "",
      "post_date": "2020-12-21T11:51:54.453000",
      "content": "<p>I finally figured out why my submission code was reporting an error, I guessed it was the leacture, but I couldn't find the exact error location, thanks!</p>\n<blockquote>\n  <p>If assigning prior_group_responses/prior_group_answers_correct values to the previous group dataframe - check, if you are doing it before or after filtering out the rows with lectures. They contain values also for lectures rows - prior_group_answers_correct is reported to contain \"-1\" for lecture rows.</p>\n</blockquote>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 1101228,
      "author_name": "Aditya Soni",
      "author_url": "",
      "post_date": "2020-12-03T18:22:40.143000",
      "content": "<p>Not quite related but might be useful to others,</p>\n<p>A good way to use kaggle kernels effectively is to do the below, </p>\n<ul>\n<li>Create independent set of snips that creates your features and caches them as well.</li>\n<li>Call these independent set of scripts one by one. It's likely that you can get them created in 16 gigs when you focus on each one of them separately rather than as a whole in one kernel itself.</li>\n<li>This trick will save you a good deal of RAM.</li>\n</ul>\n<p>Hope it helps! [Just in case you have limited access to compute like me, the above works like a charm]</p>",
      "votes": 0,
      "replies": [
        {
          "id": 1105672,
          "author_name": "jwc",
          "author_url": "",
          "post_date": "2020-12-08T04:35:00.727000",
          "content": "<p>You mean, functionize?</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 1099013,
      "author_name": "pin-rui",
      "author_url": "",
      "post_date": "2020-12-02T02:46:40.540000",
      "content": "<p>There are 104 rows in example test data for commit, but about 2500000 rows in hidden test set when submitting. Check/estimate if your predictions loop is not using too much time/memory on 2500000 rows.</p>\n<p>Following above-mentioned, how can i check if i get memory error when i submit on 2500000 hidden test set?<br>\nBefore using additional features, i can submit successfully. <br>\nAfter using some new features , i get submission error after 2~3 hours, so i doubt it is memory error, but how can i check it?<br>\nSomeone has any trick to prevent memory error?</p>",
      "votes": 0,
      "replies": [
        {
          "id": 1099116,
          "author_name": "AbdurRafae",
          "author_url": "",
          "post_date": "2020-12-02T05:15:21.593000",
          "content": "<p>enclose your feature engineering step in try and do a dummy prediction in the exception, if your code runs then the error is due to some instability of the feature engineering, otherwise it's from memory.</p>",
          "votes": 2,
          "replies": []
        },
        {
          "id": 1133190,
          "author_name": "wa007",
          "author_url": "",
          "post_date": "2020-12-31T02:36:41.667000",
          "content": "<p>I have a problem…<br>\nCould it prove no memory problem and no feature engineering problem If i run predict of test set and submit sucessful ?</p>\n<pre><code>env = riiideducation.make_env()\niter_test = env.iter_test()\nfor (current_df, sample_prediction_df) in iter_test:\n    current_df = data_process(current_df) # feature engineering\n    current_df['answered_correctly'] = model.predict(current_df[features]) # model predict \n    current_df['answered_correctly'].fillna(0.5, inplace = True)\n    current_df['answered_correctly'][current_df['answered_correctly'] &gt; 1] = 1\n    current_df['answered_correctly'][current_df['answered_correctly'] &lt; 0] = 0\n    env.predict(current_df.loc[:, ['row_id', 'answered_correctly']]) # submit \n</code></pre>\n<p>And I get sucess of Execution Info.<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1811540%2F477c4af79a720ceef31c6888fdc15d78%2Fsubmit-sucess.png?generation=1609382264895415&amp;alt=media\" alt=\"\"><br>\nThanks very much.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 1098677,
      "author_name": "AbdurRafae",
      "author_url": "",
      "post_date": "2020-12-01T19:14:03.500000",
      "content": "<p></p>\n<p>Another situation is that we get multiple task_containers for a single user_id, however only one of the received task_countainers for that user_id would be questions and all others would be lectures. </p>\n<p>Update: This doesn't seem to solve the issue</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 1094880,
      "author_name": "vvm",
      "author_url": "",
      "post_date": "2020-11-29T03:54:06.687000",
      "content": "<p>Hi everybody! Is there a specific limitation on using cuda? All of my submission failed. I am running locally with 1.5G GPU memory and 2.2RAM stable with ETA for 1M records at about 1hr58 minutes. But I've already been running longer than during submission. I tested unseen users - no problem. I generate content_id randomly - no problem. However, I have not tested NEW content_id. I am stuck. Any help/hints would be very much appreciated. This is my first kaggle competition. Please be gentle:).</p>",
      "votes": 0,
      "replies": [
        {
          "id": 1094885,
          "author_name": "Aditya Soni",
          "author_url": "",
          "post_date": "2020-11-29T03:59:28.973000",
          "content": "<blockquote>\n  <p>Hi everybody! Is there a specific limitation on using cuda?</p>\n</blockquote>\n<p>No, there's none. My sub runs in ~2.15-3 hours. Ensure you are handling new users properly, videos/non-videos, nan rows etc..</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1094895,
          "author_name": "vvm",
          "author_url": "",
          "post_date": "2020-11-29T04:16:18.203000",
          "content": "<p>Thank you, Aditya! I am not checking for nas in user_id, conten_id, content_type_id  task_container_id and etc. Are there test records with missing core key values? </p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 1094385,
      "author_name": "bturan19",
      "author_url": "",
      "post_date": "2020-11-28T15:17:42.977000",
      "content": "<p>My submission section takes 700ms in example test and it works fine. But when submit I got error in 5 minutes.</p>",
      "votes": 0,
      "replies": [
        {
          "id": 1094599,
          "author_name": "bturan19",
          "author_url": "",
          "post_date": "2020-11-28T18:52:14.950000",
          "content": "<p>Yeah, there are some groups that full of users that we didn't see in train dataset..</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1121574,
          "author_name": "Oleksandr Sirenko",
          "author_url": "",
          "post_date": "2020-12-21T18:43:24.687000",
          "content": "<p>Hi, <a href=\"https://www.kaggle.com/bturan19\" target=\"_blank\">@bturan19</a>. How did you fix it? I have the same problem with the last submission. Performed 400ms on the test example, all 4 iter pieces went smoothly but got an immediate error. </p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1126369,
          "author_name": "bturan19",
          "author_url": "",
          "post_date": "2020-12-25T14:53:20.157000",
          "content": "<p>In my case; I was concatting the dataframes with all records of a user and new records. When there is no old records for any user in the batch, it was crushing because of the 'answered correctly' column does not exist in the new batch. So, I wrote a simple check, if there is no that columns append a column with na values. It worked for me.</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 1126807,
          "author_name": "Oleksandr Sirenko",
          "author_url": "",
          "post_date": "2020-12-26T01:26:55.673000",
          "content": "<p>Thanks a lot. I'll check my old code again 👍 I burned almost 10 submissions chasing the rabbit))</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 1078269,
      "author_name": "Sunktoteca",
      "author_url": "",
      "post_date": "2020-11-14T15:19:44.147000",
      "content": "<p>The column names of my submission were \"0\" and \"1\". I changed them into \"row_id\" and \"answered_correctly\" to get rid of the submission scoring error.</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 1069983,
      "author_name": "xiaojiongzi",
      "author_url": "",
      "post_date": "2020-11-05T07:29:54.947000",
      "content": "<p>I have a problem when submitting. I can't select the output file, which shows' no output files found ', but my kernel has \"submission.csv\".</p>",
      "votes": 0,
      "replies": [
        {
          "id": 1069990,
          "author_name": "Claudio Verdú Ruiz",
          "author_url": "",
          "post_date": "2020-11-05T07:48:28.527000",
          "content": "<p>You have to save your notebook. If you do \"Quick Save\", then you have to select to save your output in advance settings (in the submission screen). \"Save &amp; Run All\" saves your outputs by default.</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 1065337,
      "author_name": "Ajay  Hayagreeve",
      "author_url": "",
      "post_date": "2020-10-31T07:18:28.863000",
      "content": "<p>Also, the test data will change after the competition right ? </p>\n<ol>\n<li>If yes, then what is the expected number of rows in the new test set after the competition and how many new users are we expected to see ( Well actually in both current test set and new ) ? </li>\n<li>If No, then Wont some people just take all the test set and then store them after commit, since they tell the answer for the previous batch (i.e score around 99% as they wont know the answer only for last batch)?</li>\n</ol>",
      "votes": 0,
      "replies": [
        {
          "id": 1065381,
          "author_name": "Aditya Soni",
          "author_url": "",
          "post_date": "2020-10-31T08:15:10.267000",
          "content": "<p>I strongly suggest, start reading the discussion posts/ data desc / kaggle EDA kernel's etc  first before asking the same thing! </p>\n<p>You don't have access to whole test set at once, you only have access to it via a generator which you can't control, just iterate over, make your press on the required columns and we are done. </p>\n<p>And we cannot commit the log of the test set as we don't have access to it when we commit on interactive mode. The dummy generator (for example_test.csvc) is replaced by the real test set generator in the commit mode which is run by kaggle in the backend in an isolated environment.<br>\nThat's it.</p>",
          "votes": -1,
          "replies": []
        }
      ]
    },
    {
      "id": 1059116,
      "author_name": "huangtaogan",
      "author_url": "",
      "post_date": "2020-10-24T16:36:18.903000",
      "content": "<p>wonderful job !<br>\nMay I ask another question: does input files(train.csv、question.csv、etc) change when it rerun my notebook?</p>",
      "votes": 0,
      "replies": [
        {
          "id": 1059674,
          "author_name": "alijs",
          "author_url": "",
          "post_date": "2020-10-25T10:33:45.533000",
          "content": "<p>No, only test data changes. Everything else (including train.csv, questions.csv) stays the same.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 1058711,
      "author_name": "",
      "author_url": "",
      "post_date": "2020-10-24T06:21:57.110000",
      "content": "",
      "votes": 0,
      "replies": [
        {
          "id": 1059682,
          "author_name": "",
          "author_url": "",
          "post_date": "2020-10-25T10:41:04.987000",
          "content": "",
          "votes": 1,
          "replies": []
        },
        {
          "id": 1064119,
          "author_name": "",
          "author_url": "",
          "post_date": "2020-10-29T18:24:46.347000",
          "content": "",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1064386,
          "author_name": "",
          "author_url": "",
          "post_date": "2020-10-30T04:18:32.417000",
          "content": "",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1064414,
          "author_name": "",
          "author_url": "",
          "post_date": "2020-10-30T05:09:09.050000",
          "content": "",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 1056021,
      "author_name": "",
      "author_url": "",
      "post_date": "2020-10-21T11:06:05.417000",
      "content": "",
      "votes": 0,
      "replies": []
    },
    {
      "id": 1125260,
      "author_name": "",
      "author_url": "",
      "post_date": "2020-12-24T14:49:21.870000",
      "content": "",
      "votes": 0,
      "replies": []
    },
    {
      "id": 1055972,
      "author_name": "",
      "author_url": "",
      "post_date": "2020-10-21T10:11:59.677000",
      "content": "",
      "votes": 0,
      "replies": []
    },
    {
      "id": 1736634,
      "author_name": "",
      "author_url": "",
      "post_date": "2022-03-27T14:52:38.697000",
      "content": "",
      "votes": 0,
      "replies": []
    },
    {
      "id": 1059121,
      "author_name": "",
      "author_url": "",
      "post_date": "2020-10-24T16:44:56.703000",
      "content": "",
      "votes": 0,
      "replies": []
    },
    {
      "id": 1056916,
      "author_name": "",
      "author_url": "",
      "post_date": "2020-10-22T07:42:49.757000",
      "content": "",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1054845": "It looks that questions about \"Submission scoring error\" are raised almost every day. I think it would save a lot of time to collect the known typical reasons for getting this error.\n\nSo here is the checklist of tricky (and less tricky) things I've observed so far, which could cause the error on submission (while potentially being fine when committing):\n* There are 104 rows in example test data for commit, but about 2500000 rows in hidden test set when submitting. Check/estimate if your predictions loop is not using too much time/memory on 2500000 rows.\n* There are unseen/new users in hidden test data. This may result in unexpected nulls when merging with features. There is however an unseen user in the example test set, too, so this is less likely to be the problem (if you have checked your solution against the example test data).\n* Unseen test rows contains not only questions but also lectures. This means:\n  * You must make predictions only for questions, not for lectures - make sure you filter lectures out before predicting.\n  * If you merge test data e.g. with questions.csv, there will be nulls for rows with lectures (if you didn't filter them out before merge).\n* If you are using *prior_group_responses* and/or *prior_group_answers_correct*:\n  * These are available only for the first row of each group - the rest are nulls. There can be also an empty list (for the first group).\n  * If assigning *prior_group_responses/prior_group_answers_correct* values to the previous group dataframe - check, if you are doing it before or after filtering out the rows with lectures. They contain values also for lectures rows - *prior_group_answers_correct* is reported to contain \"-1\" for lecture rows.\n* Check if you are not messing up the original *row_id* field values in submission file by merging, by excluding lectures or by other manipulations during feature engineering.\n\nIf you have found some other tricky reason for potentially getting \"Submission scoring error\" on submission, please share it in comments.\n",
    "1068287": "I shared an emurator for iter-test.\nI hope this notebook helps to reduce \"Submission scoring error\"!\n\nhttps://www.kaggle.com/its7171/iter-test-emulator",
    "1056282": "Good list!\nBy my observation, it also seems like the group sizes we get from the iterator must be pretty small.\n\nMy code is reasonably performant by now given the amount of feature engineering and parameter updating done, and my last submission timed out again. Performance of my code:\n- dummy test commit takes ~30 secs, so the total runtime it literally just feature engineering and sticking the result through a model\n- predict group of 1e6 rows in ~25 sec: should give a total runtime ~1 min for 2.5e6 private test rows (would be nice but not possible since maximum group size is supposedly 1000 users)\n- predict group of 1000 rows in ~0.4 sec: should give a total runtime ~1000 sec or ~20 min if we're generous\n- predict group of 100 rows in ~0.2 sec: should give a total runtime ~2 hours\n- predict group of 10 rows in ~0.12 sec: should give a total runtime ~8.5 hours, but add some uncertainty and time for the Kaggle API and it will time out (and does so apparently)\n\nSo I guess we have to assume that the size of the groups we get in the dummy test set is representative for the entire private test set, unfortunately, and at that *very* small. My takeaway therefore is: be especially efficient with feature engineering on small groups and don't worry about inefficient operations on big groups, it doesn't seem to matter.",
    "1122129": "I'm not sure this is shared before, but I burned more than 20 submissions for this annoying edge case. Those were my first lines in test set iteration loop.\n\n```\nquestion_mask = (df_test['content_type_id'] == 0).values.copy()\ndf_test = df_test[question_mask].reset_index(drop=True)\nprior_group_answers_correct = eval(df_test['prior_group_answers_correct'].iloc[0])\n```\n\nThe problem is lecture rows are dropped in df_test and then first row of prior_group_answers_correct is passed to eval. In one of the iterations, first row is actually a lecture so it is dropped here, and eval tried to execute NaN. The correct implementation should be:\n\n```\nprior_group_answers_correct = eval(df_test['prior_group_answers_correct'].iloc[0])\nquestion_mask = (df_test['content_type_id'] == 0).values.copy()\ndf_test = df_test[question_mask].reset_index(drop=True)\n```",
    "1066649": "Not exactly error but I had a few submissions with a quite lower score than expected. Turned out that I was feeding the model with columns in different order. It's obvious and at the same time it's easy to fail, specially if working with a sequence model when you have to add the previous correct answers (a new column gets added at the end by default). I know this can be off topic but I didn't find any related discussion. ",
    "1136113": "Hello. My latest submission (ver 22) errors out in 6 minutes. Ver 19 timed out. When I run the test it iteration time is 0.2 seconds consistently. No issues in local test for 1M test data. Getting 3.3 iterations/sec on average.  Is there any way to find you why submission fails. I mean would it be hard to create test set more representative (with missing values and etc) of the real test set so that it would be easier troubleshooting issues? The focus of the competition is on creating the best model, right? Otherwise, a lot of participants seem to be spending resources and time on data quality troubleshooting. I don't think it's the most green friendly approach.",
    "1058396": "Thanks! This is quite helpful.\nIt's one thing to know about the possible issues, it's another thing to identify which one it could be when you get an error 😄\n\nI'm sure a lot of competitors will need to burn some submissions before getting their pipeline right.",
    "1054908": "Nice collection! Pretty much it sums up all the gotchas i also have seen.\n\nAlso, Would just add to the points above that the given test_sample is quite small in nature ~104 rows and I am expecting anything from 1k-4/5k rows at max in each iteration  over test_set generator. Given the volume of the test_data, you will be looping anywhere from ~625 - 2.5k times and that's what takes ~2 hours as you are doing other stuffs (joins/merges/creation of features etc) inside that loop as well, So all in all, they get added up in total...\n\n>These are available only for the first row of each group - the rest are nulls. There can be also an empty list (for the first group).\n\nIt will be only for the very first row of the first group as i have successfully accumulated test data [here](https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/191856) and make a sub with 1k chunks being accumulated. \n\n>If assigning prior_group_responses/prior_group_answers_correct values to the previous group dataframe - check, if you are doing it before or after filtering out the rows with lectures. I haven't checked which is the correct way, so I'm handling both cases.\n\nThe \"prior_group_answers_correct\" has -1 for videos in it.",
    "1055130": "Helpful list, thanks!\n\n> There are unseen/new users in hidden test data. This may result in unexpected nulls when merging with features. There is however an unseen user in the example test set, too, so this is less likely to be the problem (if you have checked your solution against the example test data).\n\nSmall thing to add - this may be a bit specific to a bug I encountered (a silly index duplication issue), but handling more than one unseen user can definitely have different behavior than handling only one. I wasted a few hours debugging this -- my suggestion is to simulate multiple unseen users in the sample API data when testing your pipeline, e.g. by seeing what happens if you just drop user data for some of the other users in the sample data before running your API parsing process.  ",
    "1133026": "I have two submissions running. The one submitted later failed after 10 minutes. The first one has been running for 4 hours. Could it be the reason two were running simultaneously?",
    "1077775": "Does the sample submission: https://www.kaggle.com/sohier/quick-sample-submission cover the filtering of lectures or what exactly do you guys mean by that?",
    "1060282": "One of the very possible reason for errors is going to be a timeout error. So I feel that doing df.loc in a loop etc is painfully slow. We can try using numpy here to create new features etc, provided we know the index of the columns from  which lets say a particular feature is derived etc. And wrap the whole thing in numba's jit for a speed boost? Plus is it possible to do group by's on numpy arrays? Has anyone tried it? Ty!",
    "1119768": "> Check if you are not messing up the original row_id field values in submission file by merging, by excluding lectures or by other manipulations during feature engineering.\n\nThanks, topic starter and all involved! This is really useful information. I have interested in how the `row_id` would be the cause of raising scoring error - if anyone can explain this?\n\nIn my case, the main reason for the scoring error is the time of execution, even not memory failure. Because of micro-batches in each iteration, the data preprocessing functions do the same with the small test chunks every time, calling the base, merging, etc. So the challenge is NOT to create algorithms for \"Knowledge Tracing,\" BUT to beat the submission API.  \n\nAnd it doesn't fit in my head. Because the API doesn't complicate the task, but the task solution delivery process. Hey guys, you'll need to extract features from the 100M-rows base and then merge it 100000 times with the 25 rows)). And you have time/memory restriction... What the value of submission API for the contest problem solving - the rhetorical question.\n\nBut never the less this is one of the most interesting competition and we'll handle it!",
    "1106452": "One question if someone could help or know the answer, @sohier ?\n*(Note that we've followed all advice here and we've already some kernels working fine)*\n\nOn submission, when you get '**Submission Scoring Error**' after **9h**, does it mean the real root cause is **time out** or really a **scoring issue** (like problem with row_id or value not in 0-1.0 range or ...). Is it possible that we have a problem in our model inference after 2h but **it waits** 9h to report such error? \n\nOur previous kernels were working fine but we've such issue on a new kernel, our simulation tests give us 7h runtime (forecast from 250,000 rows tested) with at least 3GB RAM free on the 13GB. We're not able to trouble shoot and we would like to know if it's a real time out.\n",
    "1059416": "@alijs1 : Might be worth adding [this](https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/193030) to the list.",
    "1140345": "A few lessons learned. Hope this will help others.\n\nYes you can iterate in your code and do well timewise:).\n\nIter time: 0.10601687431335449 0->18 est: 4.10hrs\nIter time: 0.05148053169250488 1->27 est: 1.33hrs\nIter time: 0.05518960952758789 2->26 est: 1.48hrs\nIter time: 0.05421710014343262 3->33 est: 1.15hrs\nProcessed 104 records in 4 batches\n\nYou just have to:\n1) avoid groupby's. They are terribly slow. Use sorting instead.\n2) Pre-cache features whenever possible \n3) avoid lists and impressions.\n4) if using merges use GPU to get more total memory. Try out njit too.\n\n\nOther issues that  I was able to get help from others on.\n\n1) skip group_num column for the submission. Api doesn't report an error but you get a submission error at the end. \n2) keep rows in the same order as they come from api iterator.\n\nBurned a few submissions on that. Thanks to everyone who helped!\n",
    "1137013": "Hello everyone. I would like to share my iteration times for a provided small test set \n\nTime iter: 0.8021116256713867 <-- some initialization happen for the first group\nTime iter: 0.20076537132263184 -> (18 records) 0.20076537132263184*2500000/(18*60*60) ~= 7.74 hrs\nTime iter: 0.22262358665466309 (26 record) -> 0.22262358665466309 *2500000/(26*60*60) ~=5.946 hrs\nTime iter: 0.23348522186279297 (33 records) --> 0.23348522186279297 *2500000/(33*60*60) ~= 4.91340955098470054713 hours based on last group with 33 records \n\nProcessed 104 records in 4 batches\n\nSo based on any of the above batches (scratch the first one) the total running time should be between 5 and 8 hours. The larger the groups are the greater the processing throughput of records/sec is.\n\nAlso according to this post\nhttps://www.kaggle.com/tomooinubushi/inference-must-be-0-55-sec-iter\n0.2 - 0.3 sec per iteration should be sufficient to finish in 9 hours.\n\nI ran local test with 1M records and it finished in 53m\nThe latest submission has been running for 7 hours. \nI am trying to find any possible way to improve the run time.\n\nDoes anyone else have similar times on the test data and have a successful submission?\nAny help is much appreciated! \n\nThank you.\n\nP.S. The model is dynamic and a bit more involved and thus requires iteration. The earlier version of the code didn't use iterations but execution was not much faster. ",
    "1135717": "Can we assume that at a user level for each batch there is only a single unique task_container_id?",
    "1133315": "I got the same error after submit and I have a question...\nCould it prove no memory problem and no feature engineering problem If i run predict of test set and submit sucessful ?\n```\nenv = riiideducation.make_env()\niter_test = env.iter_test()\nfor (current_df, sample_prediction_df) in iter_test:\n    current_df = data_process(current_df) # feature engineering\n    current_df['answered_correctly'] = model.predict(current_df[features]) # model predict \n    current_df['answered_correctly'].fillna(0.5, inplace = True)\n    current_df['answered_correctly'][current_df['answered_correctly'] > 1] = 1\n    current_df['answered_correctly'][current_df['answered_correctly'] < 0] = 0\n    env.predict(current_df.loc[:, ['row_id', 'answered_correctly']]) # submit \n```\nAnd I get sucess of Execution Info.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1811540%2F477c4af79a720ceef31c6888fdc15d78%2Fsubmit-sucess.png?generation=1609382264895415&alt=media)\nThanks very much.",
    "1128934": "Somehow dropping rows requires an index reset, else I will get a \"submission scoring error\". I do not need to reset index when the exact same code was used in offline preprocessing.\n\n```\ndf.reset_index(drop=True, inplace=True)  # only required for submission notebook\ndf['timestamp'].fillna(0, inplace=True)\nmask = (df['content_type_id'] == 1) & (df['timestamp'] < 2000000)\ndf.drop(index=df[mask].index, inplace=True)\n```",
    "1121131": "I finally figured out why my submission code was reporting an error, I guessed it was the leacture, but I couldn't find the exact error location, thanks!\n> If assigning prior_group_responses/prior_group_answers_correct values to the previous group dataframe - check, if you are doing it before or after filtering out the rows with lectures. They contain values also for lectures rows - prior_group_answers_correct is reported to contain \"-1\" for lecture rows.",
    "1101228": "Not quite related but might be useful to others,\n\nA good way to use kaggle kernels effectively is to do the below, \n\n- Create independent set of snips that creates your features and caches them as well.\n- Call these independent set of scripts one by one. It's likely that you can get them created in 16 gigs when you focus on each one of them separately rather than as a whole in one kernel itself.\n- This trick will save you a good deal of RAM.\n\nHope it helps! [Just in case you have limited access to compute like me, the above works like a charm]",
    "1099013": "There are 104 rows in example test data for commit, but about 2500000 rows in hidden test set when submitting. Check/estimate if your predictions loop is not using too much time/memory on 2500000 rows.\n\nFollowing above-mentioned, how can i check if i get memory error when i submit on 2500000 hidden test set?\nBefore using additional features, i can submit successfully. \nAfter using some new features , i get submission error after 2~3 hours, so i doubt it is memory error, but how can i check it?\nSomeone has any trick to prevent memory error?",
    "1098677": "~~Don't know if someone else faced this issue or not, I think we get a new user_id with first interaction as a lecture. My submission has been failing for a while so I experimented and found this possible reason. ~~\n\nAnother situation is that we get multiple task_containers for a single user_id, however only one of the received task_countainers for that user_id would be questions and all others would be lectures. \n\nUpdate: This doesn't seem to solve the issue",
    "1094880": "Hi everybody! Is there a specific limitation on using cuda? All of my submission failed. I am running locally with 1.5G GPU memory and 2.2RAM stable with ETA for 1M records at about 1hr58 minutes. But I've already been running longer than during submission. I tested unseen users - no problem. I generate content_id randomly - no problem. However, I have not tested NEW content_id. I am stuck. Any help/hints would be very much appreciated. This is my first kaggle competition. Please be gentle:).",
    "1094385": "My submission section takes 700ms in example test and it works fine. But when submit I got error in 5 minutes.",
    "1078269": "The column names of my submission were \"0\" and \"1\". I changed them into \"row_id\" and \"answered_correctly\" to get rid of the submission scoring error.",
    "1069983": "I have a problem when submitting. I can't select the output file, which shows' no output files found ', but my kernel has \"submission.csv\".",
    "1065337": "Also, the test data will change after the competition right ? \n1. If yes, then what is the expected number of rows in the new test set after the competition and how many new users are we expected to see ( Well actually in both current test set and new ) ? \n2. If No, then Wont some people just take all the test set and then store them after commit, since they tell the answer for the previous batch (i.e score around 99% as they wont know the answer only for last batch)?",
    "1059116": "wonderful job !\nMay I ask another question: does input files(train.csv、question.csv、etc) change when it rerun my notebook?",
    "1058711": "Hello, I want to ask how does kaggle works while predict unseen dataset?\n1. Just exchange test_sample.csv to unseen dataset(For feature engineering all the same by using of user's code)\n2.after fitted the model with user's code, then predict unseen dataset with official code\n3.others way\nWhich one is more appropriate？\nThanks",
    "1056021": "Helpful list, especially for a beginner like me. Thanks!",
    "1125260": "",
    "1055972": "",
    "1736634": "thanks\nreally helpful",
    "1059121": "Thanks for this  list ! ",
    "1056916": "Thanks for this! It's really helpful."
  }
}