{
  "id": 551043,
  "title": "Simliar predictions but too poor score",
  "url": "/competitions/child-mind-institute-problematic-internet-use/discussion/551043",
  "author_name": "",
  "post_date": "2024-12-10T23:10:01.162984800Z",
  "votes": 2,
  "comment_count": 5,
  "views": 0,
  "content": "<p>I am getting similar predicted values in my notebook than other shared notebooks with score 0.49, but my score is 0.37. <br>\nTo be more precise, my notebook has only two out of the twenty samples on test dataset different than the 0.49 notebook and getting 0.37. <br>\nAm i missing something?</p>",
  "messages": [
    {
      "id": "3068982",
      "postDate": "12/10/2024 23:10:01",
      "content": "<p>I am getting similar predicted values in my notebook than other shared notebooks with score 0.49, but my score is 0.37. <br>\nTo be more precise, my notebook has only two out of the twenty samples on test dataset different than the 0.49 notebook and getting 0.37. <br>\nAm i missing something?</p>",
      "rawMarkdown": "I am getting similar predicted values in my notebook than other shared notebooks with score 0.49, but my score is 0.37. \nTo be more precise, my notebook has only two out of the twenty samples on test dataset different than the 0.49 notebook and getting 0.37. \nAm i missing something?",
      "votes": null
    },
    {
      "id": "3069004",
      "postDate": "12/11/2024 00:06:28",
      "content": "<p>Did you try to resubmit the very same 0.37 version one more time without retraining to see what the diapason of score fluctuation is?</p>",
      "rawMarkdown": "Did you try to resubmit the very same 0.37 version one more time without retraining to see what the diapason of score fluctuation is?",
      "votes": null
    },
    {
      "id": "3069281",
      "postDate": "12/11/2024 09:19:22",
      "content": "<p>That 20 tests ids are copy of 1st twenty train points, during submission they are replaced with real LB,PB test data</p>",
      "rawMarkdown": "That 20 tests ids are copy of 1st twenty train points, during submission they are replaced with real LB,PB test data",
      "votes": null
    },
    {
      "id": "3069690",
      "postDate": "12/11/2024 19:04:57",
      "content": "<p>I am now trying, because my results usually the same every time i run it (it may be becaue i am not using autoencoder) and i didn't expect to have different result. </p>\n<p>Actually i got new notebook with even less differences compared to 0.49 notebook. One run is exactly the same and other one differs only in one number.</p>\n<p>Thanks for the feedback.</p>",
      "rawMarkdown": "I am now trying, because my results usually the same every time i run it (it may be becaue i am not using autoencoder) and i didn't expect to have different result. \n\nActually i got new notebook with even less differences compared to 0.49 notebook. One run is exactly the same and other one differs only in one number.\n\nThanks for the feedback.",
      "votes": null
    },
    {
      "id": "3069691",
      "postDate": "12/11/2024 19:09:05",
      "content": "<p>I thought the new data was coming on final scoring, when the rest of the data is released… Anyways, i am trying new code and getting better scores during training and getting the same results or only a difference of single value but my score is no more than 0.438…<br>\nActually, during the data procesing i took 20 samples out of the original data and train the model without it so i can test my model with data the mdoel has never seem before, and getting 0.57 of QWK but when i submit can't get better…</p>",
      "rawMarkdown": "I thought the new data was coming on final scoring, when the rest of the data is released... Anyways, i am trying new code and getting better scores during training and getting the same results or only a difference of single value but my score is no more than 0.438...\nActually, during the data procesing i took 20 samples out of the original data and train the model without it so i can test my model with data the mdoel has never seem before, and getting 0.57 of QWK but when i submit can't get better...",
      "votes": null
    },
    {
      "id": "3069737",
      "postDate": "12/11/2024 20:08:21",
      "content": "<p>You can't compare the submission results. Since they are classified by qwk. Even if you have the same numbers, your regression numbers could be way different (for example: 0.1 is your result, 0.4 the one you compare it with and the qwk sets everything below 0.5 to 0). You can't see that in the 20 test sample results, but the leaderboard has over 1000 samples, so it will change your score for sure.</p>\n<p>tldr.: Cant compare submissions to judge LB score</p>",
      "rawMarkdown": "You can't compare the submission results. Since they are classified by qwk. Even if you have the same numbers, your regression numbers could be way different (for example: 0.1 is your result, 0.4 the one you compare it with and the qwk sets everything below 0.5 to 0). You can't see that in the 20 test sample results, but the leaderboard has over 1000 samples, so it will change your score for sure.\n\ntldr.: Cant compare submissions to judge LB score",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3069004,
      "author_name": "yekenot",
      "author_url": "",
      "post_date": "12/11/2024 00:06:28",
      "content": "<p>Did you try to resubmit the very same 0.37 version one more time without retraining to see what the diapason of score fluctuation is?</p>",
      "votes": null,
      "replies": [
        {
          "id": 3069690,
          "author_name": "mayobanexsantana",
          "author_url": "",
          "post_date": "12/11/2024 19:04:57",
          "content": "<p>I am now trying, because my results usually the same every time i run it (it may be becaue i am not using autoencoder) and i didn't expect to have different result. </p>\n<p>Actually i got new notebook with even less differences compared to 0.49 notebook. One run is exactly the same and other one differs only in one number.</p>\n<p>Thanks for the feedback.</p>",
          "votes": null,
          "replies": [
            {
              "id": 3069737,
              "author_name": "mariusheuser",
              "author_url": "",
              "post_date": "12/11/2024 20:08:21",
              "content": "<p>You can't compare the submission results. Since they are classified by qwk. Even if you have the same numbers, your regression numbers could be way different (for example: 0.1 is your result, 0.4 the one you compare it with and the qwk sets everything below 0.5 to 0). You can't see that in the 20 test sample results, but the leaderboard has over 1000 samples, so it will change your score for sure.</p>\n<p>tldr.: Cant compare submissions to judge LB score</p>",
              "votes": null,
              "replies": []
            }
          ]
        }
      ]
    },
    {
      "id": 3069281,
      "author_name": "leonixis",
      "author_url": "",
      "post_date": "12/11/2024 09:19:22",
      "content": "<p>That 20 tests ids are copy of 1st twenty train points, during submission they are replaced with real LB,PB test data</p>",
      "votes": null,
      "replies": [
        {
          "id": 3069691,
          "author_name": "mayobanexsantana",
          "author_url": "",
          "post_date": "12/11/2024 19:09:05",
          "content": "<p>I thought the new data was coming on final scoring, when the rest of the data is released… Anyways, i am trying new code and getting better scores during training and getting the same results or only a difference of single value but my score is no more than 0.438…<br>\nActually, during the data procesing i took 20 samples out of the original data and train the model without it so i can test my model with data the mdoel has never seem before, and getting 0.57 of QWK but when i submit can't get better…</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "3068982": "I am getting similar predicted values in my notebook than other shared notebooks with score 0.49, but my score is 0.37. \nTo be more precise, my notebook has only two out of the twenty samples on test dataset different than the 0.49 notebook and getting 0.37. \nAm i missing something?",
    "3069004": "Did you try to resubmit the very same 0.37 version one more time without retraining to see what the diapason of score fluctuation is?",
    "3069281": "That 20 tests ids are copy of 1st twenty train points, during submission they are replaced with real LB,PB test data",
    "3069690": "I am now trying, because my results usually the same every time i run it (it may be becaue i am not using autoencoder) and i didn't expect to have different result. \n\nActually i got new notebook with even less differences compared to 0.49 notebook. One run is exactly the same and other one differs only in one number.\n\nThanks for the feedback.",
    "3069691": "I thought the new data was coming on final scoring, when the rest of the data is released... Anyways, i am trying new code and getting better scores during training and getting the same results or only a difference of single value but my score is no more than 0.438...\nActually, during the data procesing i took 20 samples out of the original data and train the model without it so i can test my model with data the mdoel has never seem before, and getting 0.57 of QWK but when i submit can't get better...",
    "3069737": "You can't compare the submission results. Since they are classified by qwk. Even if you have the same numbers, your regression numbers could be way different (for example: 0.1 is your result, 0.4 the one you compare it with and the qwk sets everything below 0.5 to 0). You can't see that in the 20 test sample results, but the leaderboard has over 1000 samples, so it will change your score for sure.\n\ntldr.: Cant compare submissions to judge LB score"
  },
  "source": "meta"
}