{
  "id": 125312,
  "title": "scoring Issue",
  "url": "/competitions/bengaliai-cv19/discussion/125312",
  "author_name": "",
  "post_date": "2020-01-09T22:26:38.860112300Z",
  "votes": null,
  "comment_count": 7,
  "views": 0,
  "content": "<p>I have done my training local on my machine using certain framework, then i got my predictions and uploaded them in the notebook by writing them into a csv file to be able to submit them but it always giving me scoring error with no details about the error!!! BTW i can't upload my model as the notebooks don't support this frame work</p>",
  "messages": [
    {
      "id": "714888",
      "postDate": "01/09/2020 22:26:38",
      "content": "<p>I have done my training local on my machine using certain framework, then i got my predictions and uploaded them in the notebook by writing them into a csv file to be able to submit them but it always giving me scoring error with no details about the error!!! BTW i can't upload my model as the notebooks don't support this frame work</p>",
      "rawMarkdown": "I have done my training local on my machine using certain framework, then i got my predictions and uploaded them in the notebook by writing them into a csv file to be able to submit them but it always giving me scoring error with no details about the error!!! BTW i can't upload my model as the notebooks don't support this frame work",
      "votes": null
    },
    {
      "id": "715032",
      "postDate": "01/10/2020 03:51:47",
      "content": "<p>As this is a kernel-only competition, you need to submit with your model weights. When you submit, the kernel is run on a private test set, which is not given to the contestants. So submitting only predictions will not suffice, you'll need the model too.</p>\n\n<p>If you can't upload your model as a notebook kernel, upload it as a script.</p>",
      "rawMarkdown": "As this is a kernel-only competition, you need to submit with your model weights. When you submit, the kernel is run on a private test set, which is not given to the contestants. So submitting only predictions will not suffice, you'll need the model too.\n\nIf you can't upload your model as a notebook kernel, upload it as a script.",
      "votes": null
    },
    {
      "id": "715640",
      "postDate": "01/10/2020 17:22:28",
      "content": "<p>i am afraid neither notebook or script accept my model, I don't know what to do</p>",
      "rawMarkdown": "i am afraid neither notebook or script accept my model, I don't know what to do",
      "votes": null
    },
    {
      "id": "716140",
      "postDate": "01/11/2020 10:01:51",
      "content": "<p>because you only have 12 samples of test data offline, if you just copy the prediction of those samples you will got scoring error.\nwhen you submit your kernel, the kernel will be automatically rerun on about 200k samples which is invisible to you, then calculate the score.\njust take a look at public notebooks that got a score to see how they got submissions successful.</p>",
      "rawMarkdown": "because you only have 12 samples of test data offline, if you just copy the prediction of those samples you will got scoring error.\nwhen you submit your kernel, the kernel will be automatically rerun on about 200k samples which is invisible to you, then calculate the score.\njust take a look at public notebooks that got a score to see how they got submissions successful.",
      "votes": null
    },
    {
      "id": "716184",
      "postDate": "01/11/2020 11:21:16",
      "content": "<p><a href=\"/haqishen\">@haqishen</a> , hello. I'm facing kinda weird submission result and I'm guessing it may because of my evaluation process. I followed many processing pipelines for this but most of the time it behaved wrongly. Would you please evaluate my evaluation process. Note: I was using image files (128p, greyscale image) for training and so I need to resize the test data set too!). </p>\n\n<p>And by the way, I was following your score on the public LB. <strong>98 to 99</strong>, congrats, awesome indeed. Any pro tips 😄 </p>\n\n<h2>Test Data Generator</h2>\n\n<p>```\ndef test_generator(df, batch_size):\n    num_imgs = len(df)</p>\n\n<pre><code>for batch_start in range(0, num_imgs, batch_size):\n    curr_batch_size = min(num_imgs, batch_start + batch_size) - batch_start\n    idx = np.arange(batch_start, batch_start + curr_batch_size)\n\n    names_batch = df.iloc[idx, 0].values\n    imgs_batch = 255 - df.iloc[idx, 1:].values.reshape(-1, HEIGHT, WIDTH).astype(np.uint8)\n    X_batch = np.zeros((curr_batch_size, SIZE, SIZE, 1))\n\n    for j in range(curr_batch_size):\n        img = cv2.resize(imgs_batch[j,], (SIZE, SIZE))\n        img = img[:, :, np.newaxis]\n        X_batch[j,] = img\n\n    yield X_batch, names_batch\n</code></pre>\n\n<p>```</p>\n\n<h2>Parquet files, looping, prediction ...!</h2>\n\n<p>```\nTEST = [ <br>\n\"bengaliai-cv19/test_image_data_0.parquet\", \n\"bengaliai-cv19/test_image_data_1.parquet\",\n\"bengaliai-cv19/test_image_data_2.parquet\",\n\"bengaliai-cv19/test_image_data_3.parquet\",\n]</p>\n\n<p>row_id = []\ntarget = []</p>\n\n<p>for fname in TEST:\n    test_ = pd.read_parquet(fname)\n    test_gen = test_generator(test_, batch_size=batch_size)</p>\n\n<pre><code>for batch_x, batch_name in tqdm(test_gen):\n    batch_predict = model.predict(batch_x)\n    for idx, name in enumerate(batch_name):\n        row_id += [\n            f\"{name}_consonant_diacritic\",\n            f\"{name}_grapheme_root\",\n            f\"{name}_vowel_diacritic\",\n        ]\n        target += [\n            np.argmax(batch_predict[2], axis=1)[idx],\n            np.argmax(batch_predict[0], axis=1)[idx],\n            np.argmax(batch_predict[1], axis=1)[idx],\n        ]\n\ndel test_\ngc.collect()\n</code></pre>\n\n<p>df_sample = pd.DataFrame(\n    {\n        'row_id': row_id,\n        'target':target\n    },\n    columns = ['row_id','target'] \n)\ndf_sample.to_csv('submission.csv',index=False)\ngc.collect()\n```</p>\n\n<p>Any catch. TIA. </p>",
      "rawMarkdown": "haqishen , hello. I'm facing kinda weird submission result and I'm guessing it may because of my evaluation process. I followed many processing pipelines for this but most of the time it behaved wrongly. Would you please evaluate my evaluation process. Note: I was using image files (128p, greyscale image) for training and so I need to resize the test data set too!). \n\nAnd by the way, I was following your score on the public LB. **98 to 99**, congrats, awesome indeed. Any pro tips 😄 \n\n## Test Data Generator \n\n```\ndef test_generator(df, batch_size):\n    num_imgs = len(df)\n\n    for batch_start in range(0, num_imgs, batch_size):\n        curr_batch_size = min(num_imgs, batch_start + batch_size) - batch_start\n        idx = np.arange(batch_start, batch_start + curr_batch_size)\n\n        names_batch = df.iloc[idx, 0].values\n        imgs_batch = 255 - df.iloc[idx, 1:].values.reshape(-1, HEIGHT, WIDTH).astype(np.uint8)\n        X_batch = np.zeros((curr_batch_size, SIZE, SIZE, 1))\n        \n        for j in range(curr_batch_size):\n            img = cv2.resize(imgs_batch[j,], (SIZE, SIZE))\n            img = img[:, :, np.newaxis]\n            X_batch[j,] = img\n\n        yield X_batch, names_batch\n\n```\n\n## Parquet files, looping, prediction ...!\n\n```\nTEST = [  \n\"bengaliai-cv19/test_image_data_0.parquet\", \n\"bengaliai-cv19/test_image_data_1.parquet\",\n\"bengaliai-cv19/test_image_data_2.parquet\",\n\"bengaliai-cv19/test_image_data_3.parquet\",\n]\n\nrow_id = []\ntarget = []\n\nfor fname in TEST:\n    test_ = pd.read_parquet(fname)\n    test_gen = test_generator(test_, batch_size=batch_size)\n\n    for batch_x, batch_name in tqdm(test_gen):\n        batch_predict = model.predict(batch_x)\n        for idx, name in enumerate(batch_name):\n            row_id += [\n                f\"{name}_consonant_diacritic\",\n                f\"{name}_grapheme_root\",\n                f\"{name}_vowel_diacritic\",\n            ]\n            target += [\n                np.argmax(batch_predict[2], axis=1)[idx],\n                np.argmax(batch_predict[0], axis=1)[idx],\n                np.argmax(batch_predict[1], axis=1)[idx],\n            ]\n\n    del test_\n    gc.collect()\n    \ndf_sample = pd.DataFrame(\n    {\n        'row_id': row_id,\n        'target':target\n    },\n    columns = ['row_id','target'] \n)\ndf_sample.to_csv('submission.csv',index=False)\ngc.collect()\n```\n\nAny catch. TIA.",
      "votes": null
    },
    {
      "id": "716256",
      "postDate": "01/11/2020 13:32:43",
      "content": "<p>Thanks ;)\nI'm not so familiar with Keras but I've got an idea to debug it in this case.\njust load training data in your prediction notebook instead of test data and see what didn't work.\nbecasue the private test data has around 200k samples which is just as big as training data.</p>",
      "rawMarkdown": "Thanks ;)\nI'm not so familiar with Keras but I've got an idea to debug it in this case.\njust load training data in your prediction notebook instead of test data and see what didn't work.\nbecasue the private test data has around 200k samples which is just as big as training data.",
      "votes": null
    },
    {
      "id": "716263",
      "postDate": "01/11/2020 13:43:25",
      "content": "<p>Ok, that makes sense. I will try and inform you of this. Thanks 😃 </p>",
      "rawMarkdown": "Ok, that makes sense. I will try and inform you of this. Thanks 😃",
      "votes": null
    },
    {
      "id": "716689",
      "postDate": "01/12/2020 05:20:13",
      "content": "<p>can you please refer me to the part saying that test is done on hidden 200k samples</p>",
      "rawMarkdown": "can you please refer me to the part saying that test is done on hidden 200k samples",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 715032,
      "author_name": "abyaadrafid",
      "author_url": "",
      "post_date": "01/10/2020 03:51:47",
      "content": "<p>As this is a kernel-only competition, you need to submit with your model weights. When you submit, the kernel is run on a private test set, which is not given to the contestants. So submitting only predictions will not suffice, you'll need the model too.</p>\n\n<p>If you can't upload your model as a notebook kernel, upload it as a script.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 715640,
      "author_name": "sashraf93",
      "author_url": "",
      "post_date": "01/10/2020 17:22:28",
      "content": "<p>i am afraid neither notebook or script accept my model, I don't know what to do</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 716140,
      "author_name": "haqishen",
      "author_url": "",
      "post_date": "01/11/2020 10:01:51",
      "content": "<p>because you only have 12 samples of test data offline, if you just copy the prediction of those samples you will got scoring error.\nwhen you submit your kernel, the kernel will be automatically rerun on about 200k samples which is invisible to you, then calculate the score.\njust take a look at public notebooks that got a score to see how they got submissions successful.</p>",
      "votes": null,
      "replies": [
        {
          "id": 716184,
          "author_name": "ipythonx",
          "author_url": "",
          "post_date": "01/11/2020 11:21:16",
          "content": "<p><a href=\"/haqishen\">@haqishen</a> , hello. I'm facing kinda weird submission result and I'm guessing it may because of my evaluation process. I followed many processing pipelines for this but most of the time it behaved wrongly. Would you please evaluate my evaluation process. Note: I was using image files (128p, greyscale image) for training and so I need to resize the test data set too!). </p>\n\n<p>And by the way, I was following your score on the public LB. <strong>98 to 99</strong>, congrats, awesome indeed. Any pro tips 😄 </p>\n\n<h2>Test Data Generator</h2>\n\n<p>```\ndef test_generator(df, batch_size):\n    num_imgs = len(df)</p>\n\n<pre><code>for batch_start in range(0, num_imgs, batch_size):\n    curr_batch_size = min(num_imgs, batch_start + batch_size) - batch_start\n    idx = np.arange(batch_start, batch_start + curr_batch_size)\n\n    names_batch = df.iloc[idx, 0].values\n    imgs_batch = 255 - df.iloc[idx, 1:].values.reshape(-1, HEIGHT, WIDTH).astype(np.uint8)\n    X_batch = np.zeros((curr_batch_size, SIZE, SIZE, 1))\n\n    for j in range(curr_batch_size):\n        img = cv2.resize(imgs_batch[j,], (SIZE, SIZE))\n        img = img[:, :, np.newaxis]\n        X_batch[j,] = img\n\n    yield X_batch, names_batch\n</code></pre>\n\n<p>```</p>\n\n<h2>Parquet files, looping, prediction ...!</h2>\n\n<p>```\nTEST = [ <br>\n\"bengaliai-cv19/test_image_data_0.parquet\", \n\"bengaliai-cv19/test_image_data_1.parquet\",\n\"bengaliai-cv19/test_image_data_2.parquet\",\n\"bengaliai-cv19/test_image_data_3.parquet\",\n]</p>\n\n<p>row_id = []\ntarget = []</p>\n\n<p>for fname in TEST:\n    test_ = pd.read_parquet(fname)\n    test_gen = test_generator(test_, batch_size=batch_size)</p>\n\n<pre><code>for batch_x, batch_name in tqdm(test_gen):\n    batch_predict = model.predict(batch_x)\n    for idx, name in enumerate(batch_name):\n        row_id += [\n            f\"{name}_consonant_diacritic\",\n            f\"{name}_grapheme_root\",\n            f\"{name}_vowel_diacritic\",\n        ]\n        target += [\n            np.argmax(batch_predict[2], axis=1)[idx],\n            np.argmax(batch_predict[0], axis=1)[idx],\n            np.argmax(batch_predict[1], axis=1)[idx],\n        ]\n\ndel test_\ngc.collect()\n</code></pre>\n\n<p>df_sample = pd.DataFrame(\n    {\n        'row_id': row_id,\n        'target':target\n    },\n    columns = ['row_id','target'] \n)\ndf_sample.to_csv('submission.csv',index=False)\ngc.collect()\n```</p>\n\n<p>Any catch. TIA. </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 716256,
          "author_name": "haqishen",
          "author_url": "",
          "post_date": "01/11/2020 13:32:43",
          "content": "<p>Thanks ;)\nI'm not so familiar with Keras but I've got an idea to debug it in this case.\njust load training data in your prediction notebook instead of test data and see what didn't work.\nbecasue the private test data has around 200k samples which is just as big as training data.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 716263,
          "author_name": "ipythonx",
          "author_url": "",
          "post_date": "01/11/2020 13:43:25",
          "content": "<p>Ok, that makes sense. I will try and inform you of this. Thanks 😃 </p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 716689,
      "author_name": "sashraf93",
      "author_url": "",
      "post_date": "01/12/2020 05:20:13",
      "content": "<p>can you please refer me to the part saying that test is done on hidden 200k samples</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "714888": "I have done my training local on my machine using certain framework, then i got my predictions and uploaded them in the notebook by writing them into a csv file to be able to submit them but it always giving me scoring error with no details about the error!!! BTW i can't upload my model as the notebooks don't support this frame work",
    "715032": "As this is a kernel-only competition, you need to submit with your model weights. When you submit, the kernel is run on a private test set, which is not given to the contestants. So submitting only predictions will not suffice, you'll need the model too.\n\nIf you can't upload your model as a notebook kernel, upload it as a script.",
    "715640": "i am afraid neither notebook or script accept my model, I don't know what to do",
    "716140": "because you only have 12 samples of test data offline, if you just copy the prediction of those samples you will got scoring error.\nwhen you submit your kernel, the kernel will be automatically rerun on about 200k samples which is invisible to you, then calculate the score.\njust take a look at public notebooks that got a score to see how they got submissions successful.",
    "716184": "haqishen , hello. I'm facing kinda weird submission result and I'm guessing it may because of my evaluation process. I followed many processing pipelines for this but most of the time it behaved wrongly. Would you please evaluate my evaluation process. Note: I was using image files (128p, greyscale image) for training and so I need to resize the test data set too!). \n\nAnd by the way, I was following your score on the public LB. **98 to 99**, congrats, awesome indeed. Any pro tips 😄 \n\n## Test Data Generator \n\n```\ndef test_generator(df, batch_size):\n    num_imgs = len(df)\n\n    for batch_start in range(0, num_imgs, batch_size):\n        curr_batch_size = min(num_imgs, batch_start + batch_size) - batch_start\n        idx = np.arange(batch_start, batch_start + curr_batch_size)\n\n        names_batch = df.iloc[idx, 0].values\n        imgs_batch = 255 - df.iloc[idx, 1:].values.reshape(-1, HEIGHT, WIDTH).astype(np.uint8)\n        X_batch = np.zeros((curr_batch_size, SIZE, SIZE, 1))\n        \n        for j in range(curr_batch_size):\n            img = cv2.resize(imgs_batch[j,], (SIZE, SIZE))\n            img = img[:, :, np.newaxis]\n            X_batch[j,] = img\n\n        yield X_batch, names_batch\n\n```\n\n## Parquet files, looping, prediction ...!\n\n```\nTEST = [  \n\"bengaliai-cv19/test_image_data_0.parquet\", \n\"bengaliai-cv19/test_image_data_1.parquet\",\n\"bengaliai-cv19/test_image_data_2.parquet\",\n\"bengaliai-cv19/test_image_data_3.parquet\",\n]\n\nrow_id = []\ntarget = []\n\nfor fname in TEST:\n    test_ = pd.read_parquet(fname)\n    test_gen = test_generator(test_, batch_size=batch_size)\n\n    for batch_x, batch_name in tqdm(test_gen):\n        batch_predict = model.predict(batch_x)\n        for idx, name in enumerate(batch_name):\n            row_id += [\n                f\"{name}_consonant_diacritic\",\n                f\"{name}_grapheme_root\",\n                f\"{name}_vowel_diacritic\",\n            ]\n            target += [\n                np.argmax(batch_predict[2], axis=1)[idx],\n                np.argmax(batch_predict[0], axis=1)[idx],\n                np.argmax(batch_predict[1], axis=1)[idx],\n            ]\n\n    del test_\n    gc.collect()\n    \ndf_sample = pd.DataFrame(\n    {\n        'row_id': row_id,\n        'target':target\n    },\n    columns = ['row_id','target'] \n)\ndf_sample.to_csv('submission.csv',index=False)\ngc.collect()\n```\n\nAny catch. TIA.",
    "716256": "Thanks ;)\nI'm not so familiar with Keras but I've got an idea to debug it in this case.\njust load training data in your prediction notebook instead of test data and see what didn't work.\nbecasue the private test data has around 200k samples which is just as big as training data.",
    "716263": "Ok, that makes sense. I will try and inform you of this. Thanks 😃",
    "716689": "can you please refer me to the part saying that test is done on hidden 200k samples"
  },
  "source": "meta"
}