{
  "id": 238364,
  "title": "10th Place Solution",
  "url": "/competitions/hpa-single-cell-image-classification/discussion/238364",
  "author_name": "ONODERA",
  "post_date": "2021-05-12T02:13:24.904000",
  "votes": 27,
  "comment_count": 4,
  "views": 0,
  "content": "<h2>Pipeline</h2>\n<p><img src=\"https://pbs.twimg.com/media/E1JXDBBUcAItGu9?format=jpg&amp;name=large\" alt=\"pipeline\"></p>\n<h2>Filtering</h2>\n<pre><code>    # relabel and remove unmatched cell\n    ## single\n    gr = train_single.groupby(\"image_id\")\n    cell_size = gr.cell_id.transform(max)\n    for c1, c2, c3 in zip(COLS_TARGET, COLS_RELABEL, COLS_PRED):\n        # 1\n        re1 = (\n                train_single[c1] * ((gr[c3].rank() / cell_size) &gt; 0.5)\n                |\n                train_single[c1] * (train_single[c3] &gt;= 0.5)\n        )\n        re1.loc[re1==0] = np.nan\n        # 0\n        re0 = (\n                train_single[c1] * ((gr[c3].rank() / cell_size) &lt; 0.1)\n                &amp;\n                train_single[c1] * (train_single[c3] &lt; 0.1)\n        )\n        re0.loc[re0==0] = np.nan\n        re0.loc[re0==1] = 0\n        train_single[c2] = pd.concat([re0, re1], axis=1).max(1)\n        train_single.loc[train_single[c1]==0, c2] = 0\n    train_single = train_single.dropna().reset_index(drop=True)\n\n    ## multi\n    import cudf\n    cdf = cudf.DataFrame(train_multi[['image_id'] + COLS_PRED])\n    pred_quantile1 = cdf.groupby(\"image_id\").agg(lambda x: x.quantile(0.50)).to_pandas()\n    pred_quantile1 = train_multi[['image_id']].merge(pred_quantile1, on='image_id', how='left')\n    pred_quantile0 = cdf.groupby(\"image_id\").agg(lambda x: x.quantile(0.50)).to_pandas()\n    pred_quantile0 = train_multi[['image_id']].merge(pred_quantile0, on='image_id', how='left')\n    for c1, c2, c3 in zip(COLS_TARGET, COLS_RELABEL, COLS_PRED):\n        # 1\n        re1 = (\n                train_multi[c1] * (train_multi[c3] &gt;= pred_quantile1[c3])\n                |\n                train_multi[c1] * (train_multi[c3] &gt;= 0.5)\n        )\n        re1.loc[re1==0] = np.nan\n        # 0\n        re0 = (\n                train_multi[c1] * (train_multi[c3] &lt; pred_quantile0[c3])\n                &amp;\n                train_multi[c1] * (train_multi[c3] &lt; 0.1)\n        )\n        re0.loc[re0==0] = np.nan\n        re0.loc[re0==1] = 0\n        train_multi[c2] = pd.concat([re0, re1], axis=1).max(1)\n        train_multi.loc[train_multi[c1]==0, c2] = 0\n    train_multi = train_multi.dropna().reset_index(drop=True)\n</code></pre>\n<h2>Final Submission</h2>\n<p><a href=\"https://www.kaggle.com/inoueu1/hpa2021-p-all-1stx8-2ndx16-imgx16/notebook?scriptVersionId=62576779\" target=\"_blank\">https://www.kaggle.com/inoueu1/hpa2021-p-all-1stx8-2ndx16-imgx16/notebook?scriptVersionId=62576779</a></p>",
  "messages": [
    {
      "id": 1303295,
      "postDate": "2021-05-12T02:13:24.903Z",
      "content": "<h2>Pipeline</h2>\n<p><img src=\"https://pbs.twimg.com/media/E1JXDBBUcAItGu9?format=jpg&amp;name=large\" alt=\"pipeline\"></p>\n<h2>Filtering</h2>\n<pre><code>    # relabel and remove unmatched cell\n    ## single\n    gr = train_single.groupby(\"image_id\")\n    cell_size = gr.cell_id.transform(max)\n    for c1, c2, c3 in zip(COLS_TARGET, COLS_RELABEL, COLS_PRED):\n        # 1\n        re1 = (\n                train_single[c1] * ((gr[c3].rank() / cell_size) &gt; 0.5)\n                |\n                train_single[c1] * (train_single[c3] &gt;= 0.5)\n        )\n        re1.loc[re1==0] = np.nan\n        # 0\n        re0 = (\n                train_single[c1] * ((gr[c3].rank() / cell_size) &lt; 0.1)\n                &amp;\n                train_single[c1] * (train_single[c3] &lt; 0.1)\n        )\n        re0.loc[re0==0] = np.nan\n        re0.loc[re0==1] = 0\n        train_single[c2] = pd.concat([re0, re1], axis=1).max(1)\n        train_single.loc[train_single[c1]==0, c2] = 0\n    train_single = train_single.dropna().reset_index(drop=True)\n\n    ## multi\n    import cudf\n    cdf = cudf.DataFrame(train_multi[['image_id'] + COLS_PRED])\n    pred_quantile1 = cdf.groupby(\"image_id\").agg(lambda x: x.quantile(0.50)).to_pandas()\n    pred_quantile1 = train_multi[['image_id']].merge(pred_quantile1, on='image_id', how='left')\n    pred_quantile0 = cdf.groupby(\"image_id\").agg(lambda x: x.quantile(0.50)).to_pandas()\n    pred_quantile0 = train_multi[['image_id']].merge(pred_quantile0, on='image_id', how='left')\n    for c1, c2, c3 in zip(COLS_TARGET, COLS_RELABEL, COLS_PRED):\n        # 1\n        re1 = (\n                train_multi[c1] * (train_multi[c3] &gt;= pred_quantile1[c3])\n                |\n                train_multi[c1] * (train_multi[c3] &gt;= 0.5)\n        )\n        re1.loc[re1==0] = np.nan\n        # 0\n        re0 = (\n                train_multi[c1] * (train_multi[c3] &lt; pred_quantile0[c3])\n                &amp;\n                train_multi[c1] * (train_multi[c3] &lt; 0.1)\n        )\n        re0.loc[re0==0] = np.nan\n        re0.loc[re0==1] = 0\n        train_multi[c2] = pd.concat([re0, re1], axis=1).max(1)\n        train_multi.loc[train_multi[c1]==0, c2] = 0\n    train_multi = train_multi.dropna().reset_index(drop=True)\n</code></pre>\n<h2>Final Submission</h2>\n<p><a href=\"https://www.kaggle.com/inoueu1/hpa2021-p-all-1stx8-2ndx16-imgx16/notebook?scriptVersionId=62576779\" target=\"_blank\">https://www.kaggle.com/inoueu1/hpa2021-p-all-1stx8-2ndx16-imgx16/notebook?scriptVersionId=62576779</a></p>",
      "rawMarkdown": "## Pipeline\n![pipeline](https://pbs.twimg.com/media/E1JXDBBUcAItGu9?format=jpg&name=large)\n\n## Filtering\n```\n    # relabel and remove unmatched cell\n    ## single\n    gr = train_single.groupby(\"image_id\")\n    cell_size = gr.cell_id.transform(max)\n    for c1, c2, c3 in zip(COLS_TARGET, COLS_RELABEL, COLS_PRED):\n        # 1\n        re1 = (\n                train_single[c1] * ((gr[c3].rank() / cell_size) > 0.5)\n                |\n                train_single[c1] * (train_single[c3] >= 0.5)\n        )\n        re1.loc[re1==0] = np.nan\n        # 0\n        re0 = (\n                train_single[c1] * ((gr[c3].rank() / cell_size) < 0.1)\n                &\n                train_single[c1] * (train_single[c3] < 0.1)\n        )\n        re0.loc[re0==0] = np.nan\n        re0.loc[re0==1] = 0\n        train_single[c2] = pd.concat([re0, re1], axis=1).max(1)\n        train_single.loc[train_single[c1]==0, c2] = 0\n    train_single = train_single.dropna().reset_index(drop=True)\n\n    ## multi\n    import cudf\n    cdf = cudf.DataFrame(train_multi[['image_id'] + COLS_PRED])\n    pred_quantile1 = cdf.groupby(\"image_id\").agg(lambda x: x.quantile(0.50)).to_pandas()\n    pred_quantile1 = train_multi[['image_id']].merge(pred_quantile1, on='image_id', how='left')\n    pred_quantile0 = cdf.groupby(\"image_id\").agg(lambda x: x.quantile(0.50)).to_pandas()\n    pred_quantile0 = train_multi[['image_id']].merge(pred_quantile0, on='image_id', how='left')\n    for c1, c2, c3 in zip(COLS_TARGET, COLS_RELABEL, COLS_PRED):\n        # 1\n        re1 = (\n                train_multi[c1] * (train_multi[c3] >= pred_quantile1[c3])\n                |\n                train_multi[c1] * (train_multi[c3] >= 0.5)\n        )\n        re1.loc[re1==0] = np.nan\n        # 0\n        re0 = (\n                train_multi[c1] * (train_multi[c3] < pred_quantile0[c3])\n                &\n                train_multi[c1] * (train_multi[c3] < 0.1)\n        )\n        re0.loc[re0==0] = np.nan\n        re0.loc[re0==1] = 0\n        train_multi[c2] = pd.concat([re0, re1], axis=1).max(1)\n        train_multi.loc[train_multi[c1]==0, c2] = 0\n    train_multi = train_multi.dropna().reset_index(drop=True)\n\n```\n\n## Final Submission\nhttps://www.kaggle.com/inoueu1/hpa2021-p-all-1stx8-2ndx16-imgx16/notebook?scriptVersionId=62576779\n",
      "votes": 27
    },
    {
      "id": 1303347,
      "postDate": "2021-05-12T03:10:01.607Z",
      "content": "<p><a href=\"https://www.kaggle.com/onodera\" target=\"_blank\">@onodera</a>  Congratulations and Thanks for sharing your approach </p>",
      "rawMarkdown": "@onodera  Congratulations and Thanks for sharing your approach ",
      "votes": 2
    },
    {
      "id": 1308076,
      "postDate": "2021-05-15T00:42:53.720Z",
      "content": "<p>Congratulations Onodera and team. Well done!</p>",
      "rawMarkdown": "Congratulations Onodera and team. Well done!",
      "votes": 2
    },
    {
      "id": 1306159,
      "postDate": "2021-05-13T16:31:37.387Z",
      "content": "<p>Wonderful write-up. Very clean. Thank you for sharing!</p>",
      "rawMarkdown": "Wonderful write-up. Very clean. Thank you for sharing!",
      "votes": 2,
      "replies": [
        {
          "id": 1307338,
          "postDate": "2021-05-14T11:35:36.490Z",
          "content": "<p>Thanks, our solution came from quite simple pipeline.<br>\nNow we understand it's not necessary to make complex networks to get 0.53~0.54 on private LB.</p>",
          "rawMarkdown": "Thanks, our solution came from quite simple pipeline.\nNow we understand it's not necessary to make complex networks to get 0.53~0.54 on private LB.",
          "votes": 1
        }
      ]
    }
  ],
  "comments": [
    {
      "id": 1303347,
      "author_name": "Tensor Girl",
      "author_url": "",
      "post_date": "2021-05-12T03:10:01.607000",
      "content": "<p><a href=\"https://www.kaggle.com/onodera\" target=\"_blank\">@onodera</a>  Congratulations and Thanks for sharing your approach </p>",
      "votes": 2,
      "replies": []
    },
    {
      "id": 1308076,
      "author_name": "Chris Deotte",
      "author_url": "",
      "post_date": "2021-05-15T00:42:53.720000",
      "content": "<p>Congratulations Onodera and team. Well done!</p>",
      "votes": 2,
      "replies": []
    },
    {
      "id": 1306159,
      "author_name": "Darien Schettler",
      "author_url": "",
      "post_date": "2021-05-13T16:31:37.387000",
      "content": "<p>Wonderful write-up. Very clean. Thank you for sharing!</p>",
      "votes": 2,
      "replies": [
        {
          "id": 1307338,
          "author_name": "ONODERA",
          "author_url": "",
          "post_date": "2021-05-14T11:35:36.490000",
          "content": "<p>Thanks, our solution came from quite simple pipeline.<br>\nNow we understand it's not necessary to make complex networks to get 0.53~0.54 on private LB.</p>",
          "votes": 1,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1303295": "## Pipeline\n![pipeline](https://pbs.twimg.com/media/E1JXDBBUcAItGu9?format=jpg&name=large)\n\n## Filtering\n```\n    # relabel and remove unmatched cell\n    ## single\n    gr = train_single.groupby(\"image_id\")\n    cell_size = gr.cell_id.transform(max)\n    for c1, c2, c3 in zip(COLS_TARGET, COLS_RELABEL, COLS_PRED):\n        # 1\n        re1 = (\n                train_single[c1] * ((gr[c3].rank() / cell_size) > 0.5)\n                |\n                train_single[c1] * (train_single[c3] >= 0.5)\n        )\n        re1.loc[re1==0] = np.nan\n        # 0\n        re0 = (\n                train_single[c1] * ((gr[c3].rank() / cell_size) < 0.1)\n                &\n                train_single[c1] * (train_single[c3] < 0.1)\n        )\n        re0.loc[re0==0] = np.nan\n        re0.loc[re0==1] = 0\n        train_single[c2] = pd.concat([re0, re1], axis=1).max(1)\n        train_single.loc[train_single[c1]==0, c2] = 0\n    train_single = train_single.dropna().reset_index(drop=True)\n\n    ## multi\n    import cudf\n    cdf = cudf.DataFrame(train_multi[['image_id'] + COLS_PRED])\n    pred_quantile1 = cdf.groupby(\"image_id\").agg(lambda x: x.quantile(0.50)).to_pandas()\n    pred_quantile1 = train_multi[['image_id']].merge(pred_quantile1, on='image_id', how='left')\n    pred_quantile0 = cdf.groupby(\"image_id\").agg(lambda x: x.quantile(0.50)).to_pandas()\n    pred_quantile0 = train_multi[['image_id']].merge(pred_quantile0, on='image_id', how='left')\n    for c1, c2, c3 in zip(COLS_TARGET, COLS_RELABEL, COLS_PRED):\n        # 1\n        re1 = (\n                train_multi[c1] * (train_multi[c3] >= pred_quantile1[c3])\n                |\n                train_multi[c1] * (train_multi[c3] >= 0.5)\n        )\n        re1.loc[re1==0] = np.nan\n        # 0\n        re0 = (\n                train_multi[c1] * (train_multi[c3] < pred_quantile0[c3])\n                &\n                train_multi[c1] * (train_multi[c3] < 0.1)\n        )\n        re0.loc[re0==0] = np.nan\n        re0.loc[re0==1] = 0\n        train_multi[c2] = pd.concat([re0, re1], axis=1).max(1)\n        train_multi.loc[train_multi[c1]==0, c2] = 0\n    train_multi = train_multi.dropna().reset_index(drop=True)\n\n```\n\n## Final Submission\nhttps://www.kaggle.com/inoueu1/hpa2021-p-all-1stx8-2ndx16-imgx16/notebook?scriptVersionId=62576779\n",
    "1303347": "@onodera  Congratulations and Thanks for sharing your approach ",
    "1308076": "Congratulations Onodera and team. Well done!",
    "1306159": "Wonderful write-up. Very clean. Thank you for sharing!"
  }
}