{ "id": 224935, "title": "Can we do a model vote for the least accurate columns？", "url": "/competitions/ranzcr-clip-catheter-line-classification/discussion/224935", "author_name": "", "post_date": "2021-03-10T08:56:22.566973Z", "votes": 3, "comment_count": 6, "views": 0, "content": "

Obviously, most of the loss errors in the validation set were concentrated in the 'CVC-Borderline' column when training RESNET-200D, so I wondered if the prediction accuracy would be improved if multiple models with high LB scores were used for hard voting for this column rather than soft summations.

", "messages": [ { "id": "1233245", "postDate": "03/10/2021 08:56:22", "content": "

Obviously, most of the loss errors in the validation set were concentrated in the 'CVC-Borderline' column when training RESNET-200D, so I wondered if the prediction accuracy would be improved if multiple models with high LB scores were used for hard voting for this column rather than soft summations.

", "rawMarkdown": "Obviously, most of the loss errors in the validation set were concentrated in the 'CVC-Borderline' column when training RESNET-200D, so I wondered if the prediction accuracy would be improved if multiple models with high LB scores were used for hard voting for this column rather than soft summations.", "votes": null }, { "id": "1233477", "postDate": "03/10/2021 13:02:06", "content": "

Since the competition metric requires a ranking of records for each target rather than 0 or 1, I'm not sure that would really help. I guess, if you classify really well than all 1 for the positive cases and all 0 for the negative ones will be a really good ranking… I am doubtful though, but I've could try via CV.

", "rawMarkdown": "Since the competition metric requires a ranking of records for each target rather than 0 or 1, I'm not sure that would really help. I guess, if you classify really well than all 1 for the positive cases and all 0 for the negative ones will be a really good ranking... I am doubtful though, but I've could try via CV.", "votes": null }, { "id": "1233591", "postDate": "03/10/2021 14:38:34", "content": "

I try to change some doubtful answers that various from high score model to 0 or 1. And others will not do anything. I am waiting for the results from LB. If the score always be obtained from a rank number，I could do a mistake.

", "rawMarkdown": "I try to change some doubtful answers that various from high score model to 0 or 1. And others will not do anything. I am waiting for the results from LB. If the score always be obtained from a rank number，I could do a mistake.", "votes": null }, { "id": "1233593", "postDate": "03/10/2021 14:41:14", "content": "

Anyway, it may be some trick for the least column, like 'CVC-Borderline', and i have not found it

", "rawMarkdown": "Anyway, it may be some trick for the least column, like 'CVC-Borderline', and i have not found it", "votes": null }, { "id": "1233596", "postDate": "03/10/2021 14:45:33", "content": "

Here's a previous discussion post with a bunch of ideas one could use that may work better than arithmetic averages. Additionally, I would try to test this via your cross-validation, the LB is a less reliable indicator due to the smaller sample size (the CV outcome is based on almost 9 times more images).

", "rawMarkdown": "Here's a previous discussion post with a [bunch of ideas](https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/211221) one could use that may work better than arithmetic averages. Additionally, I would try to test this via your cross-validation, the LB is a less reliable indicator due to the smaller sample size (the CV outcome is based on almost 9 times more images).", "votes": null }, { "id": "1233643", "postDate": "03/10/2021 15:35:49", "content": "

Yes, it may be different. I just try about the median score between them.

", "rawMarkdown": "Yes, it may be different. I just try about the median score between them.", "votes": null }, { "id": "1241449", "postDate": "03/17/2021 03:39:08", "content": "

It should be worked when i use the magic method in my submission like this：

\n
    \n
  1. Use your local or public models to do 3~5 times simple weight average, you will get some subs;
    2. \n
  2. Convert each committed value to 1/0 using np.round and astype('int');
    3. \n
  3. Set up a meta-sub which is your best score with average before;
    4. \n
  4. Extract indexes (union set of all subs) of META-SUB and other SUB that have different values in three columns：'CVC - Borderline', 'CVC - Normal', and 'CVC - Abnormal';
    5. \n
  5. Use np.median or hard-voting's results as the value of the three columns in the index of the meta-sub
    6. \n
\n

It shows a little improvement on my submission (0.968->0.969, public & 0.970->0.971, private).
\nBut i think this way don't work all the time especially when i use low-cv model or power average. :)

", "rawMarkdown": "It should be worked when i use the magic method in my submission like this：\n1. Use your local or public models to do 3~5 times simple weight average, you will get some subs;\n2. Convert each committed value to 1/0 using np.round and astype('int');\n3. Set up a meta-sub which is your best score with average before;\n4. Extract indexes (union set of all subs) of META-SUB and other SUB that have different values in three columns：'CVC - Borderline', 'CVC - Normal', and 'CVC - Abnormal';\n5. Use np.median or hard-voting's results as the value of the three columns in the index of the meta-sub \n\nIt shows a little improvement on my submission (0.968->0.969, public & 0.970->0.971, private).\nBut i think this way don't work all the time especially when i use low-cv model or power average. :)", "votes": null } ], "comments": [ { "id": 1233477, "author_name": "bjoernholzhauer", "author_url": "", "post_date": "03/10/2021 13:02:06", "content": "

Since the competition metric requires a ranking of records for each target rather than 0 or 1, I'm not sure that would really help. I guess, if you classify really well than all 1 for the positive cases and all 0 for the negative ones will be a really good ranking… I am doubtful though, but I've could try via CV.

", "votes": null, "replies": [ { "id": 1233591, "author_name": "zhangyunsheng", "author_url": "", "post_date": "03/10/2021 14:38:34", "content": "

I try to change some doubtful answers that various from high score model to 0 or 1. And others will not do anything. I am waiting for the results from LB. If the score always be obtained from a rank number，I could do a mistake.

", "votes": null, "replies": [] }, { "id": 1233593, "author_name": "zhangyunsheng", "author_url": "", "post_date": "03/10/2021 14:41:14", "content": "

Anyway, it may be some trick for the least column, like 'CVC-Borderline', and i have not found it

", "votes": null, "replies": [] }, { "id": 1233596, "author_name": "bjoernholzhauer", "author_url": "", "post_date": "03/10/2021 14:45:33", "content": "

Here's a previous discussion post with a bunch of ideas one could use that may work better than arithmetic averages. Additionally, I would try to test this via your cross-validation, the LB is a less reliable indicator due to the smaller sample size (the CV outcome is based on almost 9 times more images).

", "votes": null, "replies": [] }, { "id": 1233643, "author_name": "zhangyunsheng", "author_url": "", "post_date": "03/10/2021 15:35:49", "content": "

Yes, it may be different. I just try about the median score between them.

", "votes": null, "replies": [] } ] }, { "id": 1241449, "author_name": "zhangyunsheng", "author_url": "", "post_date": "03/17/2021 03:39:08", "content": "

It should be worked when i use the magic method in my submission like this：

\n
    \n
  1. Use your local or public models to do 3~5 times simple weight average, you will get some subs;
    2. \n
  2. Convert each committed value to 1/0 using np.round and astype('int');
    3. \n
  3. Set up a meta-sub which is your best score with average before;
    4. \n
  4. Extract indexes (union set of all subs) of META-SUB and other SUB that have different values in three columns：'CVC - Borderline', 'CVC - Normal', and 'CVC - Abnormal';
    5. \n
  5. Use np.median or hard-voting's results as the value of the three columns in the index of the meta-sub
    6. \n
\n

It shows a little improvement on my submission (0.968->0.969, public & 0.970->0.971, private).
\nBut i think this way don't work all the time especially when i use low-cv model or power average. :)

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1233245": "Obviously, most of the loss errors in the validation set were concentrated in the 'CVC-Borderline' column when training RESNET-200D, so I wondered if the prediction accuracy would be improved if multiple models with high LB scores were used for hard voting for this column rather than soft summations.", "1233477": "Since the competition metric requires a ranking of records for each target rather than 0 or 1, I'm not sure that would really help. I guess, if you classify really well than all 1 for the positive cases and all 0 for the negative ones will be a really good ranking... I am doubtful though, but I've could try via CV.", "1233591": "I try to change some doubtful answers that various from high score model to 0 or 1. And others will not do anything. I am waiting for the results from LB. If the score always be obtained from a rank number，I could do a mistake.", "1233593": "Anyway, it may be some trick for the least column, like 'CVC-Borderline', and i have not found it", "1233596": "Here's a previous discussion post with a [bunch of ideas](https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/211221) one could use that may work better than arithmetic averages. Additionally, I would try to test this via your cross-validation, the LB is a less reliable indicator due to the smaller sample size (the CV outcome is based on almost 9 times more images).", "1233643": "Yes, it may be different. I just try about the median score between them.", "1241449": "It should be worked when i use the magic method in my submission like this：\n1. Use your local or public models to do 3~5 times simple weight average, you will get some subs;\n2. Convert each committed value to 1/0 using np.round and astype('int');\n3. Set up a meta-sub which is your best score with average before;\n4. Extract indexes (union set of all subs) of META-SUB and other SUB that have different values in three columns：'CVC - Borderline', 'CVC - Normal', and 'CVC - Abnormal';\n5. Use np.median or hard-voting's results as the value of the three columns in the index of the meta-sub \n\nIt shows a little improvement on my submission (0.968->0.969, public & 0.970->0.971, private).\nBut i think this way don't work all the time especially when i use low-cv model or power average. :)" }, "source": "meta" }