Since the evalution metric is MCC, submission with all 0 (or all 1) will give zero MCC. So, is there anyway that we can probe LB about Fault ratio?\nI guess if we know the ratio, is would help to choose best threshold on test set (or maybe overfit public LB ... ).
", "messages": [ { "id": "483680", "postDate": "03/05/2019 01:26:35", "content": "Since the evalution metric is MCC, submission with all 0 (or all 1) will give zero MCC. So, is there anyway that we can probe LB about Fault ratio?\nI guess if we know the ratio, is would help to choose best threshold on test set (or maybe overfit public LB ... ).
", "rawMarkdown": "Since the evalution metric is MCC, submission with all 0 (or all 1) will give zero MCC. So, is there anyway that we can probe LB about Fault ratio?\nI guess if we know the ratio, is would help to choose best threshold on test set (or maybe overfit public LB ... ).", "votes": null }, { "id": "486019", "postDate": "03/08/2019 08:06:38", "content": "Hi @zjucor, there is a way to probe the LB.\nIf all instances are set to class 0 except only one instance to class 1, there would be 3 cases as follows.
\n\nIf the instance of class 1 is in the private LB, then the probing MCC would be 0 because what the public LB sees are all class 0.
If the instance of class 1 is correct, then TP = 1, TN = N0, FP = 0, FN = N1-1.\nWhen we use these values in the MCC formula, we get MCC = N0 / (N1 * N0 * (N-1))^(1/2)
If the instance of class 1 is incorrect (actually it is class 0), then TP = 0, TN = N0-1, FP = 1, FN = N1.\nWhen we use these values in the MCC formula, we get MCC = -N1 / (N1 * N0 * (N-1))^(1/2)
(N is the number of all data in the public LB, N0 is the number of class 0 in the public LB and N1 is the number of class 1 in the public LB)
", "rawMarkdown": "Hi @zjucor, there is a way to probe the LB.\nIf all instances are set to class 0 except only one instance to class 1, there would be 3 cases as follows.\n\n- If the instance of class 1 is in the private LB, then the probing MCC would be 0 because what the public LB sees are all class 0.\n\n- If the instance of class 1 is correct, then TP = 1, TN = N0, FP = 0, FN = N1-1.\nWhen we use these values in the MCC formula, we get MCC = N0 / (N1 * N0 * (N-1))^(1/2)\n\n- If the instance of class 1 is incorrect (actually it is class 0), then TP = 0, TN = N0-1, FP = 1, FN = N1.\nWhen we use these values in the MCC formula, we get MCC = -N1 / (N1 * N0 * (N-1))^(1/2)\n\n(N is the number of all data in the public LB, N0 is the number of class 0 in the public LB and N1 is the number of class 1 in the public LB)", "votes": null }, { "id": "486055", "postDate": "03/08/2019 08:55:30", "content": "Hi, thanks for your reply, \n1. this is indeed a way to prob lb, but the cost is huge.\n2. And prob **private **test set maybe regarded as cheating\n3. what i want to prob is simply the ratio of class 0/ class 1. If the test metric is accuracy, we can simply get this ratio in one submission\n4. I guess this kind of probing is quite usual in other competetions. It seems that this competetion intentionally forces use not to do so.
", "rawMarkdown": "Hi, thanks for your reply, \n1. this is indeed a way to prob lb, but the cost is huge.\n2. And prob **private **test set maybe regarded as cheating\n3. what i want to prob is simply the ratio of class 0/ class 1. If the test metric is accuracy, we can simply get this ratio in one submission\n4. I guess this kind of probing is quite usual in other competetions. It seems that this competetion intentionally forces use not to do so.", "votes": null }, { "id": "486063", "postDate": "03/08/2019 09:10:18", "content": "Hi @zjucor.
\n\n1., 3. I agree that the cost is huge for checking only the ratio. There maybe some genius ways out there to check the ratio in one submission.
\n\n2., 4. I'm quite new here, so I'm not sure whether this method is considered inappropriate.
", "rawMarkdown": "Hi @zjucor.\n\n1., 3. I agree that the cost is huge for checking only the ratio. There maybe some genius ways out there to check the ratio in one submission.\n\n2., 4. I'm quite new here, so I'm not sure whether this method is considered inappropriate.", "votes": null }, { "id": "486067", "postDate": "03/08/2019 09:21:35", "content": "check this discussion for possible hints: Different distribution labels on test set?
", "rawMarkdown": "check this discussion for possible hints: *Different distribution labels on test set?*", "votes": null }, { "id": "486097", "postDate": "03/08/2019 10:12:05", "content": "thanks
", "rawMarkdown": "thanks", "votes": null }, { "id": "486102", "postDate": "03/08/2019 10:18:42", "content": "check this: https://www.kaggle.com/c/vsb-power-line-fault-detection/discussion/80674 \nit might be helpful
", "rawMarkdown": "check this: https://www.kaggle.com/c/vsb-power-line-fault-detection/discussion/80674 \nit might be helpful", "votes": null }, { "id": "487196", "postDate": "03/10/2019 10:28:37", "content": "> this is indeed a way to prob lb, but the cost is huge.
\n\nWhy do you think the cost is huge?\nSuppose you know a certain faulty signal. So you are now at the point 2 (TP = 1, TN = N0, FP = 0, FN = N1-1). You know N=0.57*20337=11592. So with a single submission you get the equation MCC=LB_ probing_score with the only one variable N1 (or N0). Assuming N1 is something like 4% of N, you should get something like 0.045 score. I didn't try it. If you wish you can try and write if it is true. :)
", "rawMarkdown": "> this is indeed a way to prob lb, but the cost is huge.\n\nWhy do you think the cost is huge?\nSuppose you know a certain faulty signal. So you are now at the point 2 (TP = 1, TN = N0, FP = 0, FN = N1-1). You know N=0.57*20337=11592. So with a single submission you get the equation MCC=LB_ probing_score with the only one variable N1 (or N0). Assuming N1 is something like 4% of N, you should get something like 0.045 score. I didn't try it. If you wish you can try and write if it is true. :)", "votes": null }, { "id": "491213", "postDate": "03/15/2019 11:00:39", "content": "yeah, you are right ...
", "rawMarkdown": "yeah, you are right ...", "votes": null } ], "comments": [ { "id": 486019, "author_name": "prasertsak", "author_url": "", "post_date": "03/08/2019 08:06:38", "content": "Hi @zjucor, there is a way to probe the LB.\nIf all instances are set to class 0 except only one instance to class 1, there would be 3 cases as follows.
\n\nIf the instance of class 1 is in the private LB, then the probing MCC would be 0 because what the public LB sees are all class 0.
If the instance of class 1 is correct, then TP = 1, TN = N0, FP = 0, FN = N1-1.\nWhen we use these values in the MCC formula, we get MCC = N0 / (N1 * N0 * (N-1))^(1/2)
If the instance of class 1 is incorrect (actually it is class 0), then TP = 0, TN = N0-1, FP = 1, FN = N1.\nWhen we use these values in the MCC formula, we get MCC = -N1 / (N1 * N0 * (N-1))^(1/2)
(N is the number of all data in the public LB, N0 is the number of class 0 in the public LB and N1 is the number of class 1 in the public LB)
", "votes": null, "replies": [ { "id": 486055, "author_name": "zjucor", "author_url": "", "post_date": "03/08/2019 08:55:30", "content": "Hi, thanks for your reply, \n1. this is indeed a way to prob lb, but the cost is huge.\n2. And prob **private **test set maybe regarded as cheating\n3. what i want to prob is simply the ratio of class 0/ class 1. If the test metric is accuracy, we can simply get this ratio in one submission\n4. I guess this kind of probing is quite usual in other competetions. It seems that this competetion intentionally forces use not to do so.
", "votes": null, "replies": [] }, { "id": 486063, "author_name": "prasertsak", "author_url": "", "post_date": "03/08/2019 09:10:18", "content": "Hi @zjucor.
\n\n1., 3. I agree that the cost is huge for checking only the ratio. There maybe some genius ways out there to check the ratio in one submission.
\n\n2., 4. I'm quite new here, so I'm not sure whether this method is considered inappropriate.
", "votes": null, "replies": [] }, { "id": 486102, "author_name": "zjucor", "author_url": "", "post_date": "03/08/2019 10:18:42", "content": "check this: https://www.kaggle.com/c/vsb-power-line-fault-detection/discussion/80674 \nit might be helpful
", "votes": null, "replies": [] }, { "id": 487196, "author_name": "sergeyzlobin", "author_url": "", "post_date": "03/10/2019 10:28:37", "content": "> this is indeed a way to prob lb, but the cost is huge.
\n\nWhy do you think the cost is huge?\nSuppose you know a certain faulty signal. So you are now at the point 2 (TP = 1, TN = N0, FP = 0, FN = N1-1). You know N=0.57*20337=11592. So with a single submission you get the equation MCC=LB_ probing_score with the only one variable N1 (or N0). Assuming N1 is something like 4% of N, you should get something like 0.045 score. I didn't try it. If you wish you can try and write if it is true. :)
", "votes": null, "replies": [] }, { "id": 491213, "author_name": "zjucor", "author_url": "", "post_date": "03/15/2019 11:00:39", "content": "yeah, you are right ...
", "votes": null, "replies": [] } ] }, { "id": 486067, "author_name": "proef2", "author_url": "", "post_date": "03/08/2019 09:21:35", "content": "check this discussion for possible hints: Different distribution labels on test set?
", "votes": null, "replies": [ { "id": 486097, "author_name": "zjucor", "author_url": "", "post_date": "03/08/2019 10:12:05", "content": "thanks
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "483680": "Since the evalution metric is MCC, submission with all 0 (or all 1) will give zero MCC. So, is there anyway that we can probe LB about Fault ratio?\nI guess if we know the ratio, is would help to choose best threshold on test set (or maybe overfit public LB ... ).", "486019": "Hi @zjucor, there is a way to probe the LB.\nIf all instances are set to class 0 except only one instance to class 1, there would be 3 cases as follows.\n\n- If the instance of class 1 is in the private LB, then the probing MCC would be 0 because what the public LB sees are all class 0.\n\n- If the instance of class 1 is correct, then TP = 1, TN = N0, FP = 0, FN = N1-1.\nWhen we use these values in the MCC formula, we get MCC = N0 / (N1 * N0 * (N-1))^(1/2)\n\n- If the instance of class 1 is incorrect (actually it is class 0), then TP = 0, TN = N0-1, FP = 1, FN = N1.\nWhen we use these values in the MCC formula, we get MCC = -N1 / (N1 * N0 * (N-1))^(1/2)\n\n(N is the number of all data in the public LB, N0 is the number of class 0 in the public LB and N1 is the number of class 1 in the public LB)", "486055": "Hi, thanks for your reply, \n1. this is indeed a way to prob lb, but the cost is huge.\n2. And prob **private **test set maybe regarded as cheating\n3. what i want to prob is simply the ratio of class 0/ class 1. If the test metric is accuracy, we can simply get this ratio in one submission\n4. I guess this kind of probing is quite usual in other competetions. It seems that this competetion intentionally forces use not to do so.", "486063": "Hi @zjucor.\n\n1., 3. I agree that the cost is huge for checking only the ratio. There maybe some genius ways out there to check the ratio in one submission.\n\n2., 4. I'm quite new here, so I'm not sure whether this method is considered inappropriate.", "486067": "check this discussion for possible hints: *Different distribution labels on test set?*", "486097": "thanks", "486102": "check this: https://www.kaggle.com/c/vsb-power-line-fault-detection/discussion/80674 \nit might be helpful", "487196": "> this is indeed a way to prob lb, but the cost is huge.\n\nWhy do you think the cost is huge?\nSuppose you know a certain faulty signal. So you are now at the point 2 (TP = 1, TN = N0, FP = 0, FN = N1-1). You know N=0.57*20337=11592. So with a single submission you get the equation MCC=LB_ probing_score with the only one variable N1 (or N0). Assuming N1 is something like 4% of N, you should get something like 0.045 score. I didn't try it. If you wish you can try and write if it is true. :)", "491213": "yeah, you are right ..." }, "source": "meta" }