I am not sure if everyone figured this out, but it is possible to check individual R,C and V scores on LB as we know the basic scores for predicting all zeros.
\n\nI prepared a script for it:\nhttps://www.kaggle.com/philippsinger/checking-r-c-v-scores
\n\nIf you for example want to know how your solution scores on R, you need to set C and V to all zeros, and only submit R. Then fill in the respective number in the corresponding cell of the notebook and you will get the score.
\n\nI believe most will have lost scores on C and R.
\n\nEdit: There appears to be an easier way which I naively assumed will lead to failed sub, you can set all other scores to out of bounds as explained below by @dipamc77
", "messages": [ { "id": 776723, "postDate": "2020-03-17T15:36:22.313Z", "content": "I am not sure if everyone figured this out, but it is possible to check individual R,C and V scores on LB as we know the basic scores for predicting all zeros.
\n\nI prepared a script for it:\nhttps://www.kaggle.com/philippsinger/checking-r-c-v-scores
\n\nIf you for example want to know how your solution scores on R, you need to set C and V to all zeros, and only submit R. Then fill in the respective number in the corresponding cell of the notebook and you will get the score.
\n\nI believe most will have lost scores on C and R.
\n\nEdit: There appears to be an easier way which I naively assumed will lead to failed sub, you can set all other scores to out of bounds as explained below by @dipamc77
", "rawMarkdown": "I am not sure if everyone figured this out, but it is possible to check individual R,C and V scores on LB as we know the basic scores for predicting all zeros.\n\nI prepared a script for it:\nhttps://www.kaggle.com/philippsinger/checking-r-c-v-scores\n\nIf you for example want to know how your solution scores on R, you need to set C and V to all zeros, and only submit R. Then fill in the respective number in the corresponding cell of the notebook and you will get the score.\n\nI believe most will have lost scores on C and R. \n\nEdit: There appears to be an easier way which I naively assumed will lead to failed sub, you can set all other scores to out of bounds as explained below by @dipamc77 ", "votes": 5 }, { "id": 776753, "postDate": "2020-03-17T15:59:59.327Z", "content": "We can't set it to all zeros, that would give you some score for the zero classes, we need to set it to 7 (or any number other than 0-6) for consonant and 11 and 168 for vowel and root respectively. We can further probe only for a particular component class by setting all others to the wrong value.
\n\nFor example - To get score for only consonant 3, when model predicts 3 keep it, otherwise set consonant to 7. (Then multiply the score by 7*4 because it its macro averaged) And set R to 168 and V to 11. C 3 and 6 were extremely poor on private set.
", "rawMarkdown": "We can't set it to all zeros, that would give you some score for the zero classes, we need to set it to 7 (or any number other than 0-6) for consonant and 11 and 168 for vowel and root respectively. We can further probe only for a particular component class by setting all others to the wrong value.\n\nFor example - To get score for only consonant 3, when model predicts 3 keep it, otherwise set consonant to 7. (Then multiply the score by 7*4 because it its macro averaged) And set R to 168 and V to 11. C 3 and 6 were extremely poor on private set. ", "votes": 1, "replies": [ { "id": 776754, "postDate": "2020-03-17T16:00:42.410Z", "content": "If you want the full score for C you can set R and V to full zeros as shown in the kernel. Setting it out of bounds is also an option of course, I never checked if it would fail on LB though, I guess there is no check for proper prediction classes.
", "rawMarkdown": "If you want the full score for C you can set R and V to full zeros as shown in the kernel. Setting it out of bounds is also an option of course, I never checked if it would fail on LB though, I guess there is no check for proper prediction classes." }, { "id": 776757, "postDate": "2020-03-17T16:03:06.833Z", "content": "I set it out of bounds. Worked perfectly.
", "rawMarkdown": "I set it out of bounds. Worked perfectly." }, { "id": 776766, "postDate": "2020-03-17T16:13:39.733Z", "content": "That's even easier then ;)
", "rawMarkdown": "That's even easier then ;)" }, { "id": 776769, "postDate": "2020-03-17T16:14:36.587Z", "content": "I'm curious what are your private scores on Consonant 3 and 6 can you please share if you have them.
", "rawMarkdown": "I'm curious what are your private scores on Consonant 3 and 6 can you please share if you have them." }, { "id": 776788, "postDate": "2020-03-17T16:28:38.637Z", "content": "Sure, will check.
", "rawMarkdown": "Sure, will check.", "votes": -1 }, { "id": 777536, "postDate": "2020-03-17T18:47:00.137Z", "content": "@dipamc77 can you share your C=3 and C=6 so that I have a reference?
", "rawMarkdown": "@dipamc77 can you share your C=3 and C=6 so that I have a reference?", "votes": -1 }, { "id": 777574, "postDate": "2020-03-17T19:29:45.070Z", "content": "If I did the calculation correctly:
\n\n0.8764 for C=3\n0.9324 for C=6
", "rawMarkdown": "If I did the calculation correctly:\n\n0.8764 for C=3\n0.9324 for C=6" }, { "id": 777604, "postDate": "2020-03-17T20:09:35.297Z", "content": "Yeah no wonder you're 2nd 👍 \n0.4984 for C=3\n0.6972 for C=6
", "rawMarkdown": "Yeah no wonder you're 2nd 👍 \n0.4984 for C=3\n0.6972 for C=6", "votes": 1 } ] }, { "id": 778077, "postDate": "2020-03-18T06:03:53.057Z", "content": "Thanks for sharing! I saw in other posts that you could see the danger of unseen graphemes from this script. I was wondering if you could elaborate how is taht accomplished. Thanks!
", "rawMarkdown": "Thanks for sharing! I saw in other posts that you could see the danger of unseen graphemes from this script. I was wondering if you could elaborate how is taht accomplished. Thanks!" }, { "id": 776730, "postDate": "2020-03-17T15:39:57.730Z", "content": "Yes, I did the same. Submitted it individually. Score on C was extremely low (coming from C=3 and C=6) , while R & V was in line with cross-validation.
", "rawMarkdown": "Yes, I did the same. Submitted it individually. Score on C was extremely low (coming from C=3 and C=6) , while R & V was in line with cross-validation.", "replies": [ { "id": 776732, "postDate": "2020-03-17T15:43:28.413Z", "content": "C was low, R was also lower for us before generalizing better for it, V was in line. I havent checked it for final sub yet, and not on private LB.
", "rawMarkdown": "C was low, R was also lower for us before generalizing better for it, V was in line. I havent checked it for final sub yet, and not on private LB.", "votes": -1 } ] }, { "id": 777521, "postDate": "2020-03-17T18:37:02.683Z", "rawMarkdown": "", "votes": -2, "isDeleted": true } ], "comments": [ { "id": 776753, "author_name": "Dipam Chakraborty", "author_url": "", "post_date": "2020-03-17T15:59:59.327000", "content": "We can't set it to all zeros, that would give you some score for the zero classes, we need to set it to 7 (or any number other than 0-6) for consonant and 11 and 168 for vowel and root respectively. We can further probe only for a particular component class by setting all others to the wrong value.
\n\nFor example - To get score for only consonant 3, when model predicts 3 keep it, otherwise set consonant to 7. (Then multiply the score by 7*4 because it its macro averaged) And set R to 168 and V to 11. C 3 and 6 were extremely poor on private set.
", "votes": 1, "replies": [ { "id": 776754, "author_name": "Psi", "author_url": "", "post_date": "2020-03-17T16:00:42.410000", "content": "If you want the full score for C you can set R and V to full zeros as shown in the kernel. Setting it out of bounds is also an option of course, I never checked if it would fail on LB though, I guess there is no check for proper prediction classes.
", "votes": 0, "replies": [] }, { "id": 776757, "author_name": "Dieter", "author_url": "", "post_date": "2020-03-17T16:03:06.833000", "content": "I set it out of bounds. Worked perfectly.
", "votes": 0, "replies": [] }, { "id": 776766, "author_name": "Psi", "author_url": "", "post_date": "2020-03-17T16:13:39.733000", "content": "That's even easier then ;)
", "votes": 0, "replies": [] }, { "id": 776769, "author_name": "Dipam Chakraborty", "author_url": "", "post_date": "2020-03-17T16:14:36.587000", "content": "I'm curious what are your private scores on Consonant 3 and 6 can you please share if you have them.
", "votes": 0, "replies": [] }, { "id": 776788, "author_name": "Psi", "author_url": "", "post_date": "2020-03-17T16:28:38.637000", "content": "Sure, will check.
", "votes": -1, "replies": [] }, { "id": 777536, "author_name": "Psi", "author_url": "", "post_date": "2020-03-17T18:47:00.137000", "content": "@dipamc77 can you share your C=3 and C=6 so that I have a reference?
", "votes": -1, "replies": [] }, { "id": 777574, "author_name": "Psi", "author_url": "", "post_date": "2020-03-17T19:29:45.070000", "content": "If I did the calculation correctly:
\n\n0.8764 for C=3\n0.9324 for C=6
", "votes": 0, "replies": [] }, { "id": 777604, "author_name": "Dipam Chakraborty", "author_url": "", "post_date": "2020-03-17T20:09:35.297000", "content": "Yeah no wonder you're 2nd 👍 \n0.4984 for C=3\n0.6972 for C=6
", "votes": 1, "replies": [] } ] }, { "id": 778077, "author_name": "Gold Retriever", "author_url": "", "post_date": "2020-03-18T06:03:53.057000", "content": "Thanks for sharing! I saw in other posts that you could see the danger of unseen graphemes from this script. I was wondering if you could elaborate how is taht accomplished. Thanks!
", "votes": 0, "replies": [] }, { "id": 776730, "author_name": "Dieter", "author_url": "", "post_date": "2020-03-17T15:39:57.730000", "content": "Yes, I did the same. Submitted it individually. Score on C was extremely low (coming from C=3 and C=6) , while R & V was in line with cross-validation.
", "votes": 0, "replies": [ { "id": 776732, "author_name": "Psi", "author_url": "", "post_date": "2020-03-17T15:43:28.413000", "content": "C was low, R was also lower for us before generalizing better for it, V was in line. I havent checked it for final sub yet, and not on private LB.
", "votes": -1, "replies": [] } ] }, { "id": 777521, "author_name": "", "author_url": "", "post_date": "2020-03-17T18:37:02.683000", "content": "", "votes": -2, "replies": [] } ], "raw_markdown_by_id": { "776723": "I am not sure if everyone figured this out, but it is possible to check individual R,C and V scores on LB as we know the basic scores for predicting all zeros.\n\nI prepared a script for it:\nhttps://www.kaggle.com/philippsinger/checking-r-c-v-scores\n\nIf you for example want to know how your solution scores on R, you need to set C and V to all zeros, and only submit R. Then fill in the respective number in the corresponding cell of the notebook and you will get the score.\n\nI believe most will have lost scores on C and R. \n\nEdit: There appears to be an easier way which I naively assumed will lead to failed sub, you can set all other scores to out of bounds as explained below by @dipamc77 ", "776753": "We can't set it to all zeros, that would give you some score for the zero classes, we need to set it to 7 (or any number other than 0-6) for consonant and 11 and 168 for vowel and root respectively. We can further probe only for a particular component class by setting all others to the wrong value.\n\nFor example - To get score for only consonant 3, when model predicts 3 keep it, otherwise set consonant to 7. (Then multiply the score by 7*4 because it its macro averaged) And set R to 168 and V to 11. C 3 and 6 were extremely poor on private set. ", "778077": "Thanks for sharing! I saw in other posts that you could see the danger of unseen graphemes from this script. I was wondering if you could elaborate how is taht accomplished. Thanks!", "776730": "Yes, I did the same. Submitted it individually. Score on C was extremely low (coming from C=3 and C=6) , while R & V was in line with cross-validation.", "777521": "" } }