The goal of optimization was wrong. The time of train model was wasted
", "messages": [ { "id": 692159, "postDate": "2019-12-11T01:17:02.647Z", "content": "The goal of optimization was wrong. The time of train model was wasted
", "rawMarkdown": "The goal of optimization was wrong. The time of train model was wasted", "votes": 10 }, { "id": 694271, "postDate": "2019-12-13T11:24:08.053Z", "content": "Update:\nThe LB match my offline score well,so the metric online is reliable.
", "rawMarkdown": "Update:\nThe LB match my offline score well,so the metric online is reliable.\n", "votes": 3, "replies": [ { "id": 694273, "postDate": "2019-12-13T11:27:47.083Z", "content": "Can you tell a bit more how you calculate the offline score? Which val set for example
", "rawMarkdown": "Can you tell a bit more how you calculate the offline score? Which val set for example\n", "votes": 1 }, { "id": 694279, "postDate": "2019-12-13T11:37:30.877Z", "content": "Just like last discussion\nLB will be higher because one prediction has many labels
", "rawMarkdown": "Just like [last discussion](https://www.kaggle.com/c/tensorflow2-question-answering/discussion/120030#686536)\nLB will be higher because one prediction has many labels", "votes": 2 }, { "id": 694319, "postDate": "2019-12-13T12:25:23.053Z", "content": "but only many labels for short answers not for long answers, or?
", "rawMarkdown": "but only many labels for short answers not for long answers, or?" }, { "id": 694322, "postDate": "2019-12-13T12:27:45.730Z", "content": "no, long answers also have many
", "rawMarkdown": "no, long answers also have many", "votes": 1 }, { "id": 694333, "postDate": "2019-12-13T12:45:46.570Z", "content": "Could you lead me to any resource (kaggle discussion or external) which is the reason for your statement? I could not find anything on kaggle or NQ repo indicating there are multiple long answers. Many thanks
", "rawMarkdown": "Could you lead me to any resource (kaggle discussion or external) which is the reason for your statement? I could not find anything on kaggle or NQ repo indicating there are multiple long answers. Many thanks" }, { "id": 694334, "postDate": "2019-12-13T12:48:54.213Z", "content": "> There may be up to five labels for long answers, and more for short
\n\nIn the Evaluation page
", "rawMarkdown": "> There may be up to five labels for long answers, and more for short\n\nIn the Evaluation page", "votes": 3 }, { "id": 694428, "postDate": "2019-12-13T15:27:15.763Z", "content": "Yes, and Phil commented that you get a credit for an NQ example if you predict at least one of the answers correctly.
", "rawMarkdown": "Yes, and Phil commented that you get a credit for an NQ example if you predict at least one of the answers correctly. ", "votes": 1 } ] }, { "id": 692296, "postDate": "2019-12-11T05:39:22.613Z", "content": "Honestly, I don't know. I implemented what I thought is the new metric and get 0.42 cv vs 0.50 LB. That seems off.
", "rawMarkdown": "Honestly, I don't know. I implemented what I thought is the new metric and get 0.42 cv vs 0.50 LB. That seems off.", "votes": 3, "replies": [ { "id": 693031, "postDate": "2019-12-12T01:05:59.647Z", "content": "On my side, I see a similar story. My long answer f1 (higher ~45%) is closer to public LB... (However full set local CV 38%-> public LB 51%) When using the previous score (thanks to your implementation) it gives a local CV of 73%, public LB 75%. I'm not sure about whether they change the distribution of the eval data-set or maybe my score is not implemented correctly.
", "rawMarkdown": "On my side, I see a similar story. My long answer f1 (higher ~45%) is closer to public LB... (However full set local CV 38%-> public LB 51%) When using the previous score (thanks to your implementation) it gives a local CV of 73%, public LB 75%. I'm not sure about whether they change the distribution of the eval data-set or maybe my score is not implemented correctly.", "votes": 1 } ] }, { "id": 692302, "postDate": "2019-12-11T05:51:30.170Z", "content": "Gradient descent to get to the correct metric is very slow ......
", "rawMarkdown": "Gradient descent to get to the correct metric is very slow ......", "votes": 1, "replies": [ { "id": 692303, "postDate": "2019-12-11T05:55:01.007Z", "content": "Especially since you don't see the code you want to fix.
", "rawMarkdown": "Especially since you don't see the code you want to fix.", "votes": 2 } ] }, { "id": 692254, "postDate": "2019-12-11T04:33:29.430Z", "rawMarkdown": "", "votes": -2, "isDeleted": true } ], "comments": [ { "id": 694271, "author_name": "sakuranew", "author_url": "", "post_date": "2019-12-13T11:24:08.053000", "content": "Update:\nThe LB match my offline score well,so the metric online is reliable.
", "votes": 3, "replies": [ { "id": 694273, "author_name": "Dieter", "author_url": "", "post_date": "2019-12-13T11:27:47.083000", "content": "Can you tell a bit more how you calculate the offline score? Which val set for example
", "votes": 1, "replies": [] }, { "id": 694279, "author_name": "sakuranew", "author_url": "", "post_date": "2019-12-13T11:37:30.877000", "content": "Just like last discussion\nLB will be higher because one prediction has many labels
", "votes": 2, "replies": [] }, { "id": 694319, "author_name": "Dieter", "author_url": "", "post_date": "2019-12-13T12:25:23.053000", "content": "but only many labels for short answers not for long answers, or?
", "votes": 0, "replies": [] }, { "id": 694322, "author_name": "sakuranew", "author_url": "", "post_date": "2019-12-13T12:27:45.730000", "content": "no, long answers also have many
", "votes": 1, "replies": [] }, { "id": 694333, "author_name": "Dieter", "author_url": "", "post_date": "2019-12-13T12:45:46.570000", "content": "Could you lead me to any resource (kaggle discussion or external) which is the reason for your statement? I could not find anything on kaggle or NQ repo indicating there are multiple long answers. Many thanks
", "votes": 0, "replies": [] }, { "id": 694334, "author_name": "sakuranew", "author_url": "", "post_date": "2019-12-13T12:48:54.213000", "content": "> There may be up to five labels for long answers, and more for short
\n\nIn the Evaluation page
", "votes": 3, "replies": [] }, { "id": 694428, "author_name": "Yury Kashnitsky", "author_url": "", "post_date": "2019-12-13T15:27:15.763000", "content": "Yes, and Phil commented that you get a credit for an NQ example if you predict at least one of the answers correctly.
", "votes": 1, "replies": [] } ] }, { "id": 692296, "author_name": "Dieter", "author_url": "", "post_date": "2019-12-11T05:39:22.613000", "content": "Honestly, I don't know. I implemented what I thought is the new metric and get 0.42 cv vs 0.50 LB. That seems off.
", "votes": 3, "replies": [ { "id": 693031, "author_name": "Shane", "author_url": "", "post_date": "2019-12-12T01:05:59.647000", "content": "On my side, I see a similar story. My long answer f1 (higher ~45%) is closer to public LB... (However full set local CV 38%-> public LB 51%) When using the previous score (thanks to your implementation) it gives a local CV of 73%, public LB 75%. I'm not sure about whether they change the distribution of the eval data-set or maybe my score is not implemented correctly.
", "votes": 1, "replies": [] } ] }, { "id": 692302, "author_name": "Yih-Dar SHIEH", "author_url": "", "post_date": "2019-12-11T05:51:30.170000", "content": "Gradient descent to get to the correct metric is very slow ......
", "votes": 1, "replies": [ { "id": 692303, "author_name": "Dieter", "author_url": "", "post_date": "2019-12-11T05:55:01.007000", "content": "Especially since you don't see the code you want to fix.
", "votes": 2, "replies": [] } ] }, { "id": 692254, "author_name": "", "author_url": "", "post_date": "2019-12-11T04:33:29.430000", "content": "", "votes": -2, "replies": [] } ], "raw_markdown_by_id": { "692159": "The goal of optimization was wrong. The time of train model was wasted", "694271": "Update:\nThe LB match my offline score well,so the metric online is reliable.\n", "692296": "Honestly, I don't know. I implemented what I thought is the new metric and get 0.42 cv vs 0.50 LB. That seems off.", "692302": "Gradient descent to get to the correct metric is very slow ......", "692254": "" } }