To better simulating LB, I create my personal validation set with 4000 videos (P:N=1:1) with no actors and scenes overlapped. However, when I monitor the Val loss history, I found that the logloss always fluctuates (see above plot).
\n\n.svg?generation=1580188608858698&alt=media)
My conclusion is that probably the size of a private test set is too small, and made the LB lose its power. Did anyone have better suggestions about building a reliable local validation set?
", "messages": [ { "id": "730910", "postDate": "01/28/2020 05:27:37", "content": "To better simulating LB, I create my personal validation set with 4000 videos (P:N=1:1) with no actors and scenes overlapped. However, when I monitor the Val loss history, I found that the logloss always fluctuates (see above plot).
\n\n.svg?generation=1580188608858698&alt=media)
My conclusion is that probably the size of a private test set is too small, and made the LB lose its power. Did anyone have better suggestions about building a reliable local validation set?
", "rawMarkdown": "To better simulating LB, I create my personal validation set with 4000 videos (P:N=1:1) with no actors and scenes overlapped. However, when I monitor the Val loss history, I found that the logloss always fluctuates (see above plot).\n\n.svg?generation=1580188608858698&alt=media)\n\nMy conclusion is that probably the size of a private test set is too small, and made the LB lose its power. Did anyone have better suggestions about building a reliable local validation set?", "votes": null }, { "id": "731049", "postDate": "01/28/2020 09:42:52", "content": "Did you build this validation set from the provided training data or from some external dataset? (If from the provided training data, I don't think you can have 4000 videos with no actors overlapping, since there are only 100 or so different actors in the dataset.)
", "rawMarkdown": "Did you build this validation set from the provided training data or from some external dataset? (If from the provided training data, I don't think you can have 4000 videos with no actors overlapping, since there are only 100 or so different actors in the dataset.)", "votes": null }, { "id": "731514", "postDate": "01/28/2020 18:24:07", "content": "Sorry to confuse you, what I mean no overlapping actors and scenes are based on the combination of them. And also due to my clustering methods, some scenes and actors might be divided into different groups.
", "rawMarkdown": "Sorry to confuse you, what I mean no overlapping actors and scenes are based on the combination of them. And also due to my clustering methods, some scenes and actors might be divided into different groups.", "votes": null } ], "comments": [ { "id": 731049, "author_name": "humananalog", "author_url": "", "post_date": "01/28/2020 09:42:52", "content": "Did you build this validation set from the provided training data or from some external dataset? (If from the provided training data, I don't think you can have 4000 videos with no actors overlapping, since there are only 100 or so different actors in the dataset.)
", "votes": null, "replies": [ { "id": 731514, "author_name": "wufanyou", "author_url": "", "post_date": "01/28/2020 18:24:07", "content": "Sorry to confuse you, what I mean no overlapping actors and scenes are based on the combination of them. And also due to my clustering methods, some scenes and actors might be divided into different groups.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "730910": "To better simulating LB, I create my personal validation set with 4000 videos (P:N=1:1) with no actors and scenes overlapped. However, when I monitor the Val loss history, I found that the logloss always fluctuates (see above plot).\n\n.svg?generation=1580188608858698&alt=media)\n\nMy conclusion is that probably the size of a private test set is too small, and made the LB lose its power. Did anyone have better suggestions about building a reliable local validation set?", "731049": "Did you build this validation set from the provided training data or from some external dataset? (If from the provided training data, I don't think you can have 4000 videos with no actors overlapping, since there are only 100 or so different actors in the dataset.)", "731514": "Sorry to confuse you, what I mean no overlapping actors and scenes are based on the combination of them. And also due to my clustering methods, some scenes and actors might be divided into different groups." }, "source": "meta" }