I submitted a few tests today; I tried to figure out how they split the public/private set. From a shakeup point of view, there are two options:
\n1, For example, every 4th image is in the public set -> minor shakeup.
\n2, split by video/sequence -> major shakeup.
I split my test data into five chunks (5 submissions for today). In every chunk, I predicted 2600 images. Except for the last one, we don't know the exact number. Based on the submission times, I think there are ~1700 images in the last chunk.
\nI used my current best model (0.676 public LB)
\n| Image idx | \nLB score | \n
|---|---|
| 0 - 2600 | \n0.035 | \n
| 2600 - 5200 | \n0.537 | \n
| 5200 - 7800 | \n0.172 | \n
| 7800 - 10400 | \n0.000 | \n
| 10400 - | \n0.000 | \n
(I assume that we iterate through the images in order, frame-by-frame)
\nBased on the results, I'd say that we have video_3 in public and video_4 in the private set.
# Change this per submission.\nPROBE_IMGS = (2600, 5200)\n\n...\nfor idx, (img, pred_df) in enumerate(tqdm(iter_test)):\n\n if idx < PROBE_IMGS[0] or idx > PROBE_IMGS[1]:\n pred_df[\"annotations\"] = \"\"\n env.predict(pred_df)\n continue\n\n ... predict as usual ...\nI submitted a few tests today; I tried to figure out how they split the public/private set. From a shakeup point of view, there are two options:
\n1, For example, every 4th image is in the public set -> minor shakeup.
\n2, split by video/sequence -> major shakeup.
I split my test data into five chunks (5 submissions for today). In every chunk, I predicted 2600 images. Except for the last one, we don't know the exact number. Based on the submission times, I think there are ~1700 images in the last chunk.
\nI used my current best model (0.676 public LB)
\n| Image idx | \nLB score | \n
|---|---|
| 0 - 2600 | \n0.035 | \n
| 2600 - 5200 | \n0.537 | \n
| 5200 - 7800 | \n0.172 | \n
| 7800 - 10400 | \n0.000 | \n
| 10400 - | \n0.000 | \n
(I assume that we iterate through the images in order, frame-by-frame)
\nBased on the results, I'd say that we have video_3 in public and video_4 in the private set.
# Change this per submission.\nPROBE_IMGS = (2600, 5200)\n\n...\nfor idx, (img, pred_df) in enumerate(tqdm(iter_test)):\n\n if idx < PROBE_IMGS[0] or idx > PROBE_IMGS[1]:\n pred_df[\"annotations\"] = \"\"\n env.predict(pred_df)\n continue\n\n ... predict as usual ...\nData page tells that:
\n\n\nExpect to see roughly 13,000 images in the test set.
\n
Which means there are 3250 images in public LB. 2500-5750 are in public probably.
", "rawMarkdown": "Data page tells that:\n> Expect to see roughly 13,000 images in the test set.\n\nWhich means there are 3250 images in public LB. 2500-5750 are in public probably.", "votes": 12, "replies": [ { "id": 1679539, "postDate": "2022-02-07T09:21:56.407Z", "content": "I think your hypothesis is more reliable than the one raised by Peter
\n\n\nBased on the results, I'd say that we have video_3 in public and video_4 in the private set.
\n
Maybe the private set holds more hard empty images, while the public part only calculates scores on images where most objects exist.
", "rawMarkdown": "I think your hypothesis is more reliable than the one raised by Peter \n\n> Based on the results, I'd say that we have video_3 in public and video_4 in the private set.\n\nMaybe the private set holds more hard empty images, while the public part only calculates scores on images where most objects exist.\n" } ] }, { "id": 1658249, "postDate": "2022-01-20T20:58:59.857Z", "content": "Fortunate side-effect: you can speed up your submission by skipping images (idx > 7800).
\nBe careful! Only use this for experimenting, do not select any of these submissions as your final one!
This is almost correct but we still have possibility that idx>7800 contains some background (no annotation frame).
", "rawMarkdown": "This is almost correct but we still have possibility that idx>7800 contains some background (no annotation frame)." }, { "id": 1658324, "postDate": "2022-01-20T23:48:20.350Z", "content": "To confirm idx>7800 contains no public test frame, we have to check if the score won’t change when we make random false predictions on that frames.
", "rawMarkdown": "To confirm idx>7800 contains no public test frame, we have to check if the score won’t change when we make random false predictions on that frames.", "votes": 1 }, { "id": 1679540, "postDate": "2022-02-07T09:22:56.057Z", "content": "Any following findings on this idea?
", "rawMarkdown": "Any following findings on this idea?" } ] }, { "id": 1663240, "postDate": "2022-01-24T22:38:28.323Z", "content": "This is very smart. This confirms the public LB is a contiguous video.
\nGiven the training data is 3 separate videos, differing in COTS size, it is probable that the hidden test set contains COTS that are larger (or even smaller) in size than those for the public LB. In that case, expect a shakeup of the LB against those that train their models (and inference resolutions) to fit the COTS size in the public LB test set!
", "rawMarkdown": "This is very smart. This confirms the public LB is a contiguous video.\n\nGiven the training data is 3 separate videos, differing in COTS size, it is probable that the hidden test set contains COTS that are larger (or even smaller) in size than those for the public LB. In that case, expect a shakeup of the LB against those that train their models (and inference resolutions) to fit the COTS size in the public LB test set!", "votes": 3 }, { "id": 1691230, "postDate": "2022-02-15T10:03:33.987Z", "content": "@pestipeti do you mind if you also reveal Private scores for the above Table?
", "rawMarkdown": "@pestipeti do you mind if you also reveal Private scores for the above Table?", "votes": 2, "replies": [ { "id": 1691505, "postDate": "2022-02-15T12:49:19.553Z", "content": "\n| Image idx | \nPublic | \nPrivate | \n
|---|---|---|
| 0 - 2600 | \n0.035 | \n0.018 | \n
| 2600 - 5200 | \n0.537 | \n0.155 | \n
| 5200 - 7800 | \n0.172 | \n0.460 | \n
| 7800 - 10400 | \n0.000 | \n0.015 | \n
| 10400 - | \n0.000 | \n0.083 | \n
| Overall | \n0.676 | \n0.646 | \n
Wow, does this mean that some frames in 5200-7800 are scored in private LB? For two of my final submissions, i only inferred frames (frame<2600)|(frame>5200). I guess this explains why they didn't do well.
So split by video?
", "rawMarkdown": "So split by video?", "votes": -1, "replies": [ { "id": 1658177, "postDate": "2022-01-20T19:58:47.833Z", "rawMarkdown": "", "votes": -3, "isDeleted": true } ] }, { "id": 1690571, "postDate": "2022-02-15T02:27:47.517Z", "content": "Nice idea!
", "rawMarkdown": "Nice idea!" }, { "id": 1659393, "postDate": "2022-01-21T19:02:03.197Z", "content": "\"1, For example, every 4th image is in the public set\"
\ni submit a simple model
\nuse all frames : LB = 5.000
\ndrop every 2nd frame (if i%=2 ==0: prediction_string = ' ') : LB = 0.274
\ndrop every 3rd frame (if i%=3 ==0: prediction_string = ' ') : LB = 0.357
\ndrop every 4th frame
\n (if i%=4 ==0: prediction_string = ' ') : LB = 0.392
\n (if (i+1)%=4 ==0: prediction_string = ' ') : LB = 0.393
\n (if (i+2)%=4 ==0: prediction_string = ' ') : LB = 0.393
Hi @hengck23, Did you mistype this?
\n\n\nuse all frames : LB = 5.000
\n
Thanks for this proof.
\n\n", "rawMarkdown": "Hi @hengck23, Did you mistype this?\n> use all frames : LB = 5.000\n\nThanks for this proof.\n> drop every 4th frame\n(if i%=4 ==0: prediction_string = ' ') : LB = 0.392\n(if (i+1)%=4 ==0: prediction_string = ' ') : LB = 0.393\n(if (i+2)%=4 ==0: prediction_string = ' ') : LB = 0.393" } ] }, { "id": 1658647, "postDate": "2022-01-21T07:40:57.193Z", "content": "drop every 4th frame
\n
\n (if i%=4 ==0: prediction_string = ' ') : LB = 0.392
\n (if (i+1)%=4 ==0: prediction_string = ' ') : LB = 0.393
\n (if (i+2)%=4 ==0: prediction_string = ' ') : LB = 0.393
host can simply replace the other parts of test data by fake videos, and replace back with real test data in the end.
", "rawMarkdown": "host can simply replace the other parts of test data by fake videos, and replace back with real test data in the end." }, { "id": 1658260, "postDate": "2022-01-20T21:16:47.777Z", "content": "Very interesting study … Looks like there will be a lot of emotions in the end
", "rawMarkdown": "Very interesting study ... Looks like there will be a lot of emotions in the end" }, { "id": 1690610, "postDate": "2022-02-15T03:06:12.520Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1658161, "postDate": "2022-01-20T19:43:09.480Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 1658184, "author_name": "Ahmet Erdem", "author_url": "", "post_date": "2022-01-20T20:09:39.777000", "content": "Data page tells that:
\n\n\nExpect to see roughly 13,000 images in the test set.
\n
Which means there are 3250 images in public LB. 2500-5750 are in public probably.
", "votes": 12, "replies": [ { "id": 1679539, "author_name": "豆柴金鯱", "author_url": "", "post_date": "2022-02-07T09:21:56.407000", "content": "I think your hypothesis is more reliable than the one raised by Peter
\n\n\nBased on the results, I'd say that we have video_3 in public and video_4 in the private set.
\n
Maybe the private set holds more hard empty images, while the public part only calculates scores on images where most objects exist.
", "votes": 0, "replies": [] } ] }, { "id": 1658249, "author_name": "Peter", "author_url": "", "post_date": "2022-01-20T20:58:59.857000", "content": "Fortunate side-effect: you can speed up your submission by skipping images (idx > 7800).
\nBe careful! Only use this for experimenting, do not select any of these submissions as your final one!
This is almost correct but we still have possibility that idx>7800 contains some background (no annotation frame).
", "votes": 0, "replies": [] }, { "id": 1658324, "author_name": "Bilzard", "author_url": "", "post_date": "2022-01-20T23:48:20.350000", "content": "To confirm idx>7800 contains no public test frame, we have to check if the score won’t change when we make random false predictions on that frames.
", "votes": 1, "replies": [] }, { "id": 1679540, "author_name": "豆柴金鯱", "author_url": "", "post_date": "2022-02-07T09:22:56.057000", "content": "Any following findings on this idea?
", "votes": 0, "replies": [] } ] }, { "id": 1663240, "author_name": "Alex Wong", "author_url": "", "post_date": "2022-01-24T22:38:28.323000", "content": "This is very smart. This confirms the public LB is a contiguous video.
\nGiven the training data is 3 separate videos, differing in COTS size, it is probable that the hidden test set contains COTS that are larger (or even smaller) in size than those for the public LB. In that case, expect a shakeup of the LB against those that train their models (and inference resolutions) to fit the COTS size in the public LB test set!
", "votes": 3, "replies": [] }, { "id": 1691230, "author_name": "Ioannis M", "author_url": "", "post_date": "2022-02-15T10:03:33.987000", "content": "@pestipeti do you mind if you also reveal Private scores for the above Table?
", "votes": 2, "replies": [ { "id": 1691505, "author_name": "Peter", "author_url": "", "post_date": "2022-02-15T12:49:19.553000", "content": "\n| Image idx | \nPublic | \nPrivate | \n
|---|---|---|
| 0 - 2600 | \n0.035 | \n0.018 | \n
| 2600 - 5200 | \n0.537 | \n0.155 | \n
| 5200 - 7800 | \n0.172 | \n0.460 | \n
| 7800 - 10400 | \n0.000 | \n0.015 | \n
| 10400 - | \n0.000 | \n0.083 | \n
| Overall | \n0.676 | \n0.646 | \n
Wow, does this mean that some frames in 5200-7800 are scored in private LB? For two of my final submissions, i only inferred frames (frame<2600)|(frame>5200). I guess this explains why they didn't do well.
So split by video?
", "votes": -1, "replies": [ { "id": 1658177, "author_name": "", "author_url": "", "post_date": "2022-01-20T19:58:47.833000", "content": "", "votes": -3, "replies": [] } ] }, { "id": 1690571, "author_name": "wgz123", "author_url": "", "post_date": "2022-02-15T02:27:47.517000", "content": "Nice idea!
", "votes": 0, "replies": [] }, { "id": 1659393, "author_name": "hengck23", "author_url": "", "post_date": "2022-01-21T19:02:03.197000", "content": "\"1, For example, every 4th image is in the public set\"
\ni submit a simple model
\nuse all frames : LB = 5.000
\ndrop every 2nd frame (if i%=2 ==0: prediction_string = ' ') : LB = 0.274
\ndrop every 3rd frame (if i%=3 ==0: prediction_string = ' ') : LB = 0.357
\ndrop every 4th frame
\n (if i%=4 ==0: prediction_string = ' ') : LB = 0.392
\n (if (i+1)%=4 ==0: prediction_string = ' ') : LB = 0.393
\n (if (i+2)%=4 ==0: prediction_string = ' ') : LB = 0.393
Hi @hengck23, Did you mistype this?
\n\n\nuse all frames : LB = 5.000
\n
Thanks for this proof.
\n\n", "votes": 0, "replies": [] } ] }, { "id": 1658647, "author_name": "dragon zhang", "author_url": "", "post_date": "2022-01-21T07:40:57.193000", "content": "drop every 4th frame
\n
\n (if i%=4 ==0: prediction_string = ' ') : LB = 0.392
\n (if (i+1)%=4 ==0: prediction_string = ' ') : LB = 0.393
\n (if (i+2)%=4 ==0: prediction_string = ' ') : LB = 0.393
host can simply replace the other parts of test data by fake videos, and replace back with real test data in the end.
", "votes": 0, "replies": [] }, { "id": 1658260, "author_name": "Robson", "author_url": "", "post_date": "2022-01-20T21:16:47.777000", "content": "Very interesting study … Looks like there will be a lot of emotions in the end
", "votes": 0, "replies": [] }, { "id": 1690610, "author_name": "", "author_url": "", "post_date": "2022-02-15T03:06:12.520000", "content": "", "votes": 0, "replies": [] }, { "id": 1658161, "author_name": "", "author_url": "", "post_date": "2022-01-20T19:43:09.480000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1658097": "I submitted a few tests today; I tried to figure out how they split the public/private set. From a shakeup point of view, there are two options:\n\n1, For example, every 4th image is in the public set -> minor shakeup.\n2, split by video/sequence -> major shakeup.\n\nI split my test data into five chunks (5 submissions for today). In every chunk, I predicted 2600 images. Except for the last one, we don't know the exact number. Based on the submission times, I think there are ~1700 images in the last chunk.\n\n\n#### Results\nI used my current best model (0.676 public LB)\n\n| **Image idx** | **LB score** |\n|:------------:|:--------:|\n| 0 - 2600 | 0.035 |\n| 2600 - 5200 | 0.537 |\n| 5200 - 7800 | 0.172 |\n| 7800 - 10400 | 0.000 |\n| 10400 - | 0.000 |\n\n(I assume that we iterate through the images in order, frame-by-frame)\nBased on the results, I'd say that we have video_3 in public and video_4 in the private set.\n\n\n\n#### Code\n```\n# Change this per submission.\nPROBE_IMGS = (2600, 5200)\n\n...\nfor idx, (img, pred_df) in enumerate(tqdm(iter_test)):\n \n if idx < PROBE_IMGS[0] or idx > PROBE_IMGS[1]:\n pred_df[\"annotations\"] = \"\"\n env.predict(pred_df)\n continue\n\n ... predict as usual ...\n\n```\n\n", "1658184": "Data page tells that:\n> Expect to see roughly 13,000 images in the test set.\n\nWhich means there are 3250 images in public LB. 2500-5750 are in public probably.", "1658249": "Fortunate side-effect: you can speed up your submission by skipping images (idx > 7800).\nBe careful! Only use this for experimenting, do not select any of these submissions as your final one!", "1663240": "This is very smart. This confirms the public LB is a contiguous video.\n\nGiven the training data is 3 separate videos, differing in COTS size, it is probable that the hidden test set contains COTS that are larger (or even smaller) in size than those for the public LB. In that case, expect a shakeup of the LB against those that train their models (and inference resolutions) to fit the COTS size in the public LB test set!", "1691230": "@pestipeti do you mind if you also reveal Private scores for the above Table?", "1658174": "So split by video?", "1690571": "Nice idea!", "1659393": "\"1, For example, every 4th image is in the public set\"\n\ni submit a simple model\nuse all frames : LB = 5.000\ndrop every 2nd frame (if i%=2 ==0: prediction\\_string = ' ') : LB = 0.274\ndrop every 3rd frame (if i%=3 ==0: prediction\\_string = ' ') : LB = 0.357\ndrop every 4th frame\n (if i%=4 ==0: prediction\\_string = ' ') : LB = 0.392\n (if (i+1)%=4 ==0: prediction\\_string = ' ') : LB = 0.393\n (if (i+2)%=4 ==0: prediction\\_string = ' ') : LB = 0.393\n", "1658647": "host can simply replace the other parts of test data by fake videos, and replace back with real test data in the end.", "1658260": "Very interesting study ... Looks like there will be a lot of emotions in the end", "1690610": "", "1658161": "" } }