![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2234817%2Fbc68deccd38a9ea06308f180ec175484%2FDALI.PNG?generation=1576202706052094&alt=media\")
Please correct me if my calculations are wrong.
\n\nWe have around 400 public test videos. When submitting the kernel, we have up to 10 times more private data, which should amount to 4000 in the worst case. OpenCV should take about 4s to read a single video (itself 10s in length, and that is without any saving). So if we want to just \"load\" the entire test set, it could take up to 16000s for the test videos, without considering saving.
\n\nDoes anyone know more efficient ways of loading videos? A faster alternative to OpenCV, or a GPU-enabled solution?
", "messages": [ { "id": "693537", "postDate": "12/12/2019 13:47:06", "content": "Please correct me if my calculations are wrong.
\n\nWe have around 400 public test videos. When submitting the kernel, we have up to 10 times more private data, which should amount to 4000 in the worst case. OpenCV should take about 4s to read a single video (itself 10s in length, and that is without any saving). So if we want to just \"load\" the entire test set, it could take up to 16000s for the test videos, without considering saving.
\n\nDoes anyone know more efficient ways of loading videos? A faster alternative to OpenCV, or a GPU-enabled solution?
", "rawMarkdown": "Please correct me if my calculations are wrong.\n\nWe have around 400 public test videos. When submitting the kernel, we have up to 10 times more private data, which should amount to 4000 in the worst case. OpenCV should take about 4s to read a single video (itself 10s in length, and that is without any saving). So if we want to just \"load\" the entire test set, it could take up to 16000s for the test videos, without considering saving.\n\nDoes anyone know more efficient ways of loading videos? A faster alternative to OpenCV, or a GPU-enabled solution?", "votes": null }, { "id": "693577", "postDate": "12/12/2019 14:24:55", "content": "NVIDIA's DALI Framework might be an alternative. \nI did an interview with @jamesdellinger where he shared many insights about it-although these were mostly related to image augmentations. Interview Video
", "rawMarkdown": "NVIDIA's [DALI Framework](https://www.google.com/search?client=safari&rls=en&q=Dali+nvidia&ie=UTF-8&oe=UTF-8) might be an alternative. \nI did an interview with @jamesdellinger where he shared many insights about it-although these were mostly related to image augmentations. [Interview Video](https://www.youtube.com/watch?v=4kMEdDcBt00)", "votes": null }, { "id": "693613", "postDate": "12/12/2019 15:21:03", "content": "Thanks for sharing this framework. Here's something I found that could be useful: https://github.com/NVIDIA/DALI/blob/b33c5acef62d16308deb2fe3d5f129e9e7566d25/docs/examples/video/video_reader_simple_example.ipynb
", "rawMarkdown": "Thanks for sharing this framework. Here's something I found that could be useful: https://github.com/NVIDIA/DALI/blob/b33c5acef62d16308deb2fe3d5f129e9e7566d25/docs/examples/video/video_reader_simple_example.ipynb", "votes": null }, { "id": "693965", "postDate": "12/13/2019 02:05:25", "content": "Thanks for the insight . I tried to use DALI in kaggle kernel and this is what i met with . Not sure ,if there is some enablement need to be done in Kaggle containers .
\n\n
Or may be I am making some mistake.
", "rawMarkdown": "Thanks for the insight . I tried to use DALI in kaggle kernel and this is what i met with . Not sure ,if there is some enablement need to be done in Kaggle containers .\n\n\n\nOr may be I am making some mistake.", "votes": null }, { "id": "698177", "postDate": "12/18/2019 22:30:06", "content": "Where are we told we have up to 10 times more private data?
", "rawMarkdown": "Where are we told we have up to 10 times more private data?", "votes": null }, { "id": "698334", "postDate": "12/19/2019 04:52:11", "content": "I'm not quite sure, I saw someone quoting that in a comment but I don't recall any official release.
", "rawMarkdown": "I'm not quite sure, I saw someone quoting that in a comment but I don't recall any official release.", "votes": null }, { "id": "698352", "postDate": "12/19/2019 05:29:26", "content": "i met this question too. Have you solved that?
", "rawMarkdown": "i met this question too. Have you solved that?", "votes": null }, { "id": "698499", "postDate": "12/19/2019 10:16:16", "content": "In \"Getting Started\" it says, \"You should anticipate the Public Test set to be 10 times the size of the Public Validation Set and budget accordingly.\"
", "rawMarkdown": "In \"Getting Started\" it says, \"You should anticipate the Public Test set to be 10 times the size of the Public Validation Set and budget accordingly.\"", "votes": null }, { "id": "699397", "postDate": "12/20/2019 11:44:34", "content": "Thank you, @humananalog
", "rawMarkdown": "Thank you, @humananalog", "votes": null }, { "id": "699433", "postDate": "12/20/2019 13:03:15", "content": "So Great to see @anokas being active on Kaggle again 😁
", "rawMarkdown": "So Great to see @anokas being active on Kaggle again 😁", "votes": null }, { "id": "701041", "postDate": "12/23/2019 02:42:08", "content": "Public Validation (400) -> Public Test (4000) -> Private Test (?????)
", "rawMarkdown": "Public Validation (400) -> Public Test (4000) -> Private Test (?????)", "votes": null }, { "id": "702015", "postDate": "12/24/2019 06:56:36", "content": "Have you successfully used that code?
", "rawMarkdown": "Have you successfully used that code?", "votes": null }, { "id": "704628", "postDate": "12/27/2019 18:38:26", "content": "I think 400 is the private test, where a fraction is used for LB scoring and the rest for prviate LB.
", "rawMarkdown": "I think 400 is the private test, where a fraction is used for LB scoring and the rest for prviate LB.", "votes": null } ], "comments": [ { "id": 693577, "author_name": "init27", "author_url": "", "post_date": "12/12/2019 14:24:55", "content": "NVIDIA's DALI Framework might be an alternative. \nI did an interview with @jamesdellinger where he shared many insights about it-although these were mostly related to image augmentations. Interview Video
", "votes": null, "replies": [ { "id": 693613, "author_name": "xhlulu", "author_url": "", "post_date": "12/12/2019 15:21:03", "content": "Thanks for sharing this framework. Here's something I found that could be useful: https://github.com/NVIDIA/DALI/blob/b33c5acef62d16308deb2fe3d5f129e9e7566d25/docs/examples/video/video_reader_simple_example.ipynb
", "votes": null, "replies": [] }, { "id": 702015, "author_name": "wegnery", "author_url": "", "post_date": "12/24/2019 06:56:36", "content": "Have you successfully used that code?
", "votes": null, "replies": [] } ] }, { "id": 693965, "author_name": "phoenix9032", "author_url": "", "post_date": "12/13/2019 02:05:25", "content": "Thanks for the insight . I tried to use DALI in kaggle kernel and this is what i met with . Not sure ,if there is some enablement need to be done in Kaggle containers .
\n\n
Or may be I am making some mistake.
", "votes": null, "replies": [ { "id": 698352, "author_name": "wegnery", "author_url": "", "post_date": "12/19/2019 05:29:26", "content": "i met this question too. Have you solved that?
", "votes": null, "replies": [] } ] }, { "id": 698177, "author_name": "anokas", "author_url": "", "post_date": "12/18/2019 22:30:06", "content": "Where are we told we have up to 10 times more private data?
", "votes": null, "replies": [ { "id": 698334, "author_name": "xhlulu", "author_url": "", "post_date": "12/19/2019 04:52:11", "content": "I'm not quite sure, I saw someone quoting that in a comment but I don't recall any official release.
", "votes": null, "replies": [] }, { "id": 698499, "author_name": "humananalog", "author_url": "", "post_date": "12/19/2019 10:16:16", "content": "In \"Getting Started\" it says, \"You should anticipate the Public Test set to be 10 times the size of the Public Validation Set and budget accordingly.\"
", "votes": null, "replies": [] }, { "id": 699397, "author_name": "anokas", "author_url": "", "post_date": "12/20/2019 11:44:34", "content": "Thank you, @humananalog
", "votes": null, "replies": [] }, { "id": 699433, "author_name": "init27", "author_url": "", "post_date": "12/20/2019 13:03:15", "content": "So Great to see @anokas being active on Kaggle again 😁
", "votes": null, "replies": [] } ] }, { "id": 701041, "author_name": "skrish13", "author_url": "", "post_date": "12/23/2019 02:42:08", "content": "Public Validation (400) -> Public Test (4000) -> Private Test (?????)
", "votes": null, "replies": [ { "id": 704628, "author_name": "xhlulu", "author_url": "", "post_date": "12/27/2019 18:38:26", "content": "I think 400 is the private test, where a fraction is used for LB scoring and the rest for prviate LB.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "693537": "Please correct me if my calculations are wrong.\n\nWe have around 400 public test videos. When submitting the kernel, we have up to 10 times more private data, which should amount to 4000 in the worst case. OpenCV should take about 4s to read a single video (itself 10s in length, and that is without any saving). So if we want to just \"load\" the entire test set, it could take up to 16000s for the test videos, without considering saving.\n\nDoes anyone know more efficient ways of loading videos? A faster alternative to OpenCV, or a GPU-enabled solution?", "693577": "NVIDIA's [DALI Framework](https://www.google.com/search?client=safari&rls=en&q=Dali+nvidia&ie=UTF-8&oe=UTF-8) might be an alternative. \nI did an interview with @jamesdellinger where he shared many insights about it-although these were mostly related to image augmentations. [Interview Video](https://www.youtube.com/watch?v=4kMEdDcBt00)", "693613": "Thanks for sharing this framework. Here's something I found that could be useful: https://github.com/NVIDIA/DALI/blob/b33c5acef62d16308deb2fe3d5f129e9e7566d25/docs/examples/video/video_reader_simple_example.ipynb", "693965": "Thanks for the insight . I tried to use DALI in kaggle kernel and this is what i met with . Not sure ,if there is some enablement need to be done in Kaggle containers .\n\n\n\nOr may be I am making some mistake.", "698177": "Where are we told we have up to 10 times more private data?", "698334": "I'm not quite sure, I saw someone quoting that in a comment but I don't recall any official release.", "698352": "i met this question too. Have you solved that?", "698499": "In \"Getting Started\" it says, \"You should anticipate the Public Test set to be 10 times the size of the Public Validation Set and budget accordingly.\"", "699397": "Thank you, @humananalog", "699433": "So Great to see @anokas being active on Kaggle again 😁", "701041": "Public Validation (400) -> Public Test (4000) -> Private Test (?????)", "702015": "Have you successfully used that code?", "704628": "I think 400 is the private test, where a fraction is used for LB scoring and the rest for prviate LB." }, "source": "meta" }