Hi, we've packaged ffmpeg/ffmprobe into a tar, extracted it, and installed it, along with a few wheels. It all works in the public validation set, however it does not work on the public test set.
\n\nOur same model works if instead of ffmpeg we use cv2.
\n\nCan anybody throw light at the differences between the dockers of public test vs. validation sets?
", "messages": [ { "id": "717560", "postDate": "01/13/2020 09:35:22", "content": "Hi, we've packaged ffmpeg/ffmprobe into a tar, extracted it, and installed it, along with a few wheels. It all works in the public validation set, however it does not work on the public test set.
\n\nOur same model works if instead of ffmpeg we use cv2.
\n\nCan anybody throw light at the differences between the dockers of public test vs. validation sets?
", "rawMarkdown": "Hi, we've packaged ffmpeg/ffmprobe into a tar, extracted it, and installed it, along with a few wheels. It all works in the public validation set, however it does not work on the public test set. \n\nOur same model works if instead of ffmpeg we use cv2.\n\nCan anybody throw light at the differences between the dockers of public test vs. validation sets?", "votes": null }, { "id": "717720", "postDate": "01/13/2020 14:21:53", "content": "check others' notebooks
", "rawMarkdown": "check others' notebooks", "votes": null }, { "id": "717735", "postDate": "01/13/2020 14:45:41", "content": "Can you be so kind to point me to a NB that:
\n\na) uses ffmpeg to extract frame pixels (not just read metadata)\nb) has a LB score (so we know it works in the public test set)
", "rawMarkdown": "Can you be so kind to point me to a NB that:\n\na) uses ffmpeg to extract frame pixels (not just read metadata)\nb) has a LB score (so we know it works in the public test set)", "votes": null }, { "id": "717842", "postDate": "01/13/2020 17:22:02", "content": "I can confirm it works. I've come to accept I'm just going to have some predictions that fail so I've built around that.
", "rawMarkdown": "I can confirm it works. I've come to accept I'm just going to have some predictions that fail so I've built around that.", "votes": null }, { "id": "719203", "postDate": "01/15/2020 08:37:37", "content": "Update here: it works but you have to bulletproof your code to skip videos ffmpeg fails to read (there are), and this only manifests itself in the public test set.
", "rawMarkdown": "Update here: it works but you have to bulletproof your code to skip videos ffmpeg fails to read (there are), and this only manifests itself in the public test set.", "votes": null }, { "id": "719322", "postDate": "01/15/2020 11:28:48", "content": "Thanks for updating. Do you mean private test set? public would be easy to spot.
", "rawMarkdown": "Thanks for updating. Do you mean *private* test set? public would be easy to spot.", "votes": null }, { "id": "719640", "postDate": "01/15/2020 17:16:34", "content": "I apparently don't have the right bullets - can you share a line or two of the code you used?
", "rawMarkdown": "I apparently don't have the right bullets - can you share a line or two of the code you used?", "votes": null }, { "id": "721967", "postDate": "01/18/2020 00:25:10", "content": "@pcjimmmy just be ready to catch an exception anywhere from spawning ffmpeg through getting your tensor ready for the net and skip the video if so. Test it by adding a few corrupted videos to your test set and check that it doesn't blow up locally before attempting to submit. That's what we did and it worked, hope it helps.
", "rawMarkdown": "pcjimmmy just be ready to catch an exception anywhere from spawning ffmpeg through getting your tensor ready for the net and skip the video if so. Test it by adding a few corrupted videos to your test set and check that it doesn't blow up locally before attempting to submit. That's what we did and it worked, hope it helps.", "votes": null }, { "id": "722007", "postDate": "01/18/2020 01:40:09", "content": "Thanks Prof
\n\nI did finally get it working on a bare bones random number type submission yesterday where I only created the audio file in the process.
\n\nI started adding a small amount of code to do the things I planned and it blew up again - Submission Scoring Error. I think that's kaggle_poop for wrong number of rows, IE, I left something out of my error trapping.
\n\nMy skill level is such that I don't think I can accurately generate corrupted videos - but that is a darn good idea. Big up vote from me for anyone who creates data set with some corrupted videos. I am 1/2 thru getting a 10 year old PC running again where I can impose some of the kaggle limits (16GB RAM, etc).
\n\nI mentioned in a post on cheating that most things done to prevent it mostly cause havoc to the good guys and only a minor inconvenience to the bad guys. For an absolute - I will NEVER do a submissions competition again. When your as old as I am you can say NEVER and have a good chance of it being true :)
\n\nI did way too much early coding on my local machine that is now cyber-dust since I can't believe that I can wheel all the libraries I used.
", "rawMarkdown": "Thanks Prof\n\nI did finally get it working on a bare bones random number type submission yesterday where I only created the audio file in the process. \n\nI started adding a small amount of code to do the things I planned and it blew up again - Submission Scoring Error. I think that's kaggle_poop for wrong number of rows, IE, I left something out of my error trapping.\n\nMy skill level is such that I don't think I can accurately generate corrupted videos - but that is a darn good idea. Big up vote from me for anyone who creates data set with some corrupted videos. I am 1/2 thru getting a 10 year old PC running again where I can impose some of the kaggle limits (16GB RAM, etc).\n\nI mentioned in a post on cheating that most things done to prevent it mostly cause havoc to the good guys and only a minor inconvenience to the bad guys. For an absolute - I will NEVER do a submissions competition again. When your as old as I am you can say NEVER and have a good chance of it being true :)\n\nI did way too much early coding on my local machine that is now cyber-dust since I can't believe that I can wheel all the libraries I used.", "votes": null } ], "comments": [ { "id": 717720, "author_name": "fionalxd", "author_url": "", "post_date": "01/13/2020 14:21:53", "content": "check others' notebooks
", "votes": null, "replies": [ { "id": 717735, "author_name": "antorsae", "author_url": "", "post_date": "01/13/2020 14:45:41", "content": "Can you be so kind to point me to a NB that:
\n\na) uses ffmpeg to extract frame pixels (not just read metadata)\nb) has a LB score (so we know it works in the public test set)
", "votes": null, "replies": [] }, { "id": 717842, "author_name": "ryches", "author_url": "", "post_date": "01/13/2020 17:22:02", "content": "I can confirm it works. I've come to accept I'm just going to have some predictions that fail so I've built around that.
", "votes": null, "replies": [] } ] }, { "id": 719203, "author_name": "antorsae", "author_url": "", "post_date": "01/15/2020 08:37:37", "content": "Update here: it works but you have to bulletproof your code to skip videos ffmpeg fails to read (there are), and this only manifests itself in the public test set.
", "votes": null, "replies": [ { "id": 719322, "author_name": "seesee", "author_url": "", "post_date": "01/15/2020 11:28:48", "content": "Thanks for updating. Do you mean private test set? public would be easy to spot.
", "votes": null, "replies": [] }, { "id": 719640, "author_name": "pcjimmmy", "author_url": "", "post_date": "01/15/2020 17:16:34", "content": "I apparently don't have the right bullets - can you share a line or two of the code you used?
", "votes": null, "replies": [] }, { "id": 721967, "author_name": "bacterio", "author_url": "", "post_date": "01/18/2020 00:25:10", "content": "@pcjimmmy just be ready to catch an exception anywhere from spawning ffmpeg through getting your tensor ready for the net and skip the video if so. Test it by adding a few corrupted videos to your test set and check that it doesn't blow up locally before attempting to submit. That's what we did and it worked, hope it helps.
", "votes": null, "replies": [] }, { "id": 722007, "author_name": "pcjimmmy", "author_url": "", "post_date": "01/18/2020 01:40:09", "content": "Thanks Prof
\n\nI did finally get it working on a bare bones random number type submission yesterday where I only created the audio file in the process.
\n\nI started adding a small amount of code to do the things I planned and it blew up again - Submission Scoring Error. I think that's kaggle_poop for wrong number of rows, IE, I left something out of my error trapping.
\n\nMy skill level is such that I don't think I can accurately generate corrupted videos - but that is a darn good idea. Big up vote from me for anyone who creates data set with some corrupted videos. I am 1/2 thru getting a 10 year old PC running again where I can impose some of the kaggle limits (16GB RAM, etc).
\n\nI mentioned in a post on cheating that most things done to prevent it mostly cause havoc to the good guys and only a minor inconvenience to the bad guys. For an absolute - I will NEVER do a submissions competition again. When your as old as I am you can say NEVER and have a good chance of it being true :)
\n\nI did way too much early coding on my local machine that is now cyber-dust since I can't believe that I can wheel all the libraries I used.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "717560": "Hi, we've packaged ffmpeg/ffmprobe into a tar, extracted it, and installed it, along with a few wheels. It all works in the public validation set, however it does not work on the public test set. \n\nOur same model works if instead of ffmpeg we use cv2.\n\nCan anybody throw light at the differences between the dockers of public test vs. validation sets?", "717720": "check others' notebooks", "717735": "Can you be so kind to point me to a NB that:\n\na) uses ffmpeg to extract frame pixels (not just read metadata)\nb) has a LB score (so we know it works in the public test set)", "717842": "I can confirm it works. I've come to accept I'm just going to have some predictions that fail so I've built around that.", "719203": "Update here: it works but you have to bulletproof your code to skip videos ffmpeg fails to read (there are), and this only manifests itself in the public test set.", "719322": "Thanks for updating. Do you mean *private* test set? public would be easy to spot.", "719640": "I apparently don't have the right bullets - can you share a line or two of the code you used?", "721967": "pcjimmmy just be ready to catch an exception anywhere from spawning ffmpeg through getting your tensor ready for the net and skip the video if so. Test it by adding a few corrupted videos to your test set and check that it doesn't blow up locally before attempting to submit. That's what we did and it worked, hope it helps.", "722007": "Thanks Prof\n\nI did finally get it working on a bare bones random number type submission yesterday where I only created the audio file in the process. \n\nI started adding a small amount of code to do the things I planned and it blew up again - Submission Scoring Error. I think that's kaggle_poop for wrong number of rows, IE, I left something out of my error trapping.\n\nMy skill level is such that I don't think I can accurately generate corrupted videos - but that is a darn good idea. Big up vote from me for anyone who creates data set with some corrupted videos. I am 1/2 thru getting a 10 year old PC running again where I can impose some of the kaggle limits (16GB RAM, etc).\n\nI mentioned in a post on cheating that most things done to prevent it mostly cause havoc to the good guys and only a minor inconvenience to the bad guys. For an absolute - I will NEVER do a submissions competition again. When your as old as I am you can say NEVER and have a good chance of it being true :)\n\nI did way too much early coding on my local machine that is now cyber-dust since I can't believe that I can wheel all the libraries I used." }, "source": "meta" }