I have spent 3 submits trying to debug my fork of kernel.
\n\nMy first fix of source notebook considered cut audio save on disk, I've read, that there is a limit on files that kernel might produce (~500?). So I've replaced ffmpeg command line to write it to stdout, and read from it, it has solved problem with Submission Scoring Error replacing it with Notebook Timeout. Notebook Timeout seems strange, kernel perfectly runs on 400 public test samples for about 5 min, and it takes >9 hours to run 4000?
There might be problem with reading audio from mp4 somewhere, other problems seem less possible, cause kernel runs without issues on publicly available data.
@admin: Is it possible to have official public notebook with valid audio loading or at least some guidelines for working with audio? With images it is easier due to preinstalled opencv, in case with audio, one need to use some kind of custom packages that probably won't work.
Thank you in advance
", "messages": [ { "id": "724660", "postDate": "01/21/2020 11:02:48", "content": "I have spent 3 submits trying to debug my fork of kernel.
\n\nMy first fix of source notebook considered cut audio save on disk, I've read, that there is a limit on files that kernel might produce (~500?). So I've replaced ffmpeg command line to write it to stdout, and read from it, it has solved problem with Submission Scoring Error replacing it with Notebook Timeout. Notebook Timeout seems strange, kernel perfectly runs on 400 public test samples for about 5 min, and it takes >9 hours to run 4000?
There might be problem with reading audio from mp4 somewhere, other problems seem less possible, cause kernel runs without issues on publicly available data.
@admin: Is it possible to have official public notebook with valid audio loading or at least some guidelines for working with audio? With images it is easier due to preinstalled opencv, in case with audio, one need to use some kind of custom packages that probably won't work.
Thank you in advance
", "rawMarkdown": "I have spent 3 submits trying to debug my fork of [kernel](https://www.kaggle.com/rakibilly/extract-audio-starter). \n\nMy first fix of source notebook considered cut audio save on disk, I've read, that there is a limit on files that kernel might produce (~500?). So I've replaced `ffmpeg` command line to write it to stdout, and read from it, it has solved problem with `Submission Scoring Error` replacing it with `Notebook Timeout`. `Notebook Timeout` seems strange, kernel perfectly runs on 400 public test samples for about 5 min, and it takes >9 hours to run 4000?\n\nThere might be problem with reading audio from `mp4` somewhere, other problems seem less possible, cause kernel runs without issues on publicly available data.\n\n@admin: Is it possible to have official public notebook with valid audio loading or at least some guidelines for working with audio? With images it is easier due to preinstalled `opencv`, in case with audio, one need to use some kind of custom packages that probably won't work.\n\nThank you in advance", "votes": null }, { "id": "725687", "postDate": "01/22/2020 11:47:56", "content": "I'm having the same problem. I'm using a CPU-only kernel that runs in ~270 seconds for the 400 public test samples, but still generates a timeout error.
", "rawMarkdown": "I'm having the same problem. I'm using a CPU-only kernel that runs in ~270 seconds for the 400 public test samples, but still generates a timeout error.", "votes": null }, { "id": "725853", "postDate": "01/22/2020 14:48:03", "content": "I'm facing similar problems, but with a regular DNN model. It runs within 15 minutes when I run the kernel on the 400 samples but getingt timeouts when submitting for competition. In my case it is a GPU kernel.
\n\nPerhaps some issues with the environment that Kaggle uses to run these submissions. Seen the three different type of kernels mentioned in this thread that exhibit this behavior, would expect something with very slow file/network IO???
", "rawMarkdown": "I'm facing similar problems, but with a regular DNN model. It runs within 15 minutes when I run the kernel on the 400 samples but getingt timeouts when submitting for competition. In my case it is a GPU kernel.\n\nPerhaps some issues with the environment that Kaggle uses to run these submissions. Seen the three different type of kernels mentioned in this thread that exhibit this behavior, would expect something with very slow file/network IO???", "votes": null }, { "id": "727452", "postDate": "01/23/2020 18:20:28", "content": "Your kernel is basis of what I am using to get audio. All my errors have been Submission Scoring Error. Did a quick look at your kernel - don't see any error trapping when your doing the ffmpeg command. I tried simple error trap (put the subprocess inside a try/except - that did not help. I than added/revised the subprocess call (its still in a try loop)
\n\ntry:\n a_wave = subprocess.call(command, shell=True)\nexcept:\n file = dataset file
\n\nif a_wave == 0:\n all is well\nelse:\n when its not zero an error has occurred \n file = dataset file
\n\nif a_wave is not zero than error has occurred. In the if/except I make the audio file a copy of file that I have included in my 1GB data set rather than the ffmpeg output. Subprocess should return a 0 for sucessful operations - I got a lot of 127 errors in other things I have tried to use ffmpeg for and 1 as value for the errors I forced when making audio.
\n\nI think if you don't error trap and than create a valid replacement your submission ends up without the right number (not enough) of rows. You might not need the replacement file but you need to be sure you have not lost a row in your submission when errors occur processing with ffmpeg.
\n\nI added a print for the time of the loop and when I process a bad file that I included with my 1GB dataset than the loop takes 30+seconds. So I think some type of timeout (30 seconds seems to be the default) occurs on failing ffmpeg calls - but that's way beyond my skill set so ?? I would assume that ffmpeg has a default timeout of 30 based on my loops but I have no idea how to change the default - but error trapping seemed to be enough to fix my issues.
", "rawMarkdown": "Your kernel is basis of what I am using to get audio. All my errors have been Submission Scoring Error. Did a quick look at your kernel - don't see any error trapping when your doing the ffmpeg command. I tried simple error trap (put the subprocess inside a try/except - that did not help. I than added/revised the subprocess call (its still in a try loop)\n\ntry:\n a_wave = subprocess.call(command, shell=True)\nexcept:\n file = dataset file\n\nif a_wave == 0:\n all is well\nelse:\n when its not zero an error has occurred \n file = dataset file\n\n\nif a_wave is not zero than error has occurred. In the if/except I make the audio file a copy of file that I have included in my 1GB data set rather than the ffmpeg output. Subprocess should return a 0 for sucessful operations - I got a lot of 127 errors in other things I have tried to use ffmpeg for and 1 as value for the errors I forced when making audio.\n\nI think if you don't error trap and than create a valid replacement your submission ends up without the right number (not enough) of rows. You might not need the replacement file but you need to be sure you have not lost a row in your submission when errors occur processing with ffmpeg.\n\nI added a print for the time of the loop and when I process a bad file that I included with my 1GB dataset than the loop takes 30+seconds. So I think some type of timeout (30 seconds seems to be the default) occurs on failing ffmpeg calls - but that's way beyond my skill set so ?? I would assume that ffmpeg has a default timeout of 30 based on my loops but I have no idea how to change the default - but error trapping seemed to be enough to fix my issues.", "votes": null }, { "id": "727988", "postDate": "01/24/2020 09:48:43", "content": "Hello, thank you for your advice. It seems that proper error handling sound as good idea, I'm going to try it.
", "rawMarkdown": "Hello, thank you for your advice. It seems that proper error handling sound as good idea, I'm going to try it.", "votes": null }, { "id": "766603", "postDate": "03/08/2020 12:54:03", "content": "I'm having this issue - we're you able to resolve?
\n\nBook runs in 25 minutes on 400 videos - should scale up to a little over 4 hours on 4,000 videos. My workflow entails extracting faces, serializing them to disk and then predicting them. I had to do it this way to avoid GPU memory issues between my face extraction model and my prediction model.
\n\nI have error handling in place for the face extraction and serialization.
\n\nHow can I make.sure that the GPU is on for the submission?
\n\nAre the limits for working disk space the same as for working kernels at 5 GB? I calculate less than 2GB of usage.
\n\nI'm deleting all working files except submission.csv at the end of my process, but it is leaving one .pyc file. What are the requirements to clean up working directory?
\n\nAny other suggestions on how to debug would be greatly appreciated.
", "rawMarkdown": "I'm having this issue - we're you able to resolve?\n\nBook runs in 25 minutes on 400 videos - should scale up to a little over 4 hours on 4,000 videos. My workflow entails extracting faces, serializing them to disk and then predicting them. I had to do it this way to avoid GPU memory issues between my face extraction model and my prediction model.\n\nI have error handling in place for the face extraction and serialization.\n\nHow can I make.sure that the GPU is on for the submission?\n\nAre the limits for working disk space the same as for working kernels at 5 GB? I calculate less than 2GB of usage.\n\nI'm deleting all working files except submission.csv at the end of my process, but it is leaving one .pyc file. What are the requirements to clean up working directory?\n\nAny other suggestions on how to debug would be greatly appreciated.", "votes": null } ], "comments": [ { "id": 725687, "author_name": "camyok", "author_url": "", "post_date": "01/22/2020 11:47:56", "content": "I'm having the same problem. I'm using a CPU-only kernel that runs in ~270 seconds for the 400 public test samples, but still generates a timeout error.
", "votes": null, "replies": [ { "id": 766603, "author_name": "calebeverett", "author_url": "", "post_date": "03/08/2020 12:54:03", "content": "I'm having this issue - we're you able to resolve?
\n\nBook runs in 25 minutes on 400 videos - should scale up to a little over 4 hours on 4,000 videos. My workflow entails extracting faces, serializing them to disk and then predicting them. I had to do it this way to avoid GPU memory issues between my face extraction model and my prediction model.
\n\nI have error handling in place for the face extraction and serialization.
\n\nHow can I make.sure that the GPU is on for the submission?
\n\nAre the limits for working disk space the same as for working kernels at 5 GB? I calculate less than 2GB of usage.
\n\nI'm deleting all working files except submission.csv at the end of my process, but it is leaving one .pyc file. What are the requirements to clean up working directory?
\n\nAny other suggestions on how to debug would be greatly appreciated.
", "votes": null, "replies": [] } ] }, { "id": 725853, "author_name": "peterdekkers101", "author_url": "", "post_date": "01/22/2020 14:48:03", "content": "I'm facing similar problems, but with a regular DNN model. It runs within 15 minutes when I run the kernel on the 400 samples but getingt timeouts when submitting for competition. In my case it is a GPU kernel.
\n\nPerhaps some issues with the environment that Kaggle uses to run these submissions. Seen the three different type of kernels mentioned in this thread that exhibit this behavior, would expect something with very slow file/network IO???
", "votes": null, "replies": [] }, { "id": 727452, "author_name": "pcjimmmy", "author_url": "", "post_date": "01/23/2020 18:20:28", "content": "Your kernel is basis of what I am using to get audio. All my errors have been Submission Scoring Error. Did a quick look at your kernel - don't see any error trapping when your doing the ffmpeg command. I tried simple error trap (put the subprocess inside a try/except - that did not help. I than added/revised the subprocess call (its still in a try loop)
\n\ntry:\n a_wave = subprocess.call(command, shell=True)\nexcept:\n file = dataset file
\n\nif a_wave == 0:\n all is well\nelse:\n when its not zero an error has occurred \n file = dataset file
\n\nif a_wave is not zero than error has occurred. In the if/except I make the audio file a copy of file that I have included in my 1GB data set rather than the ffmpeg output. Subprocess should return a 0 for sucessful operations - I got a lot of 127 errors in other things I have tried to use ffmpeg for and 1 as value for the errors I forced when making audio.
\n\nI think if you don't error trap and than create a valid replacement your submission ends up without the right number (not enough) of rows. You might not need the replacement file but you need to be sure you have not lost a row in your submission when errors occur processing with ffmpeg.
\n\nI added a print for the time of the loop and when I process a bad file that I included with my 1GB dataset than the loop takes 30+seconds. So I think some type of timeout (30 seconds seems to be the default) occurs on failing ffmpeg calls - but that's way beyond my skill set so ?? I would assume that ffmpeg has a default timeout of 30 based on my loops but I have no idea how to change the default - but error trapping seemed to be enough to fix my issues.
", "votes": null, "replies": [ { "id": 727988, "author_name": "ivanid", "author_url": "", "post_date": "01/24/2020 09:48:43", "content": "Hello, thank you for your advice. It seems that proper error handling sound as good idea, I'm going to try it.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "724660": "I have spent 3 submits trying to debug my fork of [kernel](https://www.kaggle.com/rakibilly/extract-audio-starter). \n\nMy first fix of source notebook considered cut audio save on disk, I've read, that there is a limit on files that kernel might produce (~500?). So I've replaced `ffmpeg` command line to write it to stdout, and read from it, it has solved problem with `Submission Scoring Error` replacing it with `Notebook Timeout`. `Notebook Timeout` seems strange, kernel perfectly runs on 400 public test samples for about 5 min, and it takes >9 hours to run 4000?\n\nThere might be problem with reading audio from `mp4` somewhere, other problems seem less possible, cause kernel runs without issues on publicly available data.\n\n@admin: Is it possible to have official public notebook with valid audio loading or at least some guidelines for working with audio? With images it is easier due to preinstalled `opencv`, in case with audio, one need to use some kind of custom packages that probably won't work.\n\nThank you in advance", "725687": "I'm having the same problem. I'm using a CPU-only kernel that runs in ~270 seconds for the 400 public test samples, but still generates a timeout error.", "725853": "I'm facing similar problems, but with a regular DNN model. It runs within 15 minutes when I run the kernel on the 400 samples but getingt timeouts when submitting for competition. In my case it is a GPU kernel.\n\nPerhaps some issues with the environment that Kaggle uses to run these submissions. Seen the three different type of kernels mentioned in this thread that exhibit this behavior, would expect something with very slow file/network IO???", "727452": "Your kernel is basis of what I am using to get audio. All my errors have been Submission Scoring Error. Did a quick look at your kernel - don't see any error trapping when your doing the ffmpeg command. I tried simple error trap (put the subprocess inside a try/except - that did not help. I than added/revised the subprocess call (its still in a try loop)\n\ntry:\n a_wave = subprocess.call(command, shell=True)\nexcept:\n file = dataset file\n\nif a_wave == 0:\n all is well\nelse:\n when its not zero an error has occurred \n file = dataset file\n\n\nif a_wave is not zero than error has occurred. In the if/except I make the audio file a copy of file that I have included in my 1GB data set rather than the ffmpeg output. Subprocess should return a 0 for sucessful operations - I got a lot of 127 errors in other things I have tried to use ffmpeg for and 1 as value for the errors I forced when making audio.\n\nI think if you don't error trap and than create a valid replacement your submission ends up without the right number (not enough) of rows. You might not need the replacement file but you need to be sure you have not lost a row in your submission when errors occur processing with ffmpeg.\n\nI added a print for the time of the loop and when I process a bad file that I included with my 1GB dataset than the loop takes 30+seconds. So I think some type of timeout (30 seconds seems to be the default) occurs on failing ffmpeg calls - but that's way beyond my skill set so ?? I would assume that ffmpeg has a default timeout of 30 based on my loops but I have no idea how to change the default - but error trapping seemed to be enough to fix my issues.", "727988": "Hello, thank you for your advice. It seems that proper error handling sound as good idea, I'm going to try it.", "766603": "I'm having this issue - we're you able to resolve?\n\nBook runs in 25 minutes on 400 videos - should scale up to a little over 4 hours on 4,000 videos. My workflow entails extracting faces, serializing them to disk and then predicting them. I had to do it this way to avoid GPU memory issues between my face extraction model and my prediction model.\n\nI have error handling in place for the face extraction and serialization.\n\nHow can I make.sure that the GPU is on for the submission?\n\nAre the limits for working disk space the same as for working kernels at 5 GB? I calculate less than 2GB of usage.\n\nI'm deleting all working files except submission.csv at the end of my process, but it is leaving one .pyc file. What are the requirements to clean up working directory?\n\nAny other suggestions on how to debug would be greatly appreciated." }, "source": "meta" }