Hi, my submission scheme is such that: I create a dataframe of 0.5 values per sample, then replace those values as I calculate the scores. I wrote it like this so that if my script timeouts or sth at some point, perhaps I can still get a score with the rest of the values as random.
\n\nSo this was working great but then yesterday I was gonna submit a new model. Everything was working, I once got CUDA out of memory so I decided to do 1 video batches (only 10 frames). This ran fine on the public set (commit) and I got scores for every sample. Then I submitted this commit and I **immediately **got a score of exactly random (normally it would run for 8-9 hours), meaning my code somehow failed and it just got the random score csv. I suspected CUDA out of memory again, and submitted a commit on CPU, which ended up immediately giving random again.
\n\nBesides my new model, I have exact same script as my previous submissions, so I don't see what else it could have been other than gpu memory thing.
\n\nSo Im out of my submissions for today, so would appreciate any ideas to help me debug :) Has anyone experienced sth similar?
", "messages": [ { "id": "738304", "postDate": "02/06/2020 11:03:47", "content": "Hi, my submission scheme is such that: I create a dataframe of 0.5 values per sample, then replace those values as I calculate the scores. I wrote it like this so that if my script timeouts or sth at some point, perhaps I can still get a score with the rest of the values as random.
\n\nSo this was working great but then yesterday I was gonna submit a new model. Everything was working, I once got CUDA out of memory so I decided to do 1 video batches (only 10 frames). This ran fine on the public set (commit) and I got scores for every sample. Then I submitted this commit and I **immediately **got a score of exactly random (normally it would run for 8-9 hours), meaning my code somehow failed and it just got the random score csv. I suspected CUDA out of memory again, and submitted a commit on CPU, which ended up immediately giving random again.
\n\nBesides my new model, I have exact same script as my previous submissions, so I don't see what else it could have been other than gpu memory thing.
\n\nSo Im out of my submissions for today, so would appreciate any ideas to help me debug :) Has anyone experienced sth similar?
", "rawMarkdown": "Hi, my submission scheme is such that: I create a dataframe of 0.5 values per sample, then replace those values as I calculate the scores. I wrote it like this so that if my script timeouts or sth at some point, perhaps I can still get a score with the rest of the values as random.\n\nSo this was working great but then yesterday I was gonna submit a new model. Everything was working, I once got CUDA out of memory so I decided to do 1 video batches (only 10 frames). This ran fine on the public set (commit) and I got scores for every sample. Then I submitted this commit and I **immediately **got a score of exactly random (normally it would run for 8-9 hours), meaning my code somehow failed and it just got the random score csv. I suspected CUDA out of memory again, and submitted a commit on CPU, which ended up immediately giving random again. \n\nBesides my new model, I have exact same script as my previous submissions, so I don't see what else it could have been other than gpu memory thing.\n\nSo Im out of my submissions for today, so would appreciate any ideas to help me debug :) Has anyone experienced sth similar?", "votes": null }, { "id": "738371", "postDate": "02/06/2020 12:54:04", "content": "Related additonal question: If your script has an error, but you still produce a csv, my understanding is that the submission won't show you any error? Or even if you don't produce an error, show you submission scoring error at best?
", "rawMarkdown": "Related additonal question: If your script has an error, but you still produce a csv, my understanding is that the submission won't show you any error? Or even if you don't produce an error, show you *submission scoring error* at best?", "votes": null }, { "id": "742137", "postDate": "02/11/2020 05:03:56", "content": "So if one cell produces an error, it will skip that error cell and execute the next one. If it did not output a submission.csv, it will say submission csv not found; if it did not provide an invalid submission.csv, it will say submission scoring error, if it did provide a valid submission.csv, it will normally score(probably bad).
", "rawMarkdown": "So if one cell produces an error, it will skip that error cell and execute the next one. If it did not output a submission.csv, it will say *submission csv not found*; if it did not provide an invalid submission.csv, it will say *submission scoring error*, if it did provide a valid submission.csv, it will normally score(probably bad).", "votes": null }, { "id": "745079", "postDate": "02/13/2020 13:05:08", "content": "Great thanks! I solved this btw by just copying an older submission kernel but I didn't really find a bug in the old one.
", "rawMarkdown": "Great thanks! I solved this btw by just copying an older submission kernel but I didn't really find a bug in the old one.", "votes": null } ], "comments": [ { "id": 738371, "author_name": "ipekgani", "author_url": "", "post_date": "02/06/2020 12:54:04", "content": "Related additonal question: If your script has an error, but you still produce a csv, my understanding is that the submission won't show you any error? Or even if you don't produce an error, show you submission scoring error at best?
", "votes": null, "replies": [ { "id": 742137, "author_name": "unkownhihi", "author_url": "", "post_date": "02/11/2020 05:03:56", "content": "So if one cell produces an error, it will skip that error cell and execute the next one. If it did not output a submission.csv, it will say submission csv not found; if it did not provide an invalid submission.csv, it will say submission scoring error, if it did provide a valid submission.csv, it will normally score(probably bad).
", "votes": null, "replies": [] }, { "id": 745079, "author_name": "ipekgani", "author_url": "", "post_date": "02/13/2020 13:05:08", "content": "Great thanks! I solved this btw by just copying an older submission kernel but I didn't really find a bug in the old one.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "738304": "Hi, my submission scheme is such that: I create a dataframe of 0.5 values per sample, then replace those values as I calculate the scores. I wrote it like this so that if my script timeouts or sth at some point, perhaps I can still get a score with the rest of the values as random.\n\nSo this was working great but then yesterday I was gonna submit a new model. Everything was working, I once got CUDA out of memory so I decided to do 1 video batches (only 10 frames). This ran fine on the public set (commit) and I got scores for every sample. Then I submitted this commit and I **immediately **got a score of exactly random (normally it would run for 8-9 hours), meaning my code somehow failed and it just got the random score csv. I suspected CUDA out of memory again, and submitted a commit on CPU, which ended up immediately giving random again. \n\nBesides my new model, I have exact same script as my previous submissions, so I don't see what else it could have been other than gpu memory thing.\n\nSo Im out of my submissions for today, so would appreciate any ideas to help me debug :) Has anyone experienced sth similar?", "738371": "Related additonal question: If your script has an error, but you still produce a csv, my understanding is that the submission won't show you any error? Or even if you don't produce an error, show you *submission scoring error* at best?", "742137": "So if one cell produces an error, it will skip that error cell and execute the next one. If it did not output a submission.csv, it will say *submission csv not found*; if it did not provide an invalid submission.csv, it will say *submission scoring error*, if it did provide a valid submission.csv, it will normally score(probably bad).", "745079": "Great thanks! I solved this btw by just copying an older submission kernel but I didn't really find a bug in the old one." }, "source": "meta" }