The submission could take many hours to run. When using TTA or Ensemble, it may not finish in 9 hours. It would be really helpful to estimate the model submission time if the Admin could add running time it took for each submission.
", "messages": [ { "id": "1104566", "postDate": "12/07/2020 04:21:33", "content": "The submission could take many hours to run. When using TTA or Ensemble, it may not finish in 9 hours. It would be really helpful to estimate the model submission time if the Admin could add running time it took for each submission.
", "rawMarkdown": "The submission could take many hours to run. When using TTA or Ensemble, it may not finish in 9 hours. It would be really helpful to estimate the model submission time if the Admin could add running time it took for each submission.", "votes": null }, { "id": "1104765", "postDate": "12/07/2020 07:54:34", "content": "I was a bit annoyed at not being able to see the exact run-time, too. Then I realized there might be one sensible reason for that. E.g. you could encode a lot of leaderboard probing feedback in delays you put in your program. Let's say you know the runtime of your code after testing it - see below - and you want to encode the answer to a bunch of y/n questions, then, say, encode number of predictions in class 0 within 10% of training data = 0 extra delay, outside 10% = 5 min delay and then the next question is 0 delay vs. 10 min delay and so on (you can probably encode 5 to 6 bits that way, if you think you can detect more fine-grained delays even more). Being protected from doing silly stuff like that to some extent (yes, I know, I could watch how long each active event lasts and the value of doing this must be somewhat low), has some value.
\nWhat I've done for the moment is to profile my code with the training data (i.e. see how much the pre-inference prep steps take and then seeing how much variation there is in running 1000 samples through). I'd assume that would give you the same answer (but of course does burn some GPU time).
", "rawMarkdown": "I was a bit annoyed at not being able to see the exact run-time, too. Then I realized there might be one sensible reason for that. E.g. you could encode a lot of leaderboard probing feedback in delays you put in your program. Let's say you know the runtime of your code after testing it - see below - and you want to encode the answer to a bunch of y/n questions, then, say, encode number of predictions in class 0 within 10% of training data = 0 extra delay, outside 10% = 5 min delay and then the next question is 0 delay vs. 10 min delay and so on (you can probably encode 5 to 6 bits that way, if you think you can detect more fine-grained delays even more). Being protected from doing silly stuff like that to some extent (yes, I know, I could watch how long each active event lasts and the value of doing this must be somewhat low), has some value.\n\nWhat I've done for the moment is to profile my code with the training data (i.e. see how much the pre-inference prep steps take and then seeing how much variation there is in running 1000 samples through). I'd assume that would give you the same answer (but of course does burn some GPU time).", "votes": null } ], "comments": [ { "id": 1104765, "author_name": "bjoernholzhauer", "author_url": "", "post_date": "12/07/2020 07:54:34", "content": "I was a bit annoyed at not being able to see the exact run-time, too. Then I realized there might be one sensible reason for that. E.g. you could encode a lot of leaderboard probing feedback in delays you put in your program. Let's say you know the runtime of your code after testing it - see below - and you want to encode the answer to a bunch of y/n questions, then, say, encode number of predictions in class 0 within 10% of training data = 0 extra delay, outside 10% = 5 min delay and then the next question is 0 delay vs. 10 min delay and so on (you can probably encode 5 to 6 bits that way, if you think you can detect more fine-grained delays even more). Being protected from doing silly stuff like that to some extent (yes, I know, I could watch how long each active event lasts and the value of doing this must be somewhat low), has some value.
\nWhat I've done for the moment is to profile my code with the training data (i.e. see how much the pre-inference prep steps take and then seeing how much variation there is in running 1000 samples through). I'd assume that would give you the same answer (but of course does burn some GPU time).
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1104566": "The submission could take many hours to run. When using TTA or Ensemble, it may not finish in 9 hours. It would be really helpful to estimate the model submission time if the Admin could add running time it took for each submission.", "1104765": "I was a bit annoyed at not being able to see the exact run-time, too. Then I realized there might be one sensible reason for that. E.g. you could encode a lot of leaderboard probing feedback in delays you put in your program. Let's say you know the runtime of your code after testing it - see below - and you want to encode the answer to a bunch of y/n questions, then, say, encode number of predictions in class 0 within 10% of training data = 0 extra delay, outside 10% = 5 min delay and then the next question is 0 delay vs. 10 min delay and so on (you can probably encode 5 to 6 bits that way, if you think you can detect more fine-grained delays even more). Being protected from doing silly stuff like that to some extent (yes, I know, I could watch how long each active event lasts and the value of doing this must be somewhat low), has some value.\n\nWhat I've done for the moment is to profile my code with the training data (i.e. see how much the pre-inference prep steps take and then seeing how much variation there is in running 1000 samples through). I'd assume that would give you the same answer (but of course does burn some GPU time)." }, "source": "meta" }