![\"Image\"](\"https://i.imgur.com/LAK54F7.png\")
\n(*updated on 18-DEC-2018) You can find the details of the speed prize here:
\n\nhttps://www.kaggle.com/c/airbus-ship-detection#Algorithm-Speed-Prize
\n\nPlease ask questions about it in this thread. Thx!
", "messages": [ { "id": "409837", "postDate": "10/24/2018 23:09:51", "content": "You can find the details of the speed prize here:
\n\nhttps://www.kaggle.com/c/airbus-ship-detection#Algorithm-Speed-Prize
\n\nPlease ask questions about it in this thread. Thx!
", "rawMarkdown": "You can find the details of the speed prize here:\n\nhttps://www.kaggle.com/c/airbus-ship-detection#Algorithm-Speed-Prize\n\nPlease ask questions about it in this thread. Thx!", "votes": null }, { "id": "409841", "postDate": "10/24/2018 23:30:45", "content": "Hi inversion, thanks for posting more details about the speed prize.\nTwo points came to my mind:
\n\nThank you in advance,\nRenan
", "rawMarkdown": "Hi inversion, thanks for posting more details about the speed prize.\nTwo points came to my mind:\n\n 1. \"The winner will be the fastest Kernel (that does not have significant degradation in predictive ability against the validation set)\". What do you mean by validation set? Is it the test set of the main competition? Also, could you please quantify \"significant degradation in predictive ability\"?\n 2. Do we have to use the same model design we used in the main competition (best F2 score) for the speed prize or we can design other models aiming for fastest algorithm?\n\nThank you in advance,\nRenan", "votes": null }, { "id": "410640", "postDate": "10/26/2018 11:40:19", "content": "Are there any conditions about quality of the models (minimum of the score, for example)?
\n\nP.S. I can submit empty masks very fast :D
", "rawMarkdown": "Are there any conditions about quality of the models (minimum of the score, for example)? \n\nP.S. I can submit empty masks very fast :D", "votes": null }, { "id": "411701", "postDate": "10/28/2018 20:17:35", "content": "Hi Renan,
\n\nThank you for your questions. The validation dataset is the test dataset of the main competition (called test_v2). The speed prize will be open to teams that score in top 100 of the private leaderboard. Let's call S the score of the 100th participant on the private leaderboard on the closing day of the main competition. We consider that the score of the new speed kernel on the private leaderboard should not go below S. As long as this is verified, you can use any model design that you want.
\n\nKind regards,
\n\nJeff
", "rawMarkdown": "Hi Renan,\n\nThank you for your questions. The validation dataset is the test dataset of the main competition (called test_v2). The speed prize will be open to teams that score in top 100 of the private leaderboard. Let's call S the score of the 100th participant on the private leaderboard on the closing day of the main competition. We consider that the score of the new speed kernel on the private leaderboard should not go below S. As long as this is verified, you can use any model design that you want.\n\nKind regards,\n\nJeff", "votes": null }, { "id": "411718", "postDate": "10/28/2018 21:52:42", "content": "That's very clear. Thank you Jeff!
", "rawMarkdown": "That's very clear. Thank you Jeff!", "votes": null }, { "id": "416929", "postDate": "11/07/2018 13:01:26", "content": "Can we form new teams for the speed prize competition?
", "rawMarkdown": "Can we form new teams for the speed prize competition?", "votes": null }, { "id": "419105", "postDate": "11/11/2018 09:20:11", "content": "What is the minimum system requirement for this algorithm speed challenge? like for example in Human Protein Atlas Image Classification challenge, minimum hardware limits are: CPU Cores: 2, RAM: 4GB and GPUs: Integrated Intel Graphics.. What about threshold scoring requirement for algorithm? what is the baseline score? thanks!
", "rawMarkdown": "What is the minimum system requirement for this algorithm speed challenge? like for example in Human Protein Atlas Image Classification challenge, minimum hardware limits are: CPU Cores: 2, RAM: 4GB and GPUs: Integrated Intel Graphics.. What about threshold scoring requirement for algorithm? what is the baseline score? thanks!", "votes": null }, { "id": "421401", "postDate": "11/15/2018 00:44:00", "content": "Will there be a leaderboard for speed prize ?
", "rawMarkdown": "Will there be a leaderboard for speed prize ?", "votes": null }, { "id": "421901", "postDate": "11/15/2018 14:50:18", "content": "The Kernel will be validated against a new dataset.\nfrom where we get the new data set...
", "rawMarkdown": "The Kernel will be validated against a new dataset.\nfrom where we get the new data set...", "votes": null }, { "id": "423461", "postDate": "11/18/2018 10:00:14", "content": "@jaideepvalani My guess is you can't. This prevents cheating (e.g. precompute the predictions and then just upload them.
", "rawMarkdown": "jaideepvalani My guess is you can't. This prevents cheating (e.g. precompute the predictions and then just upload them.", "votes": null }, { "id": "423555", "postDate": "11/18/2018 14:23:53", "content": "I meant in the rulesfor algo speed prize it was mentioned that new data set is going to be used.. i was needing a clarification on that..
", "rawMarkdown": "I meant in the rulesfor algo speed prize it was mentioned that new data set is going to be used.. i was needing a clarification on that..", "votes": null }, { "id": "423961", "postDate": "11/19/2018 10:29:29", "content": "Hello everybody! Regarding the speed test, should we add the model creation and weight loading time to the overall inference time?
", "rawMarkdown": "Hello everybody! Regarding the speed test, should we add the model creation and weight loading time to the overall inference time?", "votes": null }, { "id": "424436", "postDate": "11/20/2018 05:56:17", "content": "Could you, please, clarify if inference time includes the time for loading and preprocessing the data? Also, if there any way to compare the speed of my kernel with others before the submission deadline? In particular if I get, for example, 10 minutes or 1 hour, is it fast or slow?
", "rawMarkdown": "Could you, please, clarify if inference time includes the time for loading and preprocessing the data? Also, if there any way to compare the speed of my kernel with others before the submission deadline? In particular if I get, for example, 10 minutes or 1 hour, is it fast or slow?", "votes": null }, { "id": "424524", "postDate": "11/20/2018 09:01:23", "content": "@inversion Does ones entry just have to create a better submission than the current place 100: 0.84416 or is there a separate set on which our approach will be evaluated?
Hi Costas,
\n\nThanks for participating in the Algorithm Speed Prize! We expect the inference time to be only the computation time on imagery (including pre-processing, models computation and post-processing) without loading of weights and models. The rational is to think of a running docker container which would wait for imagery buffer to compute ships masks.
\n\nKind regards,
\n\nJeff.
", "rawMarkdown": "Hi Costas,\n\nThanks for participating in the Algorithm Speed Prize! We expect the inference time to be only the computation time on imagery (including pre-processing, models computation and post-processing) without loading of weights and models. The rational is to think of a running docker container which would wait for imagery buffer to compute ships masks. \n\nKind regards,\n\nJeff.", "votes": null }, { "id": "424532", "postDate": "11/20/2018 09:19:39", "content": "Hi See,
\n\nThanks for participating in the Algorithm Speed Prize! You are right. The kernel should create a submission against the current test_v2 dataset and rank better than 0.84416. You should create a comment at the end of your kernel with the score on the private LB and the inference time. We will run inference on a new test dataset just to make sure that the model still runs correctly on new imagery.
\n\nKind regards,
\n\nJeff.
", "rawMarkdown": "Hi See,\n\nThanks for participating in the Algorithm Speed Prize! You are right. The kernel should create a submission against the current test_v2 dataset and rank better than 0.84416. You should create a comment at the end of your kernel with the score on the private LB and the inference time. We will run inference on a new test dataset just to make sure that the model still runs correctly on new imagery.\n\nKind regards,\n\nJeff.", "votes": null }, { "id": "424688", "postDate": "11/20/2018 14:32:47", "content": "Thank you Jeff for your replies! So the way I understand it is that in the new test dataset you will not score the model. You will just measure the inference time! So the 0.84416 will define all eligible participants and then it is just a question of speed. I guess the private score could be used to break ties or something. Am I correct?
", "rawMarkdown": "Thank you Jeff for your replies! So the way I understand it is that in the new test dataset you will not score the model. You will just measure the inference time! So the 0.84416 will define all eligible participants and then it is just a question of speed. I guess the private score could be used to break ties or something. Am I correct?", "votes": null }, { "id": "424740", "postDate": "11/20/2018 15:50:24", "content": "Hello,\nYou should use kernels (with GPU) for the speed prize. And the minimum score is 0.84416\nKind regards,\nJeff
", "rawMarkdown": "Hello,\nYou should use kernels (with GPU) for the speed prize. And the minimum score is 0.84416\nKind regards,\nJeff", "votes": null }, { "id": "424750", "postDate": "11/20/2018 16:01:19", "content": "Yes. I will measure speed and score on test_v2. Then I will rate participants on fastest speed given that score >= 0.84416. I will use another new test dataset to measure inference and score again and verify if there is no discrepencies in the ranking (i.e. the final algorithm should perform as well on a new dataset because that is what is really important for us).
", "rawMarkdown": "Yes. I will measure speed and score on test_v2. Then I will rate participants on fastest speed given that score >= 0.84416. I will use another new test dataset to measure inference and score again and verify if there is no discrepencies in the ranking (i.e. the final algorithm should perform as well on a new dataset because that is what is really important for us).", "votes": null }, { "id": "424816", "postDate": "11/20/2018 17:48:32", "content": "Hi jeff \nthanks for replies\nI m new to kaggle competitions ,that is this first one...\n ,could u please ans below queries\n1) we only calculating here the prediction time ,so that is how long does the model takes in predicting the testv2?\n2) from where does my time starts... to predict i will have to rebuild the model object,load the trained weights finally start predicting...\n3) if i resubmit then my rank could improve also ,so new rank will be considered or first time private lb rank be considered..
", "rawMarkdown": "Hi jeff \nthanks for replies\nI m new to kaggle competitions ,that is this first one...\n ,could u please ans below queries\n1) we only calculating here the prediction time ,so that is how long does the model takes in predicting the testv2?\n2) from where does my time starts... to predict i will have to rebuild the model object,load the trained weights finally start predicting...\n3) if i resubmit then my rank could improve also ,so new rank will be considered or first time private lb rank be considered..", "votes": null }, { "id": "425219", "postDate": "11/21/2018 09:44:31", "content": "Hi iafoss,
\n\nThank you so much for participating in the Algorithm Speed Prize, especially after all the good stuff that you already shared with the community in Kernels and Discussions. Your questions are very legitimate and we are looking into ways of sharing a kind of leaderboard with the participants. I will get back to all of you in this discussion thread as soon as possible. For the inference time, I would say that fast is closer to 10 minutes than to 1 hour :)
\n\nKind regards,\nJeff.
", "rawMarkdown": "Hi iafoss,\n\nThank you so much for participating in the Algorithm Speed Prize, especially after all the good stuff that you already shared with the community in Kernels and Discussions. Your questions are very legitimate and we are looking into ways of sharing a kind of leaderboard with the participants. I will get back to all of you in this discussion thread as soon as possible. For the inference time, I would say that fast is closer to 10 minutes than to 1 hour :)\n\nKind regards,\nJeff.", "votes": null }, { "id": "425507", "postDate": "11/21/2018 17:44:16", "content": "Hi Jaideep,\nTo answer your questions :\n1.- Yes, inference time is how much the model(s) need to predict all images in test_v2\n2.- Yes, the inference time does not include setup the model, load the weights, load the images, etc... but includes pre-processing (i.e. rescaling), model prediction (with i.e. ensembling and TTA) and post processing (i.e. removing the overlaps, creating the CSV file). Everything that stays the same whatever the image should not be counted, everything that needs to be recomputed with new images needs to be included.\n3.- Yes, of course. You can submit multiple times and only your best score will count.\nKind regards,\nJeff.
", "rawMarkdown": "Hi Jaideep,\nTo answer your questions :\n1.- Yes, inference time is how much the model(s) need to predict all images in test_v2\n2.- Yes, the inference time does not include setup the model, load the weights, load the images, etc... but includes pre-processing (i.e. rescaling), model prediction (with i.e. ensembling and TTA) and post processing (i.e. removing the overlaps, creating the CSV file). Everything that stays the same whatever the image should not be counted, everything that needs to be recomputed with new images needs to be included.\n3.- Yes, of course. You can submit multiple times and only your best score will count.\nKind regards,\nJeff.", "votes": null }, { "id": "425516", "postDate": "11/21/2018 18:07:11", "content": "Jeff Faudi, thank you so much for working on it. It is really important to know how the model performs in comparison with others. My current fastest kernel takes 13 minutes, and I would really like to compare it with results of other people before the deadline.
", "rawMarkdown": "Jeff Faudi, thank you so much for working on it. It is really important to know how the model performs in comparison with others. My current fastest kernel takes 13 minutes, and I would really like to compare it with results of other people before the deadline.", "votes": null }, { "id": "425518", "postDate": "11/21/2018 18:11:09", "content": "Jeff Faudi, thank you for clarification on what must be included in the inference time.
", "rawMarkdown": "Jeff Faudi, thank you for clarification on what must be included in the inference time.", "votes": null }, { "id": "427373", "postDate": "11/25/2018 11:43:28", "content": "Hi Jeff,
\n\nYes, the inference time does not include setup the model, load the weights, load the images, etc…\nI understand your rationale. Though, for me it is not clear what you mean by excluding the image loading (reading the image string & jpeg decoding). Could you provide a modified template? I have been using the one provided by @inversion (competition tab), without the model loading.
\n\nmodel = load_model()\nimport time\ninference_start = time.time()\n###### inference code #######\ninference_end = time.time()\nprint('Inference Time: %0.2f Minutes'%((inference_end - inference_start)/60))\nEvery framework has different loaders. I am using tensorflow. Reading the image string and decoding it is part of the graph ~~and it is hard to untangle these timings~~ (see below).
", "rawMarkdown": "Hi Jeff,\n\n Yes, the inference time does not include setup the model, load the weights, load the images, etc…\n\nI understand your rationale. Though, for me it is not clear what you mean by excluding the image loading (reading the image string & jpeg decoding). Could you provide a modified template? I have been using the one provided by @inversion (competition tab), without the model loading.\n\n model = load_model()\n import time\n inference_start = time.time()\n ###### inference code #######\n inference_end = time.time()\n print('Inference Time: %0.2f Minutes'%((inference_end - inference_start)/60))\n\nEvery framework has different loaders. I am using tensorflow. Reading the image string and decoding it is part of the graph ~~and it is hard to untangle these timings~~ (see below).", "votes": null }, { "id": "427396", "postDate": "11/25/2018 12:40:38", "content": "To make it more concrete. We can exclude the time to load an image batch in RGB format as np.uint8?
\n\nmodel = load_model()\nimport time\ninference_time = 0.\nsubmission = pd.DataFrame()\n###### inference code #######\nfor k in range(len(data_loader)):\n img_batch = data_loader[k] # np.uint8 (N, H, W, C) format; not timed\n tic = time.time()\n img_batch = normalize(img_batch)\n y_pred = model(img_batch) # includes ensemble, TTA\n submission_batch = postprocess(y_pred)\n submission.append(submission_batch)\n inference_time += (time.time() - tic)\n\ntic = time.time()\nsubmission.to_csv('submission.csv', index=False)\ninference_time += (time.time() - tic)\nprint('Inference Time: %0.2f Minutes'%((inference_time) / 60))\n@See-- Yes, this is correct. You can exclude the loading of the images from the inference time.
", "rawMarkdown": "See-- Yes, this is correct. You can exclude the loading of the images from the inference time.", "votes": null }, { "id": "429080", "postDate": "11/28/2018 10:02:37", "content": "Hi to all participants in the Algorithm Speed Prize,\nThe deadline for submitting your kernel (i.e. sharing them with @jeffaudi and @inversion) is this Friday, November 30th (11:59 PM UTC). The current best time is under 10 minutes on a GPU kernel. \nGood luck,\nJeff.
", "rawMarkdown": "Hi to all participants in the Algorithm Speed Prize,\nThe deadline for submitting your kernel (i.e. sharing them with @jeffaudi and @inversion) is this Friday, November 30th (11:59 PM UTC). The current best time is under 10 minutes on a GPU kernel. \nGood luck,\nJeff.", "votes": null }, { "id": "429712", "postDate": "11/29/2018 08:27:29", "content": "Hello @jeffaudi and @inversion!\nIs it safe to assume that you will double-check the code for time calculations inside of the kernel?\nSome participants might forget to exclude image reading time.\nOthers might have unintentional mistakes in time related code.
", "rawMarkdown": "Hello @jeffaudi and @inversion!\nIs it safe to assume that you will double-check the code for time calculations inside of the kernel?\nSome participants might forget to exclude image reading time.\nOthers might have unintentional mistakes in time related code.", "votes": null }, { "id": "430020", "postDate": "11/29/2018 17:00:22", "content": "Hello,\nOh yes, we will carefully review this. We will start from the fastest kernel: check the code, re-run the kernel multiple times, verify the score, etc...\nKind regards,\nJeff.
", "rawMarkdown": "Hello,\nOh yes, we will carefully review this. We will start from the fastest kernel: check the code, re-run the kernel multiple times, verify the score, etc...\nKind regards,\nJeff.", "votes": null }, { "id": "430029", "postDate": "11/29/2018 17:35:43", "content": "this is the time for predicting 15k images
", "rawMarkdown": "this is the time for predicting 15k images", "votes": null }, { "id": "430075", "postDate": "11/29/2018 19:35:55", "content": "Impressive time... A couple of questions. Is this time measured only on model prediction? No loading of images and stuff.. The other thing I noticed is a large runtime fluctuation of the same kernel. The order of 5-10 minutes from run to run. Are you going to benchmark the codes on special \"kernels\" or on the common ones prone to this fluctuation? Thank you once more for your great efforts...
", "rawMarkdown": "Impressive time... A couple of questions. Is this time measured only on model prediction? No loading of images and stuff.. The other thing I noticed is a large runtime fluctuation of the same kernel. The order of 5-10 minutes from run to run. Are you going to benchmark the codes on special \"kernels\" or on the common ones prone to this fluctuation? Thank you once more for your great efforts...", "votes": null }, { "id": "430481", "postDate": "11/30/2018 13:06:27", "content": "Hi,\nour pipeline inference time on Kaggle kernel increased x6+ times compared to the same pipe running on 1080ti but around x3 was expected. Any legal known methods to effectively load Tesla K80? :)
", "rawMarkdown": "Hi,\nour pipeline inference time on Kaggle kernel increased x6+ times compared to the same pipe running on 1080ti but around x3 was expected. Any legal known methods to effectively load Tesla K80? :)", "votes": null }, { "id": "430578", "postDate": "11/30/2018 16:17:28", "content": "If you used half precision, which is not supported by K80, the speed reduction should be ~x6.
", "rawMarkdown": "If you used half precision, which is not supported by K80, the speed reduction should be ~x6.", "votes": null }, { "id": "430579", "postDate": "11/30/2018 16:25:03", "content": "Which is not supported by K80 :)
", "rawMarkdown": "Which is not supported by K80 :)", "votes": null }, { "id": "430643", "postDate": "11/30/2018 19:18:38", "content": "@voglinio I didn't observe that big changes. Runs are always between +-10 seconds.
", "rawMarkdown": "voglinio I didn't observe that big changes. Runs are always between +-10 seconds.", "votes": null }, { "id": "430663", "postDate": "11/30/2018 20:03:44", "content": "@See-- today things were clearly more stable.. Maybe competitions were ending yesterday. One more thing. I 've managed to run this greate examples\nhttps://github.com/jeng1220/KerasToTensorRT\non a kaggle kernel (from tensorflow.contrib import tensorrt as tftrt).\n I didn't have the time to see some substantial gain. Keep it in mind for other cases....
", "rawMarkdown": "See-- today things were clearly more stable.. Maybe competitions were ending yesterday. One more thing. I 've managed to run this greate examples\nhttps://github.com/jeng1220/KerasToTensorRT\non a kaggle kernel (from tensorflow.contrib import tensorrt as tftrt).\n I didn't have the time to see some substantial gain. Keep it in mind for other cases....", "votes": null }, { "id": "430716", "postDate": "11/30/2018 21:58:33", "content": "\n\nCould we get some confirmation that our kernel was accepted? Thanks in advance.
", "rawMarkdown": "Hi @inversion @jeffaudi\n\nCould we get some confirmation that our kernel was accepted? Thanks in advance.", "votes": null }, { "id": "431437", "postDate": "12/02/2018 09:15:34", "content": "Thank you very much for participating in the Algorithm Speed Prize! \nI am sorry but I have been unable to post on the discussion forum yesterday for some unknown reason. This seems to be OK now. I am currently verifying the kernels and compiling the results. I will notify every team that I have recorded their submission. \nKind regards,\nJeff.
", "rawMarkdown": "Thank you very much for participating in the Algorithm Speed Prize! \nI am sorry but I have been unable to post on the discussion forum yesterday for some unknown reason. This seems to be OK now. I am currently verifying the kernels and compiling the results. I will notify every team that I have recorded their submission. \nKind regards,\nJeff.", "votes": null }, { "id": "431927", "postDate": "12/03/2018 05:26:24", "content": "Submitted a couple of kernels on Friday. Did not see that they were accepted.
", "rawMarkdown": "Submitted a couple of kernels on Friday. Did not see that they were accepted.", "votes": null }, { "id": "437287", "postDate": "12/11/2018 17:18:31", "content": "Hello to everyone !
\n\nThank you so much for participating in the Algorithm Speed Prize ! This was somewhat unusual to have a follow-on challenge but we wanted it because inference speed is almost as important for us as accuracy. We need to extract ships from a full satellite image (i.e. roughly 4,500 of this competition smaller images) in a few minutes. With the below results, we are confident that we will be able to provide both accuracy and speed !
\n\nHere are the results from the Algorithm Speed Prize :\n\n(*updated on 18-DEC-2018)
Congratulations ! The first prize goes to @ddanevskyi and his team ! Kudos also to @seesee, to @marvelousninja and to @iafoss, you have done very well. And thank you to all of the participants !! I hope that you have learned a few things about putting algorithms into production during this Special Prize :) Do not hesitate to go back to me with questions or suggestions about this challenge or anything related to applying machine learning to satellite imagery.
\n\nKind regards,
\n\nJeff
", "rawMarkdown": "Hello to everyone !\n\nThank you so much for participating in the Algorithm Speed Prize ! This was somewhat unusual to have a follow-on challenge but we wanted it because inference speed is almost as important for us as accuracy. We need to extract ships from a full satellite image (i.e. roughly 4,500 of this competition smaller images) in a few minutes. With the below results, we are confident that we will be able to provide both accuracy and speed !\n\nHere are the results from the Algorithm Speed Prize :\n\n(*updated on 18-DEC-2018) \n\nCongratulations ! The first prize goes to @ddanevskyi and his team ! Kudos also to @seesee, to @marvelousninja and to @iafoss, you have done very well. And thank you to all of the participants !! I hope that you have learned a few things about putting algorithms into production during this Special Prize :) Do not hesitate to go back to me with questions or suggestions about this challenge or anything related to applying machine learning to satellite imagery. \n\nKind regards,\n\nJeff", "votes": null }, { "id": "437301", "postDate": "12/11/2018 17:35:37", "content": "Hi,
\n\nA number of comments here.\n- Did you do any sort of dockerization / hardware standardization when verifying the above results? In my runs run time within kernels varied +/- 25%, not that it matters;\n- I shared 2 additional kernels on the last day, which you ignored - my comment above also was ignored. One of these kernels explicitly showed that RAM => GPU transfer was a culprit for slow performance. I actually was able to get ~5.5 mins on my local hardware (1 GPU with full saturation). So this kind of discrepancy kind of raises the question - may be you can get a better result by exploiting some perks of kernels?;\n- 5+ minutes definitely seems doable, but on normal hardware in controlled conditions. So - the usage of kernels for this part of competitions 100% negates the purpose of the competition and (arguably) favours usage of some frameworks / optimizations;
\n\nAlso it is very interesting how 3+ minutes can be achieved.\nDid you check that the kernel contained all the preprocessing steps and the time measured was true inference time?
\n\nI actually distilled my best results into 2 networks (classifier and a small semseg model) - but it allowed me to achieve only around ~5 minutes on decent hardware.
", "rawMarkdown": "Hi,\n\n**A number of comments here.**\n- Did you do any sort of dockerization / hardware standardization when verifying the above results? In my runs run time within kernels varied +/- 25%, not that it matters;\n- I shared 2 additional kernels on the last day, which you ignored - my comment above also was ignored. One of these kernels explicitly showed that RAM => GPU transfer was a culprit for slow performance. I actually was able to get ~5.5 mins on my local hardware (1 GPU with full saturation). So this kind of discrepancy kind of raises the question - may be you can get a better result by exploiting some perks of kernels?;\n- 5+ minutes definitely seems doable, but on normal hardware in controlled conditions. So - the usage of kernels for this part of competitions 100% negates the purpose of the competition and (arguably) favours usage of some frameworks / optimizations;\n\n\nAlso it is very interesting how 3+ minutes can be achieved.\nDid you check that the kernel contained all the preprocessing steps and the time measured was true inference time?\n\nI actually distilled my best results into 2 networks (classifier and a small semseg model) - but it allowed me to achieve only around ~5 minutes on decent hardware.", "votes": null }, { "id": "437305", "postDate": "12/11/2018 17:38:06", "content": "Our team [attenion heads] also submitted a kernel, but I don't see our score in the results table.
\n\n@jeffaudi, can you elaborate on this please?
", "rawMarkdown": "Our team [attenion heads] also submitted a kernel, but I don't see our score in the results table.\n\n@jeffaudi, can you elaborate on this please?", "votes": null }, { "id": "437334", "postDate": "12/11/2018 18:18:04", "content": "Congratulations @marvelousninja . Please share some details about the winning approach. Simply loading the images (with opencv, single thread) takes ~4 minutes. 3.38 minutes seems like a miracle.
\n\n> \"may be you can get a better result by exploiting some perks of kernels?;\"
\n\nI guess so. For me, the kernel was always ~6x slower than the local pipeline.
", "rawMarkdown": "Congratulations @marvelousninja . Please share some details about the winning approach. Simply loading the images (with opencv, single thread) takes ~4 minutes. 3.38 minutes seems like a miracle.\n\n> \"may be you can get a better result by exploiting some perks of kernels?;\" \n\nI guess so. For me, the kernel was always ~6x slower than the local pipeline.", "votes": null }, { "id": "437336", "postDate": "12/11/2018 18:21:08", "content": "Hi Alexander,\nSorry for not collecting your last kernel - I will update your time to 22.56 min in the results. You are right that dockerization of the algorithms is the way to go for production of machine learning models. But in the frame of this competition, we decided to go for kernels because it would be easier to compile the results and check the inference time computation. We know that this comes with some limitations but globally it was a good choice for the speed prize. Sorry if this created some frustration over the kernels GPU architecture.\nTo answer your other questions, @inversion and myself checked that the inference time was correctly computed and we did not notice anything wrong. I also relaunched the kernels a number of time and did observed some different timing but nothing that would swap the order or at least change the winner. As for the way to achieve 3+ minutes, maybe Alexandre will want to share some tips and tricks on this discussion !\nKind regards,\nJeff.
", "rawMarkdown": "Hi Alexander,\nSorry for not collecting your last kernel - I will update your time to 22.56 min in the results. You are right that dockerization of the algorithms is the way to go for production of machine learning models. But in the frame of this competition, we decided to go for kernels because it would be easier to compile the results and check the inference time computation. We know that this comes with some limitations but globally it was a good choice for the speed prize. Sorry if this created some frustration over the kernels GPU architecture.\nTo answer your other questions, @inversion and myself checked that the inference time was correctly computed and we did not notice anything wrong. I also relaunched the kernels a number of time and did observed some different timing but nothing that would swap the order or at least change the winner. As for the way to achieve 3+ minutes, maybe Alexandre will want to share some tips and tricks on this discussion !\nKind regards,\nJeff.", "votes": null }, { "id": "437338", "postDate": "12/11/2018 18:24:40", "content": "Hi Dmitriy,\nThank you for submitting your kernel. You did not add the inference time and the score in the comments. By looking into the code, I could see \"Total time: 25727069.01 minutes.\" So I added a comment to the kernel to get your feedback on this figure because it seemed like an error. I got no answer so I kept your results out of the final list. \nKind regards,\nJeff.
", "rawMarkdown": "Hi Dmitriy,\nThank you for submitting your kernel. You did not add the inference time and the score in the comments. By looking into the code, I could see \"Total time: 25727069.01 minutes.\" So I added a comment to the kernel to get your feedback on this figure because it seemed like an error. I got no answer so I kept your results out of the final list. \nKind regards,\nJeff.", "votes": null }, { "id": "437346", "postDate": "12/11/2018 18:46:25", "content": "You might be looking on some erroneous kernel we submitted on the last minutes before the deadline. We certainly did submit proper version with time and score in comments.
", "rawMarkdown": "You might be looking on some erroneous kernel we submitted on the last minutes before the deadline. We certainly did submit proper version with time and score in comments.", "votes": null }, { "id": "437504", "postDate": "12/12/2018 03:35:59", "content": "\n\nHi Jeff,
\n\nTalked with @marvelousninja\nLooks like he also used 2 stage model (classifier + semseg, but he used LinkNet).\nUsing LinkNet would probably boost my local time to 4 ~mins, though I was skeptical about cutting the resolution drastically to 224.
\n\nAnyway my point though is about:\n- By pushing the community to use Kernels - you have only a handful of submissions, and I guess half of people stopped optimizing their pipeline on seeing how kernels perform;\n- By not counting IO time (because let's be honest - Kernels just suck) - you do no alleviate a problem, but essentially create a motivation towards solutions that half-ass the way around this, i.e. loading all images into RAM in advance - but in real life such solutions are not viable;\n- In the long term - all of this just diverts community from Kaggle;
\n\nThe only way to asses the true merit of the whole pipeline is to run it end-to-end, IO, preparation and all on a dedicated standard machine.
", "rawMarkdown": "jeffaudi\n\nHi Jeff,\n\nTalked with @marvelousninja\nLooks like he also used 2 stage model (classifier + semseg, but he used LinkNet).\nUsing LinkNet would probably boost my local time to 4 ~mins, though I was skeptical about cutting the resolution drastically to 224.\n\nAnyway my point though is about:\n- By pushing the community to use Kernels - you have only a handful of submissions, and I guess half of people stopped optimizing their pipeline on seeing how kernels perform;\n- By not counting IO time (because let's be honest - Kernels just suck) - you do no alleviate a problem, but essentially create a motivation towards solutions that half-ass the way around this, i.e. loading all images into RAM in advance - but in real life such solutions are not viable;\n- In the long term - all of this just diverts community from Kaggle;\n\nThe only way to asses the true merit of the whole pipeline is to run it end-to-end, IO, preparation and all on a dedicated standard machine.", "votes": null }, { "id": "437644", "postDate": "12/12/2018 08:55:27", "content": "Hello from team Nodalpoints (Ouranos and me). It seems the results correspond to timings and score on the 15,000 image test set. For example our timing is the same like the one we have reported on the comment section of our kernel. \nIf possible we would like to have the updated metrics (time /score) on the new images that this mini competition was based on. I believe that this will not affect the ranking but it may reduce the differences. For me going below 5 minutes for 15,000 images for classification/segmentation is blazing fast
", "rawMarkdown": "Hello from team Nodalpoints (Ouranos and me). It seems the results correspond to timings and score on the 15,000 image test set. For example our timing is the same like the one we have reported on the comment section of our kernel. \nIf possible we would like to have the updated metrics (time /score) on the new images that this mini competition was based on. I believe that this will not affect the ranking but it may reduce the differences. For me going below 5 minutes for 15,000 images for classification/segmentation is blazing fast", "votes": null }, { "id": "437649", "postDate": "12/12/2018 09:11:44", "content": "What is the running time of the whole kernel? It is ~6 minutes for me. Running an image processing algorithm (preprocessing, copying data to the GPU, launching kernels, post-processing) is faster than reading an image (SSD) & jpeg decoding. Even if the resolution is reduced to 224, this seems odd. I am happy to learn more about how this is possible.
", "rawMarkdown": "What is the running time of the whole kernel? It is ~6 minutes for me. Running an image processing algorithm (preprocessing, copying data to the GPU, launching kernels, post-processing) is faster than reading an image (SSD) & jpeg decoding. Even if the resolution is reduced to 224, this seems odd. I am happy to learn more about how this is possible.", "votes": null }, { "id": "437669", "postDate": "12/12/2018 09:45:22", "content": "@See-- , overall time was around 10 minutes. It breaks down to:
\n\nI also created a separate thread with a more detailed description:\nhttps://www.kaggle.com/c/airbus-ship-detection/discussion/74443
", "rawMarkdown": "See-- , overall time was around 10 minutes. It breaks down to:\n\n* around 3 minutes to install all pip packages\n* around 4 minutes on image I/O\n* around 3 minutes for actual inference\n\nI also created a separate thread with a more detailed description:\nhttps://www.kaggle.com/c/airbus-ship-detection/discussion/74443", "votes": null }, { "id": "437703", "postDate": "12/12/2018 10:38:58", "content": "Looks like you've missed our kernel https://www.kaggle.com/ddanevskyi/speedprize-attention-heads (shared with @inversion , @jeffaudi ) with next comment added: \"Inference time (excluding time for loading images): 4.69 minutes\nPrivate leaderboard score: 0.84509 (79 place)\" from Nov 30. Doesnt change the winner but still we spent time and that's sad :(
", "rawMarkdown": "Looks like you've missed our kernel https://www.kaggle.com/ddanevskyi/speedprize-attention-heads (shared with @inversion , @jeffaudi ) with next comment added: \"Inference time (excluding time for loading images): 4.69 minutes\nPrivate leaderboard score: 0.84509 (79 place)\" from Nov 30. Doesnt change the winner but still we spent time and that's sad :(", "votes": null }, { "id": "437705", "postDate": "12/12/2018 10:40:23", "content": "Yes. Got it! Strangely enough, I did not received a sharing email notification about your other kernel and could not see it. But it is solved now. I will update the results accordingly. Sorry for this. Jeff.
", "rawMarkdown": "Yes. Got it! Strangely enough, I did not received a sharing email notification about your other kernel and could not see it. But it is solved now. I will update the results accordingly. Sorry for this. Jeff.", "votes": null }, { "id": "437706", "postDate": "12/12/2018 10:41:31", "content": "I am correcting this right now. Good job !
", "rawMarkdown": "I am correcting this right now. Good job !", "votes": null }, { "id": "437708", "postDate": "12/12/2018 10:46:14", "content": "Thanks a lot!
", "rawMarkdown": "Thanks a lot!", "votes": null }, { "id": "437877", "postDate": "12/12/2018 17:18:17", "content": "Hello again from Nodalpoints! Can you please re-create the result table using the new images (and please report how many they are). We would also like to know the percentage of empty to non empty ships since it can also affect performance. I am not asking out of plain curiocity but out of professional interest for the business problem.
", "rawMarkdown": "Hello again from Nodalpoints! Can you please re-create the result table using the new images (and please report how many they are). We would also like to know the percentage of empty to non empty ships since it can also affect performance. I am not asking out of plain curiocity but out of professional interest for the business problem.", "votes": null }, { "id": "437886", "postDate": "12/12/2018 17:35:58", "content": ":heavy_plus_sign: I'd like to see this as well. If possible, please also add the total running time. The whole 'classifier + segmentation' approach relies on a big fraction of 'empty' images to be fast. It would be interesting to see the timings/scores for the new validation set.
\n\nI know that this requires some work, but you and we will learn a lot :)
", "rawMarkdown": ":heavy_plus_sign: I'd like to see this as well. If possible, please also add the total running time. The whole 'classifier + segmentation' approach relies on a big fraction of 'empty' images to be fast. It would be interesting to see the timings/scores for the new validation set.\n\nI know that this requires some work, but you and we will learn a lot :)", "votes": null }, { "id": "437891", "postDate": "12/12/2018 17:49:27", "content": "@jeffaudi @inversion Could please give an update to the current situation? Are these the final results? I saw you updated the results table. It is impossible to reproduce the 30 seconds classifier claimed in the winner thread. This is really disappointing. I thought you were looking into this.
", "rawMarkdown": "jeffaudi @inversion Could please give an update to the current situation? Are these the final results? I saw you updated the results table. It is impossible to reproduce the 30 seconds classifier claimed in the winner thread. This is really disappointing. I thought you were looking into this.", "votes": null }, { "id": "439007", "postDate": "12/14/2018 15:25:09", "content": "@seesee I will see if I can output this on the four top kernels.
", "rawMarkdown": "seesee I will see if I can output this on the four top kernels.", "votes": null }, { "id": "439181", "postDate": "12/14/2018 21:01:41", "content": "Sorry this is the last time I write for this subject. Is there a reason that you cannot output timings? After all timing was the issue of this mini competition. I would expect that you already compiled a simple Excel sheet that records times and score... And please report the scores on the new datasets as well. If you find it difficult just publish the new dataset with the ground truth and let us measure our performance our selves.
\n\nLast notice: such kind of speed tests are more than welcome for future competitions. Thank you for your effort to organize one. However you need to think over to the evaluation process... For example it is difficult to compare time measurements by timing blocks set by the contestants themselves... A small mistake or a forgotten block might favor one team against the others. Kaggle is famous for its bulletproof evaluation process.. this undisputable fairness is what brings the greatest added value to the competitions.
\n\nKeep up the good work! Over and out
", "rawMarkdown": "Sorry this is the last time I write for this subject. Is there a reason that you cannot output timings? After all timing was the issue of this mini competition. I would expect that you already compiled a simple Excel sheet that records times and score... And please report the scores on the new datasets as well. If you find it difficult just publish the new dataset with the ground truth and let us measure our performance our selves.\n\nLast notice: such kind of speed tests are more than welcome for future competitions. Thank you for your effort to organize one. However you need to think over to the evaluation process... For example it is difficult to compare time measurements by timing blocks set by the contestants themselves... A small mistake or a forgotten block might favor one team against the others. Kaggle is famous for its bulletproof evaluation process.. this undisputable fairness is what brings the greatest added value to the competitions.\n\nKeep up the good work! Over and out", "votes": null }, { "id": "441042", "postDate": "12/18/2018 07:49:15", "content": "\n\nAny chance we will be updated on the situation soon?
\n\nRegards,
\n\nDmitriy
", "rawMarkdown": "jeffaudi @inversion \n\nAny chance we will be updated on the situation soon?\n\nRegards,\n\nDmitriy", "votes": null }, { "id": "441109", "postDate": "12/18/2018 09:17:52", "content": "Yes, sorry that it took so long. I will update the results today.
", "rawMarkdown": "Yes, sorry that it took so long. I will update the results today.", "votes": null }, { "id": "441122", "postDate": "12/18/2018 09:31:54", "content": "Hi to everyone,
\n\nKaggle is a place to learn and share about data science. At the end of the day, we are all here to make experiments and learn from our experience. This also applies to sponsors.
\n\nWith the main Ship Detection competition, we learned that we could improve our internal results substantially but also that even large ships could still be missed in harbours or at sea on some occasions. We pushed to organize the Speed Prize because we wanted to make sure that the best models could also process our images fast enough. By deciding to measure only computation time and not loading time, I have made the rules aligned with our real production environment but I have also created extra difficulty to review the solutions. Although we have carefully rerun the kernels and reviewed the code to check that inference time was correctly measured, we were not able to spot the issue with pyTorch and asynchronism in @marvelousninja kernel immediately. This has been discovered and revealed by himself after the initial publication of the results. I am grateful to @marvelousninja for his transparency and efforts to correct the issue but together with Kaggle team, we decided that we cannot accept a new kernel after the deadline. I have updated the score accordingly and the winner team name.
\n\nOnce again, I want to thank all participants for their participation in this new type of challenge. We have a lot of lessons learned on this competition and we will be looking toward great new challenges next year !
\n\nHappy holiday season to all of you,
\n\nJeff.
", "rawMarkdown": "Hi to everyone, \n\nKaggle is a place to learn and share about data science. At the end of the day, we are all here to make experiments and learn from our experience. This also applies to sponsors. \n\nWith the main Ship Detection competition, we learned that we could improve our internal results substantially but also that even large ships could still be missed in harbours or at sea on some occasions. We pushed to organize the Speed Prize because we wanted to make sure that the best models could also process our images fast enough. By deciding to measure only computation time and not loading time, I have made the rules aligned with our real production environment but I have also created extra difficulty to review the solutions. Although we have carefully rerun the kernels and reviewed the code to check that inference time was correctly measured, we were not able to spot the issue with pyTorch and asynchronism in @marvelousninja kernel immediately. This has been discovered and revealed by himself after the initial publication of the results. I am grateful to @marvelousninja for his transparency and efforts to correct the issue but together with Kaggle team, we decided that we cannot accept a new kernel after the deadline. I have updated the score accordingly and the winner team name.\n\nOnce again, I want to thank all participants for their participation in this new type of challenge. We have a lot of lessons learned on this competition and we will be looking toward great new challenges next year ! \n\nHappy holiday season to all of you,\n\nJeff.", "votes": null }, { "id": "441134", "postDate": "12/18/2018 09:58:50", "content": "Hi Costas,\nThank you for you message and your support. It has been an interesting ride for us at Airbus - we are usually a more closed environment. We have heard the community comments about need for \"bulletproof evaluation process\" and amended the results accordingly. A lot of lessons learned through this competition - we will definitely take that into account in the future. I can assure you that we are very motivated by the results and very thankful to the whole Kaggle community. \nKind regards, Jeff.
", "rawMarkdown": "Hi Costas,\nThank you for you message and your support. It has been an interesting ride for us at Airbus - we are usually a more closed environment. We have heard the community comments about need for \"bulletproof evaluation process\" and amended the results accordingly. A lot of lessons learned through this competition - we will definitely take that into account in the future. I can assure you that we are very motivated by the results and very thankful to the whole Kaggle community. \nKind regards, Jeff.", "votes": null }, { "id": "441343", "postDate": "12/18/2018 15:09:59", "content": "Wow, this is absolutely amazing, thank you!
", "rawMarkdown": "Wow, this is absolutely amazing, thank you!", "votes": null } ], "comments": [ { "id": 409841, "author_name": "scherzo", "author_url": "", "post_date": "10/24/2018 23:30:45", "content": "Hi inversion, thanks for posting more details about the speed prize.\nTwo points came to my mind:
\n\nThank you in advance,\nRenan
", "votes": null, "replies": [ { "id": 411701, "author_name": "jeffaudi", "author_url": "", "post_date": "10/28/2018 20:17:35", "content": "Hi Renan,
\n\nThank you for your questions. The validation dataset is the test dataset of the main competition (called test_v2). The speed prize will be open to teams that score in top 100 of the private leaderboard. Let's call S the score of the 100th participant on the private leaderboard on the closing day of the main competition. We consider that the score of the new speed kernel on the private leaderboard should not go below S. As long as this is verified, you can use any model design that you want.
\n\nKind regards,
\n\nJeff
", "votes": null, "replies": [] }, { "id": 411718, "author_name": "scherzo", "author_url": "", "post_date": "10/28/2018 21:52:42", "content": "That's very clear. Thank you Jeff!
", "votes": null, "replies": [] } ] }, { "id": 410640, "author_name": "dempton", "author_url": "", "post_date": "10/26/2018 11:40:19", "content": "Are there any conditions about quality of the models (minimum of the score, for example)?
\n\nP.S. I can submit empty masks very fast :D
", "votes": null, "replies": [] }, { "id": 416929, "author_name": "petersorensen360", "author_url": "", "post_date": "11/07/2018 13:01:26", "content": "Can we form new teams for the speed prize competition?
", "votes": null, "replies": [] }, { "id": 419105, "author_name": "", "author_url": "", "post_date": "11/11/2018 09:20:11", "content": "What is the minimum system requirement for this algorithm speed challenge? like for example in Human Protein Atlas Image Classification challenge, minimum hardware limits are: CPU Cores: 2, RAM: 4GB and GPUs: Integrated Intel Graphics.. What about threshold scoring requirement for algorithm? what is the baseline score? thanks!
", "votes": null, "replies": [ { "id": 424740, "author_name": "jeffaudi", "author_url": "", "post_date": "11/20/2018 15:50:24", "content": "Hello,\nYou should use kernels (with GPU) for the speed prize. And the minimum score is 0.84416\nKind regards,\nJeff
", "votes": null, "replies": [] } ] }, { "id": 421401, "author_name": "frappuccino", "author_url": "", "post_date": "11/15/2018 00:44:00", "content": "Will there be a leaderboard for speed prize ?
", "votes": null, "replies": [] }, { "id": 421901, "author_name": "jaideepvalani", "author_url": "", "post_date": "11/15/2018 14:50:18", "content": "The Kernel will be validated against a new dataset.\nfrom where we get the new data set...
", "votes": null, "replies": [ { "id": 423461, "author_name": "seesee", "author_url": "", "post_date": "11/18/2018 10:00:14", "content": "@jaideepvalani My guess is you can't. This prevents cheating (e.g. precompute the predictions and then just upload them.
", "votes": null, "replies": [] }, { "id": 423555, "author_name": "jaideepvalani", "author_url": "", "post_date": "11/18/2018 14:23:53", "content": "I meant in the rulesfor algo speed prize it was mentioned that new data set is going to be used.. i was needing a clarification on that..
", "votes": null, "replies": [] } ] }, { "id": 423961, "author_name": "voglinio", "author_url": "", "post_date": "11/19/2018 10:29:29", "content": "Hello everybody! Regarding the speed test, should we add the model creation and weight loading time to the overall inference time?
", "votes": null, "replies": [ { "id": 424525, "author_name": "jeffaudi", "author_url": "", "post_date": "11/20/2018 09:09:33", "content": "Hi Costas,
\n\nThanks for participating in the Algorithm Speed Prize! We expect the inference time to be only the computation time on imagery (including pre-processing, models computation and post-processing) without loading of weights and models. The rational is to think of a running docker container which would wait for imagery buffer to compute ships masks.
\n\nKind regards,
\n\nJeff.
", "votes": null, "replies": [] } ] }, { "id": 424436, "author_name": "iafoss", "author_url": "", "post_date": "11/20/2018 05:56:17", "content": "Could you, please, clarify if inference time includes the time for loading and preprocessing the data? Also, if there any way to compare the speed of my kernel with others before the submission deadline? In particular if I get, for example, 10 minutes or 1 hour, is it fast or slow?
", "votes": null, "replies": [ { "id": 425219, "author_name": "jeffaudi", "author_url": "", "post_date": "11/21/2018 09:44:31", "content": "Hi iafoss,
\n\nThank you so much for participating in the Algorithm Speed Prize, especially after all the good stuff that you already shared with the community in Kernels and Discussions. Your questions are very legitimate and we are looking into ways of sharing a kind of leaderboard with the participants. I will get back to all of you in this discussion thread as soon as possible. For the inference time, I would say that fast is closer to 10 minutes than to 1 hour :)
\n\nKind regards,\nJeff.
", "votes": null, "replies": [] }, { "id": 425516, "author_name": "iafoss", "author_url": "", "post_date": "11/21/2018 18:07:11", "content": "Jeff Faudi, thank you so much for working on it. It is really important to know how the model performs in comparison with others. My current fastest kernel takes 13 minutes, and I would really like to compare it with results of other people before the deadline.
", "votes": null, "replies": [] } ] }, { "id": 424524, "author_name": "seesee", "author_url": "", "post_date": "11/20/2018 09:01:23", "content": "@inversion Does ones entry just have to create a better submission than the current place 100: 0.84416 or is there a separate set on which our approach will be evaluated?
Hi See,
\n\nThanks for participating in the Algorithm Speed Prize! You are right. The kernel should create a submission against the current test_v2 dataset and rank better than 0.84416. You should create a comment at the end of your kernel with the score on the private LB and the inference time. We will run inference on a new test dataset just to make sure that the model still runs correctly on new imagery.
\n\nKind regards,
\n\nJeff.
", "votes": null, "replies": [] }, { "id": 424688, "author_name": "voglinio", "author_url": "", "post_date": "11/20/2018 14:32:47", "content": "Thank you Jeff for your replies! So the way I understand it is that in the new test dataset you will not score the model. You will just measure the inference time! So the 0.84416 will define all eligible participants and then it is just a question of speed. I guess the private score could be used to break ties or something. Am I correct?
", "votes": null, "replies": [] }, { "id": 424750, "author_name": "jeffaudi", "author_url": "", "post_date": "11/20/2018 16:01:19", "content": "Yes. I will measure speed and score on test_v2. Then I will rate participants on fastest speed given that score >= 0.84416. I will use another new test dataset to measure inference and score again and verify if there is no discrepencies in the ranking (i.e. the final algorithm should perform as well on a new dataset because that is what is really important for us).
", "votes": null, "replies": [] }, { "id": 424816, "author_name": "jaideepvalani", "author_url": "", "post_date": "11/20/2018 17:48:32", "content": "Hi jeff \nthanks for replies\nI m new to kaggle competitions ,that is this first one...\n ,could u please ans below queries\n1) we only calculating here the prediction time ,so that is how long does the model takes in predicting the testv2?\n2) from where does my time starts... to predict i will have to rebuild the model object,load the trained weights finally start predicting...\n3) if i resubmit then my rank could improve also ,so new rank will be considered or first time private lb rank be considered..
", "votes": null, "replies": [] }, { "id": 425507, "author_name": "jeffaudi", "author_url": "", "post_date": "11/21/2018 17:44:16", "content": "Hi Jaideep,\nTo answer your questions :\n1.- Yes, inference time is how much the model(s) need to predict all images in test_v2\n2.- Yes, the inference time does not include setup the model, load the weights, load the images, etc... but includes pre-processing (i.e. rescaling), model prediction (with i.e. ensembling and TTA) and post processing (i.e. removing the overlaps, creating the CSV file). Everything that stays the same whatever the image should not be counted, everything that needs to be recomputed with new images needs to be included.\n3.- Yes, of course. You can submit multiple times and only your best score will count.\nKind regards,\nJeff.
", "votes": null, "replies": [] }, { "id": 425518, "author_name": "iafoss", "author_url": "", "post_date": "11/21/2018 18:11:09", "content": "Jeff Faudi, thank you for clarification on what must be included in the inference time.
", "votes": null, "replies": [] }, { "id": 427373, "author_name": "seesee", "author_url": "", "post_date": "11/25/2018 11:43:28", "content": "Hi Jeff,
\n\nYes, the inference time does not include setup the model, load the weights, load the images, etc…\nI understand your rationale. Though, for me it is not clear what you mean by excluding the image loading (reading the image string & jpeg decoding). Could you provide a modified template? I have been using the one provided by @inversion (competition tab), without the model loading.
\n\nmodel = load_model()\nimport time\ninference_start = time.time()\n###### inference code #######\ninference_end = time.time()\nprint('Inference Time: %0.2f Minutes'%((inference_end - inference_start)/60))\nEvery framework has different loaders. I am using tensorflow. Reading the image string and decoding it is part of the graph ~~and it is hard to untangle these timings~~ (see below).
", "votes": null, "replies": [] }, { "id": 427396, "author_name": "seesee", "author_url": "", "post_date": "11/25/2018 12:40:38", "content": "To make it more concrete. We can exclude the time to load an image batch in RGB format as np.uint8?
\n\nmodel = load_model()\nimport time\ninference_time = 0.\nsubmission = pd.DataFrame()\n###### inference code #######\nfor k in range(len(data_loader)):\n img_batch = data_loader[k] # np.uint8 (N, H, W, C) format; not timed\n tic = time.time()\n img_batch = normalize(img_batch)\n y_pred = model(img_batch) # includes ensemble, TTA\n submission_batch = postprocess(y_pred)\n submission.append(submission_batch)\n inference_time += (time.time() - tic)\n\ntic = time.time()\nsubmission.to_csv('submission.csv', index=False)\ninference_time += (time.time() - tic)\nprint('Inference Time: %0.2f Minutes'%((inference_time) / 60))\n@See-- Yes, this is correct. You can exclude the loading of the images from the inference time.
", "votes": null, "replies": [] }, { "id": 429712, "author_name": "marvelousninja", "author_url": "", "post_date": "11/29/2018 08:27:29", "content": "Hello @jeffaudi and @inversion!\nIs it safe to assume that you will double-check the code for time calculations inside of the kernel?\nSome participants might forget to exclude image reading time.\nOthers might have unintentional mistakes in time related code.
", "votes": null, "replies": [] }, { "id": 430020, "author_name": "jeffaudi", "author_url": "", "post_date": "11/29/2018 17:00:22", "content": "Hello,\nOh yes, we will carefully review this. We will start from the fastest kernel: check the code, re-run the kernel multiple times, verify the score, etc...\nKind regards,\nJeff.
", "votes": null, "replies": [] } ] }, { "id": 429080, "author_name": "jeffaudi", "author_url": "", "post_date": "11/28/2018 10:02:37", "content": "Hi to all participants in the Algorithm Speed Prize,\nThe deadline for submitting your kernel (i.e. sharing them with @jeffaudi and @inversion) is this Friday, November 30th (11:59 PM UTC). The current best time is under 10 minutes on a GPU kernel. \nGood luck,\nJeff.
", "votes": null, "replies": [ { "id": 430029, "author_name": "jaideepvalani", "author_url": "", "post_date": "11/29/2018 17:35:43", "content": "this is the time for predicting 15k images
", "votes": null, "replies": [] }, { "id": 430075, "author_name": "voglinio", "author_url": "", "post_date": "11/29/2018 19:35:55", "content": "Impressive time... A couple of questions. Is this time measured only on model prediction? No loading of images and stuff.. The other thing I noticed is a large runtime fluctuation of the same kernel. The order of 5-10 minutes from run to run. Are you going to benchmark the codes on special \"kernels\" or on the common ones prone to this fluctuation? Thank you once more for your great efforts...
", "votes": null, "replies": [] } ] }, { "id": 430481, "author_name": "yaroshevskiy", "author_url": "", "post_date": "11/30/2018 13:06:27", "content": "Hi,\nour pipeline inference time on Kaggle kernel increased x6+ times compared to the same pipe running on 1080ti but around x3 was expected. Any legal known methods to effectively load Tesla K80? :)
", "votes": null, "replies": [ { "id": 430578, "author_name": "iafoss", "author_url": "", "post_date": "11/30/2018 16:17:28", "content": "If you used half precision, which is not supported by K80, the speed reduction should be ~x6.
", "votes": null, "replies": [] }, { "id": 430579, "author_name": "yaroshevskiy", "author_url": "", "post_date": "11/30/2018 16:25:03", "content": "Which is not supported by K80 :)
", "votes": null, "replies": [] }, { "id": 430643, "author_name": "seesee", "author_url": "", "post_date": "11/30/2018 19:18:38", "content": "@voglinio I didn't observe that big changes. Runs are always between +-10 seconds.
", "votes": null, "replies": [] }, { "id": 430663, "author_name": "voglinio", "author_url": "", "post_date": "11/30/2018 20:03:44", "content": "@See-- today things were clearly more stable.. Maybe competitions were ending yesterday. One more thing. I 've managed to run this greate examples\nhttps://github.com/jeng1220/KerasToTensorRT\non a kaggle kernel (from tensorflow.contrib import tensorrt as tftrt).\n I didn't have the time to see some substantial gain. Keep it in mind for other cases....
", "votes": null, "replies": [] } ] }, { "id": 430716, "author_name": "ddanevskyi", "author_url": "", "post_date": "11/30/2018 21:58:33", "content": "\n\nCould we get some confirmation that our kernel was accepted? Thanks in advance.
", "votes": null, "replies": [] }, { "id": 431437, "author_name": "jeffaudi", "author_url": "", "post_date": "12/02/2018 09:15:34", "content": "Thank you very much for participating in the Algorithm Speed Prize! \nI am sorry but I have been unable to post on the discussion forum yesterday for some unknown reason. This seems to be OK now. I am currently verifying the kernels and compiling the results. I will notify every team that I have recorded their submission. \nKind regards,\nJeff.
", "votes": null, "replies": [] }, { "id": 431927, "author_name": "snakers41", "author_url": "", "post_date": "12/03/2018 05:26:24", "content": "Submitted a couple of kernels on Friday. Did not see that they were accepted.
", "votes": null, "replies": [] }, { "id": 437287, "author_name": "jeffaudi", "author_url": "", "post_date": "12/11/2018 17:18:31", "content": "Hello to everyone !
\n\nThank you so much for participating in the Algorithm Speed Prize ! This was somewhat unusual to have a follow-on challenge but we wanted it because inference speed is almost as important for us as accuracy. We need to extract ships from a full satellite image (i.e. roughly 4,500 of this competition smaller images) in a few minutes. With the below results, we are confident that we will be able to provide both accuracy and speed !
\n\nHere are the results from the Algorithm Speed Prize :\n\n(*updated on 18-DEC-2018)
Congratulations ! The first prize goes to @ddanevskyi and his team ! Kudos also to @seesee, to @marvelousninja and to @iafoss, you have done very well. And thank you to all of the participants !! I hope that you have learned a few things about putting algorithms into production during this Special Prize :) Do not hesitate to go back to me with questions or suggestions about this challenge or anything related to applying machine learning to satellite imagery.
\n\nKind regards,
\n\nJeff
", "votes": null, "replies": [ { "id": 437301, "author_name": "snakers41", "author_url": "", "post_date": "12/11/2018 17:35:37", "content": "Hi,
\n\nA number of comments here.\n- Did you do any sort of dockerization / hardware standardization when verifying the above results? In my runs run time within kernels varied +/- 25%, not that it matters;\n- I shared 2 additional kernels on the last day, which you ignored - my comment above also was ignored. One of these kernels explicitly showed that RAM => GPU transfer was a culprit for slow performance. I actually was able to get ~5.5 mins on my local hardware (1 GPU with full saturation). So this kind of discrepancy kind of raises the question - may be you can get a better result by exploiting some perks of kernels?;\n- 5+ minutes definitely seems doable, but on normal hardware in controlled conditions. So - the usage of kernels for this part of competitions 100% negates the purpose of the competition and (arguably) favours usage of some frameworks / optimizations;
\n\nAlso it is very interesting how 3+ minutes can be achieved.\nDid you check that the kernel contained all the preprocessing steps and the time measured was true inference time?
\n\nI actually distilled my best results into 2 networks (classifier and a small semseg model) - but it allowed me to achieve only around ~5 minutes on decent hardware.
", "votes": null, "replies": [] }, { "id": 437334, "author_name": "seesee", "author_url": "", "post_date": "12/11/2018 18:18:04", "content": "Congratulations @marvelousninja . Please share some details about the winning approach. Simply loading the images (with opencv, single thread) takes ~4 minutes. 3.38 minutes seems like a miracle.
\n\n> \"may be you can get a better result by exploiting some perks of kernels?;\"
\n\nI guess so. For me, the kernel was always ~6x slower than the local pipeline.
", "votes": null, "replies": [] }, { "id": 437336, "author_name": "jeffaudi", "author_url": "", "post_date": "12/11/2018 18:21:08", "content": "Hi Alexander,\nSorry for not collecting your last kernel - I will update your time to 22.56 min in the results. You are right that dockerization of the algorithms is the way to go for production of machine learning models. But in the frame of this competition, we decided to go for kernels because it would be easier to compile the results and check the inference time computation. We know that this comes with some limitations but globally it was a good choice for the speed prize. Sorry if this created some frustration over the kernels GPU architecture.\nTo answer your other questions, @inversion and myself checked that the inference time was correctly computed and we did not notice anything wrong. I also relaunched the kernels a number of time and did observed some different timing but nothing that would swap the order or at least change the winner. As for the way to achieve 3+ minutes, maybe Alexandre will want to share some tips and tricks on this discussion !\nKind regards,\nJeff.
", "votes": null, "replies": [] }, { "id": 437504, "author_name": "snakers41", "author_url": "", "post_date": "12/12/2018 03:35:59", "content": "\n\nHi Jeff,
\n\nTalked with @marvelousninja\nLooks like he also used 2 stage model (classifier + semseg, but he used LinkNet).\nUsing LinkNet would probably boost my local time to 4 ~mins, though I was skeptical about cutting the resolution drastically to 224.
\n\nAnyway my point though is about:\n- By pushing the community to use Kernels - you have only a handful of submissions, and I guess half of people stopped optimizing their pipeline on seeing how kernels perform;\n- By not counting IO time (because let's be honest - Kernels just suck) - you do no alleviate a problem, but essentially create a motivation towards solutions that half-ass the way around this, i.e. loading all images into RAM in advance - but in real life such solutions are not viable;\n- In the long term - all of this just diverts community from Kaggle;
\n\nThe only way to asses the true merit of the whole pipeline is to run it end-to-end, IO, preparation and all on a dedicated standard machine.
", "votes": null, "replies": [] }, { "id": 437644, "author_name": "voglinio", "author_url": "", "post_date": "12/12/2018 08:55:27", "content": "Hello from team Nodalpoints (Ouranos and me). It seems the results correspond to timings and score on the 15,000 image test set. For example our timing is the same like the one we have reported on the comment section of our kernel. \nIf possible we would like to have the updated metrics (time /score) on the new images that this mini competition was based on. I believe that this will not affect the ranking but it may reduce the differences. For me going below 5 minutes for 15,000 images for classification/segmentation is blazing fast
", "votes": null, "replies": [] }, { "id": 437649, "author_name": "seesee", "author_url": "", "post_date": "12/12/2018 09:11:44", "content": "What is the running time of the whole kernel? It is ~6 minutes for me. Running an image processing algorithm (preprocessing, copying data to the GPU, launching kernels, post-processing) is faster than reading an image (SSD) & jpeg decoding. Even if the resolution is reduced to 224, this seems odd. I am happy to learn more about how this is possible.
", "votes": null, "replies": [] }, { "id": 437669, "author_name": "marvelousninja", "author_url": "", "post_date": "12/12/2018 09:45:22", "content": "@See-- , overall time was around 10 minutes. It breaks down to:
\n\nI also created a separate thread with a more detailed description:\nhttps://www.kaggle.com/c/airbus-ship-detection/discussion/74443
", "votes": null, "replies": [] }, { "id": 437891, "author_name": "seesee", "author_url": "", "post_date": "12/12/2018 17:49:27", "content": "@jeffaudi @inversion Could please give an update to the current situation? Are these the final results? I saw you updated the results table. It is impossible to reproduce the 30 seconds classifier claimed in the winner thread. This is really disappointing. I thought you were looking into this.
", "votes": null, "replies": [] }, { "id": 441122, "author_name": "jeffaudi", "author_url": "", "post_date": "12/18/2018 09:31:54", "content": "Hi to everyone,
\n\nKaggle is a place to learn and share about data science. At the end of the day, we are all here to make experiments and learn from our experience. This also applies to sponsors.
\n\nWith the main Ship Detection competition, we learned that we could improve our internal results substantially but also that even large ships could still be missed in harbours or at sea on some occasions. We pushed to organize the Speed Prize because we wanted to make sure that the best models could also process our images fast enough. By deciding to measure only computation time and not loading time, I have made the rules aligned with our real production environment but I have also created extra difficulty to review the solutions. Although we have carefully rerun the kernels and reviewed the code to check that inference time was correctly measured, we were not able to spot the issue with pyTorch and asynchronism in @marvelousninja kernel immediately. This has been discovered and revealed by himself after the initial publication of the results. I am grateful to @marvelousninja for his transparency and efforts to correct the issue but together with Kaggle team, we decided that we cannot accept a new kernel after the deadline. I have updated the score accordingly and the winner team name.
\n\nOnce again, I want to thank all participants for their participation in this new type of challenge. We have a lot of lessons learned on this competition and we will be looking toward great new challenges next year !
\n\nHappy holiday season to all of you,
\n\nJeff.
", "votes": null, "replies": [] }, { "id": 441343, "author_name": "ddanevskyi", "author_url": "", "post_date": "12/18/2018 15:09:59", "content": "Wow, this is absolutely amazing, thank you!
", "votes": null, "replies": [] } ] }, { "id": 437305, "author_name": "ddanevskyi", "author_url": "", "post_date": "12/11/2018 17:38:06", "content": "Our team [attenion heads] also submitted a kernel, but I don't see our score in the results table.
\n\n@jeffaudi, can you elaborate on this please?
", "votes": null, "replies": [ { "id": 437338, "author_name": "jeffaudi", "author_url": "", "post_date": "12/11/2018 18:24:40", "content": "Hi Dmitriy,\nThank you for submitting your kernel. You did not add the inference time and the score in the comments. By looking into the code, I could see \"Total time: 25727069.01 minutes.\" So I added a comment to the kernel to get your feedback on this figure because it seemed like an error. I got no answer so I kept your results out of the final list. \nKind regards,\nJeff.
", "votes": null, "replies": [] }, { "id": 437346, "author_name": "ddanevskyi", "author_url": "", "post_date": "12/11/2018 18:46:25", "content": "You might be looking on some erroneous kernel we submitted on the last minutes before the deadline. We certainly did submit proper version with time and score in comments.
", "votes": null, "replies": [] }, { "id": 437705, "author_name": "jeffaudi", "author_url": "", "post_date": "12/12/2018 10:40:23", "content": "Yes. Got it! Strangely enough, I did not received a sharing email notification about your other kernel and could not see it. But it is solved now. I will update the results accordingly. Sorry for this. Jeff.
", "votes": null, "replies": [] }, { "id": 437708, "author_name": "ddanevskyi", "author_url": "", "post_date": "12/12/2018 10:46:14", "content": "Thanks a lot!
", "votes": null, "replies": [] } ] }, { "id": 437703, "author_name": "yaroshevskiy", "author_url": "", "post_date": "12/12/2018 10:38:58", "content": "Looks like you've missed our kernel https://www.kaggle.com/ddanevskyi/speedprize-attention-heads (shared with @inversion , @jeffaudi ) with next comment added: \"Inference time (excluding time for loading images): 4.69 minutes\nPrivate leaderboard score: 0.84509 (79 place)\" from Nov 30. Doesnt change the winner but still we spent time and that's sad :(
", "votes": null, "replies": [ { "id": 437706, "author_name": "jeffaudi", "author_url": "", "post_date": "12/12/2018 10:41:31", "content": "I am correcting this right now. Good job !
", "votes": null, "replies": [] } ] }, { "id": 437877, "author_name": "voglinio", "author_url": "", "post_date": "12/12/2018 17:18:17", "content": "Hello again from Nodalpoints! Can you please re-create the result table using the new images (and please report how many they are). We would also like to know the percentage of empty to non empty ships since it can also affect performance. I am not asking out of plain curiocity but out of professional interest for the business problem.
", "votes": null, "replies": [ { "id": 437886, "author_name": "seesee", "author_url": "", "post_date": "12/12/2018 17:35:58", "content": ":heavy_plus_sign: I'd like to see this as well. If possible, please also add the total running time. The whole 'classifier + segmentation' approach relies on a big fraction of 'empty' images to be fast. It would be interesting to see the timings/scores for the new validation set.
\n\nI know that this requires some work, but you and we will learn a lot :)
", "votes": null, "replies": [] }, { "id": 439007, "author_name": "jeffaudi", "author_url": "", "post_date": "12/14/2018 15:25:09", "content": "@seesee I will see if I can output this on the four top kernels.
", "votes": null, "replies": [] }, { "id": 439181, "author_name": "voglinio", "author_url": "", "post_date": "12/14/2018 21:01:41", "content": "Sorry this is the last time I write for this subject. Is there a reason that you cannot output timings? After all timing was the issue of this mini competition. I would expect that you already compiled a simple Excel sheet that records times and score... And please report the scores on the new datasets as well. If you find it difficult just publish the new dataset with the ground truth and let us measure our performance our selves.
\n\nLast notice: such kind of speed tests are more than welcome for future competitions. Thank you for your effort to organize one. However you need to think over to the evaluation process... For example it is difficult to compare time measurements by timing blocks set by the contestants themselves... A small mistake or a forgotten block might favor one team against the others. Kaggle is famous for its bulletproof evaluation process.. this undisputable fairness is what brings the greatest added value to the competitions.
\n\nKeep up the good work! Over and out
", "votes": null, "replies": [] }, { "id": 441134, "author_name": "jeffaudi", "author_url": "", "post_date": "12/18/2018 09:58:50", "content": "Hi Costas,\nThank you for you message and your support. It has been an interesting ride for us at Airbus - we are usually a more closed environment. We have heard the community comments about need for \"bulletproof evaluation process\" and amended the results accordingly. A lot of lessons learned through this competition - we will definitely take that into account in the future. I can assure you that we are very motivated by the results and very thankful to the whole Kaggle community. \nKind regards, Jeff.
", "votes": null, "replies": [] } ] }, { "id": 441042, "author_name": "ddanevskyi", "author_url": "", "post_date": "12/18/2018 07:49:15", "content": "\n\nAny chance we will be updated on the situation soon?
\n\nRegards,
\n\nDmitriy
", "votes": null, "replies": [ { "id": 441109, "author_name": "jeffaudi", "author_url": "", "post_date": "12/18/2018 09:17:52", "content": "Yes, sorry that it took so long. I will update the results today.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "409837": "You can find the details of the speed prize here:\n\nhttps://www.kaggle.com/c/airbus-ship-detection#Algorithm-Speed-Prize\n\nPlease ask questions about it in this thread. Thx!", "409841": "Hi inversion, thanks for posting more details about the speed prize.\nTwo points came to my mind:\n\n 1. \"The winner will be the fastest Kernel (that does not have significant degradation in predictive ability against the validation set)\". What do you mean by validation set? Is it the test set of the main competition? Also, could you please quantify \"significant degradation in predictive ability\"?\n 2. Do we have to use the same model design we used in the main competition (best F2 score) for the speed prize or we can design other models aiming for fastest algorithm?\n\nThank you in advance,\nRenan", "410640": "Are there any conditions about quality of the models (minimum of the score, for example)? \n\nP.S. I can submit empty masks very fast :D", "411701": "Hi Renan,\n\nThank you for your questions. The validation dataset is the test dataset of the main competition (called test_v2). The speed prize will be open to teams that score in top 100 of the private leaderboard. Let's call S the score of the 100th participant on the private leaderboard on the closing day of the main competition. We consider that the score of the new speed kernel on the private leaderboard should not go below S. As long as this is verified, you can use any model design that you want.\n\nKind regards,\n\nJeff", "411718": "That's very clear. Thank you Jeff!", "416929": "Can we form new teams for the speed prize competition?", "419105": "What is the minimum system requirement for this algorithm speed challenge? like for example in Human Protein Atlas Image Classification challenge, minimum hardware limits are: CPU Cores: 2, RAM: 4GB and GPUs: Integrated Intel Graphics.. What about threshold scoring requirement for algorithm? what is the baseline score? thanks!", "421401": "Will there be a leaderboard for speed prize ?", "421901": "The Kernel will be validated against a new dataset.\nfrom where we get the new data set...", "423461": "jaideepvalani My guess is you can't. This prevents cheating (e.g. precompute the predictions and then just upload them.", "423555": "I meant in the rulesfor algo speed prize it was mentioned that new data set is going to be used.. i was needing a clarification on that..", "423961": "Hello everybody! Regarding the speed test, should we add the model creation and weight loading time to the overall inference time?", "424436": "Could you, please, clarify if inference time includes the time for loading and preprocessing the data? Also, if there any way to compare the speed of my kernel with others before the submission deadline? In particular if I get, for example, 10 minutes or 1 hour, is it fast or slow?", "424524": "inversion Does ones entry just have to create a better submission than the current place 100: `0.84416` or is there a separate set on which our approach will be evaluated?", "424525": "Hi Costas,\n\nThanks for participating in the Algorithm Speed Prize! We expect the inference time to be only the computation time on imagery (including pre-processing, models computation and post-processing) without loading of weights and models. The rational is to think of a running docker container which would wait for imagery buffer to compute ships masks. \n\nKind regards,\n\nJeff.", "424532": "Hi See,\n\nThanks for participating in the Algorithm Speed Prize! You are right. The kernel should create a submission against the current test_v2 dataset and rank better than 0.84416. You should create a comment at the end of your kernel with the score on the private LB and the inference time. We will run inference on a new test dataset just to make sure that the model still runs correctly on new imagery.\n\nKind regards,\n\nJeff.", "424688": "Thank you Jeff for your replies! So the way I understand it is that in the new test dataset you will not score the model. You will just measure the inference time! So the 0.84416 will define all eligible participants and then it is just a question of speed. I guess the private score could be used to break ties or something. Am I correct?", "424740": "Hello,\nYou should use kernels (with GPU) for the speed prize. And the minimum score is 0.84416\nKind regards,\nJeff", "424750": "Yes. I will measure speed and score on test_v2. Then I will rate participants on fastest speed given that score >= 0.84416. I will use another new test dataset to measure inference and score again and verify if there is no discrepencies in the ranking (i.e. the final algorithm should perform as well on a new dataset because that is what is really important for us).", "424816": "Hi jeff \nthanks for replies\nI m new to kaggle competitions ,that is this first one...\n ,could u please ans below queries\n1) we only calculating here the prediction time ,so that is how long does the model takes in predicting the testv2?\n2) from where does my time starts... to predict i will have to rebuild the model object,load the trained weights finally start predicting...\n3) if i resubmit then my rank could improve also ,so new rank will be considered or first time private lb rank be considered..", "425219": "Hi iafoss,\n\nThank you so much for participating in the Algorithm Speed Prize, especially after all the good stuff that you already shared with the community in Kernels and Discussions. Your questions are very legitimate and we are looking into ways of sharing a kind of leaderboard with the participants. I will get back to all of you in this discussion thread as soon as possible. For the inference time, I would say that fast is closer to 10 minutes than to 1 hour :)\n\nKind regards,\nJeff.", "425507": "Hi Jaideep,\nTo answer your questions :\n1.- Yes, inference time is how much the model(s) need to predict all images in test_v2\n2.- Yes, the inference time does not include setup the model, load the weights, load the images, etc... but includes pre-processing (i.e. rescaling), model prediction (with i.e. ensembling and TTA) and post processing (i.e. removing the overlaps, creating the CSV file). Everything that stays the same whatever the image should not be counted, everything that needs to be recomputed with new images needs to be included.\n3.- Yes, of course. You can submit multiple times and only your best score will count.\nKind regards,\nJeff.", "425516": "Jeff Faudi, thank you so much for working on it. It is really important to know how the model performs in comparison with others. My current fastest kernel takes 13 minutes, and I would really like to compare it with results of other people before the deadline.", "425518": "Jeff Faudi, thank you for clarification on what must be included in the inference time.", "427373": "Hi Jeff,\n\n Yes, the inference time does not include setup the model, load the weights, load the images, etc…\n\nI understand your rationale. Though, for me it is not clear what you mean by excluding the image loading (reading the image string & jpeg decoding). Could you provide a modified template? I have been using the one provided by @inversion (competition tab), without the model loading.\n\n model = load_model()\n import time\n inference_start = time.time()\n ###### inference code #######\n inference_end = time.time()\n print('Inference Time: %0.2f Minutes'%((inference_end - inference_start)/60))\n\nEvery framework has different loaders. I am using tensorflow. Reading the image string and decoding it is part of the graph ~~and it is hard to untangle these timings~~ (see below).", "427396": "To make it more concrete. We can exclude the time to load an image batch in RGB format as np.uint8?\n\n\n model = load_model()\n import time\n inference_time = 0.\n submission = pd.DataFrame()\n ###### inference code #######\n for k in range(len(data_loader)):\n img_batch = data_loader[k] # np.uint8 (N, H, W, C) format; not timed\n tic = time.time()\n img_batch = normalize(img_batch)\n y_pred = model(img_batch) # includes ensemble, TTA\n submission_batch = postprocess(y_pred)\n submission.append(submission_batch)\n inference_time += (time.time() - tic)\n\n tic = time.time()\n submission.to_csv('submission.csv', index=False)\n inference_time += (time.time() - tic)\n print('Inference Time: %0.2f Minutes'%((inference_time) / 60))", "429075": "See-- Yes, this is correct. You can exclude the loading of the images from the inference time.", "429080": "Hi to all participants in the Algorithm Speed Prize,\nThe deadline for submitting your kernel (i.e. sharing them with @jeffaudi and @inversion) is this Friday, November 30th (11:59 PM UTC). The current best time is under 10 minutes on a GPU kernel. \nGood luck,\nJeff.", "429712": "Hello @jeffaudi and @inversion!\nIs it safe to assume that you will double-check the code for time calculations inside of the kernel?\nSome participants might forget to exclude image reading time.\nOthers might have unintentional mistakes in time related code.", "430020": "Hello,\nOh yes, we will carefully review this. We will start from the fastest kernel: check the code, re-run the kernel multiple times, verify the score, etc...\nKind regards,\nJeff.", "430029": "this is the time for predicting 15k images", "430075": "Impressive time... A couple of questions. Is this time measured only on model prediction? No loading of images and stuff.. The other thing I noticed is a large runtime fluctuation of the same kernel. The order of 5-10 minutes from run to run. Are you going to benchmark the codes on special \"kernels\" or on the common ones prone to this fluctuation? Thank you once more for your great efforts...", "430481": "Hi,\nour pipeline inference time on Kaggle kernel increased x6+ times compared to the same pipe running on 1080ti but around x3 was expected. Any legal known methods to effectively load Tesla K80? :)", "430578": "If you used half precision, which is not supported by K80, the speed reduction should be ~x6.", "430579": "Which is not supported by K80 :)", "430643": "voglinio I didn't observe that big changes. Runs are always between +-10 seconds.", "430663": "See-- today things were clearly more stable.. Maybe competitions were ending yesterday. One more thing. I 've managed to run this greate examples\nhttps://github.com/jeng1220/KerasToTensorRT\non a kaggle kernel (from tensorflow.contrib import tensorrt as tftrt).\n I didn't have the time to see some substantial gain. Keep it in mind for other cases....", "430716": "Hi @inversion @jeffaudi\n\nCould we get some confirmation that our kernel was accepted? Thanks in advance.", "431437": "Thank you very much for participating in the Algorithm Speed Prize! \nI am sorry but I have been unable to post on the discussion forum yesterday for some unknown reason. This seems to be OK now. I am currently verifying the kernels and compiling the results. I will notify every team that I have recorded their submission. \nKind regards,\nJeff.", "431927": "Submitted a couple of kernels on Friday. Did not see that they were accepted.", "437287": "Hello to everyone !\n\nThank you so much for participating in the Algorithm Speed Prize ! This was somewhat unusual to have a follow-on challenge but we wanted it because inference speed is almost as important for us as accuracy. We need to extract ships from a full satellite image (i.e. roughly 4,500 of this competition smaller images) in a few minutes. With the below results, we are confident that we will be able to provide both accuracy and speed !\n\nHere are the results from the Algorithm Speed Prize :\n\n(*updated on 18-DEC-2018) \n\nCongratulations ! The first prize goes to @ddanevskyi and his team ! Kudos also to @seesee, to @marvelousninja and to @iafoss, you have done very well. And thank you to all of the participants !! I hope that you have learned a few things about putting algorithms into production during this Special Prize :) Do not hesitate to go back to me with questions or suggestions about this challenge or anything related to applying machine learning to satellite imagery. \n\nKind regards,\n\nJeff", "437301": "Hi,\n\n**A number of comments here.**\n- Did you do any sort of dockerization / hardware standardization when verifying the above results? In my runs run time within kernels varied +/- 25%, not that it matters;\n- I shared 2 additional kernels on the last day, which you ignored - my comment above also was ignored. One of these kernels explicitly showed that RAM => GPU transfer was a culprit for slow performance. I actually was able to get ~5.5 mins on my local hardware (1 GPU with full saturation). So this kind of discrepancy kind of raises the question - may be you can get a better result by exploiting some perks of kernels?;\n- 5+ minutes definitely seems doable, but on normal hardware in controlled conditions. So - the usage of kernels for this part of competitions 100% negates the purpose of the competition and (arguably) favours usage of some frameworks / optimizations;\n\n\nAlso it is very interesting how 3+ minutes can be achieved.\nDid you check that the kernel contained all the preprocessing steps and the time measured was true inference time?\n\nI actually distilled my best results into 2 networks (classifier and a small semseg model) - but it allowed me to achieve only around ~5 minutes on decent hardware.", "437305": "Our team [attenion heads] also submitted a kernel, but I don't see our score in the results table.\n\n@jeffaudi, can you elaborate on this please?", "437334": "Congratulations @marvelousninja . Please share some details about the winning approach. Simply loading the images (with opencv, single thread) takes ~4 minutes. 3.38 minutes seems like a miracle.\n\n> \"may be you can get a better result by exploiting some perks of kernels?;\" \n\nI guess so. For me, the kernel was always ~6x slower than the local pipeline.", "437336": "Hi Alexander,\nSorry for not collecting your last kernel - I will update your time to 22.56 min in the results. You are right that dockerization of the algorithms is the way to go for production of machine learning models. But in the frame of this competition, we decided to go for kernels because it would be easier to compile the results and check the inference time computation. We know that this comes with some limitations but globally it was a good choice for the speed prize. Sorry if this created some frustration over the kernels GPU architecture.\nTo answer your other questions, @inversion and myself checked that the inference time was correctly computed and we did not notice anything wrong. I also relaunched the kernels a number of time and did observed some different timing but nothing that would swap the order or at least change the winner. As for the way to achieve 3+ minutes, maybe Alexandre will want to share some tips and tricks on this discussion !\nKind regards,\nJeff.", "437338": "Hi Dmitriy,\nThank you for submitting your kernel. You did not add the inference time and the score in the comments. By looking into the code, I could see \"Total time: 25727069.01 minutes.\" So I added a comment to the kernel to get your feedback on this figure because it seemed like an error. I got no answer so I kept your results out of the final list. \nKind regards,\nJeff.", "437346": "You might be looking on some erroneous kernel we submitted on the last minutes before the deadline. We certainly did submit proper version with time and score in comments.", "437504": "jeffaudi\n\nHi Jeff,\n\nTalked with @marvelousninja\nLooks like he also used 2 stage model (classifier + semseg, but he used LinkNet).\nUsing LinkNet would probably boost my local time to 4 ~mins, though I was skeptical about cutting the resolution drastically to 224.\n\nAnyway my point though is about:\n- By pushing the community to use Kernels - you have only a handful of submissions, and I guess half of people stopped optimizing their pipeline on seeing how kernels perform;\n- By not counting IO time (because let's be honest - Kernels just suck) - you do no alleviate a problem, but essentially create a motivation towards solutions that half-ass the way around this, i.e. loading all images into RAM in advance - but in real life such solutions are not viable;\n- In the long term - all of this just diverts community from Kaggle;\n\nThe only way to asses the true merit of the whole pipeline is to run it end-to-end, IO, preparation and all on a dedicated standard machine.", "437644": "Hello from team Nodalpoints (Ouranos and me). It seems the results correspond to timings and score on the 15,000 image test set. For example our timing is the same like the one we have reported on the comment section of our kernel. \nIf possible we would like to have the updated metrics (time /score) on the new images that this mini competition was based on. I believe that this will not affect the ranking but it may reduce the differences. For me going below 5 minutes for 15,000 images for classification/segmentation is blazing fast", "437649": "What is the running time of the whole kernel? It is ~6 minutes for me. Running an image processing algorithm (preprocessing, copying data to the GPU, launching kernels, post-processing) is faster than reading an image (SSD) & jpeg decoding. Even if the resolution is reduced to 224, this seems odd. I am happy to learn more about how this is possible.", "437669": "See-- , overall time was around 10 minutes. It breaks down to:\n\n* around 3 minutes to install all pip packages\n* around 4 minutes on image I/O\n* around 3 minutes for actual inference\n\nI also created a separate thread with a more detailed description:\nhttps://www.kaggle.com/c/airbus-ship-detection/discussion/74443", "437703": "Looks like you've missed our kernel https://www.kaggle.com/ddanevskyi/speedprize-attention-heads (shared with @inversion , @jeffaudi ) with next comment added: \"Inference time (excluding time for loading images): 4.69 minutes\nPrivate leaderboard score: 0.84509 (79 place)\" from Nov 30. Doesnt change the winner but still we spent time and that's sad :(", "437705": "Yes. Got it! Strangely enough, I did not received a sharing email notification about your other kernel and could not see it. But it is solved now. I will update the results accordingly. Sorry for this. Jeff.", "437706": "I am correcting this right now. Good job !", "437708": "Thanks a lot!", "437877": "Hello again from Nodalpoints! Can you please re-create the result table using the new images (and please report how many they are). We would also like to know the percentage of empty to non empty ships since it can also affect performance. I am not asking out of plain curiocity but out of professional interest for the business problem.", "437886": ":heavy_plus_sign: I'd like to see this as well. If possible, please also add the total running time. The whole 'classifier + segmentation' approach relies on a big fraction of 'empty' images to be fast. It would be interesting to see the timings/scores for the new validation set.\n\nI know that this requires some work, but you and we will learn a lot :)", "437891": "jeffaudi @inversion Could please give an update to the current situation? Are these the final results? I saw you updated the results table. It is impossible to reproduce the 30 seconds classifier claimed in the winner thread. This is really disappointing. I thought you were looking into this.", "439007": "seesee I will see if I can output this on the four top kernels.", "439181": "Sorry this is the last time I write for this subject. Is there a reason that you cannot output timings? After all timing was the issue of this mini competition. I would expect that you already compiled a simple Excel sheet that records times and score... And please report the scores on the new datasets as well. If you find it difficult just publish the new dataset with the ground truth and let us measure our performance our selves.\n\nLast notice: such kind of speed tests are more than welcome for future competitions. Thank you for your effort to organize one. However you need to think over to the evaluation process... For example it is difficult to compare time measurements by timing blocks set by the contestants themselves... A small mistake or a forgotten block might favor one team against the others. Kaggle is famous for its bulletproof evaluation process.. this undisputable fairness is what brings the greatest added value to the competitions.\n\nKeep up the good work! Over and out", "441042": "jeffaudi @inversion \n\nAny chance we will be updated on the situation soon?\n\nRegards,\n\nDmitriy", "441109": "Yes, sorry that it took so long. I will update the results today.", "441122": "Hi to everyone, \n\nKaggle is a place to learn and share about data science. At the end of the day, we are all here to make experiments and learn from our experience. This also applies to sponsors. \n\nWith the main Ship Detection competition, we learned that we could improve our internal results substantially but also that even large ships could still be missed in harbours or at sea on some occasions. We pushed to organize the Speed Prize because we wanted to make sure that the best models could also process our images fast enough. By deciding to measure only computation time and not loading time, I have made the rules aligned with our real production environment but I have also created extra difficulty to review the solutions. Although we have carefully rerun the kernels and reviewed the code to check that inference time was correctly measured, we were not able to spot the issue with pyTorch and asynchronism in @marvelousninja kernel immediately. This has been discovered and revealed by himself after the initial publication of the results. I am grateful to @marvelousninja for his transparency and efforts to correct the issue but together with Kaggle team, we decided that we cannot accept a new kernel after the deadline. I have updated the score accordingly and the winner team name.\n\nOnce again, I want to thank all participants for their participation in this new type of challenge. We have a lot of lessons learned on this competition and we will be looking toward great new challenges next year ! \n\nHappy holiday season to all of you,\n\nJeff.", "441134": "Hi Costas,\nThank you for you message and your support. It has been an interesting ride for us at Airbus - we are usually a more closed environment. We have heard the community comments about need for \"bulletproof evaluation process\" and amended the results accordingly. A lot of lessons learned through this competition - we will definitely take that into account in the future. I can assure you that we are very motivated by the results and very thankful to the whole Kaggle community. \nKind regards, Jeff.", "441343": "Wow, this is absolutely amazing, thank you!" }, "source": "meta" }