This is our first competition on kaggle, we did final submission notebook , but apparently its running more than 9 hours on private test setðŸ˜, however normally we did the estimations that it should run approximately 5 hours for private test set of 1500 samples….. May be do you know the tips how to shorten submission time? Notebook is here https://www.kaggle.com/olgapuntous/hpa-final-submission-1
", "messages": [ { "id": 1266293, "postDate": "2021-04-07T16:33:12.273Z", "content": "Hi @olgapuntous
\nI’m sorry to hear you’re having these problems.
\nUnfortunately I didn’t have time to go through your notebook. But I suppose there’s a few inefficiencies in the script that make it slow. Those can be unnecessary image reads or unnecessary for loops, for example.
\nFrom my experience with this competition, the biggest contributor to the inference time is the baseline segmentation approach suggested by the organisers.
What I suggest you could do is to try using a faster version described here: https://www.kaggle.com/c/hpa-single-cell-image-classification/discussion/222132
\nFor me it speeded things up by an order of at least 4 times. So I suppose with the rest unchanged you should fit in the timeframe if you just try a fast segmentation method.
\nGenerally, I was able to fit baseline segmentation with 2 models ensemble and test time augmentation in the 9h. So there should be enough time given the algorithm isn’t doing too much of unnecessary calculations.
\nHope, that helps!
", "rawMarkdown": "Hi @olgapuntous \nI’m sorry to hear you’re having these problems. \nUnfortunately I didn’t have time to go through your notebook. But I suppose there’s a few inefficiencies in the script that make it slow. Those can be unnecessary image reads or unnecessary for loops, for example.\nFrom my experience with this competition, the biggest contributor to the inference time is the baseline segmentation approach suggested by the organisers.\n\nWhat I suggest you could do is to try using a faster version described here: https://www.kaggle.com/c/hpa-single-cell-image-classification/discussion/222132\n\nFor me it speeded things up by an order of at least 4 times. So I suppose with the rest unchanged you should fit in the timeframe if you just try a fast segmentation method.\n\nGenerally, I was able to fit baseline segmentation with 2 models ensemble and test time augmentation in the 9h. So there should be enough time given the algorithm isn’t doing too much of unnecessary calculations. \n\nHope, that helps!", "votes": 1 }, { "id": 1265842, "postDate": "2021-04-07T08:50:01.243Z", "content": "This is our first competition on kaggle, we did final submission notebook , but apparently its running more than 9 hours on private test setðŸ˜, however normally we did the estimations that it should run approximately 5 hours for private test set of 1500 samples….. May be do you know the tips how to shorten submission time? Notebook is here https://www.kaggle.com/olgapuntous/hpa-final-submission-1
", "rawMarkdown": "This is our first competition on kaggle, we did final submission notebook , but apparently its running more than 9 hours on private test setðŸ˜, however normally we did the estimations that it should run approximately 5 hours for private test set of 1500 samples..... May be do you know the tips how to shorten submission time? Notebook is here https://www.kaggle.com/olgapuntous/hpa-final-submission-1", "votes": 1 }, { "id": 1268438, "postDate": "2021-04-09T11:50:57.457Z", "content": "Thanks a lot Dmitry @dmitryminskiy for your comments, its helped a lot to progress.
\nI've just made a resubmission with 3 line of test set only, and this time it gave a submission error. I think there is also another problem with this notebook!
\nAnyway thank you so much!
", "rawMarkdown": "Thanks a lot Dmitry @dmitryminskiy for your comments, its helped a lot to progress.\n\nI've just made a resubmission with 3 line of test set only, and this time it gave a submission error. I think there is also another problem with this notebook!\n\nAnyway thank you so much!" } ], "comments": [ { "id": 1266293, "author_name": "Dmitry Minskiy", "author_url": "", "post_date": "2021-04-07T16:33:12.273000", "content": "Hi @olgapuntous
\nI’m sorry to hear you’re having these problems.
\nUnfortunately I didn’t have time to go through your notebook. But I suppose there’s a few inefficiencies in the script that make it slow. Those can be unnecessary image reads or unnecessary for loops, for example.
\nFrom my experience with this competition, the biggest contributor to the inference time is the baseline segmentation approach suggested by the organisers.
What I suggest you could do is to try using a faster version described here: https://www.kaggle.com/c/hpa-single-cell-image-classification/discussion/222132
\nFor me it speeded things up by an order of at least 4 times. So I suppose with the rest unchanged you should fit in the timeframe if you just try a fast segmentation method.
\nGenerally, I was able to fit baseline segmentation with 2 models ensemble and test time augmentation in the 9h. So there should be enough time given the algorithm isn’t doing too much of unnecessary calculations.
\nHope, that helps!
", "votes": 1, "replies": [] }, { "id": 1268438, "author_name": "olga puntous", "author_url": "", "post_date": "2021-04-09T11:50:57.457000", "content": "Thanks a lot Dmitry @dmitryminskiy for your comments, its helped a lot to progress.
\nI've just made a resubmission with 3 line of test set only, and this time it gave a submission error. I think there is also another problem with this notebook!
\nAnyway thank you so much!
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1266293": "Hi @olgapuntous \nI’m sorry to hear you’re having these problems. \nUnfortunately I didn’t have time to go through your notebook. But I suppose there’s a few inefficiencies in the script that make it slow. Those can be unnecessary image reads or unnecessary for loops, for example.\nFrom my experience with this competition, the biggest contributor to the inference time is the baseline segmentation approach suggested by the organisers.\n\nWhat I suggest you could do is to try using a faster version described here: https://www.kaggle.com/c/hpa-single-cell-image-classification/discussion/222132\n\nFor me it speeded things up by an order of at least 4 times. So I suppose with the rest unchanged you should fit in the timeframe if you just try a fast segmentation method.\n\nGenerally, I was able to fit baseline segmentation with 2 models ensemble and test time augmentation in the 9h. So there should be enough time given the algorithm isn’t doing too much of unnecessary calculations. \n\nHope, that helps!", "1265842": "This is our first competition on kaggle, we did final submission notebook , but apparently its running more than 9 hours on private test setðŸ˜, however normally we did the estimations that it should run approximately 5 hours for private test set of 1500 samples..... May be do you know the tips how to shorten submission time? Notebook is here https://www.kaggle.com/olgapuntous/hpa-final-submission-1", "1268438": "Thanks a lot Dmitry @dmitryminskiy for your comments, its helped a lot to progress.\n\nI've just made a resubmission with 3 line of test set only, and this time it gave a submission error. I think there is also another problem with this notebook!\n\nAnyway thank you so much!" } }