Am I the only one struggling with the running time limit of 9 hours of the notebook under submission?
\nI am performing a preprocessing over the segmented volumes in order to train a segmentation CNN that will allow me to train over segmented ROIs into the competition's labels. However, the preprocessing and the inference of a volume take around 10 seconds, that's without taking into account the train time which is around 2 hours, so 10 seconds * 3000 volumes (train+test that I would need to apply it), it would take around 8 hours, and remember this is only processing the data to obtain the ROIs for another training pipeline to classify.
\nObviously this makes this type of pipelines (segmentation + classification) almost impossible, I think one could spend a lot of time trying to optimize a little bit, but it is incredible far from a feasible time which would be taking less than 2-3 seconds per volume. It is so frustrating that you cannot use your idea due to a time limitation in the notebook.
\nProbably it is part of the challenge that you cannot easily use an optimal pipeline and must go through workarounds to fit the submission into the time limitation.
", "messages": [ { "id": "1909239", "postDate": "08/22/2022 13:01:59", "content": "Am I the only one struggling with the running time limit of 9 hours of the notebook under submission?
\nI am performing a preprocessing over the segmented volumes in order to train a segmentation CNN that will allow me to train over segmented ROIs into the competition's labels. However, the preprocessing and the inference of a volume take around 10 seconds, that's without taking into account the train time which is around 2 hours, so 10 seconds * 3000 volumes (train+test that I would need to apply it), it would take around 8 hours, and remember this is only processing the data to obtain the ROIs for another training pipeline to classify.
\nObviously this makes this type of pipelines (segmentation + classification) almost impossible, I think one could spend a lot of time trying to optimize a little bit, but it is incredible far from a feasible time which would be taking less than 2-3 seconds per volume. It is so frustrating that you cannot use your idea due to a time limitation in the notebook.
\nProbably it is part of the challenge that you cannot easily use an optimal pipeline and must go through workarounds to fit the submission into the time limitation.
", "rawMarkdown": "Am I the only one struggling with the running time limit of 9 hours of the notebook under submission?\n\nI am performing a preprocessing over the segmented volumes in order to train a segmentation CNN that will allow me to train over segmented ROIs into the competition's labels. However, the preprocessing and the inference of a volume take around 10 seconds, that's without taking into account the train time which is around 2 hours, so 10 seconds * 3000 volumes (train+test that I would need to apply it), it would take around 8 hours, and remember this is only processing the data to obtain the ROIs for another training pipeline to classify. \n\nObviously this makes this type of pipelines (segmentation + classification) almost impossible, I think one could spend a lot of time trying to optimize a little bit, but it is incredible far from a feasible time which would be taking less than 2-3 seconds per volume. It is so frustrating that you cannot use your idea due to a time limitation in the notebook. \n\nProbably it is part of the challenge that you cannot easily use an optimal pipeline and must go through workarounds to fit the submission into the time limitation.", "votes": null }, { "id": "1909398", "postDate": "08/22/2022 15:33:59", "content": "I get my inference done in less than 2 hours, I am doing a bone classification, a bone ROI segmentation and a fracture classification… I think it would be more than enough if you are good enough with your optimizations, you will have to optimize it to the limit if you want this type of speed but its very much not impossible…
", "rawMarkdown": "I get my inference done in less than 2 hours, I am doing a bone classification, a bone ROI segmentation and a fracture classification... I think it would be more than enough if you are good enough with your optimizations, you will have to optimize it to the limit if you want this type of speed but its very much not impossible...", "votes": null }, { "id": "1909443", "postDate": "08/22/2022 16:02:58", "content": "@harshitsheoran I think I'm swapping to only 2D networks because I thought to use a 3D CNN to classify ROIs obtained by an UNET. But for me at least, it seems impossible to obtain a whole 3D segmentations in time and save in disk ROIs to train.
", "rawMarkdown": "harshitsheoran I think I'm swapping to only 2D networks because I thought to use a 3D CNN to classify ROIs obtained by an UNET. But for me at least, it seems impossible to obtain a whole 3D segmentations in time and save in disk ROIs to train.", "votes": null }, { "id": "1909450", "postDate": "08/22/2022 16:07:16", "content": "Whatever works for you, I am only telling you that it is possible, but getting it to work might take a really long time (still worth it imo) if you are not very well experienced in optimization.
", "rawMarkdown": "Whatever works for you, I am only telling you that it is possible, but getting it to work might take a really long time (still worth it imo) if you are not very well experienced in optimization.", "votes": null }, { "id": "1909468", "postDate": "08/22/2022 16:21:19", "content": "Ok, I will try to optimize as much as I can, thank you for the help mate.
", "rawMarkdown": "Ok, I will try to optimize as much as I can, thank you for the help mate.", "votes": null }, { "id": "1909843", "postDate": "08/23/2022 01:33:55", "content": "Train and infer in different notebooks. Save the best weights of the trained model. And use that in infernece.
", "rawMarkdown": "Train and infer in different notebooks. Save the best weights of the trained model. And use that in infernece.", "votes": null }, { "id": "1910065", "postDate": "08/23/2022 06:34:35", "content": "You mean that I can preproccess the train dataset and train in my home pc, and then upload the weights with just the inference code and submit that?? I though that you have to submit everything and they create a clean kernel and run every cell so they train in there and then run inference.
\nIf it is that way then is pretty easy to stay within the constrain of 9 hours.
", "rawMarkdown": "You mean that I can preproccess the train dataset and train in my home pc, and then upload the weights with just the inference code and submit that?? I though that you have to submit everything and they create a clean kernel and run every cell so they train in there and then run inference.\n\nIf it is that way then is pretty easy to stay within the constrain of 9 hours.", "votes": null }, { "id": "1910331", "postDate": "08/23/2022 10:53:42", "content": "Yeah that's the essence
", "rawMarkdown": "Yeah that's the essence", "votes": null }, { "id": "1917768", "postDate": "08/29/2022 03:54:05", "content": "what labels you used for bone classification? or how do you classify the bone, by knn or what?
", "rawMarkdown": "what labels you used for bone classification? or how do you classify the bone, by knn or what?", "votes": null }, { "id": "1917795", "postDate": "08/29/2022 04:48:41", "content": "My current bone classifier technique is something I would not like to tell at this point, but I am planning to change it anyways by exploring public notebooks because it is apparently not as fast or good as even machine learning public notebook (ref to .88 acc one)…
", "rawMarkdown": "My current bone classifier technique is something I would not like to tell at this point, but I am planning to change it anyways by exploring public notebooks because it is apparently not as fast or good as even machine learning public notebook (ref to .88 acc one)...", "votes": null } ], "comments": [ { "id": 1909398, "author_name": "harshitsheoran", "author_url": "", "post_date": "08/22/2022 15:33:59", "content": "I get my inference done in less than 2 hours, I am doing a bone classification, a bone ROI segmentation and a fracture classification… I think it would be more than enough if you are good enough with your optimizations, you will have to optimize it to the limit if you want this type of speed but its very much not impossible…
", "votes": null, "replies": [ { "id": 1917768, "author_name": "dragonzhang", "author_url": "", "post_date": "08/29/2022 03:54:05", "content": "what labels you used for bone classification? or how do you classify the bone, by knn or what?
", "votes": null, "replies": [] }, { "id": 1917795, "author_name": "harshitsheoran", "author_url": "", "post_date": "08/29/2022 04:48:41", "content": "My current bone classifier technique is something I would not like to tell at this point, but I am planning to change it anyways by exploring public notebooks because it is apparently not as fast or good as even machine learning public notebook (ref to .88 acc one)…
", "votes": null, "replies": [] } ] }, { "id": 1909443, "author_name": "javiervera", "author_url": "", "post_date": "08/22/2022 16:02:58", "content": "@harshitsheoran I think I'm swapping to only 2D networks because I thought to use a 3D CNN to classify ROIs obtained by an UNET. But for me at least, it seems impossible to obtain a whole 3D segmentations in time and save in disk ROIs to train.
", "votes": null, "replies": [ { "id": 1909450, "author_name": "harshitsheoran", "author_url": "", "post_date": "08/22/2022 16:07:16", "content": "Whatever works for you, I am only telling you that it is possible, but getting it to work might take a really long time (still worth it imo) if you are not very well experienced in optimization.
", "votes": null, "replies": [] }, { "id": 1909468, "author_name": "javiervera", "author_url": "", "post_date": "08/22/2022 16:21:19", "content": "Ok, I will try to optimize as much as I can, thank you for the help mate.
", "votes": null, "replies": [] } ] }, { "id": 1909843, "author_name": "robber19", "author_url": "", "post_date": "08/23/2022 01:33:55", "content": "Train and infer in different notebooks. Save the best weights of the trained model. And use that in infernece.
", "votes": null, "replies": [ { "id": 1910065, "author_name": "javiervera", "author_url": "", "post_date": "08/23/2022 06:34:35", "content": "You mean that I can preproccess the train dataset and train in my home pc, and then upload the weights with just the inference code and submit that?? I though that you have to submit everything and they create a clean kernel and run every cell so they train in there and then run inference.
\nIf it is that way then is pretty easy to stay within the constrain of 9 hours.
", "votes": null, "replies": [] }, { "id": 1910331, "author_name": "robber19", "author_url": "", "post_date": "08/23/2022 10:53:42", "content": "Yeah that's the essence
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1909239": "Am I the only one struggling with the running time limit of 9 hours of the notebook under submission?\n\nI am performing a preprocessing over the segmented volumes in order to train a segmentation CNN that will allow me to train over segmented ROIs into the competition's labels. However, the preprocessing and the inference of a volume take around 10 seconds, that's without taking into account the train time which is around 2 hours, so 10 seconds * 3000 volumes (train+test that I would need to apply it), it would take around 8 hours, and remember this is only processing the data to obtain the ROIs for another training pipeline to classify. \n\nObviously this makes this type of pipelines (segmentation + classification) almost impossible, I think one could spend a lot of time trying to optimize a little bit, but it is incredible far from a feasible time which would be taking less than 2-3 seconds per volume. It is so frustrating that you cannot use your idea due to a time limitation in the notebook. \n\nProbably it is part of the challenge that you cannot easily use an optimal pipeline and must go through workarounds to fit the submission into the time limitation.", "1909398": "I get my inference done in less than 2 hours, I am doing a bone classification, a bone ROI segmentation and a fracture classification... I think it would be more than enough if you are good enough with your optimizations, you will have to optimize it to the limit if you want this type of speed but its very much not impossible...", "1909443": "harshitsheoran I think I'm swapping to only 2D networks because I thought to use a 3D CNN to classify ROIs obtained by an UNET. But for me at least, it seems impossible to obtain a whole 3D segmentations in time and save in disk ROIs to train.", "1909450": "Whatever works for you, I am only telling you that it is possible, but getting it to work might take a really long time (still worth it imo) if you are not very well experienced in optimization.", "1909468": "Ok, I will try to optimize as much as I can, thank you for the help mate.", "1909843": "Train and infer in different notebooks. Save the best weights of the trained model. And use that in infernece.", "1910065": "You mean that I can preproccess the train dataset and train in my home pc, and then upload the weights with just the inference code and submit that?? I though that you have to submit everything and they create a clean kernel and run every cell so they train in there and then run inference.\n\nIf it is that way then is pretty easy to stay within the constrain of 9 hours.", "1910331": "Yeah that's the essence", "1917768": "what labels you used for bone classification? or how do you classify the bone, by knn or what?", "1917795": "My current bone classifier technique is something I would not like to tell at this point, but I am planning to change it anyways by exploring public notebooks because it is apparently not as fast or good as even machine learning public notebook (ref to .88 acc one)..." }, "source": "meta" }