My code executes successfully in 23 minutes, but when it comes to scoring, the process inexplicably takes over 12 hours, ultimately failing due to a notebook timeout. I need assistance understanding why the scoring step is so inefficient and how to resolve this issue.
", "messages": [ { "id": 3055256, "postDate": "2024-11-25T16:50:18.893Z", "content": "See more ideas here
\nhttps://www.kaggle.com/competitions/czii-cryo-et-object-identification/discussion/547044
I believe they run our notebooks against something like 500 files. Assuming I have that number correct that only gives you about 85 seconds per file.
", "rawMarkdown": "I believe they run our notebooks against something like 500 files. Assuming I have that number correct that only gives you about 85 seconds per file.", "votes": 2, "replies": [ { "id": 3054803, "postDate": "2024-11-25T07:40:51.243Z", "content": "thank you for your help, have a nice day
", "rawMarkdown": "thank you for your help, have a nice day", "votes": 2 } ] }, { "id": 3054681, "postDate": "2024-11-25T02:00:47.140Z", "content": "My code executes successfully in 23 minutes, but when it comes to scoring, the process inexplicably takes over 12 hours, ultimately failing due to a notebook timeout. I need assistance understanding why the scoring step is so inefficient and how to resolve this issue.
", "rawMarkdown": "My code executes successfully in 23 minutes, but when it comes to scoring, the process inexplicably takes over 12 hours, ultimately failing due to a notebook timeout. I need assistance understanding why the scoring step is so inefficient and how to resolve this issue.", "votes": 2 }, { "id": 3054763, "postDate": "2024-11-25T05:46:47.967Z", "content": "The training data has 7 tomograms, but the test set has 500. You should benchmark just the inference time (excluding the training) for the 7 tomograms, then scale it up to estimate the time required for 500.
\nIt seems like the connected component analysis seems to be the most time consuming step for a lot of people. There is some interesting discussion on how to speed up the inference here and here.
", "rawMarkdown": "The training data has 7 tomograms, but the test set has 500. You should benchmark just the inference time (excluding the training) for the 7 tomograms, then scale it up to estimate the time required for 500.\n\nIt seems like the connected component analysis seems to be the most time consuming step for a lot of people. There is some interesting discussion on how to speed up the inference [here](https://www.kaggle.com/competitions/czii-cryo-et-object-identification/discussion/547044) and [here](https://www.kaggle.com/code/hengck23/speed-up-connected-component-analysis-with-pytorch).\n\n", "votes": 4, "isDeleted": true, "replies": [ { "id": 3054802, "postDate": "2024-11-25T07:40:34.830Z", "content": "thank you so much for your help
", "rawMarkdown": "thank you so much for your help", "votes": 1 } ] } ], "comments": [ { "id": 3055256, "author_name": "DennisSakva", "author_url": "", "post_date": "2024-11-25T16:50:18.893000", "content": "See more ideas here
\nhttps://www.kaggle.com/competitions/czii-cryo-et-object-identification/discussion/547044
I believe they run our notebooks against something like 500 files. Assuming I have that number correct that only gives you about 85 seconds per file.
", "votes": 2, "replies": [ { "id": 3054803, "author_name": "Younes Hlal", "author_url": "", "post_date": "2024-11-25T07:40:51.243000", "content": "thank you for your help, have a nice day
", "votes": 2, "replies": [] } ] }, { "id": 3054763, "author_name": "", "author_url": "", "post_date": "2024-11-25T05:46:47.967000", "content": "The training data has 7 tomograms, but the test set has 500. You should benchmark just the inference time (excluding the training) for the 7 tomograms, then scale it up to estimate the time required for 500.
\nIt seems like the connected component analysis seems to be the most time consuming step for a lot of people. There is some interesting discussion on how to speed up the inference here and here.
", "votes": 4, "replies": [ { "id": 3054802, "author_name": "Younes Hlal", "author_url": "", "post_date": "2024-11-25T07:40:34.830000", "content": "thank you so much for your help
", "votes": 1, "replies": [] } ] } ], "raw_markdown_by_id": { "3055256": "See more ideas here\nhttps://www.kaggle.com/competitions/czii-cryo-et-object-identification/discussion/547044", "3054729": "I believe they run our notebooks against something like 500 files. Assuming I have that number correct that only gives you about 85 seconds per file.", "3054681": "My code executes successfully in 23 minutes, but when it comes to scoring, the process inexplicably takes over 12 hours, ultimately failing due to a notebook timeout. I need assistance understanding why the scoring step is so inefficient and how to resolve this issue.", "3054763": "The training data has 7 tomograms, but the test set has 500. You should benchmark just the inference time (excluding the training) for the 7 tomograms, then scale it up to estimate the time required for 500.\n\nIt seems like the connected component analysis seems to be the most time consuming step for a lot of people. There is some interesting discussion on how to speed up the inference [here](https://www.kaggle.com/competitions/czii-cryo-et-object-identification/discussion/547044) and [here](https://www.kaggle.com/code/hengck23/speed-up-connected-component-analysis-with-pytorch).\n\n" } }