Hi,
\nCan any one tell based on so far experience is this competition needing lot of resources ?
Hi,
\nCan any one tell based on so far experience is this competition needing lot of resources ?
Very much so in my experience so far. I'm fortunate to own a high-end PC with a RTX3090 and still it can take a couple of hours to train a model, with close to 100% GPU utilization. That means if I wanted to do 5-fold cross validation it takes the whole day. I don't have a good idea how to approach experimenting with hyperparameters and different models.
\nAnd that's on the competition dataset alone, where there are still the larger LIVCell and unsupervised datasets.
", "rawMarkdown": "Very much so in my experience so far. I'm fortunate to own a high-end PC with a RTX3090 and still it can take a couple of hours to train a model, with close to 100% GPU utilization. That means if I wanted to do 5-fold cross validation it takes the whole day. I don't have a good idea how to approach experimenting with hyperparameters and different models.\n\nAnd that's on the competition dataset alone, where there are still the larger LIVCell and unsupervised datasets.", "votes": null }, { "id": "1570992", "postDate": "11/04/2021 14:33:28", "content": "Same here. It takes 45 seconds for me train a single epoch (484 samples, 121 steps) mask r-cnn on RTX-3090.
", "rawMarkdown": "Same here. It takes 45 seconds for me train a single epoch (484 samples, 121 steps) mask r-cnn on RTX-3090.", "votes": null }, { "id": "1571019", "postDate": "11/04/2021 14:43:21", "content": "Using Detectron2 and Resnet101, on the Kaggle notebooks, a run of 40 epochs or so takes about 2 hours.
\nResnet50 is faster. Probably don't need 40 epochs for all tests.
\nOverall, compared to some of the recent imaging competitions on CT images, this seems to be more manageable.
\n-Rich
", "rawMarkdown": "Using Detectron2 and Resnet101, on the Kaggle notebooks, a run of 40 epochs or so takes about 2 hours.\n\nResnet50 is faster. Probably don't need 40 epochs for all tests.\n\nOverall, compared to some of the recent imaging competitions on CT images, this seems to be more manageable.\n\n-Rich", "votes": null }, { "id": "1571036", "postDate": "11/04/2021 14:52:27", "content": "Hey Rich!
\nDo you have some tips for working with these large models? In my previous competitions so far I was always able to find approaches that took no longer than 15 minutes to train and my strategy was to try a lot of things until something worked.
", "rawMarkdown": "Hey Rich!\n\nDo you have some tips for working with these large models? In my previous competitions so far I was always able to find approaches that took no longer than 15 minutes to train and my strategy was to try a lot of things until something worked.", "votes": null }, { "id": "1571070", "postDate": "11/04/2021 15:17:06", "content": "I don't have specific suggestions. I always worry that parameters that might work on a \"quick\" model, might not be good for a fully run model. But I note you are doing much better in this competition than I am, so you are doing something right!
", "rawMarkdown": "I don't have specific suggestions. I always worry that parameters that might work on a \"quick\" model, might not be good for a fully run model. But I note you are doing much better in this competition than I am, so you are doing something right!", "votes": null }, { "id": "1571106", "postDate": "11/04/2021 15:50:49", "content": "@slawekbiel Using a subset of the data is usually helpful to try different models quickly.
", "rawMarkdown": "slawekbiel Using a subset of the data is usually helpful to try different models quickly.", "votes": null } ], "comments": [ { "id": 1570984, "author_name": "slawekbiel", "author_url": "", "post_date": "11/04/2021 14:24:27", "content": "Very much so in my experience so far. I'm fortunate to own a high-end PC with a RTX3090 and still it can take a couple of hours to train a model, with close to 100% GPU utilization. That means if I wanted to do 5-fold cross validation it takes the whole day. I don't have a good idea how to approach experimenting with hyperparameters and different models.
\nAnd that's on the competition dataset alone, where there are still the larger LIVCell and unsupervised datasets.
", "votes": null, "replies": [ { "id": 1570992, "author_name": "gunesevitan", "author_url": "", "post_date": "11/04/2021 14:33:28", "content": "Same here. It takes 45 seconds for me train a single epoch (484 samples, 121 steps) mask r-cnn on RTX-3090.
", "votes": null, "replies": [] }, { "id": 1571106, "author_name": "tolgadincer", "author_url": "", "post_date": "11/04/2021 15:50:49", "content": "@slawekbiel Using a subset of the data is usually helpful to try different models quickly.
", "votes": null, "replies": [] } ] }, { "id": 1571019, "author_name": "richardepstein", "author_url": "", "post_date": "11/04/2021 14:43:21", "content": "Using Detectron2 and Resnet101, on the Kaggle notebooks, a run of 40 epochs or so takes about 2 hours.
\nResnet50 is faster. Probably don't need 40 epochs for all tests.
\nOverall, compared to some of the recent imaging competitions on CT images, this seems to be more manageable.
\n-Rich
", "votes": null, "replies": [ { "id": 1571036, "author_name": "slawekbiel", "author_url": "", "post_date": "11/04/2021 14:52:27", "content": "Hey Rich!
\nDo you have some tips for working with these large models? In my previous competitions so far I was always able to find approaches that took no longer than 15 minutes to train and my strategy was to try a lot of things until something worked.
", "votes": null, "replies": [] }, { "id": 1571070, "author_name": "richardepstein", "author_url": "", "post_date": "11/04/2021 15:17:06", "content": "I don't have specific suggestions. I always worry that parameters that might work on a \"quick\" model, might not be good for a fully run model. But I note you are doing much better in this competition than I am, so you are doing something right!
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1570708": "Hi,\nCan any one tell based on so far experience is this competition needing lot of resources ?", "1570984": "Very much so in my experience so far. I'm fortunate to own a high-end PC with a RTX3090 and still it can take a couple of hours to train a model, with close to 100% GPU utilization. That means if I wanted to do 5-fold cross validation it takes the whole day. I don't have a good idea how to approach experimenting with hyperparameters and different models.\n\nAnd that's on the competition dataset alone, where there are still the larger LIVCell and unsupervised datasets.", "1570992": "Same here. It takes 45 seconds for me train a single epoch (484 samples, 121 steps) mask r-cnn on RTX-3090.", "1571019": "Using Detectron2 and Resnet101, on the Kaggle notebooks, a run of 40 epochs or so takes about 2 hours.\n\nResnet50 is faster. Probably don't need 40 epochs for all tests.\n\nOverall, compared to some of the recent imaging competitions on CT images, this seems to be more manageable.\n\n-Rich", "1571036": "Hey Rich!\n\nDo you have some tips for working with these large models? In my previous competitions so far I was always able to find approaches that took no longer than 15 minutes to train and my strategy was to try a lot of things until something worked.", "1571070": "I don't have specific suggestions. I always worry that parameters that might work on a \"quick\" model, might not be good for a fully run model. But I note you are doing much better in this competition than I am, so you are doing something right!", "1571106": "slawekbiel Using a subset of the data is usually helpful to try different models quickly." }, "source": "meta" }