Welcome to Human BioMolecular Atlas Program (HuBMAP) competition. I have prepared several kernels for people who use Pytorch and fast.ai to get started with this challenge and for others to borrow some ideas and explore them more.
\nI hope you will enjoy this competition, and I wish you the best luck.
\n03/13/2021 The datasets and kernels are updated to accommodate for the new data.
Welcome to Human BioMolecular Atlas Program (HuBMAP) competition. I have prepared several kernels for people who use Pytorch and fast.ai to get started with this challenge and for others to borrow some ideas and explore them more.
\nI hope you will enjoy this competition, and I wish you the best luck.
\n03/13/2021 The datasets and kernels are updated to accommodate for the new data.
Very well done, thanks for sharing your pipeline
", "rawMarkdown": "Very well done, thanks for sharing your pipeline", "votes": null }, { "id": "1085710", "postDate": "11/21/2020 06:18:39", "content": "You are very welcome
", "rawMarkdown": "You are very welcome", "votes": null }, { "id": "1085798", "postDate": "11/21/2020 07:54:30", "content": "I have problems with inference to private test sets. I suspect it's because the private test set contains an image that's too big to read using the tifffile. I think we need to use a library that can load subregions of the image, such as pyvips. I would like to wait for an update of the kaggle environment.
", "rawMarkdown": "I have problems with inference to private test sets. I suspect it's because the private test set contains an image that's too big to read using the tifffile. I think we need to use a library that can load subregions of the image, such as pyvips. I would like to wait for an update of the kaggle environment.", "votes": null }, { "id": "1085805", "postDate": "11/21/2020 08:00:23", "content": "It is likely to be the case but the strange thing is that the notebook doesn't exit with \"out of memory\" error but rather runs to 9h limit and stops (actually it is likely trying to restart and face the same problem). Difficult to debug it without proper error message output.
", "rawMarkdown": "It is likely to be the case but the strange thing is that the notebook doesn't exit with \"out of memory\" error but rather runs to 9h limit and stops (actually it is likely trying to restart and face the same problem). Difficult to debug it without proper error message output.", "votes": null }, { "id": "1085835", "postDate": "11/21/2020 08:16:00", "content": "I have same problem. If an \"out of memory\" error occurs, there is case that it may run until the time expires instead of exiting immediately.
", "rawMarkdown": "I have same problem. If an \"out of memory\" error occurs, there is case that it may run until the time expires instead of exiting immediately.", "votes": null }, { "id": "1086321", "postDate": "11/21/2020 15:32:40", "content": "I've done successful commit on CPU instead of GPU which gives you +3 GB of RAM, so seems all is fine with file reading but problem with all memory usage.
", "rawMarkdown": "I've done successful commit on CPU instead of GPU which gives you +3 GB of RAM, so seems all is fine with file reading but problem with all memory usage.", "votes": null }, { "id": "1086323", "postDate": "11/21/2020 15:34:44", "content": "Thanks for sharing @iafoss
\nSince they provided anatomical structure annotations in the public test set do you think they are also available in the private test set?
You are welcome. It is better to ask organizers to make sure that your LB position will remain after the end of the competition and u do not relay on data not given for private test set. However, how the scoring is working is the following. When u click commit, the environment is changed to one used for evaluation, which provides an extended test set with 12 images and maybe extended json files (if u use them). Next the script is executed on this data, and a scoring script is run based on the submission file extracted from the output. So, if u use json and get reasonable results at public LB, likely it is also working at private LB, while if u get an error it may mean that there are no json files provided in test environment. But to be safe, it is better to doublecheck with organizers
", "rawMarkdown": "You are welcome. It is better to ask organizers to make sure that your LB position will remain after the end of the competition and u do not relay on data not given for private test set. However, how the scoring is working is the following. When u click commit, the environment is changed to one used for evaluation, which provides an extended test set with 12 images and maybe extended json files (if u use them). Next the script is executed on this data, and a scoring script is run based on the submission file extracted from the output. So, if u use json and get reasonable results at public LB, likely it is also working at private LB, while if u get an error it may mean that there are no json files provided in test environment. But to be safe, it is better to doublecheck with organizers", "votes": null }, { "id": "1086676", "postDate": "11/21/2020 23:25:18", "content": "The CPU environment has more memory than the GPU environment, so I think submitting without the GPU is currently the only successful way. But I want to use the GPU when submitting…
", "rawMarkdown": "The CPU environment has more memory than the GPU environment, so I think submitting without the GPU is currently the only successful way. But I want to use the GPU when submitting...", "votes": null }, { "id": "1087268", "postDate": "11/22/2020 15:00:34", "content": "HI @iafoss thanks! great work, little complicate hehe, lots to learn…
\nthanks
", "rawMarkdown": "HI @iafoss thanks! great work, little complicate hehe, lots to learn... \n\nthanks", "votes": null }, { "id": "1087393", "postDate": "11/22/2020 17:03:08", "content": "you are welcome @oscarrangel
", "rawMarkdown": "you are welcome @oscarrangel", "votes": null }, { "id": "1088051", "postDate": "11/23/2020 09:17:55", "content": "I've got the same problem… Still struggling on it
", "rawMarkdown": "I've got the same problem... Still struggling on it", "votes": null }, { "id": "1089513", "postDate": "11/24/2020 15:14:22", "content": "Hi @iafoss, was wondering how we can use one of these two to build a datablock?
\nds = HuBMAPDataset(tfms=get_aug())
\ndl = DataLoader(ds,batch_size=64,shuffle=False,num_workers=NUM_WORKERS)
or what will be the way to build a datablock with the 256x256 images?
\nThanks!
", "rawMarkdown": "Hi @iafoss, was wondering how we can use one of these two to build a datablock?\n\nds = HuBMAPDataset(tfms=get_aug())\ndl = DataLoader(ds,batch_size=64,shuffle=False,num_workers=NUM_WORKERS)\n\nor what will be the way to build a datablock with the 256x256 images?\n\nThanks!", "votes": null }, { "id": "1089652", "postDate": "11/24/2020 16:52:05", "content": "The part u are referring is just for plotting several image examples, the data structures for training are created later on with ImageDataLoaders.from_dsets. I found that the native fast.ai way to treat the data is quite inconvenient, so I create ImageDataLoaders from datasets directly.
", "rawMarkdown": "The part u are referring is just for plotting several image examples, the data structures for training are created later on with ImageDataLoaders.from_dsets. I found that the native fast.ai way to treat the data is quite inconvenient, so I create ImageDataLoaders from datasets directly.", "votes": null }, { "id": "1089714", "postDate": "11/24/2020 18:04:40", "content": "Hi @iafoss, thanks a lot for the hint! your model is very complex, so I am trying to learn from the beginning….. so I am trying this, but is giving me an error. is looking for the number of classes in the dataset… but it has none.
\ndata = ImageDataLoaders.from_dsets(ds_t,\n ds_v,\n bs=bs,\n num_workers=NUM_WORKERS,\n pin_memory=True).cuda()\n\nlearn = unet_learner(data, \n resnet34, \n loss_func=lovasz_hinge,\n metrics=[meanapv1]).to_fp16(clip=0.5)\n---------------------------------------------------------------------------\nAssertionError Traceback (most recent call last)\n<ipython-input-15-db744f424c39> in <module>\n----> 1 learn = unet_learner(data, \n 2 resnet34,\n 3 loss_func=lovasz_hinge,\n 4 metrics=[meanapv1]).to_fp16(clip=0.5)\n\n~/anaconda3/envs/torch/lib/python3.8/site-packages/fastai/vision/learner.py in unet_learner(dls, arch, loss_func, pretrained, cut, splitter, config, n_in, n_out, normalize, **kwargs)\n 193 size = dls.one_batch()[0].shape[-2:]\n 194 if n_out is None: n_out = get_c(dls)\n--> 195 assert n_out, \"`n_out` is not defined, and could not be inferred from data, set `dls.c` or pass `n_out`\"\n 196 if normalize: _add_norm(dls, meta, pretrained)\n 197 model = models.unet.DynamicUnet(body, n_out, size, **config)\n\nAssertionError: `n_out` is not defined, and could not be inferred from data, set `dls.c` or pass `n_out`\nI think fast.ai Unet class requires n_out to build the corresponding object. You can check if u can pass it to unet_learner (using ?? command). Another option is just set this.c = 1 in the dataset.
", "rawMarkdown": "I think fast.ai Unet class requires n_out to build the corresponding object. You can check if u can pass it to unet_learner (using ?? command). Another option is just set this.c = 1 in the dataset.", "votes": null }, { "id": "1089727", "postDate": "11/24/2020 18:17:09", "content": "@iafoss Thanks man! but did not work…. that is way I was trying to build something I can pass to different learners with different models, using the way you break the large images into pieces.
\nclass HuBMAPDataset(Dataset):\n def __init__(self, fold=fold, train=True, tfms=None):\n ids = pd.read_csv(LABELS).id.values\n kf = KFold(n_splits=nfolds,random_state=SEED,shuffle=True)\n ids = set(ids[list(kf.split(ids))[fold][0 if train else 1]])\n self.fnames = [fname for fname in os.listdir(TRAIN) if fname.split('_')[0] in ids]\n self.train = train\n self.tfms = tfms\n self.c = 1\n\n-------------------------------------\nModuleAttributeError: 'DynamicUnet' object has no attribute 'enc0'\nthe last error most likely is related to the way how u do the model split: u cannot use one from my kernel with a default Unet model
", "rawMarkdown": "the last error most likely is related to the way how u do the model split: u cannot use one from my kernel with a default Unet model", "votes": null }, { "id": "1089759", "postDate": "11/24/2020 18:48:53", "content": "Hi @iafoss, it worked!!!!! thanks a lot amigo, I really appreciate it!!! bless you! had to use 4 for the batch size on 1024x1024. amazing, I understand that it is not good to train models with such a low bz, as they wont generalized correctly, is this right ?
\nps.
\nis giving me another error, after the training starts
\nTypeError: list indices must be integers or slices, not str
I think it is a bad idea to start from large images. And I'm not sure about the last error u got.
", "rawMarkdown": "I think it is a bad idea to start from large images. And I'm not sure about the last error u got.", "votes": null }, { "id": "1089788", "postDate": "11/24/2020 19:27:42", "content": "@iafoss , Thanks! again. I see about the large images just wanted to test and see if I was going to be able to handle, (I have one RTX 24 Gigs) the size you recommended earlier.
\nYou got me in the beginning of the road to start learning. studying the other notebooks you have about splitting large images.
\nSo much to learn, I been doing this for a year and I still dont see the light at the end of the tunnel…. hahaha
", "rawMarkdown": "iafoss , Thanks! again. I see about the large images just wanted to test and see if I was going to be able to handle, (I have one RTX 24 Gigs) the size you recommended earlier.\n\nYou got me in the beginning of the road to start learning. studying the other notebooks you have about splitting large images. \n\nSo much to learn, I been doing this for a year and I still dont see the light at the end of the tunnel.... hahaha", "votes": null }, { "id": "1089791", "postDate": "11/24/2020 19:36:42", "content": "You are welcome
", "rawMarkdown": "You are welcome", "votes": null }, { "id": "1091290", "postDate": "11/25/2020 23:17:04", "content": "Folks, I myself had problems with memory in the private set. I solved it saving the image tiles to disk and then passing the data as a tf.Data.Dataset which reads serially from a file. Basically using the disk as working space. I am sure an equivalent approach can be used here:
\nhttps://www.kaggle.com/marcosnovaes/hubmap-memory-efficient-submission-using-disk
I'm using the tifffile in my prediction pipeline. (TensorFlow + GPU)
\nLoad the whole image -> Split to tiles -> Predict -> Combine the tiles -> RLE encoder
\nI think tifffile is OK, just don't forget to use del and gc.collect().
hi @iafoss , I was looking at images generated with the notebook 256x256 and I see that most of the tittles dont generate a mask, does that has matter with the classification job for the network to generalize?
\nThanks!
", "rawMarkdown": "hi @iafoss , I was looking at images generated with the notebook 256x256 and I see that most of the tittles dont generate a mask, does that has matter with the classification job for the network to generalize?\n\nThanks!", "votes": null }, { "id": "1093479", "postDate": "11/27/2020 19:01:52", "content": "You have just a few glomeruli, so it is not surprising. Also, that is why when u calculate the metric it is important to aggregate the Intersections and unions for the entire image, like the competition metric is evaluated. If you compute the metric in a tile wise manner, it is quite different task, and results may be misleading.
", "rawMarkdown": "You have just a few glomeruli, so it is not surprising. Also, that is why when u calculate the metric it is important to aggregate the Intersections and unions for the entire image, like the competition metric is evaluated. If you compute the metric in a tile wise manner, it is quite different task, and results may be misleading.", "votes": null }, { "id": "1093545", "postDate": "11/27/2020 20:12:03", "content": "do you think is better to eliminate those images tittles without mask ?
\nthanks!
\nPS.
\nwhat do you mean by this?
\n' If you compute the metric in a tile wise manner'
I'd expect it may degrade the score because u need to have had negative examples to train the model.
\nFor the second question, it is the way when u compute Dice tile by tile.
Thank you amigo.
", "rawMarkdown": "Thank you amigo.", "votes": null }, { "id": "1093879", "postDate": "11/28/2020 05:19:08", "content": "Hi @iafoss, I was wondering if you want to team up with me? lots of free time and good hardware
", "rawMarkdown": "Hi @iafoss, I was wondering if you want to team up with me? lots of free time and good hardware", "votes": null }, { "id": "1093960", "postDate": "11/28/2020 07:09:05", "content": "Currently I'm busy with other things, and it's too early stage of the competition for me to team up.
", "rawMarkdown": "Currently I'm busy with other things, and it's too early stage of the competition for me to team up.", "votes": null }, { "id": "1096470", "postDate": "11/30/2020 14:20:51", "content": "Hi @iafoss, I have asked this question in other chats but no one answered, I was wondering if you can tell me how many classes are there, as most of the pre-trained models need at least two, which makes sense, as I understand this is binary segmentation, but you tell me there is only one class….
\nCan you explain.
\nThanks again for all your help.
\nPS.
\nSorry it is my first segmentation exploration.
I think 2 is just a default option in fast.ai: they consider a binary classification as a multiclass classification with two classes. fast.ai , probably, try to use more uniform approach, but it makes things a little bit messier for binary classification( To get the predictions u will need to pass these two outputs through softmax (considering only one output as logits and taking a sigmoid wd be wrong). Though, it may require some coding to use single output with fast.ai. But overall those two things are nearly identical.
", "rawMarkdown": "I think 2 is just a default option in fast.ai: they consider a binary classification as a multiclass classification with two classes. fast.ai , probably, try to use more uniform approach, but it makes things a little bit messier for binary classification( To get the predictions u will need to pass these two outputs through softmax (considering only one output as logits and taking a sigmoid wd be wrong). Though, it may require some coding to use single output with fast.ai. But overall those two things are nearly identical.", "votes": null }, { "id": "1096740", "postDate": "11/30/2020 18:30:25", "content": "Thanks! @iafoss but your code considers only one, right ?
", "rawMarkdown": "Thanks! @iafoss but your code considers only one, right ?", "votes": null }, { "id": "1096766", "postDate": "11/30/2020 18:55:57", "content": "yes, I'm not pretending to write a code for multiclass problem here
", "rawMarkdown": "yes, I'm not pretending to write a code for multiclass problem here", "votes": null }, { "id": "1096973", "postDate": "11/30/2020 22:54:29", "content": "Hello @iafoss thanks for the information.
", "rawMarkdown": "Hello @iafoss thanks for the information.", "votes": null }, { "id": "1096979", "postDate": "11/30/2020 23:02:20", "content": "You are very welcome
", "rawMarkdown": "You are very welcome", "votes": null }, { "id": "1099175", "postDate": "12/02/2020 06:26:36", "content": "Does it work on private set? I use the same pipeline but the notebook just keep running 9h then report unknown error :(
", "rawMarkdown": "Does it work on private set? I use the same pipeline but the notebook just keep running 9h then report unknown error :(", "votes": null }, { "id": "1101186", "postDate": "12/03/2020 17:40:59", "content": "I had the same problem(Notebook Timeout).
\nAnd I was able to submit (Public and Private) if I tried to use less memory.
I have an update on the issue with private LB sub: the later version loads images on tile by tile base using rasterio. Please consider this version to make a proper sub.
", "rawMarkdown": "I have an update on the issue with private LB sub: the later version loads images on tile by tile base using rasterio. Please consider this version to make a proper sub.", "votes": null }, { "id": "1104588", "postDate": "12/07/2020 04:57:07", "content": "thanks a lot!
", "rawMarkdown": "thanks a lot!", "votes": null }, { "id": "1109580", "postDate": "12/11/2020 21:09:41", "content": "Hi @iafoss, what changes we need to make to the class UneXt50 in order to use it with higher resolutions?
\nThanks!
", "rawMarkdown": "Hi @iafoss, what changes we need to make to the class UneXt50 in order to use it with higher resolutions?\n\nThanks!", "votes": null }, { "id": "1109625", "postDate": "12/11/2020 22:24:24", "content": "You just need to give different input, no any changes are needed to the model itself
", "rawMarkdown": "You just need to give different input, no any changes are needed to the model itself", "votes": null }, { "id": "1109721", "postDate": "12/12/2020 02:07:17", "content": "Hi @iafoss thanks for the response, in what ways we can improve the model? adding more layers to the enconder/decoder ?
\nThanks!
", "rawMarkdown": "Hi @iafoss thanks for the response, in what ways we can improve the model? adding more layers to the enconder/decoder ?\n\nThanks!", "votes": null }, { "id": "1109786", "postDate": "12/12/2020 04:22:36", "content": "I think the best boost one could get by looking into the data
", "rawMarkdown": "I think the best boost one could get by looking into the data", "votes": null }, { "id": "1110176", "postDate": "12/12/2020 13:48:09", "content": "Coming from academia and the utopia of prepared datasets ready of modeling, a Senior Director of Artifical Intelligence at Tesla, he found that in the real world, the bread and butter of a deep learning system and where the blood, sweat, and tears would be shed, was in the data. Data acquisition, cleaning, preparation, and the day-to-day management thereof. This same sentiment can as much be inferred from any of you that watched Jeremy Howard's v2 walk through in late 20191… every single session was about getting your data modelable using the new v2 bits. That should tell you a lot!
", "rawMarkdown": "Coming from academia and the utopia of prepared datasets ready of modeling, a Senior Director of Artifical Intelligence at Tesla, he found that in the real world, the bread and butter of a deep learning system and where the blood, sweat, and tears would be shed, was in the data. Data acquisition, cleaning, preparation, and the day-to-day management thereof. This same sentiment can as much be inferred from any of you that watched Jeremy Howard's v2 walk through in late 20191... every single session was about getting your data modelable using the new v2 bits. That should tell you a lot!", "votes": null }, { "id": "1110330", "postDate": "12/12/2020 16:21:14", "content": "Hi @iafoss can you create a dataloader from your dataset? I been looking around the internet, but nothing found.
\nThanks!
", "rawMarkdown": "Hi @iafoss can you create a dataloader from your dataset? I been looking around the internet, but nothing found.\n\nThanks!", "votes": null }, { "id": "1110665", "postDate": "12/12/2020 23:55:59", "content": "If u are talking about Pytorch data loader, u can check my last submission kernel. You just create and pass the dataset u want to use to the DataLoader class.
", "rawMarkdown": "If u are talking about Pytorch data loader, u can check my last submission kernel. You just create and pass the dataset u want to use to the DataLoader class.", "votes": null }, { "id": "1110677", "postDate": "12/13/2020 00:32:12", "content": "At least I found that there are usually three main things to get a high performing model: (1) right concept for building the model for a particular task and data, (2) looking into the data and understanding what are the challenges and how to deal with them, and (3) proper validation to avoid getting \"shake up master\". Keep in mind that (1) requires (2). Other things, like backbones, hyperparameters, etc., are rather minor in many cases if one is using something reasonable enough here.
", "rawMarkdown": "At least I found that there are usually three main things to get a high performing model: (1) right concept for building the model for a particular task and data, (2) looking into the data and understanding what are the challenges and how to deal with them, and (3) proper validation to avoid getting \"shake up master\". Keep in mind that (1) requires (2). Other things, like backbones, hyperparameters, etc., are rather minor in many cases if one is using something reasonable enough here.", "votes": null }, { "id": "1237246", "postDate": "03/13/2021 23:19:52", "content": "I have updated datasets and starter kernels to accommodate for the new data. New LB is 0.905. Enjoy the competition, and the best luck.
", "rawMarkdown": "I have updated datasets and starter kernels to accommodate for the new data. New LB is 0.905. Enjoy the competition, and the best luck.", "votes": null }, { "id": "1242630", "postDate": "03/17/2021 18:11:48", "content": "Hi @Iafoss,
\nThank you for the updates. I have noticed when you train your model with higher resolution, at some point it start producing NaN values. I tried to debug it but without any success.
\nepoch train_loss valid_loss dice_soft dice_th time
\n0 0.515302 0.431999 0.833299 0.878014 19:45
\n…..
\n5 0.381877 0.357184 0.857088 0.897639 12:38
\n6 nan nan None 0.000000 12:15
\n7 nan nan None 0.000000 08:53
\n8 nan nan None 0.000000 07:15
Did you just rerun the kernel or changed something, like batch size? Check my recent reply regarding small bs here.
", "rawMarkdown": "Did you just rerun the kernel or changed something, like batch size? Check my recent reply regarding small bs [here](https://www.kaggle.com/iafoss/hubmap-pytorch-fast-ai-starter).", "votes": null }, { "id": "1243122", "postDate": "03/18/2021 03:19:53", "content": "The batch size 64 with image size 512x512 is not possible on my GPU. I tried with a batch size of 32 getting the same behaviour. I will read your reply, thanks though.
", "rawMarkdown": "The batch size 64 with image size 512x512 is not possible on my GPU. I tried with a batch size of 32 getting the same behaviour. I will read your reply, thanks though.", "votes": null }, { "id": "1287552", "postDate": "04/29/2021 07:11:29", "content": "thanks a lot!
", "rawMarkdown": "thanks a lot!", "votes": null }, { "id": "1287877", "postDate": "04/29/2021 13:42:20", "content": "you are welcome
", "rawMarkdown": "you are welcome", "votes": null } ], "comments": [ { "id": 1085545, "author_name": "matthewmasters", "author_url": "", "post_date": "11/21/2020 00:50:20", "content": "Very well done, thanks for sharing your pipeline
", "votes": null, "replies": [ { "id": 1085710, "author_name": "iafoss", "author_url": "", "post_date": "11/21/2020 06:18:39", "content": "You are very welcome
", "votes": null, "replies": [] } ] }, { "id": 1085798, "author_name": "hirune924", "author_url": "", "post_date": "11/21/2020 07:54:30", "content": "I have problems with inference to private test sets. I suspect it's because the private test set contains an image that's too big to read using the tifffile. I think we need to use a library that can load subregions of the image, such as pyvips. I would like to wait for an update of the kaggle environment.
", "votes": null, "replies": [ { "id": 1085805, "author_name": "iafoss", "author_url": "", "post_date": "11/21/2020 08:00:23", "content": "It is likely to be the case but the strange thing is that the notebook doesn't exit with \"out of memory\" error but rather runs to 9h limit and stops (actually it is likely trying to restart and face the same problem). Difficult to debug it without proper error message output.
", "votes": null, "replies": [] }, { "id": 1085835, "author_name": "hirune924", "author_url": "", "post_date": "11/21/2020 08:16:00", "content": "I have same problem. If an \"out of memory\" error occurs, there is case that it may run until the time expires instead of exiting immediately.
", "votes": null, "replies": [] }, { "id": 1086321, "author_name": "hypocrites", "author_url": "", "post_date": "11/21/2020 15:32:40", "content": "I've done successful commit on CPU instead of GPU which gives you +3 GB of RAM, so seems all is fine with file reading but problem with all memory usage.
", "votes": null, "replies": [] }, { "id": 1086676, "author_name": "hirune924", "author_url": "", "post_date": "11/21/2020 23:25:18", "content": "The CPU environment has more memory than the GPU environment, so I think submitting without the GPU is currently the only successful way. But I want to use the GPU when submitting…
", "votes": null, "replies": [] }, { "id": 1088051, "author_name": "haqishen", "author_url": "", "post_date": "11/23/2020 09:17:55", "content": "I've got the same problem… Still struggling on it
", "votes": null, "replies": [] }, { "id": 1091290, "author_name": "marcosnovaes", "author_url": "", "post_date": "11/25/2020 23:17:04", "content": "Folks, I myself had problems with memory in the private set. I solved it saving the image tiles to disk and then passing the data as a tf.Data.Dataset which reads serially from a file. Basically using the disk as working space. I am sure an equivalent approach can be used here:
\nhttps://www.kaggle.com/marcosnovaes/hubmap-memory-efficient-submission-using-disk
I'm using the tifffile in my prediction pipeline. (TensorFlow + GPU)
\nLoad the whole image -> Split to tiles -> Predict -> Combine the tiles -> RLE encoder
\nI think tifffile is OK, just don't forget to use del and gc.collect().
Does it work on private set? I use the same pipeline but the notebook just keep running 9h then report unknown error :(
", "votes": null, "replies": [] }, { "id": 1101186, "author_name": "yukkyo", "author_url": "", "post_date": "12/03/2020 17:40:59", "content": "I had the same problem(Notebook Timeout).
\nAnd I was able to submit (Public and Private) if I tried to use less memory.
Thanks for sharing @iafoss
\nSince they provided anatomical structure annotations in the public test set do you think they are also available in the private test set?
You are welcome. It is better to ask organizers to make sure that your LB position will remain after the end of the competition and u do not relay on data not given for private test set. However, how the scoring is working is the following. When u click commit, the environment is changed to one used for evaluation, which provides an extended test set with 12 images and maybe extended json files (if u use them). Next the script is executed on this data, and a scoring script is run based on the submission file extracted from the output. So, if u use json and get reasonable results at public LB, likely it is also working at private LB, while if u get an error it may mean that there are no json files provided in test environment. But to be safe, it is better to doublecheck with organizers
", "votes": null, "replies": [] } ] }, { "id": 1087268, "author_name": "oscarrangel", "author_url": "", "post_date": "11/22/2020 15:00:34", "content": "HI @iafoss thanks! great work, little complicate hehe, lots to learn…
\nthanks
", "votes": null, "replies": [ { "id": 1087393, "author_name": "iafoss", "author_url": "", "post_date": "11/22/2020 17:03:08", "content": "you are welcome @oscarrangel
", "votes": null, "replies": [] } ] }, { "id": 1089513, "author_name": "oscarrangel", "author_url": "", "post_date": "11/24/2020 15:14:22", "content": "Hi @iafoss, was wondering how we can use one of these two to build a datablock?
\nds = HuBMAPDataset(tfms=get_aug())
\ndl = DataLoader(ds,batch_size=64,shuffle=False,num_workers=NUM_WORKERS)
or what will be the way to build a datablock with the 256x256 images?
\nThanks!
", "votes": null, "replies": [ { "id": 1089652, "author_name": "iafoss", "author_url": "", "post_date": "11/24/2020 16:52:05", "content": "The part u are referring is just for plotting several image examples, the data structures for training are created later on with ImageDataLoaders.from_dsets. I found that the native fast.ai way to treat the data is quite inconvenient, so I create ImageDataLoaders from datasets directly.
", "votes": null, "replies": [] }, { "id": 1089714, "author_name": "oscarrangel", "author_url": "", "post_date": "11/24/2020 18:04:40", "content": "Hi @iafoss, thanks a lot for the hint! your model is very complex, so I am trying to learn from the beginning….. so I am trying this, but is giving me an error. is looking for the number of classes in the dataset… but it has none.
\ndata = ImageDataLoaders.from_dsets(ds_t,\n ds_v,\n bs=bs,\n num_workers=NUM_WORKERS,\n pin_memory=True).cuda()\n\nlearn = unet_learner(data, \n resnet34, \n loss_func=lovasz_hinge,\n metrics=[meanapv1]).to_fp16(clip=0.5)\n---------------------------------------------------------------------------\nAssertionError Traceback (most recent call last)\n<ipython-input-15-db744f424c39> in <module>\n----> 1 learn = unet_learner(data, \n 2 resnet34,\n 3 loss_func=lovasz_hinge,\n 4 metrics=[meanapv1]).to_fp16(clip=0.5)\n\n~/anaconda3/envs/torch/lib/python3.8/site-packages/fastai/vision/learner.py in unet_learner(dls, arch, loss_func, pretrained, cut, splitter, config, n_in, n_out, normalize, **kwargs)\n 193 size = dls.one_batch()[0].shape[-2:]\n 194 if n_out is None: n_out = get_c(dls)\n--> 195 assert n_out, \"`n_out` is not defined, and could not be inferred from data, set `dls.c` or pass `n_out`\"\n 196 if normalize: _add_norm(dls, meta, pretrained)\n 197 model = models.unet.DynamicUnet(body, n_out, size, **config)\n\nAssertionError: `n_out` is not defined, and could not be inferred from data, set `dls.c` or pass `n_out`\nI think fast.ai Unet class requires n_out to build the corresponding object. You can check if u can pass it to unet_learner (using ?? command). Another option is just set this.c = 1 in the dataset.
", "votes": null, "replies": [] }, { "id": 1089727, "author_name": "oscarrangel", "author_url": "", "post_date": "11/24/2020 18:17:09", "content": "@iafoss Thanks man! but did not work…. that is way I was trying to build something I can pass to different learners with different models, using the way you break the large images into pieces.
\nclass HuBMAPDataset(Dataset):\n def __init__(self, fold=fold, train=True, tfms=None):\n ids = pd.read_csv(LABELS).id.values\n kf = KFold(n_splits=nfolds,random_state=SEED,shuffle=True)\n ids = set(ids[list(kf.split(ids))[fold][0 if train else 1]])\n self.fnames = [fname for fname in os.listdir(TRAIN) if fname.split('_')[0] in ids]\n self.train = train\n self.tfms = tfms\n self.c = 1\n\n-------------------------------------\nModuleAttributeError: 'DynamicUnet' object has no attribute 'enc0'\nthe last error most likely is related to the way how u do the model split: u cannot use one from my kernel with a default Unet model
", "votes": null, "replies": [] }, { "id": 1089759, "author_name": "oscarrangel", "author_url": "", "post_date": "11/24/2020 18:48:53", "content": "Hi @iafoss, it worked!!!!! thanks a lot amigo, I really appreciate it!!! bless you! had to use 4 for the batch size on 1024x1024. amazing, I understand that it is not good to train models with such a low bz, as they wont generalized correctly, is this right ?
\nps.
\nis giving me another error, after the training starts
\nTypeError: list indices must be integers or slices, not str
I think it is a bad idea to start from large images. And I'm not sure about the last error u got.
", "votes": null, "replies": [] }, { "id": 1089788, "author_name": "oscarrangel", "author_url": "", "post_date": "11/24/2020 19:27:42", "content": "@iafoss , Thanks! again. I see about the large images just wanted to test and see if I was going to be able to handle, (I have one RTX 24 Gigs) the size you recommended earlier.
\nYou got me in the beginning of the road to start learning. studying the other notebooks you have about splitting large images.
\nSo much to learn, I been doing this for a year and I still dont see the light at the end of the tunnel…. hahaha
", "votes": null, "replies": [] }, { "id": 1089791, "author_name": "iafoss", "author_url": "", "post_date": "11/24/2020 19:36:42", "content": "You are welcome
", "votes": null, "replies": [] } ] }, { "id": 1093446, "author_name": "oscarrangel", "author_url": "", "post_date": "11/27/2020 18:11:00", "content": "hi @iafoss , I was looking at images generated with the notebook 256x256 and I see that most of the tittles dont generate a mask, does that has matter with the classification job for the network to generalize?
\nThanks!
", "votes": null, "replies": [ { "id": 1093479, "author_name": "iafoss", "author_url": "", "post_date": "11/27/2020 19:01:52", "content": "You have just a few glomeruli, so it is not surprising. Also, that is why when u calculate the metric it is important to aggregate the Intersections and unions for the entire image, like the competition metric is evaluated. If you compute the metric in a tile wise manner, it is quite different task, and results may be misleading.
", "votes": null, "replies": [] }, { "id": 1093545, "author_name": "oscarrangel", "author_url": "", "post_date": "11/27/2020 20:12:03", "content": "do you think is better to eliminate those images tittles without mask ?
\nthanks!
\nPS.
\nwhat do you mean by this?
\n' If you compute the metric in a tile wise manner'
I'd expect it may degrade the score because u need to have had negative examples to train the model.
\nFor the second question, it is the way when u compute Dice tile by tile.
Thank you amigo.
", "votes": null, "replies": [] } ] }, { "id": 1093879, "author_name": "oscarrangel", "author_url": "", "post_date": "11/28/2020 05:19:08", "content": "Hi @iafoss, I was wondering if you want to team up with me? lots of free time and good hardware
", "votes": null, "replies": [ { "id": 1093960, "author_name": "iafoss", "author_url": "", "post_date": "11/28/2020 07:09:05", "content": "Currently I'm busy with other things, and it's too early stage of the competition for me to team up.
", "votes": null, "replies": [] } ] }, { "id": 1096470, "author_name": "oscarrangel", "author_url": "", "post_date": "11/30/2020 14:20:51", "content": "Hi @iafoss, I have asked this question in other chats but no one answered, I was wondering if you can tell me how many classes are there, as most of the pre-trained models need at least two, which makes sense, as I understand this is binary segmentation, but you tell me there is only one class….
\nCan you explain.
\nThanks again for all your help.
\nPS.
\nSorry it is my first segmentation exploration.
I think 2 is just a default option in fast.ai: they consider a binary classification as a multiclass classification with two classes. fast.ai , probably, try to use more uniform approach, but it makes things a little bit messier for binary classification( To get the predictions u will need to pass these two outputs through softmax (considering only one output as logits and taking a sigmoid wd be wrong). Though, it may require some coding to use single output with fast.ai. But overall those two things are nearly identical.
", "votes": null, "replies": [] }, { "id": 1096740, "author_name": "oscarrangel", "author_url": "", "post_date": "11/30/2020 18:30:25", "content": "Thanks! @iafoss but your code considers only one, right ?
", "votes": null, "replies": [] }, { "id": 1096766, "author_name": "iafoss", "author_url": "", "post_date": "11/30/2020 18:55:57", "content": "yes, I'm not pretending to write a code for multiclass problem here
", "votes": null, "replies": [] } ] }, { "id": 1096973, "author_name": "puzuwe", "author_url": "", "post_date": "11/30/2020 22:54:29", "content": "Hello @iafoss thanks for the information.
", "votes": null, "replies": [ { "id": 1096979, "author_name": "iafoss", "author_url": "", "post_date": "11/30/2020 23:02:20", "content": "You are very welcome
", "votes": null, "replies": [] } ] }, { "id": 1102414, "author_name": "iafoss", "author_url": "", "post_date": "12/04/2020 22:37:21", "content": "I have an update on the issue with private LB sub: the later version loads images on tile by tile base using rasterio. Please consider this version to make a proper sub.
", "votes": null, "replies": [ { "id": 1104588, "author_name": "underwearfitting", "author_url": "", "post_date": "12/07/2020 04:57:07", "content": "thanks a lot!
", "votes": null, "replies": [] } ] }, { "id": 1109580, "author_name": "oscarrangel", "author_url": "", "post_date": "12/11/2020 21:09:41", "content": "Hi @iafoss, what changes we need to make to the class UneXt50 in order to use it with higher resolutions?
\nThanks!
", "votes": null, "replies": [ { "id": 1109625, "author_name": "iafoss", "author_url": "", "post_date": "12/11/2020 22:24:24", "content": "You just need to give different input, no any changes are needed to the model itself
", "votes": null, "replies": [] }, { "id": 1109721, "author_name": "oscarrangel", "author_url": "", "post_date": "12/12/2020 02:07:17", "content": "Hi @iafoss thanks for the response, in what ways we can improve the model? adding more layers to the enconder/decoder ?
\nThanks!
", "votes": null, "replies": [] }, { "id": 1109786, "author_name": "iafoss", "author_url": "", "post_date": "12/12/2020 04:22:36", "content": "I think the best boost one could get by looking into the data
", "votes": null, "replies": [] }, { "id": 1110176, "author_name": "oscarrangel", "author_url": "", "post_date": "12/12/2020 13:48:09", "content": "Coming from academia and the utopia of prepared datasets ready of modeling, a Senior Director of Artifical Intelligence at Tesla, he found that in the real world, the bread and butter of a deep learning system and where the blood, sweat, and tears would be shed, was in the data. Data acquisition, cleaning, preparation, and the day-to-day management thereof. This same sentiment can as much be inferred from any of you that watched Jeremy Howard's v2 walk through in late 20191… every single session was about getting your data modelable using the new v2 bits. That should tell you a lot!
", "votes": null, "replies": [] }, { "id": 1110677, "author_name": "iafoss", "author_url": "", "post_date": "12/13/2020 00:32:12", "content": "At least I found that there are usually three main things to get a high performing model: (1) right concept for building the model for a particular task and data, (2) looking into the data and understanding what are the challenges and how to deal with them, and (3) proper validation to avoid getting \"shake up master\". Keep in mind that (1) requires (2). Other things, like backbones, hyperparameters, etc., are rather minor in many cases if one is using something reasonable enough here.
", "votes": null, "replies": [] } ] }, { "id": 1110330, "author_name": "oscarrangel", "author_url": "", "post_date": "12/12/2020 16:21:14", "content": "Hi @iafoss can you create a dataloader from your dataset? I been looking around the internet, but nothing found.
\nThanks!
", "votes": null, "replies": [ { "id": 1110665, "author_name": "iafoss", "author_url": "", "post_date": "12/12/2020 23:55:59", "content": "If u are talking about Pytorch data loader, u can check my last submission kernel. You just create and pass the dataset u want to use to the DataLoader class.
", "votes": null, "replies": [] } ] }, { "id": 1237246, "author_name": "iafoss", "author_url": "", "post_date": "03/13/2021 23:19:52", "content": "I have updated datasets and starter kernels to accommodate for the new data. New LB is 0.905. Enjoy the competition, and the best luck.
", "votes": null, "replies": [ { "id": 1242630, "author_name": "aramisvesal", "author_url": "", "post_date": "03/17/2021 18:11:48", "content": "Hi @Iafoss,
\nThank you for the updates. I have noticed when you train your model with higher resolution, at some point it start producing NaN values. I tried to debug it but without any success.
\nepoch train_loss valid_loss dice_soft dice_th time
\n0 0.515302 0.431999 0.833299 0.878014 19:45
\n…..
\n5 0.381877 0.357184 0.857088 0.897639 12:38
\n6 nan nan None 0.000000 12:15
\n7 nan nan None 0.000000 08:53
\n8 nan nan None 0.000000 07:15
Did you just rerun the kernel or changed something, like batch size? Check my recent reply regarding small bs here.
", "votes": null, "replies": [] }, { "id": 1243122, "author_name": "aramisvesal", "author_url": "", "post_date": "03/18/2021 03:19:53", "content": "The batch size 64 with image size 512x512 is not possible on my GPU. I tried with a batch size of 32 getting the same behaviour. I will read your reply, thanks though.
", "votes": null, "replies": [] } ] }, { "id": 1287552, "author_name": "sasakipage", "author_url": "", "post_date": "04/29/2021 07:11:29", "content": "thanks a lot!
", "votes": null, "replies": [ { "id": 1287877, "author_name": "iafoss", "author_url": "", "post_date": "04/29/2021 13:42:20", "content": "you are welcome
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1085502": "Welcome to Human BioMolecular Atlas Program (HuBMAP) competition. I have prepared several kernels for people who use Pytorch and fast.ai to get started with this challenge and for others to borrow some ideas and explore them more. \n- https://www.kaggle.com/iafoss/256x256-images : the kernel used to cut the original images into tiles and save them to a dataset\n- https://www.kaggle.com/iafoss/hubmap-fast-ai-starter : the training kernel sharing my experience in the previous segmentation competitions and giving a Pytorch/fast.ai pipeline to start with\n- https://www.kaggle.com/iafoss/hubmap-fast-ai-starter-sub : the starter inference kernel that got 0.905 LB on 4 times lower resolution images (256x256 tile dataset)\n\nI hope you will enjoy this competition, and I wish you the best luck. \n03/13/2021 The datasets and kernels are updated to accommodate for the new data.", "1085545": "Very well done, thanks for sharing your pipeline", "1085710": "You are very welcome", "1085798": "I have problems with inference to private test sets. I suspect it's because the private test set contains an image that's too big to read using the tifffile. I think we need to use a library that can load subregions of the image, such as pyvips. I would like to wait for an update of the kaggle environment.", "1085805": "It is likely to be the case but the strange thing is that the notebook doesn't exit with \"out of memory\" error but rather runs to 9h limit and stops (actually it is likely trying to restart and face the same problem). Difficult to debug it without proper error message output.", "1085835": "I have same problem. If an \"out of memory\" error occurs, there is case that it may run until the time expires instead of exiting immediately.", "1086321": "I've done successful commit on CPU instead of GPU which gives you +3 GB of RAM, so seems all is fine with file reading but problem with all memory usage.", "1086323": "Thanks for sharing @iafoss \nSince they provided anatomical structure annotations in the public test set do you think they are also available in the private test set?", "1086478": "You are welcome. It is better to ask organizers to make sure that your LB position will remain after the end of the competition and u do not relay on data not given for private test set. However, how the scoring is working is the following. When u click commit, the environment is changed to one used for evaluation, which provides an extended test set with 12 images and maybe extended json files (if u use them). Next the script is executed on this data, and a scoring script is run based on the submission file extracted from the output. So, if u use json and get reasonable results at public LB, likely it is also working at private LB, while if u get an error it may mean that there are no json files provided in test environment. But to be safe, it is better to doublecheck with organizers", "1086676": "The CPU environment has more memory than the GPU environment, so I think submitting without the GPU is currently the only successful way. But I want to use the GPU when submitting...", "1087268": "HI @iafoss thanks! great work, little complicate hehe, lots to learn... \n\nthanks", "1087393": "you are welcome @oscarrangel", "1088051": "I've got the same problem... Still struggling on it", "1089513": "Hi @iafoss, was wondering how we can use one of these two to build a datablock?\n\nds = HuBMAPDataset(tfms=get_aug())\ndl = DataLoader(ds,batch_size=64,shuffle=False,num_workers=NUM_WORKERS)\n\nor what will be the way to build a datablock with the 256x256 images?\n\nThanks!", "1089652": "The part u are referring is just for plotting several image examples, the data structures for training are created later on with ImageDataLoaders.from_dsets. I found that the native fast.ai way to treat the data is quite inconvenient, so I create ImageDataLoaders from datasets directly.", "1089714": "Hi @iafoss, thanks a lot for the hint! your model is very complex, so I am trying to learn from the beginning..... so I am trying this, but is giving me an error. is looking for the number of classes in the dataset... but it has none.\n~~~\ndata = ImageDataLoaders.from_dsets(ds_t,\n ds_v,\n bs=bs,\n num_workers=NUM_WORKERS,\n pin_memory=True).cuda()\n\nlearn = unet_learner(data, \n resnet34, \n loss_func=lovasz_hinge,\n metrics=[meanapv1]).to_fp16(clip=0.5)\n---------------------------------------------------------------------------\nAssertionError Traceback (most recent call last)\n