Hi all!
\n\nI was inspired by @Radek generously sharing his quickdraw starter code, so I wanted to share the code I put together yesterday to enter this competition using the new version of the fastai library:
\n\nhttps://github.com/wdhorton/protein-atlas-fastai
\n\nI got 0.380 LB with my submission this morning, so this won't get you close to medaling, but I hope it's enough to get you started, and save you some time digging through the fastai code trying to get it to work with this dataset. This is a minimal starter, so there are definitely things you can tweak and play around with to try to improve your results.
\n\n(fastai v1 is not available in Kernels yet, but there is already a great Kernel based on the 0.7 version of the library https://www.kaggle.com/iafoss/pretrained-resnet34-with-rgby-0-460-public-lb which I borrowed some tricks from.)
", "messages": [ { "id": 418232, "postDate": "2018-11-09T14:11:38.977Z", "content": "Hi all!
\n\nI was inspired by @Radek generously sharing his quickdraw starter code, so I wanted to share the code I put together yesterday to enter this competition using the new version of the fastai library:
\n\nhttps://github.com/wdhorton/protein-atlas-fastai
\n\nI got 0.380 LB with my submission this morning, so this won't get you close to medaling, but I hope it's enough to get you started, and save you some time digging through the fastai code trying to get it to work with this dataset. This is a minimal starter, so there are definitely things you can tweak and play around with to try to improve your results.
\n\n(fastai v1 is not available in Kernels yet, but there is already a great Kernel based on the 0.7 version of the library https://www.kaggle.com/iafoss/pretrained-resnet34-with-rgby-0-460-public-lb which I borrowed some tricks from.)
", "rawMarkdown": "Hi all!\n\nI was inspired by @Radek generously sharing his [quickdraw starter code](https://github.com/radekosmulski/quickdraw), so I wanted to share the code I put together yesterday to enter this competition using the new version of the fastai library:\n\nhttps://github.com/wdhorton/protein-atlas-fastai\n\nI got 0.380 LB with my submission this morning, so this won't get you close to medaling, but I hope it's enough to get you started, and save you some time digging through the fastai code trying to get it to work with this dataset. This is a minimal starter, so there are definitely things you can tweak and play around with to try to improve your results.\n\n(fastai v1 is not available in Kernels yet, but there is already a great Kernel based on the 0.7 version of the library https://www.kaggle.com/iafoss/pretrained-resnet34-with-rgby-0-460-public-lb which I borrowed some tricks from.)", "votes": 58 }, { "id": 418351, "postDate": "2018-11-09T17:51:25.400Z", "content": "I think that if you use AdaptiveConcatPool2d and add extra layers on the top in resnet.py, your model may work better. You can check what fast.ai is doing by default with the model head.
", "rawMarkdown": "I think that if you use AdaptiveConcatPool2d and add extra layers on the top in resnet.py, your model may work better. You can check what fast.ai is doing by default with the model head.", "votes": 3, "replies": [ { "id": 418383, "postDate": "2018-11-09T20:17:06.790Z", "content": "Something like this:
\n\n self.out = nn.Sequential(nn.BatchNorm1d(self.bottom_channel_nr*2),\n nn.Dropout(p=self.dropout_1d, inplace=True),\n nn.Linear(self.bottom_channel_nr*2, self.bottom_channel_nr, bias=True),\n nn.ReLU(inplace=True),\n nn.BatchNorm1d(self.bottom_channel_nr),\n nn.Dropout(p=self.dropout_1d, inplace=True),\n nn.Linear(self.bottom_channel_nr, 28, bias=True)\n )\nAlso you can change the input43 layer to go from 4 to 64 channels instead and use it in place of the first resnet layer.
", "rawMarkdown": "Something like this:\n\n self.out = nn.Sequential(nn.BatchNorm1d(self.bottom_channel_nr*2),\n nn.Dropout(p=self.dropout_1d, inplace=True),\n nn.Linear(self.bottom_channel_nr*2, self.bottom_channel_nr, bias=True),\n nn.ReLU(inplace=True),\n nn.BatchNorm1d(self.bottom_channel_nr),\n nn.Dropout(p=self.dropout_1d, inplace=True),\n nn.Linear(self.bottom_channel_nr, 28, bias=True)\n )\n\nAlso you can change the input43 layer to go from 4 to 64 channels instead and use it in place of the first resnet layer." }, { "id": 418391, "postDate": "2018-11-09T20:31:00.693Z", "content": "Thank you for the suggestion. It’s definitely on my list of things to try next. I think I should be able to get the fastai default classification head in v1 by using the create_cnn function.
", "rawMarkdown": "Thank you for the suggestion. It’s definitely on my list of things to try next. I think I should be able to get the fastai default classification head in v1 by using the create_cnn function." }, { "id": 421224, "postDate": "2018-11-14T18:50:42.663Z", "content": "Great input for everyone working with fastai!
\n\nThank you @lafoss for your ship detection challenge kernals and @William Horton for this one
\n\nThey have been very helpful for getting into these competitions :D
", "rawMarkdown": "Great input for everyone working with fastai!\n\nThank you [@lafoss](https://www.kaggle.com/iafoss) for your ship detection challenge kernals and [@William Horton](https://www.kaggle.com/hortonhearsafoo) for this one\n\nThey have been very helpful for getting into these competitions :D" }, { "id": 428030, "postDate": "2018-11-26T16:34:19.783Z", "content": "Where do you learn how to put 'extra layers on the top' and 'change the input43 layer to go from 4 to 64 channels'?
\nDid Jeremy do something like this in a FastAI course somewhere?
\nMy top down education is not getting down there.
I look in the resnet.py file. I see things that are parts of the model. I type 'learn' before training and I see a model. I enter learn.unfreeze() and look at the model again and it looks unchanged. Kind of opaque as to how to modify this in any way. I changed a few lines in renset.py and dims don't match or it says such and such is not a layer in resnet.
", "rawMarkdown": "Where do you learn how to put 'extra layers on the top' and 'change the input43 layer to go from 4 to 64 channels'? \nDid Jeremy do something like this in a FastAI course somewhere? \nMy top down education is not getting down there. \n\nI look in the resnet.py file. I see things that are parts of the model. I type 'learn' before training and I see a model. I enter learn.unfreeze() and look at the model again and it looks unchanged. Kind of opaque as to how to modify this in any way. I changed a few lines in renset.py and dims don't match or it says such and such is not a layer in resnet." }, { "id": 428214, "postDate": "2018-11-26T23:38:25.810Z", "content": "@Telemachus I don’t think it’s covered in the fastai course itself.
\n\nI first learned about modifying the first conv layer of resnet from the discussion and Kernels in the TGS segmentation challenge, as some people reported improvements when changing the stride from 2 to 1.
\n\nFor this competition, I also took a look at the excellent kernel by @Iafoss
", "rawMarkdown": "@Telemachus I don’t think it’s covered in the fastai course itself.\n\nI first learned about modifying the first conv layer of resnet from the discussion and Kernels in the TGS segmentation challenge, as some people reported improvements when changing the stride from 2 to 1.\n\nFor this competition, I also took a look at the excellent kernel by @Iafoss", "votes": 1 }, { "id": 428739, "postDate": "2018-11-27T19:44:17.613Z", "content": "Thank you for that notebook. I see where the 4 channel part is and somewhat how that works. I got the part about stride and padding down.
\nIt's just that earlier comment 'use AdaptiveConcatPool2d and add extra layers on the top' by @lafoss I don't know how to work with. I'm familiar with how to change components, length, and widths of models and their layers when making a new one like with the FastAI Rossman notebook or something similar but I don't know what direction to look for adding layers or changing a pretrained model. This is pretty different than learning regular programming where you can break anything or get any stage to print and see what's happening to the bits.
\nI used an old Fastai version on lafoss's notebook that loaded the dependancies so I could play around with it but 'tfm_y' kept throwing an error so near the beginning so I gave up. Google didn't return anyone having that problem on StackOverflow.
I'm whining but at least I'm not asking for the answers. It's just working toward understanding in order to accomplish something that doesn't have any footholds.\nThere is a mountain of comments in the Salt challenge.
", "rawMarkdown": "Thank you for that notebook. I see where the 4 channel part is and somewhat how that works. I got the part about stride and padding down. \nIt's just that earlier comment 'use AdaptiveConcatPool2d and add extra layers on the top' by @lafoss I don't know how to work with. I'm familiar with how to change components, length, and widths of models and their layers when making a new one like with the FastAI Rossman notebook or something similar but I don't know what direction to look for adding layers or changing a pretrained model. This is pretty different than learning regular programming where you can break anything or get any stage to print and see what's happening to the bits. \nI used an old Fastai version on lafoss's notebook that loaded the dependancies so I could play around with it but 'tfm_y' kept throwing an error so near the beginning so I gave up. Google didn't return anyone having that problem on StackOverflow.\n\nI'm whining but at least I'm not asking for the answers. It's just working toward understanding in order to accomplish something that doesn't have any footholds.\nThere is a mountain of comments in the Salt challenge." }, { "id": 428912, "postDate": "2018-11-28T04:13:30.867Z", "content": "@Telemachus I admire your commitment! In terms of adding extra layers to the top, I'll point you to the create_cnn function in the fastai v1 library (https://github.com/fastai/fastai/blob/master/fastai/vision/learner.py#L48). If you start by studying that, you can begin to understand how to split up a model and add some of your own layers. I'd also suggest taking a look at create_head (https://github.com/fastai/fastai/blob/master/fastai/vision/learner.py#L35) to see specifically how it does the \"use AdaptiveConcatPool2d and add extra layers on the top\" part.
In terms of lessons from the Salt challenge, you can take a look at the code from the Neptune open solution to see how they took a pretrained resnet and used it to create a more complex Unet-like architecture: https://github.com/neptune-ml/open-solution-salt-identification/blob/master/common_blocks/unet_models.py#L78
", "rawMarkdown": "In terms of lessons from the Salt challenge, you can take a look at the code from the Neptune open solution to see how they took a pretrained resnet and used it to create a more complex Unet-like architecture: https://github.com/neptune-ml/open-solution-salt-identification/blob/master/common_blocks/unet_models.py#L78", "votes": 1 }, { "id": 429507, "postDate": "2018-11-29T00:29:53.650Z", "content": "Thank you. Those responses were above and beyond.
\n\nI see FastAI has docs for these functions. I'll leave the link here.
", "rawMarkdown": "Thank you. Those responses were above and beyond.\n\nI see FastAI has docs for these functions. I'll leave the link [here](https://docs.fast.ai/vision.learner.html#create_cnn).", "votes": 1 } ] }, { "id": 438184, "postDate": "2018-12-13T08:16:10.070Z", "content": "Excellent kernel, I think with 2019 version of Fast.ai course most of the people even in Kaggle will move to Fast.ai v1.
", "rawMarkdown": "Excellent kernel, I think with 2019 version of Fast.ai course most of the people even in Kaggle will move to Fast.ai v1.", "votes": 1 }, { "id": 423279, "postDate": "2018-11-17T20:53:37.380Z", "content": "UPDATE: the resnet50 basic notebook doesn't work with fastai version 1.0.25 and above, so I made another notebook to work with the new data block API. In this version, I also made changes to use the create cnn function. You can find it at resnet50 basic datablocks.ipynb.
", "rawMarkdown": "UPDATE: the resnet50 basic notebook doesn't work with fastai version 1.0.25 and above, so I made another notebook to work with the new data block API. In this version, I also made changes to use the create cnn function. You can find it at resnet50 basic datablocks.ipynb.", "votes": 1, "replies": [ { "id": 423690, "postDate": "2018-11-18T22:04:56.333Z", "content": "Thank you William, appreciate very much, will give it a go. This is a great one for me to learn so much on competitions and the tools we're learning to use in fastai.
", "rawMarkdown": "Thank you William, appreciate very much, will give it a go. This is a great one for me to learn so much on competitions and the tools we're learning to use in fastai.", "votes": 1 } ] }, { "id": 424967, "postDate": "2018-11-21T00:11:39.157Z", "content": "Made some more fixes last night based on the latest version of the library. I'm trying to keep up with the pace of development but I might have to stop updating the fastai library every week if I want to focus on this competition.
", "rawMarkdown": "Made some more fixes last night based on the latest version of the library. I'm trying to keep up with the pace of development but I might have to stop updating the fastai library every week if I want to focus on this competition.", "votes": 2 }, { "id": 423285, "postDate": "2018-11-17T21:11:28.720Z", "content": "small example of fastai v1 https://www.kaggle.com/dromosys/fast-ai-mnst
", "rawMarkdown": "small example of fastai v1 https://www.kaggle.com/dromosys/fast-ai-mnst" }, { "id": 477892, "postDate": "2019-02-25T12:33:59.423Z", "content": "hey William, thank you for sharing this, in your Kernel I keep getting this error \"got an unexpected keyword argument 'sep'\" on the ImageItemList.from_csv instruction at \"label_from_df(sep=' '.... any hints? thank u ;)
", "rawMarkdown": "hey William, thank you for sharing this, in your Kernel I keep getting this error \"got an unexpected keyword argument 'sep'\" on the ImageItemList.from_csv instruction at \"label_from_df(sep=' '.... any hints? thank u ;)", "replies": [ { "id": 477992, "postDate": "2019-02-25T15:38:56.590Z", "content": "I think it has to do with an update to the version of fastai used in the Kernels. I’ll try to see if I can figure it out tonight
", "rawMarkdown": "I think it has to do with an update to the version of fastai used in the Kernels. I’ll try to see if I can figure it out tonight", "votes": 1 }, { "id": 478055, "postDate": "2019-02-25T17:02:09.003Z", "content": "hey I found it, yes it was that, its label_delim instead of sep, thank u William
", "rawMarkdown": "hey I found it, yes it was that, its label_delim instead of sep, thank u William", "votes": 1 } ] }, { "id": 454466, "postDate": "2019-01-11T17:15:10.723Z", "content": "I get an error when running this line preds,_ = learn.get_preds(DatasetType.Test). The error I get is: \n File \"/home/devon/fastai/courses/kaggle/protein-atlas-fastai/utils.py\", line 16, in open_4_channel\n for color in colors]\n File \"/home/devon/fastai/courses/kaggle/protein-atlas-fastai/utils.py\", line 16, in \n for color in colors]\nAttributeError: 'NoneType' object has no attribute 'astype'
\n\nany ideas?
", "rawMarkdown": "I get an error when running this line preds,_ = learn.get_preds(DatasetType.Test). The error I get is: \n File \"/home/devon/fastai/courses/kaggle/protein-atlas-fastai/utils.py\", line 16, in open_4_channel\n for color in colors]\n File \"/home/devon/fastai/courses/kaggle/protein-atlas-fastai/utils.py\", line 16, in ", "replies": [ { "id": 454553, "postDate": "2019-01-11T20:24:51.987Z", "content": "It likely means some files are missing. Make sure all of the test images are in the right directory
", "rawMarkdown": "It likely means some files are missing. Make sure all of the test images are in the right directory" }, { "id": 454656, "postDate": "2019-01-11T23:35:43.300Z", "content": "Thanks for the reply. I double checked everything, and seems fine. I have a test and train folder within protein-atlas-fastai. I verified I have python3.6 and pytorch1.0.0.dev20190111. Kinda puzzled. I know you're busy and don't expect further help. Maybe other folks have some ideas.
", "rawMarkdown": "Thanks for the reply. I double checked everything, and seems fine. I have a test and train folder within protein-atlas-fastai. I verified I have python3.6 and pytorch1.0.0.dev20190111. Kinda puzzled. I know you're busy and don't expect further help. Maybe other folks have some ideas. " }, { "id": 454677, "postDate": "2019-01-12T01:15:34.740Z", "content": "Figured it out. Thanks so much for the help!
", "rawMarkdown": "Figured it out. Thanks so much for the help!" } ] }, { "id": 443861, "postDate": "2018-12-22T15:55:16.890Z", "content": "You can now find a Kernel version of this code here: https://www.kaggle.com/hortonhearsafoo/fastai-v1-starter-pack-kernel-edition-lb-0-323
", "rawMarkdown": "You can now find a Kernel version of this code here: https://www.kaggle.com/hortonhearsafoo/fastai-v1-starter-pack-kernel-edition-lb-0-323", "replies": [ { "id": 443967, "postDate": "2018-12-22T19:55:56.337Z", "content": "Thanks Dude
", "rawMarkdown": "Thanks Dude", "votes": 1 } ] }, { "id": 421340, "postDate": "2018-11-14T23:22:38Z", "content": "Thank you sharing! I'm running fastai v1 on Paperspace using Gradient, but it is not finding the ImageMultiDataset => cannot import name 'ImageMultiDataset'. Would you know how I could check the version and update to 1.0.22?
", "rawMarkdown": "Thank you sharing! I'm running fastai v1 on Paperspace using Gradient, but it is not finding the ImageMultiDataset => cannot import name 'ImageMultiDataset'. Would you know how I could check the version and update to 1.0.22?", "replies": [ { "id": 421406, "postDate": "2018-11-15T00:50:23.077Z", "content": "I think the problem might actually be that you’re on a newer version of the library. I’m going to put out a new version in the next few days that should fix the issue.
", "rawMarkdown": "I think the problem might actually be that you’re on a newer version of the library. I’m going to put out a new version in the next few days that should fix the issue." }, { "id": 421501, "postDate": "2018-11-15T04:30:53.077Z", "content": "Awesome, thank you!
", "rawMarkdown": "Awesome, thank you!" }, { "id": 423284, "postDate": "2018-11-17T21:08:11.310Z", "content": "@Jay Ratcliff this should be fixed now
", "rawMarkdown": "@Jay Ratcliff this should be fixed now" } ] }, { "id": 418607, "postDate": "2018-11-10T09:02:07.927Z", "content": "Thanks for sharing, William. May I ask, which exact version of fastai you used?
", "rawMarkdown": "Thanks for sharing, William. May I ask, which exact version of fastai you used? ", "replies": [ { "id": 418740, "postDate": "2018-11-10T14:26:48.420Z", "content": "I used 1.0.22
", "rawMarkdown": "I used 1.0.22", "votes": 2 }, { "id": 429989, "postDate": "2018-11-29T16:20:34.793Z", "content": "In a previous version of FastAI, in the 'Cats and Dogs' lesson 1 notebook they use 'differential learning rates,' stored as lr=np.array([1e-4,1e-3,1e-2]). From what I've tried v1 doesn't allow this this way. Do you know of another way to put different learning rates on different parts of the model in v1 of FastAI?
", "rawMarkdown": "In a previous version of FastAI, in the 'Cats and Dogs' lesson 1 notebook they use 'differential learning rates,' stored as lr=np.array([1e-4,1e-3,1e-2]). From what I've tried v1 doesn't allow this this way. Do you know of another way to put different learning rates on different parts of the model in v1 of FastAI?" }, { "id": 430261, "postDate": "2018-11-30T04:46:40.570Z", "content": "Nevermind. I found it in the new version's docs. I haven't tried using it yet.
", "rawMarkdown": "Nevermind. I found it in the new version's [docs](https://docs.fast.ai/basic_train.html#Learner.lr_range). I haven't tried using it yet." } ] }, { "id": 418569, "postDate": "2018-11-10T06:39:34.543Z", "content": "Why would you initialise all new weights to 0? How does this help?
", "rawMarkdown": "Why would you initialise all new weights to 0? How does this help?", "replies": [ { "id": 418772, "postDate": "2018-11-10T15:32:21.273Z", "content": "Since I added a new input channel, I had to add more weights to the first conv layer, and I used zero because that's what the other fastai kernel was doing. I think there's probably some gains to be made with a different initialization.
\n\nFor example, I just read a comment (https://www.kaggle.com/iafoss/pretrained-resnet34-with-rgby-0-460-public-lb/notebook#418042) that suggested copying the weights from one of the other channels, so I might try that next.
", "rawMarkdown": "Since I added a new input channel, I had to add more weights to the first conv layer, and I used zero because that's what the other fastai kernel was doing. I think there's probably some gains to be made with a different initialization.\n\nFor example, I just read a comment (https://www.kaggle.com/iafoss/pretrained-resnet34-with-rgby-0-460-public-lb/notebook#418042) that suggested copying the weights from one of the other channels, so I might try that next." } ] }, { "id": 432765, "postDate": "2018-12-04T09:39:55.687Z", "rawMarkdown": "", "isDeleted": true }, { "id": 430064, "postDate": "2018-11-29T19:01:37.680Z", "rawMarkdown": "", "isDeleted": true, "replies": [ { "id": 431654, "postDate": "2018-12-02T17:30:41.213Z", "content": "Hi @Telemachus,\nHere are some answers that I hope can keep you headed in the right direction:\nFor this code:\n\ndef _resnet_split(m): return (m[0][6],m[1])\n
It eventually makes it's way to the split_model function here: https://github.com/fastai/fastai/blob/7a0bd41699c15b44e8bc1df5ec0c809d9ab26115/fastai/torch_core.py#L141. Essentially you're defining parts of the overall model as the points to split on to create your layer groups. So if you take a resnet model, for example, you can index into it like this to see which layers those indexes correspond to:\n\nm = resnet34()\nprint(m[0][6])\nprint(m[1])\n
The \"cut\" removes the last 2 layers, as you can see in this code from the create_body function:\n\n return (nn.Sequential(*list(model.children())[:cut]) if cut\n else body_fn(model) if body_fn else model)\n
That can be a bit confusing, however, because those last layers are also referred to as the \"head\" of the model.
\n\nBy default, freeze freezes up to the last layer, and is called in create_cnn when using a pretrained model.\n\n def freeze(self)->None:\n \"Freeze up to last layer.\"\n assert(len(self.layer_groups)>1)\n self.freeze_to(-1)\n\nhttps://github.com/fastai/fastai/blob/05bc504819ebdab091d8308aac60aed724cd71e7/fastai/basic_train.py#L181
Thank you.
\nSo it's \"split\" but the result is that some parts are cut out? Like editing a video clip?\nOr does splitting just leave the architecture in place but wipe the information in those pre-trained layers?\nI don't understand the action the splitting is doing?
And are you splitting here because of what shapes the CNN filters has in those specific layers you are targeting?
", "rawMarkdown": "Thank you. \nSo it's \"split\" but the result is that some parts are cut out? Like editing a video clip?\nOr does splitting just leave the architecture in place but wipe the information in those pre-trained layers?\nI don't understand the action the splitting is doing?\n\nAnd are you splitting here because of what shapes the CNN filters has in those specific layers you are targeting?" } ] }, { "id": 421729, "postDate": "2018-11-15T10:42:47.217Z", "rawMarkdown": "", "isDeleted": true }, { "id": 418409, "postDate": "2018-11-09T21:42:55.970Z", "rawMarkdown": "", "isDeleted": true, "replies": [ { "id": 418413, "postDate": "2018-11-09T21:49:59.090Z", "content": "Can you include more of the error message? It should tell you what modules are missing.
\n\nYou should make sure that you have installed fastai according to the instructions in their github repo, as well as pytorch v1 (pytorch-nightly)
", "rawMarkdown": "Can you include more of the error message? It should tell you what modules are missing.\n\nYou should make sure that you have installed fastai according to the instructions in their github repo, as well as pytorch v1 (pytorch-nightly)" }, { "id": 418418, "postDate": "2018-11-09T21:56:32.020Z", "content": "Hey, thanks for the quick reply, somehow I am having issues with kaggle, I tried including the error message, but it got truncated at the end. I also tried sending you an email via kaggle and that somehow did not get send. My apologies. I tried adding the message also to this text but it still got truncated...\nI just did a git pull in my fastai folder and a conda env update, however the error persists =(
\n\nJust the message is: ModuleNotFoundError: No module named 'dataclasses'\nMaybe that helps.
", "rawMarkdown": "Hey, thanks for the quick reply, somehow I am having issues with kaggle, I tried including the error message, but it got truncated at the end. I also tried sending you an email via kaggle and that somehow did not get send. My apologies. I tried adding the message also to this text but it still got truncated...\nI just did a git pull in my fastai folder and a conda env update, however the error persists =(\n\nJust the message is: ModuleNotFoundError: No module named 'dataclasses'\nMaybe that helps." }, { "id": 418454, "postDate": "2018-11-09T23:08:57.257Z", "content": "That helps! The problem is dataclasses was added in python 3.7. Luckily there’s a backport package, so you should be able to fix the issue with pip install dataclasses
", "rawMarkdown": "That helps! The problem is dataclasses was added in python 3.7. Luckily there’s a backport package, so you should be able to fix the issue with pip install dataclasses" }, { "id": 418912, "postDate": "2018-11-10T21:13:15.993Z", "content": "Thanks! That really helped, there was another package missing, that I could just install the same way.
\n\nUnfortunately, now I am having another error: ImportError: cannot import name 'as_tensor'.\nMaybe you got another idea?
\n\nJust by the way, I also tried to install the new fast.ai version in a different conda environment, using the instructions given in https://github.com/fastai/fastai/blob/master/README.md
\n\nHowever, I get the error that the module torch is missing. Maybe there is another issue...
", "rawMarkdown": "Thanks! That really helped, there was another package missing, that I could just install the same way.\n\nUnfortunately, now I am having another error: ImportError: cannot import name 'as_tensor'.\nMaybe you got another idea?\n\nJust by the way, I also tried to install the new fast.ai version in a different conda environment, using the instructions given in https://github.com/fastai/fastai/blob/master/README.md\n\nHowever, I get the error that the module torch is missing. Maybe there is another issue..." }, { "id": 418944, "postDate": "2018-11-10T23:49:08.217Z", "content": "Did you run this? (from the README instructions you mentioned):\nconda install -c pytorch pytorch-nightly cuda92
If you run that and still get module torch is missing, you might have deeper problems. At that point I'd suggest heading to the fastai forums https://forums.fast.ai/c/fastai-users to see if anyone has had similar install issues, and if not, then to ask the question there.
", "rawMarkdown": "Did you run this? (from the README instructions you mentioned):\n`conda install -c pytorch pytorch-nightly cuda92`\n\nIf you run that and still get module torch is missing, you might have deeper problems. At that point I'd suggest heading to the fastai forums https://forums.fast.ai/c/fastai-users to see if anyone has had similar install issues, and if not, then to ask the question there." }, { "id": 419134, "postDate": "2018-11-11T11:10:28.643Z", "content": "Yep, I did that. \nGood point, I will go there and ask. Thanks for the help!
", "rawMarkdown": "Yep, I did that. \nGood point, I will go there and ask. Thanks for the help!", "votes": 1 } ] } ], "comments": [ { "id": 418351, "author_name": "Iafoss", "author_url": "", "post_date": "2018-11-09T17:51:25.400000", "content": "I think that if you use AdaptiveConcatPool2d and add extra layers on the top in resnet.py, your model may work better. You can check what fast.ai is doing by default with the model head.
", "votes": 3, "replies": [ { "id": 418383, "author_name": "Mark Worrall", "author_url": "", "post_date": "2018-11-09T20:17:06.790000", "content": "Something like this:
\n\n self.out = nn.Sequential(nn.BatchNorm1d(self.bottom_channel_nr*2),\n nn.Dropout(p=self.dropout_1d, inplace=True),\n nn.Linear(self.bottom_channel_nr*2, self.bottom_channel_nr, bias=True),\n nn.ReLU(inplace=True),\n nn.BatchNorm1d(self.bottom_channel_nr),\n nn.Dropout(p=self.dropout_1d, inplace=True),\n nn.Linear(self.bottom_channel_nr, 28, bias=True)\n )\nAlso you can change the input43 layer to go from 4 to 64 channels instead and use it in place of the first resnet layer.
", "votes": 0, "replies": [] }, { "id": 418391, "author_name": "William Horton", "author_url": "", "post_date": "2018-11-09T20:31:00.693000", "content": "Thank you for the suggestion. It’s definitely on my list of things to try next. I think I should be able to get the fastai default classification head in v1 by using the create_cnn function.
", "votes": 0, "replies": [] }, { "id": 421224, "author_name": "Cozy Doomer", "author_url": "", "post_date": "2018-11-14T18:50:42.663000", "content": "Great input for everyone working with fastai!
\n\nThank you @lafoss for your ship detection challenge kernals and @William Horton for this one
\n\nThey have been very helpful for getting into these competitions :D
", "votes": 0, "replies": [] }, { "id": 428030, "author_name": "Telemachus", "author_url": "", "post_date": "2018-11-26T16:34:19.783000", "content": "Where do you learn how to put 'extra layers on the top' and 'change the input43 layer to go from 4 to 64 channels'?
\nDid Jeremy do something like this in a FastAI course somewhere?
\nMy top down education is not getting down there.
I look in the resnet.py file. I see things that are parts of the model. I type 'learn' before training and I see a model. I enter learn.unfreeze() and look at the model again and it looks unchanged. Kind of opaque as to how to modify this in any way. I changed a few lines in renset.py and dims don't match or it says such and such is not a layer in resnet.
", "votes": 0, "replies": [] }, { "id": 428214, "author_name": "William Horton", "author_url": "", "post_date": "2018-11-26T23:38:25.810000", "content": "@Telemachus I don’t think it’s covered in the fastai course itself.
\n\nI first learned about modifying the first conv layer of resnet from the discussion and Kernels in the TGS segmentation challenge, as some people reported improvements when changing the stride from 2 to 1.
\n\nFor this competition, I also took a look at the excellent kernel by @Iafoss
", "votes": 1, "replies": [] }, { "id": 428739, "author_name": "Telemachus", "author_url": "", "post_date": "2018-11-27T19:44:17.613000", "content": "Thank you for that notebook. I see where the 4 channel part is and somewhat how that works. I got the part about stride and padding down.
\nIt's just that earlier comment 'use AdaptiveConcatPool2d and add extra layers on the top' by @lafoss I don't know how to work with. I'm familiar with how to change components, length, and widths of models and their layers when making a new one like with the FastAI Rossman notebook or something similar but I don't know what direction to look for adding layers or changing a pretrained model. This is pretty different than learning regular programming where you can break anything or get any stage to print and see what's happening to the bits.
\nI used an old Fastai version on lafoss's notebook that loaded the dependancies so I could play around with it but 'tfm_y' kept throwing an error so near the beginning so I gave up. Google didn't return anyone having that problem on StackOverflow.
I'm whining but at least I'm not asking for the answers. It's just working toward understanding in order to accomplish something that doesn't have any footholds.\nThere is a mountain of comments in the Salt challenge.
", "votes": 0, "replies": [] }, { "id": 428912, "author_name": "William Horton", "author_url": "", "post_date": "2018-11-28T04:13:30.867000", "content": "@Telemachus I admire your commitment! In terms of adding extra layers to the top, I'll point you to the create_cnn function in the fastai v1 library (https://github.com/fastai/fastai/blob/master/fastai/vision/learner.py#L48). If you start by studying that, you can begin to understand how to split up a model and add some of your own layers. I'd also suggest taking a look at create_head (https://github.com/fastai/fastai/blob/master/fastai/vision/learner.py#L35) to see specifically how it does the \"use AdaptiveConcatPool2d and add extra layers on the top\" part.
In terms of lessons from the Salt challenge, you can take a look at the code from the Neptune open solution to see how they took a pretrained resnet and used it to create a more complex Unet-like architecture: https://github.com/neptune-ml/open-solution-salt-identification/blob/master/common_blocks/unet_models.py#L78
", "votes": 1, "replies": [] }, { "id": 429507, "author_name": "Telemachus", "author_url": "", "post_date": "2018-11-29T00:29:53.650000", "content": "Thank you. Those responses were above and beyond.
\n\nI see FastAI has docs for these functions. I'll leave the link here.
", "votes": 1, "replies": [] } ] }, { "id": 438184, "author_name": "RomRoc", "author_url": "", "post_date": "2018-12-13T08:16:10.070000", "content": "Excellent kernel, I think with 2019 version of Fast.ai course most of the people even in Kaggle will move to Fast.ai v1.
", "votes": 1, "replies": [] }, { "id": 423279, "author_name": "William Horton", "author_url": "", "post_date": "2018-11-17T20:53:37.380000", "content": "UPDATE: the resnet50 basic notebook doesn't work with fastai version 1.0.25 and above, so I made another notebook to work with the new data block API. In this version, I also made changes to use the create cnn function. You can find it at resnet50 basic datablocks.ipynb.
", "votes": 1, "replies": [ { "id": 423690, "author_name": "Jay Ratcliff", "author_url": "", "post_date": "2018-11-18T22:04:56.333000", "content": "Thank you William, appreciate very much, will give it a go. This is a great one for me to learn so much on competitions and the tools we're learning to use in fastai.
", "votes": 1, "replies": [] } ] }, { "id": 424967, "author_name": "William Horton", "author_url": "", "post_date": "2018-11-21T00:11:39.157000", "content": "Made some more fixes last night based on the latest version of the library. I'm trying to keep up with the pace of development but I might have to stop updating the fastai library every week if I want to focus on this competition.
", "votes": 2, "replies": [] }, { "id": 423285, "author_name": "dromosys", "author_url": "", "post_date": "2018-11-17T21:11:28.720000", "content": "small example of fastai v1 https://www.kaggle.com/dromosys/fast-ai-mnst
", "votes": 0, "replies": [] }, { "id": 477892, "author_name": "Creative Explorer", "author_url": "", "post_date": "2019-02-25T12:33:59.423000", "content": "hey William, thank you for sharing this, in your Kernel I keep getting this error \"got an unexpected keyword argument 'sep'\" on the ImageItemList.from_csv instruction at \"label_from_df(sep=' '.... any hints? thank u ;)
", "votes": 0, "replies": [ { "id": 477992, "author_name": "William Horton", "author_url": "", "post_date": "2019-02-25T15:38:56.590000", "content": "I think it has to do with an update to the version of fastai used in the Kernels. I’ll try to see if I can figure it out tonight
", "votes": 1, "replies": [] }, { "id": 478055, "author_name": "Creative Explorer", "author_url": "", "post_date": "2019-02-25T17:02:09.003000", "content": "hey I found it, yes it was that, its label_delim instead of sep, thank u William
", "votes": 1, "replies": [] } ] }, { "id": 454466, "author_name": "Devon Kaberna", "author_url": "", "post_date": "2019-01-11T17:15:10.723000", "content": "I get an error when running this line preds,_ = learn.get_preds(DatasetType.Test). The error I get is: \n File \"/home/devon/fastai/courses/kaggle/protein-atlas-fastai/utils.py\", line 16, in open_4_channel\n for color in colors]\n File \"/home/devon/fastai/courses/kaggle/protein-atlas-fastai/utils.py\", line 16, in \n for color in colors]\nAttributeError: 'NoneType' object has no attribute 'astype'
\n\nany ideas?
", "votes": 0, "replies": [ { "id": 454553, "author_name": "William Horton", "author_url": "", "post_date": "2019-01-11T20:24:51.987000", "content": "It likely means some files are missing. Make sure all of the test images are in the right directory
", "votes": 0, "replies": [] }, { "id": 454656, "author_name": "Devon Kaberna", "author_url": "", "post_date": "2019-01-11T23:35:43.300000", "content": "Thanks for the reply. I double checked everything, and seems fine. I have a test and train folder within protein-atlas-fastai. I verified I have python3.6 and pytorch1.0.0.dev20190111. Kinda puzzled. I know you're busy and don't expect further help. Maybe other folks have some ideas.
", "votes": 0, "replies": [] }, { "id": 454677, "author_name": "Devon Kaberna", "author_url": "", "post_date": "2019-01-12T01:15:34.740000", "content": "Figured it out. Thanks so much for the help!
", "votes": 0, "replies": [] } ] }, { "id": 443861, "author_name": "William Horton", "author_url": "", "post_date": "2018-12-22T15:55:16.890000", "content": "You can now find a Kernel version of this code here: https://www.kaggle.com/hortonhearsafoo/fastai-v1-starter-pack-kernel-edition-lb-0-323
", "votes": 0, "replies": [ { "id": 443967, "author_name": "Prateek Srivastava", "author_url": "", "post_date": "2018-12-22T19:55:56.337000", "content": "Thanks Dude
", "votes": 1, "replies": [] } ] }, { "id": 421340, "author_name": "Jay Ratcliff", "author_url": "", "post_date": "2018-11-14T23:22:38", "content": "Thank you sharing! I'm running fastai v1 on Paperspace using Gradient, but it is not finding the ImageMultiDataset => cannot import name 'ImageMultiDataset'. Would you know how I could check the version and update to 1.0.22?
", "votes": 0, "replies": [ { "id": 421406, "author_name": "William Horton", "author_url": "", "post_date": "2018-11-15T00:50:23.077000", "content": "I think the problem might actually be that you’re on a newer version of the library. I’m going to put out a new version in the next few days that should fix the issue.
", "votes": 0, "replies": [] }, { "id": 421501, "author_name": "Jay Ratcliff", "author_url": "", "post_date": "2018-11-15T04:30:53.077000", "content": "Awesome, thank you!
", "votes": 0, "replies": [] }, { "id": 423284, "author_name": "William Horton", "author_url": "", "post_date": "2018-11-17T21:08:11.310000", "content": "@Jay Ratcliff this should be fixed now
", "votes": 0, "replies": [] } ] }, { "id": 418607, "author_name": "FabSchreiber", "author_url": "", "post_date": "2018-11-10T09:02:07.927000", "content": "Thanks for sharing, William. May I ask, which exact version of fastai you used?
", "votes": 0, "replies": [ { "id": 418740, "author_name": "William Horton", "author_url": "", "post_date": "2018-11-10T14:26:48.420000", "content": "I used 1.0.22
", "votes": 2, "replies": [] }, { "id": 429989, "author_name": "Telemachus", "author_url": "", "post_date": "2018-11-29T16:20:34.793000", "content": "In a previous version of FastAI, in the 'Cats and Dogs' lesson 1 notebook they use 'differential learning rates,' stored as lr=np.array([1e-4,1e-3,1e-2]). From what I've tried v1 doesn't allow this this way. Do you know of another way to put different learning rates on different parts of the model in v1 of FastAI?
", "votes": 0, "replies": [] }, { "id": 430261, "author_name": "Telemachus", "author_url": "", "post_date": "2018-11-30T04:46:40.570000", "content": "Nevermind. I found it in the new version's docs. I haven't tried using it yet.
", "votes": 0, "replies": [] } ] }, { "id": 418569, "author_name": "Rahul Deora", "author_url": "", "post_date": "2018-11-10T06:39:34.543000", "content": "Why would you initialise all new weights to 0? How does this help?
", "votes": 0, "replies": [ { "id": 418772, "author_name": "William Horton", "author_url": "", "post_date": "2018-11-10T15:32:21.273000", "content": "Since I added a new input channel, I had to add more weights to the first conv layer, and I used zero because that's what the other fastai kernel was doing. I think there's probably some gains to be made with a different initialization.
\n\nFor example, I just read a comment (https://www.kaggle.com/iafoss/pretrained-resnet34-with-rgby-0-460-public-lb/notebook#418042) that suggested copying the weights from one of the other channels, so I might try that next.
", "votes": 0, "replies": [] } ] }, { "id": 432765, "author_name": "", "author_url": "", "post_date": "2018-12-04T09:39:55.687000", "content": "", "votes": 0, "replies": [] }, { "id": 430064, "author_name": "", "author_url": "", "post_date": "2018-11-29T19:01:37.680000", "content": "", "votes": 0, "replies": [ { "id": 431654, "author_name": "William Horton", "author_url": "", "post_date": "2018-12-02T17:30:41.213000", "content": "Hi @Telemachus,\nHere are some answers that I hope can keep you headed in the right direction:\nFor this code:\n\ndef _resnet_split(m): return (m[0][6],m[1])\n
It eventually makes it's way to the split_model function here: https://github.com/fastai/fastai/blob/7a0bd41699c15b44e8bc1df5ec0c809d9ab26115/fastai/torch_core.py#L141. Essentially you're defining parts of the overall model as the points to split on to create your layer groups. So if you take a resnet model, for example, you can index into it like this to see which layers those indexes correspond to:\n\nm = resnet34()\nprint(m[0][6])\nprint(m[1])\n
The \"cut\" removes the last 2 layers, as you can see in this code from the create_body function:\n\n return (nn.Sequential(*list(model.children())[:cut]) if cut\n else body_fn(model) if body_fn else model)\n
That can be a bit confusing, however, because those last layers are also referred to as the \"head\" of the model.
\n\nBy default, freeze freezes up to the last layer, and is called in create_cnn when using a pretrained model.\n\n def freeze(self)->None:\n \"Freeze up to last layer.\"\n assert(len(self.layer_groups)>1)\n self.freeze_to(-1)\n\nhttps://github.com/fastai/fastai/blob/05bc504819ebdab091d8308aac60aed724cd71e7/fastai/basic_train.py#L181
Thank you.
\nSo it's \"split\" but the result is that some parts are cut out? Like editing a video clip?\nOr does splitting just leave the architecture in place but wipe the information in those pre-trained layers?\nI don't understand the action the splitting is doing?
And are you splitting here because of what shapes the CNN filters has in those specific layers you are targeting?
", "votes": 0, "replies": [] } ] }, { "id": 421729, "author_name": "", "author_url": "", "post_date": "2018-11-15T10:42:47.217000", "content": "", "votes": 0, "replies": [] }, { "id": 418409, "author_name": "", "author_url": "", "post_date": "2018-11-09T21:42:55.970000", "content": "", "votes": 0, "replies": [ { "id": 418413, "author_name": "William Horton", "author_url": "", "post_date": "2018-11-09T21:49:59.090000", "content": "Can you include more of the error message? It should tell you what modules are missing.
\n\nYou should make sure that you have installed fastai according to the instructions in their github repo, as well as pytorch v1 (pytorch-nightly)
", "votes": 0, "replies": [] }, { "id": 418418, "author_name": "stumpen", "author_url": "", "post_date": "2018-11-09T21:56:32.020000", "content": "Hey, thanks for the quick reply, somehow I am having issues with kaggle, I tried including the error message, but it got truncated at the end. I also tried sending you an email via kaggle and that somehow did not get send. My apologies. I tried adding the message also to this text but it still got truncated...\nI just did a git pull in my fastai folder and a conda env update, however the error persists =(
\n\nJust the message is: ModuleNotFoundError: No module named 'dataclasses'\nMaybe that helps.
", "votes": 0, "replies": [] }, { "id": 418454, "author_name": "William Horton", "author_url": "", "post_date": "2018-11-09T23:08:57.257000", "content": "That helps! The problem is dataclasses was added in python 3.7. Luckily there’s a backport package, so you should be able to fix the issue with pip install dataclasses
", "votes": 0, "replies": [] }, { "id": 418912, "author_name": "stumpen", "author_url": "", "post_date": "2018-11-10T21:13:15.993000", "content": "Thanks! That really helped, there was another package missing, that I could just install the same way.
\n\nUnfortunately, now I am having another error: ImportError: cannot import name 'as_tensor'.\nMaybe you got another idea?
\n\nJust by the way, I also tried to install the new fast.ai version in a different conda environment, using the instructions given in https://github.com/fastai/fastai/blob/master/README.md
\n\nHowever, I get the error that the module torch is missing. Maybe there is another issue...
", "votes": 0, "replies": [] }, { "id": 418944, "author_name": "William Horton", "author_url": "", "post_date": "2018-11-10T23:49:08.217000", "content": "Did you run this? (from the README instructions you mentioned):\nconda install -c pytorch pytorch-nightly cuda92
If you run that and still get module torch is missing, you might have deeper problems. At that point I'd suggest heading to the fastai forums https://forums.fast.ai/c/fastai-users to see if anyone has had similar install issues, and if not, then to ask the question there.
", "votes": 0, "replies": [] }, { "id": 419134, "author_name": "stumpen", "author_url": "", "post_date": "2018-11-11T11:10:28.643000", "content": "Yep, I did that. \nGood point, I will go there and ask. Thanks for the help!
", "votes": 1, "replies": [] } ] } ], "raw_markdown_by_id": { "418232": "Hi all!\n\nI was inspired by @Radek generously sharing his [quickdraw starter code](https://github.com/radekosmulski/quickdraw), so I wanted to share the code I put together yesterday to enter this competition using the new version of the fastai library:\n\nhttps://github.com/wdhorton/protein-atlas-fastai\n\nI got 0.380 LB with my submission this morning, so this won't get you close to medaling, but I hope it's enough to get you started, and save you some time digging through the fastai code trying to get it to work with this dataset. This is a minimal starter, so there are definitely things you can tweak and play around with to try to improve your results.\n\n(fastai v1 is not available in Kernels yet, but there is already a great Kernel based on the 0.7 version of the library https://www.kaggle.com/iafoss/pretrained-resnet34-with-rgby-0-460-public-lb which I borrowed some tricks from.)", "418351": "I think that if you use AdaptiveConcatPool2d and add extra layers on the top in resnet.py, your model may work better. You can check what fast.ai is doing by default with the model head.", "438184": "Excellent kernel, I think with 2019 version of Fast.ai course most of the people even in Kaggle will move to Fast.ai v1.", "423279": "UPDATE: the resnet50 basic notebook doesn't work with fastai version 1.0.25 and above, so I made another notebook to work with the new data block API. In this version, I also made changes to use the create cnn function. You can find it at resnet50 basic datablocks.ipynb.", "424967": "Made some more fixes last night based on the latest version of the library. I'm trying to keep up with the pace of development but I might have to stop updating the fastai library every week if I want to focus on this competition.", "423285": "small example of fastai v1 https://www.kaggle.com/dromosys/fast-ai-mnst", "477892": "hey William, thank you for sharing this, in your Kernel I keep getting this error \"got an unexpected keyword argument 'sep'\" on the ImageItemList.from_csv instruction at \"label_from_df(sep=' '.... any hints? thank u ;)", "454466": "I get an error when running this line preds,_ = learn.get_preds(DatasetType.Test). The error I get is: \n File \"/home/devon/fastai/courses/kaggle/protein-atlas-fastai/utils.py\", line 16, in open_4_channel\n for color in colors]\n File \"/home/devon/fastai/courses/kaggle/protein-atlas-fastai/utils.py\", line 16, in ", "443861": "You can now find a Kernel version of this code here: https://www.kaggle.com/hortonhearsafoo/fastai-v1-starter-pack-kernel-edition-lb-0-323", "421340": "Thank you sharing! I'm running fastai v1 on Paperspace using Gradient, but it is not finding the ImageMultiDataset => cannot import name 'ImageMultiDataset'. Would you know how I could check the version and update to 1.0.22?", "418607": "Thanks for sharing, William. May I ask, which exact version of fastai you used? ", "418569": "Why would you initialise all new weights to 0? How does this help?", "432765": "", "430064": "", "421729": "", "418409": "" } }