Can anyone help me to understand how can I add dropout, activation functions, and dense layers in the pre-trained PyTorch model? I try to search in the forum and also on stack overflow including PyTorch docs but did not get a working example of the same. Let's say we are writing out a custom class like this
\nclass CustomClassifierForPretrainedModel(nn.Module):\n def __init__(self, pretrained_model, n_class, pretrained=False):\n super().__init__()\n self.model = timm.create_model(pretrained_model, pretrained=pretrained)\n n_features = self.model.classifier.in_features \n # tweaking the model\n self.model.classifier = nn.Sequential(\n nn.Linear(n_features, 512),\n nn.Dropout(0.5),\n nn.ReLU(),\n nn.Linear(256, 128),\n nn.Dropout(0.33),\n nn.ReLU(),\n nn.Linear(256, n_class)) \n def forward(self, x):\n x = self.model(x)\n return x\nAm I doing it in the right way? However, I tried to run the same but ended with an OOM issue. Besides that, I'm trying to understand is it the right way to implement a custom pre-trained PyTorch model?
", "messages": [ { "id": "1149089", "postDate": "01/11/2021 15:25:51", "content": "Can anyone help me to understand how can I add dropout, activation functions, and dense layers in the pre-trained PyTorch model? I try to search in the forum and also on stack overflow including PyTorch docs but did not get a working example of the same. Let's say we are writing out a custom class like this
\nclass CustomClassifierForPretrainedModel(nn.Module):\n def __init__(self, pretrained_model, n_class, pretrained=False):\n super().__init__()\n self.model = timm.create_model(pretrained_model, pretrained=pretrained)\n n_features = self.model.classifier.in_features \n # tweaking the model\n self.model.classifier = nn.Sequential(\n nn.Linear(n_features, 512),\n nn.Dropout(0.5),\n nn.ReLU(),\n nn.Linear(256, 128),\n nn.Dropout(0.33),\n nn.ReLU(),\n nn.Linear(256, n_class)) \n def forward(self, x):\n x = self.model(x)\n return x\nAm I doing it in the right way? However, I tried to run the same but ended with an OOM issue. Besides that, I'm trying to understand is it the right way to implement a custom pre-trained PyTorch model?
", "rawMarkdown": "Can anyone help me to understand how can I add dropout, activation functions, and dense layers in the pre-trained PyTorch model? I try to search in the forum and also on stack overflow including PyTorch docs but did not get a working example of the same. Let's say we are writing out a custom class like this \n\n\n```\nclass CustomClassifierForPretrainedModel(nn.Module):\n def __init__(self, pretrained_model, n_class, pretrained=False):\n super().__init__()\n self.model = timm.create_model(pretrained_model, pretrained=pretrained)\n n_features = self.model.classifier.in_features \n # tweaking the model\n self.model.classifier = nn.Sequential(\n nn.Linear(n_features, 512),\n nn.Dropout(0.5),\n nn.ReLU(),\n nn.Linear(256, 128),\n nn.Dropout(0.33),\n nn.ReLU(),\n nn.Linear(256, n_class)) \n def forward(self, x):\n x = self.model(x)\n return x\n\n```\n\nAm I doing it in the right way? However, I tried to run the same but ended with an OOM issue. Besides that, I'm trying to understand is it the right way to implement a custom pre-trained PyTorch model?", "votes": null }, { "id": "1149235", "postDate": "01/11/2021 17:40:33", "content": "There is no layer named classifier in the pretrained model . It need to be self.model.fc
If you need to add other custom layers that you have already defined, you have to extract pre-classifier features of the pretrained model first and add them during the forward pass
\nExample :
\ndef forward(self, x):\n x = self.model.forward_features(x)\n x = self.mycustom_pre_pooled_layer(x) #optional\n x = self.avg_pool(x).flatten(1)\n x = self.mycustom_post_pooled_layer(x) #optional\n x = self.dropout(x)\n x = self.fc(x)\n return x\nFor the EfficientNet models the output layer is called classifier (in timm package)
Looks like your classifier has some wrong number of input/ouput in the Linear layers. Also adding some batchnorm layers can help. Here is what you should do:
\nself.model.classifier = nn.Sequential(\n nn.Linear(n_features, 512),\n nn.ReLU(),\n nn.BatchNorm1d(512)\n nn.Dropout(0.5),\n nn.Linear(512, 256),\n nn.ReLU(),\n nn.BatchNorm1d(256)\n nn.Dropout(0.33),\n nn.Linear(256, n_class)) \nYou're right.
\nI always use the pre-classifier features (forward_features) and add my own pooled/custom/fc layers. At least it avoid confusion with resnet like fc layer ^^
Thanks @yannmajewski. Your response makes sense. I will try out it :)
", "rawMarkdown": "Thanks @yannmajewski. Your response makes sense. I will try out it :)", "votes": null }, { "id": "1149453", "postDate": "01/11/2021 21:05:37", "content": "Ali has rightly pointed out the typical output layer for EfficientNet( for timm packages). However, I got the idea to follow the general pattern to use different layers for the pre-trained models suggested by Sergine. Thank you both.
", "rawMarkdown": "Ali has rightly pointed out the typical output layer for EfficientNet( for timm packages). However, I got the idea to follow the general pattern to use different layers for the pre-trained models suggested by Sergine. Thank you both.", "votes": null } ], "comments": [ { "id": 1149235, "author_name": "serigne", "author_url": "", "post_date": "01/11/2021 17:40:33", "content": "There is no layer named classifier in the pretrained model . It need to be self.model.fc
If you need to add other custom layers that you have already defined, you have to extract pre-classifier features of the pretrained model first and add them during the forward pass
\nExample :
\ndef forward(self, x):\n x = self.model.forward_features(x)\n x = self.mycustom_pre_pooled_layer(x) #optional\n x = self.avg_pool(x).flatten(1)\n x = self.mycustom_post_pooled_layer(x) #optional\n x = self.dropout(x)\n x = self.fc(x)\n return x\nFor the EfficientNet models the output layer is called classifier (in timm package)
You're right.
\nI always use the pre-classifier features (forward_features) and add my own pooled/custom/fc layers. At least it avoid confusion with resnet like fc layer ^^
Ali has rightly pointed out the typical output layer for EfficientNet( for timm packages). However, I got the idea to follow the general pattern to use different layers for the pre-trained models suggested by Sergine. Thank you both.
", "votes": null, "replies": [] } ] }, { "id": 1149257, "author_name": "yannmajewski", "author_url": "", "post_date": "01/11/2021 18:11:53", "content": "Looks like your classifier has some wrong number of input/ouput in the Linear layers. Also adding some batchnorm layers can help. Here is what you should do:
\nself.model.classifier = nn.Sequential(\n nn.Linear(n_features, 512),\n nn.ReLU(),\n nn.BatchNorm1d(512)\n nn.Dropout(0.5),\n nn.Linear(512, 256),\n nn.ReLU(),\n nn.BatchNorm1d(256)\n nn.Dropout(0.33),\n nn.Linear(256, n_class)) \nThanks @yannmajewski. Your response makes sense. I will try out it :)
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1149089": "Can anyone help me to understand how can I add dropout, activation functions, and dense layers in the pre-trained PyTorch model? I try to search in the forum and also on stack overflow including PyTorch docs but did not get a working example of the same. Let's say we are writing out a custom class like this \n\n\n```\nclass CustomClassifierForPretrainedModel(nn.Module):\n def __init__(self, pretrained_model, n_class, pretrained=False):\n super().__init__()\n self.model = timm.create_model(pretrained_model, pretrained=pretrained)\n n_features = self.model.classifier.in_features \n # tweaking the model\n self.model.classifier = nn.Sequential(\n nn.Linear(n_features, 512),\n nn.Dropout(0.5),\n nn.ReLU(),\n nn.Linear(256, 128),\n nn.Dropout(0.33),\n nn.ReLU(),\n nn.Linear(256, n_class)) \n def forward(self, x):\n x = self.model(x)\n return x\n\n```\n\nAm I doing it in the right way? However, I tried to run the same but ended with an OOM issue. Besides that, I'm trying to understand is it the right way to implement a custom pre-trained PyTorch model?", "1149235": "There is no layer named `classifier ` in the pretrained model . It need to be `self.model.fc` \n\nIf you need to add other custom layers that you have already defined, you have to extract pre-classifier features of the pretrained model first and add them during the forward pass\n\nExample : \n\n```\ndef forward(self, x):\n x = self.model.forward_features(x)\n x = self.mycustom_pre_pooled_layer(x) #optional\n x = self.avg_pool(x).flatten(1)\n x = self.mycustom_post_pooled_layer(x) #optional\n x = self.dropout(x)\n x = self.fc(x)\n return x\n```", "1149239": "For the EfficientNet models the output layer is called `classifier` (in timm package)", "1149257": "Looks like your classifier has some wrong number of input/ouput in the Linear layers. Also adding some batchnorm layers can help. Here is what you should do:\n\n```\nself.model.classifier = nn.Sequential(\n nn.Linear(n_features, 512),\n nn.ReLU(),\n nn.BatchNorm1d(512)\n nn.Dropout(0.5),\n nn.Linear(512, 256),\n nn.ReLU(),\n nn.BatchNorm1d(256)\n nn.Dropout(0.33),\n nn.Linear(256, n_class)) \n```", "1149287": "You're right. \n\nI always use the pre-classifier features (`forward_features`) and add my own pooled/custom/fc layers. At least it avoid confusion with resnet like fc layer ^^", "1149449": "Thanks @yannmajewski. Your response makes sense. I will try out it :)", "1149453": "Ali has rightly pointed out the typical output layer for EfficientNet( for timm packages). However, I got the idea to follow the general pattern to use different layers for the pre-trained models suggested by Sergine. Thank you both." }, "source": "meta" }