{
  "id": 153466,
  "title": "XLM-Roberta parameters",
  "url": "/competitions/jigsaw-multilingual-toxic-comment-classification/discussion/153466",
  "author_name": "",
  "post_date": "2020-05-24T22:12:12.816890600Z",
  "votes": 1,
  "comment_count": 6,
  "views": 0,
  "content": "<p>Hi,</p>\n\n<p>I am wondering whether anyone knows whether it is possible to access the architecture of a Keras layer containing XLM-Roberta (using TFAutoModel.from_pretrained). In my model summary, I see that this layer has 559890432 parameters (associated to ((None, 150, 1024), (None, 1024)) ), but it is unclear to me how we get such a huge number. Is there a way to access the internal structure of the layer? Thanks! </p>",
  "messages": [
    {
      "id": "859955",
      "postDate": "05/24/2020 22:12:12",
      "content": "<p>Hi,</p>\n\n<p>I am wondering whether anyone knows whether it is possible to access the architecture of a Keras layer containing XLM-Roberta (using TFAutoModel.from_pretrained). In my model summary, I see that this layer has 559890432 parameters (associated to ((None, 150, 1024), (None, 1024)) ), but it is unclear to me how we get such a huge number. Is there a way to access the internal structure of the layer? Thanks! </p>",
      "rawMarkdown": "Hi,\n\nI am wondering whether anyone knows whether it is possible to access the architecture of a Keras layer containing XLM-Roberta (using TFAutoModel.from_pretrained). In my model summary, I see that this layer has 559890432 parameters (associated to ((None, 150, 1024), (None, 1024)) ), but it is unclear to me how we get such a huge number. Is there a way to access the internal structure of the layer? Thanks!",
      "votes": null
    },
    {
      "id": "859977",
      "postDate": "05/24/2020 23:03:50",
      "content": "<p>I think the layer you mentioned is not a real layer, but the whole pretrained model (XLM-Roberta).\nYou can use sth like the following code to look the details</p>\n\n<pre><code>    for layer in model.layers:\n        for variable in layer.variables:\n            .... sth like print(variable) ....\n</code></pre>",
      "rawMarkdown": "I think the layer you mentioned is not a real layer, but the whole pretrained model (XLM-Roberta).\nYou can use sth like the following code to look the details\n\n        for layer in model.layers:\n            for variable in layer.variables:\n                .... sth like print(variable) ....",
      "votes": null
    },
    {
      "id": "860022",
      "postDate": "05/25/2020 00:48:48",
      "content": "<p>Thanks. summary() directly on the model returns the same information. I am not sure whether it is possible to know more :(</p>",
      "rawMarkdown": "Thanks. summary() directly on the model returns the same information. I am not sure whether it is possible to know more :(",
      "votes": null
    },
    {
      "id": "860563",
      "postDate": "05/25/2020 12:23:34",
      "content": "<p>Yeah the whole (headless) model is the first layer.\nYou can get the model details from pytorch:</p>\n\n<blockquote>\n  <p>from transformers import AutoModel\n  print(AutoModel.from_pretrained('xlm-roberta-large'))</p>\n</blockquote>\n\n<p><code>\nXLMRobertaModel(\n  (embeddings): RobertaEmbeddings(\n    (word_embeddings): Embedding(250002, 1024, padding_idx=1)\n    (position_embeddings): Embedding(514, 1024, padding_idx=1)\n    (token_type_embeddings): Embedding(1, 1024)\n    (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n    (dropout): Dropout(p=0.1, inplace=False)\n  )\n  (encoder): BertEncoder(\n    (layer): ModuleList(\n      (0): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (1): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (2): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (3): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (4): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (5): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (6): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (7): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (8): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (9): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (10): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (11): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (12): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (13): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (14): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (15): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (16): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (17): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (18): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (19): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (20): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (21): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (22): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (23): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n    )\n  )\n  (pooler): BertPooler(\n    (dense): Linear(in_features=1024, out_features=1024, bias=True)\n    (activation): Tanh()\n  )\n)\n</code></p>",
      "rawMarkdown": "Yeah the whole (headless) model is the first layer.\nYou can get the model details from pytorch:\n&gt; from transformers import AutoModel\nprint(AutoModel.from_pretrained('xlm-roberta-large'))\n\n```\nXLMRobertaModel(\n  (embeddings): RobertaEmbeddings(\n    (word_embeddings): Embedding(250002, 1024, padding_idx=1)\n    (position_embeddings): Embedding(514, 1024, padding_idx=1)\n    (token_type_embeddings): Embedding(1, 1024)\n    (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n    (dropout): Dropout(p=0.1, inplace=False)\n  )\n  (encoder): BertEncoder(\n    (layer): ModuleList(\n      (0): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (1): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (2): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (3): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (4): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (5): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (6): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (7): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (8): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (9): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (10): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (11): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (12): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (13): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (14): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (15): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (16): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (17): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (18): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (19): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (20): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (21): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (22): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (23): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n    )\n  )\n  (pooler): BertPooler(\n    (dense): Linear(in_features=1024, out_features=1024, bias=True)\n    (activation): Tanh()\n  )\n)\n```",
      "votes": null
    },
    {
      "id": "861111",
      "postDate": "05/25/2020 21:27:07",
      "content": "<p>wow, <code>keras.summary</code> seems not that smart here 😅 \nthanks <a href=\"/hmendonca\">@hmendonca</a> </p>",
      "rawMarkdown": "wow, `keras.summary` seems not that smart here 😅 \nthanks @hmendonca",
      "votes": null
    },
    {
      "id": "896594",
      "postDate": "06/22/2020 09:51:10",
      "content": "<p>Thanks, <a href=\"/hmendonca\">@hmendonca</a>. Am I correct that only in Pytorch I could modify some hyperparameters but not in keras?</p>",
      "rawMarkdown": "Thanks, @hmendonca. Am I correct that only in Pytorch I could modify some hyperparameters but not in keras?",
      "votes": null
    },
    {
      "id": "896694",
      "postDate": "06/22/2020 11:36:57",
      "content": "<p><a href=\"/querymaster\">@querymaster</a> I'm not sure if you can modify in-place, but you can definitely copy layers of the pre-trained model and mash them with/on your own model\ne.g. you can loop through model.layers[1].layers </p>",
      "rawMarkdown": "querymaster I'm not sure if you can modify in-place, but you can definitely copy layers of the pre-trained model and mash them with/on your own model\ne.g. you can loop through model.layers[1].layers",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 859977,
      "author_name": "yihdarshieh",
      "author_url": "",
      "post_date": "05/24/2020 23:03:50",
      "content": "<p>I think the layer you mentioned is not a real layer, but the whole pretrained model (XLM-Roberta).\nYou can use sth like the following code to look the details</p>\n\n<pre><code>    for layer in model.layers:\n        for variable in layer.variables:\n            .... sth like print(variable) ....\n</code></pre>",
      "votes": null,
      "replies": []
    },
    {
      "id": 860022,
      "author_name": "querymaster",
      "author_url": "",
      "post_date": "05/25/2020 00:48:48",
      "content": "<p>Thanks. summary() directly on the model returns the same information. I am not sure whether it is possible to know more :(</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 860563,
      "author_name": "hmendonca",
      "author_url": "",
      "post_date": "05/25/2020 12:23:34",
      "content": "<p>Yeah the whole (headless) model is the first layer.\nYou can get the model details from pytorch:</p>\n\n<blockquote>\n  <p>from transformers import AutoModel\n  print(AutoModel.from_pretrained('xlm-roberta-large'))</p>\n</blockquote>\n\n<p><code>\nXLMRobertaModel(\n  (embeddings): RobertaEmbeddings(\n    (word_embeddings): Embedding(250002, 1024, padding_idx=1)\n    (position_embeddings): Embedding(514, 1024, padding_idx=1)\n    (token_type_embeddings): Embedding(1, 1024)\n    (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n    (dropout): Dropout(p=0.1, inplace=False)\n  )\n  (encoder): BertEncoder(\n    (layer): ModuleList(\n      (0): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (1): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (2): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (3): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (4): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (5): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (6): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (7): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (8): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (9): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (10): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (11): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (12): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (13): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (14): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (15): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (16): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (17): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (18): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (19): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (20): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (21): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (22): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (23): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n    )\n  )\n  (pooler): BertPooler(\n    (dense): Linear(in_features=1024, out_features=1024, bias=True)\n    (activation): Tanh()\n  )\n)\n</code></p>",
      "votes": null,
      "replies": [
        {
          "id": 861111,
          "author_name": "ipythonx",
          "author_url": "",
          "post_date": "05/25/2020 21:27:07",
          "content": "<p>wow, <code>keras.summary</code> seems not that smart here 😅 \nthanks <a href=\"/hmendonca\">@hmendonca</a> </p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 896594,
      "author_name": "querymaster",
      "author_url": "",
      "post_date": "06/22/2020 09:51:10",
      "content": "<p>Thanks, <a href=\"/hmendonca\">@hmendonca</a>. Am I correct that only in Pytorch I could modify some hyperparameters but not in keras?</p>",
      "votes": null,
      "replies": [
        {
          "id": 896694,
          "author_name": "hmendonca",
          "author_url": "",
          "post_date": "06/22/2020 11:36:57",
          "content": "<p><a href=\"/querymaster\">@querymaster</a> I'm not sure if you can modify in-place, but you can definitely copy layers of the pre-trained model and mash them with/on your own model\ne.g. you can loop through model.layers[1].layers </p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "859955": "Hi,\n\nI am wondering whether anyone knows whether it is possible to access the architecture of a Keras layer containing XLM-Roberta (using TFAutoModel.from_pretrained). In my model summary, I see that this layer has 559890432 parameters (associated to ((None, 150, 1024), (None, 1024)) ), but it is unclear to me how we get such a huge number. Is there a way to access the internal structure of the layer? Thanks!",
    "859977": "I think the layer you mentioned is not a real layer, but the whole pretrained model (XLM-Roberta).\nYou can use sth like the following code to look the details\n\n        for layer in model.layers:\n            for variable in layer.variables:\n                .... sth like print(variable) ....",
    "860022": "Thanks. summary() directly on the model returns the same information. I am not sure whether it is possible to know more :(",
    "860563": "Yeah the whole (headless) model is the first layer.\nYou can get the model details from pytorch:\n&gt; from transformers import AutoModel\nprint(AutoModel.from_pretrained('xlm-roberta-large'))\n\n```\nXLMRobertaModel(\n  (embeddings): RobertaEmbeddings(\n    (word_embeddings): Embedding(250002, 1024, padding_idx=1)\n    (position_embeddings): Embedding(514, 1024, padding_idx=1)\n    (token_type_embeddings): Embedding(1, 1024)\n    (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n    (dropout): Dropout(p=0.1, inplace=False)\n  )\n  (encoder): BertEncoder(\n    (layer): ModuleList(\n      (0): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (1): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (2): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (3): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (4): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (5): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (6): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (7): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (8): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (9): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (10): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (11): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (12): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (13): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (14): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (15): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (16): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (17): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (18): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (19): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (20): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (21): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (22): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (23): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=1024, out_features=1024, bias=True)\n            (key): Linear(in_features=1024, out_features=1024, bias=True)\n            (value): Linear(in_features=1024, out_features=1024, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=1024, out_features=1024, bias=True)\n            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=1024, out_features=4096, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=4096, out_features=1024, bias=True)\n          (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n    )\n  )\n  (pooler): BertPooler(\n    (dense): Linear(in_features=1024, out_features=1024, bias=True)\n    (activation): Tanh()\n  )\n)\n```",
    "861111": "wow, `keras.summary` seems not that smart here 😅 \nthanks @hmendonca",
    "896594": "Thanks, @hmendonca. Am I correct that only in Pytorch I could modify some hyperparameters but not in keras?",
    "896694": "querymaster I'm not sure if you can modify in-place, but you can definitely copy layers of the pre-trained model and mash them with/on your own model\ne.g. you can loop through model.layers[1].layers"
  },
  "source": "meta"
}