I would be interested to know if anyone has tried multitask learning for captioning. As we know that InChI follows a rigorous format, each sub-component of the labels relying on the shared representation of the image features, this might motivate multitask learning for each distinct portion of the caption.
", "messages": [ { "id": "1320454", "postDate": "05/24/2021 05:19:55", "content": "I would be interested to know if anyone has tried multitask learning for captioning. As we know that InChI follows a rigorous format, each sub-component of the labels relying on the shared representation of the image features, this might motivate multitask learning for each distinct portion of the caption.
", "rawMarkdown": "I would be interested to know if anyone has tried multitask learning for captioning. As we know that InChI follows a rigorous format, each sub-component of the labels relying on the shared representation of the image features, this might motivate multitask learning for each distinct portion of the caption.", "votes": null }, { "id": "1320720", "postDate": "05/24/2021 09:27:42", "content": "I was trying that on encoder output. Validation loss went down nicely, but LD went up.
\nI didn't try it on decoder output, though.
How would you go about performing multi task learning in the decoder or encoder?
", "rawMarkdown": "How would you go about performing multi task learning in the decoder or encoder?", "votes": null }, { "id": "1320826", "postDate": "05/24/2021 10:57:32", "content": "I made 27 additional targets for each molecule (number of each atoms, does it have a layer, how many +/- in a layer, etc.)
\nEncoder:
\nDecoder:
\nThese are just what I came up with and I only tried the second one on the encoder side, with no success. I'm not aware of papers on additional targets, but you may want to check if there are some previous work on it if you are going that way.
", "rawMarkdown": "I made 27 additional targets for each molecule (number of each atoms, does it have a layer, how many +/- in a layer, etc.)\n\nEncoder:\n- You can just average/maxpool or do anything with encoder output to then create your outputs for the additional targets, or;\n- I added 32 embeddings that I concatenated to each input of encoder. I calculated a loss for the first 27 of these at encoder output.\n\nDecoder:\n- You could first predict your additional targets from your sos token, or add new tokens (that might be better) that you also predict step-by-step or at once, or you can try anything you can come up with that seems reasonable.\n\nThese are just what I came up with and I only tried the second one on the encoder side, with no success. I'm not aware of papers on additional targets, but you may want to check if there are some previous work on it if you are going that way.", "votes": null }, { "id": "1320834", "postDate": "05/24/2021 11:05:29", "content": "This sounds very interesting! Sorry for the dumb question, but what does a layer mean(for the targets)?
", "rawMarkdown": "This sounds very interesting! Sorry for the dumb question, but what does a layer mean(for the targets)?", "votes": null }, { "id": "1320846", "postDate": "05/24/2021 11:16:12", "content": "One once said to me that there are no dumb questions :)
\nAn inchi is built up from layers.
\nEach layer starts with a \"/\" sign. I think it's good to know at least a little bit about how an inchi is made up so you can judge for yourself if an idea is good or not, or even come up with new ideas yourself.
Oh I see! Thanks for the help!
", "rawMarkdown": "Oh I see! Thanks for the help!", "votes": null }, { "id": "1320853", "postDate": "05/24/2021 11:21:58", "content": "I will definitely start looking into the design of InChI.
", "rawMarkdown": "I will definitely start looking into the design of InChI.", "votes": null }, { "id": "1333504", "postDate": "06/02/2021 19:23:59", "content": "An idea I had is to place decoder heads in series. (Decoder 1 is responsible for first 'x' characters, Decoder 2 for next 'x' chars, etc). It's not multitask learning, but it does segment the output. It also drives up the number of trainable params without affecting the number of computations during inference (since these are put in series instead of stacked vertically). Training on this is finicky and I haven't yet spent a lot of time working through it.
", "rawMarkdown": "An idea I had is to place decoder heads in series. (Decoder 1 is responsible for first 'x' characters, Decoder 2 for next 'x' chars, etc). It's not multitask learning, but it does segment the output. It also drives up the number of trainable params without affecting the number of computations during inference (since these are put in series instead of stacked vertically). Training on this is finicky and I haven't yet spent a lot of time working through it.", "votes": null } ], "comments": [ { "id": 1320720, "author_name": "nofreewill", "author_url": "", "post_date": "05/24/2021 09:27:42", "content": "I was trying that on encoder output. Validation loss went down nicely, but LD went up.
\nI didn't try it on decoder output, though.
How would you go about performing multi task learning in the decoder or encoder?
", "votes": null, "replies": [] }, { "id": 1320826, "author_name": "nofreewill", "author_url": "", "post_date": "05/24/2021 10:57:32", "content": "I made 27 additional targets for each molecule (number of each atoms, does it have a layer, how many +/- in a layer, etc.)
\nEncoder:
\nDecoder:
\nThese are just what I came up with and I only tried the second one on the encoder side, with no success. I'm not aware of papers on additional targets, but you may want to check if there are some previous work on it if you are going that way.
", "votes": null, "replies": [] }, { "id": 1320834, "author_name": "andrewshao05", "author_url": "", "post_date": "05/24/2021 11:05:29", "content": "This sounds very interesting! Sorry for the dumb question, but what does a layer mean(for the targets)?
", "votes": null, "replies": [] }, { "id": 1320846, "author_name": "nofreewill", "author_url": "", "post_date": "05/24/2021 11:16:12", "content": "One once said to me that there are no dumb questions :)
\nAn inchi is built up from layers.
\nEach layer starts with a \"/\" sign. I think it's good to know at least a little bit about how an inchi is made up so you can judge for yourself if an idea is good or not, or even come up with new ideas yourself.
Oh I see! Thanks for the help!
", "votes": null, "replies": [] }, { "id": 1320853, "author_name": "andrewshao05", "author_url": "", "post_date": "05/24/2021 11:21:58", "content": "I will definitely start looking into the design of InChI.
", "votes": null, "replies": [] } ] }, { "id": 1333504, "author_name": "mvenou", "author_url": "", "post_date": "06/02/2021 19:23:59", "content": "An idea I had is to place decoder heads in series. (Decoder 1 is responsible for first 'x' characters, Decoder 2 for next 'x' chars, etc). It's not multitask learning, but it does segment the output. It also drives up the number of trainable params without affecting the number of computations during inference (since these are put in series instead of stacked vertically). Training on this is finicky and I haven't yet spent a lot of time working through it.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1320454": "I would be interested to know if anyone has tried multitask learning for captioning. As we know that InChI follows a rigorous format, each sub-component of the labels relying on the shared representation of the image features, this might motivate multitask learning for each distinct portion of the caption.", "1320720": "I was trying that on encoder output. Validation loss went down nicely, but LD went up.\nI didn't try it on decoder output, though.", "1320796": "How would you go about performing multi task learning in the decoder or encoder?", "1320826": "I made 27 additional targets for each molecule (number of each atoms, does it have a layer, how many +/- in a layer, etc.)\n\nEncoder:\n- You can just average/maxpool or do anything with encoder output to then create your outputs for the additional targets, or;\n- I added 32 embeddings that I concatenated to each input of encoder. I calculated a loss for the first 27 of these at encoder output.\n\nDecoder:\n- You could first predict your additional targets from your sos token, or add new tokens (that might be better) that you also predict step-by-step or at once, or you can try anything you can come up with that seems reasonable.\n\nThese are just what I came up with and I only tried the second one on the encoder side, with no success. I'm not aware of papers on additional targets, but you may want to check if there are some previous work on it if you are going that way.", "1320834": "This sounds very interesting! Sorry for the dumb question, but what does a layer mean(for the targets)?", "1320846": "One once said to me that there are no dumb questions :)\nAn inchi is built up from layers.\nEach layer starts with a \"/\" sign. I think it's good to know at least a little bit about how an inchi is made up so you can judge for yourself if an idea is good or not, or even come up with new ideas yourself.", "1320851": "Oh I see! Thanks for the help!", "1320853": "I will definitely start looking into the design of InChI.", "1333504": "An idea I had is to place decoder heads in series. (Decoder 1 is responsible for first 'x' characters, Decoder 2 for next 'x' chars, etc). It's not multitask learning, but it does segment the output. It also drives up the number of trainable params without affecting the number of computations during inference (since these are put in series instead of stacked vertically). Training on this is finicky and I haven't yet spent a lot of time working through it." }, "source": "meta" }