{ "id": 236125, "title": "Any idea about ensemble? ", "url": "/competitions/bms-molecular-translation/discussion/236125", "author_name": "AIFahim", "post_date": "2021-05-03T00:46:43.400000", "votes": 17, "comment_count": 29, "views": 0, "content": "

As I assume ensemble may increase the score. Is it possible to ensemble two or three generated CSVs or Any unique suggestions on ensemble?

", "messages": [ { "id": 1291355, "postDate": "2021-05-03T00:46:43.400Z", "content": "

As I assume ensemble may increase the score. Is it possible to ensemble two or three generated CSVs or Any unique suggestions on ensemble?

", "rawMarkdown": "As I assume ensemble may increase the score. Is it possible to ensemble two or three generated CSVs or Any unique suggestions on ensemble? ", "votes": 17 }, { "id": 1292449, "postDate": "2021-05-04T00:52:04.750Z", "content": "

This is my idea:
\ninput: t1, t2 (from two submissions)

\n
    \n
  1. t1 = pad(tokenizer.encode(t1))
    2. \n
  2. t2 = pad(tokenizer.encode(t2))
    3. \n
  3. t = vote(t1, t2)
    4. \n
  4. final = tokenizer.decode(t)
    5. \n
\n

This require your submission was delimited such that your tokenizer can encode the InChI correctly.

", "rawMarkdown": "This is my idea:\ninput: t1, t2 (from two submissions)\n1. t1 = pad(tokenizer.encode(t1))\n2. t2 = pad(tokenizer.encode(t2))\n3. t = vote(t1, t2)\n4. final = tokenizer.decode(t)\n\nThis require your submission was delimited such that your tokenizer can encode the InChI correctly.", "votes": 3 }, { "id": 1291648, "postDate": "2021-05-03T08:12:57.453Z", "content": "

Not on csvs but when you generate your predictions step by step. If you have two models with same vocabulary, you can ensemble them at each step.

", "rawMarkdown": "Not on csvs but when you generate your predictions step by step. If you have two models with same vocabulary, you can ensemble them at each step.", "votes": 3, "replies": [ { "id": 1291654, "postDate": "2021-05-03T08:16:44.753Z", "content": "

Thanks a lot for your insights. Can you refer some source it will help?

", "rawMarkdown": "Thanks a lot for your insights. Can you refer some source it will help?" }, { "id": 1291670, "postDate": "2021-05-03T08:29:40.227Z", "content": "

My current LB is single model with no beam-search. I just wanted a quick submission.
\nBut based on my CV, ensembling does make quite a difference.
\nIf you have more GPUs in separate computers then you better do the ensemble. But if you have one/multiple GPUs in one computer, I don't know if it is better to train multiple models or one model for longer. But I guess that in that case, ensembling is better, too.
\nI suggest searching for papers like this to make sure you go for the optimum:
\nhttps://arxiv.org/pdf/2005.00570.pdf

", "rawMarkdown": "My current LB is single model with no beam-search. I just wanted a quick submission.\nBut based on my CV, ensembling does make quite a difference.\nIf you have more GPUs in separate computers then you better do the ensemble. But if you have one/multiple GPUs in one computer, I don't know if it is better to train multiple models or one model for longer. But I **guess** that in that case, ensembling is better, too.\nI suggest searching for papers like this to make sure you go for the optimum:\nhttps://arxiv.org/pdf/2005.00570.pdf", "votes": 3 }, { "id": 1291683, "postDate": "2021-05-03T08:37:47.290Z", "content": "
\n

single model with no beam-search.

\n
\n

You achieved the top 4! Just awesome.

", "rawMarkdown": ">single model with no beam-search.\n\nYou achieved the top 4! Just awesome.", "votes": 2 }, { "id": 1291694, "postDate": "2021-05-03T08:49:47.010Z", "content": "

Thanks, I'm quite happy with that result! :D
\nBtw., a lot of people submitted a long time ago with pretty good scores. They are expected to have better scoring models now than my current submission :)
\n(I'm happy with my new 3090)

", "rawMarkdown": "Thanks, I'm quite happy with that result! :D\nBtw., a lot of people submitted a long time ago with pretty good scores. They are expected to have better scoring models now than my current submission :)\n(I'm happy with my new 3090)", "votes": 2 }, { "id": 1291803, "postDate": "2021-05-03T10:58:09.143Z", "content": "

For me a 3090 is not being enough though. 60+ hours per experiment. How long does it take to you?

", "rawMarkdown": "For me a 3090 is not being enough though. 60+ hours per experiment. How long does it take to you?" }, { "id": 1291892, "postDate": "2021-05-03T12:24:11.140Z", "content": "

@nofreewill how much did you pay for 3090?

", "rawMarkdown": "@nofreewill how much did you pay for 3090?", "votes": 1 }, { "id": 1291969, "postDate": "2021-05-03T13:43:20.587Z", "content": "

A little bit over $4k. I bought it explicitly for this competition.

", "rawMarkdown": "A little bit over $4k. I bought it explicitly for this competition.", "votes": 4 }, { "id": 1291972, "postDate": "2021-05-03T13:45:22.643Z", "content": "

Experiments take me long time, too. I don't go as far as 60 hours, though. But then comes the question, how well will the outcome extrapolate for longer training?!

", "rawMarkdown": "Experiments take me long time, too. I don't go as far as 60 hours, though. But then comes the question, how well will the outcome extrapolate for longer training?!", "votes": 1 }, { "id": 1291974, "postDate": "2021-05-03T13:46:55.463Z", "content": "

I don't mind the money, I just want a solo gold medal in my CV (Curriculum Vitae). I hope I can achieve that in this comp : )

", "rawMarkdown": "I don't mind the money, I just want a solo gold medal in my CV (Curriculum Vitae). I hope I can achieve that in this comp : )", "votes": 14, "replies": [ { "id": 1292768, "postDate": "2021-05-04T09:13:43.357Z", "content": "

Best of luck :)

", "rawMarkdown": "Best of luck :)", "votes": 1 } ] }, { "id": 1292776, "postDate": "2021-05-04T09:17:20.470Z", "content": "

Thank you! :)) You too!

", "rawMarkdown": "Thank you! :)) You too!", "votes": 1 }, { "id": 1302975, "postDate": "2021-05-11T20:17:02.890Z", "content": "

4k for 3090 is overpriced! Why not spend 5500 and get a A6000? 😃

", "rawMarkdown": "4k for 3090 is overpriced! Why not spend 5500 and get a A6000? 😃", "votes": 1 }, { "id": 1303140, "postDate": "2021-05-11T23:43:03.947Z", "content": "

That would have been great. But one only has that much money, and rent doesn't pay itself. 😅

", "rawMarkdown": "That would have been great. But one only has that much money, and rent doesn't pay itself. 😅" }, { "id": 1303159, "postDate": "2021-05-12T00:07:37.007Z", "content": "

Oh and here an A6000 is more like 6.8k :/

", "rawMarkdown": "Oh and here an A6000 is more like 6.8k :/" }, { "id": 1303192, "postDate": "2021-05-12T00:33:26.687Z", "content": "

Hungary is part of the EU, so you should be able to order GPUs from another country relatively easily (my guess). E.g. in Germany they are far cheaper. Currently the 'cheapest' RTX 3090 are for example around 2400€. At the time of your ordering they should have been around 2800€.
\nApart from that used GPUs, are probably a far better deal now, even though they should also vastly more expansive currently.

", "rawMarkdown": "Hungary is part of the EU, so you should be able to order GPUs from another country relatively easily (my guess). E.g. in Germany they are far cheaper. Currently the 'cheapest' RTX 3090 are for example around 2400€. At the time of your ordering they should have been around 2800€.\nApart from that used GPUs, are probably a far better deal now, even though they should also vastly more expansive currently." }, { "id": 1303569, "postDate": "2021-05-12T06:23:34.800Z", "content": "

I wanted to train with best reasonable gpu I can get for the longest time possible.
\nThanks for your care guys, but I'm well aware of what I wanted, and I wouldn't do anything differently if I had the chance.

", "rawMarkdown": "I wanted to train with best reasonable gpu I can get for the longest time possible.\nThanks for your care guys, but I'm well aware of what I wanted, and I wouldn't do anything differently if I had the chance." }, { "id": 1306113, "postDate": "2021-05-13T16:04:44.787Z", "content": "

\"https://i.imgflip.com/59etp4.jpg\"

", "rawMarkdown": "![https://i.imgflip.com/59etp4.jpg](https://i.imgflip.com/59etp4.jpg)", "votes": 2 } ] }, { "id": 1291395, "postDate": "2021-05-03T02:32:35.277Z", "content": "

train another seq to seq ensembling model on top of all your models

", "rawMarkdown": "train another seq to seq ensembling model on top of all your models", "votes": 3, "replies": [ { "id": 1295509, "postDate": "2021-05-06T14:08:03.167Z", "content": "

Can you explain the idea in detail, please?

", "rawMarkdown": "Can you explain the idea in detail, please?" } ] }, { "id": 1295095, "postDate": "2021-05-06T07:44:28.987Z", "content": "

If you have enough models and the same solution gets proposed by several, then a (weighted) voting ensemble is a simple idea. I'd guess the problem is what to do when every model proposes something different. You could go with the \"best\" model (based on CV) or you could go with the solution that minimizes the competition metric if you assume all proposals are equally likely (I guess you could again weight this).

\n

If you use something like beam search in each model, then you have different possible solutions coming out of each model, then I imagine that letting each model distribute it's vote(s) amongst these solutions would improve on a voting ensemble. However, I'm assuming that beam search kind of narrows the solution space, so treating what you get as probabilities is not quite right (but it may very well work, I guess just trying it out in CV is the way to determine how well that works).

", "rawMarkdown": "If you have enough models and the same solution gets proposed by several, then a (weighted) voting ensemble is a simple idea. I'd guess the problem is what to do when every model proposes something different. You could go with the \"best\" model (based on CV) or you could go with the solution that minimizes the competition metric if you assume all proposals are equally likely (I guess you could again weight this).\n\nIf you use something like beam search in each model, then you have different possible solutions coming out of each model, then I imagine that letting each model distribute it's vote(s) amongst these solutions would improve on a voting ensemble. However, I'm assuming that beam search kind of narrows the solution space, so treating what you get as probabilities is not quite right (but it may very well work, I guess just trying it out in CV is the way to determine how well that works).", "votes": 2, "replies": [ { "id": 1325005, "postDate": "2021-05-27T12:29:49.080Z", "content": "

did you try this?

", "rawMarkdown": "did you try this?" } ] }, { "id": 1326193, "postDate": "2021-05-28T09:54:38.443Z", "content": "

@nofreewill how long time to generator prediction by ensembling step by step?

", "rawMarkdown": "@nofreewill how long time to generator prediction by ensembling step by step?", "replies": [ { "id": 1326223, "postDate": "2021-05-28T10:28:27.010Z", "content": "

One model without beam search takes about 2 hours, if I remember correctly. It's on a 3090.
\nTwo models takes exactly twice as much time.

", "rawMarkdown": "One model without beam search takes about 2 hours, if I remember correctly. It's on a 3090.\nTwo models takes exactly twice as much time." }, { "id": 1326340, "postDate": "2021-05-28T12:10:18.857Z", "content": "

Thanks! Only 2 hours, it's so quick👍. I need 10 hours on a V100😭

", "rawMarkdown": "Thanks! Only 2 hours, it's so quick👍. I need 10 hours on a V100😭" }, { "id": 1326351, "postDate": "2021-05-28T12:17:07.920Z", "content": "

Do you sort the samples based on your previous prediction's token lengths? Because that gives you a pretty good speedup (×1.5-2 or somewhere around that).

", "rawMarkdown": "Do you sort the samples based on your previous prediction's token lengths? Because that gives you a pretty good speedup (×1.5-2 or somewhere around that)." } ] }, { "id": 1301083, "postDate": "2021-05-11T00:14:44.783Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1291454, "postDate": "2021-05-03T04:20:42.317Z", "rawMarkdown": "", "votes": 2, "isDeleted": true } ], "comments": [ { "id": 1292449, "author_name": "Matrix", "author_url": "", "post_date": "2021-05-04T00:52:04.750000", "content": "

This is my idea:
\ninput: t1, t2 (from two submissions)

\n
    \n
  1. t1 = pad(tokenizer.encode(t1))
    2. \n
  2. t2 = pad(tokenizer.encode(t2))
    3. \n
  3. t = vote(t1, t2)
    4. \n
  4. final = tokenizer.decode(t)
    5. \n
\n

This require your submission was delimited such that your tokenizer can encode the InChI correctly.

", "votes": 3, "replies": [] }, { "id": 1291648, "author_name": "nofreewill42", "author_url": "", "post_date": "2021-05-03T08:12:57.453000", "content": "

Not on csvs but when you generate your predictions step by step. If you have two models with same vocabulary, you can ensemble them at each step.

", "votes": 3, "replies": [ { "id": 1291654, "author_name": "AIFahim", "author_url": "", "post_date": "2021-05-03T08:16:44.753000", "content": "

Thanks a lot for your insights. Can you refer some source it will help?

", "votes": 0, "replies": [] }, { "id": 1291670, "author_name": "nofreewill42", "author_url": "", "post_date": "2021-05-03T08:29:40.227000", "content": "

My current LB is single model with no beam-search. I just wanted a quick submission.
\nBut based on my CV, ensembling does make quite a difference.
\nIf you have more GPUs in separate computers then you better do the ensemble. But if you have one/multiple GPUs in one computer, I don't know if it is better to train multiple models or one model for longer. But I guess that in that case, ensembling is better, too.
\nI suggest searching for papers like this to make sure you go for the optimum:
\nhttps://arxiv.org/pdf/2005.00570.pdf

", "votes": 3, "replies": [] }, { "id": 1291683, "author_name": "AIFahim", "author_url": "", "post_date": "2021-05-03T08:37:47.290000", "content": "
\n

single model with no beam-search.

\n
\n

You achieved the top 4! Just awesome.

", "votes": 2, "replies": [] }, { "id": 1291694, "author_name": "nofreewill42", "author_url": "", "post_date": "2021-05-03T08:49:47.010000", "content": "

Thanks, I'm quite happy with that result! :D
\nBtw., a lot of people submitted a long time ago with pretty good scores. They are expected to have better scoring models now than my current submission :)
\n(I'm happy with my new 3090)

", "votes": 2, "replies": [] }, { "id": 1291803, "author_name": "Claudio Verdú Ruiz", "author_url": "", "post_date": "2021-05-03T10:58:09.143000", "content": "

For me a 3090 is not being enough though. 60+ hours per experiment. How long does it take to you?

", "votes": 0, "replies": [] }, { "id": 1291892, "author_name": "tugstugi", "author_url": "", "post_date": "2021-05-03T12:24:11.140000", "content": "

@nofreewill how much did you pay for 3090?

", "votes": 1, "replies": [] }, { "id": 1291969, "author_name": "nofreewill42", "author_url": "", "post_date": "2021-05-03T13:43:20.587000", "content": "

A little bit over $4k. I bought it explicitly for this competition.

", "votes": 4, "replies": [] }, { "id": 1291972, "author_name": "nofreewill42", "author_url": "", "post_date": "2021-05-03T13:45:22.643000", "content": "

Experiments take me long time, too. I don't go as far as 60 hours, though. But then comes the question, how well will the outcome extrapolate for longer training?!

", "votes": 1, "replies": [] }, { "id": 1291974, "author_name": "nofreewill42", "author_url": "", "post_date": "2021-05-03T13:46:55.463000", "content": "

I don't mind the money, I just want a solo gold medal in my CV (Curriculum Vitae). I hope I can achieve that in this comp : )

", "votes": 14, "replies": [ { "id": 1292768, "author_name": "Charles", "author_url": "", "post_date": "2021-05-04T09:13:43.357000", "content": "

Best of luck :)

", "votes": 1, "replies": [] } ] }, { "id": 1292776, "author_name": "nofreewill42", "author_url": "", "post_date": "2021-05-04T09:17:20.470000", "content": "

Thank you! :)) You too!

", "votes": 1, "replies": [] }, { "id": 1302975, "author_name": "sin", "author_url": "", "post_date": "2021-05-11T20:17:02.890000", "content": "

4k for 3090 is overpriced! Why not spend 5500 and get a A6000? 😃

", "votes": 1, "replies": [] }, { "id": 1303140, "author_name": "nofreewill42", "author_url": "", "post_date": "2021-05-11T23:43:03.947000", "content": "

That would have been great. But one only has that much money, and rent doesn't pay itself. 😅

", "votes": 0, "replies": [] }, { "id": 1303159, "author_name": "nofreewill42", "author_url": "", "post_date": "2021-05-12T00:07:37.007000", "content": "

Oh and here an A6000 is more like 6.8k :/

", "votes": 0, "replies": [] }, { "id": 1303192, "author_name": "Gabriel Lindenmaier", "author_url": "", "post_date": "2021-05-12T00:33:26.687000", "content": "

Hungary is part of the EU, so you should be able to order GPUs from another country relatively easily (my guess). E.g. in Germany they are far cheaper. Currently the 'cheapest' RTX 3090 are for example around 2400€. At the time of your ordering they should have been around 2800€.
\nApart from that used GPUs, are probably a far better deal now, even though they should also vastly more expansive currently.

", "votes": 0, "replies": [] }, { "id": 1303569, "author_name": "nofreewill42", "author_url": "", "post_date": "2021-05-12T06:23:34.800000", "content": "

I wanted to train with best reasonable gpu I can get for the longest time possible.
\nThanks for your care guys, but I'm well aware of what I wanted, and I wouldn't do anything differently if I had the chance.

", "votes": 0, "replies": [] }, { "id": 1306113, "author_name": "Serigne ", "author_url": "", "post_date": "2021-05-13T16:04:44.787000", "content": "

\"https://i.imgflip.com/59etp4.jpg\"

", "votes": 2, "replies": [] } ] }, { "id": 1291395, "author_name": "hengck23", "author_url": "", "post_date": "2021-05-03T02:32:35.277000", "content": "

train another seq to seq ensembling model on top of all your models

", "votes": 3, "replies": [ { "id": 1295509, "author_name": "Mohammed Rizin V K", "author_url": "", "post_date": "2021-05-06T14:08:03.167000", "content": "

Can you explain the idea in detail, please?

", "votes": 0, "replies": [] } ] }, { "id": 1295095, "author_name": "Björn", "author_url": "", "post_date": "2021-05-06T07:44:28.987000", "content": "

If you have enough models and the same solution gets proposed by several, then a (weighted) voting ensemble is a simple idea. I'd guess the problem is what to do when every model proposes something different. You could go with the \"best\" model (based on CV) or you could go with the solution that minimizes the competition metric if you assume all proposals are equally likely (I guess you could again weight this).

\n

If you use something like beam search in each model, then you have different possible solutions coming out of each model, then I imagine that letting each model distribute it's vote(s) amongst these solutions would improve on a voting ensemble. However, I'm assuming that beam search kind of narrows the solution space, so treating what you get as probabilities is not quite right (but it may very well work, I guess just trying it out in CV is the way to determine how well that works).

", "votes": 2, "replies": [ { "id": 1325005, "author_name": "Vikrant", "author_url": "", "post_date": "2021-05-27T12:29:49.080000", "content": "

did you try this?

", "votes": 0, "replies": [] } ] }, { "id": 1326193, "author_name": "ultra_zhl", "author_url": "", "post_date": "2021-05-28T09:54:38.443000", "content": "

@nofreewill how long time to generator prediction by ensembling step by step?

", "votes": 0, "replies": [ { "id": 1326223, "author_name": "nofreewill42", "author_url": "", "post_date": "2021-05-28T10:28:27.010000", "content": "

One model without beam search takes about 2 hours, if I remember correctly. It's on a 3090.
\nTwo models takes exactly twice as much time.

", "votes": 0, "replies": [] }, { "id": 1326340, "author_name": "ultra_zhl", "author_url": "", "post_date": "2021-05-28T12:10:18.857000", "content": "

Thanks! Only 2 hours, it's so quick👍. I need 10 hours on a V100😭

", "votes": 0, "replies": [] }, { "id": 1326351, "author_name": "nofreewill42", "author_url": "", "post_date": "2021-05-28T12:17:07.920000", "content": "

Do you sort the samples based on your previous prediction's token lengths? Because that gives you a pretty good speedup (×1.5-2 or somewhere around that).

", "votes": 0, "replies": [] } ] }, { "id": 1301083, "author_name": "", "author_url": "", "post_date": "2021-05-11T00:14:44.783000", "content": "", "votes": 0, "replies": [] }, { "id": 1291454, "author_name": "", "author_url": "", "post_date": "2021-05-03T04:20:42.317000", "content": "", "votes": 2, "replies": [] } ], "raw_markdown_by_id": { "1291355": "As I assume ensemble may increase the score. Is it possible to ensemble two or three generated CSVs or Any unique suggestions on ensemble? ", "1292449": "This is my idea:\ninput: t1, t2 (from two submissions)\n1. t1 = pad(tokenizer.encode(t1))\n2. t2 = pad(tokenizer.encode(t2))\n3. t = vote(t1, t2)\n4. final = tokenizer.decode(t)\n\nThis require your submission was delimited such that your tokenizer can encode the InChI correctly.", "1291648": "Not on csvs but when you generate your predictions step by step. If you have two models with same vocabulary, you can ensemble them at each step.", "1291395": "train another seq to seq ensembling model on top of all your models", "1295095": "If you have enough models and the same solution gets proposed by several, then a (weighted) voting ensemble is a simple idea. I'd guess the problem is what to do when every model proposes something different. You could go with the \"best\" model (based on CV) or you could go with the solution that minimizes the competition metric if you assume all proposals are equally likely (I guess you could again weight this).\n\nIf you use something like beam search in each model, then you have different possible solutions coming out of each model, then I imagine that letting each model distribute it's vote(s) amongst these solutions would improve on a voting ensemble. However, I'm assuming that beam search kind of narrows the solution space, so treating what you get as probabilities is not quite right (but it may very well work, I guess just trying it out in CV is the way to determine how well that works).", "1326193": "@nofreewill how long time to generator prediction by ensembling step by step?", "1301083": "", "1291454": "" } }