{ "id": 230887, "title": "we are watching a race of gpu power", "url": "/competitions/bms-molecular-translation/discussion/230887", "author_name": "hengck23", "post_date": "2021-04-06T02:45:01.546000", "votes": 46, "comment_count": 35, "views": 0, "content": "

the loss dynamics of LSTM (and other seq model) is like this

\n\n

eventually, everyone will reach lb score of about 1 to 2 …if they have enough GPU resource and their seq model has \"enough capacity\"

\n

every one is racing now. final top lb score could be in the range of 0.5 to 0.8

", "messages": [ { "id": 1264252, "postDate": "2021-04-06T02:45:01.547Z", "content": "

the loss dynamics of LSTM (and other seq model) is like this

\n\n

eventually, everyone will reach lb score of about 1 to 2 …if they have enough GPU resource and their seq model has \"enough capacity\"

\n

every one is racing now. final top lb score could be in the range of 0.5 to 0.8

", "rawMarkdown": "the loss dynamics of LSTM (and other seq model) is like this\n- loss decrease for the head part of the seq first in early training\n- short sequences get corrected first\n- as the training goes, sometimes you see the loss is stagnant, But actually, the tail part of the long sequences are being corrected. Long sequence are rare, that is why the improvement of loss are minor\n- but as long as you keep your GPU running, lb score will increase\n\neventually, everyone will reach lb score of about 1 to 2 ...if they have enough GPU resource and their seq model has \"enough capacity\"\n\nevery one is racing now. final top lb score could be in the range of 0.5 to 0.8", "votes": 45 }, { "id": 1265554, "postDate": "2021-04-07T02:36:31.630Z", "content": "

Just manually assigned structures for 200 random images from training set, using computational chemist's intuition. 7% are wrong (7/100 and 7/100)

\n\n

Let's say 1 false prediction increases edit distance by 30 on average . According to this estimation, the winning models with LD < 2 are probably overfitting to THE particular molecular sketcher & noise generator, from a pure scientific perspective.

\n

We are watching a race of overfitting

", "rawMarkdown": "Just manually assigned structures for 200 random images from training set, using computational chemist's intuition. 7% are wrong (7/100 and 7/100)\n\n* 10 are due to *ghost* terminal methyl group (e.g., `4a4f6c36a791`)\n* 1 is due to fuzzy stereo depiction (e.g., `4080f41a9577`)\n* 2 are due to missing bonds in the complex multi-ring structure containing crossing bonds (e.g., `94a92ba4a8ae`)\n* 1 is due to extremely fuzzy fluorine depiction (e.g., `e159f8ad60e9`)\n\nLet's say 1 false prediction increases edit distance by 30 on average . According to this estimation, the winning models with LD < 2 are probably overfitting to THE particular molecular sketcher & noise generator, from a pure scientific perspective. \n\n**We are watching a race of overfitting**", "votes": 18, "replies": [ { "id": 1271090, "postDate": "2021-04-12T09:45:07.413Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1276721, "postDate": "2021-04-17T22:06:02.283Z", "content": "

This is valuable insight. Hopefully the competition sponsors are cognisant of this.

", "rawMarkdown": "This is valuable insight. Hopefully the competition sponsors are cognisant of this. " } ] }, { "id": 1264398, "postDate": "2021-04-06T06:04:23.313Z", "content": "

Kaggle should provide some GCP quota for this competition. @addisonhoward

", "rawMarkdown": "Kaggle should provide some GCP quota for this competition. @addisonhoward ", "votes": 14, "replies": [ { "id": 1271793, "postDate": "2021-04-12T23:13:05.250Z", "content": "

Yeah there must be some.

", "rawMarkdown": "Yeah there must be some." } ] }, { "id": 1276723, "postDate": "2021-04-17T22:08:37.163Z", "content": "

I am just glad that the weather here is getting colder as autumn sets in - so I am appreciative of the extra warmth in my house. =)

", "rawMarkdown": "I am just glad that the weather here is getting colder as autumn sets in - so I am appreciative of the extra warmth in my house. =)", "votes": 8 }, { "id": 1268679, "postDate": "2021-04-09T16:33:43.563Z", "content": "

Isn't it true for all image competitions? What is different from other image competitions here?

", "rawMarkdown": "Isn't it true for all image competitions? What is different from other image competitions here?", "votes": 5, "replies": [ { "id": 1269167, "postDate": "2021-04-10T08:34:40.290Z", "content": "

This competition seems way more resource hungry than other competitions. The data is huge and training longer seems to help quite a bit from what I can gather from the discussions. There are many cv competitions where you dont need much HW to score high, recent examples: VinBigData, Rainforest, NFL

", "rawMarkdown": "This competition seems way more resource hungry than other competitions. The data is huge and training longer seems to help quite a bit from what I can gather from the discussions. There are many cv competitions where you dont need much HW to score high, recent examples: VinBigData, Rainforest, NFL", "votes": 7 }, { "id": 1269177, "postDate": "2021-04-10T09:00:26.870Z", "content": "

There is a long tail problem here. you really need the number of iterations!

", "rawMarkdown": "There is a long tail problem here. you really need the number of iterations!", "votes": 7 }, { "id": 1269189, "postDate": "2021-04-10T09:18:58.327Z", "content": "

@philippsinger Sure, TGS, Bengali were small data too. But there are also large data CV competitions, this one is no the first one.

\n

@hengck23 I believe you, my point is that it is not the first compute intensive comp.

", "rawMarkdown": "@philippsinger Sure, TGS, Bengali were small data too. But there are also large data CV competitions, this one is no the first one.\n\n@hengck23 I believe you, my point is that it is not the first compute intensive comp.", "votes": 2 }, { "id": 1269514, "postDate": "2021-04-10T15:49:59.660Z", "content": "

Right, there are other compute intensive ones.

", "rawMarkdown": "Right, there are other compute intensive ones.", "votes": 1 } ] }, { "id": 1264763, "postDate": "2021-04-06T12:14:04.003Z", "content": "

This seems to be a tendency, it can be confirmed by the lack of \"end-to-end\" notebooks. Many competitions lack of notebooks with a training part, because if they train with Kaggle resources they can't achive a remarkable score.

", "rawMarkdown": "This seems to be a tendency, it can be confirmed by the lack of \"end-to-end\" notebooks. Many competitions lack of notebooks with a training part, because if they train with Kaggle resources they can't achive a remarkable score. ", "votes": 5 }, { "id": 1268773, "postDate": "2021-04-09T18:26:34.763Z", "content": "

We don't have access to 1024 A100 at NVIDIA, contrarily to what some team name could suggest…

\n

We don't even have access to A100 at this point, except for very limited test. We work with V100 GPU, which is already quite good.

\n

Just to straighten expectations.

", "rawMarkdown": "We don't have access to 1024 A100 at NVIDIA, contrarily to what some team name could suggest...\n\nWe don't even have access to A100 at this point, except for very limited test. We work with V100 GPU, which is already quite good.\n\nJust to straighten expectations.", "votes": 3 }, { "id": 1279462, "postDate": "2021-04-20T23:45:27.177Z", "content": "

I will likely switch to TF and use the full power of TPU + TFRecords.

\n

I love Pytorch but training is really slow on single V100 - 16Gb. My current val score 2.20 (not submitted yet) took
\n more than 4 days of training. Let alone inference which is even slower (I don't use fairseq, but plain Pytorch transformer.) It doesn't let me room to experiment quickly new ideas.

\n

I converted all my transformer code to run on Torch/XLA but we still have big i/o bottleneck.

", "rawMarkdown": "I will likely switch to TF and use the full power of TPU + TFRecords. \n\nI love Pytorch but training is really slow on single V100 - 16Gb. My current val score 2.20 (not submitted yet) took \n more than 4 days of training. Let alone inference which is even slower (I don't use fairseq, but plain Pytorch transformer.) It doesn't let me room to experiment quickly new ideas. \n\nI converted all my transformer code to run on Torch/XLA but we still have big i/o bottleneck. ", "votes": 1, "replies": [ { "id": 1279477, "postDate": "2021-04-21T00:30:15.767Z", "content": "

you need yoshi

\n

\"\"

", "rawMarkdown": "you need yoshi\n\n![](https://i.ibb.co/0m446GH/Selection-076.png)", "votes": 7 } ] }, { "id": 1271288, "postDate": "2021-04-12T13:00:45.650Z", "content": "

@hengck23 , I observed some thing.
\nI trained a model on 1 epoch and the results were as follows.
\nNote : I am using same code as Used by Y Nakama and using some 3 lakhs images for validation .

\n

Train Loss : 0.53
\nVal Levhestein Distance : 32.X

\n

Later I trained for 1 more epoch based on above checkpoint , Results were as follows
\nTrain Loss = 0.29
\nVal Levhestein Distance = 95.X

\n

Is this normal has anyone observed such behaviour .

", "rawMarkdown": "@hengck23 , I observed some thing.\nI trained a model on 1 epoch and the results were as follows.\nNote : I am using same code as Used by Y Nakama and using some 3 lakhs images for validation .\n\nTrain Loss : 0.53\nVal Levhestein Distance : 32.X\n\nLater I trained for 1 more epoch based on above checkpoint , Results were as follows\nTrain Loss = 0.29\nVal Levhestein Distance = 95.X\n\nIs this normal has anyone observed such behaviour .", "votes": 1, "replies": [ { "id": 1271416, "postDate": "2021-04-12T15:34:48.943Z", "content": "

I found that using the wrong mean & std for normalization leads to unstable validation performance.
\nHere are the correct ones for the data. Maybe that was your problem. But in this case it should also lead to different performances in the 1st epoch over several runs with the same parameters.

\n

Other possible cause: Model is heavily overfitting to training set, but seems unlikely. Your parameter count would have to be really high.

", "rawMarkdown": "I found that using the wrong mean & std for normalization leads to unstable validation performance.\nHere are the [correct ones ](https://www.kaggle.com/c/bms-molecular-translation/discussion/232025)for the data. Maybe that was your problem. But in this case it should also lead to different performances in the 1st epoch over several runs with the same parameters. \n\nOther possible cause: Model is heavily overfitting to training set, but seems unlikely. Your parameter count would have to be really high." }, { "id": 1279215, "postDate": "2021-04-20T17:51:00.703Z", "content": "

@sayedathar11 can I ask?

\n", "rawMarkdown": "@sayedathar11 can I ask?\n- while you were doing inferencing of validation set . Were you fixing the orientation? Someone had such a problem in the lb score first epoch it was 25.xx then I don't remember much 96.xx or 46.xx don't know clearly", "votes": 1 }, { "id": 1279426, "postDate": "2021-04-20T22:33:59.117Z", "content": "

@morizin can you specify what do you meant by fixing orientation ? I didn't get you? Can you please elaborate ?
\nIf by orientation you meant augmentation then I want to add that I was adding train augmentation as follows.

\n

A.Compose([
\n A.Resize(CFG.size, CFG.size),
\n A.OneOf([A.HorizontalFlip( p = 0.5) , # Horizontal Flip is basically rotation by an Angle , Hence its okay to use it
\n A.VerticalFlip(p=0.5)] , p = 0.3 ),
\n A.RandomRotate90(p = 0.4),
\n Normalize(
\n mean=[0.485, 0.456, 0.406],
\n std=[0.229, 0.224, 0.225],
\n ),
\n ToTensorV2(),
\n ])

\n

And on validation set I was applying augmentation as follows .
\nCompose([
\n A.Resize(CFG.size, CFG.size),
\n A.Normalize(
\n mean=[0.485, 0.456, 0.406],
\n std=[0.229, 0.224, 0.225],
\n ),
\n ToTensorV2(),
\n ])

\n

Is there something problematic here ?

", "rawMarkdown": "@morizin can you specify what do you meant by fixing orientation ? I didn't get you? Can you please elaborate ?\nIf by orientation you meant augmentation then I want to add that I was adding train augmentation as follows.\n\nA.Compose([\n A.Resize(CFG.size, CFG.size),\n A.OneOf([A.HorizontalFlip( p = 0.5) , # Horizontal Flip is basically rotation by an Angle , Hence its okay to use it \n A.VerticalFlip(p=0.5)] , p = 0.3 ),\n A.RandomRotate90(p = 0.4),\n Normalize(\n mean=[0.485, 0.456, 0.406],\n std=[0.229, 0.224, 0.225],\n ),\n ToTensorV2(),\n ])\n\n\nAnd on validation set I was applying augmentation as follows .\nCompose([\n A.Resize(CFG.size, CFG.size),\n A.Normalize(\n mean=[0.485, 0.456, 0.406],\n std=[0.229, 0.224, 0.225],\n ),\n ToTensorV2(),\n ])\n\nIs there something problematic here ?" }, { "id": 1279439, "postDate": "2021-04-20T23:04:45.547Z", "content": "

Is there something problematic here ?
\nYes, you are using mean & std from image net for normalization. As long as you don't use a model pretrained on image net your normalization is useless.
\nThat is most likely the cause for your instability, as already mentioned by me above.

", "rawMarkdown": "`Is there something problematic here ?`\nYes, you are using mean & std from image net for normalization. As long as you don't use a model pretrained on image net your normalization is useless.\nThat is most likely the cause for your instability, as already mentioned by me above.", "votes": -1 }, { "id": 1279445, "postDate": "2021-04-20T23:11:08.607Z", "content": "

I am ensuring that my model uses pretrained weights of imagenet and also others are doing same I guess there is most probably some other error can you please let @morizin answer this .

", "rawMarkdown": "I am ensuring that my model uses pretrained weights of imagenet and also others are doing same I guess there is most probably some other error can you please let @morizin answer this .", "votes": -1 }, { "id": 1279449, "postDate": "2021-04-20T23:18:17.900Z", "content": "

Try the correct mean & std anyway to check, because your normalized images should have mean of 0 and std 1 anyway. I argue it is better to use mean & std of actual data if they are to far of from the original training data.
\nAnd @morizin can still answer, no problem.
\nI mean it is your problem, I just want to help. And I had the exact same problems and was using the same normalization as you.

", "rawMarkdown": "Try the correct mean & std anyway to check, because your normalized images should have mean of 0 and std 1 anyway. I argue it is better to use mean & std of actual data if they are to far of from the original training data. \nAnd @morizin can still answer, no problem.\nI mean it is your problem, I just want to help. And I had the **exact** same problems and was using the same normalization as you.", "votes": -1 }, { "id": 1281217, "postDate": "2021-04-22T17:57:26.753Z", "content": "

@sayedathar11 sorry for late reply
\nactually the some images of test is rotated which our training model finds weird
\ni am not sure that your problem is same as this. but some experienced this weird behavior in LB but not in CV.
\nhttps://www.kaggle.com/c/bms-molecular-translation/discussion/224257#1237479
\nI just started to explore the competition didnt try much till now.

\n

Is this normal has anyone observed such behaviour .
\ncan you check what happens in epoch 3 please. try only rotate as augmentation please for now

\n

because i think horizontal and vertical flip in train aug causes some problem

\n

\"\"

", "rawMarkdown": "@sayedathar11 sorry for late reply\nactually the some images of test is rotated which our training model finds weird\ni am not sure that your problem is same as this. but some experienced this weird behavior in LB but not in CV.\nhttps://www.kaggle.com/c/bms-molecular-translation/discussion/224257#1237479\nI just started to explore the competition didnt try much till now.\n\n`Is this normal has anyone observed such behaviour .`\ncan you check what happens in epoch 3 please. try only rotate as augmentation please for now\n\nbecause i think horizontal and vertical flip in train aug causes some problem\n\n![](https://th.bing.com/th/id/R85095686b39fa2fe6648eaf0c70e3999?rik=H%2feB0Rv55fOCrA&riu=http%3a%2f%2fwww.online-image-editor.com%2fhelp%2fimages%2frotate_flip_02.png)", "votes": 1 }, { "id": 1282024, "postDate": "2021-04-23T14:15:55.810Z", "content": "

@morizin I guess the problem was arising due to using some augmentations which causes instability in training ,I have removed those augmentations and Trying Efficientnet B1 as encoder , still will inform if I see any changes in 3rd epoch.

", "rawMarkdown": "@morizin I guess the problem was arising due to using some augmentations which causes instability in training ,I have removed those augmentations and Trying Efficientnet B1 as encoder , still will inform if I see any changes in 3rd epoch." } ] }, { "id": 1268334, "postDate": "2021-04-09T10:07:35.787Z", "content": "

i find an interesting results:

\n

\"\"
\nhttps://arxiv.org/pdf/2103.17239.pdf

\n

data-mining : let GPU run and mine information from data

", "rawMarkdown": "i find an interesting results:\n\n![](https://i.ibb.co/wJ9n3LV/Selection-054.png)\nhttps://arxiv.org/pdf/2103.17239.pdf\n\ndata-mining : let GPU run and mine information from data", "votes": 1, "replies": [ { "id": 1268376, "postDate": "2021-04-09T10:59:48.540Z", "content": "

\"\"

\n

how to improve your image transformer (depth and distillation)

", "rawMarkdown": "![](https://i.ibb.co/3NKgCHc/Selection-058.png)\n\nhow to improve your image transformer (depth and distillation)", "votes": 1 } ] }, { "id": 1281126, "postDate": "2021-04-22T16:30:55.207Z", "content": "

Just wanted to comment and post a photo showing results from my model on a holdout validation dataset. The model is an EfficientNetB7-->Attn/LSTM run for around 6 epochs. Scores a public LB of 5.77. Note that although I say \"INCHI LENGTH\" below, I am referring to the number of tokens that make up the ground truth InChI string.

\n

\"validation\"

\n

I've noticed that if I train for more epochs the LSD continues to get better for the longer sequences. I don't have a photo showing this but as you pointed out above… this is obviously what's supposed to happen.

\n

I did wonder how much of an LSD drop we should expect for longer sequences when compared to shorter sequences (assuming an infinite amount of training)? Essentially, should I assume that this model is capable of scoring ~1.5 if trained for a LOoooong time? Or will there be some remaining negative effects as longer sequences may just be more difficult to predict?

\n

PS:
\nAs a random aside, my model currently only predicts ~1.5% valid inchi strings when submitting.

", "rawMarkdown": "Just wanted to comment and post a photo showing results from my model on a holdout validation dataset. The model is an EfficientNetB7-->Attn/LSTM run for around 6 epochs. Scores a public LB of 5.77. Note that although I say \"INCHI LENGTH\" below, I am referring to the number of tokens that make up the ground truth InChI string.\n\n![validation](https://i.ibb.co/H7Ngq5J/Screen-Shot-2021-04-22-at-12-23-29-PM.png)\n\nI've noticed that if I train for more epochs the LSD continues to get better for the longer sequences. I don't have a photo showing this but as you pointed out above... this is obviously what's supposed to happen.\n\nI did wonder how much of an LSD drop we should expect for longer sequences when compared to shorter sequences (assuming an infinite amount of training)? Essentially, should I assume that this model is capable of scoring ~1.5 if trained for a LOoooong time? Or will there be some remaining negative effects as longer sequences may just be more difficult to predict?\n\n\nPS:\nAs a random aside, my model currently only predicts ~1.5% valid inchi strings when submitting.\n", "votes": 2, "replies": [ { "id": 1281318, "postDate": "2021-04-22T19:30:27.727Z", "content": "

I think LD~3 is totally achievable. Not sure about 1.5. But thanks for sharing valid InChI percent.

\n

Here is my baseline model on holdout set (N = 303k):
\n100% valid InChI. 78% perfect LD = 0 predictions. Mean LD = 7.2

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
mean LDperfect %NInChI length
6.33333367%320
3.86046578%107540
2.57459187%1771060
2.41655890%5958680
3.36763587%71533100
5.56301180%58537120
8.59069073%47304140
13.62956662%26145160
20.81064851%12585180
30.79225339%5757200
43.46851929%2160220
65.28977319%704240
106.1712069%257260
149.3897064%136280
182.1048390%124300
194.8641980%81320
207.9852941.5%68340
234.8260870%23360
247.5000000%6380
", "rawMarkdown": "I think LD~3 is totally achievable. Not sure about 1.5. But thanks for sharing valid InChI percent. \n\nHere is my baseline model on holdout set (N = 303k): \n100% valid InChI. 78% perfect LD = 0 predictions. Mean LD = 7.2\n\n| mean LD | perfect %| N | InChI length |\n| --- | --- | --- | --- |\n|6.333333 | 67% | 3 | 20|\n|3.860465 |78% |1075 | 40|\n|2.574591| 87% |17710| 60|\n|2.416558 |90%| 59586 | 80|\n|3.367635| 87%| 71533| 100|\n|5.563011 |80%|58537| 120|\n|8.590690 |73%|47304| 140|\n|13.629566 |62%|26145| 160|\n|20.810648| 51%|12585| 180|\n|30.792253 |39%|5757| 200|\n|43.468519| 29%| 2160 |220|\n|65.289773 |19%|704| 240|\n|106.171206 | 9%|257 |260|\n|149.389706 | 4%|136| 280|\n|182.104839 |0%|124| 300|\n|194.864198 | 0%|81 |320|\n|207.985294 | 1.5%|68 |340|\n|234.826087 | 0%|23 |360|\n|247.500000 | 0%|6 |380|", "votes": 2 } ] }, { "id": 1274357, "postDate": "2021-04-15T08:19:26.297Z", "content": "

Is it worth it to keep training or use the GPU power to mine crypto then ? 😄

", "rawMarkdown": "Is it worth it to keep training or use the GPU power to mine crypto then ? 😄", "votes": 2 }, { "id": 1298495, "postDate": "2021-05-08T23:44:27.817Z", "content": "

Many thanks for this super valuable insight.

", "rawMarkdown": "Many thanks for this super valuable insight." }, { "id": 1269444, "postDate": "2021-04-10T14:38:05.810Z", "content": "

Is there a way to sample more Long sequences to make training more efficient?

", "rawMarkdown": "Is there a way to sample more Long sequences to make training more efficient?", "replies": [ { "id": 1270013, "postDate": "2021-04-11T07:06:34.627Z", "content": "

\"Is there a way to sample more Long sequences to make training more efficient?\"

\n

we should find out the reason why long sequence get worst results first.
\ne.g. because the are bigger in image and resizing them to a fixed size causes loss of information?
\ne.g. we need to model larger range dependency because the atoms are far apart?
\ne.g. is there a correlation of sequence length and image size?

\n

when the reason become known, then we are decide how to sample, etc

", "rawMarkdown": "\"Is there a way to sample more Long sequences to make training more efficient?\"\n\nwe should find out the reason why long sequence get worst results first.\ne.g. because the are bigger in image and resizing them to a fixed size causes loss of information?\ne.g. we need to model larger range dependency because the atoms are far apart?\ne.g. is there a correlation of sequence length and image size?\n\nwhen the reason become known, then we are decide how to sample, etc", "votes": 5 } ] }, { "id": 1265427, "postDate": "2021-04-06T21:44:38.773Z", "content": "

Thanks for the observation. This suggests that we should use whatever GPU resources we have as efficiently as we can. If I understand correctly, it would seem that changing the mix of the training data in later epochs, to include more longer sequences (either by sampling or augmentation) would help to speed up the process of correcting the long sequences.

", "rawMarkdown": "Thanks for the observation. This suggests that we should use whatever GPU resources we have as efficiently as we can. If I understand correctly, it would seem that changing the mix of the training data in later epochs, to include more longer sequences (either by sampling or augmentation) would help to speed up the process of correcting the long sequences.", "replies": [ { "id": 1266750, "postDate": "2021-04-08T03:52:08.410Z", "content": "

note that there is smart batching in training transformer:

\n

https://towardsdatascience.com/divide-hugging-face-transformers-training-time-by-2-or-more-21bf7129db9q-21bf7129db9e

\n

https://wandb.ai/pommedeterresautee/speed_training/reports/Train-HuggingFace-Models-Twice-As-Fast--VmlldzoxMDgzOTI

\n

\"smart batching (named uniform length batching in the article, because experiments have been logged with this stupid name, I will keep it for this report).\"

\n
\n

you can check the following papers, etc:

\n

Training Tips for the Transformer Model
\nhttps://ufal.mff.cuni.cz/pbml/110/art-popel-bojar.pdf

\n

Tips on max_length
\n\"So in our case, if the batch size is high enough, the max_length has almost no
\neffect on BLEU, but this should be checked for each new dataset.\"

\n
\n

https://www.aclweb.org/anthology/W17-5708.pdf
\nA Bag of Useful Tricks for Practical Neural Machine Translation:
\nEmbedding Layer Initialization and Large Batch Size

\n

https://github.com/neubig/nmt-tips

", "rawMarkdown": "note that there is smart batching in training transformer:\n\nhttps://towardsdatascience.com/divide-hugging-face-transformers-training-time-by-2-or-more-21bf7129db9q-21bf7129db9e\n\nhttps://wandb.ai/pommedeterresautee/speed_training/reports/Train-HuggingFace-Models-Twice-As-Fast--VmlldzoxMDgzOTI\n\n\"smart batching (named uniform length batching in the article, because experiments have been logged with this stupid name, I will keep it for this report).\"\n\n----\n\nyou can check the following papers, etc:\n\nTraining Tips for the Transformer Model\nhttps://ufal.mff.cuni.cz/pbml/110/art-popel-bojar.pdf\n\nTips on max_length\n\"So in our case, if the batch size is high enough, the max_length has almost no\neffect on BLEU, but this should be checked for each new dataset.\"\n\n---\n\nhttps://www.aclweb.org/anthology/W17-5708.pdf\nA Bag of Useful Tricks for Practical Neural Machine Translation:\nEmbedding Layer Initialization and Large Batch Size\n\nhttps://github.com/neubig/nmt-tips", "votes": 6 }, { "id": 1271708, "postDate": "2021-04-12T19:59:42.010Z", "content": "

@hengck23, thanks for your response and super helpful links… yes I can see a path to faster results!

", "rawMarkdown": "@hengck23, thanks for your response and super helpful links... yes I can see a path to faster results!" } ] } ], "comments": [ { "id": 1265554, "author_name": "human intelligence", "author_url": "", "post_date": "2021-04-07T02:36:31.630000", "content": "

Just manually assigned structures for 200 random images from training set, using computational chemist's intuition. 7% are wrong (7/100 and 7/100)

\n\n

Let's say 1 false prediction increases edit distance by 30 on average . According to this estimation, the winning models with LD < 2 are probably overfitting to THE particular molecular sketcher & noise generator, from a pure scientific perspective.

\n

We are watching a race of overfitting

", "votes": 18, "replies": [ { "id": 1271090, "author_name": "", "author_url": "", "post_date": "2021-04-12T09:45:07.413000", "content": "", "votes": 0, "replies": [] }, { "id": 1276721, "author_name": "Charles", "author_url": "", "post_date": "2021-04-17T22:06:02.283000", "content": "

This is valuable insight. Hopefully the competition sponsors are cognisant of this.

", "votes": 0, "replies": [] } ] }, { "id": 1264398, "author_name": "Johnny Lee", "author_url": "", "post_date": "2021-04-06T06:04:23.313000", "content": "

Kaggle should provide some GCP quota for this competition. @addisonhoward

", "votes": 14, "replies": [ { "id": 1271793, "author_name": "Santosh kumar", "author_url": "", "post_date": "2021-04-12T23:13:05.250000", "content": "

Yeah there must be some.

", "votes": 0, "replies": [] } ] }, { "id": 1276723, "author_name": "Charles", "author_url": "", "post_date": "2021-04-17T22:08:37.163000", "content": "

I am just glad that the weather here is getting colder as autumn sets in - so I am appreciative of the extra warmth in my house. =)

", "votes": 8, "replies": [] }, { "id": 1268679, "author_name": "CPMP", "author_url": "", "post_date": "2021-04-09T16:33:43.563000", "content": "

Isn't it true for all image competitions? What is different from other image competitions here?

", "votes": 5, "replies": [ { "id": 1269167, "author_name": "Psi", "author_url": "", "post_date": "2021-04-10T08:34:40.290000", "content": "

This competition seems way more resource hungry than other competitions. The data is huge and training longer seems to help quite a bit from what I can gather from the discussions. There are many cv competitions where you dont need much HW to score high, recent examples: VinBigData, Rainforest, NFL

", "votes": 7, "replies": [] }, { "id": 1269177, "author_name": "hengck23", "author_url": "", "post_date": "2021-04-10T09:00:26.870000", "content": "

There is a long tail problem here. you really need the number of iterations!

", "votes": 7, "replies": [] }, { "id": 1269189, "author_name": "CPMP", "author_url": "", "post_date": "2021-04-10T09:18:58.327000", "content": "

@philippsinger Sure, TGS, Bengali were small data too. But there are also large data CV competitions, this one is no the first one.

\n

@hengck23 I believe you, my point is that it is not the first compute intensive comp.

", "votes": 2, "replies": [] }, { "id": 1269514, "author_name": "Psi", "author_url": "", "post_date": "2021-04-10T15:49:59.660000", "content": "

Right, there are other compute intensive ones.

", "votes": 1, "replies": [] } ] }, { "id": 1264763, "author_name": "Hiram Coria 🧬", "author_url": "", "post_date": "2021-04-06T12:14:04.003000", "content": "

This seems to be a tendency, it can be confirmed by the lack of \"end-to-end\" notebooks. Many competitions lack of notebooks with a training part, because if they train with Kaggle resources they can't achive a remarkable score.

", "votes": 5, "replies": [] }, { "id": 1268773, "author_name": "CPMP", "author_url": "", "post_date": "2021-04-09T18:26:34.763000", "content": "

We don't have access to 1024 A100 at NVIDIA, contrarily to what some team name could suggest…

\n

We don't even have access to A100 at this point, except for very limited test. We work with V100 GPU, which is already quite good.

\n

Just to straighten expectations.

", "votes": 3, "replies": [] }, { "id": 1279462, "author_name": "Serigne ", "author_url": "", "post_date": "2021-04-20T23:45:27.177000", "content": "

I will likely switch to TF and use the full power of TPU + TFRecords.

\n

I love Pytorch but training is really slow on single V100 - 16Gb. My current val score 2.20 (not submitted yet) took
\n more than 4 days of training. Let alone inference which is even slower (I don't use fairseq, but plain Pytorch transformer.) It doesn't let me room to experiment quickly new ideas.

\n

I converted all my transformer code to run on Torch/XLA but we still have big i/o bottleneck.

", "votes": 1, "replies": [ { "id": 1279477, "author_name": "hengck23", "author_url": "", "post_date": "2021-04-21T00:30:15.767000", "content": "

you need yoshi

\n

\"\"

", "votes": 7, "replies": [] } ] }, { "id": 1271288, "author_name": "Athar Sayed", "author_url": "", "post_date": "2021-04-12T13:00:45.650000", "content": "

@hengck23 , I observed some thing.
\nI trained a model on 1 epoch and the results were as follows.
\nNote : I am using same code as Used by Y Nakama and using some 3 lakhs images for validation .

\n

Train Loss : 0.53
\nVal Levhestein Distance : 32.X

\n

Later I trained for 1 more epoch based on above checkpoint , Results were as follows
\nTrain Loss = 0.29
\nVal Levhestein Distance = 95.X

\n

Is this normal has anyone observed such behaviour .

", "votes": 1, "replies": [ { "id": 1271416, "author_name": "Gabriel Lindenmaier", "author_url": "", "post_date": "2021-04-12T15:34:48.943000", "content": "

I found that using the wrong mean & std for normalization leads to unstable validation performance.
\nHere are the correct ones for the data. Maybe that was your problem. But in this case it should also lead to different performances in the 1st epoch over several runs with the same parameters.

\n

Other possible cause: Model is heavily overfitting to training set, but seems unlikely. Your parameter count would have to be really high.

", "votes": 0, "replies": [] }, { "id": 1279215, "author_name": "Mohammed Rizin V K", "author_url": "", "post_date": "2021-04-20T17:51:00.703000", "content": "

@sayedathar11 can I ask?

\n", "votes": 1, "replies": [] }, { "id": 1279426, "author_name": "Athar Sayed", "author_url": "", "post_date": "2021-04-20T22:33:59.117000", "content": "

@morizin can you specify what do you meant by fixing orientation ? I didn't get you? Can you please elaborate ?
\nIf by orientation you meant augmentation then I want to add that I was adding train augmentation as follows.

\n

A.Compose([
\n A.Resize(CFG.size, CFG.size),
\n A.OneOf([A.HorizontalFlip( p = 0.5) , # Horizontal Flip is basically rotation by an Angle , Hence its okay to use it
\n A.VerticalFlip(p=0.5)] , p = 0.3 ),
\n A.RandomRotate90(p = 0.4),
\n Normalize(
\n mean=[0.485, 0.456, 0.406],
\n std=[0.229, 0.224, 0.225],
\n ),
\n ToTensorV2(),
\n ])

\n

And on validation set I was applying augmentation as follows .
\nCompose([
\n A.Resize(CFG.size, CFG.size),
\n A.Normalize(
\n mean=[0.485, 0.456, 0.406],
\n std=[0.229, 0.224, 0.225],
\n ),
\n ToTensorV2(),
\n ])

\n

Is there something problematic here ?

", "votes": 0, "replies": [] }, { "id": 1279439, "author_name": "Gabriel Lindenmaier", "author_url": "", "post_date": "2021-04-20T23:04:45.547000", "content": "

Is there something problematic here ?
\nYes, you are using mean & std from image net for normalization. As long as you don't use a model pretrained on image net your normalization is useless.
\nThat is most likely the cause for your instability, as already mentioned by me above.

", "votes": -1, "replies": [] }, { "id": 1279445, "author_name": "Athar Sayed", "author_url": "", "post_date": "2021-04-20T23:11:08.607000", "content": "

I am ensuring that my model uses pretrained weights of imagenet and also others are doing same I guess there is most probably some other error can you please let @morizin answer this .

", "votes": -1, "replies": [] }, { "id": 1279449, "author_name": "Gabriel Lindenmaier", "author_url": "", "post_date": "2021-04-20T23:18:17.900000", "content": "

Try the correct mean & std anyway to check, because your normalized images should have mean of 0 and std 1 anyway. I argue it is better to use mean & std of actual data if they are to far of from the original training data.
\nAnd @morizin can still answer, no problem.
\nI mean it is your problem, I just want to help. And I had the exact same problems and was using the same normalization as you.

", "votes": -1, "replies": [] }, { "id": 1281217, "author_name": "Mohammed Rizin V K", "author_url": "", "post_date": "2021-04-22T17:57:26.753000", "content": "

@sayedathar11 sorry for late reply
\nactually the some images of test is rotated which our training model finds weird
\ni am not sure that your problem is same as this. but some experienced this weird behavior in LB but not in CV.
\nhttps://www.kaggle.com/c/bms-molecular-translation/discussion/224257#1237479
\nI just started to explore the competition didnt try much till now.

\n

Is this normal has anyone observed such behaviour .
\ncan you check what happens in epoch 3 please. try only rotate as augmentation please for now

\n

because i think horizontal and vertical flip in train aug causes some problem

\n

\"\"

", "votes": 1, "replies": [] }, { "id": 1282024, "author_name": "Athar Sayed", "author_url": "", "post_date": "2021-04-23T14:15:55.810000", "content": "

@morizin I guess the problem was arising due to using some augmentations which causes instability in training ,I have removed those augmentations and Trying Efficientnet B1 as encoder , still will inform if I see any changes in 3rd epoch.

", "votes": 0, "replies": [] } ] }, { "id": 1268334, "author_name": "hengck23", "author_url": "", "post_date": "2021-04-09T10:07:35.787000", "content": "

i find an interesting results:

\n

\"\"
\nhttps://arxiv.org/pdf/2103.17239.pdf

\n

data-mining : let GPU run and mine information from data

", "votes": 1, "replies": [ { "id": 1268376, "author_name": "hengck23", "author_url": "", "post_date": "2021-04-09T10:59:48.540000", "content": "

\"\"

\n

how to improve your image transformer (depth and distillation)

", "votes": 1, "replies": [] } ] }, { "id": 1281126, "author_name": "Darien Schettler", "author_url": "", "post_date": "2021-04-22T16:30:55.207000", "content": "

Just wanted to comment and post a photo showing results from my model on a holdout validation dataset. The model is an EfficientNetB7-->Attn/LSTM run for around 6 epochs. Scores a public LB of 5.77. Note that although I say \"INCHI LENGTH\" below, I am referring to the number of tokens that make up the ground truth InChI string.

\n

\"validation\"

\n

I've noticed that if I train for more epochs the LSD continues to get better for the longer sequences. I don't have a photo showing this but as you pointed out above… this is obviously what's supposed to happen.

\n

I did wonder how much of an LSD drop we should expect for longer sequences when compared to shorter sequences (assuming an infinite amount of training)? Essentially, should I assume that this model is capable of scoring ~1.5 if trained for a LOoooong time? Or will there be some remaining negative effects as longer sequences may just be more difficult to predict?

\n

PS:
\nAs a random aside, my model currently only predicts ~1.5% valid inchi strings when submitting.

", "votes": 2, "replies": [ { "id": 1281318, "author_name": "human intelligence", "author_url": "", "post_date": "2021-04-22T19:30:27.727000", "content": "

I think LD~3 is totally achievable. Not sure about 1.5. But thanks for sharing valid InChI percent.

\n

Here is my baseline model on holdout set (N = 303k):
\n100% valid InChI. 78% perfect LD = 0 predictions. Mean LD = 7.2

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
mean LDperfect %NInChI length
6.33333367%320
3.86046578%107540
2.57459187%1771060
2.41655890%5958680
3.36763587%71533100
5.56301180%58537120
8.59069073%47304140
13.62956662%26145160
20.81064851%12585180
30.79225339%5757200
43.46851929%2160220
65.28977319%704240
106.1712069%257260
149.3897064%136280
182.1048390%124300
194.8641980%81320
207.9852941.5%68340
234.8260870%23360
247.5000000%6380
", "votes": 2, "replies": [] } ] }, { "id": 1274357, "author_name": "Yijie Xu", "author_url": "", "post_date": "2021-04-15T08:19:26.297000", "content": "

Is it worth it to keep training or use the GPU power to mine crypto then ? 😄

", "votes": 2, "replies": [] }, { "id": 1298495, "author_name": "HinePo", "author_url": "", "post_date": "2021-05-08T23:44:27.817000", "content": "

Many thanks for this super valuable insight.

", "votes": 0, "replies": [] }, { "id": 1269444, "author_name": "ZavodRobotov", "author_url": "", "post_date": "2021-04-10T14:38:05.810000", "content": "

Is there a way to sample more Long sequences to make training more efficient?

", "votes": 0, "replies": [ { "id": 1270013, "author_name": "hengck23", "author_url": "", "post_date": "2021-04-11T07:06:34.627000", "content": "

\"Is there a way to sample more Long sequences to make training more efficient?\"

\n

we should find out the reason why long sequence get worst results first.
\ne.g. because the are bigger in image and resizing them to a fixed size causes loss of information?
\ne.g. we need to model larger range dependency because the atoms are far apart?
\ne.g. is there a correlation of sequence length and image size?

\n

when the reason become known, then we are decide how to sample, etc

", "votes": 5, "replies": [] } ] }, { "id": 1265427, "author_name": "Andy Penrose", "author_url": "", "post_date": "2021-04-06T21:44:38.773000", "content": "

Thanks for the observation. This suggests that we should use whatever GPU resources we have as efficiently as we can. If I understand correctly, it would seem that changing the mix of the training data in later epochs, to include more longer sequences (either by sampling or augmentation) would help to speed up the process of correcting the long sequences.

", "votes": 0, "replies": [ { "id": 1266750, "author_name": "hengck23", "author_url": "", "post_date": "2021-04-08T03:52:08.410000", "content": "

note that there is smart batching in training transformer:

\n

https://towardsdatascience.com/divide-hugging-face-transformers-training-time-by-2-or-more-21bf7129db9q-21bf7129db9e

\n

https://wandb.ai/pommedeterresautee/speed_training/reports/Train-HuggingFace-Models-Twice-As-Fast--VmlldzoxMDgzOTI

\n

\"smart batching (named uniform length batching in the article, because experiments have been logged with this stupid name, I will keep it for this report).\"

\n
\n

you can check the following papers, etc:

\n

Training Tips for the Transformer Model
\nhttps://ufal.mff.cuni.cz/pbml/110/art-popel-bojar.pdf

\n

Tips on max_length
\n\"So in our case, if the batch size is high enough, the max_length has almost no
\neffect on BLEU, but this should be checked for each new dataset.\"

\n
\n

https://www.aclweb.org/anthology/W17-5708.pdf
\nA Bag of Useful Tricks for Practical Neural Machine Translation:
\nEmbedding Layer Initialization and Large Batch Size

\n

https://github.com/neubig/nmt-tips

", "votes": 6, "replies": [] }, { "id": 1271708, "author_name": "Andy Penrose", "author_url": "", "post_date": "2021-04-12T19:59:42.010000", "content": "

@hengck23, thanks for your response and super helpful links… yes I can see a path to faster results!

", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "1264252": "the loss dynamics of LSTM (and other seq model) is like this\n- loss decrease for the head part of the seq first in early training\n- short sequences get corrected first\n- as the training goes, sometimes you see the loss is stagnant, But actually, the tail part of the long sequences are being corrected. Long sequence are rare, that is why the improvement of loss are minor\n- but as long as you keep your GPU running, lb score will increase\n\neventually, everyone will reach lb score of about 1 to 2 ...if they have enough GPU resource and their seq model has \"enough capacity\"\n\nevery one is racing now. final top lb score could be in the range of 0.5 to 0.8", "1265554": "Just manually assigned structures for 200 random images from training set, using computational chemist's intuition. 7% are wrong (7/100 and 7/100)\n\n* 10 are due to *ghost* terminal methyl group (e.g., `4a4f6c36a791`)\n* 1 is due to fuzzy stereo depiction (e.g., `4080f41a9577`)\n* 2 are due to missing bonds in the complex multi-ring structure containing crossing bonds (e.g., `94a92ba4a8ae`)\n* 1 is due to extremely fuzzy fluorine depiction (e.g., `e159f8ad60e9`)\n\nLet's say 1 false prediction increases edit distance by 30 on average . According to this estimation, the winning models with LD < 2 are probably overfitting to THE particular molecular sketcher & noise generator, from a pure scientific perspective. \n\n**We are watching a race of overfitting**", "1264398": "Kaggle should provide some GCP quota for this competition. @addisonhoward ", "1276723": "I am just glad that the weather here is getting colder as autumn sets in - so I am appreciative of the extra warmth in my house. =)", "1268679": "Isn't it true for all image competitions? What is different from other image competitions here?", "1264763": "This seems to be a tendency, it can be confirmed by the lack of \"end-to-end\" notebooks. Many competitions lack of notebooks with a training part, because if they train with Kaggle resources they can't achive a remarkable score. ", "1268773": "We don't have access to 1024 A100 at NVIDIA, contrarily to what some team name could suggest...\n\nWe don't even have access to A100 at this point, except for very limited test. We work with V100 GPU, which is already quite good.\n\nJust to straighten expectations.", "1279462": "I will likely switch to TF and use the full power of TPU + TFRecords. \n\nI love Pytorch but training is really slow on single V100 - 16Gb. My current val score 2.20 (not submitted yet) took \n more than 4 days of training. Let alone inference which is even slower (I don't use fairseq, but plain Pytorch transformer.) It doesn't let me room to experiment quickly new ideas. \n\nI converted all my transformer code to run on Torch/XLA but we still have big i/o bottleneck. ", "1271288": "@hengck23 , I observed some thing.\nI trained a model on 1 epoch and the results were as follows.\nNote : I am using same code as Used by Y Nakama and using some 3 lakhs images for validation .\n\nTrain Loss : 0.53\nVal Levhestein Distance : 32.X\n\nLater I trained for 1 more epoch based on above checkpoint , Results were as follows\nTrain Loss = 0.29\nVal Levhestein Distance = 95.X\n\nIs this normal has anyone observed such behaviour .", "1268334": "i find an interesting results:\n\n![](https://i.ibb.co/wJ9n3LV/Selection-054.png)\nhttps://arxiv.org/pdf/2103.17239.pdf\n\ndata-mining : let GPU run and mine information from data", "1281126": "Just wanted to comment and post a photo showing results from my model on a holdout validation dataset. The model is an EfficientNetB7-->Attn/LSTM run for around 6 epochs. Scores a public LB of 5.77. Note that although I say \"INCHI LENGTH\" below, I am referring to the number of tokens that make up the ground truth InChI string.\n\n![validation](https://i.ibb.co/H7Ngq5J/Screen-Shot-2021-04-22-at-12-23-29-PM.png)\n\nI've noticed that if I train for more epochs the LSD continues to get better for the longer sequences. I don't have a photo showing this but as you pointed out above... this is obviously what's supposed to happen.\n\nI did wonder how much of an LSD drop we should expect for longer sequences when compared to shorter sequences (assuming an infinite amount of training)? Essentially, should I assume that this model is capable of scoring ~1.5 if trained for a LOoooong time? Or will there be some remaining negative effects as longer sequences may just be more difficult to predict?\n\n\nPS:\nAs a random aside, my model currently only predicts ~1.5% valid inchi strings when submitting.\n", "1274357": "Is it worth it to keep training or use the GPU power to mine crypto then ? 😄", "1298495": "Many thanks for this super valuable insight.", "1269444": "Is there a way to sample more Long sequences to make training more efficient?", "1265427": "Thanks for the observation. This suggests that we should use whatever GPU resources we have as efficiently as we can. If I understand correctly, it would seem that changing the mix of the training data in later epochs, to include more longer sequences (either by sampling or augmentation) would help to speed up the process of correcting the long sequences." } }