Hi there,
\nI have created a notebook that implements the EfficientNetV2 model architecture (in Tensorflow) to function as the image encoder.
\nNot only that. I implement it in such a way that BOTH THE TRAINING AND THE MODEL INFERENCE ON THE TEST SET can happen in APPROXIMATELY 3 HOURS (OR LESS!)
\nTL;DR ––––––– TF+TPU+EfficientNetV2 –––> 🔥🔥🔥🔥🔥🔥🔥🔥
\nA caveat to the work I've completed. As this notebook implements both of these things, I don't have a comparative version with EfficientNetB2 to benchmark against. If I have time in the future I will do this just so we can all see the comparative performance differences.
\nA second caveat is that I did the notebook development and initial submission within an interactive session. Therefore, there MAY be problems with a save-run-all scenario… I hope not. If I have TPU near the end of the week I will re-run this notebook using the run-all save option. If someone else does this and it succeeds with no alterations please let me know. Thanks 😊
\nBr is a token, 191 is a token, etc.)Timing
\nValidation Levenshtein Distance (up to half max length) By Epoch
\nFinal Validation Levenshtein Distance
\nPublic Leaderboard Score
\nSIDE NOTE: If you fine-tune on the training dataset for another 3-5 epochs with the max length being the correct value, I would estimate an improvement of 0.75-1.5 LSD on the LB… just a guestimate from previous experience though…
\nNOTE:
\nThe discrepancy between public LB LSD and validation LSD is to be expected for 2 reasons.
\nI hope this helps!
\np.s. I will be updating the notebook tomorrow or early this week to include an option to train the EfficientNetV2 model from the provided pre-trained ImageNet weights.
\np.p.s. I will also tack on the RDKit normalization procedure to the end of this notebook as per @nofreewill's notebook. Not sure if it will help… but why not.
", "messages": [ { "id": "1310687", "postDate": "05/16/2021 21:00:28", "content": "Hi there,
\nI have created a notebook that implements the EfficientNetV2 model architecture (in Tensorflow) to function as the image encoder.
\nNot only that. I implement it in such a way that BOTH THE TRAINING AND THE MODEL INFERENCE ON THE TEST SET can happen in APPROXIMATELY 3 HOURS (OR LESS!)
\nTL;DR ––––––– TF+TPU+EfficientNetV2 –––> 🔥🔥🔥🔥🔥🔥🔥🔥
\nA caveat to the work I've completed. As this notebook implements both of these things, I don't have a comparative version with EfficientNetB2 to benchmark against. If I have time in the future I will do this just so we can all see the comparative performance differences.
\nA second caveat is that I did the notebook development and initial submission within an interactive session. Therefore, there MAY be problems with a save-run-all scenario… I hope not. If I have TPU near the end of the week I will re-run this notebook using the run-all save option. If someone else does this and it succeeds with no alterations please let me know. Thanks 😊
\nBr is a token, 191 is a token, etc.)Timing
\nValidation Levenshtein Distance (up to half max length) By Epoch
\nFinal Validation Levenshtein Distance
\nPublic Leaderboard Score
\nSIDE NOTE: If you fine-tune on the training dataset for another 3-5 epochs with the max length being the correct value, I would estimate an improvement of 0.75-1.5 LSD on the LB… just a guestimate from previous experience though…
\nNOTE:
\nThe discrepancy between public LB LSD and validation LSD is to be expected for 2 reasons.
\nI hope this helps!
\np.s. I will be updating the notebook tomorrow or early this week to include an option to train the EfficientNetV2 model from the provided pre-trained ImageNet weights.
\np.p.s. I will also tack on the RDKit normalization procedure to the end of this notebook as per @nofreewill's notebook. Not sure if it will help… but why not.
", "rawMarkdown": "Hi there, \n\nI have created a notebook that implements the EfficientNetV2 model architecture (in Tensorflow) to function as the image encoder.\n\nNot only that. I implement it in such a way that **BOTH THE TRAINING AND THE MODEL INFERENCE ON THE TEST SET** can happen in **APPROXIMATELY 3 HOURS (OR LESS!)** \n\n---\n\nI am currently trying a similar strategy of tuning on shorter sequences and then fine-tuning on full length. I see that your performance improved after choosing longer sequences but mine converges to a worse point. Is ther anything you tried to allow the model to converge better on the full sequences? Thanks!
", "rawMarkdown": "I am currently trying a similar strategy of tuning on shorter sequences and then fine-tuning on full length. I see that your performance improved after choosing longer sequences but mine converges to a worse point. Is ther anything you tried to allow the model to converge better on the full sequences? Thanks!", "votes": null }, { "id": "1316412", "postDate": "05/20/2021 14:21:51", "content": "You have to be very careful with the learning rate schedule. i.e. long ramp-ups and lower peak.
\nOther than that I have not done anything fancy.
One note:
\nI do not CHOOSE longer or shorter sequences. I simply truncate all of the sequences to a smaller fixed max length for the initial rounds of training and then remove that truncation for fine-tuning. I have not done it with the model above… but using an EfficientNetB5 we achieved an improvement in the order of 0.75-1.25 LSD by fine-tuning on the non-truncated sequences.
", "rawMarkdown": "You have to be very careful with the learning rate schedule. i.e. long ramp-ups and lower peak.\nOther than that I have not done anything fancy. \n\nOne note:\n\nI do not CHOOSE longer or shorter sequences. I simply truncate all of the sequences to a smaller fixed max length for the initial rounds of training and then remove that truncation for fine-tuning. I have not done it with the model above... but using an EfficientNetB5 we achieved an improvement in the order of 0.75-1.25 LSD by fine-tuning on the non-truncated sequences.", "votes": null }, { "id": "1316429", "postDate": "05/20/2021 14:40:34", "content": "I see! I did filter out the longer sequences. Perhaps my model biased and then lost many features after adding back in longer sequences. Thanks for all the help.
", "rawMarkdown": "I see! I did filter out the longer sequences. Perhaps my model biased and then lost many features after adding back in longer sequences. Thanks for all the help.", "votes": null }, { "id": "1316468", "postDate": "05/20/2021 15:27:16", "content": "Great notebook. I also tried training from scratch but I found out training from pretrained weight helped a lot. You are right that rotation if h > w does not perform quite well, one example in the test data set is d83bc1c3b5ec. This image has h>w , but in fact it should not be rotated based on the element orientation.
", "rawMarkdown": "Great notebook. I also tried training from scratch but I found out training from pretrained weight helped a lot. You are right that rotation if h > w does not perform quite well, one example in the test data set is d83bc1c3b5ec. This image has h>w , but in fact it should not be rotated based on the element orientation.", "votes": null }, { "id": "1316487", "postDate": "05/20/2021 15:33:59", "content": "Really! Can you share a benchmark with starting from pretrained?
", "rawMarkdown": "Really! Can you share a benchmark with starting from pretrained?", "votes": null }, { "id": "1317123", "postDate": "05/21/2021 07:03:20", "content": "Great job! A question: you have a PREFIX_ORDERING which is not used anywhere - what is the purpose of it?
", "rawMarkdown": "Great job! A question: you have a PREFIX_ORDERING which is not used anywhere - what is the purpose of it?", "votes": null }, { "id": "1317539", "postDate": "05/21/2021 13:31:43", "content": "It’s a vestigial thing I created when I was doing my EDA. It just shows the order that the InChI sub layers go in.
\nI don’t use it anywhere. Safe to delete!
", "rawMarkdown": "It’s a vestigial thing I created when I was doing my EDA. It just shows the order that the InChI sub layers go in.\n\nI don’t use it anywhere. Safe to delete!", "votes": null }, { "id": "1318960", "postDate": "05/22/2021 17:54:58", "content": "Hey! Great effort, just a question, which notebook did you use for generatin the tfrecords??
\nThanks
See here -> https://www.kaggle.com/dschettler8845/bms-tfrecord-creation-192x384-train-half
", "rawMarkdown": "See here -> https://www.kaggle.com/dschettler8845/bms-tfrecord-creation-192x384-train-half", "votes": null }, { "id": "1322993", "postDate": "05/25/2021 21:19:15", "content": "404 notebook is private or does not exist, but appreciate the answer, your other notebooks are following the same process to generate the tfrecords, thanks!
", "rawMarkdown": "404 notebook is private or does not exist, but appreciate the answer, your other notebooks are following the same process to generate the tfrecords, thanks!", "votes": null }, { "id": "1323060", "postDate": "05/25/2021 23:46:38", "content": "Updated it to be public! My bad!
", "rawMarkdown": "Updated it to be public! My bad!", "votes": null } ], "comments": [ { "id": 1316322, "author_name": "andrewshao05", "author_url": "", "post_date": "05/20/2021 13:02:33", "content": "I am currently trying a similar strategy of tuning on shorter sequences and then fine-tuning on full length. I see that your performance improved after choosing longer sequences but mine converges to a worse point. Is ther anything you tried to allow the model to converge better on the full sequences? Thanks!
", "votes": null, "replies": [ { "id": 1316412, "author_name": "dschettler8845", "author_url": "", "post_date": "05/20/2021 14:21:51", "content": "You have to be very careful with the learning rate schedule. i.e. long ramp-ups and lower peak.
\nOther than that I have not done anything fancy.
One note:
\nI do not CHOOSE longer or shorter sequences. I simply truncate all of the sequences to a smaller fixed max length for the initial rounds of training and then remove that truncation for fine-tuning. I have not done it with the model above… but using an EfficientNetB5 we achieved an improvement in the order of 0.75-1.25 LSD by fine-tuning on the non-truncated sequences.
", "votes": null, "replies": [] }, { "id": 1316429, "author_name": "andrewshao05", "author_url": "", "post_date": "05/20/2021 14:40:34", "content": "I see! I did filter out the longer sequences. Perhaps my model biased and then lost many features after adding back in longer sequences. Thanks for all the help.
", "votes": null, "replies": [] } ] }, { "id": 1316468, "author_name": "joblessphysicist", "author_url": "", "post_date": "05/20/2021 15:27:16", "content": "Great notebook. I also tried training from scratch but I found out training from pretrained weight helped a lot. You are right that rotation if h > w does not perform quite well, one example in the test data set is d83bc1c3b5ec. This image has h>w , but in fact it should not be rotated based on the element orientation.
", "votes": null, "replies": [ { "id": 1316487, "author_name": "dschettler8845", "author_url": "", "post_date": "05/20/2021 15:33:59", "content": "Really! Can you share a benchmark with starting from pretrained?
", "votes": null, "replies": [] } ] }, { "id": 1317123, "author_name": "jhasanov", "author_url": "", "post_date": "05/21/2021 07:03:20", "content": "Great job! A question: you have a PREFIX_ORDERING which is not used anywhere - what is the purpose of it?
", "votes": null, "replies": [ { "id": 1317539, "author_name": "dschettler8845", "author_url": "", "post_date": "05/21/2021 13:31:43", "content": "It’s a vestigial thing I created when I was doing my EDA. It just shows the order that the InChI sub layers go in.
\nI don’t use it anywhere. Safe to delete!
", "votes": null, "replies": [] } ] }, { "id": 1318960, "author_name": "enric1296", "author_url": "", "post_date": "05/22/2021 17:54:58", "content": "Hey! Great effort, just a question, which notebook did you use for generatin the tfrecords??
\nThanks
See here -> https://www.kaggle.com/dschettler8845/bms-tfrecord-creation-192x384-train-half
", "votes": null, "replies": [] }, { "id": 1322993, "author_name": "jesucristo", "author_url": "", "post_date": "05/25/2021 21:19:15", "content": "404 notebook is private or does not exist, but appreciate the answer, your other notebooks are following the same process to generate the tfrecords, thanks!
", "votes": null, "replies": [] }, { "id": 1323060, "author_name": "dschettler8845", "author_url": "", "post_date": "05/25/2021 23:46:38", "content": "Updated it to be public! My bad!
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1310687": "Hi there, \n\nI have created a notebook that implements the EfficientNetV2 model architecture (in Tensorflow) to function as the image encoder.\n\nNot only that. I implement it in such a way that **BOTH THE TRAINING AND THE MODEL INFERENCE ON THE TEST SET** can happen in **APPROXIMATELY 3 HOURS (OR LESS!)** \n\n---\n\n