![\"\"](\"https://i.ibb.co/KmD952w/Selection-041.png\")
\n
![\"\"](\"https://i.ibb.co/qdg981H/Selection-038.png\")
i was trying to analyze the relationship for prediction verus sequence length and image size.
\ni discover something fishy ….
\n
\n
i was trying to analyze the relationship for prediction verus sequence length and image size.
\ni discover something fishy ….
\n
\n
This is very interesting…I am thinking we can use this in a couple ways:
\nFirst idea
\n Use original image size as a meta feature - might help model determine the desired sequence length from the image size alone.
Second idea
\nTrain different models on different sequence length ranges and then use test image sizes to determine how to infer with these models. For example:
Third idea
\nSame as second idea, but infer on all images with all three \"specialized\" models but use sequence length and test image sizes to determine best blend.
You can apply the second and third ideas without this sequence length / image size correlation, but perhaps we will get better results using both the sequence length of a previous submission and information about the image size together.
\nWe could also train these groups with different resolutions: smaller sequences use lower resolution so you can save time, but the larger sequence models can be trained at fairly high resolution.
", "rawMarkdown": "This is very interesting...I am thinking we can use this in a couple ways:\n\n**First idea**\n Use original image size as a meta feature - might help model determine the desired sequence length from the image size alone.\n\n**Second idea**\nTrain different models on different sequence length ranges and then use test image sizes to determine how to infer with these models. For example:\n* Train 3 models. 1st with sequence length min length - 100, 2nd with 101 - 150, 3rd with 151 - max length\n* Estimate test image sequence length based on previous submission coupled with information about their image sizes (as you've shown above)\n* Infer with 1st model on test images with sequence length min length - 100, infer with 2nd on sequence lengths 101 - 150, etc.\n\n**Third idea**\nSame as second idea, but infer on all images with all three \"specialized\" models but use sequence length and test image sizes to determine best blend. \n\n---------------------------------------------------------------------------------------------------------------------------\nYou can apply the second and third ideas without this sequence length / image size correlation, but perhaps we will get better results using both the sequence length of a previous submission and information about the image size together.\n\nWe could also train these groups with different resolutions: smaller sequences use lower resolution so you can save time, but the larger sequence models can be trained at fairly high resolution.", "votes": null }, { "id": "1277628", "postDate": "04/19/2021 02:25:22", "content": "i was training and validating only big images.
\nthen i train another model using big + some small images. I want to see if small images can help to improve the prediction of big images. if there is no improvement, then one might as well separate big and small images
", "rawMarkdown": "i was training and validating only big images.\n\nthen i train another model using big + some small images. I want to see if small images can help to improve the prediction of big images. if there is no improvement, then one might as well separate big and small images", "votes": null }, { "id": "1277633", "postDate": "04/19/2021 02:35:26", "content": "add the coordinate layer as input may help.
\ncoordinate layer can be normalised to local image size and/or to common size.
\nbecause of the \"flaw\" the data is prepared, pixel location has information.
mean image of similar size images show this
\n![\"\"](\"https://1fykyq3mdn5r21tpna3wkdyi-wpengine.netdna-ssl.com/wp-content/uploads/2018/07/Featured-Image-1-768x329.png\")
I had not heard of a coordconv layer before…thanks for sharing. I found a PyTorch implementation of it and will run some tests
", "rawMarkdown": "I had not heard of a coordconv layer before...thanks for sharing. I found a PyTorch implementation of it and will run some tests", "votes": null }, { "id": "1277644", "postDate": "04/19/2021 03:10:28", "content": "![\"\"](\"https://i.ibb.co/tKTFvNP/Selection-044.png\")
a better idea is to get rid of the image copletely
\n
make a mistake. should be 2**(16x16)
\nmuch of the image are empty, so we can encode it sparsely. i think we can can run very fast
", "rawMarkdown": "\n\n\n\na better idea is to get rid of the image copletely\n~~even without auto encoder, because our image is binary, we only have 2**16 = 65536 discrete values. just by clustering, we can retrieve top 4k word~~\n\nmake a mistake. should be 2**(16x16)\n\nmuch of the image are empty, so we can encode it sparsely. i think we can can run very fast", "votes": null }, { "id": "1277824", "postDate": "04/19/2021 09:11:28", "content": "very neat! 🙂 could I please ask how you are detecting orientation?
\nEDIT: from taking a look at the data it doesn't seem that they are ever rotated counter-clockwise or upside down… in this case that would be straightforward by just looking at the dimensions…
", "rawMarkdown": "very neat! 🙂 could I please ask how you are detecting orientation?\n\nEDIT: from taking a look at the data it doesn't seem that they are ever rotated counter-clockwise or upside down... in this case that would be straightforward by just looking at the dimensions...", "votes": null }, { "id": "1277840", "postDate": "04/19/2021 09:35:52", "content": "h>w -> rotate; but there are some images that don't follow this rule
", "rawMarkdown": "h>w -> rotate; but there are some images that don't follow this rule", "votes": null }, { "id": "1277848", "postDate": "04/19/2021 09:42:57", "content": "I train an orientation detection and scale detector using resnet34.
", "rawMarkdown": "I train an orientation detection and scale detector using resnet34.", "votes": null }, { "id": "1278007", "postDate": "04/19/2021 13:30:22", "content": "You have to keep the positions of those image patches in mind. And as the encoded patches still would need some kind of embedding vectors I don't see a big difference to given setups. I mean you would do Autoencoder -> image patch encoding -> patch embedding sequence. Now we have: image model -> image features -> shape-transformed feature sequence
", "rawMarkdown": "You have to keep the positions of those image patches in mind. And as the encoded patches still would need some kind of embedding vectors I don't see a big difference to given setups. I mean you would do Autoencoder -> image patch encoding -> patch embedding sequence. Now we have: image model -> image features -> shape-transformed feature sequence", "votes": null }, { "id": "1278012", "postDate": "04/19/2021 13:33:50", "content": "there is a difference if you have big and small images.
\n(in this case, you have to set image size to the largest image size. or you can train in batches of similar sizes, from smallest to largest)
i am skipping all empty patches
", "rawMarkdown": "there is a difference if you have big and small images.\n(in this case, you have to set image size to the largest image size. or you can train in batches of similar sizes, from smallest to largest)\n\ni am skipping all empty patches", "votes": null }, { "id": "1278019", "postDate": "04/19/2021 13:39:54", "content": "coordinate layer
\nThanks! I knew that this existed and searched for it, but couldn't find it anymore.
\npaperswithcode - Here are some implementations and the paper
Edit: I have my doubts, that coordinates are worth the effort. The images have different scales. And while you can correct rotation, the sizes are still different. And the molecular structures of the same kind are all over the place. So I predict that you get only a very small performance gain.
", "rawMarkdown": "`coordinate layer`\nThanks! I knew that this existed and searched for it, but couldn't find it anymore.\n[paperswithcode - Here are some implementations and the paper](https://paperswithcode.com/paper/an-intriguing-failing-of-convolutional-neural)\n\nEdit: I have my doubts, that coordinates are worth the effort. The images have different scales. And while you can correct rotation, the sizes are still different. And the molecular structures of the same kind are all over the place. So I predict that you get only a very small performance gain.", "votes": null }, { "id": "1278044", "postDate": "04/19/2021 14:11:13", "content": "Watch out that being too greedy with \"smart batching\" can lead to unstable training. I've lost a hole week because of it (smartbatched by inchi length).
", "rawMarkdown": "Watch out that being too greedy with \"smart batching\" can lead to unstable training. I've lost a hole week because of it (smartbatched by inchi length).", "votes": null }, { "id": "1278114", "postDate": "04/19/2021 15:07:09", "content": "\n\n(smartbatched by inchi length).
\n
How so? I have been doing this for years for text data, and it works great. What you have to keep in mind though is: Don't sort after length over your whole training data, do it for windows where a window equals x batch sizes. Then you split the sorted windows again into batches and randomize them within the window. The latter, because so you don't get windows with batches of increasing lengths.
\nIf you do it over your whole training data, then yes, it won't generalize well. It might not even learn well in the first place and be unstable.
\n\nthere is a difference if you have big and small images.
\n
Resizing it to the largest size would still make no real difference to given approach. The scale is nevertheless different for the images. And empty patches should be ignored by image model anyway.
\nImportant question: Does it actually work better than an image model? In this case you would be right.
\n\nHow so?
\n
As you just described it. :)
\nAnother way of the randomization: sorted(lambda x: x[1], key=len(x)*(1+random.random()*0.1-0.05))
", "rawMarkdown": "> How so?\n\nAs you just described it. :)\n\nAnother way of the randomization: sorted(lambda x: x[1], key=len(x)\\*(1+random.random()\\*0.1-0.05))", "votes": null }, { "id": "1278132", "postDate": "04/19/2021 15:29:14", "content": "\"The scale is nevertheless different for the images.\"
\nno. there is only 2 scales in the test and train images.
\nthe bigger molecule image has a border of 58 pixel and the smaller one has a border of 36 pixel
\"Does it actually work better than an image model? \"
\nI don't think it will be more accurate. But i hope it would be much faster
As you just described it. :)
\nI meant, how does it come to be unstable. Unless I described that also with my last part.
there is only 2 scales in the test and train images.
\nI see more than 2 different sizes of molecules. You can see it through the size of the letters or pentagons very well. Maybe the amount they where scaled afterwards has only two different sizes, but that doesn't change the absolute size.
But i hope it would be much faster
\nI see. But I doubt that a lot. You still need the AutoEncoder + Sequence-Encoder VS. (CNN) Image Model.
\nEdit: I think you could go with a Transformer for images and just exclude the white patches and use a smart positional embedding which takes that into account. That should work as well as a classical approach and be faster (than at least other transformers).
\n\nUnless I described that also with my last part.
\n\n\nIf you do it over your whole training data, then yes, it won't generalize well. It might not even learn well in the first place and be unstable.
\n
I guess it comes to that in a batch it sees that all sequences are of length L and it probably drives the model a lot in that direction.
", "rawMarkdown": "> Unless I described that also with my last part.\n> > If you do it over your whole training data, then yes, it won't generalize well. It might not even learn well in the first place and be unstable.\n\nI guess it comes to that in a batch it sees that all sequences are of length L and it probably drives the model a lot in that direction.", "votes": null }, { "id": "1278261", "postDate": "04/19/2021 17:40:14", "content": "that in a batch it sees that all sequences are of length L...
\nYes, the gradient goes to much in a specific, specialized direction. And if you don't permute if afterwards, the length increases/decreases the whole epoch. This means the model forgets earlier patters for different lengths.
\nThat's why I use sorting within a window of x*batch_size. And then permute the order of those locally sorted batches again. That can accelerate the training speed usually by 2x (for the sequence model) and also improve the loss slightly. Although the optimal window sizes for both objectives might be different.
\nx is usually in [2, 30]. So it sorts elements of up to 30 batches. For this competition I use currently x = 4, but haven't tuned it yet.
Why can local length sorting improve the loss slightly? Well, first the sequences are more similar, but still random enough so it gives a slightly better training signal. Second, you have less padding per sequence on average which means less noise (You would have to pad and probably reset to zero otherwise). The second part is even valid for a auto-regressive model because the weights that process the hidden embeddings are shared over all time-steps. Of course you just can mask the padding always (with zeros) - which I don't do.
\nThat worked well also for such benchmarks as Wikitext-103 where you
", "rawMarkdown": "`that in a batch it sees that all sequences are of length L...`\nYes, the gradient goes to much in a specific, *specialized* direction. And if you don't permute if afterwards, the length increases/decreases the whole epoch. This means the model forgets earlier patters for different lengths. \nThat's why I use sorting within a window of x*batch_size. And then permute the order of those locally sorted batches again. That can accelerate the training speed usually by 2x (for the sequence model) and also improve the loss slightly. Although the optimal window sizes for both objectives might be different. \nx is usually in [2, 30]. So it sorts elements of up to 30 batches. For this competition I use currently x = 4, but haven't tuned it yet.\n\nWhy can local length sorting improve the loss slightly? Well, first the sequences are more similar, but still random enough so it gives a slightly better training signal. Second, you have less padding per sequence on average which means less noise (You would have to pad and probably reset to zero otherwise). The second part is even valid for a auto-regressive model because the weights that process the hidden embeddings are shared over all time-steps. Of course you just can mask the padding always (with zeros) - which I don't do.\n\nThat worked well also for such benchmarks as Wikitext-103 where you", "votes": null }, { "id": "1278269", "postDate": "04/19/2021 17:51:39", "content": "\n\naccelerate the training speed usually by 2x
\n
I doubt that. The speedup is around 2x just for the decoder part. If you use fixed sized images, then the time spent on the encoder is the same regardless of smart batching.
\nAlso, can you please compress your thoughts?
", "rawMarkdown": "> accelerate the training speed usually by 2x\n\nI doubt that. The speedup is around 2x just for the decoder part. If you use fixed sized images, then the time spent on the encoder is the same regardless of smart batching.\n\nAlso, can you please compress your thoughts?", "votes": null }, { "id": "1278276", "postDate": "04/19/2021 17:55:39", "content": "Your are right, in this case it isn't 2x. But I meant just the sequence part, forgot to add that.
\nI will add a TLDR in the future.
I hope you don't mind the critique!
", "rawMarkdown": "I hope you don't mind the critique!", "votes": null }, { "id": "1278298", "postDate": "04/19/2021 18:05:00", "content": "I hope you don't mind the critique!
\nNo. Yours is based on arguments (apart from the compress part), so it's welcome.
\nI also hope I could help with my length sorting description.
\n\n(apart from the compress part)
\n
I was referring to that under critique :D
\nSo I hope you don't mind that!
It's fine. Some people don't understand it with a long explanation, others don't want/need those. You get a TLDR in case of long posts.
", "rawMarkdown": "It's fine. Some people don't understand it with a long explanation, others don't want/need those. You get a TLDR in case of long posts.", "votes": null }, { "id": "1280476", "postDate": "04/22/2021 02:52:19", "content": "another way to do text-to-text:
\n(use ascii art)
O\n //\n Cl--{/\n \\\n \\_____\n / --- \\\n / \\\n \\\\ //\n \\_____/\n OH\n /\n C --C O\n // \\\\ //\nC C---C\n \\ __ / \\\n C --C OH\nThis is very interesting…I am thinking we can use this in a couple ways:
\nFirst idea
\n Use original image size as a meta feature - might help model determine the desired sequence length from the image size alone.
Second idea
\nTrain different models on different sequence length ranges and then use test image sizes to determine how to infer with these models. For example:
Third idea
\nSame as second idea, but infer on all images with all three \"specialized\" models but use sequence length and test image sizes to determine best blend.
You can apply the second and third ideas without this sequence length / image size correlation, but perhaps we will get better results using both the sequence length of a previous submission and information about the image size together.
\nWe could also train these groups with different resolutions: smaller sequences use lower resolution so you can save time, but the larger sequence models can be trained at fairly high resolution.
", "votes": null, "replies": [ { "id": 1277628, "author_name": "hengck23", "author_url": "", "post_date": "04/19/2021 02:25:22", "content": "i was training and validating only big images.
\nthen i train another model using big + some small images. I want to see if small images can help to improve the prediction of big images. if there is no improvement, then one might as well separate big and small images
", "votes": null, "replies": [] }, { "id": 1277633, "author_name": "hengck23", "author_url": "", "post_date": "04/19/2021 02:35:26", "content": "add the coordinate layer as input may help.
\ncoordinate layer can be normalised to local image size and/or to common size.
\nbecause of the \"flaw\" the data is prepared, pixel location has information.
mean image of similar size images show this
\n
I had not heard of a coordconv layer before…thanks for sharing. I found a PyTorch implementation of it and will run some tests
", "votes": null, "replies": [] }, { "id": 1278019, "author_name": "cepheidq", "author_url": "", "post_date": "04/19/2021 13:39:54", "content": "coordinate layer
\nThanks! I knew that this existed and searched for it, but couldn't find it anymore.
\npaperswithcode - Here are some implementations and the paper
Edit: I have my doubts, that coordinates are worth the effort. The images have different scales. And while you can correct rotation, the sizes are still different. And the molecular structures of the same kind are all over the place. So I predict that you get only a very small performance gain.
", "votes": null, "replies": [] } ] }, { "id": 1277644, "author_name": "hengck23", "author_url": "", "post_date": "04/19/2021 03:10:28", "content": "
a better idea is to get rid of the image copletely
\n
make a mistake. should be 2**(16x16)
\nmuch of the image are empty, so we can encode it sparsely. i think we can can run very fast
", "votes": null, "replies": [ { "id": 1278007, "author_name": "cepheidq", "author_url": "", "post_date": "04/19/2021 13:30:22", "content": "You have to keep the positions of those image patches in mind. And as the encoded patches still would need some kind of embedding vectors I don't see a big difference to given setups. I mean you would do Autoencoder -> image patch encoding -> patch embedding sequence. Now we have: image model -> image features -> shape-transformed feature sequence
", "votes": null, "replies": [] }, { "id": 1278012, "author_name": "hengck23", "author_url": "", "post_date": "04/19/2021 13:33:50", "content": "there is a difference if you have big and small images.
\n(in this case, you have to set image size to the largest image size. or you can train in batches of similar sizes, from smallest to largest)
i am skipping all empty patches
", "votes": null, "replies": [] }, { "id": 1278044, "author_name": "nofreewill", "author_url": "", "post_date": "04/19/2021 14:11:13", "content": "Watch out that being too greedy with \"smart batching\" can lead to unstable training. I've lost a hole week because of it (smartbatched by inchi length).
", "votes": null, "replies": [] }, { "id": 1278114, "author_name": "cepheidq", "author_url": "", "post_date": "04/19/2021 15:07:09", "content": "\n\n(smartbatched by inchi length).
\n
How so? I have been doing this for years for text data, and it works great. What you have to keep in mind though is: Don't sort after length over your whole training data, do it for windows where a window equals x batch sizes. Then you split the sorted windows again into batches and randomize them within the window. The latter, because so you don't get windows with batches of increasing lengths.
\nIf you do it over your whole training data, then yes, it won't generalize well. It might not even learn well in the first place and be unstable.
\n\nthere is a difference if you have big and small images.
\n
Resizing it to the largest size would still make no real difference to given approach. The scale is nevertheless different for the images. And empty patches should be ignored by image model anyway.
\nImportant question: Does it actually work better than an image model? In this case you would be right.
\n\nHow so?
\n
As you just described it. :)
\nAnother way of the randomization: sorted(lambda x: x[1], key=len(x)*(1+random.random()*0.1-0.05))
", "votes": null, "replies": [] }, { "id": 1278132, "author_name": "hengck23", "author_url": "", "post_date": "04/19/2021 15:29:14", "content": "\"The scale is nevertheless different for the images.\"
\nno. there is only 2 scales in the test and train images.
\nthe bigger molecule image has a border of 58 pixel and the smaller one has a border of 36 pixel
\"Does it actually work better than an image model? \"
\nI don't think it will be more accurate. But i hope it would be much faster
As you just described it. :)
\nI meant, how does it come to be unstable. Unless I described that also with my last part.
there is only 2 scales in the test and train images.
\nI see more than 2 different sizes of molecules. You can see it through the size of the letters or pentagons very well. Maybe the amount they where scaled afterwards has only two different sizes, but that doesn't change the absolute size.
But i hope it would be much faster
\nI see. But I doubt that a lot. You still need the AutoEncoder + Sequence-Encoder VS. (CNN) Image Model.
\nEdit: I think you could go with a Transformer for images and just exclude the white patches and use a smart positional embedding which takes that into account. That should work as well as a classical approach and be faster (than at least other transformers).
\n\nUnless I described that also with my last part.
\n\n\nIf you do it over your whole training data, then yes, it won't generalize well. It might not even learn well in the first place and be unstable.
\n
I guess it comes to that in a batch it sees that all sequences are of length L and it probably drives the model a lot in that direction.
", "votes": null, "replies": [] }, { "id": 1278261, "author_name": "cepheidq", "author_url": "", "post_date": "04/19/2021 17:40:14", "content": "that in a batch it sees that all sequences are of length L...
\nYes, the gradient goes to much in a specific, specialized direction. And if you don't permute if afterwards, the length increases/decreases the whole epoch. This means the model forgets earlier patters for different lengths.
\nThat's why I use sorting within a window of x*batch_size. And then permute the order of those locally sorted batches again. That can accelerate the training speed usually by 2x (for the sequence model) and also improve the loss slightly. Although the optimal window sizes for both objectives might be different.
\nx is usually in [2, 30]. So it sorts elements of up to 30 batches. For this competition I use currently x = 4, but haven't tuned it yet.
Why can local length sorting improve the loss slightly? Well, first the sequences are more similar, but still random enough so it gives a slightly better training signal. Second, you have less padding per sequence on average which means less noise (You would have to pad and probably reset to zero otherwise). The second part is even valid for a auto-regressive model because the weights that process the hidden embeddings are shared over all time-steps. Of course you just can mask the padding always (with zeros) - which I don't do.
\nThat worked well also for such benchmarks as Wikitext-103 where you
", "votes": null, "replies": [] }, { "id": 1278269, "author_name": "nofreewill", "author_url": "", "post_date": "04/19/2021 17:51:39", "content": "\n\naccelerate the training speed usually by 2x
\n
I doubt that. The speedup is around 2x just for the decoder part. If you use fixed sized images, then the time spent on the encoder is the same regardless of smart batching.
\nAlso, can you please compress your thoughts?
", "votes": null, "replies": [] }, { "id": 1278276, "author_name": "cepheidq", "author_url": "", "post_date": "04/19/2021 17:55:39", "content": "Your are right, in this case it isn't 2x. But I meant just the sequence part, forgot to add that.
\nI will add a TLDR in the future.
I hope you don't mind the critique!
", "votes": null, "replies": [] }, { "id": 1278298, "author_name": "cepheidq", "author_url": "", "post_date": "04/19/2021 18:05:00", "content": "I hope you don't mind the critique!
\nNo. Yours is based on arguments (apart from the compress part), so it's welcome.
\nI also hope I could help with my length sorting description.
\n\n(apart from the compress part)
\n
I was referring to that under critique :D
\nSo I hope you don't mind that!
It's fine. Some people don't understand it with a long explanation, others don't want/need those. You get a TLDR in case of long posts.
", "votes": null, "replies": [] } ] }, { "id": 1277824, "author_name": "radek1", "author_url": "", "post_date": "04/19/2021 09:11:28", "content": "very neat! 🙂 could I please ask how you are detecting orientation?
\nEDIT: from taking a look at the data it doesn't seem that they are ever rotated counter-clockwise or upside down… in this case that would be straightforward by just looking at the dimensions…
", "votes": null, "replies": [ { "id": 1277840, "author_name": "nofreewill", "author_url": "", "post_date": "04/19/2021 09:35:52", "content": "h>w -> rotate; but there are some images that don't follow this rule
", "votes": null, "replies": [] }, { "id": 1277848, "author_name": "hengck23", "author_url": "", "post_date": "04/19/2021 09:42:57", "content": "I train an orientation detection and scale detector using resnet34.
", "votes": null, "replies": [] } ] }, { "id": 1280476, "author_name": "hengck23", "author_url": "", "post_date": "04/22/2021 02:52:19", "content": "another way to do text-to-text:
\n(use ascii art)
O\n //\n Cl--{/\n \\\n \\_____\n / --- \\\n / \\\n \\\\ //\n \\_____/\n OH\n /\n C --C O\n // \\\\ //\nC C---C\n \\ __ / \\\n C --C OH\n