One simple idea that I have was to concate additional information like history_positions, speed, history_yaws to the final dense layers with the resnet output. And, throw a few dense layer after that. However, I found that doing so can lead to overfit pretty easily (huge gap between train and validation score), and give you worse result.
\nI was wondering if anyone has tried similar thing and got similar problem? And, why does this happen?
Also, is it a general pattern that adding dense layers after these deep neural net in image recognition will lead to worse result?
", "messages": [ { "id": "1093535", "postDate": "11/27/2020 19:51:32", "content": "One simple idea that I have was to concate additional information like history_positions, speed, history_yaws to the final dense layers with the resnet output. And, throw a few dense layer after that. However, I found that doing so can lead to overfit pretty easily (huge gap between train and validation score), and give you worse result.
\nI was wondering if anyone has tried similar thing and got similar problem? And, why does this happen?
Also, is it a general pattern that adding dense layers after these deep neural net in image recognition will lead to worse result?
", "rawMarkdown": "One simple idea that I have was to concate additional information like `history_positions`, speed, history_yaws to the final dense layers with the resnet output. And, throw a few dense layer after that. However, I found that doing so can lead to overfit pretty easily (huge gap between train and validation score), and give you worse result. \nI was wondering if anyone has tried similar thing and got similar problem? And, why does this happen?\n\nAlso, is it a general pattern that adding dense layers after these deep neural net in image recognition will lead to worse result?", "votes": null }, { "id": "1093547", "postDate": "11/27/2020 20:15:56", "content": "Yes. I tried the same and got the same result. I think it is because the network might have already picked up on this information from the raster? Not too sure though.
", "rawMarkdown": "Yes. I tried the same and got the same result. I think it is because the network might have already picked up on this information from the raster? Not too sure though.", "votes": null }, { "id": "1093698", "postDate": "11/28/2020 00:03:51", "content": "deeper heads didn't work for us (2 linear layers max)
\nalso, usage of additional data such as velocity only lead to a faster loss (also visible in validation) reduction but didn't yield a better result at the end of training. I guess the models were strong enough to pick all the information up from the image already and use it for the feature extraction with better results than late concatenation would give.
Glad that 1st place also got the similar observation. It sounds like making the problem easier for a deep neural net is not always a good thing.
\nFrom your experience to other competition, do you think this apply in general to image recognition problem or deep learning? Or, is it just specific for this problem?
I would say with any competition you hardly know before trying if auxiliary loss/data helps. Here I was pretty sure the NN will \"learn\" velocity and acceleration just from difference of frames, we tried anyways.
", "rawMarkdown": "I would say with any competition you hardly know before trying if auxiliary loss/data helps. Here I was pretty sure the NN will \"learn\" velocity and acceleration just from difference of frames, we tried anyways.", "votes": null }, { "id": "1094270", "postDate": "11/28/2020 12:59:32", "content": "I had this in my solution (BTW, it is all single model) and I added date, hour, extra info. Did it contribute much to the results, it is hard to confirm given the extra long run time and the nonlinear exploration path I went through till reaching a good score.
", "rawMarkdown": "I had this in my solution (BTW, it is all single model) and I added date, hour, extra info. Did it contribute much to the results, it is hard to confirm given the extra long run time and the nonlinear exploration path I went through till reaching a good score.", "votes": null }, { "id": "1094696", "postDate": "11/28/2020 20:27:21", "content": "Interesting! How did you merge them with resenet output? Did you do many regular fully connected layers and relu after that?
", "rawMarkdown": "Interesting! How did you merge them with resenet output? Did you do many regular fully connected layers and relu after that?", "votes": null }, { "id": "1094758", "postDate": "11/28/2020 22:21:04", "content": "Fully connected, relu, then output. Something like this:
\nx = torch.cat((x, y), axis=-1)\nx = self.backbone.fc(x)\nx = F.relu(x)\nx = self.output(x)\nWe also tried add some information, like agent type.
\nWe tried to input this additional data as multiplication between image feature vectors and additional metadata feature vectors, instead of concatenation, but not worked well.
Yes. I tried the same and got the same result. I think it is because the network might have already picked up on this information from the raster? Not too sure though.
", "votes": null, "replies": [] }, { "id": 1093698, "author_name": "ilu000", "author_url": "", "post_date": "11/28/2020 00:03:51", "content": "deeper heads didn't work for us (2 linear layers max)
\nalso, usage of additional data such as velocity only lead to a faster loss (also visible in validation) reduction but didn't yield a better result at the end of training. I guess the models were strong enough to pick all the information up from the image already and use it for the feature extraction with better results than late concatenation would give.
Glad that 1st place also got the similar observation. It sounds like making the problem easier for a deep neural net is not always a good thing.
\nFrom your experience to other competition, do you think this apply in general to image recognition problem or deep learning? Or, is it just specific for this problem?
I would say with any competition you hardly know before trying if auxiliary loss/data helps. Here I was pretty sure the NN will \"learn\" velocity and acceleration just from difference of frames, we tried anyways.
", "votes": null, "replies": [] } ] }, { "id": 1094270, "author_name": "ggrizzly", "author_url": "", "post_date": "11/28/2020 12:59:32", "content": "I had this in my solution (BTW, it is all single model) and I added date, hour, extra info. Did it contribute much to the results, it is hard to confirm given the extra long run time and the nonlinear exploration path I went through till reaching a good score.
", "votes": null, "replies": [ { "id": 1094696, "author_name": "louis925", "author_url": "", "post_date": "11/28/2020 20:27:21", "content": "Interesting! How did you merge them with resenet output? Did you do many regular fully connected layers and relu after that?
", "votes": null, "replies": [] }, { "id": 1094758, "author_name": "ggrizzly", "author_url": "", "post_date": "11/28/2020 22:21:04", "content": "Fully connected, relu, then output. Something like this:
\nx = torch.cat((x, y), axis=-1)\nx = self.backbone.fc(x)\nx = F.relu(x)\nx = self.output(x)\nWe also tried add some information, like agent type.
\nWe tried to input this additional data as multiplication between image feature vectors and additional metadata feature vectors, instead of concatenation, but not worked well.