![](\"https://i.imgur.com/tTOe7cV.png\")
\n\n\n
Howdy!!! Seems like Patrick is lost, and SpongeBob and others need our help to find him. I was discussing with my friends about different ideas we can use to make a better model(and find Patrick). We came up with an idea that is using a kind of Sequential model like LSTM or RNN to capture the sequential information and then do Object detection. The Idea is similar to video object detection. As most people are doing only object detection, it will be great to throw a Sequential model with object detection and see if that can help or not. It is like taking the context of the previous frame and using it to predict the object's position for the next frame.
If you see the video from the Overview tab, it also mentions that model related to time series. And one picture from the video,
\n\n![](\"https://i.imgur.com/XesZdt4.png\")
\n
also shows[not sure that info is right or wrong] that starfishes borns in groups like you might not find a single(or only 2 or 3) starfish in the sea, it means if you see a starfish that means you will find starfish in some nearby areas too. So, taking this into context and using the sequential image data[kind of like a video] it would be great to create a video object detection model that can detect starfishes(and help find Patrick).
\nIt also opens a lot of opportunities for attention and transformer-based models[DART might do really well]. I don't have experience in video object detection but here are some Benchmark papers on video object detection that i could find,
Check more here -> https://paperswithcode.com/task/video-object-detection
\n\nI will try to implement a model based on this idea. Would like to hear your thoughts on this, and any point or advice you want to giveš. Happy kaggling, and Thanksgiving!
", "messages": [ { "id": "1593743", "postDate": "11/24/2021 08:19:24", "content": "\n\n
Howdy!!! Seems like Patrick is lost, and SpongeBob and others need our help to find him. I was discussing with my friends about different ideas we can use to make a better model(and find Patrick). We came up with an idea that is using a kind of Sequential model like LSTM or RNN to capture the sequential information and then do Object detection. The Idea is similar to video object detection. As most people are doing only object detection, it will be great to throw a Sequential model with object detection and see if that can help or not. It is like taking the context of the previous frame and using it to predict the object's position for the next frame.
If you see the video from the Overview tab, it also mentions that model related to time series. And one picture from the video,
\n\n\n
also shows[not sure that info is right or wrong] that starfishes borns in groups like you might not find a single(or only 2 or 3) starfish in the sea, it means if you see a starfish that means you will find starfish in some nearby areas too. So, taking this into context and using the sequential image data[kind of like a video] it would be great to create a video object detection model that can detect starfishes(and help find Patrick).
\nIt also opens a lot of opportunities for attention and transformer-based models[DART might do really well]. I don't have experience in video object detection but here are some Benchmark papers on video object detection that i could find,
Check more here -> https://paperswithcode.com/task/video-object-detection
\n\nI will try to implement a model based on this idea. Would like to hear your thoughts on this, and any point or advice you want to giveš. Happy kaggling, and Thanksgiving!
", "rawMarkdown": "\n\n
\n\n
How long sequence you are going to use in one sample?
\nHere we have a limited amount of video. Will be there a high chance of overfitting?
In addition to your brilliant idea - how about Inflated 3D convolution (I am not familiar with transformed yet, sorry). May it be helpful to perform convolution by width/height and time as well?
", "rawMarkdown": "How long sequence you are going to use in one sample?\nHere we have a limited amount of video. Will be there a high chance of overfitting?\n\nIn addition to your brilliant idea - how about Inflated 3D convolution (I am not familiar with transformed yet, sorry). May it be helpful to perform convolution by width/height and time as well?", "votes": null }, { "id": "1593816", "postDate": "11/24/2021 09:48:07", "content": "yeah, I guess it is a very bad idea to go with, so many problems, sorry for posting it. i guess its not much helpful taking sequence information into consideration after all.
\nit's just my opinion though.
", "rawMarkdown": "yeah, I guess it is a very bad idea to go with, so many problems, sorry for posting it. i guess its not much helpful taking sequence information into consideration after all. \n\nit's just my opinion though.", "votes": null }, { "id": "1593825", "postDate": "11/24/2021 09:53:49", "content": "Hey, mate. I believe your approach is good. Anyway, thank you for posting your thoughts!
", "rawMarkdown": "Hey, mate. I believe your approach is good. Anyway, thank you for posting your thoughts!", "votes": null }, { "id": "1594332", "postDate": "11/24/2021 17:52:23", "content": "I strongly agree that using multiple frames is likely to be fruitful. Another point Iād make is that, unlike general challenges with object tracking, the COTS are (essentially) stationery and the tow is (relatively) constant and unidirectional. So as the tow passes over a target, we can assume that the target will shift, over the course of n frames, not much in x but increasing in y.
That might be projecting too much heuristic thinking ā ājust let the model decideā ā but given the desire for efficiency, there is a part of me that is looking for a solution that takes advantage of that specific aspect of the domain.
", "rawMarkdown": "I strongly agree that using multiple frames is likely to be fruitful. Another point Iād make is that, unlike general challenges with object tracking, the COTS are (essentially) stationery and the tow is (relatively) constant and unidirectional. So as the tow passes over a target, we can assume that the target will shift, over the course of `n` frames, not much in `x` but increasing in `y`. \n\nThat might be projecting too much heuristic thinking ā ājust let the model decideā ā but given the desire for efficiency, there _is_ a part of me that is looking for a solution that takes advantage of that specific aspect of the domain.", "votes": null }, { "id": "1594489", "postDate": "11/24/2021 21:51:30", "content": "Thank you @lobrien , I really appreciate your thoughtfulness.
\nSo as the tow passes over a target, we can assume that the target will shift, over the course of n frames, not much in x but increasing in y.
\nby that, you mean that over the course of n frames right after passing a target, the next target would be a bit higher[increase in y] compared to the previous target but the difference in x won't be much compared to y?
\n![](\"https://i.imgur.com/3vPdBlK.png\")
\n
My interpretation might be wrong though.
", "rawMarkdown": "Thank you @lobrien , I really appreciate your thoughtfulness. \n`So as the tow passes over a target, we can assume that the target will shift, over the course of n frames, not much in x but increasing in y.` \nby that, you mean that over the course of n frames right after passing a target, the next target would be a bit higher[increase in y] compared to the previous target but the difference in x won't be much compared to y? \n\n- This might be true cause I have observed in the data that there are many times when I saw that the camera angle was from the top[not straight top though], so we are getting a tilted top view, like looking at something at the bottom from a tilted top. And when you see something from the top, then the next element looks a bit further than the previous element. but at the same time, the next element might be at the same x, this will happen if the targets happen to be in a row.\n\n\n\n\n
hey @meowmeowmeowmeowmeow , your point about the less data is true, and I also saw that there are many empty annotations in the dataset, so using a pre-trained model might yield some good results. and if someone wants to try it then, they can check out the total length of the video, and divide them into equal parts accordingly. I don't know the specifics though, cause I have not tried video object detection before, I'm sorry about that.
", "rawMarkdown": "hey @meowmeowmeowmeowmeow , your point about the less data is true, and I also saw that there are many empty annotations in the dataset, so using a pre-trained model might yield some good results. and if someone wants to try it then, they can check out the total length of the video, and divide them into equal parts accordingly. I don't know the specifics though, cause I have not tried video object detection before, I'm sorry about that.", "votes": null }, { "id": "1596486", "postDate": "11/26/2021 14:41:05", "content": "Great catch! I got the same idea when I checked the training data carefully. The main question is: can we expect that the test data will contains frames from one video in order? If yes, then in this case modified tracking algos (e.g.: IoU tracking) would help a lot.
", "rawMarkdown": "Great catch! I got the same idea when I checked the training data carefully. The main question is: can we expect that the test data will contains frames from one video in order? If yes, then in this case modified tracking algos (e.g.: IoU tracking) would help a lot.", "votes": null }, { "id": "1596555", "postDate": "11/26/2021 16:11:29", "content": "In that competition, test results must be submitted via a special API that feeds data frame by frame in the correct order.
", "rawMarkdown": "In that competition, test results must be submitted via a special API that feeds data frame by frame in the correct order.", "votes": null }, { "id": "1596699", "postDate": "11/26/2021 18:31:38", "content": "Yeah as @meowmeowmeowmeowmeow said, this is from the data tab,
\nThis competition uses a hidden test set that will be served by an API to ensure you evaluate the images in the same order they were recorded within each video.
Well, turns out I was wrong that \"ā¦as the tow passes over a target, we can assume that the target will shift, over the course of n frames, not much in x but increasing in y.\"
\nI plotted the bounding boxes from video 0, frames 16-32 (lines 18-34 in train.csv):
There was actually a significant shift in X and the shift in Y does not increase every time.
\nWhy I was wrong: as the FOV moves, anything not in the very center is going to shift in X and Y due to both the camera's motion in Z, but because of the angle in the viewing frustrum. Objects to either side of the vertical centerline will shift further to the side as the camera passes by, objects well above the horizontal centerline may initially shift up in Y as distance decreases.
", "rawMarkdown": "Well, turns out I was wrong that \"...as the tow passes over a target, we can assume that the target will shift, over the course of n frames, not much in x but increasing in y.\"\n\nI plotted the bounding boxes from video 0, frames 16-32 (lines 18-34 in `train.csv`):\n\n\n\nThere was actually a significant shift in X and the shift in Y does not increase every time. \n\nWhy I was wrong: as the FOV moves, anything not in the very center is going to shift in X and Y due to both the camera's motion in Z, but because of the angle in the viewing frustrum. Objects to either side of the vertical centerline will shift further to the side as the camera passes by, objects well above the horizontal centerline may initially shift up in Y as distance decreases.", "votes": null }, { "id": "1598699", "postDate": "11/28/2021 17:49:29", "content": "Oh, I see, thank you for explaining @lobrien, I guess my interpretation was wrong.
\nI'm really curious what is this tow that you are referring to?
I mean the camera that is being towed. While it is likely true that every frame is basically the reef \"passing under\" the towed camera, you can see from my graph that the region of interest is not just \"pretty much constant in X, uniformly increasing in Y.\"
", "rawMarkdown": "I mean the camera that is being towed. While it is likely true that every frame is basically the reef \"passing under\" the towed camera, you can see from my graph that the region of interest is not just \"pretty much constant in X, uniformly increasing in Y.\"", "votes": null }, { "id": "1599347", "postDate": "11/29/2021 11:10:32", "content": "This is natural that BB coordinates change linearly. Moreover, size of BB changes as well. This is due to camera optical distortion. But anyway, we should expect COTS in the same region (with some padding of course) on the next frame.
\nCan we try to feed the expected region as an additional ROI to the FasterRCNN, for example?
", "rawMarkdown": "This is natural that BB coordinates change linearly. Moreover, size of BB changes as well. This is due to camera optical distortion. But anyway, we should expect COTS in the same region (with some padding of course) on the next frame. \n\nCan we try to feed the expected region as an additional ROI to the FasterRCNN, for example?", "votes": null }, { "id": "1599645", "postDate": "11/29/2021 16:41:00", "content": "Yes, I'd like to try something like that, but whether to do it manually or with something like a TimeDistributed layer, I don't know. I'm concerned about the memory cost of using TimeDistributed or Conv3D since performance is a goal of the contest.
", "rawMarkdown": "Yes, I'd like to try something like that, but whether to do it manually or with something like a TimeDistributed layer, I don't know. I'm concerned about the memory cost of using TimeDistributed or Conv3D since performance is a goal of the contest.", "votes": null } ], "comments": [ { "id": 1593810, "author_name": "meowmeowmeowmeowmeow", "author_url": "", "post_date": "11/24/2021 09:43:00", "content": "How long sequence you are going to use in one sample?
\nHere we have a limited amount of video. Will be there a high chance of overfitting?
In addition to your brilliant idea - how about Inflated 3D convolution (I am not familiar with transformed yet, sorry). May it be helpful to perform convolution by width/height and time as well?
", "votes": null, "replies": [ { "id": 1593816, "author_name": "soumya9977", "author_url": "", "post_date": "11/24/2021 09:48:07", "content": "yeah, I guess it is a very bad idea to go with, so many problems, sorry for posting it. i guess its not much helpful taking sequence information into consideration after all.
\nit's just my opinion though.
", "votes": null, "replies": [] }, { "id": 1593825, "author_name": "meowmeowmeowmeowmeow", "author_url": "", "post_date": "11/24/2021 09:53:49", "content": "Hey, mate. I believe your approach is good. Anyway, thank you for posting your thoughts!
", "votes": null, "replies": [] }, { "id": 1594499, "author_name": "soumya9977", "author_url": "", "post_date": "11/24/2021 21:59:13", "content": "hey @meowmeowmeowmeowmeow , your point about the less data is true, and I also saw that there are many empty annotations in the dataset, so using a pre-trained model might yield some good results. and if someone wants to try it then, they can check out the total length of the video, and divide them into equal parts accordingly. I don't know the specifics though, cause I have not tried video object detection before, I'm sorry about that.
", "votes": null, "replies": [] } ] }, { "id": 1594332, "author_name": "lobrien", "author_url": "", "post_date": "11/24/2021 17:52:23", "content": "I strongly agree that using multiple frames is likely to be fruitful. Another point Iād make is that, unlike general challenges with object tracking, the COTS are (essentially) stationery and the tow is (relatively) constant and unidirectional. So as the tow passes over a target, we can assume that the target will shift, over the course of n frames, not much in x but increasing in y.
That might be projecting too much heuristic thinking ā ājust let the model decideā ā but given the desire for efficiency, there is a part of me that is looking for a solution that takes advantage of that specific aspect of the domain.
", "votes": null, "replies": [ { "id": 1594489, "author_name": "soumya9977", "author_url": "", "post_date": "11/24/2021 21:51:30", "content": "Thank you @lobrien , I really appreciate your thoughtfulness.
\nSo as the tow passes over a target, we can assume that the target will shift, over the course of n frames, not much in x but increasing in y.
\nby that, you mean that over the course of n frames right after passing a target, the next target would be a bit higher[increase in y] compared to the previous target but the difference in x won't be much compared to y?
\n\n
My interpretation might be wrong though.
", "votes": null, "replies": [] }, { "id": 1596730, "author_name": "lobrien", "author_url": "", "post_date": "11/26/2021 19:17:55", "content": "Well, turns out I was wrong that \"ā¦as the tow passes over a target, we can assume that the target will shift, over the course of n frames, not much in x but increasing in y.\"
\nI plotted the bounding boxes from video 0, frames 16-32 (lines 18-34 in train.csv):
There was actually a significant shift in X and the shift in Y does not increase every time.
\nWhy I was wrong: as the FOV moves, anything not in the very center is going to shift in X and Y due to both the camera's motion in Z, but because of the angle in the viewing frustrum. Objects to either side of the vertical centerline will shift further to the side as the camera passes by, objects well above the horizontal centerline may initially shift up in Y as distance decreases.
", "votes": null, "replies": [] }, { "id": 1598699, "author_name": "soumya9977", "author_url": "", "post_date": "11/28/2021 17:49:29", "content": "Oh, I see, thank you for explaining @lobrien, I guess my interpretation was wrong.
\nI'm really curious what is this tow that you are referring to?
I mean the camera that is being towed. While it is likely true that every frame is basically the reef \"passing under\" the towed camera, you can see from my graph that the region of interest is not just \"pretty much constant in X, uniformly increasing in Y.\"
", "votes": null, "replies": [] }, { "id": 1599347, "author_name": "meowmeowmeowmeowmeow", "author_url": "", "post_date": "11/29/2021 11:10:32", "content": "This is natural that BB coordinates change linearly. Moreover, size of BB changes as well. This is due to camera optical distortion. But anyway, we should expect COTS in the same region (with some padding of course) on the next frame.
\nCan we try to feed the expected region as an additional ROI to the FasterRCNN, for example?
", "votes": null, "replies": [] }, { "id": 1599645, "author_name": "lobrien", "author_url": "", "post_date": "11/29/2021 16:41:00", "content": "Yes, I'd like to try something like that, but whether to do it manually or with something like a TimeDistributed layer, I don't know. I'm concerned about the memory cost of using TimeDistributed or Conv3D since performance is a goal of the contest.
", "votes": null, "replies": [] } ] }, { "id": 1596486, "author_name": "benkelaci", "author_url": "", "post_date": "11/26/2021 14:41:05", "content": "Great catch! I got the same idea when I checked the training data carefully. The main question is: can we expect that the test data will contains frames from one video in order? If yes, then in this case modified tracking algos (e.g.: IoU tracking) would help a lot.
", "votes": null, "replies": [ { "id": 1596555, "author_name": "meowmeowmeowmeowmeow", "author_url": "", "post_date": "11/26/2021 16:11:29", "content": "In that competition, test results must be submitted via a special API that feeds data frame by frame in the correct order.
", "votes": null, "replies": [] }, { "id": 1596699, "author_name": "soumya9977", "author_url": "", "post_date": "11/26/2021 18:31:38", "content": "Yeah as @meowmeowmeowmeowmeow said, this is from the data tab,
\nThis competition uses a hidden test set that will be served by an API to ensure you evaluate the images in the same order they were recorded within each video.
\n\n
\n\n
\n\n
![\"2](\"https://imgur.com/a/MqU2PjI\")