I’m guessing that success is going to require multiframe inference rather than just per-image work. It looks to me like Keras has 2 pre-written layers that are natural fits for working with video sequences: Conv3D is the obvious match but there’s also the TimeDistributed layer, which could be used to wrap a Conv2D. The temptation with TimeDistributed is that it seems like it might be easier to integrate with known object detection architectures. The temptation with a Conv3D-based architecture is that it feels like there’s better affinity between the features.
Any experiences or thoughts with these options or others?
", "messages": [ { "id": "1595384", "postDate": "11/25/2021 16:37:01", "content": "I’m guessing that success is going to require multiframe inference rather than just per-image work. It looks to me like Keras has 2 pre-written layers that are natural fits for working with video sequences: Conv3D is the obvious match but there’s also the TimeDistributed layer, which could be used to wrap a Conv2D. The temptation with TimeDistributed is that it seems like it might be easier to integrate with known object detection architectures. The temptation with a Conv3D-based architecture is that it feels like there’s better affinity between the features.
Any experiences or thoughts with these options or others?
", "rawMarkdown": "I’m guessing that success is going to require multiframe inference rather than just per-image work. It looks to me like Keras has 2 pre-written layers that are natural fits for working with video sequences: `Conv3D` is the obvious match but there’s also the `TimeDistributed` layer, which could be used to wrap a `Conv2D`. The temptation with `TimeDistributed` is that it seems like it might be easier to integrate with known object detection architectures. The temptation with a `Conv3D`-based architecture is that it feels like there’s better affinity between the features. \n\nAny experiences or thoughts with these options or others?", "votes": null }, { "id": "1595422", "postDate": "11/25/2021 17:17:16", "content": "What about Conv3D - I think that Conv3D will information about multiple frames. My intuition says me that it would be useful for predicting one thing in multiple frames (e.g. some video action). Would it be useful in our case when we need to predict boxes per frame?
\nOr may we pass only several subsequence frames to the Conv3D? In such a case ground truth bounding boxes are not changed dramatically. But frames are very similar as well.
\nAny thought about it?
Well, the thing is that ROIs do shift over frames as the tow passes over the reef. I was thinking of doing some of that visualization today, at least for those sequences that do not have many ROIs in the same frame (there’s one sequence that has 18 COTS in the image). Once you have multiple or overlapping ROIs, the difficult part of object tracking kicks in.
", "rawMarkdown": "Well, the thing is that ROIs _do_ shift over frames as the tow passes over the reef. I was thinking of doing some of that visualization today, at least for those sequences that do not have many ROIs in the same frame (there’s one sequence that has 18 COTS in the image). Once you have multiple or overlapping ROIs, the difficult part of object tracking kicks in.", "votes": null } ], "comments": [ { "id": 1595422, "author_name": "meowmeowmeowmeowmeow", "author_url": "", "post_date": "11/25/2021 17:17:16", "content": "What about Conv3D - I think that Conv3D will information about multiple frames. My intuition says me that it would be useful for predicting one thing in multiple frames (e.g. some video action). Would it be useful in our case when we need to predict boxes per frame?
\nOr may we pass only several subsequence frames to the Conv3D? In such a case ground truth bounding boxes are not changed dramatically. But frames are very similar as well.
\nAny thought about it?
Well, the thing is that ROIs do shift over frames as the tow passes over the reef. I was thinking of doing some of that visualization today, at least for those sequences that do not have many ROIs in the same frame (there’s one sequence that has 18 COTS in the image). Once you have multiple or overlapping ROIs, the difficult part of object tracking kicks in.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1595384": "I’m guessing that success is going to require multiframe inference rather than just per-image work. It looks to me like Keras has 2 pre-written layers that are natural fits for working with video sequences: `Conv3D` is the obvious match but there’s also the `TimeDistributed` layer, which could be used to wrap a `Conv2D`. The temptation with `TimeDistributed` is that it seems like it might be easier to integrate with known object detection architectures. The temptation with a `Conv3D`-based architecture is that it feels like there’s better affinity between the features. \n\nAny experiences or thoughts with these options or others?", "1595422": "What about `Conv3D` - I think that `Conv3D` will information about multiple frames. My intuition says me that it would be useful for predicting one thing in multiple frames (e.g. some video action). Would it be useful in our case when we need to predict boxes per frame?\nOr may we pass only several subsequence frames to the `Conv3D`? In such a case ground truth bounding boxes are not changed dramatically. But frames are very similar as well.\nAny thought about it?", "1595705": "Well, the thing is that ROIs _do_ shift over frames as the tow passes over the reef. I was thinking of doing some of that visualization today, at least for those sequences that do not have many ROIs in the same frame (there’s one sequence that has 18 COTS in the image). Once you have multiple or overlapping ROIs, the difficult part of object tracking kicks in." }, "source": "meta" }