\n
![\"https://i.ibb.co/TP72zHd/Selection-999-1549.png\"](\"https://i.ibb.co/TP72zHd/Selection-999-1549.png\")
I found an interesting paper that describes a methodology for using the path signature method for classification problems that are similar to the ASL problem. Their approach used the signature method to transform a data stream into a set of features, which were fed into a simple MLP that was able to outperform more complex models such as LSTM in classifying . The signature method essentially takes a stream of data and returns a series of iterated integrals for that stream, which are generally unique to each stream, with the only exception being that if a path were to retrace over itself, it would not be unique from a path that did not retrace over the same points. This limitation can be handled by adding time/frames as a dimension. Some current packages that implement this method are signatory, iisignature, and esig. Unfortunately, to my understanding, doing this sort of feature engineering must be done in TensorFlow or PyTorch in order to be converted into TF-Lite, so we can't use those packages. I've tried but have found it too challenging for myself to implement this method in Torch. Hopefully, someone else can and can improve their model.
\n\"Developing the Path Signature Methodology and its Application to Landmark-based Human Action Recognition\" https://arxiv.org/abs/1707.03993
\nAnother paper, \"Deep Signature Transforms\" https://arxiv.org/abs/1905.08494, was written by the developer for Signatory, which uses the transformation as layers in an NN. Though this method will be even more difficult to implement in TF-Lite.
", "messages": [ { "id": 2190128, "postDate": "2023-03-21T04:04:13.463Z", "content": "I found an interesting paper that describes a methodology for using the path signature method for classification problems that are similar to the ASL problem. Their approach used the signature method to transform a data stream into a set of features, which were fed into a simple MLP that was able to outperform more complex models such as LSTM in classifying . The signature method essentially takes a stream of data and returns a series of iterated integrals for that stream, which are generally unique to each stream, with the only exception being that if a path were to retrace over itself, it would not be unique from a path that did not retrace over the same points. This limitation can be handled by adding time/frames as a dimension. Some current packages that implement this method are signatory, iisignature, and esig. Unfortunately, to my understanding, doing this sort of feature engineering must be done in TensorFlow or PyTorch in order to be converted into TF-Lite, so we can't use those packages. I've tried but have found it too challenging for myself to implement this method in Torch. Hopefully, someone else can and can improve their model.
\n\"Developing the Path Signature Methodology and its Application to Landmark-based Human Action Recognition\" https://arxiv.org/abs/1707.03993
\nAnother paper, \"Deep Signature Transforms\" https://arxiv.org/abs/1905.08494, was written by the developer for Signatory, which uses the transformation as layers in an NN. Though this method will be even more difficult to implement in TF-Lite.
", "rawMarkdown": "I found an interesting paper that describes a methodology for using the path signature method for classification problems that are similar to the ASL problem. Their approach used the signature method to transform a data stream into a set of features, which were fed into a simple MLP that was able to outperform more complex models such as LSTM in classifying . The signature method essentially takes a stream of data and returns a series of iterated integrals for that stream, which are generally unique to each stream, with the only exception being that if a path were to retrace over itself, it would not be unique from a path that did not retrace over the same points. This limitation can be handled by adding time/frames as a dimension. Some current packages that implement this method are signatory, iisignature, and esig. Unfortunately, to my understanding, doing this sort of feature engineering must be done in TensorFlow or PyTorch in order to be converted into TF-Lite, so we can't use those packages. I've tried but have found it too challenging for myself to implement this method in Torch. Hopefully, someone else can and can improve their model.\n\n\"Developing the Path Signature Methodology and its Application to Landmark-based Human Action Recognition\" https://arxiv.org/abs/1707.03993\n\nAnother paper, \"Deep Signature Transforms\" https://arxiv.org/abs/1905.08494, was written by the developer for Signatory, which uses the transformation as layers in an NN. Though this method will be even more difficult to implement in TF-Lite.", "votes": 3 }, { "id": 2190148, "postDate": "2023-03-21T04:31:50.410Z", "content": "\"Though this method will be even more difficult to implement in TF-Lite\"
\nyou can always use this as a teacher for distillation
", "rawMarkdown": "\"Though this method will be even more difficult to implement in TF-Lite\"\n\nyou can always use this as a teacher for distillation", "votes": 2, "replies": [ { "id": 2190154, "postDate": "2023-03-21T04:38:48.693Z", "content": "I haven't heard of this before, and a quick google search has me interested. Thanks for the advice
", "rawMarkdown": "I haven't heard of this before, and a quick google search has me interested. Thanks for the advice", "votes": 1, "replies": [ { "id": 2190232, "postDate": "2023-03-21T05:59:55.240Z", "content": "take the famous DEIT in vision transformer as an example:
\n
the extra distllation token and loss force the model to learn information not only from the train data but also from other model (teacher)
\nanother way to think if it as aux loss with teacher label, etc as aux target
", "rawMarkdown": "take the famous DEIT in vision transformer as an example:\n\n\nthe extra distllation token and loss force the model to learn information not only from the train data but also from other model (teacher)\n\nanother way to think if it as aux loss with teacher label, etc as aux target", "votes": 3 } ] } ] }, { "id": 2232510, "postDate": "2023-04-24T11:57:43.910Z", "content": "Thanks for posting! I tried to find the discussion where this was posted a few days ago, but couldn't.
\nReviewing this has helped me learn quite a bit.
\nThis video explains some of the background. It begins very abstractly, but eventually comes around:
\nThis repo also has useful resources for background on signatures:
\nThe signatory library can calculate the required dimensionality like so
\nFor sequences of length 10 sampled from 5D space to depth k=3, each signature requires 155 dimensions. For points sampled from 3D that would be significantly less, but I haven't calculated it yet. Not every hand landmark needs to be included. It's not clear that 10 frames would be sufficient.
\nThere are alot of discussions involving performance cutoffs. Does anyone have any feedback on how this works for TF Lite?
", "rawMarkdown": "Thanks for posting! I tried to find the discussion where this was posted a few days ago, but couldn't. \n\nReviewing this has helped me learn quite a bit.\n\n[This video](https://www.youtube.com/watch?v=Lj_vs0nq1NA) explains some of the background. It begins very abstractly, but eventually comes around: \n\n[This repo](https://github.com/kormilitzin/the-signature-method-in-machine-learning) also has useful resources for background on signatures: \n\nThe signatory library can calculate the required dimensionality [like so](https://signatory.readthedocs.io/en/latest/pages/examples/simple.html)\n\nFor sequences of length 10 sampled from 5D space to depth k=3, each signature requires 155 dimensions. For points sampled from 3D that would be significantly less, but I haven't calculated it yet. Not every hand landmark needs to be included. It's not clear that 10 frames would be sufficient.\n\nThere are alot of discussions involving performance cutoffs. Does anyone have any feedback on how this works for TF Lite?" }, { "id": 2232501, "postDate": "2023-04-24T11:54:49.040Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 2190148, "author_name": "hengck23", "author_url": "", "post_date": "2023-03-21T04:31:50.410000", "content": "\"Though this method will be even more difficult to implement in TF-Lite\"
\nyou can always use this as a teacher for distillation
", "votes": 2, "replies": [ { "id": 2190154, "author_name": "Jordan Connolly", "author_url": "", "post_date": "2023-03-21T04:38:48.693000", "content": "I haven't heard of this before, and a quick google search has me interested. Thanks for the advice
", "votes": 1, "replies": [ { "id": 2190232, "author_name": "hengck23", "author_url": "", "post_date": "2023-03-21T05:59:55.240000", "content": "take the famous DEIT in vision transformer as an example:
\n
the extra distllation token and loss force the model to learn information not only from the train data but also from other model (teacher)
\nanother way to think if it as aux loss with teacher label, etc as aux target
", "votes": 3, "replies": [] } ] } ] }, { "id": 2232510, "author_name": "davidconner", "author_url": "", "post_date": "2023-04-24T11:57:43.910000", "content": "Thanks for posting! I tried to find the discussion where this was posted a few days ago, but couldn't.
\nReviewing this has helped me learn quite a bit.
\nThis video explains some of the background. It begins very abstractly, but eventually comes around:
\nThis repo also has useful resources for background on signatures:
\nThe signatory library can calculate the required dimensionality like so
\nFor sequences of length 10 sampled from 5D space to depth k=3, each signature requires 155 dimensions. For points sampled from 3D that would be significantly less, but I haven't calculated it yet. Not every hand landmark needs to be included. It's not clear that 10 frames would be sufficient.
\nThere are alot of discussions involving performance cutoffs. Does anyone have any feedback on how this works for TF Lite?
", "votes": 0, "replies": [] }, { "id": 2232501, "author_name": "", "author_url": "", "post_date": "2023-04-24T11:54:49.040000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2190128": "I found an interesting paper that describes a methodology for using the path signature method for classification problems that are similar to the ASL problem. Their approach used the signature method to transform a data stream into a set of features, which were fed into a simple MLP that was able to outperform more complex models such as LSTM in classifying . The signature method essentially takes a stream of data and returns a series of iterated integrals for that stream, which are generally unique to each stream, with the only exception being that if a path were to retrace over itself, it would not be unique from a path that did not retrace over the same points. This limitation can be handled by adding time/frames as a dimension. Some current packages that implement this method are signatory, iisignature, and esig. Unfortunately, to my understanding, doing this sort of feature engineering must be done in TensorFlow or PyTorch in order to be converted into TF-Lite, so we can't use those packages. I've tried but have found it too challenging for myself to implement this method in Torch. Hopefully, someone else can and can improve their model.\n\n\"Developing the Path Signature Methodology and its Application to Landmark-based Human Action Recognition\" https://arxiv.org/abs/1707.03993\n\nAnother paper, \"Deep Signature Transforms\" https://arxiv.org/abs/1905.08494, was written by the developer for Signatory, which uses the transformation as layers in an NN. Though this method will be even more difficult to implement in TF-Lite.", "2190148": "\"Though this method will be even more difficult to implement in TF-Lite\"\n\nyou can always use this as a teacher for distillation", "2232510": "Thanks for posting! I tried to find the discussion where this was posted a few days ago, but couldn't. \n\nReviewing this has helped me learn quite a bit.\n\n[This video](https://www.youtube.com/watch?v=Lj_vs0nq1NA) explains some of the background. It begins very abstractly, but eventually comes around: \n\n[This repo](https://github.com/kormilitzin/the-signature-method-in-machine-learning) also has useful resources for background on signatures: \n\nThe signatory library can calculate the required dimensionality [like so](https://signatory.readthedocs.io/en/latest/pages/examples/simple.html)\n\nFor sequences of length 10 sampled from 5D space to depth k=3, each signature requires 155 dimensions. For points sampled from 3D that would be significantly less, but I haven't calculated it yet. Not every hand landmark needs to be included. It's not clear that 10 frames would be sufficient.\n\nThere are alot of discussions involving performance cutoffs. Does anyone have any feedback on how this works for TF Lite?", "2232501": "" } }