{ "id": 391812, "title": "🗢 Lovely lips", "url": "/competitions/asl-signs/discussion/391812", "author_name": "Andrew", "post_date": "2023-03-02T18:07:42.354000", "votes": 45, "comment_count": 18, "views": 0, "content": "

Summary: Paying attention to lip data gives a 1% boost over a model that ignores it.

\n

My best model, which scored 0.44 on the LB, used data from the whole body (face, pose, LH, RH). When I removed the face data, my validation and LB scores improved by 2 percentage points to 0.46 (and the model trained 10x faster). When I then added just the 40 lip points back again, my validation and LB scores increased by an additional percentage point to 0.47.

\n

Out of curiosity, I trained a lip-reading model with access to the lip positions only, using the same model architecture. It achieved a score for 0.12 (which is 30x better than guessing at random and surprisingly good since it only had mean/std data for each point). Clearly never going to win any prizes on its own, but interesting all the same.

\n

All of which answers my earlier question/comment about lip reading.

", "messages": [ { "id": 2166281, "postDate": "2023-03-02T18:07:42.353Z", "content": "

Summary: Paying attention to lip data gives a 1% boost over a model that ignores it.

\n

My best model, which scored 0.44 on the LB, used data from the whole body (face, pose, LH, RH). When I removed the face data, my validation and LB scores improved by 2 percentage points to 0.46 (and the model trained 10x faster). When I then added just the 40 lip points back again, my validation and LB scores increased by an additional percentage point to 0.47.

\n

Out of curiosity, I trained a lip-reading model with access to the lip positions only, using the same model architecture. It achieved a score for 0.12 (which is 30x better than guessing at random and surprisingly good since it only had mean/std data for each point). Clearly never going to win any prizes on its own, but interesting all the same.

\n

All of which answers my earlier question/comment about lip reading.

", "rawMarkdown": "**Summary**: Paying attention to lip data gives a 1% boost over a model that ignores it.\n\nMy best model, which scored 0.44 on the LB, used data from the whole body (face, pose, LH, RH). When I removed the face data, my validation and LB scores improved by 2 percentage points to 0.46 (and the model trained 10x faster). When I then added just the 40 lip points back again, my validation and LB scores increased by an additional percentage point to 0.47.\n\nOut of curiosity, I trained a lip-reading model with access to the lip positions only, using the same model architecture. It achieved a score for 0.12 (which is 30x better than guessing at random and surprisingly good since it only had mean/std data for each point). Clearly never going to win any prizes on its own, but interesting all the same.\n\nAll of which answers my [earlier question/comment about lip reading](https://www.kaggle.com/competitions/asl-signs/discussion/390509).", "votes": 45 }, { "id": 2167319, "postDate": "2023-03-03T12:07:58.173Z", "content": "

Thanks for this! Could you please tell me which keypoints are the lips?

", "rawMarkdown": "Thanks for this! Could you please tell me which keypoints are the lips?\n", "votes": 1, "replies": [ { "id": 2168354, "postDate": "2023-03-04T06:47:29.747Z", "content": "

Possibly these here: https://github.com/tensorflow/tfjs-models/blob/838611c02f51159afdd77469ce67f0e26b7bbb23/face-landmarks-detection/src/mediapipe-facemesh/keypoints.ts

\n

But you'll probably want to plot and visualize it and check, because the file above includes an extra 10 points (0-477, so 478 total) with right eye iris and left eye iris being the extra 10. So good idea to double check whether the 468 we're given really do map up to 0-467 in that file.

", "rawMarkdown": "Possibly these here: https://github.com/tensorflow/tfjs-models/blob/838611c02f51159afdd77469ce67f0e26b7bbb23/face-landmarks-detection/src/mediapipe-facemesh/keypoints.ts\n\nBut you'll probably want to plot and visualize it and check, because the file above includes an extra 10 points (0-477, so 478 total) with right eye iris and left eye iris being the extra 10. So good idea to double check whether the 468 we're given really do map up to 0-467 in that file.", "votes": 6 }, { "id": 2170708, "postDate": "2023-03-06T08:08:22.100Z", "content": "

I've been using these…

\n
    lip_marks = [61, 185, 40, 39, 37, 0, 267, 269, 270, 409, 291, 78, 191, 80, 81, 82, 13, 312, 311, 310, 415, 308, 95, 88, 178, 87, 14, 317, 402, 318, 324, 146, 91, 181, 84, 17, 314, 405, 321, 375]    \n
\n

…although I haven't checked them.

", "rawMarkdown": "I've been using these...\n\n```\n lip_marks = [61, 185, 40, 39, 37, 0, 267, 269, 270, 409, 291, 78, 191, 80, 81, 82, 13, 312, 311, 310, 415, 308, 95, 88, 178, 87, 14, 317, 402, 318, 324, 146, 91, 181, 84, 17, 314, 405, 321, 375] \n```\n\n...although I haven't checked them.", "votes": 6, "replies": [ { "id": 2170997, "postDate": "2023-03-06T12:54:21.527Z", "content": "

I've compared both lists of landmarks, @andrewrrose and @roberthatch and they are the same, just with some landmarks in different places.

", "rawMarkdown": "I've compared both lists of landmarks, @andrewrrose and @roberthatch and they are the same, just with some landmarks in different places." }, { "id": 2176086, "postDate": "2023-03-10T11:39:54.473Z", "content": "

Thanks a lot @andrewrrose

", "rawMarkdown": "Thanks a lot @andrewrrose " } ] }, { "id": 2173355, "postDate": "2023-03-08T10:37:00.093Z", "content": "

Here is a visual of all the face landmarks (it's crowded, so you'll need to zoom in a bit). There are more points on the lips, but those on the outer and inner rim are precisely the ones posted by @roberthatch and @andrewrrose.

", "rawMarkdown": "Here is a [visual](https://www.kaggle.com/code/lcdrdata/face-landmarks) of all the face landmarks (it's crowded, so you'll need to zoom in a bit). There are more points on the lips, but those on the outer and inner rim are precisely the ones posted by @roberthatch and @andrewrrose.", "votes": 2 } ] }, { "id": 2197702, "postDate": "2023-03-26T10:41:11.383Z", "content": "

In case anyone is looking on how to get points for different face landmarks:

\n
import mediapipe as mp\n\nlips = set()\n\nfor elem in mp.solutions.face_mesh_connections.FACEMESH_LIPS:\n    lips.add(elem[0])\n    lips.add(elem[1])\n
\n

You can get different face landmarks similarly.

", "rawMarkdown": "In case anyone is looking on how to get points for different face landmarks:\n```\nimport mediapipe as mp\n\nlips = set()\n\nfor elem in mp.solutions.face_mesh_connections.FACEMESH_LIPS:\n lips.add(elem[0])\n lips.add(elem[1])\n```\n\nYou can get different face landmarks similarly.", "votes": 2, "replies": [ { "id": 2197708, "postDate": "2023-03-26T10:49:27.870Z", "content": "

I would like to note here, that hands and body landmarks indexes start from 0, so you need to add offsets. Just in case someone doesn't aware of it

", "rawMarkdown": "I would like to note here, that hands and body landmarks indexes start from 0, so you need to add offsets. Just in case someone doesn't aware of it", "votes": 1, "replies": [ { "id": 2206102, "postDate": "2023-04-02T09:38:59.277Z", "content": "

Does this not give 0 indexed landmarks?

", "rawMarkdown": "Does this not give 0 indexed landmarks?" }, { "id": 2206110, "postDate": "2023-04-02T09:48:21.350Z", "content": "

It does, but indexing doesn't start from 0 for each part - face, left hand, pose, and right hand. You should add an offset per each part

", "rawMarkdown": "It does, but indexing doesn't start from 0 for each part - face, left hand, pose, and right hand. You should add an offset per each part" } ] }, { "id": 2231895, "postDate": "2023-04-23T19:32:51.930Z", "content": "

I would add this line since there are a couple of duplicated values, and we get the same list as here
\nlips = list(dict.fromkeys(lips))

", "rawMarkdown": "I would add this line since there are a couple of duplicated values, and we get the same list as [here](https://www.kaggle.com/competitions/asl-signs/discussion/391812#2170708) \n`lips = list(dict.fromkeys(lips))`" } ] }, { "id": 2168356, "postDate": "2023-03-04T06:48:36.867Z", "content": "

Finally have my own baseline, I'll try lips tomorrow and report back how it does for me! :)

", "rawMarkdown": "Finally have my own baseline, I'll try lips tomorrow and report back how it does for me! :)", "replies": [ { "id": 2168948, "postDate": "2023-03-04T16:48:23.887Z", "content": "

Adding the lips gave me a big boost of +0.03CV (!)

\n

I'll probably run various other experiments before submitting anything to LB.

", "rawMarkdown": "Adding the lips gave me a big boost of +0.03CV (!)\n\nI'll probably run various other experiments before submitting anything to LB.", "votes": 2 }, { "id": 2169298, "postDate": "2023-03-05T02:06:16.753Z", "content": "

Dropping pose also seemed to help, and all told this helped me get 0.63 LB - and notebook is now public :)

", "rawMarkdown": "Dropping pose also seemed to help, and all told this helped me get 0.63 LB - and notebook is now public :)", "votes": 3, "replies": [ { "id": 2169332, "postDate": "2023-03-05T03:14:52.853Z", "content": "

Thank you!

", "rawMarkdown": "Thank you!" } ] } ] }, { "id": 2166566, "postDate": "2023-03-02T21:08:49.313Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2176309, "postDate": "2023-03-10T14:43:20.683Z", "content": "

Thanks for sharing!

", "rawMarkdown": "Thanks for sharing!", "votes": 1 }, { "id": 2211680, "postDate": "2023-04-06T07:30:11.210Z", "content": "

Thanks! This is helpful.

", "rawMarkdown": "Thanks! This is helpful." } ], "comments": [ { "id": 2167319, "author_name": "nyakasko", "author_url": "", "post_date": "2023-03-03T12:07:58.173000", "content": "

Thanks for this! Could you please tell me which keypoints are the lips?

", "votes": 1, "replies": [ { "id": 2168354, "author_name": "Robert Hatch", "author_url": "", "post_date": "2023-03-04T06:47:29.747000", "content": "

Possibly these here: https://github.com/tensorflow/tfjs-models/blob/838611c02f51159afdd77469ce67f0e26b7bbb23/face-landmarks-detection/src/mediapipe-facemesh/keypoints.ts

\n

But you'll probably want to plot and visualize it and check, because the file above includes an extra 10 points (0-477, so 478 total) with right eye iris and left eye iris being the extra 10. So good idea to double check whether the 468 we're given really do map up to 0-467 in that file.

", "votes": 6, "replies": [] }, { "id": 2170708, "author_name": "Andrew", "author_url": "", "post_date": "2023-03-06T08:08:22.100000", "content": "

I've been using these…

\n
    lip_marks = [61, 185, 40, 39, 37, 0, 267, 269, 270, 409, 291, 78, 191, 80, 81, 82, 13, 312, 311, 310, 415, 308, 95, 88, 178, 87, 14, 317, 402, 318, 324, 146, 91, 181, 84, 17, 314, 405, 321, 375]    \n
\n

…although I haven't checked them.

", "votes": 6, "replies": [ { "id": 2170997, "author_name": "Juan Olano", "author_url": "", "post_date": "2023-03-06T12:54:21.527000", "content": "

I've compared both lists of landmarks, @andrewrrose and @roberthatch and they are the same, just with some landmarks in different places.

", "votes": 0, "replies": [] }, { "id": 2176086, "author_name": "Vikram Sandu", "author_url": "", "post_date": "2023-03-10T11:39:54.473000", "content": "

Thanks a lot @andrewrrose

", "votes": 0, "replies": [] } ] }, { "id": 2173355, "author_name": "lcdrdata", "author_url": "", "post_date": "2023-03-08T10:37:00.093000", "content": "

Here is a visual of all the face landmarks (it's crowded, so you'll need to zoom in a bit). There are more points on the lips, but those on the outer and inner rim are precisely the ones posted by @roberthatch and @andrewrrose.

", "votes": 2, "replies": [] } ] }, { "id": 2197702, "author_name": "r0", "author_url": "", "post_date": "2023-03-26T10:41:11.383000", "content": "

In case anyone is looking on how to get points for different face landmarks:

\n
import mediapipe as mp\n\nlips = set()\n\nfor elem in mp.solutions.face_mesh_connections.FACEMESH_LIPS:\n    lips.add(elem[0])\n    lips.add(elem[1])\n
\n

You can get different face landmarks similarly.

", "votes": 2, "replies": [ { "id": 2197708, "author_name": "Mykola", "author_url": "", "post_date": "2023-03-26T10:49:27.870000", "content": "

I would like to note here, that hands and body landmarks indexes start from 0, so you need to add offsets. Just in case someone doesn't aware of it

", "votes": 1, "replies": [ { "id": 2206102, "author_name": "r0", "author_url": "", "post_date": "2023-04-02T09:38:59.277000", "content": "

Does this not give 0 indexed landmarks?

", "votes": 0, "replies": [] }, { "id": 2206110, "author_name": "Mykola", "author_url": "", "post_date": "2023-04-02T09:48:21.350000", "content": "

It does, but indexing doesn't start from 0 for each part - face, left hand, pose, and right hand. You should add an offset per each part

", "votes": 0, "replies": [] } ] }, { "id": 2231895, "author_name": "Giancarlo Poémape", "author_url": "", "post_date": "2023-04-23T19:32:51.930000", "content": "

I would add this line since there are a couple of duplicated values, and we get the same list as here
\nlips = list(dict.fromkeys(lips))

", "votes": 0, "replies": [] } ] }, { "id": 2168356, "author_name": "Robert Hatch", "author_url": "", "post_date": "2023-03-04T06:48:36.867000", "content": "

Finally have my own baseline, I'll try lips tomorrow and report back how it does for me! :)

", "votes": 0, "replies": [ { "id": 2168948, "author_name": "Robert Hatch", "author_url": "", "post_date": "2023-03-04T16:48:23.887000", "content": "

Adding the lips gave me a big boost of +0.03CV (!)

\n

I'll probably run various other experiments before submitting anything to LB.

", "votes": 2, "replies": [] }, { "id": 2169298, "author_name": "Robert Hatch", "author_url": "", "post_date": "2023-03-05T02:06:16.753000", "content": "

Dropping pose also seemed to help, and all told this helped me get 0.63 LB - and notebook is now public :)

", "votes": 3, "replies": [ { "id": 2169332, "author_name": "Juan Olano", "author_url": "", "post_date": "2023-03-05T03:14:52.853000", "content": "

Thank you!

", "votes": 0, "replies": [] } ] } ] }, { "id": 2166566, "author_name": "", "author_url": "", "post_date": "2023-03-02T21:08:49.313000", "content": "", "votes": 0, "replies": [] }, { "id": 2176309, "author_name": "mLiammm", "author_url": "", "post_date": "2023-03-10T14:43:20.683000", "content": "

Thanks for sharing!

", "votes": 1, "replies": [] }, { "id": 2211680, "author_name": "Yongce Li", "author_url": "", "post_date": "2023-04-06T07:30:11.210000", "content": "

Thanks! This is helpful.

", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2166281": "**Summary**: Paying attention to lip data gives a 1% boost over a model that ignores it.\n\nMy best model, which scored 0.44 on the LB, used data from the whole body (face, pose, LH, RH). When I removed the face data, my validation and LB scores improved by 2 percentage points to 0.46 (and the model trained 10x faster). When I then added just the 40 lip points back again, my validation and LB scores increased by an additional percentage point to 0.47.\n\nOut of curiosity, I trained a lip-reading model with access to the lip positions only, using the same model architecture. It achieved a score for 0.12 (which is 30x better than guessing at random and surprisingly good since it only had mean/std data for each point). Clearly never going to win any prizes on its own, but interesting all the same.\n\nAll of which answers my [earlier question/comment about lip reading](https://www.kaggle.com/competitions/asl-signs/discussion/390509).", "2167319": "Thanks for this! Could you please tell me which keypoints are the lips?\n", "2197702": "In case anyone is looking on how to get points for different face landmarks:\n```\nimport mediapipe as mp\n\nlips = set()\n\nfor elem in mp.solutions.face_mesh_connections.FACEMESH_LIPS:\n lips.add(elem[0])\n lips.add(elem[1])\n```\n\nYou can get different face landmarks similarly.", "2168356": "Finally have my own baseline, I'll try lips tomorrow and report back how it does for me! :)", "2166566": "", "2176309": "Thanks for sharing!", "2211680": "Thanks! This is helpful." } }