Hi all
\n\nI hacked together a web application so we can work on annotating the training set. You find the first version here. Before launching the effort for real, I need some feedback on a few things though:
\n\nWhat parts of the image should we annotate?\nAs a start, I configured three ROIs per image: head, steering wheel and left arm. But we need a consensus as which parts we want to annotate before starting the effort.
\n\nIn what order should the images appear for tagging?\nAt the moment, the images are given in the order they are globbed from the filesystem. But this is suboptimal. One suggestion would be to bundle images per driver and in consequence to allow only as many people as there are distinct drivers to work in parallel. When doing it this way, for each new image the previously made annotations could be suggested, which would speed up annotations for a sequence of very similar images considerably.
\n\nHow could handling be improved?\nPlease play with it a little, and report ideas on how the annotation process can be more speedy or more convenient.
\n\nBugs?\nThey are somewhere, please report any sightings.
\n\nHow should the (final) output be formatted?\nThis is maybe less important, but at the moment output is simply all fields of the database dumped into json. If someone has a good suggestion, I can make output more convenient.
\n\nSome more notes:
\n\nTo access the results, use the following two links to either get the results of one single image, or of all images annotated so far:\nhttp://tobycheese.pythonanywhere.com/annotated/img_72999\nhttp://tobycheese.pythonanywhere.com/annotated
\n\nIf you have already spent a substantial amount of work or money on annotating images and want to contribute this, we will find a way. At first I need all image ids that you have already annotated, so I can filter them out to avoid double work. Then (after we have decided on which ROIs to annotate), I can set up a second instance of the web application with just your images to annotate the missing ROIS (in case the ROIs you annotated differ from those here). Then your work can be integrated here.
\n\nThings still on the todo list:
\n\nEdit:
\n\nChanges since initial post:
\n\n@Toby -
\n\nI'm taking a guess here. Either (a) people think hand annotations will be too much work, and might not give incremental benefit over, e.g., pre-trained networks, or (b) people are annotating their images themselves to keep it as a competitive advantage.
", "rawMarkdown": "@Toby -\r\n\r\nI'm taking a guess here. Either (a) people think hand annotations will be too much work, and might not give incremental benefit over, e.g., pre-trained networks, or (b) people are annotating their images themselves to keep it as a competitive advantage.", "votes": 3 }, { "id": 116193, "postDate": "2016-04-22T13:37:24.587Z", "content": "Hi all
\n\nI hacked together a web application so we can work on annotating the training set. You find the first version here. Before launching the effort for real, I need some feedback on a few things though:
\n\nWhat parts of the image should we annotate?\nAs a start, I configured three ROIs per image: head, steering wheel and left arm. But we need a consensus as which parts we want to annotate before starting the effort.
\n\nIn what order should the images appear for tagging?\nAt the moment, the images are given in the order they are globbed from the filesystem. But this is suboptimal. One suggestion would be to bundle images per driver and in consequence to allow only as many people as there are distinct drivers to work in parallel. When doing it this way, for each new image the previously made annotations could be suggested, which would speed up annotations for a sequence of very similar images considerably.
\n\nHow could handling be improved?\nPlease play with it a little, and report ideas on how the annotation process can be more speedy or more convenient.
\n\nBugs?\nThey are somewhere, please report any sightings.
\n\nHow should the (final) output be formatted?\nThis is maybe less important, but at the moment output is simply all fields of the database dumped into json. If someone has a good suggestion, I can make output more convenient.
\n\nSome more notes:
\n\nTo access the results, use the following two links to either get the results of one single image, or of all images annotated so far:\nhttp://tobycheese.pythonanywhere.com/annotated/img_72999\nhttp://tobycheese.pythonanywhere.com/annotated
\n\nIf you have already spent a substantial amount of work or money on annotating images and want to contribute this, we will find a way. At first I need all image ids that you have already annotated, so I can filter them out to avoid double work. Then (after we have decided on which ROIs to annotate), I can set up a second instance of the web application with just your images to annotate the missing ROIS (in case the ROIs you annotated differ from those here). Then your work can be integrated here.
\n\nThings still on the todo list:
\n\nEdit:
\n\nChanges since initial post:
\n\nOK, thank to all who replied.
\n\nI'll dig this back out once the next computer vision competition is launched where there are things to detect that are not already handled by a pretrained net (such as whales :-) )
", "rawMarkdown": "OK, thank to all who replied.\r\n\r\nI'll dig this back out once the next computer vision competition is launched where there are things to detect that are not already handled by a pretrained net (such as whales :-) )", "votes": 1 }, { "id": 117259, "postDate": "2016-04-28T08:04:42.217Z", "content": "@Toby\nProbably a little bit of both. There is no clear benefits (it is relatively easy to get face and hands using simple image processing - but even that may be not needed - or even make things worse - for the CNN) and installing things could be a headache.
", "rawMarkdown": "@Toby\r\nProbably a little bit of both. There is no clear benefits (it is relatively easy to get face and hands using simple image processing - but even that may be not needed - or even make things worse - for the CNN) and installing things could be a headache. ", "votes": 1 }, { "id": 117145, "postDate": "2016-04-27T15:17:05.687Z", "content": "Right, so now I even have to reply to my own posts, since nobody else does :-)
\n\nSeriously though, I am a bit surprised that my attempt to start and guide a community effort attracted almost zero reaction so far, and I am wondering why that is or what I did wrong in the process.
\n\nIs it simply because nobody needs an annotated training set in this competition?
\n\nOr is the part where one has to install an extension for chrome too complicated, or too suspicious? This seems possible, because if the demo I put up was tried by anyone at all, no one has actually saved an annotated image so far. But then again, my personal opinion is that serving the image files from one’s local machine is actually a clever move (if I may say so). The images load and display instantly and there is no wasted bandwidth for downloading files that everyone already has downloaded.
\n\nSo, I am grateful for all feedback on why there is no feedback :-)
", "rawMarkdown": "Right, so now I even have to reply to my own posts, since nobody else does :-)\r\n\r\nSeriously though, I am a bit surprised that my attempt to start and guide a community effort attracted almost zero reaction so far, and I am wondering why that is or what I did wrong in the process.\r\n\r\nIs it simply because nobody needs an annotated training set in this competition?\r\n\r\nOr is the part where one has to install an extension for chrome too complicated, or too suspicious? This seems possible, because if the demo I put up was tried by anyone at all, no one has actually saved an annotated image so far. But then again, my personal opinion is that serving the image files from one’s local machine is actually a clever move (if I may say so). The images load and display instantly and there is no wasted bandwidth for downloading files that everyone already has downloaded.\r\n\r\nSo, I am grateful for all feedback on why there is no feedback :-)\r\n" }, { "id": 117301, "postDate": "2016-04-28T12:23:43.727Z", "rawMarkdown": "", "votes": 1, "isDeleted": true } ], "comments": [ { "id": 117150, "author_name": "inversion", "author_url": "", "post_date": "2016-04-27T15:46:27.187000", "content": "@Toby -
\n\nI'm taking a guess here. Either (a) people think hand annotations will be too much work, and might not give incremental benefit over, e.g., pre-trained networks, or (b) people are annotating their images themselves to keep it as a competitive advantage.
", "votes": 3, "replies": [] }, { "id": 127296, "author_name": "Toby Cheese", "author_url": "", "post_date": "2016-07-13T20:08:59.030000", "content": "OK, thank to all who replied.
\n\nI'll dig this back out once the next computer vision competition is launched where there are things to detect that are not already handled by a pretrained net (such as whales :-) )
", "votes": 1, "replies": [] }, { "id": 117259, "author_name": "==>(AL)<==", "author_url": "", "post_date": "2016-04-28T08:04:42.217000", "content": "@Toby\nProbably a little bit of both. There is no clear benefits (it is relatively easy to get face and hands using simple image processing - but even that may be not needed - or even make things worse - for the CNN) and installing things could be a headache.
", "votes": 1, "replies": [] }, { "id": 117145, "author_name": "Toby Cheese", "author_url": "", "post_date": "2016-04-27T15:17:05.687000", "content": "Right, so now I even have to reply to my own posts, since nobody else does :-)
\n\nSeriously though, I am a bit surprised that my attempt to start and guide a community effort attracted almost zero reaction so far, and I am wondering why that is or what I did wrong in the process.
\n\nIs it simply because nobody needs an annotated training set in this competition?
\n\nOr is the part where one has to install an extension for chrome too complicated, or too suspicious? This seems possible, because if the demo I put up was tried by anyone at all, no one has actually saved an annotated image so far. But then again, my personal opinion is that serving the image files from one’s local machine is actually a clever move (if I may say so). The images load and display instantly and there is no wasted bandwidth for downloading files that everyone already has downloaded.
\n\nSo, I am grateful for all feedback on why there is no feedback :-)
", "votes": 0, "replies": [] }, { "id": 117301, "author_name": "", "author_url": "", "post_date": "2016-04-28T12:23:43.727000", "content": "", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "117150": "@Toby -\r\n\r\nI'm taking a guess here. Either (a) people think hand annotations will be too much work, and might not give incremental benefit over, e.g., pre-trained networks, or (b) people are annotating their images themselves to keep it as a competitive advantage.", "116193": "Hi all\r\n\r\nI hacked together a web application so we can work on annotating the training set. You find the first version [here][1]. Before launching the effort for real, I need some feedback on a few things though:\r\n\r\n\r\n**What parts of the image should we annotate**?\r\nAs a start, I configured three ROIs per image: head, steering wheel and left arm. But we need a consensus as which parts we want to annotate before starting the effort.\r\n\r\n\r\n**In what order should the images appear for tagging?**\r\nAt the moment, the images are given in the order they are globbed from the filesystem. But this is suboptimal. One suggestion would be to bundle images per driver and in consequence to allow only as many people as there are distinct drivers to work in parallel. When doing it this way, for each new image the previously made annotations could be suggested, which would speed up annotations for a sequence of very similar images considerably. \r\n\r\n**How could handling be improved?**\r\nPlease play with it a little, and report ideas on how the annotation process can be more speedy or more convenient.\r\n\r\n**Bugs?**\r\nThey are somewhere, please report any sightings.\r\n\r\n**How should the (final) output be formatted?**\r\nThis is maybe less important, but at the moment output is simply all fields of the database dumped into json. If someone has a good suggestion, I can make output more convenient.\r\n\r\n\r\n**Some more notes:**\r\n\r\nTo access the results, use the following two links to either get the results of one single image, or of all images annotated so far:\r\n[http://tobycheese.pythonanywhere.com/annotated/img_72999][2]\r\n[http://tobycheese.pythonanywhere.com/annotated][3]\r\n\r\nIf you have already spent a substantial amount of work or money on annotating images and want to contribute this, we will find a way. At first I need all image ids that you have already annotated, so I can filter them out to avoid double work. Then (after we have decided on which ROIs to annotate), I can set up a second instance of the web application with just your images to annotate the missing ROIS (in case the ROIs you annotated differ from those here). Then your work can be integrated here.\r\n\r\n**Things still on the todo list:**\r\n\r\n - Get answers to above questions and implement the consequences\r\n - Add keyboard shortcuts for 'Next' and 'Reset Annotations'\r\n - Add a cleanup process to remove locked files\r\n - Make it SQL injection safe\r\n - Maybe make it look a bit nicer, it really is just a quick hack\r\n - Polish code and put it on github\r\n\r\n\r\n----------\r\nEdit:\r\n\r\n**Changes since initial post:**\r\n\r\n - Reset button actually does something now\r\n - Added a reference to libraries used\r\n\r\n [1]: http://tobycheese.pythonanywhere.com/\r\n [2]: http://tobycheese.pythonanywhere.com/annotated/img_72999\r\n [3]: http://tobycheese.pythonanywhere.com/annotated/", "127296": "OK, thank to all who replied.\r\n\r\nI'll dig this back out once the next computer vision competition is launched where there are things to detect that are not already handled by a pretrained net (such as whales :-) )", "117259": "@Toby\r\nProbably a little bit of both. There is no clear benefits (it is relatively easy to get face and hands using simple image processing - but even that may be not needed - or even make things worse - for the CNN) and installing things could be a headache. ", "117145": "Right, so now I even have to reply to my own posts, since nobody else does :-)\r\n\r\nSeriously though, I am a bit surprised that my attempt to start and guide a community effort attracted almost zero reaction so far, and I am wondering why that is or what I did wrong in the process.\r\n\r\nIs it simply because nobody needs an annotated training set in this competition?\r\n\r\nOr is the part where one has to install an extension for chrome too complicated, or too suspicious? This seems possible, because if the demo I put up was tried by anyone at all, no one has actually saved an annotated image so far. But then again, my personal opinion is that serving the image files from one’s local machine is actually a clever move (if I may say so). The images load and display instantly and there is no wasted bandwidth for downloading files that everyone already has downloaded.\r\n\r\nSo, I am grateful for all feedback on why there is no feedback :-)\r\n", "117301": "" } }