A good way to conceptualize what a dataloader vs a dataset does is that a dataset prescribes how to fetch an example with a given ID and a dataloader handles doing the actual reading (reading in parallel for instance) and combining examples into batches.
\n\nPyTorch documentation is a really good place to start and the base Dataset class - which comprises only a couple of lines of code - is a really good read.
\n\nHaving said that, unfortunately none of this will be very helpful for this competition on its own. The challenge here is how to process the bounding box information and how to present it so that we can run a loss function against our model's predictions. With image recognition - telling what class the main object in an image belongs to - this is easy, we just put a softmax at the end of our model and we are done. Here this is the part that requires much more thought.
\n\nThe best material that speaks to this that I am aware of is the beginning of part 2 of the fastai course which is available on the Internet for free. All this is not all that complicated but explaining it is quite involved and I am not aware of there existing any tutorial doing a good job at this (might be beyond the scope of a single article, a lecture might work better).
\n\nFrom a practical standpoint, you might be better off just going with the object detection functionality that the google team shares here which is developed in Tensorflow. The problem is that this code is not easy to understand and is really hard to learn from if you don't already have quite a bit of understanding of what is going on. For the results in this competition you might be best off by figuring out what is going on in the repo I linked - for building up your understanding you might benefit most from doing the fastai course or studying some other library.
", "rawMarkdown": "A good way to conceptualize what a dataloader vs a dataset does is that a dataset prescribes how to fetch an example with a given ID and a dataloader handles doing the actual reading (reading in parallel for instance) and combining examples into batches.\n\nPyTorch documentation is a really good place to start and the base Dataset class - which comprises only a couple of lines of code - is a really good read.\n\nHaving said that, unfortunately none of this will be very helpful for this competition on its own. The challenge here is how to process the bounding box information and how to present it so that we can run a loss function against our model's predictions. With image recognition - telling what class the main object in an image belongs to - this is easy, we just put a softmax at the end of our model and we are done. Here this is the part that requires much more thought.\n\nThe best material that speaks to this that I am aware of is the beginning of [part 2 of the fastai course][1] which is available on the Internet for free. All this is not all that complicated but explaining it is quite involved and I am not aware of there existing any tutorial doing a good job at this (might be beyond the scope of a single article, a lecture might work better).\n\nFrom a practical standpoint, you might be better off just going with the object detection functionality that the google team shares [here][2] which is developed in Tensorflow. The problem is that this code is not easy to understand and is really hard to learn from if you don't already have quite a bit of understanding of what is going on. For the results in this competition you might be best off by figuring out what is going on in the repo I linked - for building up your understanding you might benefit most from doing the fastai course or studying some other library.\n\n\n [1]: http://course.fast.ai/part2.html\n [2]: https://github.com/tensorflow/models/tree/master/research/object_detection", "votes": null }, { "id": "358195", "postDate": "07/17/2018 17:37:51", "content": "I thought this YouTube Tutorial was very clear. \nAfter that I'd take a look at Stanford's tutorial, PyTorch's website and lastly, the PyTorch Forums have lots of examples and common confusions with great answers from the Dev Team & community. For example, here.\nHope this helps!
", "rawMarkdown": "I thought this [YouTube Tutorial][1] was very clear. \nAfter that I'd take a look at [Stanford's tutorial][2], [PyTorch's website][3] and lastly, the PyTorch Forums have lots of examples and common confusions with great answers from the Dev Team & community. For example, [here][4].\nHope this helps!\n\n\n [1]: https://www.youtube.com/watch?v=zN49HdDxHi8\n [2]: https://stanford.edu/~shervine/blog/pytorch-how-to-generate-data-parallel.html\n [3]: https://pytorch.org/tutorials/beginner/data_loading_tutorial.html\n [4]: https://discuss.pytorch.org/t/trying-to-build-a-dataloader/12618/4", "votes": null }, { "id": "358983", "postDate": "07/19/2018 09:30:45", "content": "Thank you. This cleared a lot of things
", "rawMarkdown": "Thank you. This cleared a lot of things", "votes": null }, { "id": "359549", "postDate": "07/20/2018 10:29:46", "content": "@Mario Aspiris
\n\nWe have implemented dataset/dataloaders for this competition in this open solution repo
\n\nThe datasets/loaders are here
\n\nThe retinanet model itself is training and we should have first reasonable results after the weekend.\nYou can monitor the progress here if you like.
", "rawMarkdown": "Mario Aspiris\n\nWe have implemented dataset/dataloaders for this competition in this open solution [repo](https://github.com/neptune-ml/open-solution-googleai-object-detection)\n\nThe datasets/loaders are [here](https://github.com/neptune-ml/open-solution-googleai-object-detection/blob/master/src/loaders.py)\n\nThe retinanet model itself is training and we should have first reasonable results after the weekend.\nYou can monitor the progress [here](https://app.neptune.ml/neptune-ml/Google-AI-Object-Detection-Challenge) if you like.", "votes": null } ], "comments": [ { "id": 357362, "author_name": "radek1", "author_url": "", "post_date": "07/15/2018 20:00:45", "content": "A good way to conceptualize what a dataloader vs a dataset does is that a dataset prescribes how to fetch an example with a given ID and a dataloader handles doing the actual reading (reading in parallel for instance) and combining examples into batches.
\n\nPyTorch documentation is a really good place to start and the base Dataset class - which comprises only a couple of lines of code - is a really good read.
\n\nHaving said that, unfortunately none of this will be very helpful for this competition on its own. The challenge here is how to process the bounding box information and how to present it so that we can run a loss function against our model's predictions. With image recognition - telling what class the main object in an image belongs to - this is easy, we just put a softmax at the end of our model and we are done. Here this is the part that requires much more thought.
\n\nThe best material that speaks to this that I am aware of is the beginning of part 2 of the fastai course which is available on the Internet for free. All this is not all that complicated but explaining it is quite involved and I am not aware of there existing any tutorial doing a good job at this (might be beyond the scope of a single article, a lecture might work better).
\n\nFrom a practical standpoint, you might be better off just going with the object detection functionality that the google team shares here which is developed in Tensorflow. The problem is that this code is not easy to understand and is really hard to learn from if you don't already have quite a bit of understanding of what is going on. For the results in this competition you might be best off by figuring out what is going on in the repo I linked - for building up your understanding you might benefit most from doing the fastai course or studying some other library.
", "votes": null, "replies": [] }, { "id": 358195, "author_name": "differentialmind", "author_url": "", "post_date": "07/17/2018 17:37:51", "content": "I thought this YouTube Tutorial was very clear. \nAfter that I'd take a look at Stanford's tutorial, PyTorch's website and lastly, the PyTorch Forums have lots of examples and common confusions with great answers from the Dev Team & community. For example, here.\nHope this helps!
", "votes": null, "replies": [ { "id": 358983, "author_name": "pejibaye", "author_url": "", "post_date": "07/19/2018 09:30:45", "content": "Thank you. This cleared a lot of things
", "votes": null, "replies": [] } ] }, { "id": 359549, "author_name": "jakubczakon", "author_url": "", "post_date": "07/20/2018 10:29:46", "content": "@Mario Aspiris
\n\nWe have implemented dataset/dataloaders for this competition in this open solution repo
\n\nThe datasets/loaders are here
\n\nThe retinanet model itself is training and we should have first reasonable results after the weekend.\nYou can monitor the progress here if you like.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "356957": "", "357362": "A good way to conceptualize what a dataloader vs a dataset does is that a dataset prescribes how to fetch an example with a given ID and a dataloader handles doing the actual reading (reading in parallel for instance) and combining examples into batches.\n\nPyTorch documentation is a really good place to start and the base Dataset class - which comprises only a couple of lines of code - is a really good read.\n\nHaving said that, unfortunately none of this will be very helpful for this competition on its own. The challenge here is how to process the bounding box information and how to present it so that we can run a loss function against our model's predictions. With image recognition - telling what class the main object in an image belongs to - this is easy, we just put a softmax at the end of our model and we are done. Here this is the part that requires much more thought.\n\nThe best material that speaks to this that I am aware of is the beginning of [part 2 of the fastai course][1] which is available on the Internet for free. All this is not all that complicated but explaining it is quite involved and I am not aware of there existing any tutorial doing a good job at this (might be beyond the scope of a single article, a lecture might work better).\n\nFrom a practical standpoint, you might be better off just going with the object detection functionality that the google team shares [here][2] which is developed in Tensorflow. The problem is that this code is not easy to understand and is really hard to learn from if you don't already have quite a bit of understanding of what is going on. For the results in this competition you might be best off by figuring out what is going on in the repo I linked - for building up your understanding you might benefit most from doing the fastai course or studying some other library.\n\n\n [1]: http://course.fast.ai/part2.html\n [2]: https://github.com/tensorflow/models/tree/master/research/object_detection", "358195": "I thought this [YouTube Tutorial][1] was very clear. \nAfter that I'd take a look at [Stanford's tutorial][2], [PyTorch's website][3] and lastly, the PyTorch Forums have lots of examples and common confusions with great answers from the Dev Team & community. For example, [here][4].\nHope this helps!\n\n\n [1]: https://www.youtube.com/watch?v=zN49HdDxHi8\n [2]: https://stanford.edu/~shervine/blog/pytorch-how-to-generate-data-parallel.html\n [3]: https://pytorch.org/tutorials/beginner/data_loading_tutorial.html\n [4]: https://discuss.pytorch.org/t/trying-to-build-a-dataloader/12618/4", "358983": "Thank you. This cleared a lot of things", "359549": "Mario Aspiris\n\nWe have implemented dataset/dataloaders for this competition in this open solution [repo](https://github.com/neptune-ml/open-solution-googleai-object-detection)\n\nThe datasets/loaders are [here](https://github.com/neptune-ml/open-solution-googleai-object-detection/blob/master/src/loaders.py)\n\nThe retinanet model itself is training and we should have first reasonable results after the weekend.\nYou can monitor the progress [here](https://app.neptune.ml/neptune-ml/Google-AI-Object-Detection-Challenge) if you like." }, "source": "meta" }