I've taken the habit to try to reflect on competitions once they're done and see what I've learned from them. Not only on the technical aspect (learned a lot of new cool tricks, augmentations, models, etc.) but also on how to actually handle such a competition. There is a lot of discussion out there about why model X is better than model Y, but I found that there was pretty few discussion on how to start, organize and get through a multiple months long project like this. I like to learn on how to be more efficient, and the end of a competition is IMHO a great exercise for that.
\n\nFirst of all, I want to give a huge thanks to the members of my team @rsmits @acmilannesta & @nerin0707 for the awesome work we managed to pull off, and getting what is my first medal and a silver one that is!
\n\nNow, these guys had some really shinny hardware to throw some models at, but I don't, and that's also one of the reasons why I love computer vision tasks to much: they require thinking about optimizations, tricks, ways to get around long training times, and I find that just as interesting as trying to find the best model whatever the training times.
\n\nI'll be referring quite a bit to the previous post like this that I did on the Understanding Clouds competition, where I was giving my middle-rank guys' perspective on the competition. In that competition, I got really frustrated that I did not have time to develop a pipeline quickly enough to try all the ideas that I wanted, I spend most of the time setting up a pipeline.
\n\nFirst take-away, which I applied here and that helped a lot: iterate fast as early as possible. The first thing I did was to create a new dataset with cropped & resized images to only keep the \"useful\" part of the image. The idea here is to get rid of as much data as possible, while keeping as much of the useful data as possible. Reducing image size is a great way to do so. A 137 x 236 has 32,332 pixels in it. By cropping to the character and resizing the image to a 100x100 we get images of 10,000 pixels, which 3.23 times less. We barely loose any information, if any at all, and we can train a model more than 3 times faster. Along side this, I start by taking a stratified subset of the dataset and only train on that. The idea isn't to get a good score, but to get a score at all. Again, iterating fast is the name of the game here. Being able to create a pipeline that outputs a result is a really important first step. And for that, each run has to take seconds, not hours. If it takes seconds, you're never leaving your keyboard and in an hour you can go through 10, 15, 20 iterations and start getting a working pipeline. This is especially important when you don't have access to hardware, those 30h of GPU time are very precious. I always have a TESTING_PIPELINE = True flag at the top of all my notebook. If set to True this only uses a few hundred images to run the whole pipeline. That way, I can change my model, add augmentations, you name it, and check in a minute if it doesn't break anything. Ideally you'd have unit tests for that, but I haven't (yet) figured a way to efficiently do that in a notebook. Fast iteration is by far the biggest win for me in this competition.
Second topic that I spend a lot of time refining is how to use limited GPU time as efficiently as possible. I ended up upgrading to Colab Pro about 2 weeks before the end of the competition and it opened up the possibilities for me. But it also has a lot of drawbacks (session has to be left open, and everything is still in 1 notebook) that require some thinking. As much as I could, I tried to move pre (and post) processing outside of GPU times. That flag for testing was also very useful in making sure that I wouldn't end up with a notebook that didn't output whenever I tried something new. While a model is training, I also got the weights from a previous model and started doing error anlaysis on my laptop. I do have a dedicated GPU on my laptop, bu it's a 2Gb mx150, which is basically the bottom of the barrel. But still, it might not be good for training, but it's great for inference! I feel like error analysis is pretty overlooked, or at least not very discussed in Kaggle competitions yet there is a lot to learn from what types of errors our models make. For example, I dug deeper into class activation maps (to the point of @rsmits nick naming me after them) and how they might provide useful info on what our models were doing. I didn't get it to where I wanted to, but had I had great hardware, I would have never taken the time to look into it in the first place. I think this is where people with limited hardware should think: A lot of people are really focused on \"blindly\" trying new stuff, throwing it at their GPU and waiting to see what will happen: will recall go up, or down? I think taking the time to investigate how it makes our metric go up or down is just as important. Ideally, I would have loved to have time to find the least corrolated models to create even better ensembles. But like everyone, we had a deadline for this competition.
\n\nThis leads me to something else that I took from this competition, and that many really good Kagglers keep repeating: having a solid model, capable of holding the shake-up. I have to really thing the guys in my team for all the heavy lifting they did on model tuning and training on that side. But we had quite a bit of talk on who=ich models to take and how to ensemble them. This is IMHO ultimately what made us jump from below bronze in public to silver on private. And again, this can be done without much hardware once the models are trained. That's the cool thing about these computer vision tasks! There's so much more than just raw model training.
\n\nI also want to touch upon the fact that teaming up with different people for this competition was great, and I highly recommend it to anybody getting started in their competition as much as possible. I feel like I got to ask a bunch of the questions I had, but also re-evaluate things I thought I understood when came time to explain them to someone on the team. Those are part of the exercise.
\n\nFinally, I'd like to talk about all the info that can be found on the forum. There is so much info that has been posted in this competition, it was really interesting! Being able to follow all of this was a great experience even without being able to replicate all of it once again due to hardware limitations. But just trying to sum it up, and push the useful parts forward to the guys on my team that then could focus on training the model was a way to contribute.
\n\nIn short, I hope this will be useful to people that like me don't have access to super high end hardware but want to give a try at computer vision. I'm not even the best person to talk about this, as @aerdem4 managed to solo the 21st place only using Kaggle kernels. But this also for people that aren grand masters, like me, and that might want to know that it's still possible to reach a medal when teaming up and contributing in a way that brings good ideas to the table. I really really like Kaggling with limited hardware, I do believe limitations lead to creativity and to having to think and rethink of ways things are done.
\n\nOnce again, thanks to the team for getting me in even though I didn't have a 2080Ti!
", "messages": [ { "id": "778912", "postDate": "03/18/2020 21:08:22", "content": "I've taken the habit to try to reflect on competitions once they're done and see what I've learned from them. Not only on the technical aspect (learned a lot of new cool tricks, augmentations, models, etc.) but also on how to actually handle such a competition. There is a lot of discussion out there about why model X is better than model Y, but I found that there was pretty few discussion on how to start, organize and get through a multiple months long project like this. I like to learn on how to be more efficient, and the end of a competition is IMHO a great exercise for that.
\n\nFirst of all, I want to give a huge thanks to the members of my team @rsmits @acmilannesta & @nerin0707 for the awesome work we managed to pull off, and getting what is my first medal and a silver one that is!
\n\nNow, these guys had some really shinny hardware to throw some models at, but I don't, and that's also one of the reasons why I love computer vision tasks to much: they require thinking about optimizations, tricks, ways to get around long training times, and I find that just as interesting as trying to find the best model whatever the training times.
\n\nI'll be referring quite a bit to the previous post like this that I did on the Understanding Clouds competition, where I was giving my middle-rank guys' perspective on the competition. In that competition, I got really frustrated that I did not have time to develop a pipeline quickly enough to try all the ideas that I wanted, I spend most of the time setting up a pipeline.
\n\nFirst take-away, which I applied here and that helped a lot: iterate fast as early as possible. The first thing I did was to create a new dataset with cropped & resized images to only keep the \"useful\" part of the image. The idea here is to get rid of as much data as possible, while keeping as much of the useful data as possible. Reducing image size is a great way to do so. A 137 x 236 has 32,332 pixels in it. By cropping to the character and resizing the image to a 100x100 we get images of 10,000 pixels, which 3.23 times less. We barely loose any information, if any at all, and we can train a model more than 3 times faster. Along side this, I start by taking a stratified subset of the dataset and only train on that. The idea isn't to get a good score, but to get a score at all. Again, iterating fast is the name of the game here. Being able to create a pipeline that outputs a result is a really important first step. And for that, each run has to take seconds, not hours. If it takes seconds, you're never leaving your keyboard and in an hour you can go through 10, 15, 20 iterations and start getting a working pipeline. This is especially important when you don't have access to hardware, those 30h of GPU time are very precious. I always have a TESTING_PIPELINE = True flag at the top of all my notebook. If set to True this only uses a few hundred images to run the whole pipeline. That way, I can change my model, add augmentations, you name it, and check in a minute if it doesn't break anything. Ideally you'd have unit tests for that, but I haven't (yet) figured a way to efficiently do that in a notebook. Fast iteration is by far the biggest win for me in this competition.
Second topic that I spend a lot of time refining is how to use limited GPU time as efficiently as possible. I ended up upgrading to Colab Pro about 2 weeks before the end of the competition and it opened up the possibilities for me. But it also has a lot of drawbacks (session has to be left open, and everything is still in 1 notebook) that require some thinking. As much as I could, I tried to move pre (and post) processing outside of GPU times. That flag for testing was also very useful in making sure that I wouldn't end up with a notebook that didn't output whenever I tried something new. While a model is training, I also got the weights from a previous model and started doing error anlaysis on my laptop. I do have a dedicated GPU on my laptop, bu it's a 2Gb mx150, which is basically the bottom of the barrel. But still, it might not be good for training, but it's great for inference! I feel like error analysis is pretty overlooked, or at least not very discussed in Kaggle competitions yet there is a lot to learn from what types of errors our models make. For example, I dug deeper into class activation maps (to the point of @rsmits nick naming me after them) and how they might provide useful info on what our models were doing. I didn't get it to where I wanted to, but had I had great hardware, I would have never taken the time to look into it in the first place. I think this is where people with limited hardware should think: A lot of people are really focused on \"blindly\" trying new stuff, throwing it at their GPU and waiting to see what will happen: will recall go up, or down? I think taking the time to investigate how it makes our metric go up or down is just as important. Ideally, I would have loved to have time to find the least corrolated models to create even better ensembles. But like everyone, we had a deadline for this competition.
\n\nThis leads me to something else that I took from this competition, and that many really good Kagglers keep repeating: having a solid model, capable of holding the shake-up. I have to really thing the guys in my team for all the heavy lifting they did on model tuning and training on that side. But we had quite a bit of talk on who=ich models to take and how to ensemble them. This is IMHO ultimately what made us jump from below bronze in public to silver on private. And again, this can be done without much hardware once the models are trained. That's the cool thing about these computer vision tasks! There's so much more than just raw model training.
\n\nI also want to touch upon the fact that teaming up with different people for this competition was great, and I highly recommend it to anybody getting started in their competition as much as possible. I feel like I got to ask a bunch of the questions I had, but also re-evaluate things I thought I understood when came time to explain them to someone on the team. Those are part of the exercise.
\n\nFinally, I'd like to talk about all the info that can be found on the forum. There is so much info that has been posted in this competition, it was really interesting! Being able to follow all of this was a great experience even without being able to replicate all of it once again due to hardware limitations. But just trying to sum it up, and push the useful parts forward to the guys on my team that then could focus on training the model was a way to contribute.
\n\nIn short, I hope this will be useful to people that like me don't have access to super high end hardware but want to give a try at computer vision. I'm not even the best person to talk about this, as @aerdem4 managed to solo the 21st place only using Kaggle kernels. But this also for people that aren grand masters, like me, and that might want to know that it's still possible to reach a medal when teaming up and contributing in a way that brings good ideas to the table. I really really like Kaggling with limited hardware, I do believe limitations lead to creativity and to having to think and rethink of ways things are done.
\n\nOnce again, thanks to the team for getting me in even though I didn't have a 2080Ti!
", "rawMarkdown": "I've taken the habit to try to reflect on competitions once they're done and see what I've learned from them. Not only on the technical aspect (learned a lot of new cool tricks, augmentations, models, etc.) but also on how to actually handle such a competition. There is a lot of discussion out there about why model X is better than model Y, but I found that there was pretty few discussion on how to start, organize and get through a multiple months long project like this. I like to learn on how to be more efficient, and the end of a competition is IMHO a great exercise for that.\n\nFirst of all, I want to give a huge thanks to the members of my team @rsmits @acmilannesta & @nerin0707 for the awesome work we managed to pull off, and getting what is my first medal and a silver one that is!\n\nNow, these guys had some really shinny hardware to throw some models at, but I don't, and that's also one of the reasons why I love computer vision tasks to much: they require thinking about optimizations, tricks, ways to get around long training times, and I find that just as interesting as trying to find the best model whatever the training times.\n\n## Lessons applied from previous experience\n\nI'll be referring quite a bit to the previous post like this that I did on the Understanding Clouds competition, where I was giving my [middle-rank guys' perspective on the competition](https://www.kaggle.com/c/understanding_cloud_organization/discussion/118328). In that competition, I got really frustrated that I did not have time to develop a pipeline quickly enough to try all the ideas that I wanted, I spend most of the time setting up a pipeline.\n\nFirst take-away, which I applied here and that helped a lot: **iterate fast as early as possible**. The first thing I did was to create a new dataset with [cropped & resized images](https://www.kaggle.com/maxlenormand/cropping-to-character-resizing-images) to only keep the \"useful\" part of the image. The idea here is to get rid of as much data as possible, while keeping as much of the *useful* data as possible. Reducing image size is a great way to do so. A 137 x 236 has 32,332 pixels in it. By cropping to the character and resizing the image to a 100x100 we get images of 10,000 pixels, which 3.23 times less. We barely loose any information, if any at all, and we can train a model more than 3 times faster. Along side this, I start by taking a stratified subset of the dataset and only train on that. The idea isn't to get a good score, but to get a score at all. Again, iterating fast is the name of the game here. Being able to create a pipeline that outputs a result is a really important first step. And for that, each run has to take seconds, not hours. If it takes seconds, you're never leaving your keyboard and in an hour you can go through 10, 15, 20 iterations and start getting a working pipeline. This is especially important when you don't have access to hardware, those 30h of GPU time are very precious. I always have a `TESTING_PIPELINE = True` flag at the top of all my notebook. If set to `True` this only uses a few hundred images to run the whole pipeline. That way, I can change my model, add augmentations, you name it, and check in a minute if it doesn't break anything. Ideally you'd have unit tests for that, but I haven't (yet) figured a way to efficiently do that in a notebook. Fast iteration is by far the biggest win for me in this competition. \n\nSecond topic that I spend a lot of time refining is how to use limited GPU time as efficiently as possible. I ended up upgrading to Colab Pro about 2 weeks before the end of the competition and it opened up the possibilities for me. But it also has a lot of drawbacks (session has to be left open, and everything is still in 1 notebook) that require some thinking. As much as I could, I tried to move pre (and post) processing outside of GPU times. That flag for testing was also very useful in making sure that I wouldn't end up with a notebook that didn't output whenever I tried something new. While a model is training, I also got the weights from a previous model and started doing error anlaysis on my laptop. I do have a dedicated GPU on my laptop, bu it's a 2Gb mx150, which is basically the bottom of the barrel. But still, it might not be good for training, but it's great for inference! I feel like error analysis is pretty overlooked, or at least not very discussed in Kaggle competitions yet there is a lot to learn from what types of errors our models make. For example, I dug deeper into class activation maps (to the point of @rsmits nick naming me after them) and how they might provide useful info on what our models were doing. I didn't get it to where I wanted to, but had I had great hardware, I would have never taken the time to look into it in the first place. I think this is where people with limited hardware should think: A lot of people are really focused on \"blindly\" trying new stuff, throwing it at their GPU and waiting to see what will happen: will recall go up, or down? I think taking the time to investigate *how* it makes our metric go up or down is just as important. Ideally, I would have loved to have time to find the least corrolated models to create even better ensembles. But like everyone, we had a deadline for this competition.\n\nThis leads me to something else that I took from this competition, and that many really good Kagglers keep repeating: having a solid model, capable of holding the shake-up. I have to really thing the guys in my team for all the heavy lifting they did on model tuning and training on that side. But we had quite a bit of talk on who=ich models to take and how to ensemble them. This is IMHO ultimately what made us jump from below bronze in public to silver on private. And again, this can be done without much hardware once the models are trained. That's the cool thing about these computer vision tasks! There's so much more than just raw model training.\n\n## Take-away from this competition\n\nI also want to touch upon the fact that teaming up with different people for this competition was great, and I highly recommend it to anybody getting started in their competition as much as possible. I feel like I got to ask a bunch of the questions I had, but also re-evaluate things I thought I understood when came time to explain them to someone on the team. Those are part of the exercise.\n\nFinally, I'd like to talk about all the info that can be found on the forum. There is **so** much info that has been posted in this competition, it was really interesting! Being able to follow all of this was a great experience even without being able to replicate all of it once again due to hardware limitations. But just trying to sum it up, and push the useful parts forward to the guys on my team that then could focus on training the model was a way to contribute.\n\nIn short, I hope this will be useful to people that like me don't have access to super high end hardware but want to give a try at computer vision. I'm not even the best person to talk about this, as @aerdem4 managed to solo the 21st place [only using Kaggle kernels](https://www.kaggle.com/c/bengaliai-cv19/discussion/136056). But this also for people that aren grand masters, like me, and that might want to know that it's still possible to reach a medal when teaming up and contributing in a way that brings good ideas to the table. I really really like Kaggling with limited hardware, I do believe limitations lead to creativity and to having to think and rethink of ways things are done. \n\nOnce again, thanks to the team for getting me in even though I didn't have a 2080Ti!", "votes": null }, { "id": "779198", "postDate": "03/19/2020 05:09:46", "content": "Words of wisdom! also, congratulations on your result.
", "rawMarkdown": "Words of wisdom! also, congratulations on your result.", "votes": null }, { "id": "779244", "postDate": "03/19/2020 06:15:39", "content": "Thanks! :)
", "rawMarkdown": "Thanks! :)", "votes": null }, { "id": "779344", "postDate": "03/19/2020 08:37:47", "content": "Some great advice there! And congrats on your silver.\nI created a very small toy dataset by doing a stratified split on the original data and tested my models on that.\nSadly, things didn't replicate for the whole data and eventually I could never submit because dealing with free tier colab is a very tedious task. Although, I did read a lot of papers and tried to implement everything from scratch myself which was fun in itself. \nReally looking forward to the next CV competiion now.
", "rawMarkdown": "Some great advice there! And congrats on your silver.\nI created a very small toy dataset by doing a stratified split on the original data and tested my models on that.\nSadly, things didn't replicate for the whole data and eventually I could never submit because dealing with free tier colab is a very tedious task. Although, I did read a lot of papers and tried to implement everything from scratch myself which was fun in itself. \nReally looking forward to the next CV competiion now.", "votes": null }, { "id": "779364", "postDate": "03/19/2020 09:08:33", "content": "I think especially in this competition, where there was a bit of work required to actually be able to make a submission that worked in the first place, using stratified subsets was a great way to get started.
", "rawMarkdown": "I think especially in this competition, where there was a bit of work required to actually be able to make a submission that worked in the first place, using stratified subsets was a great way to get started.", "votes": null }, { "id": "779838", "postDate": "03/19/2020 18:24:52", "content": "Thanks - very useful!
", "rawMarkdown": "Thanks - very useful!", "votes": null }, { "id": "779906", "postDate": "03/19/2020 19:48:59", "content": "Feel free to add if you have any additional points!
", "rawMarkdown": "Feel free to add if you have any additional points!", "votes": null }, { "id": "780923", "postDate": "03/20/2020 18:30:14", "content": "Thanks for your detailed sharing. Fast iteration and training with a small training set are really good suggestions. I wanted to set a small train set but had no clue how to do it. May I ask how do you choose samples for this small training set without losing generalization ability?
", "rawMarkdown": "Thanks for your detailed sharing. Fast iteration and training with a small training set are really good suggestions. I wanted to set a small train set but had no clue how to do it. May I ask how do you choose samples for this small training set without losing generalization ability?", "votes": null }, { "id": "780996", "postDate": "03/20/2020 19:54:38", "content": "Sure!
\n\nI usually do this in multiple steps. I started by making a single class classifier (only trying to predict consonants, as this was the symbol with the least amount of class, thus requiring less data to get a few images per class). For this I simply use train_test_split from sklearn with the stratify option set. This is very helpful to get the data shapes / formats working at first and get a feeling of how to handle the data from the competition.
Then, in this particular competition I discovered this very useful library getting this stratified split for multiple labels.
\n\nTo come back to the point on having a strong cross-validation, you can then create K-Folds (also stratified) which can also be done with the iterative-stratification mentioned.
Hope that helps!
", "rawMarkdown": "Sure!\n\nI usually do this in multiple steps. I started by making a single class classifier (only trying to predict consonants, as this was the symbol with the least amount of class, thus requiring less data to get a few images per class). For this I simply use `train_test_split` from `sklearn` with the [stratify option set](https://stackoverflow.com/questions/29438265/stratified-train-test-split-in-scikit-learn). This is very helpful to get the data shapes / formats working at first and get a feeling of how to handle the data from the competition.\n\nThen, in this particular competition I discovered [this very useful library](https://github.com/trent-b/iterative-stratification) getting this stratified split for multiple labels.\n\nTo come back to the point on having a strong cross-validation, you can then create K-Folds (also stratified) which can also be done with the `iterative-stratification` mentioned.\n\nHope that helps!", "votes": null }, { "id": "781029", "postDate": "03/20/2020 20:56:04", "content": "Thanks for your detailed reply. That helps a lot! So just for clarification, is the stratified subset of the dataset one of your K-folds created by iterative-stratification lib? If so, what's your k value? I used k equal to five, which means 80% dataset data used for training, still too heavy for training. I used 112*112*3 input, each epoch took around 30 mins. It was too slow.
", "rawMarkdown": "Thanks for your detailed reply. That helps a lot! So just for clarification, is the stratified subset of the dataset one of your K-folds created by iterative-stratification lib? If so, what's your k value? I used k equal to five, which means 80% dataset data used for training, still too heavy for training. I used 112\\*112\\*3 input, each epoch took around 30 mins. It was too slow.", "votes": null }, { "id": "781389", "postDate": "03/21/2020 08:13:25", "content": "That's a good question!
\n\nChoosing a good k value is tricky, but I also usually go for 5. Though first I tinker around with a 80 / 20 stratified split, which allows me to only train the model once. 5-fold means you'll train 5 times with 80% of the data, which takes, well 5 times longer. So I experiment with a 80 / 20 first and if something interesting comes up, I try it on the 5-Fold. Again the name of the game for me is smart subsetting to speed up training. Especially with limited hardware access!
", "rawMarkdown": "That's a good question!\n\nChoosing a good k value is tricky, but I also usually go for 5. Though first I tinker around with a 80 / 20 stratified split, which allows me to only train the model once. 5-fold means you'll train 5 times with 80% of the data, which takes, well 5 times longer. So I experiment with a 80 / 20 first and if something interesting comes up, I try it on the 5-Fold. Again the name of the game for me is smart subsetting to speed up training. Especially with limited hardware access!", "votes": null }, { "id": "781881", "postDate": "03/21/2020 18:08:35", "content": "Thanks a lot! That's really helpful! I'd like to try this in my next competition. 👍
", "rawMarkdown": "Thanks a lot! That's really helpful! I'd like to try this in my next competition. 👍", "votes": null }, { "id": "1177304", "postDate": "01/30/2021 07:44:42", "content": "Thank you for sharing your experience @maxlenormand. I am new to kaggle competitions and working on images. Your post was very helpful and informative.
", "rawMarkdown": "Thank you for sharing your experience @maxlenormand. I am new to kaggle competitions and working on images. Your post was very helpful and informative.", "votes": null } ], "comments": [ { "id": 1177304, "author_name": "mayank1101sharma", "author_url": "", "post_date": "01/30/2021 07:44:42", "content": "Thank you for sharing your experience @maxlenormand. I am new to kaggle competitions and working on images. Your post was very helpful and informative.
", "votes": null, "replies": [] }, { "id": 779198, "author_name": "datamafia7", "author_url": "", "post_date": "03/19/2020 05:09:46", "content": "Words of wisdom! also, congratulations on your result.
", "votes": null, "replies": [ { "id": 779244, "author_name": "maxlenormand", "author_url": "", "post_date": "03/19/2020 06:15:39", "content": "Thanks! :)
", "votes": null, "replies": [] } ] }, { "id": 779344, "author_name": "timetraveller98", "author_url": "", "post_date": "03/19/2020 08:37:47", "content": "Some great advice there! And congrats on your silver.\nI created a very small toy dataset by doing a stratified split on the original data and tested my models on that.\nSadly, things didn't replicate for the whole data and eventually I could never submit because dealing with free tier colab is a very tedious task. Although, I did read a lot of papers and tried to implement everything from scratch myself which was fun in itself. \nReally looking forward to the next CV competiion now.
", "votes": null, "replies": [ { "id": 779364, "author_name": "maxlenormand", "author_url": "", "post_date": "03/19/2020 09:08:33", "content": "I think especially in this competition, where there was a bit of work required to actually be able to make a submission that worked in the first place, using stratified subsets was a great way to get started.
", "votes": null, "replies": [] } ] }, { "id": 779838, "author_name": "gpamoukoff", "author_url": "", "post_date": "03/19/2020 18:24:52", "content": "Thanks - very useful!
", "votes": null, "replies": [ { "id": 779906, "author_name": "maxlenormand", "author_url": "", "post_date": "03/19/2020 19:48:59", "content": "Feel free to add if you have any additional points!
", "votes": null, "replies": [] } ] }, { "id": 780923, "author_name": "yuanlin08", "author_url": "", "post_date": "03/20/2020 18:30:14", "content": "Thanks for your detailed sharing. Fast iteration and training with a small training set are really good suggestions. I wanted to set a small train set but had no clue how to do it. May I ask how do you choose samples for this small training set without losing generalization ability?
", "votes": null, "replies": [ { "id": 780996, "author_name": "maxlenormand", "author_url": "", "post_date": "03/20/2020 19:54:38", "content": "Sure!
\n\nI usually do this in multiple steps. I started by making a single class classifier (only trying to predict consonants, as this was the symbol with the least amount of class, thus requiring less data to get a few images per class). For this I simply use train_test_split from sklearn with the stratify option set. This is very helpful to get the data shapes / formats working at first and get a feeling of how to handle the data from the competition.
Then, in this particular competition I discovered this very useful library getting this stratified split for multiple labels.
\n\nTo come back to the point on having a strong cross-validation, you can then create K-Folds (also stratified) which can also be done with the iterative-stratification mentioned.
Hope that helps!
", "votes": null, "replies": [] }, { "id": 781029, "author_name": "yuanlin08", "author_url": "", "post_date": "03/20/2020 20:56:04", "content": "Thanks for your detailed reply. That helps a lot! So just for clarification, is the stratified subset of the dataset one of your K-folds created by iterative-stratification lib? If so, what's your k value? I used k equal to five, which means 80% dataset data used for training, still too heavy for training. I used 112*112*3 input, each epoch took around 30 mins. It was too slow.
", "votes": null, "replies": [] }, { "id": 781389, "author_name": "maxlenormand", "author_url": "", "post_date": "03/21/2020 08:13:25", "content": "That's a good question!
\n\nChoosing a good k value is tricky, but I also usually go for 5. Though first I tinker around with a 80 / 20 stratified split, which allows me to only train the model once. 5-fold means you'll train 5 times with 80% of the data, which takes, well 5 times longer. So I experiment with a 80 / 20 first and if something interesting comes up, I try it on the 5-Fold. Again the name of the game for me is smart subsetting to speed up training. Especially with limited hardware access!
", "votes": null, "replies": [] }, { "id": 781881, "author_name": "yuanlin08", "author_url": "", "post_date": "03/21/2020 18:08:35", "content": "Thanks a lot! That's really helpful! I'd like to try this in my next competition. 👍
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "778912": "I've taken the habit to try to reflect on competitions once they're done and see what I've learned from them. Not only on the technical aspect (learned a lot of new cool tricks, augmentations, models, etc.) but also on how to actually handle such a competition. There is a lot of discussion out there about why model X is better than model Y, but I found that there was pretty few discussion on how to start, organize and get through a multiple months long project like this. I like to learn on how to be more efficient, and the end of a competition is IMHO a great exercise for that.\n\nFirst of all, I want to give a huge thanks to the members of my team @rsmits @acmilannesta & @nerin0707 for the awesome work we managed to pull off, and getting what is my first medal and a silver one that is!\n\nNow, these guys had some really shinny hardware to throw some models at, but I don't, and that's also one of the reasons why I love computer vision tasks to much: they require thinking about optimizations, tricks, ways to get around long training times, and I find that just as interesting as trying to find the best model whatever the training times.\n\n## Lessons applied from previous experience\n\nI'll be referring quite a bit to the previous post like this that I did on the Understanding Clouds competition, where I was giving my [middle-rank guys' perspective on the competition](https://www.kaggle.com/c/understanding_cloud_organization/discussion/118328). In that competition, I got really frustrated that I did not have time to develop a pipeline quickly enough to try all the ideas that I wanted, I spend most of the time setting up a pipeline.\n\nFirst take-away, which I applied here and that helped a lot: **iterate fast as early as possible**. The first thing I did was to create a new dataset with [cropped & resized images](https://www.kaggle.com/maxlenormand/cropping-to-character-resizing-images) to only keep the \"useful\" part of the image. The idea here is to get rid of as much data as possible, while keeping as much of the *useful* data as possible. Reducing image size is a great way to do so. A 137 x 236 has 32,332 pixels in it. By cropping to the character and resizing the image to a 100x100 we get images of 10,000 pixels, which 3.23 times less. We barely loose any information, if any at all, and we can train a model more than 3 times faster. Along side this, I start by taking a stratified subset of the dataset and only train on that. The idea isn't to get a good score, but to get a score at all. Again, iterating fast is the name of the game here. Being able to create a pipeline that outputs a result is a really important first step. And for that, each run has to take seconds, not hours. If it takes seconds, you're never leaving your keyboard and in an hour you can go through 10, 15, 20 iterations and start getting a working pipeline. This is especially important when you don't have access to hardware, those 30h of GPU time are very precious. I always have a `TESTING_PIPELINE = True` flag at the top of all my notebook. If set to `True` this only uses a few hundred images to run the whole pipeline. That way, I can change my model, add augmentations, you name it, and check in a minute if it doesn't break anything. Ideally you'd have unit tests for that, but I haven't (yet) figured a way to efficiently do that in a notebook. Fast iteration is by far the biggest win for me in this competition. \n\nSecond topic that I spend a lot of time refining is how to use limited GPU time as efficiently as possible. I ended up upgrading to Colab Pro about 2 weeks before the end of the competition and it opened up the possibilities for me. But it also has a lot of drawbacks (session has to be left open, and everything is still in 1 notebook) that require some thinking. As much as I could, I tried to move pre (and post) processing outside of GPU times. That flag for testing was also very useful in making sure that I wouldn't end up with a notebook that didn't output whenever I tried something new. While a model is training, I also got the weights from a previous model and started doing error anlaysis on my laptop. I do have a dedicated GPU on my laptop, bu it's a 2Gb mx150, which is basically the bottom of the barrel. But still, it might not be good for training, but it's great for inference! I feel like error analysis is pretty overlooked, or at least not very discussed in Kaggle competitions yet there is a lot to learn from what types of errors our models make. For example, I dug deeper into class activation maps (to the point of @rsmits nick naming me after them) and how they might provide useful info on what our models were doing. I didn't get it to where I wanted to, but had I had great hardware, I would have never taken the time to look into it in the first place. I think this is where people with limited hardware should think: A lot of people are really focused on \"blindly\" trying new stuff, throwing it at their GPU and waiting to see what will happen: will recall go up, or down? I think taking the time to investigate *how* it makes our metric go up or down is just as important. Ideally, I would have loved to have time to find the least corrolated models to create even better ensembles. But like everyone, we had a deadline for this competition.\n\nThis leads me to something else that I took from this competition, and that many really good Kagglers keep repeating: having a solid model, capable of holding the shake-up. I have to really thing the guys in my team for all the heavy lifting they did on model tuning and training on that side. But we had quite a bit of talk on who=ich models to take and how to ensemble them. This is IMHO ultimately what made us jump from below bronze in public to silver on private. And again, this can be done without much hardware once the models are trained. That's the cool thing about these computer vision tasks! There's so much more than just raw model training.\n\n## Take-away from this competition\n\nI also want to touch upon the fact that teaming up with different people for this competition was great, and I highly recommend it to anybody getting started in their competition as much as possible. I feel like I got to ask a bunch of the questions I had, but also re-evaluate things I thought I understood when came time to explain them to someone on the team. Those are part of the exercise.\n\nFinally, I'd like to talk about all the info that can be found on the forum. There is **so** much info that has been posted in this competition, it was really interesting! Being able to follow all of this was a great experience even without being able to replicate all of it once again due to hardware limitations. But just trying to sum it up, and push the useful parts forward to the guys on my team that then could focus on training the model was a way to contribute.\n\nIn short, I hope this will be useful to people that like me don't have access to super high end hardware but want to give a try at computer vision. I'm not even the best person to talk about this, as @aerdem4 managed to solo the 21st place [only using Kaggle kernels](https://www.kaggle.com/c/bengaliai-cv19/discussion/136056). But this also for people that aren grand masters, like me, and that might want to know that it's still possible to reach a medal when teaming up and contributing in a way that brings good ideas to the table. I really really like Kaggling with limited hardware, I do believe limitations lead to creativity and to having to think and rethink of ways things are done. \n\nOnce again, thanks to the team for getting me in even though I didn't have a 2080Ti!", "779198": "Words of wisdom! also, congratulations on your result.", "779244": "Thanks! :)", "779344": "Some great advice there! And congrats on your silver.\nI created a very small toy dataset by doing a stratified split on the original data and tested my models on that.\nSadly, things didn't replicate for the whole data and eventually I could never submit because dealing with free tier colab is a very tedious task. Although, I did read a lot of papers and tried to implement everything from scratch myself which was fun in itself. \nReally looking forward to the next CV competiion now.", "779364": "I think especially in this competition, where there was a bit of work required to actually be able to make a submission that worked in the first place, using stratified subsets was a great way to get started.", "779838": "Thanks - very useful!", "779906": "Feel free to add if you have any additional points!", "780923": "Thanks for your detailed sharing. Fast iteration and training with a small training set are really good suggestions. I wanted to set a small train set but had no clue how to do it. May I ask how do you choose samples for this small training set without losing generalization ability?", "780996": "Sure!\n\nI usually do this in multiple steps. I started by making a single class classifier (only trying to predict consonants, as this was the symbol with the least amount of class, thus requiring less data to get a few images per class). For this I simply use `train_test_split` from `sklearn` with the [stratify option set](https://stackoverflow.com/questions/29438265/stratified-train-test-split-in-scikit-learn). This is very helpful to get the data shapes / formats working at first and get a feeling of how to handle the data from the competition.\n\nThen, in this particular competition I discovered [this very useful library](https://github.com/trent-b/iterative-stratification) getting this stratified split for multiple labels.\n\nTo come back to the point on having a strong cross-validation, you can then create K-Folds (also stratified) which can also be done with the `iterative-stratification` mentioned.\n\nHope that helps!", "781029": "Thanks for your detailed reply. That helps a lot! So just for clarification, is the stratified subset of the dataset one of your K-folds created by iterative-stratification lib? If so, what's your k value? I used k equal to five, which means 80% dataset data used for training, still too heavy for training. I used 112\\*112\\*3 input, each epoch took around 30 mins. It was too slow.", "781389": "That's a good question!\n\nChoosing a good k value is tricky, but I also usually go for 5. Though first I tinker around with a 80 / 20 stratified split, which allows me to only train the model once. 5-fold means you'll train 5 times with 80% of the data, which takes, well 5 times longer. So I experiment with a 80 / 20 first and if something interesting comes up, I try it on the 5-Fold. Again the name of the game for me is smart subsetting to speed up training. Especially with limited hardware access!", "781881": "Thanks a lot! That's really helpful! I'd like to try this in my next competition. 👍", "1177304": "Thank you for sharing your experience @maxlenormand. I am new to kaggle competitions and working on images. Your post was very helpful and informative." }, "source": "meta" }