Congratulations to all the winners and all those who have done well in this competition through their hard work! Thanks also to Professor Heikki Huttunen for hosting this exciting competition! I had a lot of fun and learned plenty of lessons.
\n\nMy code with detailed comments is available on my GitHub repo.
\n\nFirst of all, I cleaned the training dataset. It can be seen that image classes may contain peculiar or odd images, for example, 000118_07.jpgin Limousine class. Besides, certain images belong to several different classes at the same time. For instance, 000040_15.jpg in the Van class is the image 000040_09.jpg in the Ambulance class. Therefore, to enhance the dataset quality, I filtered the data and removed these \"outliers\" from the training set.
At the next step, our team trained a bunch of Tensorflow/Keras classification models as well as models from this repo. Additionally, my teammate Roman was working with Wolfram Mathematica and trained models there.
\n\nWe tried the following architectures: Xception, VGG16, VGG19, InceptionV3, InceptionResNetV2, MobileNet, MobileNetV2, NASNetMobile, NASNetLarge, ResNet18, ResNet34, ResNet50, ResNet101, ResNet152, SeNet154, WideResNetV2, ResNeXt50, ResNeXt101. Different image sizes starting from 224 and up to 331 as well as batch sizes were used. Also, I employed pretrained EfficientNet models from here by training EfficientNetB4 and EfficientNetB7.
\n\nTo load images, I used Keras Image Generators, which generate batches of tensor image data with real-time data augmentation. In particular, I used horizontal flip, rotation, width shift, and height shift. In this setup, zoom augmentation did not give any gain in the evaluation score.
\n\nIt can be noticed, that certain classes are dominant in the data set. To resolve this issue, I used class weights for each class. I utilized compute_class_weight function from sklearn.utils with 'balanced' distribution. In this case, class weights are given by the following formula: n_samples/(n_classes * np.bincount(y)).
What we also used/tried:\n- Layers freezing\n- Usage of dropouts\n- Reducing the learning rates\n- Data distillation\n- Label smoothing
\n\nUsing simple arithmetic average and maximization over class probabilities, we ensembled results of various models, which gave us approximately 1% gain in the public score. Additionally, we attempted to blend Kaggle submissions, i.e. tried majority voting. This technique returned a nearly 0.5% increase.
\n\nThe main breakthrough of the blending is the weighted average for each category separately. From the confusion matrixes of the CV set, we noticed that certain models classify specific classes better than others. Therefore, models from the top-10 list were chosen and applying a matrix of weights, the average was calculated. This category-specific average significantly outperforms the simple average and maximization methods over class probabilities.
\n\nCongratulations to all the winners and all those who have done well in this competition through their hard work! Thanks also to Professor Heikki Huttunen for hosting this exciting competition! I had a lot of fun and learned plenty of lessons.
\n\nMy code with detailed comments is available on my GitHub repo.
\n\nFirst of all, I cleaned the training dataset. It can be seen that image classes may contain peculiar or odd images, for example, 000118_07.jpgin Limousine class. Besides, certain images belong to several different classes at the same time. For instance, 000040_15.jpg in the Van class is the image 000040_09.jpg in the Ambulance class. Therefore, to enhance the dataset quality, I filtered the data and removed these \"outliers\" from the training set.
At the next step, our team trained a bunch of Tensorflow/Keras classification models as well as models from this repo. Additionally, my teammate Roman was working with Wolfram Mathematica and trained models there.
\n\nWe tried the following architectures: Xception, VGG16, VGG19, InceptionV3, InceptionResNetV2, MobileNet, MobileNetV2, NASNetMobile, NASNetLarge, ResNet18, ResNet34, ResNet50, ResNet101, ResNet152, SeNet154, WideResNetV2, ResNeXt50, ResNeXt101. Different image sizes starting from 224 and up to 331 as well as batch sizes were used. Also, I employed pretrained EfficientNet models from here by training EfficientNetB4 and EfficientNetB7.
\n\nTo load images, I used Keras Image Generators, which generate batches of tensor image data with real-time data augmentation. In particular, I used horizontal flip, rotation, width shift, and height shift. In this setup, zoom augmentation did not give any gain in the evaluation score.
\n\nIt can be noticed, that certain classes are dominant in the data set. To resolve this issue, I used class weights for each class. I utilized compute_class_weight function from sklearn.utils with 'balanced' distribution. In this case, class weights are given by the following formula: n_samples/(n_classes * np.bincount(y)).
What we also used/tried:\n- Layers freezing\n- Usage of dropouts\n- Reducing the learning rates\n- Data distillation\n- Label smoothing
\n\nUsing simple arithmetic average and maximization over class probabilities, we ensembled results of various models, which gave us approximately 1% gain in the public score. Additionally, we attempted to blend Kaggle submissions, i.e. tried majority voting. This technique returned a nearly 0.5% increase.
\n\nThe main breakthrough of the blending is the weighted average for each category separately. From the confusion matrixes of the CV set, we noticed that certain models classify specific classes better than others. Therefore, models from the top-10 list were chosen and applying a matrix of weights, the average was calculated. This category-specific average significantly outperforms the simple average and maximization methods over class probabilities.
\n\nHow was the experience training networks in Mathematica? I haven't seen anybody training deep neural networks on large datasets in Mathematica, so I am curious about it's performance and efficiency.
", "rawMarkdown": "How was the experience training networks in Mathematica? I haven't seen anybody training deep neural networks on large datasets in Mathematica, so I am curious about it's performance and efficiency.", "votes": null }, { "id": "709257", "postDate": "01/03/2020 09:13:51", "content": "Hi Mark! The training process and overall performance were good. We didn't compare the training time with Keras due to a bit different strategies.
\n\nMathematica repository has quite a lot of pretrained models, and I used only those. I tried to import models from Keras but did not succeed. Not sure if it is possible. There were also some issues with the cleaning of the graphic card memory after the training finish, so I had to restart the kernel sometimes.
\n\nOverall, I find the process of training neural networks quite easy and intuitive for those who used to Mathematica language. Wolfram Mathematica's documentation with lots of examples helps a lot.
\n\nI plan to write a separate, more detailed report/tutorial on my Mathematica implementation then I have free time.
", "rawMarkdown": "Hi Mark! The training process and overall performance were good. We didn't compare the training time with Keras due to a bit different strategies.\n\nMathematica repository has quite a lot of pretrained [models](https://resources.wolframcloud.com/NeuralNetRepository/), and I used only those. I tried to import models from Keras but did not succeed. Not sure if it is possible. There were also some issues with the cleaning of the graphic card memory after the training finish, so I had to restart the kernel sometimes. \n\nOverall, I find the process of training neural networks quite easy and intuitive for those who used to Mathematica language. Wolfram Mathematica's documentation with lots of examples helps a lot.\n\nI plan to write a separate, more detailed report/tutorial on my Mathematica implementation then I have free time.", "votes": null } ], "comments": [ { "id": 703894, "author_name": "markseliaev", "author_url": "", "post_date": "12/26/2019 19:15:49", "content": "How was the experience training networks in Mathematica? I haven't seen anybody training deep neural networks on large datasets in Mathematica, so I am curious about it's performance and efficiency.
", "votes": null, "replies": [ { "id": 709257, "author_name": "romankovalchukov", "author_url": "", "post_date": "01/03/2020 09:13:51", "content": "Hi Mark! The training process and overall performance were good. We didn't compare the training time with Keras due to a bit different strategies.
\n\nMathematica repository has quite a lot of pretrained models, and I used only those. I tried to import models from Keras but did not succeed. Not sure if it is possible. There were also some issues with the cleaning of the graphic card memory after the training finish, so I had to restart the kernel sometimes.
\n\nOverall, I find the process of training neural networks quite easy and intuitive for those who used to Mathematica language. Wolfram Mathematica's documentation with lots of examples helps a lot.
\n\nI plan to write a separate, more detailed report/tutorial on my Mathematica implementation then I have free time.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "696426": "Congratulations to all the winners and all those who have done well in this competition through their hard work! Thanks also to Professor [Heikki Huttunen](https://www.kaggle.com/mahehu) for hosting this exciting competition! I had a lot of fun and learned plenty of lessons.\n\nMy code with detailed comments is available on my [GitHub repo](https://github.com/nikitataf/vehicle-recognition).\n\n# Data cleaning\n\nFirst of all, I cleaned the training dataset. It can be seen that image classes may contain peculiar or odd images, for example, `000118_07.jpg `in Limousine class. Besides, certain images belong to several different classes at the same time. For instance, `000040_15.jpg` in the Van class is the image `000040_09.jpg` in the Ambulance class. Therefore, to enhance the dataset quality, I filtered the data and removed these \"outliers\" from the training set.\n\n# CNN\n\nAt the next step, our team trained a bunch of Tensorflow/Keras classification models as well as models from this [repo](https://github.com/qubvel/classification_models ). Additionally, my teammate [Roman](https://www.kaggle.com/romankovalchukov) was working with Wolfram Mathematica and trained models there.\n\nWe tried the following architectures: Xception, VGG16, VGG19, InceptionV3, InceptionResNetV2, MobileNet, MobileNetV2, NASNetMobile, NASNetLarge, ResNet18, ResNet34, ResNet50, ResNet101, ResNet152, SeNet154, WideResNetV2, ResNeXt50, ResNeXt101. Different image sizes starting from 224 and up to 331 as well as batch sizes were used. Also, I employed pretrained EfficientNet models from [here](https://github.com/qubvel/efficientnet) by training EfficientNetB4 and EfficientNetB7. \n\nTo load images, I used Keras Image Generators, which generate batches of tensor image data with real-time data augmentation. In particular, I used horizontal flip, rotation, width shift, and height shift. In this setup, zoom augmentation did not give any gain in the evaluation score.\n\nIt can be noticed, that certain classes are dominant in the data set. To resolve this issue, I used class weights for each class. I utilized `compute_class_weight` function from `sklearn.utils` with `'balanced'` distribution. In this case, class weights are given by the following formula: ```n_samples/(n_classes * np.bincount(y))```.\n\nWhat we also used/tried:\n- Layers freezing\n- Usage of dropouts\n- Reducing the learning rates\n- Data distillation\n- Label smoothing\n\n# Blending/Ensembling\n\nUsing simple arithmetic average and maximization over class probabilities, we ensembled results of various models, which gave us approximately 1% gain in the public score. Additionally, we attempted to blend Kaggle submissions, i.e. tried majority voting. This technique returned a nearly 0.5% increase.\n\nThe main breakthrough of the blending is the weighted average for each category separately. From the confusion matrixes of the CV set, we noticed that certain models classify specific classes better than others. Therefore, models from the top-10 list were chosen and applying a matrix of weights, the average was calculated. This category-specific average significantly outperforms the simple average and maximization methods over class probabilities.\n\n# Key takeaways\n\n- Study past solutions from similar competitions\n- Usage of stacking\n- A thoughtful choice of final submissions :)", "703894": "How was the experience training networks in Mathematica? I haven't seen anybody training deep neural networks on large datasets in Mathematica, so I am curious about it's performance and efficiency.", "709257": "Hi Mark! The training process and overall performance were good. We didn't compare the training time with Keras due to a bit different strategies.\n\nMathematica repository has quite a lot of pretrained [models](https://resources.wolframcloud.com/NeuralNetRepository/), and I used only those. I tried to import models from Keras but did not succeed. Not sure if it is possible. There were also some issues with the cleaning of the graphic card memory after the training finish, so I had to restart the kernel sometimes. \n\nOverall, I find the process of training neural networks quite easy and intuitive for those who used to Mathematica language. Wolfram Mathematica's documentation with lots of examples helps a lot.\n\nI plan to write a separate, more detailed report/tutorial on my Mathematica implementation then I have free time." }, "source": "meta" }