As per title, I am still new to Kaggle, do someone mind to explain how data from TTA is actually fitted into the unaugmented submission csv?
\n\nThanks in advance!
", "messages": [ { "id": "901497", "postDate": "06/25/2020 14:24:35", "content": "As per title, I am still new to Kaggle, do someone mind to explain how data from TTA is actually fitted into the unaugmented submission csv?
\n\nThanks in advance!
", "rawMarkdown": "As per title, I am still new to Kaggle, do someone mind to explain how data from TTA is actually fitted into the unaugmented submission csv?\n\nThanks in advance!", "votes": null }, { "id": "901556", "postDate": "06/25/2020 15:03:38", "content": "@fadzlinrafi TTA is Test Time Augmentation. \nThe idea is to augment the test data just as you do for the training data and to do prediction on the augmented test data. If you do it only once, your LB will degrade, but you should repeat the process several times and ensemble the outputs (usually a simple mean will do). This way you average the predictions on the same image when it is augmented in different random ways.
", "rawMarkdown": "fadzlinrafi TTA is Test Time Augmentation. \nThe idea is to augment the test data just as you do for the training data and to do prediction on the augmented test data. If you do it only once, your LB will degrade, but you should repeat the process several times and ensemble the outputs (usually a simple mean will do). This way you average the predictions on the same image when it is augmented in different random ways.", "votes": null }, { "id": "901733", "postDate": "06/25/2020 16:45:28", "content": "Here was some work I did on tta for the flowers competition:
\n\nhttps://www.kaggle.com/calebeverett/comparison-of-tta-prediction-procedures
", "rawMarkdown": "Here was some work I did on tta for the flowers competition:\n\nhttps://www.kaggle.com/calebeverett/comparison-of-tta-prediction-procedures", "votes": null }, { "id": "901862", "postDate": "06/25/2020 18:50:13", "content": "Hello @yuval6967 . I've a small question regarding working with TTA and ensambling models together. Like we should first ensemble models and then perform TTA (ensembling the outputs) or we should first perform TTA (esemble the outputs) for every model and then ensemble the models all together. There's any best practices out there or it's the part of the experiment (what's working for the dataset and what's not)
", "rawMarkdown": "Hello @yuval6967 . I've a small question regarding working with TTA and ensambling models together. Like we should first ensemble models and then perform TTA (ensembling the outputs) or we should first perform TTA (esemble the outputs) for every model and then ensemble the models all together. There's any best practices out there or it's the part of the experiment (what's working for the dataset and what's not)", "votes": null }, { "id": "901892", "postDate": "06/25/2020 19:18:31", "content": "I had a similar question - answered here: https://www.kaggle.com/c/flower-classification-with-tpus/discussion/142136
\n\nIt may be useful to think about ensembling and test time augmentation as separate dimensions. Test time augmentation gets applied to the test dataset and ensembling relates to the number of models you run your test dataset through. You could aggregate predictions for individual test examples by averaging across all predictions for all tta versions from each model, or you could choose different aggregation methods for each of tta and model dimensions, i.e., you could take the max of each tta version and then average across each of your models, as one possible example.
", "rawMarkdown": "I had a similar question - answered here: https://www.kaggle.com/c/flower-classification-with-tpus/discussion/142136\n\nIt may be useful to think about ensembling and test time augmentation as separate dimensions. Test time augmentation gets applied to the test dataset and ensembling relates to the number of models you run your test dataset through. You could aggregate predictions for individual test examples by averaging across all predictions for all tta versions from each model, or you could choose different aggregation methods for each of tta and model dimensions, i.e., you could take the max of each tta version and then average across each of your models, as one possible example.", "votes": null }, { "id": "902062", "postDate": "06/25/2020 22:39:47", "content": "Thanks for your explanation!
", "rawMarkdown": "Thanks for your explanation!", "votes": null }, { "id": "902065", "postDate": "06/25/2020 22:41:19", "content": "That's a really informative notebook, thank you!
", "rawMarkdown": "That's a really informative notebook, thank you!", "votes": null }, { "id": "902342", "postDate": "06/26/2020 05:02:16", "content": "https://towardsdatascience.com/test-time-augmentation-tta-and-how-to-perform-it-with-keras-4ac19b67fb4d \nThis article will help you a lot @fadzlinrafi . I also had the same doubt.
", "rawMarkdown": "https://towardsdatascience.com/test-time-augmentation-tta-and-how-to-perform-it-with-keras-4ac19b67fb4d \nThis article will help you a lot @fadzlinrafi . I also had the same doubt.", "votes": null }, { "id": "902431", "postDate": "06/26/2020 06:26:45", "content": "@rahulgulia \nI usually do it together and before the softmax/sigmoid. like this:
\n\n\npreds=[]\nfor model in models:\n for fold in folds:\n for i in range(n):\n ds = test_dataset_with_random_augmentation\n preds.append(model(fold).predict(ds))\np=np.stack(preds,0).mean(0)\nout = sigmoid(p)\n
This is the simplest way that works for me most of the times, if you want you can add weights on the models (never on the tta, and usually not on the folds)
", "rawMarkdown": "rahulgulia \nI usually do it together and before the softmax/sigmoid. like this:\n\n```\npreds=[]\nfor model in models:\n for fold in folds:\n for i in range(n):\n ds = test_dataset_with_random_augmentation\n preds.append(model(fold).predict(ds))\np=np.stack(preds,0).mean(0)\nout = sigmoid(p)\n```\n\nThis is the simplest way that works for me most of the times, if you want you can add weights on the models (never on the tta, and usually not on the folds)", "votes": null }, { "id": "902685", "postDate": "06/26/2020 09:56:51", "content": "thanks @calebeverett and @yuval6967 for your assistance 😄
", "rawMarkdown": "thanks @calebeverett and @yuval6967 for your assistance 😄", "votes": null }, { "id": "902945", "postDate": "06/26/2020 13:28:21", "content": "Nice explanation, thanks
", "rawMarkdown": "Nice explanation, thanks", "votes": null }, { "id": "903457", "postDate": "06/26/2020 20:51:29", "content": "@yuval6967 @calebeverett Thanks for the explaination , I am interested in using TTA now.
", "rawMarkdown": "yuval6967 @calebeverett Thanks for the explaination , I am interested in using TTA now.", "votes": null } ], "comments": [ { "id": 901556, "author_name": "yuval6967", "author_url": "", "post_date": "06/25/2020 15:03:38", "content": "@fadzlinrafi TTA is Test Time Augmentation. \nThe idea is to augment the test data just as you do for the training data and to do prediction on the augmented test data. If you do it only once, your LB will degrade, but you should repeat the process several times and ensemble the outputs (usually a simple mean will do). This way you average the predictions on the same image when it is augmented in different random ways.
", "votes": null, "replies": [ { "id": 902062, "author_name": "fadzlinrafi", "author_url": "", "post_date": "06/25/2020 22:39:47", "content": "Thanks for your explanation!
", "votes": null, "replies": [] }, { "id": 902945, "author_name": "ragnar123", "author_url": "", "post_date": "06/26/2020 13:28:21", "content": "Nice explanation, thanks
", "votes": null, "replies": [] } ] }, { "id": 901733, "author_name": "calebeverett", "author_url": "", "post_date": "06/25/2020 16:45:28", "content": "Here was some work I did on tta for the flowers competition:
\n\nhttps://www.kaggle.com/calebeverett/comparison-of-tta-prediction-procedures
", "votes": null, "replies": [ { "id": 902065, "author_name": "fadzlinrafi", "author_url": "", "post_date": "06/25/2020 22:41:19", "content": "That's a really informative notebook, thank you!
", "votes": null, "replies": [] } ] }, { "id": 901862, "author_name": "rahulgulia", "author_url": "", "post_date": "06/25/2020 18:50:13", "content": "Hello @yuval6967 . I've a small question regarding working with TTA and ensambling models together. Like we should first ensemble models and then perform TTA (ensembling the outputs) or we should first perform TTA (esemble the outputs) for every model and then ensemble the models all together. There's any best practices out there or it's the part of the experiment (what's working for the dataset and what's not)
", "votes": null, "replies": [ { "id": 901892, "author_name": "calebeverett", "author_url": "", "post_date": "06/25/2020 19:18:31", "content": "I had a similar question - answered here: https://www.kaggle.com/c/flower-classification-with-tpus/discussion/142136
\n\nIt may be useful to think about ensembling and test time augmentation as separate dimensions. Test time augmentation gets applied to the test dataset and ensembling relates to the number of models you run your test dataset through. You could aggregate predictions for individual test examples by averaging across all predictions for all tta versions from each model, or you could choose different aggregation methods for each of tta and model dimensions, i.e., you could take the max of each tta version and then average across each of your models, as one possible example.
", "votes": null, "replies": [] }, { "id": 902431, "author_name": "yuval6967", "author_url": "", "post_date": "06/26/2020 06:26:45", "content": "@rahulgulia \nI usually do it together and before the softmax/sigmoid. like this:
\n\n\npreds=[]\nfor model in models:\n for fold in folds:\n for i in range(n):\n ds = test_dataset_with_random_augmentation\n preds.append(model(fold).predict(ds))\np=np.stack(preds,0).mean(0)\nout = sigmoid(p)\n
This is the simplest way that works for me most of the times, if you want you can add weights on the models (never on the tta, and usually not on the folds)
", "votes": null, "replies": [] }, { "id": 902685, "author_name": "rahulgulia", "author_url": "", "post_date": "06/26/2020 09:56:51", "content": "thanks @calebeverett and @yuval6967 for your assistance 😄
", "votes": null, "replies": [] }, { "id": 903457, "author_name": "yash612", "author_url": "", "post_date": "06/26/2020 20:51:29", "content": "@yuval6967 @calebeverett Thanks for the explaination , I am interested in using TTA now.
", "votes": null, "replies": [] } ] }, { "id": 902342, "author_name": "ishalgarg", "author_url": "", "post_date": "06/26/2020 05:02:16", "content": "https://towardsdatascience.com/test-time-augmentation-tta-and-how-to-perform-it-with-keras-4ac19b67fb4d \nThis article will help you a lot @fadzlinrafi . I also had the same doubt.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "901497": "As per title, I am still new to Kaggle, do someone mind to explain how data from TTA is actually fitted into the unaugmented submission csv?\n\nThanks in advance!", "901556": "fadzlinrafi TTA is Test Time Augmentation. \nThe idea is to augment the test data just as you do for the training data and to do prediction on the augmented test data. If you do it only once, your LB will degrade, but you should repeat the process several times and ensemble the outputs (usually a simple mean will do). This way you average the predictions on the same image when it is augmented in different random ways.", "901733": "Here was some work I did on tta for the flowers competition:\n\nhttps://www.kaggle.com/calebeverett/comparison-of-tta-prediction-procedures", "901862": "Hello @yuval6967 . I've a small question regarding working with TTA and ensambling models together. Like we should first ensemble models and then perform TTA (ensembling the outputs) or we should first perform TTA (esemble the outputs) for every model and then ensemble the models all together. There's any best practices out there or it's the part of the experiment (what's working for the dataset and what's not)", "901892": "I had a similar question - answered here: https://www.kaggle.com/c/flower-classification-with-tpus/discussion/142136\n\nIt may be useful to think about ensembling and test time augmentation as separate dimensions. Test time augmentation gets applied to the test dataset and ensembling relates to the number of models you run your test dataset through. You could aggregate predictions for individual test examples by averaging across all predictions for all tta versions from each model, or you could choose different aggregation methods for each of tta and model dimensions, i.e., you could take the max of each tta version and then average across each of your models, as one possible example.", "902062": "Thanks for your explanation!", "902065": "That's a really informative notebook, thank you!", "902342": "https://towardsdatascience.com/test-time-augmentation-tta-and-how-to-perform-it-with-keras-4ac19b67fb4d \nThis article will help you a lot @fadzlinrafi . I also had the same doubt.", "902431": "rahulgulia \nI usually do it together and before the softmax/sigmoid. like this:\n\n```\npreds=[]\nfor model in models:\n for fold in folds:\n for i in range(n):\n ds = test_dataset_with_random_augmentation\n preds.append(model(fold).predict(ds))\np=np.stack(preds,0).mean(0)\nout = sigmoid(p)\n```\n\nThis is the simplest way that works for me most of the times, if you want you can add weights on the models (never on the tta, and usually not on the folds)", "902685": "thanks @calebeverett and @yuval6967 for your assistance 😄", "902945": "Nice explanation, thanks", "903457": "yuval6967 @calebeverett Thanks for the explaination , I am interested in using TTA now." }, "source": "meta" }