Hi , I have tried to replicate solution present in the notebook : https://www.kaggle.com/martinpiotte/whale-recognition-model-with-score-0-78563/output\nusing FASTAI library.
\n\nTill now I have been able to incorporate the score matrix part into the data loader and siamese network architecture in the model creation part.
\n\nLink to my code: \nhttps://github.com/SwatiTiwarii/whale_competition/\nmartin's_siamese_network_fastai.ipynb
\n\nPlease provide your feedbacks and suggestions regarding improvements in the above solution approach.
\n\nI am working on training model for longer epochs with score matrix updates (based upon training images scores) but the work is still in progress.
\n\nEdit : I have added on_epoch_end callback to update the score matrix. Updated code :
\n\nhttps://github.com/SwatiTiwarii/whale_competition/ \nmartin's_siamese_network_fastai_V3.ipynb
", "messages": [ { "id": 456733, "postDate": "2019-01-16T12:17:03.980Z", "content": "Hi , I have tried to replicate solution present in the notebook : https://www.kaggle.com/martinpiotte/whale-recognition-model-with-score-0-78563/output\nusing FASTAI library.
\n\nTill now I have been able to incorporate the score matrix part into the data loader and siamese network architecture in the model creation part.
\n\nLink to my code: \nhttps://github.com/SwatiTiwarii/whale_competition/\nmartin's_siamese_network_fastai.ipynb
\n\nPlease provide your feedbacks and suggestions regarding improvements in the above solution approach.
\n\nI am working on training model for longer epochs with score matrix updates (based upon training images scores) but the work is still in progress.
\n\nEdit : I have added on_epoch_end callback to update the score matrix. Updated code :
\n\nhttps://github.com/SwatiTiwarii/whale_competition/ \nmartin's_siamese_network_fastai_V3.ipynb
", "rawMarkdown": "Hi , I have tried to replicate solution present in the notebook : https://www.kaggle.com/martinpiotte/whale-recognition-model-with-score-0-78563/output\nusing FASTAI library. \n\nTill now I have been able to incorporate the **score** matrix part into the data loader and **siamese network** architecture in the model creation part. \n\nLink to my code: \nhttps://github.com/SwatiTiwarii/whale_competition/\nmartin's_siamese_network_fastai.ipynb\n\nPlease provide your feedbacks and suggestions regarding improvements in the above solution approach. \n\nI am working on training model for longer epochs with score matrix updates (based upon training images scores) but the work is still in progress. \n\n\nEdit : I have added on_epoch_end callback to update the score matrix. Updated code : \n\nhttps://github.com/SwatiTiwarii/whale_competition/ \nmartin's_siamese_network_fastai_V3.ipynb", "votes": 19 }, { "id": 480354, "postDate": "2019-02-28T05:04:19.117Z", "content": "Hi Swati, thanks for your excellent work.
\n\nAbout your UpdateScoreMatrix callback, does it work for you as written in the notebook?\nIt worked like this\n @dataclass\n class UpdateScoreMatrix(LearnerCallback):
\n\n def __init__(self, update=True):\n super().__init__(learn)\n\n if update:\n learn.data.train_dl.dataset.on_epoch_end()\nbut I don't know if this is wrong.
", "rawMarkdown": "Hi Swati, thanks for your excellent work. \n\nAbout your UpdateScoreMatrix callback, does it work for you as written in the notebook?\nIt worked like this\n @dataclass\n class UpdateScoreMatrix(LearnerCallback):\n \n def __init__(self, update=True):\n super().__init__(learn)\n\n if update:\n learn.data.train_dl.dataset.on_epoch_end()\n\nbut I don't know if this is wrong.", "votes": 1, "replies": [ { "id": 480373, "postDate": "2019-02-28T05:31:18.807Z", "content": "yes , my version works as well becuase it is printing # of steps left after each update.
", "rawMarkdown": "yes , my version works as well becuase it is printing # of steps left after each update.", "votes": 1 } ] }, { "id": 476614, "postDate": "2019-02-22T12:19:28.610Z", "content": "Hallo @Swati, \nTwo suggestions:\n1. the main problem lies in the function score_generation, because ImageItemList shuffles the images automatically. So the answer (x,y) of Lap is not the real order in self.ds of TwoImDataset. This is why Lap doesn't work. \n2. Do not shuffle the data in train_dl Dataloader in order to get 50% positive and 50% negative examples. And in fact you have shuffled the data in Class TwoImDataset.
", "rawMarkdown": "Hallo @Swati, \nTwo suggestions:\n1. the main problem lies in the function score_generation, because ImageItemList shuffles the images automatically. So the answer (x,y) of Lap is not the real order in self.ds of TwoImDataset. This is why Lap doesn't work. \n2. Do not shuffle the data in train_dl Dataloader in order to get 50% positive and 50% negative examples. And in fact you have shuffled the data in Class TwoImDataset. ", "votes": 2, "replies": [ { "id": 479320, "postDate": "2019-02-27T04:17:01.397Z", "content": "Thanks a lot Sir. This is very very useful insight. Would have been very difficult for me alone to understand this. This was my motivation while publicly posting my efforts :) Thank you very much once again.
", "rawMarkdown": "Thanks a lot Sir. This is very very useful insight. Would have been very difficult for me alone to understand this. This was my motivation while publicly posting my efforts :) Thank you very much once again." }, { "id": 479951, "postDate": "2019-02-27T15:57:10.280Z", "content": "hi kclick.\nWould it make a diff if we generate lap pairs just once statically and then use list generated every time ?This is to improve the performance.
", "rawMarkdown": "hi kclick.\nWould it make a diff if we generate lap pairs just once statically and then use list generated every time ?This is to improve the performance." } ] }, { "id": 460519, "postDate": "2019-01-23T21:50:37.023Z", "content": "Nice one. As you probably already know, pytorch's datasets don't have an on_epoch_end callback as you've coded. You can however use fastai learner callbacks to change dataloaders on the fly eg each epoch, if that's what you seek.\nMy fast.ai siamese model (no lapjv, low augmentation) is 0.785 after 'just' 30 epochs (edit: 0.835 after 125), so keep at it.
Hi @robga\nI'm curious how you get 0.785 without lapjv, because my siamese only got about 0.3. Any more details? Thanks!
", "rawMarkdown": "Hi @robga\nI'm curious how you get 0.785 without lapjv, because my siamese only got about 0.3. Any more details? Thanks!" }, { "id": 460722, "postDate": "2019-01-24T10:03:01.097Z", "rawMarkdown": "", "isDeleted": true }, { "id": 460820, "postDate": "2019-01-24T13:27:46.967Z", "content": "I think what Mrtin mean is that insted of exact but costly training examples, we actually only need some heuristic and cheap examples. However, just training from random picked data cannot get a good model, IMHO. I guess to get 0.785, you need cropped images and some method to select hard examples.
", "rawMarkdown": "I think what Mrtin mean is that insted of exact but costly training examples, we actually only need some heuristic and cheap examples. However, just training from random picked data cannot get a good model, IMHO. I guess to get 0.785, you need cropped images and some method to select hard examples." }, { "id": 463807, "postDate": "2019-01-30T16:39:13.743Z", "content": "Hi @rogba , thanks for your inputs. I have added on_epoch_end() functionality and have cleaned the code a little bit. I have added the path to the updated notebook. \nI have added the results till 40 epochs, but loss function is performing poorly. One reason which i can think of is , i should have a base learner , which is very good as separating matching vs non-matching images , then i should add lapvj part. \nCan you also suggest some other methods for coming up with good training sample pairs.
", "rawMarkdown": "Hi @rogba , thanks for your inputs. I have added on_epoch_end() functionality and have cleaned the code a little bit. I have added the path to the updated notebook. \nI have added the results till 40 epochs, but loss function is performing poorly. One reason which i can think of is , i should have a base learner , which is very good as separating matching vs non-matching images , then i should add lapvj part. \nCan you also suggest some other methods for coming up with good training sample pairs. " }, { "id": 463923, "postDate": "2019-01-30T22:50:27.380Z", "content": "I think Martin’s similarity function, which you’ve replicated, is a good one. No doubt we will find out at the end of competition that better metrics have been discovered.
\n\nI don’t have much/any time for the rest of the competition so am happy to share some of my solution.
\n\nI haven’t looked at your amended notebook but I achieved 0.845 in 100ish epochs in this way with resnet18 as a base. I forewent any elegance of a derangement. I mean, so what if the negative part of a hard pair is a repeat? It seems an unnessary hoop. The hardest pairs are those that have been historically most wrong. And what measures that? Periodic predictions. If hypothetically you have a whale image that the model says is most similar to 10 other whales, why not tell the model in the next epoch it is wrong 10 times. So feed the most wrong to the model ASAP and see how it copes, not bothering with derangement. (in Martins original notebook, he was apprehensive to send hard examples early.) Keep predicting and sending the most confounding as the next sequence of epochs training data.
\n\nIn fastai/pytorch I just send a supersized set of pairs to a dataset and then using a callback have it use a slice of the dataset - different pairs - depending on the epoch number. No doubt there is a better way to do this sampling than my clumsy hacking :)
\n\nBest of luck.
", "rawMarkdown": "I think Martin’s similarity function, which you’ve replicated, is a good one. No doubt we will find out at the end of competition that better metrics have been discovered. \n\nI don’t have much/any time for the rest of the competition so am happy to share some of my solution. \n\nI haven’t looked at your amended notebook but I achieved 0.845 in 100ish epochs in this way with resnet18 as a base. I forewent any elegance of a derangement. I mean, so what if the negative part of a hard pair is a repeat? It seems an unnessary hoop. The hardest pairs are those that have been historically most wrong. And what measures that? Periodic predictions. If hypothetically you have a whale image that the model says is most similar to 10 other whales, why not tell the model in the next epoch it is wrong 10 times. So feed the most wrong to the model ASAP and see how it copes, not bothering with derangement. (in Martins original notebook, he was apprehensive to send hard examples early.) Keep predicting and sending the most confounding as the next sequence of epochs training data. \n\nIn fastai/pytorch I just send a supersized set of pairs to a dataset and then using a callback have it use a slice of the dataset - different pairs - depending on the epoch number. No doubt there is a better way to do this sampling than my clumsy hacking :)\n\nBest of luck. ", "votes": 5 }, { "id": 467587, "postDate": "2019-02-07T11:35:08.933Z", "content": "Thanks for your help @robga! If I may ask, are you using an ensemble and optimizing using a metric learning loss like contrastive or triplet? Or are you doing like Martin and adding a head model on top of the ensemble which is responsible to predicting a score given two whales embeddings as input?
", "rawMarkdown": "Thanks for your help @robga! If I may ask, are you using an ensemble and optimizing using a metric learning loss like contrastive or triplet? Or are you doing like Martin and adding a head model on top of the ensemble which is responsible to predicting a score given two whales embeddings as input?" }, { "id": 473662, "postDate": "2019-02-18T10:34:13.093Z", "content": "@swati hw are you predicting new whales....
", "rawMarkdown": "@swati hw are you predicting new whales.... " } ] }, { "id": 473661, "postDate": "2019-02-18T10:33:20.010Z", "content": "how are you predicting the new_whales in updated notebook ..
", "rawMarkdown": "how are you predicting the new_whales in updated notebook ..", "replies": [ { "id": 480332, "postDate": "2019-02-28T04:08:38.023Z", "content": "hi , I am doing it , similar to radek , finding a threshold for whale probability by adding around 1000 new whales in validation set, then i apply same threshold on test set.
", "rawMarkdown": "hi , I am doing it , similar to radek , finding a threshold for whale probability by adding around 1000 new whales in validation set, then i apply same threshold on test set. " } ] }, { "id": 472292, "postDate": "2019-02-15T16:42:18.823Z", "content": "Hi @Swati,\nThanks for the Kernel. This has been a great help.\nOne suggestion to improve score.\nIn your score_generation routine, please put model back to train() mode from eval() mode after calculating the score matrix each time. As I understand, in eval() mode both batch-norm and dropout layers are not active(may be some other layers too!). so the model can learn only so much without these two layers.\nI think we miss this step as it is not common to call eval() in the middle of training as we do in these kernels.\nHope this helps.
", "rawMarkdown": "Hi @Swati,\nThanks for the Kernel. This has been a great help.\nOne suggestion to improve score.\nIn your score_generation routine, please put model back to train() mode from eval() mode after calculating the score matrix each time. As I understand, in eval() mode both batch-norm and dropout layers are not active(may be some other layers too!). so the model can learn only so much without these two layers.\nI think we miss this step as it is not common to call eval() in the middle of training as we do in these kernels.\nHope this helps.\n", "replies": [ { "id": 476387, "postDate": "2019-02-22T04:05:59.260Z", "content": "hi Subra, actually after score_generation routine , i am creating new data object and new Learner object. and hence this learner object will be in train mode itself by default.
", "rawMarkdown": "hi Subra, actually after score_generation routine , i am creating new data object and new Learner object. and hence this learner object will be in train mode itself by default. " } ] }, { "id": 468431, "postDate": "2019-02-08T21:16:43.837Z", "content": "Thank you!!\nI had tried to duplicate siamese network using fastai v1, but I failed so many times and then I gave up...
", "rawMarkdown": "Thank you!!\nI had tried to duplicate siamese network using fastai v1, but I failed so many times and then I gave up...", "replies": [ { "id": 470527, "postDate": "2019-02-13T05:50:17.737Z", "content": "Hi Jolyon, thanks a lot for appreciating it. Let me know your progress if possible. Because i have not been able to reach beyond 0.65 using fast ai v1 as of now.
", "rawMarkdown": "Hi Jolyon, thanks a lot for appreciating it. Let me know your progress if possible. Because i have not been able to reach beyond 0.65 using fast ai v1 as of now." } ] }, { "id": 460581, "postDate": "2019-01-24T02:37:36.810Z", "content": "What leaderboard score does your notebook get?
", "rawMarkdown": "What leaderboard score does your notebook get?" } ], "comments": [ { "id": 480354, "author_name": "interneuron", "author_url": "", "post_date": "2019-02-28T05:04:19.117000", "content": "Hi Swati, thanks for your excellent work.
\n\nAbout your UpdateScoreMatrix callback, does it work for you as written in the notebook?\nIt worked like this\n @dataclass\n class UpdateScoreMatrix(LearnerCallback):
\n\n def __init__(self, update=True):\n super().__init__(learn)\n\n if update:\n learn.data.train_dl.dataset.on_epoch_end()\nbut I don't know if this is wrong.
", "votes": 1, "replies": [ { "id": 480373, "author_name": "SwatiTiwari", "author_url": "", "post_date": "2019-02-28T05:31:18.807000", "content": "yes , my version works as well becuase it is printing # of steps left after each update.
", "votes": 1, "replies": [] } ] }, { "id": 476614, "author_name": "klickmal", "author_url": "", "post_date": "2019-02-22T12:19:28.610000", "content": "Hallo @Swati, \nTwo suggestions:\n1. the main problem lies in the function score_generation, because ImageItemList shuffles the images automatically. So the answer (x,y) of Lap is not the real order in self.ds of TwoImDataset. This is why Lap doesn't work. \n2. Do not shuffle the data in train_dl Dataloader in order to get 50% positive and 50% negative examples. And in fact you have shuffled the data in Class TwoImDataset.
", "votes": 2, "replies": [ { "id": 479320, "author_name": "SwatiTiwari", "author_url": "", "post_date": "2019-02-27T04:17:01.397000", "content": "Thanks a lot Sir. This is very very useful insight. Would have been very difficult for me alone to understand this. This was my motivation while publicly posting my efforts :) Thank you very much once again.
", "votes": 0, "replies": [] }, { "id": 479951, "author_name": "Jaideep", "author_url": "", "post_date": "2019-02-27T15:57:10.280000", "content": "hi kclick.\nWould it make a diff if we generate lap pairs just once statically and then use list generated every time ?This is to improve the performance.
", "votes": 0, "replies": [] } ] }, { "id": 460519, "author_name": "robga", "author_url": "", "post_date": "2019-01-23T21:50:37.023000", "content": "Nice one. As you probably already know, pytorch's datasets don't have an on_epoch_end callback as you've coded. You can however use fastai learner callbacks to change dataloaders on the fly eg each epoch, if that's what you seek.\nMy fast.ai siamese model (no lapjv, low augmentation) is 0.785 after 'just' 30 epochs (edit: 0.835 after 125), so keep at it.
Hi @robga\nI'm curious how you get 0.785 without lapjv, because my siamese only got about 0.3. Any more details? Thanks!
", "votes": 0, "replies": [] }, { "id": 460722, "author_name": "", "author_url": "", "post_date": "2019-01-24T10:03:01.097000", "content": "", "votes": 0, "replies": [] }, { "id": 460820, "author_name": "benwu232", "author_url": "", "post_date": "2019-01-24T13:27:46.967000", "content": "I think what Mrtin mean is that insted of exact but costly training examples, we actually only need some heuristic and cheap examples. However, just training from random picked data cannot get a good model, IMHO. I guess to get 0.785, you need cropped images and some method to select hard examples.
", "votes": 0, "replies": [] }, { "id": 463807, "author_name": "SwatiTiwari", "author_url": "", "post_date": "2019-01-30T16:39:13.743000", "content": "Hi @rogba , thanks for your inputs. I have added on_epoch_end() functionality and have cleaned the code a little bit. I have added the path to the updated notebook. \nI have added the results till 40 epochs, but loss function is performing poorly. One reason which i can think of is , i should have a base learner , which is very good as separating matching vs non-matching images , then i should add lapvj part. \nCan you also suggest some other methods for coming up with good training sample pairs.
", "votes": 0, "replies": [] }, { "id": 463923, "author_name": "robga", "author_url": "", "post_date": "2019-01-30T22:50:27.380000", "content": "I think Martin’s similarity function, which you’ve replicated, is a good one. No doubt we will find out at the end of competition that better metrics have been discovered.
\n\nI don’t have much/any time for the rest of the competition so am happy to share some of my solution.
\n\nI haven’t looked at your amended notebook but I achieved 0.845 in 100ish epochs in this way with resnet18 as a base. I forewent any elegance of a derangement. I mean, so what if the negative part of a hard pair is a repeat? It seems an unnessary hoop. The hardest pairs are those that have been historically most wrong. And what measures that? Periodic predictions. If hypothetically you have a whale image that the model says is most similar to 10 other whales, why not tell the model in the next epoch it is wrong 10 times. So feed the most wrong to the model ASAP and see how it copes, not bothering with derangement. (in Martins original notebook, he was apprehensive to send hard examples early.) Keep predicting and sending the most confounding as the next sequence of epochs training data.
\n\nIn fastai/pytorch I just send a supersized set of pairs to a dataset and then using a callback have it use a slice of the dataset - different pairs - depending on the epoch number. No doubt there is a better way to do this sampling than my clumsy hacking :)
\n\nBest of luck.
", "votes": 5, "replies": [] }, { "id": 467587, "author_name": "Eduardo Rocha de Andrade", "author_url": "", "post_date": "2019-02-07T11:35:08.933000", "content": "Thanks for your help @robga! If I may ask, are you using an ensemble and optimizing using a metric learning loss like contrastive or triplet? Or are you doing like Martin and adding a head model on top of the ensemble which is responsible to predicting a score given two whales embeddings as input?
", "votes": 0, "replies": [] }, { "id": 473662, "author_name": "Jaideep", "author_url": "", "post_date": "2019-02-18T10:34:13.093000", "content": "@swati hw are you predicting new whales....
", "votes": 0, "replies": [] } ] }, { "id": 473661, "author_name": "Jaideep", "author_url": "", "post_date": "2019-02-18T10:33:20.010000", "content": "how are you predicting the new_whales in updated notebook ..
", "votes": 0, "replies": [ { "id": 480332, "author_name": "SwatiTiwari", "author_url": "", "post_date": "2019-02-28T04:08:38.023000", "content": "hi , I am doing it , similar to radek , finding a threshold for whale probability by adding around 1000 new whales in validation set, then i apply same threshold on test set.
", "votes": 0, "replies": [] } ] }, { "id": 472292, "author_name": "Subra", "author_url": "", "post_date": "2019-02-15T16:42:18.823000", "content": "Hi @Swati,\nThanks for the Kernel. This has been a great help.\nOne suggestion to improve score.\nIn your score_generation routine, please put model back to train() mode from eval() mode after calculating the score matrix each time. As I understand, in eval() mode both batch-norm and dropout layers are not active(may be some other layers too!). so the model can learn only so much without these two layers.\nI think we miss this step as it is not common to call eval() in the middle of training as we do in these kernels.\nHope this helps.
", "votes": 0, "replies": [ { "id": 476387, "author_name": "SwatiTiwari", "author_url": "", "post_date": "2019-02-22T04:05:59.260000", "content": "hi Subra, actually after score_generation routine , i am creating new data object and new Learner object. and hence this learner object will be in train mode itself by default.
", "votes": 0, "replies": [] } ] }, { "id": 468431, "author_name": "Jolyon", "author_url": "", "post_date": "2019-02-08T21:16:43.837000", "content": "Thank you!!\nI had tried to duplicate siamese network using fastai v1, but I failed so many times and then I gave up...
", "votes": 0, "replies": [ { "id": 470527, "author_name": "SwatiTiwari", "author_url": "", "post_date": "2019-02-13T05:50:17.737000", "content": "Hi Jolyon, thanks a lot for appreciating it. Let me know your progress if possible. Because i have not been able to reach beyond 0.65 using fast ai v1 as of now.
", "votes": 0, "replies": [] } ] }, { "id": 460581, "author_name": "impulsecorp", "author_url": "", "post_date": "2019-01-24T02:37:36.810000", "content": "What leaderboard score does your notebook get?
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "456733": "Hi , I have tried to replicate solution present in the notebook : https://www.kaggle.com/martinpiotte/whale-recognition-model-with-score-0-78563/output\nusing FASTAI library. \n\nTill now I have been able to incorporate the **score** matrix part into the data loader and **siamese network** architecture in the model creation part. \n\nLink to my code: \nhttps://github.com/SwatiTiwarii/whale_competition/\nmartin's_siamese_network_fastai.ipynb\n\nPlease provide your feedbacks and suggestions regarding improvements in the above solution approach. \n\nI am working on training model for longer epochs with score matrix updates (based upon training images scores) but the work is still in progress. \n\n\nEdit : I have added on_epoch_end callback to update the score matrix. Updated code : \n\nhttps://github.com/SwatiTiwarii/whale_competition/ \nmartin's_siamese_network_fastai_V3.ipynb", "480354": "Hi Swati, thanks for your excellent work. \n\nAbout your UpdateScoreMatrix callback, does it work for you as written in the notebook?\nIt worked like this\n @dataclass\n class UpdateScoreMatrix(LearnerCallback):\n \n def __init__(self, update=True):\n super().__init__(learn)\n\n if update:\n learn.data.train_dl.dataset.on_epoch_end()\n\nbut I don't know if this is wrong.", "476614": "Hallo @Swati, \nTwo suggestions:\n1. the main problem lies in the function score_generation, because ImageItemList shuffles the images automatically. So the answer (x,y) of Lap is not the real order in self.ds of TwoImDataset. This is why Lap doesn't work. \n2. Do not shuffle the data in train_dl Dataloader in order to get 50% positive and 50% negative examples. And in fact you have shuffled the data in Class TwoImDataset. ", "460519": "Nice one. As you probably already know, pytorch's datasets don't have an `on_epoch_end` callback as you've coded. You can however use fastai learner callbacks to change dataloaders on the fly eg each epoch, if that's what you seek.\nMy fast.ai siamese model (no lapjv, low augmentation) is 0.785 after 'just' 30 epochs (edit: 0.835 after 125), so keep at it.", "473661": "how are you predicting the new_whales in updated notebook ..", "472292": "Hi @Swati,\nThanks for the Kernel. This has been a great help.\nOne suggestion to improve score.\nIn your score_generation routine, please put model back to train() mode from eval() mode after calculating the score matrix each time. As I understand, in eval() mode both batch-norm and dropout layers are not active(may be some other layers too!). so the model can learn only so much without these two layers.\nI think we miss this step as it is not common to call eval() in the middle of training as we do in these kernels.\nHope this helps.\n", "468431": "Thank you!!\nI had tried to duplicate siamese network using fastai v1, but I failed so many times and then I gave up...", "460581": "What leaderboard score does your notebook get?" } }