I trying to train a simple CNN architecture and when I am trying to have linear layer with 80000 out channels I am running out of memory. How are you dealing with so many classes?
\nNote: I have done all of my work on CPU till now. Is GPU allowing to have 80000 out channels in the last layer?
I trying to train a simple CNN architecture and when I am trying to have linear layer with 80000 out channels I am running out of memory. How are you dealing with so many classes?
\nNote: I have done all of my work on CPU till now. Is GPU allowing to have 80000 out channels in the last layer?
https://github.com/mayukh18/Google-Landmark-Recognition-Retrieval-2019
\nYou can look this code written by Mayukh Bhattacharyya.
Memory is definitively a problem when dealing with such large number of classes. What you could do to reduce the memory requirements and number of parameters of your model is to reduce the number of features of your ensemble.
\nFor example, ResNet101 has 2048 convolutional features so your last FC layer will have 2048x80313 parameters. However, if you add an extra FC layer with bias after pooling to reduce the 2048 features to 512 your model will have 2048x512 + 512x80313, which is roughly 3.9 times less parameters and operations. Additionally, this extra FC + bias layer will also work towards whitening your features, which reduce co-occurrences and helps the score for retrieval tasks.
\nAnother option is to train your model without using classification as a proxy. In this case, you would need to train using contrastive or triplet loss.
\nBut bear in mind that this competition has a lot of data, which means you won't be able to do much using CPU. You'll likely need some GPUs or a TPU to do some work.
", "rawMarkdown": "Memory is definitively a problem when dealing with such large number of classes. What you could do to reduce the memory requirements and number of parameters of your model is to reduce the number of features of your ensemble. \n\nFor example, ResNet101 has 2048 convolutional features so your last FC layer will have 2048x80313 parameters. However, if you add an extra FC layer with bias after pooling to reduce the 2048 features to 512 your model will have 2048x512 + 512x80313, which is roughly 3.9 times less parameters and operations. Additionally, this extra FC + bias layer will also work towards whitening your features, which reduce co-occurrences and helps the score for retrieval tasks. \n\nAnother option is to train your model without using classification as a proxy. In this case, you would need to train using contrastive or triplet loss.\n\nBut bear in mind that this competition has a lot of data, which means you won't be able to do much using CPU. You'll likely need some GPUs or a TPU to do some work.", "votes": null }, { "id": "1003561", "postDate": "09/09/2020 05:03:10", "content": "Hi, Is it mandatory to have training loops(i.e to use private train set) in submission code ? I have pre-trained model which is trained with public training set and in submission code i have only inference part, i always get 0.0000 as public score? Can you please help me
", "rawMarkdown": "Hi, Is it mandatory to have training loops(i.e to use private train set) in submission code ? I have pre-trained model which is trained with public training set and in submission code i have only inference part, i always get 0.0000 as public score? Can you please help me", "votes": null }, { "id": "1004206", "postDate": "09/09/2020 14:33:52", "content": "@shanmugam212 have you got an answer to this? Even I am stuck. Somewhere they are saying that we can train outside. But yeah training loops are still there for private dataset.
\nPeople are saying even for baseline model, train embeddings can be generated outside and just loaded from a file and used. But what if my embeddings are having some files which are not in their private training dataset?
", "rawMarkdown": "shanmugam212 have you got an answer to this? Even I am stuck. Somewhere they are saying that we can train outside. But yeah training loops are still there for private dataset.\n\nPeople are saying even for baseline model, train embeddings can be generated outside and just loaded from a file and used. But what if my embeddings are having some files which are not in their private training dataset?", "votes": null } ], "comments": [ { "id": 996703, "author_name": "shivam17818", "author_url": "", "post_date": "09/03/2020 13:54:59", "content": "https://github.com/mayukh18/Google-Landmark-Recognition-Retrieval-2019
\nYou can look this code written by Mayukh Bhattacharyya.
Memory is definitively a problem when dealing with such large number of classes. What you could do to reduce the memory requirements and number of parameters of your model is to reduce the number of features of your ensemble.
\nFor example, ResNet101 has 2048 convolutional features so your last FC layer will have 2048x80313 parameters. However, if you add an extra FC layer with bias after pooling to reduce the 2048 features to 512 your model will have 2048x512 + 512x80313, which is roughly 3.9 times less parameters and operations. Additionally, this extra FC + bias layer will also work towards whitening your features, which reduce co-occurrences and helps the score for retrieval tasks.
\nAnother option is to train your model without using classification as a proxy. In this case, you would need to train using contrastive or triplet loss.
\nBut bear in mind that this competition has a lot of data, which means you won't be able to do much using CPU. You'll likely need some GPUs or a TPU to do some work.
", "votes": null, "replies": [ { "id": 1003561, "author_name": "shanmugam212", "author_url": "", "post_date": "09/09/2020 05:03:10", "content": "Hi, Is it mandatory to have training loops(i.e to use private train set) in submission code ? I have pre-trained model which is trained with public training set and in submission code i have only inference part, i always get 0.0000 as public score? Can you please help me
", "votes": null, "replies": [] }, { "id": 1004206, "author_name": "rohitdeepu17", "author_url": "", "post_date": "09/09/2020 14:33:52", "content": "@shanmugam212 have you got an answer to this? Even I am stuck. Somewhere they are saying that we can train outside. But yeah training loops are still there for private dataset.
\nPeople are saying even for baseline model, train embeddings can be generated outside and just loaded from a file and used. But what if my embeddings are having some files which are not in their private training dataset?
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "996630": "I trying to train a simple CNN architecture and when I am trying to have linear layer with 80000 out channels I am running out of memory. How are you dealing with so many classes?\nNote: I have done all of my work on CPU till now. Is GPU allowing to have 80000 out channels in the last layer?", "996703": "https://github.com/mayukh18/Google-Landmark-Recognition-Retrieval-2019\nYou can look this code written by Mayukh Bhattacharyya.", "996948": "Memory is definitively a problem when dealing with such large number of classes. What you could do to reduce the memory requirements and number of parameters of your model is to reduce the number of features of your ensemble. \n\nFor example, ResNet101 has 2048 convolutional features so your last FC layer will have 2048x80313 parameters. However, if you add an extra FC layer with bias after pooling to reduce the 2048 features to 512 your model will have 2048x512 + 512x80313, which is roughly 3.9 times less parameters and operations. Additionally, this extra FC + bias layer will also work towards whitening your features, which reduce co-occurrences and helps the score for retrieval tasks. \n\nAnother option is to train your model without using classification as a proxy. In this case, you would need to train using contrastive or triplet loss.\n\nBut bear in mind that this competition has a lot of data, which means you won't be able to do much using CPU. You'll likely need some GPUs or a TPU to do some work.", "1003561": "Hi, Is it mandatory to have training loops(i.e to use private train set) in submission code ? I have pre-trained model which is trained with public training set and in submission code i have only inference part, i always get 0.0000 as public score? Can you please help me", "1004206": "shanmugam212 have you got an answer to this? Even I am stuck. Somewhere they are saying that we can train outside. But yeah training loops are still there for private dataset.\n\nPeople are saying even for baseline model, train embeddings can be generated outside and just loaded from a file and used. But what if my embeddings are having some files which are not in their private training dataset?" }, "source": "meta" }