i spend time to investigate why some public kernel are better than others.
\ni used one week to discover this:
\nhttps://www.kaggle.com/c/humpback-whale-identification/discussion/82409
\n\"BatchNorm vs LayerNorm before L2 normalization: LayerNorm better\"
https://www.kaggle.com/c/recursion-cellular-image-classification/discussion/109987
\n\"Batch normalization layer is vital just before the L2-normalized features multiplication by the weights. It is indeed mentioned in the original article, but I didn't pay attention at first, and ArcFace started working for me only after I added it.\"
i spend time to investigate why some public kernel are better than others.
\ni used one week to discover this:
\nhttps://www.kaggle.com/c/humpback-whale-identification/discussion/82409
\n\"BatchNorm vs LayerNorm before L2 normalization: LayerNorm better\"
https://www.kaggle.com/c/recursion-cellular-image-classification/discussion/109987
\n\"Batch normalization layer is vital just before the L2-normalized features multiplication by the weights. It is indeed mentioned in the original article, but I didn't pay attention at first, and ArcFace started working for me only after I added it.\"
The characteristics of this game are overfitting the training set. It is useful to add constraints and tricks at the feature level.
\nUsing bn layer and feature loss is equivalent to assuming that the feature space, increases the features to distinguish, also eased the overfitting.
i have a feeling that normalisation layer is smiliar is the PCA trick before deep learning metric learning is \"invented\".
\nIn traditional old ML methods, it is necessary to perform PCA to ensure that the distance is spherical in the embedding space. cosine distance is only meaningful if the embedding space is spherical
\nit is also a known trick that batch size should be large enough for metric learning/retrival/face recognition problem/contrastive learning
", "rawMarkdown": "i have a feeling that normalisation layer is smiliar is the PCA trick before deep learning metric learning is \"invented\". \n\nIn traditional old ML methods, it is necessary to perform PCA to ensure that the distance is spherical in the embedding space. cosine distance is only meaningful if the embedding space is spherical\n\nit is also a known trick that batch size should be large enough for metric learning/retrival/face recognition problem/contrastive learning", "votes": null }, { "id": "1736096", "postDate": "03/27/2022 00:28:57", "content": "Is this for training or inference or both?
", "rawMarkdown": "Is this for training or inference or both?", "votes": null }, { "id": "1738614", "postDate": "03/29/2022 12:04:35", "content": "Yep, in original paper they say, that:
\n\n", "rawMarkdown": "Yep, in [original paper](https://arxiv.org/pdf/1801.07698.pdf) they say, that:\n> After the last convolutional layer, we explore the BN-Dropout-FC-BN structure to get the final 512-D embedding feature.", "votes": null }, { "id": "1741804", "postDate": "04/01/2022 07:10:07", "content": "After the last convolutional layer, we explore the BN-Dropout-FC-BN structure to get the final 512-D embedding feature.
\n
Batch norm parameters are being updated during training
\nIn the inference phase, the batch norm parameters called gamma and beta (which are learned in the training phase) and their running mean and std are fixed and will be used for computing the output.
", "rawMarkdown": "Batch norm parameters are being updated during training\n\nIn the inference phase, the batch norm parameters called gamma and beta (which are learned in the training phase) and their running mean and std are fixed and will be used for computing the output.", "votes": null }, { "id": "1750734", "postDate": "04/10/2022 02:52:51", "content": "Thank you for sharing great information!
\nCould you show me some example of kernels better than others, or give me hint to find better notebook.
\nI can't judge what notebook is better or not…
Most of the time there is no wrong or right in machine learning. Try all options and see what works best.
", "rawMarkdown": "Most of the time there is no wrong or right in machine learning. Try all options and see what works best.", "votes": null }, { "id": "1751128", "postDate": "04/10/2022 12:01:00", "content": "Indeed. There's still a lot to be studied, so it's hard to say if something is right or wrong.
", "rawMarkdown": "Indeed. There's still a lot to be studied, so it's hard to say if something is right or wrong.", "votes": null } ], "comments": [ { "id": 1735584, "author_name": "biglafe", "author_url": "", "post_date": "03/26/2022 12:29:26", "content": "The characteristics of this game are overfitting the training set. It is useful to add constraints and tricks at the feature level.
\nUsing bn layer and feature loss is equivalent to assuming that the feature space, increases the features to distinguish, also eased the overfitting.
i have a feeling that normalisation layer is smiliar is the PCA trick before deep learning metric learning is \"invented\".
\nIn traditional old ML methods, it is necessary to perform PCA to ensure that the distance is spherical in the embedding space. cosine distance is only meaningful if the embedding space is spherical
\nit is also a known trick that batch size should be large enough for metric learning/retrival/face recognition problem/contrastive learning
", "votes": null, "replies": [] } ] }, { "id": 1736096, "author_name": "joejeo1", "author_url": "", "post_date": "03/27/2022 00:28:57", "content": "Is this for training or inference or both?
", "votes": null, "replies": [ { "id": 1741804, "author_name": "ptran1203", "author_url": "", "post_date": "04/01/2022 07:10:07", "content": "Batch norm parameters are being updated during training
\nIn the inference phase, the batch norm parameters called gamma and beta (which are learned in the training phase) and their running mean and std are fixed and will be used for computing the output.
", "votes": null, "replies": [] } ] }, { "id": 1738614, "author_name": "vadbeg", "author_url": "", "post_date": "03/29/2022 12:04:35", "content": "Yep, in original paper they say, that:
\n\n", "votes": null, "replies": [] }, { "id": 1750734, "author_name": "whitelily", "author_url": "", "post_date": "04/10/2022 02:52:51", "content": "After the last convolutional layer, we explore the BN-Dropout-FC-BN structure to get the final 512-D embedding feature.
\n
Thank you for sharing great information!
\nCould you show me some example of kernels better than others, or give me hint to find better notebook.
\nI can't judge what notebook is better or not…
Most of the time there is no wrong or right in machine learning. Try all options and see what works best.
", "votes": null, "replies": [ { "id": 1751128, "author_name": "robsonsan", "author_url": "", "post_date": "04/10/2022 12:01:00", "content": "Indeed. There's still a lot to be studied, so it's hard to say if something is right or wrong.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1735555": "i spend time to investigate why some public kernel are better than others.\n\ni used one week to discover this:\n- norm (batchnorm or llayernorm) before L2 is necessary\n\nhttps://www.kaggle.com/c/humpback-whale-identification/discussion/82409\n\"BatchNorm vs LayerNorm before L2 normalization: LayerNorm better\"\n\nhttps://www.kaggle.com/c/recursion-cellular-image-classification/discussion/109987\n\"Batch normalization layer is vital just before the L2-normalized features multiplication by the weights. It is indeed mentioned in the original article, but I didn't pay attention at first, and ArcFace started working for me only after I added it.\"", "1735584": "The characteristics of this game are overfitting the training set. It is useful to add constraints and tricks at the feature level.\nUsing bn layer and feature loss is equivalent to assuming that the feature space, increases the features to distinguish, also eased the overfitting.", "1735647": "i have a feeling that normalisation layer is smiliar is the PCA trick before deep learning metric learning is \"invented\". \n\nIn traditional old ML methods, it is necessary to perform PCA to ensure that the distance is spherical in the embedding space. cosine distance is only meaningful if the embedding space is spherical\n\nit is also a known trick that batch size should be large enough for metric learning/retrival/face recognition problem/contrastive learning", "1736096": "Is this for training or inference or both?", "1738614": "Yep, in [original paper](https://arxiv.org/pdf/1801.07698.pdf) they say, that:\n> After the last convolutional layer, we explore the BN-Dropout-FC-BN structure to get the final 512-D embedding feature.", "1741804": "Batch norm parameters are being updated during training\n\nIn the inference phase, the batch norm parameters called gamma and beta (which are learned in the training phase) and their running mean and std are fixed and will be used for computing the output.", "1750734": "Thank you for sharing great information!\nCould you show me some example of kernels better than others, or give me hint to find better notebook.\nI can't judge what notebook is better or not...", "1750945": "Most of the time there is no wrong or right in machine learning. Try all options and see what works best.", "1751128": "Indeed. There's still a lot to be studied, so it's hard to say if something is right or wrong." }, "source": "meta" }