\nAs we can the case in our problem too where the individuals histogram shows us the unbalance of training samples
\n
![\"\"](\"https://i.imgur.com/5NxHJOD.png\")
\nSo, what can we do to compensate this ? Most easy to implement solutions are:
As we know that the class label assigned to a query vector is determined by a majority vote amongst the k-nearest neighbors of the query vector. For simplicity, let us assume that we are using Euclidean distance as the distance metric here and its a binary classification problem. The inductive bias of knn classification in this case assumes that the space of all samples can be partitioned into two disjoint subspaces. If training set is large enough and balanced, it'd seem as if the subspaces were sampled uniformly for generating the training set. If the training set were to be imbalanced, and if the uniform distribution assumption were to still hold, the probability that the k nearest neighbors of any random query point will belong to the class with more examples becomes higher. So, the closest neighbor of the query point may still belong to the class with less examples, but if rest (k-1) points belong to the other class (because of its higher density in the space), the point will get misclassified.
\nAs we can the case in our problem too where the individuals histogram shows us the unbalance of training samples
\n
\nSo, what can we do to compensate this ? Most easy to implement solutions are:
But there are much more solutions and/or classifiers that can help. Below are 2 links with generical approaches when our targets are imbalanced
\nBiography:
\nhttps://www.site.uottawa.ca/~nat/Workshop2003/jzhang.pdf
\nhttps://stats.stackexchange.com/questions/341/knn-and-unbalanced-classes
\nhttps://www.quora.com/Why-does-knn-get-effected-by-the-class-imbalance
As we know that the class label assigned to a query vector is determined by a majority vote amongst the k-nearest neighbors of the query vector. For simplicity, let us assume that we are using Euclidean distance as the distance metric here and its a binary classification problem. The inductive bias of knn classification in this case assumes that the space of all samples can be partitioned into two disjoint subspaces. If training set is large enough and balanced, it'd seem as if the subspaces were sampled uniformly for generating the training set. If the training set were to be imbalanced, and if the uniform distribution assumption were to still hold, the probability that the k nearest neighbors of any random query point will belong to the class with more examples becomes higher. So, the closest neighbor of the query point may still belong to the class with less examples, but if rest (k-1) points belong to the other class (because of its higher density in the space), the point will get misclassified.
\nAs we can the case in our problem too where the individuals histogram shows us the unbalance of training samples
\n
\nSo, what can we do to compensate this ? Most easy to implement solutions are:
But there are much more solutions and/or classifiers that can help. Below are 2 links with generical approaches when our targets are imbalanced
\nBiography:
\nhttps://www.site.uottawa.ca/~nat/Workshop2003/jzhang.pdf
\nhttps://stats.stackexchange.com/questions/341/knn-and-unbalanced-classes
\nhttps://www.quora.com/Why-does-knn-get-effected-by-the-class-imbalance
Can we just calculate the centroid of all clusters? Assuming that all clusters are spheres, will it bring the same result as a uniformly balanced dataset?
", "rawMarkdown": "Can we just calculate the centroid of all clusters? Assuming that all clusters are spheres, will it bring the same result as a uniformly balanced dataset?", "votes": null }, { "id": "1697328", "postDate": "02/19/2022 14:42:07", "content": "ICYMI: Grandmaster Awsaf has made a neat kernel showcasing the distributions
", "rawMarkdown": "ICYMI: Grandmaster Awsaf has made a neat [kernel](https://www.kaggle.com/awsaf49/happywhale-data-distribution) showcasing the distributions", "votes": null }, { "id": "1697376", "postDate": "02/19/2022 15:18:59", "content": "@meowmeowmeowmeowmeow I am not sure that I understand but we may think of similar solutions. I am thinking of making just one data point for every individual_id by making for example the mean (for every dimension of the 512) of all samples for that individual so in this way every individual will have just one point (considered the centroid) then for inference calculate the distance from the unknown embedding to each of the centroids and classify regarding that distance
", "rawMarkdown": "meowmeowmeowmeowmeow I am not sure that I understand but we may think of similar solutions. I am thinking of making just one data point for every individual_id by making for example the mean (for every dimension of the 512) of all samples for that individual so in this way every individual will have just one point (considered the centroid) then for inference calculate the distance from the unknown embedding to each of the centroids and classify regarding that distance", "votes": null } ], "comments": [ { "id": 1697316, "author_name": "meowmeowmeowmeowmeow", "author_url": "", "post_date": "02/19/2022 14:28:41", "content": "Can we just calculate the centroid of all clusters? Assuming that all clusters are spheres, will it bring the same result as a uniformly balanced dataset?
", "votes": null, "replies": [ { "id": 1697328, "author_name": "init27", "author_url": "", "post_date": "02/19/2022 14:42:07", "content": "ICYMI: Grandmaster Awsaf has made a neat kernel showcasing the distributions
", "votes": null, "replies": [] }, { "id": 1697376, "author_name": "vladvdv", "author_url": "", "post_date": "02/19/2022 15:18:59", "content": "@meowmeowmeowmeowmeow I am not sure that I understand but we may think of similar solutions. I am thinking of making just one data point for every individual_id by making for example the mean (for every dimension of the 512) of all samples for that individual so in this way every individual will have just one point (considered the centroid) then for inference calculate the distance from the unknown embedding to each of the centroids and classify regarding that distance
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1696935": "As we know that the class label assigned to a query vector is determined by a majority vote amongst the k-nearest neighbors of the query vector. For simplicity, let us assume that we are using Euclidean distance as the distance metric here and its a binary classification problem. The inductive bias of knn classification in this case assumes that the space of all samples can be partitioned into two disjoint subspaces. If training set is large enough and balanced, it'd seem as if the subspaces were sampled uniformly for generating the training set. If the training set were to be imbalanced, and if the uniform distribution assumption were to still hold, the probability that the k nearest neighbors of any random query point will belong to the class with more examples becomes higher. So, the closest neighbor of the query point may still belong to the class with less examples, but if rest (k-1) points belong to the other class (because of its higher density in the space), the point will get misclassified. \nAs we can the case in our problem too where the individuals histogram shows us the unbalance of training samples\n\nSo, what can we do to compensate this ? Most easy to implement solutions are:\n\n- weighting neighbors by the inverse of their class size converts neighbor counts into the fraction of each class that falls in your K nearest neighbors\n- weighting neighbors by their distances\n- using a radius-based rule for gathering neighbors instead of the K nearest ones (often implemented in KNN packages)\n\nBut there are much more solutions and/or classifiers that can help. Below are 2 links with generical approaches when our targets are imbalanced\n- https://machinelearningmastery.com/tactics-to-combat-imbalanced-classes-in-your-machine-learning-dataset/\n- https://elitedatascience.com/imbalanced-classes\n\nBiography:\nhttps://www.site.uottawa.ca/~nat/Workshop2003/jzhang.pdf\nhttps://stats.stackexchange.com/questions/341/knn-and-unbalanced-classes\nhttps://www.quora.com/Why-does-knn-get-effected-by-the-class-imbalance", "1697316": "Can we just calculate the centroid of all clusters? Assuming that all clusters are spheres, will it bring the same result as a uniformly balanced dataset?", "1697328": "ICYMI: Grandmaster Awsaf has made a neat [kernel](https://www.kaggle.com/awsaf49/happywhale-data-distribution) showcasing the distributions", "1697376": "meowmeowmeowmeowmeow I am not sure that I understand but we may think of similar solutions. I am thinking of making just one data point for every individual_id by making for example the mean (for every dimension of the 512) of all samples for that individual so in this way every individual will have just one point (considered the centroid) then for inference calculate the distance from the unknown embedding to each of the centroids and classify regarding that distance" }, "source": "meta" }