I was planning to add a new feature for each numeric feature to tell the model whether a sample for that feature is an outlier or not. So, basically I thought the new feature would have a 1 against samples that are outliers and 0 against samples that aren't. Since we're supposed to detect fraud, I thought it would be useful to let the model know which values are unusual. Do you guys think this would be useful? I would love to hear your thoughts. I'm sorry if the question is stupid.
\nThanks.
", "messages": [ { "id": "1850757", "postDate": "07/10/2022 17:54:13", "content": "I was planning to add a new feature for each numeric feature to tell the model whether a sample for that feature is an outlier or not. So, basically I thought the new feature would have a 1 against samples that are outliers and 0 against samples that aren't. Since we're supposed to detect fraud, I thought it would be useful to let the model know which values are unusual. Do you guys think this would be useful? I would love to hear your thoughts. I'm sorry if the question is stupid.
\nThanks.
", "rawMarkdown": "I was planning to add a new feature for each numeric feature to tell the model whether a sample for that feature is an outlier or not. So, basically I thought the new feature would have a 1 against samples that are outliers and 0 against samples that aren't. Since we're supposed to detect fraud, I thought it would be useful to let the model know which values are unusual. Do you guys think this would be useful? I would love to hear your thoughts. I'm sorry if the question is stupid. \n\nThanks.", "votes": null }, { "id": "1850778", "postDate": "07/10/2022 18:22:54", "content": "I feel this is very good idea.
", "rawMarkdown": "I feel this is very good idea.", "votes": null }, { "id": "1850781", "postDate": "07/10/2022 18:28:00", "content": "I'll try it out. Thanks.
", "rawMarkdown": "I'll try it out. Thanks.", "votes": null }, { "id": "1851004", "postDate": "07/11/2022 01:11:48", "content": "This could definitely be useful, although it might not be necessary to create a new feature for each numeric feature - you could also generate a single outlier feature that is based on all of the numeric features.
\nOutliers can be tricky to deal with, so it's definitely worth experiment with different approaches to see what works best on your data.
\nGood luck!
Thanks…I was actually planning to calculate Z-score across each feature and have an outlier feature for each individual feature. I don't understand how I could replace the outliers with a single feature? Can you please explain about that?
", "rawMarkdown": "Thanks...I was actually planning to calculate Z-score across each feature and have an outlier feature for each individual feature. I don't understand how I could replace the outliers with a single feature? Can you please explain about that?", "votes": null }, { "id": "1851353", "postDate": "07/11/2022 07:23:35", "content": "That is a great idea. we know that outliers can be problematic for boosting methods (because boosting builds each tree on previous trees' residuals/errors). I suggest two experiments:
\nAssuming that your model is based on decision trees, I doubt that the additional features would be useful. A decision tree detects by itself on what feature and what threshold for this feature it has to branch. A boolean feature for outliers doesn't provide any additional value to the algorithm.
\nIf you belong to the minority who uses linear models for this competition 😏, the situation would be different: Linear models are very bad at dealing with outliers, and you'd have to apply suitable feature engineering.
", "rawMarkdown": "Assuming that your model is based on decision trees, I doubt that the additional features would be useful. A decision tree detects by itself on what feature and what threshold for this feature it has to branch. A boolean feature for outliers doesn't provide any additional value to the algorithm.\n\nIf you belong to the minority who uses linear models for this competition 😏, the situation would be different: Linear models are very bad at dealing with outliers, and you'd have to apply suitable feature engineering.", "votes": null }, { "id": "1851464", "postDate": "07/11/2022 09:04:47", "content": "Thanks..I'll try both. I'm already trying 2nd actually..After that I'll try first one too.
", "rawMarkdown": "Thanks..I'll try both. I'm already trying 2nd actually..After that I'll try first one too.", "votes": null }, { "id": "1851468", "postDate": "07/11/2022 09:11:48", "content": "Thanks for the tip. I'm certainly not using linear models😄
", "rawMarkdown": "Thanks for the tip. I'm certainly not using linear models😄", "votes": null }, { "id": "1851894", "postDate": "07/11/2022 15:58:49", "content": "Was thinking on similar lines .I assume that one that @ragnar123 showcased is basically like a difference from mean X_last - Mean .. which seems to be have some resemblance with Z scores and detecting anomalies that way . I might be wrong but this is the only possible reasoning I thought .
\nBut as Ambrose commented boosting models are really good at figuring those out as we have seen with some folks generating a lot of em and boosting models finding the best fit .
", "rawMarkdown": "Was thinking on similar lines .I assume that one that @ragnar123 showcased is basically like a difference from mean *X_last - Mean* .. which seems to be have some resemblance with Z scores and detecting anomalies that way . I might be wrong but this is the only possible reasoning I thought .\n\nBut as Ambrose commented boosting models are really good at figuring those out as we have seen with some folks generating a lot of em and boosting models finding the best fit .", "votes": null }, { "id": "1851968", "postDate": "07/11/2022 17:16:47", "content": "I believe you are thinking to implement something similar to : Calculate Z score. If Z score>3, print it as an outlier
\nBut before doing this we need to check if the distribution is normal or not, if normal distribution then only outlier detection using z-score can be implemented. - others can comment and add upon this.
\nNote : not sure if we can apply the same with non-normal distribution, as we have most of the numerical feature in this competition are not normally distributed.
But AI/ML is all about experimenting , so go for it. All the best.
", "rawMarkdown": "I believe you are thinking to implement something similar to : **Calculate Z score. If Z score>3, print it as an outlier**\nBut before doing this we need to check if the distribution is normal or not, if normal distribution then only outlier detection using z-score can be implemented. - others can comment and add upon this.\nNote : not sure if we can apply the same with non-normal distribution, as we have most of the numerical feature in this competition are not normally distributed.\n\nBut AI/ML is all about experimenting , so go for it. All the best.", "votes": null }, { "id": "1851971", "postDate": "07/11/2022 17:17:31", "content": "good luck!
", "rawMarkdown": "good luck!", "votes": null } ], "comments": [ { "id": 1850778, "author_name": "maheshak04", "author_url": "", "post_date": "07/10/2022 18:22:54", "content": "I feel this is very good idea.
", "votes": null, "replies": [ { "id": 1850781, "author_name": "vivekjan14", "author_url": "", "post_date": "07/10/2022 18:28:00", "content": "I'll try it out. Thanks.
", "votes": null, "replies": [] } ] }, { "id": 1851004, "author_name": "thedevastator", "author_url": "", "post_date": "07/11/2022 01:11:48", "content": "This could definitely be useful, although it might not be necessary to create a new feature for each numeric feature - you could also generate a single outlier feature that is based on all of the numeric features.
\nOutliers can be tricky to deal with, so it's definitely worth experiment with different approaches to see what works best on your data.
\nGood luck!
Thanks…I was actually planning to calculate Z-score across each feature and have an outlier feature for each individual feature. I don't understand how I could replace the outliers with a single feature? Can you please explain about that?
", "votes": null, "replies": [] }, { "id": 1851968, "author_name": "abhianalytic", "author_url": "", "post_date": "07/11/2022 17:16:47", "content": "I believe you are thinking to implement something similar to : Calculate Z score. If Z score>3, print it as an outlier
\nBut before doing this we need to check if the distribution is normal or not, if normal distribution then only outlier detection using z-score can be implemented. - others can comment and add upon this.
\nNote : not sure if we can apply the same with non-normal distribution, as we have most of the numerical feature in this competition are not normally distributed.
But AI/ML is all about experimenting , so go for it. All the best.
", "votes": null, "replies": [] } ] }, { "id": 1851353, "author_name": "mohammadrahmati", "author_url": "", "post_date": "07/11/2022 07:23:35", "content": "That is a great idea. we know that outliers can be problematic for boosting methods (because boosting builds each tree on previous trees' residuals/errors). I suggest two experiments:
\nThanks..I'll try both. I'm already trying 2nd actually..After that I'll try first one too.
", "votes": null, "replies": [] }, { "id": 1851971, "author_name": "mohammadrahmati", "author_url": "", "post_date": "07/11/2022 17:17:31", "content": "good luck!
", "votes": null, "replies": [] } ] }, { "id": 1851369, "author_name": "ambrosm", "author_url": "", "post_date": "07/11/2022 07:34:40", "content": "Assuming that your model is based on decision trees, I doubt that the additional features would be useful. A decision tree detects by itself on what feature and what threshold for this feature it has to branch. A boolean feature for outliers doesn't provide any additional value to the algorithm.
\nIf you belong to the minority who uses linear models for this competition 😏, the situation would be different: Linear models are very bad at dealing with outliers, and you'd have to apply suitable feature engineering.
", "votes": null, "replies": [ { "id": 1851468, "author_name": "vivekjan14", "author_url": "", "post_date": "07/11/2022 09:11:48", "content": "Thanks for the tip. I'm certainly not using linear models😄
", "votes": null, "replies": [] } ] }, { "id": 1851894, "author_name": "gauravbrills", "author_url": "", "post_date": "07/11/2022 15:58:49", "content": "Was thinking on similar lines .I assume that one that @ragnar123 showcased is basically like a difference from mean X_last - Mean .. which seems to be have some resemblance with Z scores and detecting anomalies that way . I might be wrong but this is the only possible reasoning I thought .
\nBut as Ambrose commented boosting models are really good at figuring those out as we have seen with some folks generating a lot of em and boosting models finding the best fit .
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1850757": "I was planning to add a new feature for each numeric feature to tell the model whether a sample for that feature is an outlier or not. So, basically I thought the new feature would have a 1 against samples that are outliers and 0 against samples that aren't. Since we're supposed to detect fraud, I thought it would be useful to let the model know which values are unusual. Do you guys think this would be useful? I would love to hear your thoughts. I'm sorry if the question is stupid. \n\nThanks.", "1850778": "I feel this is very good idea.", "1850781": "I'll try it out. Thanks.", "1851004": "This could definitely be useful, although it might not be necessary to create a new feature for each numeric feature - you could also generate a single outlier feature that is based on all of the numeric features.\nOutliers can be tricky to deal with, so it's definitely worth experiment with different approaches to see what works best on your data.\nGood luck!", "1851192": "Thanks...I was actually planning to calculate Z-score across each feature and have an outlier feature for each individual feature. I don't understand how I could replace the outliers with a single feature? Can you please explain about that?", "1851353": "That is a great idea. we know that outliers can be problematic for boosting methods (because boosting builds each tree on previous trees' residuals/errors). I suggest two experiments: \n1. remove the outliers and run your baseline code with 3 different seed and average CV scores\n2. add new features stating if there is outlier or not\nI feel the first option will give you better results.", "1851369": "Assuming that your model is based on decision trees, I doubt that the additional features would be useful. A decision tree detects by itself on what feature and what threshold for this feature it has to branch. A boolean feature for outliers doesn't provide any additional value to the algorithm.\n\nIf you belong to the minority who uses linear models for this competition 😏, the situation would be different: Linear models are very bad at dealing with outliers, and you'd have to apply suitable feature engineering.", "1851464": "Thanks..I'll try both. I'm already trying 2nd actually..After that I'll try first one too.", "1851468": "Thanks for the tip. I'm certainly not using linear models😄", "1851894": "Was thinking on similar lines .I assume that one that @ragnar123 showcased is basically like a difference from mean *X_last - Mean* .. which seems to be have some resemblance with Z scores and detecting anomalies that way . I might be wrong but this is the only possible reasoning I thought .\n\nBut as Ambrose commented boosting models are really good at figuring those out as we have seen with some folks generating a lot of em and boosting models finding the best fit .", "1851968": "I believe you are thinking to implement something similar to : **Calculate Z score. If Z score>3, print it as an outlier**\nBut before doing this we need to check if the distribution is normal or not, if normal distribution then only outlier detection using z-score can be implemented. - others can comment and add upon this.\nNote : not sure if we can apply the same with non-normal distribution, as we have most of the numerical feature in this competition are not normally distributed.\n\nBut AI/ML is all about experimenting , so go for it. All the best.", "1851971": "good luck!" }, "source": "meta" }