thanks to @raddar for valuble insights into the data patterns.
\nReference: here
I have created a notebook to illustrate the following patterns from the data through quantile graph representation.
\nThe values are clipped as follows for the series .
\nq01 = np.quantile(s, 0.01)
\nq99 = np.quantile(s, 0.99)
\nd = q99 - q01
\ns = np.clip(s, q01-2d, q99+2d)
Discreete values
\nWe are able to see few quantile graphs to have a sharp elbows in the quantile graph in case of outliers or having discreete values.
\nExample: S15 -> looks from the quantile graph to have atleast 7 discreete values.
Gaussian Distribution
\nP_2, D_47 from the Normal QQ plot looks to follow gaussian plot.
\nwhere as few other features R_13, R_17 will have high kurtosis
B_31 have single value across all the quantiles.
\nwith the reference from @cdeotte comment on R_26 has 6 discreete values when zooming-in. which is not showing clearly from my notebook. May need to remove the outliers and check, since the gap is so small.
\nLike R_26, many other features have a big jump in the outliers, that are inhibiting to show the other values like D_94, S_26.
\nplease let me know if any furthur processing, insights and interpretation which might be interesting to look at.
\nlink to my notebook
\nhere
thanks to @raddar for valuble insights into the data patterns.
\nReference: here
I have created a notebook to illustrate the following patterns from the data through quantile graph representation.
\nThe values are clipped as follows for the series .
\nq01 = np.quantile(s, 0.01)
\nq99 = np.quantile(s, 0.99)
\nd = q99 - q01
\ns = np.clip(s, q01-2d, q99+2d)
Discreete values
\nWe are able to see few quantile graphs to have a sharp elbows in the quantile graph in case of outliers or having discreete values.
\nExample: S15 -> looks from the quantile graph to have atleast 7 discreete values.
Gaussian Distribution
\nP_2, D_47 from the Normal QQ plot looks to follow gaussian plot.
\nwhere as few other features R_13, R_17 will have high kurtosis
B_31 have single value across all the quantiles.
\nwith the reference from @cdeotte comment on R_26 has 6 discreete values when zooming-in. which is not showing clearly from my notebook. May need to remove the outliers and check, since the gap is so small.
\nLike R_26, many other features have a big jump in the outliers, that are inhibiting to show the other values like D_94, S_26.
\nplease let me know if any furthur processing, insights and interpretation which might be interesting to look at.
\nlink to my notebook
\nhere
Link to the notebook is broken :)
", "rawMarkdown": "Link to the notebook is broken :)", "votes": null }, { "id": "1828458", "postDate": "06/21/2022 17:55:46", "content": "fixed. thnks
", "rawMarkdown": "fixed. thnks", "votes": null }, { "id": "1831133", "postDate": "06/24/2022 01:26:42", "content": "Hello doteeee, excellent work! As a statistician, I'm very happy when I see distributions, quantiles and so on.
\nSome ideas popped to my mind while I was reading your notebook:
\nShould we apply concave transformations (log, sqrt) to several variables ? A lot of them seems to have a quadratic -like shaped, all of them with a very discrepant point (in most cases, this point is 1.0). Applying this transformation,we could , in a way, \"soft\" the distributions, vanquishing the effects of the outliers and skewness. For a very few variables we could make the opposite: apply some transformation like x**2 or exp(x) for the same reasons.
From the plots, it seems that R9, S23, D51, D70, D82, D122, D136, D138 are counting variables, aren't they?
One could say we drop B_31, D_87 for having an unique value across all the quantiles but, if we fill the missing values of this variable, the distribution of the target change for missing/non-missing values ? Have you already done this analysis ?
Again, excellent analysis. Upvoted!
", "rawMarkdown": "Hello [doteeee](https://www.kaggle.com/narendra), excellent work! As a statistician, I'm very happy when I see distributions, quantiles and so on.\n\nSome ideas popped to my mind while I was reading your notebook:\n\n1. Should we apply concave transformations (log, sqrt) to several variables ? A lot of them seems to have a quadratic -like shaped, all of them with a very discrepant point (in most cases, this point is 1.0). Applying this transformation,we could , in a way, \"soft\" the distributions, vanquishing the effects of the outliers and skewness. For a very few variables we could make the opposite: apply some transformation like x**2 or exp(x) for the same reasons.\n\n2. From the plots, it seems that R9, S23, D51, D70, D82, D122, D136, D138 are counting variables, aren't they?\n\n3. One could say we drop B_31, D_87 for having an unique value across all the quantiles but, if we fill the missing values of this variable, the distribution of the target change for missing/non-missing values ? Have you already done this analysis ?\n\nAgain, excellent analysis. Upvoted!", "votes": null }, { "id": "1832471", "postDate": "06/25/2022 04:38:32", "content": "Hi @carlosasdesouza ,
\nShould we apply concave transformations (log, sqrt) to several variables ? A lot of them seems to have a quadratic -like shaped, all of them with a very discrepant point (in most cases, this point is 1.0). Applying this transformation,we could , in a way, \"soft\" the distributions, vanquishing the effects of the outliers and skewness. For a very few variables we could make the opposite: apply some transformation like x^^2 or exp(x) for the same reasons.
\nyes, i observe that few variables are deviating from the normal distribution due to the outliers. Along with the above said transaformations we can use quantile transformation and check. these yet to try.
From the plots, it seems that R9, S23, D51, D70, D82, D122, D136, D138 are counting variables, aren't they?
\nApart from S23, the rest looks to be counting variables. outliers effected the visual representation of the graph, you can look at the normal plot and say many of them are not zeros.
\nwhen we zoom-in [0.131, 0.14] --> uniform distribution, < 0 -> Left-Skewed.
\nThe sharp bends from the graph also represents possible outliers.
3.One could say we drop B_31, D_87 for having an unique value across all the quantiles but, if we fill the missing values of this variable, the distribution of the target change for missing/non-missing values ? Have you already done this analysis ?
\nD_87 --> I think we can remove this variable and check the performance of the model.
\n0.926% are missing values.
\nOf the missing values 0.751335 (target=0), 0.248665(target=1), which had lesser distingushing capacity, atleast from analysis. here.
B_31 --> some mistake of me representing the values, it actually had 2 values(0,1). we can check from the z-score, the range of values are different.
", "rawMarkdown": "Hi @carlosasdesouza , \n\n1. **Should we apply concave transformations (log, sqrt) to several variables ? A lot of them seems to have a quadratic -like shaped, all of them with a very discrepant point (in most cases, this point is 1.0). Applying this transformation,we could , in a way, \"soft\" the distributions, vanquishing the effects of the outliers and skewness. For a very few variables we could make the opposite: apply some transformation like x^^2 or exp(x) for the same reasons.**\n\n yes, i observe that few variables are deviating from the normal distribution due to the outliers. Along with the above said transaformations we can use quantile transformation and check. these yet to try.\n\n\n\n\n2. **From the plots, it seems that R9, S23, D51, D70, D82, D122, D136, D138 are counting variables, aren't they?**\nApart from S23, the rest looks to be counting variables. outliers effected the visual representation of the graph, you can look at the normal plot and say many of them are not zeros.\nwhen we zoom-in [0.131, 0.14] --> uniform distribution, < 0 -> Left-Skewed.\nThe sharp bends from the graph also represents possible outliers.\n\n\n\n\n3.**One could say we drop B_31, D_87 for having an unique value across all the quantiles but, if we fill the missing values of this variable, the distribution of the target change for missing/non-missing values ? Have you already done this analysis ?**\n\n\n D_87 --> I think we can remove this variable and check the performance of the model.\n0.926% are missing values. \nOf the missing values 0.751335 (target=0), 0.248665(target=1), which had lesser distingushing capacity, atleast from analysis. [here](https://www.kaggle.com/code/narendra/amex-high-level-analysis).\n\nB_31 --> some mistake of me representing the values, it actually had 2 values(0,1). we can check from the z-score, the range of values are different.", "votes": null } ], "comments": [ { "id": 1828450, "author_name": "raddar", "author_url": "", "post_date": "06/21/2022 17:35:35", "content": "Link to the notebook is broken :)
", "votes": null, "replies": [ { "id": 1828458, "author_name": "narendra", "author_url": "", "post_date": "06/21/2022 17:55:46", "content": "fixed. thnks
", "votes": null, "replies": [] } ] }, { "id": 1831133, "author_name": "carlosasdesouza", "author_url": "", "post_date": "06/24/2022 01:26:42", "content": "Hello doteeee, excellent work! As a statistician, I'm very happy when I see distributions, quantiles and so on.
\nSome ideas popped to my mind while I was reading your notebook:
\nShould we apply concave transformations (log, sqrt) to several variables ? A lot of them seems to have a quadratic -like shaped, all of them with a very discrepant point (in most cases, this point is 1.0). Applying this transformation,we could , in a way, \"soft\" the distributions, vanquishing the effects of the outliers and skewness. For a very few variables we could make the opposite: apply some transformation like x**2 or exp(x) for the same reasons.
From the plots, it seems that R9, S23, D51, D70, D82, D122, D136, D138 are counting variables, aren't they?
One could say we drop B_31, D_87 for having an unique value across all the quantiles but, if we fill the missing values of this variable, the distribution of the target change for missing/non-missing values ? Have you already done this analysis ?
Again, excellent analysis. Upvoted!
", "votes": null, "replies": [ { "id": 1832471, "author_name": "narendra", "author_url": "", "post_date": "06/25/2022 04:38:32", "content": "Hi @carlosasdesouza ,
\nShould we apply concave transformations (log, sqrt) to several variables ? A lot of them seems to have a quadratic -like shaped, all of them with a very discrepant point (in most cases, this point is 1.0). Applying this transformation,we could , in a way, \"soft\" the distributions, vanquishing the effects of the outliers and skewness. For a very few variables we could make the opposite: apply some transformation like x^^2 or exp(x) for the same reasons.
\nyes, i observe that few variables are deviating from the normal distribution due to the outliers. Along with the above said transaformations we can use quantile transformation and check. these yet to try.
From the plots, it seems that R9, S23, D51, D70, D82, D122, D136, D138 are counting variables, aren't they?
\nApart from S23, the rest looks to be counting variables. outliers effected the visual representation of the graph, you can look at the normal plot and say many of them are not zeros.
\nwhen we zoom-in [0.131, 0.14] --> uniform distribution, < 0 -> Left-Skewed.
\nThe sharp bends from the graph also represents possible outliers.
3.One could say we drop B_31, D_87 for having an unique value across all the quantiles but, if we fill the missing values of this variable, the distribution of the target change for missing/non-missing values ? Have you already done this analysis ?
\nD_87 --> I think we can remove this variable and check the performance of the model.
\n0.926% are missing values.
\nOf the missing values 0.751335 (target=0), 0.248665(target=1), which had lesser distingushing capacity, atleast from analysis. here.
B_31 --> some mistake of me representing the values, it actually had 2 values(0,1). we can check from the z-score, the range of values are different.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1828438": "thanks to @raddar for valuble insights into the data patterns. \nReference: [here](https://www.kaggle.com/competitions/amex-default-prediction/discussion/328514)\n\nI have created a notebook to illustrate the following patterns from the data through quantile graph representation.\n1. Highly discreete values \n2. Outlier detection.\n3. follows gaussian distribution/not\n4. skewness and kurtosis \n\n\n\n##### Quick Insights\n\nThe values are clipped as follows for the series .\nq01 = np.quantile(s, 0.01)\nq99 = np.quantile(s, 0.99)\nd = q99 - q01\ns = np.clip(s, q01-2*d, q99+2*d)\n\n\n**Discreete values**\nWe are able to see few quantile graphs to have a sharp elbows in the quantile graph in case of outliers or having discreete values.\nExample: S15 -> looks from the quantile graph to have atleast 7 discreete values.\n\n**Gaussian Distribution**\nP_2, D_47 from the Normal QQ plot looks to follow gaussian plot.\nwhere as few other features R_13, R_17 will have high kurtosis\n\nB_31 have single value across all the quantiles.\n\n\n\nwith the reference from @cdeotte comment on R_26 has 6 discreete values when zooming-in. which is not showing clearly from my notebook. May need to remove the outliers and check, since the gap is so small.\n\nLike R_26, many other features have a big jump in the outliers, that are inhibiting to show the other values like D_94, S_26.\n\n\nplease let me know if any furthur processing, insights and interpretation which might be interesting to look at.\n\nlink to my notebook \n[here](https://www.kaggle.com/code/narendra/understanding-features-with-quantiles)", "1828450": "Link to the notebook is broken :)", "1828458": "fixed. thnks", "1831133": "Hello [doteeee](https://www.kaggle.com/narendra), excellent work! As a statistician, I'm very happy when I see distributions, quantiles and so on.\n\nSome ideas popped to my mind while I was reading your notebook:\n\n1. Should we apply concave transformations (log, sqrt) to several variables ? A lot of them seems to have a quadratic -like shaped, all of them with a very discrepant point (in most cases, this point is 1.0). Applying this transformation,we could , in a way, \"soft\" the distributions, vanquishing the effects of the outliers and skewness. For a very few variables we could make the opposite: apply some transformation like x**2 or exp(x) for the same reasons.\n\n2. From the plots, it seems that R9, S23, D51, D70, D82, D122, D136, D138 are counting variables, aren't they?\n\n3. One could say we drop B_31, D_87 for having an unique value across all the quantiles but, if we fill the missing values of this variable, the distribution of the target change for missing/non-missing values ? Have you already done this analysis ?\n\nAgain, excellent analysis. Upvoted!", "1832471": "Hi @carlosasdesouza , \n\n1. **Should we apply concave transformations (log, sqrt) to several variables ? A lot of them seems to have a quadratic -like shaped, all of them with a very discrepant point (in most cases, this point is 1.0). Applying this transformation,we could , in a way, \"soft\" the distributions, vanquishing the effects of the outliers and skewness. For a very few variables we could make the opposite: apply some transformation like x^^2 or exp(x) for the same reasons.**\n\n yes, i observe that few variables are deviating from the normal distribution due to the outliers. Along with the above said transaformations we can use quantile transformation and check. these yet to try.\n\n\n\n\n2. **From the plots, it seems that R9, S23, D51, D70, D82, D122, D136, D138 are counting variables, aren't they?**\nApart from S23, the rest looks to be counting variables. outliers effected the visual representation of the graph, you can look at the normal plot and say many of them are not zeros.\nwhen we zoom-in [0.131, 0.14] --> uniform distribution, < 0 -> Left-Skewed.\nThe sharp bends from the graph also represents possible outliers.\n\n\n\n\n3.**One could say we drop B_31, D_87 for having an unique value across all the quantiles but, if we fill the missing values of this variable, the distribution of the target change for missing/non-missing values ? Have you already done this analysis ?**\n\n\n D_87 --> I think we can remove this variable and check the performance of the model.\n0.926% are missing values. \nOf the missing values 0.751335 (target=0), 0.248665(target=1), which had lesser distingushing capacity, atleast from analysis. [here](https://www.kaggle.com/code/narendra/amex-high-level-analysis).\n\nB_31 --> some mistake of me representing the values, it actually had 2 values(0,1). we can check from the z-score, the range of values are different." }, "source": "meta" }