In case of a few segment ids, all values of some sensors are missing, maybe due to sensor malfunction. I've tried imputing with single mean values of other sensors and row mean(axis=1). In both cases, the performance decreases in comparison to leaving NaNs as it is.
\nHow are you dealing with NaNs ?
", "messages": [ { "id": 1110073, "postDate": "2020-12-12T11:59:24.397Z", "content": "hi,
\nI'm still trying to impute missing values right. I am working on the calculated characteristics and first I tried KNN, I had very bad results. Then with median and improved a little. now i am using the mean and it improved another bit. I'm still trying to improve this though.
Hey, thanks for sharing.
\nIn case where all values of a sensor column are missing, I tried imputing with row mean. The results were bad. How are you doing in cases where the entire column is NaNs ?
", "rawMarkdown": "Hey, thanks for sharing. \n\nIn case where all values of a sensor column are missing, I tried imputing with row mean. The results were bad. How are you doing in cases where the entire column is NaNs ?" }, { "id": 1110592, "postDate": "2020-12-12T22:21:48.993Z", "content": "Are you working directly on the time series or on created features?
\nIf it is about time series, I don't know if there is a way to make a good estimate for impute when all data is missing. In the case of partial losses you could do mean, median or linear regression (handle-missing-values-in-time-series-for-beginners)
\nIf it is with created characteristics, it is possible to make an impute of a data using the mean or median of the data column, also something a little more complex is to estimate with random forest or knn.
\nIf all the data in a column are NA, you should check how you are calculating the feature and / or delete it. Because it is strange that a column of features created from a time series returns all values as NA.
I am working on created features.
\nAnd all values are NA in case of some segments of time series data, not the created features. I tried imputing the NAs in created features with mean and median but no improvement.
\nMy cv score is in the range of 2.5M but lb is 5M+.
", "rawMarkdown": "I am working on created features. \n\nAnd all values are NA in case of some segments of time series data, not the created features. I tried imputing the NAs in created features with mean and median but no improvement. \n\nMy cv score is in the range of 2.5M but lb is 5M+." }, { "id": 1113879, "postDate": "2020-12-15T18:50:24.183Z", "content": "The problem may not necessarily just be NAs, it may also be overfitting.
\nAs the training set is so small (approx. 4000 rows) you have to check very well how you are training, the sampling method, the outliers, etc.
\nIf it is the NAs that cause you the most problems, you would have to work without these observations, but losing information.
\nIn my case I am using impute to mean, plus impute outliers, this increased the performance a bit from 5.0M to 4.7M, but I still have a lot of overfitting.
\nNow I am fixed at 4.7M and I have not been able to get out of there.
", "rawMarkdown": "The problem may not necessarily just be NAs, it may also be overfitting.\n\nAs the training set is so small (approx. 4000 rows) you have to check very well how you are training, the sampling method, the outliers, etc.\n\nIf it is the NAs that cause you the most problems, you would have to work without these observations, but losing information.\n\nIn my case I am using impute to mean, plus impute outliers, this increased the performance a bit from 5.0M to 4.7M, but I still have a lot of overfitting.\n\nNow I am fixed at 4.7M and I have not been able to get out of there." } ] }, { "id": 1110004, "postDate": "2020-12-12T10:07:57.803Z", "content": "In case of a few segment ids, all values of some sensors are missing, maybe due to sensor malfunction. I've tried imputing with single mean values of other sensors and row mean(axis=1). In both cases, the performance decreases in comparison to leaving NaNs as it is.
\nHow are you dealing with NaNs ?
", "rawMarkdown": "In case of a few segment ids, all values of some sensors are missing, maybe due to sensor malfunction. I've tried imputing with single mean values of other sensors and row mean(axis=1). In both cases, the performance decreases in comparison to leaving NaNs as it is. \n\nHow are you dealing with NaNs ? " } ], "comments": [ { "id": 1110073, "author_name": "Desareca", "author_url": "", "post_date": "2020-12-12T11:59:24.397000", "content": "hi,
\nI'm still trying to impute missing values right. I am working on the calculated characteristics and first I tried KNN, I had very bad results. Then with median and improved a little. now i am using the mean and it improved another bit. I'm still trying to improve this though.
Hey, thanks for sharing.
\nIn case where all values of a sensor column are missing, I tried imputing with row mean. The results were bad. How are you doing in cases where the entire column is NaNs ?
", "votes": 0, "replies": [] }, { "id": 1110592, "author_name": "Desareca", "author_url": "", "post_date": "2020-12-12T22:21:48.993000", "content": "Are you working directly on the time series or on created features?
\nIf it is about time series, I don't know if there is a way to make a good estimate for impute when all data is missing. In the case of partial losses you could do mean, median or linear regression (handle-missing-values-in-time-series-for-beginners)
\nIf it is with created characteristics, it is possible to make an impute of a data using the mean or median of the data column, also something a little more complex is to estimate with random forest or knn.
\nIf all the data in a column are NA, you should check how you are calculating the feature and / or delete it. Because it is strange that a column of features created from a time series returns all values as NA.
I am working on created features.
\nAnd all values are NA in case of some segments of time series data, not the created features. I tried imputing the NAs in created features with mean and median but no improvement.
\nMy cv score is in the range of 2.5M but lb is 5M+.
", "votes": 0, "replies": [] }, { "id": 1113879, "author_name": "Desareca", "author_url": "", "post_date": "2020-12-15T18:50:24.183000", "content": "The problem may not necessarily just be NAs, it may also be overfitting.
\nAs the training set is so small (approx. 4000 rows) you have to check very well how you are training, the sampling method, the outliers, etc.
\nIf it is the NAs that cause you the most problems, you would have to work without these observations, but losing information.
\nIn my case I am using impute to mean, plus impute outliers, this increased the performance a bit from 5.0M to 4.7M, but I still have a lot of overfitting.
\nNow I am fixed at 4.7M and I have not been able to get out of there.
", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "1110073": "hi, \nI'm still trying to impute missing values right. I am working on the calculated characteristics and first I tried KNN, I had very bad results. Then with median and improved a little. now i am using the mean and it improved another bit. I'm still trying to improve this though.", "1110004": "In case of a few segment ids, all values of some sensors are missing, maybe due to sensor malfunction. I've tried imputing with single mean values of other sensors and row mean(axis=1). In both cases, the performance decreases in comparison to leaving NaNs as it is. \n\nHow are you dealing with NaNs ? " } }