I need help applying a function over a moving window. How to do this? Suppose I have an array shape of (n, m) and I need to apply a function lets say mean, std, min, max, etc. to a window of 10. What is the code? I need an output array of length of 1000, each containing the function applied over the 10 period hypothetical window. Is there a numpy function for this? Thanks!
\n\nconsider array:
\n\n> np.ones(10000)
\n\nFull Disclosure: I'm also participating in a kaggle research study so I need a couple of instances where I asked for help and received it.
", "messages": [ { "id": "537937", "postDate": "05/27/2019 22:08:26", "content": "I need help applying a function over a moving window. How to do this? Suppose I have an array shape of (n, m) and I need to apply a function lets say mean, std, min, max, etc. to a window of 10. What is the code? I need an output array of length of 1000, each containing the function applied over the 10 period hypothetical window. Is there a numpy function for this? Thanks!
\n\nconsider array:
\n\n> np.ones(10000)
\n\nFull Disclosure: I'm also participating in a kaggle research study so I need a couple of instances where I asked for help and received it.
", "rawMarkdown": "I need help applying a function over a moving window. How to do this? Suppose I have an array shape of (n, m) and I need to apply a function lets say mean, std, min, max, etc. to a window of 10. What is the code? I need an output array of length of 1000, each containing the function applied over the 10 period hypothetical window. Is there a numpy function for this? Thanks!\n\nconsider array:\n\n> np.ones(10000)\n\n\n\n__Full Disclosure__: I'm also participating in a kaggle research study so I need a couple of instances where I asked for help and received it.", "votes": null }, { "id": "537941", "postDate": "05/27/2019 22:16:27", "content": "If you don't write your own moving window computation, try the pandas.DataFrame.rolling:\nhttps://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.rolling.html
\n\nFor example, df.rolling(2).sum().
Thank you. I think I'm going to try a custom function
", "rawMarkdown": "Thank you. I think I'm going to try [a custom function](https://rigtorp.se/2011/01/01/rolling-statistics-numpy.html)", "votes": null }, { "id": "538133", "postDate": "05/28/2019 07:44:40", "content": "You can do a custom function using rolling\nHere is an example https://www.kaggle.com/scirpus/earthquake-gp-filter\nYou are mad not to use rolling in pandas as this is performance optimized. Writing it from scratch is a bit of a waste of time IMHO.
", "rawMarkdown": "You can do a custom function using rolling\nHere is an example https://www.kaggle.com/scirpus/earthquake-gp-filter\nYou are mad not to use rolling in pandas as this is performance optimized. Writing it from scratch is a bit of a waste of time IMHO.", "votes": null }, { "id": "538159", "postDate": "05/28/2019 08:43:57", "content": "Thank you. Do you have any basic benchmarks performed verifying this claim? I'm not doubting you, just, healthy skepticism ;). The only thing I really need is the implementation of a variable stride, and the ability to modify the datatype of the rolling output. I've ran into memory issues with certain functions, and high window sizes.
", "rawMarkdown": "Thank you. Do you have any basic benchmarks performed verifying this claim? I'm not doubting you, just, healthy skepticism ;). The only thing I really need is the implementation of a variable stride, and the ability to modify the datatype of the rolling output. I've ran into memory issues with certain functions, and high window sizes.", "votes": null }, { "id": "538173", "postDate": "05/28/2019 09:10:01", "content": "You could look at pandas code as it is open source.
\n\nStart here for roll your own https://stackoverflow.com/questions/25583494/faster-rolling-apply-on-a-pandas-dataframe (pun intended)
", "rawMarkdown": "You could look at pandas code as it is open source.\n\nStart here for roll your own https://stackoverflow.com/questions/25583494/faster-rolling-apply-on-a-pandas-dataframe (pun intended)", "votes": null }, { "id": "539287", "postDate": "05/29/2019 20:35:24", "content": "Weird how my question and request for help could get downvoted. I'm guessing it's someone who found it jeapordizing their chance at winning. That's good to know.
", "rawMarkdown": "Weird how my question and request for help could get downvoted. I'm guessing it's someone who found it jeapordizing their chance at winning. That's good to know.", "votes": null } ], "comments": [ { "id": 537941, "author_name": "pukkinming", "author_url": "", "post_date": "05/27/2019 22:16:27", "content": "If you don't write your own moving window computation, try the pandas.DataFrame.rolling:\nhttps://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.rolling.html
\n\nFor example, df.rolling(2).sum().
Thank you. I think I'm going to try a custom function
", "votes": null, "replies": [] } ] }, { "id": 538133, "author_name": "scirpus", "author_url": "", "post_date": "05/28/2019 07:44:40", "content": "You can do a custom function using rolling\nHere is an example https://www.kaggle.com/scirpus/earthquake-gp-filter\nYou are mad not to use rolling in pandas as this is performance optimized. Writing it from scratch is a bit of a waste of time IMHO.
", "votes": null, "replies": [ { "id": 538159, "author_name": "teeyee314", "author_url": "", "post_date": "05/28/2019 08:43:57", "content": "Thank you. Do you have any basic benchmarks performed verifying this claim? I'm not doubting you, just, healthy skepticism ;). The only thing I really need is the implementation of a variable stride, and the ability to modify the datatype of the rolling output. I've ran into memory issues with certain functions, and high window sizes.
", "votes": null, "replies": [] }, { "id": 538173, "author_name": "scirpus", "author_url": "", "post_date": "05/28/2019 09:10:01", "content": "You could look at pandas code as it is open source.
\n\nStart here for roll your own https://stackoverflow.com/questions/25583494/faster-rolling-apply-on-a-pandas-dataframe (pun intended)
", "votes": null, "replies": [] } ] }, { "id": 539287, "author_name": "teeyee314", "author_url": "", "post_date": "05/29/2019 20:35:24", "content": "Weird how my question and request for help could get downvoted. I'm guessing it's someone who found it jeapordizing their chance at winning. That's good to know.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "537937": "I need help applying a function over a moving window. How to do this? Suppose I have an array shape of (n, m) and I need to apply a function lets say mean, std, min, max, etc. to a window of 10. What is the code? I need an output array of length of 1000, each containing the function applied over the 10 period hypothetical window. Is there a numpy function for this? Thanks!\n\nconsider array:\n\n> np.ones(10000)\n\n\n\n__Full Disclosure__: I'm also participating in a kaggle research study so I need a couple of instances where I asked for help and received it.", "537941": "If you don't write your own moving window computation, try the pandas.DataFrame.rolling:\nhttps://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.rolling.html\n\nFor example, `df.rolling(2).sum()`.", "537942": "Thank you. I think I'm going to try [a custom function](https://rigtorp.se/2011/01/01/rolling-statistics-numpy.html)", "538133": "You can do a custom function using rolling\nHere is an example https://www.kaggle.com/scirpus/earthquake-gp-filter\nYou are mad not to use rolling in pandas as this is performance optimized. Writing it from scratch is a bit of a waste of time IMHO.", "538159": "Thank you. Do you have any basic benchmarks performed verifying this claim? I'm not doubting you, just, healthy skepticism ;). The only thing I really need is the implementation of a variable stride, and the ability to modify the datatype of the rolling output. I've ran into memory issues with certain functions, and high window sizes.", "538173": "You could look at pandas code as it is open source.\n\nStart here for roll your own https://stackoverflow.com/questions/25583494/faster-rolling-apply-on-a-pandas-dataframe (pun intended)", "539287": "Weird how my question and request for help could get downvoted. I'm guessing it's someone who found it jeapordizing their chance at winning. That's good to know." }, "source": "meta" }