{ "id": 237900, "title": "How is circular/cyclic data (e.g. month, longitude, etc.) represented in ML?", "url": "/competitions/birdclef-2021/discussion/237900", "author_name": "", "post_date": "2021-05-10T15:51:57.726069500Z", "votes": 4, "comment_count": 11, "views": 0, "content": "

Hi everyone, I haven't done many competitions and it is not until this time that I came to wonder

\n
\n

How to better represent cyclic data like month of the year, longitude on Earth, etc. in a way that might reveal that December is actually quite close to January and 179 degree is closer to -179 degree than 150 degree (longitude)?

\n
\n

I never thought about this, and would like to hear other people's idea, or just seeking for a bit of an interaction :) Thanks in advance.

", "messages": [ { "id": "1300642", "postDate": "05/10/2021 15:51:57", "content": "

Hi everyone, I haven't done many competitions and it is not until this time that I came to wonder

\n
\n

How to better represent cyclic data like month of the year, longitude on Earth, etc. in a way that might reveal that December is actually quite close to January and 179 degree is closer to -179 degree than 150 degree (longitude)?

\n
\n

I never thought about this, and would like to hear other people's idea, or just seeking for a bit of an interaction :) Thanks in advance.

", "rawMarkdown": "Hi everyone, I haven't done many competitions and it is not until this time that I came to wonder\n\n> How to better represent cyclic data like month of the year, longitude on Earth, etc. in a way that might reveal that `December` is actually quite close to `January` and `179` degree is closer to `-179` degree than `150` degree (longitude)?\n\nI never thought about this, and would like to hear other people's idea, or just seeking for a bit of an interaction :) Thanks in advance.", "votes": null }, { "id": "1300699", "postDate": "05/10/2021 16:32:55", "content": "

That's quite an interesting question I personally never thought about it before, so thank you!
\nI found some info on stackexchange, which has links on how to approach it.

", "rawMarkdown": "That's quite an interesting question I personally never thought about it before, so thank you!\nI found some info on [stackexchange](https://stats.stackexchange.com/questions/311494/best-practice-for-encoding-datetime-in-machine-learning), which has links on how to approach it.", "votes": null }, { "id": "1300746", "postDate": "05/10/2021 17:17:01", "content": "

I quite like the idea of taking sine and cosine in the link. Thank you :)

", "rawMarkdown": "I quite like the idea of taking sine and cosine in the link. Thank you :)", "votes": null }, { "id": "1301048", "postDate": "05/10/2021 23:38:18", "content": "

Sometimes it is represented with sine and cos.

", "rawMarkdown": "Sometimes it is represented with sine and cos.", "votes": null }, { "id": "1301132", "postDate": "05/11/2021 00:39:57", "content": "

Ja. Thanks (y)

", "rawMarkdown": "Ja. Thanks (y)", "votes": null }, { "id": "1301136", "postDate": "05/11/2021 00:41:26", "content": "

Maybe some will use a Graph Neural Network? Although I knew little about GNN, but it seems possible.

", "rawMarkdown": "Maybe some will use a Graph Neural Network? Although I knew little about GNN, but it seems possible.", "votes": null }, { "id": "1301554", "postDate": "05/11/2021 06:18:55", "content": "

What graph would you start from? Just curious.

", "rawMarkdown": "What graph would you start from? Just curious.", "votes": null }, { "id": "1301695", "postDate": "05/11/2021 07:43:06", "content": "

I did not mean to use GNN on birdclef2021. If only applied to cyclic data like hours, months, etc. maybe just the same number of vertices as the original data with edge weights being the distances btw the vertices. It just occurred to me as an idea this morning, not necessarily a good one.

", "rawMarkdown": "I did not mean to use GNN on birdclef2021. If only applied to cyclic data like hours, months, etc. maybe just the same number of vertices as the original data with edge weights being the distances btw the vertices. It just occurred to me as an idea this morning, not necessarily a good one.", "votes": null }, { "id": "1319513", "postDate": "05/23/2021 09:33:04", "content": "

I have created this toy dataset of telling which one of four seasons we are in, according to the date of the year and the longitude/latitude, so that I can explore the benefit(?) of introducing sine and cosine. In case this might interest someone out there, free feel to play with the repo and give some advices. https://github.com/phunc20/DL/tree/main/cyclic_data

", "rawMarkdown": "I have created this toy dataset of telling which one of four seasons we are in, according to the date of the year and the longitude/latitude, so that I can explore the benefit(?) of introducing sine and cosine. In case this might interest someone out there, free feel to play with the repo and give some advices. [https://github.com/phunc20/DL/tree/main/cyclic_data](url)", "votes": null }, { "id": "1321739", "postDate": "05/24/2021 22:51:46", "content": "

If you really want to dig in on this, I highly recommend this book
\nhttps://www.cambridge.org/core/books/statistical-analysis-of-circular-data/324A46F3941A5CD641ED0B0910B2C33F

\n

Used it in my PhD (looking at animal movement) and statistically analyzing turn angles and interpolating dew drying out time based on geographic aspect.

", "rawMarkdown": "If you really want to dig in on this, I highly recommend this book\nhttps://www.cambridge.org/core/books/statistical-analysis-of-circular-data/324A46F3941A5CD641ED0B0910B2C33F\n\nUsed it in my PhD (looking at animal movement) and statistically analyzing turn angles and interpolating dew drying out time based on geographic aspect.", "votes": null }, { "id": "1321797", "postDate": "05/25/2021 01:33:34", "content": "

@thesteve0 Thanks, Steve. I'll see if I can get a copy of it to read.

", "rawMarkdown": "thesteve0 Thanks, Steve. I'll see if I can get a copy of it to read.", "votes": null }, { "id": "1322808", "postDate": "05/25/2021 17:35:28", "content": "

I faced a similar problem a few years back while handling sales predictions, that particular product had weekly, monthly and yearly seasonality, plus other impacting things like holidays etc.

\n

The solution implemented was to keep one column as the primary date index as datetime(YYYYMMDD), that would always be continuous plus it was only 1 record per day.

\n

The weekdays, weeknum, etc were broken into binary columns using the get_dummies funcion from pandas, i even found the report of it (kaggle disabled the attachments, here's the extract)

\n
\n

Preprocess Feature Columns
\n I've mentioned that some of the data seems numeric but is not. This is so because some columns means
\n things like days of the week and can't be threated as numbers, one example is the column NUM_DIA_SMN,
\n when 1 represents sunday, 2 represents monday and so on, as you can see, if we threat this as numbers, the
\n algorithm will think that 1 is closer from 2 than from 7, but in reality, saturday and monday are both 1 day
\n away from sunday.
\n As you can see, there are several non-numeric columns that need to be converted. A few of them are already
\n binary, like IND_DIA_UTL, others like NUM_DIA_MES have more than 2 values assigned to them, those
\n need to be converted.
\n The recommended way to handle such a column is to create as many columns as possible values (e.g.
\n NUM_DIA_SMN_MONDAY, NUM_DIA_SMN_TUESDAY, NUM_DIA_SMN_WEDNESDAY, etc.), and assign a 1 to one of
\n them and 0 to all others.
\n These generated columns are sometimes called dummy variables, we will use the pandas.get_dummies()
\n (https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.get_dummies.html) function to perform this transformation on the code below.

\n
\n

This approch got the job done for that particular problem, mainly because there were not that many possible values for the columns there were processed, in the case of lat and long (or any truly cyclic data), i can't see this working, i wonder what can be done

", "rawMarkdown": "I faced a similar problem a few years back while handling sales predictions, that particular product had weekly, monthly and yearly seasonality, plus other impacting things like holidays etc.\n\nThe solution implemented was to keep one column as the primary date index as **datetime(YYYYMMDD)**, that would always be continuous plus it was only 1 record per day.\n\nThe **weekdays**, **weeknum**, etc were broken into binary columns using the get_dummies funcion from pandas, i even found the report of it (kaggle disabled the attachments, here's the extract)\n\n\n> Preprocess Feature Columns\nI've mentioned that some of the data seems numeric but is not. This is so because some columns means\nthings like days of the week and can't be threated as numbers, one example is the column NUM_DIA_SMN,\nwhen 1 represents sunday, 2 represents monday and so on, as you can see, if we threat this as numbers, the\nalgorithm will think that 1 is closer from 2 than from 7, but in reality, saturday and monday are both 1 day\naway from sunday.\nAs you can see, there are several non-numeric columns that need to be converted. A few of them are already\nbinary, like IND_DIA_UTL, others like NUM_DIA_MES have more than 2 values assigned to them, those\nneed to be converted.\nThe recommended way to handle such a column is to create as many columns as possible values (e.g.\nNUM_DIA_SMN_MONDAY, NUM_DIA_SMN_TUESDAY, NUM_DIA_SMN_WEDNESDAY, etc.), and assign a 1 to one of\nthem and 0 to all others.\nThese generated columns are sometimes called dummy variables, we will use the pandas.get_dummies()\n(https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.get_dummies.html) function to perform this transformation on the code below.\n\nThis approch got the job done for that particular problem, mainly because there were not that many possible values for the columns there were processed, in the case of lat and long (or any truly cyclic data), i can't see this working, i wonder what can be done", "votes": null } ], "comments": [ { "id": 1300699, "author_name": "shtrausslearning", "author_url": "", "post_date": "05/10/2021 16:32:55", "content": "

That's quite an interesting question I personally never thought about it before, so thank you!
\nI found some info on stackexchange, which has links on how to approach it.

", "votes": null, "replies": [ { "id": 1300746, "author_name": "wucf20", "author_url": "", "post_date": "05/10/2021 17:17:01", "content": "

I quite like the idea of taking sine and cosine in the link. Thank you :)

", "votes": null, "replies": [] } ] }, { "id": 1301048, "author_name": "hidehisaarai1213", "author_url": "", "post_date": "05/10/2021 23:38:18", "content": "

Sometimes it is represented with sine and cos.

", "votes": null, "replies": [ { "id": 1301132, "author_name": "wucf20", "author_url": "", "post_date": "05/11/2021 00:39:57", "content": "

Ja. Thanks (y)

", "votes": null, "replies": [] } ] }, { "id": 1301136, "author_name": "wucf20", "author_url": "", "post_date": "05/11/2021 00:41:26", "content": "

Maybe some will use a Graph Neural Network? Although I knew little about GNN, but it seems possible.

", "votes": null, "replies": [ { "id": 1301554, "author_name": "cpmpml", "author_url": "", "post_date": "05/11/2021 06:18:55", "content": "

What graph would you start from? Just curious.

", "votes": null, "replies": [] }, { "id": 1301695, "author_name": "wucf20", "author_url": "", "post_date": "05/11/2021 07:43:06", "content": "

I did not mean to use GNN on birdclef2021. If only applied to cyclic data like hours, months, etc. maybe just the same number of vertices as the original data with edge weights being the distances btw the vertices. It just occurred to me as an idea this morning, not necessarily a good one.

", "votes": null, "replies": [] } ] }, { "id": 1319513, "author_name": "wucf20", "author_url": "", "post_date": "05/23/2021 09:33:04", "content": "

I have created this toy dataset of telling which one of four seasons we are in, according to the date of the year and the longitude/latitude, so that I can explore the benefit(?) of introducing sine and cosine. In case this might interest someone out there, free feel to play with the repo and give some advices. https://github.com/phunc20/DL/tree/main/cyclic_data

", "votes": null, "replies": [] }, { "id": 1321739, "author_name": "thesteve0", "author_url": "", "post_date": "05/24/2021 22:51:46", "content": "

If you really want to dig in on this, I highly recommend this book
\nhttps://www.cambridge.org/core/books/statistical-analysis-of-circular-data/324A46F3941A5CD641ED0B0910B2C33F

\n

Used it in my PhD (looking at animal movement) and statistically analyzing turn angles and interpolating dew drying out time based on geographic aspect.

", "votes": null, "replies": [ { "id": 1321797, "author_name": "wucf20", "author_url": "", "post_date": "05/25/2021 01:33:34", "content": "

@thesteve0 Thanks, Steve. I'll see if I can get a copy of it to read.

", "votes": null, "replies": [] } ] }, { "id": 1322808, "author_name": "victorasso", "author_url": "", "post_date": "05/25/2021 17:35:28", "content": "

I faced a similar problem a few years back while handling sales predictions, that particular product had weekly, monthly and yearly seasonality, plus other impacting things like holidays etc.

\n

The solution implemented was to keep one column as the primary date index as datetime(YYYYMMDD), that would always be continuous plus it was only 1 record per day.

\n

The weekdays, weeknum, etc were broken into binary columns using the get_dummies funcion from pandas, i even found the report of it (kaggle disabled the attachments, here's the extract)

\n
\n

Preprocess Feature Columns
\n I've mentioned that some of the data seems numeric but is not. This is so because some columns means
\n things like days of the week and can't be threated as numbers, one example is the column NUM_DIA_SMN,
\n when 1 represents sunday, 2 represents monday and so on, as you can see, if we threat this as numbers, the
\n algorithm will think that 1 is closer from 2 than from 7, but in reality, saturday and monday are both 1 day
\n away from sunday.
\n As you can see, there are several non-numeric columns that need to be converted. A few of them are already
\n binary, like IND_DIA_UTL, others like NUM_DIA_MES have more than 2 values assigned to them, those
\n need to be converted.
\n The recommended way to handle such a column is to create as many columns as possible values (e.g.
\n NUM_DIA_SMN_MONDAY, NUM_DIA_SMN_TUESDAY, NUM_DIA_SMN_WEDNESDAY, etc.), and assign a 1 to one of
\n them and 0 to all others.
\n These generated columns are sometimes called dummy variables, we will use the pandas.get_dummies()
\n (https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.get_dummies.html) function to perform this transformation on the code below.

\n
\n

This approch got the job done for that particular problem, mainly because there were not that many possible values for the columns there were processed, in the case of lat and long (or any truly cyclic data), i can't see this working, i wonder what can be done

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1300642": "Hi everyone, I haven't done many competitions and it is not until this time that I came to wonder\n\n> How to better represent cyclic data like month of the year, longitude on Earth, etc. in a way that might reveal that `December` is actually quite close to `January` and `179` degree is closer to `-179` degree than `150` degree (longitude)?\n\nI never thought about this, and would like to hear other people's idea, or just seeking for a bit of an interaction :) Thanks in advance.", "1300699": "That's quite an interesting question I personally never thought about it before, so thank you!\nI found some info on [stackexchange](https://stats.stackexchange.com/questions/311494/best-practice-for-encoding-datetime-in-machine-learning), which has links on how to approach it.", "1300746": "I quite like the idea of taking sine and cosine in the link. Thank you :)", "1301048": "Sometimes it is represented with sine and cos.", "1301132": "Ja. Thanks (y)", "1301136": "Maybe some will use a Graph Neural Network? Although I knew little about GNN, but it seems possible.", "1301554": "What graph would you start from? Just curious.", "1301695": "I did not mean to use GNN on birdclef2021. If only applied to cyclic data like hours, months, etc. maybe just the same number of vertices as the original data with edge weights being the distances btw the vertices. It just occurred to me as an idea this morning, not necessarily a good one.", "1319513": "I have created this toy dataset of telling which one of four seasons we are in, according to the date of the year and the longitude/latitude, so that I can explore the benefit(?) of introducing sine and cosine. In case this might interest someone out there, free feel to play with the repo and give some advices. [https://github.com/phunc20/DL/tree/main/cyclic_data](url)", "1321739": "If you really want to dig in on this, I highly recommend this book\nhttps://www.cambridge.org/core/books/statistical-analysis-of-circular-data/324A46F3941A5CD641ED0B0910B2C33F\n\nUsed it in my PhD (looking at animal movement) and statistically analyzing turn angles and interpolating dew drying out time based on geographic aspect.", "1321797": "thesteve0 Thanks, Steve. I'll see if I can get a copy of it to read.", "1322808": "I faced a similar problem a few years back while handling sales predictions, that particular product had weekly, monthly and yearly seasonality, plus other impacting things like holidays etc.\n\nThe solution implemented was to keep one column as the primary date index as **datetime(YYYYMMDD)**, that would always be continuous plus it was only 1 record per day.\n\nThe **weekdays**, **weeknum**, etc were broken into binary columns using the get_dummies funcion from pandas, i even found the report of it (kaggle disabled the attachments, here's the extract)\n\n\n> Preprocess Feature Columns\nI've mentioned that some of the data seems numeric but is not. This is so because some columns means\nthings like days of the week and can't be threated as numbers, one example is the column NUM_DIA_SMN,\nwhen 1 represents sunday, 2 represents monday and so on, as you can see, if we threat this as numbers, the\nalgorithm will think that 1 is closer from 2 than from 7, but in reality, saturday and monday are both 1 day\naway from sunday.\nAs you can see, there are several non-numeric columns that need to be converted. A few of them are already\nbinary, like IND_DIA_UTL, others like NUM_DIA_MES have more than 2 values assigned to them, those\nneed to be converted.\nThe recommended way to handle such a column is to create as many columns as possible values (e.g.\nNUM_DIA_SMN_MONDAY, NUM_DIA_SMN_TUESDAY, NUM_DIA_SMN_WEDNESDAY, etc.), and assign a 1 to one of\nthem and 0 to all others.\nThese generated columns are sometimes called dummy variables, we will use the pandas.get_dummies()\n(https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.get_dummies.html) function to perform this transformation on the code below.\n\nThis approch got the job done for that particular problem, mainly because there were not that many possible values for the columns there were processed, in the case of lat and long (or any truly cyclic data), i can't see this working, i wonder what can be done" }, "source": "meta" }