![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1659719%2F593e310c99d10311e44d575bed88484b%2FProof.png?generation=1598903196314916&alt=media\")
I've noticed that most of popular notebooks with Quantile Regression have the same data preparation. It's at first glance hard to understand in some parts. In this topic I will write some improvements of these parts without any time improvement and only for better understanding. You can use them if you want (for instance, if you are a perfectionist in the code style or just for ideas for the future). I will extend this topic with each improvement that I can find in the next days.
\nImprovements:
\n1. This one:
\nbase = data.loc[data.Weeks == data.min_week]\nbase = base[['Patient','FVC']].copy()\nbase.columns = ['Patient','min_FVC']\nbase['nb'] = 1\nbase['nb'] = base.groupby('Patient')['nb'].transform('cumsum')\nbase = base[base.nb==1]\nbase.drop('nb', axis=1, inplace=True)\ninto this one:
\nbase = (\n data\n .loc[data.Weeks == data.min_week][['Patient','FVC']]\n .rename({'FVC': 'min_FVC'}, axis=1)\n .groupby('Patient')\n .first()\n .reset_index()\n)\n2. This one:
\nCOLS = ['Sex','SmokingStatus'] #,'Age'\nFE = []\nfor col in COLS:\n for mod in sorted(data[col].unique()):\n FE.append(mod)\n data[mod] = (data[col] == mod).astype(int)\ninto this one:
\nFE = list(data.Sex.unique()) + list(data.SmokingStatus.unique())\ndata = pd.concat([\n data,\n pd.get_dummies(data.Sex),\n pd.get_dummies(data.SmokingStatus)\n], axis=1)\n3.[optional] This one:
\ndata['age'] = (data['Age'] - data['Age'].min()) / (data['Age'].max() - data['Age'].min())\ndata['BASE'] = (data['min_FVC'] - data['min_FVC'].min() ) / ( data['min_FVC'].max() - data['min_FVC'].min() )\ndata['week'] = (data['base_week'] - data['base_week'].min() ) / ( data['base_week'].max() - data['base_week'].min() )\ndata['percent'] = (data['Percent'] - data['Percent'].min() ) / ( data['Percent'].max() - data['Percent'].min() )\nFE += ['age','percent','week','BASE']\ninto this one:
\ndef get_fillness(series):\n return (series - series.min()) / (series.max() - series.min())\n\ndata['age'] = get_fillness(data['Age'])\ndata['BASE'] = get_fillness(data['min_FVC'])\ndata['week'] = get_fillness(data['base_week'])\ndata['percent'] = get_fillness(data['Percent'])\n\nFE += ['age','percent','week','BASE']\nProofs:
\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1659719%2F007734f1ed6097bd990fe7a7bca70dea%2FProof2.png?generation=1598903680222466&alt=media\")
I've noticed that most of popular notebooks with Quantile Regression have the same data preparation. It's at first glance hard to understand in some parts. In this topic I will write some improvements of these parts without any time improvement and only for better understanding. You can use them if you want (for instance, if you are a perfectionist in the code style or just for ideas for the future). I will extend this topic with each improvement that I can find in the next days.
\nImprovements:
\n1. This one:
\nbase = data.loc[data.Weeks == data.min_week]\nbase = base[['Patient','FVC']].copy()\nbase.columns = ['Patient','min_FVC']\nbase['nb'] = 1\nbase['nb'] = base.groupby('Patient')['nb'].transform('cumsum')\nbase = base[base.nb==1]\nbase.drop('nb', axis=1, inplace=True)\ninto this one:
\nbase = (\n data\n .loc[data.Weeks == data.min_week][['Patient','FVC']]\n .rename({'FVC': 'min_FVC'}, axis=1)\n .groupby('Patient')\n .first()\n .reset_index()\n)\n2. This one:
\nCOLS = ['Sex','SmokingStatus'] #,'Age'\nFE = []\nfor col in COLS:\n for mod in sorted(data[col].unique()):\n FE.append(mod)\n data[mod] = (data[col] == mod).astype(int)\ninto this one:
\nFE = list(data.Sex.unique()) + list(data.SmokingStatus.unique())\ndata = pd.concat([\n data,\n pd.get_dummies(data.Sex),\n pd.get_dummies(data.SmokingStatus)\n], axis=1)\n3.[optional] This one:
\ndata['age'] = (data['Age'] - data['Age'].min()) / (data['Age'].max() - data['Age'].min())\ndata['BASE'] = (data['min_FVC'] - data['min_FVC'].min() ) / ( data['min_FVC'].max() - data['min_FVC'].min() )\ndata['week'] = (data['base_week'] - data['base_week'].min() ) / ( data['base_week'].max() - data['base_week'].min() )\ndata['percent'] = (data['Percent'] - data['Percent'].min() ) / ( data['Percent'].max() - data['Percent'].min() )\nFE += ['age','percent','week','BASE']\ninto this one:
\ndef get_fillness(series):\n return (series - series.min()) / (series.max() - series.min())\n\ndata['age'] = get_fillness(data['Age'])\ndata['BASE'] = get_fillness(data['min_FVC'])\ndata['week'] = get_fillness(data['base_week'])\ndata['percent'] = get_fillness(data['Percent'])\n\nFE += ['age','percent','week','BASE']\nProofs:
\n
Thanks for sharing!
\nAbout normalization: most of kernels normalize the tabular data with the train AND the test sets (which is actually hidden and made during submission because the training portion is included in the inference kernel). However it's a good practice to normalize train data with only train data and test data with only test data because of potential leaks. In this challenge we only want that our solution performs the best on the competition set so should we continue to normalize with (train+test) ? so if we voluntary introduces a leak during training, would the performance on the competition test set be better ?
Agree, in my opinion that's a data leakage. Thank you for pay attention.
", "rawMarkdown": "Agree, in my opinion that's a data leakage. Thank you for pay attention.", "votes": 1 }, { "id": 998624, "postDate": "2020-09-04T22:13:45.077Z", "content": "It actually goes further than this. In a cross-validation we should be normalising based only on the training fold not the complete training data.
", "rawMarkdown": "It actually goes further than this. In a cross-validation we should be normalising based only on the training fold not the complete training data.", "votes": 3 } ] }, { "id": 994028, "postDate": "2020-09-01T10:10:11.937Z", "content": "Thanks for sharing. I always use my own code because of this reason. Some of them are still too long though. You can use this one-liner for baseline features. (You have to make sure the data is sorted by [Patient, Weeks])
df['FVC_Baseline'] = df.groupby('Patient').transform('first')['FVC']
Agree, good optimization! Before grouping we can apply .sort(by=(Patient, Weeks)) and it will work in every case. In the evening I will update the topic with corresponding check, that results are the same.
Hey @koza4ukdmitrij, very clean approach to optimize some code which I used..thanks for the hints! 👍👍
", "rawMarkdown": "Hey @koza4ukdmitrij, very clean approach to optimize some code which I used..thanks for the hints! 👍👍", "votes": 1, "replies": [ { "id": 998441, "postDate": "2020-09-04T18:16:26.700Z", "content": "You're welcome!
", "rawMarkdown": "You're welcome! " } ] }, { "id": 995388, "postDate": "2020-09-02T12:02:14.140Z", "content": "Thanks for sharing , helps to simplify the data preparation script 😃
", "rawMarkdown": "Thanks for sharing , helps to simplify the data preparation script 😃", "votes": 1, "replies": [ { "id": 995402, "postDate": "2020-09-02T12:10:34.427Z", "content": "Yeah, that's one of the aims)
", "rawMarkdown": "Yeah, that's one of the aims)", "votes": 1 } ] }, { "id": 994241, "postDate": "2020-09-01T13:34:15.193Z", "content": "This is pretty good. I actually used some of this myself and you might not have any problems with this data set but when you do the pd.get_dummies() it's better to make the column into categorical first because if for e.g., only males were present on the train set then you would not get the right one-hot-encoding.
", "rawMarkdown": "This is pretty good. I actually used some of this myself and you might not have any problems with this data set but when you do the pd.get_dummies() it's better to make the column into categorical first because if for e.g., only males were present on the train set then you would not get the right one-hot-encoding.", "votes": 1, "replies": [ { "id": 994252, "postDate": "2020-09-01T13:41:24.537Z", "content": "Thanks! Looks like there is no such problems in the old version too, cause it was sorted(data['Sex'].unique()) instead of ['male', \"female'].
Thanks for making this clean. I did use this code first, but then I was having a hard time to understand the script. I decided to rewrite almost the whole notebook with the addition of comments, and now navigating to a section and understanding scripts feels easy. @chrisden did a good job with his well-documented notebook.
", "rawMarkdown": "Thanks for making this clean. I did use this code first, but then I was having a hard time to understand the script. I decided to rewrite almost the whole notebook with the addition of comments, and now navigating to a section and understanding scripts feels easy. @chrisden did a good job with his [well-documented notebook](https://www.kaggle.com/chrisden/6-82-quantile-reg-lr-schedulers-checkpoints).", "votes": 2, "replies": [ { "id": 998397, "postDate": "2020-09-04T17:47:40.123Z", "content": "Thanks a lot for your notebook anyway: writing a code with an attractive result is more usefull than the one with optimization and lower metric on my side. You're really welcome to use code from the topic.
", "rawMarkdown": "Thanks a lot for your notebook anyway: writing a code with an attractive result is more usefull than the one with optimization and lower metric on my side. You're really welcome to use code from the topic. " }, { "id": 998436, "postDate": "2020-09-04T18:13:39.703Z", "content": "Dear @aadhavvignesh, thanks a lot for the Kudos for my Notebook! 😃
", "rawMarkdown": "Dear @aadhavvignesh, thanks a lot for the Kudos for my Notebook! 😃", "votes": 1 } ] }, { "id": 994222, "postDate": "2020-09-01T13:26:41.427Z", "content": "Thanks for sharing.
", "rawMarkdown": "Thanks for sharing. ", "replies": [ { "id": 994239, "postDate": "2020-09-01T13:33:49.267Z", "content": "You're welcome!
", "rawMarkdown": "You're welcome!", "votes": 2 } ] }, { "id": 1000457, "postDate": "2020-09-06T14:39:50.263Z", "rawMarkdown": "", "votes": -1, "isDeleted": true, "replies": [ { "id": 1000480, "postDate": "2020-09-06T14:55:29.687Z", "content": "Thanks for your response ;)
", "rawMarkdown": "Thanks for your response ;)", "votes": 1 } ] }, { "id": 1001145, "postDate": "2020-09-07T05:11:08.307Z", "content": "Thanks for making this clean.
", "rawMarkdown": "Thanks for making this clean." } ], "comments": [ { "id": 994317, "author_name": "Alex", "author_url": "", "post_date": "2020-09-01T14:13:57.027000", "content": "Thanks for sharing!
\nAbout normalization: most of kernels normalize the tabular data with the train AND the test sets (which is actually hidden and made during submission because the training portion is included in the inference kernel). However it's a good practice to normalize train data with only train data and test data with only test data because of potential leaks. In this challenge we only want that our solution performs the best on the competition set so should we continue to normalize with (train+test) ? so if we voluntary introduces a leak during training, would the performance on the competition test set be better ?
Agree, in my opinion that's a data leakage. Thank you for pay attention.
", "votes": 1, "replies": [] }, { "id": 998624, "author_name": "jameschapman19", "author_url": "", "post_date": "2020-09-04T22:13:45.077000", "content": "It actually goes further than this. In a cross-validation we should be normalising based only on the training fold not the complete training data.
", "votes": 3, "replies": [] } ] }, { "id": 994028, "author_name": "Gunes Evitan", "author_url": "", "post_date": "2020-09-01T10:10:11.937000", "content": "Thanks for sharing. I always use my own code because of this reason. Some of them are still too long though. You can use this one-liner for baseline features. (You have to make sure the data is sorted by [Patient, Weeks])
df['FVC_Baseline'] = df.groupby('Patient').transform('first')['FVC']
Agree, good optimization! Before grouping we can apply .sort(by=(Patient, Weeks)) and it will work in every case. In the evening I will update the topic with corresponding check, that results are the same.
Hey @koza4ukdmitrij, very clean approach to optimize some code which I used..thanks for the hints! 👍👍
", "votes": 1, "replies": [ { "id": 998441, "author_name": "Dmitrij Kozachuk", "author_url": "", "post_date": "2020-09-04T18:16:26.700000", "content": "You're welcome!
", "votes": 0, "replies": [] } ] }, { "id": 995388, "author_name": "D1nall", "author_url": "", "post_date": "2020-09-02T12:02:14.140000", "content": "Thanks for sharing , helps to simplify the data preparation script 😃
", "votes": 1, "replies": [ { "id": 995402, "author_name": "Dmitrij Kozachuk", "author_url": "", "post_date": "2020-09-02T12:10:34.427000", "content": "Yeah, that's one of the aims)
", "votes": 1, "replies": [] } ] }, { "id": 994241, "author_name": "johnny", "author_url": "", "post_date": "2020-09-01T13:34:15.193000", "content": "This is pretty good. I actually used some of this myself and you might not have any problems with this data set but when you do the pd.get_dummies() it's better to make the column into categorical first because if for e.g., only males were present on the train set then you would not get the right one-hot-encoding.
", "votes": 1, "replies": [ { "id": 994252, "author_name": "Dmitrij Kozachuk", "author_url": "", "post_date": "2020-09-01T13:41:24.537000", "content": "Thanks! Looks like there is no such problems in the old version too, cause it was sorted(data['Sex'].unique()) instead of ['male', \"female'].
Thanks for making this clean. I did use this code first, but then I was having a hard time to understand the script. I decided to rewrite almost the whole notebook with the addition of comments, and now navigating to a section and understanding scripts feels easy. @chrisden did a good job with his well-documented notebook.
", "votes": 2, "replies": [ { "id": 998397, "author_name": "Dmitrij Kozachuk", "author_url": "", "post_date": "2020-09-04T17:47:40.123000", "content": "Thanks a lot for your notebook anyway: writing a code with an attractive result is more usefull than the one with optimization and lower metric on my side. You're really welcome to use code from the topic.
", "votes": 0, "replies": [] }, { "id": 998436, "author_name": "from coffee import *", "author_url": "", "post_date": "2020-09-04T18:13:39.703000", "content": "Dear @aadhavvignesh, thanks a lot for the Kudos for my Notebook! 😃
", "votes": 1, "replies": [] } ] }, { "id": 994222, "author_name": "Smuch", "author_url": "", "post_date": "2020-09-01T13:26:41.427000", "content": "Thanks for sharing.
", "votes": 0, "replies": [ { "id": 994239, "author_name": "Dmitrij Kozachuk", "author_url": "", "post_date": "2020-09-01T13:33:49.267000", "content": "You're welcome!
", "votes": 2, "replies": [] } ] }, { "id": 1000457, "author_name": "", "author_url": "", "post_date": "2020-09-06T14:39:50.263000", "content": "", "votes": -1, "replies": [ { "id": 1000480, "author_name": "Dmitrij Kozachuk", "author_url": "", "post_date": "2020-09-06T14:55:29.687000", "content": "Thanks for your response ;)
", "votes": 1, "replies": [] } ] }, { "id": 1001145, "author_name": "Ranejeb", "author_url": "", "post_date": "2020-09-07T05:11:08.307000", "content": "Thanks for making this clean.
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "993847": "I've noticed that most of popular notebooks with Quantile Regression have the same data preparation. It's at first glance hard to understand in some parts. In this topic I will write some improvements of these parts without any time improvement and only for better understanding. You can use them if you want (for instance, if you are a perfectionist in the code style or just for ideas for the future). I will extend this topic with each improvement that I can find in the next days.\n\n**Improvements:**\n\n**1.** This one:\n```\nbase = data.loc[data.Weeks == data.min_week]\nbase = base[['Patient','FVC']].copy()\nbase.columns = ['Patient','min_FVC']\nbase['nb'] = 1\nbase['nb'] = base.groupby('Patient')['nb'].transform('cumsum')\nbase = base[base.nb==1]\nbase.drop('nb', axis=1, inplace=True)\n```\ninto this one:\n```\nbase = (\n data\n .loc[data.Weeks == data.min_week][['Patient','FVC']]\n .rename({'FVC': 'min_FVC'}, axis=1)\n .groupby('Patient')\n .first()\n .reset_index()\n)\n```\n**2.** This one:\n```\nCOLS = ['Sex','SmokingStatus'] #,'Age'\nFE = []\nfor col in COLS:\n for mod in sorted(data[col].unique()):\n FE.append(mod)\n data[mod] = (data[col] == mod).astype(int)\n```\ninto this one:\n```\nFE = list(data.Sex.unique()) + list(data.SmokingStatus.unique())\ndata = pd.concat([\n data,\n pd.get_dummies(data.Sex),\n pd.get_dummies(data.SmokingStatus)\n], axis=1)\n```\n**3.**[optional] This one:\n```\ndata['age'] = (data['Age'] - data['Age'].min()) / (data['Age'].max() - data['Age'].min())\ndata['BASE'] = (data['min_FVC'] - data['min_FVC'].min() ) / ( data['min_FVC'].max() - data['min_FVC'].min() )\ndata['week'] = (data['base_week'] - data['base_week'].min() ) / ( data['base_week'].max() - data['base_week'].min() )\ndata['percent'] = (data['Percent'] - data['Percent'].min() ) / ( data['Percent'].max() - data['Percent'].min() )\nFE += ['age','percent','week','BASE']\n```\ninto this one:\n```\ndef get_fillness(series):\n return (series - series.min()) / (series.max() - series.min())\n\ndata['age'] = get_fillness(data['Age'])\ndata['BASE'] = get_fillness(data['min_FVC'])\ndata['week'] = get_fillness(data['base_week'])\ndata['percent'] = get_fillness(data['Percent'])\n\nFE += ['age','percent','week','BASE']\n```\n\n**Proofs:**\n\n\n\n", "994317": "Thanks for sharing!\nAbout normalization: most of kernels normalize the tabular data with the train AND the test sets (which is actually hidden and made during submission because the training portion is included in the inference kernel). However it's a good practice to normalize train data with only train data and test data with only test data because of potential leaks. In this challenge we only want that our solution performs the best on the competition set so should we continue to normalize with (train+test) ? so if we voluntary introduces a leak during training, would the performance on the competition test set be better ?", "994028": "Thanks for sharing. I always use my own code because of this reason. Some of them are still too long though. You can use this one-liner for baseline features. (You have to make sure the data is sorted by `[Patient, Weeks]`)\n\n`df['FVC_Baseline'] = df.groupby('Patient').transform('first')['FVC']`", "998433": "Hey @koza4ukdmitrij, very clean approach to optimize some code which I used..thanks for the hints! 👍👍", "995388": "Thanks for sharing , helps to simplify the data preparation script 😃", "994241": "This is pretty good. I actually used some of this myself and you might not have any problems with this data set but when you do the pd.get_dummies() it's better to make the column into categorical first because if for e.g., only males were present on the train set then you would not get the right one-hot-encoding.", "998377": "Thanks for making this clean. I did use this code first, but then I was having a hard time to understand the script. I decided to rewrite almost the whole notebook with the addition of comments, and now navigating to a section and understanding scripts feels easy. @chrisden did a good job with his [well-documented notebook](https://www.kaggle.com/chrisden/6-82-quantile-reg-lr-schedulers-checkpoints).", "994222": "Thanks for sharing. ", "1000457": "", "1001145": "Thanks for making this clean." } }