{"cells":[{"metadata":{"_uuid":"b299f7111c79d7894fa0670deba36014221ca272"},"cell_type":"markdown","source":"![](https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcRw7Pl0gvH5M6O39YTvooScgZ8AG6sJ-dZkW_zc60ilB3-H8BW7)"},{"metadata":{"_uuid":"dfae6755bdfe16c0bc43cf43ca5fdebbfb6a17a2"},"cell_type":"markdown","source":"# Predicting sickness using weather data\n\n### This kernel should show how to answer data science question(s) using one owns data (not from kaggle)\n\n\n\nWe will cover all of the data-science steps and eventually implement neural network and random forest models for regression. Together with hyper-parameter optimization. But let us take one step at the time:"},{"metadata":{"_uuid":"9c9a3919844309289e36f3e98cf855819829c015"},"cell_type":"markdown","source":"# **NOTE** It has come to my attention after learning about it that I performed [data leakage](https://machinelearningmastery.com/data-leakage-machine-learning/). \n\nWhere I did not split the data right at the beggining and performed the pre-processing seperately on different folds (Here I had only train and test and test_2018 sub-data sets). What happend is that the data rescaling process that I performed had knowledge of the full distribution of data in the training dataset when calculating the scaling factors (like min and max or mean and standard deviation). This knowledge was stamped into the rescaled values and exploited by all algorithms later on. Now I could just correCt it and have a perfect kernel but then reader wont learn about this important miss-step, since in real world application sdata leakage is serious. You are thinking you have a very good model but in reality you just over-fitted on the data.\n\n**I will** leave it as it is since all of the other steps are ok, but one should be carefull about it and do the pre-processing seperately on different subsets.\n**Note** please do note that this is not the only way to commit data leakage, but the only one I did."},{"metadata":{"_uuid":"879cb9fceb7138e78072918e8a6323f0b83ae525"},"cell_type":"markdown","source":"Import the necessary modules"},{"metadata":{"trusted":true,"_uuid":"ab74fd84e77fc519144f10cefea6927f23de9888","_kg_hide-input":true},"cell_type":"code","source":"\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport os\nimport datetime\nimport warnings\nwarnings.filterwarnings(\"ignore\")\nimport seaborn as sns\nimport scipy\n\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f61a4dbbb2efb05e83f4259485b58515ae4acf6f"},"cell_type":"markdown","source":"## DATA\n\nTo investigate a potential relationship, we will use two datasets:\n * daily weather observation data in Vienna (2012-2018)\n * weekly reports on [new influenza infections](https://www.data.gv.at/katalog/dataset/grippemeldedienst-stadt-wien) in Vienna (2009-2018).(flu-sickness)\n"},{"metadata":{"_uuid":"2745611b41069d987d4376eb53ad9d2a1a7c97bd"},"cell_type":"markdown","source":"## Load the data\nData in csv files is not in an optimal format, we want to make it a 4 level hiararchical index. (year,month,week and day) Week is not given hence we are going to use isocalendar() function of datetime to derive it and create a column. Furthermore we want to make sure that data is sorted (from the deepest level-day up until the year level). Another thing that is implemented in the load_weather_data function is a search for leap year, i.e. 53 weeks. We need to indentify it at label it correctly.\n\n### Weather observations \n"},{"metadata":{"trusted":true,"_uuid":"8b68c8cd67fc9e0ed07d6133a7423037a7c6c342"},"cell_type":"code","source":"def sortByDate(year_data): # Additional function needed for data-loading. Without it we get unsorted values. For example 4-th day gets thrown into second week and so on...\n    \n    month_data = year_data.loc[year_data['month'] == 1]\n    sorted_df = month_data.sort_values(by = 'day')\n    \n    for month in range(2,13):\n        month_data = year_data.loc[year_data['month'] == month]\n        sorted_month_data = month_data.sort_values(by = 'day')\n        sorted_df = pd.concat([sorted_df, sorted_month_data],)\n    \n    return sorted_df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8f6d528b655754da52364649b7ec4e6038398a1e"},"cell_type":"code","source":"def load_weather_data():\n    \"\"\" \n    Load all weather data files and combine them into a single Pandas DataFrame.\n    Add a week column and a hierarchical index (year, month, week, day)\n    \n    Returns\n    --------\n    weather_data: data frame containing the weather data\n    \"\"\"\n    \n    years_to_load = ['2012','2013','2014','2015','2016','2017','2018']\n    path=[\"../input/sickness-and-weather-data/weather_2012.csv\",\"../input/sickness-and-weather-data/weather_2013.csv\",\"../input/sickness-and-weather-data/weather_2014.csv\",\"../input/sickness-and-weather-data/weather_2015.csv\",\"../input/sickness-and-weather-data/weather_2016.csv\",\"../input/sickness-and-weather-data/weather_2017.csv\",\"../input/sickness-and-weather-data/weather_2018.csv\"]\n    col = list()\n    datalist = list()\n    week_col = list()\n    load_years=list()\n\n    first_year = True\n    for filename in path:\n        year_data = pd.read_csv(filename)\n        year_data = sortByDate(year_data)\n        if first_year:\n            weather_data = year_data\n            first_year = False\n        else:\n            weather_data = pd.concat([weather_data, year_data], sort = True)\n\n    year,week,day = datetime.date(int(years_to_load[0]),1,1).isocalendar()\n    count = 8-day\n    rows, cols = weather_data.shape\n    first_year=int(years_to_load[0])\n    \n    \n    for i in range(count):\n        week_col.append(week)\n    if week==52:\n        if datetime.date(year,12,28).isocalendar()[1] == 53: # every 4-th year, check it- add it!\n            week = 53\n        else:\n            week = 1\n            if year == first_year:\n                first_year +=1\n    elif week == 53:\n        week = 1\n        if year == first_year:\n            first_year += 1        \n        week=1\n    else:\n        week+=1\n        \n        \n        \n    while len(week_col)<rows:\n        for i in range(7):\n            if len(week_col)<rows:\n                week_col.append(week)\n        if week==52:\n            if datetime.date(int(first_year),12,28).isocalendar()[1] == 53: #53 woche?\n                week = 53\n            else:\n                week = 1\n                first_year += 1\n        \n        elif week == 53:\n            week = 1 \n            first_year += 1\n        \n        else:\n            week += 1\n            \n            \n            \n            \n    weather_data.drop(\"Unnamed: 0\",axis=1,inplace=True)\n   \n    weather_data.insert(loc=0, column='week', value = week_col)\n    weather_data.set_index([\"year\",\"month\",\"week\",\"day\"],inplace=True)\n    return weather_data    ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a2fbc57a3ea846dd49255226cc5d3c1068b8c549"},"cell_type":"markdown","source":"**Columns** Are really standard ones, humidtiy measured every 7,14 and 21 hours, wind speed, temperature and other weather specific indicators taken at different times of the day"},{"metadata":{"trusted":true,"_uuid":"6c86ae541e4f5c5e67fab41d82d41c5dd461164e"},"cell_type":"code","source":"data_weather=load_weather_data()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"cf9bd70280fa1d7a300c9027534eafec66dd2505"},"cell_type":"code","source":"data_weather","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a0fdec77136380ab6752b6564da68d287515b66f"},"cell_type":"markdown","source":"### Influenza infections  load the data\nAgain write a function and make sure that dataframe is in the suitable format. Data frame containing measuraments of new cases of flu."},{"metadata":{"trusted":true,"_uuid":"5562eae179f1d76e9bbb8a7fc91d0fa387a7eec2"},"cell_type":"code","source":"def load_influenza_data():\n    \"\"\" \n    Load and prepare the influenza data file\n    \n    Returns\n    --------\n    influenza_data: data frame containing the influenza data\n    \"\"\"\n    influenza_data=pd.read_csv('../input/sickness-and-weather-data/influenza.csv')\n    influenza_data = pd.DataFrame(influenza_data)\n    influenza_data= influenza_data[[\"Neuerkrankungen pro Woche\",\"Jahr\",\"Kalenderwoche\"]]\n    new_names = {'Neuerkrankungen pro Woche':'weekly_infections',\"Jahr\":\"year\",\"Kalenderwoche\":\"week\"}\n    \n\n    influenza_data.rename(index=str, columns=new_names,inplace=True)\n    influenza_data['week'] = influenza_data['week'].str.replace('Woche', '')\n    influenza_data['week'] = influenza_data['week'].str.replace('.', '')\n    influenza_data['year'] = influenza_data['year'].astype(int)\n    influenza_data['week'] = influenza_data['week'].astype(int)\n    influenza_data.set_index([\"year\",\"week\"],inplace=True)\n    return influenza_data\n\ndata_influenza = load_influenza_data()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"427465902254f739a3bdbc5b6f53f44036e95b0f"},"cell_type":"code","source":"data_influenza","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"df166a6256872802536cbf593a2f040969b27801"},"cell_type":"markdown","source":"Check for missing values:"},{"metadata":{"trusted":true,"_uuid":"e23d2bb9b382c777997c1dcdc79b2bcfba620526"},"cell_type":"code","source":"\n\ndata_weather.isnull().any().any()\n\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2fe8687d5fd7299015d45bbb770a5241b59437a1"},"cell_type":"markdown","source":"##  Handling Missing values"},{"metadata":{"_uuid":"a5cd98b8be2c0960ffd00b6196cf009bf1c68eb4"},"cell_type":"markdown","source":"If you take a closer look at the data, you will notice that a few of the observations are missing.\n\nThere are a wide range of standard strategies to deal with such missing values, including:\n\n- row deletion\n- substitution methods (e.g., replace with mean or median)\n- hot-/cold-deck methods (impute from a randomly selected similar record)\n- regression methods\n\nTo decide which strategy is appropriate, it is important to investigate the mechanism that led to the missing values to find out whether the missing data is missing completely at random (MCAR), missing at random (MAR), or missing not at random (MNAR). \n\n - **MCAR** means that there is no relationship between the missingness of the data and any of the values.\n - **MAR** means that that there is a systematic relationship between the propensity of missing values and the observed data, but not the missing data.\n - **MNAR** means that there is a systematic relationship between the propensity of a value to be missing and its values. \n\nTo find out more about what mechanisms may have caused the missing values, you talked to the metereologist that compiled the data. \nShe told you that she does not know why some of the temperature readings are missing, but that it may be that someone forgot to record them. In any case, it is likely that the propensity of a temperature value to be missing does not have anything to do with the weather itself.\n\nAs far as the missing humidity readings are concerned, she says that according to her experience, she suspects that the humidity sensor is less reliable in hot weather.\n\nThe missing wind speed and direction sensor readings were due to a hardware defect.\n"},{"metadata":{"trusted":true,"_uuid":"7ec0dd14750d16c53a666e3e01cb3e99796eb60a"},"cell_type":"code","source":"def handle_missingValues_simple(incomplete_data):\n    \"\"\" \n    Parameters\n    --------\n    incomplete_data: data frame containing missing values \n    \n    Returns\n    --------\n    complete_data: data frame not containing any missing values\n    \"\"\"\n    #See description\n   \n\n    wind=incomplete_data[[\"wind_mSec\"]].interpolate(method='linear')\n    temp7h = incomplete_data['temp_7h'].fillna(method='bfill')\n    temp14h = incomplete_data['temp_14h'].fillna(method='bfill')\n    temp19h = incomplete_data['temp_19h'].fillna(method='bfill')\n    hum_missing = incomplete_data[[\"hum_14h\",\"hum_19h\",\"hum_7h\"]]\n    hum_missing = hum_missing.reset_index()\n    hum_missing = hum_missing.interpolate(method = 'piecewise_polynomial')\n    hum_missing.set_index([\"year\",\"month\",\"week\",\"day\"],inplace = True)\n    \n    #2012 to 2018_01 are ok (just a bit of NaN)\n    wind_degrees_only = incomplete_data[[\"wind_degrees\"]]\n    wind_degrees_2012_until_2018_01 = wind_degrees_only.iloc[0:2223]\n    wind_degrees_2012_until_2018_01 = wind_degrees_2012_until_2018_01.interpolate(method='linear')\n    \n    #Tricky part(find the indices before)---description\n    wind_degrees_2018_Feb = wind_degrees_only.iloc[2223:2251]     \n    wind_degrees_2018_Feb.iloc[:] = wind_degrees_only.iloc[1858:1886].values\n    \n    wind_degrees_2018_March = wind_degrees_only.iloc[2251:2282] #no NaN\n    \n    wind_degrees_2018_April = wind_degrees_only.iloc[2282:2312]\n    wind_degrees_2018_April.iloc[:] = wind_degrees_only.iloc[1917:1947].values\n    \n    wind_degrees_2018_May = wind_degrees_only.iloc[2312:2343]\n    wind_degrees_2018_May.iloc[:] = wind_degrees_only.iloc[1947:1978].values\n    \n    wind_degrees_2018_June_July_August = wind_degrees_only.iloc[2343:] #no NaN\n    \n    skyCover_and_sun = incomplete_data[['skyCover_14h','skyCover_19h','skyCover_7h','sun_hours']].interpolate(method='linear')\n    \n    \n    wind_degrees = pd.concat([wind_degrees_2012_until_2018_01, wind_degrees_2018_Feb, wind_degrees_2018_March,\n                              wind_degrees_2018_April,wind_degrees_2018_May, wind_degrees_2018_June_July_August])\n   \n\n\n    complete_data = pd.concat([ incomplete_data[['temp_dailyMax','temp_dailyMean',\n                                                'temp_dailyMin','temp_minGround','hum_dailyMean','precip']],hum_missing,wind,temp7h,temp14h,temp19h,skyCover_and_sun], axis = 1, sort = True)\n\n    return complete_data\n\n\ndef handle_missingValues_advanced (incomplete_data):\n    \"\"\" \n    Parameters\n    --------\n    data: data frame containing missing values \n    \n\n    Returns\n    --------\n    data: data frame not containing any missing values\n    \"\"\"\n    \n    return complete_data\n    \ndata_weather_complete = handle_missingValues_simple(data_weather)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2f39239a3ec5e46fde276ff07f8c5e2ecc1be504"},"cell_type":"code","source":"data_weather_complete\n\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"dc91d9e5403f4ff1d5760510eb1aebac92788a98"},"cell_type":"markdown","source":"###  Discussion\n\n#### Pros and Cons of strategies for dealing with missing data\n\n"},{"metadata":{"_uuid":"780bdf1b59aed4335527e160ca7852699f930e89"},"cell_type":"markdown","source":"\nBefore we evan start discussing missing values imputation, we need to define the scope of a definition missing values. Is it just NaN, can it be 0 or 999 or some other value? For that we ought to plot the variables and check our usual suspects. Other tactic would also be to ask meterologist. It can happen that for example humidity readings machine gives 999 when in defect. Let us assume we are only dealing with NaN.\n\nweather.dropna()-quickest and dumbest solution. We will lose a lot of data. Makes sense if number of missing values is small. But evan than we should use our brains.\n\n\nweather.fillna(method='pad') \nweather.fillna(method='bfill') Foreward, backward filling. It is reasonable. Data point in previous or next step should also be the one missing. It is robust in the sense that we do not have to make any assumptions about the data. Imputation works best when many variables are missing in small proportions. The power boost is much less impressive when one important variable is missing in 70% of cases, because the uncertainty in estimates will yield highly varying imputed datasets.\n\n\nweather = weather.fillna(weather.mean()) Mean is prone to outliers, and imputation would fail. It also reduces the variance in the data. We could opt for median to avoid outliers. Assumptions for pad and bfill also hold for this imputation\n\n\nweather = df.weather(method='linear') or polynomial of higher order would also make sense, at the end we are finding a function that dsecribes our data (column). But we will not be able to generalise, over-fitting.\n\nmodel for NaN- same story it would be in most cases the perfect solution to handle missing values. But we are giving up a few things. OVerfitting,time,simplicity to name a few.\n\n\nLastly for MAR (humidity) we will be using Multiple Imputation using MICE (Multiple Imputation by Chained Equations).\nBasic idea is that we are going to run multiple regression models and each missing value is modeled conditionally depending on the observed (non-missing) values, andthen average it over the number of regressions. Papers show that this ought to be the best solution for MAR-data. We are going to use neat package called fancyimpute. There we could have used kNN (Which also very reasonable) to imputata. Please note that it works only on numpy array, so we need to convert back and fort!\n"},{"metadata":{"_uuid":"0f56980e6ee23296f0cbe34d93918d21ba143c93"},"cell_type":"markdown","source":"#### Explanation of our own strategy"},{"metadata":{"_uuid":"1d09c44ca939c00d0274f33accd96b4a40229528"},"cell_type":"markdown","source":"\nStrategy is as follows (general one) We got some pointers on where and what type of missing variables are in certain columns. But there are much more missing values than that. We need to check each column(variable) individually and than depending on the findings (amount of missing data, where are they missing, the variable) we will perform the optimal imputing.\n\nLet us start with no missing data-columns, How do we check ? For example, for the daily maximal temperature we would do (I deleted these cells) data_weather[data_weather['temp_dailyMax'].isnull()] \"temp_dailyMean\", 'temp_dailyMin', 'temp_minGround', 'hum_dailyMean', \"hum_dailyMean\" columns have no missing values. Let us check for data_weather[data_weather['temp_7h'].isnull()] there are some missing values and they seem to be rather randomly scattered. Taking that into account we are going to use simple backwardfill here. Analog for 'temp_14h' and 'temp_19h'.\nLet us move further on. 'wind_mSec' has only 7 missing values in all of the 7 years. Simple interpolate() ought to do the trick.\n'wind_degrees' is the interesting one. There a couple of missing values in the beginning but in the last year we observe that data is missing for whole months at a part. Logically we ought to do simple interpolate() for the few values that are missing in the beginning and than just copy the months from previous year (2017) or any other year for that matter. We only need to find the row indices of the previous year months and than replace it with the missing values in 2018. Finally 'skyCover_14h','skyCover_19h','skyCover_7h','sun_hours' are all missing values at the same time (just a few of them) once again I opt for simple interpolate().\n\nTemperature is MCAR\nThe missing wind speed readings is MNAR\nMissing humidity readings MAR. If we compare given and the alternative definitions following tactics would be reasonable choices.\n\nFor Temperature, independent what the time of the year is temperature should be close to the previous day/following day. So backward fill seams reasonable.\n\nWind i described above, but it depends on what wind column exactly.\n\nHumidity NaNs (there are 3 columns in regard to humidity) could be removed with a machine learning model. We already now that temperature is a independent variable, we could also test whether other variables have predictive power and do a regression. But after consulting online it seems that Multiple imputation with MICE is the best option, that is why I wanted to use that. Unfortunately I did not manage to install the library fancyimputer, so I opted for polynomial interpolation (\"piecewise_poly\") there I hope that polynomial function will catch trends and predict/imputate missing values.\n\n"},{"metadata":{"trusted":true,"_uuid":"dd36c3492171b5c7340afef4be63aa307e26bd87"},"cell_type":"code","source":"data_weather_complete.isnull().any().any()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"99d99a6bb2476f20089cf68d4080d94ed92933b2"},"cell_type":"markdown","source":"##  Handling Outliers"},{"metadata":{"_uuid":"37e910813bfd8e57ae07bb8fcf6c8fb30e921cad"},"cell_type":"markdown","source":"If you take a closer look at some of the observations, you should notice that some of the temperature values are not particularly plausible "},{"metadata":{"trusted":true,"_uuid":"ca1bb01189ced33c7fd5ebd6a0e0b547d065fbf4"},"cell_type":"code","source":"fig, ax = plt.subplots(4, 2)\n    \n    \n    \nax[0, 0].hist(data_weather_complete.temp_14h, normed=True, bins=30)\nax[1, 0].hist(data_weather_complete.temp_19h, normed=True, bins=30)\nax[0, 1].hist(data_weather_complete.temp_7h, normed=True, bins=30) \nax[1, 1].hist(data_weather_complete.temp_dailyMax, normed=True, bins=30) \nax[2, 0].hist(data_weather_complete.temp_dailyMean, normed=True, bins=30)\nax[2, 1].hist(data_weather_complete.temp_dailyMin, normed=True, bins=30)\nax[3, 0].hist(data_weather_complete.temp_minGround, normed=True, bins=30)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1849c95d13237eee8f080eda6ff37578ac98f4fd"},"cell_type":"code","source":"# Before excluding certain values in handle_outliers function underneath, we are going to compare two methods and different paramaeters\n# All to see which number of outliers seems reasonable, than we are going to exclude entire row that has this outlier\n#It will be only a few since we will opt for the most extreme case, where deviation from the mean is really ridiculous.\n\ndef out_std(s, nstd=3.0, return_thresholds=False):\n\n    data_mean, data_std = s.mean(), s.std()\n    cut_off = data_std * nstd\n    lower, upper = data_mean - cut_off, data_mean + cut_off\n    if return_thresholds:\n        return lower, upper\n    else:\n        return [False if x < lower or x > upper else True for x in s]\n    \n\n    \n    \nstd2 = data_weather_complete.apply(out_std, nstd=1.8)\nstd3 = data_weather_complete.apply(out_std, nstd=3.0)\nstd4 = data_weather_complete.apply(out_std, nstd=4.0)\n\n    \n    \nf, ((ax1, ax2, ax3)) = plt.subplots(ncols=3, nrows=1, figsize=(22, 12));\nax1.set_title('Outliers with 1.8 standard deviations');\nax2.set_title('Outliers using 3 standard deviations');\nax3.set_title('Outliers using 4 standard deviations');\n\nsns.heatmap(std2, cmap='Blues', ax=ax1);\nsns.heatmap(std3, cmap='Blues', ax=ax2);\nsns.heatmap(std4, cmap='Blues', ax=ax3);\n\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"dc878a66d4336f5fcc3a4a5c66b94c12769f004d"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8928fe27d1706743ee75e73575baf3f1719f6cf7"},"cell_type":"code","source":"def handle_outliers(noisy_data):\n    \"\"\" \n    Parameters\n    --------\n    noisy_data: data frame that contains outliers\n    \n    Returns\n    --------\n    cleaned_data: data frame with outliers\n    \"\"\"\n    noisy_data=noisy_data[std2]\n    noisy_data.loc[noisy_data.temp_14h >= 40, 'temp_14h'] = np.NaN\n    \n    noisy_data = noisy_data.reset_index()\n    noisy_data = noisy_data.interpolate(method = 'piecewise_polynomial')\n    noisy_data.set_index([\"year\",\"month\",\"week\",\"day\"],inplace = True)\n    \n    cleaned_data = noisy_data\n    return cleaned_data\n    \ndata_weather_cleaned = handle_outliers(data_weather_complete)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"617527fd780995983f0f57e154cbec0e5ba7334c"},"cell_type":"code","source":"data_weather_cleaned[[\"temp_19h\"]].plot()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c651876f5c2d103ff4d619c388214262e8ba7b5a"},"cell_type":"markdown","source":"#### Strategy explanation below\n\n"},{"metadata":{"_uuid":"aa7c71627eecf3e51307d9535b10170e747ac382"},"cell_type":"markdown","source":"\nThere is no precise way to define and identify outliers in general because of the specifics of each dataset. Instead, you, or a domain expert, must interpret the raw observations and decide whether a value is an outlier or not. Nevertheless, we can use statistical methods to identify observations that appear to be rare or unlikely given the available data. We will assume that outliers are only to be found in temperature measuraments. From the plots above we can see there are some points that fall far out of the normal one or two points standard deviation. With three we should be certain that it is an outlier and that is why we should remove it. Alternatively we can also say that standard deviation is heavily influenced with the outliers. To remedy that I also implemented a an IQR (inter quartile range) method that removes all values that fall below or above IQR (which is Q1-Q3). With these two relatively simple methods we should be able to recognize most of the outliers. Qualitative examination would also be beneficial (and they were since we had couple of very hot days in winter ;)), along with other more advanced techniques.\n\nThen I tested it and IQR method was not very effiecent even with high koefficients, so I stuck with std method. I tried different values, and it seems that with std2 that is koefficient of 1.8*std removes most of the outliers.\nFor the missing values I just interpolated the missing values (standard imputation technique). But inspecting the data I see that there some values that are 40 degrees in winter (in temp_14columns). So we also need to make sure that these are also gone, since they are definitely an outlier. So I located these valus and set them to NaN then i interpolated all of the NaN with polynomial interpolation.\n\n\n\n"},{"metadata":{"_uuid":"3658dfdad7e5da014b2f5b93d5554dced82a58f2"},"cell_type":"markdown","source":"## Aggregate values \n\nAggregation of the observations on a weekly basis. Returns a data frame with a hierarchical index (levels `year` and `week`) on the vertical axis and the following weekly aggregations as columns:\n\n- `temp_weeklyMin`: minimum of `temp_dailyMin`\n- `temp_weeklyMax`: mean of `temp_dailyMax`\n- `temp_weeklyMean`: mean of `temp_dailyMean`\n- `temp_7h_weeklyMedian`: median of `temp_7h`\n- `temp_14h_weeklyMedian`: median of `temp_14h`\n- `temp_19h_weeklyMedian`: median of `temp_19h`\n\n- `hum_weeklyMean`: mean of `hum_dailyMean`\n- `hum_7h_weeklyMedian`: median of `hum_7h`\n- `hum_14h_weeklyMedian`: median of `hum_14h`\n- `hum_19h_weeklyMedian`: median of `hum_19h`\n\n- `precip_weeklyMean`: mean of `precip`\n- `wind_mSec_mean`: mean of `wind_mSec`"},{"metadata":{"trusted":true,"_uuid":"dcdc7889e68c2e5caa1b6c2282c02b62f1c70657"},"cell_type":"code","source":"def aggregate_weekly(data):\n    \"\"\" \n    Parameters\n    --------\n    data: weather data frame\n    \n    Returns\n    --------\n    weekly_stats: data frame that contains statistics aggregated on a weekly basis\n    \"\"\"\n    \n   \n    data=data.reset_index()\n    data = data.iloc[1:] #Aggregation with a 2012 1 1 is actually week 52 from 2011 (because of the way they count it) so we should not include it in aggregation\n    data.set_index([\"year\",\"week\"],inplace=True)\n\n    data[\"temp_weeklyMin\"] = data.pivot_table('temp_dailyMin', index=[\"year\",'week'],aggfunc=min)\n    data[\"temp_weeklyMax\"] = data.pivot_table('temp_dailyMax', index=[\"year\",'week'],aggfunc=np.mean)\n    data[\"temp_weeklyMean\"] = data.pivot_table('temp_dailyMean', index=[\"year\",'week'],aggfunc=np.mean)\n    data[\"temp_7h_weeklyMedian\"] = data.pivot_table('temp_7h', index=[\"year\",'week'],aggfunc=np.median)\n    data[\"temp_14h_weeklyMedian\"] = data.pivot_table('temp_14h', index=[\"year\",'week'],aggfunc=np.median)\n    data[\"temp_19h_weeklyMedian\"] = data.pivot_table('temp_19h', index=[\"year\",'week'],aggfunc=np.median)\n    data[\"hum_weeklyMean\"] = data.pivot_table('hum_dailyMean', index=[\"year\",'week'],aggfunc=np.mean)\n    data[\"hum_7h_weeklyMedian\"] = data.pivot_table('hum_7h', index=[\"year\",'week'],aggfunc=np.median)\n    data[\"hum_14h_weeklyMedian\"] = data.pivot_table('hum_14h', index=[\"year\",'week'],aggfunc=np.median)\n    data[\"hum_19h_weeklyMedian\"] = data.pivot_table('hum_19h', index=[\"year\",'week'],aggfunc=np.median)\n    data[\"precip_weeklyMean\"] = data.pivot_table('precip', index=[\"year\",'week'],aggfunc=np.mean)\n    data[\"wind_mSec_mean\"] = data.pivot_table('wind_mSec', index=[\"year\",'week'],aggfunc=np.mean)\n    \n    weekly_weather_data = data.drop([\"temp_minGround\",\"sun_hours\",\"skyCover_7h\",\"skyCover_19h\",\"skyCover_14h\",'temp_dailyMin', 'temp_dailyMax','temp_dailyMean','temp_7h','temp_14h','temp_19h','hum_dailyMean','hum_7h','hum_14h','hum_19h','precip','wind_mSec'], 1)\n    ww2012=weekly_weather_data.xs(2012,level='year')\n    ww2012=ww2012[~ww2012.index.get_level_values(0).duplicated()]\n    ww2013=weekly_weather_data.xs(2013,level='year')\n    ww2013=ww2013[~ww2013.index.get_level_values(0).duplicated()]\n    ww2014=weekly_weather_data.xs(2014,level='year')\n    ww2014=ww2014[~ww2014.index.get_level_values(0).duplicated()]\n    ww2015=weekly_weather_data.xs(2015,level='year')\n    ww2015=ww2015[~ww2015.index.get_level_values(0).duplicated()]\n    ww2016=weekly_weather_data.xs(2016,level='year')\n    ww2016=ww2016[~ww2016.index.get_level_values(0).duplicated()]\n    ww2017=weekly_weather_data.xs(2017,level='year')\n    ww2017=ww2017[~ww2017.index.get_level_values(0).duplicated()]\n    ww2018=weekly_weather_data.xs(2018,level='year')\n    ww2018=ww2018[~ww2018.index.get_level_values(0).duplicated()]\n    weekly_weather_data = pd.concat([ww2012,ww2013,ww2014,ww2015,ww2016,ww2017,ww2018], keys=['2012','2013',\"2014\",\"2015\",\"2016\",\"2017\",\"2018\"])\n    weekly_weather_data.index.names = ['year','week']\n    weekly_weather_data.reset_index(inplace=True)\n    weekly_weather_data['year'] = pd.to_numeric(weekly_weather_data['year'])\n    weekly_weather_data['week'] = pd.to_numeric(weekly_weather_data['week'])\n    weekly_weather_data.set_index([\"year\",\"week\"],inplace=True)\n    \n    return weekly_weather_data\n\ndata_weather_weekly = aggregate_weekly(data_weather_cleaned)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a2446c55978b17f8292588c7de2991c246afa505"},"cell_type":"code","source":"data_weather_weekly","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d1962b812c3c038503a8c7627881a8f0ff6cc7da"},"cell_type":"markdown","source":"## Merging influenza and weather datasets"},{"metadata":{"_uuid":"e7ce5779212c8c476b165f6a4ab480eed594704b"},"cell_type":"markdown","source":"Merge the `data_weather_weekly` and `data_influenza` datasets."},{"metadata":{"trusted":true,"_uuid":"c30bb280641a0a4768eaa75dc53a46e7ebfa1d1a"},"cell_type":"code","source":"\ndef merge_data(weather_df, influenza_df):\n    \"\"\" \n    Parameters\n    --------\n    weather_df: weekly weather data frame\n    influenza_df: influenza data frame\n    \n    Returns\n    --------\n    merged_data: merged data frame that contains both weekly weather observations and prevalence of influence infections\n    \"\"\"\n    merged_data = weather_df.join(influenza_df)\n    merged_data=merged_data.reset_index()\n    merged_data = merged_data.apply(pd.to_numeric)\n    merged_data[\"weekly_infections\"]= merged_data[\"weekly_infections\"].interpolate(method = 'linear')\n    \n    merged_data.iloc[327:,14] = merged_data.iloc[275:297,14].values\n    merged_data.set_index(\"year\",\"week\")\n    return merged_data\n\ndata_merged = merge_data(data_weather_weekly, data_influenza)","execution_count":null,"outputs":[]},{"metadata":{"scrolled":true,"trusted":true,"_uuid":"3d216c947c94d883609ff8ef0a5b346affdf0631"},"cell_type":"code","source":"data_merged","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6f43c3262194f8a254e4d4b256bb877cf9616977"},"cell_type":"markdown","source":"##  Visualization"},{"metadata":{"_uuid":"5f543a805d3f77981631c484b9b4cffa73cc8b50"},"cell_type":"markdown","source":"To get a better understanding of the dataset, create visualizations of the merged data set that help to explore the potential relationships between the variables before starting to develop a model.\n\n"},{"metadata":{"trusted":true,"_uuid":"9b94afe135e1cdf9bd84ecd8f50e88aa33571768"},"cell_type":"code","source":"data_merged2=data_merged[[\"temp_weeklyMin\",\"temp_weeklyMax\",\"temp_weeklyMean\",\"temp_7h_weeklyMedian\",\"hum_weeklyMean\",\"hum_7h_weeklyMedian\",\"precip_weeklyMean\",\"wind_mSec_mean\",\"weekly_infections\"]].reset_index()\n\nsns_plot2=sns.pairplot(data_merged2)\nsns_plot2.savefig(\"01527395_01.png\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ffda12a8283cc9dc9794b16c02d7d222a52a8852"},"cell_type":"code","source":"b=data_merged.drop([\"weekly_infections\"], axis=1).reset_index()\nb=pd.DataFrame(b)\nmelted = pd.melt(b, ['year',\"week\",\"index\"])\n    \nmelted.drop(['year',\"week\",\"index\"],axis=1,inplace=True)\nmelted[\"value\"] = pd.to_numeric(melted[\"value\"])\nmelted\nsns_plot1=sns.boxplot(x=\"variable\", y=\"value\", data=melted)\nsns_plot1.set_xticklabels(sns_plot1.get_xticklabels(), rotation = 90, fontsize = 10)\nsns_plot1.figure.savefig(\"01527395_02.png\")\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"401a8451d11471f34c41c38ca68b0cb0b1587516"},"cell_type":"code","source":"\ndata_merged1=data_merged.reset_index()\ndata_merged1.drop(['year',\"week\"],axis=1,inplace=True)\n\n# calculate the correlation matrix\ncorr = data_merged1.corr()\n\n# plot the heatmap\nsns_plot2=sns.heatmap(corr, \n        xticklabels=corr.columns,\n        yticklabels=corr.columns)\nsns_plot2.figure.savefig(\"01527395_03.png\")\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"150f93980f41216fd148495434830b8290fed3db"},"cell_type":"markdown","source":"##  Influenza prediction model \n\n\nBuild a model to predict the number of influenza incidents for the year 2018 (discarding all the data available for 2018) based on data of previous year using."},{"metadata":{"trusted":true,"_uuid":"3f4eb1e7d3dcceb544cc21d5e9d0ae5a51dc7df3"},"cell_type":"code","source":"from sklearn.model_selection import train_test_split\nfrom keras.models import Sequential\nfrom keras.layers import Dense\nfrom keras.wrappers.scikit_learn import KerasRegressor\nimport keras\nfrom sklearn.preprocessing import LabelEncoder\nfrom sklearn.preprocessing import LabelBinarizer\nfrom sklearn import preprocessing\nimport sklearn\n\ndata_merged1=data_merged.reset_index()\nyear=data_merged1[\"year\"]\nweek=data_merged1[\"week\"]\ndata_merged1=data_merged1.drop([\"year\",\"week\"],axis=1)\ndm1_col = data_merged1.columns\nmin_max_scaler = preprocessing.MinMaxScaler()\nnp_scaled = min_max_scaler.fit_transform(data_merged1)\ndata_merged2=pd.DataFrame(np_scaled,columns =dm1_col)\n\nX=data_merged2.drop([\"weekly_infections\"], axis=1)\nX[\"year\"]=year\nX[\"week\"]=week\nX_2018=X.loc[X['year'] == 2018]\nX=X.loc[X['year'] != 2018]\n\n\n\n\ny=data_merged2[\"weekly_infections\"].reset_index()\n\n# y.drop([\"year\",\"week\"], axis=1,inplace=True)\ny[\"year\"]=year\ny[\"week\"]=week\ny_2018=y.loc[y['year'] == 2018]\ny_2018.drop([\"year\",\"week\",\"index\"], axis=1,inplace=True)\ny=y.loc[y['year'] != 2018]\ny.drop([\"year\",\"week\",\"index\"], axis=1,inplace=True)\nX_train,X_test,y_train,y_test=sklearn.model_selection.train_test_split(X,y,test_size=0.4,random_state=42)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6947bd31c8da5b0a38f5994450a374c8239c5e9d"},"cell_type":"code","source":"\nfrom keras.models import Sequential\nfrom sklearn.metrics import accuracy_score\nfrom keras.layers import Dense\nfrom keras.wrappers.scikit_learn import KerasRegressor\nfrom sklearn.model_selection import GridSearchCV\nfrom sklearn.metrics import mean_squared_error\nfrom math import sqrt\n\nfrom numpy.random import seed\nseed(1)\nfrom tensorflow import set_random_seed\nset_random_seed(1)\n\nkeras.initializers.RandomUniform(minval=-0.05, maxval=0.05, seed=123)\n\ndef baseline_model():\n    model = Sequential()\n    model.add(Dense(200, input_dim=17, kernel_initializer='random_uniform', activation='relu'))\n    model.add(Dense(100, kernel_initializer='normal', activation='relu'))\n    model.add(Dense(50, kernel_initializer='normal', activation='relu'))\n    model.add(Dense(25, kernel_initializer='normal', activation='relu'))\n    model.add(Dense(1, kernel_initializer='normal'))\n    model.compile(loss='mean_squared_error', optimizer=keras.optimizers.Adam())\n    return model\n\nestimator = KerasRegressor(build_fn=baseline_model, nb_epoch=100, batch_size=32, verbose=False)\nestimator.fit(X_train, y_train)\n\nprediction = estimator.predict(X_train)\nrmse_nn = sqrt(mean_squared_error(y_train, prediction))\n\nprediction1 = estimator.predict(X_test)\nrmse_nn1 = sqrt(mean_squared_error(y_test, prediction1))\n\nprediction2 = estimator.predict(X_2018)\nrmse_nn2 = sqrt(mean_squared_error(y_2018, prediction2))\n\n\n\nprint(rmse_nn,rmse_nn1,rmse_nn2)\n    \n    \n    \n    \n#batch_size = [20, 40, 60, 80, 100]\n#epochs = [10, 50, 100,250,500]\n#learn_rate = [0.001, 0.01, 0.1, 0.2, 0.3]\n#momentum = [0.0, 0.2, 0.4, 0.6, 0.8, 0.9]\n#weight_constraint = [1, 2, 3, 4, 5]\n#dropout_rate = [0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9]\n#neurons = [1, 5, 10, 15, 20, 25, 30]\n#param_grid = dict(neurons=neurons, batch_size=batch_size, epochs=epochs, learn_rate=learn_rate, \n#                 momentum=momentum, dropout_rate=dropout_rate, weight_constraint=weight_constraint)\n# param_grid = dict(batch_size=batch_size)\n# grid = GridSearchCV(estimator=estimator, param_grid=param_grid, n_jobs=-1,cv=10)\n# grid_result = grid.fit(X_train, y_train)\n# summarize results\n# print(\"Best: %f using %s\" % (grid_result.best_score_, grid_result.best_params_))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6753359e94e3dca9af38804c06960f4564d3c507"},"cell_type":"code","source":"from sklearn import *\n\nmodel2 = ensemble.RandomForestRegressor(n_estimators=25, random_state=11, max_depth=1,\n                                        min_weight_fraction_leaf=0.122)\ndef RMSLE(y, pred):\n    return metrics.mean_squared_error(y, pred)**0.5\n\n\nmodel2.fit(X_train, np.log1p(y_train.values))\nprint(\"Model2 trained\")\npreds2 = model2.predict(X_test)\npreds3 = model2.predict(X_2018)\nprint('RMSE RandomForestRegressor on validation data: ', RMSLE(y_test, preds2))\nprint('RMSE RandomForestRegressor on test data: ', RMSLE(y_2018, preds3))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a638e4ba965d4d5d73c80a0acec75ac34ffb0bf1"},"cell_type":"code","source":"\nfrom sklearn.model_selection import RandomizedSearchCV\nfrom scipy.stats import randint as sp_randint\n# parameters for GridSearchCV\n# specify parameters and distributions to sample from\nparam_dist = {\"n_estimators\": [3,4,5,6,7,8,9,11,15,18,24,25,26,27,28,40,50,80],\n              \"min_weight_fraction_leaf\": [3.03030303e-03,   4.04040404e-03,   5.05050505e-03,\n         6.06060606e-03,   7.07070707e-03,   8.08080808e-03,\n         9.09090909e-03,   1.01010101e-02,   1.11111111e-02,\n         1.21212121e-02,   1.31313131e-02,   1.41414141e-02,\n         1.51515152e-02,   1.61616162e-02,   1.71717172e-02,0.0002,0.0003,0.0004,0.0001,0.0009,0.0008],\n              \"max_features\": sp_randint(1, 11),\n              \"min_samples_split\": sp_randint(2, 30),\n              \"min_samples_leaf\": sp_randint(1, 30),\n              \"bootstrap\": [True, False],\n              \"max_depth\": [3,4,5,6,7,8,9,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,30,40,50,80],\n              \"random_state\": sp_randint(1, 30)\n             }\n# run randomized search\nn_iter_search = 30\nrandom_search = RandomizedSearchCV(model2, param_distributions=param_dist,\n                                   n_iter=n_iter_search,cv=10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"af2f786c38c6509f353afa1eeb2e78ab47ebee10","_kg_hide-input":false,"_kg_hide-output":true},"cell_type":"code","source":"random_search.fit(X_train, np.log1p(y_train.values))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8bf4188384b082f002420bbe9f0f7aa16e5e3019"},"cell_type":"code","source":"def report(results, n_top=3):\n    for i in range(1, n_top + 1):\n        candidates = np.flatnonzero(results['rank_test_score'] == i)\n        for candidate in candidates:\n            print(\"Model with rank: {0}\".format(i))\n            print(\"Mean validation score: {0:.3f} (std: {1:.3f})\".format(\n                  results['mean_test_score'][candidate],\n                  results['std_test_score'][candidate]))\n            print(\"Parameters: {0}\".format(results['params'][candidate]))\n            print(\"\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"28c15cb4d46b8fe2912495f0074c81fe8984cbb4"},"cell_type":"code","source":"report(random_search.cv_results_)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e4309884fd349afa8a0284e21fd7d4f182390f16"},"cell_type":"code","source":"finalmodel = ensemble.RandomForestRegressor(n_estimators=5, random_state=25, max_depth=21,\n                                        min_weight_fraction_leaf=0.0131313131,bootstrap=False,max_features=8,min_samples_leaf=1,min_samples_split=12)\ndef RMSLE(y, pred):\n    return metrics.mean_squared_error(y, pred)**0.5\n\n\nfinalmodel.fit(X_train, np.log1p(y_train.values))\nprint(\"Model2 trained\")\npreds2 = finalmodel.predict(X_test)\nprint('RMSE RandomForestRegressor: ', RMSLE(y_test, preds2))\n\npreds3 = finalmodel.predict(X_2018)\nprint('RMSE RandomForestRegressor on test data: ', RMSLE(y_2018, preds3))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c4f2451de7d9568788eabd33b6e4307947cff287"},"cell_type":"markdown","source":"## Approach explanation and algorithm motivation:\n"},{"metadata":{"_uuid":"1af6ac0d0af8f1d0f87324b188a0ac16f9ce61bb"},"cell_type":"markdown","source":" Let us first talk what we did additionally with the data (pre-processing, assumptions etc...) than the models: Label encoding, important but than I realised that they are already label encoded (not hot-one!) I would like to stay with this assumption since week category obviously matters but also years will be different from each other(global warming etc.) Than I normalized all of the columns. Than I extracted the 2018 data which is our test data(y_2018,X_2018). And with help of train_test from sklearn I created training and validation sets (X_train,X_test)\n\n\nFirst model---NN. Started of with manually checking the hyperparameters-of course very bad tactic but atleast I got a feeling of the hyperparameters. Then I implemented a grid search for neural network. Time expensive! I commented the code out because it takes way to long. Results? Training set (in-sample error: 0.17964286018957273  Validation set:0.18736383060332323  Test set (out-of sample error): 0.27193468030735335\nNothing special! We should definately find better parameters. LEt us now try 70 30 split . The results are 0.23348722898699353 0.24321321252264558 0.3280610208183715 respectively. Now that is already better and 80/20 : 0.21209512149397639 0.22662413117187602 0.3066506555408157. I have to be honest I was suprised, but it just goes to show case the difference in using different splits when training. NN had more data to learn on (with 80/20, but 70/30 was not efficient), hence it had better results. NOTE: For neural networks is seems to be impossible to achieve absolute reproducibility. I understand that randomness comes from 2 sources. First of all when I use cross validation train_test_split function but there I defined random_state=42. Another source is the initialisation of the weights. Every time it is going to be different. I tried to tackle that on 2 ways: \n\n\nfrom numpy.random import seed\n\nseed(1)\n\nfrom tensorflow import set_random_seed\n\nset_random_seed(1)\n\nand\n\nkeras.initializers.RandomUniform(minval=-0.05, maxval=0.05, seed=123)\n\n\nWhat I did achieve is that the variability(reproducability) was smaller but it was not perfekt as with random forest. There I can achieve same results with every run, (even tough there is internal randomness in the RF algorithm itself) So I know it has something to do with NN. Luckily for me RF gives much better and stable results and that is mine final solution, but I spend to much time on NN just to throw it away.\n\n\n\n\nSecond model---random forest---RF  Than I tried to implement another tactic namely randomized search(for Hyperparameter tuning but this time random forest instead of NN. Then the process is straight foreward, I experimented a bit to know which parameters are good (low RMSE) and than I used this knowledge to built interval of parameters that I gave to the randomizedsearchCV. Eventually I do find better parameters (and actually a better model, RF outperforms NN) And the results are as follows:\nRandom forest on mine trial-and error hyperparameters validation set: 0.15017251130121254\n\n and test data (2018 data) RMSE: 0.1941060627117721 now when I used randomized search I get following results: RMSE on validation data set: 0.11813299983524722\n\nand RMSE on test data set (2018)  0.15562591092506128\n\nObviously we can see that new parameters perform better on 2018 (what we would hope for)\n\n\n\n"},{"metadata":{"_uuid":"6e8e6716ca0d29254f04cb39967dddd2f9566a0c"},"cell_type":"markdown","source":"## Findings ans lessons learned:\n"},{"metadata":{"_uuid":"887dd2495715af48bf076830a7e8f0e52418f9b8"},"cell_type":"markdown","source":"\n\n<span style=\"color:blue\">**Findings:**</span> There is not much to say here other than we confirmed quantitively what was already familiar, namely there is more flu in winter time, temperature, humidity and other factors influence/are connected in certain ways to the outcome of the epidemy. Certain values fall in the range of normal/expected values of every variable, everything beyond that is simply outlier. I would say that finding fall in scope of normal reasoning.\n\n<span style=\"color:blue\">**Lessons:**</span>I think the reader of this assignment can learn more about neural network hyper-parameter hardship. Trial and error wont work and we need to have more quantitative approach, like randomized-search. Grid search is out of the question until we get quant computers (run-time is awfull). Problems with reproducibility of NN, making the same initial weights for the artificial neural network is pretty hard. How can we implement randomized search for random forests. Different imputation techniques and also to think qualitatively and inspect the data, not only blindly apply the techniques(thats how I found out that some of the outliers in winter were deliberately set to 40, have I only applied those two techniques with standard deviation or inter-quartile range, I would have missed it!). Furthermore what are some interesting aggregation measures that we can introduce to our dataset(feature engineering, we can potentially get a better prediction if we make the features ourselves). How to compactly visualize the dataset in 3 plots, cross-validation with model_selection from sklearn and datetime library together with isocalendar function... There were a lot of interesting lessons learned, these are just a few that stuck for sure.\n\n"},{"metadata":{"_uuid":"b08a55d0b4c337948145d56b42d1cb00a99f6fb0"},"cell_type":"markdown","source":"**If you benefited in any way please upvote :)**"}],"metadata":{"hide_input":false,"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}