{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "0e0dcd94-0d70-276c-aa66-66b894038d45"
      },
      "source": [
        "Credits to Jonathan Dekthiar (@Born2Data).\n",
        "None of this is mine, except for some comments. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "a83368ab-5673-6cb8-89aa-e04c26c03cdb"
      },
      "outputs": [],
      "source": [
        "import os # appel syst\u00e8me\n",
        "import numpy as np # manipulations matricielles, un peu d'alg\u00e8bre lin\u00e9aire\n",
        "import pandas as pd # manipulation de tableau, jointure SQL, etc."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "a07afbf6-ba0b-53d6-1f71-c65be7988db5"
      },
      "source": [
        "Le pipeline est un tuyau dans lequel on envoie une donn\u00e9e brute \u00e0 l'entr\u00e9e, et on r\u00e9cup\u00e8re le r\u00e9sultat final \u00e0 la sortie.\n",
        "\n",
        "A l'entr\u00e9e, on va donner train_users_2.csv, et on devrait obtenir ce que l'algorithme en fait. Toutes les op\u00e9rations vont s'encha\u00eener \u00e0 l'int\u00e9rieur m\u00eame du pipeline. On va cr\u00e9er une ou des fonctions, \u00e0 appliquer \u00e0 l'int\u00e9rieur du tube, qui vont donner le mod\u00e8le de sortie.\n",
        "\n",
        "On doit d\u00e9clarer ces diff\u00e9rentes \u00e9tapes : on \u00e9tablit une routine de nettoyage. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "05dce959-13f1-7aae-cf66-0e00522fa58a"
      },
      "outputs": [],
      "source": [
        "df_train = pd.read_csv(\"../input/train_users_2.csv\")\n",
        "df_train.sample(n=5) # Ne montre que quelques lignes choisie au hasard"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "a4122beb-7bd7-f576-8227-0f97724ccb48"
      },
      "source": [
        "On charge les donn\u00e9es de test. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "20c7e8e6-a064-6b1e-97a3-505911fa47b7"
      },
      "outputs": [],
      "source": [
        "df_test = pd.read_csv(\"../input/test_users.csv\")\n",
        "df_test.sample(n=5)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "8299d744-4199-1f34-aa2c-3768ac421a2d"
      },
      "source": [
        "On souhaite appliquer un traitement sur les colonnes du train ET du test sans avoir \u00e0 \u00e9crire la ligne de code 2 fois. \n",
        "On va alors vouloir empiler les 2 tableaux.\n",
        "\n",
        "Il faut se m\u00e9fier de la colonne de premi\u00e8re r\u00e9servation"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "e25f2602-d1c4-0b6c-dabd-b37999bf2dce"
      },
      "outputs": [],
      "source": [
        "#On combine les 2 tableaux\n",
        "\n",
        "df_all = pd.concat((df_train, df_test), axis = 0, ignore_index = True)\n",
        "# on importe pas l'index car pandas num\u00e9rote les lignes et on ne veut pas que \u00e7a collisione\n",
        "df_all.head(n=5)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "ace2c391-4775-7e4a-f2ae-58679361e809"
      },
      "outputs": [],
      "source": [
        "# On supprime la colonne de 1\u00e8re r\u00e9servation qui est emb\u00eatante\n",
        "df_all.drop('date_first_booking', axis = 1, inplace = True)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "50ed73c5-b498-628c-58c5-01a713139177"
      },
      "outputs": [],
      "source": [
        "df_all.sample(n=5)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "1a819858-a8a4-987b-d4a3-2bff4afca0e1"
      },
      "source": [
        "Le standard de date n'est pas le m\u00eame dans le timestamp et le date_account_created..."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "57c0a935-8371-a568-494f-cab21d6194e1"
      },
      "outputs": [],
      "source": [
        "df_all['date_account_created'] = pd.to_datetime(df_all['date_account_created'], format = '%Y-%m-%d')\n",
        "df_all['timestamp_first_active'] = pd.to_datetime(df_all['timestamp_first_active'], format = '%Y%m%d%H%M%S')\n",
        "\n",
        "df_all.sample(n=5)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "51dbe20b-e1a6-6688-98e8-c7fea21bf222"
      },
      "source": [
        "On va maintenant nettoyer les valeurs aberrantes gr\u00e2ce \u00e0 cette petite fonction.\n",
        "\n",
        "On va consid\u00e9rer les personnes de - 15 ans et de + 90 ans comme des valeurs aberrantes et on va les supprimer de notre dataset."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "44cd1c85-124b-0b91-e4a2-934fcad776ce"
      },
      "outputs": [],
      "source": [
        "def remove_age_outliers(x, min_value=15, max_value=90): #operations logiques sur des tableaux\n",
        "    if np.logical_or(x<=min_value, x>=max_value):\n",
        "        return np.nan\n",
        "    else:\n",
        "        return x\n",
        "    "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "38868800-3d13-081c-72b0-dc67aa54f84b"
      },
      "outputs": [],
      "source": [
        "df_all['age'] = df_all['age'].apply(lambda x: remove_age_outliers(x) if(not np.isnan(x)) else x)\n",
        "# Pandas accepte qu'on applique une fonction sur toutes les valeurs d'une ligne ou d'une colonne\n",
        "# est-ce que naN est superieur ou \u00e9gal \u00e0 90? Comparaison pas toujours possible\n",
        "\n",
        "df_all['age'].head(n=20)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "700628b6-10c7-269d-1ca5-49295dfdfea3"
      },
      "outputs": [],
      "source": [
        "# on remplace les NaN par -1\n",
        "df_all['age'].fillna(-1, inplace=True)\n",
        "df_all.head(n=10)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "e4f90ec9-227b-f620-ca8f-877a0f9aeeab"
      },
      "outputs": [],
      "source": [
        "# L'age est \u00e9crit comme n r\u00e9el ! Conversion en entier.\n",
        "\n",
        "df_all.age = df_all.age.astype(int)\n",
        "df_all.age.sample(n=10)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "88d0b0cb-856c-00a1-2a85-4d8c6bd2eb5a"
      },
      "source": [
        "On va maintenant chercher les lignes avec des NaN pour s'en d\u00e9barasser."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "b7f98406-b46e-768f-fcb3-2e98bd9879e5"
      },
      "outputs": [],
      "source": [
        "def check_NaN_values_in_df(df):\n",
        "    for col in df: # col va \u00eatre chacune des colonnes\n",
        "        nan_count = df[col].isnull().sum() #nombre de valeurs nulles\n",
        "        \n",
        "        if nan_count != 0:\n",
        "            print(col + \" => \" + str(nan_count) + \" NaN values\") #nan_count is int => string"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "64aebf01-4a3d-fe4d-2757-dee9b81ed7da"
      },
      "outputs": [],
      "source": [
        "check_NaN_values_in_df(df_all)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "7ea23d86-3dc8-5bf2-f6c6-05818d81fd3c"
      },
      "outputs": [],
      "source": [
        "df_all['first_affiliate_tracked'].fillna(-1, inplace = True)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "bcb59520-777e-1677-ddf4-42e6b9855155"
      },
      "outputs": [],
      "source": [
        "check_NaN_values_in_df(df_all)\n",
        "df_all.sample(n=5)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "39c2613b-0471-b7fb-d99e-5836d0f9f848"
      },
      "source": [
        "On enl\u00e8ve le timestamp qui va pas nous servir \u00e9norm\u00e9ment et qui a un format qui ne pla\u00eet pas."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "dac6667e-fa98-556e-f9b7-a428561939fd"
      },
      "outputs": [],
      "source": [
        "df_all.drop('timestamp_first_active', axis = 1, inplace = True)\n",
        "df_all.sample(n=5)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "ff4a0805-b942-e35e-f35b-728e82edc8ce"
      },
      "outputs": [],
      "source": [
        "# Il faut faire attention avec ce que l'on supprime. N\u00e9anmoins,\n",
        "# il se peut qu'on retire de grosses informations (patterns)\n",
        "df_all.drop('language', axis = 1, inplace = True)\n",
        "df_all.sample(n=5)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "49c7641d-ab3f-fd94-34b3-de5c989766f9"
      },
      "source": [
        "Les utilisateurs sont les plus anciens sont particuliers : ils sont aventureux et veulent tester l'appli. Ils n'ont pas le comportement habituel et on d\u00e9cide de les retirer."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "e390fc46-71a5-c44b-a0f1-1e419064a08d"
      },
      "outputs": [],
      "source": [
        "df_all = df_all[df_all['date_account_created'] > '2013-02-01']\n",
        "df_all.sample(n=5)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "266e192a-2a85-2f68-51c7-14891dd9e7b4"
      },
      "outputs": [],
      "source": [
        "#creation du directory si n\u00e9cessaire\n",
        "if not os.path.exists(\"output\"):\n",
        "    os.makedirs(\"output\")\n",
        "    \n",
        "#exportation en CSV\n",
        "df_all.to_csv(\"output/cleaned.csv\", sep = \",\", index = False)"
      ]
    }
  ],
  "metadata": {
    "_change_revision": 0,
    "_is_fork": false,
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.0"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}