{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "08ed7b38-b846-6f47-8862-37ac60133a41"
      },
      "source": [
        "technical_20 and technical_30 - two sides of one coin"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "85d6aa18-b11c-a9a6-6d35-64273703a7f4"
      },
      "outputs": [],
      "source": [
        "import numpy as np \n",
        "import pandas as pd"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "b0476265-ee1e-1d71-c692-3676f5a380ef"
      },
      "outputs": [],
      "source": [
        "data = pd.read_hdf('../input/train.h5')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "99803f34-9acf-e405-7970-4f0b005599a8"
      },
      "source": [
        "I don't know if  somebody has noticed, data set is divided on 3 parts: technical_20 has nonzero value, technical_30 has nonzero value, they both zero or null. And first two part are separate! We have one of this features nonzero only when second is zero or null.  \n",
        "When i say nonzero, i mean (!= 0 and != null)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "a3228b3a-ca99-0a8f-983e-6c19bb1f2845"
      },
      "outputs": [],
      "source": [
        "t20_size = ((data.technical_20 != 0) & (~data.technical_20.isnull())).sum()\n",
        "t30_size = ((data.technical_30 != 0) & (~data.technical_30.isnull())).sum()\n",
        "other = data.shape[0] - t20_size - t30_size"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "9bf83749-6232-0ade-062a-423a712f1ba4"
      },
      "outputs": [],
      "source": [
        "t20_rows = (data.technical_20 != 0) & (~data.technical_20.isnull())\n",
        "t30_rows = (data.technical_30 != 0) & (~data.technical_30.isnull())\n",
        "joined_rows = t20_rows & t30_rows\n",
        "other_rows = (~t20_rows) & (~t30_rows)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "cb3c3325-490b-07ff-14b0-d9d26b3eaa9b"
      },
      "source": [
        "Proof"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "e4d394fa-5226-b9ca-3b30-beca7dca34e5"
      },
      "outputs": [],
      "source": [
        "print ('Full dataset ->', data.shape[0],'\\n','technical_20 has value ->', t20_size,'\\n',\n",
        "       'technical_30 has a value ->', t30_size,'\\n',       \n",
        "      'both technical_20 and technical_30 does not have a value->',other,'\\n',\n",
        "      'both technical_20 and technical_30 has a value->',joined_rows.sum(),'\\n')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "965eafe8-e663-5a70-f83a-560bfde6d2be"
      },
      "source": [
        "One more proof"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "e45d4a28-7b01-43d2-49fa-ff6d4902ae83"
      },
      "outputs": [],
      "source": [
        "data.loc[t20_rows, 'technical_20'].describe()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "831e0fab-c9e6-abcc-4f54-db6bb3cd463b"
      },
      "outputs": [],
      "source": [
        "data.loc[t30_rows, 'technical_20'].describe()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "0d6edc36-431f-9564-60cd-099f0e6e467c"
      },
      "outputs": [],
      "source": [
        "data.loc[t30_rows, 'technical_30'].describe()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "7dd651bb-0d75-9a3c-8d21-e73cc4cad468"
      },
      "outputs": [],
      "source": [
        "data.loc[t20_rows, 'technical_30'].describe()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "f68014b9-1cae-8e18-ac60-832308a9b74c"
      },
      "source": [
        "I saw a lot of competitors made a regression just on two of this values and filled NA by means. So for 1/3 of data set, they produced output = weighted means."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "71e0d746-464a-8320-b755-ff531c1b5114"
      },
      "source": [
        "And furthermore, mean of y, when technical_20 is nonzero and mean of y, when technical_30 is nonzero, is on the different sides, even by sign"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "ff523d27-6fec-e3b5-4d62-4d1f8a1d6793"
      },
      "outputs": [],
      "source": [
        "print ('y mean when technical_20 nonzero -> ','{:.4}'.format(data.loc[t20_rows,'y'].mean()),'\\n',\n",
        "       'y mean in residual data set -> ','{:.4}'.format(data.loc[other_rows,'y'].mean()),'\\n',\n",
        "      'y mean when technical_30 nonzero -> ','{:.4}'.format(data.loc[t30_rows,'y'].mean()))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "df94e2db-c88d-2f12-9386-8ee1ff00c4c7"
      },
      "outputs": [],
      "source": [
        ""
      ]
    }
  ],
  "metadata": {
    "_change_revision": 0,
    "_is_fork": false,
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.0"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}