{ "cells": [ { "cell_type": "markdown", "metadata": { "_cell_guid": "2bbbbc7f-c1d7-27ba-91f9-29595edf1aa8" }, "source": [ "I am curious about how to use the information in the column \"Description\". I got some idea when I loged in the RentHop. As we can see from the website, some location information is posted on it. Such as \"tribeca\" ,\"bronx\" and so on. These words are also the key words in the column \"Description\". So I did some analysis about the location information. This make me some little improvement. But I think there still some important information I didn't find in the column \"Description\". I hope we can talk about it to make more improvement." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "cc53f53a-1f9c-1f6c-55d3-991347fe95e7" }, "outputs": [], "source": [ "#import some packages.\n", "import numpy as np\n", "import pandas as pd\n", "import string\n", "from sklearn.feature_extraction.text import CountVectorizer" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "f6ab1a06-bd1f-022b-f3da-8112c8b2f977" }, "source": [ "**We should do some analysis about how these key words impact the distribution of 'interest_level'.**" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "f0e45f75-49e6-e905-ef0b-788939c1e52f" }, "outputs": [], "source": [ "# Load traing data.\n", "data_path = \"../input/\"\n", "train_file = data_path + \"train.json\"\n", "train_df = pd.read_json(train_file).reset_index(drop=True)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "42b1706b-77fb-20d0-88af-bf071c8646e2" }, "outputs": [], "source": [ "# Compute the ratio of different interest_level.\n", "num = train_df.shape[0]\n", "low_num = np.where(train_df.interest_level == 'low')[0].shape[0]\n", "medium_num = np.where(train_df.interest_level == 'medium')[0].shape[0]\n", "high_num = np.where(train_df.interest_level == 'high')[0].shape[0]\n", "print('low:',low_num/float(num))\n", "print('medium:',medium_num/float(num))\n", "print('high:',high_num/float(num))" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "89356c97-b5a2-1e6e-27f1-f5a915b72bf9" }, "outputs": [], "source": [ "# Let's do some pretratment for the column \"Description\".Code tin this cell was obtained in MosMK's kernal.\n", "\n", "\n", "train_df['desc'] = train_df['description']\n", "train_df['desc'] = train_df['desc'].apply(lambda x: x.replace('


', ''))\n", "\n", "string.punctuation.__add__('!!')\n", "string.punctuation.__add__('(')\n", "string.punctuation.__add__(')')\n", "\n", "remove_punct_map = dict.fromkeys(map(ord, string.punctuation))\n", "\n", "train_df['desc'] = train_df['desc'].apply(lambda x: x.translate(remove_punct_map))" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "3940a446-9826-25fa-d7cc-0d823c9f207e" }, "outputs": [], "source": [ "# I change all the str into lower and drop all the blank.\n", "train_df['desc_str'] = train_df.desc.apply(lambda x : x.lower())\n", "train_df.desc_str = train_df.desc_str.apply(lambda x : x.replace(' ',''))" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "e376a9e6-717c-9060-5ef5-dda7c7e8dcfb" }, "outputs": [], "source": [ "# Define a function to analys the ratio of interest_level.\n", "def analysis_desc(df,str1):\n", " df['contain_'+str1] = train_df.desc_str.apply(lambda x : 1 if str1 in x else 0)\n", " df['contain_'+str1] = train_df.desc_str.apply(lambda x : 1 if str1 in x else 0)\n", " df['contain_'+str1] = train_df.desc_str.apply(lambda x : 1 if str1 in x else 0)\n", " df['contain_'+str1] = train_df.desc_str.apply(lambda x : 1 if str1 in x else 0)\n", " num = np.where(df['contain_'+str1] == 1)[0].shape[0]\n", " print(str1,':',num)\n", " low = np.where((df['contain_'+str1] == 1)&(df.interest_level == 'low'))[0].shape[0]\n", " medium = np.where((df['contain_'+str1] == 1)&(df.interest_level == 'medium'))[0].shape[0]\n", " high = np.where((df['contain_'+str1] == 1)&(df.interest_level == 'high'))[0].shape[0]\n", " print(low / float(num))\n", " print(medium / float(num))\n", " print(high / float(num))" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "742fd19c-fd2f-d403-a10d-d19717352c19" }, "outputs": [], "source": [ "# The features I found.\n", "\n", "desc_feat = ['batteryparkcity','bowery','chinatown','eastvillage','greenwichvillage','littleitaly',\n", " 'lowereastside','noho','nolita','soho','tribeca','twobridges','westvillage','chelsea',\n", " 'flatirondistrict','garmentdistrict','gramercypark','hellskitchen','kipsbay','midtowneast',\n", " 'murrayhill','nomad','stuyvesant','centralharlem','centralpark','eastharlem','inwood',\n", " 'uppereastside','upperwestside','washingtonheights','westharlem','rooseveltisland',\n", " 'randalls','bedfordstuyvesant','bushwick','greenpoint','williamsburg','boerumhill',\n", " 'carrollgardens','cobblehill','gowanus','parkslope','prospectpark','redhook','sunsetpark',\n", " 'windsorterrace','crownheights','flatbush','kensington','midwood','brooklynheights',\n", " 'clintonhill','dumbo','downtownbrooklyn','fortgreene','prospectheights','bayridge',\n", " 'bensonhurst','astoria','corona','elmhurst','foresthills','glendale','jacksonheights',\n", " 'longislandcity','regopark','ridgewood','sunnyside','woodside','millbasin','flushing',\n", " 'flushingmeadowscoronapark','kewgardens','kewgardenshills','whitestone','briarwood',\n", " 'forestpark','howar','ozonepark','woodhaven','bedfordpark','bronx','concourse','fordham',\n", " 'highbridge','kingsbridge','morris','mount','riverdale','university','morrispark',\n", " 'westchester','thewaterfront','westside','shore']\n", "for place in desc_feat:\n", " analysis_desc(train_df,place)" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "0cb5a29f-7645-7181-1ef2-7dc7eacb729f" }, "source": [ "**I listed more than ninety key words. Some of these features can affect the ratio of interest_level. You can choose the key words by yourself. I will give the function to save the selected features.**" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "854d18e7-1b61-a89d-b937-ec83ec6d9925" }, "outputs": [], "source": [ "# Load data.\n", "data_path = \"../input/\"\n", "train_file = data_path + \"train.json\"\n", "test_file = data_path + \"test.json\"\n", "train_df = pd.read_json(train_file)\n", "test_df = pd.read_json(test_file)\n", "train_test = pd.concat((train_df, test_df), axis=0).reset_index(drop=True)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "01794a19-e454-c420-4191-b4dd229d78c8" }, "outputs": [], "source": [ "# Some pretratment.\n", "train_test['desc'] = train_test['description']\n", "train_test['desc'] = train_test['desc'].apply(lambda x: x.replace('


', ''))\n", "string.punctuation.__add__('!!')\n", "string.punctuation.__add__('(')\n", "string.punctuation.__add__(')')\n", "remove_punct_map = dict.fromkeys(map(ord, string.punctuation))\n", "train_test['desc'] = train_test['desc'].apply(lambda x: x.translate(remove_punct_map))\n", "train_test['desc_str'] = train_test.desc.apply(lambda x:x.lower())\n", "train_test.desc_str = train_test.desc_str.apply(lambda x : x.replace(' ',''))" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "2f66e5fa-069b-29b8-bf8d-3908aeb940d2" }, "outputs": [], "source": [ "# The function used to save features.\n", "def analysis_desc(df,str1):\n", " df['contain_'+str1] = df.desc_str.apply(lambda x : 1 if str1 in x else 0)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "92364cf4-137f-e011-eaac-95f19bd1aa47" }, "outputs": [], "source": [ "# Add your features.\n", "desc_feat_example = desc_feat" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "96acf0ef-90ce-3152-6643-76dd93f9fd70" }, "outputs": [], "source": [ "# Create a new dataframe to save featues.\n", "desc_fea = pd.DataFrame()\n", "desc_feat_names = []\n", "for feat in desc_feat_example:\n", " analysis_desc(train_test,feat)\n", " temp = 'contain_'+feat\n", " desc_feat_names.append(temp)\n", " desc_fea = pd.concat((desc_fea,train_test[temp]),axis=1).reset_index(drop=True)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "bc403da6-96b9-bcdc-e06b-5cd321b886aa" }, "outputs": [], "source": [ "# Save features to your own path.\n", "desc_fea.to_csv('desc_fea.csv',index=False)" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "36b2f0c6-b8ed-62d3-ad75-e924388d726e" }, "source": [ "You can select the features that are sute for you. I can't guarantee that these features are food for you. But I think these location information did reflact some rule of interest_level. I hope we can talk more about the column \"Description\"." ] } ], "metadata": { "_change_revision": 0, "_is_fork": false, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.0" } }, "nbformat": 4, "nbformat_minor": 0 }