{"cells":[{"metadata":{"_uuid":"4b828c46f913d3767c01de7e2d6192d5706e4619","colab_type":"text","id":"BGi45RunhgSq"},"cell_type":"markdown","source":"## Data Cleaning and Shape Examining \n"},{"metadata":{"_uuid":"e3b4336bd38793d5b3337c475d2897ac9d534369"},"cell_type":"markdown","source":"## Home Depot Product Search Relevance\n\n"},{"metadata":{"_uuid":"2ad7ddccf61efdb2e9e7dd131aa7ae38f6dc6543","colab":{"base_uri":"https://localhost:8080/","height":419},"colab_type":"code","id":"a0Sqe1x7hjzu","outputId":"da28f8a4-fb3c-4505-d9c5-34726bfb4246"},"cell_type":"markdown","source":"Shoppers rely on Home Depot’s product authority to find and buy the latest products and to get timely solutions to their home improvement needs. From installing a new ceiling fan to remodeling an entire kitchen, with the click of a mouse or tap of the screen, customers expect the correct results to their queries – quickly. Speed, accuracy and delivering a frictionless customer experience are essential.\n\nIn this competition, Home Depot is asking Kagglers to help them improve their customers' shopping experience by developing a model that can accurately predict the relevance of search results.\n\nSearch relevancy is an implicit measure Home Depot uses to gauge how quickly they can get customers to the right products. Currently, human raters evaluate the impact of potential changes to their search algorithms, which is a slow and subjective process. By removing or minimizing human input in search relevance evaluation, Home Depot hopes to increase the number of iterations their team can perform on the current search algorithms."},{"metadata":{"_uuid":"51f84f9e6e4eb70cc4a3f2bfb3123cb780793b4d"},"cell_type":"markdown","source":"### Data description"},{"metadata":{"_uuid":"f4b1ab735d3340ec4e30709deb4d6ee818da81bb"},"cell_type":"markdown","source":"This data set contains a number of products and real customer search terms from Home Depot's website. The challenge is to predict a relevance score for the provided combinations of search terms and products. To create the ground truth labels, Home Depot has crowdsourced the search/product pairs to multiple human raters.\n\nThe relevance is a number between 1 (not relevant) to 3 (highly relevant). For example, a search for \"AA battery\" would be considered highly relevant to a pack of size AA batteries (relevance = 3), mildly relevant to a cordless drill battery (relevance = 2), and not relevant to a snow shovel (relevance = 1).\n\nEach pair was evaluated by at least three human raters. The provided relevance scores are the average value of the ratings. There are three additional things to know about the ratings:\n\nThe specific instructions given to the raters is provided in relevance_instructions.docx.\nRaters did not have access to the attributes.\nRaters had access to product images, while the competition does not include images.\nYour task is to predict the relevance for each pair listed in the test set. Note that the test set contains both seen and unseen search terms.\n\n"},{"metadata":{"_uuid":"829d3edad1ab04e981e64897e3a3b9fe6aeb8c22"},"cell_type":"markdown","source":"### File descriptions\n\n- train.csv - the training set, contains products, searches, and relevance scores\n- test.csv - the test set, contains products and searches. You must predict the relevance for these pairs.\n- product_descriptions.csv - contains a text description of each product. You may join this table to the training or test set via the product_uid.\n- attributes.csv -  provides extended information about a subset of the products (typically representing detailed technical specifications). Not every product will have attributes.\n- sample_submission.csv - a file showing the correct submission format\n- relevance_instructions.docx - the instructions provided to human raters\n\n### Data fields\n\n- id - a unique Id field which represents a (search_term, product_uid) pair\n- product_uid - an id for the products\n- product_title - the product title\n- product_description - the text description of the product (may contain HTML content)\n- search_term - the search query\n- relevance - the average of the relevance ratings for a given id\n- name - an attribute name\n- value - the attribute's value"},{"metadata":{"_uuid":"1720d9873a1549cd29ce34e9c821ca3628dfd3fe","colab":{},"colab_type":"code","id":"nqvGk3kkhgSv","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nfrom sklearn.ensemble import RandomForestRegressor, BaggingRegressor\nfrom nltk.stem.snowball import SnowballStemmer\n\nstemmer = SnowballStemmer('english')\n\n# training_data = pd.read_csv(\"input/train.csv.zip\", encoding=\"ISO-8859-1\")\n# testing_data = pd.read_csv(\"input/test.csv.zip\", encoding=\"ISO-8859-1\")\n# attribute_data = pd.read_csv('input/attributes.csv.zip')\n# descriptions = pd.read_csv('input/product_descriptions.csv.zip')\n\n\ntraining_data = pd.read_csv(\"../input/train.csv\", encoding=\"ISO-8859-1\")\ntesting_data = pd.read_csv(\"../input/test.csv\", encoding=\"ISO-8859-1\")\nattribute_data = pd.read_csv('../input/attributes.csv')\ndescriptions = pd.read_csv('../input/product_descriptions.csv')\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"020cf9f3a3e83fe0699443affc49d8916d77a7a9"},"cell_type":"markdown","source":"Let's try to examing the data and try to spot if there are anything suspicious about it"},{"metadata":{"_uuid":"7f54e9a8f8cc18605e917d47aca6feebfb97b151","colab":{"base_uri":"https://localhost:8080/","height":34},"colab_type":"code","id":"78PcQC__hgS4","outputId":"862af9df-97c3-49be-ed7e-5d19ed08c58d","trusted":true},"cell_type":"code","source":"print(\"training data shape is:\",training_data.shape)\nprint(\"testing data shape is:\",testing_data.shape)\nprint(\"attribute data shape is:\",attribute_data.shape)\nprint(\"description data shape is:\",descriptions.shape)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b3db1baf79ee00bfc2c6849f4da2531140fe812c","trusted":true},"cell_type":"code","source":"print(\"training data has empty values:\",training_data.isnull().values.any())\nprint(\"testing data has empty values:\",testing_data.isnull().values.any())\nprint(\"attribute data has empty values:\",attribute_data.isnull().values.any())\nprint(\"description data has empty values:\",descriptions.isnull().values.any())","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ba6adde21c36f96b628d62ada0237fca408a2c9e","trusted":true},"cell_type":"code","source":"training_data.head(10)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c49f6559257da5a67a964a6789c28f06e747dea8","trusted":true},"cell_type":"code","source":"print(\"there are in total {} products \".format(len(training_data.product_title.unique())))\nprint(\"there are in total {} search query \".format(len(training_data.search_term.unique())))\nprint(\"there are in total {} product_uid\".format(len(training_data.product_uid.unique())))\n\n\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"80a1108ce36b78864021d1c5155a507f7c669849","trusted":true},"cell_type":"code","source":"testing_data.head(10)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"399d2e99fcdd1e640732df5a2a36210100e22578","trusted":true},"cell_type":"code","source":"print(\"there are in total {} products \".format(len(testing_data.product_title.unique())))\nprint(\"there are in total {} search query \".format(len(testing_data.search_term.unique())))\nprint(\"there are in total {} product_uid\".format(len(testing_data.product_uid.unique())))\n\n\n\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9fad4704439fb863feb80a4ff55e2ee24cceb990","trusted":true},"cell_type":"code","source":"attribute_data.head(10)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"582d10cd507c4701d165bd404301ccc2bd2d5dbb","trusted":true},"cell_type":"code","source":"print(\"there are in total {} product_uid \".format(len(attribute_data.product_uid.unique())))\nprint(\"there are in total {} names \".format(len(attribute_data.name.unique())))\nprint(\"there are in total {} values\".format(len(attribute_data.value.unique())))\n\n\n\n\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2006babf19f052461630781ae3f55f72ce1b736a","trusted":true},"cell_type":"code","source":"descriptions.head(10)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9d9a3ab307cfa242c5c2aa6dbf831582f928542d","trusted":true},"cell_type":"code","source":"print(\"there are in total {} product_uid \".format(len(descriptions.product_uid.unique())))\nprint(\"there are in total {} product_descriptions \".format(len(descriptions.product_description.unique())))\n\n\n\n\n\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"88c82f14d1b984f0d9b087bad0cdb24ddb79265e","trusted":true},"cell_type":"code","source":"(descriptions.product_description.str.count('\\d+') + 1).hist(bins=30)\n(descriptions.product_description.str.count('\\W')+1).hist(bins=30)\n\n\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0fe9e2b18bb62e69cd224063e2555e545db44366","trusted":true},"cell_type":"code","source":"(training_data.product_title.str.count(\"\\\\d+\") + 1).hist(bins=30)#plot number of digits in title\n(training_data.product_title.str.count(\"\\\\w+\") + 1).hist(bins=30)#plot number of digits in title\n\n\n\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7dbbf22bf617684b4596582184d201c00f475a8d","trusted":true},"cell_type":"code","source":"(training_data.search_term.str.count(\"\\\\w+\") + 1).hist(bins=30) #plot number of words in search therms\n(training_data.search_term.str.count(\"\\\\d+\") + 1).hist(bins=30) #plot number of digits in search terms\n\n\n\n\n\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8b057f501ef6959d717cd296b7f081e0c352734f","trusted":true},"cell_type":"code","source":"(training_data.relevance ).hist(bins=30)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0d2f171acac6e498c91babd27138a124a2940d85","trusted":true},"cell_type":"code","source":"%matplotlib inline\nimport matplotlib.pyplot as plt\nfrom scipy.stats import norm  \n\ntraining_data.relevance.plot(kind='hist', normed=True)\n\nmu, std = norm.fit(training_data.relevance)\n\nxmin, xmax = plt.xlim()\nx = np.linspace(xmin, xmax, 100)\np = norm.pdf(x, mu, std)\nplt.plot(x, p, 'k', linewidth=2)\ntitle = \"Fit results: mu = %.2f,  std = %.2f\" % (mu, std)\nplt.title(title)\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"839220074019d4a13f9a32ccbbef421b34571d41","trusted":true},"cell_type":"code","source":"print('total data has html tags in',descriptions.product_description.str.count('<br$').values.sum())","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4d6ca2cb53fb5a63331b3b2afeaef2931cb53bdb","scrolled":true,"trusted":true},"cell_type":"code","source":"descriptions[descriptions.product_description.str.contains(\"<br\")].values.tolist()[:3]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"20f52a4878122cf8bc113a6cb5999d10ff107489","trusted":true},"cell_type":"code","source":"descriptions.product_description.str.contains(\"Click here to review our return policy for additional information regarding returns\").values.sum()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2627b4064d30cbf897dee3b908a0307f0bb33bbd","trusted":true},"cell_type":"code","source":"training_data[training_data.search_term.str.contains(\"^\\\\d+ . \\\\d+$\")].head(10)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"48ea05bbb1a1862c0f948c4a1ef81c67e5dea0c5","trusted":true},"cell_type":"code","source":"training_data[training_data.product_uid==100030]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1b299add8862de2f5bc1b3f52d053171edf5630f"},"cell_type":"markdown","source":"from above the following conclusion follows. \n   - At first there exists fields which has html tags in for __description__ dataset. (maybe and error made by the scrapper) along with _Click here to review our return policy_\n   - There is no missing/empty values in any of these datasets \n   - in dataset __description__ field product_description contains more digits than word characters\n   - some query in dataset __training__ are too straight, it's hard to guess exactly what user meant in terms of broad  sense\n   - some of the search query in dataset __training__ has too specific meaning like 8 4616809045 9\t\n   - number of diggits appearence in the product title tends to be greater number of characters for dataset __training__ (and the same is true for search query field)\n   - the relevancy score is between 1 and 3. Because the density of product whose relevancy score is between 2 and 3 is higher we can conclude that most of search query has been classifield between 2 and 3\n   - The histogram of relevancy score doesn't follow standard distribution pattern\n   \n   \n   \nIn order to continue the analysis we will need the whole datasets\n   - description datasets might be joined together to training by the product_uid (the same holds for attribute datasets) then clean the html parts\n   \n   "},{"metadata":{"_uuid":"4b29cc4a47fe31f09dc5ec91aa251767201f72cc"},"cell_type":"markdown","source":"## Data cleaning"},{"metadata":{"_uuid":"66018a689e75b4413ae9f944f625378e42e6bb55","colab":{"base_uri":"https://localhost:8080/","height":314},"colab_type":"code","id":"mD3zyodThgTD","outputId":"e4611f88-2a09-4554-dcf3-f6995b02796a","trusted":true},"cell_type":"code","source":"## let's create first the cleaning functions\nfrom bs4 import BeautifulSoup\nimport lxml\nimport re\nimport nltk\nfrom nltk.corpus import stopwords # Import the stop word list\nfrom nltk.metrics import edit_distance\nfrom string import punctuation\nfrom collections import Counter\n\n\ndef remove_html_tag(text):\n    soup = BeautifulSoup(text, 'lxml')\n    text = soup.get_text().replace('Click here to review our return policy for additional information regarding returns', '')\n    return text\n\ndef str_stemmer(doc):\n    # split into tokens by white space\n    tokens = doc.split()\n    # remove punctuation from each token\n    table = str.maketrans('', '', punctuation)\n    tokens = [w.translate(table) for w in tokens]\n    # remove remaining tokens that are not alphabetic\n    tokens = [word for word in tokens if word.isalpha()]\n    # filter out stop words\n    stop_words = set(stopwords.words('english'))\n    tokens = [w for w in tokens if not w in stop_words]\n    # filter out short tokens\n    tokens = [word for word in tokens if len(word) > 1]\n    return ' '.join(tokens)\n\ndef str_stemmer_title(s):\n#     return \" \".join([stemmer.stem(word) for word in s.lower().split()])\n    return \" \".join(map(stemmer.stem, s.lower().split()))\n\ndef str_common_word(str1, str2):\n    whole_set = set(str1.split())\n#     return sum(int(str2.find(word)>=0) for word in whole_set)\n    return sum(int(str2.find(word)>=0) for word in whole_set)\n\n\ndef get_shared_words(row_data):\n    return np.sum([str_common_word(*row_data[:-1]), str_common_word(*row_data[1:])])\n\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"71ff92472875e008fe1a04bc2c64267bd960c76b","trusted":true},"cell_type":"code","source":"############### cleaning html tags ##################\nhas_tag_in = descriptions.product_description.str.contains('<br')\ndescriptions.loc[has_tag_in, 'product_description'] = descriptions.loc[has_tag_in, 'product_description'].map(lambda x:remove_html_tag(x))\n###############","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a6295bb7eca8efb42e364a95169bec7b576dcae5","colab_type":"text","id":"n-UOBiz0hgTt"},"cell_type":"markdown","source":"Examing the search query in the datasets __training__, there some misspelings for field _search_term_ contains a lot of misspelling (more than 3000). This might be fixed by using Google API "},{"metadata":{"_uuid":"ab4ac5796055b4d664d838a62ec7a5cf56947e85","trusted":true},"cell_type":"code","source":"import requests\nimport re\nimport time\nfrom random import randint\n\nSTART_SPELL_CHECK=\"<span class=\\\"spell\\\">Showing results for</span>\"\nEND_SPELL_CHECK=\"<br><span class=\\\"spell_orig\\\">Search instead for\"\nHTML_Codes = ((\"'\", '&#39;'),('\"', '&quot;'),('>', '&gt;'),('<', '&lt;'),('&', '&amp;'))\n\ndef spell_check(s):\n    q = '+'.join(s.split())\n    time.sleep(  randint(0,1) ) #relax and don't let google be angry\n    r = requests.get(\"https://www.google.co.uk/search?q=\"+q)\n    content = r.text\n    start=content.find(START_SPELL_CHECK) \n    if ( start > -1 ):\n        start = start + len(START_SPELL_CHECK)\n        end=content.find(END_SPELL_CHECK)\n        search= content[start:end]\n        search = re.sub(r'<[^>]+>', '', search)\n        for code in HTML_Codes:\n            search = search.replace(code[1], code[0])\n        search = search[1:]\n    else:\n        search = s\n    return search ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8e4afafc8871c75d1395afe354536facd601c1fd"},"cell_type":"markdown","source":"Indeed correcting the misspelings words might help, due to ability of reproducing the result at Kaggle, we won't do spell correction"},{"metadata":{"_uuid":"c86493591b2c80347dbb30794d17752dccb2999c","colab":{"base_uri":"https://localhost:8080/","height":175},"colab_type":"code","id":"qLK6NtZ7hgUJ","outputId":"36df4ff3-d9c4-4a17-a38a-859fc59ea915","trusted":true},"cell_type":"code","source":"training_data = pd.merge(training_data, descriptions, \n                         on=\"product_uid\", how=\"left\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e4a223cf4cf2162a5e723d69bd29287e958eb434","colab":{"base_uri":"https://localhost:8080/","height":175},"colab_type":"code","id":"BgBG3ClMhgUQ","outputId":"7e8fddb7-b57b-4ed1-c1d0-120cdf68b63d","trusted":true},"cell_type":"code","source":"print(\"It has blank/empty fields \",training_data.isnull().values.sum())\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"639f7f67ed43914827dfd58d792fd1bbf2b37173"},"cell_type":"markdown","source":"## Feature Engineering"},{"metadata":{"_uuid":"5311e61cddb38adad6b0ff78227d66ac23b28123"},"cell_type":"markdown","source":"### Plan\nWe are going to do the following:\n0. Join dataset __training__ with __description__  by  _product uid_ (already done)\n\n2. Create num columns based on text columns\n    - count number of words from search query which appears both in product_title and product_description\n    - compute edit distnace from search query which appears both in product_title and product_title\n    - compute the cosine similarity between search query, product_title and product_description\n    - count number of words in the product description\n    - create new columns for each pair\n    \n3. Remove all text columns\n\nAs a result we will have vectors of numbers that suites well for the machine learning."},{"metadata":{"_uuid":"8a1fa3b6e3db9ff746ac1fc82425b27482892611","trusted":true},"cell_type":"code","source":"print(\"has blank/empty values\",training_data.isnull().values.any())","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"909e0b15aad161808c49ba64d0427b9acf5d6ef8","trusted":true},"cell_type":"code","source":"from nltk.corpus import brown, stopwords\nfrom nltk.cluster.util import cosine_distance\nfrom sklearn.feature_extraction.text import CountVectorizer\nfrom sklearn.metrics.pairwise import cosine_similarity\nfrom collections import Counter\n\n\ndef sentence_similarity(columns,stopwords=None):\n    sent1, sent2 = columns[0], columns[1]\n    sent1 = sent1.split(' ')\n    sent2 = sent2.split(' ')\n    if stopwords is None:\n        stopwords = []\n \n    sent1 = [w.lower() for w in sent1]\n    sent2 = [w.lower() for w in sent2]\n \n    all_words = list(set(sent1 + sent2))\n \n    vector1 = [0] * len(all_words)\n    vector2 = [0] * len(all_words)\n \n    # build the vector for the first sentence\n    for w in sent1:\n        if w in stopwords:\n            continue\n        vector1[all_words.index(w)] += 1\n \n    # build the vector for the second sentence\n    for w in sent2:\n        if w in stopwords:\n            continue\n        vector2[all_words.index(w)] += 1\n \n    return 1 - cosine_distance(vector1, vector2)\n\ndef get_jaccard_sim(columns): \n    str1, str2 = columns[0], columns[1]\n    a = set(str1.split()) \n    b = set(str2.split())\n    c = a.intersection(b)\n    return float(len(c)) / (len(a) + len(b) - len(c))\n\n\ndef calc_edit_dist(row):\n    return edit_distance(*row)\n\n    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b163917befd2cbd5ce2ef996b2dce14e668d0fe0"},"cell_type":"code","source":"################begin testing\n## let's create first the cleaning functions\nfrom bs4 import BeautifulSoup\nimport lxml\nimport re\nimport nltk\nfrom nltk.corpus import stopwords # Import the stop word list\nfrom nltk.metrics import edit_distance\nfrom string import punctuation\nfrom collections import Counter\n\n\ndef remove_html_tag(text):\n    soup = BeautifulSoup(text, 'lxml')\n    text = soup.get_text().replace('Click here to review our return policy for additional information regarding returns', '')\n    return text\n\ndef str_stemmer(doc):\n    # split into tokens by white space\n    tokens = doc.split()\n    # remove punctuation from each token\n    table = str.maketrans('', '', punctuation)\n    tokens = [w.translate(table) for w in tokens]\n    # remove remaining tokens that are not alphabetic\n    tokens = [word for word in tokens if word.isalpha()]\n    # filter out stop words\n    stop_words = set(stopwords.words('english'))\n    tokens = [w for w in tokens if not w in stop_words]\n    # filter out short tokens\n    tokens = [word for word in tokens if len(word) > 1]\n    return ' '.join(tokens)\n\n\ndef str_stemmer_tokens(tokens):\n    # split into tokens by white space\n#     tokens = doc.split()\n    # remove punctuation from each token\n    table = str.maketrans('', '', punctuation)\n    tokens = [w.translate(table) for w in tokens]\n    # remove remaining tokens that are not alphabetic\n    tokens = [word for word in tokens if word.isalpha()]\n    # filter out stop words\n    stop_words = set(stopwords.words('english'))\n    tokens = [w for w in tokens if not w in stop_words]\n    # filter out short tokens\n    tokens = [word for word in tokens if len(word) > 1]\n    return ' '.join(tokens)\n\ndef str_stemmer_title(s):\n    return \" \".join(map(stemmer.stem, s))\n\ndef str_common_word(str1, str2):\n    whole_set = set(str1.split())\n#     return sum(int(str2.find(word)>=0) for word in whole_set)\n    return sum(int(str2.find(word)>=0) for word in whole_set)\n\n\n# def str_common_word(str1, str2):\n#     return sum(int(str2.find(word)>=0) for word in str1.split())\n\n\ndef str_common_word2(str1, str2):\n    part_of_first = set(str1)\n    return sum(1 for word in str2 if word in part_of_first)\n#     return sum(int(str2.find(word)>=0) for word in str1.split())\n\ndef get_shared_words_mut(row_data):\n    return np.sum([str_common_word2(*row_data[:-1]), str_common_word2(*row_data[1:])])\n\n\ndef get_shared_words_imut(row_data):\n    return np.sum([str_common_word(*row_data[:-1]), str_common_word2(*row_data[1:])])\n    \nfrom nltk.corpus import brown, stopwords\nfrom nltk.cluster.util import cosine_distance\nfrom sklearn.feature_extraction.text import CountVectorizer\nfrom sklearn.metrics.pairwise import cosine_similarity\nfrom collections import Counter\n\n\ndef sentence_similarity(columns,stopwords=None):\n    sent1, sent2 = columns[0], columns[1]\n    if stopwords is None:\n        stopwords = []\n \n    sent1 = [w.lower() for w in sent1]\n    sent2 = [w.lower() for w in sent2]\n \n    all_words = list(set(sent1 + sent2))\n \n    vector1 = [0] * len(all_words)\n    vector2 = [0] * len(all_words)\n \n    # build the vector for the first sentence\n    for w in sent1:\n        if w in stopwords:\n            continue\n        vector1[all_words.index(w)] += 1\n \n    # build the vector for the second sentence\n    for w in sent2:\n        if w in stopwords:\n            continue\n        vector2[all_words.index(w)] += 1\n \n    return 1 - cosine_distance(vector1, vector2)\n\ndef get_jaccard_sim(columns): \n    str1, str2 = columns[0], columns[1]\n    a = set(str1) \n    b = set(str2)\n    c = a.intersection(b)\n    return float(len(c)) / (len(a) + len(b) - len(c))\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"909e0b15aad161808c49ba64d0427b9acf5d6ef8","trusted":true},"cell_type":"code","source":"############## apply stemming #####################\n#  also .apply(, raw=True) might be a good options\n# https://github.com/s-heisler/pycon2017-optimizing-pandas to see why it is done on this way\n############## apply stemming #####################\n# training_data['search_term'] = list(map(str_stemmer_title,training_data['search_term'].values))\n# training_data['product_title'] = list(map(str_stemmer, training_data['product_title'].values))\n# training_data['product_description'] = list(map(str_stemmer, training_data['product_description'].values))\n\n# training_data['shared_words'] = list(map(get_shared_words, training_data[['search_term','product_description', 'product_title']].values))\n\n# training_data[\"edistance_sprot\"] = list(map(calc_edit_dist, training_data[[\"search_term\",\"product_title\"]].values))\n# training_data[\"edistance_sd\"] = list(map(calc_edit_dist, training_data[[\"search_term\",\"product_description\"]].values))\n\n\n# training_data['cos_dis_sqt'] = list(map(sentence_similarity ,training_data[[\"search_term\",\"product_title\"]].values))\n# training_data['cos_dis_sqd'] = list(map(sentence_similarity, training_data[[\"search_term\",\"product_description\"]].values))\n\n\n# training_data['j_dis_sqt'] = list(map(get_jaccard_sim, training_data[[\"search_term\",\"product_title\"]].values))\n# training_data['j_dis_sqd'] = list(map(get_jaccard_sim, training_data[[\"search_term\",\"product_description\"]].values))\n\n# training_data['j_dis_sqt'] = list(map(get_jaccard_sim, training_data[[\"search_term\",\"product_title\"]].values))\n# training_data['j_dis_sqd'] = list(map(get_jaccard_sim, training_data[[\"search_term\",\"product_description\"]].values))\n\n# training_data['search_query_length'] = training_data.search_term.str.len()\n# training_data['number_of_words_in_descr'] = training_data.product_description.str.count(\"\\\\w+\")\n\n\n\ntraining_data['search_term_tokens'] = training_data.search_term.str.lower().str.split()\ntraining_data['product_title_tokens'] = training_data.product_title.str.lower().str.split()\ntraining_data['product_description_tokens'] = training_data.product_description.str.lower().str.split()\n\ntraining_data['search_term'] = [str_stemmer_title(_) for _ in training_data.search_term_tokens.values.tolist()]\ntraining_data['product_title'] = [str_stemmer_tokens(_) for _ in training_data.product_title_tokens.values.tolist()]\ntraining_data['product_description'] = [str_stemmer_tokens(_) for _ in training_data.product_description_tokens.values.tolist()]\n\n\ntraining_data['shared_words_mut'] = [get_shared_words_mut(columns)\n                         for columns in \n                         training_data[['search_term_tokens', 'product_title_tokens', 'product_description_tokens']].values.tolist()\n                        ]\n\ntraining_data['shared_words'] = list(map(get_shared_words_imut, training_data[['search_term','product_description', 'product_title']].values))\n\n\n\ntraining_data[\"edistance_sprot\"] = [edit_distance(word1, word2) for word1, word2 in\n                                    training_data[[\"search_term\",\"product_title\"]].values.tolist()]\n\n\ntraining_data[\"edistance_sd\"] = [edit_distance(word1, word2) for word1, word2 in\n                                    training_data[[\"search_term\",\"product_description\"]].values.tolist()]\n\ntraining_data['j_dis_sqt'] = [get_jaccard_sim(rows) for rows in training_data[[\"search_term_tokens\",\"product_title_tokens\"]].values]\ntraining_data['j_dis_sqd'] = [get_jaccard_sim(rows) for rows in training_data[[\"search_term_tokens\",\"product_description_tokens\"]].values]\n\ntraining_data['search_query_length'] = training_data.search_term.str.len()\ntraining_data['number_of_words_in_descr'] = training_data.product_description.str.count(\"\\\\w+\")\n\n\ntraining_data['cos_dis_sqt'] = [ sentence_similarity(rows) for rows in training_data[[\"search_term\",\"product_title\"]].values]\ntraining_data['cos_dis_sqd'] = [sentence_similarity(rows) for rows in training_data[[\"search_term\",\"product_description\"]].values]\n\n\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3f43b737fe8f26d6d1ff698a46b60487904407a1","trusted":true},"cell_type":"code","source":"# training_data.corr()\ntraining_data.head(3)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"06dcccfb8ce72535c7047348a9907238ca239337"},"cell_type":"markdown","source":"__test dataset__\nwe have to have to apply symmetric transformation for both data set, except relevance score field since it is target field. Except we are not allow to take any actions which might lead to overfitting the data"},{"metadata":{"_uuid":"ba3393fdd9c87acbc22a31433dc80d58ed29d0ee","trusted":true},"cell_type":"code","source":"testing_data = pd.merge(testing_data, descriptions, \n                         on=\"product_uid\", how=\"left\")\nprint(\"has blank/empty values\",testing_data.isnull().values.any())","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8a340438dab93bc53d312c0877fdf0be72c3330e","trusted":true},"cell_type":"code","source":"############## apply stemming for test data #####################\n# testing_data['search_term'] = list(map(str_stemmer_title, testing_data['search_term'].values))\n# testing_data['product_title'] = list(map(str_stemmer, testing_data['product_title'].values))\n# testing_data['product_description'] = list(map(str_stemmer, testing_data['product_description'].values))\ntesting_data['search_term_tokens'] = testing_data.search_term.str.lower().str.split()\ntesting_data['product_title_tokens'] = testing_data.product_title.str.lower().str.split()\ntesting_data['product_description_tokens'] = testing_data.product_description.str.lower().str.split()\n\ntesting_data['search_term'] = [str_stemmer_title(_) for _ in testing_data.search_term_tokens.values.tolist()]\ntesting_data['product_title'] = [str_stemmer_tokens(_) for _ in testing_data.product_title_tokens.values.tolist()]\ntesting_data['product_description'] = [str_stemmer_tokens(_) for _ in testing_data.product_description_tokens.values.tolist()]\n\n############## end stemming #####################","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"be5828d4e0308d63f5144c328334b5c41968b196","trusted":true},"cell_type":"code","source":"############## building custome feature for test data, let's build a few of them before compare which one is the best ###########\n# testing_data['shared_words'] = list(map(get_shared_words, testing_data[['search_term','product_description', 'product_title']].values))\n# testing_data[\"edistance_sprot\"] = list(map(calc_edit_dist, testing_data[[\"search_term\",\"product_title\"]].values))\n# testing_data[\"edistance_sd\"] = list(map(calc_edit_dist, testing_data[[\"search_term\",\"product_description\"]].values))\n\n\n# testing_data['cos_dis_sqt'] = list(map(sentence_similarity ,testing_data[[\"search_term\",\"product_title\"]].values))\n# testing_data['cos_dis_sqd'] = list(map(sentence_similarity, testing_data[[\"search_term\",\"product_description\"]].values))\n\n\n\n# testing_data['j_dis_sqt'] = list(map(get_jaccard_sim, testing_data[[\"search_term\",\"product_title\"]].values))\n# testing_data['j_dis_sqd'] = list(map(get_jaccard_sim, testing_data[[\"search_term\",\"product_description\"]].values))\n\n# testing_data['j_dis_sqt'] = list(map(get_jaccard_sim, testing_data[[\"search_term\",\"product_title\"]].values))\n# testing_data['j_dis_sqd'] = list(map(get_jaccard_sim, testing_data[[\"search_term\",\"product_description\"]].values))\n\n# testing_data['search_query_length'] = testing_data.search_term.str.len()\n# testing_data['number_of_words_in_descr'] = testing_data.product_description.str.count(\"\\\\w+\")\n\ntesting_data['shared_words_mut'] = [get_shared_words_mut(columns)\n                         for columns in \n                         testing_data[['search_term_tokens', 'product_title_tokens', 'product_description_tokens']].values.tolist()\n                        ]\n\ntesting_data['shared_words'] = list(map(get_shared_words_imut, testing_data[['search_term','product_description', 'product_title']].values))\n\n\n\ntesting_data[\"edistance_sprot\"] = [edit_distance(word1, word2) for word1, word2 in\n                                    testing_data[[\"search_term\",\"product_title\"]].values.tolist()]\n\n\ntesting_data[\"edistance_sd\"] = [edit_distance(word1, word2) for word1, word2 in\n                                    testing_data[[\"search_term\",\"product_description\"]].values.tolist()]\n\ntesting_data['j_dis_sqt'] = [get_jaccard_sim(rows) for rows in testing_data[[\"search_term_tokens\",\"product_title_tokens\"]].values]\ntesting_data['j_dis_sqd'] = [get_jaccard_sim(rows) for rows in testing_data[[\"search_term_tokens\",\"product_description_tokens\"]].values]\n\ntesting_data['search_query_length'] = testing_data.search_term.str.len()\ntesting_data['number_of_words_in_descr'] = testing_data.product_description.str.count(\"\\\\w+\")\n\n\ntesting_data['cos_dis_sqt'] = [ sentence_similarity(rows) for rows in testing_data[[\"search_term\",\"product_title\"]].values]\ntesting_data['cos_dis_sqd'] = [sentence_similarity(rows) for rows in testing_data[[\"search_term\",\"product_description\"]].values]\n\n\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ed42e568ad949c896622ec112b85d743f0d0012d","trusted":true},"cell_type":"code","source":"testing_data.corr()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"682a529c53afac3f1fcc29c0dc1f9965c4783818","trusted":true},"cell_type":"code","source":"training_data.describe()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6c99783ec017adde41cb7c464a3b352059565533","trusted":true},"cell_type":"code","source":"testing_data.describe()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"219e83dd3f31a7e6662d64763487d988faf278b7","trusted":true},"cell_type":"code","source":"import seaborn as sns\nplt.figure(figsize=(12, 12))\ntemp = training_data.drop(['product_uid','id'],axis=1)\nsns.heatmap(temp.corr(), annot=True)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6d6aba06d70d4a1f9c89926a7204d0dd69ead412","trusted":true},"cell_type":"code","source":"import seaborn as sns\nplt.figure(figsize=(12, 12))\ntemp = testing_data.drop(['product_uid','id'],axis=1)\nsns.heatmap(temp.corr(), annot=True)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"173357cb05308b6f11d08afb7c6bff8fcf137a2b","trusted":true},"cell_type":"code","source":"%matplotlib inline\nimport matplotlib.pyplot as plt\nfrom scipy.stats import norm  \n\ntraining_data.cos_dis_sqd.plot(kind='hist', normed=True)\n\nmu, std = norm.fit(training_data.cos_dis_sqd)\n\nxmin, xmax = plt.xlim()\nx = np.linspace(xmin, xmax, 100)\np = norm.pdf(x, mu, std)\nplt.plot(x, p, 'k', linewidth=2)\ntitle = \"Fit results: mu = %.2f,  std = %.2f\" % (mu, std)\nplt.title(title)\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f8f5f60c55ccd45fdce53a3f5db6880f88fdab24"},"cell_type":"markdown","source":"let's check wheather this is follows Gaussian distribution or not. Indeed it doesn't follow Gaussian distribution, that follows from Shapiro test"},{"metadata":{"trusted":true,"_uuid":"39dd3d967188d6020517f4b3bcbe3154186ecf2a"},"cell_type":"code","source":"from statsmodels.graphics.gofplots import qqplot\nfrom scipy.stats import shapiro\n\n\nfrom matplotlib import pyplot\nqqplot(training_data.cos_dis_sqd, line='s')\npyplot.show()\n\nstat, p = shapiro(training_data.cos_dis_sqd)\nprint('Statistics=%.3f, p=%.3f' % (stat, p))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6600f0a94ff610787287c77fc21816fe792ba9e4"},"cell_type":"markdown","source":"let's try to find out if wheather it follows normal distribution or not, by doing a few others test"},{"metadata":{"trusted":true,"_uuid":"20b636dd45b2a82ad7759f53e16262efe299556c"},"cell_type":"code","source":"from scipy.stats import normaltest\n\nstat, p = normaltest(training_data.cos_dis_sqd)\nprint('Statistics=%.3f, p=%.3f' % (stat, p))\n# interpret\nalpha = 0.05\nif p > alpha:\n    print('Sample looks Gaussian (fail to reject H0)')\nelse:\n    print('Sample does not look Gaussian (reject H0)')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6e1136dad497dfe845f8fe94c35af63572387d4e"},"cell_type":"code","source":"from scipy.stats import anderson\n\nresult = anderson(training_data.cos_dis_sqd)\nprint('Statistic: %.3f' % result.statistic)\np = 0\nfor i in range(len(result.critical_values)):\n    sl, cv = result.significance_level[i], result.critical_values[i]\n    if result.statistic < result.critical_values[i]:\n        print('%.3f: %.3f, data looks normal (fail to reject H0)' % (sl, cv))\n    else:\n        print('%.3f: %.3f, data does not look normal (reject H0)' % (sl, cv))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"380e4558d674a83e269de4ecd4f2c9cf7868b14a","trusted":true},"cell_type":"code","source":"%matplotlib inline\nimport matplotlib.pyplot as plt\nfrom scipy.stats import norm  \n\ntraining_data.cos_dis_sqt.plot(kind='hist', normed=True)\n\nmu, std = norm.fit(training_data.cos_dis_sqt)\n\nxmin, xmax = plt.xlim()\nx = np.linspace(xmin, xmax, 100)\np = norm.pdf(x, mu, std)\nplt.plot(x, p, 'k', linewidth=2)\ntitle = \"Fit results: mu = %.2f,  std = %.2f\" % (mu, std)\nplt.title(title)\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"27bd5e0746e197f53e4eb130c3281d9e6d6994b0"},"cell_type":"code","source":"from matplotlib import pyplot\nqqplot(training_data.cos_dis_sqt, line='s')\npyplot.show()\n\nstat, p = shapiro(training_data.cos_dis_sqt)\nprint('Statistics=%.3f, p=%.3f' % (stat, p))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d45e77790ede7edbfad1ddaaa2490d70bde0675a"},"cell_type":"markdown","source":"From the below histogram we can conclude that the sum of shared words between search_query product_title, and product description follows the standard distribution.\n"},{"metadata":{"_uuid":"d8b112dffee530550ca51ac9f508275784b88569","trusted":true},"cell_type":"code","source":"%matplotlib inline\nimport matplotlib.pyplot as plt\nfrom scipy.stats import norm  \n\ntraining_data.shared_words.plot(kind='hist', normed=True)\n\nmu, std = norm.fit(training_data.shared_words)\n\nxmin, xmax = plt.xlim()\nx = np.linspace(xmin, xmax, 100)\np = norm.pdf(x, mu, std)\nplt.plot(x, p, 'k', linewidth=2)\ntitle = \"Fit results: mu = %.2f,  std = %.2f\" % (mu, std)\nplt.title(title)\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2fdd2342a0f7a4caee54753c63295ed41780d5b5"},"cell_type":"code","source":"from statsmodels.graphics.gofplots import qqplot\nfrom scipy.stats import shapiro\n\n\nfrom matplotlib import pyplot\nqqplot(training_data.shared_words, line='s')\npyplot.show()\n\nstat, p = shapiro(training_data.shared_words)\nprint('Statistics=%.3f, p=%.3f' % (stat, p))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"56242e46f3fcabc393b5b274a874bca98c80d329","trusted":true},"cell_type":"code","source":"%matplotlib inline\nimport matplotlib.pyplot as plt\nfrom scipy.stats import norm  \n\ntraining_data.edistance_sprot.plot(kind='hist', normed=True)\n\nmu, std = norm.fit(training_data.edistance_sprot)\n\nxmin, xmax = plt.xlim()\nx = np.linspace(xmin, xmax, 100)\np = norm.pdf(x, mu, std)\nplt.plot(x, p, 'k', linewidth=2)\ntitle = \"Fit results: mu = %.2f,  std = %.2f\" % (mu, std)\nplt.title(title)\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1aecd01bdb065413cbe4631050f0c9e1483dbed4","trusted":true},"cell_type":"code","source":"%matplotlib inline\nimport matplotlib.pyplot as plt\nfrom scipy.stats import norm  \n\ntraining_data.search_query_length.plot(kind='hist', normed=True)\n\nmu, std = norm.fit(training_data.search_query_length)\n\nxmin, xmax = plt.xlim()\nx = np.linspace(xmin, xmax, 100)\np = norm.pdf(x, mu, std)\nplt.plot(x, p, 'k', linewidth=2)\ntitle = \"Fit results: mu = %.2f,  std = %.2f\" % (mu, std)\nplt.title(title)\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"411fa86b72928bc9e7ccb0e30e9f59197117f925"},"cell_type":"markdown","source":"let's examing if the same behaviour can be spotted on __testing__ dataset"},{"metadata":{"_uuid":"852bb2f9f363cbee240edf50c156a94fc3b55f62","trusted":true},"cell_type":"code","source":"%matplotlib inline\nimport matplotlib.pyplot as plt\nfrom scipy.stats import norm  \n\ntesting_data.cos_dis_sqd.plot(kind='hist', normed=True)\n\nmu, std = norm.fit(testing_data.cos_dis_sqd)\n\nxmin, xmax = plt.xlim()\nx = np.linspace(xmin, xmax, 100)\np = norm.pdf(x, mu, std)\nplt.plot(x, p, 'k', linewidth=2)\ntitle = \"Fit results: mu = %.2f,  std = %.2f\" % (mu, std)\nplt.title(title)\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fb9283894ce26c917dc8d123d6daf0140db201a3","trusted":true},"cell_type":"code","source":"%matplotlib inline\nimport matplotlib.pyplot as plt\nfrom scipy.stats import norm  \n\ntesting_data.cos_dis_sqt.plot(kind='hist', normed=True)\n\nmu, std = norm.fit(testing_data.cos_dis_sqt)\n\nxmin, xmax = plt.xlim()\nx = np.linspace(xmin, xmax, 100)\np = norm.pdf(x, mu, std)\nplt.plot(x, p, 'k', linewidth=2)\ntitle = \"Fit results: mu = %.2f,  std = %.2f\" % (mu, std)\nplt.title(title)\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"89f1aafac081c6a55d02d7690c35d530b6941bf5","trusted":true},"cell_type":"code","source":"%matplotlib inline\nimport matplotlib.pyplot as plt\nfrom scipy.stats import norm  \n\ntesting_data.shared_words.plot(kind='hist', normed=True)\n\nmu, std = norm.fit(testing_data.shared_words)\n\nxmin, xmax = plt.xlim()\nx = np.linspace(xmin, xmax, 100)\np = norm.pdf(x, mu, std)\nplt.plot(x, p, 'k', linewidth=2)\ntitle = \"Fit results: mu = %.2f,  std = %.2f\" % (mu, std)\nplt.title(title)\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"89fa01e465c0239dd6e974ec480b6a5dd8e53e13","trusted":true},"cell_type":"code","source":"%matplotlib inline\nimport matplotlib.pyplot as plt\nfrom scipy.stats import norm  \n\ntesting_data.edistance_sprot.plot(kind='hist', normed=True)\n\nmu, std = norm.fit(testing_data.edistance_sprot)\n\nxmin, xmax = plt.xlim()\nx = np.linspace(xmin, xmax, 100)\np = norm.pdf(x, mu, std)\nplt.plot(x, p, 'k', linewidth=2)\ntitle = \"Fit results: mu = %.2f,  std = %.2f\" % (mu, std)\nplt.title(title)\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"49093ea64d015233570197a9da1e0ba8aa34d837","trusted":true},"cell_type":"code","source":"%matplotlib inline\nimport matplotlib.pyplot as plt\nfrom scipy.stats import norm  \n\ntesting_data.search_query_length.plot(kind='hist', normed=True)\n\nmu, std = norm.fit(testing_data.search_query_length)\n\nxmin, xmax = plt.xlim()\nx = np.linspace(xmin, xmax, 100)\np = norm.pdf(x, mu, std)\nplt.plot(x, p, 'k', linewidth=2)\ntitle = \"Fit results: mu = %.2f,  std = %.2f\" % (mu, std)\nplt.title(title)\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"347dfa3828630e7f857375f738a4e6c5a16034c7"},"cell_type":"code","source":"sns.pairplot(training_data)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d9ee5b98b695c13f15db89e186ed6f611de9b1c1"},"cell_type":"code","source":"sns.pairplot(testing_dataing_data)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3a6b49bb5111f1c8144752b4dd5e5c16edb6ca30"},"cell_type":"markdown","source":"# 3. Let's start machine learning\nfirst of all let's create training and test data sets\n"},{"metadata":{"_uuid":"fa44b78d93d48499d68d6372e7169fe224f7cba5"},"cell_type":"markdown","source":"We are going to apply the following models:\n1. RandomForestRegressor\n2. LinearRegression\n4. GradientBoostingRegressor \n5. BaggingRegressor\n6. Chain model withing pipeline\n7. XGBoost\n8. CatBoost\n9. Naive Baies\n10. PolynomialFeatures for all previous algorithms\n\n\n### Plan\nWe are going to do the following:\n0. Define pipeline\n1. drop non numeric columns because these information has been already transformed to numberic\n2. Apply the model which has been mentioned above within pipeline mode and outside pipeline\n3. Train models and compare their result on __test__ dataset\n4. write a summary about it\n\n\n\n"},{"metadata":{"_uuid":"8f5c81c64e7c6de8a67fd927db73a646648731cf","trusted":true},"cell_type":"code","source":"df_training = training_data.drop(['product_title','search_term','product_description', 'product_title_tokens', 'product_description_tokens','product_title_tokens','search_term_tokens'],axis=1)\n\ny_train = df_training['relevance'].values\nX_train = df_training.drop(['id','relevance'],axis=1).values","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a19375601be05d47734234ceaeaab8650e7af297","trusted":true},"cell_type":"code","source":"df_training.head(3)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f9e515a47dee0cd6a62721e01ad17dbadf1d0d93","trusted":true},"cell_type":"code","source":"# X_test = testing_data.drop(['id','product_title','search_term','product_description'],axis=1).values\nX_test = testing_data.drop(['id','product_title','search_term','product_description', 'product_title_tokens', 'product_description_tokens','product_title_tokens','search_term_tokens'],axis=1).values\n\nid_test = testing_data['id']\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"bce862e09c7f91b1d739c9445976bc53c40d0bfb"},"cell_type":"markdown","source":"## 3.1 RandomForestRegressor"},{"metadata":{"_uuid":"cd1012ab9c1a9dbdf69ec884cf6baece0880809a","trusted":true},"cell_type":"code","source":"from sklearn.ensemble import RandomForestRegressor\nrfr = RandomForestRegressor(n_estimators = 3, n_jobs = -1, random_state = 17, verbose = 1)\nrfr.fit(X_train, y_train)\n\ny_pred = rfr.predict(X_test)\n\npd.DataFrame({\"id\": id_test, \"relevance\": y_pred}).to_csv('submission.csv',index=False)\n\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f5b7cdf091c5cda2500f2a47cbc5e2ba586848de"},"cell_type":"markdown","source":"## 3.2 LinearRegression\n"},{"metadata":{"_uuid":"a2f4716438151edff4cfea0128a35fb6cb6cad3f","trusted":true},"cell_type":"code","source":"from sklearn.linear_model import LinearRegression\nlr = LinearRegression(n_jobs = -1)\nlr.fit(X_train, y_train)\ny_pred = lr.predict(X_test)\n\n# pd.DataFrame({\"id\": id_test, \"relevance\": y_pred}).to_csv('submission.csv',index=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"04ed36140ee9fc199b827f2299985e24a8c37a71"},"cell_type":"markdown","source":"## 3.3 GradientBoostingRegressor"},{"metadata":{"_uuid":"87ca7a0260efc79ecd01e3df6f5a1fd8ccd46a90","trusted":true},"cell_type":"code","source":"import sklearn\nfrom sklearn.ensemble import GradientBoostingRegressor\n\nparam_grid = {\n                'loss' : ['ls'],\n                'n_estimators' : [3], \n                'max_depth' : [9],\n                'max_features' : ['auto'] \n             }\n\ngbr = GradientBoostingRegressor()\n\nmodel_gbr = sklearn.model_selection.GridSearchCV(estimator = gbr, n_jobs = -1, param_grid = param_grid)\nmodel_gbr.fit(X_train, y_train)\n\ny_pred = model_gbr.predict(X_test)\n\n# pd.DataFrame({\"id\": id_test, \"relevance\": y_pred}).to_csv('submission.csv',index=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"84dea397721702e8c8e7d5ddf363978ba51b2e65"},"cell_type":"markdown","source":"## 3.4 BaggingRegressor based on  RandomForestRegressor"},{"metadata":{"_uuid":"9a5363458373fdac548d79300e7c9e4b835f4201","trusted":true},"cell_type":"code","source":"from sklearn.ensemble import BaggingRegressor\nrf = RandomForestRegressor(max_depth = 20, max_features =  'sqrt', n_estimators = 3)\nclf = BaggingRegressor(rf, n_estimators=3, max_samples=0.1, random_state=25)\n\nclf.fit(X_train, y_train)\n\ny_pred = clf.predict(X_test)\n\n# pd.DataFrame({\"id\": id_test, \"relevance\": y_pred}).to_csv('submission.csv',index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b8c993a5cf22f0d3488c4c5d5f09e88503920602"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"83e9b6fe50f964b9e51c4a49b992f564e7407589"},"cell_type":"markdown","source":"## 3.5 Chain model withing pipeline\n"},{"metadata":{"_uuid":"be917c00bedc186b865e104ed2255c8634df065f","trusted":true},"cell_type":"code","source":"# define models which will be chained togher in a bigger model, which aims to predict the relevancy score\nimport numpy as np\nimport pandas as pd\nfrom sklearn.ensemble import RandomForestRegressor, BaggingRegressor\nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn.pipeline import Pipeline\nfrom sklearn.model_selection import StratifiedKFold\n\n#define standard scaler\nscaler = StandardScaler()\nscaler.fit(X_train, y_train)\nscaled_train_data = scaler.transform(X_train)\nscaled_test_data = scaler.transform(X_test)\n\n\nrf = RandomForestRegressor(n_estimators=4, max_depth=6, random_state=0)\nclf = BaggingRegressor(rf, n_estimators=4, max_samples=0.1, random_state=25)\n\n\npipeline = Pipeline(steps = [('scaling', scaler), ('baggingregressor', clf)])\n#end pipeline \npipeline.fit(X_train, y_train)\ny_pred = pipeline.predict(X_test)\n\n# pd.DataFrame({\"id\": id_test, \"relevance\": y_pred}).to_csv('submission.csv',index=False)\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"729e1d4d673a6da75803ab51ce24ddc1023323e5"},"cell_type":"markdown","source":"## 3.6 Naive Bayes"},{"metadata":{"_uuid":"3d1c8e741f3d5da7537bf97c957b14fe59e89bac","trusted":true},"cell_type":"code","source":"from sklearn.linear_model import BayesianRidge\n\ngnb = BayesianRidge()\nparam_grid = {}\nmodel_nb = sklearn.model_selection.GridSearchCV(estimator = gnb, param_grid = param_grid, n_jobs = -1)\nmodel_nb.fit(X_train, y_train)\n\ny_pred = model_nb.predict(X_test)\n# pd.DataFrame({\"id\": id_test, \"relevance\": y_pred}).to_csv('submission.csv',index=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"95a3a308e84eb4c6d9de0822c53a32d011aa7b76"},"cell_type":"markdown","source":"## 3.7 XGBoost"},{"metadata":{"_uuid":"3dcbca60fc7b7b2a353cefe9dc0d853a7b3bb100","trusted":true},"cell_type":"code","source":"from xgboost import XGBRegressor\n\nxgb = XGBRegressor()\nparam_grid = {'max_depth':[5, 6], \n              'n_estimators': [130, 150, 170], \n              'learning_rate' : [0.1]}\nmodel_xgb = sklearn.model_selection.GridSearchCV(estimator = xgb, param_grid = param_grid, n_jobs = -1)\nmodel_xgb.fit(X_train, y_train)\n\ny_pred = model_xgb.predict(X_test)\n# pd.DataFrame({\"id\": id_test, \"relevance\": y_pred}).to_csv('submission.csv',index=False)\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"55e4d09819db3cb05f1f111cf4a5c4e39069d11d"},"cell_type":"markdown","source":"# Results"},{"metadata":{"_uuid":"370cd2a887776cbe72fbca392c6c4725af417be4"},"cell_type":"markdown","source":"\n\n|Regressor|Train|Kaggle\n|:----------------------|----------|----------|\n|CatBoostRegressor|-|-|\n|XGBRegressor|-|-|\n|GradientBoostingRegressor|-|-|\n|PolynomialFeatures on GradientBoostingRegressor|-|-|\n|PolynomialFeatures on XGBRegressor|-|-|\n|PolynomialFeatures on LinearRegression|-|-|\n|PolynomialFeatures on BaggingRegressor on RandomForestRegressor|-|-|\n|BaggingRegressor on RandomForestRegressor|0.53480|0.53427|\n|Chaining toghether using Pipeline|0.53063|0.53100|\n|BayesianRidge|-|-|\n|LinearRegression|-|-|\n|PolynomialFeatures on BayesianRidge|-|-|\n|RandomForestRegressor|0.59063|0.58869|\n"}],"metadata":{"colab":{"collapsed_sections":[],"name":"EDA and ML.ipynb","provenance":[],"version":"0.3.2"},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}