{"cells":[{"metadata":{"_uuid":"ce7a4fa64ec3575b9e22af3536ba0c28c312681c","_cell_guid":"7cf95c58-521c-4cdf-af36-cd079b15056a"},"cell_type":"markdown","source":"#   Word Embedding Idea."},{"metadata":{"_uuid":"d5c846c2dcabb06b11be6abd5bfa77501d83ec0d","_cell_guid":"674422e9-d0fa-4108-b798-3e04d5501142"},"cell_type":"markdown","source":"The reason i think Word2Vec might working here is dataset is sequential. If you sorted by ip and click time, you will make your datset as a sequence of clicks by different ip. If you take all clicks from one ip, that shows the whole process of that 'ip' 's behavioral pattern. It is sure that for different ips their behavior should be different. But how about commons? How could we extract the relationships of app, device, os and channel? What is the relationship between app 9 and app 10? That is something we need to take into consideration. By applying Word2Vec here, I diminished the specific pattern from different ips and get the 'word embeddings' of each categories in app, device, os and channel based on their own characteristics. You might experienced an ad for game app that pop up during you play a mobile game and after you click that ad another ad for another game might pop up. This is the 'charateristic' that I want to figure out and include in my training dataset."},{"metadata":{"_uuid":"5de51d3c881ac965d1311bcf184a3d1754fafbee","_cell_guid":"2ceb234b-3f7d-4513-94ab-4333138d6bb8"},"cell_type":"markdown","source":"Personally I dont think there will be big difference between applying XGBoost on original dataset and on embedded dataset. The reason is obvious that decision tree works in a way that dimension expand doesn't really make effort on increasing the accuracy. But I still apply it here becuase i want to see how are those word embeddings performs and may be find out how long should my word embedding is. I use 3 as length for each predictors here, as you see from feature importance, it is a little bit suprising that for word embedding 'app', x2 doesn't play a same role as x1 and x3. So may be when we increase the dimension of word embeddings and set col_sample by tree a small value, we may get some unexpected good result."},{"metadata":{"_uuid":"2ef44574fc436e3248b3bfd6e550d11be642b247","_cell_guid":"720ddc83-76fa-4c03-9d65-dad2156f4467"},"cell_type":"markdown","source":" Now i increase the dimension of word embeddings to a size of 300. The result AUC score goes up dramatially. (This kernel is an extention of my original kernel. https://www.kaggle.com/jingqliu/xgboost-nn-on-small-sample-with-word2vec)"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import pandas as pd\nimport tensorflow as tf\nimport numpy as np\nfrom sklearn.model_selection import train_test_split\nimport gc\nimport xgboost as xgb\nfrom xgboost.sklearn import XGBClassifier\nfrom sklearn import cross_validation, metrics\nfrom sklearn.grid_search import GridSearchCV\nimport matplotlib as plt\nfrom gensim.models import Word2Vec","execution_count":1,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true,"collapsed":true},"cell_type":"code","source":"train = pd.read_csv('../input/readyforuse/ready.csv',nrows = 3698078*4)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"8c50fc01-b5b0-4fbd-94a9-170d4ef6070e","collapsed":true,"_uuid":"cd95529964194280b526a471742d1bb5c079c573","trusted":false},"cell_type":"code","source":"test = pd.read_csv('../input/readyforuse/testready.csv')","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"b40d1988-d364-47e0-8ae6-eee0c56f7ffc","_uuid":"9f7dd9035c9ca8617814b998881328cd6e86e096","collapsed":true,"trusted":false},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"0d530ccd-e352-4b0c-a42f-2e47bffdcba4","_uuid":"602b81bd36cbcff389dff11c58aef0a9e5401df9","collapsed":true,"trusted":false},"cell_type":"code","source":"train = train.iloc[1479231:,:]","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"36f73e13-5f32-4aba-bc89-5f12427a4985","_uuid":"3bbedaf1544c99c9d44219bb85b3d5a6702f0c14","collapsed":true,"trusted":false},"cell_type":"code","source":"model_app = Word2Vec.load('../input/large-word-embeddings/vec_app.txt')\nmodel_channel = Word2Vec.load('../input/large-word-embeddings/vec_channel.txt')\nmodel_device = Word2Vec.load('../input/large-word-embeddings/vec_device.txt')\nmodel_os = Word2Vec.load('../input/large-word-embeddings/vec_os.txt')","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"4c23dd0a-dd86-4f27-9bb1-9585e3a6f18a","_uuid":"322def294c8f9dedea315f86960cb48a939c41a0","collapsed":true,"trusted":false},"cell_type":"code","source":"train['app'] = train['app'].astype(str)\ntrain['device'] = train['device'].astype(str)\ntrain['os'] = train['os'].astype(str)\ntrain['channel'] = train['channel'].astype(str)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"cd5e9bef-9745-4cf9-94f1-e819fe72dd47","collapsed":true,"_uuid":"ee9a2e1244d88020f1238a4ac077433e1e54df1d","trusted":false},"cell_type":"code","source":"test['app'] = test['app'].astype(str)\ntest['device'] = test['device'].astype(str)\ntest['os'] = test['os'].astype(str)\ntest['channel'] = test['channel'].astype(str)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0b57995ff18851f7ff6ff2fb54900093b23d8257","_cell_guid":"cc41fc2f-b5ab-4e03-860e-c802eb1766e6"},"cell_type":"markdown","source":"# Neural Network on Embedded dataset."},{"metadata":{"_cell_guid":"cb2c26c7-3df6-4f32-9f04-97cad1c4e154","_uuid":"26e86d7ffc2fa2d39c695ceeece28b5eb5d2e345","collapsed":true,"trusted":false},"cell_type":"code","source":"input_x = tf.placeholder(tf.float32, [None, 301])\ninput_y = tf.placeholder(tf.float32, [None, 2])","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"0132d2f1-f4e6-486a-87e0-b6fcdffc7cc2","_uuid":"5daa4021654c445f0a9af11af86b2a08f37b3dd7","collapsed":true,"trusted":false},"cell_type":"code","source":"w1 = tf.Variable(tf.random_normal([301, 301], stddev = 0.05), name = 'w1')\nb1 = tf.Variable(tf.random_normal([301], stddev = 0.05), name = 'b1')\nw2 = tf.Variable(tf.random_normal([301, 150], stddev = 0.05), name = 'w2')\nb2 = tf.Variable(tf.random_normal([150], stddev = 0.05), name = 'b2')\nw3 = tf.Variable(tf.random_normal([150, 150], stddev = 0.05), name = 'w3')\nb3 = tf.Variable(tf.random_normal([150], stddev = 0.05), name = 'b3')\nw4 = tf.Variable(tf.random_normal([150, 150], stddev = 0.05), name = 'w4')\nb4 = tf.Variable(tf.random_normal([150], stddev = 0.05), name = 'b4')\nw5 = tf.Variable(tf.random_normal([150, 2], stddev = 0.05), name = 'w5')\nb5 = tf.Variable(tf.random_normal([2], stddev = 0.05), name = 'b5')","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"739875bf-8ac6-43f3-bc92-b0eb57dc8829","_uuid":"ca75ac0f5f9f56cc73df3dfebf80c4feaafc0b4a","collapsed":true,"trusted":false},"cell_type":"code","source":"layer1 = tf.nn.xw_plus_b(input_x, w1, b1, name = 'layer1')\nlayer1 = tf.nn.relu(layer1)\nlayer1 = tf.nn.dropout(layer1,0.8)\nlayer2 = tf.nn.xw_plus_b(layer1, w2, b2, name = 'layer2')\nlayer2 = tf.nn.relu(layer2)\nlayer3 = tf.nn.xw_plus_b(layer2, w3, b3, name = 'layer3')\nlayer3 = tf.nn.relu(layer3)\nlayer4 = tf.nn.xw_plus_b(layer3, w4, b4, name = 'layer4')\nlayer4 = tf.nn.relu(layer4)\nlayer5 = tf.nn.xw_plus_b(layer4, w5, b5, name = 'layer5')\nprediction = tf.nn.softmax(layer5)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"9b0d30b3-91ac-481a-9d00-ac2dd06756e3","_uuid":"d6adfeaa0229b02513429b605b99ecaa1ab7dd49","collapsed":true,"trusted":false},"cell_type":"code","source":"loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits_v2(logits = layer5, labels = input_y))\noptimizer = tf.train.AdamOptimizer(learning_rate=0.0016).minimize(loss)\naccuracy = tf.reduce_mean(tf.cast(tf.equal(tf.argmax(input_y, 1), tf.argmax(prediction, 1)), tf.float32))","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"1c6ac2a5-96f5-4ce7-a25c-28fc42107ad5","_uuid":"f5b5027067c6bf14c131fc52b767d700583f0d1e","collapsed":true,"trusted":false},"cell_type":"code","source":"def generate_batch(data, batch_size, num_epochs, shuffle=True):\n    data = np.array(data)\n    data_size = len(data)\n    num_batches_per_epoch = int((len(data)-1)/batch_size) + 1\n    l = 0\n    for epoch in range(num_epochs):\n        l += 1\n        if shuffle:\n            shuffle_indices = np.random.permutation(np.arange(data_size))\n            shuffled_data = data[shuffle_indices]\n        else:\n            shuffled_data = data\n        for batch_num in range(num_batches_per_epoch):\n            start_index = batch_num * batch_size\n            end_index = min((batch_num + 1) * batch_size, data_size)\n            yield shuffled_data[start_index:end_index]","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"8f57effe-469c-4fb0-aa85-e76cfb2f082f","_uuid":"539428a1b031e6f4b6975fab4601f51375bc005e","collapsed":true,"trusted":false},"cell_type":"code","source":"train1 = train.iloc[0:3698078,:]\ntrain2 = train.iloc[3698078:3698078*2,:]\ntrain3 = train.iloc[3698078*2:3698078*3,:]\ntrain4 = train.iloc[3698078*3:,:]","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"926d4f58-0534-40dc-a360-b356daaee2ef","_uuid":"f1bcbe3c726d144b3eab9beeeacbb3e033f49de5","collapsed":true,"trusted":false},"cell_type":"code","source":"batches1 = generate_batch(train1,10000,1)\nbatches2 = generate_batch(train2,10000,1)\nbatches3 = generate_batch(train3,10000,1)\nbatches4 = generate_batch(train4,10000,1)\nbatches5 = generate_batch(train1,10000,1)\nbatches6 = generate_batch(train2,10000,1)\nbatches7 = generate_batch(train3,10000,1)\nbatches8 = generate_batch(train4,10000,1)\nbatches9 = generate_batch(train1,10000,1)\nbatches10 = generate_batch(train2,10000,1)\nbatches11 = generate_batch(train3,10000,1)\nbatches12 = generate_batch(train4,10000,1)\nbatch_bag1 = [batches1,batches2,batches3,batches4]\nbatch_bag2 = [batches5,batches6,batches7,batches8]\nbatch_bag3 = [batches9,batches10,batches11,batches12]\nbatch_bags = [batch_bag1, batch_bag2, batch_bag3]","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"00cbc736-96fa-44e2-9964-db7828007208","_uuid":"00f2af928783b8945fa5ab22983eea9c9c7431dd","collapsed":true,"trusted":false},"cell_type":"code","source":"(int((len(train1)-1)/10000) + 1) * 4","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"f680d19c-d772-40bc-88e4-491191e6e6c9","_uuid":"8661d9a49963ec09140029d50facd2ae73f6e9c9","collapsed":true,"trusted":false},"cell_type":"code","source":"test_blocks = generate_batch(test, 20000, 1, shuffle = False)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"a5d8a44c-5f81-4852-815d-942690423e8d","_uuid":"9eee779653ddd49dae17ebda8ef85ea78d736361","collapsed":true,"trusted":false},"cell_type":"code","source":"init_op = tf.global_variables_initializer()\nwith tf.Session() as sess:\n    \n    sess.run(init_op)\n    \n    print('Start!')\n    i = 0\n    for batch_bag in batch_bags:\n        i += 1\n        print('Epoch ' + str(i) + ' Start!')\n        avg_loss = 0\n        avg_acc = 0\n        avg_auc = 0\n        for batches in batch_bag:\n            for batch in batches:\n                batch = pd.DataFrame(batch, columns = ['app','device','os','channel','is_attributed','count'])\n                x_batch = batch.loc[:, batch.columns != 'is_attributed']\n                y_batch = batch.loc[:, batch.columns == 'is_attributed']\n                x_batch = pd.concat([x_batch, pd.DataFrame(model_app.wv[x_batch['app']]), pd.DataFrame(model_channel.wv[x_batch['channel']]), pd.DataFrame(model_os.wv[x_batch['os']]), pd.DataFrame(model_device.wv[x_batch['device']])],axis = 1)\n                x_batch = x_batch.drop(columns = ['app','os','device','channel'])\n                y_batch['is_not_attributed'] = 1 - y_batch['is_attributed']\n                _,c, acc, pred = sess.run([optimizer, loss, accuracy, prediction],feed_dict = {input_x: x_batch, input_y: y_batch})\n                avg_loss += c\n                avg_acc += acc\n                avg_auc += metrics.roc_auc_score(y_batch['is_attributed'].astype(int), pred[:,0])\n        print('Average loss is: ' + str(avg_loss/1480) + ', Average accuracy is: ' + str(avg_acc/1480) + ', Average AUC is: ' + str(avg_auc/1480))\n    \n    print('Prediction Start!')\n    \n    df = pd.DataFrame()\n    for block in test_blocks:\n        block = pd.DataFrame(block, columns = ['app', 'device', 'os', 'channel', 'count'])\n        block = pd.concat([block, pd.DataFrame(model_app.wv[block['app']]), pd.DataFrame(model_channel.wv[block['channel']]), pd.DataFrame(model_os.wv[block['os']]), pd.DataFrame(model_device.wv[block['device']])],axis = 1)\n        block = block.drop(columns = ['app','device','os','channel'])\n        pred = sess.run(prediction, feed_dict = {input_x: block})\n        df = df.append(pd.DataFrame(pred))\n    \n    print('Finish!')","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"8e4fdabf-79ce-4a31-9a8b-7df10cd1468b","_uuid":"59b4ac450ec2d592739722109cf8a55ff258a5ff","collapsed":true,"trusted":false},"cell_type":"code","source":"df.round().mean()","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"e570a34e-967b-49dd-877f-656081ab6f97","collapsed":true,"_uuid":"d5e0cfe6a93f54f9c6ab63d81b9ec0e586c3c0f7","trusted":false},"cell_type":"code","source":"submission = pd.read_csv(\"../input/talkingdata-adtracking-fraud-detection/sample_submission.csv\")\ndf.columns = ['is_attributed','b']\nsubmission['is_attributed'] = np.array(df['is_attributed'])\nsubmission.to_csv(\"submission.csv\", index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.5","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}