{"cells":[{"metadata":{"_uuid":"d844b86d521eb8059fc3674078b437dc689f377a"},"cell_type":"markdown","source":"## Using text features to predict image_top_1\n\nThis kernel aims to demonstrate that training an image_top_1 model using the text fields, might come handy for 3 things:\n\n* Train word embeddings that have more focus on visual content\n* predict image_top_1 which are NaN\n* use the difference of our image top 1 prediction and the actual prediction as feature (not implemented)\n"},{"metadata":{"trusted":true,"_uuid":"0f24299762166d4dc4fa7eff9bccacc2d360c88a","collapsed":true,"_kg_hide-input":true},"cell_type":"code","source":"import pandas as pd\nimport time\nimport gc","execution_count":1,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"854ed79532084b5365d620807e38ad14851cef5b"},"cell_type":"markdown","source":"Let's start by loading train and test data and concatenate for easier processing. We still keep the indices to split it up later again. Additionally we get the indices of items that have nan in the image_top_1 field. We don't want thos in our model.\n"},{"metadata":{"trusted":true,"_uuid":"89b39cf794f49dfdb8ee2aa8ec096577cd0213e1","collapsed":true},"cell_type":"code","source":"text_cols = ['param_1','param_2','param_3','title','description']\nprint('loading train...')\ntrain = pd.read_csv('../input/train.csv', index_col = 'item_id', usecols = text_cols + ['item_id','image_top_1'])\ntrain_indices = train.index\nprint('loading test')\ntest = pd.read_csv('../input/test.csv', index_col = 'item_id', usecols = text_cols + ['item_id','image_top_1'])\ntest_indices = test.index\nprint('concat dfs')\ndf = pd.concat([train,test])\nnan_indices = df[pd.isnull(df['image_top_1'])].index\nnot_nan_indices = df[pd.notnull(df['image_top_1'])].index\n\n#df = df[pd.notnull(df['image_top_1'])]\n\ndel train, test","execution_count":3,"outputs":[]},{"metadata":{"_uuid":"fe0fd156e65b5d095f5bda637d01eb8b56d3dabc"},"cell_type":"markdown","source":"Join the text for having one text feature."},{"metadata":{"trusted":true,"_uuid":"ca25a8d6009a3e6bf0d0d96e98342c2281a92ef5","collapsed":true},"cell_type":"code","source":"print('cleaning text')\n\nfor col in text_cols:\n    df[col] = df[col].fillna('nan').astype(str)\nprint('concat text')\ndf['text'] = df[text_cols].apply(lambda x: ' '.join(x), axis=1)\ndf.drop(text_cols,axis = 1, inplace = True)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"29322f58c2abb1ad44108711789471f8ed342e0d"},"cell_type":"markdown","source":"Next we fit the keras build in tokenizer on our text, and pad the sequences. We already split by those items that have image_top_1 (X_train) and those which haven't (X_test). Keep in mind that the tokenizer will have our word2index dictionary, whcih we will need later to build our embeddings dictionary."},{"metadata":{"trusted":true,"_uuid":"c0db8c620112d43f8c7e10ef3819985b110868e6","collapsed":true},"cell_type":"code","source":"from keras.preprocessing import text\nfrom keras.preprocessing.sequence import pad_sequences\n\nmax_features = 100000 # max amount of words considered\nmax_len = 100 #maximum length of text\ndim = 100 #dimension of embedding\n\n\nprint('tokenizing...',end='')\ntic = time.time()\ntokenizer = text.Tokenizer(num_words=max_features)\ntokenizer.fit_on_texts(list(df['text'].values))\ntoc = time.time()\nprint('done. {}'.format(toc-tic))\n\ncol = 'text'\nprint(\"   Transforming {} to seq...\".format(col))\ntic = time.time()\ndf[col] = tokenizer.texts_to_sequences(df[col])\ntoc = time.time()\nprint('done. {}'.format(toc-tic))\n\nprint('padding X_train')\ntic = time.time()\nX_train = pad_sequences(df.loc[not_nan_indices,col], maxlen=max_len)\ntoc = time.time()\nprint('done. {}'.format(toc-tic))\n\nprint('padding X_nan')\ntic = time.time()\nX_nan = pad_sequences(df.loc[nan_indices,col], maxlen=max_len)\ntoc = time.time()\nprint('done. {}'.format(toc-tic))\n\ndf.drop(['text'], axis = 1, inplace=True)","execution_count":5,"outputs":[]},{"metadata":{"_uuid":"330d562b9ebac29d03523470e4b520ce162dc34b"},"cell_type":"markdown","source":"As target value we take the image_top_1 values. I first had them one-hot-encoded with categorical_crossentropy as loss but using sparse_categorical_crossentropy is more memory efficient."},{"metadata":{"trusted":true,"_uuid":"e7303884e216a21b4eed2b35b9d833e9a70a1fa1","collapsed":true},"cell_type":"code","source":"y = df.loc[not_nan_indices,'image_top_1'].values\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"77d88c89a69434fa6135e32d3d1a3b00eb34b4e4"},"cell_type":"markdown","source":"Ok, now we have our input data and label. So let's patch up a model. Feel free to improve :)"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"collapsed":true},"cell_type":"code","source":"import numpy as np\nfrom keras.layers import Input,PReLU,BatchNormalization, GlobalMaxPooling1D, GlobalAveragePooling1D, CuDNNGRU, Bidirectional, Dense, Embedding\nfrom keras.layers import Concatenate, Flatten, Bidirectional\nfrom keras.optimizers import Adam\nfrom keras.initializers import he_uniform\nfrom keras.models import Model\nfrom keras.callbacks import ModelCheckpoint, EarlyStopping\n\n\nfrom keras.losses import categorical_crossentropy, sparse_categorical_crossentropy\n\n\n\ndef all_pool(tensor):\n    avg_tensor = GlobalAveragePooling1D()(tensor)\n    max_tensor = GlobalMaxPooling1D()(tensor)\n    res_tensor = Concatenate()([avg_tensor, max_tensor])\n    return res_tensor\n\ndef build_model():\n    inp = Input(shape=(max_len,))\n\n    embedding = Embedding(max_features + 1, dim)(inp)\n    x = Bidirectional(CuDNNGRU(64,return_sequences=True))(embedding)\n    x = all_pool(x)\n    x = BatchNormalization()(x)\n    x = Dense(256, activation = 'relu')(x)\n    out = Dense(3067, activation='softmax')(x)\n\n    model = Model(inputs=inp, outputs=out)\n\n    model.compile(optimizer=Adam(lr=0.0005), loss=sparse_categorical_crossentropy)\n    return model\n\nmodel = build_model()\nmodel.summary()\n","execution_count":10,"outputs":[]},{"metadata":{"_uuid":"1f22275224b2552a2926298aa7390966848de08a"},"cell_type":"markdown","source":"Let's use some build in functionality of keras to make our life easier. check_point for saving, early stopping to stop at plateau and a validation_split for cv."},{"metadata":{"trusted":true,"_uuid":"63decdebd8fd260ae7b78986bb223b08a4baf50a","collapsed":true},"cell_type":"code","source":"early_stop = EarlyStopping(patience=2)\ncheck_point = ModelCheckpoint('model.hdf5', monitor = \"val_loss\", mode = \"min\", save_best_only = True, verbose = 1)\n\nhistory = model.fit(X_train, y, batch_size = 512, epochs = 10,\n                verbose = 1, validation_split=0.1,callbacks=[early_stop,check_point])\n\n","execution_count":11,"outputs":[]},{"metadata":{"_uuid":"e9e482e3d379d90ec13b2e2c4e429ee9015579f7"},"cell_type":"markdown","source":"Nice, now we have a model predicting image_top_1 value from text. So what can we do with it? First let's extract the trained embeddings. We can use them directly in any NN architecture. I would not sorely use them but they might be worth concatenating with other embeddings. "},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","collapsed":true,"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"id2word = {tokenizer.word_index[word]:word for word in tokenizer.word_index}\nweights = model.layers[1].get_weights()[0]\nembedding_dict = {}\nfor id in id2word:\n    if id <= weights.shape[0]-1:\n        embedding_dict[id2word[id]] = weights[id]\n\n","execution_count":12,"outputs":[]},{"metadata":{"_uuid":"b6b53a990d88bafe66b7d3a04ca0240cbc5199a5"},"cell_type":"markdown","source":"Since now they are already trained, let's save for other people :)"},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"6121bb88f97787035321a4ac1de25d722e6cefca"},"cell_type":"code","source":"import pickle\nwith open('embedding_dict.p','wb') as f:\n    pickle.dump(embedding_dict,f)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"340228b5d3b2a42671635d94dfbe3d0b4c2c06a6"},"cell_type":"markdown","source":"So lets get the predictions for our missing image classes and put them into our train-test dataframe. Here you can play around with a confidence-threshold (I set 0.1) . You don't want to messup your image_top_1 data with classes your not confident with, and since we just used the text there might be some items where we don't have any info at all."},{"metadata":{"trusted":true,"_uuid":"cd90226f2e93432d0de4be437c4c35de2b22931e","collapsed":true},"cell_type":"code","source":"preds = model.predict(X_nan,verbose=1)","execution_count":13,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f92a5d24f4906e05b3c12355abdc97a73c040ea4","collapsed":true},"cell_type":"code","source":"k = 0\nclasses = np.zeros(shape=np.argmax(preds,axis = 1).shape)\nfor i in range(preds.shape[0]):\n    if np.max(preds[i]) > 0.1:\n        k+=1\n        classes[i] = np.argmax(preds[i])\n    else:\n        classes[i] = np.nan\ndf.loc[nan_indices,'image_top_1'] = classes\n","execution_count":18,"outputs":[]},{"metadata":{"_uuid":"f0658b5de5c2c0556541d023e5bd3dc57c527b30"},"cell_type":"markdown","source":"Let's also save our \"corrected\" image_top_1 column for others. Cheers and happy kaggling"},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"b59be2f86330eecc80c8b032b9cf38bba39a93b6"},"cell_type":"code","source":"df.loc[train_indices].to_csv('train_image_top_1_features.csv')\ndf.loc[test_indices].to_csv('test_image_top_1_features.csv')\n","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}