{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"markdown","source":"# Data leak? similar/same validataoin samples on test data\nThere migiht be leaked samples from validation data, ~600 samples on test data.  \nI could confirm some minor changed validation samples on test data or the exact same samples on test data like below.  \nPlease check this notebook.\n\nAnd also I tried whether LB score is improved with this leak.  \nBut I got worse result if I use this leak.(0.9462 -> 0.9459: base submission file from https://www.kaggle.com/hamditarek/ensemble ).  \nOne possible reason for this is my rough treatment after 128 tokens. \n\n```\n--------------------------------------------------------------------------------\n>> TEST \n ID:8667, LANG:tr, Duplicated Num:2\nCOMMET_TEXT:Vikipedi de deneme yaptığınız için teşekkürler. Denemeniz çalıştı, ancak şu anda ya    geri alındı    ya da    silindi   . Başka bir deneme yapmak istiyorsanız lütfen    deneme tahtasını    kullanın. Ansiklopedimize nasıl katkıda bulunabileceğiniz hakkında daha fazla bilgi edinmek istiyorsanız    hoşgeldin    sayfasına bir göz atın. Teşekkürler.\n\n>> VALIDATION \n ID:2162, LANG:tr, Duplicated Num:1, TOXIC:0\nCOMMET_TEXT:Vikipedi de deneme yaptığınız için teşekkürler. Denemeniz çalıştı, ancak şu anda ya geri alındı ya da silindi. Başka bir deneme yapmak istiyorsanız lütfen deneme tahtasını kullanın. Ansiklopedimize nasıl katkıda bulunabileceğiniz hakkında daha fazla bilgi edinmek istiyorsanız hoşgeldin sayfasına bir göz atın. Teşekkürler.\n\n>> diff result\n*** TEST\n--- VAL\n***************\n*** 14,21 ****\n  alındı  ya  da! silindi! .  Başka  bir  deneme--- 14,20 ----\n  alındı  ya  da! silindi.  Başka  bir  deneme\n\n>> TEST ENGLISH Translation\nThank you for doing experiments in Wikipedia. Your experiment worked, but at the moment either rolled back or deleted. If you want to make another attempt, please use the sandbox. If you want to learn more about how you might be contributing to our encyclopedia Take a look at the welcome page. Thank you.\n\n```","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import os\nimport random\nimport difflib\nfrom binascii import crc32\nimport sys \nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\n\nDATA_PATH = \"/kaggle/input/jigsaw-multilingual-toxic-comment-classification\"\nval_df = pd.read_csv(os.path.join(DATA_PATH, \"validation.csv\"))\ntest_df = pd.read_csv(os.path.join(DATA_PATH, \"test.csv\"))\ntest_en_df = pd.read_csv(\"../input/test-en-df/test_en.csv\")\n\nval_ids_df = pd.read_csv(os.path.join(DATA_PATH, \"validation-processed-seqlen128.csv\"))\ntest_ids_df = pd.read_csv(os.path.join(DATA_PATH, \"test-processed-seqlen128.csv\"))\ntest_ids_df = pd.merge(test_ids_df, test_df[[\"id\", \"lang\"]], on=\"id\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There are no ovelappings with \"comment_text\" between test and validation.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"target_column = \"comment_text\"\ntest_ids_df.loc[test_ids_df[target_column].isin(val_ids_df[target_column]), ]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### There are 641 test \"input_word_ids\" samples which are confirmed at validation data. And there are also deplicated samples.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"target_column = \"input_word_ids\"\noverlapped_test_df = test_ids_df.loc[test_ids_df[target_column].isin(val_ids_df[target_column]), ]\noverlapped_val_df = val_ids_df.loc[val_ids_df[target_column].isin(test_ids_df[target_column]), ]\noverlapped_test_df.nunique()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's check these samples.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"test_ids_df[\"token_hash\"] = test_ids_df[\"input_word_ids\"].apply(lambda x: crc32(x.encode()) & 0xffffffff)\nval_ids_df[\"token_hash\"] = val_ids_df[\"input_word_ids\"].apply(lambda x: crc32(x.encode()) & 0xffffffff)\ntarget_column = \"token_hash\"\noverlapped_hash = test_ids_df.loc[test_ids_df[target_column].isin(val_ids_df[target_column]), target_column]\noverlapped_hash = np.unique(overlapped_hash.values)\noverlapped_hash.sort()\noverlapped_hash.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def display_test_val_comment(test_ids_df, val_ids_df, target_column=\"token_hash\", target_value=0, test_en_df=None, ind_=0):\n    test_sample = test_ids_df.query(f\"{target_column} == {target_value}\")\n    val_sample = val_ids_df.query(f\"{target_column} == {target_value}\")\n    \n    test_comment = test_sample.comment_text.values[ind_]\n    val_comment = val_sample.comment_text.values[0]\n    print(\"{}\".format(\"-\"*80))\n\n    print(f\"{target_column}:{target_hash}\")\n    print(f\">> TEST \\n ID:{test_sample.id.values[ind_]}, LANG:{test_sample.lang.values[ind_]}, Duplicated Num:{len(test_sample)}\")\n    print(f\"COMMET_TEXT:{test_comment}\\n\")\n\n    print(f\">> VALIDATION \\n ID:{val_sample.id.values[0]}, LANG:{val_sample.lang.values[0]}, Duplicated Num:{len(val_sample)}, TOXIC:{val_sample.toxic.values[0]}\")\n    print(f\"COMMET_TEXT:{val_comment}\\n\")\n\n    # diff = difflib.unified_diff(test_comment.replace(\" \", \"\\n \").split(), val_comment.replace(\" \", \"\\n \").split(), \"TEST\", \"VAL\", lineterm='\\n')\n    diff = difflib.context_diff(test_comment.replace(\" \", \"\\n \").split(), val_comment.replace(\" \", \"\\n \").split(), \"TEST\", \"VAL\", lineterm='\\n')\n    print(\">> diff result\")\n    sys.stdout.writelines(diff)\n\n    if test_en_df is not None:\n        test_en_comment = test_en_df.query(f\"id == {test_sample.id.values[ind_]}\").content_en.values[0]\n        print(\"\\n\\n>> TEST ENGLISH Translation\")\n        print(f\"{test_en_comment}\\n\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Display all overlapped samples.","execution_count":null},{"metadata":{"trusted":true,"collapsed":true},"cell_type":"code","source":"for i, target_hash in enumerate(overlapped_hash):\n    display_test_val_comment(test_ids_df, val_ids_df, target_column=\"token_hash\", target_value=target_hash, test_en_df=test_en_df)\n    # comment out lines below if you want to check all samples\n    if i >= 2:\n        break","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Make a submission\nLet's check this leak with LB score.  But current apporach doen't work well, you will get worse result... 0.9462 -> 0.9459","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# submission file from https://www.kaggle.com/hamditarek/ensemble by Tarek Hamdi, its LB score is 0.9462.\nsub_df = pd.read_csv(\"../input/ensemble/submission.csv\")\nsub_df = pd.merge(sub_df, test_ids_df, on=\"id\")\nsub_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Split the overlapped samples with validation toxicity, \"toxic\" and \"non_toxic\".","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"non_toxic_hash = val_ids_df.loc[val_ids_df.token_hash.isin(overlapped_hash), :].query(\"toxic == 0\").token_hash\ntoxic_hash = val_ids_df.loc[val_ids_df.token_hash.isin(overlapped_hash), :].query(\"toxic == 1\").token_hash\nnon_toxic_hash = np.unique(non_toxic_hash)\ntoxic_hash = np.unique(toxic_hash)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub_df.loc[sub_df.token_hash.isin(non_toxic_hash), \"toxic\"].hist()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub_df.loc[sub_df.token_hash.isin(non_toxic_hash), :].sort_values(\"toxic\").tail()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub_df.query(\"token_hash == 3880127965\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"At \"token_hash == 3880127965\" these samples share the header, \"........\", and not leak.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"non_toxic_hash = non_toxic_hash[non_toxic_hash != 3880127965]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub_df.loc[sub_df.token_hash.isin(non_toxic_hash), \"toxic\"].hist()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub_df.loc[sub_df.token_hash.isin(toxic_hash), \"toxic\"].hist()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub_df.loc[sub_df.token_hash.isin(non_toxic_hash), \"toxic\"] = 0.0\nsub_df.loc[sub_df.token_hash.isin(toxic_hash), \"toxic\"]= 1.0","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub_df[[\"id\", \"toxic\"]].to_csv(\"submission.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":" # Appendix: Similar samples for each validatoin and test data.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"test_ids_df.duplicated(\"input_word_ids\", keep=False).sum()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"val_ids_df.duplicated(\"input_word_ids\", keep=False).sum()","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}