{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# About the competition","metadata":{}},{"cell_type":"markdown","source":"## Wikipedia - Image/Caption Matching\n### Retrieve captions based on images\n\n\n### Description\nA picture is worth a thousand words, yet sometimes a few will do. We all rely on online images for knowledge sharing, learning, and understanding. Even the largest websites are missing visual content and metadata to pair with their images. Captions and “alt text” increase accessibility and enable better search. The majority of images on Wikipedia articles, for example, don't have any written context connected to the image. Open models could help anyone improve accessibility and learning for all.\n\nCurrent solutions rely on simple methods based on translations or page interlinks, which have limited coverage. Even the most advanced computer vision image captioning isn't suitable for images with complex semantics.\n\n### Data\nThe objective of this competition is to predict the target caption_title_and_reference_description given information about an images. The targets for this competition are in multiple languages.\n\n### Files\n- train-{0000x}-of-00005.tsv - the training data (tab delimited)\n- test.tsv - the test data; the objective is to predict the target caption_title_and_reference_description for each row id\n- sample_submission.csv - a sample submission file in the correct format; note that multiple predictions (up to 5) are allowed for each id in the test data.\n- image_data_test/\n - image_pixels/test_image_pixels_part-{0000x}.csv.gz\n - image_url: url to the original image file, e.g. https://upload.wikimedia.org/wikipedia/commons/e/ec/Hovden.jpg\n - b64_bytes: base64 encoded bytes of the image file at a 300px resolution\n - metadata_url: url to the commons page of the image, e.g. https://commons.wikimedia.org/wiki/File:Hovden.jpg\n - resnet_embeddings/test_resnet_embeddings_part-{0000x}.csv.gz\n - image_url: url to the original image file, e.g. https://upload.wikimedia.org/wikipedia/commons/e/ec/Hovden.jpg\n - embedding: a comma separated list of 2048 float values\n- image_data_train - Due to the size of the training image data (~275 Gb), it is hosted separately and can be found here. Note that not all of the training observations have corresponding image data.\n \n*kaggle competitions download -c wikipedia-image-caption*\n\n### Submission\nSubmissions will be evaluated using NDCG@5 (Normalized Discounted Cumulative Gain).\n\nThe submission should be a list of id,caption_title_and_reference_description pairs ranked from top to bottom according to their relevance (i.e., the top id is the most relevant caption_title_and_reference_description), with up to 5 predictions per id. Each line should be a single id,caption_title_and_reference_description pair.\n\n## Prizes\nThe top three winning teams will receive Wikipedia-branded merchandise","metadata":{}},{"cell_type":"markdown","source":"# Importing the libs and data ","metadata":{}},{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-09-16T06:03:32.956727Z","iopub.execute_input":"2021-09-16T06:03:32.957124Z","iopub.status.idle":"2021-09-16T06:03:32.972743Z","shell.execute_reply.started":"2021-09-16T06:03:32.957079Z","shell.execute_reply":"2021-09-16T06:03:32.971950Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import os\nimport requests\n\nimport os\nimport gc\nimport glob\n\nfrom tqdm.notebook import tqdm\ntqdm.pandas()\n\nimport plotly.express as px\nfrom plotly.offline import init_notebook_mode, iplot\ninit_notebook_mode(connected=True) \nimport plotly.graph_objs as go\n\nfrom plotnine import *\nimport pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport plotly\nimport plotly.express as px\nimport plotly.graph_objs as go\nfrom geopy.geocoders import Nominatim\nimport folium\nfrom folium.plugins import HeatMap\nfrom folium.plugins import FastMarkerCluster\nfrom plotly import tools\nimport re\nfrom plotly.offline import init_notebook_mode, plot, iplot\nfrom wordcloud import WordCloud, STOPWORDS \nfrom warnings import filterwarnings\nfilterwarnings('ignore')\nimport missingno as msno\nimport glob\n\nimport pandas as pd\nimport numpy as np\nimport matplotlib\nimport matplotlib.pyplot as plt\nimport plotly.graph_objs as go\nimport plotly.express as px\nimport PIL.Image\nimport cv2\nimport urllib\nfrom IPython.display import Image, display\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport urllib\n\nimport warnings\nwarnings.simplefilter(action='ignore', category=FutureWarning)","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-09-16T06:03:32.973724Z","iopub.execute_input":"2021-09-16T06:03:32.973956Z","iopub.status.idle":"2021-09-16T06:03:32.989450Z","shell.execute_reply.started":"2021-09-16T06:03:32.973928Z","shell.execute_reply":"2021-09-16T06:03:32.988504Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"os.listdir('../input/wikipedia-image-caption/')","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-09-16T06:03:32.990735Z","iopub.execute_input":"2021-09-16T06:03:32.991143Z","iopub.status.idle":"2021-09-16T06:03:33.003879Z","shell.execute_reply.started":"2021-09-16T06:03:32.991085Z","shell.execute_reply":"2021-09-16T06:03:33.002808Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Loading the test data**","metadata":{}},{"cell_type":"code","source":"test_file = pd.read_csv('../input/wikipedia-image-caption/test.tsv', sep='\\t')\ntest_file.head(1)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-09-16T06:03:53.987521Z","iopub.execute_input":"2021-09-16T06:03:53.987829Z","iopub.status.idle":"2021-09-16T06:03:54.244473Z","shell.execute_reply.started":"2021-09-16T06:03:53.987796Z","shell.execute_reply":"2021-09-16T06:03:54.243519Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tesdt_file = pd.read_csv('../input/wikipedia-image-caption/test_caption_list.csv')\ntesdt_file.head()","metadata":{"execution":{"iopub.status.busy":"2021-09-16T06:04:44.572420Z","iopub.execute_input":"2021-09-16T06:04:44.572720Z","iopub.status.idle":"2021-09-16T06:04:44.933803Z","shell.execute_reply.started":"2021-09-16T06:04:44.572690Z","shell.execute_reply":"2021-09-16T06:04:44.933032Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Loading the wiki data**","metadata":{}},{"cell_type":"code","source":"wiki_df = pd.read_csv('../input/wikipedia-image-caption/image_data_test/image_pixels/test_image_pixels_part-00000.csv', \n                      sep='\\t', names=['image_url', 'b64_bytes', 'metadata_url'])\ndf = pd.read_csv('../input/wikipedia-image-caption/image_data_test/image_pixels/test_image_pixels_part-00000.csv', sep='\\t', names=['image_url', 'b64_bytes', 'metadata_url'])\ndf\nwiki_df.head(1)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-09-16T06:04:49.035222Z","iopub.execute_input":"2021-09-16T06:04:49.035491Z","iopub.status.idle":"2021-09-16T06:05:01.333440Z","shell.execute_reply.started":"2021-09-16T06:04:49.035463Z","shell.execute_reply":"2021-09-16T06:05:01.332603Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Loading the Submission data**","metadata":{}},{"cell_type":"code","source":"sub_file = pd.read_csv('../input/wikipedia-image-caption/sample_submission.csv')\nsub_file.head(1)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-09-16T06:05:01.335038Z","iopub.execute_input":"2021-09-16T06:05:01.335285Z","iopub.status.idle":"2021-09-16T06:05:01.922095Z","shell.execute_reply.started":"2021-09-16T06:05:01.335258Z","shell.execute_reply":"2021-09-16T06:05:01.921256Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Shape of the data we have!**","metadata":{}},{"cell_type":"code","source":"print(\"the shape of the wiki file is : \" , wiki_df.shape)\nprint(\"the shape of the test file is : \", test_file.shape)\nprint(\"the shape of the sub file is  : \" ,sub_file.shape)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-09-16T06:05:01.923169Z","iopub.execute_input":"2021-09-16T06:05:01.923412Z","iopub.status.idle":"2021-09-16T06:05:01.929679Z","shell.execute_reply.started":"2021-09-16T06:05:01.923385Z","shell.execute_reply":"2021-09-16T06:05:01.928659Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# EDA","metadata":{}},{"cell_type":"markdown","source":"## Loading the images and exploring the images ","metadata":{}},{"cell_type":"code","source":"def get_links(df, num):\n    return df.image_url[:num].values\n\nlinks = get_links(df, 10)\n\n\ndef load_images(links):\n    images = []\n    \n    for link in links:\n        URL = link\n        try:\n\n            with urllib.request.urlopen(URL) as url:\n                with open('./temp.jpg', 'wb') as f:\n                    f.write(url.read())\n\n            img = PIL.Image.open('./temp.jpg')\n            img = np.asarray(img)\n            images.append(img)\n        except:\n            continue\n    return images\n\ndef display_images(images, title=None): \n    f, ax = plt.subplots(2,5, figsize=(18,12))\n    if title:\n        f.suptitle(title, fontsize = 30)\n\n    for i, image_id in enumerate(images):\n        ax[i//5, i%5].imshow(image_id) \n   \n        ax[i//5, i%5].axis('off')\n\n    plt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-09-16T06:05:01.931518Z","iopub.execute_input":"2021-09-16T06:05:01.931731Z","iopub.status.idle":"2021-09-16T06:05:01.949939Z","shell.execute_reply.started":"2021-09-16T06:05:01.931706Z","shell.execute_reply":"2021-09-16T06:05:01.948976Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"images = load_images(links)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-09-16T06:05:01.951077Z","iopub.execute_input":"2021-09-16T06:05:01.951321Z","iopub.status.idle":"2021-09-16T06:05:11.978190Z","shell.execute_reply.started":"2021-09-16T06:05:01.951293Z","shell.execute_reply":"2021-09-16T06:05:11.977277Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display_images(images)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-09-16T06:05:11.979634Z","iopub.execute_input":"2021-09-16T06:05:11.979851Z","iopub.status.idle":"2021-09-16T06:05:16.336983Z","shell.execute_reply.started":"2021-09-16T06:05:11.979826Z","shell.execute_reply":"2021-09-16T06:05:16.336241Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"links = df.image_url[20:32].values\nimages = load_images(links)\ndisplay_images(images)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-09-16T06:05:16.338484Z","iopub.execute_input":"2021-09-16T06:05:16.338723Z","iopub.status.idle":"2021-09-16T06:05:29.609109Z","shell.execute_reply.started":"2021-09-16T06:05:16.338696Z","shell.execute_reply":"2021-09-16T06:05:29.608206Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Exploring the test file\n### Languages in the test file\n\nthis won't work after the data was updated","metadata":{}},{"cell_type":"code","source":"# from matplotlib import rcParams\n\n# a = sns.displot(x='language', data=test_file,color='#73C6B6',height=8, aspect=20/8)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-09-16T06:07:05.236824Z","iopub.execute_input":"2021-09-16T06:07:05.237768Z","iopub.status.idle":"2021-09-16T06:07:05.241504Z","shell.execute_reply.started":"2021-09-16T06:07:05.237716Z","shell.execute_reply":"2021-09-16T06:07:05.240643Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# import plotly.graph_objects as go    \n\n# fig = go.Figure(\n#     data=[ go.Bar(x=test_file['language'].value_counts().index, \n#             y=test_file['language'].value_counts().values,\n#             text=test_file['language'].value_counts().values,\n#             textposition='auto',name='Count',\n#            marker_color='#73C6B6')],\n#     layout_title_text=\"Language Distribution : using plotly v2\"\n# )\n# fig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-09-16T06:07:05.454603Z","iopub.execute_input":"2021-09-16T06:07:05.455045Z","iopub.status.idle":"2021-09-16T06:07:05.458773Z","shell.execute_reply.started":"2021-09-16T06:07:05.454998Z","shell.execute_reply":"2021-09-16T06:07:05.457853Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# import matplotlib.pyplot as plt\n# import squarify    # pip install squarify (algorithm for treemap)\n# plt.figure(figsize=(25,8))\n# squarify.plot(sizes=test_file['language'].value_counts().values, \n#               label=test_file['language'].value_counts().index, \n#               color=[\"#73C6B6\",\"lightgreen\",\"cyan\", \"c\"],\n#               alpha=.8 )\n# plt.title(\"A square graph for the same :D\")\n# plt.axis('off')\n# plt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-09-16T06:07:05.693871Z","iopub.execute_input":"2021-09-16T06:07:05.694354Z","iopub.status.idle":"2021-09-16T06:07:05.699322Z","shell.execute_reply.started":"2021-09-16T06:07:05.694306Z","shell.execute_reply":"2021-09-16T06:07:05.698105Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## We look at a English artices in a bit more detail, I will be using just the first 10000 rows ","metadata":{}},{"cell_type":"code","source":"df = pd.read_csv('../input/wikipedia-image-caption/train-00000-of-00005.tsv', sep='\\t',nrows=10000)\ndf.head(1)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-09-16T06:07:06.202825Z","iopub.execute_input":"2021-09-16T06:07:06.203413Z","iopub.status.idle":"2021-09-16T06:07:06.848677Z","shell.execute_reply.started":"2021-09-16T06:07:06.203375Z","shell.execute_reply":"2021-09-16T06:07:06.847888Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df = df[df['language']==\"en\"]\ndf.head(1)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-09-16T06:07:06.850065Z","iopub.execute_input":"2021-09-16T06:07:06.850289Z","iopub.status.idle":"2021-09-16T06:07:07.051851Z","shell.execute_reply.started":"2021-09-16T06:07:06.850264Z","shell.execute_reply":"2021-09-16T06:07:07.051015Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def display_images(images, title=None): \n    f, ax = plt.subplots(1,1, figsize=(18,12))\n    if title:\n        f.suptitle(title, fontsize = 30)\n\n    for i, image_id in enumerate(images):\n        ax.imshow(image_id) \n        ax.axis('off')\n\n    plt.show()\n\nlinks = df.image_url[0:1]\nimages = load_images(links)\nprint(\"The title of the image is \",df.page_title[0:1])\ndisplay_images(images)\n\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-09-16T06:07:07.053793Z","iopub.execute_input":"2021-09-16T06:07:07.054268Z","iopub.status.idle":"2021-09-16T06:07:09.061280Z","shell.execute_reply.started":"2021-09-16T06:07:07.054227Z","shell.execute_reply":"2021-09-16T06:07:09.060515Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import plotly.graph_objects as go    \n\nfig = go.Figure(\n    data=[ go.Bar(x=df['page_changed_recently'].value_counts().index, \n            y=df['page_changed_recently'].value_counts().values,\n            text=df['page_changed_recently'].value_counts().values,\n            textposition='auto',name='Count',\n           marker_color='#73C6B6')],\n    layout_title_text=\"Has the page been changed recently?\"\n)\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-09-16T06:07:09.062877Z","iopub.execute_input":"2021-09-16T06:07:09.063653Z","iopub.status.idle":"2021-09-16T06:07:09.138307Z","shell.execute_reply.started":"2021-09-16T06:07:09.063614Z","shell.execute_reply":"2021-09-16T06:07:09.137488Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import plotly.graph_objects as go    \n\nfig = go.Figure(\n    data=[ go.Bar(x=df['mime_type'].value_counts().index, \n            y=df['mime_type'].value_counts().values,\n            text=df['mime_type'].value_counts().values,\n            textposition='auto',name='Count',\n           marker_color='#73C6B6')],\n    layout_title_text=\"What is the Distribution of the various file types\"\n)\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-09-16T06:07:09.139836Z","iopub.execute_input":"2021-09-16T06:07:09.140176Z","iopub.status.idle":"2021-09-16T06:07:09.160451Z","shell.execute_reply.started":"2021-09-16T06:07:09.140115Z","shell.execute_reply":"2021-09-16T06:07:09.159547Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import plotly.graph_objects as go    \n\nfig = go.Figure(\n    data=[ go.Bar(x=df['is_main_image'].value_counts().index, \n            y=df['is_main_image'].value_counts().values,\n            text=df['is_main_image'].value_counts().values,\n            textposition='auto',name='Count',\n           marker_color='#73C6B6')],\n    layout_title_text=\"Is the image the main image of the article?\"\n)\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-09-16T06:07:09.162236Z","iopub.execute_input":"2021-09-16T06:07:09.162461Z","iopub.status.idle":"2021-09-16T06:07:09.182817Z","shell.execute_reply.started":"2021-09-16T06:07:09.162435Z","shell.execute_reply":"2021-09-16T06:07:09.181808Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cloud = WordCloud(width=1440, height=1080,stopwords={'nan'},colormap='Greens',background_color='white').generate(\" \".join(df['page_title'].astype(str)))\nplt.figure(figsize=(16, 10))\nplt.title('A WordCloud of the various pages in the file',fontsize=20,pad=40)\nplt.imshow(cloud)\nplt.axis('off')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-09-16T06:07:09.184548Z","iopub.execute_input":"2021-09-16T06:07:09.185078Z","iopub.status.idle":"2021-09-16T06:07:12.288510Z","shell.execute_reply.started":"2021-09-16T06:07:09.185037Z","shell.execute_reply":"2021-09-16T06:07:12.287902Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cloud = WordCloud(width=1440, height=1080,stopwords={'nan'},colormap='Greens',background_color='white').generate(\" \".join(df['section_title'].astype(str)))\nplt.figure(figsize=(16, 10))\nplt.title('A WordCloud of the various section for the pages in the file',fontsize=20,pad=40)\nplt.imshow(cloud)\nplt.axis('off')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-09-16T06:07:12.289526Z","iopub.execute_input":"2021-09-16T06:07:12.289816Z","iopub.status.idle":"2021-09-16T06:07:15.268254Z","shell.execute_reply.started":"2021-09-16T06:07:12.289790Z","shell.execute_reply":"2021-09-16T06:07:15.267672Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Now looking at some data with German language ","metadata":{}},{"cell_type":"code","source":"df = pd.read_csv('../input/wikipedia-image-caption/train-00000-of-00005.tsv', sep='\\t',nrows=10000)\ndf.head(1)\ndf = df[df['language']==\"de\"]\ndf.head(1)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-09-16T06:07:15.269367Z","iopub.execute_input":"2021-09-16T06:07:15.269703Z","iopub.status.idle":"2021-09-16T06:07:15.615684Z","shell.execute_reply.started":"2021-09-16T06:07:15.269664Z","shell.execute_reply":"2021-09-16T06:07:15.615097Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def display_images(images, title=None): \n    f, ax = plt.subplots(1,1, figsize=(18,12))\n    if title:\n        f.suptitle(title, fontsize = 30)\n\n    for i, image_id in enumerate(images):\n        ax.imshow(image_id) \n        ax.axis('off')\n\n    plt.show()\n\nlinks = df.image_url[0:1]\nimages = load_images(links)\nprint(\"The title of the image is \",df.page_title[0:1])\ndisplay_images(images)\n\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-09-16T06:07:15.616678Z","iopub.execute_input":"2021-09-16T06:07:15.616989Z","iopub.status.idle":"2021-09-16T06:07:17.098218Z","shell.execute_reply.started":"2021-09-16T06:07:15.616964Z","shell.execute_reply":"2021-09-16T06:07:17.097316Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cloud = WordCloud(width=1440, height=1080,stopwords={'nan'},colormap='Greens',background_color='white').generate(\" \".join(df['page_title'].astype(str)))\nplt.figure(figsize=(16, 10))\nplt.title('A WordCloud of the various pages in the file',fontsize=20,pad=40)\nplt.imshow(cloud)\nplt.axis('off')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-09-16T06:07:17.099465Z","iopub.execute_input":"2021-09-16T06:07:17.099715Z","iopub.status.idle":"2021-09-16T06:07:20.158501Z","shell.execute_reply.started":"2021-09-16T06:07:17.099669Z","shell.execute_reply":"2021-09-16T06:07:20.157569Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I'll try to read the whole tsv","metadata":{}},{"cell_type":"markdown","source":"If there are any suggesion for the notebook please comment, that would be helpful. Also please upvote if you liked it! Thank you!!\n\nSome of my other works:\n\n* [TPS- APR](https://www.kaggle.com/udbhavpangotra/tps-apr21-eda-model) \n* [HEART ATTACKS](https://www.kaggle.com/udbhavpangotra/heart-attacks-extensive-eda-and-visualizations) \n* [YOUTUBE DATA EXPLORATION](https://www.kaggle.com/udbhavpangotra/what-do-people-use-youtube-for-in-great-britain)\n* [TPS MAY](https://www.kaggle.com/udbhavpangotra/tps-may-21-extensive-eda-catboost-shap)\n* [COVID-19 DIGITAL LEARNING](https://www.kaggle.com/udbhavpangotra/how-did-covid-19-impact-digital-learning-eda)\n* [TPS - SEPT](https://www.kaggle.com/udbhavpangotra/extensive-eda-baseline-shap)\n\n* [also try this dataset ReliefWeb Crisis Figures Data](https://www.kaggle.com/udbhavpangotra/reliefweb-crisis-figures-data)","metadata":{}},{"cell_type":"code","source":"%%html\n<marquee style='width: 90% ;height:70%; color: #45B39D ;'>\n    <b>Do UPVOTE if you like my work, I will be adding some more content to this kernel post understanding the files :) </b></marquee>","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-09-16T06:07:22.323705Z","iopub.execute_input":"2021-09-16T06:07:22.324163Z","iopub.status.idle":"2021-09-16T06:07:22.330051Z","shell.execute_reply.started":"2021-09-16T06:07:22.324099Z","shell.execute_reply":"2021-09-16T06:07:22.329287Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Credits to the people who did some EDA before me helping me in doing the same! \n\n1.  [KALILUR RAHMAN](https://www.kaggle.com/kalilurrahman/wikimedia-image-text-matching-eda), I loved the square plot! \n2.  [RADMIR ZOSIMOV](https://www.kaggle.com/hijest/wikipedia-image-caption-matching-starter-eda)\n3.  [MARÍLIA PRATA](https://www.kaggle.com/mpwolke/wikimedia-urllib)","metadata":{}}]}