{ "cells": [ { "cell_type": "markdown", "metadata": { "_cell_guid": "813e742d-62d6-8148-90ab-6986dd4c70fd" }, "source": [ "# YouTube8M EDA\n", "\n", "The YouTube8M challenge is a multi-class classification problem, where we are asked to predict for each video, given video & frame level audio and frame RGB features, to which group of categories it belongs to.\n", "\n", "For this task it is important to know:\n", "\n", "* the total **number of classes/video categories**\n", "* the **distribution of classes in the training set**\n", "\n", "In the first part I'll be focusing on how to take apart detailed information about the labels for the training data. Especially taking into account frequent patterns that occur in the data. We will also be looking at co-dependencies of the most frequent label categories.\n", "\n", "I've added a small section about the dependence of different video categories to the number of labels for each sample.\n", "\n", "\n", "----------" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "58e52a94-b0b7-40a7-3395-ce9a00e24103" }, "outputs": [], "source": [ "import numpy as np # linear algebra\n", "import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n", "import tensorflow as tf\n", "import seaborn as sns\n", "from IPython.display import YouTubeVideo\n", "import matplotlib.pyplot as plt\n", "\n", "from subprocess import check_output\n", "print(check_output([\"ls\", \"../input\"]).decode(\"utf8\"))\n", "print(check_output([\"ls\", \"../input/video_level\"]).decode(\"utf8\"))" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "ba78b60a-2feb-97b5-1baa-40c159dfe488" }, "source": [ "### Read label names and count the distinct number of labels" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "ade18c08-f02f-7c0c-5349-cd04131b6732" }, "outputs": [], "source": [ "labels_df = pd.read_csv('../input/label_names.csv')\n", "filenames = [\"../input/video_level/train-{}.tfrecord\".format(i) for i in range(10)]\n", "print(\"we have {} unique labels in the dataset\".format(len(labels_df['label_name'].unique())))" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "93c07a94-bcd3-f08e-1103-d1482cd154a1" }, "source": [ "### read labels for all training samples and map to textual representation\n", "\n", "in `'../input/label_names.csv'` there are a few label ids missing" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "05374563-73fb-e441-de58-8cdad85faeae" }, "outputs": [], "source": [ "labels_df = pd.read_csv('../input/label_names.csv')\n", "labels = []\n", "textual_labels = []\n", "textual_labels_nested = []\n", "filenames = [\"../input/video_level/train-{}.tfrecord\".format(i) for i in range(10)]\n", "total_sample_counter = 0\n", "\n", "label_counts = []\n", "\n", "for filename in filenames:\n", " for example in tf.python_io.tf_record_iterator(filename):\n", " total_sample_counter += 1\n", " tf_example = tf.train.Example.FromString(example)\n", "\n", " label_example = list(tf_example.features.feature['labels'].int64_list.value)\n", " label_counts.append(len(label_example))\n", " labels = labels + label_example\n", " label_example_textual = list(labels_df[labels_df['label_id'].isin(label_example)]['label_name'])\n", " textual_labels_nested.append(set(label_example_textual))\n", " textual_labels = textual_labels + label_example_textual\n", " if len(label_example_textual) != len(label_example):\n", " print('label names lookup failed: {} vs {}'.format(label_example, label_example_textual))\n", "\n", "print('label ids missing from label_names.csv: {}'.format(sorted(set(labels) - set(labels_df['label_id']))))\n", "print('Found {} samples in all of the 10 available tfrecords'.format(total_sample_counter))" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "74c39e96-7687-65e9-a93c-34795f2a5a17" }, "source": [ "### Label count distribution\n", "\n", "For each sample in the available training set, we've counted the number of labels assigned to it. From this subset of the training data we can now estimate the true distribution of number of labels.\n", "\n", "As we can see, the majority of samples has rather low number of labels attached." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "91446f4a-2731-6a4c-167a-b443ba105015" }, "outputs": [], "source": [ "sns.distplot(label_counts, kde=False)\n", "plt.title('distribution of number of labels')" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "8362cf31-0a2a-cb75-916e-4c3ffe808bc5" }, "source": [ "## Now, lets have a look at the most common labels\n", "\n", "First, we have a look at single labels, simply by counting all of the labels." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "afa9901d-39f5-becb-474b-feba0beb658d" }, "outputs": [], "source": [ "# define helper function to group data\n", "def grouped_data_for(l):\n", " # wrap the grouped data into dataframe, since the inner is pd.Series, not what we need\n", " l_with_c = pd.DataFrame(\n", " pd.DataFrame({'label': l}).groupby('label').size().rename('n')\n", " ).sort_values('n', ascending=False).reset_index()\n", " return l_with_c" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "c5196276-4f83-b5a0-2d86-0c2aa90c7b07" }, "source": [ "Show top 20 labels by occurence in the subsample of the training set available to kernels" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "8bf985a7-a800-3bd6-1219-5f7a958f6a98" }, "outputs": [], "source": [ "N = 20\n", "\n", "textual_labels_with_counts_all = grouped_data_for(textual_labels)\n", "\n", "sns.barplot(y='label', x='n', data=textual_labels_with_counts_all.iloc[0:N, :])\n", "plt.title('Top {} labels with sample count'.format(N))" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "40f2d181-f326-7407-b39b-f9ddfa29ef0d" }, "outputs": [], "source": [ "#singlets = [x for x in textual_labels_nested if len(x) == 1]\n", "# flatten\n", "#singlets = [item for sublist in singlets for item in sublist]\n", "#N = 20\n", "#textual_labels_with_counts = grouped_data_for(singlets)\n", "#sns.barplot(y='label', x='n', data=textual_labels_with_counts.iloc[0:N, :])\n", "#plt.title('Top {} labels with sample count from only samples with one label'.format(N))" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "a60b1973-35e6-7ef6-8092-2f02b7a73f93" }, "source": [ "Most common label combinations for 2-element label assignments" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "037e27f5-e813-4ab7-fd47-365a68104845" }, "outputs": [], "source": [ "two_element_labels = ['|'.join(sorted(x)) for x in textual_labels_nested if len(x) == 2]\n", "\n", "N = 20\n", "\n", "textual_labels_with_counts = grouped_data_for(two_element_labels)\n", "\n", "sns.barplot(y='label', x='n', data=textual_labels_with_counts.iloc[0:N, :])\n", "plt.title('Top {} labels with sample count from only samples with two labels'.format(N))" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "d96ea9df-d2ed-30ff-d5c0-d4076168ccc5" }, "source": [ "For label triplets" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "e3cabc2d-6452-4bc9-d571-a743232a8e31" }, "outputs": [], "source": [ "two_element_labels = ['|'.join(sorted(x)) for x in textual_labels_nested if len(x) == 3]\n", "\n", "N = 20\n", "\n", "textual_labels_with_counts = grouped_data_for(two_element_labels)\n", "\n", "sns.barplot(y='label', x='n', data=textual_labels_with_counts.iloc[0:N, :])\n", "plt.title('Top {} labels with sample count from only samples with three labels'.format(N))" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "38b68839-e1e6-ba60-2602-b1f420555152" }, "source": [ "## For each of the most occuring labels, how does the group size in average differ?\n", "\n", "For this, we take the top 50 labels by count, and compute for each how many other labels are assigned to samples that also have the respective label from the top 50." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "d26656c1-c9a6-550c-14ed-19bd74b31b71" }, "outputs": [], "source": [ "top_50_labels = list(textual_labels_with_counts_all['label'][0:50].values)\n", "top_50_labels" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "72b4508b-d5ff-3da4-a51c-60395007b969" }, "outputs": [], "source": [ "label_group_counts = []\n", "labels_for_group_counts = []\n", "for label in top_50_labels:\n", " # this is a list of lists and\n", " # for each of the inner lists we want to know how many elements there are\n", " nested_labels_with_label = [x for x in textual_labels_nested if label in x]\n", " group_counts = [len(x) for x in nested_labels_with_label]\n", " label_group_counts = label_group_counts + group_counts\n", " labels_for_group_counts = labels_for_group_counts + [label]*len(group_counts)\n", "\n", "count_df = pd.DataFrame({'label': labels_for_group_counts, 'group_size': label_group_counts})\n", "count_df.head()" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "2c084345-7c67-7db7-e8cf-e6b601a0d3ab" }, "source": [ "If we look at the extremes, we can see that for example if **Football** is in the labels of one example, you would find a higher count of accompanying labels when compared to samples that have **Food** label." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "1e083bd4-aba3-c862-1640-0644f8f85714" }, "outputs": [], "source": [ "plt.figure(figsize=(12,8))\n", "sns.barplot(y='label', x='group_size', data=count_df)\n", "plt.title('avg number of labels per top 50 categories')" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "cbf81bc7-1d4c-0427-608e-fe10ac77c239" }, "source": [ "### Distributions of label counts split by a few top labels\n", "\n", "Here we can see that, depending on the category, the group size distribution varies quite a lot. This information can be used to regularize the number of labels for a given video, e.g. if we are confident that the video is about **Vehicle** we can then constrain the number of additional labels for that sample to be within the distribution that we estimated from the training data." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "f3430e55-bb4e-a8b4-580b-a996e81861b9" }, "outputs": [], "source": [ "bins = [0, 1, 3, 14]\n", "colors = ['r', 'g', 'b', 'y']\n", "\n", "plt.figure(figsize=(12,8))\n", "for bin, color in zip(bins, colors):\n", " sns.distplot(count_df[count_df['label'] == top_50_labels[bin]]['group_size'], kde=True, color=color)\n", "\n", "plt.legend([top_50_labels[bin] for bin in bins])" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "2e91eedd-4ff9-c2fd-6ad2-333bff0ea0bf" }, "source": [ "### Estimate the probability of label occurence, given another label from training data\n", "\n", "This probability can be interpreted as a similarity measure between labels." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "10bfccab-62b2-daef-d7a7-83c61878f7a0" }, "outputs": [], "source": [ "K_labels = []\n", "\n", "for i in top_50_labels:\n", " row = []\n", " for j in top_50_labels:\n", " # find all records that have label `i` in them\n", " i_occurs = [x for x in textual_labels_nested if i in x]\n", " # how often does j occur in total in them?\n", " j_and_i_occurs = [x for x in i_occurs if j in x]\n", " k = 1.0*len(j_and_i_occurs)/len(i_occurs)\n", " row.append(k)\n", " K_labels.append(row)\n", "\n", "K_labels = np.array(K_labels)\n", "K_labels = pd.DataFrame(K_labels)\n", "K_labels.columns = top_50_labels\n", "K_labels.index = top_50_labels" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "4380f8cd-8ffe-e37f-c41e-100034346eec" }, "outputs": [], "source": [ "K_labels.head()" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "a1ee067b-2002-437d-c17a-de4798c9fe63" }, "source": [ "The similarity matrix is not symmetric because observing label A given B is not the same as observing B given A.\n", "\n", "For rows:\n", "\n", "* football, the sims, call of duty, racing and race track\n", "\n", "it intuitively makes sense to observe high likelihood of *games* label." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "8be5253d-5e01-f008-0912-4743ab1e4e8b" }, "outputs": [], "source": [ "plt.figure(figsize=(12,8))\n", "sns.heatmap(K_labels)\n", "# probability of observing column label given row label\n", "plt.title('P(column|row)')" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "af9bae12-13a6-1957-9fa0-c27d742c52a1" }, "source": [ "****" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "1e3733c8-df4e-db18-266a-8ec9334944d4" }, "source": [ "# Video level RGB features\n", "\n", "For this section I'll be going a little deeper in the video level RGB features, to get an idea about how they vary across different labels.\n", "The basic idea is to compare the distributions of the averaged video level features. To do this, we are going to take the average of the video level RGB features, which is a very compressed format of that feature and compare the distribution of these values across top labels." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "0a8d977d-7767-8583-4376-367a74b8288c" }, "outputs": [], "source": [ "filenames = [\"../input/video_level/train-{}.tfrecord\".format(i) for i in range(10)]\n", "\n", "cosmetics = []\n", "games = []\n", "car = []\n", "vehicle = []\n", "animal = []\n", "\n", "for filename in filenames:\n", " for example in tf.python_io.tf_record_iterator(filename):\n", " tf_example = tf.train.Example.FromString(example)\n", " label_example = list(tf_example.features.feature['labels'].int64_list.value)\n", " label_example_textual = list(labels_df[labels_df['label_id'].isin(label_example)]['label_name'])\n", " mean_mean_rgb = np.mean(tf_example.features.feature['mean_rgb'].float_list.value)\n", " for label in label_example_textual:\n", " if label == 'Cosmetics':\n", " cosmetics.append(mean_mean_rgb)\n", " elif label == 'Games':\n", " games.append(mean_mean_rgb)\n", " elif label == 'Car':\n", " car.append(mean_mean_rgb)\n", " elif label == 'Vehicle':\n", " vehicle.append(mean_mean_rgb)\n", " elif label == 'Animal':\n", " animal.append(mean_mean_rgb)\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "c7b77e5f-7078-6365-5504-e929056afbf8" }, "outputs": [], "source": [ "len(cosmetics), len(games), len(car), len(vehicle), len(animal)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "1b8ec1b6-53be-78ce-4cd5-56735854e799" }, "outputs": [], "source": [ "sns.distplot(cosmetics, color='b')\n", "sns.distplot(games, color='r')\n", "sns.distplot(car, color='k')\n", "sns.distplot(vehicle, color='c')\n", "sns.distplot(animal, color='y')" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "a3d84734-1ee0-f44f-60a3-71af44e429ae" }, "source": [ "As expected the distributions don't differ a lot by just looking at them. You could now do a one way ANOVA test to find out whether the mean of any pair of these groups differ significantly, but let's not do that now.\n", "\n", "----------\n" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "3c03b468-9208-c326-192e-6cd8c085f265" }, "source": [ "# Frame level features\n", "\n", "For this section, we are going to have a detailed look at the audio frame level features of a few selected videos.\n", "The frame level features, audio and video, were extracted by taking a sample each second uniformly across the whole video.\n", "\n", "See [this][1] forum post for details.\n", "\n", "\n", " [1]: https://www.kaggle.com/c/youtube8m/discussion/28848#162409" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "dbaa76bc-bb41-5e82-28a8-83445fb6d6cc" }, "outputs": [], "source": [ "frame_lvl_record = \"../input/frame_level/train-1.tfrecord\"\n", "\n", "def rgb_and_audio_from(tf_seq_example):\n", " feat_rgb = []\n", " feat_audio = []\n", " n_frames = len(tf_seq_example.feature_lists.feature_list['audio'].feature)\n", " sess = tf.InteractiveSession()\n", " rgb_frame = []\n", " audio_frame = []\n", " # iterate through frames for that example\n", " for i in range(n_frames):\n", " rgb_frame.append(tf.cast(tf.decode_raw(\n", " tf_seq_example.feature_lists.feature_list['rgb'].feature[i].bytes_list.value[0],tf.uint8)\n", " ,tf.float32).eval())\n", " audio_frame.append(tf.cast(tf.decode_raw(\n", " tf_seq_example.feature_lists.feature_list['audio'].feature[i].bytes_list.value[0],tf.uint8)\n", " ,tf.float32).eval())\n", " \n", " \n", " sess.close()\n", " feat_rgb.append(rgb_frame)\n", " feat_audio.append(audio_frame)\n", " return feat_rgb, feat_audio\n", "\n", "# now, let's read the frame-level data\n", "# due to execution time, we're only going to read the first video\n", "\n", "i = 0\n", "for example in tf.python_io.tf_record_iterator(frame_lvl_record): \n", "# i+=1\n", "# if i < 3:\n", "# continue\n", " tf_seq_example = tf.train.SequenceExample.FromString(example)\n", " video_id = tf_seq_example.context.feature['video_id'].bytes_list.value[0]\n", " print('https://www.youtube.com/watch?v={}'.format(str(video_id)))\n", " rgb, audio = rgb_and_audio_from(tf_seq_example)\n", " break" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "20f71a2b-6189-594e-ac28-407b83e8e2a7" }, "outputs": [], "source": [ "#YouTubeVideo('-1xYbUNeA7U')" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "21b2322d-f06a-0900-3963-c913fa6f279a" }, "outputs": [], "source": [ "audio = np.array(audio).squeeze()\n", "# audio now in (128, 168)\n", "audio = np.transpose(audio, (1, 0))\n", "print(audio.shape)" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "03b7f7af-8373-fa03-2d38-9c794d6cb08a" }, "source": [ "So we have a two-dimensional array for the audio frame level feature of one video. Each feature has a dimensionality of **128**, so the video is 161 seconds long.\n", "As it says in the post above, both audio and video frame level features were transformed with a fixed point PCA. However for illustration I'll just consider each of the 161 audio features as a 128-point FFT, now let's see if there is some structure once we have reshaped it and plotted it.\n", "\n", "We can then interpret this reshaped data as a [spectrogram][1], very frequently used in digital audio signal processing, where each row corresponds to a specific frequency, each column to a timestep (for us 1 second) and the color of each of the cells to the magnitude of the audio frequency in that frequency bin.\n", "\n", "\n", " [1]: https://en.wikipedia.org/wiki/Spectrogram" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "2e637423-601a-53fd-d2b7-c9b7e505ad24" }, "outputs": [], "source": [ "cmap = plt.get_cmap('viridis') # plasma\n", "# let's also look at the first order diff across time\n", "daudio = np.diff(audio, axis=1)\n", "plt.figure(figsize=(16,8))\n", "plt.subplot(121)\n", "sns.heatmap(audio, cmap=cmap)\n", "plt.axvline(x=24, ymin=0, ymax=128, color='r')\n", "plt.axvline(x=40, ymin=0, ymax=128, color='r')\n", "plt.axvline(x=62, ymin=0, ymax=128, color='r')\n", "plt.axvline(x=91, ymin=0, ymax=128, color='r')\n", "plt.subplot(122)\n", "sns.heatmap(daudio, cmap=cmap)" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "547da857-9b66-1ce3-bfc9-e21547a1a968" }, "source": [ "You can see vertical lines of audio segments over time, they are also visible in the first order difference. I've also marked the beginnings with red lines. Can we still make these out when averaging over the 128 dimensions?" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "2cd7a4f0-fcda-3ee4-4a3a-14b0d1372ea6" }, "outputs": [], "source": [ "plt.figure()\n", "plt.plot(audio.mean(axis=0))\n", "# plot the same indicator lines\n", "plt.axvline(x=24, ymin=0, ymax=200, color='r')\n", "plt.axvline(x=40, ymin=0, ymax=200, color='r')\n", "plt.axvline(x=62, ymin=0, ymax=200, color='r')\n", "plt.axvline(x=91, ymin=0, ymax=200, color='r')" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "af26748b-9eaf-75c1-ba4a-757db81d1dbe" }, "source": [ "" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "52f1c3de-1d8f-6603-23fd-daee995f5309" }, "outputs": [], "source": [ "" ] } ], "metadata": { "_change_revision": 0, "_is_fork": false, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.0" } }, "nbformat": 4, "nbformat_minor": 0 }