{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"
\n\n#
Stable Diffusion: Detecting Noise in Text Encoders
\n\n### Table of Contents\n\nThe notebook will be divided into separate sections to provide an organized walk through for the process used. The sections are:\n\n
\n
    \n
  1. Introduction
    2. \n
  2. Import libraries
    3. \n
  3. Load Data
    4. \n
  4. Utils
    5. \n
  5. Build Vocabularies
    6. \n
  6. Check Similarity
    7. \n
\n
","metadata":{"_uuid":"5fb07eae7b445e3bf358222065133a144b7c4ade"}},{"cell_type":"markdown","source":"# 1 | Introduction [↑](#top) \n***\n\nThe goal of this notebook is to detect vocabulary similarity between our input prompts and the text encoder used in this competition. This competition uses the `all-MiniLM-L6-v2` sentence transformer which can be found [here](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2).\n\nWe want to detect **unseen words** by the transformer's tokenizer which are present in our input vocabulary. \n\nUltimately, the best case scenario would be to end up with prompts where its vocabulary is as close as possbile to the hidden test set vocabulary. But we don't know anything about the hidden test set vocabulary.\n\nWhat happens when an unseen word (word not present in tokenizer's vocabulary) is input to our model? Well, transformers can 'fallback' to subword/character-level(-like) embedding if an unseen word is present, but seen words have better embedding representations.\n\nMost likely, the hidden test set won't use prompts with words that come only from the tokenizer's vocabulary, but we can check how similar is our vocabulary to the tokenizer's vocabulary and check if too many unseen words affect performance.\n\n\nThis notebook was based on @christofhenkel excellent [notebook](https://www.kaggle.com/code/christofhenkel/how-to-preprocessing-when-using-embeddings) on how to do pre-processing when using embeddings. ","metadata":{"_uuid":"ec55ed96a9cab0ac75b22f9710c0393b17cfefd2"}},{"cell_type":"markdown","source":"# 2 | Import libraries [↑](#top) \n***","metadata":{}},{"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport numpy as np\nimport operator \nimport pandas as pd\nimport sys\n\n\nfrom pprint import pprint\nfrom tqdm import tqdm","metadata":{"_uuid":"b378958a9606ac48fe0dc54e24bed4cd503e0ac7","execution":{"iopub.status.busy":"2023-03-16T22:46:55.508398Z","iopub.execute_input":"2023-03-16T22:46:55.509074Z","iopub.status.idle":"2023-03-16T22:46:55.559538Z","shell.execute_reply.started":"2023-03-16T22:46:55.509010Z","shell.execute_reply":"2023-03-16T22:46:55.558332Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Paths for different datasets you might want to check","metadata":{"_uuid":"6e432bb170a329c79f58f78527bbbe4e857b2c41"}},{"cell_type":"code","source":"class paths:\n DIFUSSION_DB_META = \"/kaggle/input/diffusiondb-metadata/metadata.parquet\"\n GUSTAVOSTA = \"/kaggle/input/gustavosta-stable-diffusion-prompts-sd2/train.csv\"","metadata":{"execution":{"iopub.status.busy":"2023-03-16T22:46:55.562488Z","iopub.execute_input":"2023-03-16T22:46:55.563555Z","iopub.status.idle":"2023-03-16T22:46:55.571032Z","shell.execute_reply.started":"2023-03-16T22:46:55.563498Z","shell.execute_reply":"2023-03-16T22:46:55.569080Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 3 | Load Data [↑](#top) \n***\n\nLet's load our data. We will be using the DiffusionDB metadata by Alexandre Teles.","metadata":{}},{"cell_type":"code","source":"train_df = pd.read_parquet(paths.DIFUSSION_DB_META)\ntrain_df.rename(columns={\"Prompt\":\"prompt\"}, inplace=True)\ntrain_df[\"prompt\"] = train_df[\"prompt\"].astype(str) # convert prompts to strings\ntrain_df[\"prompt\"] = train_df[\"prompt\"].apply(lambda x: x.lower()) # lower-case prompts\nprint(f\"Train shape: {train_df.shape}\") # print shape\ntrain_df.head()","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2023-03-16T22:46:55.576158Z","iopub.execute_input":"2023-03-16T22:46:55.576902Z","iopub.status.idle":"2023-03-16T22:47:06.510543Z","shell.execute_reply.started":"2023-03-16T22:46:55.576802Z","shell.execute_reply":"2023-03-16T22:47:06.508430Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 4 | Utils [↑](#top) \n***\n\nUtility functions we will use throughout the notebook.","metadata":{}},{"cell_type":"code","source":"def build_vocab(sentences, verbose=True):\n \"\"\"\n Builds a vocabulary dictionary where keys are the unique words in our sentences and \n the values are the word counts.\n :param sentences: list of list of words.\n :return: dictionary of words and their count.\n \"\"\"\n vocab = {}\n for sentence in tqdm(sentences, disable = (not verbose)):\n for word in sentence:\n try:\n vocab[word] += 1\n except KeyError:\n vocab[word] = 1\n return vocab\n\n\ndef division(n, d):\n \"\"\"Avoid zero division\"\"\"\n return n / d if d else 0\n\n\ndef check_coverage(vocab, embeddings_index):\n \"\"\"\n :param vocab: a python dictionary with all the words in our dataframe as keys and their count as value.\n :param embeddings_index: a dict-like object where its keys are words and the values are index or the corresponding word's embedding.\n \"\"\"\n a = {} # auxiliary dict\n oov = {} # out of vocabulary\n k = 0 # match counter\n i = 0 # no match counter\n # === Iterate over our dataframe's words ===\n for word in tqdm(vocab): \n try:\n a[word] = embeddings_index[word] # check if word exists in embeddings\n k += vocab[word] # if it exists add to match counter\n except:\n oov[word] = vocab[word] # if word not exists in embeddings add to out of vocabulary dict\n i += vocab[word] # if it does not exists add to no match counter\n pass\n\n print('Found embeddings for {:.2%} of vocab'.format(len(a) / len(vocab)))\n print('Found embeddings for {:.2%} of all text'.format(k / (k + i)))\n sorted_x = sorted(oov.items(), key=operator.itemgetter(1))[::-1]\n\n return sorted_x\n\n\ndef count_sentence_match(sentences, embeddings_index):\n \"\"\"\n :param sentences: list of list of words\n :return: dictionary of words and their count\n \"\"\"\n sentences_matches = []\n for sentence in tqdm(sentences):\n match = 0 # match counter\n no_match = 0 # no match counter\n for word in sentence:\n try:\n embeddings_index[word]\n match += 1\n except KeyError:\n no_match += 1\n sentences_matches.append(division(match,(match + no_match)))\n \n return sentences_matches\n\n\ndef count_word_exists(sentences, vocab_dict):\n \"\"\"\n :param sentences: list of list of words\n :return: dictionary of words and values\n \"\"\"\n sentences_matches = []\n word_appeared = []\n for sentence in tqdm(sentences):\n exist = 0 # match counter\n word_list = []\n for word in sentence:\n try:\n vocab_dict[word]\n exist += 1\n word_list.append(word)\n except KeyError:\n pass\n sentences_matches.append(exist)\n word_appeared.append(word_list)\n \n \n return sentences_matches, word_appeared\n\n\ndef check_intersection(vocab_input, vocab_tokenizer):\n vocab_input = list(vocab_input.keys())\n vocab_tokenizer = list(vocab_tokenizer.keys())\n intersection = list(set(vocab_input) & set(vocab_tokenizer))\n input_percentage = len(intersection)/len(vocab_input)\n tokenizer_percentage = len(intersection)/len(vocab_tokenizer)\n print(f\"Input vocabulary has {len(vocab_input)} words\")\n print(f\"Tokenizer vocabulary has {len(vocab_tokenizer)} words\")\n print(f\"There are {len(intersection)} words in common.\")\n print(f\"Input vocabulary has {100*round(input_percentage,3)}% similar words with tokenizer.\")\n print(f\"Tokenizer vocabulary has {100*round(tokenizer_percentage,3)}% similar words to input.\")","metadata":{"_uuid":"3e050f2fa9668c765466d766cfd79720cf6cc819","_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-03-16T22:47:06.513600Z","iopub.execute_input":"2023-03-16T22:47:06.514061Z","iopub.status.idle":"2023-03-16T22:47:06.545283Z","shell.execute_reply.started":"2023-03-16T22:47:06.514020Z","shell.execute_reply":"2023-03-16T22:47:06.541275Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 5 | Build Vocabularies [↑](#top) \n***\n\nWe will build our input vocabulary, which is a dictionary with unique words as keys and word counts as values.","metadata":{}},{"cell_type":"code","source":"tqdm.pandas()\nsentences = train_df[\"prompt\"].progress_apply(lambda x: x.split()).values\nvocab_input = build_vocab(sentences)\nprint(f\"There are {len(vocab_input)} unique words in our vocabulary\")\npprint({k: vocab_input[k] for k in list(vocab_input)[:5]})","metadata":{"execution":{"iopub.status.busy":"2023-03-16T22:47:06.548526Z","iopub.execute_input":"2023-03-16T22:47:06.549209Z","iopub.status.idle":"2023-03-16T22:47:40.734795Z","shell.execute_reply.started":"2023-03-16T22:47:06.549147Z","shell.execute_reply":"2023-03-16T22:47:40.733897Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Next, we will get our tokenizer's vocabulary. We will compare it later to our input vocabulary.\n\n
\n
NOTE:
\n \nPlease note that the tokenizer vocabulary contains word tokens, subword tokens (tokens with `##`) and character-level tokens. Also, the values is not word count like in our input vocabulary, but rather an index.\n
","metadata":{"_uuid":"b5f9048e53a5cb91bdaa518d33fda756f1038562"}},{"cell_type":"code","source":"from transformers import AutoTokenizer\n\n# === Get Tokenizer ===\ntokenizer = AutoTokenizer.from_pretrained('sentence-transformers/all-MiniLM-L6-v2')\nvocab_tokenizer = tokenizer.get_vocab() # get vocabulary\nprint(f\"There are {len(vocab_tokenizer)} unique words in our vocabulary\")\npprint({k: vocab_tokenizer[k] for k in list(vocab_tokenizer)[:5]})","metadata":{"execution":{"iopub.status.busy":"2023-03-16T22:47:40.736068Z","iopub.execute_input":"2023-03-16T22:47:40.736416Z","iopub.status.idle":"2023-03-16T22:47:56.142460Z","shell.execute_reply.started":"2023-03-16T22:47:40.736382Z","shell.execute_reply":"2023-03-16T22:47:56.140712Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 6 | Check Similarity [↑](#top) \n***\n\nWe will check how many words are in common between our two vocabularies and get a `match` score for each prompt in our input vocabulary.","metadata":{}},{"cell_type":"code","source":"oov = check_coverage(vocab_input, vocab_tokenizer)\nsentences_matches = count_sentence_match(sentences, vocab_tokenizer)\ncheck_intersection(vocab_input, vocab_tokenizer)\ntrain_df[\"match\"] = sentences_matches\ntrain_df.head()","metadata":{"execution":{"iopub.status.busy":"2023-03-16T22:47:56.144487Z","iopub.execute_input":"2023-03-16T22:47:56.145709Z","iopub.status.idle":"2023-03-16T22:48:14.270333Z","shell.execute_reply.started":"2023-03-16T22:47:56.145648Z","shell.execute_reply":"2023-03-16T22:48:14.269203Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Some conclusions:\n1. From **all unique words** in our input vocabulary, only 7.21% are present in the tokenizer vocabulary.\n2. From **all words** in our input vocabulary, 63.64% are present in the tokenizer vocabulary.\n3. Our input vocabulary and our tokenizer vocabulary have 20613 words in common. This represents 7.2% of the input vocabulary and 67.5% of the tokenizer vocabulary.\n4. Please note that, since the tokenizer have subword embeddings, the similarity can be much higher. Our input vocabulary has word-level tokens which do not give matches here, but a word can be accurately represented by sub-word tokens.\n\n### Match distribution\n\nLet's check our `match` distribution. The `match` column is a value between 0 and 1. For each input prompt it indicates the percentage of the input prompt words which are common/seen by the tokenizer. For example, a value of 1 indicates that all words present in the prompt were seen by the tokenizer.","metadata":{}},{"cell_type":"code","source":"_ = plt.hist(train_df[\"match\"], bins=50)\nplt.xlabel(\"Match percentage\")\nplt.ylabel(\"Frequency\")\nplt.title(\"DiffusionDB metadata\")","metadata":{"execution":{"iopub.status.busy":"2023-03-16T22:48:14.271497Z","iopub.execute_input":"2023-03-16T22:48:14.271875Z","iopub.status.idle":"2023-03-16T22:48:14.746788Z","shell.execute_reply.started":"2023-03-16T22:48:14.271827Z","shell.execute_reply":"2023-03-16T22:48:14.745414Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df[\"match\"].describe()","metadata":{"execution":{"iopub.status.busy":"2023-03-16T22:48:14.748904Z","iopub.execute_input":"2023-03-16T22:48:14.750237Z","iopub.status.idle":"2023-03-16T22:48:14.863996Z","shell.execute_reply.started":"2023-03-16T22:48:14.750171Z","shell.execute_reply":"2023-03-16T22:48:14.862649Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.to_csv(\"train_meta.csv\", index=False)","metadata":{"execution":{"iopub.status.busy":"2023-03-16T22:49:11.362373Z","iopub.execute_input":"2023-03-16T22:49:11.362844Z","iopub.status.idle":"2023-03-16T22:49:49.156939Z","shell.execute_reply.started":"2023-03-16T22:49:11.362790Z","shell.execute_reply":"2023-03-16T22:49:49.155889Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"What to do next:\n\n- You can select a subset of your data with the amount of `match` you want and experiment.\n- Train with high match only prompts (probably won't be a very robust model).\n- Train with low match only prompts (probably too much noise (?))\n- Try cleaning the input vocabulary.","metadata":{}}]}