{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":71549,"databundleVersionId":8561470,"sourceType":"competition"},{"sourceId":51817,"sourceType":"modelInstanceVersion","modelInstanceId":43578}],"dockerImageVersionId":30699,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"Published on May 19, 2024. By Marília Prata, mpwolke","metadata":{}},{"cell_type":"markdown","source":"# Introduction\n\nOpenLLM-Ro is a new initiative to develop a Romanian LLM, based on existing large language models. The first version is based on Llama2, fine-tuned with a corpus of Romanian language sources data.\n\nFor more details about the project, check this link: https://ilds.ro/llm-for-romanian/\n        \n#The current model - By Gabriel Preda \n\nThe model we are using here was obtained by quantization of the [OpenLLM-Ro/RoLlama2-7b-Instruct](https://huggingface.co/OpenLLM-Ro/RoLlama2-7b-Instruct) model using the notebook [Quantized Romanian LLM (OpenLLM-Ro) Tests](https://www.kaggle.com/code/gpreda/quantized-romanian-llm-openllm-ro-tests).\n\n","metadata":{}},{"cell_type":"markdown","source":"# Load the model","metadata":{}},{"cell_type":"code","source":"!pip install -U transformers accelerate bitsandbytes","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2024-05-19T17:43:24.416220Z","iopub.execute_input":"2024-05-19T17:43:24.416569Z","iopub.status.idle":"2024-05-19T17:43:55.319902Z","shell.execute_reply.started":"2024-05-19T17:43:24.416536Z","shell.execute_reply":"2024-05-19T17:43:55.318962Z"},"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from time import time\nfrom IPython.display import display, Markdown\nfrom transformers import pipeline","metadata":{"execution":{"iopub.status.busy":"2024-05-19T17:44:09.722710Z","iopub.execute_input":"2024-05-19T17:44:09.723063Z","iopub.status.idle":"2024-05-19T17:44:27.042234Z","shell.execute_reply.started":"2024-05-19T17:44:09.723029Z","shell.execute_reply":"2024-05-19T17:44:27.041296Z"},"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pipe = pipeline(\"text-generation\", \n                model=\"/kaggle/input/openllm-ro-q4b/transformers/7b-instruct-q4b/1/RoLlama2-7b-Instruct-v1-q4b\",\n               kwargs=['_load_in_4bit'])","metadata":{"execution":{"iopub.status.busy":"2024-05-19T17:44:37.039087Z","iopub.execute_input":"2024-05-19T17:44:37.039707Z","iopub.status.idle":"2024-05-19T17:45:24.371091Z","shell.execute_reply.started":"2024-05-19T17:44:37.039679Z","shell.execute_reply":"2024-05-19T17:45:24.369980Z"},"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Query function","metadata":{}},{"cell_type":"code","source":"def query_model(\n        system_message,\n        user_message,\n        temperature=0.7,\n        max_length=1024\n        ):\n    start_time = time()\n    user_message = \"Question: \" + user_message + \" Answer:\"\n    messages = [\n        {\"role\": \"system\", \"content\": system_message},\n        {\"role\": \"user\", \"content\": user_message},\n        ]\n    prompt = pipe.tokenizer.apply_chat_template(\n        messages, \n        tokenize=False, \n        add_generation_prompt=True\n        )\n    terminators = [\n        pipe.tokenizer.eos_token_id,\n        pipe.tokenizer.convert_tokens_to_ids(\"<|eot_id|>\")\n    ]\n    sequences = pipe(\n        prompt,\n        do_sample=True,\n        top_p=0.9,\n        temperature=temperature,\n        #num_return_sequences=1,\n        eos_token_id=terminators,\n        max_new_tokens=max_length,\n        return_full_text=False,\n        pad_token_id=pipe.model.config.eos_token_id\n    )\n    #answer = f\"{sequences[0]['generated_text'][len(prompt):]}\\n\"\n    answer = sequences[0]['generated_text']\n    end_time = time()\n    ttime = f\"Total time: {round(end_time-start_time, 2)} sec.\"\n\n    return user_message + \" \" + answer  + \" \" +  ttime\n\n\nsystem_message = \"\"\"\nYou are an AI assistant designed to answer simple questions.\nPlease restrict your answer to the exact question asked.\n\"\"\"","metadata":{"execution":{"iopub.status.busy":"2024-05-19T17:45:51.848177Z","iopub.execute_input":"2024-05-19T17:45:51.848563Z","iopub.status.idle":"2024-05-19T17:45:51.857337Z","shell.execute_reply.started":"2024-05-19T17:45:51.848532Z","shell.execute_reply":"2024-05-19T17:45:51.856385Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Test with a few questions in English and Romanian. ","metadata":{}},{"cell_type":"code","source":"def colorize_text(text):\n    for word, color in zip([\"Reasoning\", \"Question\", \"Answer\", \"Total time\"], [\"blue\", \"red\", \"green\", \"magenta\"]):\n        text = text.replace(f\"{word}:\", f\"\\n\\n**<font color='{color}'>{word}:</font>**\")\n    return text","metadata":{"execution":{"iopub.status.busy":"2024-05-19T17:45:59.517795Z","iopub.execute_input":"2024-05-19T17:45:59.518185Z","iopub.status.idle":"2024-05-19T17:45:59.523434Z","shell.execute_reply.started":"2024-05-19T17:45:59.518156Z","shell.execute_reply":"2024-05-19T17:45:59.522420Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#What are the severity levels of Spinal Canal Stenosis?\n\n","metadata":{}},{"cell_type":"code","source":"response = query_model(\n    system_message,\n    user_message=\"What are the severity levels of Spinal Canal Stenosis?\",\n    temperature=0.1,\n    max_length=32)\ndisplay(Markdown(colorize_text(response)))","metadata":{"execution":{"iopub.status.busy":"2024-05-19T17:46:20.132534Z","iopub.execute_input":"2024-05-19T17:46:20.132904Z","iopub.status.idle":"2024-05-19T17:46:24.444409Z","shell.execute_reply.started":"2024-05-19T17:46:20.132874Z","shell.execute_reply":"2024-05-19T17:46:24.443508Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Care sunt nivelurile de severitate ale stenozei canalului spinal?","metadata":{}},{"cell_type":"code","source":"response = query_model(\n    system_message,\n    user_message=\"Care sunt nivelurile de severitate ale stenozei canalului spinal?\",\n    temperature=0.1,\n    max_length=32)\ndisplay(Markdown(colorize_text(response)))","metadata":{"execution":{"iopub.status.busy":"2024-05-19T17:49:30.612906Z","iopub.execute_input":"2024-05-19T17:49:30.613785Z","iopub.status.idle":"2024-05-19T17:49:33.883269Z","shell.execute_reply.started":"2024-05-19T17:49:30.613755Z","shell.execute_reply":"2024-05-19T17:49:33.882322Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Which is the percent of Moderate level in right_subarticular_stenosis_l1_l2?","metadata":{}},{"cell_type":"code","source":"response = query_model(\n    system_message,\n    user_message=\"Which is the percent of Moderate level in right_subarticular_stenosis_l1_l2?\",\n    temperature=0.1,\n    max_length=64)\ndisplay(Markdown(colorize_text(response)))","metadata":{"execution":{"iopub.status.busy":"2024-05-19T17:46:35.288236Z","iopub.execute_input":"2024-05-19T17:46:35.289112Z","iopub.status.idle":"2024-05-19T17:46:41.011324Z","shell.execute_reply.started":"2024-05-19T17:46:35.289067Z","shell.execute_reply":"2024-05-19T17:46:41.010394Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Care este procentul de nivel moderat în stenoza_subarticulară_dreaptă_l1_l2?","metadata":{}},{"cell_type":"code","source":"response = query_model(\n    system_message,\n    user_message=\"Care este procentul de nivel moderat în stenoza_subarticulară_dreaptă_l1_l2?\",\n    temperature=0.1,\n    max_length=64)\ndisplay(Markdown(colorize_text(response)))","metadata":{"execution":{"iopub.status.busy":"2024-05-19T17:51:02.885098Z","iopub.execute_input":"2024-05-19T17:51:02.885487Z","iopub.status.idle":"2024-05-19T17:51:06.486118Z","shell.execute_reply.started":"2024-05-19T17:51:02.885460Z","shell.execute_reply":"2024-05-19T17:51:06.485082Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#What are the core conditions of Lumbar Spine degeneration?","metadata":{}},{"cell_type":"code","source":"response = query_model(\n    system_message,\n    user_message=\"What are the core conditions of Lumbar Spine degeneration?\",\n    temperature=0.1,\n    max_length=64)\ndisplay(Markdown(colorize_text(response)))","metadata":{"execution":{"iopub.status.busy":"2024-05-19T17:46:52.148885Z","iopub.execute_input":"2024-05-19T17:46:52.149290Z","iopub.status.idle":"2024-05-19T17:46:57.960073Z","shell.execute_reply.started":"2024-05-19T17:46:52.149260Z","shell.execute_reply":"2024-05-19T17:46:57.959154Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Care sunt condițiile de bază ale degenerescenței coloanei vertebrale lombare?","metadata":{}},{"cell_type":"code","source":"response = query_model(\n    system_message,\n    user_message=\"Care sunt condițiile de bază ale degenerescenței coloanei vertebrale lombare?\",\n    temperature=0.1,\n    max_length=64)\ndisplay(Markdown(colorize_text(response)))","metadata":{"execution":{"iopub.status.busy":"2024-05-19T17:52:36.469294Z","iopub.execute_input":"2024-05-19T17:52:36.470148Z","iopub.status.idle":"2024-05-19T17:52:42.258430Z","shell.execute_reply.started":"2024-05-19T17:52:36.470118Z","shell.execute_reply":"2024-05-19T17:52:42.257536Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Citați câțiva dintre marii utilizatori români Kaggle.\n\nQuote some of The Great romanian kagglers.","metadata":{}},{"cell_type":"code","source":"response = query_model(\n    system_message,\n    user_message=\"Citați câțiva dintre marii utilizatori români Kaggle.\",\n    temperature=0.1,\n    max_length=64)\ndisplay(Markdown(colorize_text(response)))","metadata":{"execution":{"iopub.status.busy":"2024-05-19T18:19:27.937920Z","iopub.execute_input":"2024-05-19T18:19:27.938783Z","iopub.status.idle":"2024-05-19T18:19:33.726793Z","shell.execute_reply.started":"2024-05-19T18:19:27.938752Z","shell.execute_reply":"2024-05-19T18:19:33.725961Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"For the record, I've never heard about Alex Iacobescu on Kaggle.","metadata":{}},{"cell_type":"markdown","source":"#Care utilizator Kaggle a scris cartea „Developing Kaggle Notebooks”?","metadata":{}},{"cell_type":"code","source":"response = query_model(\n    system_message,\n    user_message=\"Care utilizator Kaggle a scris cartea „Developing Kaggle Notebooks”?\",\n    temperature=0.1,\n    max_length=64)\ndisplay(Markdown(colorize_text(response)))","metadata":{"execution":{"iopub.status.busy":"2024-05-19T18:05:13.959541Z","iopub.execute_input":"2024-05-19T18:05:13.960276Z","iopub.status.idle":"2024-05-19T18:05:16.804063Z","shell.execute_reply.started":"2024-05-19T18:05:13.960247Z","shell.execute_reply":"2024-05-19T18:05:16.803159Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Wrong! The book \"Developping Kaggle Notebooks\"was written By Gabriel Preda.","metadata":{}},{"cell_type":"markdown","source":"# Conclusions\n\nUsing the quantized model derived from OpenLLM-Ro 7B Instruct (v1) model published by **ILDS**, we can run fast queries in Romanian language.","metadata":{}},{"cell_type":"markdown","source":"#Acknowledgements:\n\nGabriel Preda https://www.kaggle.com/code/gpreda/test-with-quantized-openllm-ro-model","metadata":{}}]}