{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":38760,"databundleVersionId":4493939,"sourceType":"competition"},{"sourceId":1187,"sourceType":"datasetVersion","datasetId":626},{"sourceId":2350192,"sourceType":"datasetVersion","datasetId":1418899}],"dockerImageVersionId":30587,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\n# for dirname, _, filenames in os.walk('/kaggle/input'):\n#     for filename in filenames:\n#         print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2024-02-17T03:08:25.156368Z","iopub.execute_input":"2024-02-17T03:08:25.156721Z","iopub.status.idle":"2024-02-17T03:08:25.491795Z","shell.execute_reply.started":"2024-02-17T03:08:25.156690Z","shell.execute_reply":"2024-02-17T03:08:25.490839Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Source\n - https://github.com/NicolasHug/Surprise","metadata":{}},{"cell_type":"markdown","source":"### Recommender systems \n - Recommender systemsare algorithms and techniques designed to suggest items or information to users based on their preferences or behavior. There are several types of recommender systems, and they can be broadly classified into three main categories: collaborative filtering, content-based filtering, and hybrid methods. Here's an overview of each type:\n\n  - Collaborative Filtering: \n        - User-based Collaborative Filtering (UBCF): This approach recommends items to a user based on the preferences and behaviors of users who are similar to them. If user A and user B have similar tastes and preferences, and user A likes an item that user B has not yet seen, the system may recommend that item to user B.\n        - Item-based Collaborative Filtering (IBCF): This method recommends items that are similar to those the user has shown interest in. It identifies relationships between items based on user preferences and recommends items that are similar to the ones the user has liked or interacted with.\n\n- Content-Based Filtering:\n\n        - Content-based recommender systems: These systems recommend items to users based on the characteristics or features of the items and a profile of the user's preferences. For example, in a movie recommendation system, if a user has liked action movies in the past, the system might recommend other action movies.\n        - TF-IDF (Term Frequency-Inverse Document Frequency): This is a common technique used in content-based filtering to represent the importance of terms in a document.\n        \n\n- Hybrid Methods:\n\n        - Combination of Collaborative Filtering and Content-Based Filtering: Hybrid recommender systems aim to leverage the strengths of both collaborative and content-based methods. They can be implemented in various ways, such as by making separate predictions with each method and combining them, or by integrating the models into a single model.\n         - Model-based Hybrid Methods: These approaches use machine learning algorithms to combine collaborative and content-based methods in a unified model.\n         \n         \n- Knowledge-based recommender systems: These systems recommend items based on explicit knowledge about the user's preferences and the items themselves. They often use rule-based systems or knowledge graphs to infer recommendations.\n\n- Context-aware recommender systems: These systems take into account additional contextual information such as time, location, or the user's current activity to provide more personalized recommendations.\n\n- Neural Collaborative Filtering (NCF): This is an approach that uses neural networks to model user-item interactions. It combines the strengths of collaborative filtering and deep learning techniques.","metadata":{}},{"cell_type":"markdown","source":"- Example of  recommendation system in Python using collaborative filtering, which is a common approach in recommendation systems","metadata":{}},{"cell_type":"code","source":"import numpy as np\nfrom sklearn.metrics.pairwise import cosine_similarity\n\n# Sample user-item matrix (user x item)\n# In a real-world scenario, you would have a much larger and sparse matrix\nuser_item_matrix = np.array([\n    [1, 0, 1, 0, 1],\n    [0, 1, 1, 1, 0],\n    [1, 0, 0, 1, 1],\n    [0, 1, 0, 1, 0],\n])\n\n# Sample user-item interactions (1 indicates interaction, 0 indicates no interaction)\n# In a real-world scenario, obtain this from user data\nuser_interactions = np.array([\n    [1, 0, 1, 0, 0],\n    [0, 1, 0, 1, 0],\n    [1, 0, 0, 0, 1],\n    [0, 0, 1, 1, 0],\n])\n\n# Compute similarity matrix using cosine similarity\nsimilarity_matrix = cosine_similarity(user_item_matrix)\n\n# Make recommendations for a target user\ntarget_user_index = 0\ntarget_user_interactions = user_interactions[target_user_index]\n\n# Set the similarity of the target user to themselves to 0 (ignore self-similarity)\nsimilarity_matrix[target_user_index, target_user_index] = 0\n\n# Calculate recommendation scores\nrecommendation_scores = np.dot(similarity_matrix[target_user_index], user_interactions)\n\n# Sort recommendations by score\nsorted_recommendations = np.argsort(recommendation_scores)[::-1]\n\n# Print top N recommendations\ntop_n = 2\ntop_recommendations = sorted_recommendations[:top_n]\n\nprint(f\"Top {top_n} recommendations for user {target_user_index}: {top_recommendations}\")\n","metadata":{"execution":{"iopub.status.busy":"2023-12-15T04:47:15.758270Z","iopub.execute_input":"2023-12-15T04:47:15.759017Z","iopub.status.idle":"2023-12-15T04:47:16.630589Z","shell.execute_reply.started":"2023-12-15T04:47:15.758974Z","shell.execute_reply":"2023-12-15T04:47:16.629702Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# collaborative filtering with the Surprise library\nimport pandas as pd\nfrom surprise import Dataset, Reader\nfrom surprise.model_selection import train_test_split\nfrom surprise import KNNBasic\nfrom surprise import accuracy\n\n# Sample data (user_id, item_id, rating)\ndata = [\n    (1, 'Movie1', 5),\n    (1, 'Movie2', 4),\n    (1, 'Movie3', 2),\n    (2, 'Movie1', 3),\n    (2, 'Movie2', 5),\n    (2, 'Movie3', 4),\n    (3, 'Movie1', 4),\n    (3, 'Movie2', 2),\n    (3, 'Movie3', 5),\n]\n\n# Convert data to a pandas DataFrame\ndf = pd.DataFrame(data, columns=['user_id', 'item_id', 'rating'])\n\n# Define the data format for Surprise\nreader = Reader(rating_scale=(1, 5))\ndataset = Dataset.load_from_df(df[['user_id', 'item_id', 'rating']], reader)\n\n# Split the data into training and testing sets\ntrainset, testset = train_test_split(dataset, test_size=0.25)\n\n# Use user-based collaborative filtering with the k-nearest neighbors algorithm\nsim_options = {\n    'name': 'cosine',\n    'user_based': True\n}\nknn_model = KNNBasic(sim_options=sim_options)\n\n# Train the model\nknn_model.fit(trainset)\n\n# Make predictions on the test set\npredictions = knn_model.test(testset)\n\n# Evaluate the model\naccuracy.rmse(predictions)\n\n# Make recommendations for a specific user (user_id = 1)\nuser_id_to_recommend = 1\nitems_to_recommend = set(df['item_id']) - set(df[df['user_id'] == user_id_to_recommend]['item_id'])\nuser_not_rated_items = [(user_id_to_recommend, item, 0) for item in items_to_recommend]\nuser_recommendations = knn_model.test(user_not_rated_items)\n\n# Get top N recommendations\ntop_n = 3\nuser_recommendations.sort(key=lambda x: x.est, reverse=True)\ntop_recommendations = user_recommendations[:top_n]\n\n# Display top recommendations\nprint(f\"Top {top_n} recommendations for User {user_id_to_recommend}:\")\nfor recommendation in top_recommendations:\n    print(f\"Item {recommendation.iid} with predicted rating: {recommendation.est}\")\n","metadata":{"execution":{"iopub.status.busy":"2023-12-15T04:47:16.632073Z","iopub.execute_input":"2023-12-15T04:47:16.632750Z","iopub.status.idle":"2023-12-15T04:47:16.690587Z","shell.execute_reply.started":"2023-12-15T04:47:16.632722Z","shell.execute_reply":"2023-12-15T04:47:16.689727Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### User-based Collaborative Filtering (UBCF)\n- user_item_matrix is a matrix where each row corresponds to a user, and each column corresponds to an item. Zeros in the matrix represent missing ratings. The cosine_similarity to compute the cosine similarity between users based on their ratings.\n- predict_user_based function predicts ratings for the missing entries in the user_item_matrix based on the ratings of similar users. The predictions are stored in the predicted_ratings matrix.","metadata":{}},{"cell_type":"code","source":"import numpy as np\nfrom sklearn.metrics.pairwise import cosine_similarity\n\n# Sample user-item matrix (rows are users, columns are items)\n# Each row represents a user's ratings for different items\nuser_item_matrix = np.array([\n    [5, 4, 0, 5, 0],\n    [4, 0, 0, 4, 5],\n    [0, 5, 4, 0, 4],\n    [5, 0, 3, 4, 0],\n    [0, 4, 5, 0, 3],\n])\n\n# Compute cosine similarity between users\nuser_similarity = cosine_similarity(user_item_matrix)\n\ndef predict_user_based(user_item_matrix, user_similarity):\n    # Predict ratings for items the user hasn't rated\n    num_users, num_items = user_item_matrix.shape\n    predicted_ratings = np.zeros((num_users, num_items))\n\n    for i in range(num_users):\n        for j in range(num_items):\n            if user_item_matrix[i, j] == 0:  # If the user hasn't rated the item\n                # Find users who have rated the same item\n                rated_indices = user_item_matrix[:, j].nonzero()[0]\n\n                # Calculate the weighted sum of ratings from similar users\n                weighted_sum = sum(\n                    user_similarity[i, k] * user_item_matrix[k, j]\n                    for k in rated_indices\n                )\n\n                # Normalize by the sum of similarity scores\n                normalization = sum(user_similarity[i, k] for k in rated_indices)\n\n                # Make the prediction\n                if normalization != 0:\n                    predicted_ratings[i, j] = weighted_sum / normalization\n\n    return predicted_ratings\n\n# Get the predicted ratings\npredicted_ratings = predict_user_based(user_item_matrix, user_similarity)\n\nprint(\"User-Item Matrix:\")\nprint(user_item_matrix)\nprint(\"\\nUser Similarity Matrix:\")\nprint(user_similarity)\nprint(\"\\nPredicted Ratings:\")\nprint(predicted_ratings)\n","metadata":{"execution":{"iopub.status.busy":"2023-12-15T04:47:16.692709Z","iopub.execute_input":"2023-12-15T04:47:16.693102Z","iopub.status.idle":"2023-12-15T04:47:16.706792Z","shell.execute_reply.started":"2023-12-15T04:47:16.693076Z","shell.execute_reply":"2023-12-15T04:47:16.705786Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- Item-based Collaborative Filtering (IBCF) in Python involves calculating the similarity between items based on user preferences and using this information to make recommendations. \n\n    - calculate cosine similarity between items using scikit-learn's cosine_similarity function.       - get_item_recommendations function takes a user's ratings and the item similarity matrix to provide item recommendations.","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nfrom sklearn.metrics.pairwise import cosine_similarity\n\n# Sample user-item ratings data\ndata = {\n    'User1': {'Item1': 4, 'Item2': 5, 'Item3': 3, 'Item4': 2},\n    'User2': {'Item1': 5, 'Item2': 2, 'Item3': 4, 'Item4': 3},\n    'User3': {'Item1': 2, 'Item2': 4, 'Item3': 1, 'Item4': 5},\n    'User4': {'Item1': 4, 'Item2': 3, 'Item3': 5, 'Item4': 1},\n}\n\n# Convert the data to a DataFrame\ndf = pd.DataFrame(data)\n\n# Transpose DataFrame to have items as rows and users as columns\ndf = df.transpose()\n\n# Calculate item-item similarity using cosine similarity\nitem_similarity = cosine_similarity(df.fillna(0))\n\n# Convert the similarity matrix to a DataFrame for better readability\nitem_similarity_df = pd.DataFrame(item_similarity, index=df.index, columns=df.index)\n\n# Get item recommendations based on user preferences\ndef get_item_recommendations(user_ratings, item_similarity):\n    # Weighted sum of item ratings based on similarity\n    weighted_sum = (user_ratings * item_similarity).sum(axis=0)\n    \n    # Normalize by sum of similarities\n    normalized_scores = weighted_sum / item_similarity.sum(axis=0)\n    \n    # Sort items by the normalized scores in descending order\n    recommendations = normalized_scores.sort_values(ascending=False)\n    \n    return recommendations\n\n# Example: Get item recommendations for 'User1'\nuser1_ratings = df.loc['User1'].fillna(0)\nuser1_recommendations = get_item_recommendations(user1_ratings, item_similarity_df)\n\nprint(\"Item Recommendations for User1:\")\nprint(user1_recommendations)\n","metadata":{"execution":{"iopub.status.busy":"2023-12-15T04:47:16.707969Z","iopub.execute_input":"2023-12-15T04:47:16.708301Z","iopub.status.idle":"2023-12-15T04:47:16.735638Z","shell.execute_reply.started":"2023-12-15T04:47:16.708273Z","shell.execute_reply":"2023-12-15T04:47:16.734798Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Item-based Collaborative Filtering (IBCF) using surprise\nfrom surprise import Dataset, Reader\nfrom surprise.model_selection import train_test_split\nfrom surprise import KNNBasic\nfrom surprise import accuracy\n\n# user-item ratings data\ndata = {\n    'User1': {'Item1': 4, 'Item2': 5, 'Item3': 3, 'Item4': 2},\n    'User2': {'Item1': 5, 'Item2': 2, 'Item3': 4, 'Item4': 3},\n    'User3': {'Item1': 2, 'Item2': 4, 'Item3': 1, 'Item4': 5},\n    'User4': {'Item1': 4, 'Item2': 3, 'Item3': 5, 'Item4': 1},\n}\n\n# data to DataFrame\nimport pandas as pd\ndf = pd.DataFrame([(user, item, rating) for user, items in data.items() for item, rating in items.items()], columns=['user', 'item', 'rating'])\n\n# Surprise Reader\nreader = Reader(rating_scale=(1, 5))\n\n# Load data into a Surprise Dataset\nsurprise_data = Dataset.load_from_df(df, reader)\n\n# Split data into training and testing sets\ntrainset, testset = train_test_split(surprise_data, test_size=0.2, random_state=42)\n\n# Use Item-based Collaborative Filtering with k-NN algorithm\nsim_options = {\n    'name': 'cosine',\n    'user_based': False  # Item-based Collaborative Filtering\n}\n\nmodel = KNNBasic(sim_options=sim_options)\nmodel.fit(trainset)\n\n# Make predictions on the test set\npredictions = model.test(testset)\n\n# Evaluate the model\naccuracy.rmse(predictions)\n\n# Function to get item recommendations for a user\ndef get_item_recommendations(user_id, model, n=5):\n    # Get all items not rated by the user\n    items_not_rated = [item for item in df['item'].unique() if item not in data[user_id]]\n\n    # Predict ratings for items not rated by the user\n    predictions = [model.predict(user_id, item) for item in items_not_rated]\n\n    # Sort predictions by estimated rating in descending order\n    sorted_predictions = sorted(predictions, key=lambda x: x.est, reverse=True)\n\n    # Return the top n recommended items\n    top_n_recommendations = [(prediction.iid, prediction.est) for prediction in sorted_predictions[:n]]\n    return top_n_recommendations\n\n# Example: Get item recommendations for 'User3'\nuser1_recommendations = get_item_recommendations('User1', model)\nprint(\"Item Recommendations for User1:\")\nprint(user1_recommendations)\n","metadata":{"execution":{"iopub.status.busy":"2023-12-15T04:47:16.737328Z","iopub.execute_input":"2023-12-15T04:47:16.737712Z","iopub.status.idle":"2023-12-15T04:47:16.759003Z","shell.execute_reply.started":"2023-12-15T04:47:16.737677Z","shell.execute_reply":"2023-12-15T04:47:16.758173Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n### content-based recommender system \n- TF-IDF (Term Frequency-Inverse Document Frequency) vectorizer to convert movie descriptions into numerical vectors. We compute the cosine similarity between these vectors to measure the similarity between movies. Finally, the get_recommendations function takes a movie title as input and returns recommended movies based on similarity.","metadata":{}},{"cell_type":"code","source":"# content-based recommender system \nimport pandas as pd\nfrom sklearn.feature_extraction.text import TfidfVectorizer\nfrom sklearn.metrics.pairwise import linear_kernel\n\n# Sample movie data\nmovies_data = {\n    'Title': ['Movie 1', 'Movie 2', 'Movie 3', 'Movie 4'],\n    'Genre': ['Action', 'Comedy', 'Action', 'Drama'],\n    'Description': ['Action-packed movie with thrilling scenes.',\n                    'A hilarious comedy that will make you laugh.',\n                    'Another action movie with intense sequences.',\n                    'Drama film with emotional storytelling.']\n}\n\nmovies_df = pd.DataFrame(movies_data)\n\n# TF-IDF Vectorization\ntfidf_vectorizer = TfidfVectorizer(stop_words='english')\ntfidf_matrix = tfidf_vectorizer.fit_transform(movies_df['Description'])\n\n# Compute cosine similarity\ncosine_sim = linear_kernel(tfidf_matrix, tfidf_matrix)\n\n# get movie recommendations\ndef get_recommendations(title, cosine_sim=cosine_sim):\n    idx = movies_df.index[movies_df['Title'] == title].tolist()[0]\n    sim_scores = list(enumerate(cosine_sim[idx]))\n    sim_scores = sorted(sim_scores, key=lambda x: x[1], reverse=True)\n    sim_scores = sim_scores[1:3]  # Get the top 2 most similar movies (excluding itself)\n\n    movie_indices = [i[0] for i in sim_scores]\n    return movies_df['Title'].iloc[movie_indices]\n\n# Example usage\nmovie_title = 'Movie 1'\nrecommendations = get_recommendations(movie_title)\n\nprint(f\"Recommendations for {movie_title}:\")\nprint(recommendations)\n","metadata":{"execution":{"iopub.status.busy":"2023-12-15T04:47:16.760344Z","iopub.execute_input":"2023-12-15T04:47:16.761094Z","iopub.status.idle":"2023-12-15T04:47:16.798630Z","shell.execute_reply.started":"2023-12-15T04:47:16.761062Z","shell.execute_reply":"2023-12-15T04:47:16.797764Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# knowledge-based recommender system\nclass Movie:\n    def __init__(self, title, genre, release_year):\n        self.title = title\n        self.genre = genre\n        self.release_year = release_year\n\nclass KnowledgeBasedRecommender:\n    def __init__(self, movie_database):\n        self.movie_database = movie_database\n\n    def recommend_movies(self, user_preferences):\n        recommended_movies = []\n\n        for movie in self.movie_database:\n            # Check if the movie's genre matches the user's preference\n            if movie.genre in user_preferences['genres']:\n                # Check if the movie's release year is within the user's preferred range\n                if user_preferences['min_year'] <= movie.release_year <= user_preferences['max_year']:\n                    recommended_movies.append(movie.title)\n\n        return recommended_movies\n\n# Sample movie database\nmovies = [\n    Movie(\"The Shawshank Redemption\", \"Drama\", 1994),\n    Movie(\"The Godfather\", \"Crime\", 1972),\n    Movie(\"The Dark Knight\", \"Action\", 2008),\n    # Add more movies to the database\n]\n\n# Sample user preferences\nuser_preferences = {\n    'genres': ['Drama', 'Crime'],\n    'min_year': 1990,\n    'max_year': 2022\n}\n\n# Create a knowledge-based recommender system\nrecommender_system = KnowledgeBasedRecommender(movies)\n\n# Get recommended movies based on user preferences\nrecommendations = recommender_system.recommend_movies(user_preferences)\n\n# Print recommended movies\nprint(\"Recommended Movies:\")\nfor movie in recommendations:\n    print(movie)\n","metadata":{"execution":{"iopub.status.busy":"2023-12-15T04:47:16.799747Z","iopub.execute_input":"2023-12-15T04:47:16.800002Z","iopub.status.idle":"2023-12-15T04:47:16.809434Z","shell.execute_reply.started":"2023-12-15T04:47:16.799980Z","shell.execute_reply":"2023-12-15T04:47:16.808566Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Neural Collaborative Filtering (NCF) recommendation system ","metadata":{}},{"cell_type":"code","source":"from sklearn.model_selection import train_test_split\nfrom tensorflow.keras.models import Model\nfrom tensorflow.keras.layers import Input, Embedding, Flatten, Concatenate, Dense\n\n# Create a synthetic dataset\nnp.random.seed(42)\ndata = pd.DataFrame({\n    'user_id': np.random.randint(1, 100, 1000),\n    'item_id': np.random.randint(1, 50, 1000),\n    'rating': np.random.randint(1, 6, 1000)\n})\n","metadata":{"execution":{"iopub.status.busy":"2023-12-15T04:47:16.810748Z","iopub.execute_input":"2023-12-15T04:47:16.810990Z","iopub.status.idle":"2023-12-15T04:47:28.143264Z","shell.execute_reply.started":"2023-12-15T04:47:16.810969Z","shell.execute_reply":"2023-12-15T04:47:28.142312Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Split the data into training and testing sets\ntrain_data, test_data = train_test_split(data, test_size=0.2, random_state=42)\n\n# NCF model\ndef create_ncf_model(num_users, num_items, embedding_size=50, hidden_units=(100, 50, 25)):\n    user_input = Input(shape=(1,), name='user_input')\n    item_input = Input(shape=(1,), name='item_input')\n\n    user_embedding = Embedding(input_dim=num_users, output_dim=embedding_size, input_length=1)(user_input)\n    item_embedding = Embedding(input_dim=num_items, output_dim=embedding_size, input_length=1)(item_input)\n\n    user_flatten = Flatten()(user_embedding)\n    item_flatten = Flatten()(item_embedding)\n\n    concat = Concatenate()([user_flatten, item_flatten])\n\n    for units in hidden_units:\n        concat = Dense(units, activation='relu')(concat)\n\n    output = Dense(1, activation='linear')(concat)\n\n    model = Model(inputs=[user_input, item_input], outputs=output)\n    model.compile(optimizer='adam', loss='mean_squared_error')\n    return model","metadata":{"execution":{"iopub.status.busy":"2023-12-15T04:47:28.146356Z","iopub.execute_input":"2023-12-15T04:47:28.146869Z","iopub.status.idle":"2023-12-15T04:47:28.156751Z","shell.execute_reply.started":"2023-12-15T04:47:28.146843Z","shell.execute_reply":"2023-12-15T04:47:28.155868Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Get the number of unique users and items\nnum_users = data['user_id'].nunique()\nnum_items = data['item_id'].nunique()\n\n# Create and train the model\nmodel = create_ncf_model(num_users, num_items)\nmodel.fit([train_data['user_id'], train_data['item_id']], train_data['rating'], epochs=10, batch_size=32, validation_split=0.2)\n","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-12-15T04:47:28.157784Z","iopub.execute_input":"2023-12-15T04:47:28.158073Z","iopub.status.idle":"2023-12-15T04:47:42.316572Z","shell.execute_reply.started":"2023-12-15T04:47:28.158043Z","shell.execute_reply":"2023-12-15T04:47:42.315434Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Evaluate model on test set\ntest_loss = model.evaluate([test_data['user_id'], test_data['item_id']], test_data['rating'])\nprint(f'Test Loss: {test_loss}')","metadata":{"execution":{"iopub.status.busy":"2023-12-15T04:47:42.317917Z","iopub.execute_input":"2023-12-15T04:47:42.318286Z","iopub.status.idle":"2023-12-15T04:47:42.581178Z","shell.execute_reply.started":"2023-12-15T04:47:42.318254Z","shell.execute_reply":"2023-12-15T04:47:42.580256Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Word2Vec-based recommender system","metadata":{}},{"cell_type":"code","source":"!pip install gensim","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-12-15T04:47:42.582954Z","iopub.execute_input":"2023-12-15T04:47:42.583346Z","iopub.status.idle":"2023-12-15T04:47:55.009692Z","shell.execute_reply.started":"2023-12-15T04:47:42.583312Z","shell.execute_reply":"2023-12-15T04:47:55.008683Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Import necessary libraries\nfrom gensim.models import Word2Vec\nfrom nltk.tokenize import word_tokenize\nimport nltk\nnltk.download('punkt')  # Download the punkt tokenizer\n\n# Sample dataset\ncorpus = [\n    \"I like natural language processing\",\n    \"Word embeddings capture semantic meaning\",\n    \"Recommender systems are useful\",\n    \"Natural language understanding is challenging\",\n    \"Word2Vec is a popular embedding technique\",\n]\n\n# Tokenize the sentences\ntokenized_corpus = [word_tokenize(sentence.lower()) for sentence in corpus]","metadata":{"execution":{"iopub.status.busy":"2023-12-15T04:47:55.011364Z","iopub.execute_input":"2023-12-15T04:47:55.012242Z","iopub.status.idle":"2023-12-15T04:48:11.274478Z","shell.execute_reply.started":"2023-12-15T04:47:55.012203Z","shell.execute_reply":"2023-12-15T04:48:11.273551Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Train Word2Vec model\nmodel = Word2Vec(sentences=tokenized_corpus, vector_size=100, window=5, min_count=1, workers=4)\n\n# Save model\nmodel.save(\"word2vec_model\")\n\n# finding similar words using the trained model\nsimilar_words = model.wv.most_similar(\"natural\", topn=3)\n\nprint(\"Similar words to 'natural':\")\nfor word, similarity in similar_words:\n    print(f\"{word}: {similarity}\")","metadata":{"execution":{"iopub.status.busy":"2023-12-15T04:48:11.275849Z","iopub.execute_input":"2023-12-15T04:48:11.276238Z","iopub.status.idle":"2023-12-15T04:48:11.296393Z","shell.execute_reply.started":"2023-12-15T04:48:11.276205Z","shell.execute_reply":"2023-12-15T04:48:11.295440Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# simple recommender function word2vec\ndef recommend_similar_words(word, topn=3):\n    similar_words = model.wv.most_similar(word, topn=topn)\n    return similar_words\n\n# recommender function\nrecommendations = recommend_similar_words(\"language\")\nprint(\"\\nRecommendations for 'language':\")\nfor word, similarity in recommendations:\n    print(f\"{word}: {similarity}\")","metadata":{"execution":{"iopub.status.busy":"2023-12-15T04:48:11.297505Z","iopub.execute_input":"2023-12-15T04:48:11.297787Z","iopub.status.idle":"2023-12-15T04:48:11.304155Z","shell.execute_reply.started":"2023-12-15T04:48:11.297755Z","shell.execute_reply":"2023-12-15T04:48:11.303236Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# OTTO – Multi-Objective Recommender System\n\n- Training data is huge and has 12,899,778 sessions (216,716,095 actions such as clikcs or cart-added), therefore try to download 100k sessions only\n- word2vec model - first read 100k sessions from train.jsonl with values like session , events , aid(article id) , ts(timestamp of event) , type(event type) and use word2vec model predict the aid values for each session type thats occur after the last timestamp ts in the test session\n","metadata":{}},{"cell_type":"code","source":"# import json\n# import random\n# # number of sessions to download\n# num_sessions_to_download = 100000\n\n# # Load and preprocess a subset of the data\n# sessions = []\n# with open('/kaggle/input/otto-recommender-system/train.jsonl', 'r') as file:\n#     for line in file:\n#         entry = json.loads(line)\n#         session_events = [event['aid'] for event in entry['events'] if event['type'] in ['click', 'cart', 'order']]\n#         sessions.append(session_events)","metadata":{"execution":{"iopub.status.busy":"2023-12-15T04:48:11.305227Z","iopub.execute_input":"2023-12-15T04:48:11.305477Z","iopub.status.idle":"2023-12-15T04:48:11.314613Z","shell.execute_reply.started":"2023-12-15T04:48:11.305455Z","shell.execute_reply":"2023-12-15T04:48:11.313747Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# import json\n\n# # Specify the path to your train.jsonl file\n# file_path = '/kaggle/input/otto-recommender-system/train.jsonl'\n\n# # Specify the number of sessions to read (e.g., 100,000)\n# num_sessions_to_read = 100000\n\n# # List to store the extracted sessions\n# sessions = []\n\n# # Read the JSONL file line by line\n# with open(file_path, 'r', encoding='utf-8') as file:\n#     for line_number, line in enumerate(file):\n#         # Load the JSON data from each line\n#         data = json.loads(line)\n\n#         # Assuming each line represents a session, you can append it to the list\n#         sessions.append(data)\n\n#         # Break the loop if the desired number of sessions is reached\n#         if len(sessions) == num_sessions_to_read:\n#             break\n","metadata":{"execution":{"iopub.status.busy":"2023-12-15T04:48:11.315801Z","iopub.execute_input":"2023-12-15T04:48:11.316157Z","iopub.status.idle":"2023-12-15T04:48:11.325354Z","shell.execute_reply.started":"2023-12-15T04:48:11.316123Z","shell.execute_reply":"2023-12-15T04:48:11.324546Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# from gensim.models import Word2Vec\n# import json\n# import random\n\n# # Define the number of sessions to download\n# num_sessions_to_download = 100000\n\n# # Load and preprocess a subset of the data\n# sessions = []\n# with open('/kaggle/input/otto-recommender-system/train.jsonl', 'r') as file:\n#     # Read all lines into a list\n#     all_lines = file.readlines()\n\n#     # Randomly sample the specified number of sessions\n#     sampled_lines = random.sample(all_lines, num_sessions_to_download)\n\n#     # Parse and preprocess sampled sessions\n#     for line in sampled_lines:\n#         entry = json.loads(line)\n#         session_events = [event['aid'] for event in entry['events'] if event['type'] in ['click', 'cart', 'order']]\n#         sessions.append(session_events)","metadata":{"execution":{"iopub.status.busy":"2023-12-15T04:48:11.326350Z","iopub.execute_input":"2023-12-15T04:48:11.326599Z","iopub.status.idle":"2023-12-15T04:48:11.339752Z","shell.execute_reply.started":"2023-12-15T04:48:11.326577Z","shell.execute_reply":"2023-12-15T04:48:11.339018Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Task recommendation system","metadata":{}},{"cell_type":"code","source":"# Basic task recommendation system\nimport pandas as pd\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.ensemble import RandomForestClassifier\nfrom sklearn.metrics import accuracy_score\nfrom sklearn.preprocessing import LabelEncoder\nfrom sklearn.feature_extraction.text import TfidfVectorizer\n\n# In areal-world scenario, data would come from your virtual workspace application\ndata = {\n    'Task': ['Task1', 'Task2', 'Task3', 'Task4', 'Task5'],\n    'Description': ['Do something important', 'Review document', 'Attend meeting', 'Submit report', 'Code review'],\n    'Category': ['Important', 'Review', 'Meeting', 'Report', 'Code']\n}\n\ndf = pd.DataFrame(data)\n\n# Encode task categories\nle = LabelEncoder()\ndf['CategoryEncoded'] = le.fit_transform(df['Category'])\n\nX_train, X_test, y_train, y_test = train_test_split(df['Description'], df['CategoryEncoded'], test_size=0.2, random_state=42)\n\n# Convert task descriptions to TF-IDF features\ntfidf_vectorizer = TfidfVectorizer(stop_words='english')\nX_train_tfidf = tfidf_vectorizer.fit_transform(X_train)\nX_test_tfidf = tfidf_vectorizer.transform(X_test)\n\n# RandomForestClassifier as a simple task category prediction model\nmodel = RandomForestClassifier(n_estimators=100, random_state=42)\nmodel.fit(X_train_tfidf, y_train)\n\npredictions = model.predict(X_test_tfidf)\n\naccuracy = accuracy_score(y_test, predictions)\nprint(f'Model Accuracy: {accuracy}')\n\n# Use trained model to predict task categories for new tasks\nnew_task_description = ['New task description']\nnew_task_tfidf = tfidf_vectorizer.transform(new_task_description)\npredicted_category = le.inverse_transform(model.predict(new_task_tfidf))\n\nprint(f'Predicted Category for New Task: {predicted_category}')\n","metadata":{"execution":{"iopub.status.busy":"2023-12-15T04:48:11.342433Z","iopub.execute_input":"2023-12-15T04:48:11.342700Z","iopub.status.idle":"2023-12-15T04:48:11.686039Z","shell.execute_reply.started":"2023-12-15T04:48:11.342677Z","shell.execute_reply":"2023-12-15T04:48:11.685093Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Basic task recommendation system using SVD algorithm and collaborative filtering","metadata":{}},{"cell_type":"code","source":"\nfrom surprise import Dataset, Reader, SVD\nfrom surprise.model_selection import train_test_split\nfrom surprise import accuracy\n\n# Sample data (user, task, rating)\ndata = {\n    'user': ['User1', 'User1', 'User2', 'User2', 'User3', 'User3', 'User4', 'User4'],\n    'task': ['TaskA', 'TaskB', 'TaskA', 'TaskC', 'TaskB', 'TaskC', 'TaskD', 'TaskE'],\n    'rating': [5, 4, 3, 2, 4, 1, 5, 3],\n}\n\n# DataFrame from data\nimport pandas as pd\ndf = pd.DataFrame(data)\n\n# Define Reader object\nreader = Reader(rating_scale=(1, 5))\n\n# Load dataset\ndataset = Dataset.load_from_df(df[['user', 'task', 'rating']], reader)\n\ntrainset, testset = train_test_split(dataset, test_size=0.2, random_state=42)\n\n# Build recommendation model (SVD algorithm)\nmodel = SVD()\nmodel.fit(trainset)\n\npredictions = model.test(testset)\n\n# Evaluate model\naccuracy.rmse(predictions)\n\n# Function to get task recommendations for a given user\ndef get_top_n_recommendations(predictions, user_id, n=5):\n    top_n = {}\n    for uid, iid, true_r, est, _ in predictions:\n        if uid not in top_n:\n            top_n[uid] = []\n        top_n[uid].append((iid, est))\n\n    # Sort recommendations for specified user\n    if user_id in top_n:\n        top_n[user_id].sort(key=lambda x: x[1], reverse=True)\n        return top_n[user_id][:n]\n    else:\n        return []\n\n# recommendations for specific user\nuser_id = 'User1'\nuser_recommendations = get_top_n_recommendations(predictions, user_id, n=3)\n\nif user_recommendations:\n    print(f\"Top 3 task recommendations for {user_id}: {user_recommendations}\")\nelse:\n    print(f\"No recommendations available for {user_id}\")\n","metadata":{"execution":{"iopub.status.busy":"2023-12-15T05:01:42.524318Z","iopub.execute_input":"2023-12-15T05:01:42.524687Z","iopub.status.idle":"2023-12-15T05:01:42.541652Z","shell.execute_reply.started":"2023-12-15T05:01:42.524658Z","shell.execute_reply":"2023-12-15T05:01:42.540402Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Task management recommendation system\n- TF-IDF vectorizer and cosine similarity to recommend tasks based on their descriptions","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics.pairwise import cosine_similarity\nfrom sklearn.feature_extraction.text import TfidfVectorizer\n\n# Sample data \ndata = {\n    'TaskID': [1, 2, 3, 4, 5],\n    'TaskDescription': [\n        'Complete project proposal',\n        'Review client feedback',\n        'Test new software release',\n        'Prepare presentation',\n        'Attend team meeting'\n    ],\n    'AssignedTo': ['John', 'Alice', 'Bob', 'Alice', 'John']\n}\n\ntasks_df = pd.DataFrame(data)\n\ntrain_data, test_data = train_test_split(tasks_df, test_size=0.2, random_state=42)\n\n# Vectorize task descriptions using TF-IDF\nvectorizer = TfidfVectorizer(stop_words='english')\ntfidf_matrix = vectorizer.fit_transform(train_data['TaskDescription'])\n\n# Compute cosine similarity between tasks based on TF-IDF vectors\ncosine_sim = cosine_similarity(tfidf_matrix, tfidf_matrix)\n\n# Function to get task recommendations\ndef get_task_recommendations(task_id, cosine_sim=cosine_sim, df=train_data):\n    task_index = df[df['TaskID'] == task_id].index[0]\n    sim_scores = list(enumerate(cosine_sim[task_index]))\n    sim_scores = sorted(sim_scores, key=lambda x: x[1], reverse=True)\n    sim_scores = sim_scores[1:4]  # Get top 3 similar tasks (excluding itself)\n\n    task_indices = [i[0] for i in sim_scores]\n    return df['TaskID'].iloc[task_indices]\n\n# Get task recommendations for TaskID 1\nrecommendations = get_task_recommendations(1)\nprint(\"Task Recommendations:\", recommendations.values)\n","metadata":{"execution":{"iopub.status.busy":"2023-12-15T05:05:03.919338Z","iopub.execute_input":"2023-12-15T05:05:03.919811Z","iopub.status.idle":"2023-12-15T05:05:03.940085Z","shell.execute_reply.started":"2023-12-15T05:05:03.919780Z","shell.execute_reply":"2023-12-15T05:05:03.938972Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas as pd\nfrom sklearn.metrics.pairwise import cosine_similarity\n\nmovies_df = pd.read_csv('/kaggle/input/netflix-movie-rating-dataset/Netflix_Dataset_Movie.csv')\nratings_df = pd.read_csv('/kaggle/input/netflix-movie-rating-dataset/Netflix_Dataset_Rating.csv')\n\nmerged_df = pd.merge(ratings_df, movies_df, on='Movie_ID')\n\n# Step 2: Data Exploration\nprint(merged_df.head())\nprint(merged_df.describe())\n\n# Step 3: User-Item Matrix\nuser_item_matrix = merged_df.pivot_table(index='User_ID', columns='Name', values='Rating')\n\n# Step 4: Collaborative Filtering\ndef get_movie_recommendations(user_item_matrix, user_id, num_recommendations=5):\n    user_ratings = user_item_matrix.loc[user_id].dropna()\n    similar_to_user = user_item_matrix.corrwith(user_ratings)\n    similar_to_user = similar_to_user.dropna().sort_values(ascending=False)\n    \n    recommendations = pd.DataFrame(similar_to_user, columns=['correlation'])\n    \n    if 'Rating' in user_item_matrix.columns:\n        user_has_rated = user_ratings.index\n        unrated_movies = user_item_matrix.columns.difference(user_has_rated)\n        recommendations = recommendations.join(user_item_matrix[unrated_movies], how='left')\n    else:\n        recommendations['Rating'] = None\n    \n    recommendations = recommendations[recommendations['Rating'].isnull()]\n    recommendations = recommendations.drop(columns=['Rating'])\n    \n    return recommendations.head(num_recommendations)\n\n\n# Step 5: Recommendation Generation\nuser_id = 712664  \nrecommendations = get_movie_recommendations(user_item_matrix, user_id)\nprint(\"Recommended Movies for User\", user_id)\nprint(recommendations)\n","metadata":{"execution":{"iopub.status.busy":"2024-02-17T01:21:12.286798Z","iopub.execute_input":"2024-02-17T01:21:12.287190Z","iopub.status.idle":"2024-02-17T01:22:38.353684Z","shell.execute_reply.started":"2024-02-17T01:21:12.287160Z","shell.execute_reply":"2024-02-17T01:22:38.352566Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# import joblib\n\n# movies_df = pd.read_csv('/kaggle/input/netflix-movie-rating-dataset/Netflix_Dataset_Movie.csv')\n# ratings_df = pd.read_csv('/kaggle/input/netflix-movie-rating-dataset/Netflix_Dataset_Rating.csv')\n\n# merged_df = pd.merge(ratings_df, movies_df, on='Movie_ID')\n\n# user_item_matrix = merged_df.pivot_table(index='User_ID', columns='Name', values='Rating')\n\n# def get_movie_recommendations(user_id, user_item_matrix):\n#     user_ratings = user_item_matrix.loc[user_id].dropna()\n#     similar_to_user = user_item_matrix.corrwith(user_ratings)\n#     similar_to_user = similar_to_user.dropna().sort_values(ascending=False)\n    \n#     recommendations = pd.DataFrame(similar_to_user, columns=['correlation'])\n    \n#     if 'Rating' in user_item_matrix.columns:\n#         user_has_rated = user_ratings.index\n#         unrated_movies = user_item_matrix.columns.difference(user_has_rated)\n#         recommendations = recommendations.join(user_item_matrix[unrated_movies], how='left')\n#     else:\n#         recommendations['Rating'] = None\n    \n#     recommendations = recommendations[recommendations['Rating'].isnull()]\n#     recommendations = recommendations.drop(columns=['Rating'])\n    \n#     return recommendations.head(num_recommendations)\n\n# # number of recommendations to generate\n# num_recommendations = 5\n\n# # recommendations for all users in parallel\n# all_user_ids = user_item_matrix.index.tolist()\n\n# # function execution in parallel\n# def generate_recommendations(user_id):\n#     return get_movie_recommendations(user_id, user_item_matrix)\n\n# # Parallelized computation\n# recommendations_list = joblib.Parallel(n_jobs=-1, verbose=10)(\n#     joblib.delayed(generate_recommendations)(user_id) for user_id in all_user_ids\n# )\n\n# # Concatenate all recommendations \n# all_recommendations = pd.concat(recommendations_list, ignore_index=True)\n\n\n# all_recommendations.to_csv('submission.csv', index=False)\n","metadata":{"execution":{"iopub.status.busy":"2024-02-17T04:08:21.173560Z","iopub.execute_input":"2024-02-17T04:08:21.173931Z","iopub.status.idle":"2024-02-17T04:08:21.179951Z","shell.execute_reply.started":"2024-02-17T04:08:21.173902Z","shell.execute_reply":"2024-02-17T04:08:21.178969Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport joblib\n\nmovies_df = pd.read_csv('/kaggle/input/netflix-movie-rating-dataset/Netflix_Dataset_Movie.csv')\nratings_df = pd.read_csv('/kaggle/input/netflix-movie-rating-dataset/Netflix_Dataset_Rating.csv')\n\nmerged_df = pd.merge(ratings_df, movies_df, on='Movie_ID')\n\nuser_item_matrix = merged_df.pivot_table(index='User_ID', columns='Name', values='Rating')\n\nnum_recommendations = 5\n\n# recommendations for a single user\ndef get_movie_recommendations(user_id):\n    user_ratings = user_item_matrix.loc[user_id].dropna()\n    similar_to_user = user_item_matrix.corrwith(user_ratings)\n    similar_to_user = similar_to_user.dropna().sort_values(ascending=False)\n    \n    # unrated movies\n    unrated_movies = user_item_matrix.columns.difference(user_ratings.index)\n    \n    # recommendations calculation\n    recommendations = similar_to_user.to_frame(name='correlation').join(user_item_matrix[unrated_movies])\n    recommendations = recommendations[recommendations.iloc[:, 1:].isna().all(axis=1)].head(num_recommendations)\n    \n    return recommendations\n\n# subset of user IDs for parallelization\nsubset_user_ids = user_item_matrix.index[:100]\n\n# Parallelize recommendation generation\nrecommendations_list = joblib.Parallel(n_jobs=-1, verbose=10)(\n    joblib.delayed(get_movie_recommendations)(user_id) for user_id in subset_user_ids\n)\n\nall_recommendations = pd.concat(recommendations_list, ignore_index=True)\n\nall_recommendations.to_csv('submission.csv', index=False)\n","metadata":{"execution":{"iopub.status.busy":"2024-02-17T04:30:58.985727Z","iopub.execute_input":"2024-02-17T04:30:58.986974Z","iopub.status.idle":"2024-02-17T05:08:03.936372Z","shell.execute_reply.started":"2024-02-17T04:30:58.986892Z","shell.execute_reply":"2024-02-17T05:08:03.934925Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport seaborn as sns\n\nprint(all_recommendations.head())\nall_recommendations.info()\n\n# Correlation Score Distribution\nprint(all_recommendations['correlation'].describe())\nprint(all_recommendations['correlation'].unique())\n\nall_recommendations.dropna(subset=['correlation'], inplace=True)\n\nplt.figure(figsize=(10, 6))\nsns.histplot(all_recommendations['correlation'], bins=20, kde=True)\nplt.title('Distribution of Correlation Scores for Recommendations')\nplt.xlabel('Correlation Score')\nplt.ylabel('Frequency')\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2024-02-17T05:14:32.236273Z","iopub.execute_input":"2024-02-17T05:14:32.236836Z","iopub.status.idle":"2024-02-17T05:14:32.732164Z","shell.execute_reply.started":"2024-02-17T05:14:32.236791Z","shell.execute_reply":"2024-02-17T05:14:32.730955Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}