{
  "id": 365917,
  "title": "RePlay - opensource RecSys constructor",
  "url": "/competitions/otto-recommender-system/discussion/365917",
  "author_name": "Alexander Ryzhkov",
  "post_date": "2022-11-13T20:56:06.587000",
  "votes": 18,
  "comment_count": 0,
  "views": 0,
  "content": "<p><a href=\"https://github.com/sb-ai-lab/RePlay\" target=\"_blank\">RePlay</a> is a library providing tools for all stages of creating a recommendation system, from data preprocessing to model evaluation and comparison. RePlay uses PySpark to handle big data.</p>\n<p>You can:</p>\n<ul>\n<li>Filter and split data</li>\n<li>Train models</li>\n<li>Optimize hyper parameters</li>\n<li>Evaluate predictions with metrics</li>\n<li>Combine predictions from different models</li>\n<li>Create a two-level model</li>\n</ul>\n<p>Documentation is available <a href=\"https://sb-ai-lab.github.io/RePlay/\" target=\"_blank\">here</a>.</p>\n<h2>Installation</h2>\n<p>Use Linux machine with Python 3.7-3.9, Java 8+ and C++ compiler.</p>\n<pre><code>pip install replay-rec\n</code></pre>\n<p>To get the latest development version or RePlay, <a href=\"https://sb-ai-lab.github.io/RePlay/pages/installation.html#development\" target=\"_blank\">install it from the GitHub repository</a>. It is preferable to use a virtual environment for your installation.</p>\n<h2>Resources</h2>\n<h3>Videos and papers</h3>\n<ul>\n<li><p><strong>Video guides</strong>:</p>\n<ul>\n<li><a href=\"https://www.youtube.com/watch?v=ejQZKGAG0xs\" target=\"_blank\">Replay for offline recommendations, AI Journey 2021</a></li></ul></li>\n<li><p><strong>Research papers</strong>:</p>\n<ul>\n<li>(CORE A RecSys 2021) Yan-Martin Tamm, Rinchin Damdinov, Alexey Vasilev <a href=\"https://dl.acm.org/doi/10.1145/3460231.3478848\" target=\"_blank\">Quality Metrics in Recommender Systems: Do We Calculate Metrics Consistently?</a></li></ul></li>\n</ul>\n<h3>Usage examples</h3>\n<ol>\n<li><a href=\"https://github.com/sb-ai-lab/RePlay/blob/main/experiments/01_replay_basics.ipynb\" target=\"_blank\">01_replay_basics.ipynb</a> - get started with RePlay.</li>\n<li><a href=\"https://github.com/sb-ai-lab/RePlay/blob/main/experiments/02_models_comparison.ipynb\" target=\"_blank\">02_models_comparison.ipynb</a> - reproducible models comparison on <a href=\"https://grouplens.org/datasets/movielens/1m/\" target=\"_blank\">MovieLens-1M dataset</a>.</li>\n<li><a href=\"https://github.com/sb-ai-lab/RePlay/blob/main/experiments/03_features_preprocessing_and_lightFM.ipynb\" target=\"_blank\">03_features_preprocessing_and_lightFM.ipynb</a> - LightFM example with pyspark for feature preprocessing.</li>\n</ol>\n<h2>Quickstart</h2>\n<pre><code> rs_datasets  MovieLens\n\n replay.data_preparator  DataPreparator, Indexer\n replay.metrics  HitRate, NDCG\n replay.models  ItemKNN\n replay.session_handler  State\n replay.splitters  UserSplitter\n\nspark = State().session\n\nml_1m = MovieLens()\n\n\npreparator = DataPreparator()\nlog = preparator.transform(\n    columns_mapping={\n        : ,\n        : ,\n        : ,\n        : \n    }, \n    data=ml_1m.ratings\n)\nindexer = Indexer(user_col=, item_col=)\nindexer.fit(users=log.select(), items=log.select())\nlog_replay = indexer.transform(df=log)\n\n\nuser_splitter = UserSplitter(\n    item_test_size=,\n    user_test_size=,\n    drop_cold_items=,\n    drop_cold_users=,\n    shuffle=,\n    seed=,\n)\ntrain, test = user_splitter.split(log_replay)\n\n\nmodel = ItemKNN()\nmodel.fit(train)\n\n\nrecs = model.predict(\n    log=train,\n    k=K,\n    users=test.select().distinct(),\n    filter_seen_items=,\n)\n\n\nmetrics = Experiment(test,  {NDCG(): K, HitRate(): K})\nmetrics.add_result(, recs)\n</code></pre>\n<h2>Contributing to RePlay</h2>\n<p>For more details visit <a href=\"https://sb-ai-lab.github.io/RePlay/pages/installation.html#development\" target=\"_blank\">development section in docs</a></p>",
  "messages": [
    {
      "id": 2028372,
      "postDate": "2022-11-13T20:56:06.587Z",
      "content": "<p><a href=\"https://github.com/sb-ai-lab/RePlay\" target=\"_blank\">RePlay</a> is a library providing tools for all stages of creating a recommendation system, from data preprocessing to model evaluation and comparison. RePlay uses PySpark to handle big data.</p>\n<p>You can:</p>\n<ul>\n<li>Filter and split data</li>\n<li>Train models</li>\n<li>Optimize hyper parameters</li>\n<li>Evaluate predictions with metrics</li>\n<li>Combine predictions from different models</li>\n<li>Create a two-level model</li>\n</ul>\n<p>Documentation is available <a href=\"https://sb-ai-lab.github.io/RePlay/\" target=\"_blank\">here</a>.</p>\n<h2>Installation</h2>\n<p>Use Linux machine with Python 3.7-3.9, Java 8+ and C++ compiler.</p>\n<pre><code>pip install replay-rec\n</code></pre>\n<p>To get the latest development version or RePlay, <a href=\"https://sb-ai-lab.github.io/RePlay/pages/installation.html#development\" target=\"_blank\">install it from the GitHub repository</a>. It is preferable to use a virtual environment for your installation.</p>\n<h2>Resources</h2>\n<h3>Videos and papers</h3>\n<ul>\n<li><p><strong>Video guides</strong>:</p>\n<ul>\n<li><a href=\"https://www.youtube.com/watch?v=ejQZKGAG0xs\" target=\"_blank\">Replay for offline recommendations, AI Journey 2021</a></li></ul></li>\n<li><p><strong>Research papers</strong>:</p>\n<ul>\n<li>(CORE A RecSys 2021) Yan-Martin Tamm, Rinchin Damdinov, Alexey Vasilev <a href=\"https://dl.acm.org/doi/10.1145/3460231.3478848\" target=\"_blank\">Quality Metrics in Recommender Systems: Do We Calculate Metrics Consistently?</a></li></ul></li>\n</ul>\n<h3>Usage examples</h3>\n<ol>\n<li><a href=\"https://github.com/sb-ai-lab/RePlay/blob/main/experiments/01_replay_basics.ipynb\" target=\"_blank\">01_replay_basics.ipynb</a> - get started with RePlay.</li>\n<li><a href=\"https://github.com/sb-ai-lab/RePlay/blob/main/experiments/02_models_comparison.ipynb\" target=\"_blank\">02_models_comparison.ipynb</a> - reproducible models comparison on <a href=\"https://grouplens.org/datasets/movielens/1m/\" target=\"_blank\">MovieLens-1M dataset</a>.</li>\n<li><a href=\"https://github.com/sb-ai-lab/RePlay/blob/main/experiments/03_features_preprocessing_and_lightFM.ipynb\" target=\"_blank\">03_features_preprocessing_and_lightFM.ipynb</a> - LightFM example with pyspark for feature preprocessing.</li>\n</ol>\n<h2>Quickstart</h2>\n<pre><code> rs_datasets  MovieLens\n\n replay.data_preparator  DataPreparator, Indexer\n replay.metrics  HitRate, NDCG\n replay.models  ItemKNN\n replay.session_handler  State\n replay.splitters  UserSplitter\n\nspark = State().session\n\nml_1m = MovieLens()\n\n\npreparator = DataPreparator()\nlog = preparator.transform(\n    columns_mapping={\n        : ,\n        : ,\n        : ,\n        : \n    }, \n    data=ml_1m.ratings\n)\nindexer = Indexer(user_col=, item_col=)\nindexer.fit(users=log.select(), items=log.select())\nlog_replay = indexer.transform(df=log)\n\n\nuser_splitter = UserSplitter(\n    item_test_size=,\n    user_test_size=,\n    drop_cold_items=,\n    drop_cold_users=,\n    shuffle=,\n    seed=,\n)\ntrain, test = user_splitter.split(log_replay)\n\n\nmodel = ItemKNN()\nmodel.fit(train)\n\n\nrecs = model.predict(\n    log=train,\n    k=K,\n    users=test.select().distinct(),\n    filter_seen_items=,\n)\n\n\nmetrics = Experiment(test,  {NDCG(): K, HitRate(): K})\nmetrics.add_result(, recs)\n</code></pre>\n<h2>Contributing to RePlay</h2>\n<p>For more details visit <a href=\"https://sb-ai-lab.github.io/RePlay/pages/installation.html#development\" target=\"_blank\">development section in docs</a></p>",
      "rawMarkdown": "[RePlay](https://github.com/sb-ai-lab/RePlay) is a library providing tools for all stages of creating a recommendation system, from data preprocessing to model evaluation and comparison. RePlay uses PySpark to handle big data.\n\nYou can:\n- Filter and split data\n- Train models\n- Optimize hyper parameters\n- Evaluate predictions with metrics\n- Combine predictions from different models\n- Create a two-level model\n\nDocumentation is available [here](https://sb-ai-lab.github.io/RePlay/).\n\n## Installation\n\nUse Linux machine with Python 3.7-3.9, Java 8+ and C++ compiler.\n\n```bash\npip install replay-rec\n```\n\nTo get the latest development version or RePlay, [install it from the GitHub repository](https://sb-ai-lab.github.io/RePlay/pages/installation.html#development). It is preferable to use a virtual environment for your installation.\n\n## Resources\n\n### Videos and papers\n* **Video guides**:\n\t- [Replay for offline recommendations, AI Journey 2021](https://www.youtube.com/watch?v=ejQZKGAG0xs)\n\n* **Research papers**:\n\t- (CORE A RecSys 2021) Yan-Martin Tamm, Rinchin Damdinov, Alexey Vasilev [Quality Metrics in Recommender Systems: Do We Calculate Metrics Consistently?](https://dl.acm.org/doi/10.1145/3460231.3478848)\n\n### Usage examples\n1. [01_replay_basics.ipynb](https://github.com/sb-ai-lab/RePlay/blob/main/experiments/01_replay_basics.ipynb) - get started with RePlay.\n2. [02_models_comparison.ipynb](https://github.com/sb-ai-lab/RePlay/blob/main/experiments/02_models_comparison.ipynb) - reproducible models comparison on [MovieLens-1M dataset](https://grouplens.org/datasets/movielens/1m/).\n3. [03_features_preprocessing_and_lightFM.ipynb](https://github.com/sb-ai-lab/RePlay/blob/main/experiments/03_features_preprocessing_and_lightFM.ipynb) - LightFM example with pyspark for feature preprocessing.\n\n## Quickstart\n\n```python\nfrom rs_datasets import MovieLens\n\nfrom replay.data_preparator import DataPreparator, Indexer\nfrom replay.metrics import HitRate, NDCG\nfrom replay.models import ItemKNN\nfrom replay.session_handler import State\nfrom replay.splitters import UserSplitter\n\nspark = State().session\n\nml_1m = MovieLens(\"1m\")\n\n# data preprocessing\npreparator = DataPreparator()\nlog = preparator.transform(\n    columns_mapping={\n        'user_id': 'user_id',\n        'item_id': 'item_id',\n        'relevance': 'rating',\n        'timestamp': 'timestamp'\n    }, \n    data=ml_1m.ratings\n)\nindexer = Indexer(user_col='user_id', item_col='item_id')\nindexer.fit(users=log.select('user_id'), items=log.select('item_id'))\nlog_replay = indexer.transform(df=log)\n\n# data splitting\nuser_splitter = UserSplitter(\n    item_test_size=10,\n    user_test_size=500,\n    drop_cold_items=True,\n    drop_cold_users=True,\n    shuffle=True,\n    seed=42,\n)\ntrain, test = user_splitter.split(log_replay)\n\n# model training\nmodel = ItemKNN()\nmodel.fit(train)\n\n# model inference\nrecs = model.predict(\n    log=train,\n    k=K,\n    users=test.select('user_idx').distinct(),\n    filter_seen_items=True,\n)\n\n# model evaluation\nmetrics = Experiment(test,  {NDCG(): K, HitRate(): K})\nmetrics.add_result(\"knn\", recs)\n```\n\n## Contributing to RePlay\n\nFor more details visit [development section in docs](https://sb-ai-lab.github.io/RePlay/pages/installation.html#development)\n",
      "votes": 18
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "2028372": "[RePlay](https://github.com/sb-ai-lab/RePlay) is a library providing tools for all stages of creating a recommendation system, from data preprocessing to model evaluation and comparison. RePlay uses PySpark to handle big data.\n\nYou can:\n- Filter and split data\n- Train models\n- Optimize hyper parameters\n- Evaluate predictions with metrics\n- Combine predictions from different models\n- Create a two-level model\n\nDocumentation is available [here](https://sb-ai-lab.github.io/RePlay/).\n\n## Installation\n\nUse Linux machine with Python 3.7-3.9, Java 8+ and C++ compiler.\n\n```bash\npip install replay-rec\n```\n\nTo get the latest development version or RePlay, [install it from the GitHub repository](https://sb-ai-lab.github.io/RePlay/pages/installation.html#development). It is preferable to use a virtual environment for your installation.\n\n## Resources\n\n### Videos and papers\n* **Video guides**:\n\t- [Replay for offline recommendations, AI Journey 2021](https://www.youtube.com/watch?v=ejQZKGAG0xs)\n\n* **Research papers**:\n\t- (CORE A RecSys 2021) Yan-Martin Tamm, Rinchin Damdinov, Alexey Vasilev [Quality Metrics in Recommender Systems: Do We Calculate Metrics Consistently?](https://dl.acm.org/doi/10.1145/3460231.3478848)\n\n### Usage examples\n1. [01_replay_basics.ipynb](https://github.com/sb-ai-lab/RePlay/blob/main/experiments/01_replay_basics.ipynb) - get started with RePlay.\n2. [02_models_comparison.ipynb](https://github.com/sb-ai-lab/RePlay/blob/main/experiments/02_models_comparison.ipynb) - reproducible models comparison on [MovieLens-1M dataset](https://grouplens.org/datasets/movielens/1m/).\n3. [03_features_preprocessing_and_lightFM.ipynb](https://github.com/sb-ai-lab/RePlay/blob/main/experiments/03_features_preprocessing_and_lightFM.ipynb) - LightFM example with pyspark for feature preprocessing.\n\n## Quickstart\n\n```python\nfrom rs_datasets import MovieLens\n\nfrom replay.data_preparator import DataPreparator, Indexer\nfrom replay.metrics import HitRate, NDCG\nfrom replay.models import ItemKNN\nfrom replay.session_handler import State\nfrom replay.splitters import UserSplitter\n\nspark = State().session\n\nml_1m = MovieLens(\"1m\")\n\n# data preprocessing\npreparator = DataPreparator()\nlog = preparator.transform(\n    columns_mapping={\n        'user_id': 'user_id',\n        'item_id': 'item_id',\n        'relevance': 'rating',\n        'timestamp': 'timestamp'\n    }, \n    data=ml_1m.ratings\n)\nindexer = Indexer(user_col='user_id', item_col='item_id')\nindexer.fit(users=log.select('user_id'), items=log.select('item_id'))\nlog_replay = indexer.transform(df=log)\n\n# data splitting\nuser_splitter = UserSplitter(\n    item_test_size=10,\n    user_test_size=500,\n    drop_cold_items=True,\n    drop_cold_users=True,\n    shuffle=True,\n    seed=42,\n)\ntrain, test = user_splitter.split(log_replay)\n\n# model training\nmodel = ItemKNN()\nmodel.fit(train)\n\n# model inference\nrecs = model.predict(\n    log=train,\n    k=K,\n    users=test.select('user_idx').distinct(),\n    filter_seen_items=True,\n)\n\n# model evaluation\nmetrics = Experiment(test,  {NDCG(): K, HitRate(): K})\nmetrics.add_result(\"knn\", recs)\n```\n\n## Contributing to RePlay\n\nFor more details visit [development section in docs](https://sb-ai-lab.github.io/RePlay/pages/installation.html#development)\n"
  }
}