Hey!
\nVery excited to finally see a Recommender systems competition. I hope some creative solutions will give birth to new models. Keep in mind that the last big Recsys competition gave us Matrix Factorization :D
\nI wanted to take the opportunity and tickle your minds about how to properly evaluate Recommender systems, both in training and testing.
A well-known issue in recommender systems is the way the data is collected. In a perfect supervised learning setting, we make sure that the data is balanced, labels are correct, points are iid, etc… In Recsys, data is generated by users and this throws most traditional Machine Learning assumptions out of the window. For instance, a user cannot interact with an item without seeing it first. therefore, the collected data heavily depends on the previous exposure of the user (this exposure can come from a previous recommender system, the popularity of the item, a friend recommendation, another platform, an ad…)
\nWhy is this important? This exposure bias creates a Missing Not At Random problem that biases the estimation of the loss functions, evaluation functions, etc…
\nWhat I mean by biasing an estimation is that you can no longer trust that the loss value you calculated is unbiased and hence it may give you the wrong estimation of the true performance of the model.
\nTo understand more about how this exposure may become a problem I invite you to read this paper:
\nhttps://arxiv.org/abs/1602.05352 and other papers that cite it.
In testing, it is even harder. How do you know that you recommended a negative item without showing it to the user first and getting their impression? That is why in industry, online A/B testing is used extensively to gauge the performance of a new model. A/B testing is however expensive. So we try to come up with other ways of offline evaluation. These offline evaluations can be very tricky as we don't really know if the user has seen and did not like the item or the user simply was not exposed to the item. Many papers discuss this issue and come up with more rigorous evaluation techniques that can mitigate this bias:
\nhttps://www.kdd.org/kdd2020/accepted-papers/view/on-sampled-metrics-for-item-recommendation a great paper explaining the effect of negative sampling on different evaluation metrics
A nice paper discussing the different offline evaluation methods in Recsys
\nhttp://adrem.uantwerpen.be/bibrem/pubs/JeunenRecSys19_DoctoralSymposium.pdf
Very excited to see the validation strategies that people are going to use in this competition as it is going to be a major deciding factor.
\nGoog luck everyone
Hey!
\nVery excited to finally see a Recommender systems competition. I hope some creative solutions will give birth to new models. Keep in mind that the last big Recsys competition gave us Matrix Factorization :D
\nI wanted to take the opportunity and tickle your minds about how to properly evaluate Recommender systems, both in training and testing.
A well-known issue in recommender systems is the way the data is collected. In a perfect supervised learning setting, we make sure that the data is balanced, labels are correct, points are iid, etc… In Recsys, data is generated by users and this throws most traditional Machine Learning assumptions out of the window. For instance, a user cannot interact with an item without seeing it first. therefore, the collected data heavily depends on the previous exposure of the user (this exposure can come from a previous recommender system, the popularity of the item, a friend recommendation, another platform, an ad…)
\nWhy is this important? This exposure bias creates a Missing Not At Random problem that biases the estimation of the loss functions, evaluation functions, etc…
\nWhat I mean by biasing an estimation is that you can no longer trust that the loss value you calculated is unbiased and hence it may give you the wrong estimation of the true performance of the model.
\nTo understand more about how this exposure may become a problem I invite you to read this paper:
\nhttps://arxiv.org/abs/1602.05352 and other papers that cite it.
In testing, it is even harder. How do you know that you recommended a negative item without showing it to the user first and getting their impression? That is why in industry, online A/B testing is used extensively to gauge the performance of a new model. A/B testing is however expensive. So we try to come up with other ways of offline evaluation. These offline evaluations can be very tricky as we don't really know if the user has seen and did not like the item or the user simply was not exposed to the item. Many papers discuss this issue and come up with more rigorous evaluation techniques that can mitigate this bias:
\nhttps://www.kdd.org/kdd2020/accepted-papers/view/on-sampled-metrics-for-item-recommendation a great paper explaining the effect of negative sampling on different evaluation metrics
A nice paper discussing the different offline evaluation methods in Recsys
\nhttp://adrem.uantwerpen.be/bibrem/pubs/JeunenRecSys19_DoctoralSymposium.pdf
Very excited to see the validation strategies that people are going to use in this competition as it is going to be a major deciding factor.
\nGoog luck everyone