Just a shower thought whilst running code; although ostensibly each of us are working alone to win this competition, we are in some way forming the \"Random Kaggle estimator\".
\nEach team is a \"weak\" learner (not meant in any pejorative sense) akin to a decision tree estimator. However, collectively we are like the strong Random Forest leading to an almost optimal solution, using the Public Leaderboard as our objective function. We are very much guided by the work of others, either directly (for example forking and tweaking public notebooks) or indirectly (ideas as to what to explore further, or what others have already tried and do not seem to work, or even simply observing a sudden jump in the top LB score indicating that there is some magic to be found). The ideas and code do not even have to originate from this competition, but maybe are from a similar kaggle competition in the (perhaps distant) past, and some of the more recent Tabular Playground Series write-ups are astounding and truly inspiring!
\nIt is this 'human-in-the-loop' meta-estimator that can be found only on kaggle!
\nAnyway, I think my script has just finished, so back to work…
\nGood luck to all; the better you do, the better everyone does!
", "messages": [ { "id": "1827527", "postDate": "06/21/2022 05:39:27", "content": "Just a shower thought whilst running code; although ostensibly each of us are working alone to win this competition, we are in some way forming the \"Random Kaggle estimator\".
\nEach team is a \"weak\" learner (not meant in any pejorative sense) akin to a decision tree estimator. However, collectively we are like the strong Random Forest leading to an almost optimal solution, using the Public Leaderboard as our objective function. We are very much guided by the work of others, either directly (for example forking and tweaking public notebooks) or indirectly (ideas as to what to explore further, or what others have already tried and do not seem to work, or even simply observing a sudden jump in the top LB score indicating that there is some magic to be found). The ideas and code do not even have to originate from this competition, but maybe are from a similar kaggle competition in the (perhaps distant) past, and some of the more recent Tabular Playground Series write-ups are astounding and truly inspiring!
\nIt is this 'human-in-the-loop' meta-estimator that can be found only on kaggle!
\nAnyway, I think my script has just finished, so back to work…
\nGood luck to all; the better you do, the better everyone does!
", "rawMarkdown": "Just a shower thought whilst running code; although ostensibly each of us are working alone to win this competition, we are in some way forming the \"Random Kaggle estimator\".\n\nEach team is a \"weak\" learner (not meant in any pejorative sense) akin to a decision tree estimator. However, collectively we are like the strong Random Forest leading to an almost optimal solution, using the Public Leaderboard as our objective function. We are very much guided by the work of others, either directly (for example forking and tweaking public notebooks) or indirectly (ideas as to what to explore further, or what others have already tried and do not seem to work, or even simply observing a sudden jump in the top LB score indicating that there is some *magic* to be found). The ideas and code do not even have to originate from this competition, but maybe are from a similar kaggle competition in the (perhaps distant) past, and some of the more recent Tabular Playground Series write-ups are astounding and truly inspiring!\n\nIt is this 'human-in-the-loop' meta-estimator that can be found only on kaggle! \n\nAnyway, I think my script has just finished, so back to work...\n\nGood luck to all; the better you do, the better everyone does!", "votes": null }, { "id": "1827661", "postDate": "06/21/2022 07:58:36", "content": "if all team can made their submission csv file public …. then it will be interesting
", "rawMarkdown": "if all team can made their submission csv file public .... then it will be interesting", "votes": null }, { "id": "1827673", "postDate": "06/21/2022 08:09:53", "content": "Dear @hengck23
\nAmusing idea! One thing about stand-alone submission.csv files is that they are a bit like sausages; you don't know what went into making them. Indeed perhaps all closed-source blending notebooks should even come with a caveat emptor. For example, they must clearly say on the packet whether they have B_29 added into the mix or not 😃
All the best,
\ncarl
We are all part of a big ensemble.
", "rawMarkdown": "We are all part of a big ensemble.", "votes": null }, { "id": "1827816", "postDate": "06/21/2022 10:04:28", "content": "Hi @carlmcbrideellis I would be curious if there is some gain in averaging lots of score or if it is somehow capped. I would be also curious of innovative weighting scheme that could be used. I know that some market finance techniques (Hierarchical Risk Parity) was proposed.
", "rawMarkdown": "Hi @carlmcbrideellis I would be curious if there is some gain in averaging lots of score or if it is somehow capped. I would be also curious of innovative weighting scheme that could be used. I know that some market finance techniques (Hierarchical Risk Parity) was proposed.", "votes": null }, { "id": "1827819", "postDate": "06/21/2022 10:09:33", "content": "Dear @lucasmorin
\nI am completely unfamiliar with \"Hierarchical Risk Parity\", do you have any links?
\nAll the best,
\ncarl
well at least B_29 is at the bottom of the list in terms how useful it is: )
Dear @raddar
\nOnly for the training data and the Public chunk; ¿what happens when it 'kicks-in' on the Private data is the question?
\nThat said, if the estimators they use employ just a smidgen of regularization (i.e. L2, or perhaps better still L1) they probably create a model that diminishes (L2) or disregards (L1) the feature B_29 anyway…
All the best,
\ncarl
HRP Paper: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2708678
\nHRP Slides: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2713516
I've seen the idea of applying portfolio selection approach to model ensembling as the problem are relatively similar (can't find a good ressource right now).
", "rawMarkdown": "HRP Paper: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2708678\nHRP Slides: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2713516\n\nI've seen the idea of applying portfolio selection approach to model ensembling as the problem are relatively similar (can't find a good ressource right now).", "votes": null }, { "id": "1828000", "postDate": "06/21/2022 12:50:08", "content": "We demonstrate the real work behind winning any Kaggle competition is actually accomplished through a human-computer collaboration. Teams are treated as individual weak learners who compete against one another by deriving optimized estimation solutions on the public leaderboard. Mutually and unconsciously, competitors in turn capitalize on the strengths and mitigate the weaknesses of other competitors by re-reporting their findings, when necessary, and ultimately developing their own solutions. By assembling the individual solutions together, we obtain an ideal solution, which is almost optimal. This \"Random Kaggle estimator\" enables us to demonstrate how humans, as part of an ML system, significantly contribute to its overall ability to solve complex problems. The more competitors, the more distinct differences there are between them (a min. two requisite), and the greater the leaderboard fluctuations, the faster and better the meta-estimation solution can be developed.
\nFurthermore, we define a measure of the degree to which random actions by humans are required to accomplish a task relative to the task's complexity - \"Stochastic Leaderboard Descent\". We will demonstrate that the more complex a problem-solving task requires a greater number of independent teams working in unison to accomplish the ML task, with increasing importance placed on each team as the degree of stochasticity increases. We also show that although some of the ideas and code do not originate from the same competition (or are from the same competition at a different time), there are still ideas that we can not attribute to being influenced by any specific historical source.
\nSuggesting that such ideas result from pure creativity and are driven by intuition, a human strength that extends far beyond ML tasks.
\nIn the world of Kaggle, very often only the top spot on the leaderboard matters. On the other hand, once the leaderboard is released, it becomes a valuable tool for researchers to learn from and obtain new insights, especially in competitions with a sufficient number of competitors.
\nIn this paper, we show that there is much more to leaderboards than the winner's name. Using a recent Kaggle competition as an example, we illustrate that competitors are able to adapt to one another and progressively improve their leaderboard score by benefiting from the cumulative effect of multiple contributions. We formulate a \"Random Kaggle estimator\" which, although simple, is able to reproduce the leaderboard evolution curve and we derive some insights from this model.
\nWe then demonstrate how the leaderboard evolution curve itself can be used to define an optimal estimation model, which we call the \"stochastic leaderboard descent\".
\nDefinition 1: Solution Blending
\n\n\nThe process of assembling the predictions of multiple solutions in order to produce a new and improved estimate of the target value.
\n
Therefore, it follows from Definition 1 that a blended solution is the combination of multiple solutions that can be assembled to produce a more accurate estimate of the target value. As the blended solution is a combination of multiple solutions, it is a collaborative ensemble of solutions. Therefore, we can define a blended solution as follows:
\n$$\\beta(x_i) = \\sum_j \\alpha_j(x_i) x_j(x_i)$$
\nWhere β is the blended solution, 𝛼 is the solution weight and x_j(x_i) is the prediction for the training instance x_i.
\nIdeas Blending
\nBefore we define we must first define Idea:
\nData scientists come up with a metric and then optimize that metric.
\nAs we can see the information of the target outcome is linked to both the solution and the methods used to optimize the solution. We can represent a solution as follows:
\n$$x_i(z) = \\Phi_i(z)\\phi_i(z)$$
\nWhere z is the data, Φ is the encoder (Optimization method) and 𝜙 is the decoder (model), the feature-based solution.
\nLastly, we are able to formulate the idea of data scientist as:
\n$$y_i(z) = M(x_i(z))$$
\nWhere y_i(z) is the data science outcome and M is the information linking the respective solution to the metric.
\nDefinition 2: Ideas Blending
\n\n\nThe process of assembling the predictions of multiple ideas in order to produce a new and improved estimate of the target value.
\n
From Definition 1, we can define Ideas Blending as:
\n$$y_i = \\arg \\min_{\\beta(z)} \\left[\\sum_j \\alpha_j y_j(z)\\right]$$
\nSince the scientists try to blend ideas in order to minimize his leaderboard position.
\nDefinition 3: Stochastic Leaderboard Convergence (SLC)
\n\n\nA Stochastic Leaderboard Convergence is a process of obtaining an improved estimate of the target value by re-estimating the target value after a new public leaderboard score has been reported.
\n
It follows from Definition 1 that a Stochastic Leaderboard Convergence (SLC) is a process of obtaining an improved estimate of the target value by re-estimating the target value after a new public leaderboard score has been reported. The idea behind SLC is that the more estimates of the target value are reported on the public leaderboard, the better the estimate of the target value will be.
\nWe can therefore formulate the Stochastic Leaderboard Convergence (SLC) as follows:
\n$$\\hat{y} = \\arg \\min_{\\beta(x_i)} \\left[\\sum_i \\left[\\sum_j \\alpha_j(x_i) x_j(x_i) - y_i\\right]^2\\right]$$
\nDefinition 4: Public Leaderboard Density
\n\n\nThe density of the number of public leaderboard scores reported.
\n
The Public Leaderboard Density (PLD) is the number of reported public leaderboard scores per unit area (competition timeframe). We can formulate the PLD as follows:
\n$$PLD = \\frac{N}{T}$$
\nWhere N is the number of public leaderboard scores reported, and T is the competition timeframe.
\nDefinition 4: Stochastic Leaderboard Descent (SLD)
\n\n\nA Stochastic Leaderboard Descent is a process of obtaining an improved estimate of the target value by taking the current public leaderboard score as the starting point and applying stochastic gradient descent to obtain a better estimate of the target value.
\n
Bayesian Blending
\nBayesian Blending is a process of obtaining an improved estimate of the target value by taking the current top public solution as the starting point and applying stochastic notebook descent to obtain a better estimate of the target value.
\nSince there are:
We propse to define the blended solution as a linear combination of both a solution term and an idea term:
$$\\omega(x_i) = \\sum_j \\alpha_j n_j(z) + \\sum_k \\beta_k M(x_k(z))$$
\nWhere n_j is the notebook's code used (function of the data z) and M(x_k(z)) again is the information extracted from an idea x_i (function of the competition's data). The weights 𝛼 and 𝛽 are the probabilistic weight of the notebooks and ideas.
\nWe therefore can define the direction of the kaggle bayesian blending process as the gradient of the notebook-idea-blended solution.
\n$$\\nabla_k(z) = \\sum_j \\alpha_j(z) \\nabla_k n_j(z) + \\sum_j \\beta_j(z) \\nabla_k M(x_j(z))$$
\nTherefore, the notebook idea blended solution can be formulated as a kernel function:
\n$$K_j(z,z') = \\nabla_k(z) x_j(z')$$
\nTo optimize the kernel function, we can introduce a kernel matrix of all the kernels and their respective gradients for each notebook-idea-blended solution.
\n$$\\hat{K}_{jk}(z,z') = \\nabla_k(z) x_j(z')$$
\nHistorical influence can sometime be attributed to some public submissions. Also with the recent social activity made popular by covid lockdown elimination this type of influence becomes more and more of a factor. Thus, from second law of medaldynamics we can assume that just like entropy, ideas flow from low medal density areas to high medal density areas, so we can define the past idea-solution as:
\n$$n_j(z) = \\sum_i \\nabla_{i}(z)$$
\nWhere 𝛼 is the coefficient for idea divergence (The second law of medaldynamics) and 𝛽 is the past idea-solution.
\nWe can therefore define the emergence of creativity as the process of extracting the solution that is orthogonal to past influence.
", "rawMarkdown": "### Kaggle Random Walk\n##### Regularized Stochastic Leaderboard Descent\n_____\n\n### Abstract\nWe demonstrate the real work behind winning any Kaggle competition is actually accomplished through a human-computer collaboration. Teams are treated as individual weak learners who compete against one another by deriving optimized estimation solutions on the public leaderboard. Mutually and unconsciously, competitors in turn capitalize on the strengths and mitigate the weaknesses of other competitors by re-reporting their findings, when necessary, and ultimately developing their own solutions. By assembling the individual solutions together, we obtain an ideal solution, which is almost optimal. This \"Random Kaggle estimator\" enables us to demonstrate how humans, as part of an ML system, significantly contribute to its overall ability to solve complex problems. The more competitors, the more distinct differences there are between them (a min. two requisite), and the greater the leaderboard fluctuations, the faster and better the meta-estimation solution can be developed.\n\nFurthermore, we define a measure of the degree to which random actions by humans are required to accomplish a task relative to the task's complexity - \"Stochastic Leaderboard Descent\". We will demonstrate that the more complex a problem-solving task requires a greater number of independent teams working in unison to accomplish the ML task, with increasing importance placed on each team as the degree of stochasticity increases. We also show that although some of the ideas and code do not originate from the same competition (or are from the same competition at a different time), there are still ideas that we can not attribute to being influenced by any specific historical source.\n\nSuggesting that such ideas result from pure creativity and are driven by intuition, a human strength that extends far beyond ML tasks.\n\n#### Introduction\n\nIn the world of Kaggle, very often only the top spot on the leaderboard matters. On the other hand, once the leaderboard is released, it becomes a valuable tool for researchers to learn from and obtain new insights, especially in competitions with a sufficient number of competitors.\n\nIn this paper, we show that there is much more to leaderboards than the winner's name. Using a recent Kaggle competition as an example, we illustrate that competitors are able to adapt to one another and progressively improve their leaderboard score by benefiting from the cumulative effect of multiple contributions. We formulate a \"Random Kaggle estimator\" which, although simple, is able to reproduce the leaderboard evolution curve and we derive some insights from this model.\n\nWe then demonstrate how the leaderboard evolution curve itself can be used to define an optimal estimation model, which we call the \"stochastic leaderboard descent\".\n\n### Preliminaries\n\n#### Generalized Blending Theory\n\n**Definition 1: Solution Blending**\n> The process of assembling the predictions of multiple solutions in order to produce a new and improved estimate of the target value.\n\nTherefore, it follows from Definition 1 that a blended solution is the combination of multiple solutions that can be assembled to produce a more accurate estimate of the target value. As the blended solution is a combination of multiple solutions, it is a collaborative ensemble of solutions. Therefore, we can define a blended solution as follows:\n\n$$\\beta(x_i) = \\sum_j \\alpha_j(x_i) x_j(x_i)$$\n\nWhere *β* is the blended solution, *𝛼* is the solution weight and *x_j(x_i)* is the prediction for the training instance *x_i*.\n\n**Ideas Blending**\n\nBefore we define we must first define *Idea*:\n\nData scientists come up with a metric and then optimize that metric.\n\nAs we can see the information of the target outcome is linked to both the solution and the methods used to optimize the solution. We can represent a solution as follows:\n\n$$x_i(z) = \\Phi_i(z)\\phi_i(z)$$\n\nWhere *z* is the data, *Φ* is the encoder (Optimization method) and 𝜙 is the decoder (model), the feature-based solution.\n\nLastly, we are able to formulate the idea of data scientist as:\n\n$$y_i(z) = M(x_i(z))$$\n\nWhere *y_i(z)* is the data science outcome and *M* is the information linking the respective solution to the metric.\n\n**Definition 2: Ideas Blending**\n> The process of assembling the predictions of multiple ideas in order to produce a new and improved estimate of the target value.\n\nFrom Definition 1, we can define *Ideas Blending* as:\n\n$$y_i = \\arg \\min_{\\beta(z)} \\left[\\sum_j \\alpha_j y_j(z)\\right]$$\n\nSince the scientists try to blend ideas in order to minimize his leaderboard position.\n\n**Definition 3: Stochastic Leaderboard Convergence (SLC)**\n> A Stochastic Leaderboard Convergence is a process of obtaining an improved estimate of the target value by re-estimating the target value after a new public leaderboard score has been reported.\n\nIt follows from Definition 1 that a Stochastic Leaderboard Convergence (SLC) is a process of obtaining an improved estimate of the target value by re-estimating the target value after a new public leaderboard score has been reported. The idea behind SLC is that the more estimates of the target value are reported on the public leaderboard, the better the estimate of the target value will be.\n\nWe can therefore formulate the Stochastic Leaderboard Convergence (SLC) as follows:\n\n$$\\hat{y} = \\arg \\min_{\\beta(x_i)} \\left[\\sum_i \\left[\\sum_j \\alpha_j(x_i) x_j(x_i) - y_i\\right]^2\\right]$$\n\n**Definition 4: Public Leaderboard Density**\n> The density of the number of public leaderboard scores reported.\n\nThe Public Leaderboard Density (PLD) is the number of reported public leaderboard scores per unit area (competition timeframe). We can formulate the PLD as follows:\n\n$$PLD = \\frac{N}{T}$$\n\nWhere *N* is the number of public leaderboard scores reported, and *T* is the competition timeframe.\n\n**Definition 4: Stochastic Leaderboard Descent (SLD)**\n> A Stochastic Leaderboard Descent is a process of obtaining an improved estimate of the target value by taking the current public leaderboard score as the starting point and applying stochastic gradient descent to obtain a better estimate of the target value.\n\n#### Leaderboard Monte Carlo Estimation\n\n**Bayesian Blending**\n\nBayesian Blending is a process of obtaining an improved estimate of the target value by taking the current top public solution as the starting point and applying stochastic notebook descent to obtain a better estimate of the target value.\nSince there are: \n\n- **Solutions** released as notebooks\n- **Ideas** released on the discussions forums\n\nWe propse to define the blended solution as a linear combination of both a `solution term` and an `idea term`:\n\n$$\\omega(x_i) = \\sum_j \\alpha_j n_j(z) + \\sum_k \\beta_k M(x_k(z))$$\n\nWhere *n_j* is the notebook's code used (function of the data *z*) and *M(x_k(z))* again is the information extracted from an idea *x_i* (function of the competition's data). The weights *𝛼* and *𝛽* are the probabilistic weight of the notebooks and ideas.\n\nWe therefore can define the direction of the kaggle bayesian blending process as the gradient of the notebook-idea-blended solution.\n\n$$\\nabla_k(z) = \\sum_j \\alpha_j(z) \\nabla_k n_j(z) + \\sum_j \\beta_j(z) \\nabla_k M(x_j(z))$$\n\nTherefore, the notebook idea blended solution can be formulated as a kernel function:\n\n$$K_j(z,z') = \\nabla_k(z) x_j(z')$$\n\nTo optimize the kernel function, we can introduce a kernel matrix of all the kernels and their respective gradients for each notebook-idea-blended solution.\n\n$$\\hat{K}_{jk}(z,z') = \\nabla_k(z) x_j(z')$$\n\n#### Past solution influence\n\nHistorical influence can sometime be attributed to some public submissions. Also with the recent social activity made popular by covid lockdown elimination this type of influence becomes more and more of a factor. Thus, from second law of medaldynamics we can assume that just like entropy, ideas flow from low medal density areas to high medal density areas, so we can define the past idea-solution as:\n\n$$n_j(z) = \\sum_i \\nabla_{i}(z)$$\n\nWhere *𝛼* is the coefficient for idea divergence *(The second law of medaldynamics)* and *𝛽* is the past idea-solution.\n\nWe can therefore define the **emergence of creativity** as the process of extracting the solution that is orthogonal to past influence.", "votes": null }, { "id": "1828038", "postDate": "06/21/2022 13:03:12", "content": "Dear @thedevastator
\nEr, wow! I guess that is why you are called \"The Devastator\"… 😃
\nThat said, somewhat more than your \"Regularized Stochastic Leaderboard Descent\"(!) I am sort of reminded of the genetic algorithms that were very much the fashion some 20-30 years ago….
\nAll the best,
\ncarl
Kagglers…Assemble.
", "rawMarkdown": "Kagglers...Assemble.", "votes": null }, { "id": "1828393", "postDate": "06/21/2022 16:45:49", "content": "You should get likes simply for attempting to write this!!🙌
", "rawMarkdown": "You should get likes simply for attempting to write this!!🙌", "votes": null }, { "id": "1828405", "postDate": "06/21/2022 16:58:26", "content": "IIRC, in the past, organizers have created aggregate models from submissions after the end of some competitions. I don’t remember which competititions but it might have been some of the basketball or american football ones.
", "rawMarkdown": "IIRC, in the past, organizers have created aggregate models from submissions after the end of some competitions. I don’t remember which competititions but it might have been some of the basketball or american football ones.", "votes": null }, { "id": "1828408", "postDate": "06/21/2022 17:02:09", "content": "Thank you, Thank you 😁
", "rawMarkdown": "Thank you, Thank you 😁", "votes": null }, { "id": "1828419", "postDate": "06/21/2022 17:11:18", "content": "Dear @christoffer
\nThat is interesting. Although from an organizers point of view it is worth distinguishing between ensembling a diverse set of models using the provided code (as per the \"Kaggle Ensembling Guide\") which can (could? \"Netflix Never Used Its $1 Million Algorithm Due To Engineering Costs\") be beneficial to the organizers, as opposed to the blending of raw submission.csv files together, which may also do well on a kaggle competition, but on the other hand provide little (basically nothing) of use to the organizers, as they are solutions specific to the dataset provided. (PS: I am a little surprised that this is not a \"Code\" competition).
All the best,
\ncarl
Thanks for that link. I've been learning much more about ensembles and missed that resource.
", "rawMarkdown": "Thanks for that link. I've been learning much more about ensembles and missed that resource.", "votes": null }, { "id": "1829757", "postDate": "06/22/2022 22:58:44", "content": "Beautifully written. I scout the site for technical insights but, occasionally, I stumble on such meta posts which make the process not just useful, but also pleasurable.
", "rawMarkdown": "Beautifully written. I scout the site for technical insights but, occasionally, I stumble on such meta posts which make the process not just useful, but also pleasurable.", "votes": null }, { "id": "1829985", "postDate": "06/23/2022 04:54:26", "content": "Dear @edwardakalarrywelch
\nMotivated by your comment I have just posted the link as a Topic. As this is the first big \"Featured\" non-forecasting tabular competition in quite a while, and could hit 4-5k participants, and there will be quite a few people who have not seen \"kaggle ensembling\" in the text books. Furthermore, the site MLwave went down about a year ago, but thankfully the content was rescued by the Wayback Machine.
\nAll the best,
\ncarl
People at AMEX were probably like:
\n— Hey we need an insane model for this business problem…
\n— Lets just fit a Kaggle Random Forest.
Hola @heyspaceturtle
\nMaybe, but it takes 3 months to run, and doesn't come cheap with $100,000 for prize money and an unknown amount in compute + storage…
\nThat said, credit default could cost somebody like American Express $M each year, so even a slightly better model could be worth the outlay.
\nUn saludo muy codrial,
\ncarl
True… but they still get the chance to find talented data scientists for their team :)
\nSaludos!
", "rawMarkdown": "True... but they still get the chance to find talented data scientists for their team :) \n\nSaludos!", "votes": null }, { "id": "1832792", "postDate": "06/25/2022 11:54:52", "content": "Great work @thedevastator, upvoted :)
", "rawMarkdown": "Great work @thedevastator, upvoted :)", "votes": null }, { "id": "1836195", "postDate": "06/28/2022 12:50:06", "content": "nice one…!
", "rawMarkdown": "nice one...!", "votes": null } ], "comments": [ { "id": 1827661, "author_name": "hengck23", "author_url": "", "post_date": "06/21/2022 07:58:36", "content": "if all team can made their submission csv file public …. then it will be interesting
", "votes": null, "replies": [ { "id": 1827673, "author_name": "carlmcbrideellis", "author_url": "", "post_date": "06/21/2022 08:09:53", "content": "Dear @hengck23
\nAmusing idea! One thing about stand-alone submission.csv files is that they are a bit like sausages; you don't know what went into making them. Indeed perhaps all closed-source blending notebooks should even come with a caveat emptor. For example, they must clearly say on the packet whether they have B_29 added into the mix or not 😃
All the best,
\ncarl
well at least B_29 is at the bottom of the list in terms how useful it is: )
Dear @raddar
\nOnly for the training data and the Public chunk; ¿what happens when it 'kicks-in' on the Private data is the question?
\nThat said, if the estimators they use employ just a smidgen of regularization (i.e. L2, or perhaps better still L1) they probably create a model that diminishes (L2) or disregards (L1) the feature B_29 anyway…
All the best,
\ncarl
We are all part of a big ensemble.
", "votes": null, "replies": [] }, { "id": 1827816, "author_name": "lucasmorin", "author_url": "", "post_date": "06/21/2022 10:04:28", "content": "Hi @carlmcbrideellis I would be curious if there is some gain in averaging lots of score or if it is somehow capped. I would be also curious of innovative weighting scheme that could be used. I know that some market finance techniques (Hierarchical Risk Parity) was proposed.
", "votes": null, "replies": [ { "id": 1827819, "author_name": "carlmcbrideellis", "author_url": "", "post_date": "06/21/2022 10:09:33", "content": "Dear @lucasmorin
\nI am completely unfamiliar with \"Hierarchical Risk Parity\", do you have any links?
\nAll the best,
\ncarl
HRP Paper: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2708678
\nHRP Slides: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2713516
I've seen the idea of applying portfolio selection approach to model ensembling as the problem are relatively similar (can't find a good ressource right now).
", "votes": null, "replies": [] } ] }, { "id": 1828000, "author_name": "thedevastator", "author_url": "", "post_date": "06/21/2022 12:50:08", "content": "We demonstrate the real work behind winning any Kaggle competition is actually accomplished through a human-computer collaboration. Teams are treated as individual weak learners who compete against one another by deriving optimized estimation solutions on the public leaderboard. Mutually and unconsciously, competitors in turn capitalize on the strengths and mitigate the weaknesses of other competitors by re-reporting their findings, when necessary, and ultimately developing their own solutions. By assembling the individual solutions together, we obtain an ideal solution, which is almost optimal. This \"Random Kaggle estimator\" enables us to demonstrate how humans, as part of an ML system, significantly contribute to its overall ability to solve complex problems. The more competitors, the more distinct differences there are between them (a min. two requisite), and the greater the leaderboard fluctuations, the faster and better the meta-estimation solution can be developed.
\nFurthermore, we define a measure of the degree to which random actions by humans are required to accomplish a task relative to the task's complexity - \"Stochastic Leaderboard Descent\". We will demonstrate that the more complex a problem-solving task requires a greater number of independent teams working in unison to accomplish the ML task, with increasing importance placed on each team as the degree of stochasticity increases. We also show that although some of the ideas and code do not originate from the same competition (or are from the same competition at a different time), there are still ideas that we can not attribute to being influenced by any specific historical source.
\nSuggesting that such ideas result from pure creativity and are driven by intuition, a human strength that extends far beyond ML tasks.
\nIn the world of Kaggle, very often only the top spot on the leaderboard matters. On the other hand, once the leaderboard is released, it becomes a valuable tool for researchers to learn from and obtain new insights, especially in competitions with a sufficient number of competitors.
\nIn this paper, we show that there is much more to leaderboards than the winner's name. Using a recent Kaggle competition as an example, we illustrate that competitors are able to adapt to one another and progressively improve their leaderboard score by benefiting from the cumulative effect of multiple contributions. We formulate a \"Random Kaggle estimator\" which, although simple, is able to reproduce the leaderboard evolution curve and we derive some insights from this model.
\nWe then demonstrate how the leaderboard evolution curve itself can be used to define an optimal estimation model, which we call the \"stochastic leaderboard descent\".
\nDefinition 1: Solution Blending
\n\n\nThe process of assembling the predictions of multiple solutions in order to produce a new and improved estimate of the target value.
\n
Therefore, it follows from Definition 1 that a blended solution is the combination of multiple solutions that can be assembled to produce a more accurate estimate of the target value. As the blended solution is a combination of multiple solutions, it is a collaborative ensemble of solutions. Therefore, we can define a blended solution as follows:
\n$$\\beta(x_i) = \\sum_j \\alpha_j(x_i) x_j(x_i)$$
\nWhere β is the blended solution, 𝛼 is the solution weight and x_j(x_i) is the prediction for the training instance x_i.
\nIdeas Blending
\nBefore we define we must first define Idea:
\nData scientists come up with a metric and then optimize that metric.
\nAs we can see the information of the target outcome is linked to both the solution and the methods used to optimize the solution. We can represent a solution as follows:
\n$$x_i(z) = \\Phi_i(z)\\phi_i(z)$$
\nWhere z is the data, Φ is the encoder (Optimization method) and 𝜙 is the decoder (model), the feature-based solution.
\nLastly, we are able to formulate the idea of data scientist as:
\n$$y_i(z) = M(x_i(z))$$
\nWhere y_i(z) is the data science outcome and M is the information linking the respective solution to the metric.
\nDefinition 2: Ideas Blending
\n\n\nThe process of assembling the predictions of multiple ideas in order to produce a new and improved estimate of the target value.
\n
From Definition 1, we can define Ideas Blending as:
\n$$y_i = \\arg \\min_{\\beta(z)} \\left[\\sum_j \\alpha_j y_j(z)\\right]$$
\nSince the scientists try to blend ideas in order to minimize his leaderboard position.
\nDefinition 3: Stochastic Leaderboard Convergence (SLC)
\n\n\nA Stochastic Leaderboard Convergence is a process of obtaining an improved estimate of the target value by re-estimating the target value after a new public leaderboard score has been reported.
\n
It follows from Definition 1 that a Stochastic Leaderboard Convergence (SLC) is a process of obtaining an improved estimate of the target value by re-estimating the target value after a new public leaderboard score has been reported. The idea behind SLC is that the more estimates of the target value are reported on the public leaderboard, the better the estimate of the target value will be.
\nWe can therefore formulate the Stochastic Leaderboard Convergence (SLC) as follows:
\n$$\\hat{y} = \\arg \\min_{\\beta(x_i)} \\left[\\sum_i \\left[\\sum_j \\alpha_j(x_i) x_j(x_i) - y_i\\right]^2\\right]$$
\nDefinition 4: Public Leaderboard Density
\n\n\nThe density of the number of public leaderboard scores reported.
\n
The Public Leaderboard Density (PLD) is the number of reported public leaderboard scores per unit area (competition timeframe). We can formulate the PLD as follows:
\n$$PLD = \\frac{N}{T}$$
\nWhere N is the number of public leaderboard scores reported, and T is the competition timeframe.
\nDefinition 4: Stochastic Leaderboard Descent (SLD)
\n\n\nA Stochastic Leaderboard Descent is a process of obtaining an improved estimate of the target value by taking the current public leaderboard score as the starting point and applying stochastic gradient descent to obtain a better estimate of the target value.
\n
Bayesian Blending
\nBayesian Blending is a process of obtaining an improved estimate of the target value by taking the current top public solution as the starting point and applying stochastic notebook descent to obtain a better estimate of the target value.
\nSince there are:
We propse to define the blended solution as a linear combination of both a solution term and an idea term:
$$\\omega(x_i) = \\sum_j \\alpha_j n_j(z) + \\sum_k \\beta_k M(x_k(z))$$
\nWhere n_j is the notebook's code used (function of the data z) and M(x_k(z)) again is the information extracted from an idea x_i (function of the competition's data). The weights 𝛼 and 𝛽 are the probabilistic weight of the notebooks and ideas.
\nWe therefore can define the direction of the kaggle bayesian blending process as the gradient of the notebook-idea-blended solution.
\n$$\\nabla_k(z) = \\sum_j \\alpha_j(z) \\nabla_k n_j(z) + \\sum_j \\beta_j(z) \\nabla_k M(x_j(z))$$
\nTherefore, the notebook idea blended solution can be formulated as a kernel function:
\n$$K_j(z,z') = \\nabla_k(z) x_j(z')$$
\nTo optimize the kernel function, we can introduce a kernel matrix of all the kernels and their respective gradients for each notebook-idea-blended solution.
\n$$\\hat{K}_{jk}(z,z') = \\nabla_k(z) x_j(z')$$
\nHistorical influence can sometime be attributed to some public submissions. Also with the recent social activity made popular by covid lockdown elimination this type of influence becomes more and more of a factor. Thus, from second law of medaldynamics we can assume that just like entropy, ideas flow from low medal density areas to high medal density areas, so we can define the past idea-solution as:
\n$$n_j(z) = \\sum_i \\nabla_{i}(z)$$
\nWhere 𝛼 is the coefficient for idea divergence (The second law of medaldynamics) and 𝛽 is the past idea-solution.
\nWe can therefore define the emergence of creativity as the process of extracting the solution that is orthogonal to past influence.
", "votes": null, "replies": [ { "id": 1828038, "author_name": "carlmcbrideellis", "author_url": "", "post_date": "06/21/2022 13:03:12", "content": "Dear @thedevastator
\nEr, wow! I guess that is why you are called \"The Devastator\"… 😃
\nThat said, somewhat more than your \"Regularized Stochastic Leaderboard Descent\"(!) I am sort of reminded of the genetic algorithms that were very much the fashion some 20-30 years ago….
\nAll the best,
\ncarl
You should get likes simply for attempting to write this!!🙌
", "votes": null, "replies": [] }, { "id": 1828408, "author_name": "thedevastator", "author_url": "", "post_date": "06/21/2022 17:02:09", "content": "Thank you, Thank you 😁
", "votes": null, "replies": [] }, { "id": 1832792, "author_name": "saurabhbagchi", "author_url": "", "post_date": "06/25/2022 11:54:52", "content": "Great work @thedevastator, upvoted :)
", "votes": null, "replies": [] } ] }, { "id": 1828351, "author_name": "susnato", "author_url": "", "post_date": "06/21/2022 16:20:54", "content": "Kagglers…Assemble.
", "votes": null, "replies": [ { "id": 1836195, "author_name": "mohammadrahmati", "author_url": "", "post_date": "06/28/2022 12:50:06", "content": "nice one…!
", "votes": null, "replies": [] } ] }, { "id": 1828405, "author_name": "christoffer", "author_url": "", "post_date": "06/21/2022 16:58:26", "content": "IIRC, in the past, organizers have created aggregate models from submissions after the end of some competitions. I don’t remember which competititions but it might have been some of the basketball or american football ones.
", "votes": null, "replies": [ { "id": 1828419, "author_name": "carlmcbrideellis", "author_url": "", "post_date": "06/21/2022 17:11:18", "content": "Dear @christoffer
\nThat is interesting. Although from an organizers point of view it is worth distinguishing between ensembling a diverse set of models using the provided code (as per the \"Kaggle Ensembling Guide\") which can (could? \"Netflix Never Used Its $1 Million Algorithm Due To Engineering Costs\") be beneficial to the organizers, as opposed to the blending of raw submission.csv files together, which may also do well on a kaggle competition, but on the other hand provide little (basically nothing) of use to the organizers, as they are solutions specific to the dataset provided. (PS: I am a little surprised that this is not a \"Code\" competition).
All the best,
\ncarl
Thanks for that link. I've been learning much more about ensembles and missed that resource.
", "votes": null, "replies": [] }, { "id": 1829985, "author_name": "carlmcbrideellis", "author_url": "", "post_date": "06/23/2022 04:54:26", "content": "Dear @edwardakalarrywelch
\nMotivated by your comment I have just posted the link as a Topic. As this is the first big \"Featured\" non-forecasting tabular competition in quite a while, and could hit 4-5k participants, and there will be quite a few people who have not seen \"kaggle ensembling\" in the text books. Furthermore, the site MLwave went down about a year ago, but thankfully the content was rescued by the Wayback Machine.
\nAll the best,
\ncarl
Beautifully written. I scout the site for technical insights but, occasionally, I stumble on such meta posts which make the process not just useful, but also pleasurable.
", "votes": null, "replies": [] }, { "id": 1830437, "author_name": "heyspaceturtle", "author_url": "", "post_date": "06/23/2022 12:49:08", "content": "People at AMEX were probably like:
\n— Hey we need an insane model for this business problem…
\n— Lets just fit a Kaggle Random Forest.
Hola @heyspaceturtle
\nMaybe, but it takes 3 months to run, and doesn't come cheap with $100,000 for prize money and an unknown amount in compute + storage…
\nThat said, credit default could cost somebody like American Express $M each year, so even a slightly better model could be worth the outlay.
\nUn saludo muy codrial,
\ncarl
True… but they still get the chance to find talented data scientists for their team :)
\nSaludos!
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1827527": "Just a shower thought whilst running code; although ostensibly each of us are working alone to win this competition, we are in some way forming the \"Random Kaggle estimator\".\n\nEach team is a \"weak\" learner (not meant in any pejorative sense) akin to a decision tree estimator. However, collectively we are like the strong Random Forest leading to an almost optimal solution, using the Public Leaderboard as our objective function. We are very much guided by the work of others, either directly (for example forking and tweaking public notebooks) or indirectly (ideas as to what to explore further, or what others have already tried and do not seem to work, or even simply observing a sudden jump in the top LB score indicating that there is some *magic* to be found). The ideas and code do not even have to originate from this competition, but maybe are from a similar kaggle competition in the (perhaps distant) past, and some of the more recent Tabular Playground Series write-ups are astounding and truly inspiring!\n\nIt is this 'human-in-the-loop' meta-estimator that can be found only on kaggle! \n\nAnyway, I think my script has just finished, so back to work...\n\nGood luck to all; the better you do, the better everyone does!", "1827661": "if all team can made their submission csv file public .... then it will be interesting", "1827673": "Dear @hengck23 \n\nAmusing idea! One thing about stand-alone `submission.csv` files is that they are a bit like sausages; you don't know what went into making them. Indeed perhaps all closed-source blending notebooks should even come with a *caveat emptor*. For example, they must clearly say on the packet whether they have [`B_29`](https://www.kaggle.com/competitions/amex-default-prediction/discussion/328756) added into the mix or not 😃\n\nAll the best,\ncarl", "1827784": "We are all part of a big ensemble.", "1827816": "Hi @carlmcbrideellis I would be curious if there is some gain in averaging lots of score or if it is somehow capped. I would be also curious of innovative weighting scheme that could be used. I know that some market finance techniques (Hierarchical Risk Parity) was proposed.", "1827819": "Dear @lucasmorin \n\nI am completely unfamiliar with \"Hierarchical Risk Parity\", do you have any links?\n\nAll the best,\ncarl", "1827849": "well at least `B_29` is at the bottom of the list in terms how useful it is: )", "1827856": "Dear @raddar \n\nOnly for the training data and the Public chunk; ¿what happens when it 'kicks-in' on the Private data is the question?\n\nThat said, if the estimators they use employ just a smidgen of regularization (*i.e.* L2, or perhaps better still L1) they probably create a model that diminishes (L2) or disregards (L1) the feature `B_29` anyway...\n\nAll the best,\ncarl", "1827938": "HRP Paper: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2708678\nHRP Slides: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2713516\n\nI've seen the idea of applying portfolio selection approach to model ensembling as the problem are relatively similar (can't find a good ressource right now).", "1828000": "### Kaggle Random Walk\n##### Regularized Stochastic Leaderboard Descent\n_____\n\n### Abstract\nWe demonstrate the real work behind winning any Kaggle competition is actually accomplished through a human-computer collaboration. Teams are treated as individual weak learners who compete against one another by deriving optimized estimation solutions on the public leaderboard. Mutually and unconsciously, competitors in turn capitalize on the strengths and mitigate the weaknesses of other competitors by re-reporting their findings, when necessary, and ultimately developing their own solutions. By assembling the individual solutions together, we obtain an ideal solution, which is almost optimal. This \"Random Kaggle estimator\" enables us to demonstrate how humans, as part of an ML system, significantly contribute to its overall ability to solve complex problems. The more competitors, the more distinct differences there are between them (a min. two requisite), and the greater the leaderboard fluctuations, the faster and better the meta-estimation solution can be developed.\n\nFurthermore, we define a measure of the degree to which random actions by humans are required to accomplish a task relative to the task's complexity - \"Stochastic Leaderboard Descent\". We will demonstrate that the more complex a problem-solving task requires a greater number of independent teams working in unison to accomplish the ML task, with increasing importance placed on each team as the degree of stochasticity increases. We also show that although some of the ideas and code do not originate from the same competition (or are from the same competition at a different time), there are still ideas that we can not attribute to being influenced by any specific historical source.\n\nSuggesting that such ideas result from pure creativity and are driven by intuition, a human strength that extends far beyond ML tasks.\n\n#### Introduction\n\nIn the world of Kaggle, very often only the top spot on the leaderboard matters. On the other hand, once the leaderboard is released, it becomes a valuable tool for researchers to learn from and obtain new insights, especially in competitions with a sufficient number of competitors.\n\nIn this paper, we show that there is much more to leaderboards than the winner's name. Using a recent Kaggle competition as an example, we illustrate that competitors are able to adapt to one another and progressively improve their leaderboard score by benefiting from the cumulative effect of multiple contributions. We formulate a \"Random Kaggle estimator\" which, although simple, is able to reproduce the leaderboard evolution curve and we derive some insights from this model.\n\nWe then demonstrate how the leaderboard evolution curve itself can be used to define an optimal estimation model, which we call the \"stochastic leaderboard descent\".\n\n### Preliminaries\n\n#### Generalized Blending Theory\n\n**Definition 1: Solution Blending**\n> The process of assembling the predictions of multiple solutions in order to produce a new and improved estimate of the target value.\n\nTherefore, it follows from Definition 1 that a blended solution is the combination of multiple solutions that can be assembled to produce a more accurate estimate of the target value. As the blended solution is a combination of multiple solutions, it is a collaborative ensemble of solutions. Therefore, we can define a blended solution as follows:\n\n$$\\beta(x_i) = \\sum_j \\alpha_j(x_i) x_j(x_i)$$\n\nWhere *β* is the blended solution, *𝛼* is the solution weight and *x_j(x_i)* is the prediction for the training instance *x_i*.\n\n**Ideas Blending**\n\nBefore we define we must first define *Idea*:\n\nData scientists come up with a metric and then optimize that metric.\n\nAs we can see the information of the target outcome is linked to both the solution and the methods used to optimize the solution. We can represent a solution as follows:\n\n$$x_i(z) = \\Phi_i(z)\\phi_i(z)$$\n\nWhere *z* is the data, *Φ* is the encoder (Optimization method) and 𝜙 is the decoder (model), the feature-based solution.\n\nLastly, we are able to formulate the idea of data scientist as:\n\n$$y_i(z) = M(x_i(z))$$\n\nWhere *y_i(z)* is the data science outcome and *M* is the information linking the respective solution to the metric.\n\n**Definition 2: Ideas Blending**\n> The process of assembling the predictions of multiple ideas in order to produce a new and improved estimate of the target value.\n\nFrom Definition 1, we can define *Ideas Blending* as:\n\n$$y_i = \\arg \\min_{\\beta(z)} \\left[\\sum_j \\alpha_j y_j(z)\\right]$$\n\nSince the scientists try to blend ideas in order to minimize his leaderboard position.\n\n**Definition 3: Stochastic Leaderboard Convergence (SLC)**\n> A Stochastic Leaderboard Convergence is a process of obtaining an improved estimate of the target value by re-estimating the target value after a new public leaderboard score has been reported.\n\nIt follows from Definition 1 that a Stochastic Leaderboard Convergence (SLC) is a process of obtaining an improved estimate of the target value by re-estimating the target value after a new public leaderboard score has been reported. The idea behind SLC is that the more estimates of the target value are reported on the public leaderboard, the better the estimate of the target value will be.\n\nWe can therefore formulate the Stochastic Leaderboard Convergence (SLC) as follows:\n\n$$\\hat{y} = \\arg \\min_{\\beta(x_i)} \\left[\\sum_i \\left[\\sum_j \\alpha_j(x_i) x_j(x_i) - y_i\\right]^2\\right]$$\n\n**Definition 4: Public Leaderboard Density**\n> The density of the number of public leaderboard scores reported.\n\nThe Public Leaderboard Density (PLD) is the number of reported public leaderboard scores per unit area (competition timeframe). We can formulate the PLD as follows:\n\n$$PLD = \\frac{N}{T}$$\n\nWhere *N* is the number of public leaderboard scores reported, and *T* is the competition timeframe.\n\n**Definition 4: Stochastic Leaderboard Descent (SLD)**\n> A Stochastic Leaderboard Descent is a process of obtaining an improved estimate of the target value by taking the current public leaderboard score as the starting point and applying stochastic gradient descent to obtain a better estimate of the target value.\n\n#### Leaderboard Monte Carlo Estimation\n\n**Bayesian Blending**\n\nBayesian Blending is a process of obtaining an improved estimate of the target value by taking the current top public solution as the starting point and applying stochastic notebook descent to obtain a better estimate of the target value.\nSince there are: \n\n- **Solutions** released as notebooks\n- **Ideas** released on the discussions forums\n\nWe propse to define the blended solution as a linear combination of both a `solution term` and an `idea term`:\n\n$$\\omega(x_i) = \\sum_j \\alpha_j n_j(z) + \\sum_k \\beta_k M(x_k(z))$$\n\nWhere *n_j* is the notebook's code used (function of the data *z*) and *M(x_k(z))* again is the information extracted from an idea *x_i* (function of the competition's data). The weights *𝛼* and *𝛽* are the probabilistic weight of the notebooks and ideas.\n\nWe therefore can define the direction of the kaggle bayesian blending process as the gradient of the notebook-idea-blended solution.\n\n$$\\nabla_k(z) = \\sum_j \\alpha_j(z) \\nabla_k n_j(z) + \\sum_j \\beta_j(z) \\nabla_k M(x_j(z))$$\n\nTherefore, the notebook idea blended solution can be formulated as a kernel function:\n\n$$K_j(z,z') = \\nabla_k(z) x_j(z')$$\n\nTo optimize the kernel function, we can introduce a kernel matrix of all the kernels and their respective gradients for each notebook-idea-blended solution.\n\n$$\\hat{K}_{jk}(z,z') = \\nabla_k(z) x_j(z')$$\n\n#### Past solution influence\n\nHistorical influence can sometime be attributed to some public submissions. Also with the recent social activity made popular by covid lockdown elimination this type of influence becomes more and more of a factor. Thus, from second law of medaldynamics we can assume that just like entropy, ideas flow from low medal density areas to high medal density areas, so we can define the past idea-solution as:\n\n$$n_j(z) = \\sum_i \\nabla_{i}(z)$$\n\nWhere *𝛼* is the coefficient for idea divergence *(The second law of medaldynamics)* and *𝛽* is the past idea-solution.\n\nWe can therefore define the **emergence of creativity** as the process of extracting the solution that is orthogonal to past influence.", "1828038": "Dear @thedevastator \n\nEr, wow! I guess that is why you are called \"The Devastator\"... 😃\n\nThat said, somewhat more than your \"*Regularized Stochastic Leaderboard Descent*\"(!) I am sort of reminded of the [genetic algorithms](https://en.wikipedia.org/wiki/Genetic_algorithm) that were very much the fashion some 20-30 years ago....\n\nAll the best,\ncarl", "1828351": "Kagglers...Assemble.", "1828393": "You should get likes simply for attempting to write this!!🙌", "1828405": "IIRC, in the past, organizers have created aggregate models from submissions after the end of some competitions. I don’t remember which competititions but it might have been some of the basketball or american football ones.", "1828408": "Thank you, Thank you 😁", "1828419": "Dear @christoffer \n\nThat is interesting. Although from an organizers point of view it is worth distinguishing between ensembling a diverse set of models using the provided code (as per the [\"Kaggle Ensembling Guide\"](https://web.archive.org/web/20160304031055/http://mlwave.com/kaggle-ensembling-guide/)) which can (could? [\"*Netflix Never Used Its $1 Million Algorithm Due To Engineering Costs*\"](https://www.wired.com/2012/04/netflix-prize-costs/)) be beneficial to the organizers, as opposed to the blending of raw `submission.csv` files together, which may also do well on a kaggle competition, but on the other hand provide little (basically nothing) of use to the organizers, as they are solutions specific to the dataset provided. (PS: I am a little surprised that this is not a \"Code\" competition).\n\nAll the best,\ncarl", "1829214": "Thanks for that link. I've been learning much more about ensembles and missed that resource.", "1829757": "Beautifully written. I scout the site for technical insights but, occasionally, I stumble on such meta posts which make the process not just useful, but also pleasurable.", "1829985": "Dear @edwardakalarrywelch \n\nMotivated by your comment I have just posted the link as a Topic. As this is the first big \"Featured\" non-forecasting tabular competition in quite a while, and could hit 4-5k participants, and there will be quite a few people who have not seen \"kaggle ensembling\" in the text books. Furthermore, the site MLwave went down about a year ago, but thankfully the content was rescued by the Wayback Machine. \n\nAll the best,\ncarl", "1830437": "People at AMEX were probably like: \n— Hey we need an insane model for this business problem...\n— Lets just fit a Kaggle Random Forest.", "1830440": "Hola @heyspaceturtle \n\nMaybe, but it takes 3 months to run, and doesn't come cheap with $100,000 for prize money and an unknown amount in compute + storage...\n\nThat said, credit default could cost somebody like American Express $M each year, so even a slightly better model could be worth the outlay.\n\nUn saludo muy codrial,\ncarl", "1830464": "True... but they still get the chance to find talented data scientists for their team :) \n\nSaludos!", "1832792": "Great work @thedevastator, upvoted :)", "1836195": "nice one...!" }, "source": "meta" }