{ "id": 240342, "title": "2nd Place Solution (Overall)", "url": "/competitions/indoor-location-navigation/writeups/myrcj-2nd-place-solution-overall", "author_name": "", "post_date": "2022-02-21T00:42:05.003Z", "votes": 59, "comment_count": 10, "views": 0, "content": "

Thank you my briliant teammates ( @ymatioun, @vaghefi, @demonen, @rsakata ), all teams who competed with us, and all people who participated in this competition. Congrats @tvdwiele @areehdot @dott1718, who won this competition with an incredible performance!

\n

I'm happy because I finally became Kaggle Grandmaster with 5 gold medals, and @ymatioun became kaggle Master in this competition.
\nIt's my 5th competition and it was fun to compete with my past teammate ( @vicensgaitan ) and people who I competed with past competitions ( @takoihiraokazu )!

\n

Here, I will explain the overall approach of our team's solution. For more details of (especially) absolute position prediction models and delta prediction models, please check our teammate's solutions.

\n\n

Code and the visualization of our prediction are here.

\n\n

For our solution, our team does not use any leakage (e.g. start/end point leakage, raw timestamp leakage, device leakage). We tried to use these leakages, but none of them improved the score and sometimes degraded the score, because our floor model and our WIFI model are already very good without leakage.

\n

Validation Set Construction

\n

By EDA, I found the samples of the test set meet these conditions:

\n
    \n
  1. No non-standard floor is used, as stated in the evaluation section.
    2. \n
  2. Brand is always OPPO.
    3. \n
  3. max(timestamp) - min(timestamp) is larger than 60310.
    4. \n
  4. TYPE_ACCELEROMETER_acc is not all nan.
    5. \n
  5. TYPE_ACCELEROMETER_UNCALIBRATED_acc is not all nan.
    6. \n
  6. TYPE_GYROSCOPE_UNCALIBRATED_acc is not all nan.
    7. \n
  7. TYPE_WIFI_ssid is not all nan.
    8. \n
\n

I made these 7 masks and mulitiply them and get one mask. I chose the samples from training set using this mask. Then, from those samples, I chose 548 paths and use these 548 paths as validation set. LB and CV was always correlated using this validation set. I guess the host wants to use the good and clean paths for the evaluation of the performance, which is reasonable.

\n

Absolute/Delta Prediction Models

\n

We have 5 absolute position prediction models, and 3 delta prediction models.
\nThese are the lists of our absolute position prediction models:

\n\n

These are the lists of our delta prediction models:

\n\n

For our ensembled delta prediction, we simply tried weighted average of these 3 models:
\nensemble_delta_prediction = Mamas_MLP * 0.2 + Youri_PDR * 0.33 + Christoffer_LSTM * 0.47
\nOur delta prediction is not so good, MAE = 1.06 in CV.

\n

For our ensembled absolute position prediction, we tried weighted average of these 5 models after applying akio's cost minimization using the ensembled delta prediction. When applying akio's cost minimization to Youri's Fingerprinting model and Reza's WKNN models, we consider the uncertainty of the prediction in akio's cost minimization. To be more specific, I modified the code of akio's cost minimization as follows.

\n
def get_xy_star_weight(delta_xy_hat, xy_hat, a, b, w):\n    N = xy_hat.shape[0]\n    alpha = w * a * np.ones(N)\n    beta  = b * np.ones_like(delta_xy_hat[:, 0])\n    A = scipy.sparse.spdiags(alpha, [0], N, N)\n    B = scipy.sparse.spdiags( beta, [0], N-1, N-1)\n    D = scipy.sparse.spdiags(np.stack([-np.ones(N), np.ones(N)]), [0, 1], N-1, N)\n    Q = A + (D.T @ B @ D)\n    c = (A @ xy_hat) + (D.T @ (B @ delta_xy_hat))\n    A = scipy.sparse.block_diag((Q, Q))\n    b = np.r_[c[:,0], c[:,1]]\n    xy_star = scipy.optimize.nnls(A.toarray(), b)[0].reshape(2,-1).T\n    return xy_star\n
\n

where w is given by

\n
w = (Reza or Youri's prediction's uncertainty) ** (-2)\n
\n

Finally, the ensembled prediction is given by

\n
ensembled_absolute_prediction = akio_costmin(mamas_RNN) * 0.04 + akio_weighted_costmin(youri_fingerprint) * 0.17 + akio_weighted_costmin(reza_WKNN_euclid) * 0.11 + akio_costmin(Jack_multiclass) * 0.34 + akio_weighted_costmin(reza_WKNN_corr) * 0.34\n
\n

The ensembled absolute prediction scored 4.206 in CV and 3.171 in LB.

\n

Iterative snap to corridor & akio's cost minimization

\n

After the weighted averaging, We tried to move the prediction to the corridor. Because we didn't use additional generated waypoints unlike the 1st team, the predictions before the discrete optimization must be in the corridor, because some of these predictions will be used as the final prediction when no grid is selected in the discrete optimization process.
\nFirst, we fixed the incorrect map according to @chris62's great thread. Then, Christoffer automatically eliminated the stores on which the grids are present. This is the example of the problematic cases: The left image shows the map before the Christoffer's fixing map and the right image shows the map after the Christoffer's fixing map.
\n
\nThis is the only case we didn't eliminate the store, because we thought it can give a bad effect in snap-to-corridor. The effect should be smaller than 0.005, though.
\n

\n

After fixing the maps, We applied snap to corridor and the akio's cost minimization iteratively (100 iterations). After the iterative snap to corridor & akio's cost minimization, CV score is 3.882 and LB score is 2.740.

\n

Discrete Optimization

\n

Like the 1st place solution, Discrete optimization using proper penalties is the true key of this competition. We used these 6 penalties and corresponding 6 parameters (alpha, beta_1, beta_2, gamma, kappa, tau). However, kappa and tau is not so important and they didn't improve the score a lot. Gamma is really important and this is the reason our method worked well without additional generated grids.

\n
    \n
  1. alpha * (euclid distance between grids and the absolute prediction)
    2. \n
  2. beta_1 * abs(the angle of predicted delta - the angle of the delta between two grids.)
    3. \n
  3. beta_2 * abs(l2-norm of the predicted delta - l2-norm of the delta between two grids)
    4. \n
  4. gamma * (1 when no grid is chosen, 0 otherwise)
    5. \n
  5. kappa * abs(actual distance estimated by dijkstra algorithm - manhattan distance between 2 grids)
    6. \n
  6. tau * (1 when the delta between two grids are zero, 0 otherwise)
    7. \n
\n

The reason we estimated the actual distance using dijkstra algorithm between 2 grids are, the predictions are sometimes strange like this.
\n

\n

After applying a cost that considers actual distance, the prediction became like this.
\n(I'm sorry these predictions are not the one from our best submissions, but the very old predictions. These are just examples)
\n

\n

This is the example of the actual distance calculated by dijkstra algorithm (by Jack)
\n

\n

In the optimizaiton process, I used a simple greedy optimization algorithm, while the 1st team used the beam search, which is I guess one of the reason we lost. I have no experience in heuristics contest like Santa and couldn't do better in the optimization process. To be more specific, I split the paths by chunksize = 7, and I searched the 8 closest grids for each prediction. Then, I calculated the cost for all the possible patterns ((8 + 1) ** 7 = 4782969) and chose the grids (or sometimes original predictions) that minimize the cost. For the calculation of the backward delta of the first sample of a specific chunk, mamas's discrete optimization uses the last sample of the previous chunk.
\nAfter mamas's discrete optimization, CV score is 2.523 and LB score is 1.392, which is our final solution.

\n

Comments

\n

I told I would become the winner in the next competition in the 2nd place solution of Riiid competition, which was held 4 months ago, but I couldn't keep my promise. However, I have no regret because I did my best with really great teammates with the clean and completely no leakage solution and I really enjoyed this competition. Moreover, I learned a lot from @tvdwiele's team's great solution, which I think is one of the greatest solution I saw in the kaggle.
\nI agree with Reza that hiding LB score should not become a habit in Kaggle competitions, because if we all hide a score the competition will become not enjoyable for participants and probably will give a not good effect on the kaggle community. However, I sincerely respect the 1st team, because I'm sure we couldn't have won if they didn't hide a score, and their great solution is clearly a tremendous contribution to the community!
\nAnyway, I really enjoyed this competition. I will take a rest for a while, and will surely come back to Kaggle again !!!

\n

Thanks all, see you again!

", "messages": [ { "id": "1314827", "postDate": "05/19/2021 12:03:56", "content": "

Thank you my briliant teammates ( @ymatioun, @vaghefi, @demonen, @rsakata ), all teams who competed with us, and all people who participated in this competition. Congrats @tvdwiele @areehdot @dott1718, who won this competition with an incredible performance!

\n

I'm happy because I finally became Kaggle Grandmaster with 5 gold medals, and @ymatioun became kaggle Master in this competition.
\nIt's my 5th competition and it was fun to compete with my past teammate ( @vicensgaitan ) and people who I competed with past competitions ( @takoihiraokazu )!

\n

Here, I will explain the overall approach of our team's solution. For more details of (especially) absolute position prediction models and delta prediction models, please check our teammate's solutions.

\n\n

Code and the visualization of our prediction are here.

\n\n

For our solution, our team does not use any leakage (e.g. start/end point leakage, raw timestamp leakage, device leakage). We tried to use these leakages, but none of them improved the score and sometimes degraded the score, because our floor model and our WIFI model are already very good without leakage.

\n

Validation Set Construction

\n

By EDA, I found the samples of the test set meet these conditions:

\n
    \n
  1. No non-standard floor is used, as stated in the evaluation section.
    2. \n
  2. Brand is always OPPO.
    3. \n
  3. max(timestamp) - min(timestamp) is larger than 60310.
    4. \n
  4. TYPE_ACCELEROMETER_acc is not all nan.
    5. \n
  5. TYPE_ACCELEROMETER_UNCALIBRATED_acc is not all nan.
    6. \n
  6. TYPE_GYROSCOPE_UNCALIBRATED_acc is not all nan.
    7. \n
  7. TYPE_WIFI_ssid is not all nan.
    8. \n
\n

I made these 7 masks and mulitiply them and get one mask. I chose the samples from training set using this mask. Then, from those samples, I chose 548 paths and use these 548 paths as validation set. LB and CV was always correlated using this validation set. I guess the host wants to use the good and clean paths for the evaluation of the performance, which is reasonable.

\n

Absolute/Delta Prediction Models

\n

We have 5 absolute position prediction models, and 3 delta prediction models.
\nThese are the lists of our absolute position prediction models:

\n\n

These are the lists of our delta prediction models:

\n\n

For our ensembled delta prediction, we simply tried weighted average of these 3 models:
\nensemble_delta_prediction = Mamas_MLP * 0.2 + Youri_PDR * 0.33 + Christoffer_LSTM * 0.47
\nOur delta prediction is not so good, MAE = 1.06 in CV.

\n

For our ensembled absolute position prediction, we tried weighted average of these 5 models after applying akio's cost minimization using the ensembled delta prediction. When applying akio's cost minimization to Youri's Fingerprinting model and Reza's WKNN models, we consider the uncertainty of the prediction in akio's cost minimization. To be more specific, I modified the code of akio's cost minimization as follows.

\n
def get_xy_star_weight(delta_xy_hat, xy_hat, a, b, w):\n    N = xy_hat.shape[0]\n    alpha = w * a * np.ones(N)\n    beta  = b * np.ones_like(delta_xy_hat[:, 0])\n    A = scipy.sparse.spdiags(alpha, [0], N, N)\n    B = scipy.sparse.spdiags( beta, [0], N-1, N-1)\n    D = scipy.sparse.spdiags(np.stack([-np.ones(N), np.ones(N)]), [0, 1], N-1, N)\n    Q = A + (D.T @ B @ D)\n    c = (A @ xy_hat) + (D.T @ (B @ delta_xy_hat))\n    A = scipy.sparse.block_diag((Q, Q))\n    b = np.r_[c[:,0], c[:,1]]\n    xy_star = scipy.optimize.nnls(A.toarray(), b)[0].reshape(2,-1).T\n    return xy_star\n
\n

where w is given by

\n
w = (Reza or Youri's prediction's uncertainty) ** (-2)\n
\n

Finally, the ensembled prediction is given by

\n
ensembled_absolute_prediction = akio_costmin(mamas_RNN) * 0.04 + akio_weighted_costmin(youri_fingerprint) * 0.17 + akio_weighted_costmin(reza_WKNN_euclid) * 0.11 + akio_costmin(Jack_multiclass) * 0.34 + akio_weighted_costmin(reza_WKNN_corr) * 0.34\n
\n

The ensembled absolute prediction scored 4.206 in CV and 3.171 in LB.

\n

Iterative snap to corridor & akio's cost minimization

\n

After the weighted averaging, We tried to move the prediction to the corridor. Because we didn't use additional generated waypoints unlike the 1st team, the predictions before the discrete optimization must be in the corridor, because some of these predictions will be used as the final prediction when no grid is selected in the discrete optimization process.
\nFirst, we fixed the incorrect map according to @chris62's great thread. Then, Christoffer automatically eliminated the stores on which the grids are present. This is the example of the problematic cases: The left image shows the map before the Christoffer's fixing map and the right image shows the map after the Christoffer's fixing map.
\n
\nThis is the only case we didn't eliminate the store, because we thought it can give a bad effect in snap-to-corridor. The effect should be smaller than 0.005, though.
\n

\n

After fixing the maps, We applied snap to corridor and the akio's cost minimization iteratively (100 iterations). After the iterative snap to corridor & akio's cost minimization, CV score is 3.882 and LB score is 2.740.

\n

Discrete Optimization

\n

Like the 1st place solution, Discrete optimization using proper penalties is the true key of this competition. We used these 6 penalties and corresponding 6 parameters (alpha, beta_1, beta_2, gamma, kappa, tau). However, kappa and tau is not so important and they didn't improve the score a lot. Gamma is really important and this is the reason our method worked well without additional generated grids.

\n
    \n
  1. alpha * (euclid distance between grids and the absolute prediction)
    2. \n
  2. beta_1 * abs(the angle of predicted delta - the angle of the delta between two grids.)
    3. \n
  3. beta_2 * abs(l2-norm of the predicted delta - l2-norm of the delta between two grids)
    4. \n
  4. gamma * (1 when no grid is chosen, 0 otherwise)
    5. \n
  5. kappa * abs(actual distance estimated by dijkstra algorithm - manhattan distance between 2 grids)
    6. \n
  6. tau * (1 when the delta between two grids are zero, 0 otherwise)
    7. \n
\n

The reason we estimated the actual distance using dijkstra algorithm between 2 grids are, the predictions are sometimes strange like this.
\n

\n

After applying a cost that considers actual distance, the prediction became like this.
\n(I'm sorry these predictions are not the one from our best submissions, but the very old predictions. These are just examples)
\n

\n

This is the example of the actual distance calculated by dijkstra algorithm (by Jack)
\n

\n

In the optimizaiton process, I used a simple greedy optimization algorithm, while the 1st team used the beam search, which is I guess one of the reason we lost. I have no experience in heuristics contest like Santa and couldn't do better in the optimization process. To be more specific, I split the paths by chunksize = 7, and I searched the 8 closest grids for each prediction. Then, I calculated the cost for all the possible patterns ((8 + 1) ** 7 = 4782969) and chose the grids (or sometimes original predictions) that minimize the cost. For the calculation of the backward delta of the first sample of a specific chunk, mamas's discrete optimization uses the last sample of the previous chunk.
\nAfter mamas's discrete optimization, CV score is 2.523 and LB score is 1.392, which is our final solution.

\n

Comments

\n

I told I would become the winner in the next competition in the 2nd place solution of Riiid competition, which was held 4 months ago, but I couldn't keep my promise. However, I have no regret because I did my best with really great teammates with the clean and completely no leakage solution and I really enjoyed this competition. Moreover, I learned a lot from @tvdwiele's team's great solution, which I think is one of the greatest solution I saw in the kaggle.
\nI agree with Reza that hiding LB score should not become a habit in Kaggle competitions, because if we all hide a score the competition will become not enjoyable for participants and probably will give a not good effect on the kaggle community. However, I sincerely respect the 1st team, because I'm sure we couldn't have won if they didn't hide a score, and their great solution is clearly a tremendous contribution to the community!
\nAnyway, I really enjoyed this competition. I will take a rest for a while, and will surely come back to Kaggle again !!!

\n

Thanks all, see you again!

", "rawMarkdown": "Thank you my briliant teammates ( @ymatioun, @vaghefi, @demonen, @rsakata ), all teams who competed with us, and all people who participated in this competition. Congrats @tvdwiele @areehdot @dott1718, who won this competition with an incredible performance!\n\nI'm happy because I finally became Kaggle Grandmaster with 5 gold medals, and @ymatioun became kaggle Master in this competition. \nIt's my 5th competition and it was fun to compete with my past teammate ( @vicensgaitan ) and people who I competed with past competitions ( @takoihiraokazu )!\n\nHere, I will explain the overall approach of our team's solution. For more details of (especially) absolute position prediction models and delta prediction models, please check our teammate's solutions. \n- Youri's part: https://www.kaggle.com/c/indoor-location-navigation/discussion/239880\n- Reza's part: https://www.kaggle.com/c/indoor-location-navigation/discussion/240197\n- Christoffer's part: https://www.kaggle.com/c/indoor-location-navigation/discussion/240025\n- Jack's part: https://www.kaggle.com/c/indoor-location-navigation/discussion/240141\n\nCode and the visualization of our prediction are here. \n- Discrete optimization code (simple version): https://www.kaggle.com/mamasinkgs/discrete-optimization-in-2nd-place-solution\n- The valid/test predictions of our team (public 1.41, private 2.18) : https://drive.google.com/drive/folders/1lxC0DYc9K84mKaNZoFdRVk9n8YCIrQbf\n\nFor our solution, **our team does not use any leakage (e.g. start/end point leakage, raw timestamp leakage, device leakage)**. We tried to use these leakages, but none of them improved the score and sometimes degraded the score, because our floor model and our WIFI model are already very good without leakage. \n \n# Validation Set Construction\nBy EDA, I found the samples of the test set meet these conditions: \n1. No non-standard floor is used, as stated in the evaluation section.\n2. Brand is always OPPO.\n3. max(timestamp) - min(timestamp) is larger than 60310.\n4. TYPE_ACCELEROMETER_acc is not all nan.\n5. TYPE_ACCELEROMETER_UNCALIBRATED_acc is not all nan.\n6. TYPE_GYROSCOPE_UNCALIBRATED_acc is not all nan.\n7. TYPE_WIFI_ssid is not all nan.\n\nI made these 7 masks and mulitiply them and get one mask. I chose the samples from training set using this mask. Then, from those samples, I chose 548 paths and use these 548 paths as validation set. LB and CV was always correlated using this validation set. I guess the host wants to use the good and clean paths for the evaluation of the performance, which is reasonable. \n\n# Absolute/Delta Prediction Models\nWe have 5 absolute position prediction models, and 3 delta prediction models. \nThese are the lists of our absolute position prediction models: \n- Reza's WKNN model (euclid distance)\n- Reza's WKNN model (correlation distance)\n- Jack's floor-level multiclass classification model\n- Youri's Fingerprinting model \n- Mamas's RNN model \n\nThese are the lists of our delta prediction models: \n- Youri's PDR model\n- Christoffer's LSTM\n- Mamas's MLP\n\nFor our ensembled delta prediction, we simply tried weighted average of these 3 models:\n`ensemble_delta_prediction = Mamas_MLP * 0.2 + Youri_PDR * 0.33 + Christoffer_LSTM * 0.47`\nOur delta prediction is not so good, MAE = 1.06 in CV.\n\nFor our ensembled absolute position prediction, we tried weighted average of these 5 models after applying [akio's cost minimization](https://www.kaggle.com/saitodevel01/indoor-post-processing-by-cost-minimization) using the ensembled delta prediction. When applying akio's cost minimization to Youri's Fingerprinting model and Reza's WKNN models, we consider the uncertainty of the prediction in akio's cost minimization. To be more specific, I modified the code of akio's cost minimization as follows. \n```\ndef get_xy_star_weight(delta_xy_hat, xy_hat, a, b, w):\n N = xy_hat.shape[0]\n alpha = w * a * np.ones(N)\n beta = b * np.ones_like(delta_xy_hat[:, 0])\n A = scipy.sparse.spdiags(alpha, [0], N, N)\n B = scipy.sparse.spdiags( beta, [0], N-1, N-1)\n D = scipy.sparse.spdiags(np.stack([-np.ones(N), np.ones(N)]), [0, 1], N-1, N)\n Q = A + (D.T @ B @ D)\n c = (A @ xy_hat) + (D.T @ (B @ delta_xy_hat))\n A = scipy.sparse.block_diag((Q, Q))\n b = np.r_[c[:,0], c[:,1]]\n xy_star = scipy.optimize.nnls(A.toarray(), b)[0].reshape(2,-1).T\n return xy_star\n```\nwhere w is given by\n```\nw = (Reza or Youri's prediction's uncertainty) ** (-2)\n```\nFinally, the ensembled prediction is given by \n```\nensembled_absolute_prediction = akio_costmin(mamas_RNN) * 0.04 + akio_weighted_costmin(youri_fingerprint) * 0.17 + akio_weighted_costmin(reza_WKNN_euclid) * 0.11 + akio_costmin(Jack_multiclass) * 0.34 + akio_weighted_costmin(reza_WKNN_corr) * 0.34\n```\nThe ensembled absolute prediction scored 4.206 in CV and 3.171 in LB.\n\n# Iterative snap to corridor & akio's cost minimization\nAfter the weighted averaging, We tried to move the prediction to the corridor. Because we didn't use additional generated waypoints unlike the 1st team, **the predictions before the discrete optimization must be in the corridor**, because some of these predictions will be used as the final prediction when no grid is selected in the discrete optimization process.\nFirst, we fixed the incorrect map according to @chris62's [great thread](https://www.kaggle.com/c/indoor-location-navigation/discussion/230558). Then, Christoffer automatically eliminated the stores on which the grids are present. This is the example of the problematic cases: The left image shows the map before the Christoffer's fixing map and the right image shows the map after the Christoffer's fixing map. \n\nThis is the only case we didn't eliminate the store, because we thought it can give a bad effect in snap-to-corridor. The effect should be smaller than 0.005, though. \n\n\nAfter fixing the maps, We applied snap to corridor and the akio's cost minimization iteratively (100 iterations). After the iterative snap to corridor & akio's cost minimization, CV score is 3.882 and LB score is 2.740. \n\n# Discrete Optimization\nLike the 1st place solution, Discrete optimization using proper penalties is the true key of this competition. We used these 6 penalties and corresponding 6 parameters (alpha, beta_1, beta_2, gamma, kappa, tau). However, kappa and tau is not so important and they didn't improve the score a lot. Gamma is really important and this is the reason our method worked well without additional generated grids. \n\n1. alpha * (euclid distance between grids and the absolute prediction)\n2. beta_1 * abs(**the angle of predicted delta** - **the angle of the delta between two grids**.)\n3. beta_2 * abs(**l2-norm of the predicted delta** - **l2-norm of the delta between two grids**)\n4. gamma * (1 when no grid is chosen, 0 otherwise)\n5. kappa * abs(**actual distance estimated by dijkstra algorithm** - **manhattan distance between 2 grids**)\n6. tau * (1 when the delta between two grids are zero, 0 otherwise)\n\nThe reason we estimated the actual distance using dijkstra algorithm between 2 grids are, the predictions are sometimes strange like this. \n\n\nAfter applying a cost that considers actual distance, the prediction became like this. \n(I'm sorry these predictions are not the one from our best submissions, but the very old predictions. These are just examples)\n\n\nThis is the example of the actual distance calculated by dijkstra algorithm (by Jack)\n\n\nIn the optimizaiton process, I used a simple greedy optimization algorithm, while the 1st team used the beam search, which is I guess one of the reason we lost. I have no experience in heuristics contest like Santa and couldn't do better in the optimization process. To be more specific, I split the paths by chunksize = 7, and I searched the 8 closest grids for each prediction. Then, I calculated the cost for all the possible patterns ((8 + 1) ** 7 = 4782969) and chose the grids (or sometimes original predictions) that minimize the cost. For the calculation of the backward delta of the first sample of a specific chunk, mamas's discrete optimization uses the last sample of the previous chunk.\nAfter mamas's discrete optimization, CV score is 2.523 and LB score is 1.392, which is our final solution. \n\n# Comments\nI told I would become the winner in the next competition in the [2nd place solution of Riiid competition](https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/210113), which was held 4 months ago, but I couldn't keep my promise. However, I have no regret because I did my best with really great teammates with the clean and completely no leakage solution and I really enjoyed this competition. Moreover, I learned a lot from @tvdwiele's team's great solution, which I think is one of the greatest solution I saw in the kaggle. \nI agree with Reza that hiding LB score should not become a habit in Kaggle competitions, because if we all hide a score the competition will become not enjoyable for participants and probably will give a not good effect on the kaggle community. However, I sincerely respect the 1st team, because I'm sure we couldn't have won if they didn't hide a score, and their great solution is clearly a tremendous contribution to the community!\nAnyway, I really enjoyed this competition. I will take a rest for a while, and will surely come back to Kaggle again !!! \n\n# Thanks all, see you again!", "votes": null }, { "id": "1314876", "postDate": "05/19/2021 12:26:45", "content": "

Congratulations on your second place!
\nAlso big congrats for became a Grand Master!

\n

I'm always impressed with your performance, amazing!
\nAs well as the attitude toward competitions.
\nI'm glad that I was able to compete with you… or just seeing from far from you. It's just enjoyable.

\n

I really hope that I could see you again on kaggle, thank you so much!

", "rawMarkdown": "Congratulations on your second place! \nAlso big congrats for became a Grand Master! \n\nI'm always impressed with your performance, amazing! \nAs well as the attitude toward competitions. \nI'm glad that I was able to compete with you... or just seeing from far from you. It's just enjoyable. \n\nI really hope that I could see you again on kaggle, thank you so much!", "votes": null }, { "id": "1314895", "postDate": "05/19/2021 12:41:55", "content": "

Thanks Kouki, your great notebook helped me a lot in the beginning of the competition.
\nI also hope I could see you again on kaggle!

", "rawMarkdown": "Thanks Kouki, your great notebook helped me a lot in the beginning of the competition. \nI also hope I could see you again on kaggle!", "votes": null }, { "id": "1315140", "postDate": "05/19/2021 15:36:25", "content": "

Thanks for sharing details solution. Learn alot from your team @mamasinkgs

", "rawMarkdown": "Thanks for sharing details solution. Learn alot from your team @mamasinkgs", "votes": null }, { "id": "1315438", "postDate": "05/19/2021 19:30:35", "content": "

Congratulations on your second place!😄

", "rawMarkdown": "Congratulations on your second place!😄", "votes": null }, { "id": "1315445", "postDate": "05/19/2021 19:40:50", "content": "

I uploaded discrete optimization code that runs within about 15 mins. This is a much simpler version, but achieves 1.49 public/ 2.24 private. https://www.kaggle.com/mamasinkgs/discrete-optimization-in-2nd-place-solution

", "rawMarkdown": "I uploaded discrete optimization code that runs within about 15 mins. This is a much simpler version, but achieves 1.49 public/ 2.24 private. https://www.kaggle.com/mamasinkgs/discrete-optimization-in-2nd-place-solution", "votes": null }, { "id": "1315488", "postDate": "05/19/2021 20:44:12", "content": "

Congratulations to the second place winner and grandmaster ! ! !

\n

Also, thank you for making the competition more exciting with your discussions (and tweet).
\nIf there is a competition next time you make it exciting, I would like to participate !

", "rawMarkdown": "Congratulations to the second place winner and grandmaster ! ! !\n\nAlso, thank you for making the competition more exciting with your discussions (and tweet).\nIf there is a competition next time you make it exciting, I would like to participate !", "votes": null }, { "id": "1317524", "postDate": "05/21/2021 13:10:15", "content": "

Congratulations for your second place and your grandmaster title!

", "rawMarkdown": "Congratulations for your second place and your grandmaster title!", "votes": null }, { "id": "1318077", "postDate": "05/22/2021 01:32:51", "content": "

Again, congrats on second place!

\n

There are really many things I need to learn from your solution, but I would like to ask you to teach me about one of simpler points.
\nIt's about \"Validation Set Construction\" section.
\nI had thought that the most important thing to correlate validation to LB was the sample size. But your very reasonable approach made me change my mind immediately.

\n

However, I also think that for some other tasks, it may not be possible to construct a reliable validation set in this way. For example, I have no idea how to adopt a similar approach for image processing tasks. In such cases, do you think it is advisable to validate with OOF and so on? Or do you think that a reliable validation set can be constructed by careful EDA for many kinds of tasks?
\nI would be happy to hear your opinion.

", "rawMarkdown": "Again, congrats on second place!\n\nThere are really many things I need to learn from your solution, but I would like to ask you to teach me about one of simpler points.\nIt's about \"Validation Set Construction\" section.\nI had thought that the most important thing to correlate validation to LB was the sample size. But your very reasonable approach made me change my mind immediately.\n\nHowever, I also think that for some other tasks, it may not be possible to construct a reliable validation set in this way. For example, I have no idea how to adopt a similar approach for image processing tasks. In such cases, do you think it is advisable to validate with OOF and so on? Or do you think that a reliable validation set can be constructed by careful EDA for many kinds of tasks?\nI would be happy to hear your opinion.", "votes": null }, { "id": "1318334", "postDate": "05/22/2021 08:17:04", "content": "

Hi housuke,
\nfor other tasks (e.g. image data), I think adversarial validation or careful extraction of the statistics from an image may work. I think it's also ok to try simple random split, but the optimal parameters of the postprocessing in validation set will be different from that in the test set.

", "rawMarkdown": "Hi housuke, \nfor other tasks (e.g. image data), I think adversarial validation or careful extraction of the statistics from an image may work. I think it's also ok to try simple random split, but the optimal parameters of the postprocessing in validation set will be different from that in the test set.", "votes": null }, { "id": "1318351", "postDate": "05/22/2021 08:41:51", "content": "

I see. So, the same approach is effective.
\nAnd I can understand that it is especially important to construct a reliable validation set when determining the parameters of the pp.
\nThank you, mamas!

", "rawMarkdown": "I see. So, the same approach is effective.\nAnd I can understand that it is especially important to construct a reliable validation set when determining the parameters of the pp.\nThank you, mamas!", "votes": null } ], "comments": [ { "id": 1314876, "author_name": "kokitanisaka", "author_url": "", "post_date": "05/19/2021 12:26:45", "content": "

Congratulations on your second place!
\nAlso big congrats for became a Grand Master!

\n

I'm always impressed with your performance, amazing!
\nAs well as the attitude toward competitions.
\nI'm glad that I was able to compete with you… or just seeing from far from you. It's just enjoyable.

\n

I really hope that I could see you again on kaggle, thank you so much!

", "votes": null, "replies": [ { "id": 1314895, "author_name": "mamasinkgs", "author_url": "", "post_date": "05/19/2021 12:41:55", "content": "

Thanks Kouki, your great notebook helped me a lot in the beginning of the competition.
\nI also hope I could see you again on kaggle!

", "votes": null, "replies": [] } ] }, { "id": 1315140, "author_name": "duykhanh99", "author_url": "", "post_date": "05/19/2021 15:36:25", "content": "

Thanks for sharing details solution. Learn alot from your team @mamasinkgs

", "votes": null, "replies": [] }, { "id": 1315438, "author_name": "mohamedbakrey", "author_url": "", "post_date": "05/19/2021 19:30:35", "content": "

Congratulations on your second place!😄

", "votes": null, "replies": [] }, { "id": 1315445, "author_name": "mamasinkgs", "author_url": "", "post_date": "05/19/2021 19:40:50", "content": "

I uploaded discrete optimization code that runs within about 15 mins. This is a much simpler version, but achieves 1.49 public/ 2.24 private. https://www.kaggle.com/mamasinkgs/discrete-optimization-in-2nd-place-solution

", "votes": null, "replies": [] }, { "id": 1315488, "author_name": "dehokanta", "author_url": "", "post_date": "05/19/2021 20:44:12", "content": "

Congratulations to the second place winner and grandmaster ! ! !

\n

Also, thank you for making the competition more exciting with your discussions (and tweet).
\nIf there is a competition next time you make it exciting, I would like to participate !

", "votes": null, "replies": [] }, { "id": 1317524, "author_name": "dwin183287", "author_url": "", "post_date": "05/21/2021 13:10:15", "content": "

Congratulations for your second place and your grandmaster title!

", "votes": null, "replies": [] }, { "id": 1318077, "author_name": "horsek", "author_url": "", "post_date": "05/22/2021 01:32:51", "content": "

Again, congrats on second place!

\n

There are really many things I need to learn from your solution, but I would like to ask you to teach me about one of simpler points.
\nIt's about \"Validation Set Construction\" section.
\nI had thought that the most important thing to correlate validation to LB was the sample size. But your very reasonable approach made me change my mind immediately.

\n

However, I also think that for some other tasks, it may not be possible to construct a reliable validation set in this way. For example, I have no idea how to adopt a similar approach for image processing tasks. In such cases, do you think it is advisable to validate with OOF and so on? Or do you think that a reliable validation set can be constructed by careful EDA for many kinds of tasks?
\nI would be happy to hear your opinion.

", "votes": null, "replies": [ { "id": 1318334, "author_name": "mamasinkgs", "author_url": "", "post_date": "05/22/2021 08:17:04", "content": "

Hi housuke,
\nfor other tasks (e.g. image data), I think adversarial validation or careful extraction of the statistics from an image may work. I think it's also ok to try simple random split, but the optimal parameters of the postprocessing in validation set will be different from that in the test set.

", "votes": null, "replies": [] }, { "id": 1318351, "author_name": "horsek", "author_url": "", "post_date": "05/22/2021 08:41:51", "content": "

I see. So, the same approach is effective.
\nAnd I can understand that it is especially important to construct a reliable validation set when determining the parameters of the pp.
\nThank you, mamas!

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1314827": "Thank you my briliant teammates ( @ymatioun, @vaghefi, @demonen, @rsakata ), all teams who competed with us, and all people who participated in this competition. Congrats @tvdwiele @areehdot @dott1718, who won this competition with an incredible performance!\n\nI'm happy because I finally became Kaggle Grandmaster with 5 gold medals, and @ymatioun became kaggle Master in this competition. \nIt's my 5th competition and it was fun to compete with my past teammate ( @vicensgaitan ) and people who I competed with past competitions ( @takoihiraokazu )!\n\nHere, I will explain the overall approach of our team's solution. For more details of (especially) absolute position prediction models and delta prediction models, please check our teammate's solutions. \n- Youri's part: https://www.kaggle.com/c/indoor-location-navigation/discussion/239880\n- Reza's part: https://www.kaggle.com/c/indoor-location-navigation/discussion/240197\n- Christoffer's part: https://www.kaggle.com/c/indoor-location-navigation/discussion/240025\n- Jack's part: https://www.kaggle.com/c/indoor-location-navigation/discussion/240141\n\nCode and the visualization of our prediction are here. \n- Discrete optimization code (simple version): https://www.kaggle.com/mamasinkgs/discrete-optimization-in-2nd-place-solution\n- The valid/test predictions of our team (public 1.41, private 2.18) : https://drive.google.com/drive/folders/1lxC0DYc9K84mKaNZoFdRVk9n8YCIrQbf\n\nFor our solution, **our team does not use any leakage (e.g. start/end point leakage, raw timestamp leakage, device leakage)**. We tried to use these leakages, but none of them improved the score and sometimes degraded the score, because our floor model and our WIFI model are already very good without leakage. \n \n# Validation Set Construction\nBy EDA, I found the samples of the test set meet these conditions: \n1. No non-standard floor is used, as stated in the evaluation section.\n2. Brand is always OPPO.\n3. max(timestamp) - min(timestamp) is larger than 60310.\n4. TYPE_ACCELEROMETER_acc is not all nan.\n5. TYPE_ACCELEROMETER_UNCALIBRATED_acc is not all nan.\n6. TYPE_GYROSCOPE_UNCALIBRATED_acc is not all nan.\n7. TYPE_WIFI_ssid is not all nan.\n\nI made these 7 masks and mulitiply them and get one mask. I chose the samples from training set using this mask. Then, from those samples, I chose 548 paths and use these 548 paths as validation set. LB and CV was always correlated using this validation set. I guess the host wants to use the good and clean paths for the evaluation of the performance, which is reasonable. \n\n# Absolute/Delta Prediction Models\nWe have 5 absolute position prediction models, and 3 delta prediction models. \nThese are the lists of our absolute position prediction models: \n- Reza's WKNN model (euclid distance)\n- Reza's WKNN model (correlation distance)\n- Jack's floor-level multiclass classification model\n- Youri's Fingerprinting model \n- Mamas's RNN model \n\nThese are the lists of our delta prediction models: \n- Youri's PDR model\n- Christoffer's LSTM\n- Mamas's MLP\n\nFor our ensembled delta prediction, we simply tried weighted average of these 3 models:\n`ensemble_delta_prediction = Mamas_MLP * 0.2 + Youri_PDR * 0.33 + Christoffer_LSTM * 0.47`\nOur delta prediction is not so good, MAE = 1.06 in CV.\n\nFor our ensembled absolute position prediction, we tried weighted average of these 5 models after applying [akio's cost minimization](https://www.kaggle.com/saitodevel01/indoor-post-processing-by-cost-minimization) using the ensembled delta prediction. When applying akio's cost minimization to Youri's Fingerprinting model and Reza's WKNN models, we consider the uncertainty of the prediction in akio's cost minimization. To be more specific, I modified the code of akio's cost minimization as follows. \n```\ndef get_xy_star_weight(delta_xy_hat, xy_hat, a, b, w):\n N = xy_hat.shape[0]\n alpha = w * a * np.ones(N)\n beta = b * np.ones_like(delta_xy_hat[:, 0])\n A = scipy.sparse.spdiags(alpha, [0], N, N)\n B = scipy.sparse.spdiags( beta, [0], N-1, N-1)\n D = scipy.sparse.spdiags(np.stack([-np.ones(N), np.ones(N)]), [0, 1], N-1, N)\n Q = A + (D.T @ B @ D)\n c = (A @ xy_hat) + (D.T @ (B @ delta_xy_hat))\n A = scipy.sparse.block_diag((Q, Q))\n b = np.r_[c[:,0], c[:,1]]\n xy_star = scipy.optimize.nnls(A.toarray(), b)[0].reshape(2,-1).T\n return xy_star\n```\nwhere w is given by\n```\nw = (Reza or Youri's prediction's uncertainty) ** (-2)\n```\nFinally, the ensembled prediction is given by \n```\nensembled_absolute_prediction = akio_costmin(mamas_RNN) * 0.04 + akio_weighted_costmin(youri_fingerprint) * 0.17 + akio_weighted_costmin(reza_WKNN_euclid) * 0.11 + akio_costmin(Jack_multiclass) * 0.34 + akio_weighted_costmin(reza_WKNN_corr) * 0.34\n```\nThe ensembled absolute prediction scored 4.206 in CV and 3.171 in LB.\n\n# Iterative snap to corridor & akio's cost minimization\nAfter the weighted averaging, We tried to move the prediction to the corridor. Because we didn't use additional generated waypoints unlike the 1st team, **the predictions before the discrete optimization must be in the corridor**, because some of these predictions will be used as the final prediction when no grid is selected in the discrete optimization process.\nFirst, we fixed the incorrect map according to @chris62's [great thread](https://www.kaggle.com/c/indoor-location-navigation/discussion/230558). Then, Christoffer automatically eliminated the stores on which the grids are present. This is the example of the problematic cases: The left image shows the map before the Christoffer's fixing map and the right image shows the map after the Christoffer's fixing map. \n\nThis is the only case we didn't eliminate the store, because we thought it can give a bad effect in snap-to-corridor. The effect should be smaller than 0.005, though. \n\n\nAfter fixing the maps, We applied snap to corridor and the akio's cost minimization iteratively (100 iterations). After the iterative snap to corridor & akio's cost minimization, CV score is 3.882 and LB score is 2.740. \n\n# Discrete Optimization\nLike the 1st place solution, Discrete optimization using proper penalties is the true key of this competition. We used these 6 penalties and corresponding 6 parameters (alpha, beta_1, beta_2, gamma, kappa, tau). However, kappa and tau is not so important and they didn't improve the score a lot. Gamma is really important and this is the reason our method worked well without additional generated grids. \n\n1. alpha * (euclid distance between grids and the absolute prediction)\n2. beta_1 * abs(**the angle of predicted delta** - **the angle of the delta between two grids**.)\n3. beta_2 * abs(**l2-norm of the predicted delta** - **l2-norm of the delta between two grids**)\n4. gamma * (1 when no grid is chosen, 0 otherwise)\n5. kappa * abs(**actual distance estimated by dijkstra algorithm** - **manhattan distance between 2 grids**)\n6. tau * (1 when the delta between two grids are zero, 0 otherwise)\n\nThe reason we estimated the actual distance using dijkstra algorithm between 2 grids are, the predictions are sometimes strange like this. \n\n\nAfter applying a cost that considers actual distance, the prediction became like this. \n(I'm sorry these predictions are not the one from our best submissions, but the very old predictions. These are just examples)\n\n\nThis is the example of the actual distance calculated by dijkstra algorithm (by Jack)\n\n\nIn the optimizaiton process, I used a simple greedy optimization algorithm, while the 1st team used the beam search, which is I guess one of the reason we lost. I have no experience in heuristics contest like Santa and couldn't do better in the optimization process. To be more specific, I split the paths by chunksize = 7, and I searched the 8 closest grids for each prediction. Then, I calculated the cost for all the possible patterns ((8 + 1) ** 7 = 4782969) and chose the grids (or sometimes original predictions) that minimize the cost. For the calculation of the backward delta of the first sample of a specific chunk, mamas's discrete optimization uses the last sample of the previous chunk.\nAfter mamas's discrete optimization, CV score is 2.523 and LB score is 1.392, which is our final solution. \n\n# Comments\nI told I would become the winner in the next competition in the [2nd place solution of Riiid competition](https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/210113), which was held 4 months ago, but I couldn't keep my promise. However, I have no regret because I did my best with really great teammates with the clean and completely no leakage solution and I really enjoyed this competition. Moreover, I learned a lot from @tvdwiele's team's great solution, which I think is one of the greatest solution I saw in the kaggle. \nI agree with Reza that hiding LB score should not become a habit in Kaggle competitions, because if we all hide a score the competition will become not enjoyable for participants and probably will give a not good effect on the kaggle community. However, I sincerely respect the 1st team, because I'm sure we couldn't have won if they didn't hide a score, and their great solution is clearly a tremendous contribution to the community!\nAnyway, I really enjoyed this competition. I will take a rest for a while, and will surely come back to Kaggle again !!! \n\n# Thanks all, see you again!", "1314876": "Congratulations on your second place! \nAlso big congrats for became a Grand Master! \n\nI'm always impressed with your performance, amazing! \nAs well as the attitude toward competitions. \nI'm glad that I was able to compete with you... or just seeing from far from you. It's just enjoyable. \n\nI really hope that I could see you again on kaggle, thank you so much!", "1314895": "Thanks Kouki, your great notebook helped me a lot in the beginning of the competition. \nI also hope I could see you again on kaggle!", "1315140": "Thanks for sharing details solution. Learn alot from your team @mamasinkgs", "1315438": "Congratulations on your second place!😄", "1315445": "I uploaded discrete optimization code that runs within about 15 mins. This is a much simpler version, but achieves 1.49 public/ 2.24 private. https://www.kaggle.com/mamasinkgs/discrete-optimization-in-2nd-place-solution", "1315488": "Congratulations to the second place winner and grandmaster ! ! !\n\nAlso, thank you for making the competition more exciting with your discussions (and tweet).\nIf there is a competition next time you make it exciting, I would like to participate !", "1317524": "Congratulations for your second place and your grandmaster title!", "1318077": "Again, congrats on second place!\n\nThere are really many things I need to learn from your solution, but I would like to ask you to teach me about one of simpler points.\nIt's about \"Validation Set Construction\" section.\nI had thought that the most important thing to correlate validation to LB was the sample size. But your very reasonable approach made me change my mind immediately.\n\nHowever, I also think that for some other tasks, it may not be possible to construct a reliable validation set in this way. For example, I have no idea how to adopt a similar approach for image processing tasks. In such cases, do you think it is advisable to validate with OOF and so on? Or do you think that a reliable validation set can be constructed by careful EDA for many kinds of tasks?\nI would be happy to hear your opinion.", "1318334": "Hi housuke, \nfor other tasks (e.g. image data), I think adversarial validation or careful extraction of the statistics from an image may work. I think it's also ok to try simple random split, but the optimal parameters of the postprocessing in validation set will be different from that in the test set.", "1318351": "I see. So, the same approach is effective.\nAnd I can understand that it is especially important to construct a reliable validation set when determining the parameters of the pp.\nThank you, mamas!" }, "source": "meta" }