{ "id": 359674, "title": "Need help with settings keras!", "url": "/competitions/tabular-playground-series-oct-2022/discussion/359674", "author_name": "", "post_date": "2022-10-13T05:02:27.863795300Z", "votes": 1, "comment_count": 11, "views": 0, "content": "

Hello to everyone. I am going slightly mad. Please I need you help.

\n

The question 1.
\nIs it possible to get good result with kras?

\n

The question 2.
\nThe more epochs i set up the better result I have with training set but the worse result with finish test set.

\n

My settings. Should I change my metrics here? I hope it is the case of overfitting, but I can't understand why?
\nI have around 0.21 result with epochs = 5 and around 0.25 with epoch = 50.

\n
from tensorflow import keras\nfrom tensorflow.keras import layers\n\ninput_shape = [X_train.shape[1]]\n\nmodelA = keras.Sequential([\nlayers.BatchNormalization(input_shape = input_shape),\nlayers.Dense(512,activation='relu'),\nlayers.BatchNormalization(),\nlayers.Dropout(0.3),\nlayers.Dense(512,activation='relu'),\nlayers.BatchNormalization(),\nlayers.Dropout(0.3),\nlayers.Dense(512,activation='relu'),\nlayers.BatchNormalization(),\nlayers.Dropout(0.3),\nlayers.Dense(1,activation='sigmoid'),\n])`\n\nmodelA.compile( optimizer='adam', loss='binary_crossentropy', metrics=['binary_accuracy'], )\n\nearly_stopping = keras.callbacks.EarlyStopping( patience=5, min_delta=0.001, restore_best_weights=True, ) history = modelA.fit( X_train, y_A_train, validation_data=(X_valid, y_A_valid), batch_size=2000, epochs=50, callbacks=[early_stopping], )\n\n2022-10-12 05:42:24.574704: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:185] None of the MLIR \n
\n

Optimization Passes are enabled (registered 2)
\nEpoch 1/50
\n753/753 [==============================] - 87s 113ms/step - loss: 0.2470 - binary_accuracy: 0.9248 - val_loss: 0.2009 - val_binary_accuracy: 0.9436
\nEpoch 2/50
\n753/753 [==============================] - 83s 110ms/step - loss: 0.2060 - binary_accuracy: 0.9420 - val_loss: 0.1975 - val_binary_accuracy: 0.9439
\nEpoch 3/50
\n753/753 [==============================] - 84s 111ms/step - loss: 0.2031 - binary_accuracy: 0.9424 - val_loss: 0.1959 - val_binary_accuracy: 0.9441
\nEpoch 4/50
\n753/753 [==============================] - 85s 113ms/step - loss: 0.2009 - binary_accuracy: 0.9425 - val_loss: 0.1939 - val_binary_accuracy: 0.9442
\nEpoch 5/50
\n753/753 [==============================] - 85s 113ms/step - loss: 0.1985 - binary_accuracy: 0.9428 - val_loss: 0.1912 - val_binary_accuracy: 0.9443
\nEpoch 6/50
\n753/753 [==============================] - 86s 114ms/step - loss: 0.1963 - binary_accuracy: 0.9428 - val_loss: 0.1878 - val_binary_accuracy: 0.9444
\nEpoch 7/50
\n753/753 [==============================] - 85s 113ms/step - loss: 0.1934 - binary_accuracy: 0.9430 - val_loss: 0.1839 - val_binary_accuracy: 0.9446
\nEpoch 8/50
\n753/753 [==============================] - 84s 112ms/step - loss: 0.1902 - binary_accuracy: 0.9432 - val_loss: 0.1791 - val_binary_accuracy: 0.9448
\nEpoch 9/50
\n753/753 [==============================] - 84s 112ms/step - loss: 0.1870 - binary_accuracy: 0.9435 - val_loss: 0.1740 - val_binary_accuracy: 0.9451
\nEpoch 10/50
\n753/753 [==============================] - 85s 112ms/step - loss: 0.1838 - binary_accuracy: 0.9436 - val_loss: 0.1700 - val_binary_accuracy: 0.9457
\nEpoch 11/50
\n753/753 [==============================] - 84s 112ms/step - loss: 0.1804 - binary_accuracy: 0.9438 - val_loss: 0.1653 - val_binary_accuracy: 0.9460
\nEpoch 12/50
\n753/753 [==============================] - 84s 112ms/step - loss: 0.1778 - binary_accuracy: 0.9441 - val_loss: 0.1601 - val_binary_accuracy: 0.9463
\nEpoch 13/50
\n753/753 [==============================] - 86s 114ms/step - loss: 0.1749 - binary_accuracy: 0.9445 - val_loss: 0.1567 - val_binary_accuracy: 0.9472
\nEpoch 14/50
\n753/753 [==============================] - 84s 111ms/step - loss: 0.1721 - binary_accuracy: 0.9447 - val_loss: 0.1523 - val_binary_accuracy: 0.9482
\nEpoch 15/50
\n753/753 [==============================] - 85s 113ms/step - loss: 0.1696 - binary_accuracy: 0.9451 - val_loss: 0.1485 - val_binary_accuracy: 0.9486
\nEpoch 16/50
\n753/753 [==============================] - 87s 116ms/step - loss: 0.1673 - binary_accuracy: 0.9454 - val_loss: 0.1441 - val_binary_accuracy: 0.9497
\nEpoch 17/50
\n753/753 [==============================] - 87s 116ms/step - loss: 0.1652 - binary_accuracy: 0.9458 - val_loss: 0.1422 - val_binary_accuracy: 0.9498
\nEpoch 18/50
\n753/753 [==============================] - 87s 115ms/step - loss: 0.1631 - binary_accuracy: 0.9461 - val_loss: 0.1379 - val_binary_accuracy: 0.9514
\nEpoch 19/50
\n753/753 [==============================] - 86s 115ms/step - loss: 0.1609 - binary_accuracy: 0.9466 - val_loss: 0.1350 - val_binary_accuracy: 0.9520
\nEpoch 20/50
\n753/753 [==============================] - 86s 115ms/step - loss: 0.1594 - binary_accuracy: 0.9468 - val_loss: 0.1330 - val_binary_accuracy: 0.9522
\nEpoch 21/50
\n753/753 [==============================] - 86s 114ms/step - loss: 0.1579 - binary_accuracy: 0.9471 - val_loss: 0.1297 - val_binary_accuracy: 0.9535
\nEpoch 22/50
\n753/753 [==============================] - 87s 116ms/step - loss: 0.1562 - binary_accuracy: 0.9475 - val_loss: 0.1282 - val_binary_accuracy: 0.9543
\nEpoch 23/50
\n753/753 [==============================] - 88s 118ms/step - loss: 0.1548 - binary_accuracy: 0.9477 - val_loss: 0.1260 - val_binary_accuracy: 0.9549
\nEpoch 24/50
\n753/753 [==============================] - 88s 118ms/step - loss: 0.1534 - binary_accuracy: 0.9480 - val_loss: 0.1230 - val_binary_accuracy: 0.9562
\nEpoch 25/50
\n753/753 [==============================] - 87s 115ms/step - loss: 0.1520 - binary_accuracy: 0.9484 - val_loss: 0.1226 - val_binary_accuracy: 0.9560
\nEpoch 26/50
\n753/753 [==============================] - 87s 115ms/step - loss: 0.1512 - binary_accuracy: 0.9485 - val_loss: 0.1197 - val_binary_accuracy: 0.9565
\nEpoch 27/50
\n753/753 [==============================] - 89s 118ms/step - loss: 0.1501 - binary_accuracy: 0.9488 - val_loss: 0.1174 - val_binary_accuracy: 0.9581
\nEpoch 28/50
\n753/753 [==============================] - 87s 116ms/step - loss: 0.1489 - binary_accuracy: 0.9490 - val_loss: 0.1167 - val_binary_accuracy: 0.9584
\nEpoch 29/50
\n753/753 [==============================] - 87s 115ms/step - loss: 0.1477 - binary_accuracy: 0.9493 - val_loss: 0.1153 - val_binary_accuracy: 0.9590
\nEpoch 30/50
\n753/753 [==============================] - 90s 119ms/step - loss: 0.1469 - binary_accuracy: 0.9495 - val_loss: 0.1140 - val_binary_accuracy: 0.9591
\nEpoch 31/50
\n753/753 [==============================] - 86s 115ms/step - loss: 0.1463 - binary_accuracy: 0.9498 - val_loss: 0.1121 - val_binary_accuracy: 0.9599
\nEpoch 32/50
\n753/753 [==============================] - 86s 114ms/step - loss: 0.1452 - binary_accuracy: 0.9501 - val_loss: 0.1113 - val_binary_accuracy: 0.9597
\nEpoch 33/50
\n753/753 [==============================] - 86s 114ms/step - loss: 0.1447 - binary_accuracy: 0.9503 - val_loss: 0.1093 - val_binary_accuracy: 0.9615
\nEpoch 34/50
\n753/753 [==============================] - 87s 116ms/step - loss: 0.1437 - binary_accuracy: 0.9505 - val_loss: 0.1088 - val_binary_accuracy: 0.9610
\nEpoch 35/50
\n753/753 [==============================] - 86s 114ms/step - loss: 0.1430 - binary_accuracy: 0.9507 - val_loss: 0.1074 - val_binary_accuracy: 0.9626
\nEpoch 36/50
\n753/753 [==============================] - 86s 114ms/step - loss: 0.1422 - binary_accuracy: 0.9508 - val_loss: 0.1055 - val_binary_accuracy: 0.9621
\nEpoch 37/50
\n753/753 [==============================] - 87s 115ms/step - loss: 0.1417 - binary_accuracy: 0.9508 - val_loss: 0.1049 - val_binary_accuracy: 0.9621
\nEpoch 38/50
\n753/753 [==============================] - 87s 116ms/step - loss: 0.1408 - binary_accuracy: 0.9513 - val_loss: 0.1042 - val_binary_accuracy: 0.9625
\nEpoch 39/50
\n753/753 [==============================] - 87s 116ms/step - loss: 0.1400 - binary_accuracy: 0.9515 - val_loss: 0.1026 - val_binary_accuracy: 0.9626
\nEpoch 40/50
\n753/753 [==============================] - 88s 117ms/step - loss: 0.1395 - binary_accuracy: 0.9515 - val_loss: 0.1025 - val_binary_accuracy: 0.9633
\nEpoch 41/50
\n753/753 [==============================] - 88s 117ms/step - loss: 0.1390 - binary_accuracy: 0.9516 - val_loss: 0.1011 - val_binary_accuracy: 0.9633
\nEpoch 42/50
\n753/753 [==============================] - 87s 116ms/step - loss: 0.1384 - binary_accuracy: 0.9521 - val_loss: 0.1017 - val_binary_accuracy: 0.9630
\nEpoch 43/50
\n753/753 [==============================] - 85s 114ms/step - loss: 0.1376 - binary_accuracy: 0.9520 - val_loss: 0.0992 - val_binary_accuracy: 0.9645
\nEpoch 44/50
\n753/753 [==============================] - 88s 117ms/step - loss: 0.1372 - binary_accuracy: 0.9522 - val_loss: 0.0989 - val_binary_accuracy: 0.9646
\nEpoch 45/50
\n753/753 [==============================] - 87s 116ms/step - loss: 0.1371 - binary_accuracy: 0.9522 - val_loss: 0.0982 - val_binary_accuracy: 0.9650
\nEpoch 46/50
\n753/753 [==============================] - 88s 116ms/step - loss: 0.1359 - binary_accuracy: 0.9527 - val_loss: 0.0983 - val_binary_accuracy: 0.9647
\nEpoch 47/50
\n753/753 [==============================] - 85s 113ms/step - loss: 0.1362 - binary_accuracy: 0.9526 - val_loss: 0.0966 - val_binary_accuracy: 0.9652
\nEpoch 48/50
\n753/753 [==============================] - 89s 118ms/step - loss: 0.1352 - binary_accuracy: 0.9527 - val_loss: 0.0953 - val_binary_accuracy: 0.9663
\nEpoch 49/50
\n753/753 [==============================] - 87s 115ms/step - loss: 0.1350 - binary_accuracy: 0.9528 - val_loss: 0.0956 - val_binary_accuracy: 0.9653
\nEpoch 50/50
\n753/753 [==============================] - 86s 115ms/step - loss: 0.1342 - binary_accuracy: 0.9531 - val_loss: 0.0946 - val_binary_accuracy: 0.9661

", "messages": [ { "id": "1985079", "postDate": "10/13/2022 05:02:27", "content": "

Hello to everyone. I am going slightly mad. Please I need you help.

\n

The question 1.
\nIs it possible to get good result with kras?

\n

The question 2.
\nThe more epochs i set up the better result I have with training set but the worse result with finish test set.

\n

My settings. Should I change my metrics here? I hope it is the case of overfitting, but I can't understand why?
\nI have around 0.21 result with epochs = 5 and around 0.25 with epoch = 50.

\n
from tensorflow import keras\nfrom tensorflow.keras import layers\n\ninput_shape = [X_train.shape[1]]\n\nmodelA = keras.Sequential([\nlayers.BatchNormalization(input_shape = input_shape),\nlayers.Dense(512,activation='relu'),\nlayers.BatchNormalization(),\nlayers.Dropout(0.3),\nlayers.Dense(512,activation='relu'),\nlayers.BatchNormalization(),\nlayers.Dropout(0.3),\nlayers.Dense(512,activation='relu'),\nlayers.BatchNormalization(),\nlayers.Dropout(0.3),\nlayers.Dense(1,activation='sigmoid'),\n])`\n\nmodelA.compile( optimizer='adam', loss='binary_crossentropy', metrics=['binary_accuracy'], )\n\nearly_stopping = keras.callbacks.EarlyStopping( patience=5, min_delta=0.001, restore_best_weights=True, ) history = modelA.fit( X_train, y_A_train, validation_data=(X_valid, y_A_valid), batch_size=2000, epochs=50, callbacks=[early_stopping], )\n\n2022-10-12 05:42:24.574704: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:185] None of the MLIR \n
\n

Optimization Passes are enabled (registered 2)
\nEpoch 1/50
\n753/753 [==============================] - 87s 113ms/step - loss: 0.2470 - binary_accuracy: 0.9248 - val_loss: 0.2009 - val_binary_accuracy: 0.9436
\nEpoch 2/50
\n753/753 [==============================] - 83s 110ms/step - loss: 0.2060 - binary_accuracy: 0.9420 - val_loss: 0.1975 - val_binary_accuracy: 0.9439
\nEpoch 3/50
\n753/753 [==============================] - 84s 111ms/step - loss: 0.2031 - binary_accuracy: 0.9424 - val_loss: 0.1959 - val_binary_accuracy: 0.9441
\nEpoch 4/50
\n753/753 [==============================] - 85s 113ms/step - loss: 0.2009 - binary_accuracy: 0.9425 - val_loss: 0.1939 - val_binary_accuracy: 0.9442
\nEpoch 5/50
\n753/753 [==============================] - 85s 113ms/step - loss: 0.1985 - binary_accuracy: 0.9428 - val_loss: 0.1912 - val_binary_accuracy: 0.9443
\nEpoch 6/50
\n753/753 [==============================] - 86s 114ms/step - loss: 0.1963 - binary_accuracy: 0.9428 - val_loss: 0.1878 - val_binary_accuracy: 0.9444
\nEpoch 7/50
\n753/753 [==============================] - 85s 113ms/step - loss: 0.1934 - binary_accuracy: 0.9430 - val_loss: 0.1839 - val_binary_accuracy: 0.9446
\nEpoch 8/50
\n753/753 [==============================] - 84s 112ms/step - loss: 0.1902 - binary_accuracy: 0.9432 - val_loss: 0.1791 - val_binary_accuracy: 0.9448
\nEpoch 9/50
\n753/753 [==============================] - 84s 112ms/step - loss: 0.1870 - binary_accuracy: 0.9435 - val_loss: 0.1740 - val_binary_accuracy: 0.9451
\nEpoch 10/50
\n753/753 [==============================] - 85s 112ms/step - loss: 0.1838 - binary_accuracy: 0.9436 - val_loss: 0.1700 - val_binary_accuracy: 0.9457
\nEpoch 11/50
\n753/753 [==============================] - 84s 112ms/step - loss: 0.1804 - binary_accuracy: 0.9438 - val_loss: 0.1653 - val_binary_accuracy: 0.9460
\nEpoch 12/50
\n753/753 [==============================] - 84s 112ms/step - loss: 0.1778 - binary_accuracy: 0.9441 - val_loss: 0.1601 - val_binary_accuracy: 0.9463
\nEpoch 13/50
\n753/753 [==============================] - 86s 114ms/step - loss: 0.1749 - binary_accuracy: 0.9445 - val_loss: 0.1567 - val_binary_accuracy: 0.9472
\nEpoch 14/50
\n753/753 [==============================] - 84s 111ms/step - loss: 0.1721 - binary_accuracy: 0.9447 - val_loss: 0.1523 - val_binary_accuracy: 0.9482
\nEpoch 15/50
\n753/753 [==============================] - 85s 113ms/step - loss: 0.1696 - binary_accuracy: 0.9451 - val_loss: 0.1485 - val_binary_accuracy: 0.9486
\nEpoch 16/50
\n753/753 [==============================] - 87s 116ms/step - loss: 0.1673 - binary_accuracy: 0.9454 - val_loss: 0.1441 - val_binary_accuracy: 0.9497
\nEpoch 17/50
\n753/753 [==============================] - 87s 116ms/step - loss: 0.1652 - binary_accuracy: 0.9458 - val_loss: 0.1422 - val_binary_accuracy: 0.9498
\nEpoch 18/50
\n753/753 [==============================] - 87s 115ms/step - loss: 0.1631 - binary_accuracy: 0.9461 - val_loss: 0.1379 - val_binary_accuracy: 0.9514
\nEpoch 19/50
\n753/753 [==============================] - 86s 115ms/step - loss: 0.1609 - binary_accuracy: 0.9466 - val_loss: 0.1350 - val_binary_accuracy: 0.9520
\nEpoch 20/50
\n753/753 [==============================] - 86s 115ms/step - loss: 0.1594 - binary_accuracy: 0.9468 - val_loss: 0.1330 - val_binary_accuracy: 0.9522
\nEpoch 21/50
\n753/753 [==============================] - 86s 114ms/step - loss: 0.1579 - binary_accuracy: 0.9471 - val_loss: 0.1297 - val_binary_accuracy: 0.9535
\nEpoch 22/50
\n753/753 [==============================] - 87s 116ms/step - loss: 0.1562 - binary_accuracy: 0.9475 - val_loss: 0.1282 - val_binary_accuracy: 0.9543
\nEpoch 23/50
\n753/753 [==============================] - 88s 118ms/step - loss: 0.1548 - binary_accuracy: 0.9477 - val_loss: 0.1260 - val_binary_accuracy: 0.9549
\nEpoch 24/50
\n753/753 [==============================] - 88s 118ms/step - loss: 0.1534 - binary_accuracy: 0.9480 - val_loss: 0.1230 - val_binary_accuracy: 0.9562
\nEpoch 25/50
\n753/753 [==============================] - 87s 115ms/step - loss: 0.1520 - binary_accuracy: 0.9484 - val_loss: 0.1226 - val_binary_accuracy: 0.9560
\nEpoch 26/50
\n753/753 [==============================] - 87s 115ms/step - loss: 0.1512 - binary_accuracy: 0.9485 - val_loss: 0.1197 - val_binary_accuracy: 0.9565
\nEpoch 27/50
\n753/753 [==============================] - 89s 118ms/step - loss: 0.1501 - binary_accuracy: 0.9488 - val_loss: 0.1174 - val_binary_accuracy: 0.9581
\nEpoch 28/50
\n753/753 [==============================] - 87s 116ms/step - loss: 0.1489 - binary_accuracy: 0.9490 - val_loss: 0.1167 - val_binary_accuracy: 0.9584
\nEpoch 29/50
\n753/753 [==============================] - 87s 115ms/step - loss: 0.1477 - binary_accuracy: 0.9493 - val_loss: 0.1153 - val_binary_accuracy: 0.9590
\nEpoch 30/50
\n753/753 [==============================] - 90s 119ms/step - loss: 0.1469 - binary_accuracy: 0.9495 - val_loss: 0.1140 - val_binary_accuracy: 0.9591
\nEpoch 31/50
\n753/753 [==============================] - 86s 115ms/step - loss: 0.1463 - binary_accuracy: 0.9498 - val_loss: 0.1121 - val_binary_accuracy: 0.9599
\nEpoch 32/50
\n753/753 [==============================] - 86s 114ms/step - loss: 0.1452 - binary_accuracy: 0.9501 - val_loss: 0.1113 - val_binary_accuracy: 0.9597
\nEpoch 33/50
\n753/753 [==============================] - 86s 114ms/step - loss: 0.1447 - binary_accuracy: 0.9503 - val_loss: 0.1093 - val_binary_accuracy: 0.9615
\nEpoch 34/50
\n753/753 [==============================] - 87s 116ms/step - loss: 0.1437 - binary_accuracy: 0.9505 - val_loss: 0.1088 - val_binary_accuracy: 0.9610
\nEpoch 35/50
\n753/753 [==============================] - 86s 114ms/step - loss: 0.1430 - binary_accuracy: 0.9507 - val_loss: 0.1074 - val_binary_accuracy: 0.9626
\nEpoch 36/50
\n753/753 [==============================] - 86s 114ms/step - loss: 0.1422 - binary_accuracy: 0.9508 - val_loss: 0.1055 - val_binary_accuracy: 0.9621
\nEpoch 37/50
\n753/753 [==============================] - 87s 115ms/step - loss: 0.1417 - binary_accuracy: 0.9508 - val_loss: 0.1049 - val_binary_accuracy: 0.9621
\nEpoch 38/50
\n753/753 [==============================] - 87s 116ms/step - loss: 0.1408 - binary_accuracy: 0.9513 - val_loss: 0.1042 - val_binary_accuracy: 0.9625
\nEpoch 39/50
\n753/753 [==============================] - 87s 116ms/step - loss: 0.1400 - binary_accuracy: 0.9515 - val_loss: 0.1026 - val_binary_accuracy: 0.9626
\nEpoch 40/50
\n753/753 [==============================] - 88s 117ms/step - loss: 0.1395 - binary_accuracy: 0.9515 - val_loss: 0.1025 - val_binary_accuracy: 0.9633
\nEpoch 41/50
\n753/753 [==============================] - 88s 117ms/step - loss: 0.1390 - binary_accuracy: 0.9516 - val_loss: 0.1011 - val_binary_accuracy: 0.9633
\nEpoch 42/50
\n753/753 [==============================] - 87s 116ms/step - loss: 0.1384 - binary_accuracy: 0.9521 - val_loss: 0.1017 - val_binary_accuracy: 0.9630
\nEpoch 43/50
\n753/753 [==============================] - 85s 114ms/step - loss: 0.1376 - binary_accuracy: 0.9520 - val_loss: 0.0992 - val_binary_accuracy: 0.9645
\nEpoch 44/50
\n753/753 [==============================] - 88s 117ms/step - loss: 0.1372 - binary_accuracy: 0.9522 - val_loss: 0.0989 - val_binary_accuracy: 0.9646
\nEpoch 45/50
\n753/753 [==============================] - 87s 116ms/step - loss: 0.1371 - binary_accuracy: 0.9522 - val_loss: 0.0982 - val_binary_accuracy: 0.9650
\nEpoch 46/50
\n753/753 [==============================] - 88s 116ms/step - loss: 0.1359 - binary_accuracy: 0.9527 - val_loss: 0.0983 - val_binary_accuracy: 0.9647
\nEpoch 47/50
\n753/753 [==============================] - 85s 113ms/step - loss: 0.1362 - binary_accuracy: 0.9526 - val_loss: 0.0966 - val_binary_accuracy: 0.9652
\nEpoch 48/50
\n753/753 [==============================] - 89s 118ms/step - loss: 0.1352 - binary_accuracy: 0.9527 - val_loss: 0.0953 - val_binary_accuracy: 0.9663
\nEpoch 49/50
\n753/753 [==============================] - 87s 115ms/step - loss: 0.1350 - binary_accuracy: 0.9528 - val_loss: 0.0956 - val_binary_accuracy: 0.9653
\nEpoch 50/50
\n753/753 [==============================] - 86s 115ms/step - loss: 0.1342 - binary_accuracy: 0.9531 - val_loss: 0.0946 - val_binary_accuracy: 0.9661

", "rawMarkdown": "Hello to everyone. I am going slightly mad. Please I need you help.\n\nThe question 1.\nIs it possible to get good result with kras?\n\nThe question 2.\nThe more epochs i set up the better result I have with training set but the worse result with finish test set.\n\nMy settings. Should I change my metrics here? I hope it is the case of overfitting, but I can't understand why?\nI have around 0.21 result with epochs = 5 and around 0.25 with epoch = 50.\n\n```\nfrom tensorflow import keras\nfrom tensorflow.keras import layers\n\ninput_shape = [X_train.shape[1]]\n\nmodelA = keras.Sequential([\nlayers.BatchNormalization(input_shape = input_shape),\nlayers.Dense(512,activation='relu'),\nlayers.BatchNormalization(),\nlayers.Dropout(0.3),\nlayers.Dense(512,activation='relu'),\nlayers.BatchNormalization(),\nlayers.Dropout(0.3),\nlayers.Dense(512,activation='relu'),\nlayers.BatchNormalization(),\nlayers.Dropout(0.3),\nlayers.Dense(1,activation='sigmoid'),\n])`\n\nmodelA.compile( optimizer='adam', loss='binary_crossentropy', metrics=['binary_accuracy'], )\n\nearly_stopping = keras.callbacks.EarlyStopping( patience=5, min_delta=0.001, restore_best_weights=True, ) history = modelA.fit( X_train, y_A_train, validation_data=(X_valid, y_A_valid), batch_size=2000, epochs=50, callbacks=[early_stopping], )\n\n2022-10-12 05:42:24.574704: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:185] None of the MLIR \n```\nOptimization Passes are enabled (registered 2)\nEpoch 1/50\n753/753 [==============================] - 87s 113ms/step - loss: 0.2470 - binary_accuracy: 0.9248 - val_loss: 0.2009 - val_binary_accuracy: 0.9436\nEpoch 2/50\n753/753 [==============================] - 83s 110ms/step - loss: 0.2060 - binary_accuracy: 0.9420 - val_loss: 0.1975 - val_binary_accuracy: 0.9439\nEpoch 3/50\n753/753 [==============================] - 84s 111ms/step - loss: 0.2031 - binary_accuracy: 0.9424 - val_loss: 0.1959 - val_binary_accuracy: 0.9441\nEpoch 4/50\n753/753 [==============================] - 85s 113ms/step - loss: 0.2009 - binary_accuracy: 0.9425 - val_loss: 0.1939 - val_binary_accuracy: 0.9442\nEpoch 5/50\n753/753 [==============================] - 85s 113ms/step - loss: 0.1985 - binary_accuracy: 0.9428 - val_loss: 0.1912 - val_binary_accuracy: 0.9443\nEpoch 6/50\n753/753 [==============================] - 86s 114ms/step - loss: 0.1963 - binary_accuracy: 0.9428 - val_loss: 0.1878 - val_binary_accuracy: 0.9444\nEpoch 7/50\n753/753 [==============================] - 85s 113ms/step - loss: 0.1934 - binary_accuracy: 0.9430 - val_loss: 0.1839 - val_binary_accuracy: 0.9446\nEpoch 8/50\n753/753 [==============================] - 84s 112ms/step - loss: 0.1902 - binary_accuracy: 0.9432 - val_loss: 0.1791 - val_binary_accuracy: 0.9448\nEpoch 9/50\n753/753 [==============================] - 84s 112ms/step - loss: 0.1870 - binary_accuracy: 0.9435 - val_loss: 0.1740 - val_binary_accuracy: 0.9451\nEpoch 10/50\n753/753 [==============================] - 85s 112ms/step - loss: 0.1838 - binary_accuracy: 0.9436 - val_loss: 0.1700 - val_binary_accuracy: 0.9457\nEpoch 11/50\n753/753 [==============================] - 84s 112ms/step - loss: 0.1804 - binary_accuracy: 0.9438 - val_loss: 0.1653 - val_binary_accuracy: 0.9460\nEpoch 12/50\n753/753 [==============================] - 84s 112ms/step - loss: 0.1778 - binary_accuracy: 0.9441 - val_loss: 0.1601 - val_binary_accuracy: 0.9463\nEpoch 13/50\n753/753 [==============================] - 86s 114ms/step - loss: 0.1749 - binary_accuracy: 0.9445 - val_loss: 0.1567 - val_binary_accuracy: 0.9472\nEpoch 14/50\n753/753 [==============================] - 84s 111ms/step - loss: 0.1721 - binary_accuracy: 0.9447 - val_loss: 0.1523 - val_binary_accuracy: 0.9482\nEpoch 15/50\n753/753 [==============================] - 85s 113ms/step - loss: 0.1696 - binary_accuracy: 0.9451 - val_loss: 0.1485 - val_binary_accuracy: 0.9486\nEpoch 16/50\n753/753 [==============================] - 87s 116ms/step - loss: 0.1673 - binary_accuracy: 0.9454 - val_loss: 0.1441 - val_binary_accuracy: 0.9497\nEpoch 17/50\n753/753 [==============================] - 87s 116ms/step - loss: 0.1652 - binary_accuracy: 0.9458 - val_loss: 0.1422 - val_binary_accuracy: 0.9498\nEpoch 18/50\n753/753 [==============================] - 87s 115ms/step - loss: 0.1631 - binary_accuracy: 0.9461 - val_loss: 0.1379 - val_binary_accuracy: 0.9514\nEpoch 19/50\n753/753 [==============================] - 86s 115ms/step - loss: 0.1609 - binary_accuracy: 0.9466 - val_loss: 0.1350 - val_binary_accuracy: 0.9520\nEpoch 20/50\n753/753 [==============================] - 86s 115ms/step - loss: 0.1594 - binary_accuracy: 0.9468 - val_loss: 0.1330 - val_binary_accuracy: 0.9522\nEpoch 21/50\n753/753 [==============================] - 86s 114ms/step - loss: 0.1579 - binary_accuracy: 0.9471 - val_loss: 0.1297 - val_binary_accuracy: 0.9535\nEpoch 22/50\n753/753 [==============================] - 87s 116ms/step - loss: 0.1562 - binary_accuracy: 0.9475 - val_loss: 0.1282 - val_binary_accuracy: 0.9543\nEpoch 23/50\n753/753 [==============================] - 88s 118ms/step - loss: 0.1548 - binary_accuracy: 0.9477 - val_loss: 0.1260 - val_binary_accuracy: 0.9549\nEpoch 24/50\n753/753 [==============================] - 88s 118ms/step - loss: 0.1534 - binary_accuracy: 0.9480 - val_loss: 0.1230 - val_binary_accuracy: 0.9562\nEpoch 25/50\n753/753 [==============================] - 87s 115ms/step - loss: 0.1520 - binary_accuracy: 0.9484 - val_loss: 0.1226 - val_binary_accuracy: 0.9560\nEpoch 26/50\n753/753 [==============================] - 87s 115ms/step - loss: 0.1512 - binary_accuracy: 0.9485 - val_loss: 0.1197 - val_binary_accuracy: 0.9565\nEpoch 27/50\n753/753 [==============================] - 89s 118ms/step - loss: 0.1501 - binary_accuracy: 0.9488 - val_loss: 0.1174 - val_binary_accuracy: 0.9581\nEpoch 28/50\n753/753 [==============================] - 87s 116ms/step - loss: 0.1489 - binary_accuracy: 0.9490 - val_loss: 0.1167 - val_binary_accuracy: 0.9584\nEpoch 29/50\n753/753 [==============================] - 87s 115ms/step - loss: 0.1477 - binary_accuracy: 0.9493 - val_loss: 0.1153 - val_binary_accuracy: 0.9590\nEpoch 30/50\n753/753 [==============================] - 90s 119ms/step - loss: 0.1469 - binary_accuracy: 0.9495 - val_loss: 0.1140 - val_binary_accuracy: 0.9591\nEpoch 31/50\n753/753 [==============================] - 86s 115ms/step - loss: 0.1463 - binary_accuracy: 0.9498 - val_loss: 0.1121 - val_binary_accuracy: 0.9599\nEpoch 32/50\n753/753 [==============================] - 86s 114ms/step - loss: 0.1452 - binary_accuracy: 0.9501 - val_loss: 0.1113 - val_binary_accuracy: 0.9597\nEpoch 33/50\n753/753 [==============================] - 86s 114ms/step - loss: 0.1447 - binary_accuracy: 0.9503 - val_loss: 0.1093 - val_binary_accuracy: 0.9615\nEpoch 34/50\n753/753 [==============================] - 87s 116ms/step - loss: 0.1437 - binary_accuracy: 0.9505 - val_loss: 0.1088 - val_binary_accuracy: 0.9610\nEpoch 35/50\n753/753 [==============================] - 86s 114ms/step - loss: 0.1430 - binary_accuracy: 0.9507 - val_loss: 0.1074 - val_binary_accuracy: 0.9626\nEpoch 36/50\n753/753 [==============================] - 86s 114ms/step - loss: 0.1422 - binary_accuracy: 0.9508 - val_loss: 0.1055 - val_binary_accuracy: 0.9621\nEpoch 37/50\n753/753 [==============================] - 87s 115ms/step - loss: 0.1417 - binary_accuracy: 0.9508 - val_loss: 0.1049 - val_binary_accuracy: 0.9621\nEpoch 38/50\n753/753 [==============================] - 87s 116ms/step - loss: 0.1408 - binary_accuracy: 0.9513 - val_loss: 0.1042 - val_binary_accuracy: 0.9625\nEpoch 39/50\n753/753 [==============================] - 87s 116ms/step - loss: 0.1400 - binary_accuracy: 0.9515 - val_loss: 0.1026 - val_binary_accuracy: 0.9626\nEpoch 40/50\n753/753 [==============================] - 88s 117ms/step - loss: 0.1395 - binary_accuracy: 0.9515 - val_loss: 0.1025 - val_binary_accuracy: 0.9633\nEpoch 41/50\n753/753 [==============================] - 88s 117ms/step - loss: 0.1390 - binary_accuracy: 0.9516 - val_loss: 0.1011 - val_binary_accuracy: 0.9633\nEpoch 42/50\n753/753 [==============================] - 87s 116ms/step - loss: 0.1384 - binary_accuracy: 0.9521 - val_loss: 0.1017 - val_binary_accuracy: 0.9630\nEpoch 43/50\n753/753 [==============================] - 85s 114ms/step - loss: 0.1376 - binary_accuracy: 0.9520 - val_loss: 0.0992 - val_binary_accuracy: 0.9645\nEpoch 44/50\n753/753 [==============================] - 88s 117ms/step - loss: 0.1372 - binary_accuracy: 0.9522 - val_loss: 0.0989 - val_binary_accuracy: 0.9646\nEpoch 45/50\n753/753 [==============================] - 87s 116ms/step - loss: 0.1371 - binary_accuracy: 0.9522 - val_loss: 0.0982 - val_binary_accuracy: 0.9650\nEpoch 46/50\n753/753 [==============================] - 88s 116ms/step - loss: 0.1359 - binary_accuracy: 0.9527 - val_loss: 0.0983 - val_binary_accuracy: 0.9647\nEpoch 47/50\n753/753 [==============================] - 85s 113ms/step - loss: 0.1362 - binary_accuracy: 0.9526 - val_loss: 0.0966 - val_binary_accuracy: 0.9652\nEpoch 48/50\n753/753 [==============================] - 89s 118ms/step - loss: 0.1352 - binary_accuracy: 0.9527 - val_loss: 0.0953 - val_binary_accuracy: 0.9663\nEpoch 49/50\n753/753 [==============================] - 87s 115ms/step - loss: 0.1350 - binary_accuracy: 0.9528 - val_loss: 0.0956 - val_binary_accuracy: 0.9653\nEpoch 50/50\n753/753 [==============================] - 86s 115ms/step - loss: 0.1342 - binary_accuracy: 0.9531 - val_loss: 0.0946 - val_binary_accuracy: 0.9661", "votes": null }, { "id": "1985082", "postDate": "10/13/2022 05:04:29", "content": "

Maybe something wrong with \"binary_accuracy\". I should use some sort of log loss right, shouldn't I?

", "rawMarkdown": "Maybe something wrong with \"binary_accuracy\". I should use some sort of log loss right, shouldn't I?", "votes": null }, { "id": "1985180", "postDate": "10/13/2022 07:04:17", "content": "

How have you split train and validation? The val_loss looks too good to be true.

", "rawMarkdown": "How have you split train and validation? The val_loss looks too good to be true.", "votes": null }, { "id": "1985227", "postDate": "10/13/2022 07:29:12", "content": "

Thank you for the response.
\nI do it in this way!

\n
X = train.fillna(0).copy()\n\nX_train = X.sample(frac=0.7, random_state=0)\nX_valid = X.drop(X_train.index)\n\ny_A=train['team_A_scoring_within_10sec']\ny_B=train['team_B_scoring_within_10sec']\n\ny_A_train = y_A.iloc[X_train.index]\ny_B_train = y_B.iloc[X_train.index]\ny_A_valid = y_A.drop(X_train.index)\ny_B_valid = y_B.drop(X_train.index)\n
", "rawMarkdown": "Thank you for the response.\nI do it in this way!\n ```\nX = train.fillna(0).copy()\n\nX_train = X.sample(frac=0.7, random_state=0)\nX_valid = X.drop(X_train.index)\n\ny_A=train['team_A_scoring_within_10sec']\ny_B=train['team_B_scoring_within_10sec']\n\ny_A_train = y_A.iloc[X_train.index]\ny_B_train = y_B.iloc[X_train.index]\ny_A_valid = y_A.drop(X_train.index)\ny_B_valid = y_B.drop(X_train.index)\n```", "votes": null }, { "id": "1985256", "postDate": "10/13/2022 07:50:09", "content": "

You haven't yet removed the target columns from the X_train dataframe (apologies if this sounds like \"have you tried switching it off and on again\")

", "rawMarkdown": "You haven't yet removed the target columns from the X_train dataframe *(apologies if this sounds like \"have you tried switching it off and on again\")*", "votes": null }, { "id": "1985257", "postDate": "10/13/2022 07:50:17", "content": "

You have some data leakage. Frames from the same game can appear both in the train and validation sets. It is better to group by event or game to prevent this. GroupKFold is one way to do it.

", "rawMarkdown": "You have some data leakage. Frames from the same game can appear both in the train and validation sets. It is better to group by event or game to prevent this. GroupKFold is one way to do it.", "votes": null }, { "id": "1985275", "postDate": "10/13/2022 07:58:47", "content": "

Thank you, i will check your idea. But in test set we don't have series of games. We have random test set. Why if we choose train and validation randomly it leads us to overfitting and the data leakage? Because it is possible that in train set in this way we get games and rows without winning and in validation set for example only 10 sec winning rows, it is not balanced, isn't it?

", "rawMarkdown": "Thank you, i will check your idea. But in test set we don't have series of games. We have random test set. Why if we choose train and validation randomly it leads us to overfitting and the data leakage? Because it is possible that in train set in this way we get games and rows without winning and in validation set for example only 10 sec winning rows, it is not balanced, isn't it?", "votes": null }, { "id": "1985301", "postDate": "10/13/2022 08:13:04", "content": "

https://www.kaggle.com/code/viktortaran/tps-oct-2022

\n

I published my notebook.

", "rawMarkdown": "https://www.kaggle.com/code/viktortaran/tps-oct-2022\n\nI published my notebook.", "votes": null }, { "id": "1985302", "postDate": "10/13/2022 08:13:11", "content": "

https://www.kaggle.com/code/viktortaran/tps-oct-2022

", "rawMarkdown": "https://www.kaggle.com/code/viktortaran/tps-oct-2022", "votes": null }, { "id": "1985329", "postDate": "10/13/2022 08:49:13", "content": "

@samuelcortinhas is quite right. You have leaked information by not splitting on whole games.

\n

Changing:
\n X_train = X.sample(frac=0.7, random_state=0)
\nto:
\n split_idx = int(len(X) * 0.7)
\n X_train = X[:split_idx]

\n

Will illustrate.

\n

The validation loss no longer improves and stays around ~0.2 and early stopping kicks in after 5 epochs.

\n
\n

Why if we choose train and validation randomly it leads us to overfitting and the data leakage?

\n
\n

There are lots of very similar rows when a goal is scored. Splitting them between train and validation gives the model a big clue.

", "rawMarkdown": "samuelcortinhas is quite right. You have leaked information by not splitting on whole games. \n\nChanging: \n X_train = X.sample(frac=0.7, random_state=0)\nto: \n split_idx = int(len(X) * 0.7)\n X_train = X[:split_idx]\n\nWill illustrate. \n\nThe validation loss no longer improves and stays around ~0.2 and early stopping kicks in after 5 epochs.\n\n>Why if we choose train and validation randomly it leads us to overfitting and the data leakage?\n\nThere are lots of *very similar rows* when a goal is scored. Splitting them between train and validation gives the model a big clue.", "votes": null }, { "id": "1985348", "postDate": "10/13/2022 09:08:10", "content": "

Yeah, thx, I need sleep on it.

", "rawMarkdown": "Yeah, thx, I need sleep on it.", "votes": null }, { "id": "1985509", "postDate": "10/13/2022 11:34:43", "content": "

Your advice are beautiful. I got 0.195 for A and 0.189 for B.

", "rawMarkdown": "Your advice are beautiful. I got 0.195 for A and 0.189 for B.", "votes": null } ], "comments": [ { "id": 1985082, "author_name": "viktortaran", "author_url": "", "post_date": "10/13/2022 05:04:29", "content": "

Maybe something wrong with \"binary_accuracy\". I should use some sort of log loss right, shouldn't I?

", "votes": null, "replies": [] }, { "id": 1985180, "author_name": "paddykb", "author_url": "", "post_date": "10/13/2022 07:04:17", "content": "

How have you split train and validation? The val_loss looks too good to be true.

", "votes": null, "replies": [ { "id": 1985227, "author_name": "viktortaran", "author_url": "", "post_date": "10/13/2022 07:29:12", "content": "

Thank you for the response.
\nI do it in this way!

\n
X = train.fillna(0).copy()\n\nX_train = X.sample(frac=0.7, random_state=0)\nX_valid = X.drop(X_train.index)\n\ny_A=train['team_A_scoring_within_10sec']\ny_B=train['team_B_scoring_within_10sec']\n\ny_A_train = y_A.iloc[X_train.index]\ny_B_train = y_B.iloc[X_train.index]\ny_A_valid = y_A.drop(X_train.index)\ny_B_valid = y_B.drop(X_train.index)\n
", "votes": null, "replies": [] }, { "id": 1985256, "author_name": "paddykb", "author_url": "", "post_date": "10/13/2022 07:50:09", "content": "

You haven't yet removed the target columns from the X_train dataframe (apologies if this sounds like \"have you tried switching it off and on again\")

", "votes": null, "replies": [] }, { "id": 1985257, "author_name": "samuelcortinhas", "author_url": "", "post_date": "10/13/2022 07:50:17", "content": "

You have some data leakage. Frames from the same game can appear both in the train and validation sets. It is better to group by event or game to prevent this. GroupKFold is one way to do it.

", "votes": null, "replies": [] }, { "id": 1985275, "author_name": "viktortaran", "author_url": "", "post_date": "10/13/2022 07:58:47", "content": "

Thank you, i will check your idea. But in test set we don't have series of games. We have random test set. Why if we choose train and validation randomly it leads us to overfitting and the data leakage? Because it is possible that in train set in this way we get games and rows without winning and in validation set for example only 10 sec winning rows, it is not balanced, isn't it?

", "votes": null, "replies": [] }, { "id": 1985302, "author_name": "viktortaran", "author_url": "", "post_date": "10/13/2022 08:13:11", "content": "

https://www.kaggle.com/code/viktortaran/tps-oct-2022

", "votes": null, "replies": [] }, { "id": 1985329, "author_name": "paddykb", "author_url": "", "post_date": "10/13/2022 08:49:13", "content": "

@samuelcortinhas is quite right. You have leaked information by not splitting on whole games.

\n

Changing:
\n X_train = X.sample(frac=0.7, random_state=0)
\nto:
\n split_idx = int(len(X) * 0.7)
\n X_train = X[:split_idx]

\n

Will illustrate.

\n

The validation loss no longer improves and stays around ~0.2 and early stopping kicks in after 5 epochs.

\n
\n

Why if we choose train and validation randomly it leads us to overfitting and the data leakage?

\n
\n

There are lots of very similar rows when a goal is scored. Splitting them between train and validation gives the model a big clue.

", "votes": null, "replies": [] }, { "id": 1985348, "author_name": "viktortaran", "author_url": "", "post_date": "10/13/2022 09:08:10", "content": "

Yeah, thx, I need sleep on it.

", "votes": null, "replies": [] }, { "id": 1985509, "author_name": "viktortaran", "author_url": "", "post_date": "10/13/2022 11:34:43", "content": "

Your advice are beautiful. I got 0.195 for A and 0.189 for B.

", "votes": null, "replies": [] } ] }, { "id": 1985301, "author_name": "viktortaran", "author_url": "", "post_date": "10/13/2022 08:13:04", "content": "

https://www.kaggle.com/code/viktortaran/tps-oct-2022

\n

I published my notebook.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1985079": "Hello to everyone. I am going slightly mad. Please I need you help.\n\nThe question 1.\nIs it possible to get good result with kras?\n\nThe question 2.\nThe more epochs i set up the better result I have with training set but the worse result with finish test set.\n\nMy settings. Should I change my metrics here? I hope it is the case of overfitting, but I can't understand why?\nI have around 0.21 result with epochs = 5 and around 0.25 with epoch = 50.\n\n```\nfrom tensorflow import keras\nfrom tensorflow.keras import layers\n\ninput_shape = [X_train.shape[1]]\n\nmodelA = keras.Sequential([\nlayers.BatchNormalization(input_shape = input_shape),\nlayers.Dense(512,activation='relu'),\nlayers.BatchNormalization(),\nlayers.Dropout(0.3),\nlayers.Dense(512,activation='relu'),\nlayers.BatchNormalization(),\nlayers.Dropout(0.3),\nlayers.Dense(512,activation='relu'),\nlayers.BatchNormalization(),\nlayers.Dropout(0.3),\nlayers.Dense(1,activation='sigmoid'),\n])`\n\nmodelA.compile( optimizer='adam', loss='binary_crossentropy', metrics=['binary_accuracy'], )\n\nearly_stopping = keras.callbacks.EarlyStopping( patience=5, min_delta=0.001, restore_best_weights=True, ) history = modelA.fit( X_train, y_A_train, validation_data=(X_valid, y_A_valid), batch_size=2000, epochs=50, callbacks=[early_stopping], )\n\n2022-10-12 05:42:24.574704: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:185] None of the MLIR \n```\nOptimization Passes are enabled (registered 2)\nEpoch 1/50\n753/753 [==============================] - 87s 113ms/step - loss: 0.2470 - binary_accuracy: 0.9248 - val_loss: 0.2009 - val_binary_accuracy: 0.9436\nEpoch 2/50\n753/753 [==============================] - 83s 110ms/step - loss: 0.2060 - binary_accuracy: 0.9420 - val_loss: 0.1975 - val_binary_accuracy: 0.9439\nEpoch 3/50\n753/753 [==============================] - 84s 111ms/step - loss: 0.2031 - binary_accuracy: 0.9424 - val_loss: 0.1959 - val_binary_accuracy: 0.9441\nEpoch 4/50\n753/753 [==============================] - 85s 113ms/step - loss: 0.2009 - binary_accuracy: 0.9425 - val_loss: 0.1939 - val_binary_accuracy: 0.9442\nEpoch 5/50\n753/753 [==============================] - 85s 113ms/step - loss: 0.1985 - binary_accuracy: 0.9428 - val_loss: 0.1912 - val_binary_accuracy: 0.9443\nEpoch 6/50\n753/753 [==============================] - 86s 114ms/step - loss: 0.1963 - binary_accuracy: 0.9428 - val_loss: 0.1878 - val_binary_accuracy: 0.9444\nEpoch 7/50\n753/753 [==============================] - 85s 113ms/step - loss: 0.1934 - binary_accuracy: 0.9430 - val_loss: 0.1839 - val_binary_accuracy: 0.9446\nEpoch 8/50\n753/753 [==============================] - 84s 112ms/step - loss: 0.1902 - binary_accuracy: 0.9432 - val_loss: 0.1791 - val_binary_accuracy: 0.9448\nEpoch 9/50\n753/753 [==============================] - 84s 112ms/step - loss: 0.1870 - binary_accuracy: 0.9435 - val_loss: 0.1740 - val_binary_accuracy: 0.9451\nEpoch 10/50\n753/753 [==============================] - 85s 112ms/step - loss: 0.1838 - binary_accuracy: 0.9436 - val_loss: 0.1700 - val_binary_accuracy: 0.9457\nEpoch 11/50\n753/753 [==============================] - 84s 112ms/step - loss: 0.1804 - binary_accuracy: 0.9438 - val_loss: 0.1653 - val_binary_accuracy: 0.9460\nEpoch 12/50\n753/753 [==============================] - 84s 112ms/step - loss: 0.1778 - binary_accuracy: 0.9441 - val_loss: 0.1601 - val_binary_accuracy: 0.9463\nEpoch 13/50\n753/753 [==============================] - 86s 114ms/step - loss: 0.1749 - binary_accuracy: 0.9445 - val_loss: 0.1567 - val_binary_accuracy: 0.9472\nEpoch 14/50\n753/753 [==============================] - 84s 111ms/step - loss: 0.1721 - binary_accuracy: 0.9447 - val_loss: 0.1523 - val_binary_accuracy: 0.9482\nEpoch 15/50\n753/753 [==============================] - 85s 113ms/step - loss: 0.1696 - binary_accuracy: 0.9451 - val_loss: 0.1485 - val_binary_accuracy: 0.9486\nEpoch 16/50\n753/753 [==============================] - 87s 116ms/step - loss: 0.1673 - binary_accuracy: 0.9454 - val_loss: 0.1441 - val_binary_accuracy: 0.9497\nEpoch 17/50\n753/753 [==============================] - 87s 116ms/step - loss: 0.1652 - binary_accuracy: 0.9458 - val_loss: 0.1422 - val_binary_accuracy: 0.9498\nEpoch 18/50\n753/753 [==============================] - 87s 115ms/step - loss: 0.1631 - binary_accuracy: 0.9461 - val_loss: 0.1379 - val_binary_accuracy: 0.9514\nEpoch 19/50\n753/753 [==============================] - 86s 115ms/step - loss: 0.1609 - binary_accuracy: 0.9466 - val_loss: 0.1350 - val_binary_accuracy: 0.9520\nEpoch 20/50\n753/753 [==============================] - 86s 115ms/step - loss: 0.1594 - binary_accuracy: 0.9468 - val_loss: 0.1330 - val_binary_accuracy: 0.9522\nEpoch 21/50\n753/753 [==============================] - 86s 114ms/step - loss: 0.1579 - binary_accuracy: 0.9471 - val_loss: 0.1297 - val_binary_accuracy: 0.9535\nEpoch 22/50\n753/753 [==============================] - 87s 116ms/step - loss: 0.1562 - binary_accuracy: 0.9475 - val_loss: 0.1282 - val_binary_accuracy: 0.9543\nEpoch 23/50\n753/753 [==============================] - 88s 118ms/step - loss: 0.1548 - binary_accuracy: 0.9477 - val_loss: 0.1260 - val_binary_accuracy: 0.9549\nEpoch 24/50\n753/753 [==============================] - 88s 118ms/step - loss: 0.1534 - binary_accuracy: 0.9480 - val_loss: 0.1230 - val_binary_accuracy: 0.9562\nEpoch 25/50\n753/753 [==============================] - 87s 115ms/step - loss: 0.1520 - binary_accuracy: 0.9484 - val_loss: 0.1226 - val_binary_accuracy: 0.9560\nEpoch 26/50\n753/753 [==============================] - 87s 115ms/step - loss: 0.1512 - binary_accuracy: 0.9485 - val_loss: 0.1197 - val_binary_accuracy: 0.9565\nEpoch 27/50\n753/753 [==============================] - 89s 118ms/step - loss: 0.1501 - binary_accuracy: 0.9488 - val_loss: 0.1174 - val_binary_accuracy: 0.9581\nEpoch 28/50\n753/753 [==============================] - 87s 116ms/step - loss: 0.1489 - binary_accuracy: 0.9490 - val_loss: 0.1167 - val_binary_accuracy: 0.9584\nEpoch 29/50\n753/753 [==============================] - 87s 115ms/step - loss: 0.1477 - binary_accuracy: 0.9493 - val_loss: 0.1153 - val_binary_accuracy: 0.9590\nEpoch 30/50\n753/753 [==============================] - 90s 119ms/step - loss: 0.1469 - binary_accuracy: 0.9495 - val_loss: 0.1140 - val_binary_accuracy: 0.9591\nEpoch 31/50\n753/753 [==============================] - 86s 115ms/step - loss: 0.1463 - binary_accuracy: 0.9498 - val_loss: 0.1121 - val_binary_accuracy: 0.9599\nEpoch 32/50\n753/753 [==============================] - 86s 114ms/step - loss: 0.1452 - binary_accuracy: 0.9501 - val_loss: 0.1113 - val_binary_accuracy: 0.9597\nEpoch 33/50\n753/753 [==============================] - 86s 114ms/step - loss: 0.1447 - binary_accuracy: 0.9503 - val_loss: 0.1093 - val_binary_accuracy: 0.9615\nEpoch 34/50\n753/753 [==============================] - 87s 116ms/step - loss: 0.1437 - binary_accuracy: 0.9505 - val_loss: 0.1088 - val_binary_accuracy: 0.9610\nEpoch 35/50\n753/753 [==============================] - 86s 114ms/step - loss: 0.1430 - binary_accuracy: 0.9507 - val_loss: 0.1074 - val_binary_accuracy: 0.9626\nEpoch 36/50\n753/753 [==============================] - 86s 114ms/step - loss: 0.1422 - binary_accuracy: 0.9508 - val_loss: 0.1055 - val_binary_accuracy: 0.9621\nEpoch 37/50\n753/753 [==============================] - 87s 115ms/step - loss: 0.1417 - binary_accuracy: 0.9508 - val_loss: 0.1049 - val_binary_accuracy: 0.9621\nEpoch 38/50\n753/753 [==============================] - 87s 116ms/step - loss: 0.1408 - binary_accuracy: 0.9513 - val_loss: 0.1042 - val_binary_accuracy: 0.9625\nEpoch 39/50\n753/753 [==============================] - 87s 116ms/step - loss: 0.1400 - binary_accuracy: 0.9515 - val_loss: 0.1026 - val_binary_accuracy: 0.9626\nEpoch 40/50\n753/753 [==============================] - 88s 117ms/step - loss: 0.1395 - binary_accuracy: 0.9515 - val_loss: 0.1025 - val_binary_accuracy: 0.9633\nEpoch 41/50\n753/753 [==============================] - 88s 117ms/step - loss: 0.1390 - binary_accuracy: 0.9516 - val_loss: 0.1011 - val_binary_accuracy: 0.9633\nEpoch 42/50\n753/753 [==============================] - 87s 116ms/step - loss: 0.1384 - binary_accuracy: 0.9521 - val_loss: 0.1017 - val_binary_accuracy: 0.9630\nEpoch 43/50\n753/753 [==============================] - 85s 114ms/step - loss: 0.1376 - binary_accuracy: 0.9520 - val_loss: 0.0992 - val_binary_accuracy: 0.9645\nEpoch 44/50\n753/753 [==============================] - 88s 117ms/step - loss: 0.1372 - binary_accuracy: 0.9522 - val_loss: 0.0989 - val_binary_accuracy: 0.9646\nEpoch 45/50\n753/753 [==============================] - 87s 116ms/step - loss: 0.1371 - binary_accuracy: 0.9522 - val_loss: 0.0982 - val_binary_accuracy: 0.9650\nEpoch 46/50\n753/753 [==============================] - 88s 116ms/step - loss: 0.1359 - binary_accuracy: 0.9527 - val_loss: 0.0983 - val_binary_accuracy: 0.9647\nEpoch 47/50\n753/753 [==============================] - 85s 113ms/step - loss: 0.1362 - binary_accuracy: 0.9526 - val_loss: 0.0966 - val_binary_accuracy: 0.9652\nEpoch 48/50\n753/753 [==============================] - 89s 118ms/step - loss: 0.1352 - binary_accuracy: 0.9527 - val_loss: 0.0953 - val_binary_accuracy: 0.9663\nEpoch 49/50\n753/753 [==============================] - 87s 115ms/step - loss: 0.1350 - binary_accuracy: 0.9528 - val_loss: 0.0956 - val_binary_accuracy: 0.9653\nEpoch 50/50\n753/753 [==============================] - 86s 115ms/step - loss: 0.1342 - binary_accuracy: 0.9531 - val_loss: 0.0946 - val_binary_accuracy: 0.9661", "1985082": "Maybe something wrong with \"binary_accuracy\". I should use some sort of log loss right, shouldn't I?", "1985180": "How have you split train and validation? The val_loss looks too good to be true.", "1985227": "Thank you for the response.\nI do it in this way!\n ```\nX = train.fillna(0).copy()\n\nX_train = X.sample(frac=0.7, random_state=0)\nX_valid = X.drop(X_train.index)\n\ny_A=train['team_A_scoring_within_10sec']\ny_B=train['team_B_scoring_within_10sec']\n\ny_A_train = y_A.iloc[X_train.index]\ny_B_train = y_B.iloc[X_train.index]\ny_A_valid = y_A.drop(X_train.index)\ny_B_valid = y_B.drop(X_train.index)\n```", "1985256": "You haven't yet removed the target columns from the X_train dataframe *(apologies if this sounds like \"have you tried switching it off and on again\")*", "1985257": "You have some data leakage. Frames from the same game can appear both in the train and validation sets. It is better to group by event or game to prevent this. GroupKFold is one way to do it.", "1985275": "Thank you, i will check your idea. But in test set we don't have series of games. We have random test set. Why if we choose train and validation randomly it leads us to overfitting and the data leakage? Because it is possible that in train set in this way we get games and rows without winning and in validation set for example only 10 sec winning rows, it is not balanced, isn't it?", "1985301": "https://www.kaggle.com/code/viktortaran/tps-oct-2022\n\nI published my notebook.", "1985302": "https://www.kaggle.com/code/viktortaran/tps-oct-2022", "1985329": "samuelcortinhas is quite right. You have leaked information by not splitting on whole games. \n\nChanging: \n X_train = X.sample(frac=0.7, random_state=0)\nto: \n split_idx = int(len(X) * 0.7)\n X_train = X[:split_idx]\n\nWill illustrate. \n\nThe validation loss no longer improves and stays around ~0.2 and early stopping kicks in after 5 epochs.\n\n>Why if we choose train and validation randomly it leads us to overfitting and the data leakage?\n\nThere are lots of *very similar rows* when a goal is scored. Splitting them between train and validation gives the model a big clue.", "1985348": "Yeah, thx, I need sleep on it.", "1985509": "Your advice are beautiful. I got 0.195 for A and 0.189 for B." }, "source": "meta" }