{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":59094,"databundleVersionId":7010844,"sourceType":"competition"},{"sourceId":7151495,"sourceType":"datasetVersion","datasetId":3765688}],"dockerImageVersionId":30615,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Single-Cell Perturbations (Part A-Model Training)","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"}},{"cell_type":"markdown","source":"## Feature engineering\n\nAfter many attempts, we finally adopted the following two features from the public notebook \"[OP2: feature engineering](https://www.kaggle.com/code/antoninadolgorukova/op2-feature-engineering/notebook)\" as our training and testing feature.\n\n*(1). PCA followed by target encoding (cell type + drug) without noise (pca_target_encoded_features)*   \n*(2). PCA followed by target encoding (cell type + drug) with noise (pca_target_encoded_features_s0.1)*\n\n(1) is subjected to PCA on 18,211 target variables and produced features representing cell type means for drugs and cell type means for compounds. And by using features (2) that add random noise, we believe this will make the model more generalizable.\n\nThe following public dataset is used to import there two features: \"[OP2: supplementary calcs & data for ML](https://www.kaggle.com/datasets/antoninadolgorukova/op2-supplementary-calcs-for-ml/)\". ","metadata":{}},{"cell_type":"code","source":"import pandas as pd\n\nfrom tensorflow.keras.layers import Dense, Dropout, BatchNormalization, Activation\nfrom tensorflow.keras.models import Sequential\nimport tensorflow as tf\nimport numpy as np","metadata":{"execution":{"iopub.status.busy":"2023-12-08T06:16:09.408793Z","iopub.execute_input":"2023-12-08T06:16:09.409795Z","iopub.status.idle":"2023-12-08T06:16:09.418842Z","shell.execute_reply.started":"2023-12-08T06:16:09.409752Z","shell.execute_reply":"2023-12-08T06:16:09.417167Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"full_features = pd.read_csv(\"/kaggle/input/op2-supplementary-calcs-for-ml/pca_TE_features/pca_target_encoded_features_train.csv\")\nprint(full_features.shape)\nfull_features","metadata":{"execution":{"iopub.status.busy":"2023-12-08T06:16:09.421359Z","iopub.execute_input":"2023-12-08T06:16:09.421749Z","iopub.status.idle":"2023-12-08T06:16:09.817617Z","shell.execute_reply.started":"2023-12-08T06:16:09.421707Z","shell.execute_reply":"2023-12-08T06:16:09.816363Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"full_features_noise = pd.read_csv(\"/kaggle/input/op2-supplementary-calcs-for-ml/pca_TE_features/pca_target_encoded_features_s0.1_train.csv\")\nprint(full_features_noise.shape)\nfull_features_noise","metadata":{"execution":{"iopub.status.busy":"2023-12-08T06:16:09.820174Z","iopub.execute_input":"2023-12-08T06:16:09.820544Z","iopub.status.idle":"2023-12-08T06:16:10.108819Z","shell.execute_reply.started":"2023-12-08T06:16:09.820515Z","shell.execute_reply":"2023-12-08T06:16:10.107212Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"de_train =   pd.read_parquet(\"/kaggle/input/open-problems-single-cell-perturbations/de_train.parquet\")\nlabels_columns=[\"cell_type\",\"sm_name\",\"sm_lincs_id\",\"SMILES\",\"control\"]\nlabels = de_train.drop(columns=labels_columns)\nlabels","metadata":{"execution":{"iopub.status.busy":"2023-12-08T06:16:10.110332Z","iopub.execute_input":"2023-12-08T06:16:10.110662Z","iopub.status.idle":"2023-12-08T06:16:11.907875Z","shell.execute_reply.started":"2023-12-08T06:16:10.110632Z","shell.execute_reply":"2023-12-08T06:16:11.906950Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Use full features for final training \nprint(\"full_features shape:\", full_features.shape)\nprint(\"full_labels shape:\", labels.shape)","metadata":{"execution":{"iopub.status.busy":"2023-12-08T06:16:11.936364Z","iopub.execute_input":"2023-12-08T06:16:11.936950Z","iopub.status.idle":"2023-12-08T06:16:11.943510Z","shell.execute_reply.started":"2023-12-08T06:16:11.936897Z","shell.execute_reply":"2023-12-08T06:16:11.941896Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Get test data \ntest_features = pd.read_csv(\"/kaggle/input/op2-supplementary-calcs-for-ml/pca_TE_features/pca_target_encoded_features_test.csv\")\nprint(test_features.shape)\ntest_features","metadata":{"execution":{"iopub.status.busy":"2023-12-08T06:16:11.945268Z","iopub.execute_input":"2023-12-08T06:16:11.945651Z","iopub.status.idle":"2023-12-08T06:16:12.110014Z","shell.execute_reply.started":"2023-12-08T06:16:11.945618Z","shell.execute_reply":"2023-12-08T06:16:12.108972Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_features_noise = pd.read_csv(\"/kaggle/input/op2-supplementary-calcs-for-ml/pca_TE_features/pca_target_encoded_features_s0.1_test.csv\")\nprint(test_features_noise.shape)\ntest_features_noise","metadata":{"execution":{"iopub.status.busy":"2023-12-08T06:16:12.111546Z","iopub.execute_input":"2023-12-08T06:16:12.111975Z","iopub.status.idle":"2023-12-08T06:16:12.270579Z","shell.execute_reply.started":"2023-12-08T06:16:12.111918Z","shell.execute_reply":"2023-12-08T06:16:12.269418Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_submission = pd.read_csv('../input/open-problems-single-cell-perturbations/sample_submission.csv')\nsample_submission","metadata":{"execution":{"iopub.status.busy":"2023-12-08T06:16:12.271814Z","iopub.execute_input":"2023-12-08T06:16:12.272181Z","iopub.status.idle":"2023-12-08T06:16:15.384503Z","shell.execute_reply.started":"2023-12-08T06:16:12.272152Z","shell.execute_reply":"2023-12-08T06:16:15.383624Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Model structure\n- NN with Fully Connected Layers, as well as BatchNormalization, Dropout, ReLU, and Linear Activation Functions.\n- The initial seed is set to 42 and is fixed. The loss function is mae, the optimizer is Adam\n\nIn **Part A**, we build NN models with the following 4 structures. ","metadata":{}},{"cell_type":"code","source":"def custom_mean_rowwise_rmse(y_true, y_pred):\n    \"\"\"\n    Custom metric to calculate the Mean Rowwise Root Mean Squared Error (RMSE).\n\n    Parameters:\n    - y_true: The true target values.\n    - y_pred: The predicted values.\n\n    Returns:\n    - Mean Rowwise RMSE as a scalar tensor.\n    \"\"\"\n    # Calculate RMSE for each row\n    rmse_per_row = tf.sqrt(tf.reduce_mean(tf.square(y_true - y_pred), axis=1))\n    # Calculate the mean of RMSE values across all rows\n    mean_rmse = tf.reduce_mean(rmse_per_row)\n    \n    return mean_rmse\n\ndef calculate_mae_and_mrrmse(model, data, y_true):\n    \"\"\"\n    Calculate Mean Absolute Error (MAE) and Mean Rowwise Root Mean Squared Error (MRRMSE).\n\n    Parameters:\n    - model: The trained  model.\n    - data: The input data for prediction.\n    - y_true: The true target values.\n    - scaler: The scaler used for data normalization.\n\n    Returns:\n    - None\n    \"\"\"\n    # Predict using the model\n    y_pred_original = model.predict(data, batch_size=1)\n    \n    # Calculate Mean Absolute Error (MAE)\n    mae = mean_absolute_error(y_true , y_pred_original)\n    \n    # Calculate Mean Rowwise Root Mean Squared Error (MRRMSE)\n    rowwise_rmse = np.sqrt(np.mean(np.square(y_true - y_pred_original), axis=1))\n    mrrmse_score = np.mean(rowwise_rmse)\n    \n    # Print the results\n    print(f\"Mean Absolute Error (MAE): {mae}\")\n    print(f\"Mean Rowwise Root Mean Squared Error (MRRMSE): {mrrmse_score}\")","metadata":{"execution":{"iopub.status.busy":"2023-12-08T06:16:15.385671Z","iopub.execute_input":"2023-12-08T06:16:15.386171Z","iopub.status.idle":"2023-12-08T06:16:15.395130Z","shell.execute_reply.started":"2023-12-08T06:16:15.386142Z","shell.execute_reply":"2023-12-08T06:16:15.393691Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Model 1\ntf.random.set_seed(42)\n\nmodel1 = Sequential([ \n    Dense(1228),\n    BatchNormalization(),\n    Activation(\"relu\"),\n    Dropout(0.2),\n    Dense(768),\n    BatchNormalization(),\n    Activation(\"relu\"),\n    Dropout(0.2),\n    Dense(614),\n    BatchNormalization(),\n    Activation(\"relu\"),\n    Dropout(0.2),\n    Dense(512, activation=\"relu\"),\n    Dropout(0.2),\n    Dense(256, activation=\"relu\"),\n    BatchNormalization(),\n    Dropout(0.2),\n    Dense(128, activation=\"relu\"),\n    Dropout(0.2),\n    Dense(64, activation=\"relu\"),\n    Dropout(0.1),\n    Dense(18211,activation= \"linear\")\n])\n\nmodel1.compile(loss=\"mae\", \n                optimizer=tf.keras.optimizers.Adam(),\n                metrics=[custom_mean_rowwise_rmse])","metadata":{"execution":{"iopub.status.busy":"2023-12-08T06:16:15.397011Z","iopub.execute_input":"2023-12-08T06:16:15.397499Z","iopub.status.idle":"2023-12-08T06:16:15.484944Z","shell.execute_reply.started":"2023-12-08T06:16:15.397454Z","shell.execute_reply":"2023-12-08T06:16:15.483642Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#model2\ntf.random.set_seed(42)\n\nmodel2 = Sequential([ \n    Dense(1228),\n    BatchNormalization(),\n    Activation(\"relu\"),\n    Dropout(0.2),\n    Dense(614),\n    BatchNormalization(),\n    Activation(\"relu\"),\n    Dropout(0.2),\n    Dense(512, activation=\"relu\"),\n    Dropout(0.2),\n    Dense(256, activation=\"relu\"),\n    BatchNormalization(),\n    Dropout(0.2),\n    Dense(128, activation=\"relu\"),\n    Dropout(0.2),\n    Dense(64, activation=\"relu\"),\n    Dropout(0.1),\n    Dense(18211,activation= \"linear\")\n])\n\nmodel2.compile(loss=\"mae\", \n                optimizer=tf.keras.optimizers.Adam(),\n                metrics=[custom_mean_rowwise_rmse])","metadata":{"execution":{"iopub.status.busy":"2023-12-08T06:16:15.489674Z","iopub.execute_input":"2023-12-08T06:16:15.490248Z","iopub.status.idle":"2023-12-08T06:16:15.539297Z","shell.execute_reply.started":"2023-12-08T06:16:15.490201Z","shell.execute_reply":"2023-12-08T06:16:15.538126Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#model3\ntf.random.set_seed(42)\n\nmodel3 = Sequential([ \n    Dense(1228),\n    BatchNormalization(),\n    Activation(\"relu\"),\n    Dropout(0.2),\n    Dense(614),\n    BatchNormalization(),\n    Activation(\"relu\"),\n    Dropout(0.2),\n    Dense(512, activation=\"relu\"),\n    Dropout(0.2),\n    Dense(256, activation=\"relu\"),\n    BatchNormalization(),\n    Dropout(0.2),\n    Dense(128, activation=\"relu\"),\n    Dropout(0.1),\n    Dense(18211,activation= \"linear\")\n])\n\nmodel3.compile(loss=\"mae\", \n                optimizer=tf.keras.optimizers.Adam(),\n                metrics=[custom_mean_rowwise_rmse])","metadata":{"execution":{"iopub.status.busy":"2023-12-08T06:16:15.540877Z","iopub.execute_input":"2023-12-08T06:16:15.542335Z","iopub.status.idle":"2023-12-08T06:16:15.593578Z","shell.execute_reply.started":"2023-12-08T06:16:15.542276Z","shell.execute_reply":"2023-12-08T06:16:15.592213Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tf.random.set_seed(42)\n\nmodel4 = Sequential([ \n    Dense(1228),\n    BatchNormalization(),\n    Activation(\"relu\"),\n    Dropout(0.2),\n    Dense(614),\n    Activation(\"relu\"),\n    Dropout(0.2),\n    Dense(512, activation=\"relu\"),\n    BatchNormalization(),\n    Dropout(0.2),\n    Dense(256, activation=\"relu\"),\n    Dropout(0.1),\n    Dense(18211,activation= \"linear\")\n])\n\nmodel4.compile(loss=\"mae\", \n                optimizer=tf.keras.optimizers.Adam(),\n                metrics=[custom_mean_rowwise_rmse])","metadata":{"execution":{"iopub.status.busy":"2023-12-08T06:16:15.595552Z","iopub.execute_input":"2023-12-08T06:16:15.595973Z","iopub.status.idle":"2023-12-08T06:16:15.643096Z","shell.execute_reply.started":"2023-12-08T06:16:15.595918Z","shell.execute_reply":"2023-12-08T06:16:15.641653Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model training example\nBy changing the NN models, input features (with or without noise), and training epochs. We successfully obtained a set of individual models that scored 0.567-0.582 on LB.\n\nThe following is an example of model training and submission file generation. Due to the uncertainty of network prediction of sequence data, we used a large number of submissions to obtain the best possible single models in both CV and LB.\n\nWe then tried multiple ensembles of these models and finally settled on 7 models to determine **Part A**.\n\n","metadata":{}},{"cell_type":"code","source":"history = model3.fit(full_features_noise, labels,\n                       epochs=450,\n                       verbose=1)\n\n#calculate_mae_and_mrrmse(model=model, data=X_val, y_true=y_val)\nprint(\"Done!\")\npreds = model3.predict(test_features_noise, batch_size=1)\n\n\nsample_columns = sample_submission.columns\nsample_columns= sample_columns[1:]\nsubmission_df = pd.DataFrame(preds, columns=sample_columns)\nsubmission_df.insert(0, 'id', range(255))\nsubmission_df.to_csv(\"sub_submission.csv\", index=False)","metadata":{"execution":{"iopub.status.busy":"2023-12-08T06:16:15.645328Z","iopub.execute_input":"2023-12-08T06:16:15.646112Z","iopub.status.idle":"2023-12-08T06:21:22.088853Z","shell.execute_reply.started":"2023-12-08T06:16:15.646066Z","shell.execute_reply":"2023-12-08T06:21:22.086709Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission_df","metadata":{},"execution_count":null,"outputs":[]}]}