{"metadata":{"kernelspec":{"name":"python3","display_name":"Python 3","language":"python"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":59094,"databundleVersionId":7010844,"sourceType":"competition"}],"dockerImageVersionId":30616,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## Please Read","metadata":{}},{"cell_type":"markdown","source":"The following notebook is an adaptation of a popular [notebook](https://www.kaggle.com/code/jhoward/getting-started-with-nlp-for-absolute-beginners) by Jeremy Howard discussed in fast.ai's [Practical Deep Learning for Coders – Lesson 4](https://course.fast.ai/Lessons/lesson4.html).\n\nIn order for this notebook to run on Kaggle in a reasonable amount of time (~85 min) the model will be trained on only 5 % of the training data sampled randomly. Please note that this notebook serves only as a training demo, since for best results the model needs to be fine-tuned on all training data. For context: Fine-tuning the model for 5 epochs on an A10 GPU using a batch size of 256 and a train-valid-split of 80/20 took about 6 hours. For predictions submitted to the Open Problems – Single-Cell Perturbation competition the model was fine-tuned for 20–30 epochs.","metadata":{}},{"cell_type":"markdown","source":"## Imports","metadata":{}},{"cell_type":"code","source":"import time\nt0start = time.time()\nfrom fastai.collab import *\nfrom fastai.tabular.all import *\nimport warnings\nwarnings.filterwarnings('ignore')","metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","tags":[],"execution":{"iopub.status.busy":"2023-12-08T20:11:01.212993Z","iopub.execute_input":"2023-12-08T20:11:01.213344Z","iopub.status.idle":"2023-12-08T20:11:07.179885Z","shell.execute_reply.started":"2023-12-08T20:11:01.213314Z","shell.execute_reply":"2023-12-08T20:11:07.179048Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"random_seed = 42","metadata":{"tags":[],"execution":{"iopub.status.busy":"2023-12-08T20:11:07.181889Z","iopub.execute_input":"2023-12-08T20:11:07.182404Z","iopub.status.idle":"2023-12-08T20:11:07.187529Z","shell.execute_reply.started":"2023-12-08T20:11:07.182366Z","shell.execute_reply":"2023-12-08T20:11:07.186517Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from transformers import set_seed\nset_seed(random_seed)","metadata":{"tags":[],"execution":{"iopub.status.busy":"2023-12-08T20:11:07.188607Z","iopub.execute_input":"2023-12-08T20:11:07.188902Z","iopub.status.idle":"2023-12-08T20:11:19.885198Z","shell.execute_reply.started":"2023-12-08T20:11:07.188856Z","shell.execute_reply":"2023-12-08T20:11:19.883933Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Prepare Train Set","metadata":{}},{"cell_type":"markdown","source":"Here I read the training data and melt it to yield a ```DataFrame``` with three categorical features (```cell_type```, ```sm_name```, and ```gene```) and one target (```value```).","metadata":{}},{"cell_type":"code","source":"%%time\nfn = '/kaggle/input/open-problems-single-cell-perturbations/de_train.parquet'\ndf_de_train = pd.read_parquet(fn)# , index_col = 0)\ntrain_df = df_de_train.drop(columns=['sm_lincs_id', 'SMILES', 'control'])\ntrdf = train_df.melt(id_vars=['cell_type', 'sm_name'], value_vars=train_df.iloc[:,2:].columns, var_name='gene', value_name='value')","metadata":{"tags":[],"execution":{"iopub.status.busy":"2023-12-08T20:11:19.887474Z","iopub.execute_input":"2023-12-08T20:11:19.888078Z","iopub.status.idle":"2023-12-08T20:11:24.869458Z","shell.execute_reply.started":"2023-12-08T20:11:19.888048Z","shell.execute_reply":"2023-12-08T20:11:24.868534Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The next line samples 5 % of the training data to demonstrate model training in a reasonable time frame:","metadata":{}},{"cell_type":"code","source":"trdf = trdf.sample(frac=0.05).reset_index(drop=true)","metadata":{"execution":{"iopub.status.busy":"2023-12-08T20:11:24.870906Z","iopub.execute_input":"2023-12-08T20:11:24.871334Z","iopub.status.idle":"2023-12-08T20:11:25.732742Z","shell.execute_reply.started":"2023-12-08T20:11:24.871294Z","shell.execute_reply":"2023-12-08T20:11:25.731610Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Prepare Test Set","metadata":{}},{"cell_type":"markdown","source":"Similar procedure for the test set.","metadata":{}},{"cell_type":"code","source":"fn = '/kaggle/input/open-problems-single-cell-perturbations/id_map.csv'\ndf_id_map = pd.read_csv(fn)\nfn = '/kaggle/input/open-problems-single-cell-perturbations/sample_submission.csv'\ndf = pd.read_csv(fn, index_col = 0)\n\ncols_to_add = df_de_train.iloc[:,5:].columns\ncols_to_add\n\ndf_zeros = pd.DataFrame(0.0, columns=cols_to_add, index=df_id_map.index)\ndf_zeros\n\ndf_id_map_preds = pd.concat([df_id_map, df_zeros], axis=1)\ntsdf = df_id_map_preds.melt(id_vars=['cell_type', 'sm_name'], value_vars=df_id_map_preds.iloc[:,3:].columns, var_name='gene', value_name='value')","metadata":{"tags":[],"execution":{"iopub.status.busy":"2023-12-08T20:11:25.734226Z","iopub.execute_input":"2023-12-08T20:11:25.734579Z","iopub.status.idle":"2023-12-08T20:11:30.949465Z","shell.execute_reply.started":"2023-12-08T20:11:25.734550Z","shell.execute_reply":"2023-12-08T20:11:30.948658Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Inspect Train and Test Set","metadata":{}},{"cell_type":"code","source":"trdf","metadata":{"tags":[],"execution":{"iopub.status.busy":"2023-12-08T20:11:30.950586Z","iopub.execute_input":"2023-12-08T20:11:30.950907Z","iopub.status.idle":"2023-12-08T20:11:30.971105Z","shell.execute_reply.started":"2023-12-08T20:11:30.950878Z","shell.execute_reply":"2023-12-08T20:11:30.970038Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tsdf","metadata":{"tags":[],"execution":{"iopub.status.busy":"2023-12-08T20:11:30.972394Z","iopub.execute_input":"2023-12-08T20:11:30.972752Z","iopub.status.idle":"2023-12-08T20:11:30.996498Z","shell.execute_reply.started":"2023-12-08T20:11:30.972722Z","shell.execute_reply":"2023-12-08T20:11:30.995582Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Prepare Transformer Input","metadata":{}},{"cell_type":"code","source":"trdf['input'] = \"Estimate the −log(p-value) confidence for change in gene expression of \" + trdf.gene + \" in \" + trdf.cell_type + \" when treated with \" + trdf.sm_name + \" compared to DMSO.\"\nprint(trdf.input[0])\ntrdf.input.head()","metadata":{"tags":[],"execution":{"iopub.status.busy":"2023-12-08T20:11:30.997788Z","iopub.execute_input":"2023-12-08T20:11:30.998092Z","iopub.status.idle":"2023-12-08T20:11:32.040437Z","shell.execute_reply.started":"2023-12-08T20:11:30.998062Z","shell.execute_reply":"2023-12-08T20:11:32.039329Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tsdf['input'] = \"Estimate the −log(p-value) confidence for change in gene expression of \" + tsdf.gene + \" in \" + tsdf.cell_type + \" when treated with \" + tsdf.sm_name + \" compared to DMSO.\"\nprint(trdf.input[0])\nprint(tsdf.input[0])","metadata":{"tags":[],"execution":{"iopub.status.busy":"2023-12-08T20:11:32.046912Z","iopub.execute_input":"2023-12-08T20:11:32.047244Z","iopub.status.idle":"2023-12-08T20:11:40.951152Z","shell.execute_reply.started":"2023-12-08T20:11:32.047209Z","shell.execute_reply":"2023-12-08T20:11:40.950114Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from datasets import Dataset, DatasetDict\ntrds = Dataset.from_pandas(trdf)\ntrds","metadata":{"tags":[],"execution":{"iopub.status.busy":"2023-12-08T20:11:40.952369Z","iopub.execute_input":"2023-12-08T20:11:40.952719Z","iopub.status.idle":"2023-12-08T20:11:41.711708Z","shell.execute_reply.started":"2023-12-08T20:11:40.952687Z","shell.execute_reply":"2023-12-08T20:11:41.710640Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tsds = Dataset.from_pandas(tsdf)\ntsds","metadata":{"tags":[],"execution":{"iopub.status.busy":"2023-12-08T20:11:41.713397Z","iopub.execute_input":"2023-12-08T20:11:41.714467Z","iopub.status.idle":"2023-12-08T20:11:45.496553Z","shell.execute_reply.started":"2023-12-08T20:11:41.714426Z","shell.execute_reply":"2023-12-08T20:11:45.495405Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Prepare Model","metadata":{}},{"cell_type":"markdown","source":"### Tokenize","metadata":{}},{"cell_type":"code","source":"from transformers import AutoModelForSequenceClassification, AutoTokenizer\nmodel_nm = 'nlpie/tiny-biobert'\ntokz = AutoTokenizer.from_pretrained(model_nm)\n\n# @misc{https://doi.org/10.48550/arxiv.2209.03182,\n#   doi = {10.48550/ARXIV.2209.03182},\n#   url = {https://arxiv.org/abs/2209.03182},\n#   author = {Rohanian, Omid and Nouriborji, Mohammadmahdi and Kouchaki, Samaneh and Clifton, David A.},\n#   keywords = {Computation and Language (cs.CL), Machine Learning (cs.LG), FOS: Computer and information sciences, FOS: Computer and information sciences, 68T50},\n#   title = {On the Effectiveness of Compact Biomedical Transformers},\n#   publisher = {arXiv},\n#   year = {2022}, \n#   copyright = {arXiv.org perpetual, non-exclusive license}\n# }\n","metadata":{"tags":[],"execution":{"iopub.status.busy":"2023-12-08T20:11:45.497681Z","iopub.execute_input":"2023-12-08T20:11:45.497997Z","iopub.status.idle":"2023-12-08T20:11:46.618964Z","shell.execute_reply.started":"2023-12-08T20:11:45.497970Z","shell.execute_reply":"2023-12-08T20:11:46.618106Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"L(tokz.tokenize(trds['input'][0]))","metadata":{"tags":[],"execution":{"iopub.status.busy":"2023-12-08T20:11:46.620413Z","iopub.execute_input":"2023-12-08T20:11:46.620832Z","iopub.status.idle":"2023-12-08T20:11:47.725053Z","shell.execute_reply.started":"2023-12-08T20:11:46.620794Z","shell.execute_reply":"2023-12-08T20:11:47.723935Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('Started tokenization...')\ndef tok_func(x): return tokz(x[\"input\"])\n\ntok_trds = trds.map(tok_func, batched = True)\ntok_tsds = tsds.map(tok_func, batched = True)","metadata":{"tags":[],"execution":{"iopub.status.busy":"2023-12-08T20:11:47.726169Z","iopub.execute_input":"2023-12-08T20:11:47.726467Z","iopub.status.idle":"2023-12-08T20:20:45.032270Z","shell.execute_reply.started":"2023-12-08T20:11:47.726431Z","shell.execute_reply":"2023-12-08T20:20:45.030933Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(tok_trds[0]['input']), L(tok_trds[0]['input_ids'])","metadata":{"tags":[],"execution":{"iopub.status.busy":"2023-12-08T20:20:45.034126Z","iopub.execute_input":"2023-12-08T20:20:45.034571Z","iopub.status.idle":"2023-12-08T20:20:45.044438Z","shell.execute_reply.started":"2023-12-08T20:20:45.034530Z","shell.execute_reply":"2023-12-08T20:20:45.043420Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(tok_tsds[0]['input']), L(tok_tsds[0]['input_ids'])","metadata":{"tags":[],"execution":{"iopub.status.busy":"2023-12-08T20:20:45.045772Z","iopub.execute_input":"2023-12-08T20:20:45.046184Z","iopub.status.idle":"2023-12-08T20:20:45.062978Z","shell.execute_reply.started":"2023-12-08T20:20:45.046155Z","shell.execute_reply":"2023-12-08T20:20:45.062053Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"len(tokz)","metadata":{"tags":[],"execution":{"iopub.status.busy":"2023-12-08T20:20:45.064367Z","iopub.execute_input":"2023-12-08T20:20:45.065372Z","iopub.status.idle":"2023-12-08T20:20:45.079581Z","shell.execute_reply.started":"2023-12-08T20:20:45.065336Z","shell.execute_reply":"2023-12-08T20:20:45.078587Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for i in tok_tsds[0]['input_ids']:\n    print(tokz.decode(i))","metadata":{"tags":[],"execution":{"iopub.status.busy":"2023-12-08T20:20:45.080691Z","iopub.execute_input":"2023-12-08T20:20:45.080996Z","iopub.status.idle":"2023-12-08T20:20:45.091824Z","shell.execute_reply.started":"2023-12-08T20:20:45.080970Z","shell.execute_reply":"2023-12-08T20:20:45.090921Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tok_trds = tok_trds.rename_columns({\"value\":\"labels\"})\ntok_trds","metadata":{"tags":[],"execution":{"iopub.status.busy":"2023-12-08T20:20:45.093027Z","iopub.execute_input":"2023-12-08T20:20:45.093317Z","iopub.status.idle":"2023-12-08T20:20:45.105895Z","shell.execute_reply.started":"2023-12-08T20:20:45.093291Z","shell.execute_reply":"2023-12-08T20:20:45.105086Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Split Train Data","metadata":{}},{"cell_type":"code","source":"dds = tok_trds.train_test_split(0.20, seed=random_seed)\ndds","metadata":{"tags":[],"execution":{"iopub.status.busy":"2023-12-08T20:20:45.107060Z","iopub.execute_input":"2023-12-08T20:20:45.107352Z","iopub.status.idle":"2023-12-08T20:20:45.324705Z","shell.execute_reply.started":"2023-12-08T20:20:45.107327Z","shell.execute_reply":"2023-12-08T20:20:45.323737Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Define Model","metadata":{}},{"cell_type":"code","source":"bs = 256\nepochs = 5\nlr = 8e-5","metadata":{"tags":[],"execution":{"iopub.status.busy":"2023-12-08T20:20:45.325760Z","iopub.execute_input":"2023-12-08T20:20:45.326068Z","iopub.status.idle":"2023-12-08T20:20:45.331009Z","shell.execute_reply.started":"2023-12-08T20:20:45.326042Z","shell.execute_reply":"2023-12-08T20:20:45.329905Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"steps = len(dds['train']) // bs\nsteps","metadata":{"tags":[],"execution":{"iopub.status.busy":"2023-12-08T20:20:45.332234Z","iopub.execute_input":"2023-12-08T20:20:45.332527Z","iopub.status.idle":"2023-12-08T20:20:45.343843Z","shell.execute_reply.started":"2023-12-08T20:20:45.332501Z","shell.execute_reply":"2023-12-08T20:20:45.342884Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from transformers import TrainingArguments, Trainer\n\nargs = TrainingArguments('outputs', save_steps=steps, learning_rate=lr, warmup_ratio=0.1, lr_scheduler_type='cosine', fp16=True,\n    evaluation_strategy=\"epoch\", per_device_train_batch_size=bs, per_device_eval_batch_size=bs*2,\n    num_train_epochs=epochs, weight_decay=0.01, report_to='none', seed=random_seed) # resume_from_checkpoint=\"/kaggle/working/20231005/outputs/checkpoint-104826\", fp16=True","metadata":{"tags":[],"execution":{"iopub.status.busy":"2023-12-08T20:20:45.345273Z","iopub.execute_input":"2023-12-08T20:20:45.345637Z","iopub.status.idle":"2023-12-08T20:20:45.601600Z","shell.execute_reply.started":"2023-12-08T20:20:45.345596Z","shell.execute_reply":"2023-12-08T20:20:45.600652Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from transformers import EvalPrediction\n\ndef compute_metrics(p: EvalPrediction):\n    preds = p.predictions\n    labels = p.label_ids\n    rmse = np.sqrt(((preds - labels) ** 2).mean())\n    return {\"rmse\": rmse}","metadata":{"tags":[],"execution":{"iopub.status.busy":"2023-12-08T20:20:45.602991Z","iopub.execute_input":"2023-12-08T20:20:45.603652Z","iopub.status.idle":"2023-12-08T20:20:45.609933Z","shell.execute_reply.started":"2023-12-08T20:20:45.603614Z","shell.execute_reply":"2023-12-08T20:20:45.608792Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model = AutoModelForSequenceClassification.from_pretrained(model_nm, num_labels=1) # for checkpoint use e.g. \"/kaggle/working/20231005/outputs/checkpoint-104826\" instead of model_nm\ntrainer = Trainer(model, args, train_dataset=dds['train'], eval_dataset=dds['test'],\n                  tokenizer=tokz)#, compute_metrics=compute_metrics)","metadata":{"tags":[],"execution":{"iopub.status.busy":"2023-12-08T20:20:45.611171Z","iopub.execute_input":"2023-12-08T20:20:45.611452Z","iopub.status.idle":"2023-12-08T20:20:53.419767Z","shell.execute_reply.started":"2023-12-08T20:20:45.611427Z","shell.execute_reply":"2023-12-08T20:20:53.418937Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Training","metadata":{}},{"cell_type":"code","source":"print('Started training...')\ntrainer.train()","metadata":{"tags":[],"execution":{"iopub.status.busy":"2023-12-08T20:20:53.420850Z","iopub.execute_input":"2023-12-08T20:20:53.421167Z","iopub.status.idle":"2023-12-08T20:59:40.782084Z","shell.execute_reply.started":"2023-12-08T20:20:53.421141Z","shell.execute_reply":"2023-12-08T20:59:40.781147Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model_string = 'tinybiobert_tasked_one_sentence_5eps42'\ntrainer.save_model(f'./{model_string}/')","metadata":{"tags":[],"execution":{"iopub.status.busy":"2023-12-08T20:59:40.786983Z","iopub.execute_input":"2023-12-08T20:59:40.787275Z","iopub.status.idle":"2023-12-08T20:59:40.941001Z","shell.execute_reply.started":"2023-12-08T20:59:40.787249Z","shell.execute_reply":"2023-12-08T20:59:40.939921Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"valid_loss_history = [log['eval_loss'] for log in trainer.state.log_history if 'eval_loss' in log]\nvalid_step_history = [log['step'] for log in trainer.state.log_history if 'eval_loss' in log]\ntrain_loss_history = [log['loss'] for log in trainer.state.log_history if 'loss' in log]\ntrain_step_history = [log['step'] for log in trainer.state.log_history if 'loss' in log]\nlrate_history = [log['learning_rate'] for log in trainer.state.log_history if 'loss' in log]\nepoch_history = [log['epoch'] for log in trainer.state.log_history if 'eval_loss' in log]\n\nfig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 5))\n\nax1.plot(valid_step_history, valid_loss_history)\nax1.plot(train_step_history, train_loss_history)\nax1.set_xlabel('Step')\nax1.set_ylabel('Loss')\nax1.set_title('Loss History')\n\nax2.plot(train_step_history, lrate_history)\nax2.set_xlabel('Step')\nax2.set_ylabel('Learning Rate')\nax2.set_title('Learning Rate History')\n\nplt.savefig(f'{model_string}_training.pdf', format='pdf')\nplt.tight_layout\nplt.show()","metadata":{"tags":[],"execution":{"iopub.status.busy":"2023-12-08T20:59:40.942252Z","iopub.execute_input":"2023-12-08T20:59:40.942570Z","iopub.status.idle":"2023-12-08T20:59:42.018846Z","shell.execute_reply.started":"2023-12-08T20:59:40.942543Z","shell.execute_reply":"2023-12-08T20:59:42.017788Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with open(f'{model_string}_training_history.txt', 'w') as file:\n    file.write(str(trainer.state.log_history))","metadata":{"tags":[],"execution":{"iopub.status.busy":"2023-12-08T20:59:42.020252Z","iopub.execute_input":"2023-12-08T20:59:42.021108Z","iopub.status.idle":"2023-12-08T20:59:42.026829Z","shell.execute_reply.started":"2023-12-08T20:59:42.021077Z","shell.execute_reply":"2023-12-08T20:59:42.025704Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Predict","metadata":{}},{"cell_type":"code","source":"print('Started inference...')\npreds = trainer.predict(tok_tsds).predictions.astype(float)\npreds","metadata":{"tags":[],"execution":{"iopub.status.busy":"2023-12-08T20:59:42.028376Z","iopub.execute_input":"2023-12-08T20:59:42.028751Z","iopub.status.idle":"2023-12-08T21:36:07.408329Z","shell.execute_reply.started":"2023-12-08T20:59:42.028725Z","shell.execute_reply":"2023-12-08T21:36:07.407177Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"preds.max(), preds.min()","metadata":{"tags":[],"execution":{"iopub.status.busy":"2023-12-08T21:36:07.409660Z","iopub.execute_input":"2023-12-08T21:36:07.410060Z","iopub.status.idle":"2023-12-08T21:36:07.422144Z","shell.execute_reply.started":"2023-12-08T21:36:07.410029Z","shell.execute_reply":"2023-12-08T21:36:07.421045Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"preds_rounded = np.round_(preds, 4)\npreds_rounded","metadata":{"tags":[],"execution":{"iopub.status.busy":"2023-12-08T21:36:07.423604Z","iopub.execute_input":"2023-12-08T21:36:07.424040Z","iopub.status.idle":"2023-12-08T21:36:07.450407Z","shell.execute_reply.started":"2023-12-08T21:36:07.424007Z","shell.execute_reply":"2023-12-08T21:36:07.449377Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In the last step I reshape the predictions back into a 255 x 18211 tensor for submission:","metadata":{}},{"cell_type":"code","source":"to_submit = preds_rounded.reshape(18211, -1).T","metadata":{"tags":[],"execution":{"iopub.status.busy":"2023-12-08T21:36:07.452121Z","iopub.execute_input":"2023-12-08T21:36:07.452429Z","iopub.status.idle":"2023-12-08T21:36:07.456569Z","shell.execute_reply.started":"2023-12-08T21:36:07.452391Z","shell.execute_reply":"2023-12-08T21:36:07.455653Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submit = pd.DataFrame(to_submit, columns=df_de_train.iloc[:,5:].columns)\nsubmit.index.name = 'id'\nsubmit","metadata":{"tags":[],"execution":{"iopub.status.busy":"2023-12-08T21:36:07.457954Z","iopub.execute_input":"2023-12-08T21:36:07.458248Z","iopub.status.idle":"2023-12-08T21:36:07.534512Z","shell.execute_reply.started":"2023-12-08T21:36:07.458222Z","shell.execute_reply":"2023-12-08T21:36:07.533635Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submit.to_csv(f'submission.csv')","metadata":{"tags":[],"execution":{"iopub.status.busy":"2023-12-08T21:36:07.535761Z","iopub.execute_input":"2023-12-08T21:36:07.536081Z","iopub.status.idle":"2023-12-08T21:36:14.835102Z","shell.execute_reply.started":"2023-12-08T21:36:07.536053Z","shell.execute_reply":"2023-12-08T21:36:14.834155Z"},"trusted":true},"execution_count":null,"outputs":[]}]}