{"cells":[{"metadata":{},"cell_type":"markdown","source":"### Model Validation & Learning Rate Monitoring\n\nWrote a quick script in case you print out Train/Val Loss and Learning rate per epoch which is always good practice to understand how the learning rate is scheduled and to make sure you are not overfitting on train set. Monitoring the learning rate vs Val loss can be useful in understanding things like which rate has the highest gradient in val loss.\n\nFormat expected is something like this but it is easy to change based on your output in the regex lines here"},{"metadata":{"trusted":true},"cell_type":"code","source":"fold_line = '.+Fold (.+?)\\s.+'\ntrain_loss_line = 'Epoch (.+?): Train loss: (.+?)\\s.*LR: (.+?)\\s.+'\nval_loss_line = 'Epoch (.+?): Val loss: (.+?)$'","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Full code"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import re\nimport argparse\nimport plotly.graph_objects as go\nfrom plotly.subplots import make_subplots\n\n\"\"\"\nparser = argparse.ArgumentParser()\nparser.add_argument(\"-l\", \"--log\", help=\"Path to ml training log\", required=True)\nargs = parser.parse_args()\nml_log_file = args.log\n\n\"\"\"\n\nml_log_file = '../input/ml-logs/training_log'\n\nfold_line = '.+Fold (.+?)\\s.+'\ntrain_loss_line = 'Epoch (.+?): Train loss: (.+?)\\s.*LR: (.+?)\\s.+'\nval_loss_line = 'Epoch (.+?): Val loss: (.+?)$'\n\nloss_dict = dict()\ntrain_losses = []\nval_losses = []\nlrs = []\n\n#print(f'Parsing {ml_log_file}')\nwith open(ml_log_file) as fp:\n    line = fp.readline()\n    while line:\n        # Extract fold\n        match = re.match(fold_line, line, re.I)\n        if match:\n            if train_losses:\n                loss_dict[fold] = {'train_loss': train_losses, 'val_loss': val_losses, 'lr': lrs}\n            fold = match[1]\n            train_losses = []\n            val_losses = []\n        # Extract train loss\n        match = re.match(train_loss_line, line, re.I)\n        if match:\n            epoch = match[1]\n            train_loss = float(match[2])\n            lr = float(match[3])\n            train_losses.append(train_loss)\n            lrs.append(lr)\n        # Extract val loss\n        match = re.match(val_loss_line, line, re.I)\n        if match:\n            epoch = int(match[1])\n            val_loss = float(match[2])\n            val_losses.append(val_loss)\n\n        line = fp.readline()\n\n\nfor fold in loss_dict:\n    fig = make_subplots(rows=1, cols=2, subplot_titles=(\"Train & Val Loss\", \"Val Loss vs LR\"))\n    fig.update_layout(title=f'Fold: {fold}')\n    train_loss = loss_dict[fold]['train_loss']\n    val_loss = loss_dict[fold]['val_loss']\n    fig.add_trace(go.Scatter(y=train_loss,\n                             mode='lines',\n                             name='Train Loss'), row=1, col=1)\n    fig.add_trace(go.Scatter(y=val_loss,\n                             mode='lines',\n                             name='Val Loss'), row=1, col=1)\n\n    fig.update_layout(title=f'Fold: {fold}')\n    lr = loss_dict[fold]['lr']\n    fig.add_trace(go.Scatter(y=val_loss, x=lr,\n                             mode='lines',\n                             name='Val Loss vs LR'),row=1, col=2)\n    fig.update_layout(height=400, width=1200, title_text=\"Model Validation\")\n    fig.show()\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### The above charts can be used to check a few things:\n* Train & Val Loss vs Epochs Chart - Make sure that val loss continues to decrease and doesnt plateau or worse start increasing as train loss decreases. Overfitting on train cean be avoided using things like Early Stopping\n* Val Loss vs LR curve - Monitor how val loss decreases with learning rate. This can be helpful in finding a good set of learning rates. "}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}