{"cells":[{"metadata":{},"cell_type":"markdown","source":"Hi to all! \n\nFrom the very beginning of this competition it was clear that when using StratifiedKFold we get very different results on each KFold. \nFor example, according to my observations in my model (https://www.kaggle.com/aikhmelnytskyy/resnet-wavenet-my-best-single-model-ensemble) val_lwlrap at 3 KFold is 0.88-0.89 and at two 0.85-0.86. And the results on the public date network range from 0.830 to 850. To solve this problem, I decided to train 10 identical models, but with different SEED parameters. That each KFold got different pieces from a training set. The https://www.kaggle.com/aikhmelnytskyy/rainforest-connection-species-audio-detectionv1 dataset includes all 50 models (5 models for each SEED value). The training code is published in version 3 of this notebook: https://www.kaggle.com/aikhmelnytskyy/resnet-tpu-on-colab-and-kaggle. I ran all the code on Google Colab in parallel, so the code is not very elegant, but if you want you can improve it. This model is different from the one I published in this notebook: https://www.kaggle.com/aikhmelnytskyy/resnet-wavenet-my-best-single-model-ensemble. The reason is the banal possibility of Google Colab is smaller than in the kaggle notebook, so I had to reduce the number of parameters in the models.\n\nNow, after training 50 models, I plan to average them. In my head my ideas are like some Bagging option. Correct me in the comments if I'm wrong\n\nFor the experiment in version 1 of this notebook, I average the best models from each SEED parameter, and in version 2 the worst models. Version 3 is the average of all models.\n\nIn version 4, I average only the 5 best models. I hope you understand that the result of version 3 is the most credible."},{"metadata":{},"cell_type":"markdown","source":"Version 4 showed why you should not rely only on the best validation models :-)\nGood luck to all!!!"},{"metadata":{},"cell_type":"markdown","source":"In version 8, I tried to build an ensemble of this model with the \"best\" public ensemble, the result on the one hand has not changed, but on the other hand has not deteriorated, which is good. I had free attempts so I tried a few more simple ensembles in versions 9,10,11. I hope you have already understood that this ensemble may show good results on a private test set, but it is still extremely unreliable even by the standards of public notebooks."},{"metadata":{},"cell_type":"markdown","source":""},{"metadata":{},"cell_type":"markdown","source":"Version 1 - 0.842\n\nVersion 2 - 0.835\n\nVersion 3 - 0.840\n\nVersion 4 - 0.828\n\nVersion 8,9,10,11 - ensemble\n\nVersion 15 - 0.878 - An ensemble with my new best model. How to get this model I described in these discussions: https://www.kaggle.com/c/rfcx-species-audio-detection/discussion/217301"},{"metadata":{},"cell_type":"markdown","source":"Auxiliary functions"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"from kaggle_datasets import KaggleDatasets\n\n\nimport pandas as pd\n\ndef read_train_csv(n_trains,path):\n    base_df=pd.DataFrame()\n    for i in range(n_trains):   \n        \n        base_df=base_df.append(pd.read_csv(path+f'train_n_{i}.csv'), ignore_index = True) \n    return base_df\n\ndef find_best_train_csv(n_trains,path,n_bests):\n    base_df=pd.DataFrame()\n    for i in range(n_trains):   \n        df=pd.read_csv(path+f'train_n_{i}.csv')                \n        base_df=base_df.append(df.nlargest(n_bests, ['best_score']) , ignore_index = True) \n    return base_df\n\ndef find_worst_train_csv(n_trains,path,n_worst):\n    base_df=pd.DataFrame()\n    for i in range(n_trains):   \n        df=pd.read_csv(path+f'train_n_{i}.csv')                \n        base_df=base_df.append(df.nsmallest(n_worst, ['best_score']) , ignore_index = True) \n    return base_df\n\ndef average_csv(csv_list):\n    BLEND=pd.read_csv('../input/resnet-wavenet-my-best-single-model-ensemble/submission.csv')\n    \n    BLEND.iloc[:,1:]=pd.read_csv(csv_list[0])\n    for sub_csv in csv_list[1:]:\n        \n        BLEND.iloc[:,1:]+=pd.read_csv(sub_csv)\n        \n    BLEND.iloc[:,1:]=BLEND.iloc[:,1:]/len(csv_list)   \n    return BLEND\n    \n    \ndef create_csv_list(df,path):\n    df['CSV'] = path + df['CSV'].astype(str)\n    return df[\"CSV\"].tolist()\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We average all models"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"print (KaggleDatasets().get_gcs_path('rfcx-species-audio-detection'))\n\nn_trains=10\npath='../input/rainforest-connection-species-audio-detectionv1/KAggle/'\n\ntrain_data=read_train_csv(n_trains,path)\nprint(train_data)\nBLEND_all=create_csv_list(train_data,path)\nBLEND_all=average_csv(BLEND_all)\nprint(BLEND_all)\n\n\nBLEND_all.to_csv('submission.csv',index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\nNow let's try an ensemble of public notebooks.\n\nLet's be careful overfitting"},{"metadata":{"trusted":true},"cell_type":"code","source":"sub2 = pd.read_csv('../input/automl-inference-audio-detection-soliset/submission.csv')#0.876\nsub3 = pd.read_csv('../input/fork-of-bagging-rainforest/submission.csv')#0.856\nBLEND_all.iloc[:,1:]=(BLEND_all.iloc[:,1:]*0.30+sub3.iloc[:,1:]*0.70)*0.40+sub2.iloc[:,1:]*0.60/5\nBLEND_all.to_csv('submission_ensemble.csv',index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We average the models with the best val_lwlrap in each SEED"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_best=find_best_train_csv(n_trains,path,3)\nprint(train_best)\n\nBLEND_best=create_csv_list(train_best,path)\nBLEND_best=average_csv(BLEND_best)\nprint(BLEND_best)\nBLEND_best.to_csv('submission_best.csv',index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* We average the models with the worst val_lwlrap in each SEED"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_worst=find_worst_train_csv(n_trains,path,2)\nprint(train_worst)\n\nBLEND_worst=create_csv_list(train_worst,path)\nBLEND_worst=average_csv(BLEND_worst)\nprint(BLEND_worst)\nBLEND_worst.to_csv('submission_worst.csv',index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We average the models with the best val_lwlrap"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data=read_train_csv(n_trains,path)\nprint(train_data)\nBLEND_all=create_csv_list(train_data.nlargest(5, ['best_score']),path)\nBLEND_all=average_csv(BLEND_all)\nprint(BLEND_all)\nBLEND_all.to_csv('submission_best_5.csv',index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Add Post Process from here: https://www.kaggle.com/cdeotte/rainforest-post-process-lb-0-970"},{"metadata":{"trusted":true},"cell_type":"code","source":"MODE = 3\n\n# LOAD SUBMISSION\nimport pandas as pd, numpy as np\nFUDGE = 2.0\nFILE = 'submission.csv'\ndf = BLEND_all\nfor k in range(24):\n    df.iloc[:,1+k] -= df.iloc[:,1+k].min()\n    df.iloc[:,1+k] /= df.iloc[:,1+k].max()\n\n# CONVERT PROBS TO ODDS, APPLY MULTIPLIER, CONVERT BACK TO PROBS\ndef scale(probs, factor):\n    probs = probs.copy()\n    idx = np.where(probs!=1)[0]\n    odds = factor * probs[idx] / (1-probs[idx])\n    probs[idx] =  odds/(1+odds)\n    return probs\n\n# DIFFERENT DISTRIBUTIONS\nd1 = df.iloc[:,1:].mean().values\nd2 = np.array([113, 204, 44, 923, 53, 41, 3, 213, 44, 23, 26, 149, 255,  \n    14, 123, 222, 46, 6, 474, 4, 17, 18, 23, 72])/1000.\n\nfor k in range(24):\n    if MODE==1: d = FUDGE\n    if MODE==2: d = d1[k]/(1-d1[k])\n    if MODE==3: s = d2[k] / d1[k]\n    else: s = (d2[k]/(1-d2[k]))/d\n    df.iloc[:,k+1] = scale(df.iloc[:,k+1].values,s)\n    \nsub4 = pd.read_csv('../input/lb0984rainforestcomparativemethodpartb/submission.csv')#0.984   \n    \nBLEND_all.iloc[:,1:]=df.iloc[:,1:]*0.10+sub4.iloc[:,1:]*0.90\ndf.to_csv('submission_with_pp.csv',index=False)\n\n\n\n\n","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}