{"cells":[{"metadata":{},"cell_type":"markdown","source":"<div class=\"alert alert-success\">  \n</div>"},{"metadata":{},"cell_type":"markdown","source":"<div>\n    <h1 align=\"center\">\"AutoML in Kaggle Kernels\"</h1></h1>\n    <h1 align=\"center\">Rainforest Connection Species Audio Detection</h1>\n    <h4 align=\"center\">By: Somayyeh Gholami & Mehran Kazeminia</h4>\n</div>"},{"metadata":{},"cell_type":"markdown","source":"<div class=\"alert alert-success\">  \n</div>"},{"metadata":{},"cell_type":"markdown","source":"### Description:\n- This type of method only works for some challenges and is not a general method.\n\n- If we could test our answers unlimitedly (it means that there were no five tries limit per day or we could create a Kaggle simulator), we would complete our notebook to automatically optimize the results of all kernels. In this case, even non-experts could optimize the results of the experts' work, and our notebook could approach the concept of \"AutoML\". But our method currently works only empirically as well as \"trial and error\"."},{"metadata":{},"cell_type":"markdown","source":"<div class=\"alert alert-success\">  \n</div>"},{"metadata":{},"cell_type":"markdown","source":"### Import & Data Set"},{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd \nimport matplotlib.pyplot as plt\n\n%matplotlib inline\n\n# _______________________________________\n\n# Kernels Data (Public Score & File Path)\n\nked = pd.DataFrame({      \n    'Kernel ID': ['A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I'],\n    'Symbol':    ['SoliSet', '[Inference] ResNest RFCX Audio Detection',  'notebookba481ef16a', 'All-in-one RFCX baseline for beginners', 'RFCX: train resnet50 with TPU',  'RFCX Resnet50 TPU', 'ResNet34 More Augmentations+Mixup+TTA (Inference)', '[Inference][TPU] RFCX Audio Detection Fast++', 'RFCX Bagging'],\n    'Score':     [ 0.589 , 0.594 , 0.613 , 0.748 , 0.793 , 0.824 , 0.845 , 0.861 , 0.871 ],\n    'File Path': ['../input/audio-detection-soliset-201/submission.csv', '../input/inference-resnest-rfcx-audio-detection/submission.csv', '../input/minimal-fastai-solution-score-0-61/submission.csv', '../input/all-in-one-rfcx-baseline-for-beginners/submission.csv', '../input/rfcx-train-resnet50-with-tpu/submission.csv', '../input/rfcx-resnet50-tpu/submission.csv', '../input/resnet34-more-augmentations-mixup-tta-inference/submission.csv', '../input/inference-tpu-rfcx-audio-detection-fast/submission.csv', '../input/rfcx-bagging-with-different-weights-0871-score/submission.csv'],        \n    'Note'     : ['xgboost & cuml(https://rapids.ai)', 'torch & resnest50', 'fastai.vision & torchaudio', 'torch & resnest50', 'tensorflow & tf.keras.Sequential', 'tensorflow & tf.keras.Sequential', 'tensorflow & classification_models.keras', 'torch & resnest50', 'To sort the scores and use their ranks.']                                                  \n})    \n    \nked    ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<div class=\"alert alert-success\">  \n</div>"},{"metadata":{},"cell_type":"markdown","source":"### Kernel Class & Instances"},{"metadata":{"trusted":true},"cell_type":"code","source":"class Kernel():    \n    '''\n       Class Kernel V 1.0\n       Input Argument:       \n       - symbol      (kernel name OR author)       \n       - score       (Score for the kernel)\n       - file_path   (CSV file address)\n    ''' \n      \n    def __init__(self, symbol, score, file_path):  \n        \n        self.symbol = symbol\n        self.score = score\n        \n        self.file_path = file_path\n        self.sub = pd.read_csv(self.file_path)\n        \n            \n    def __str__(self):\n        return f'Kernel: {self.symbol}\\t| Score: {self.score}'\n\n    \n    def __repr__(self):\n        return f'Class: {self.__class__.__name__}\\nName: {repr(self.symbol)}\\t| Score: {self.score}'   \n\n        \n    def print_head(self):\n        print(self)\n        print(f'\\nHead:\\n')\n        print(self.sub.head())        \n    \n    \n    def print_description(self):\n        print(self)      \n        print(f'\\nDescription:\\n')\n        print(self.sub.describe())\n        \n        \n    def generation(self, other, coeff):\n        g1 = self.sub.copy()\n        g2 = self.sub.copy()\n        g3 = self.sub.copy()\n        g4 = self.sub.copy() \n        \n        if isinstance(other, Kernel):             \n            for i in self.sub.columns[1:]: \n                \n                lm, Is = [], []                \n                lm = self.sub[i].tolist()\n                ls = other.sub[i].tolist()        \n                res1, res2, res3, res4 = [], [], [], []  \n                \n                for j in range(len(self.sub)): \n                    \n                    res1.append(max(lm[j] , ls[j]))\n                    res2.append(min(lm[j] , ls[j]))\n                    res3.append((lm[j] + ls[j]) / 2)\n                    res4.append((lm[j] * coeff) + (ls[j] * (1.- coeff)))        \n        \n                g1[i] = res1\n                g2[i] = res2\n                g3[i] = res3\n                g4[i] = res4\n                \n        return g1,g2,g3,g4   \n    \n# ____________________________________________\n    \n# Seven instance of \"Kernel\" class is defined.\n\nfor i in range(9):   \n    ked.iloc[i, 0] = Kernel(ked.iloc[i, 1], ked.iloc[i, 2], ked.iloc[i, 3])     \n#    ked.iloc[i, 0].print_head() \n#    ked.iloc[i, 0].print_description() \n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# print(ked.iloc[0, 0])\n# ked.iloc[0, 0].sub.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# print(ked.iloc[1, 0])\n# ked.iloc[1, 0].sub.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# print(ked.iloc[2, 0])\n# ked.iloc[2, 0].sub.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# print(ked.iloc[3, 0])\n# ked.iloc[3, 0].sub.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# print(ked.iloc[4, 0])\n# ked.iloc[4, 0].sub.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(ked.iloc[5, 0])\nked.iloc[5, 0].sub.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(ked.iloc[6, 0])\nked.iloc[6, 0].sub.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(ked.iloc[7, 0])\nked.iloc[7, 0].sub.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(ked.iloc[8, 0])\nked.iloc[8, 0].sub.describe()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<div class=\"alert alert-success\">  \n</div>"},{"metadata":{},"cell_type":"markdown","source":"### Increase the best score.\nCan the results of the better kernels support each other? YES:)"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Auxiliary function\ndef generate(main, support, coeff):\n    g1 = main.copy()\n    g2 = main.copy()\n    g3 = main.copy()\n    g4 = main.copy()\n    \n    for i in main.columns[1:]:\n        lm, Is = [], []                \n        lm = main[i].tolist()\n        ls = support[i].tolist() \n        \n        res1, res2, res3, res4 = [], [], [], []          \n        for j in range(len(main)):\n            res1.append(max(lm[j] , ls[j]))\n            res2.append(min(lm[j] , ls[j]))\n            res3.append((lm[j] + ls[j]) / 2)\n            res4.append((lm[j] * coeff) + (ls[j] * (1.- coeff)))\n            \n        g1[i] = res1\n        g2[i] = res2\n        g3[i] = res3\n        g4[i] = res4\n        \n    return g1,g2,g3,g4\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<div class=\"alert alert-success\">  \n</div>"},{"metadata":{},"cell_type":"markdown","source":"## Example:1\nTo increase the score of the best kernel (Score: 0.845), we get help from the kernel with a score of 0.824."},{"metadata":{"trusted":true},"cell_type":"code","source":"g1,g2,g3,g4 = generate(ked.iloc[6, 0].sub, ked.iloc[5, 0].sub, 0.8)\n\n# g1,g2,g3,g4 = ked.iloc[6, 0].generation(ked.iloc[5, 0], 0.8)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# print('Maximum function    | Score: 0.828')\n# g1.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Minimum function    | Score: 0.848')\ng2.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# print('Mean function    | Score: 0.845')\n# g3.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# print('Coefficient function (Coeff: 0.8, 0.2)    | Score: 0.847')\n# g4.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Version 1\n# We have now selected the minimum function.\n\n# sub = g2","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Version-1 results\n### We have now selected the minimum function (g2).\n### [G: (Score: 0.845), F: (Score: 0.824)] >>> g2: (Score: 0.848)"},{"metadata":{},"cell_type":"markdown","source":"## visualization\n### We draw the values of a column (for example, column 17).\n### For better visualization, you can draw the remaining 23 columns in the same way."},{"metadata":{"trusted":true},"cell_type":"code","source":"main = ked.iloc[6, 0].sub\nX  = main.iloc[:, 17]\n\nsupport = ked.iloc[5, 0].sub\nY1 = support.iloc[:, 17]\n\nY2 = g2.iloc[:, 17]\n\nplt.style.use('seaborn-whitegrid')    \nplt.figure(figsize=(8, 8), facecolor='lightgray')\nplt.title(f'<<< S17 >>>\\n\\nOn the X axis >>> G: (Score: 0.845)\\nOn the Y axis >>> F: (Score: 0.824)')\nfor k in range(1992):            \n    plt.scatter(X[k], Y1[k], s=20, alpha=0.8)\nplt.show() \n\nplt.style.use('seaborn-whitegrid')    \nplt.figure(figsize=(8, 8), facecolor='lightgray')\nplt.title(f'<<< S17 >>>\\n\\nOn the X axis >>> G: (Score: 0.845)\\nOn the Y axis >>> g2: (Score: 0.848)')          \nfor k in range(1992):            \n    plt.scatter(X[k], Y2[k], s=20, alpha=0.8)\nplt.show() \n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<div class=\"alert alert-success\">  \n</div>"},{"metadata":{},"cell_type":"markdown","source":"## Example: 2\nTo increase the score of the best kernel (Score: 0.861), we get help from the g2 kernel with a score of 0.848."},{"metadata":{"trusted":true},"cell_type":"code","source":"f1,f2,f3,f4 = generate(ked.iloc[7, 0].sub, g2, 0.8)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# print('Maximum function    | Score: ----')\n# f1.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Minimum function    | Score: 0.866')\nf2.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# print('Mean function    | Score: ----')\n# f3.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# print('Coefficient function (Coeff: 0.8, 0.2)    | Score: ----')\n# f4.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Version 2\n# We have now selected the minimum function.\n\n# sub = f2","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Version-2 results\n### We have now selected the minimum function (f2).\n### [H: (Score: 0.861), g2: (Score: 0.848)] >>> f2: (Score: 0.866)"},{"metadata":{},"cell_type":"markdown","source":"## visualization\n### We draw the values of a column (for example, column 11).\n### For better visualization, you can draw the remaining 23 columns in the same way."},{"metadata":{"trusted":true},"cell_type":"code","source":"main = ked.iloc[7, 0].sub\nX  = main.iloc[:, 11]\n\nsupport = g2\nY1 = support.iloc[:, 11]\n\nY2 = f2.iloc[:, 11]\n\nplt.style.use('seaborn-whitegrid')    \nplt.figure(figsize=(8, 8), facecolor='lightgray')\nplt.title(f'<<< S11 >>>\\n\\nOn the X axis >>> H: (Score: 0.861)\\nOn the Y axis >>> g2: (Score: 0.848)')\nfor k in range(1992):            \n    plt.scatter(X[k], Y1[k], s=20, alpha=0.8)\nplt.show() \n\nplt.style.use('seaborn-whitegrid')    \nplt.figure(figsize=(8, 8), facecolor='lightgray')\nplt.title(f'<<< S11 >>>\\n\\nOn the X axis >>> H: (Score: 0.861)\\nOn the Y axis >>> f2: (Score: 0.866)')          \nfor k in range(1992):            \n    plt.scatter(X[k], Y2[k], s=20, alpha=0.8)\nplt.show() \n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<div class=\"alert alert-success\">  \n</div>"},{"metadata":{},"cell_type":"markdown","source":"## Example: 3\nTo increase the score of the best kernel (Score: 0.871), we get help from the f2 kernel with a score of 0.866."},{"metadata":{"trusted":true},"cell_type":"code","source":"e1,e2,e3,e4 = generate(ked.iloc[8, 0].sub, f2, 0.7)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# print('Maximum function    | Score: ----')\n# e1.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# print('Minimum function    | Score: ----')\n# e2.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Mean function    | Score: 0.874')\ne3.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# print('Coefficient function (Coeff: 0.7, 0.3)    | Score: ----')\n# e4.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Version 3\n# We have selected the mean function.\n\n# sub = e3","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Version-3 results\n### We have now selected the mean function (e3).\n### [I: (Score: 0.871), f2: (Score: 0.866)] >>> e3: (Score: 0.874)"},{"metadata":{},"cell_type":"markdown","source":"## visualization\n### We draw the values of a column (for example, column 21).\n### For better visualization, you can draw the remaining 23 columns in the same way."},{"metadata":{"trusted":true},"cell_type":"code","source":"main = ked.iloc[8, 0].sub\nX  = main.iloc[:, 21]\n\nsupport = f2\nY1 = support.iloc[:, 21]\n\nY2 = e3.iloc[:, 21]\n\nplt.style.use('seaborn-whitegrid')    \nplt.figure(figsize=(8, 8), facecolor='lightgray')\nplt.title(f'<<< S21 >>>\\n\\nOn the X axis >>> I: (Score: 0.871)\\nOn the Y axis >>> f2: (Score: 0.866)')\nfor k in range(1992):            \n    plt.scatter(X[k], Y1[k], s=20, alpha=0.8)\nplt.show() \n\nplt.style.use('seaborn-whitegrid')    \nplt.figure(figsize=(8, 8), facecolor='lightgray')\nplt.title(f'<<< S21 >>>\\n\\nOn the X axis >>> I: (Score: 0.871)\\nOn the Y axis >>> e3: (Score: 0.874)')          \nfor k in range(1992):            \n    plt.scatter(X[k], Y2[k], s=20, alpha=0.8)\nplt.show() \n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<div class=\"alert alert-success\">  \n</div>"},{"metadata":{},"cell_type":"markdown","source":"## Example: 4\nWe want to optimize the previous example (Example # 3), because a notebook with a higher score has not been released yet. that's mean:\n\nTo increase the score of the best kernel (Score: 0.871), we get help from the f2 kernel with a score of 0.866."},{"metadata":{"trusted":true},"cell_type":"code","source":"d1,d2,d3,d4 = generate(ked.iloc[8, 0].sub, f2, 0.45)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# print('Maximum function    | Score: ----')\n# d1.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# print('Minimum function    | Score: ----')\n# d2.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# print('Mean function    | Score: 0.874')\n# d3.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Coefficient function (Coeff: 0.45, 0.55)    | Score: 0.876')\nd4.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Version 4\n# We have selected the Coefficient function.\n\n# sub = d4","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Version-4 results\n### We have now selected the Coefficient function (d4).\n### [I: (Score: 0.871), f2: (Score: 0.866)] >>> d4: (Score: 0.876)"},{"metadata":{},"cell_type":"markdown","source":"## visualization\n### We draw the values of a column (for example, column 21).\n### For better visualization, you can draw the remaining 23 columns in the same way."},{"metadata":{"trusted":true},"cell_type":"code","source":"main = ked.iloc[8, 0].sub\nX  = main.iloc[:, 21]\n\nsupport = f2\nY1 = support.iloc[:, 21]\n\nY2 = d4.iloc[:, 21]\n\nplt.style.use('seaborn-whitegrid')    \nplt.figure(figsize=(8, 8), facecolor='lightgray')\nplt.title(f'<<< S21 >>>\\n\\nOn the X axis >>> I: (Score: 0.871)\\nOn the Y axis >>> f2: (Score: 0.866)')\nfor k in range(1992):            \n    plt.scatter(X[k], Y1[k], s=20, alpha=0.8)\nplt.show() \n\nplt.style.use('seaborn-whitegrid')    \nplt.figure(figsize=(8, 8), facecolor='lightgray')\nplt.title(f'<<< S21 >>>\\n\\nOn the X axis >>> I: (Score: 0.871)\\nOn the Y axis >>> d4: (Score: 0.876)')          \nfor k in range(1992):            \n    plt.scatter(X[k], Y2[k], s=20, alpha=0.8)\nplt.show() \n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<div class=\"alert alert-success\">  \n</div>"},{"metadata":{},"cell_type":"markdown","source":"## Example: 5\nAgain, we want to optimize Example 3, because a notebook with a higher score has not been published yet. It means:\n\nTo increase the score of the best kernel (Score: 0.871), we get help from the f2 kernel with a score of 0.866."},{"metadata":{"trusted":true},"cell_type":"code","source":"c1,c2,c3,c4 = generate(ked.iloc[8, 0].sub, f2, 0.475)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# print('Maximum function    | Score: ----')\n# c1.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# print('Minimum function    | Score: ----')\n# c2.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# print('Mean function    | Score: 0.874')\n# c3.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Coefficient function (Coeff: 0.475, 0.525)    | Score: 0.876')\nc4.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Version 5\n# We have selected the coefficient function.\n\nsub = c4","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub.to_csv(\"submission.csv\", index=False)\n\nc1.to_csv(\"submission1.csv\", index=False)\nc2.to_csv(\"submission2.csv\", index=False)\nc3.to_csv(\"submission3.csv\", index=False)\nc4.to_csv(\"submission4.csv\", index=False)\n\n!ls","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<div class=\"alert alert-success\">  \n</div>"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}