{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Why is it worth it?\n\nOur world is **highly dynamic** and even if we find periodic patterns we can't assume that they will stay there forever. When we just train models on data that was collected in the past and apply them only for inference on future data we will likely experience a break-down of our evaluation criteria someday. Especially the pandemic of covid-19 should have opened the eyes that **stochastic events can cause changes in the overall system that are not easy to foresee and/or somehow irrational. In such cases machine learning models that are not able to adapt to environmental changes like a chameleon are doomed to fail.**\n\n\nWith this notebook I like to learn more about evolutionary algorithms and share my experiences with the community. If you like to play with code and ideas, just fork. ;-)\n\nHappy kaggling!\n\n\n","metadata":{}},{"cell_type":"markdown","source":"## Table of contents\n\n1. [What is Differential Evolution?](#differential_evolution)\n2. [How does it work?](#how)\n3. [Code to built the model](#de_model)\n4. [Some simple examples](#simple_examples)\n5. [Prepare for the Titanic](#prepare_titanic)\n6. [Build the NNs for Titanic](#nn_titanic)\n7. [Build an individual for DE](#individual)\n8. [Differential Evolution for Titanic](#de_titanic)\n9. [Understanding the evolution of hyperparameters](#de_hyperpar_evolution)\n10. [Hyperparameter evolution for digits and flowers dataset](#de_digits_and_flowers)\n11. [Conclusion](#conclusion)","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\n\nimport matplotlib.pyplot as plt\n%matplotlib inline\n\nimport seaborn as sns\nsns.set()\n\nfrom matplotlib.colors import BoundaryNorm\nfrom matplotlib.ticker import FixedLocator, MaxNLocator\nimport matplotlib.animation as animation\nfrom matplotlib import rc\nrc('animation', html='html5')\n\nfrom tensorflow import keras\n\nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn.model_selection import train_test_split","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-08T20:01:17.103013Z","iopub.execute_input":"2022-02-08T20:01:17.103484Z","iopub.status.idle":"2022-02-08T20:01:24.366542Z","shell.execute_reply.started":"2022-02-08T20:01:17.103456Z","shell.execute_reply":"2022-02-08T20:01:24.365333Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# What is Differential Evolution? \n\n* An evolutionary algorithm based on a popultion of models/individuals.\n* It's able to find global optima in optimisation landscapes with many local minima.\n* It recombines one individual of the population with a new candidate **(trial)** that is obtained by computing the difference between two further individuals in the population.\n* By computing these so called difference vectors the algorithm is able to self-adapt the stepsize for exploiting the parameter space of the optimisation. \n* It can also be applied in settings where there is no optimisation function at all but some kind of feedback how good the result is.\n* It can also deal with optimisation landscapes that change over time.","metadata":{}},{"cell_type":"markdown","source":"# How does it work? \n\nLet's directly use an example by tuning the hyperparameters of neural networks.\n\n* For this kind of optimisation the function is unknown. We have no mathematical equation that tells us how the scores or error losses behave when we change the learning rate or increase the dropout rate etc.. \n* In this problem setup, each set of parameters we could chose for a neural network is one individual. When we initialize different parameter sets then we obtain our first population.\n\nNow the following recipe has to be followed! ;-)\n\n1. Generate mutant using three different individuals $p_{i}$:\n\n * $ \\vec{m} = \\vec{p}_{A} - s \\cdot \\left( \\vec{p}_{B}-\\vec{p}_{C} \\right)$\n2. Compute a trial individual $\\vec{c}$ by doing crossover using the mutant $\\vec{m}$ and a randomly selected target individual $\\vec{t}$:\n\n * $ \\vec{c} = copy \\left( \\vec{t} \\right) $\n * Randomly select elements $c_{i}$ for being replaced with the mutant element $m_{i}$:\n \n $$c_{i}=\\begin{cases}\n m_{i}, & \\text{if } r R\\\\\n c_{i}, & \\text{else }\n\\end{cases}$$\n \n \n3. Compute losses of trials and parents.\n4. Perform selection by comparing trials and parents and choosing the best for the next generation.\n5. Redo until stopping criertion has been reached. ","metadata":{}},{"cell_type":"markdown","source":"# Code to built the model ","metadata":{}},{"cell_type":"code","source":"class DifferentialEvolution:\n\n def __init__(self, scaling_rate, crossover_rate, population_size):\n self.scaling_rate = scaling_rate\n self.crossover_rate = crossover_rate\n self.population_size = population_size\n self.ranges = []\n \n self.generation_counter = 0\n \n self.all_generations = []\n self.min_std_criterion = 0.001\n \n def set_range(self, paramkey, range):\n self.ranges.append([paramkey, range])\n \n def set_objective(self, objective):\n self.objective = objective\n \n def init_generation(self):\n self.n_params = len(self.ranges)\n self.generation = np.zeros(shape=(self.population_size, self.n_params))\n for p in range(self.n_params):\n low = self.ranges[p][1][0]\n high = self.ranges[p][1][1]\n self.generation[:, p] = np.random.uniform(low, high, size=self.population_size)\n \n def get_trials(self):\n self.trials = np.zeros(shape=(self.population_size, self.n_params))\n for i in range(self.population_size):\n target = self.generation[i, :]\n mutant = self.mutate(i)\n self.trials[i, :] = self.crossover(mutant, target)\n \n def get_candidates(self, i):\n to_select = list(np.arange(self.population_size))\n to_select.remove(i)\n candidates = np.random.choice(to_select, 3, replace=False)\n return candidates\n \n def mutate(self, i):\n candidates = self.get_candidates(i)\n difference_vector = self.generation[candidates[1]] - self.generation[candidates[2]]\n mutant = self.generation[candidates[0]] + self.scaling_rate * difference_vector\n return mutant\n \n def crossover(self, mutant, target):\n crossover_units = np.random.uniform(0, 1, self.n_params)\n trial = np.copy(target)\n random_parameter = np.random.choice(self.n_params)\n for param in range(self.n_params):\n if crossover_units[param] <= self.crossover_rate or param == random_parameter:\n trial[param] = mutant[param]\n return trial\n \n def select(self, generation_costs, trials_costs):\n idx = np.where(trials_costs < generation_costs)[0]\n for i in idx:\n self.generation[i, :] = self.trials[i, :]\n\n def compute_cost(self):\n generation_costs = self.objective(self.generation)\n trials_cost = self.objective(self.trials)\n return generation_costs, trials_cost\n \n def evolve(self):\n self.init_generation()\n self.all_generations.append(self.generation)\n self.best_solutions = []\n gen_costs = self.objective(self.generation)\n while ((np.std(gen_costs) > self.min_std_criterion) and (self.generation_counter < 200)):\n self.get_trials()\n gen_costs, trials_costs = self.compute_cost()\n self.select(gen_costs, trials_costs)\n self.all_generations.append(np.copy(self.generation))\n self.generation_counter += 1\n self.best_solutions.append(np.min(gen_costs))\n print(\"Stopped at generation {}\".format(self.generation_counter))","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-08T20:01:24.369563Z","iopub.execute_input":"2022-02-08T20:01:24.370035Z","iopub.status.idle":"2022-02-08T20:01:24.408074Z","shell.execute_reply.started":"2022-02-08T20:01:24.369990Z","shell.execute_reply":"2022-02-08T20:01:24.406364Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Some simple examples \n\nLet's get a first feeling about how this algorithm works and abouts its performance by searching for optima in the parameter space of:\n\n* the rosenbrock function (coming soon)\n* the griewank function\n* the himmelblau function\n\nJust use an example of your choice and watch Differential Evolution during optimisation.","metadata":{}},{"cell_type":"code","source":"example = \"griewank\"","metadata":{"execution":{"iopub.status.busy":"2022-02-08T20:01:24.410381Z","iopub.execute_input":"2022-02-08T20:01:24.410949Z","iopub.status.idle":"2022-02-08T20:01:24.427680Z","shell.execute_reply.started":"2022-02-08T20:01:24.410902Z","shell.execute_reply":"2022-02-08T20:01:24.426184Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def rosenbrock(w, a=1, b=100):\n return np.power(a-w[:,0],2)+b*np.power(w[:,1]-w[:,0]**2, 2)\n\ndef griewank(w):\n sum_part = np.zeros(w.shape[0])\n cos_part = np.ones(w.shape[0])\n for i in range(w.shape[1]):\n sum_part += np.power(w[:, i], 2)\n cos_part *= np.cos(w[:, i] / np.sqrt(i+1))\n return np.ones(w.shape[0]) + 1/4000 * sum_part - cos_part\n\n\ndef himmelblau(w):\n part1 = np.power(np.power(w[:, 0], 2) + w[:, 1] - 11, 2)\n part2 = np.power(w[:, 0] + np.power(w[:, 1], 2) - 7, 2)\n return part1 + part2 \n\n\nexamples = {\"rosenbrock\": {\"objective\": rosenbrock, \"range\": [-2,3], \"delta\": 0.1}, \n \"himmelblau\": {\"objective\": himmelblau, \"range\": [-6, 6], \"delta\": 0.1},\n \"griewank\": {\"objective\": griewank, \"range\": [-50, 50], \"delta\": 0.5}}","metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-08T20:01:24.430971Z","iopub.execute_input":"2022-02-08T20:01:24.431372Z","iopub.status.idle":"2022-02-08T20:01:24.449541Z","shell.execute_reply.started":"2022-02-08T20:01:24.431327Z","shell.execute_reply":"2022-02-08T20:01:24.448201Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class ShowEvolution:\n\n def __init__(self, example):\n self.model = DifferentialEvolution(scaling_rate=0.7, crossover_rate=0.6, population_size=15)\n self.example = example\n self.range = examples[example][\"range\"]\n self.objective = examples[example][\"objective\"]\n self.delta = examples[example][\"delta\"]\n self.model.set_objective(self.objective)\n self.model.set_range(\"w1\", self.range)\n self.model.set_range(\"w2\", self.range)\n self.model.evolve()\n\n self.dx, self.dy = self.delta, self.delta\n\n tau = np.mgrid[slice(self.range[0], self.range[1] + self.dy, self.dy),\n slice(self.range[0], self.range[1] + self.dx, self.dx)]\n\n self.X = tau[0, :-1, :-1]\n self.Y = tau[1, :-1, :-1]\n self.Z = np.zeros(shape=(tau.shape[1], tau.shape[2]))\n \n for m in range(tau.shape[1]):\n for n in range(tau.shape[2]):\n self.Z[n, m] = self.objective(tau[:, n, m].reshape(1, -1))\n\n def show(self):\n self.fig, self.ax = plt.subplots()\n self.make_nice_contours(self.ax, self.fig)\n\n self.ani = animation.FuncAnimation(self.fig, self.update,\n init_func=self.setup_plot,\n frames=np.arange(len(self.model.all_generations)),\n interval=50,\n blit=True,\n repeat=False)\n Writer = animation.writers['ffmpeg']\n writer = Writer(fps=15, metadata=dict(artist='Me'), bitrate=1800)\n self.ani.save('example.mp4', writer=writer)\n plt.close()\n return self.ani\n \n \n def make_nice_contours(self, ax, fig):\n Z = self.Z[:-1, :-1]\n \n if self.example == \"griewank\":\n levels = MaxNLocator(nbins=50).tick_values(Z.min(), Z.max())\n \n else:\n levels = FixedLocator([0.05, 0.1, 0.25, 0.5, 1, 2.5, 5 , 7.5,\n 10, 20, 40, 80, 160, 320, 640, 1280]).tick_values(vmin=0, vmax=1280)\n \n cmap = plt.get_cmap('coolwarm')\n norm = BoundaryNorm(levels, ncolors=cmap.N, clip=True)\n\n cf = ax.contourf(self.X + self.dx / 2., self.Y + self.dy / 2., Z,\n levels=levels,\n cmap=cmap,\n norm=norm)\n ax.set_xlabel(\"$p_{1}$\")\n ax.set_ylabel(\"$p_{2}$\")\n return plt\n \n def update(self, i):\n \"\"\"Update the scatter plot.\"\"\"\n data = self.model.all_generations[i]\n self.scat.set_offsets(data)\n return self.scat,\n\n def setup_plot(self):\n data = self.model.all_generations[0]\n x = data[:, 0]\n y = data[:, 1]\n self.scat = self.ax.scatter(x, y, c=\"black\", marker='o', s=15)\n self.ax.set_xlim([self.range[0], self.range[1]])\n self.ax.set_ylim([self.range[0], self.range[1]])\n return self.scat,","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-08T20:01:24.452741Z","iopub.execute_input":"2022-02-08T20:01:24.453259Z","iopub.status.idle":"2022-02-08T20:01:24.492855Z","shell.execute_reply.started":"2022-02-08T20:01:24.453210Z","shell.execute_reply":"2022-02-08T20:01:24.491970Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"gwc = ShowEvolution(example)\ngwc.show()","metadata":{"execution":{"iopub.status.busy":"2022-02-08T20:01:24.495474Z","iopub.execute_input":"2022-02-08T20:01:24.495953Z","iopub.status.idle":"2022-02-08T20:02:04.368853Z","shell.execute_reply.started":"2022-02-08T20:01:24.495872Z","shell.execute_reply":"2022-02-08T20:02:04.367390Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Prepare the Titanic \n\nOk, let's start simple by building a feedforward neural network to classify the survival of the titanic. As I don't want to win this competition but rather to show what you can do with Differential Evolution I like to reduce the feature space by only using:\n\n* Pclass\n* Sex\n* Age\n* Embarked\n* Fare\n* Family size\n\nIt should be sufficient. ","metadata":{}},{"cell_type":"code","source":"titanic = pd.read_csv(\"../input/titanic/train.csv\")\ntitanic.head()","metadata":{"execution":{"iopub.status.busy":"2022-02-08T20:02:04.370671Z","iopub.execute_input":"2022-02-08T20:02:04.371280Z","iopub.status.idle":"2022-02-08T20:02:04.422004Z","shell.execute_reply.started":"2022-02-08T20:02:04.371226Z","shell.execute_reply":"2022-02-08T20:02:04.420453Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We have to be careful with the fare as it is a price per group/family and not per person. You can see that every family member given by the lastname and the ticket has the same fare (std of 0):","metadata":{}},{"cell_type":"code","source":"titanic.loc[:, \"lastname\"] = titanic.Name.str.split(\",\").apply(lambda l: l[0])\ntitanic.groupby([\"lastname\",\"Ticket\"]).Fare.std().fillna(0).max()","metadata":{"execution":{"iopub.status.busy":"2022-02-08T20:02:04.424172Z","iopub.execute_input":"2022-02-08T20:02:04.424600Z","iopub.status.idle":"2022-02-08T20:02:04.444607Z","shell.execute_reply.started":"2022-02-08T20:02:04.424565Z","shell.execute_reply":"2022-02-08T20:02:04.443493Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we can compute the price per person which should work better together with features like Pclass. Otherwise we would conclude for big families that a high price corresponds to a higher Pclass.","metadata":{}},{"cell_type":"code","source":"price_map = titanic.groupby([\"lastname\", \"Ticket\"]).Fare.min() / titanic.groupby([\"lastname\", \"Ticket\"]).size()\nfamily_size_map = titanic.groupby([\"lastname\", \"Ticket\"]).size()\ntitanic = titanic.set_index([\"lastname\", \"Ticket\"], drop=True)\ntitanic.loc[:, \"price\"] = titanic.index.map(price_map)\ntitanic.loc[:, \"family_size\"] = titanic.index.map(family_size_map)\ntitanic = titanic.reset_index()","metadata":{"execution":{"iopub.status.busy":"2022-02-08T20:02:04.445833Z","iopub.execute_input":"2022-02-08T20:02:04.446070Z","iopub.status.idle":"2022-02-08T20:02:04.506530Z","shell.execute_reply.started":"2022-02-08T20:02:04.446045Z","shell.execute_reply":"2022-02-08T20:02:04.505229Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Ok, let's get one-hot encoded values for Pclass, Embarked and Sex:","metadata":{}},{"cell_type":"code","source":"titanic = pd.get_dummies(titanic, columns=[\"Pclass\", \"Embarked\", \"Sex\"])","metadata":{"execution":{"iopub.status.busy":"2022-02-08T20:02:04.508460Z","iopub.execute_input":"2022-02-08T20:02:04.509091Z","iopub.status.idle":"2022-02-08T20:02:04.529476Z","shell.execute_reply.started":"2022-02-08T20:02:04.509027Z","shell.execute_reply":"2022-02-08T20:02:04.527710Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"And fill missing ages with median:","metadata":{}},{"cell_type":"code","source":"titanic.Age = titanic.Age.fillna(titanic.Age.median())","metadata":{"execution":{"iopub.status.busy":"2022-02-08T20:02:04.531556Z","iopub.execute_input":"2022-02-08T20:02:04.532308Z","iopub.status.idle":"2022-02-08T20:02:04.542451Z","shell.execute_reply.started":"2022-02-08T20:02:04.532247Z","shell.execute_reply":"2022-02-08T20:02:04.541432Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Furthermore we need to transform Age and price such that they look more normally distributed. The gradients of neural networks are direclty influenced by the features themselves and can be misled when outliers are present.","metadata":{}},{"cell_type":"code","source":"fig, ax = plt.subplots(1,4,figsize=(20,5))\nsns.distplot(titanic.Age, ax=ax[0], color=\"Orange\")\nsns.distplot(titanic.Age.apply(lambda l: np.log(l+20)), ax=ax[1], color=\"Orange\")\nsns.distplot(titanic.price, ax=ax[2], color=\"Purple\")\nsns.distplot(titanic.price.apply(lambda l: np.log(l+0.5)), ax=ax[3], color=\"Purple\");","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-08T20:02:04.544083Z","iopub.execute_input":"2022-02-08T20:02:04.544537Z","iopub.status.idle":"2022-02-08T20:02:05.759873Z","shell.execute_reply.started":"2022-02-08T20:02:04.544493Z","shell.execute_reply":"2022-02-08T20:02:05.758693Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"titanic.Age = titanic.Age.apply(lambda l: np.log(l+20))\ntitanic.price = titanic.price.apply(lambda l: np.log(l+0.5))","metadata":{"execution":{"iopub.status.busy":"2022-02-08T20:02:05.761049Z","iopub.execute_input":"2022-02-08T20:02:05.761289Z","iopub.status.idle":"2022-02-08T20:02:05.771337Z","shell.execute_reply.started":"2022-02-08T20:02:05.761263Z","shell.execute_reply":"2022-02-08T20:02:05.770013Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"to_drop = [\"PassengerId\", \"Ticket\", \"lastname\", \"Sex_male\", \"Name\", \"SibSp\", \"Parch\", \"Cabin\", \"Fare\"]\ntitanic = titanic.drop(to_drop, axis=1)\ntitanic.head()","metadata":{"execution":{"iopub.status.busy":"2022-02-08T20:02:05.772743Z","iopub.execute_input":"2022-02-08T20:02:05.773046Z","iopub.status.idle":"2022-02-08T20:02:05.798264Z","shell.execute_reply.started":"2022-02-08T20:02:05.773017Z","shell.execute_reply":"2022-02-08T20:02:05.797577Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Ok, that's it! :-) Let's now build a simple neural network and define the hyperparameters we like to tune using Differential Evolution.","metadata":{}},{"cell_type":"code","source":"num_titanic_features=titanic.drop(\"Survived\", axis=1).shape[1]\nnum_titanic_features","metadata":{"execution":{"iopub.status.busy":"2022-02-08T20:02:05.799252Z","iopub.execute_input":"2022-02-08T20:02:05.799579Z","iopub.status.idle":"2022-02-08T20:02:05.806542Z","shell.execute_reply.started":"2022-02-08T20:02:05.799553Z","shell.execute_reply":"2022-02-08T20:02:05.805885Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Build the NNs for Titanic ","metadata":{}},{"cell_type":"markdown","source":"Ok, we have optimised some test optimisation functions to see how differential evolution solves the task. Now I like to explore how it will solve hyperparameter tuning for a simple feedforward neural network. I'm especially interested in:\n\n* The number of neurons compared to regularisation given by dropout rates or weight regularisation.\n* The number of epochs compared to the learning rate.\n\nLet's go! :-)","metadata":{}},{"cell_type":"code","source":"def build_titanic_nn(num_neurons_1, dropout_1,\n num_neurons_2, dropout_2,\n input_shape=(num_titanic_features,)):\n \n model = keras.Sequential()\n model.add(keras.Input(shape=input_shape))\n model.add(keras.layers.Dense(num_neurons_1, activation=\"relu\"))\n model.add(keras.layers.Dropout(dropout_1))\n model.add(keras.layers.Dense(num_neurons_2, activation=\"relu\"))\n model.add(keras.layers.Dropout(dropout_2))\n model.add(keras.layers.Dense(1, activation=\"sigmoid\"))\n return model","metadata":{"execution":{"iopub.status.busy":"2022-02-08T20:02:05.807679Z","iopub.execute_input":"2022-02-08T20:02:05.807975Z","iopub.status.idle":"2022-02-08T20:02:05.821828Z","shell.execute_reply.started":"2022-02-08T20:02:05.807947Z","shell.execute_reply":"2022-02-08T20:02:05.820705Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Build an individual for DE \n\n","metadata":{}},{"cell_type":"markdown","source":"Before we can use Differential Evolution to search for the best neural network hyperparameters, we need a class for individual neural networks that captures the logic for training and evaluating each net and for storing its parameters:","metadata":{}},{"cell_type":"code","source":"from collections import OrderedDict\nimport copy\n\nclass Individual_Titanic:\n \n def __init__(self, params):\n self.parameters = copy.deepcopy(params)\n self.history = None\n \n def init_current(self, r_state):\n next_state = 0\n for par in self.parameters.keys():\n state = r_state + next_state\n if self.parameters[par][\"val_range\"].dtype == np.int:\n self.parameters[par][\"current\"] = np.random.RandomState(state).randint(\n low=self.parameters[par][\"val_range\"][0],\n high=self.parameters[par][\"val_range\"][1]) \n else:\n self.parameters[par][\"current\"] = np.random.RandomState(state).uniform(\n low=self.parameters[par][\"val_range\"][0],\n high=self.parameters[par][\"val_range\"][1])\n next_state += 1\n \n \n def build_model(self):\n model = build_titanic_nn(num_neurons_1=self.parameters[\"num_neurons_1\"][\"current\"],\n dropout_1=self.parameters[\"dropout_1\"][\"current\"],\n num_neurons_2=self.parameters[\"num_neurons_2\"][\"current\"],\n dropout_2=self.parameters[\"dropout_2\"][\"current\"])\n return model\n \n \n def train(self, x_train, y_train, x_dev, y_dev, verbose=0):\n model = self.build_model()\n model.compile(loss=keras.losses.BinaryCrossentropy(),\n optimizer=keras.optimizers.Adam(learning_rate=self.parameters[\"lr\"][\"current\"]),\n metrics=[\"accuracy\"])\n self.history = model.fit(x_train, y_train,\n batch_size=32,\n epochs=self.parameters[\"epochs\"][\"current\"],\n verbose=verbose,\n workers=4)\n dev_loss, dev_acc = model.evaluate(x_dev, y_dev)\n self.score = dev_acc\n self.loss = dev_loss\n del model\n ","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-08T20:02:05.823399Z","iopub.execute_input":"2022-02-08T20:02:05.824019Z","iopub.status.idle":"2022-02-08T20:02:05.846954Z","shell.execute_reply.started":"2022-02-08T20:02:05.823972Z","shell.execute_reply":"2022-02-08T20:02:05.845548Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X = titanic.drop(\"Survived\", axis=1).values\nY = titanic.Survived.values\nx_train, x_dev, y_train, y_dev = train_test_split(X, Y, stratify=Y, random_state=0)\nprint(x_train.shape, x_dev.shape)","metadata":{"execution":{"iopub.status.busy":"2022-02-08T20:02:05.848387Z","iopub.execute_input":"2022-02-08T20:02:05.849060Z","iopub.status.idle":"2022-02-08T20:02:05.873110Z","shell.execute_reply.started":"2022-02-08T20:02:05.849011Z","shell.execute_reply":"2022-02-08T20:02:05.871024Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here is a small example:","metadata":{}},{"cell_type":"code","source":"parameters = {\n \"dropout_1\": {\"val_range\": np.array([0.1,0.8]).astype(np.float), \"current\": None},\n \"dropout_2\": {\"val_range\": np.array([0.1,0.8]).astype(np.float), \"current\": None},\n \"num_neurons_1\": {\"val_range\": np.array([10,300]).astype(np.int), \"current\": None},\n \"num_neurons_2\": {\"val_range\": np.array([10,300]).astype(np.int), \"current\": None},\n \"lr\": {\"val_range\": np.array([0.001,0.5]).astype(np.float), \"current\": None},\n \"epochs\": {\"val_range\": np.array([1,20]).astype(np.int), \"current\": None}\n}","metadata":{"execution":{"iopub.status.busy":"2022-02-08T20:02:05.874946Z","iopub.execute_input":"2022-02-08T20:02:05.875370Z","iopub.status.idle":"2022-02-08T20:02:05.884525Z","shell.execute_reply.started":"2022-02-08T20:02:05.875342Z","shell.execute_reply":"2022-02-08T20:02:05.883295Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"individual = Individual_Titanic(parameters)\n\nindividual.init_current(0)\nfor par in individual.parameters.keys():\n print(\"Par {}, {}\".format(par, individual.parameters[par][\"current\"]))\nindividual.train(x_train, y_train, x_dev, y_dev, verbose=0)\nindividual.score","metadata":{"execution":{"iopub.status.busy":"2022-02-08T20:02:05.886496Z","iopub.execute_input":"2022-02-08T20:02:05.887008Z","iopub.status.idle":"2022-02-08T20:02:07.054954Z","shell.execute_reply.started":"2022-02-08T20:02:05.886967Z","shell.execute_reply":"2022-02-08T20:02:07.053362Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Ok, now we like to know how we need to change the parameters such that the accuracy increases. ","metadata":{}},{"cell_type":"markdown","source":"# Differential Evolution for Titanic ","metadata":{}},{"cell_type":"code","source":"class DE_NeuralNets:\n \n def __init__(self, X, Y,\n scaling_rate=0.5, \n crossover_rate=0.5, \n population_size=20,\n max_generation_steps=25):\n self.scaling_rate = scaling_rate\n self.crossover_rate = crossover_rate\n self.population_size = population_size\n \n self.generation_counter = 0\n \n self.all_generations = []\n self.max_generation_steps = max_generation_steps\n self.x_train, self.x_dev, self.y_train, self.y_dev = train_test_split(\n X, Y, stratify=Y, random_state=0)\n \n def add_params(self, params):\n for par in params.keys():\n val_range = params[par][\"val_range\"]\n if not isinstance(val_range, np.ndarray):\n raise TypeError(\"The value range must be of type list.\")\n if len(val_range) != 2:\n pass\n if val_range[0] > val_range[1]:\n pass\n self.parameters = params\n \n def init_generation(self):\n self.generation = []\n self.generation_costs = []\n for i in range(self.population_size):\n individual = Individual_Titanic(self.parameters)\n random_state = i*len(self.parameters)\n individual.init_current(random_state)\n individual.train(x_train=self.x_train,\n y_train=self.y_train,\n x_dev=self.x_dev,\n y_dev=self.y_dev,\n verbose=0)\n self.generation_costs.append(individual.score)\n self.generation.append(individual)\n \n def get_trials(self):\n print(\"-- Trials are computed --\")\n self.trials = []\n self.trials_costs = []\n for i in range(self.population_size):\n target = self.generation[i]\n mutant = self.mutate(i)\n trial = self.crossover(mutant, target)\n trial.train(x_train=self.x_train,\n y_train=self.y_train,\n x_dev=self.x_dev,\n y_dev=self.y_dev,\n verbose=0)\n self.trials_costs.append(trial.score)\n self.trials.append(trial)\n \n def mutate(self, i):\n candidates = self.get_candidates(i)\n mutant = Individual_Titanic(self.generation[candidates[0]].parameters.copy())\n for par in self.parameters.keys():\n A = self.generation[candidates[1]].parameters[par][\"current\"]\n B = self.generation[candidates[2]].parameters[par][\"current\"]\n if type(A) is float:\n mutation = mutant.parameters[par][\"current\"] + self.scaling_rate * (A-B)\n else:\n mutation = mutant.parameters[par][\"current\"] + np.int(np.round(self.scaling_rate * (A-B)))\n mutation = self.check_boundaries(mutation,\n mutant.parameters[par][\"val_range\"][0],\n mutant.parameters[par][\"val_range\"][1])\n mutant.parameters[par][\"current\"] = mutation\n return mutant\n \n def check_boundaries(self, mutation, low_bound, high_bound):\n if mutation < low_bound:\n mutation = low_bound\n elif mutation > high_bound:\n mutation = high_bound\n return mutation\n \n def get_candidates(self, i):\n to_select = list(np.arange(self.population_size))\n to_select.remove(i)\n candidates = np.random.choice(to_select, 3, replace=False)\n return candidates\n \n def crossover(self, mutant, target):\n param_list = list(self.parameters.keys())\n random_par = np.random.choice(param_list)\n trial = Individual_Titanic(target.parameters.copy())\n for par in param_list:\n crossover_unit = np.random.uniform(0, 1)\n if crossover_unit <= self.crossover_rate or par == random_par:\n trial.parameters[par][\"current\"] = mutant.parameters[par][\"current\"]\n return trial\n \n def select(self):\n \n idx = np.where(np.array(self.trials_costs) > np.array(self.generation_costs))[0]\n \n # fuse trials and parents to form a candidate pool\n fusion = self.trials_costs[:]\n fusion.extend(self.generation_costs[:])\n candidate_pool = self.trials[:]\n candidate_pool.extend(self.generation[:]) \n \n # sort score (descending)\n idx = np.argsort(np.array(fusion))[::-1]\n \n # only select the best individuals of both (generation & trials)\n for i in range(len(self.generation)):\n self.generation[i] = candidate_pool[idx[i]]\n self.generation_costs[i] = fusion[idx[i]]\n \n def select_v2(self):\n idx = np.where(np.array(self.trials_costs) > np.array(self.generation_costs))[0]\n for i in idx:\n self.generation[i] = self.trials[i]\n self.generation_costs[i] = self.trials_costs[i]\n \n def evolve(self):\n self.best_solutions = []\n self.worst_solutions = []\n self.init_generation()\n self.all_generations.append(self.generation[:])\n while(self.generation_counter < self.max_generation_steps):\n self.get_trials()\n self.select()\n self.all_generations.append(self.generation[:])\n self.generation_counter += 1\n self.best_solutions.append(np.max(self.generation_costs))\n self.worst_solutions.append(np.min(self.generation_costs))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-08T20:02:07.056115Z","iopub.execute_input":"2022-02-08T20:02:07.056381Z","iopub.status.idle":"2022-02-08T20:02:07.091820Z","shell.execute_reply.started":"2022-02-08T20:02:07.056352Z","shell.execute_reply":"2022-02-08T20:02:07.090155Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"eve = DE_NeuralNets(X, Y)\neve.add_params(parameters)\neve.evolve()","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-02-08T20:02:07.093647Z","iopub.execute_input":"2022-02-08T20:02:07.094014Z","iopub.status.idle":"2022-02-08T20:11:45.055067Z","shell.execute_reply.started":"2022-02-08T20:02:07.093980Z","shell.execute_reply":"2022-02-08T20:11:45.053784Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# How does the solution look like?\n\nDifferential evolution is able to find the global optimum for the data and model given but it is not guaranteed to converge. In our case I expect that all individuals of the updated generation improve their scores during hyperparameter search. While the worst and best accuracy score might have a big difference in the beginning I assume that they become closer during iteration:\n","metadata":{}},{"cell_type":"code","source":"plt.plot(eve.best_solutions, 'o-', label=\"best\")\nplt.plot(eve.worst_solutions, '+-', label=\"worst\")\nplt.ylabel(\"Best val accuracy in generation i\")\nplt.xlabel(\"Generation i\")\nplt.title(\"Evolution of best individuals\")\nplt.legend()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-08T20:11:45.056709Z","iopub.execute_input":"2022-02-08T20:11:45.057153Z","iopub.status.idle":"2022-02-08T20:11:45.304210Z","shell.execute_reply.started":"2022-02-08T20:11:45.057100Z","shell.execute_reply":"2022-02-08T20:11:45.302803Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"For the titanic validation dataset and the chosen features we can see that some score above 0.8 is best.","metadata":{}},{"cell_type":"markdown","source":"# Understanding the evolution of hyperparameters \n\nOne part I find most interesting in using DE for hyperparameter search is to observe the evolution of hyperparameters and their related score or loss values:","metadata":{}},{"cell_type":"code","source":"plt.rcParams[\"animation.html\"] = \"jshtml\" \n \nclass ShowHyperparameterEvolution:\n\n def __init__(self, model):\n self.eve = model\n self.num_generations = len(self.eve.all_generations)\n \n self.fig, self.ax = plt.subplots(1,3,figsize=(20,5))\n \n self.ax[0].set_xlim([0, 500])\n self.ax[0].set_ylim([0, 500])\n self.ax[0].set_xlabel(\"Neurons in layer 1\")\n self.ax[0].set_ylabel(\"Neurons in layer 2\")\n \n self.ax[1].set_xlim([0,1])\n self.ax[1].set_ylim([0,1])\n self.ax[1].set_xlabel(\"Dropout layer 1\")\n self.ax[1].set_ylabel(\"Dropout layer 2\")\n \n self.ax[2].set_xlim([0,0.8])\n self.ax[2].set_ylim([0,25])\n self.ax[2].set_xlabel(\"Learning rate\")\n self.ax[2].set_ylabel(\"Number of epochs\")\n \n \n \n def show(self):\n self.ani = animation.FuncAnimation(self.fig, self.update,\n init_func=self.init_fun,\n frames=self.num_generations,\n interval=900,\n blit=False,\n repeat=False)\n #Writer = animation.writers['ffmpeg']\n #writer = Writer(fps=15, metadata=dict(artist='Me'), bitrate=1800)\n #self.ani.save('example.mp4', writer=writer)\n plt.close()\n return self.ani\n\n def get_data(self, generation):\n data = np.zeros(shape=(len(generation), 6))\n for i in range(len(generation)):\n data[i,0] = generation[i].parameters[\"num_neurons_1\"][\"current\"]\n data[i,1] = generation[i].parameters[\"num_neurons_2\"][\"current\"]\n data[i,2] = generation[i].parameters[\"dropout_1\"][\"current\"]\n data[i,3] = generation[i].parameters[\"dropout_2\"][\"current\"]\n data[i,4] = generation[i].parameters[\"lr\"][\"current\"]\n data[i,5] = generation[i].parameters[\"epochs\"][\"current\"]\n return data\n \n def update(self, i):\n \"\"\"Update the scatter plot.\"\"\"\n generation = self.eve.all_generations[i]\n data = self.get_data(generation)\n if i==0:\n self.scat1 = self.ax[0].scatter(data[:, 0], data[:, 1], c=\"mediumseagreen\", marker='o', s=15) \n self.scat2 = self.ax[1].scatter(data[:, 2], data[:, 3], c=\"red\", marker=\"o\", s=15)\n self.scat3 = self.ax[2].scatter(data[:, 4], data[:, 5], c=\"purple\", marker=\"o\", s=15)\n else:\n self.scat1.set_offsets(data[:,0:2])\n self.scat2.set_offsets(data[:,2:4])\n self.scat3.set_offsets(data[:,4:6])\n #return self.scat1,\n\n def init_fun(self):\n return [self.ax]","metadata":{"_kg_hide-output":false,"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-08T20:11:45.306890Z","iopub.execute_input":"2022-02-08T20:11:45.307267Z","iopub.status.idle":"2022-02-08T20:11:45.344125Z","shell.execute_reply.started":"2022-02-08T20:11:45.307229Z","shell.execute_reply":"2022-02-08T20:11:45.342444Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"gwc = ShowHyperparameterEvolution(eve)\ngwc.show()","metadata":{"execution":{"iopub.status.busy":"2022-02-08T20:11:45.345904Z","iopub.execute_input":"2022-02-08T20:11:45.346304Z","iopub.status.idle":"2022-02-08T20:11:51.770434Z","shell.execute_reply.started":"2022-02-08T20:11:45.346174Z","shell.execute_reply":"2022-02-08T20:11:51.769316Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Insights\n\n* We can see that DE finds several solutions that are similar good with close scores. \n* Furthermore we can see an interesting pattern: \n * The number of neurons in layer 1 seem to be more peaked than in layer 2.\n * The dropout rate of layer 1 spreads a lot while the rates of layer 2 are more dense. \n * The number of epochs vary quite a lot! \n\nProbably there is a dependency between these hyperparameters that we can use to improve our optimization and for a fruitful rule of thumb. ","metadata":{}},{"cell_type":"markdown","source":"# Hyperparameter evolution for the digits dataset ","metadata":{}},{"cell_type":"markdown","source":"**Work in progress**","metadata":{}},{"cell_type":"code","source":"def build_digits_cnn(num_neurons_1, dropout_1, input_shape=(28, 28, 1)):\n model = keras.Sequential()\n model.add(keras.Input(shape=input_shape))\n model.add(keras.layers.Conv2D(filters=32, kernel_size=3, activation=\"relu\"))\n model.add(keras.layers.MaxPooling2D(pool_size=2))\n model.add(keras.layers.Conv2D(filters=64, kernel_size=3, activation=\"relu\"))\n #model.add(keras.layers.MaxPooling2D(pool_size=2))\n #model.add(keras.layers.Conv2D(filters=128, kernel_size=3, activation=\"relu\"))\n model.add(keras.layers.Flatten())\n model.add(keras.layers.Dense(num_neurons_1, activation=\"relu\"))\n model.add(keras.layers.Dropout(dropout_1))\n model.add(keras.layers.Dense(10, activation=\"softmax\"))\n return model","metadata":{"execution":{"iopub.status.busy":"2022-02-08T20:11:51.771601Z","iopub.execute_input":"2022-02-08T20:11:51.771885Z","iopub.status.idle":"2022-02-08T20:11:51.780257Z","shell.execute_reply.started":"2022-02-08T20:11:51.771858Z","shell.execute_reply":"2022-02-08T20:11:51.778982Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class Individual_Digits:\n \n def __init__(self, params):\n self.parameters = copy.deepcopy(params)\n self.history = None\n \n def init_current(self, r_state):\n next_state = 0\n for par in self.parameters.keys():\n state = r_state + next_state\n if self.parameters[par][\"val_range\"].dtype == np.int:\n self.parameters[par][\"current\"] = np.random.RandomState(state).randint(\n low=self.parameters[par][\"val_range\"][0],\n high=self.parameters[par][\"val_range\"][1]) \n else:\n self.parameters[par][\"current\"] = np.random.RandomState(state).uniform(\n low=self.parameters[par][\"val_range\"][0],\n high=self.parameters[par][\"val_range\"][1])\n next_state += 1\n \n def build_model(self):\n model = build_digits_cnn(num_neurons_1=self.parameters[\"num_neurons_1\"][\"current\"],\n dropout_1=self.parameters[\"dropout_1\"][\"current\"])\n return model\n \n def train(self, x_train, y_train, x_dev, y_dev, verbose=0):\n model = self.build_model()\n model.compile(loss=keras.losses.SparseCategoricalCrossentropy(),\n optimizer=keras.optimizers.Adam(learning_rate=self.parameters[\"lr\"][\"current\"]),\n metrics=[\"accuracy\"])\n self.history = model.fit(x_train, y_train,\n batch_size=16*self.parameters[\"batchsize\"][\"current\"],\n epochs=5,\n verbose=verbose,\n workers=4)\n dev_loss, dev_acc = model.evaluate(x_dev, y_dev)\n self.score = dev_acc\n self.loss = dev_loss\n del model","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-08T20:11:51.781798Z","iopub.execute_input":"2022-02-08T20:11:51.782072Z","iopub.status.idle":"2022-02-08T20:11:51.796089Z","shell.execute_reply.started":"2022-02-08T20:11:51.782042Z","shell.execute_reply":"2022-02-08T20:11:51.795154Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from tensorflow.keras.datasets import mnist\n\n(train_images, train_labels), (test_images, test_labels) = mnist.load_data()\ntrain_images = train_images.reshape((60000, 28, 28, 1))\ntrain_images = train_images.astype(\"float32\") / 255\ntest_images = test_images.reshape((10000, 28, 28, 1))\ntest_images = test_images.astype(\"float32\") / 255","metadata":{"execution":{"iopub.status.busy":"2022-02-08T20:11:51.797301Z","iopub.execute_input":"2022-02-08T20:11:51.797542Z","iopub.status.idle":"2022-02-08T20:11:52.242170Z","shell.execute_reply.started":"2022-02-08T20:11:51.797516Z","shell.execute_reply":"2022-02-08T20:11:52.240863Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"parameters = {\n \"dropout_1\": {\"val_range\": np.array([0.1,0.8]).astype(np.float), \"current\": None},\n \"num_neurons_1\": {\"val_range\": np.array([10,300]).astype(np.int), \"current\": None},\n \"lr\": {\"val_range\": np.array([0.005,0.01]).astype(np.float), \"current\": None},\n \"batchsize\": {\"val_range\": np.array([4,8]).astype(np.int), \"current\": None}\n}","metadata":{"execution":{"iopub.status.busy":"2022-02-08T20:11:52.243580Z","iopub.execute_input":"2022-02-08T20:11:52.243940Z","iopub.status.idle":"2022-02-08T20:11:52.251582Z","shell.execute_reply.started":"2022-02-08T20:11:52.243907Z","shell.execute_reply":"2022-02-08T20:11:52.250579Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"individual = Individual_Digits(parameters)\n\nindividual.init_current(2)\nfor par in individual.parameters.keys():\n print(\"Par {}, {}\".format(par, individual.parameters[par][\"current\"]))\nindividual.train(train_images, train_labels, test_images, test_labels, verbose=1)\nindividual.score","metadata":{"execution":{"iopub.status.busy":"2022-02-08T20:11:52.253161Z","iopub.execute_input":"2022-02-08T20:11:52.253420Z","iopub.status.idle":"2022-02-08T20:13:39.064735Z","shell.execute_reply.started":"2022-02-08T20:11:52.253387Z","shell.execute_reply":"2022-02-08T20:13:39.063861Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class DE_NeuralNets_Digits(DE_NeuralNets):\n \n def __init__(self,\n scaling_rate=0.5, \n crossover_rate=0.5, \n population_size=6,\n max_generation_steps=10):\n self.scaling_rate = scaling_rate\n self.crossover_rate = crossover_rate\n self.population_size = population_size\n \n self.generation_counter = 0\n \n self.all_generations = []\n self.max_generation_steps = max_generation_steps\n \n (train_images, self.y_train), (test_images, self.y_dev) = mnist.load_data()\n train_images = train_images.reshape((60000, 28, 28, 1))\n self.x_train = train_images.astype(\"float32\") / 255\n test_images = test_images.reshape((10000, 28, 28, 1))\n self.x_dev = test_images.astype(\"float32\") / 255\n \n def init_generation(self):\n self.generation = []\n self.generation_costs = []\n for i in range(self.population_size):\n individual = Individual_Digits(self.parameters)\n random_state = i*len(self.parameters)\n individual.init_current(random_state)\n individual.train(x_train=self.x_train,\n y_train=self.y_train,\n x_dev=self.x_dev,\n y_dev=self.y_dev,\n verbose=0)\n self.generation_costs.append(individual.score)\n self.generation.append(individual)\n \n def mutate(self, i):\n candidates = self.get_candidates(i)\n mutant = Individual_Digits(self.generation[candidates[0]].parameters.copy())\n for par in self.parameters.keys():\n A = self.generation[candidates[1]].parameters[par][\"current\"]\n B = self.generation[candidates[2]].parameters[par][\"current\"]\n if type(A) is float:\n mutation = mutant.parameters[par][\"current\"] + self.scaling_rate * (A-B)\n else:\n mutation = mutant.parameters[par][\"current\"] + np.int(np.round(self.scaling_rate * (A-B)))\n mutation = self.check_boundaries(mutation,\n mutant.parameters[par][\"val_range\"][0],\n mutant.parameters[par][\"val_range\"][1])\n mutant.parameters[par][\"current\"] = mutation\n return mutant \n \n \n def crossover(self, mutant, target):\n param_list = list(self.parameters.keys())\n random_par = np.random.choice(param_list)\n trial = Individual_Digits(target.parameters.copy())\n for par in param_list:\n crossover_unit = np.random.uniform(0, 1)\n if crossover_unit <= self.crossover_rate or par == random_par:\n trial.parameters[par][\"current\"] = mutant.parameters[par][\"current\"]\n return trial","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-08T20:13:39.066145Z","iopub.execute_input":"2022-02-08T20:13:39.066600Z","iopub.status.idle":"2022-02-08T20:13:39.094799Z","shell.execute_reply.started":"2022-02-08T20:13:39.066559Z","shell.execute_reply":"2022-02-08T20:13:39.093409Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"eve = DE_NeuralNets_Digits()\neve.add_params(parameters)\neve.evolve()","metadata":{"execution":{"iopub.status.busy":"2022-02-08T20:13:39.099488Z","iopub.execute_input":"2022-02-08T20:13:39.100048Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.plot(eve.best_solutions, 'o-', label=\"best\")\nplt.plot(eve.worst_solutions, '+-', label=\"worst\")\nplt.ylabel(\"Best val accuracy in generation i\")\nplt.xlabel(\"Generation i\")\nplt.title(\"Evolution of best individuals\")\nplt.legend()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}