Evolutionary training of deep neural networks on heterogeneous computing environments

Deep neural networks are typically trained using gradient-based optimizers such as error backpropagation. This study proposes a framework based on Evolutionary Algorithms (EAs) to train deep neural networks without gradients. The network parameters, which may vary up to millions, are considered optimization variables. We demonstrate the training of an encoder-decoder segmentation network (U-Net) and Long Short-Term Memory (LSTM) model using (μ + λ)-ES, Genetic Algorithm, and Particle Swarm Optimization. The framework can train models with forward propagation on machines with different hardware in a cluster computing environment. We compare prediction results from the two models trained using our framework and backpropagation. We show that the neural networks can be trained in less time on CPUs as compared to the training on specialized compute-intensive GPUs.