Deep Learning Alternatives to Error Backpropagation: A Review

Abstract

Error Back Propagation have become the dominant training method for deep neural networks, but it is counter-intuitive to the way biological mind works (biological implausibility) and requires substantial compute resources, so much so that the ownership of adequately trained large language models have become the domain of a few resource rich organiza-tions. This review examines the current trends in the research into alternatives to deep learning, such as biologically-inspired approaches, local learning rules, and novel optimization strategies. The author surveys key advancements mostly from 2020 onwards, discussing the promising approaches such as Forward-Forward algorithms, Hebbian Learning, Equilibrium Propa-gation, and Predictive Coding Frameworks. The error Back-Propagation (eBP) is currently the most successful approach to train deep networks and the alternatives are only chasing accuracies achieved by this method. All major AI models use Deep Learning with eBP and this more-is-more approach has effected alternative forms to AI research [1]. Nevertheless, if the al-ternative methods begin to approach benchmarks set by eBP, a few percentage points drop in accuracy may be acceptable provided the training is possible at a fraction of the compute cost of eBP. Such reduction in requirement for resources is most desirable and, in addition to being beneficial for developers, users and environment, shall democratise the Artificial Intelli-gence ecosystem.