Neural network approximation: Three hidden layers are enough.

Abstract

A three-hidden-layer neural network with super approximation power is introduced. This network is built with the floor function (⌊x⌋), the exponential function (2^x), the step function (1_x≥0), or their compositions as the activation function in each neuron and hence we call such networks as Floor-Exponential-Step (FLES) networks. For any width hyper-parameter N∈N⁺, it is shown that FLES networks with width max{d,N} and three hidden layers can uniformly approximate a Hölder continuous function f on [0,1]^d with an exponential approximation rate 3λ(2d)^α2^-αN, where α∈(0,1] and λ>0 are the Hölder order and constant, respectively. More generally for an arbitrary continuous function f on [0,1]^d with a modulus of continuity ω_f(⋅), the constructive approximation rate is 2ω_f(2d)2^-N+ω_f(2d2^-N). Moreover, we extend such a result to general bounded continuous functions on a bounded set E⊆R^d. As a consequence, this new class of networks overcomes the curse of dimensionality in approximation power when the variation of ω_f(r) as r→0 is moderate (e.g., ω_f(r)≲r^α for Hölder continuous functions), since the major term to be concerned in our approximation rate is essentially d times a function of N independent of d within the modulus of continuity. Finally, we extend our analysis to derive similar approximation results in the L^p-norm for p∈[1,∞) via replacing Floor-Exponential-Step activation functions by continuous activation functions.