Smoothing of cost function leads to faster convergence of neural network learning

Li-Qun Xu; Trevor J. Hall

doi:10.1117/12.169959

2 March 1994 Smoothing of cost function leads to faster convergence of neural network learning

Li-Qun Xu, Trevor J. Hall

Proceedings Volume 2243, Applications of Artificial Neural Networks V; (1994) https://doi.org/10.1117/12.169959
Event: SPIE's International Symposium on Optical Engineering and Photonics in Aerospace Sensing, 1994, Orlando, FL, United States

Abstract

One of the major problems in supervised learning of neural networks is the inevitable local minima inherent in the cost function f(W,D). This often makes classic gradient-descent-based learning algorithms that calculate the weight updates for each iteration according to (Delta) W(t) equals -(eta) (DOT)$DEL_wf(W,D) powerless. In this paper we describe a new strategy to solve this problem, which, adaptively, changes the learning rate and manipulates the gradient estimator simultaneously. The idea is to implicitly convert the local- minima-laden cost function f((DOT)) into a sequence of its smoothed versions {f_(beta _t)}^T_tequals1, which, subject to the parameter (beta) _t, bears less details at time t equals 1 and gradually more later on, the learning is actually performed on this sequence of functionals. The corresponding smoothed global minima obtained in this way, {W_t}^T_tequals1, thus progressively approximate W--the desired global minimum. Experimental results on a nonconvex function minimization problem and a typical neural network learning task are given, analyses and discussions of some important issues are provided.

Citation Download Citation

Li-Qun Xu and Trevor J. Hall "Smoothing of cost function leads to faster convergence of neural network learning", Proc. SPIE 2243, Applications of Artificial Neural Networks V, (2 March 1994); https://doi.org/10.1117/12.169959

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available