Using the Quasi-Newton Method to Solve Nonlinear Least Squares Regression Problems

Document Type : Original Article

Authors

Faculty of Computers and Artificial Intelligence, Benha University, Benha, Qalyubia Governorate, Egypt.

Abstract

In regression modeling, Gradient Descent (GD) is widely used to update parameters by iteratively minimizing a cost function. However, GD often converges slowly and may suffer from instability due to its reliance on first-order derivatives only. To improve convergence speed and stability, Newton’s method utilizes second-order derivative information, aiming to find the point where the gradient vanishes. While Newton’s method offers faster convergence, computing the exact Hessian matrix is often computationally expensive or infeasible.

Quasi-Newton methods overcome this limitation by approximating the Hessian matrix. These methods iteratively update an estimate of the Hessian, typically denoted as H_k≈∇^2 f(x_k), to guide the search direction. A notable quasi-Newton algorithm is the Broyden–Fletcher–Goldfarb–Shanno (BFGS) method.

In this paper, we apply BFGS and its variants—including Memoryless BFGS, Limited-memory BFGS (L-BFGS), and Scaled BFGS—to nonlinear least squares regression problems. Their performance is compared with traditional Gradient Descent and Newton’s method, focusing on convergence behavior and optimization efficiency. The results demonstrate the potential advantages of quasi-Newton approaches in practical regression scenarios.

Keywords

Main Subjects

Files

Share

How to cite

Statistics