Analysis of Adaptive Gradient Algorithms: From Theory to MATLAB Practice

What is the Adaptive Gradient Algorithm?

In the field of deep learning optimization, adaptive gradient algorithms have become the preferred method for training neural networks. Unlike traditional gradient descent methods that use a uniform learning rate, adaptive algorithms assign different learning rates to each parameter, making the optimization process more efficient and stable.

The core idea of the adaptive gradient algorithm is very simple: assign a smaller learning rate to frequently updated parameters and a larger learning rate to infrequently updated parameters. This ensures both training speed and improved convergence stability.

Introduction to Mainstream Adaptive Gradient Algorithms

1. AdaGrad (Adaptive Gradient)

AdaGrad is one of the earliest adaptive algorithms, which accumulates the sum of the squares of historical gradients to adjust the learning rate:

θ_t+1 = θ_t - (η/√(G_t + ε)) ⊙ g_t

where G_t is the accumulation of all squared gradients up to time t.

Advantages: Suitable for sparse data, small updates for frequently occurring features, and large updates for rare features. Disadvantages: The learning rate monotonically decreases, which may lead to premature stopping of learning.

2. RMSprop

RMSprop is an improvement over AdaGrad, introducing a decay factor to solve the problem of continuously decreasing learning rates:

E[g²]_t = γE[g²]_{t-1} + (1-γ)g_tθ_t+1 = θ_t - (η/√(E[g²]_t + ε)) ⊙ g_t

3. Adam (Adaptive Moment Estimation)

Adam combines the advantages of momentum methods and RMSprop, making it the most popular adaptive algorithm today:

m_t = β₁m_{t-1} + (1-β₁)g_tv_t = β₂v_{t-1} + (1-β₂)g_t²m̂_t = m_t/(1-β₁^t)v̂_t = v_t/(1-β₂^t)θ_t+1 = θ_t - η m̂_t/(√v̂_t + ε)

MATLAB Implementation and Comparison

Below, we implement several adaptive algorithms in MATLAB and compare their performance on a test function.

clear; close all; clc;
%% Rosenbrock function and gradientfprintf('=== Comparison of Adaptive Gradient Algorithms ===\n\n');
% Rosenbrock function - a classic test function with a minimum value of 0 at (1,1)rosenbrock = @(x) (1 - x(1))^2 + 100 * (x(2) - x(1)^2)^2;
% Correct gradient calculationrosenbrock_grad = @(x) [    -2*(1 - x(1)) - 400*x(1)*(x(2) - x(1)^2);    200*(x(2) - x(1)^2)];
% Verify gradient calculationtest_point = [0.5; 0.5];f_val = rosenbrock(test_point);grad_val = rosenbrock_grad(test_point);fprintf('Gradient verification: f(%.1f,%.1f)=%.4f, grad=[%.4f;%.4f]\n', ...        test_point(1), test_point(2), f_val, grad_val(1), grad_val(2));
%% Optimization parameter settingsx0 = [-1.2; 1.0];  % Standard initial pointmax_iter = 10000;tolerance = 1e-8;
% Set appropriate learning rates for different algorithmslr_gd = 1e-4;      % Gradient descent requires a smaller learning ratelr_adaptive = 1e-2; % Adaptive algorithms can use a larger learning rate
fprintf('Initial point: (%.2f, %.2f)\n', x0(1), x0(2));fprintf('Target minimum point: (1.00, 1.00), f=0\n\n');
%% 1. Standard Gradient Descentfprintf('1. Running Standard Gradient Descent...\n');x = x0;traj_gd = zeros(2, max_iter);loss_gd = zeros(1, max_iter);
for iter = 1:max_iter    traj_gd(:, iter) = x;    loss_gd(iter) = rosenbrock(x);    grad = rosenbrock_grad(x);
    x = x - lr_gd * grad;
    if norm(grad) < tolerance || iter == max_iter        fprintf('   Iteration count: %d, Final position: (%.6f, %.6f), Final loss: %.2e\n', ...                iter, x(1), x(2), rosenbrock(x));        traj_gd = traj_gd(:, 1:iter);        loss_gd = loss_gd(1:iter);        break;    endend
%% 2. AdaGradfprintf('2. Running AdaGrad...\n');x = x0;g_accum = zeros(2, 1);traj_ada = zeros(2, max_iter);loss_ada = zeros(1, max_iter);
for iter = 1:max_iter    traj_ada(:, iter) = x;    loss_ada(iter) = rosenbrock(x);    grad = rosenbrock_grad(x);
    g_accum = g_accum + grad.^2;    adjusted_lr = lr_adaptive ./ (sqrt(g_accum) + 1e-8);    x = x - adjusted_lr .* grad;
    if norm(grad) < tolerance || iter == max_iter        fprintf('   Iteration count: %d, Final position: (%.6f, %.6f), Final loss: %.2e\n', ...                iter, x(1), x(2), rosenbrock(x));        traj_ada = traj_ada(:, 1:iter);        loss_ada = loss_ada(1:iter);        break;    endend
%% 3. RMSpropfprintf('3. Running RMSprop...\n');x = x0;g_sq_avg = zeros(2, 1);gamma = 0.9;traj_rms = zeros(2, max_iter);loss_rms = zeros(1, max_iter);
for iter = 1:max_iter    traj_rms(:, iter) = x;    loss_rms(iter) = rosenbrock(x);    grad = rosenbrock_grad(x);
    g_sq_avg = gamma * g_sq_avg + (1 - gamma) * grad.^2;    adjusted_lr = lr_adaptive ./ (sqrt(g_sq_avg) + 1e-8);    x = x - adjusted_lr .* grad;
    if norm(grad) < tolerance || iter == max_iter        fprintf('   Iteration count: %d, Final position: (%.6f, %.6f), Final loss: %.2e\n', ...                iter, x(1), x(2), rosenbrock(x));        traj_rms = traj_rms(:, 1:iter);        loss_rms = loss_rms(1:iter);        break;    endend
%% 4. Adamfprintf('4. Running Adam...\n');x = x0;m = zeros(2, 1);v = zeros(2, 1);beta1 = 0.9;beta2 = 0.999;traj_adam = zeros(2, max_iter);loss_adam = zeros(1, max_iter);
for iter = 1:max_iter    traj_adam(:, iter) = x;    loss_adam(iter) = rosenbrock(x);    grad = rosenbrock_grad(x);
    m = beta1 * m + (1 - beta1) * grad;    v = beta2 * v + (1 - beta2) * grad.^2;
    m_hat = m / (1 - beta1^iter);    v_hat = v / (1 - beta2^iter);
    x = x - lr_adaptive * m_hat ./ (sqrt(v_hat) + 1e-8);
    if norm(grad) < tolerance || iter == max_iter        fprintf('   Iteration count: %d, Final position: (%.6f, %.6f), Final loss: %.2e\n', ...                iter, x(1), x(2), rosenbrock(x));        traj_adam = traj_adam(:, 1:iter);        loss_adam = loss_adam(1:iter);        break;    endend
%% Visualizing Resultsfprintf('\nGenerating visualization results...\n');
% Create contour datax1 = linspace(-1.5, 1.5, 200);x2 = linspace(-0.5, 2.0, 200);[X1, X2] = meshgrid(x1, x2);Z = zeros(size(X1));
for i = 1:numel(X1)    Z(i) = rosenbrock([X1(i); X2(i)]);end
% Figure 1: Optimization trajectories of each algorithmfigure('Position', [100, 100, 1400, 1000]);
subplot(2, 2, 1);contour(X1, X2, log10(Z + 1), 30);hold on;plot(traj_gd(1, :), traj_gd(2, :), 'r.-', 'MarkerSize', 8, 'LineWidth', 1.5);plot(1, 1, 'g*', 'MarkerSize', 15, 'LineWidth', 2);plot(traj_gd(1, 1), traj_gd(2, 1), 'bo', 'MarkerSize', 10, 'LineWidth', 2);title('Gradient Descent');xlabel('x1'); ylabel('x2');legend('Contour', 'Optimization Path', 'Optimal Point', 'Starting Point', 'Location', 'best');grid on;
subplot(2, 2, 2);contour(X1, X2, log10(Z + 1), 30);hold on;plot(traj_ada(1, :), traj_ada(2, :), 'r.-', 'MarkerSize', 8, 'LineWidth', 1.5);plot(1, 1, 'g*', 'MarkerSize', 15, 'LineWidth', 2);plot(traj_ada(1, 1), traj_ada(2, 1), 'bo', 'MarkerSize', 10, 'LineWidth', 2);title('AdaGrad');xlabel('x1'); ylabel('x2');legend('Contour', 'Optimization Path', 'Optimal Point', 'Starting Point', 'Location', 'best');grid on;
subplot(2, 2, 3);contour(X1, X2, log10(Z + 1), 30);hold on;plot(traj_rms(1, :), traj_rms(2, :), 'r.-', 'MarkerSize', 8, 'LineWidth', 1.5);plot(1, 1, 'g*', 'MarkerSize', 15, 'LineWidth', 2);plot(traj_rms(1, 1), traj_rms(2, 1), 'bo', 'MarkerSize', 10, 'LineWidth', 2);title('RMSprop');xlabel('x1'); ylabel('x2');legend('Contour', 'Optimization Path', 'Optimal Point', 'Starting Point', 'Location', 'best');grid on;
subplot(2, 2, 4);contour(X1, X2, log10(Z + 1), 30);hold on;plot(traj_adam(1, :), traj_adam(2, :), 'r.-', 'MarkerSize', 8, 'LineWidth', 1.5);plot(1, 1, 'g*', 'MarkerSize', 15, 'LineWidth', 2);plot(traj_adam(1, 1), traj_adam(2, 1), 'bo', 'MarkerSize', 10, 'LineWidth', 2);title('Adam');xlabel('x1'); ylabel('x2');legend('Contour', 'Optimization Path', 'Optimal Point', 'Starting Point', 'Location', 'best');grid on;
sgtitle('Comparison of Optimization Trajectories of Adaptive Gradient Algorithms', 'FontSize', 16, 'FontWeight', 'bold');
% Figure 2: Comparison of loss descent curvesfigure('Position', [100, 100, 1200, 500]);
subplot(1, 2, 1);semilogy(loss_gd, 'r-', 'LineWidth', 2); hold on;semilogy(loss_ada, 'b-', 'LineWidth', 2);semilogy(loss_rms, 'g-', 'LineWidth', 2);semilogy(loss_adam, 'm-', 'LineWidth', 2);xlabel('Iteration Count');ylabel('Loss Value (log scale)');title('Loss Descent Curve');legend('Gradient Descent', 'AdaGrad', 'RMSprop', 'Adam', 'Location', 'best');grid on;
subplot(1, 2, 2);% Display the first 200 iterations to observe early convergencemin_len = min([length(loss_gd), length(loss_ada), length(loss_rms), length(loss_adam), 200]);iter_range = 1:min_len;semilogy(iter_range, loss_gd(iter_range), 'r-', 'LineWidth', 2); hold on;semilogy(iter_range, loss_ada(iter_range), 'b-', 'LineWidth', 2);semilogy(iter_range, loss_rms(iter_range), 'g-', 'LineWidth', 2);semilogy(iter_range, loss_adam(iter_range), 'm-', 'LineWidth', 2);xlabel('Iteration Count');ylabel('Loss Value (log scale)');title('Early Loss Descent (First 200 Iterations)');legend('Gradient Descent', 'AdaGrad', 'RMSprop', 'Adam', 'Location', 'best');grid on;
sgtitle('Comparison of Loss Descent Processes of Optimization Algorithms', 'FontSize', 16, 'FontWeight', 'bold');
% Figure 3: Comprehensive Comparisonfigure('Position', [100, 100, 1000, 800]);
% Create finer contourscontourf(X1, X2, log10(Z + 1), 50, 'LineColor', 'none');colormap(parula);colorbar;hold on;
% Plot all trajectoriesplot(traj_gd(1, :), traj_gd(2, :), 'r.-', 'MarkerSize', 10, 'LineWidth', 2, 'DisplayName', 'Gradient Descent');plot(traj_ada(1, :), traj_ada(2, :), 'b.-', 'MarkerSize', 10, 'LineWidth', 2, 'DisplayName', 'AdaGrad');plot(traj_rms(1, :), traj_rms(2, :), 'g.-', 'MarkerSize', 10, 'LineWidth', 2, 'DisplayName', 'RMSprop');plot(traj_adam(1, :), traj_adam(2, :), 'm.-', 'MarkerSize', 10, 'LineWidth', 2, 'DisplayName', 'Adam');
% Mark key pointsplot(1, 1, 'y*', 'MarkerSize', 20, 'LineWidth', 3, 'DisplayName', 'Global Optimal Point (1,1)');plot(x0(1), x0(2), 'ko', 'MarkerSize', 12, 'LineWidth', 2, 'MarkerFaceColor', 'w', 'DisplayName', 'Starting Point');
title('Comprehensive Comparison of Adaptive Gradient Algorithms');xlabel('x1');ylabel('x2');legend('Location', 'best');grid on;
%% Performance Analysisfprintf('\n=== Performance Analysis ===\n');fprintf('Algorithm       Iteration Count   Final Loss       Final Position\n');fprintf('---------------------------------------------\n');fprintf('Gradient Descent   %-8d   %.2e   (%.4f, %.4f)\n', length(loss_gd), loss_gd(end), traj_gd(1,end), traj_gd(2,end));fprintf('AdaGrad    %-8d   %.2e   (%.4f, %.4f)\n', length(loss_ada), loss_ada(end), traj_ada(1,end), traj_ada(2,end));fprintf('RMSprop    %-8d   %.2e   (%.4f, %.4f)\n', length(loss_rms), loss_rms(end), traj_rms(1,end), traj_rms(2,end));fprintf('Adam       %-8d   %.2e   (%.4f, %.4f)\n', length(loss_adam), loss_adam(end), traj_adam(1,end), traj_adam(2,end));
% Calculate convergence speedtarget_loss = 1e-4;fprintf('\nIterations required to reach loss %.0e:\n', target_loss);for i = 1:4    switch i        case 1            loss = loss_gd; name = 'Gradient Descent';        case 2            loss = loss_ada; name = 'AdaGrad';        case 3            loss = loss_rms; name = 'RMSprop';        case 4            loss = loss_adam; name = 'Adam';    end    idx = find(loss < target_loss, 1);    if ~isempty(idx)        fprintf('  %s: %d iterations\n', name, idx);    else        fprintf('  %s: Not reached\n', name);    endend
fprintf('\nProgram completed!\n');