Gradient descent is an optimization algorithm used to minimize a function's value iteratively. It involves taking steps proportional to the negative of the gradient (slope) of the function at each point. By repeatedly updating the parameters, such as weights or coefficients, in the direction of the steepest descent, the algorithm moves closer to the function's minimum. The size of the steps is determined by a learning rate, which controls the convergence speed. Gradient descent is widely used in machine learning to optimize models by minimizing a loss function, allowing the model to find the best set of parameters for accurate predictions or fitting data.
Batch GD:
Batch gradient descent is a variant of the gradient descent algorithm used in machine learning. In this approach, the entire training dataset is divided into batches, and the model's parameters are updated based on the average gradient computed over each batch. Unlike other variants of gradient descent that update parameters after processing each individual data point (stochastic gradient descent) or a subset of the data (mini-batch gradient descent), batch gradient descent computes the gradients using the entire dataset. This results in a more stable and precise update direction, but it can be computationally expensive for large datasets. However, batch gradient descent is often used when computational resources permit to achieve high-quality convergence.
Stochastic GD:
Stochastic gradient descent (SGD) is an optimization algorithm commonly used in machine learning. Unlike batch gradient descent, which updates parameters based on the average gradient computed over batches of data, SGD updates parameters after processing each individual data point. This approach introduces randomness into the parameter updates, making it more computationally efficient, especially for large datasets. SGD approximates the true gradient by randomly sampling a single data point or a small subset (mini-batch) of data at each iteration. While this introduces more noise in the gradient estimation, it allows for faster convergence and better exploration of the optimization landscape. SGD is particularly useful in scenarios with large datasets and in online learning settings where new data arrives continuously.