Gentle Introduction to Gradients and Level Curves (Surfaces)

In this introductory optimization tutorial, we provide a gentle introduction to gradients and level curves (surfaces). Gradients are fundamental mathematical objects that appear in many scientific, mathematics, engineering, and physics fields, such as optimization, signal processing, control theory, fluid dynamics, physics, etc. The YouTube video accompanying this tutorial is given below.

What is a Gradient?

Let us first explain the mathematical concept of a gradient. Let us consider the following function

(1)  

where and are independent variables, and is a real scalar value. This function is illustrated in the figure below.

Now, the question is:

What is the gradient of this function at a certain point?

From the mathematical point of view, the gradient is a vector defined by the following equation

(2)  

where

• (nabla) is a symbol transforming f into the gradient
• is the gradient vector
• and are the partial derivatives of with respect to and .

Let us compute the gradient of the function (1). The gradient is given by the following equation

(3)  

The gradient at the point is equal to

(4)  

This gradient is shown in the figure below (red arrow).

When the function depends on variables: , , …, , then the gradient is defined by

(5)  

For presentation clarity, let us return to the example of the function of two variables. Everything explained in this tutorial can easily be generalized to the case of functions depending on three or more variables. Several important facts about this gradient should be observed:

• The gradient is equal to zero at the point . This point is actually the minimum of the function. That is, the function achieves the minimum at the point for which the gradient is equal to zero. We have
  

  (6)  

  
  This expression is zero for and , and from the form of the function (1), we conclude that the minimum is achieved at and .
  
  On the other hand, if our function was for example
  
  

  (7)  

  
  
  Then, the gradient is
  
  

  (8)  

  
  
  The function achieves a maximum at the point , that is, it achieves the maximum value at the point for which its gradient is equal to zero. These simple examples confirm that the candidate points for (local) minimum and maximum of functions are the points for which their gradients become equal to zero. Keep in mind that these are only necessary conditions for a local extremum. The sufficient conditions involve Hessians of functions. This topic is out of the scope of this tutorial.
  
• The gradient is a vector whose value at a certain point is the direction and rate of the (locally) fastest increase of the function . For example, the gradient at the point of the function defined by the equation (1) is
  
  

  (9)  

  
  That is, the direction of the fastest increase is a vector
  
  

  (10)  

  
  Let us numerically illustrate this fact. That is, let us show that starting at the point , the direction of the fastest increase of the function is positively collinear with the vector (10). Suppose that at the point , we move in the direction of a vector that is positively collinear with the gradient vector. We assume that we take a small step in this direction. We can select an infinite number of vectors that are collinear with the gradient vector. The most natural choice is the following unit vector:
  
  

  (11)  

  
  That is, new point at which we want to evaluate the value of the function is
  
  

  (12)  

  
  The value of the function at this point is
  
  

  (13)  

  
  
  Now, let us assume that at the point , we move in the direction of another unit vector, which makes an angle of 60 degrees with respect to the x-axis. Again, we assume that we take small steps in this direction. The vector is
  
  

  (14)  

  
  Let us evaluate the value of the function at a new point in the direction of this vector. The new point is defined by
  
  

  (15)  

  
  The value of the function at this point is
  
  

  (16)  

  
  And obviously, this value is smaller than the value of the function at the point in the direction of the gradient vector. Similarly, it can be numerically verified that if we take any direction, along any unit vector from the point , except for the direction of the unit vector that is positively collinear with the gradient, and take a small step, we will always have a smaller function value than the value of the function in the direction of the unit vector positively collinear with the gradient vector.
  
  

To properly understand gradients, it is also important to introduce the concept of level curves for the case when the function depends on two variables and the concept of level surfaces for the case when the function depends on three or more variables. Consider the function . The level curve of this function is defined by

(17)  

where is a constant. Obviously, the equation (17) defines a curve that is the intersection of the function with the horizontal plane parallel to the xy plane with the distance of from the xy plane.

In the case of the function defined by (1) the level curves are obviously circles centered at with the radius of :

(18)  

The figure below illustrates the level curve.

In the same manner, we can define the level surfaces for functions with three or more variables. For example, for the function

(19)  

The level surface is given by the following equation

(20)  

Obviously, this is a sphere with the radius of centered at zero.

Why are the level curves and level surfaces relevant to the concept of gradients?

Because of this reason:

• The gradient at point is perpendicular to the level curve (level surface) that passes through that point. More precisely, the gradient at the point is perpendicular to the tangent (the tangent plane in the case of level surfaces) of the level curve (level surface) at the point . This is illustrated in the figure below for the function (1) and for .