Optima of a function

We all know that the differentiation of a function is zero at point x if the function is optimum (or saddle point) at x. We will try to see what does that means intuitively and using a graph. For more formulas on calculus check out my other blog. (Calculus in AI and ML)

Theory:

Single Variable Function:

The function has optima at x if its first derivative equals 0 at x. We follow the below process to find the maxima or minima of a function.

\[\begin{align*} &\text{For function } y=f(x),\\ &\text{1. Evaulate c where } \frac{\mathrm{d} y}{\mathrm{d} x}\mid_{x=c} = f'(c) = 0 \\ &\text{There can be more than one c where} f'(c)=0\\ &\text{2. For each c calculate } \frac{\mathrm{d^2} y}{\mathrm{d} x^2} = f''(c) \\ &\text{if }f''(c) \begin{cases} & <0 \text{ There is Maxima at c} \\ & >0 \text{ There is Minima at c} \\ & =0 \text{ Can't be determined} \end{cases} \end{align*}\]

Intuition:

Let's take function f, its first derivative f' and second derivative f'' and plot them on a graph.

\[\begin{align*} f(x) &= x^{3}-20x^{2}+500\\ \frac{\mathrm{d} f}{\mathrm{d} x} = f'(x) &= 3x^2 - 40x\\ \frac{\mathrm{d^2} f}{\mathrm{d} x^2} = f''(x) &= 6x - 40 \end{align*}\]

Optima of a single variable function

Now we find the roots of f'(x) = 0, which are 

A = 0 and B = 13.33

As per the theory, we find f'' at 0 and 13.33

f''(0) = -40.......<0 So f is maxima at x = 0

f''(13.33) = 40....>0 So f is minima at x = 13.33

Now we calculate f at 0 and 13.33

f(0) = 500 (maxima)

f(13.33) = -685.2 (minima).

How on earth is this working?

f is a function. Its derivative f' represents its rate of change at a given point. So if f is increasing we can observe f' is positive and if f is decreasing we can observe f' is negative.

Between point A and B function f is decreasing so value of f' is negative.

If f' is zero that represents function is neither increasing nor decreasing. At point A and B, f' is zero and function is neither increasing nor decreasing. But function changes its path at A and B either from increasing to decreasing or vice versa.

At point A or H, the function f was increasing and then it starts decreasing. So it is maxima there. We climb to the peak and then started to walk down.

At point B or I, the function f was decreasing and then it starts increasing. So it is minima there. We walked to the valley and then started to climb up.

How can we check if it's minima or maxima if f' is zero?

Novice way: We can check the value of the function before and after at the optima points to check if it's peak or valley. But it is lengthy and can cause an error if the function is changing rapidly with a small change in x.

Math way: Here the second derivative comes into the picture. First derivative tells us the state of the function- if it is increasing, decreasing or optimum. But at optima f' is zero. So to check if the function is increasing or decreasing after the optimum point, we can take the derivative of the first derivative which is nothing but the second derivative of the function.

The second derivative represents the rate of change of the first derivative, So if the second derivative is positive f' is increasing, and if it is negative f' is decreasing. Let's understand case by case,

Case 1 f'' < 0: (point A, x = 0, f' = 0, f'' = -40)

  f' is decreasing at x = 0 because f'' is negative. So that we can say that f was optimum and start decreasing as f' is negative because it is decreasing from zero.

  So if f is optimum and then start decreasing means it was maxima (f = 500) there. Because you are walking down from the peak.

Case 2 f'' > 0: (point B, x = 13.33, f' = 0, f'' = 40)

  f' is increasing at x = 13.33 because f'' is positive. So that we can say that f was optimum and start increasing as f' is positive because it is increasing from zero.

  So if f is optimum and then start increasing means it was minima (f = -685.2) there. Because you are walking up from the valley.