Lagrangians – derivation under constraints

In this post I will describe a formal technique which can help in finding derivatives of functions under certain constraints. In order to see what I mean, take the following two examples:

1. If you differentiate the function {x \mapsto f} with respect to {x}, you get zero, obviously.

2. If you differentiate the function {x \mapsto f} under the constraint that {f^2=x} (supposing that {x>0}) then it is not hard to see that {f = \sqrt{x}} and the derivative will be {1/(2\sqrt{x})}.

This shows that adding certain constraints will change the derivative and when dealing with more sophisticated constraints, like PDE or differential equations, things get less clear.

The question of the existence of derivatives with respect to some variables, when dealing with constraints, is usually dealt with by using the implicit function theorem: this basically says that if some variables are linked by some smooth equations and that you can locally invert the dependence, then you can infer differentiability.

The method presented below skips this step and goes directly for the computation of the derivative. This is a common used technique in control theory or shape optimization, where computing derivatives is essential in numerical simulations. I will come back to derivatives linked to shape optimization in some further posts. For now, let’s see how one can use the Lagrangian method for computing derivatives in some simple cases.

Example 1. Let’s compute the derivative of {x \mapsto f} under the constraint {f^2 = x}. In order to do this we introduce a Lagrangian, which has the following form

\displaystyle \text{(objective function) } + \text{ (penalization of the constraints)}.

The penalization of the constraints is done using Lagrange multipliers. In our case we can define the Lagrangian {\mathcal L} in the following way:

\displaystyle \mathcal L(x,g,q) = g + (g^2-x)q

where {g} takes the place of {g} and {q} is the Lagrange multiplier. When introducing the Lagrangian it is important to make all variables independent so that when dealing with derivatives, we won’t have to worry about derivatives of composite functions. Moreover, note that if we take {g = f = \sqrt{x}} then

\displaystyle \mathcal L(x,g,q) = f,

so we recover the functional we wish to differentiate. Now imagine we differentiate {\mathcal L} directly w.r.t {x}:

\displaystyle f'(x) = \frac{\partial L}{\partial x}(x,f,q)+\frac{\partial L}{\partial g}(x,f,q)\frac{df}{dx}+\frac{\partial L}{\partial q}(x,f,q)\frac{dq}{dx}.

The fact that we have the derivatives of {f} and {q} which appear makes the formulas more complicated. In order to eliminate such terms we may search for the so-called saddle-points w.r.t. variables {g} and {q}. Let’s investigate where these partial derivatives vanish:

  • (a) The derivative w.r.t. {q}:

    \displaystyle \frac{\partial L}{\partial q}(x,f,q)=g^2-x=0,

    which gives {g=\sqrt{x}}. This is the state equation, and obtaining it is natural when differentiating w.r.t. Lagrange multipliers.

  • (b) The derivative w.r.t. {g}:

    \displaystyle \frac{\partial L}{\partial g}(x,f,q) =1+2gq = 0,

    giving {q = 1/2g}. This is the adjoint equation, giving the Lagrange multiplier once the state is known.

Now let’s suppose that {f = f(x)} and {f} verifies the state equation {f^2=x}. Then

\displaystyle f(x) = \mathcal{L}(x,f(x),q).

Differntiating with respect to {x} we get

\displaystyle f'(x) = \frac{\partial L}{\partial x}(x,f(x),q)+\frac{\partial L}{\partial g}(x,f(x),q)\frac{df}{dx}+\frac{\partial L}{\partial q}(x,f(x),q)\frac{dq}{dx},

and this holds for all {q}. Note that the second term vanishes, as {f} is a solution of the state equation. We are free to choose a convenient value for {q} and the natural choice is to cancel the third term. For this we choose {q = 1/(2g) = 1/(2f(x))}. Now it remains to compute the derivative of {\mathcal L} w.r.t. {x} and this gives

\displaystyle \frac{\partial L}{\partial x}(x,f,q) = -q

In conclusion we have

\displaystyle f'(x) = \frac{\partial L}{\partial x}(x,f(x),-1/(2f(x))) = \frac{1}{2f(x)} = \frac{1}{2\sqrt{x}}

where we used the fact that {f(x)^2=x} and therefore {f(x) = \sqrt{x}}.

Example 2. Let’s find the derivative of {f} under the constraint {\exp(f) = x}. This is equivalent to finding the derivative of {\ln x} (for {x>0}, of course).

Define the Lagrangian

\displaystyle \mathcal L(x,g,q) = g+(\exp(g)-x)q.

As before, we look for points where derivatives of {\mathcal L} w.r.t. {g} and {q} vanish:

  • (a) The derivative w.r.t. {q}:

    \displaystyle \frac{\partial L}{\partial q}(x,f,q)=\exp(g)-x=0,

    which gives {g=\ln{x}}.

  • (b) The derivative w.r.t. {g}:

    \displaystyle \frac{\partial L}{\partial g}(x,f,q) =1+\exp(g)q = 0,

    giving {q = -1/\exp(g)}.

Now we recall that if {\exp(f(x))=x} then

\displaystyle f(x) = \mathcal L(x,f(x),q)

for all {q}. When differentiating, if we replace {q} using the relation given by the adjoint state then we’ll cancel the partial derivative w.r.t. {q}. Therefore we are left with

\displaystyle f'(x) = \frac{\partial L}{\partial x} \mathcal L(x,f(x),-1/\exp(f(x)))=-q=\frac{1}{\exp(f(x))} = \frac{1}{\exp(\ln x))} = \frac{1}{x},

as expected.

Example 3. (do it yourself) Use the Lagrangian formulation to recover the computation of the derivative of the inverse function.

  1. No comments yet.
  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: