Notes on automatic differentiation#

Preliminaries#

Definition of normalised derivative:

(1)#

x^{[n]} (t) = \frac{1}{n!} x^{(n)} (t) .

General Leibniz rule: given $a (t) = b (t) c (t)$ , then

(2)#

a^{[n]} (t) = \sum_{j = 0}^{n} b^{[n - j]} (t) c^{[j]} (t) .

Basic arithmetic#

Addition and subtraction#

Given $a (t) = b (t) \pm c (t)$ , trivially

(3)#

a^{[n]} (t) = b^{[n]} (t) \pm c^{[n]} (t) .

Multiplication#

Given $a (t) = b (t) c (t)$ , the derivative $a^{[n]} (t)$ is given directly by the application of the general Leibniz rule (2).

Division#

Given $a (t) = \frac{b (t)}{c (t)}$ , we can write

(4)#

a (t) c (t) = b (t) .

We can now apply the normalised derivative of order $n$ to both sides, use (2) and re-arrange to obtain:

(5)#

a^{[n]} (t) = \frac{1}{c^{[0]} (t)} [b^{[n]} (t) - \sum_{j = 1}^{n} a^{[n - j]} (t) c^{[j]} (t)] .

Squaring#

Given $a (t) = b {(t)}^{2}$ , the computation of $a^{[n]} (t)$ is a special case of (2) in which we take advantage of the summation’s symmetry in order to halve the computational complexity:

(6)#

\begin{array}{r} a^{[n]} (t) = {\begin{cases} 2 \sum_{j = 0}^{\frac{n}{2} - 1} b^{[n - j]} (t) b^{[j]} (t) + {(b^{[\frac{n}{2}]} (t))}^{2} if n is even, \\ 2 \sum_{j = 0}^{\frac{n - 1}{2}} b^{[n - j]} (t) b^{[j]} (t) if n is odd . \end{cases} \end{array}

Square root#

Given $a (t) = \sqrt{b (t)}$ , we can write

(7)#

a {(t)}^{2} = b (t) .

We can apply the normalised derivative of order $n$ to both sides, and, with the help of (6), we obtain:

(8)#

\begin{array}{r} b^{[n]} (t) = {\begin{cases} 2 \sum_{j = 0}^{\frac{n}{2} - 1} a^{[n - j]} (t) a^{[j]} (t) + {(a^{[\frac{n}{2}]} (t))}^{2} if n is even, \\ 2 \sum_{j = 0}^{\frac{n - 1}{2}} a^{[n - j]} (t) a^{[j]} (t) if n is odd . \end{cases} \end{array}

We can then isolate $a^{[n]} (t)$ to obtain:

(9)#

\begin{array}{r} a^{[n]} (t) = {\begin{cases} \frac{1}{2 a^{[0]} (t)} [b^{[n]} (t) - 2 \sum_{j = 1}^{\frac{n}{2} - 1} a^{[n - j]} (t) a^{[j]} (t) - {(a^{[\frac{n}{2}]} (t))}^{2}] if n is even, \\ \frac{1}{2 a^{[0]} (t)} [b^{[n]} (t) - 2 \sum_{j = 0}^{\frac{n - 1}{2}} a^{[n - j]} (t) a^{[j]} (t)] if n is odd . \end{cases} \end{array}

Exponentiation#

Given $a (t) = b {(t)}^{α}$ , with $α \neq 0$ , we have

(10)#

a^{'} (t) = α b {(t)}^{α - 1} b^{'} (t) .

By multiplying both sides by $b (t)$ we obtain

(11)#

\begin{array}{r} \begin{aligned} b (t) a^{'} (t) & = b (t) α b {(t)}^{α - 1} b^{'} (t) \\ = α b^{'} (t) a (t) . \end{aligned} \end{array}

We can now apply the normalised derivative of order $n - 1$ to both sides, use (2) and re-arrange to obtain, for $n > 0$ :

(12)#

a^{[n]} (t) = \frac{1}{n b^{[0]} (t)} \sum_{j = 0}^{n - 1} [n α - j (α + 1)] b^{[n - j]} (t) a^{[j]} (t) .

Exponentials#

Natural exponential#

Given $a (t) = e^{b (t)}$ , we have

(13)#

a^{'} (t) = e^{b (t)} b^{'} (t) = a (t) b^{'} (t) .

We can now apply the normalised derivative of order $n - 1$ to both sides, use (1) and (2) and obtain, for $n > 0$ :

(14)#

a^{[n]} (t) = \frac{1}{n} \sum_{j = 1}^{n} j a^{[n - j]} (t) b^{[j]} (t) .

Standard logistic function#

Given $a (t) = sig b (t)$ , where $sig (x)$ is the standard logistic function

(15)#

sig (x) = \frac{1}{1 + e^{- x}},

we have

(16)#

a^{'} (t) = sig b (t) [1 - sig b (t)] b^{'} (t) = a (t) [1 - a (t)] b^{'} (t),

which, after the introduction of the auxiliary function

(17)#

c (t) = a^{2} (t),

becomes

(18)#

a^{'} (t) = [a (t) - c (t)] b^{'} (t) .

After applying the normalised derivative of order $n - 1$ to both sides, we can use (1), (2) and (3) to obtain, for $n > 0$ :

(19)#

a^{[n]} (t) = \frac{1}{n} \sum_{j = 1}^{n} j [a^{[n - j]} (t) - c^{[n - j]} (t)] b^{[j]} (t) .

Logarithms#

Natural logarithm#

Given $a (t) = \log b (t)$ , we have

(20)#

a^{'} (t) = \frac{b^{'} (t)}{b (t)},

or, equivalently,

(21)#

b (t) a^{'} (t) = b^{'} (t) .

We can now apply the normalised derivative of order $n - 1$ to both sides, use (1) and (2) and re-arrange to obtain, for $n > 0$ :

(22)#

a^{[n]} (t) = \frac{1}{n b^{[0]} (t)} [n b^{[n]} (t) - \sum_{j = 1}^{n - 1} j b^{[n - j]} (t) a^{[j]} (t)] .

Trigonometric functions#

Tangent#

Given $a (t) = \tan b (t)$ , we have

(23)#

a^{'} (t) = [\tan^{2} b (t) + 1] b^{'} (t) = a^{2} (t) b^{'} (t) + b^{'} (t),

which, after the introduction of the auxiliary function

(24)#

c (t) = a^{2} (t),

becomes

(25)#

a^{'} (t) = c (t) b^{'} (t) + b^{'} (t) .

After applying the normalised derivative of order $n - 1$ to both sides, we can use (1), (2) and (3) to obtain, for $n > 0$ :

(26)#

a^{[n]} (t) = \frac{1}{n} \sum_{j = 1}^{n} j c^{[n - j]} (t) b^{[j]} (t) + b^{[n]} (t) .

Inverse trigonometric functions#

Inverse sine#

Given $a (t) = \arcsin b (t)$ , we have

(27)#

a^{'} (t) = \frac{b^{'} (t)}{\sqrt{1 - b^{2} (t)}},

or, equivalently,

(28)#

a^{'} (t) \sqrt{1 - b^{2} (t)} = b^{'} (t) .

We introduce the auxiliary function

(29)#

c (t) = \sqrt{1 - b^{2} (t)},

so that (28) can be rewritten as

(30)#

a^{'} (t) c (t) = b^{'} (t) .

Applying the normalised derivative of order $n - 1$ to both sides yields, via (1):

(31)#

{[a^{'} (t) c (t)]}^{[n - 1]} = n b^{[n]} (t) .

We can now apply the general Leibniz rule (2) to the left-hand side and re-arrange the terms to obtain, for $n > 0$ :

(32)#

a^{[n]} (t) = \frac{1}{n c^{[0]} (t)} [n b^{[n]} (t) - \sum_{j = 1}^{n - 1} j c^{[n - j]} (t) a^{[j]} (t)] .

Inverse cosine#

The derivation is identical to the inverse sine, apart from a sign change. Given $a (t) = \arccos b (t)$ , the final result is, for $n > 0$ :

(33)#

a^{[n]} (t) = - \frac{1}{n c^{[0]} (t)} [n b^{[n]} (t) + \sum_{j = 1}^{n - 1} j c^{[n - j]} (t) a^{[j]} (t)],

with $c (t)$ defined as:

(34)#

c (t) = \sqrt{1 - b^{2} (t)} .

Inverse tangent#

Given $a (t) = \arctan b (t)$ , we have

(35)#

a^{'} (t) = \frac{b^{'} (t)}{1 + b^{2} (t)},

or, equivalently,

(36)#

a^{'} (t) [1 + b^{2} (t)] = b^{'} (t) .

We introduce the auxiliary function

(37)#

c (t) = b^{2} (t),

so that (36) can be rewritten as

(38)#

a^{'} (t) + a^{'} (t) c (t) = b^{'} (t) .

Applying the normalised derivative of order $n - 1$ to both sides yields, via (1) and (3):

(39)#

n a^{[n]} (t) + {[a^{'} (t) c (t)]}^{[n - 1]} = n b^{[n]} (t) .

With the help of the general Leibniz rule (2), after re-arranging we obtain, for $n > 0$ :

(40)#

a^{[n]} (t) = \frac{1}{n [c^{[0]} (t) + 1]} [n b^{[n]} (t) - \sum_{j = 1}^{n - 1} j c^{[n - j]} (t) a^{[j]} (t)] .

Two-argument inverse tangent#

Given $a (t) = \arctan 2 (b (t), c (t))$ , we have

(41)#

a^{'} (t) = \frac{c (t) b^{'} (t) - b (t) c^{'} (t)}{b^{2} (t) + c^{2} (t)} .

After the introduction of the auxiliary function

(42)#

d (t) = b^{2} (t) + c^{2} (t),

(41) can be rewritten as

(43)#

d (t) a^{'} (t) = c (t) b^{'} (t) - b (t) c^{'} (t) .

We can now apply the normalised derivative of order $n - 1$ to both sides, and, via (2), obtain, for $n > 0$ :

(44)#

\begin{array}{r} \begin{aligned} a^{[n]} (t) & = \frac{1}{n d^{[0]} (t)} [n (c^{[0]} (t) b^{[n]} (t) - b^{[0]} (t) c^{[n]} (t)) \\ + \sum_{j = 1}^{n - 1} j (c^{[n - j]} (t) b^{[j]} (t) - b^{[n - j]} (t) c^{[j]} (t) - d^{[n - j]} (t) a^{[j]} (t))] . \end{aligned} \end{array}

Hyperbolic functions#

Hyperbolic sine#

Given $a (t) = \sinh b (t)$ , we have

(45)#

a^{'} (t) = b^{'} (t) \cosh b (t) .

We introduce the auxiliary function

(46)#

c (t) = \cosh b (t),

so that (45) can be rewritten as

(47)#

a^{'} (t) = c (t) b^{'} (t) .

We can now apply the normalised derivative of order $n - 1$ to both sides, and, via (2), obtain, for $n > 0$ :

(48)#

a^{[n]} (t) = \frac{1}{n} \sum_{j = 1}^{n} j c^{[n - j]} (t) b^{[j]} (t) .

Hyperbolic cosine#

Given $a (t) = \cosh b (t)$ , the process of deriving $a^{[n]} (t)$ is identical to the hyperbolic sine. After the definition of the auxiliary function

(49)#

s (t) = \sinh b (t),

the final result, for $n > 0$ , is:

(50)#

a^{[n]} (t) = \frac{1}{n} \sum_{j = 1}^{n} j s^{[n - j]} (t) b^{[j]} (t) .

Hyperbolic tangent#

Given $a (t) = \tanh b (t)$ , the process of deriving $a^{[n]} (t)$ is identical to the tangent, apart from a sign change. After the definition of the auxiliary function

(51)#

c (t) = a^{2} (t),

the final result, for $n > 0$ , is:

(52)#

a^{[n]} (t) = b^{[n]} (t) - \frac{1}{n} \sum_{j = 1}^{n} j c^{[n - j]} (t) b^{[j]} (t) .

Inverse hyperbolic functions#

Inverse hyperbolic sine#

Given $a (t) = arsinh b (t)$ , the process of deriving $a^{[n]} (t)$ is identical to the inverse sine, apart from a sign change. After the definition of the auxiliary function

(53)#

c (t) = \sqrt{1 + b^{2} (t)},

the final result, for $n > 0$ , is:

(54)#

a^{[n]} (t) = \frac{1}{n c^{[0]} (t)} [n b^{[n]} (t) - \sum_{j = 1}^{n - 1} j c^{[n - j]} (t) a^{[j]} (t)] .

Inverse hyperbolic cosine#

Given $a (t) = arcosh b (t)$ , the process of deriving $a^{[n]} (t)$ is identical to the inverse hyperbolic sine, apart from a sign change. After the definition of the auxiliary function

(55)#

c (t) = \sqrt{b^{2} (t) - 1},

the final result, for $n > 0$ , is:

(56)#

a^{[n]} (t) = \frac{1}{n c^{[0]} (t)} [n b^{[n]} (t) - \sum_{j = 1}^{n - 1} j c^{[n - j]} (t) a^{[j]} (t)] .

Inverse hyperbolic tangent#

Given $a (t) = artanh b (t)$ , the process of deriving $a^{[n]} (t)$ is identical to the inverse tangent, apart from a sign change. After the definition of the auxiliary function

(57)#

c (t) = b^{2} (t),

the final result, for $n > 0$ , is:

(58)#

a^{[n]} (t) = \frac{1}{n [1 - c^{[0]} (t)]} [n b^{[n]} (t) + \sum_{j = 1}^{n - 1} j c^{[n - j]} (t) a^{[j]} (t)] .

Special functions#

Error function#

Given $a (t) = \erf b (t)$ , we have

(59)#

a^{'} (t) = \frac{2}{\sqrt{π}} \exp [- b^{2} (t)] b^{'} (t),

which, after the introduction of the auxiliary function

(60)#

c (t) = \exp [- b^{2} (t)],

becomes

(61)#

a^{'} (t) = \frac{2}{\sqrt{π}} c (t) b^{'} (t) .

After applying the normalised derivative of order $n - 1$ to both sides, we can use (1) and (2) to obtain, for $n > 0$ :

(62)#

a^{[n]} (t) = \frac{1}{n} \frac{2}{\sqrt{π}} \sum_{j = 1}^{n} j c^{[n - j]} (t) b^{[j]} (t) .

Celestial mechanics#

Kepler’s eccentric anomaly#

The eccentric anomaly is the bivariate function $E = E (e, M)$ implicitly defined by the trascendental equation

(63)#

M = E - e \sin E,

with $e \in [0, 1)$ . Given $a (t) = E (e (t), M (t))$ , we have

(64)#

a^{'} (t) = \frac{\partial E}{\partial e} e^{'} (t) + \frac{\partial E}{\partial M} M^{'} (t),

where the partial derivatives are

(65)#

\begin{array}{r} {\begin{cases} \frac{\partial E}{\partial e} = \frac{\sin E}{1 - e \cos E}, \\ \frac{\partial E}{\partial M} = \frac{1}{1 - e \cos E} . \end{cases} \end{array}

Expanding the partial derivatives yields

(66)#

a^{'} (t) = \frac{e^{'} (t) \sin a (t) + M^{'} (t)}{1 - e (t) \cos a (t)},

or, equivalently,

(67)#

a^{'} (t) - a^{'} (t) e (t) \cos a (t) = e^{'} (t) \sin a (t) + M^{'} (t) .

We can now introduce the auxiliary functions

(68)#

\begin{array}{r} {\begin{cases} c (t) = e (t) \cos a (t), \\ d (t) = \sin a (t), \end{cases} \end{array}

so that (67) can be rewritten as

(69)#

a^{'} (t) - a^{'} (t) c (t) = e^{'} (t) d (t) + M^{'} (t) .

After applying the normalised derivative of order $n - 1$ to both sides, we can use (1) and (2) and re-arrange to obtain, for $n > 0$ :

(70)#

a^{[n]} (t) = \frac{1}{n (1 - c^{[0]} (t))} [n (e^{[n]} (t) d^{[0]} (t) + M^{[n]} (t)) + \sum_{j = 1}^{n - 1} j (c^{[n - j]} (t) a^{[j]} (t) + d^{[n - j]} (t) e^{[j]} (t))] .

Eccentric longitude#

The eccentric longitude is the trivariate function $F = F (h, k, λ)$ implicitly defined by the trascendental equation

(71)#

λ = F + h \cos F - k \sin F,

with $h^{2} + k^{2} < 1$ . Given $a (t) = F (h (t), k (t), λ (t))$ , we have

(72)#

a^{'} (t) = \frac{k^{'} (t) \sin a (t) - h^{'} (t) \cos a (t) + λ^{'} (t)}{1 - h (t) \sin a (t) - k (t) \cos a (t)} .

After the introduction of the auxiliary functions

(73)#

\begin{array}{r} {\begin{cases} c (t) = h (t) \sin a (t), \\ d (t) = k (t) \cos a (t), \\ e (t) = \sin a (t), \\ f (t) = \cos a (t), \end{cases} \end{array}

we can then proceed in the same way as explained for the eccentric anomaly. The final result, for $n > 0$ , is:

(74)#

a^{[n]} (t) = \frac{1}{n (1 - c^{[0]} (t) - d^{[0]} (t))} {n (k^{[n]} (t) e^{[0]} (t) - h^{[n]} (t) f^{[0]} (t) + λ^{[n]} (t)) + \sum_{j = 1}^{n - 1} j [a^{[j]} (t) (c^{[n - j]} (t) + d^{[n - j]} (t)) + k^{[j]} (t) e^{[n - j]} (t) - h^{[j]} (t) f^{[n - j]} (t)]} .

Time functions#

Time polynomials#

Given the time polynomial of order $n$

(75)#

p_{n} (t) = \sum_{i = 0}^{n} a_{i} t^{i},

its derivative of order $j$ is

(76)#

{(p_{n} (t))}^{(j)} = \sum_{i = j}^{n} {(i)}_{j} a_{i} t^{i - j},

where ${(i)}_{j}$ is the falling factorial. The normalised derivative of order $j$ is

(77)#

{(p_{n} (t))}^{[j]} = \frac{1}{j!} \sum_{i = j}^{n} {(i)}_{j} a_{i} t^{i - j},

which, with the help of elementary relations involving factorials and after re-arranging the indices, can be rewritten as

(78)#

{(p_{n} (t))}^{[j]} = \sum_{i = 0}^{n - j} (\binom{i + j}{j}) a_{i + j} t^{i} .

Notes on automatic differentiation

Contents

Notes on automatic differentiation#

Preliminaries#

Basic arithmetic#

Addition and subtraction#

Multiplication#

Division#

Squaring#

Square root#

Exponentiation#

Exponentials#

Natural exponential#

Standard logistic function#

Logarithms#

Natural logarithm#

Trigonometric functions#

Tangent#

Inverse trigonometric functions#

Inverse sine#

Inverse cosine#

Inverse tangent#

Two-argument inverse tangent#

Hyperbolic functions#

Hyperbolic sine#

Hyperbolic cosine#

Hyperbolic tangent#

Inverse hyperbolic functions#

Inverse hyperbolic sine#

Inverse hyperbolic cosine#

Inverse hyperbolic tangent#

Special functions#

Error function#

Celestial mechanics#

Kepler’s eccentric anomaly#

Eccentric longitude#

Time functions#

Time polynomials#