Neural ODEs - II

We here implement, in heyoka, a standard traning pipeline for NeuralODE replicating this torcheqdiff demo.

We will be using the very same parameters and dynamics, of the original demo but only use a simple fixed learning rate stochastic gradient descent. Note that it is also possible to use more advanced optimizers such as those implemented in pytorch while computing the gradients using heyoka, but in this tutorial we keep it simple and just update the gradient naively.

See also:

• Neural ODEs I

• Neural Hamiltonian ODEs

# The main imports
import numpy as np
import time
from itertools import batched
from copy import deepcopy

# Import Heyoka
import heyoka as hy

%matplotlib inline
import matplotlib.pyplot as plt

Generating the ground truth

The dynamics we consider is a simple third order system:

\[\begin{split} \left\{ \begin{array}{l} \dot x = -0.1 x^3 - 2y^3 \\ \dot y = 2 x^3 - 0.1y^3 \end{array} \right. \end{split}\]

which we implement in heyoka as follows:

# The symolic variables representing the system state.
x, y = hy.make_vars("x", "y")

# We now assemble the dynamics.
A = np.array([[-0.1, 2.0], [-2.0, -0.1]])
tmp = np.array([[x*x*x, y*y*y]]) @ A
dyn = [(x, tmp[0][0]), (y, tmp[0][1])]

# For convenience we store here the dimension of the state.
dim = len(dyn)

To generate groundtruth observations from this system we will use the following parameters:

# This is the number of observations (i.e. the ground truth).
data_size = 1000
# This is the initial condition used to generate the observations (i.e. the ground truth).
ic = [2.0, 0.0]
# This is the time grid used to generate the observations (i.e. the ground truth).
t_grid = np.linspace(0.0, 25, data_size)

We start by instantiating a heyoka.taylor_adaptive object that will generate our observations. These will be considered as ground truth. Note that in other settings this data may be observed from a real phenomena, rather than generated numerically. In that case there will be noise which is here not modelled.

start_time = time.time()
ta = hy.taylor_adaptive(
    # The ODEs.
    dyn,
    # The initial conditions.
    ic,
    # Do not operate in compact mode.
    compact_mode=False,
    # Define the tolerance
    tol=1e-18,
)
print(
    "--- %s seconds --- to build (jit) the Taylor integrator"
    % (time.time() - start_time)
)

--- 0.07422232627868652 seconds --- to build (jit) the Taylor integrator

We are thus ready to generate the the ground truth using the method propagate_grid method of the Taylor adaptive integrator.

# It is good practice to always set initial conditions explicitly
ta.time = 0
ta.state[:] = ic
# This is our ground truth (gt)
gt = ta.propagate_grid(t_grid)[5]

Let us have a look at the generated observations.

plt.plot(gt[:, 0], gt[:, 1], 'k.')

[<matplotlib.lines.Line2D at 0x7fdcc5ea6e40>]

Instantiating the neural ODE

So far we have only generated the data we want to teach our Neural ODE to replicate. We now need to instantiate the Neural ODE system so that we can make predictions and compute the gradient of our loss. Let us start with instantiating a feed-forward neural network using heyoka.models.ffnn.

# We define as nonlinearity a simple linear layer
linear = lambda inp: inp

# We call the factory to construct a FFNN. Following the torcheqdiff example here we put cubes as inputs:
ffnn = hy.model.ffnn(
    inputs=[x*x*x, y*y*y],
    nn_hidden=[50],
    n_out=2,
    activations=[hy.tanh, linear],
)

We are now able to build a taylor_adaptive_batch numerical integrator which will propagate the dynamics in batches as to leverage SIMD instructions:

\[\begin{split} \left\{ \begin{array}{l} \dot x = \mathcal N_{\theta, x}(x^3, y^3) \\ \dot y = \mathcal N_{\theta, y}(x^3, y^3) \\ \end{array} \right. \end{split}\]

And since we will want to be able to compute the sensitivities of this ODE we are first going to augment such dynamics with the first order variational equations:

# We decide on whether to use single or double precision. In this example it may
# Not be too important, but in other settings it can give a speedup when connected with batch and ensamble propagation.
# The single precision does effect, though, the quality of thegradient direction.
precision = np.double  # np.single
batch_size_simd = hy.recommended_simd_size(fp_type=precision)
print("Batch size for SIMD use in integration: ", batch_size_simd)

# We thus define the neural dynamics
dyn_n = [(x, ffnn[0]), (y, ffnn[1])]

# We augment it with the variational one (only order 1 - i.e. gradient)
var_dyn_n = hy.var_ode_sys(dyn_n, args=hy.var_args.params, order=1)

Batch size for SIMD use in integration:  4

… and instantiate the variational integrator. For convenience we also instantiate the non batched version

# This is the batched version, which we will use for gradient computation
ta_var_b = hy.taylor_adaptive_batch(
    # The ODEs.
    var_dyn_n,
    # The initial conditions.
    np.ones((dim, batch_size_simd), dtype=precision),
    # Operate in compact mode.
    compact_mode=True,
    # Define the tolerance (low tolerance is enough here)
    tol=precision(1e-4),
    # Single precision is enough here
    fp_type=precision,
)

# This is the normal version used for wuick plotting and visualization of a single solution
ta_var = hy.taylor_adaptive(
    # The ODEs.
    var_dyn_n,
    # The initial conditions.
    ic,
    # Operate in compact mode.
    compact_mode=True,
    # Define the tolerance (low tolerance is enough here)
    tol=precision(1e-4),
    # Single precision is enough here
    fp_type=precision,
)

# We store for convenience the initial conditions in a numpy array (including the variational state)
ic_var_b = np.array(ta_var_b.state)
ic_var = np.array(ta_var.state)

Before starting the training, we define the details on the batch we are going to use in the generic step of stochastic gradient descent:

# This is the number of initial conditions to use in one batch (M)
batch_size_ic = 20
# This is the number of observations to predict from each sampled initial condition (T)
batch_size_time = 10
# We check that the observation batch size is a multiple of the simd batch size
print("This must be an integer: ", batch_size_ic/batch_size_simd)

This must be an integer:  5.0

This helper function creates a randomly generated batch from the available observation:

def get_batch(t_grid, gt, batch_size_ic=20, batch_size_time=10):
    # We select the initial conditions from which generate predictions
    s = np.random.choice(
        np.arange(gt.shape[0] - batch_size_time, dtype=np.int64),
        batch_size_ic,
        replace=False,
    )
    batch_ic = gt[s, :]  # (M, D)
    # Assuming uniform grid and a non-autonomous system, all predictions will be made on the same time grid
    batch_t_grid = t_grid[:batch_size_time]  # (T)
    batch_y = np.stack([gt[s + i] for i in range(batch_size_time)])  # (T,M,D)
    return batch_ic, batch_t_grid, batch_y

Let us visualize what a batch looks like. First we generate one:

batch_ic, batch_t_grid, batch_y = get_batch(t_grid, gt, batch_size_ic, batch_size_time)

# We need the tgrid repeated as per heyoka API of the taylor adaptive batch
t_grid_b = np.repeat(batch_t_grid, batch_size_simd).reshape(-1, batch_size_simd)

Then we plot it overlapped to all the observations:

plt.plot(gt[:, 0], gt[:, 1])
for i in range(batch_size_ic):
    plt.plot(batch_y[:, i, 0], batch_y[:, i, 1], "k.")

We initialize the network weights and biases.

# We compute initialization values for weigths / biases
n_pars = len(ta_var_b.pars)
nn_wb = np.random.normal(loc=0, scale=0.1, size=(n_pars,))

And visualize the prediction of the starting random network.

plt.plot(gt[:, 0], gt[:, 1])
for i in range(batch_size_ic):
    plt.plot(batch_y[:, i, 0], batch_y[:, i, 1], "k.")

start_time = time.time()
for batch in batched(batch_ic, batch_size_simd):
    ta_var_b.set_time(precision(0.))
    ta_var_b.state[:,:] = deepcopy(ic_var_b)
    ta_var_b.state[:dim,:] = np.array(batch).T
    ta_var_b.pars[:] = np.tile(nn_wb, (batch_size_simd,1)).T
    sol_b = ta_var_b.propagate_grid(np.array(t_grid_b, dtype=precision))[1]
    for sol in [sol_b[:,:,i] for i in range(batch_size_simd)]:
        plt.plot(sol[:, 0], sol[:, 1], ".")
print(
    "--- %s seconds --- to compute prediction over the whole batch"
    % (time.time() - start_time)
)

--- 0.007964611053466797 seconds --- to compute prediction over the whole batch

Training

To train we need to first compute the gradient of the loss over the batch (M initial conditions and T times). Formally

\[ \mathcal L_{\theta} = \sum_{i=0}^{M}\sum_{j=0}^T (\pmb y_{ij} - \hat {\pmb y}_{ij})\cdot (\pmb y_{ij} - \hat {\pmb y}_{ij}) \]

\[ \frac{\partial L_\theta}{\partial \theta} = 2 \sum_{i=0}^{M}\sum_{j=0}^T (\pmb y_{ij} - \hat {\pmb y}_{ij})\cdot \frac{\partial \pmb y_{ij}}{\partial \theta} \]

This helper function computes the loss and the gradient over one batch. The gradient is computed rearranging the ODE sensitivities computed propagating the variational dynamics:

def loss_and_gradient(nn_wb, batch_ic, batch_y):
    # We reset the batch_loss value
    batch_loss = 0
    # We reset the gradient loss
    grad = np.array([0.0] * ta_var_b.pars.shape[0])
    counter=0
    # We process the batch (of the dataset) in batches for SIMD efficient processing
    for batch_simd in batched(batch_ic, batch_size_simd):
        ta_var_b.set_time(precision(0.))
        ta_var_b.state[:,:] = deepcopy(ic_var_b)
        ta_var_b.state[:dim,:] = np.array(batch_simd).T
        ta_var_b.pars[:] = np.tile(nn_wb, (batch_size_simd,1)).T
        sol_b = ta_var_b.propagate_grid(np.array(t_grid_b, dtype=precision))[1]
        # The numerical integration on the simd_batch is done, we process its result 
        # accumulating loss and gradient on the predictions
        for sol in [sol_b[:,:,i] for i in range(batch_size_simd)]:
            # Here is the term (y-y_hat)
            diff = sol[:, :dim] - batch_y[:, counter, :]
            counter+=1
            # Which is then summed sum sum (y-y_hat).(y-y_hat)
            batch_loss += np.sum(diff**2)
            # And the gradient computed as 2 sum (y-y_hat).dy
            for dy in sol[:, dim:].reshape(batch_y.shape[0], dim, -1):
                grad += 2 * np.sum(diff @ dy, axis=0)
    # We then take the mean over the points used
    batch_loss /= batch_size_ic * batch_size_time
    grad /= batch_size_ic * batch_size_time
    return batch_loss, grad

Using the various blocks defined above the training loop takes the form:

# We reset initialization values for weigths / biases
n_pars = len(ta_var_b.pars)
nn_wb = np.random.normal(loc=0, scale=0.1, size=(n_pars,))

# We loop over the epochs
n_epochs = 3000
start_time = time.time()
for i in range(n_epochs):
    # 1 - generate a batch
    batch_ic, batch_t_grid, batch_y = get_batch(
        t_grid, gt, batch_size_ic=batch_size_ic, batch_size_time=batch_size_time
    )
    # 2 - compute the loss and gradient on the batch
    batch_loss, grad = loss_and_gradient(nn_wb, batch_ic, batch_y)
    print(f"iter: {i}, loss : {batch_loss:.3e}", end="\r")
    # 3 - update the weights and biases (learning rate here is fixed, we use a naive update rule which in this case works)
    nn_wb = nn_wb - 0.1 * grad
print(f"iter: {i}, loss : {batch_loss:.3e}", end="\n")
print(
    "--- %s seconds --- to perform %i epochs"
    % (time.time() - start_time, n_epochs)
)

iter: 0, loss : 6.678e-02
iter: 1, loss : 6.151e-02
iter: 2, loss : 1.305e-01
iter: 3, loss : 1.324e-01
iter: 4, loss : 1.742e-01
iter: 5, loss : 4.954e-02
iter: 6, loss : 1.397e-02
iter: 7, loss : 1.696e-01
iter: 8, loss : 3.531e-02
iter: 9, loss : 1.447e-01
iter: 10, loss : 3.426e-02
iter: 11, loss : 5.243e-02
iter: 12, loss : 2.239e-02
iter: 13, loss : 8.538e-02
iter: 14, loss : 1.094e-02
iter: 15, loss : 7.628e-02
iter: 16, loss : 1.199e-01
iter: 17, loss : 9.249e-02
iter: 18, loss : 8.393e-03
iter: 19, loss : 9.384e-02
iter: 20, loss : 5.339e-02
iter: 21, loss : 6.150e-02
iter: 22, loss : 1.600e-02
iter: 23, loss : 1.319e-01

iter: 24, loss : 1.404e-01
iter: 25, loss : 1.710e-02
iter: 26, loss : 3.647e-02
iter: 27, loss : 3.654e-03
iter: 28, loss : 1.644e-02
iter: 29, loss : 8.769e-02
iter: 30, loss : 5.403e-03
iter: 31, loss : 4.024e-02
iter: 32, loss : 1.251e-02
iter: 33, loss : 4.089e-02
iter: 34, loss : 2.726e-02
iter: 35, loss : 7.505e-02
iter: 36, loss : 4.793e-02
iter: 37, loss : 5.925e-03
iter: 38, loss : 6.319e-02
iter: 39, loss : 6.831e-03
iter: 40, loss : 8.957e-03
iter: 41, loss : 2.998e-02
iter: 42, loss : 9.031e-02
iter: 43, loss : 2.573e-02
iter: 44, loss : 6.715e-02
iter: 45, loss : 4.136e-02
iter: 46, loss : 1.772e-03
iter: 47, loss : 1.008e-01
iter: 48, loss : 1.599e-02

iter: 49, loss : 4.543e-03
iter: 50, loss : 2.154e-02
iter: 51, loss : 4.726e-02
iter: 52, loss : 1.280e-03
iter: 53, loss : 8.777e-03
iter: 54, loss : 3.185e-02
iter: 55, loss : 4.867e-03
iter: 56, loss : 2.541e-02
iter: 57, loss : 5.390e-02
iter: 58, loss : 7.247e-03
iter: 59, loss : 2.135e-03
iter: 60, loss : 5.285e-03
iter: 61, loss : 6.612e-03
iter: 62, loss : 1.264e-02
iter: 63, loss : 1.287e-03
iter: 64, loss : 9.859e-03
iter: 65, loss : 1.564e-02
iter: 66, loss : 2.565e-02
iter: 67, loss : 1.423e-03
iter: 68, loss : 7.551e-03
iter: 69, loss : 1.180e-02

iter: 70, loss : 1.417e-03
iter: 71, loss : 1.255e-02
iter: 72, loss : 3.469e-02
iter: 73, loss : 6.139e-03
iter: 74, loss : 6.514e-03
iter: 75, loss : 1.913e-02
iter: 76, loss : 1.823e-02
iter: 77, loss : 2.450e-03
iter: 78, loss : 2.843e-03
iter: 79, loss : 1.561e-04
iter: 80, loss : 7.859e-04
iter: 81, loss : 4.084e-04
iter: 82, loss : 2.115e-03
iter: 83, loss : 1.810e-02
iter: 84, loss : 6.714e-03
iter: 85, loss : 1.698e-03
iter: 86, loss : 5.234e-05
iter: 87, loss : 3.861e-03
iter: 88, loss : 3.340e-03
iter: 89, loss : 1.184e-03
iter: 90, loss : 2.769e-03
iter: 91, loss : 4.675e-02

iter: 92, loss : 1.934e-03
iter: 93, loss : 6.614e-03
iter: 94, loss : 1.811e-02
iter: 95, loss : 1.096e-02
iter: 96, loss : 3.542e-02
iter: 97, loss : 1.225e-02
iter: 98, loss : 6.870e-04
iter: 99, loss : 2.493e-04
iter: 100, loss : 1.610e-03
iter: 101, loss : 1.357e-02
iter: 102, loss : 6.580e-03
iter: 103, loss : 6.392e-04
iter: 104, loss : 9.254e-03
iter: 105, loss : 5.465e-04
iter: 106, loss : 1.035e-02
iter: 107, loss : 7.401e-03
iter: 108, loss : 3.786e-04
iter: 109, loss : 7.914e-04
iter: 110, loss : 2.733e-04

iter: 111, loss : 1.201e-02
iter: 112, loss : 5.041e-04
iter: 113, loss : 1.876e-02
iter: 114, loss : 6.753e-03
iter: 115, loss : 4.516e-03
iter: 116, loss : 7.926e-04
iter: 117, loss : 1.060e-02
iter: 118, loss : 1.346e-03
iter: 119, loss : 5.908e-03
iter: 120, loss : 2.419e-02
iter: 121, loss : 6.222e-03
iter: 122, loss : 7.529e-03
iter: 123, loss : 4.507e-03
iter: 124, loss : 1.127e-03
iter: 125, loss : 1.522e-03
iter: 126, loss : 5.635e-03
iter: 127, loss : 7.907e-04
iter: 128, loss : 1.866e-03
iter: 129, loss : 1.133e-02
iter: 130, loss : 4.595e-03

iter: 131, loss : 5.152e-04
iter: 132, loss : 7.673e-03
iter: 133, loss : 1.483e-02
iter: 134, loss : 1.808e-03
iter: 135, loss : 1.636e-03
iter: 136, loss : 5.460e-03
iter: 137, loss : 1.142e-03
iter: 138, loss : 9.275e-04
iter: 139, loss : 6.820e-03
iter: 140, loss : 1.724e-03
iter: 141, loss : 8.209e-03
iter: 142, loss : 1.420e-03
iter: 143, loss : 7.568e-03
iter: 144, loss : 8.288e-03
iter: 145, loss : 1.889e-03
iter: 146, loss : 1.703e-02
iter: 147, loss : 1.565e-03
iter: 148, loss : 1.333e-03
iter: 149, loss : 3.553e-03

iter: 150, loss : 1.950e-02
iter: 151, loss : 1.826e-03
iter: 152, loss : 8.297e-04
iter: 153, loss : 1.385e-03
iter: 154, loss : 9.560e-04
iter: 155, loss : 2.081e-03
iter: 156, loss : 1.002e-03
iter: 157, loss : 7.839e-04
iter: 158, loss : 7.211e-04
iter: 159, loss : 7.069e-04
iter: 160, loss : 3.314e-03
iter: 161, loss : 3.469e-03
iter: 162, loss : 3.022e-03
iter: 163, loss : 1.150e-03
iter: 164, loss : 1.151e-03
iter: 165, loss : 5.526e-04
iter: 166, loss : 5.816e-04
iter: 167, loss : 4.251e-03
iter: 168, loss : 1.509e-03
iter: 169, loss : 1.040e-03

iter: 170, loss : 1.157e-02
iter: 171, loss : 1.031e-03
iter: 172, loss : 2.488e-03
iter: 173, loss : 5.064e-04
iter: 174, loss : 2.548e-03
iter: 175, loss : 6.689e-03
iter: 176, loss : 1.754e-03
iter: 177, loss : 1.148e-03
iter: 178, loss : 1.976e-03
iter: 179, loss : 1.293e-03
iter: 180, loss : 7.870e-04
iter: 181, loss : 8.069e-03
iter: 182, loss : 8.175e-03
iter: 183, loss : 1.374e-03
iter: 184, loss : 2.513e-03
iter: 185, loss : 1.259e-02
iter: 186, loss : 4.710e-03
iter: 187, loss : 1.491e-03
iter: 188, loss : 8.278e-03

iter: 189, loss : 1.144e-03
iter: 190, loss : 3.210e-03
iter: 191, loss : 9.671e-04
iter: 192, loss : 2.652e-03
iter: 193, loss : 7.642e-03
iter: 194, loss : 6.889e-04
iter: 195, loss : 7.450e-04
iter: 196, loss : 1.488e-03
iter: 197, loss : 7.107e-04
iter: 198, loss : 4.850e-04
iter: 199, loss : 1.293e-03
iter: 200, loss : 5.543e-04
iter: 201, loss : 3.388e-03
iter: 202, loss : 1.026e-03
iter: 203, loss : 2.686e-03
iter: 204, loss : 2.729e-03
iter: 205, loss : 6.541e-03
iter: 206, loss : 1.666e-03
iter: 207, loss : 1.094e-03
iter: 208, loss : 1.653e-03

iter: 209, loss : 9.834e-03
iter: 210, loss : 7.158e-04
iter: 211, loss : 1.533e-03
iter: 212, loss : 1.110e-03
iter: 213, loss : 8.124e-04
iter: 214, loss : 1.467e-03
iter: 215, loss : 1.219e-02
iter: 216, loss : 7.194e-04
iter: 217, loss : 7.235e-04
iter: 218, loss : 4.769e-04
iter: 219, loss : 1.331e-02
iter: 220, loss : 5.633e-04
iter: 221, loss : 1.073e-03
iter: 222, loss : 4.434e-03
iter: 223, loss : 9.186e-03
iter: 224, loss : 4.329e-04
iter: 225, loss : 8.188e-03
iter: 226, loss : 1.244e-03
iter: 227, loss : 8.746e-04
iter: 228, loss : 8.725e-04

iter: 229, loss : 1.055e-03
iter: 230, loss : 3.172e-03
iter: 231, loss : 3.098e-03
iter: 232, loss : 1.908e-03
iter: 233, loss : 6.639e-04
iter: 234, loss : 9.945e-04
iter: 235, loss : 1.514e-03
iter: 236, loss : 8.964e-04
iter: 237, loss : 7.318e-03
iter: 238, loss : 6.737e-03
iter: 239, loss : 3.419e-04
iter: 240, loss : 6.309e-04
iter: 241, loss : 8.164e-03
iter: 242, loss : 1.107e-03
iter: 243, loss : 7.875e-04
iter: 244, loss : 1.165e-03
iter: 245, loss : 8.270e-04
iter: 246, loss : 7.452e-04
iter: 247, loss : 1.969e-02
iter: 248, loss : 9.485e-04

iter: 249, loss : 4.359e-03
iter: 250, loss : 7.743e-04
iter: 251, loss : 6.805e-03
iter: 252, loss : 6.125e-03
iter: 253, loss : 5.612e-04
iter: 254, loss : 2.459e-03
iter: 255, loss : 3.974e-03
iter: 256, loss : 2.186e-03
iter: 257, loss : 1.872e-03
iter: 258, loss : 2.853e-03
iter: 259, loss : 8.978e-04
iter: 260, loss : 1.158e-02
iter: 261, loss : 8.542e-04
iter: 262, loss : 2.231e-03
iter: 263, loss : 4.586e-03
iter: 264, loss : 4.707e-03
iter: 265, loss : 6.411e-04
iter: 266, loss : 3.354e-03
iter: 267, loss : 1.447e-03
iter: 268, loss : 1.277e-03

iter: 269, loss : 2.244e-03
iter: 270, loss : 7.160e-04
iter: 271, loss : 1.916e-03
iter: 272, loss : 1.349e-03
iter: 273, loss : 1.227e-03
iter: 274, loss : 1.292e-03
iter: 275, loss : 7.272e-04
iter: 276, loss : 6.132e-04
iter: 277, loss : 5.603e-04
iter: 278, loss : 8.242e-04
iter: 279, loss : 5.409e-04
iter: 280, loss : 7.542e-04
iter: 281, loss : 2.868e-04
iter: 282, loss : 2.963e-04
iter: 283, loss : 8.332e-04
iter: 284, loss : 6.122e-04
iter: 285, loss : 5.559e-04
iter: 286, loss : 2.703e-03
iter: 287, loss : 6.680e-04
iter: 288, loss : 7.588e-04

iter: 289, loss : 2.463e-03
iter: 290, loss : 5.047e-04
iter: 291, loss : 3.917e-04
iter: 292, loss : 6.194e-04
iter: 293, loss : 1.296e-02
iter: 294, loss : 9.620e-04
iter: 295, loss : 5.866e-03
iter: 296, loss : 9.446e-04
iter: 297, loss : 3.880e-03
iter: 298, loss : 5.454e-03
iter: 299, loss : 5.613e-04
iter: 300, loss : 9.234e-04
iter: 301, loss : 7.277e-04
iter: 302, loss : 2.689e-04
iter: 303, loss : 8.203e-03
iter: 304, loss : 6.007e-04
iter: 305, loss : 9.998e-04
iter: 306, loss : 3.777e-04
iter: 307, loss : 7.774e-04
iter: 308, loss : 7.735e-04

iter: 309, loss : 1.784e-02
iter: 310, loss : 6.551e-04
iter: 311, loss : 3.936e-04
iter: 312, loss : 4.730e-03
iter: 313, loss : 6.659e-04
iter: 314, loss : 1.763e-03
iter: 315, loss : 4.053e-03
iter: 316, loss : 2.208e-03
iter: 317, loss : 7.395e-04
iter: 318, loss : 9.317e-04
iter: 319, loss : 5.442e-04
iter: 320, loss : 1.901e-04
iter: 321, loss : 1.290e-02
iter: 322, loss : 9.657e-04
iter: 323, loss : 2.445e-03
iter: 324, loss : 1.033e-03
iter: 325, loss : 4.146e-04
iter: 326, loss : 3.761e-04
iter: 327, loss : 2.902e-04
iter: 328, loss : 7.813e-04

iter: 329, loss : 6.461e-03
iter: 330, loss : 5.196e-04
iter: 331, loss : 5.659e-04
iter: 332, loss : 6.274e-04
iter: 333, loss : 4.296e-03
iter: 334, loss : 6.831e-03
iter: 335, loss : 7.075e-04
iter: 336, loss : 7.153e-04
iter: 337, loss : 7.788e-04
iter: 338, loss : 5.278e-04
iter: 339, loss : 7.096e-04
iter: 340, loss : 1.461e-03
iter: 341, loss : 1.516e-02
iter: 342, loss : 5.078e-03
iter: 343, loss : 7.199e-03
iter: 344, loss : 1.349e-03
iter: 345, loss : 5.106e-04
iter: 346, loss : 2.306e-03
iter: 347, loss : 3.093e-03
iter: 348, loss : 3.388e-04

iter: 349, loss : 3.192e-03
iter: 350, loss : 5.691e-04
iter: 351, loss : 4.684e-03
iter: 352, loss : 8.023e-04
iter: 353, loss : 1.037e-03
iter: 354, loss : 8.778e-03
iter: 355, loss : 3.171e-04
iter: 356, loss : 6.986e-04
iter: 357, loss : 2.366e-03
iter: 358, loss : 8.257e-04
iter: 359, loss : 6.906e-04
iter: 360, loss : 7.471e-04
iter: 361, loss : 4.390e-04
iter: 362, loss : 8.745e-04
iter: 363, loss : 4.667e-04
iter: 364, loss : 3.636e-04
iter: 365, loss : 8.267e-04
iter: 366, loss : 3.638e-04
iter: 367, loss : 1.753e-03
iter: 368, loss : 6.468e-04

iter: 369, loss : 4.556e-04
iter: 370, loss : 2.169e-03
iter: 371, loss : 4.051e-04
iter: 372, loss : 5.707e-04
iter: 373, loss : 2.647e-04
iter: 374, loss : 4.226e-03
iter: 375, loss : 4.253e-04
iter: 376, loss : 4.553e-04
iter: 377, loss : 4.288e-04
iter: 378, loss : 4.741e-04
iter: 379, loss : 5.368e-04
iter: 380, loss : 2.432e-03
iter: 381, loss : 6.981e-04
iter: 382, loss : 1.359e-02
iter: 383, loss : 4.553e-03
iter: 384, loss : 9.524e-04
iter: 385, loss : 3.674e-04
iter: 386, loss : 7.222e-04
iter: 387, loss : 3.529e-04
iter: 388, loss : 9.800e-04
iter: 389, loss : 8.164e-04

iter: 390, loss : 3.959e-04
iter: 391, loss : 9.755e-03
iter: 392, loss : 5.220e-04
iter: 393, loss : 4.195e-03
iter: 394, loss : 4.782e-04
iter: 395, loss : 5.912e-04
iter: 396, loss : 2.761e-03
iter: 397, loss : 6.489e-04
iter: 398, loss : 7.922e-04
iter: 399, loss : 1.064e-02
iter: 400, loss : 6.242e-03
iter: 401, loss : 8.058e-04
iter: 402, loss : 4.670e-04
iter: 403, loss : 2.614e-03
iter: 404, loss : 6.577e-04
iter: 405, loss : 5.690e-04
iter: 406, loss : 3.744e-04
iter: 407, loss : 4.942e-04
iter: 408, loss : 5.832e-04
iter: 409, loss : 6.939e-04

iter: 410, loss : 4.229e-04
iter: 411, loss : 2.207e-03
iter: 412, loss : 2.882e-04
iter: 413, loss : 7.631e-04
iter: 414, loss : 3.319e-04
iter: 415, loss : 3.776e-04
iter: 416, loss : 1.943e-03
iter: 417, loss : 4.682e-03
iter: 418, loss : 3.829e-04
iter: 419, loss : 2.107e-03
iter: 420, loss : 6.916e-04
iter: 421, loss : 3.089e-04
iter: 422, loss : 5.674e-04
iter: 423, loss : 6.940e-04
iter: 424, loss : 2.076e-04
iter: 425, loss : 2.681e-04
iter: 426, loss : 4.571e-04
iter: 427, loss : 2.499e-04
iter: 428, loss : 1.054e-02
iter: 429, loss : 5.138e-04
iter: 430, loss : 3.125e-03

iter: 431, loss : 6.631e-04
iter: 432, loss : 6.546e-04
iter: 433, loss : 2.966e-04
iter: 434, loss : 5.142e-04
iter: 435, loss : 6.928e-04
iter: 436, loss : 2.219e-03
iter: 437, loss : 4.755e-04
iter: 438, loss : 2.941e-04
iter: 439, loss : 1.693e-03
iter: 440, loss : 4.918e-04
iter: 441, loss : 6.198e-04
iter: 442, loss : 3.395e-04
iter: 443, loss : 2.057e-04
iter: 444, loss : 5.347e-04
iter: 445, loss : 1.684e-03
iter: 446, loss : 2.027e-03
iter: 447, loss : 5.961e-04
iter: 448, loss : 2.705e-04
iter: 449, loss : 6.096e-03
iter: 450, loss : 1.216e-03

iter: 451, loss : 2.408e-04
iter: 452, loss : 8.758e-04
iter: 453, loss : 3.490e-04
iter: 454, loss : 3.564e-03
iter: 455, loss : 4.994e-04
iter: 456, loss : 3.629e-03
iter: 457, loss : 8.944e-04
iter: 458, loss : 3.611e-03
iter: 459, loss : 4.998e-04
iter: 460, loss : 8.955e-04
iter: 461, loss : 6.534e-04
iter: 462, loss : 5.817e-04
iter: 463, loss : 3.996e-04
iter: 464, loss : 5.049e-04
iter: 465, loss : 1.757e-03
iter: 466, loss : 2.702e-04
iter: 467, loss : 4.512e-03
iter: 468, loss : 3.738e-03
iter: 469, loss : 4.621e-04
iter: 470, loss : 4.244e-03

iter: 471, loss : 1.139e-03
iter: 472, loss : 2.992e-04
iter: 473, loss : 1.685e-04
iter: 474, loss : 1.590e-03
iter: 475, loss : 6.816e-04
iter: 476, loss : 2.958e-03
iter: 477, loss : 2.172e-03
iter: 478, loss : 8.485e-04
iter: 479, loss : 4.499e-04
iter: 480, loss : 5.228e-04
iter: 481, loss : 3.200e-04
iter: 482, loss : 4.393e-03
iter: 483, loss : 2.602e-03
iter: 484, loss : 8.281e-04
iter: 485, loss : 3.364e-04
iter: 486, loss : 5.785e-04
iter: 487, loss : 2.903e-04
iter: 488, loss : 1.622e-03
iter: 489, loss : 5.753e-04
iter: 490, loss : 3.905e-04

iter: 491, loss : 2.776e-04
iter: 492, loss : 2.345e-04
iter: 493, loss : 6.354e-04
iter: 494, loss : 2.861e-04
iter: 495, loss : 2.719e-03
iter: 496, loss : 4.839e-04
iter: 497, loss : 2.657e-03
iter: 498, loss : 2.898e-04
iter: 499, loss : 2.344e-04
iter: 500, loss : 7.420e-04
iter: 501, loss : 4.035e-04
iter: 502, loss : 2.694e-04
iter: 503, loss : 3.400e-04
iter: 504, loss : 3.962e-04
iter: 505, loss : 2.143e-04
iter: 506, loss : 4.647e-04
iter: 507, loss : 3.986e-04
iter: 508, loss : 1.356e-02
iter: 509, loss : 4.726e-04
iter: 510, loss : 2.266e-03
iter: 511, loss : 2.183e-04

iter: 512, loss : 1.390e-04
iter: 513, loss : 6.595e-04
iter: 514, loss : 5.023e-04
iter: 515, loss : 1.588e-03
iter: 516, loss : 5.377e-04
iter: 517, loss : 8.276e-03
iter: 518, loss : 6.994e-04
iter: 519, loss : 5.627e-04
iter: 520, loss : 4.526e-04
iter: 521, loss : 2.002e-04
iter: 522, loss : 3.560e-04
iter: 523, loss : 1.583e-03
iter: 524, loss : 4.728e-04
iter: 525, loss : 7.289e-04
iter: 526, loss : 3.013e-04
iter: 527, loss : 3.520e-03
iter: 528, loss : 1.236e-04
iter: 529, loss : 2.565e-03
iter: 530, loss : 4.117e-04
iter: 531, loss : 1.830e-03

iter: 532, loss : 2.613e-04
iter: 533, loss : 1.239e-03
iter: 534, loss : 6.692e-04
iter: 535, loss : 6.553e-04
iter: 536, loss : 1.290e-03
iter: 537, loss : 4.169e-03
iter: 538, loss : 4.640e-04
iter: 539, loss : 1.741e-04
iter: 540, loss : 2.964e-04
iter: 541, loss : 2.471e-03
iter: 542, loss : 8.423e-04
iter: 543, loss : 4.210e-03
iter: 544, loss : 2.955e-03
iter: 545, loss : 6.562e-04
iter: 546, loss : 1.063e-03
iter: 547, loss : 6.220e-04
iter: 548, loss : 5.204e-04
iter: 549, loss : 2.818e-03
iter: 550, loss : 3.959e-04
iter: 551, loss : 5.077e-04
iter: 552, loss : 1.915e-04

iter: 553, loss : 6.442e-04
iter: 554, loss : 9.816e-04
iter: 555, loss : 3.885e-04
iter: 556, loss : 3.964e-04
iter: 557, loss : 3.530e-04
iter: 558, loss : 9.606e-04
iter: 559, loss : 1.715e-04
iter: 560, loss : 6.490e-04
iter: 561, loss : 7.638e-04
iter: 562, loss : 2.748e-03
iter: 563, loss : 3.986e-04
iter: 564, loss : 2.914e-03
iter: 565, loss : 2.798e-04
iter: 566, loss : 9.625e-04
iter: 567, loss : 2.201e-04
iter: 568, loss : 1.635e-03
iter: 569, loss : 1.651e-04
iter: 570, loss : 7.894e-03
iter: 571, loss : 7.198e-04
iter: 572, loss : 5.470e-04

iter: 573, loss : 5.906e-04
iter: 574, loss : 5.273e-04
iter: 575, loss : 6.160e-04
iter: 576, loss : 2.070e-03
iter: 577, loss : 1.360e-03
iter: 578, loss : 2.716e-04
iter: 579, loss : 8.371e-04
iter: 580, loss : 3.993e-04
iter: 581, loss : 1.833e-03
iter: 582, loss : 2.564e-04
iter: 583, loss : 3.813e-03
iter: 584, loss : 3.745e-04
iter: 585, loss : 3.548e-04
iter: 586, loss : 6.564e-04
iter: 587, loss : 5.512e-04
iter: 588, loss : 3.446e-04
iter: 589, loss : 6.048e-04
iter: 590, loss : 2.139e-04
iter: 591, loss : 2.277e-03
iter: 592, loss : 4.497e-04
iter: 593, loss : 2.483e-04

iter: 594, loss : 2.080e-04
iter: 595, loss : 2.996e-04
iter: 596, loss : 4.691e-03
iter: 597, loss : 1.350e-03
iter: 598, loss : 4.840e-04
iter: 599, loss : 2.324e-04
iter: 600, loss : 1.562e-03
iter: 601, loss : 1.117e-03
iter: 602, loss : 4.876e-03
iter: 603, loss : 1.161e-03
iter: 604, loss : 3.528e-04
iter: 605, loss : 2.578e-04
iter: 606, loss : 1.056e-02
iter: 607, loss : 3.055e-03
iter: 608, loss : 8.016e-04
iter: 609, loss : 4.838e-03
iter: 610, loss : 5.055e-04
iter: 611, loss : 9.427e-04
iter: 612, loss : 8.404e-04
iter: 613, loss : 1.879e-04

iter: 614, loss : 2.261e-04
iter: 615, loss : 1.611e-03
iter: 616, loss : 2.767e-04
iter: 617, loss : 8.669e-03
iter: 618, loss : 4.737e-04
iter: 619, loss : 6.599e-04
iter: 620, loss : 4.050e-03
iter: 621, loss : 1.451e-04
iter: 622, loss : 4.688e-04
iter: 623, loss : 2.235e-03
iter: 624, loss : 3.040e-04
iter: 625, loss : 9.429e-04
iter: 626, loss : 8.785e-04
iter: 627, loss : 4.445e-03
iter: 628, loss : 7.460e-04
iter: 629, loss : 4.070e-04
iter: 630, loss : 2.548e-04
iter: 631, loss : 1.718e-04
iter: 632, loss : 8.148e-03

iter: 633, loss : 1.044e-03
iter: 634, loss : 8.811e-04
iter: 635, loss : 1.150e-03
iter: 636, loss : 5.137e-04
iter: 637, loss : 2.150e-04
iter: 638, loss : 1.247e-03
iter: 639, loss : 7.566e-04
iter: 640, loss : 7.370e-04
iter: 641, loss : 1.373e-04
iter: 642, loss : 7.744e-03
iter: 643, loss : 3.136e-03
iter: 644, loss : 6.879e-04
iter: 645, loss : 3.129e-04
iter: 646, loss : 4.852e-04
iter: 647, loss : 5.847e-04
iter: 648, loss : 3.408e-04
iter: 649, loss : 4.384e-04
iter: 650, loss : 5.222e-04
iter: 651, loss : 4.020e-04
iter: 652, loss : 4.531e-04
iter: 653, loss : 3.550e-04

iter: 654, loss : 5.227e-03
iter: 655, loss : 3.169e-04
iter: 656, loss : 1.007e-03
iter: 657, loss : 1.510e-03
iter: 658, loss : 2.298e-03
iter: 659, loss : 8.052e-04
iter: 660, loss : 2.770e-03
iter: 661, loss : 4.170e-04
iter: 662, loss : 4.694e-04
iter: 663, loss : 2.936e-03
iter: 664, loss : 2.412e-04
iter: 665, loss : 5.668e-04
iter: 666, loss : 3.616e-03
iter: 667, loss : 1.188e-03
iter: 668, loss : 4.491e-04
iter: 669, loss : 4.793e-03
iter: 670, loss : 4.716e-04
iter: 671, loss : 8.532e-04

iter: 672, loss : 9.410e-04
iter: 673, loss : 3.965e-04
iter: 674, loss : 2.699e-03
iter: 675, loss : 3.732e-04
iter: 676, loss : 7.568e-04
iter: 677, loss : 1.199e-03
iter: 678, loss : 2.676e-04
iter: 679, loss : 6.395e-04
iter: 680, loss : 7.330e-04
iter: 681, loss : 1.294e-04
iter: 682, loss : 2.231e-03
iter: 683, loss : 4.822e-04
iter: 684, loss : 3.401e-04
iter: 685, loss : 4.703e-04
iter: 686, loss : 1.745e-03
iter: 687, loss : 8.446e-03
iter: 688, loss : 8.265e-04
iter: 689, loss : 4.660e-04
iter: 690, loss : 1.830e-03
iter: 691, loss : 7.549e-03
iter: 692, loss : 4.935e-04
iter: 693, loss : 4.611e-04

iter: 694, loss : 2.123e-03
iter: 695, loss : 5.652e-04
iter: 696, loss : 4.220e-03
iter: 697, loss : 8.018e-04
iter: 698, loss : 1.690e-03
iter: 699, loss : 4.691e-04
iter: 700, loss : 4.081e-03
iter: 701, loss : 2.919e-04
iter: 702, loss : 6.785e-04
iter: 703, loss : 9.803e-04
iter: 704, loss : 6.277e-04
iter: 705, loss : 5.555e-04
iter: 706, loss : 1.170e-03
iter: 707, loss : 3.295e-04
iter: 708, loss : 6.863e-04
iter: 709, loss : 1.021e-03
iter: 710, loss : 5.133e-04
iter: 711, loss : 4.233e-04

iter: 712, loss : 4.960e-04
iter: 713, loss : 7.534e-03
iter: 714, loss : 3.169e-03
iter: 715, loss : 4.222e-04
iter: 716, loss : 8.677e-04
iter: 717, loss : 6.108e-04
iter: 718, loss : 4.421e-04
iter: 719, loss : 5.487e-04
iter: 720, loss : 3.387e-04
iter: 721, loss : 1.636e-04
iter: 722, loss : 1.790e-04
iter: 723, loss : 8.721e-03
iter: 724, loss : 5.623e-04
iter: 725, loss : 2.310e-04
iter: 726, loss : 4.905e-04
iter: 727, loss : 5.096e-04
iter: 728, loss : 4.022e-04
iter: 729, loss : 3.051e-04
iter: 730, loss : 9.019e-04
iter: 731, loss : 1.349e-03
iter: 732, loss : 2.397e-03
iter: 733, loss : 6.207e-04

iter: 734, loss : 2.857e-04
iter: 735, loss : 1.850e-04
iter: 736, loss : 7.015e-03
iter: 737, loss : 7.934e-04
iter: 738, loss : 3.290e-04
iter: 739, loss : 3.265e-04
iter: 740, loss : 1.472e-03
iter: 741, loss : 2.129e-03
iter: 742, loss : 3.184e-04
iter: 743, loss : 3.386e-04
iter: 744, loss : 2.096e-03
iter: 745, loss : 3.410e-04
iter: 746, loss : 1.009e-03
iter: 747, loss : 6.181e-04
iter: 748, loss : 6.357e-03
iter: 749, loss : 1.759e-03
iter: 750, loss : 1.964e-04
iter: 751, loss : 2.417e-04

iter: 752, loss : 8.221e-03
iter: 753, loss : 5.751e-03
iter: 754, loss : 1.705e-03
iter: 755, loss : 5.685e-04
iter: 756, loss : 5.130e-03
iter: 757, loss : 3.821e-04
iter: 758, loss : 7.337e-04
iter: 759, loss : 9.680e-04
iter: 760, loss : 4.613e-04
iter: 761, loss : 7.708e-04
iter: 762, loss : 6.513e-03
iter: 763, loss : 6.286e-03
iter: 764, loss : 5.812e-04
iter: 765, loss : 6.921e-04
iter: 766, loss : 5.862e-04
iter: 767, loss : 1.060e-03
iter: 768, loss : 4.807e-04
iter: 769, loss : 1.885e-03
iter: 770, loss : 4.502e-04
iter: 771, loss : 1.550e-03
iter: 772, loss : 2.366e-03

iter: 773, loss : 1.966e-03
iter: 774, loss : 1.242e-03
iter: 775, loss : 8.153e-04
iter: 776, loss : 6.461e-04
iter: 777, loss : 2.037e-04
iter: 778, loss : 2.656e-04
iter: 779, loss : 1.812e-03
iter: 780, loss : 5.854e-04
iter: 781, loss : 4.873e-04
iter: 782, loss : 2.111e-03
iter: 783, loss : 4.369e-04
iter: 784, loss : 1.045e-03
iter: 785, loss : 5.332e-04
iter: 786, loss : 2.185e-04
iter: 787, loss : 1.319e-03
iter: 788, loss : 5.377e-04
iter: 789, loss : 4.767e-04
iter: 790, loss : 5.170e-04

iter: 791, loss : 3.967e-04
iter: 792, loss : 1.406e-04
iter: 793, loss : 2.585e-04
iter: 794, loss : 3.127e-04
iter: 795, loss : 1.997e-04
iter: 796, loss : 2.566e-04
iter: 797, loss : 4.736e-03
iter: 798, loss : 1.148e-04
iter: 799, loss : 3.050e-04
iter: 800, loss : 1.322e-03
iter: 801, loss : 3.006e-04
iter: 802, loss : 2.543e-04
iter: 803, loss : 8.951e-04
iter: 804, loss : 3.888e-04
iter: 805, loss : 5.501e-03
iter: 806, loss : 2.025e-04
iter: 807, loss : 2.625e-04
iter: 808, loss : 3.013e-04
iter: 809, loss : 2.601e-03
iter: 810, loss : 2.677e-04
iter: 811, loss : 2.834e-04
iter: 812, loss : 2.507e-04

iter: 813, loss : 2.823e-04
iter: 814, loss : 2.463e-04
iter: 815, loss : 4.745e-04
iter: 816, loss : 2.009e-03
iter: 817, loss : 4.634e-04
iter: 818, loss : 2.306e-04
iter: 819, loss : 2.500e-04
iter: 820, loss : 1.151e-04
iter: 821, loss : 8.063e-05
iter: 822, loss : 2.478e-04
iter: 823, loss : 3.011e-04
iter: 824, loss : 1.283e-03
iter: 825, loss : 1.797e-03
iter: 826, loss : 2.303e-04
iter: 827, loss : 1.810e-04
iter: 828, loss : 1.411e-03
iter: 829, loss : 1.576e-04
iter: 830, loss : 4.594e-04
iter: 831, loss : 2.427e-04

iter: 832, loss : 3.661e-04
iter: 833, loss : 1.681e-04
iter: 834, loss : 7.528e-03
iter: 835, loss : 3.330e-04
iter: 836, loss : 1.377e-04
iter: 837, loss : 2.361e-04
iter: 838, loss : 2.555e-03
iter: 839, loss : 2.465e-04
iter: 840, loss : 2.941e-04
iter: 841, loss : 3.823e-04
iter: 842, loss : 1.693e-04
iter: 843, loss : 1.896e-04
iter: 844, loss : 1.108e-04
iter: 845, loss : 1.051e-03
iter: 846, loss : 1.156e-04
iter: 847, loss : 3.126e-04
iter: 848, loss : 7.749e-03
iter: 849, loss : 2.471e-04
iter: 850, loss : 1.816e-03
iter: 851, loss : 2.032e-04
iter: 852, loss : 2.474e-04

iter: 853, loss : 2.427e-04
iter: 854, loss : 1.098e-04
iter: 855, loss : 2.952e-04
iter: 856, loss : 1.754e-04
iter: 857, loss : 2.584e-04
iter: 858, loss : 1.316e-03
iter: 859, loss : 5.442e-03
iter: 860, loss : 3.195e-04
iter: 861, loss : 4.892e-03
iter: 862, loss : 3.404e-04
iter: 863, loss : 7.123e-04
iter: 864, loss : 1.345e-03
iter: 865, loss : 2.468e-03
iter: 866, loss : 1.481e-03
iter: 867, loss : 5.348e-04
iter: 868, loss : 6.874e-04
iter: 869, loss : 3.176e-04
iter: 870, loss : 1.804e-04
iter: 871, loss : 3.948e-04

iter: 872, loss : 2.282e-03
iter: 873, loss : 2.148e-04
iter: 874, loss : 2.402e-03
iter: 875, loss : 9.007e-04
iter: 876, loss : 4.774e-03
iter: 877, loss : 3.011e-04
iter: 878, loss : 1.434e-03
iter: 879, loss : 4.817e-04
iter: 880, loss : 4.667e-04
iter: 881, loss : 6.727e-04
iter: 882, loss : 2.871e-04
iter: 883, loss : 3.806e-04
iter: 884, loss : 1.218e-03
iter: 885, loss : 2.379e-04
iter: 886, loss : 1.063e-03
iter: 887, loss : 5.831e-04
iter: 888, loss : 3.228e-04
iter: 889, loss : 2.072e-04
iter: 890, loss : 2.031e-03
iter: 891, loss : 2.902e-03
iter: 892, loss : 4.300e-04

iter: 893, loss : 4.342e-04
iter: 894, loss : 3.657e-04
iter: 895, loss : 3.204e-04
iter: 896, loss : 3.242e-04
iter: 897, loss : 1.744e-03
iter: 898, loss : 3.193e-04
iter: 899, loss : 2.023e-04
iter: 900, loss : 1.902e-04
iter: 901, loss : 2.308e-04
iter: 902, loss : 1.216e-03
iter: 903, loss : 9.139e-04
iter: 904, loss : 2.406e-04
iter: 905, loss : 1.866e-04
iter: 906, loss : 2.283e-04
iter: 907, loss : 1.385e-04
iter: 908, loss : 2.174e-04
iter: 909, loss : 2.665e-03
iter: 910, loss : 8.760e-04
iter: 911, loss : 1.831e-04
iter: 912, loss : 2.505e-04

iter: 913, loss : 9.146e-04
iter: 914, loss : 5.140e-05
iter: 915, loss : 3.858e-04
iter: 916, loss : 9.697e-04
iter: 917, loss : 5.157e-03
iter: 918, loss : 3.843e-04
iter: 919, loss : 1.596e-04
iter: 920, loss : 3.462e-04
iter: 921, loss : 2.874e-04
iter: 922, loss : 2.874e-04
iter: 923, loss : 4.517e-04
iter: 924, loss : 2.846e-03
iter: 925, loss : 3.442e-04
iter: 926, loss : 3.774e-03
iter: 927, loss : 2.785e-04
iter: 928, loss : 1.801e-03
iter: 929, loss : 2.132e-03
iter: 930, loss : 1.702e-03
iter: 931, loss : 1.731e-04
iter: 932, loss : 3.473e-04
iter: 933, loss : 6.066e-03

iter: 934, loss : 6.376e-04
iter: 935, loss : 3.357e-04
iter: 936, loss : 1.928e-04
iter: 937, loss : 1.470e-03
iter: 938, loss : 1.297e-03
iter: 939, loss : 1.963e-03
iter: 940, loss : 1.809e-04
iter: 941, loss : 1.442e-04
iter: 942, loss : 2.338e-04
iter: 943, loss : 3.381e-04
iter: 944, loss : 9.004e-05
iter: 945, loss : 1.098e-03
iter: 946, loss : 6.984e-04
iter: 947, loss : 3.235e-04
iter: 948, loss : 2.336e-04
iter: 949, loss : 1.749e-04
iter: 950, loss : 2.427e-04
iter: 951, loss : 1.972e-04
iter: 952, loss : 4.901e-04
iter: 953, loss : 2.160e-04

iter: 954, loss : 2.823e-04
iter: 955, loss : 2.137e-04
iter: 956, loss : 2.231e-04
iter: 957, loss : 4.169e-03
iter: 958, loss : 4.322e-04
iter: 959, loss : 7.350e-03
iter: 960, loss : 3.495e-03
iter: 961, loss : 6.610e-04
iter: 962, loss : 2.831e-03
iter: 963, loss : 3.964e-03
iter: 964, loss : 2.673e-04
iter: 965, loss : 4.786e-04
iter: 966, loss : 4.866e-04
iter: 967, loss : 2.070e-04
iter: 968, loss : 1.954e-04
iter: 969, loss : 2.132e-04
iter: 970, loss : 3.031e-04
iter: 971, loss : 1.680e-04
iter: 972, loss : 1.983e-04
iter: 973, loss : 1.391e-03
iter: 974, loss : 1.413e-03
iter: 975, loss : 4.295e-03

iter: 976, loss : 3.218e-04
iter: 977, loss : 2.815e-04
iter: 978, loss : 8.110e-04
iter: 979, loss : 6.801e-04
iter: 980, loss : 6.129e-04
iter: 981, loss : 1.149e-03
iter: 982, loss : 2.141e-04
iter: 983, loss : 1.068e-03
iter: 984, loss : 4.734e-04
iter: 985, loss : 2.773e-04
iter: 986, loss : 4.781e-04
iter: 987, loss : 3.864e-04
iter: 988, loss : 3.982e-04
iter: 989, loss : 2.762e-04
iter: 990, loss : 1.841e-04
iter: 991, loss : 2.323e-04
iter: 992, loss : 1.929e-04
iter: 993, loss : 3.410e-04
iter: 994, loss : 1.912e-04

iter: 995, loss : 1.978e-04
iter: 996, loss : 2.407e-04
iter: 997, loss : 2.791e-04
iter: 998, loss : 1.825e-03
iter: 999, loss : 1.723e-04
iter: 1000, loss : 6.162e-04
iter: 1001, loss : 3.455e-04
iter: 1002, loss : 2.801e-04
iter: 1003, loss : 4.183e-04
iter: 1004, loss : 2.825e-04
iter: 1005, loss : 3.950e-04
iter: 1006, loss : 1.516e-03
iter: 1007, loss : 3.242e-04
iter: 1008, loss : 1.265e-03
iter: 1009, loss : 2.411e-04
iter: 1010, loss : 1.670e-03
iter: 1011, loss : 5.121e-04
iter: 1012, loss : 7.636e-04
iter: 1013, loss : 2.556e-04
iter: 1014, loss : 1.929e-04
iter: 1015, loss : 2.414e-04
iter: 1016, loss : 2.492e-04

iter: 1017, loss : 1.085e-03
iter: 1018, loss : 1.227e-03
iter: 1019, loss : 1.971e-04
iter: 1020, loss : 3.772e-04
iter: 1021, loss : 1.547e-04
iter: 1022, loss : 5.107e-03
iter: 1023, loss : 1.923e-04
iter: 1024, loss : 1.043e-04
iter: 1025, loss : 9.704e-05
iter: 1026, loss : 6.842e-05
iter: 1027, loss : 1.338e-04
iter: 1028, loss : 6.432e-04
iter: 1029, loss : 1.571e-03
iter: 1030, loss : 6.514e-03
iter: 1031, loss : 1.003e-04
iter: 1032, loss : 2.073e-04
iter: 1033, loss : 6.160e-04
iter: 1034, loss : 7.180e-04

iter: 1035, loss : 2.450e-04
iter: 1036, loss : 1.548e-03
iter: 1037, loss : 2.002e-04
iter: 1038, loss : 4.462e-03
iter: 1039, loss : 1.012e-03
iter: 1040, loss : 2.183e-04
iter: 1041, loss : 5.187e-04
iter: 1042, loss : 3.338e-03
iter: 1043, loss : 2.296e-04
iter: 1044, loss : 1.545e-04
iter: 1045, loss : 4.071e-04
iter: 1046, loss : 3.936e-04
iter: 1047, loss : 2.483e-04
iter: 1048, loss : 3.195e-04
iter: 1049, loss : 3.161e-04
iter: 1050, loss : 3.020e-04
iter: 1051, loss : 1.922e-04
iter: 1052, loss : 2.784e-04
iter: 1053, loss : 1.774e-04
iter: 1054, loss : 8.083e-05
iter: 1055, loss : 2.013e-04
iter: 1056, loss : 2.564e-04
iter: 1057, loss : 1.895e-04

iter: 1058, loss : 2.019e-04
iter: 1059, loss : 2.306e-04
iter: 1060, loss : 1.056e-03
iter: 1061, loss : 4.813e-04
iter: 1062, loss : 1.076e-03
iter: 1063, loss : 9.743e-04
iter: 1064, loss : 2.347e-04
iter: 1065, loss : 2.014e-04
iter: 1066, loss : 1.544e-04
iter: 1067, loss : 1.448e-04
iter: 1068, loss : 2.343e-04
iter: 1069, loss : 2.022e-03
iter: 1070, loss : 2.820e-04
iter: 1071, loss : 2.956e-04
iter: 1072, loss : 3.656e-03
iter: 1073, loss : 3.870e-04
iter: 1074, loss : 2.622e-04
iter: 1075, loss : 3.152e-04

iter: 1076, loss : 1.189e-03
iter: 1077, loss : 6.071e-04
iter: 1078, loss : 5.059e-04
iter: 1079, loss : 5.627e-03
iter: 1080, loss : 2.111e-04
iter: 1081, loss : 7.611e-04
iter: 1082, loss : 5.908e-04
iter: 1083, loss : 2.830e-04
iter: 1084, loss : 4.183e-04
iter: 1085, loss : 3.508e-04
iter: 1086, loss : 1.336e-04
iter: 1087, loss : 1.914e-04
iter: 1088, loss : 1.520e-03
iter: 1089, loss : 2.219e-04
iter: 1090, loss : 3.218e-03
iter: 1091, loss : 8.040e-04
iter: 1092, loss : 4.967e-03
iter: 1093, loss : 3.138e-04
iter: 1094, loss : 8.014e-04
iter: 1095, loss : 3.074e-04
iter: 1096, loss : 4.054e-04
iter: 1097, loss : 3.590e-04

iter: 1098, loss : 2.705e-04
iter: 1099, loss : 1.329e-03
iter: 1100, loss : 2.982e-03
iter: 1101, loss : 3.842e-04
iter: 1102, loss : 6.471e-04
iter: 1103, loss : 4.150e-03
iter: 1104, loss : 2.721e-03
iter: 1105, loss : 1.404e-03
iter: 1106, loss : 4.922e-04
iter: 1107, loss : 2.273e-04
iter: 1108, loss : 2.374e-04
iter: 1109, loss : 1.267e-04
iter: 1110, loss : 6.242e-04
iter: 1111, loss : 4.997e-04
iter: 1112, loss : 1.346e-03
iter: 1113, loss : 2.923e-04
iter: 1114, loss : 6.036e-04
iter: 1115, loss : 3.225e-04

iter: 1116, loss : 4.443e-04
iter: 1117, loss : 3.421e-04
iter: 1118, loss : 3.059e-04
iter: 1119, loss : 4.112e-04
iter: 1120, loss : 9.381e-05
iter: 1121, loss : 6.386e-03
iter: 1122, loss : 4.297e-04
iter: 1123, loss : 8.228e-04
iter: 1124, loss : 2.737e-03
iter: 1125, loss : 2.949e-04
iter: 1126, loss : 5.371e-04
iter: 1127, loss : 4.364e-04
iter: 1128, loss : 1.577e-03
iter: 1129, loss : 4.087e-04
iter: 1130, loss : 1.702e-03
iter: 1131, loss : 6.816e-04
iter: 1132, loss : 3.201e-04
iter: 1133, loss : 2.040e-04
iter: 1134, loss : 1.360e-04
iter: 1135, loss : 2.745e-04
iter: 1136, loss : 1.337e-03
iter: 1137, loss : 5.284e-03

iter: 1138, loss : 5.835e-04
iter: 1139, loss : 2.330e-04
iter: 1140, loss : 1.180e-03
iter: 1141, loss : 1.781e-04
iter: 1142, loss : 3.755e-03
iter: 1143, loss : 4.947e-04
iter: 1144, loss : 1.706e-04
iter: 1145, loss : 7.104e-04
iter: 1146, loss : 4.711e-04
iter: 1147, loss : 3.579e-04
iter: 1148, loss : 1.418e-04
iter: 1149, loss : 3.683e-04
iter: 1150, loss : 7.218e-05
iter: 1151, loss : 2.319e-03
iter: 1152, loss : 2.819e-03
iter: 1153, loss : 7.348e-04
iter: 1154, loss : 5.139e-04
iter: 1155, loss : 3.151e-04

iter: 1156, loss : 4.567e-04
iter: 1157, loss : 6.857e-04
iter: 1158, loss : 4.238e-04
iter: 1159, loss : 1.812e-04
iter: 1160, loss : 3.538e-04
iter: 1161, loss : 3.436e-04
iter: 1162, loss : 8.888e-04
iter: 1163, loss : 2.636e-04
iter: 1164, loss : 2.129e-04
iter: 1165, loss : 3.724e-04
iter: 1166, loss : 1.084e-04
iter: 1167, loss : 3.813e-04
iter: 1168, loss : 4.309e-04
iter: 1169, loss : 2.340e-04
iter: 1170, loss : 9.042e-04
iter: 1171, loss : 1.450e-04
iter: 1172, loss : 2.204e-04
iter: 1173, loss : 5.571e-05
iter: 1174, loss : 2.911e-03
iter: 1175, loss : 1.254e-04
iter: 1176, loss : 1.418e-03
iter: 1177, loss : 9.835e-04

iter: 1178, loss : 1.067e-03
iter: 1179, loss : 1.545e-04
iter: 1180, loss : 2.741e-04
iter: 1181, loss : 2.504e-04
iter: 1182, loss : 1.044e-04
iter: 1183, loss : 8.848e-05
iter: 1184, loss : 1.622e-04
iter: 1185, loss : 7.625e-04
iter: 1186, loss : 2.472e-03
iter: 1187, loss : 9.079e-04
iter: 1188, loss : 5.322e-04
iter: 1189, loss : 1.916e-04
iter: 1190, loss : 1.975e-04
iter: 1191, loss : 1.556e-04
iter: 1192, loss : 2.831e-03
iter: 1193, loss : 2.068e-03
iter: 1194, loss : 4.869e-04
iter: 1195, loss : 4.869e-04
iter: 1196, loss : 1.353e-03

iter: 1197, loss : 1.303e-04
iter: 1198, loss : 5.741e-04
iter: 1199, loss : 2.655e-03
iter: 1200, loss : 4.969e-04
iter: 1201, loss : 8.718e-04
iter: 1202, loss : 6.960e-04
iter: 1203, loss : 2.069e-04
iter: 1204, loss : 8.317e-04
iter: 1205, loss : 7.903e-04
iter: 1206, loss : 3.855e-04
iter: 1207, loss : 4.555e-04
iter: 1208, loss : 3.609e-04
iter: 1209, loss : 1.981e-04
iter: 1210, loss : 2.231e-03
iter: 1211, loss : 1.360e-04
iter: 1212, loss : 6.646e-04
iter: 1213, loss : 5.170e-04
iter: 1214, loss : 2.014e-04
iter: 1215, loss : 2.745e-04
iter: 1216, loss : 2.333e-04
iter: 1217, loss : 6.965e-04

iter: 1218, loss : 1.823e-04
iter: 1219, loss : 1.579e-04
iter: 1220, loss : 5.133e-04
iter: 1221, loss : 3.287e-04
iter: 1222, loss : 7.335e-04
iter: 1223, loss : 2.988e-04
iter: 1224, loss : 2.557e-04
iter: 1225, loss : 3.197e-04
iter: 1226, loss : 2.935e-04
iter: 1227, loss : 2.570e-04
iter: 1228, loss : 1.972e-03
iter: 1229, loss : 2.474e-04
iter: 1230, loss : 6.651e-04
iter: 1231, loss : 1.531e-04
iter: 1232, loss : 9.412e-04
iter: 1233, loss : 5.984e-03
iter: 1234, loss : 7.555e-04
iter: 1235, loss : 2.218e-04
iter: 1236, loss : 1.966e-03

iter: 1237, loss : 7.568e-04
iter: 1238, loss : 3.993e-04
iter: 1239, loss : 2.027e-04
iter: 1240, loss : 7.576e-04
iter: 1241, loss : 1.511e-03
iter: 1242, loss : 1.619e-04
iter: 1243, loss : 2.653e-04
iter: 1244, loss : 1.750e-03
iter: 1245, loss : 2.917e-04
iter: 1246, loss : 2.801e-03
iter: 1247, loss : 5.417e-04
iter: 1248, loss : 2.408e-04
iter: 1249, loss : 3.184e-04
iter: 1250, loss : 1.208e-03
iter: 1251, loss : 4.874e-04
iter: 1252, loss : 2.296e-04
iter: 1253, loss : 1.082e-03
iter: 1254, loss : 1.232e-03
iter: 1255, loss : 9.258e-04
iter: 1256, loss : 7.104e-04

iter: 1257, loss : 4.149e-04
iter: 1258, loss : 4.397e-04
iter: 1259, loss : 2.855e-04
iter: 1260, loss : 2.559e-03
iter: 1261, loss : 3.377e-04
iter: 1262, loss : 1.717e-04
iter: 1263, loss : 1.804e-03
iter: 1264, loss : 4.239e-04
iter: 1265, loss : 4.017e-04
iter: 1266, loss : 9.644e-04
iter: 1267, loss : 2.335e-04
iter: 1268, loss : 3.098e-04
iter: 1269, loss : 2.606e-04
iter: 1270, loss : 9.547e-05
iter: 1271, loss : 8.228e-04
iter: 1272, loss : 7.602e-04
iter: 1273, loss : 2.119e-04
iter: 1274, loss : 2.990e-03
iter: 1275, loss : 2.618e-04

iter: 1276, loss : 6.155e-04
iter: 1277, loss : 2.806e-03
iter: 1278, loss : 1.673e-04
iter: 1279, loss : 2.435e-04
iter: 1280, loss : 1.170e-04
iter: 1281, loss : 2.368e-04
iter: 1282, loss : 8.421e-04
iter: 1283, loss : 4.856e-03
iter: 1284, loss : 1.592e-04
iter: 1285, loss : 2.667e-04
iter: 1286, loss : 7.542e-04
iter: 1287, loss : 9.495e-04
iter: 1288, loss : 2.991e-04
iter: 1289, loss : 2.590e-04
iter: 1290, loss : 1.134e-04
iter: 1291, loss : 3.673e-04
iter: 1292, loss : 4.082e-04
iter: 1293, loss : 3.515e-04
iter: 1294, loss : 4.623e-04
iter: 1295, loss : 1.203e-04
iter: 1296, loss : 1.735e-04
iter: 1297, loss : 3.258e-04

iter: 1298, loss : 2.160e-04
iter: 1299, loss : 2.594e-04
iter: 1300, loss : 8.210e-05
iter: 1301, loss : 1.600e-04
iter: 1302, loss : 2.032e-04
iter: 1303, loss : 1.972e-04
iter: 1304, loss : 2.646e-04
iter: 1305, loss : 1.399e-03
iter: 1306, loss : 1.197e-03
iter: 1307, loss : 2.249e-04
iter: 1308, loss : 1.507e-04
iter: 1309, loss : 2.043e-04
iter: 1310, loss : 5.236e-03
iter: 1311, loss : 1.761e-04
iter: 1312, loss : 3.047e-04
iter: 1313, loss : 1.199e-03
iter: 1314, loss : 1.812e-04
iter: 1315, loss : 2.696e-04
iter: 1316, loss : 2.342e-04

iter: 1317, loss : 2.470e-04
iter: 1318, loss : 2.646e-04
iter: 1319, loss : 8.206e-05
iter: 1320, loss : 2.847e-04
iter: 1321, loss : 2.215e-04
iter: 1322, loss : 1.803e-04
iter: 1323, loss : 5.402e-04
iter: 1324, loss : 1.203e-04
iter: 1325, loss : 2.078e-04
iter: 1326, loss : 9.370e-05
iter: 1327, loss : 1.802e-04
iter: 1328, loss : 1.704e-04
iter: 1329, loss : 2.245e-03
iter: 1330, loss : 3.336e-04
iter: 1331, loss : 1.156e-03
iter: 1332, loss : 3.787e-03
iter: 1333, loss : 7.252e-04
iter: 1334, loss : 7.593e-05
iter: 1335, loss : 4.151e-04
iter: 1336, loss : 5.601e-04
iter: 1337, loss : 1.034e-04
iter: 1338, loss : 2.018e-04

iter: 1339, loss : 8.486e-04
iter: 1340, loss : 5.678e-03
iter: 1341, loss : 1.178e-03
iter: 1342, loss : 3.470e-03
iter: 1343, loss : 3.314e-04
iter: 1344, loss : 4.225e-04
iter: 1345, loss : 6.285e-04
iter: 1346, loss : 7.818e-04
iter: 1347, loss : 2.232e-04
iter: 1348, loss : 5.115e-04
iter: 1349, loss : 4.322e-04
iter: 1350, loss : 3.491e-03
iter: 1351, loss : 1.661e-04
iter: 1352, loss : 2.929e-04
iter: 1353, loss : 1.127e-03
iter: 1354, loss : 3.136e-04
iter: 1355, loss : 1.178e-04
iter: 1356, loss : 4.466e-03

iter: 1357, loss : 3.363e-04
iter: 1358, loss : 2.900e-04
iter: 1359, loss : 3.164e-04
iter: 1360, loss : 6.260e-04
iter: 1361, loss : 1.165e-04
iter: 1362, loss : 3.449e-03
iter: 1363, loss : 3.188e-04
iter: 1364, loss : 3.163e-04
iter: 1365, loss : 3.706e-04
iter: 1366, loss : 2.859e-04
iter: 1367, loss : 7.235e-04
iter: 1368, loss : 2.486e-04
iter: 1369, loss : 1.404e-04
iter: 1370, loss : 2.331e-04
iter: 1371, loss : 1.835e-04
iter: 1372, loss : 1.580e-04
iter: 1373, loss : 1.516e-03
iter: 1374, loss : 9.724e-04
iter: 1375, loss : 2.075e-04
iter: 1376, loss : 3.337e-04
iter: 1377, loss : 4.358e-04
iter: 1378, loss : 8.640e-04

iter: 1379, loss : 2.841e-04
iter: 1380, loss : 2.593e-03
iter: 1381, loss : 3.319e-04
iter: 1382, loss : 7.428e-04
iter: 1383, loss : 1.931e-04
iter: 1384, loss : 1.976e-04
iter: 1385, loss : 6.744e-05
iter: 1386, loss : 9.080e-04
iter: 1387, loss : 6.324e-04
iter: 1388, loss : 2.005e-04
iter: 1389, loss : 3.140e-04
iter: 1390, loss : 6.009e-04
iter: 1391, loss : 1.645e-04
iter: 1392, loss : 5.860e-04
iter: 1393, loss : 3.551e-03
iter: 1394, loss : 2.197e-04
iter: 1395, loss : 7.432e-04
iter: 1396, loss : 3.577e-04

iter: 1397, loss : 4.947e-04
iter: 1398, loss : 7.884e-04
iter: 1399, loss : 9.052e-04
iter: 1400, loss : 3.042e-04
iter: 1401, loss : 1.288e-03
iter: 1402, loss : 3.332e-04
iter: 1403, loss : 1.027e-03
iter: 1404, loss : 1.006e-03
iter: 1405, loss : 5.410e-04
iter: 1406, loss : 3.769e-04
iter: 1407, loss : 3.771e-04
iter: 1408, loss : 2.582e-04
iter: 1409, loss : 4.112e-04
iter: 1410, loss : 2.249e-04
iter: 1411, loss : 2.648e-04
iter: 1412, loss : 8.985e-04
iter: 1413, loss : 4.204e-04
iter: 1414, loss : 1.868e-04
iter: 1415, loss : 2.735e-03
iter: 1416, loss : 3.058e-04
iter: 1417, loss : 2.789e-04
iter: 1418, loss : 2.987e-04

iter: 1419, loss : 9.114e-04
iter: 1420, loss : 9.270e-05
iter: 1421, loss : 4.884e-03
iter: 1422, loss : 2.935e-04
iter: 1423, loss : 3.558e-04
iter: 1424, loss : 4.150e-04
iter: 1425, loss : 2.013e-04
iter: 1426, loss : 2.458e-04
iter: 1427, loss : 2.814e-04
iter: 1428, loss : 1.788e-04
iter: 1429, loss : 7.486e-04
iter: 1430, loss : 1.710e-04
iter: 1431, loss : 6.303e-04
iter: 1432, loss : 4.264e-04
iter: 1433, loss : 1.706e-04
iter: 1434, loss : 1.888e-04
iter: 1435, loss : 1.744e-04
iter: 1436, loss : 1.609e-04

iter: 1437, loss : 5.535e-04
iter: 1438, loss : 1.718e-04
iter: 1439, loss : 1.146e-04
iter: 1440, loss : 6.977e-05
iter: 1441, loss : 1.707e-04
iter: 1442, loss : 5.809e-05
iter: 1443, loss : 1.323e-04
iter: 1444, loss : 6.462e-05
iter: 1445, loss : 1.524e-04
iter: 1446, loss : 2.347e-03
iter: 1447, loss : 2.076e-04
iter: 1448, loss : 1.559e-04
iter: 1449, loss : 2.279e-04
iter: 1450, loss : 8.346e-05
iter: 1451, loss : 1.203e-04
iter: 1452, loss : 7.649e-05
iter: 1453, loss : 1.120e-04
iter: 1454, loss : 4.978e-04
iter: 1455, loss : 1.180e-04
iter: 1456, loss : 1.104e-04
iter: 1457, loss : 7.551e-05
iter: 1458, loss : 2.560e-04

iter: 1459, loss : 3.712e-04
iter: 1460, loss : 8.704e-05
iter: 1461, loss : 8.382e-05
iter: 1462, loss : 3.157e-03
iter: 1463, loss : 1.581e-04
iter: 1464, loss : 6.883e-04
iter: 1465, loss : 8.641e-05
iter: 1466, loss : 1.097e-03
iter: 1467, loss : 1.443e-04
iter: 1468, loss : 5.170e-04
iter: 1469, loss : 1.429e-04
iter: 1470, loss : 2.037e-04
iter: 1471, loss : 1.387e-04
iter: 1472, loss : 1.969e-04
iter: 1473, loss : 9.084e-04
iter: 1474, loss : 4.443e-04
iter: 1475, loss : 2.376e-03
iter: 1476, loss : 2.807e-04

iter: 1477, loss : 3.970e-04
iter: 1478, loss : 6.091e-04
iter: 1479, loss : 8.025e-05
iter: 1480, loss : 1.516e-04
iter: 1481, loss : 2.917e-03
iter: 1482, loss : 4.764e-03
iter: 1483, loss : 1.368e-04
iter: 1484, loss : 1.744e-03
iter: 1485, loss : 2.322e-04
iter: 1486, loss : 8.773e-04
iter: 1487, loss : 8.024e-04
iter: 1488, loss : 1.300e-04
iter: 1489, loss : 4.769e-04
iter: 1490, loss : 3.605e-04
iter: 1491, loss : 1.690e-04
iter: 1492, loss : 8.405e-04
iter: 1493, loss : 3.607e-04
iter: 1494, loss : 7.118e-04
iter: 1495, loss : 2.788e-04
iter: 1496, loss : 4.305e-04
iter: 1497, loss : 2.453e-04
iter: 1498, loss : 3.473e-04

iter: 1499, loss : 2.510e-04
iter: 1500, loss : 6.012e-04
iter: 1501, loss : 1.347e-04
iter: 1502, loss : 4.727e-03
iter: 1503, loss : 6.862e-03
iter: 1504, loss : 6.869e-04
iter: 1505, loss : 3.048e-04
iter: 1506, loss : 3.247e-04
iter: 1507, loss : 7.896e-04
iter: 1508, loss : 2.469e-04
iter: 1509, loss : 9.118e-05
iter: 1510, loss : 5.803e-04
iter: 1511, loss : 1.242e-04
iter: 1512, loss : 2.476e-04
iter: 1513, loss : 4.481e-04
iter: 1514, loss : 4.362e-04
iter: 1515, loss : 2.933e-04
iter: 1516, loss : 1.298e-04

iter: 1517, loss : 2.244e-04
iter: 1518, loss : 3.086e-04
iter: 1519, loss : 1.701e-04
iter: 1520, loss : 2.379e-04
iter: 1521, loss : 1.383e-04
iter: 1522, loss : 1.187e-04
iter: 1523, loss : 1.172e-04
iter: 1524, loss : 8.784e-05
iter: 1525, loss : 1.296e-04
iter: 1526, loss : 1.617e-04
iter: 1527, loss : 1.450e-04
iter: 1528, loss : 1.179e-04
iter: 1529, loss : 1.944e-04
iter: 1530, loss : 9.174e-04
iter: 1531, loss : 2.792e-04
iter: 1532, loss : 1.860e-04
iter: 1533, loss : 1.164e-04
iter: 1534, loss : 1.269e-03
iter: 1535, loss : 1.914e-04
iter: 1536, loss : 1.354e-03
iter: 1537, loss : 8.778e-05
iter: 1538, loss : 1.743e-04
iter: 1539, loss : 2.295e-04

iter: 1540, loss : 2.496e-04
iter: 1541, loss : 9.867e-04
iter: 1542, loss : 2.506e-04
iter: 1543, loss : 1.745e-04
iter: 1544, loss : 1.119e-04
iter: 1545, loss : 7.495e-04
iter: 1546, loss : 1.436e-04
iter: 1547, loss : 1.521e-04
iter: 1548, loss : 4.373e-03
iter: 1549, loss : 1.445e-04
iter: 1550, loss : 1.369e-04
iter: 1551, loss : 2.097e-03
iter: 1552, loss : 1.754e-04
iter: 1553, loss : 1.789e-04
iter: 1554, loss : 3.242e-04
iter: 1555, loss : 1.811e-03
iter: 1556, loss : 5.160e-04
iter: 1557, loss : 7.621e-04

iter: 1558, loss : 4.325e-04
iter: 1559, loss : 7.994e-04
iter: 1560, loss : 3.692e-04
iter: 1561, loss : 1.509e-03
iter: 1562, loss : 1.728e-04
iter: 1563, loss : 5.698e-04
iter: 1564, loss : 8.725e-04
iter: 1565, loss : 5.966e-04
iter: 1566, loss : 2.915e-04
iter: 1567, loss : 1.407e-04
iter: 1568, loss : 2.649e-04
iter: 1569, loss : 3.018e-04
iter: 1570, loss : 2.514e-04
iter: 1571, loss : 5.857e-05
iter: 1572, loss : 1.358e-04
iter: 1573, loss : 1.556e-04
iter: 1574, loss : 4.688e-03
iter: 1575, loss : 4.331e-03
iter: 1576, loss : 2.371e-04
iter: 1577, loss : 2.800e-04
iter: 1578, loss : 2.501e-04
iter: 1579, loss : 1.909e-04

iter: 1580, loss : 3.490e-04
iter: 1581, loss : 6.818e-04
iter: 1582, loss : 4.861e-04
iter: 1583, loss : 1.307e-04
iter: 1584, loss : 2.993e-04
iter: 1585, loss : 3.721e-04
iter: 1586, loss : 1.774e-03
iter: 1587, loss : 3.227e-04
iter: 1588, loss : 1.372e-04
iter: 1589, loss : 2.090e-04
iter: 1590, loss : 8.659e-05
iter: 1591, loss : 2.331e-03
iter: 1592, loss : 1.043e-04
iter: 1593, loss : 5.198e-04
iter: 1594, loss : 5.781e-04
iter: 1595, loss : 1.450e-04
iter: 1596, loss : 4.913e-04
iter: 1597, loss : 1.223e-03

iter: 1598, loss : 2.839e-04
iter: 1599, loss : 1.041e-04
iter: 1600, loss : 2.259e-03
iter: 1601, loss : 9.155e-04
iter: 1602, loss : 3.916e-04
iter: 1603, loss : 4.451e-04
iter: 1604, loss : 2.217e-04
iter: 1605, loss : 3.335e-04
iter: 1606, loss : 4.515e-03
iter: 1607, loss : 2.458e-04
iter: 1608, loss : 2.647e-04
iter: 1609, loss : 1.810e-04
iter: 1610, loss : 5.446e-05
iter: 1611, loss : 2.234e-04
iter: 1612, loss : 1.965e-04
iter: 1613, loss : 3.213e-03
iter: 1614, loss : 1.285e-04
iter: 1615, loss : 6.028e-04
iter: 1616, loss : 3.152e-04
iter: 1617, loss : 3.137e-04
iter: 1618, loss : 1.944e-03
iter: 1619, loss : 3.186e-04

iter: 1620, loss : 4.514e-04
iter: 1621, loss : 1.567e-04
iter: 1622, loss : 2.397e-04
iter: 1623, loss : 2.550e-04
iter: 1624, loss : 1.650e-03
iter: 1625, loss : 3.277e-04
iter: 1626, loss : 1.964e-04
iter: 1627, loss : 4.388e-04
iter: 1628, loss : 1.839e-04
iter: 1629, loss : 2.142e-04
iter: 1630, loss : 1.051e-03
iter: 1631, loss : 4.453e-04
iter: 1632, loss : 3.008e-04
iter: 1633, loss : 3.148e-03
iter: 1634, loss : 4.096e-04
iter: 1635, loss : 1.410e-04
iter: 1636, loss : 1.298e-03
iter: 1637, loss : 9.877e-05

iter: 1638, loss : 3.249e-04
iter: 1639, loss : 1.036e-03
iter: 1640, loss : 2.357e-04
iter: 1641, loss : 8.661e-05
iter: 1642, loss : 3.063e-04
iter: 1643, loss : 7.111e-04
iter: 1644, loss : 1.834e-04
iter: 1645, loss : 2.726e-03
iter: 1646, loss : 7.864e-04
iter: 1647, loss : 2.981e-04
iter: 1648, loss : 3.897e-04
iter: 1649, loss : 1.582e-03
iter: 1650, loss : 1.518e-04
iter: 1651, loss : 2.978e-03
iter: 1652, loss : 5.535e-04
iter: 1653, loss : 2.475e-04
iter: 1654, loss : 5.263e-04
iter: 1655, loss : 2.521e-04
iter: 1656, loss : 2.319e-04
iter: 1657, loss : 1.999e-04
iter: 1658, loss : 7.803e-04
iter: 1659, loss : 6.940e-04

iter: 1660, loss : 2.029e-04
iter: 1661, loss : 2.854e-04
iter: 1662, loss : 2.008e-04
iter: 1663, loss : 1.145e-04
iter: 1664, loss : 3.089e-04
iter: 1665, loss : 5.504e-04
iter: 1666, loss : 1.787e-04
iter: 1667, loss : 3.830e-03
iter: 1668, loss : 3.426e-04
iter: 1669, loss : 8.924e-04
iter: 1670, loss : 2.846e-04
iter: 1671, loss : 3.230e-04
iter: 1672, loss : 1.140e-03
iter: 1673, loss : 1.714e-04
iter: 1674, loss : 2.982e-04
iter: 1675, loss : 2.392e-04
iter: 1676, loss : 2.458e-04
iter: 1677, loss : 2.405e-04

iter: 1678, loss : 1.569e-04
iter: 1679, loss : 2.965e-04
iter: 1680, loss : 1.621e-04
iter: 1681, loss : 7.806e-05
iter: 1682, loss : 2.576e-04
iter: 1683, loss : 5.195e-04
iter: 1684, loss : 1.292e-03
iter: 1685, loss : 3.325e-04
iter: 1686, loss : 2.971e-04
iter: 1687, loss : 6.447e-04
iter: 1688, loss : 1.941e-04
iter: 1689, loss : 3.393e-03
iter: 1690, loss : 6.703e-05
iter: 1691, loss : 1.489e-04
iter: 1692, loss : 1.865e-04
iter: 1693, loss : 1.728e-04
iter: 1694, loss : 7.450e-05
iter: 1695, loss : 3.398e-04
iter: 1696, loss : 1.736e-04
iter: 1697, loss : 4.454e-03
iter: 1698, loss : 3.103e-04
iter: 1699, loss : 1.551e-04

iter: 1700, loss : 2.396e-04
iter: 1701, loss : 1.830e-04
iter: 1702, loss : 2.699e-04
iter: 1703, loss : 1.433e-04
iter: 1704, loss : 1.983e-04
iter: 1705, loss : 7.559e-04
iter: 1706, loss : 1.668e-04
iter: 1707, loss : 2.712e-03
iter: 1708, loss : 6.603e-04
iter: 1709, loss : 2.245e-04
iter: 1710, loss : 1.135e-04
iter: 1711, loss : 5.189e-03
iter: 1712, loss : 1.876e-04
iter: 1713, loss : 2.947e-04
iter: 1714, loss : 3.472e-04
iter: 1715, loss : 1.188e-03
iter: 1716, loss : 7.524e-04
iter: 1717, loss : 1.650e-04

iter: 1718, loss : 4.585e-04
iter: 1719, loss : 2.855e-04
iter: 1720, loss : 5.464e-04
iter: 1721, loss : 2.996e-03
iter: 1722, loss : 6.895e-04
iter: 1723, loss : 4.404e-04
iter: 1724, loss : 2.597e-04
iter: 1725, loss : 2.272e-03
iter: 1726, loss : 8.589e-04
iter: 1727, loss : 1.281e-03
iter: 1728, loss : 5.251e-04
iter: 1729, loss : 1.212e-04
iter: 1730, loss : 1.573e-04
iter: 1731, loss : 2.246e-04
iter: 1732, loss : 6.143e-04
iter: 1733, loss : 2.144e-04
iter: 1734, loss : 2.123e-04
iter: 1735, loss : 1.172e-03
iter: 1736, loss : 9.052e-04
iter: 1737, loss : 1.283e-04
iter: 1738, loss : 1.965e-04
iter: 1739, loss : 2.728e-04

iter: 1740, loss : 2.258e-04
iter: 1741, loss : 6.224e-04
iter: 1742, loss : 2.147e-04
iter: 1743, loss : 1.648e-04
iter: 1744, loss : 3.259e-04
iter: 1745, loss : 7.706e-04
iter: 1746, loss : 9.360e-04
iter: 1747, loss : 3.124e-03
iter: 1748, loss : 2.799e-04
iter: 1749, loss : 3.924e-04
iter: 1750, loss : 2.171e-04
iter: 1751, loss : 5.896e-04
iter: 1752, loss : 1.738e-04
iter: 1753, loss : 2.385e-04
iter: 1754, loss : 2.613e-04
iter: 1755, loss : 3.548e-04
iter: 1756, loss : 1.580e-04
iter: 1757, loss : 5.045e-05

iter: 1758, loss : 2.595e-04
iter: 1759, loss : 1.325e-04
iter: 1760, loss : 7.428e-04
iter: 1761, loss : 8.595e-05
iter: 1762, loss : 6.811e-04
iter: 1763, loss : 3.289e-04
iter: 1764, loss : 2.534e-04
iter: 1765, loss : 9.751e-04
iter: 1766, loss : 6.864e-04
iter: 1767, loss : 2.925e-04
iter: 1768, loss : 2.816e-04
iter: 1769, loss : 4.390e-04
iter: 1770, loss : 9.029e-04
iter: 1771, loss : 5.987e-04
iter: 1772, loss : 6.492e-04
iter: 1773, loss : 2.765e-04
iter: 1774, loss : 1.256e-03
iter: 1775, loss : 2.701e-05
iter: 1776, loss : 1.070e-03
iter: 1777, loss : 1.256e-04
iter: 1778, loss : 5.550e-04
iter: 1779, loss : 1.581e-03

iter: 1780, loss : 3.029e-04
iter: 1781, loss : 3.746e-04
iter: 1782, loss : 5.106e-04
iter: 1783, loss : 4.057e-04
iter: 1784, loss : 1.485e-04
iter: 1785, loss : 1.345e-04
iter: 1786, loss : 1.732e-04
iter: 1787, loss : 4.284e-04
iter: 1788, loss : 4.427e-04
iter: 1789, loss : 2.339e-04
iter: 1790, loss : 1.652e-04
iter: 1791, loss : 2.231e-04
iter: 1792, loss : 1.366e-04
iter: 1793, loss : 7.088e-04
iter: 1794, loss : 1.647e-03
iter: 1795, loss : 9.973e-04
iter: 1796, loss : 1.909e-04
iter: 1797, loss : 1.479e-04

iter: 1798, loss : 1.886e-03
iter: 1799, loss : 3.245e-04
iter: 1800, loss : 4.895e-04
iter: 1801, loss : 4.740e-04
iter: 1802, loss : 6.086e-04
iter: 1803, loss : 1.789e-04
iter: 1804, loss : 1.561e-04
iter: 1805, loss : 2.440e-04
iter: 1806, loss : 2.961e-03
iter: 1807, loss : 1.626e-04
iter: 1808, loss : 7.584e-04
iter: 1809, loss : 6.961e-04
iter: 1810, loss : 5.021e-04
iter: 1811, loss : 1.267e-03
iter: 1812, loss : 1.157e-03
iter: 1813, loss : 4.026e-04
iter: 1814, loss : 3.716e-04
iter: 1815, loss : 3.921e-04
iter: 1816, loss : 2.528e-04
iter: 1817, loss : 1.384e-04
iter: 1818, loss : 1.542e-04
iter: 1819, loss : 1.168e-04

iter: 1820, loss : 3.837e-04
iter: 1821, loss : 2.131e-04
iter: 1822, loss : 7.166e-04
iter: 1823, loss : 7.039e-04
iter: 1824, loss : 3.264e-04
iter: 1825, loss : 1.163e-04
iter: 1826, loss : 2.743e-03
iter: 1827, loss : 1.503e-04
iter: 1828, loss : 1.553e-04
iter: 1829, loss : 2.580e-04
iter: 1830, loss : 3.476e-04
iter: 1831, loss : 1.385e-03
iter: 1832, loss : 3.240e-04
iter: 1833, loss : 2.131e-04
iter: 1834, loss : 3.574e-04
iter: 1835, loss : 7.046e-04
iter: 1836, loss : 1.403e-04
iter: 1837, loss : 1.582e-04

iter: 1838, loss : 3.994e-04
iter: 1839, loss : 1.528e-03
iter: 1840, loss : 5.726e-04
iter: 1841, loss : 1.466e-03
iter: 1842, loss : 3.069e-04
iter: 1843, loss : 8.838e-04
iter: 1844, loss : 2.787e-04
iter: 1845, loss : 5.385e-05
iter: 1846, loss : 1.271e-04
iter: 1847, loss : 1.321e-04
iter: 1848, loss : 1.471e-04
iter: 1849, loss : 9.827e-04
iter: 1850, loss : 1.112e-04
iter: 1851, loss : 5.566e-05
iter: 1852, loss : 2.245e-04
iter: 1853, loss : 5.656e-04
iter: 1854, loss : 5.289e-04
iter: 1855, loss : 1.137e-04
iter: 1856, loss : 1.559e-04
iter: 1857, loss : 2.393e-04
iter: 1858, loss : 1.521e-03
iter: 1859, loss : 1.207e-04

iter: 1860, loss : 1.853e-04
iter: 1861, loss : 5.646e-04
iter: 1862, loss : 2.197e-04
iter: 1863, loss : 6.684e-04
iter: 1864, loss : 5.015e-04
iter: 1865, loss : 2.300e-04
iter: 1866, loss : 4.774e-04
iter: 1867, loss : 1.113e-03
iter: 1868, loss : 2.773e-04
iter: 1869, loss : 2.743e-04
iter: 1870, loss : 5.244e-04
iter: 1871, loss : 1.387e-04
iter: 1872, loss : 4.587e-05
iter: 1873, loss : 1.445e-04
iter: 1874, loss : 1.216e-04
iter: 1875, loss : 3.427e-04
iter: 1876, loss : 3.954e-04
iter: 1877, loss : 3.508e-03

iter: 1878, loss : 7.432e-04
iter: 1879, loss : 8.896e-04
iter: 1880, loss : 3.120e-04
iter: 1881, loss : 2.483e-04
iter: 1882, loss : 6.792e-04
iter: 1883, loss : 1.382e-04
iter: 1884, loss : 8.870e-05
iter: 1885, loss : 1.931e-03
iter: 1886, loss : 4.676e-04
iter: 1887, loss : 5.028e-04
iter: 1888, loss : 2.651e-04
iter: 1889, loss : 2.238e-04
iter: 1890, loss : 1.138e-04
iter: 1891, loss : 4.106e-03
iter: 1892, loss : 9.824e-04
iter: 1893, loss : 3.416e-04
iter: 1894, loss : 5.498e-04
iter: 1895, loss : 9.820e-05
iter: 1896, loss : 2.626e-04
iter: 1897, loss : 1.056e-03
iter: 1898, loss : 1.470e-04
iter: 1899, loss : 2.219e-04

iter: 1900, loss : 1.487e-04
iter: 1901, loss : 2.270e-04
iter: 1902, loss : 1.571e-04
iter: 1903, loss : 3.769e-04
iter: 1904, loss : 1.760e-03
iter: 1905, loss : 2.767e-04
iter: 1906, loss : 1.982e-04
iter: 1907, loss : 6.263e-04
iter: 1908, loss : 1.023e-03
iter: 1909, loss : 3.784e-04
iter: 1910, loss : 1.805e-04
iter: 1911, loss : 3.620e-04
iter: 1912, loss : 2.923e-04
iter: 1913, loss : 4.539e-04
iter: 1914, loss : 2.678e-04
iter: 1915, loss : 2.372e-04
iter: 1916, loss : 6.228e-04
iter: 1917, loss : 4.694e-04

iter: 1918, loss : 3.202e-04
iter: 1919, loss : 4.676e-04
iter: 1920, loss : 4.362e-03
iter: 1921, loss : 2.452e-04
iter: 1922, loss : 2.693e-04
iter: 1923, loss : 4.629e-04
iter: 1924, loss : 3.001e-04
iter: 1925, loss : 1.318e-04
iter: 1926, loss : 1.670e-04
iter: 1927, loss : 3.232e-04
iter: 1928, loss : 1.033e-04
iter: 1929, loss : 1.731e-04
iter: 1930, loss : 6.935e-04
iter: 1931, loss : 1.268e-03
iter: 1932, loss : 1.919e-04
iter: 1933, loss : 6.628e-04
iter: 1934, loss : 3.024e-04
iter: 1935, loss : 6.957e-04
iter: 1936, loss : 3.864e-03
iter: 1937, loss : 3.232e-04
iter: 1938, loss : 2.914e-04
iter: 1939, loss : 5.921e-04

iter: 1940, loss : 1.636e-04
iter: 1941, loss : 2.259e-04
iter: 1942, loss : 6.620e-05
iter: 1943, loss : 2.553e-04
iter: 1944, loss : 1.624e-04
iter: 1945, loss : 1.323e-03
iter: 1946, loss : 1.203e-04
iter: 1947, loss : 2.724e-03
iter: 1948, loss : 4.041e-04
iter: 1949, loss : 1.863e-04
iter: 1950, loss : 8.929e-05
iter: 1951, loss : 1.038e-04
iter: 1952, loss : 1.308e-03
iter: 1953, loss : 5.182e-04
iter: 1954, loss : 2.154e-04
iter: 1955, loss : 3.187e-04
iter: 1956, loss : 1.696e-04
iter: 1957, loss : 2.232e-04

iter: 1958, loss : 1.681e-04
iter: 1959, loss : 1.335e-04
iter: 1960, loss : 2.057e-04
iter: 1961, loss : 1.880e-04
iter: 1962, loss : 2.816e-04
iter: 1963, loss : 2.706e-04
iter: 1964, loss : 9.753e-04
iter: 1965, loss : 1.032e-03
iter: 1966, loss : 2.396e-04
iter: 1967, loss : 2.780e-04
iter: 1968, loss : 2.613e-04
iter: 1969, loss : 2.364e-04
iter: 1970, loss : 5.628e-05
iter: 1971, loss : 4.669e-04
iter: 1972, loss : 2.634e-04
iter: 1973, loss : 3.076e-04
iter: 1974, loss : 6.256e-04
iter: 1975, loss : 2.454e-04
iter: 1976, loss : 4.981e-04
iter: 1977, loss : 2.230e-04
iter: 1978, loss : 6.957e-04
iter: 1979, loss : 1.539e-04

iter: 1980, loss : 4.108e-04
iter: 1981, loss : 1.859e-03
iter: 1982, loss : 1.775e-04
iter: 1983, loss : 2.580e-04
iter: 1984, loss : 1.141e-04
iter: 1985, loss : 1.105e-04
iter: 1986, loss : 2.574e-04
iter: 1987, loss : 1.082e-04
iter: 1988, loss : 2.231e-04
iter: 1989, loss : 4.022e-03
iter: 1990, loss : 1.203e-04
iter: 1991, loss : 1.269e-04
iter: 1992, loss : 2.586e-04
iter: 1993, loss : 6.354e-04
iter: 1994, loss : 1.842e-04
iter: 1995, loss : 1.270e-04
iter: 1996, loss : 9.528e-05
iter: 1997, loss : 1.290e-04

iter: 1998, loss : 1.138e-04
iter: 1999, loss : 6.110e-04
iter: 2000, loss : 1.925e-04
iter: 2001, loss : 9.558e-04
iter: 2002, loss : 2.603e-03
iter: 2003, loss : 3.094e-04
iter: 2004, loss : 4.823e-04
iter: 2005, loss : 1.677e-04
iter: 2006, loss : 3.563e-04
iter: 2007, loss : 1.517e-04
iter: 2008, loss : 1.503e-04
iter: 2009, loss : 2.442e-04
iter: 2010, loss : 4.129e-04
iter: 2011, loss : 2.247e-04
iter: 2012, loss : 8.453e-05
iter: 2013, loss : 9.505e-05
iter: 2014, loss : 1.794e-03
iter: 2015, loss : 6.235e-04
iter: 2016, loss : 9.842e-05
iter: 2017, loss : 7.679e-04
iter: 2018, loss : 1.633e-04
iter: 2019, loss : 1.446e-04
iter: 2020, loss : 4.543e-04

iter: 2021, loss : 1.903e-04
iter: 2022, loss : 1.096e-04
iter: 2023, loss : 2.497e-04
iter: 2024, loss : 4.423e-05
iter: 2025, loss : 1.344e-04
iter: 2026, loss : 1.755e-04
iter: 2027, loss : 1.274e-04
iter: 2028, loss : 9.229e-04
iter: 2029, loss : 2.700e-04
iter: 2030, loss : 2.046e-04
iter: 2031, loss : 6.189e-04
iter: 2032, loss : 2.112e-03
iter: 2033, loss : 2.707e-04
iter: 2034, loss : 2.612e-04
iter: 2035, loss : 1.025e-03
iter: 2036, loss : 2.595e-04
iter: 2037, loss : 2.967e-04

iter: 2038, loss : 2.559e-04
iter: 2039, loss : 2.143e-04
iter: 2040, loss : 5.648e-04
iter: 2041, loss : 5.096e-05
iter: 2042, loss : 8.843e-05
iter: 2043, loss : 1.105e-04
iter: 2044, loss : 1.187e-04
iter: 2045, loss : 5.126e-04
iter: 2046, loss : 1.215e-04
iter: 2047, loss : 1.480e-04
iter: 2048, loss : 1.461e-04
iter: 2049, loss : 4.616e-05
iter: 2050, loss : 1.440e-03
iter: 2051, loss : 8.283e-04
iter: 2052, loss : 1.903e-04
iter: 2053, loss : 1.067e-04
iter: 2054, loss : 3.000e-04
iter: 2055, loss : 4.499e-04
iter: 2056, loss : 2.200e-04
iter: 2057, loss : 1.340e-03
iter: 2058, loss : 2.055e-03
iter: 2059, loss : 1.658e-03
iter: 2060, loss : 7.101e-04

iter: 2061, loss : 8.152e-04
iter: 2062, loss : 8.675e-05
iter: 2063, loss : 3.290e-04
iter: 2064, loss : 1.384e-04
iter: 2065, loss : 1.875e-04
iter: 2066, loss : 7.424e-05
iter: 2067, loss : 4.687e-04
iter: 2068, loss : 6.771e-04
iter: 2069, loss : 5.949e-04
iter: 2070, loss : 1.267e-04
iter: 2071, loss : 8.001e-04
iter: 2072, loss : 1.109e-04
iter: 2073, loss : 1.295e-04
iter: 2074, loss : 1.068e-04
iter: 2075, loss : 7.684e-04
iter: 2076, loss : 5.336e-04
iter: 2077, loss : 1.013e-04

iter: 2078, loss : 6.042e-04
iter: 2079, loss : 2.345e-04
iter: 2080, loss : 3.789e-04
iter: 2081, loss : 2.564e-04
iter: 2082, loss : 1.401e-04
iter: 2083, loss : 3.803e-04
iter: 2084, loss : 1.639e-04
iter: 2085, loss : 1.603e-04
iter: 2086, loss : 3.062e-05
iter: 2087, loss : 1.323e-04
iter: 2088, loss : 2.426e-04
iter: 2089, loss : 4.030e-04
iter: 2090, loss : 1.944e-04
iter: 2091, loss : 6.335e-05
iter: 2092, loss : 2.183e-04
iter: 2093, loss : 1.455e-04
iter: 2094, loss : 5.365e-04
iter: 2095, loss : 1.160e-03
iter: 2096, loss : 4.646e-05
iter: 2097, loss : 1.362e-04
iter: 2098, loss : 9.289e-05
iter: 2099, loss : 1.181e-04
iter: 2100, loss : 2.993e-03
iter: 2101, loss : 2.253e-04

iter: 2102, loss : 1.702e-03
iter: 2103, loss : 4.570e-04
iter: 2104, loss : 7.671e-05
iter: 2105, loss : 2.457e-04
iter: 2106, loss : 3.242e-04
iter: 2107, loss : 3.230e-04
iter: 2108, loss : 3.358e-03
iter: 2109, loss : 3.653e-04
iter: 2110, loss : 5.268e-04
iter: 2111, loss : 4.716e-04
iter: 2112, loss : 1.230e-04
iter: 2113, loss : 8.869e-04
iter: 2114, loss : 5.392e-04
iter: 2115, loss : 5.217e-04
iter: 2116, loss : 2.367e-04
iter: 2117, loss : 9.846e-05

iter: 2118, loss : 5.452e-04
iter: 2119, loss : 2.167e-04
iter: 2120, loss : 1.202e-04
iter: 2121, loss : 5.172e-04
iter: 2122, loss : 2.872e-04
iter: 2123, loss : 1.171e-03
iter: 2124, loss : 7.559e-04
iter: 2125, loss : 2.949e-04
iter: 2126, loss : 3.059e-04
iter: 2127, loss : 2.597e-03
iter: 2128, loss : 2.824e-04
iter: 2129, loss : 3.389e-04
iter: 2130, loss : 2.836e-04
iter: 2131, loss : 2.507e-04
iter: 2132, loss : 1.982e-04
iter: 2133, loss : 2.434e-04
iter: 2134, loss : 3.030e-04
iter: 2135, loss : 1.668e-04
iter: 2136, loss : 2.432e-04
iter: 2137, loss : 1.880e-04
iter: 2138, loss : 1.623e-04
iter: 2139, loss : 1.546e-04
iter: 2140, loss : 2.289e-04
iter: 2141, loss : 8.227e-05

iter: 2142, loss : 1.591e-03
iter: 2143, loss : 1.134e-04
iter: 2144, loss : 2.609e-04
iter: 2145, loss : 1.644e-04
iter: 2146, loss : 1.284e-04
iter: 2147, loss : 7.549e-05
iter: 2148, loss : 1.321e-04
iter: 2149, loss : 3.180e-04
iter: 2150, loss : 8.238e-04
iter: 2151, loss : 5.122e-05
iter: 2152, loss : 1.507e-04
iter: 2153, loss : 1.204e-04
iter: 2154, loss : 2.029e-04
iter: 2155, loss : 1.761e-04
iter: 2156, loss : 3.561e-05
iter: 2157, loss : 1.279e-04
iter: 2158, loss : 8.860e-05

iter: 2159, loss : 8.617e-05
iter: 2160, loss : 1.847e-04
iter: 2161, loss : 1.225e-03
iter: 2162, loss : 1.780e-04
iter: 2163, loss : 5.053e-04
iter: 2164, loss : 6.645e-05
iter: 2165, loss : 9.418e-05
iter: 2166, loss : 3.048e-05
iter: 2167, loss : 8.505e-05
iter: 2168, loss : 1.334e-04
iter: 2169, loss : 1.133e-04
iter: 2170, loss : 7.674e-05
iter: 2171, loss : 8.061e-05
iter: 2172, loss : 2.891e-04
iter: 2173, loss : 4.709e-05
iter: 2174, loss : 9.927e-05
iter: 2175, loss : 3.062e-03
iter: 2176, loss : 4.614e-04
iter: 2177, loss : 2.225e-04
iter: 2178, loss : 7.366e-04
iter: 2179, loss : 1.529e-04
iter: 2180, loss : 1.863e-04
iter: 2181, loss : 1.927e-03
iter: 2182, loss : 1.053e-04
iter: 2183, loss : 7.816e-04

iter: 2184, loss : 1.600e-04
iter: 2185, loss : 4.104e-04
iter: 2186, loss : 3.486e-04
iter: 2187, loss : 3.450e-04
iter: 2188, loss : 3.504e-04
iter: 2189, loss : 2.733e-04
iter: 2190, loss : 1.988e-04
iter: 2191, loss : 1.331e-03
iter: 2192, loss : 7.539e-04
iter: 2193, loss : 8.189e-05
iter: 2194, loss : 1.013e-04
iter: 2195, loss : 2.525e-04
iter: 2196, loss : 1.284e-04
iter: 2197, loss : 7.801e-05
iter: 2198, loss : 1.454e-04
iter: 2199, loss : 7.191e-05

iter: 2200, loss : 2.937e-04
iter: 2201, loss : 1.179e-04
iter: 2202, loss : 3.223e-04
iter: 2203, loss : 2.005e-04
iter: 2204, loss : 1.427e-04
iter: 2205, loss : 1.323e-04
iter: 2206, loss : 1.370e-04
iter: 2207, loss : 3.865e-04
iter: 2208, loss : 1.203e-04
iter: 2209, loss : 1.031e-04
iter: 2210, loss : 1.637e-04
iter: 2211, loss : 1.504e-04
iter: 2212, loss : 1.678e-03
iter: 2213, loss : 9.515e-05
iter: 2214, loss : 1.279e-04
iter: 2215, loss : 1.194e-04
iter: 2216, loss : 1.394e-04
iter: 2217, loss : 6.159e-04
iter: 2218, loss : 2.346e-03
iter: 2219, loss : 2.053e-04
iter: 2220, loss : 1.741e-03
iter: 2221, loss : 5.213e-04
iter: 2222, loss : 3.447e-04
iter: 2223, loss : 1.434e-03
iter: 2224, loss : 1.716e-04

iter: 2225, loss : 1.838e-04
iter: 2226, loss : 1.030e-03
iter: 2227, loss : 4.062e-04
iter: 2228, loss : 4.143e-04
iter: 2229, loss : 1.422e-03
iter: 2230, loss : 6.077e-04
iter: 2231, loss : 1.296e-04
iter: 2232, loss : 5.206e-04
iter: 2233, loss : 4.809e-04
iter: 2234, loss : 2.350e-04
iter: 2235, loss : 2.901e-04
iter: 2236, loss : 6.141e-04
iter: 2237, loss : 2.253e-04
iter: 2238, loss : 3.907e-04
iter: 2239, loss : 3.454e-03

iter: 2240, loss : 5.410e-04
iter: 2241, loss : 6.012e-04
iter: 2242, loss : 4.490e-04
iter: 2243, loss : 3.041e-04
iter: 2244, loss : 2.097e-04
iter: 2245, loss : 2.134e-04
iter: 2246, loss : 9.818e-05
iter: 2247, loss : 1.717e-04
iter: 2248, loss : 7.281e-04
iter: 2249, loss : 3.113e-04
iter: 2250, loss : 2.694e-04
iter: 2251, loss : 3.517e-03
iter: 2252, loss : 4.498e-04
iter: 2253, loss : 2.890e-04
iter: 2254, loss : 1.341e-04
iter: 2255, loss : 1.095e-04
iter: 2256, loss : 2.185e-04
iter: 2257, loss : 1.190e-04
iter: 2258, loss : 1.199e-04
iter: 2259, loss : 6.428e-04
iter: 2260, loss : 2.001e-04
iter: 2261, loss : 9.162e-05
iter: 2262, loss : 1.550e-04
iter: 2263, loss : 5.476e-04
iter: 2264, loss : 3.650e-04

iter: 2265, loss : 2.974e-04
iter: 2266, loss : 1.915e-04
iter: 2267, loss : 1.412e-03
iter: 2268, loss : 7.386e-05
iter: 2269, loss : 1.500e-04
iter: 2270, loss : 1.523e-04
iter: 2271, loss : 2.425e-04
iter: 2272, loss : 4.345e-04
iter: 2273, loss : 2.133e-04
iter: 2274, loss : 5.617e-04
iter: 2275, loss : 8.447e-04
iter: 2276, loss : 2.746e-04
iter: 2277, loss : 1.079e-04
iter: 2278, loss : 6.771e-04
iter: 2279, loss : 8.238e-05

iter: 2280, loss : 2.579e-04
iter: 2281, loss : 8.414e-05
iter: 2282, loss : 5.107e-04
iter: 2283, loss : 2.295e-04
iter: 2284, loss : 6.025e-04
iter: 2285, loss : 1.816e-04
iter: 2286, loss : 2.334e-04
iter: 2287, loss : 8.015e-05
iter: 2288, loss : 1.453e-04
iter: 2289, loss : 2.021e-04
iter: 2290, loss : 2.043e-04
iter: 2291, loss : 2.864e-03
iter: 2292, loss : 1.635e-04
iter: 2293, loss : 8.572e-05
iter: 2294, loss : 2.727e-04
iter: 2295, loss : 2.406e-03
iter: 2296, loss : 1.274e-04
iter: 2297, loss : 2.765e-04
iter: 2298, loss : 3.465e-03
iter: 2299, loss : 1.413e-04
iter: 2300, loss : 1.765e-04
iter: 2301, loss : 2.591e-04
iter: 2302, loss : 2.196e-04
iter: 2303, loss : 1.578e-04
iter: 2304, loss : 8.121e-05

iter: 2305, loss : 3.594e-04
iter: 2306, loss : 4.292e-04
iter: 2307, loss : 4.090e-04
iter: 2308, loss : 3.605e-04
iter: 2309, loss : 1.104e-04
iter: 2310, loss : 1.598e-04
iter: 2311, loss : 1.913e-04
iter: 2312, loss : 5.069e-04
iter: 2313, loss : 2.315e-03
iter: 2314, loss : 2.085e-04
iter: 2315, loss : 6.569e-05
iter: 2316, loss : 2.374e-04
iter: 2317, loss : 2.630e-04
iter: 2318, loss : 5.609e-05
iter: 2319, loss : 5.477e-03

iter: 2320, loss : 5.868e-04
iter: 2321, loss : 4.521e-04
iter: 2322, loss : 5.101e-04
iter: 2323, loss : 4.959e-04
iter: 2324, loss : 5.510e-05
iter: 2325, loss : 1.267e-04
iter: 2326, loss : 1.654e-04
iter: 2327, loss : 1.657e-04
iter: 2328, loss : 2.316e-04
iter: 2329, loss : 2.414e-04
iter: 2330, loss : 9.238e-05
iter: 2331, loss : 2.212e-04
iter: 2332, loss : 4.185e-04
iter: 2333, loss : 1.263e-04
iter: 2334, loss : 1.104e-04
iter: 2335, loss : 3.837e-04
iter: 2336, loss : 4.130e-04
iter: 2337, loss : 1.479e-04
iter: 2338, loss : 7.866e-04
iter: 2339, loss : 1.226e-04
iter: 2340, loss : 7.421e-05
iter: 2341, loss : 9.075e-05
iter: 2342, loss : 3.689e-04
iter: 2343, loss : 1.937e-04

iter: 2344, loss : 2.053e-04
iter: 2345, loss : 2.443e-04
iter: 2346, loss : 1.266e-03
iter: 2347, loss : 1.787e-04
iter: 2348, loss : 2.592e-04
iter: 2349, loss : 1.073e-04
iter: 2350, loss : 3.251e-04
iter: 2351, loss : 3.616e-04
iter: 2352, loss : 1.186e-03
iter: 2353, loss : 3.749e-04
iter: 2354, loss : 1.707e-04
iter: 2355, loss : 1.894e-04
iter: 2356, loss : 4.529e-04
iter: 2357, loss : 7.165e-04
iter: 2358, loss : 1.350e-04
iter: 2359, loss : 8.728e-04

iter: 2360, loss : 1.764e-04
iter: 2361, loss : 1.481e-04
iter: 2362, loss : 3.382e-04
iter: 2363, loss : 8.752e-05
iter: 2364, loss : 5.400e-04
iter: 2365, loss : 3.404e-04
iter: 2366, loss : 3.890e-04
iter: 2367, loss : 1.901e-04
iter: 2368, loss : 1.910e-04
iter: 2369, loss : 1.022e-04
iter: 2370, loss : 1.235e-04
iter: 2371, loss : 1.725e-03
iter: 2372, loss : 2.205e-04
iter: 2373, loss : 5.945e-04
iter: 2374, loss : 8.308e-05
iter: 2375, loss : 4.399e-04
iter: 2376, loss : 1.288e-03
iter: 2377, loss : 3.733e-04
iter: 2378, loss : 3.381e-04
iter: 2379, loss : 1.939e-04
iter: 2380, loss : 1.783e-04
iter: 2381, loss : 1.470e-04
iter: 2382, loss : 1.324e-04
iter: 2383, loss : 1.670e-04

iter: 2384, loss : 1.604e-04
iter: 2385, loss : 1.450e-04
iter: 2386, loss : 2.545e-04
iter: 2387, loss : 2.999e-05
iter: 2388, loss : 1.625e-03
iter: 2389, loss : 1.198e-04
iter: 2390, loss : 3.716e-04
iter: 2391, loss : 4.561e-04
iter: 2392, loss : 3.058e-04
iter: 2393, loss : 1.104e-04
iter: 2394, loss : 6.942e-04
iter: 2395, loss : 1.640e-04
iter: 2396, loss : 2.952e-03
iter: 2397, loss : 5.246e-04
iter: 2398, loss : 3.049e-03
iter: 2399, loss : 7.582e-04

iter: 2400, loss : 4.662e-04
iter: 2401, loss : 2.973e-03
iter: 2402, loss : 6.804e-05
iter: 2403, loss : 1.033e-03
iter: 2404, loss : 4.318e-04
iter: 2405, loss : 5.554e-04
iter: 2406, loss : 6.594e-04
iter: 2407, loss : 2.707e-03
iter: 2408, loss : 3.134e-04
iter: 2409, loss : 7.768e-04
iter: 2410, loss : 1.627e-04
iter: 2411, loss : 3.661e-04
iter: 2412, loss : 1.795e-04
iter: 2413, loss : 2.254e-04
iter: 2414, loss : 1.350e-04
iter: 2415, loss : 1.691e-04
iter: 2416, loss : 9.335e-04
iter: 2417, loss : 3.858e-04
iter: 2418, loss : 6.165e-04
iter: 2419, loss : 1.078e-04
iter: 2420, loss : 1.937e-04
iter: 2421, loss : 2.491e-04
iter: 2422, loss : 6.731e-05
iter: 2423, loss : 1.112e-04

iter: 2424, loss : 2.120e-04
iter: 2425, loss : 1.800e-03
iter: 2426, loss : 1.397e-04
iter: 2427, loss : 1.208e-04
iter: 2428, loss : 9.156e-05
iter: 2429, loss : 9.875e-05
iter: 2430, loss : 2.794e-04
iter: 2431, loss : 1.331e-04
iter: 2432, loss : 7.700e-05
iter: 2433, loss : 1.387e-04
iter: 2434, loss : 1.218e-04
iter: 2435, loss : 3.858e-04
iter: 2436, loss : 1.925e-04
iter: 2437, loss : 4.965e-04
iter: 2438, loss : 8.043e-04
iter: 2439, loss : 4.125e-04

iter: 2440, loss : 9.873e-05
iter: 2441, loss : 2.746e-04
iter: 2442, loss : 7.266e-04
iter: 2443, loss : 1.061e-03
iter: 2444, loss : 2.479e-04
iter: 2445, loss : 9.191e-05
iter: 2446, loss : 4.102e-04
iter: 2447, loss : 4.648e-04
iter: 2448, loss : 2.511e-04
iter: 2449, loss : 1.373e-03
iter: 2450, loss : 1.335e-04
iter: 2451, loss : 1.844e-04
iter: 2452, loss : 1.079e-04
iter: 2453, loss : 6.949e-04
iter: 2454, loss : 1.211e-04
iter: 2455, loss : 2.383e-04
iter: 2456, loss : 1.293e-04
iter: 2457, loss : 9.953e-05
iter: 2458, loss : 3.804e-04
iter: 2459, loss : 6.103e-05
iter: 2460, loss : 1.154e-04
iter: 2461, loss : 2.500e-03
iter: 2462, loss : 3.072e-03
iter: 2463, loss : 1.108e-04

iter: 2464, loss : 2.433e-04
iter: 2465, loss : 2.092e-03
iter: 2466, loss : 2.069e-04
iter: 2467, loss : 8.483e-05
iter: 2468, loss : 1.954e-04
iter: 2469, loss : 1.800e-04
iter: 2470, loss : 2.004e-04
iter: 2471, loss : 1.254e-04
iter: 2472, loss : 2.691e-04
iter: 2473, loss : 8.920e-05
iter: 2474, loss : 5.207e-04
iter: 2475, loss : 9.623e-05
iter: 2476, loss : 9.026e-04
iter: 2477, loss : 2.182e-04
iter: 2478, loss : 3.343e-04
iter: 2479, loss : 1.291e-04

iter: 2480, loss : 1.519e-04
iter: 2481, loss : 9.351e-05
iter: 2482, loss : 1.249e-03
iter: 2483, loss : 3.424e-04
iter: 2484, loss : 4.953e-04
iter: 2485, loss : 9.176e-05
iter: 2486, loss : 5.197e-05
iter: 2487, loss : 1.126e-04
iter: 2488, loss : 2.868e-03
iter: 2489, loss : 8.525e-04
iter: 2490, loss : 2.377e-04
iter: 2491, loss : 1.998e-03
iter: 2492, loss : 1.012e-03
iter: 2493, loss : 4.792e-04
iter: 2494, loss : 5.193e-04
iter: 2495, loss : 6.441e-04
iter: 2496, loss : 3.698e-04
iter: 2497, loss : 3.897e-04
iter: 2498, loss : 1.734e-04
iter: 2499, loss : 1.770e-04
iter: 2500, loss : 4.366e-04
iter: 2501, loss : 1.176e-03
iter: 2502, loss : 9.111e-05
iter: 2503, loss : 8.043e-04

iter: 2504, loss : 3.018e-04
iter: 2505, loss : 1.846e-04
iter: 2506, loss : 3.415e-04
iter: 2507, loss : 1.487e-04
iter: 2508, loss : 1.623e-04
iter: 2509, loss : 4.789e-04
iter: 2510, loss : 1.040e-04
iter: 2511, loss : 1.538e-04
iter: 2512, loss : 2.619e-04
iter: 2513, loss : 1.872e-04
iter: 2514, loss : 8.943e-04
iter: 2515, loss : 2.221e-04
iter: 2516, loss : 2.364e-04
iter: 2517, loss : 4.173e-04
iter: 2518, loss : 1.758e-04
iter: 2519, loss : 1.605e-04

iter: 2520, loss : 2.895e-04
iter: 2521, loss : 8.684e-04
iter: 2522, loss : 7.456e-05
iter: 2523, loss : 6.400e-04
iter: 2524, loss : 6.273e-04
iter: 2525, loss : 3.900e-04
iter: 2526, loss : 4.976e-04
iter: 2527, loss : 1.265e-04
iter: 2528, loss : 3.205e-04
iter: 2529, loss : 2.158e-03
iter: 2530, loss : 1.349e-04
iter: 2531, loss : 1.529e-04
iter: 2532, loss : 1.867e-04
iter: 2533, loss : 9.152e-05
iter: 2534, loss : 2.741e-04
iter: 2535, loss : 3.711e-04
iter: 2536, loss : 1.697e-04
iter: 2537, loss : 7.727e-05
iter: 2538, loss : 3.155e-04
iter: 2539, loss : 2.857e-04
iter: 2540, loss : 6.126e-04
iter: 2541, loss : 4.174e-04
iter: 2542, loss : 1.312e-04
iter: 2543, loss : 1.119e-04

iter: 2544, loss : 3.788e-04
iter: 2545, loss : 2.202e-04
iter: 2546, loss : 1.550e-04
iter: 2547, loss : 1.812e-04
iter: 2548, loss : 9.570e-05
iter: 2549, loss : 1.131e-04
iter: 2550, loss : 2.413e-04
iter: 2551, loss : 1.515e-04
iter: 2552, loss : 1.363e-04
iter: 2553, loss : 9.160e-04
iter: 2554, loss : 4.316e-04
iter: 2555, loss : 1.825e-04
iter: 2556, loss : 1.863e-04
iter: 2557, loss : 6.858e-04
iter: 2558, loss : 5.346e-04
iter: 2559, loss : 1.630e-04

iter: 2560, loss : 2.143e-04
iter: 2561, loss : 3.207e-05
iter: 2562, loss : 3.218e-04
iter: 2563, loss : 4.450e-04
iter: 2564, loss : 2.059e-04
iter: 2565, loss : 1.910e-04
iter: 2566, loss : 1.984e-04
iter: 2567, loss : 2.224e-04
iter: 2568, loss : 6.038e-05
iter: 2569, loss : 1.270e-04
iter: 2570, loss : 6.026e-05
iter: 2571, loss : 1.363e-04
iter: 2572, loss : 1.154e-04
iter: 2573, loss : 4.138e-04
iter: 2574, loss : 1.273e-03
iter: 2575, loss : 4.161e-04
iter: 2576, loss : 3.680e-04
iter: 2577, loss : 1.966e-04
iter: 2578, loss : 1.618e-04
iter: 2579, loss : 1.704e-04
iter: 2580, loss : 1.664e-04
iter: 2581, loss : 1.912e-04
iter: 2582, loss : 6.899e-05
iter: 2583, loss : 4.738e-04

iter: 2584, loss : 1.507e-03
iter: 2585, loss : 3.820e-04
iter: 2586, loss : 7.160e-05
iter: 2587, loss : 1.319e-04
iter: 2588, loss : 2.250e-04
iter: 2589, loss : 4.926e-05
iter: 2590, loss : 1.411e-04
iter: 2591, loss : 1.510e-04
iter: 2592, loss : 7.058e-05
iter: 2593, loss : 5.665e-03
iter: 2594, loss : 1.573e-04
iter: 2595, loss : 2.019e-04
iter: 2596, loss : 2.057e-04
iter: 2597, loss : 1.419e-04
iter: 2598, loss : 9.221e-05
iter: 2599, loss : 2.125e-04
iter: 2600, loss : 7.010e-05

iter: 2601, loss : 1.386e-04
iter: 2602, loss : 3.720e-04
iter: 2603, loss : 6.549e-05
iter: 2604, loss : 1.635e-04
iter: 2605, loss : 1.095e-04
iter: 2606, loss : 5.597e-04
iter: 2607, loss : 3.319e-03
iter: 2608, loss : 7.451e-05
iter: 2609, loss : 7.680e-05
iter: 2610, loss : 4.792e-05
iter: 2611, loss : 2.513e-04
iter: 2612, loss : 1.470e-04
iter: 2613, loss : 3.422e-04
iter: 2614, loss : 6.697e-05
iter: 2615, loss : 6.685e-05
iter: 2616, loss : 2.132e-04
iter: 2617, loss : 1.773e-04
iter: 2618, loss : 2.188e-03
iter: 2619, loss : 2.100e-04
iter: 2620, loss : 9.086e-05
iter: 2621, loss : 4.682e-04
iter: 2622, loss : 4.107e-04
iter: 2623, loss : 3.674e-04
iter: 2624, loss : 4.350e-04

iter: 2625, loss : 1.022e-04
iter: 2626, loss : 8.138e-04
iter: 2627, loss : 2.609e-04
iter: 2628, loss : 1.105e-04
iter: 2629, loss : 1.360e-04
iter: 2630, loss : 3.684e-05
iter: 2631, loss : 1.238e-04
iter: 2632, loss : 1.780e-04
iter: 2633, loss : 1.005e-04
iter: 2634, loss : 1.394e-04
iter: 2635, loss : 9.170e-05
iter: 2636, loss : 1.442e-04
iter: 2637, loss : 1.506e-04
iter: 2638, loss : 6.217e-05
iter: 2639, loss : 1.177e-03
iter: 2640, loss : 9.359e-04
iter: 2641, loss : 3.988e-04

iter: 2642, loss : 4.829e-04
iter: 2643, loss : 1.217e-04
iter: 2644, loss : 1.185e-04
iter: 2645, loss : 1.677e-04
iter: 2646, loss : 1.374e-03
iter: 2647, loss : 2.394e-04
iter: 2648, loss : 2.067e-04
iter: 2649, loss : 1.023e-04
iter: 2650, loss : 8.542e-05
iter: 2651, loss : 2.433e-04
iter: 2652, loss : 1.536e-04
iter: 2653, loss : 2.340e-03
iter: 2654, loss : 3.562e-04
iter: 2655, loss : 7.831e-05
iter: 2656, loss : 1.352e-04
iter: 2657, loss : 1.061e-04
iter: 2658, loss : 3.971e-04
iter: 2659, loss : 3.597e-04
iter: 2660, loss : 2.106e-03
iter: 2661, loss : 1.351e-04
iter: 2662, loss : 4.927e-04

iter: 2663, loss : 3.255e-04
iter: 2664, loss : 1.838e-04
iter: 2665, loss : 2.174e-04
iter: 2666, loss : 1.047e-04
iter: 2667, loss : 2.503e-04
iter: 2668, loss : 3.146e-04
iter: 2669, loss : 3.717e-04
iter: 2670, loss : 1.755e-04
iter: 2671, loss : 1.950e-03
iter: 2672, loss : 1.614e-04
iter: 2673, loss : 3.883e-04
iter: 2674, loss : 1.580e-04
iter: 2675, loss : 5.969e-05
iter: 2676, loss : 2.587e-04
iter: 2677, loss : 1.846e-04
iter: 2678, loss : 1.150e-03
iter: 2679, loss : 1.395e-04

iter: 2680, loss : 4.967e-04
iter: 2681, loss : 1.714e-04
iter: 2682, loss : 1.488e-04
iter: 2683, loss : 4.072e-04
iter: 2684, loss : 6.758e-05
iter: 2685, loss : 4.136e-04
iter: 2686, loss : 2.082e-04
iter: 2687, loss : 1.824e-04
iter: 2688, loss : 2.663e-04
iter: 2689, loss : 9.755e-05
iter: 2690, loss : 3.940e-04
iter: 2691, loss : 4.902e-04
iter: 2692, loss : 3.435e-04
iter: 2693, loss : 9.598e-05
iter: 2694, loss : 4.181e-05
iter: 2695, loss : 1.758e-04
iter: 2696, loss : 1.392e-03
iter: 2697, loss : 1.737e-04
iter: 2698, loss : 6.560e-04
iter: 2699, loss : 1.218e-04
iter: 2700, loss : 1.193e-04
iter: 2701, loss : 4.085e-04
iter: 2702, loss : 2.570e-05
iter: 2703, loss : 7.619e-05

iter: 2704, loss : 5.900e-04
iter: 2705, loss : 4.584e-04
iter: 2706, loss : 3.974e-04
iter: 2707, loss : 9.476e-05
iter: 2708, loss : 2.460e-04
iter: 2709, loss : 6.897e-05
iter: 2710, loss : 5.778e-04
iter: 2711, loss : 1.594e-04
iter: 2712, loss : 2.100e-03
iter: 2713, loss : 2.707e-04
iter: 2714, loss : 2.079e-04
iter: 2715, loss : 1.578e-04
iter: 2716, loss : 9.417e-05
iter: 2717, loss : 1.052e-04
iter: 2718, loss : 4.670e-04
iter: 2719, loss : 3.647e-04

iter: 2720, loss : 1.727e-03
iter: 2721, loss : 2.853e-04
iter: 2722, loss : 1.835e-04
iter: 2723, loss : 3.103e-04
iter: 2724, loss : 1.039e-03
iter: 2725, loss : 1.731e-04
iter: 2726, loss : 5.769e-04
iter: 2727, loss : 3.667e-04
iter: 2728, loss : 1.257e-04
iter: 2729, loss : 5.197e-05
iter: 2730, loss : 2.585e-04
iter: 2731, loss : 2.428e-04
iter: 2732, loss : 1.874e-04
iter: 2733, loss : 1.627e-04
iter: 2734, loss : 1.512e-04
iter: 2735, loss : 8.297e-05
iter: 2736, loss : 1.067e-03
iter: 2737, loss : 1.342e-04
iter: 2738, loss : 1.678e-04
iter: 2739, loss : 1.336e-04
iter: 2740, loss : 5.272e-04
iter: 2741, loss : 9.869e-05
iter: 2742, loss : 3.841e-04
iter: 2743, loss : 1.978e-04

iter: 2744, loss : 6.983e-04
iter: 2745, loss : 2.869e-04
iter: 2746, loss : 9.921e-04
iter: 2747, loss : 3.373e-04
iter: 2748, loss : 8.290e-05
iter: 2749, loss : 3.013e-04
iter: 2750, loss : 7.368e-04
iter: 2751, loss : 1.169e-04
iter: 2752, loss : 3.969e-04
iter: 2753, loss : 2.166e-04
iter: 2754, loss : 5.928e-04
iter: 2755, loss : 3.237e-04
iter: 2756, loss : 3.975e-05
iter: 2757, loss : 3.768e-04
iter: 2758, loss : 7.127e-05
iter: 2759, loss : 1.860e-04

iter: 2760, loss : 7.763e-04
iter: 2761, loss : 1.571e-04
iter: 2762, loss : 1.582e-04
iter: 2763, loss : 1.574e-04
iter: 2764, loss : 1.693e-03
iter: 2765, loss : 2.010e-04
iter: 2766, loss : 2.899e-04
iter: 2767, loss : 1.511e-04
iter: 2768, loss : 1.027e-04
iter: 2769, loss : 4.155e-04
iter: 2770, loss : 3.467e-04
iter: 2771, loss : 1.057e-04
iter: 2772, loss : 1.731e-04
iter: 2773, loss : 9.547e-05
iter: 2774, loss : 2.606e-03
iter: 2775, loss : 1.018e-04
iter: 2776, loss : 2.650e-04
iter: 2777, loss : 2.324e-04
iter: 2778, loss : 6.558e-05
iter: 2779, loss : 1.016e-04
iter: 2780, loss : 4.156e-04
iter: 2781, loss : 6.910e-05
iter: 2782, loss : 1.584e-04
iter: 2783, loss : 1.489e-04

iter: 2784, loss : 9.244e-04
iter: 2785, loss : 5.886e-04
iter: 2786, loss : 2.318e-04
iter: 2787, loss : 2.929e-04
iter: 2788, loss : 1.763e-04
iter: 2789, loss : 1.602e-04
iter: 2790, loss : 9.794e-05
iter: 2791, loss : 1.732e-04
iter: 2792, loss : 1.483e-03
iter: 2793, loss : 3.525e-04
iter: 2794, loss : 2.091e-04
iter: 2795, loss : 1.375e-04
iter: 2796, loss : 1.143e-04
iter: 2797, loss : 1.177e-04
iter: 2798, loss : 1.428e-04
iter: 2799, loss : 2.625e-04
iter: 2800, loss : 9.437e-05

iter: 2801, loss : 1.486e-04
iter: 2802, loss : 6.250e-05
iter: 2803, loss : 8.687e-05
iter: 2804, loss : 1.199e-04
iter: 2805, loss : 7.491e-04
iter: 2806, loss : 3.062e-03
iter: 2807, loss : 1.261e-04
iter: 2808, loss : 7.117e-04
iter: 2809, loss : 2.457e-04
iter: 2810, loss : 1.322e-04
iter: 2811, loss : 1.751e-04
iter: 2812, loss : 1.073e-04
iter: 2813, loss : 1.155e-04
iter: 2814, loss : 3.856e-04
iter: 2815, loss : 7.912e-04
iter: 2816, loss : 5.906e-04
iter: 2817, loss : 7.656e-04
iter: 2818, loss : 2.925e-04
iter: 2819, loss : 3.603e-04
iter: 2820, loss : 2.448e-04
iter: 2821, loss : 1.321e-04
iter: 2822, loss : 1.391e-04
iter: 2823, loss : 1.228e-04
iter: 2824, loss : 6.787e-05

iter: 2825, loss : 8.773e-05
iter: 2826, loss : 4.560e-04
iter: 2827, loss : 1.797e-04
iter: 2828, loss : 1.072e-04
iter: 2829, loss : 1.101e-04
iter: 2830, loss : 4.423e-04
iter: 2831, loss : 1.161e-04
iter: 2832, loss : 1.367e-04
iter: 2833, loss : 8.818e-05
iter: 2834, loss : 6.718e-04
iter: 2835, loss : 6.021e-05
iter: 2836, loss : 1.274e-04
iter: 2837, loss : 1.101e-04
iter: 2838, loss : 2.100e-03
iter: 2839, loss : 1.643e-04
iter: 2840, loss : 2.799e-04

iter: 2841, loss : 1.286e-04
iter: 2842, loss : 1.774e-04
iter: 2843, loss : 1.624e-04
iter: 2844, loss : 1.585e-04
iter: 2845, loss : 8.579e-05
iter: 2846, loss : 2.693e-03
iter: 2847, loss : 6.801e-05
iter: 2848, loss : 9.736e-05
iter: 2849, loss : 5.420e-05
iter: 2850, loss : 2.776e-04
iter: 2851, loss : 1.388e-04
iter: 2852, loss : 1.133e-04
iter: 2853, loss : 2.426e-04
iter: 2854, loss : 7.584e-05
iter: 2855, loss : 4.283e-04
iter: 2856, loss : 9.964e-05
iter: 2857, loss : 1.827e-04
iter: 2858, loss : 1.704e-04
iter: 2859, loss : 3.587e-03
iter: 2860, loss : 1.845e-04
iter: 2861, loss : 1.991e-03
iter: 2862, loss : 4.178e-04
iter: 2863, loss : 2.492e-04
iter: 2864, loss : 1.671e-03

iter: 2865, loss : 4.651e-04
iter: 2866, loss : 3.349e-04
iter: 2867, loss : 2.723e-04
iter: 2868, loss : 2.636e-04
iter: 2869, loss : 2.149e-04
iter: 2870, loss : 1.943e-04
iter: 2871, loss : 9.669e-05
iter: 2872, loss : 1.721e-04
iter: 2873, loss : 1.959e-04
iter: 2874, loss : 1.742e-04
iter: 2875, loss : 3.870e-04
iter: 2876, loss : 1.750e-04
iter: 2877, loss : 1.294e-04
iter: 2878, loss : 1.792e-03
iter: 2879, loss : 5.730e-04

iter: 2880, loss : 2.079e-04
iter: 2881, loss : 2.181e-04
iter: 2882, loss : 6.269e-04
iter: 2883, loss : 4.026e-04
iter: 2884, loss : 3.173e-04
iter: 2885, loss : 2.861e-04
iter: 2886, loss : 3.722e-04
iter: 2887, loss : 2.032e-04
iter: 2888, loss : 1.329e-04
iter: 2889, loss : 7.378e-05
iter: 2890, loss : 3.663e-04
iter: 2891, loss : 3.655e-05
iter: 2892, loss : 1.143e-04
iter: 2893, loss : 4.475e-04
iter: 2894, loss : 7.745e-04
iter: 2895, loss : 3.689e-04
iter: 2896, loss : 2.878e-04
iter: 2897, loss : 2.171e-04
iter: 2898, loss : 2.003e-03
iter: 2899, loss : 3.690e-04
iter: 2900, loss : 2.113e-03
iter: 2901, loss : 3.123e-04
iter: 2902, loss : 1.442e-04
iter: 2903, loss : 2.680e-04

iter: 2904, loss : 1.678e-04
iter: 2905, loss : 1.473e-04
iter: 2906, loss : 2.594e-04
iter: 2907, loss : 2.973e-04
iter: 2908, loss : 1.229e-04
iter: 2909, loss : 6.911e-05
iter: 2910, loss : 1.327e-04
iter: 2911, loss : 7.055e-04
iter: 2912, loss : 1.381e-04
iter: 2913, loss : 2.675e-04
iter: 2914, loss : 1.445e-04
iter: 2915, loss : 8.247e-05
iter: 2916, loss : 8.287e-05
iter: 2917, loss : 8.502e-04
iter: 2918, loss : 2.985e-04
iter: 2919, loss : 1.841e-04

iter: 2920, loss : 3.357e-04
iter: 2921, loss : 1.547e-04
iter: 2922, loss : 2.249e-04
iter: 2923, loss : 3.111e-04
iter: 2924, loss : 9.561e-05
iter: 2925, loss : 1.592e-04
iter: 2926, loss : 1.325e-04
iter: 2927, loss : 1.010e-04
iter: 2928, loss : 8.190e-05
iter: 2929, loss : 5.988e-04
iter: 2930, loss : 4.704e-04
iter: 2931, loss : 6.186e-05
iter: 2932, loss : 6.045e-05
iter: 2933, loss : 4.466e-04
iter: 2934, loss : 1.786e-04
iter: 2935, loss : 1.485e-04
iter: 2936, loss : 2.175e-03
iter: 2937, loss : 1.136e-04
iter: 2938, loss : 1.267e-04
iter: 2939, loss : 1.718e-04
iter: 2940, loss : 9.191e-05
iter: 2941, loss : 1.361e-04
iter: 2942, loss : 7.765e-05
iter: 2943, loss : 7.719e-05
iter: 2944, loss : 5.587e-04

iter: 2945, loss : 7.499e-05
iter: 2946, loss : 3.432e-04
iter: 2947, loss : 1.658e-04
iter: 2948, loss : 9.438e-04
iter: 2949, loss : 2.099e-04
iter: 2950, loss : 2.013e-04
iter: 2951, loss : 1.902e-04
iter: 2952, loss : 3.386e-04
iter: 2953, loss : 7.377e-05
iter: 2954, loss : 1.019e-04
iter: 2955, loss : 1.461e-04
iter: 2956, loss : 2.761e-05
iter: 2957, loss : 5.114e-04
iter: 2958, loss : 8.850e-04
iter: 2959, loss : 6.388e-04
iter: 2960, loss : 2.125e-04

iter: 2961, loss : 8.027e-05
iter: 2962, loss : 2.898e-04
iter: 2963, loss : 9.605e-05
iter: 2964, loss : 2.498e-04
iter: 2965, loss : 1.226e-04
iter: 2966, loss : 3.396e-04
iter: 2967, loss : 3.977e-04
iter: 2968, loss : 1.535e-04
iter: 2969, loss : 4.952e-05
iter: 2970, loss : 4.256e-04
iter: 2971, loss : 4.449e-04
iter: 2972, loss : 9.544e-05
iter: 2973, loss : 5.274e-05
iter: 2974, loss : 2.880e-03
iter: 2975, loss : 2.604e-03
iter: 2976, loss : 2.272e-04
iter: 2977, loss : 2.134e-04
iter: 2978, loss : 1.139e-04
iter: 2979, loss : 3.352e-04
iter: 2980, loss : 9.534e-04
iter: 2981, loss : 3.208e-04
iter: 2982, loss : 1.568e-04
iter: 2983, loss : 1.567e-04
iter: 2984, loss : 1.082e-04

iter: 2985, loss : 3.576e-04
iter: 2986, loss : 1.705e-04
iter: 2987, loss : 5.855e-04
iter: 2988, loss : 6.828e-05
iter: 2989, loss : 4.157e-04
iter: 2990, loss : 1.296e-03
iter: 2991, loss : 6.416e-04
iter: 2992, loss : 5.208e-04
iter: 2993, loss : 3.953e-04
iter: 2994, loss : 8.810e-05
iter: 2995, loss : 1.117e-03
iter: 2996, loss : 3.592e-04
iter: 2997, loss : 2.250e-04
iter: 2998, loss : 2.483e-04
iter: 2999, loss : 1.174e-04
iter: 2999, loss : 1.174e-04
--- 15.171650409698486 seconds --- to perform 3000 epochs

NOTE: For comparison we also called the torcheqdiff demo using the following command line ‘python ode_demo.py –niters 3000’. This triggers a comparable number of gradient evaluations on CPU. The iterations done in torcheqdiff complete in 348 seconds showing how on common CPUs, the use of heyoka may offer a significant speedup for NeuralODE research.

We are now ready to visualize the result of the learned neural ODE:

plt.plot(gt[:, 0], gt[:, 1])

ta_var.time = precision(0)
ta_var.state[:] = list(ic_var)
ta_var.state[:2] = ic
ta_var.pars[:] = np.array(nn_wb, dtype=np.single)
sol = ta_var.propagate_grid(np.array(t_grid, dtype=precision))[5]
plt.plot(sol[:, 0], sol[:, 1], ".")

[<matplotlib.lines.Line2D at 0x7fdcbd112c90>]

It is also of interest to visualize the predictions of the trained system over a new random batch, as opposed to the predictions of the entire trajectory from the first observation:

batch_ic, batch_t_grid, batch_y = get_batch(t_grid, gt, batch_size_ic, batch_size_time)

plt.plot(gt[:, 0], gt[:, 1])
for i in range(batch_size_ic):
    plt.plot(batch_y[:, i, 0], batch_y[:, i, 1], "k.")

start_time = time.time()
for batch in batched(batch_ic, batch_size_simd):
        ta_var_b.set_time(precision(0.))
        ta_var_b.state[:,:] = deepcopy(ic_var_b)
        ta_var_b.state[:dim,:] = np.array(batch).T
        ta_var_b.pars[:] = np.tile(nn_wb, (batch_size_simd,1)).T
        sol_b = ta_var_b.propagate_grid(np.array(t_grid_b, dtype=precision))[1]
        for sol in [sol_b[:,:,i] for i in range(batch_size_simd)]:
            plt.plot(sol[:, 0], sol[:, 1], ".")
print(
    "--- %s seconds --- to compute prediction over the whole batch"
    % (time.time() - start_time)
)

--- 0.007982254028320312 seconds --- to compute prediction over the whole batch