Neural ODEs - II#
We here implement, in heyoka, a standard traning pipeline for NeuralODE replicating this torcheqdiff demo.
We will be using the very same parameters and dynamics, of the original demo but only use a simple fixed learning rate stochastic gradient descent. Note that it is also possible to use more advanced optimizers such as those implemented in pytorch while computing the gradients using heyoka, but in this tutorial we keep it simple and just update the gradient naively.
See also:
# The main imports
import numpy as np
import time
from itertools import batched
from copy import deepcopy
# Import Heyoka
import heyoka as hy
%matplotlib inline
import matplotlib.pyplot as plt
Generating the ground truth#
The dynamics we consider is a simple third order system:
\[\begin{split}
\left\{
\begin{array}{l}
\dot x = -0.1 x^3 - 2y^3 \\
\dot y = 2 x^3 - 0.1y^3
\end{array}
\right.
\end{split}\]
which we implement in heyoka as follows:
# The symolic variables representing the system state.
x, y = hy.make_vars("x", "y")
# We now assemble the dynamics.
A = np.array([[-0.1, 2.0], [-2.0, -0.1]])
tmp = np.array([[x * x * x, y * y * y]]) @ A
dyn = [(x, tmp[0][0]), (y, tmp[0][1])]
# For convenience we store here the dimension of the state.
dim = len(dyn)
To generate groundtruth observations from this system we will use the following parameters:
# This is the number of observations (i.e. the ground truth).
data_size = 1000
# This is the initial condition used to generate the observations (i.e. the ground truth).
ic = [2.0, 0.0]
# This is the time grid used to generate the observations (i.e. the ground truth).
t_grid = np.linspace(0.0, 25, data_size)
We start by instantiating a heyoka.taylor_adaptive object that will generate our observations. These will be considered as ground truth. Note that in other settings this data may be observed from a real phenomena, rather than generated numerically. In that case there will be noise which is here not modelled.
start_time = time.time()
ta = hy.taylor_adaptive(
# The ODEs.
dyn,
# The initial conditions.
ic,
# Do not operate in compact mode.
compact_mode=False,
# Define the tolerance
tol=1e-18,
)
print(
"--- %s seconds --- to build (jit) the Taylor integrator"
% (time.time() - start_time)
)
--- 0.07111859321594238 seconds --- to build (jit) the Taylor integrator
We are thus ready to generate the the ground truth using the method propagate_grid method of the Taylor adaptive integrator.
# It is good practice to always set initial conditions explicitly
ta.time = 0
ta.state[:] = ic
# This is our ground truth (gt)
gt = ta.propagate_grid(t_grid)[5]
Let us have a look at the generated observations.
plt.plot(gt[:, 0], gt[:, 1], "k.")
[<matplotlib.lines.Line2D at 0x7fdddf1f3890>]
Instantiating the neural ODE#
So far we have only generated the data we want to teach our Neural ODE to replicate. We now need to instantiate the Neural ODE system so that we can make predictions and compute the gradient of our loss.
Let us start with instantiating a feed-forward neural network using heyoka.models.ffnn.
# We define as nonlinearity a simple linear layer
linear = lambda inp: inp
# We call the factory to construct a FFNN. Following the torcheqdiff example here we put cubes as inputs:
ffnn = hy.model.ffnn(
inputs=[x * x * x, y * y * y],
nn_hidden=[50],
n_out=2,
activations=[hy.tanh, linear],
)
We are now able to build a taylor_adaptive_batch numerical integrator which will propagate the dynamics in batches as to leverage SIMD instructions:
\[\begin{split}
\left\{
\begin{array}{l}
\dot x = \mathcal N_{\theta, x}(x^3, y^3) \\
\dot y = \mathcal N_{\theta, y}(x^3, y^3) \\
\end{array}
\right.
\end{split}\]
And since we will want to be able to compute the sensitivities of this ODE we are first going to augment such dynamics with the first order variational equations:
# We decide on whether to use single or double precision. In this example it may
# Not be too important, but in other settings it can give a speedup when connected with batch and ensamble propagation.
# The single precision does effect, though, the quality of thegradient direction.
precision = np.double # np.single
batch_size_simd = hy.recommended_simd_size(fp_type=precision)
print("Batch size for SIMD use in integration: ", batch_size_simd)
# We thus define the neural dynamics
dyn_n = [(x, ffnn[0]), (y, ffnn[1])]
# We augment it with the variational one (only order 1 - i.e. gradient)
var_dyn_n = hy.var_ode_sys(dyn_n, args=hy.var_args.params, order=1)
Batch size for SIMD use in integration: 4
… and instantiate the variational integrator. For convenience we also instantiate the non batched version
# This is the batched version, which we will use for gradient computation
ta_var_b = hy.taylor_adaptive_batch(
# The ODEs.
var_dyn_n,
# The initial conditions.
np.ones((dim, batch_size_simd), dtype=precision),
# Operate in compact mode.
compact_mode=True,
# Define the tolerance (low tolerance is enough here)
tol=precision(1e-4),
# Single precision is enough here
fp_type=precision,
)
# This is the normal version used for wuick plotting and visualization of a single solution
ta_var = hy.taylor_adaptive(
# The ODEs.
var_dyn_n,
# The initial conditions.
ic,
# Operate in compact mode.
compact_mode=True,
# Define the tolerance (low tolerance is enough here)
tol=precision(1e-4),
# Single precision is enough here
fp_type=precision,
)
# We store for convenience the initial conditions in a numpy array (including the variational state)
ic_var_b = np.array(ta_var_b.state)
ic_var = np.array(ta_var.state)
Before starting the training, we define the details on the batch we are going to use in the generic step of stochastic gradient descent:
# This is the number of initial conditions to use in one batch (M)
batch_size_ic = 20
# This is the number of observations to predict from each sampled initial condition (T)
batch_size_time = 10
# We check that the observation batch size is a multiple of the simd batch size
print("This must be an integer: ", batch_size_ic / batch_size_simd)
This must be an integer: 5.0
This helper function creates a randomly generated batch from the available observation:
def get_batch(t_grid, gt, batch_size_ic=20, batch_size_time=10):
# We select the initial conditions from which generate predictions
s = np.random.choice(
np.arange(gt.shape[0] - batch_size_time, dtype=np.int64),
batch_size_ic,
replace=False,
)
batch_ic = gt[s, :] # (M, D)
# Assuming uniform grid and a non-autonomous system, all predictions will be made on the same time grid
batch_t_grid = t_grid[:batch_size_time] # (T)
batch_y = np.stack([gt[s + i] for i in range(batch_size_time)]) # (T,M,D)
return batch_ic, batch_t_grid, batch_y
Let us visualize what a batch looks like. First we generate one:
batch_ic, batch_t_grid, batch_y = get_batch(t_grid, gt, batch_size_ic, batch_size_time)
# We need the tgrid repeated as per heyoka API of the taylor adaptive batch
t_grid_b = np.repeat(batch_t_grid, batch_size_simd).reshape(-1, batch_size_simd)
Then we plot it overlapped to all the observations:
plt.plot(gt[:, 0], gt[:, 1])
for i in range(batch_size_ic):
plt.plot(batch_y[:, i, 0], batch_y[:, i, 1], "k.")
We initialize the network weights and biases.
# We compute initialization values for weigths / biases
n_pars = len(ta_var_b.pars)
nn_wb = np.random.normal(loc=0, scale=0.1, size=(n_pars,))
And visualize the prediction of the starting random network.
plt.plot(gt[:, 0], gt[:, 1])
for i in range(batch_size_ic):
plt.plot(batch_y[:, i, 0], batch_y[:, i, 1], "k.")
start_time = time.time()
for batch in batched(batch_ic, batch_size_simd):
ta_var_b.set_time(precision(0.0))
ta_var_b.state[:, :] = deepcopy(ic_var_b)
ta_var_b.state[:dim, :] = np.array(batch).T
ta_var_b.pars[:] = np.tile(nn_wb, (batch_size_simd, 1)).T
sol_b = ta_var_b.propagate_grid(np.array(t_grid_b, dtype=precision))[1]
for sol in [sol_b[:, :, i] for i in range(batch_size_simd)]:
plt.plot(sol[:, 0], sol[:, 1], ".")
print(
"--- %s seconds --- to compute prediction over the whole batch"
% (time.time() - start_time)
)
--- 0.00741267204284668 seconds --- to compute prediction over the whole batch
Training#
To train we need to first compute the gradient of the loss over the batch (M initial conditions and T times). Formally
\[
\mathcal L_{\theta} = \sum_{i=0}^{M}\sum_{j=0}^T (\pmb y_{ij} - \hat {\pmb y}_{ij})\cdot (\pmb y_{ij} - \hat {\pmb y}_{ij})
\]
\[
\frac{\partial L_\theta}{\partial \theta} = 2 \sum_{i=0}^{M}\sum_{j=0}^T (\pmb y_{ij} - \hat {\pmb y}_{ij})\cdot \frac{\partial \pmb y_{ij}}{\partial \theta}
\]
This helper function computes the loss and the gradient over one batch. The gradient is computed rearranging the ODE sensitivities computed propagating the variational dynamics:
def loss_and_gradient(nn_wb, batch_ic, batch_y):
# We reset the batch_loss value
batch_loss = 0
# We reset the gradient loss
grad = np.array([0.0] * ta_var_b.pars.shape[0])
counter = 0
# We process the batch (of the dataset) in batches for SIMD efficient processing
for batch_simd in batched(batch_ic, batch_size_simd):
ta_var_b.set_time(precision(0.0))
ta_var_b.state[:, :] = deepcopy(ic_var_b)
ta_var_b.state[:dim, :] = np.array(batch_simd).T
ta_var_b.pars[:] = np.tile(nn_wb, (batch_size_simd, 1)).T
sol_b = ta_var_b.propagate_grid(np.array(t_grid_b, dtype=precision))[1]
# The numerical integration on the simd_batch is done, we process its result
# accumulating loss and gradient on the predictions
for sol in [sol_b[:, :, i] for i in range(batch_size_simd)]:
# Here is the term (y-y_hat)
diff = sol[:, :dim] - batch_y[:, counter, :]
counter += 1
# Which is then summed sum sum (y-y_hat).(y-y_hat)
batch_loss += np.sum(diff**2)
# And the gradient computed as 2 sum (y-y_hat).dy
for dy in sol[:, dim:].reshape(batch_y.shape[0], dim, -1):
grad += 2 * np.sum(diff @ dy, axis=0)
# We then take the mean over the points used
batch_loss /= batch_size_ic * batch_size_time
grad /= batch_size_ic * batch_size_time
return batch_loss, grad
Using the various blocks defined above the training loop takes the form:
# We reset initialization values for weigths / biases
n_pars = len(ta_var_b.pars)
nn_wb = np.random.normal(loc=0, scale=0.1, size=(n_pars,))
# We loop over the epochs
n_epochs = 3000
start_time = time.time()
for i in range(n_epochs):
# 1 - generate a batch
batch_ic, batch_t_grid, batch_y = get_batch(
t_grid, gt, batch_size_ic=batch_size_ic, batch_size_time=batch_size_time
)
# 2 - compute the loss and gradient on the batch
batch_loss, grad = loss_and_gradient(nn_wb, batch_ic, batch_y)
print(f"iter: {i}, loss : {batch_loss:.3e}", end="\r")
# 3 - update the weights and biases (learning rate here is fixed, we use a naive update rule which in this case works)
nn_wb = nn_wb - 0.1 * grad
print(f"iter: {i}, loss : {batch_loss:.3e}", end="\n")
print("--- %s seconds --- to perform %i epochs" % (time.time() - start_time, n_epochs))
iter: 0, loss : 3.059e-02
iter: 1, loss : 6.223e-02
iter: 2, loss : 2.774e-02
iter: 3, loss : 6.435e-02
iter: 4, loss : 2.331e-02
iter: 5, loss : 9.834e-02
iter: 6, loss : 1.631e-01
iter: 7, loss : 4.619e-02
iter: 8, loss : 1.951e-02
iter: 9, loss : 2.853e-02
iter: 10, loss : 7.630e-02
iter: 11, loss : 4.642e-02
iter: 12, loss : 1.921e-01
iter: 13, loss : 1.602e-02
iter: 14, loss : 9.984e-02
iter: 15, loss : 1.133e-01
iter: 16, loss : 3.377e-02
iter: 17, loss : 2.230e-02
iter: 18, loss : 9.931e-02
iter: 19, loss : 4.866e-02
iter: 20, loss : 2.388e-02
iter: 21, loss : 6.396e-02
iter: 22, loss : 1.072e-01
iter: 23, loss : 3.484e-02
iter: 24, loss : 9.988e-02
iter: 25, loss : 2.231e-02
iter: 26, loss : 8.580e-02
iter: 27, loss : 1.580e-02
iter: 28, loss : 2.732e-02
iter: 29, loss : 8.874e-02
iter: 30, loss : 2.441e-02
iter: 31, loss : 4.764e-02
iter: 32, loss : 3.925e-03
iter: 33, loss : 5.459e-02
iter: 34, loss : 9.242e-02
iter: 35, loss : 4.738e-03
iter: 36, loss : 9.766e-03
iter: 37, loss : 1.304e-02
iter: 38, loss : 9.760e-03
iter: 39, loss : 7.829e-02
iter: 40, loss : 4.520e-02
iter: 41, loss : 4.058e-02
iter: 42, loss : 1.314e-02
iter: 43, loss : 4.814e-03
iter: 44, loss : 7.401e-02
iter: 45, loss : 1.106e-02
iter: 46, loss : 8.431e-03
iter: 47, loss : 1.562e-02
iter: 48, loss : 4.843e-02
iter: 49, loss : 6.022e-03
iter: 50, loss : 1.571e-02
iter: 51, loss : 2.827e-02
iter: 52, loss : 1.939e-02
iter: 53, loss : 7.860e-02
iter: 54, loss : 9.256e-02
iter: 55, loss : 4.768e-02
iter: 56, loss : 2.999e-02
iter: 57, loss : 1.121e-02
iter: 58, loss : 4.946e-03
iter: 59, loss : 9.571e-02
iter: 60, loss : 1.423e-02
iter: 61, loss : 3.572e-02
iter: 62, loss : 7.377e-04
iter: 63, loss : 7.526e-03
iter: 64, loss : 1.116e-02
iter: 65, loss : 4.043e-02
iter: 66, loss : 4.550e-02
iter: 67, loss : 1.805e-03
iter: 68, loss : 1.130e-02
iter: 69, loss : 1.187e-02
iter: 70, loss : 2.169e-02
iter: 71, loss : 2.107e-02
iter: 72, loss : 2.270e-03
iter: 73, loss : 3.146e-02
iter: 74, loss : 1.653e-02
iter: 75, loss : 1.898e-02
iter: 76, loss : 3.434e-02
iter: 77, loss : 5.683e-03
iter: 78, loss : 1.860e-02
iter: 79, loss : 2.770e-02
iter: 80, loss : 3.012e-03
iter: 81, loss : 1.085e-03
iter: 82, loss : 4.349e-03
iter: 83, loss : 5.138e-04
iter: 84, loss : 1.309e-03
iter: 85, loss : 4.261e-04
iter: 86, loss : 2.259e-02
iter: 87, loss : 2.795e-03
iter: 88, loss : 2.778e-02
iter: 89, loss : 2.579e-04
iter: 90, loss : 3.163e-03
iter: 91, loss : 2.125e-02
iter: 92, loss : 2.358e-02
iter: 93, loss : 3.157e-04
iter: 94, loss : 1.596e-02
iter: 95, loss : 1.273e-03
iter: 96, loss : 1.664e-04
iter: 97, loss : 8.104e-05
iter: 98, loss : 1.391e-02
iter: 99, loss : 1.149e-02
iter: 100, loss : 1.988e-04
iter: 101, loss : 8.258e-05
iter: 102, loss : 1.784e-04
iter: 103, loss : 7.777e-03
iter: 104, loss : 2.378e-02
iter: 105, loss : 1.268e-04
iter: 106, loss : 1.124e-03
iter: 107, loss : 6.694e-03
iter: 108, loss : 6.174e-03
iter: 109, loss : 9.731e-04
iter: 110, loss : 2.276e-04
iter: 111, loss : 2.195e-04
iter: 112, loss : 1.371e-03
iter: 113, loss : 1.215e-02
iter: 114, loss : 2.177e-04
iter: 115, loss : 9.073e-05
iter: 116, loss : 1.028e-04
iter: 117, loss : 1.168e-02
iter: 118, loss : 1.857e-04
iter: 119, loss : 1.172e-03
iter: 120, loss : 8.889e-04
iter: 121, loss : 9.275e-04
iter: 122, loss : 3.112e-04
iter: 123, loss : 3.104e-03
iter: 124, loss : 1.526e-03
iter: 125, loss : 2.327e-04
iter: 126, loss : 5.284e-03
iter: 127, loss : 9.413e-03
iter: 128, loss : 9.311e-04
iter: 129, loss : 1.825e-02
iter: 130, loss : 1.083e-03
iter: 131, loss : 3.339e-03
iter: 132, loss : 1.016e-03
iter: 133, loss : 4.650e-04
iter: 134, loss : 2.826e-04
iter: 135, loss : 8.259e-03
iter: 136, loss : 1.707e-02
iter: 137, loss : 1.458e-02
iter: 138, loss : 5.954e-04
iter: 139, loss : 4.591e-03
iter: 140, loss : 7.740e-04
iter: 141, loss : 8.651e-04
iter: 142, loss : 2.400e-04
iter: 143, loss : 4.301e-04
iter: 144, loss : 7.758e-04
iter: 145, loss : 3.400e-04
iter: 146, loss : 9.315e-03
iter: 147, loss : 1.376e-02
iter: 148, loss : 6.179e-04
iter: 149, loss : 2.413e-04
iter: 150, loss : 8.450e-04
iter: 151, loss : 1.217e-02
iter: 152, loss : 6.946e-04
iter: 153, loss : 1.036e-02
iter: 154, loss : 1.631e-02
iter: 155, loss : 1.164e-02
iter: 156, loss : 1.899e-03
iter: 157, loss : 3.311e-03
iter: 158, loss : 4.920e-03
iter: 159, loss : 2.724e-03
iter: 160, loss : 7.873e-04
iter: 161, loss : 8.131e-04
iter: 162, loss : 1.965e-02
iter: 163, loss : 1.414e-03
iter: 164, loss : 5.469e-04
iter: 165, loss : 1.150e-03
iter: 166, loss : 6.982e-04
iter: 167, loss : 1.318e-03
iter: 168, loss : 9.483e-04
iter: 169, loss : 7.114e-04
iter: 170, loss : 7.698e-04
iter: 171, loss : 1.310e-03
iter: 172, loss : 1.925e-02
iter: 173, loss : 3.575e-04
iter: 174, loss : 7.926e-04
iter: 175, loss : 5.973e-03
iter: 176, loss : 6.541e-04
iter: 177, loss : 2.122e-03
iter: 178, loss : 2.641e-03
iter: 179, loss : 1.486e-03
iter: 180, loss : 6.300e-04
iter: 181, loss : 7.530e-04
iter: 182, loss : 4.963e-04
iter: 183, loss : 4.839e-04
iter: 184, loss : 1.480e-03
iter: 185, loss : 3.108e-03
iter: 186, loss : 7.219e-03
iter: 187, loss : 1.116e-02
iter: 188, loss : 1.038e-02
iter: 189, loss : 5.383e-03
iter: 190, loss : 1.243e-02
iter: 191, loss : 4.337e-03
iter: 192, loss : 1.203e-03
iter: 193, loss : 9.117e-03
iter: 194, loss : 1.293e-03
iter: 195, loss : 1.038e-03
iter: 196, loss : 1.078e-03
iter: 197, loss : 6.695e-03
iter: 198, loss : 1.298e-03
iter: 199, loss : 7.068e-04
iter: 200, loss : 6.053e-04
iter: 201, loss : 4.284e-03
iter: 202, loss : 2.333e-03
iter: 203, loss : 3.730e-03
iter: 204, loss : 1.107e-03
iter: 205, loss : 3.224e-03
iter: 206, loss : 1.483e-03
iter: 207, loss : 5.582e-04
iter: 208, loss : 3.334e-03
iter: 209, loss : 2.994e-03
iter: 210, loss : 8.839e-04
iter: 211, loss : 1.039e-03
iter: 212, loss : 1.304e-03
iter: 213, loss : 3.335e-03
iter: 214, loss : 4.435e-03
iter: 215, loss : 3.131e-03
iter: 216, loss : 3.684e-03
iter: 217, loss : 8.809e-04
iter: 218, loss : 1.162e-03
iter: 219, loss : 8.939e-04
iter: 220, loss : 4.499e-04
iter: 221, loss : 1.386e-03
iter: 222, loss : 2.881e-03
iter: 223, loss : 1.226e-03
iter: 224, loss : 4.754e-04
iter: 225, loss : 7.333e-04
iter: 226, loss : 9.889e-04
iter: 227, loss : 5.828e-03
iter: 228, loss : 1.080e-03
iter: 229, loss : 1.115e-03
iter: 230, loss : 5.155e-04
iter: 231, loss : 3.334e-03
iter: 232, loss : 1.056e-03
iter: 233, loss : 8.003e-04
iter: 234, loss : 2.292e-03
iter: 235, loss : 1.324e-02
iter: 236, loss : 1.025e-03
iter: 237, loss : 1.426e-03
iter: 238, loss : 8.783e-04
iter: 239, loss : 8.044e-04
iter: 240, loss : 3.060e-03
iter: 241, loss : 1.653e-03
iter: 242, loss : 6.814e-04
iter: 243, loss : 5.109e-03
iter: 244, loss : 1.008e-02
iter: 245, loss : 6.853e-03
iter: 246, loss : 1.085e-02
iter: 247, loss : 5.294e-04
iter: 248, loss : 3.452e-03
iter: 249, loss : 1.840e-03
iter: 250, loss : 7.499e-04
iter: 251, loss : 7.611e-04
iter: 252, loss : 1.184e-02
iter: 253, loss : 1.286e-02
iter: 254, loss : 1.262e-03
iter: 255, loss : 1.609e-03
iter: 256, loss : 4.714e-03
iter: 257, loss : 2.972e-04
iter: 258, loss : 1.145e-03
iter: 259, loss : 3.221e-03
iter: 260, loss : 4.206e-03
iter: 261, loss : 4.618e-03
iter: 262, loss : 1.162e-03
iter: 263, loss : 5.937e-04
iter: 264, loss : 7.398e-04
iter: 265, loss : 7.234e-03
iter: 266, loss : 1.710e-03
iter: 267, loss : 7.836e-04
iter: 268, loss : 1.188e-03
iter: 269, loss : 1.220e-03
iter: 270, loss : 3.275e-03
iter: 271, loss : 4.345e-04
iter: 272, loss : 8.413e-03
iter: 273, loss : 4.826e-04
iter: 274, loss : 7.862e-04
iter: 275, loss : 9.835e-04
iter: 276, loss : 1.344e-03
iter: 277, loss : 4.525e-04
iter: 278, loss : 1.078e-02
iter: 279, loss : 1.160e-02
iter: 280, loss : 4.257e-04
iter: 281, loss : 3.583e-03
iter: 282, loss : 5.039e-03
iter: 283, loss : 7.140e-03
iter: 284, loss : 1.487e-03
iter: 285, loss : 5.619e-03
iter: 286, loss : 1.073e-03
iter: 287, loss : 1.205e-03
iter: 288, loss : 8.605e-04
iter: 289, loss : 1.179e-03
iter: 290, loss : 2.321e-03
iter: 291, loss : 7.297e-04
iter: 292, loss : 8.235e-04
iter: 293, loss : 7.736e-04
iter: 294, loss : 6.081e-04
iter: 295, loss : 1.323e-03
iter: 296, loss : 8.719e-04
iter: 297, loss : 1.266e-03
iter: 298, loss : 6.778e-03
iter: 299, loss : 4.054e-04
iter: 300, loss : 2.301e-03
iter: 301, loss : 9.558e-03
iter: 302, loss : 9.091e-04
iter: 303, loss : 6.857e-03
iter: 304, loss : 1.869e-03
iter: 305, loss : 4.892e-04
iter: 306, loss : 5.921e-04
iter: 307, loss : 9.923e-03
iter: 308, loss : 9.321e-04
iter: 309, loss : 1.136e-03
iter: 310, loss : 1.175e-03
iter: 311, loss : 5.383e-04
iter: 312, loss : 3.752e-04
iter: 313, loss : 4.225e-03
iter: 314, loss : 8.063e-03
iter: 315, loss : 2.381e-03
iter: 316, loss : 6.093e-04
iter: 317, loss : 6.627e-04
iter: 318, loss : 4.532e-03
iter: 319, loss : 7.295e-04
iter: 320, loss : 9.867e-04
iter: 321, loss : 5.530e-04
iter: 322, loss : 4.239e-04
iter: 323, loss : 2.553e-03
iter: 324, loss : 8.366e-04
iter: 325, loss : 6.864e-04
iter: 326, loss : 2.716e-03
iter: 327, loss : 9.536e-04
iter: 328, loss : 4.645e-04
iter: 329, loss : 7.834e-04
iter: 330, loss : 8.110e-04
iter: 331, loss : 9.790e-04
iter: 332, loss : 1.173e-03
iter: 333, loss : 6.873e-04
iter: 334, loss : 7.584e-04
iter: 335, loss : 4.656e-04
iter: 336, loss : 3.186e-04
iter: 337, loss : 1.074e-02
iter: 338, loss : 3.938e-04
iter: 339, loss : 7.422e-04
iter: 340, loss : 5.323e-04
iter: 341, loss : 7.405e-04
iter: 342, loss : 3.713e-04
iter: 343, loss : 4.827e-04
iter: 344, loss : 1.269e-02
iter: 345, loss : 6.315e-04
iter: 346, loss : 7.923e-04
iter: 347, loss : 6.564e-04
iter: 348, loss : 5.756e-04
iter: 349, loss : 2.549e-04
iter: 350, loss : 6.908e-04
iter: 351, loss : 1.571e-04
iter: 352, loss : 4.829e-04
iter: 353, loss : 4.575e-04
iter: 354, loss : 8.496e-04
iter: 355, loss : 5.068e-04
iter: 356, loss : 6.456e-04
iter: 357, loss : 5.279e-03
iter: 358, loss : 2.923e-04
iter: 359, loss : 3.075e-04
iter: 360, loss : 8.342e-04
iter: 361, loss : 8.196e-03
iter: 362, loss : 5.688e-03
iter: 363, loss : 6.296e-04
iter: 364, loss : 5.528e-04
iter: 365, loss : 8.663e-03
iter: 366, loss : 3.437e-04
iter: 367, loss : 8.826e-04
iter: 368, loss : 4.483e-04
iter: 369, loss : 4.501e-04
iter: 370, loss : 8.763e-04
iter: 371, loss : 2.902e-03
iter: 372, loss : 5.582e-04
iter: 373, loss : 8.379e-04
iter: 374, loss : 2.712e-03
iter: 375, loss : 5.624e-04
iter: 376, loss : 4.022e-03
iter: 377, loss : 6.248e-04
iter: 378, loss : 2.330e-04
iter: 379, loss : 2.481e-03
iter: 380, loss : 3.546e-03
iter: 381, loss : 5.357e-04
iter: 382, loss : 1.038e-02
iter: 383, loss : 6.464e-04
iter: 384, loss : 5.558e-03
iter: 385, loss : 5.830e-04
iter: 386, loss : 7.630e-04
iter: 387, loss : 4.916e-04
iter: 388, loss : 5.360e-04
iter: 389, loss : 2.040e-03
iter: 390, loss : 3.001e-03
iter: 391, loss : 8.405e-04
iter: 392, loss : 6.699e-04
iter: 393, loss : 1.228e-02
iter: 394, loss : 6.047e-04
iter: 395, loss : 8.576e-04
iter: 396, loss : 6.760e-04
iter: 397, loss : 3.067e-04
iter: 398, loss : 4.798e-04
iter: 399, loss : 4.712e-04
iter: 400, loss : 5.811e-03
iter: 401, loss : 4.914e-04
iter: 402, loss : 1.080e-02
iter: 403, loss : 2.058e-04
iter: 404, loss : 4.835e-04
iter: 405, loss : 7.063e-04
iter: 406, loss : 1.200e-03
iter: 407, loss : 3.067e-04
iter: 408, loss : 2.801e-03
iter: 409, loss : 6.814e-03
iter: 410, loss : 1.621e-03
iter: 411, loss : 7.820e-03
iter: 412, loss : 6.902e-04
iter: 413, loss : 8.853e-04
iter: 414, loss : 6.205e-03
iter: 415, loss : 5.561e-03
iter: 416, loss : 7.859e-04
iter: 417, loss : 7.628e-04
iter: 418, loss : 8.062e-04
iter: 419, loss : 1.051e-02
iter: 420, loss : 1.018e-03
iter: 421, loss : 3.040e-03
iter: 422, loss : 2.814e-03
iter: 423, loss : 9.394e-04
iter: 424, loss : 9.748e-04
iter: 425, loss : 6.530e-04
iter: 426, loss : 2.495e-03
iter: 427, loss : 8.150e-04
iter: 428, loss : 1.916e-03
iter: 429, loss : 1.051e-03
iter: 430, loss : 5.464e-04
iter: 431, loss : 2.269e-03
iter: 432, loss : 5.562e-04
iter: 433, loss : 1.573e-03
iter: 434, loss : 2.632e-04
iter: 435, loss : 7.110e-03
iter: 436, loss : 7.470e-04
iter: 437, loss : 7.358e-04
iter: 438, loss : 4.750e-04
iter: 439, loss : 2.834e-04
iter: 440, loss : 4.540e-04
iter: 441, loss : 2.825e-04
iter: 442, loss : 2.508e-04
iter: 443, loss : 4.949e-04
iter: 444, loss : 4.316e-04
iter: 445, loss : 6.207e-03
iter: 446, loss : 3.961e-04
iter: 447, loss : 7.759e-04
iter: 448, loss : 2.112e-03
iter: 449, loss : 4.243e-03
iter: 450, loss : 6.466e-04
iter: 451, loss : 8.274e-04
iter: 452, loss : 7.750e-04
iter: 453, loss : 5.067e-03
iter: 454, loss : 4.104e-03
iter: 455, loss : 7.277e-04
iter: 456, loss : 7.401e-04
iter: 457, loss : 2.611e-04
iter: 458, loss : 2.943e-04
iter: 459, loss : 3.093e-03
iter: 460, loss : 4.287e-04
iter: 461, loss : 2.792e-03
iter: 462, loss : 4.419e-03
iter: 463, loss : 4.023e-04
iter: 464, loss : 5.246e-04
iter: 465, loss : 3.982e-04
iter: 466, loss : 4.612e-04
iter: 467, loss : 4.835e-03
iter: 468, loss : 4.538e-04
iter: 469, loss : 2.974e-04
iter: 470, loss : 5.362e-03
iter: 471, loss : 5.992e-04
iter: 472, loss : 1.084e-03
iter: 473, loss : 4.711e-04
iter: 474, loss : 2.362e-04
iter: 475, loss : 7.709e-04
iter: 476, loss : 1.640e-03
iter: 477, loss : 3.055e-03
iter: 478, loss : 2.751e-04
iter: 479, loss : 9.992e-04
iter: 480, loss : 2.754e-03
iter: 481, loss : 7.271e-04
iter: 482, loss : 4.728e-04
iter: 483, loss : 5.768e-04
iter: 484, loss : 3.287e-03
iter: 485, loss : 6.466e-04
iter: 486, loss : 7.336e-04
iter: 487, loss : 4.907e-04
iter: 488, loss : 5.561e-03
iter: 489, loss : 5.513e-04
iter: 490, loss : 4.978e-04
iter: 491, loss : 1.793e-03
iter: 492, loss : 2.610e-04
iter: 493, loss : 3.791e-04
iter: 494, loss : 7.183e-04
iter: 495, loss : 2.958e-04
iter: 496, loss : 3.933e-04
iter: 497, loss : 7.353e-04
iter: 498, loss : 3.130e-03
iter: 499, loss : 6.546e-04
iter: 500, loss : 3.424e-04
iter: 501, loss : 9.268e-03
iter: 502, loss : 5.604e-04
iter: 503, loss : 5.311e-04
iter: 504, loss : 5.324e-04
iter: 505, loss : 3.172e-04
iter: 506, loss : 4.020e-04
iter: 507, loss : 4.658e-04
iter: 508, loss : 1.744e-03
iter: 509, loss : 2.349e-04
iter: 510, loss : 3.767e-04
iter: 511, loss : 5.176e-04
iter: 512, loss : 4.965e-04
iter: 513, loss : 4.510e-04
iter: 514, loss : 5.088e-04
iter: 515, loss : 4.717e-04
iter: 516, loss : 4.900e-04
iter: 517, loss : 2.804e-03
iter: 518, loss : 1.434e-04
iter: 519, loss : 3.431e-04
iter: 520, loss : 4.102e-03
iter: 521, loss : 4.244e-04
iter: 522, loss : 5.643e-04
iter: 523, loss : 2.764e-03
iter: 524, loss : 8.773e-04
iter: 525, loss : 5.224e-04
iter: 526, loss : 2.636e-04
iter: 527, loss : 2.574e-03
iter: 528, loss : 9.503e-03
iter: 529, loss : 2.476e-04
iter: 530, loss : 4.202e-03
iter: 531, loss : 9.997e-04
iter: 532, loss : 8.264e-04
iter: 533, loss : 6.739e-04
iter: 534, loss : 7.174e-03
iter: 535, loss : 4.842e-03
iter: 536, loss : 8.819e-04
iter: 537, loss : 2.919e-04
iter: 538, loss : 9.122e-04
iter: 539, loss : 8.022e-04
iter: 540, loss : 4.657e-03
iter: 541, loss : 8.361e-04
iter: 542, loss : 9.406e-04
iter: 543, loss : 4.468e-04
iter: 544, loss : 6.601e-04
iter: 545, loss : 5.775e-04
iter: 546, loss : 3.866e-04
iter: 547, loss : 7.856e-03
iter: 548, loss : 5.463e-04
iter: 549, loss : 3.341e-03
iter: 550, loss : 3.736e-04
iter: 551, loss : 4.571e-04
iter: 552, loss : 7.097e-04
iter: 553, loss : 4.885e-04
iter: 554, loss : 3.825e-04
iter: 555, loss : 4.714e-03
iter: 556, loss : 4.168e-04
iter: 557, loss : 5.762e-04
iter: 558, loss : 3.755e-04
iter: 559, loss : 2.203e-04
iter: 560, loss : 1.132e-02
iter: 561, loss : 4.779e-04
iter: 562, loss : 1.678e-03
iter: 563, loss : 2.800e-04
iter: 564, loss : 7.261e-04
iter: 565, loss : 4.153e-04
iter: 566, loss : 3.471e-04
iter: 567, loss : 4.213e-04
iter: 568, loss : 5.809e-04
iter: 569, loss : 1.255e-02
iter: 570, loss : 1.967e-03
iter: 571, loss : 6.878e-03
iter: 572, loss : 4.192e-03
iter: 573, loss : 5.747e-04
iter: 574, loss : 6.941e-04
iter: 575, loss : 5.638e-04
iter: 576, loss : 1.220e-03
iter: 577, loss : 5.559e-03
iter: 578, loss : 1.042e-03
iter: 579, loss : 7.284e-04
iter: 580, loss : 3.062e-03
iter: 581, loss : 9.825e-04
iter: 582, loss : 2.490e-03
iter: 583, loss : 2.183e-03
iter: 584, loss : 8.358e-03
iter: 585, loss : 9.478e-04
iter: 586, loss : 6.392e-04
iter: 587, loss : 4.490e-04
iter: 588, loss : 2.541e-04
iter: 589, loss : 4.099e-04
iter: 590, loss : 3.016e-03
iter: 591, loss : 3.662e-04
iter: 592, loss : 4.854e-04
iter: 593, loss : 2.340e-03
iter: 594, loss : 4.958e-04
iter: 595, loss : 5.295e-04
iter: 596, loss : 5.910e-04
iter: 597, loss : 3.653e-04
iter: 598, loss : 5.269e-04
iter: 599, loss : 2.262e-04
iter: 600, loss : 4.363e-04
iter: 601, loss : 1.652e-04
iter: 602, loss : 2.467e-04
iter: 603, loss : 4.635e-04
iter: 604, loss : 1.333e-03
iter: 605, loss : 3.292e-04
iter: 606, loss : 3.823e-04
iter: 607, loss : 5.521e-03
iter: 608, loss : 4.546e-04
iter: 609, loss : 5.291e-04
iter: 610, loss : 7.404e-03
iter: 611, loss : 5.215e-03
iter: 612, loss : 2.912e-03
iter: 613, loss : 1.915e-03
iter: 614, loss : 3.955e-04
iter: 615, loss : 6.675e-03
iter: 616, loss : 4.020e-04
iter: 617, loss : 7.119e-04
iter: 618, loss : 5.077e-04
iter: 619, loss : 4.718e-04
iter: 620, loss : 5.527e-04
iter: 621, loss : 3.767e-04
iter: 622, loss : 2.140e-04
iter: 623, loss : 5.785e-04
iter: 624, loss : 4.526e-04
iter: 625, loss : 3.759e-04
iter: 626, loss : 1.718e-03
iter: 627, loss : 1.857e-04
iter: 628, loss : 1.773e-03
iter: 629, loss : 2.229e-04
iter: 630, loss : 4.195e-04
iter: 631, loss : 5.339e-03
iter: 632, loss : 2.010e-03
iter: 633, loss : 4.175e-04
iter: 634, loss : 5.407e-04
iter: 635, loss : 2.888e-03
iter: 636, loss : 2.880e-04
iter: 637, loss : 2.266e-04
iter: 638, loss : 4.182e-04
iter: 639, loss : 4.463e-04
iter: 640, loss : 5.058e-04
iter: 641, loss : 3.450e-04
iter: 642, loss : 5.223e-04
iter: 643, loss : 4.887e-04
iter: 644, loss : 2.182e-03
iter: 645, loss : 3.396e-04
iter: 646, loss : 2.434e-03
iter: 647, loss : 2.355e-04
iter: 648, loss : 3.240e-04
iter: 649, loss : 3.281e-04
iter: 650, loss : 2.675e-04
iter: 651, loss : 2.838e-04
iter: 652, loss : 1.599e-03
iter: 653, loss : 7.624e-03
iter: 654, loss : 4.167e-04
iter: 655, loss : 4.584e-04
iter: 656, loss : 5.058e-04
iter: 657, loss : 1.575e-03
iter: 658, loss : 4.065e-04
iter: 659, loss : 1.779e-04
iter: 660, loss : 2.679e-04
iter: 661, loss : 1.675e-03
iter: 662, loss : 3.379e-04
iter: 663, loss : 3.602e-04
iter: 664, loss : 3.656e-04
iter: 665, loss : 5.152e-04
iter: 666, loss : 4.079e-04
iter: 667, loss : 1.875e-04
iter: 668, loss : 4.806e-03
iter: 669, loss : 1.620e-03
iter: 670, loss : 7.459e-03
iter: 671, loss : 6.161e-03
iter: 672, loss : 1.218e-03
iter: 673, loss : 3.610e-04
iter: 674, loss : 5.290e-04
iter: 675, loss : 3.693e-04
iter: 676, loss : 1.058e-03
iter: 677, loss : 5.075e-04
iter: 678, loss : 1.495e-03
iter: 679, loss : 3.981e-04
iter: 680, loss : 3.965e-04
iter: 681, loss : 1.414e-03
iter: 682, loss : 6.861e-04
iter: 683, loss : 3.388e-03
iter: 684, loss : 6.787e-03
iter: 685, loss : 6.215e-04
iter: 686, loss : 4.366e-03
iter: 687, loss : 5.398e-04
iter: 688, loss : 1.412e-03
iter: 689, loss : 9.866e-04
iter: 690, loss : 4.948e-04
iter: 691, loss : 4.202e-04
iter: 692, loss : 5.059e-04
iter: 693, loss : 4.032e-04
iter: 694, loss : 2.771e-04
iter: 695, loss : 1.401e-03
iter: 696, loss : 9.388e-05
iter: 697, loss : 4.628e-04
iter: 698, loss : 4.270e-04
iter: 699, loss : 2.146e-04
iter: 700, loss : 3.733e-03
iter: 701, loss : 6.602e-03
iter: 702, loss : 5.258e-04
iter: 703, loss : 2.532e-03
iter: 704, loss : 4.584e-04
iter: 705, loss : 1.954e-03
iter: 706, loss : 1.667e-03
iter: 707, loss : 1.575e-03
iter: 708, loss : 1.635e-03
iter: 709, loss : 4.343e-03
iter: 710, loss : 5.685e-04
iter: 711, loss : 7.374e-04
iter: 712, loss : 1.437e-03
iter: 713, loss : 7.441e-04
iter: 714, loss : 2.085e-03
iter: 715, loss : 3.635e-03
iter: 716, loss : 2.392e-04
iter: 717, loss : 5.557e-04
iter: 718, loss : 5.486e-03
iter: 719, loss : 9.809e-03
iter: 720, loss : 9.391e-04
iter: 721, loss : 3.208e-04
iter: 722, loss : 6.685e-04
iter: 723, loss : 5.156e-03
iter: 724, loss : 2.603e-03
iter: 725, loss : 2.670e-04
iter: 726, loss : 5.014e-04
iter: 727, loss : 1.402e-03
iter: 728, loss : 2.006e-03
iter: 729, loss : 1.179e-03
iter: 730, loss : 6.979e-04
iter: 731, loss : 7.663e-03
iter: 732, loss : 7.706e-04
iter: 733, loss : 9.234e-04
iter: 734, loss : 5.498e-04
iter: 735, loss : 5.847e-04
iter: 736, loss : 1.621e-03
iter: 737, loss : 7.545e-03
iter: 738, loss : 1.021e-03
iter: 739, loss : 2.702e-04
iter: 740, loss : 3.617e-03
iter: 741, loss : 6.076e-04
iter: 742, loss : 1.034e-03
iter: 743, loss : 5.800e-04
iter: 744, loss : 7.131e-03
iter: 745, loss : 5.223e-04
iter: 746, loss : 3.513e-04
iter: 747, loss : 1.097e-03
iter: 748, loss : 1.101e-03
iter: 749, loss : 8.668e-04
iter: 750, loss : 1.181e-03
iter: 751, loss : 7.577e-04
iter: 752, loss : 1.211e-03
iter: 753, loss : 1.904e-03
iter: 754, loss : 4.094e-04
iter: 755, loss : 1.337e-03
iter: 756, loss : 1.377e-03
iter: 757, loss : 9.025e-04
iter: 758, loss : 4.918e-04
iter: 759, loss : 7.714e-03
iter: 760, loss : 4.576e-04
iter: 761, loss : 4.735e-03
iter: 762, loss : 7.606e-04
iter: 763, loss : 4.932e-04
iter: 764, loss : 6.636e-04
iter: 765, loss : 1.363e-03
iter: 766, loss : 3.618e-04
iter: 767, loss : 1.273e-03
iter: 768, loss : 4.343e-04
iter: 769, loss : 2.796e-04
iter: 770, loss : 1.198e-03
iter: 771, loss : 7.756e-03
iter: 772, loss : 7.271e-03
iter: 773, loss : 8.333e-04
iter: 774, loss : 3.170e-03
iter: 775, loss : 8.898e-04
iter: 776, loss : 8.782e-04
iter: 777, loss : 3.070e-03
iter: 778, loss : 6.633e-04
iter: 779, loss : 5.422e-04
iter: 780, loss : 5.908e-04
iter: 781, loss : 1.520e-03
iter: 782, loss : 1.359e-03
iter: 783, loss : 4.199e-04
iter: 784, loss : 6.611e-04
iter: 785, loss : 8.771e-04
iter: 786, loss : 5.841e-03
iter: 787, loss : 5.279e-04
iter: 788, loss : 3.522e-03
iter: 789, loss : 4.829e-04
iter: 790, loss : 5.509e-04
iter: 791, loss : 1.322e-03
iter: 792, loss : 4.209e-04
iter: 793, loss : 3.490e-03
iter: 794, loss : 5.176e-04
iter: 795, loss : 7.687e-04
iter: 796, loss : 2.780e-04
iter: 797, loss : 3.704e-04
iter: 798, loss : 3.543e-04
iter: 799, loss : 3.633e-04
iter: 800, loss : 3.177e-03
iter: 801, loss : 1.520e-04
iter: 802, loss : 5.123e-04
iter: 803, loss : 3.769e-04
iter: 804, loss : 1.377e-03
iter: 805, loss : 1.964e-04
iter: 806, loss : 1.473e-03
iter: 807, loss : 4.785e-04
iter: 808, loss : 3.154e-03
iter: 809, loss : 6.010e-04
iter: 810, loss : 4.840e-04
iter: 811, loss : 1.704e-03
iter: 812, loss : 2.225e-04
iter: 813, loss : 2.006e-04
iter: 814, loss : 1.201e-03
iter: 815, loss : 9.648e-04
iter: 816, loss : 2.819e-03
iter: 817, loss : 3.111e-04
iter: 818, loss : 1.279e-03
iter: 819, loss : 3.917e-04
iter: 820, loss : 5.726e-03
iter: 821, loss : 3.926e-04
iter: 822, loss : 6.569e-04
iter: 823, loss : 5.256e-04
iter: 824, loss : 7.571e-03
iter: 825, loss : 2.930e-04
iter: 826, loss : 7.372e-04
iter: 827, loss : 6.551e-04
iter: 828, loss : 1.862e-03
iter: 829, loss : 1.924e-04
iter: 830, loss : 1.754e-04
iter: 831, loss : 6.093e-04
iter: 832, loss : 3.153e-04
iter: 833, loss : 2.706e-04
iter: 834, loss : 3.527e-04
iter: 835, loss : 3.649e-04
iter: 836, loss : 4.676e-04
iter: 837, loss : 1.941e-04
iter: 838, loss : 2.104e-04
iter: 839, loss : 4.131e-04
iter: 840, loss : 1.001e-04
iter: 841, loss : 2.885e-04
iter: 842, loss : 2.296e-04
iter: 843, loss : 1.475e-04
iter: 844, loss : 5.409e-03
iter: 845, loss : 4.260e-04
iter: 846, loss : 1.313e-03
iter: 847, loss : 1.271e-03
iter: 848, loss : 1.468e-04
iter: 849, loss : 3.959e-04
iter: 850, loss : 3.107e-04
iter: 851, loss : 3.090e-04
iter: 852, loss : 3.617e-03
iter: 853, loss : 2.192e-03
iter: 854, loss : 5.216e-04
iter: 855, loss : 2.905e-04
iter: 856, loss : 5.455e-03
iter: 857, loss : 5.778e-04
iter: 858, loss : 3.197e-04
iter: 859, loss : 2.706e-04
iter: 860, loss : 5.220e-03
iter: 861, loss : 4.769e-04
iter: 862, loss : 8.475e-04
iter: 863, loss : 5.461e-04
iter: 864, loss : 6.180e-04
iter: 865, loss : 2.939e-04
iter: 866, loss : 4.177e-04
iter: 867, loss : 1.470e-03
iter: 868, loss : 1.699e-03
iter: 869, loss : 2.115e-04
iter: 870, loss : 3.623e-04
iter: 871, loss : 1.503e-03
iter: 872, loss : 3.453e-04
iter: 873, loss : 1.118e-03
iter: 874, loss : 3.065e-04
iter: 875, loss : 7.028e-04
iter: 876, loss : 4.282e-03
iter: 877, loss : 5.989e-04
iter: 878, loss : 7.334e-04
iter: 879, loss : 3.466e-04
iter: 880, loss : 2.177e-04
iter: 881, loss : 7.656e-05
iter: 882, loss : 5.379e-03
iter: 883, loss : 2.413e-04
iter: 884, loss : 3.033e-04
iter: 885, loss : 5.622e-04
iter: 886, loss : 1.476e-03
iter: 887, loss : 1.883e-04
iter: 888, loss : 5.389e-04
iter: 889, loss : 8.536e-04
iter: 890, loss : 3.124e-04
iter: 891, loss : 1.202e-03
iter: 892, loss : 2.876e-04
iter: 893, loss : 1.651e-04
iter: 894, loss : 1.617e-04
iter: 895, loss : 3.692e-04
iter: 896, loss : 3.433e-03
iter: 897, loss : 3.014e-03
iter: 898, loss : 5.034e-04
iter: 899, loss : 2.744e-04
iter: 900, loss : 3.209e-04
iter: 901, loss : 2.697e-03
iter: 902, loss : 2.571e-04
iter: 903, loss : 3.672e-03
iter: 904, loss : 2.322e-04
iter: 905, loss : 2.166e-03
iter: 906, loss : 7.054e-04
iter: 907, loss : 2.439e-04
iter: 908, loss : 1.426e-03
iter: 909, loss : 1.489e-04
iter: 910, loss : 2.198e-03
iter: 911, loss : 7.379e-03
iter: 912, loss : 7.542e-04
iter: 913, loss : 9.305e-04
iter: 914, loss : 2.985e-03
iter: 915, loss : 7.546e-04
iter: 916, loss : 1.521e-03
iter: 917, loss : 1.570e-03
iter: 918, loss : 6.508e-04
iter: 919, loss : 5.814e-04
iter: 920, loss : 1.723e-03
iter: 921, loss : 7.491e-04
iter: 922, loss : 1.121e-04
iter: 923, loss : 2.123e-03
iter: 924, loss : 2.751e-04
iter: 925, loss : 8.075e-04
iter: 926, loss : 3.197e-04
iter: 927, loss : 2.120e-03
iter: 928, loss : 6.271e-04
iter: 929, loss : 2.785e-04
iter: 930, loss : 2.478e-03
iter: 931, loss : 3.851e-04
iter: 932, loss : 1.412e-04
iter: 933, loss : 1.110e-04
iter: 934, loss : 4.650e-03
iter: 935, loss : 5.884e-03
iter: 936, loss : 5.658e-04
iter: 937, loss : 2.142e-03
iter: 938, loss : 7.044e-04
iter: 939, loss : 5.338e-04
iter: 940, loss : 3.294e-04
iter: 941, loss : 1.077e-03
iter: 942, loss : 4.974e-04
iter: 943, loss : 2.117e-04
iter: 944, loss : 2.662e-04
iter: 945, loss : 2.551e-04
iter: 946, loss : 3.183e-04
iter: 947, loss : 1.228e-03
iter: 948, loss : 3.342e-04
iter: 949, loss : 4.114e-04
iter: 950, loss : 2.139e-04
iter: 951, loss : 1.312e-03
iter: 952, loss : 7.493e-04
iter: 953, loss : 4.018e-03
iter: 954, loss : 1.117e-03
iter: 955, loss : 1.698e-04
iter: 956, loss : 1.512e-03
iter: 957, loss : 6.322e-04
iter: 958, loss : 2.629e-04
iter: 959, loss : 6.891e-03
iter: 960, loss : 4.872e-04
iter: 961, loss : 2.020e-04
iter: 962, loss : 3.192e-04
iter: 963, loss : 4.617e-04
iter: 964, loss : 2.075e-03
iter: 965, loss : 1.580e-03
iter: 966, loss : 9.117e-04
iter: 967, loss : 9.490e-04
iter: 968, loss : 1.858e-04
iter: 969, loss : 9.109e-04
iter: 970, loss : 4.611e-04
iter: 971, loss : 1.418e-04
iter: 972, loss : 4.673e-04
iter: 973, loss : 2.253e-04
iter: 974, loss : 2.381e-03
iter: 975, loss : 5.485e-04
iter: 976, loss : 3.584e-03
iter: 977, loss : 4.087e-04
iter: 978, loss : 7.855e-04
iter: 979, loss : 1.331e-04
iter: 980, loss : 3.512e-04
iter: 981, loss : 2.177e-03
iter: 982, loss : 7.488e-04
iter: 983, loss : 3.060e-04
iter: 984, loss : 3.686e-04
iter: 985, loss : 1.344e-04
iter: 986, loss : 3.666e-04
iter: 987, loss : 2.836e-04
iter: 988, loss : 1.253e-03
iter: 989, loss : 3.748e-04
iter: 990, loss : 9.793e-04
iter: 991, loss : 1.964e-04
iter: 992, loss : 3.085e-04
iter: 993, loss : 1.809e-03
iter: 994, loss : 3.614e-04
iter: 995, loss : 3.557e-04
iter: 996, loss : 4.850e-04
iter: 997, loss : 2.295e-04
iter: 998, loss : 5.325e-04
iter: 999, loss : 5.115e-04
iter: 1000, loss : 3.723e-04
iter: 1001, loss : 3.281e-04
iter: 1002, loss : 3.281e-04
iter: 1003, loss : 2.834e-04
iter: 1004, loss : 2.062e-04
iter: 1005, loss : 5.414e-03
iter: 1006, loss : 1.875e-04
iter: 1007, loss : 1.838e-04
iter: 1008, loss : 2.986e-03
iter: 1009, loss : 2.381e-04
iter: 1010, loss : 3.248e-04
iter: 1011, loss : 9.244e-04
iter: 1012, loss : 2.585e-04
iter: 1013, loss : 1.576e-04
iter: 1014, loss : 1.292e-04
iter: 1015, loss : 3.390e-04
iter: 1016, loss : 3.141e-04
iter: 1017, loss : 4.462e-03
iter: 1018, loss : 1.300e-03
iter: 1019, loss : 2.387e-03
iter: 1020, loss : 9.581e-04
iter: 1021, loss : 2.979e-04
iter: 1022, loss : 3.687e-04
iter: 1023, loss : 4.779e-04
iter: 1024, loss : 1.865e-03
iter: 1025, loss : 1.588e-03
iter: 1026, loss : 4.497e-04
iter: 1027, loss : 1.111e-03
iter: 1028, loss : 4.367e-04
iter: 1029, loss : 1.975e-04
iter: 1030, loss : 2.539e-03
iter: 1031, loss : 6.340e-04
iter: 1032, loss : 6.316e-04
iter: 1033, loss : 5.722e-04
iter: 1034, loss : 3.225e-03
iter: 1035, loss : 7.138e-04
iter: 1036, loss : 2.772e-03
iter: 1037, loss : 8.316e-04
iter: 1038, loss : 5.573e-04
iter: 1039, loss : 4.021e-04
iter: 1040, loss : 1.808e-03
iter: 1041, loss : 2.170e-04
iter: 1042, loss : 6.217e-03
iter: 1043, loss : 2.175e-04
iter: 1044, loss : 8.024e-04
iter: 1045, loss : 7.390e-04
iter: 1046, loss : 2.128e-04
iter: 1047, loss : 5.745e-04
iter: 1048, loss : 9.404e-04
iter: 1049, loss : 2.264e-04
iter: 1050, loss : 3.625e-04
iter: 1051, loss : 2.193e-04
iter: 1052, loss : 5.617e-04
iter: 1053, loss : 2.503e-04
iter: 1054, loss : 2.436e-04
iter: 1055, loss : 1.815e-04
iter: 1056, loss : 3.054e-04
iter: 1057, loss : 2.722e-04
iter: 1058, loss : 2.787e-04
iter: 1059, loss : 6.367e-03
iter: 1060, loss : 2.556e-04
iter: 1061, loss : 3.138e-04
iter: 1062, loss : 4.594e-03
iter: 1063, loss : 2.718e-04
iter: 1064, loss : 4.082e-04
iter: 1065, loss : 2.493e-04
iter: 1066, loss : 1.158e-04
iter: 1067, loss : 2.236e-03
iter: 1068, loss : 4.183e-04
iter: 1069, loss : 4.307e-04
iter: 1070, loss : 1.006e-03
iter: 1071, loss : 8.384e-04
iter: 1072, loss : 8.990e-04
iter: 1073, loss : 4.150e-04
iter: 1074, loss : 1.921e-03
iter: 1075, loss : 1.761e-04
iter: 1076, loss : 4.223e-04
iter: 1077, loss : 3.471e-04
iter: 1078, loss : 2.595e-04
iter: 1079, loss : 4.777e-04
iter: 1080, loss : 1.020e-03
iter: 1081, loss : 3.786e-04
iter: 1082, loss : 1.297e-04
iter: 1083, loss : 3.740e-03
iter: 1084, loss : 2.466e-04
iter: 1085, loss : 4.311e-04
iter: 1086, loss : 8.487e-04
iter: 1087, loss : 1.351e-04
iter: 1088, loss : 1.608e-04
iter: 1089, loss : 2.851e-04
iter: 1090, loss : 1.266e-03
iter: 1091, loss : 1.194e-04
iter: 1092, loss : 7.432e-05
iter: 1093, loss : 2.609e-04
iter: 1094, loss : 1.336e-04
iter: 1095, loss : 3.439e-04
iter: 1096, loss : 1.033e-04
iter: 1097, loss : 1.748e-04
iter: 1098, loss : 2.309e-04
iter: 1099, loss : 1.150e-04
iter: 1100, loss : 8.339e-05
iter: 1101, loss : 2.100e-04
iter: 1102, loss : 2.299e-04
iter: 1103, loss : 2.418e-04
iter: 1104, loss : 2.770e-03
iter: 1105, loss : 2.803e-04
iter: 1106, loss : 1.798e-03
iter: 1107, loss : 5.425e-04
iter: 1108, loss : 2.929e-04
iter: 1109, loss : 1.922e-04
iter: 1110, loss : 3.617e-03
iter: 1111, loss : 3.102e-03
iter: 1112, loss : 1.694e-03
iter: 1113, loss : 2.276e-04
iter: 1114, loss : 1.833e-03
iter: 1115, loss : 1.031e-03
iter: 1116, loss : 3.339e-04
iter: 1117, loss : 2.642e-04
iter: 1118, loss : 4.128e-04
iter: 1119, loss : 2.532e-04
iter: 1120, loss : 2.863e-04
iter: 1121, loss : 1.382e-03
iter: 1122, loss : 3.141e-04
iter: 1123, loss : 2.668e-04
iter: 1124, loss : 1.907e-04
iter: 1125, loss : 2.625e-04
iter: 1126, loss : 1.746e-04
iter: 1127, loss : 1.060e-03
iter: 1128, loss : 2.975e-04
iter: 1129, loss : 1.908e-04
iter: 1130, loss : 3.276e-04
iter: 1131, loss : 2.114e-04
iter: 1132, loss : 9.614e-04
iter: 1133, loss : 5.030e-03
iter: 1134, loss : 1.225e-03
iter: 1135, loss : 3.962e-04
iter: 1136, loss : 3.106e-04
iter: 1137, loss : 1.722e-04
iter: 1138, loss : 1.589e-04
iter: 1139, loss : 5.667e-04
iter: 1140, loss : 1.923e-04
iter: 1141, loss : 5.849e-04
iter: 1142, loss : 3.562e-04
iter: 1143, loss : 7.993e-04
iter: 1144, loss : 2.576e-04
iter: 1145, loss : 2.267e-04
iter: 1146, loss : 3.040e-03
iter: 1147, loss : 4.027e-04
iter: 1148, loss : 8.092e-04
iter: 1149, loss : 5.734e-03
iter: 1150, loss : 1.287e-03
iter: 1151, loss : 3.030e-03
iter: 1152, loss : 1.609e-04
iter: 1153, loss : 6.670e-04
iter: 1154, loss : 2.039e-03
iter: 1155, loss : 4.571e-04
iter: 1156, loss : 1.199e-03
iter: 1157, loss : 2.118e-04
iter: 1158, loss : 9.576e-04
iter: 1159, loss : 6.072e-04
iter: 1160, loss : 4.787e-04
iter: 1161, loss : 4.502e-04
iter: 1162, loss : 4.403e-04
iter: 1163, loss : 2.287e-04
iter: 1164, loss : 4.138e-04
iter: 1165, loss : 3.081e-04
iter: 1166, loss : 2.677e-04
iter: 1167, loss : 2.890e-04
iter: 1168, loss : 1.363e-04
iter: 1169, loss : 2.010e-04
iter: 1170, loss : 3.874e-03
iter: 1171, loss : 4.598e-04
iter: 1172, loss : 7.291e-04
iter: 1173, loss : 2.531e-03
iter: 1174, loss : 5.416e-04
iter: 1175, loss : 4.467e-04
iter: 1176, loss : 4.776e-04
iter: 1177, loss : 4.005e-04
iter: 1178, loss : 3.363e-04
iter: 1179, loss : 2.007e-04
iter: 1180, loss : 1.987e-04
iter: 1181, loss : 1.176e-04
iter: 1182, loss : 2.679e-04
iter: 1183, loss : 2.868e-04
iter: 1184, loss : 8.920e-04
iter: 1185, loss : 2.383e-04
iter: 1186, loss : 2.182e-04
iter: 1187, loss : 1.173e-03
iter: 1188, loss : 1.355e-04
iter: 1189, loss : 1.580e-03
iter: 1190, loss : 7.821e-04
iter: 1191, loss : 3.017e-04
iter: 1192, loss : 3.004e-04
iter: 1193, loss : 1.421e-04
iter: 1194, loss : 1.360e-04
iter: 1195, loss : 3.412e-04
iter: 1196, loss : 1.867e-04
iter: 1197, loss : 1.881e-04
iter: 1198, loss : 3.034e-04
iter: 1199, loss : 8.637e-04
iter: 1200, loss : 2.365e-04
iter: 1201, loss : 2.711e-04
iter: 1202, loss : 9.888e-05
iter: 1203, loss : 4.176e-04
iter: 1204, loss : 1.432e-04
iter: 1205, loss : 1.125e-04
iter: 1206, loss : 7.043e-04
iter: 1207, loss : 1.622e-04
iter: 1208, loss : 9.443e-05
iter: 1209, loss : 4.108e-04
iter: 1210, loss : 1.433e-04
iter: 1211, loss : 1.437e-04
iter: 1212, loss : 1.308e-03
iter: 1213, loss : 7.444e-04
iter: 1214, loss : 2.198e-04
iter: 1215, loss : 1.651e-04
iter: 1216, loss : 7.940e-04
iter: 1217, loss : 6.502e-04
iter: 1218, loss : 2.687e-04
iter: 1219, loss : 2.972e-03
iter: 1220, loss : 1.387e-03
iter: 1221, loss : 2.416e-04
iter: 1222, loss : 1.724e-04
iter: 1223, loss : 1.403e-03
iter: 1224, loss : 2.177e-04
iter: 1225, loss : 1.837e-04
iter: 1226, loss : 1.297e-04
iter: 1227, loss : 1.170e-03
iter: 1228, loss : 4.980e-04
iter: 1229, loss : 2.574e-04
iter: 1230, loss : 2.802e-04
iter: 1231, loss : 2.150e-04
iter: 1232, loss : 2.447e-04
iter: 1233, loss : 2.093e-04
iter: 1234, loss : 7.144e-04
iter: 1235, loss : 4.147e-04
iter: 1236, loss : 1.212e-04
iter: 1237, loss : 1.959e-04
iter: 1238, loss : 4.981e-03
iter: 1239, loss : 1.749e-04
iter: 1240, loss : 2.650e-04
iter: 1241, loss : 1.822e-04
iter: 1242, loss : 6.232e-04
iter: 1243, loss : 2.068e-04
iter: 1244, loss : 2.314e-04
iter: 1245, loss : 1.406e-03
iter: 1246, loss : 2.366e-03
iter: 1247, loss : 1.849e-04
iter: 1248, loss : 3.879e-04
iter: 1249, loss : 9.219e-04
iter: 1250, loss : 2.126e-04
iter: 1251, loss : 2.516e-04
iter: 1252, loss : 1.780e-04
iter: 1253, loss : 2.071e-03
iter: 1254, loss : 1.432e-04
iter: 1255, loss : 1.289e-04
iter: 1256, loss : 2.397e-04
iter: 1257, loss : 8.847e-04
iter: 1258, loss : 3.812e-04
iter: 1259, loss : 3.996e-03
iter: 1260, loss : 9.261e-04
iter: 1261, loss : 4.321e-04
iter: 1262, loss : 8.503e-04
iter: 1263, loss : 1.495e-03
iter: 1264, loss : 4.705e-03
iter: 1265, loss : 3.354e-04
iter: 1266, loss : 9.612e-04
iter: 1267, loss : 3.024e-04
iter: 1268, loss : 2.818e-04
iter: 1269, loss : 1.311e-04
iter: 1270, loss : 4.472e-04
iter: 1271, loss : 2.608e-04
iter: 1272, loss : 2.381e-04
iter: 1273, loss : 3.784e-04
iter: 1274, loss : 1.884e-04
iter: 1275, loss : 1.549e-04
iter: 1276, loss : 2.750e-03
iter: 1277, loss : 3.836e-04
iter: 1278, loss : 2.414e-04
iter: 1279, loss : 1.182e-04
iter: 1280, loss : 1.229e-03
iter: 1281, loss : 3.156e-04
iter: 1282, loss : 3.217e-04
iter: 1283, loss : 1.691e-04
iter: 1284, loss : 1.942e-04
iter: 1285, loss : 4.859e-04
iter: 1286, loss : 5.523e-04
iter: 1287, loss : 1.641e-04
iter: 1288, loss : 1.446e-04
iter: 1289, loss : 1.390e-04
iter: 1290, loss : 1.236e-04
iter: 1291, loss : 1.605e-04
iter: 1292, loss : 2.872e-04
iter: 1293, loss : 1.396e-04
iter: 1294, loss : 1.299e-04
iter: 1295, loss : 1.022e-03
iter: 1296, loss : 1.127e-03
iter: 1297, loss : 3.394e-04
iter: 1298, loss : 1.832e-04
iter: 1299, loss : 2.703e-04
iter: 1300, loss : 2.259e-03
iter: 1301, loss : 3.114e-04
iter: 1302, loss : 9.083e-05
iter: 1303, loss : 1.626e-04
iter: 1304, loss : 2.407e-04
iter: 1305, loss : 7.967e-05
iter: 1306, loss : 3.313e-03
iter: 1307, loss : 4.573e-05
iter: 1308, loss : 2.180e-04
iter: 1309, loss : 9.714e-04
iter: 1310, loss : 2.969e-03
iter: 1311, loss : 4.000e-04
iter: 1312, loss : 2.298e-04
iter: 1313, loss : 5.576e-03
iter: 1314, loss : 4.061e-04
iter: 1315, loss : 5.625e-04
iter: 1316, loss : 3.795e-03
iter: 1317, loss : 1.288e-03
iter: 1318, loss : 3.439e-04
iter: 1319, loss : 1.545e-04
iter: 1320, loss : 4.036e-04
iter: 1321, loss : 1.830e-03
iter: 1322, loss : 2.750e-03
iter: 1323, loss : 6.247e-04
iter: 1324, loss : 2.604e-04
iter: 1325, loss : 2.615e-04
iter: 1326, loss : 2.678e-04
iter: 1327, loss : 1.241e-04
iter: 1328, loss : 1.854e-04
iter: 1329, loss : 2.833e-03
iter: 1330, loss : 1.193e-03
iter: 1331, loss : 2.433e-03
iter: 1332, loss : 5.531e-04
iter: 1333, loss : 2.923e-04
iter: 1334, loss : 2.873e-04
iter: 1335, loss : 1.125e-03
iter: 1336, loss : 1.432e-03
iter: 1337, loss : 3.722e-03
iter: 1338, loss : 1.857e-03
iter: 1339, loss : 4.969e-04
iter: 1340, loss : 6.634e-04
iter: 1341, loss : 6.917e-04
iter: 1342, loss : 2.582e-04
iter: 1343, loss : 1.394e-04
iter: 1344, loss : 2.103e-04
iter: 1345, loss : 1.186e-03
iter: 1346, loss : 5.208e-04
iter: 1347, loss : 6.003e-04
iter: 1348, loss : 2.744e-04
iter: 1349, loss : 2.008e-04
iter: 1350, loss : 1.475e-04
iter: 1351, loss : 1.047e-04
iter: 1352, loss : 2.962e-04
iter: 1353, loss : 1.758e-04
iter: 1354, loss : 2.771e-04
iter: 1355, loss : 4.318e-04
iter: 1356, loss : 2.558e-04
iter: 1357, loss : 1.696e-04
iter: 1358, loss : 2.085e-04
iter: 1359, loss : 3.002e-04
iter: 1360, loss : 2.697e-04
iter: 1361, loss : 2.218e-03
iter: 1362, loss : 1.462e-03
iter: 1363, loss : 6.967e-04
iter: 1364, loss : 3.277e-04
iter: 1365, loss : 2.725e-04
iter: 1366, loss : 2.222e-04
iter: 1367, loss : 2.244e-03
iter: 1368, loss : 2.955e-04
iter: 1369, loss : 2.870e-03
iter: 1370, loss : 1.038e-03
iter: 1371, loss : 2.484e-04
iter: 1372, loss : 5.005e-04
iter: 1373, loss : 5.491e-04
iter: 1374, loss : 2.406e-04
iter: 1375, loss : 2.902e-03
iter: 1376, loss : 4.512e-04
iter: 1377, loss : 3.190e-03
iter: 1378, loss : 2.656e-04
iter: 1379, loss : 2.639e-04
iter: 1380, loss : 4.739e-04
iter: 1381, loss : 2.192e-04
iter: 1382, loss : 2.929e-04
iter: 1383, loss : 9.196e-04
iter: 1384, loss : 5.586e-04
iter: 1385, loss : 2.783e-03
iter: 1386, loss : 2.652e-04
iter: 1387, loss : 2.163e-04
iter: 1388, loss : 1.653e-04
iter: 1389, loss : 2.483e-04
iter: 1390, loss : 1.463e-04
iter: 1391, loss : 3.539e-04
iter: 1392, loss : 3.026e-04
iter: 1393, loss : 6.102e-03
iter: 1394, loss : 3.983e-04
iter: 1395, loss : 2.911e-04
iter: 1396, loss : 4.543e-04
iter: 1397, loss : 2.649e-04
iter: 1398, loss : 6.894e-04
iter: 1399, loss : 1.646e-03
iter: 1400, loss : 6.091e-04
iter: 1401, loss : 3.131e-04
iter: 1402, loss : 1.933e-04
iter: 1403, loss : 1.637e-03
iter: 1404, loss : 3.047e-04
iter: 1405, loss : 6.757e-04
iter: 1406, loss : 4.223e-04
iter: 1407, loss : 1.794e-03
iter: 1408, loss : 3.083e-04
iter: 1409, loss : 5.507e-04
iter: 1410, loss : 4.846e-03
iter: 1411, loss : 5.193e-04
iter: 1412, loss : 1.920e-04
iter: 1413, loss : 8.967e-04
iter: 1414, loss : 2.200e-04
iter: 1415, loss : 3.418e-04
iter: 1416, loss : 7.978e-04
iter: 1417, loss : 2.147e-04
iter: 1418, loss : 9.839e-04
iter: 1419, loss : 1.211e-03
iter: 1420, loss : 3.015e-04
iter: 1421, loss : 6.713e-04
iter: 1422, loss : 2.162e-04
iter: 1423, loss : 7.873e-04
iter: 1424, loss : 5.048e-03
iter: 1425, loss : 2.714e-04
iter: 1426, loss : 1.267e-04
iter: 1427, loss : 2.588e-04
iter: 1428, loss : 4.046e-04
iter: 1429, loss : 4.626e-04
iter: 1430, loss : 2.772e-04
iter: 1431, loss : 1.904e-04
iter: 1432, loss : 1.555e-03
iter: 1433, loss : 2.759e-04
iter: 1434, loss : 1.407e-04
iter: 1435, loss : 1.998e-04
iter: 1436, loss : 5.115e-03
iter: 1437, loss : 2.090e-04
iter: 1438, loss : 7.639e-05
iter: 1439, loss : 2.332e-04
iter: 1440, loss : 2.448e-04
iter: 1441, loss : 1.875e-04
iter: 1442, loss : 1.196e-03
iter: 1443, loss : 1.512e-04
iter: 1444, loss : 3.918e-04
iter: 1445, loss : 3.703e-04
iter: 1446, loss : 1.289e-04
iter: 1447, loss : 1.715e-04
iter: 1448, loss : 7.776e-04
iter: 1449, loss : 1.957e-04
iter: 1450, loss : 1.793e-04
iter: 1451, loss : 1.192e-04
iter: 1452, loss : 1.557e-04
iter: 1453, loss : 2.220e-03
iter: 1454, loss : 9.543e-04
iter: 1455, loss : 4.946e-04
iter: 1456, loss : 1.125e-03
iter: 1457, loss : 1.171e-04
iter: 1458, loss : 8.057e-04
iter: 1459, loss : 2.938e-04
iter: 1460, loss : 6.166e-04
iter: 1461, loss : 3.577e-03
iter: 1462, loss : 1.558e-04
iter: 1463, loss : 1.198e-03
iter: 1464, loss : 1.498e-03
iter: 1465, loss : 4.559e-03
iter: 1466, loss : 5.232e-04
iter: 1467, loss : 1.263e-03
iter: 1468, loss : 1.807e-04
iter: 1469, loss : 3.180e-04
iter: 1470, loss : 5.821e-04
iter: 1471, loss : 8.332e-04
iter: 1472, loss : 7.287e-04
iter: 1473, loss : 2.672e-04
iter: 1474, loss : 3.061e-04
iter: 1475, loss : 2.347e-04
iter: 1476, loss : 3.075e-04
iter: 1477, loss : 9.677e-04
iter: 1478, loss : 2.468e-04
iter: 1479, loss : 2.301e-04
iter: 1480, loss : 1.661e-04
iter: 1481, loss : 1.759e-03
iter: 1482, loss : 1.428e-03
iter: 1483, loss : 4.262e-04
iter: 1484, loss : 2.922e-04
iter: 1485, loss : 4.184e-04
iter: 1486, loss : 3.916e-04
iter: 1487, loss : 2.177e-04
iter: 1488, loss : 3.584e-04
iter: 1489, loss : 2.511e-03
iter: 1490, loss : 3.123e-04
iter: 1491, loss : 3.161e-03
iter: 1492, loss : 4.131e-03
iter: 1493, loss : 4.383e-04
iter: 1494, loss : 6.799e-04
iter: 1495, loss : 2.575e-04
iter: 1496, loss : 8.280e-04
iter: 1497, loss : 3.209e-04
iter: 1498, loss : 3.069e-04
iter: 1499, loss : 3.887e-04
iter: 1500, loss : 5.171e-04
iter: 1501, loss : 3.545e-04
iter: 1502, loss : 1.928e-03
iter: 1503, loss : 2.872e-04
iter: 1504, loss : 1.890e-03
iter: 1505, loss : 1.057e-03
iter: 1506, loss : 1.552e-04
iter: 1507, loss : 1.913e-04
iter: 1508, loss : 4.363e-03
iter: 1509, loss : 2.675e-04
iter: 1510, loss : 1.858e-04
iter: 1511, loss : 1.848e-03
iter: 1512, loss : 2.050e-04
iter: 1513, loss : 4.824e-04
iter: 1514, loss : 1.485e-03
iter: 1515, loss : 3.397e-04
iter: 1516, loss : 2.249e-04
iter: 1517, loss : 1.776e-04
iter: 1518, loss : 9.256e-05
iter: 1519, loss : 3.072e-03
iter: 1520, loss : 3.007e-04
iter: 1521, loss : 6.963e-04
iter: 1522, loss : 3.124e-04
iter: 1523, loss : 2.154e-04
iter: 1524, loss : 4.606e-04
iter: 1525, loss : 1.046e-03
iter: 1526, loss : 6.503e-05
iter: 1527, loss : 1.940e-04
iter: 1528, loss : 4.350e-03
iter: 1529, loss : 2.048e-04
iter: 1530, loss : 2.721e-04
iter: 1531, loss : 9.272e-04
iter: 1532, loss : 5.249e-05
iter: 1533, loss : 6.680e-04
iter: 1534, loss : 2.601e-04
iter: 1535, loss : 8.056e-04
iter: 1536, loss : 6.033e-04
iter: 1537, loss : 5.674e-04
iter: 1538, loss : 9.920e-05
iter: 1539, loss : 1.798e-04
iter: 1540, loss : 1.344e-03
iter: 1541, loss : 1.839e-04
iter: 1542, loss : 1.702e-04
iter: 1543, loss : 1.268e-03
iter: 1544, loss : 2.851e-04
iter: 1545, loss : 3.143e-03
iter: 1546, loss : 3.643e-04
iter: 1547, loss : 3.185e-04
iter: 1548, loss : 9.269e-04
iter: 1549, loss : 2.512e-04
iter: 1550, loss : 3.934e-04
iter: 1551, loss : 2.274e-03
iter: 1552, loss : 2.985e-04
iter: 1553, loss : 3.598e-04
iter: 1554, loss : 9.438e-05
iter: 1555, loss : 2.534e-04
iter: 1556, loss : 2.072e-04
iter: 1557, loss : 1.628e-03
iter: 1558, loss : 2.014e-04
iter: 1559, loss : 1.636e-03
iter: 1560, loss : 7.596e-04
iter: 1561, loss : 3.195e-04
iter: 1562, loss : 1.845e-03
iter: 1563, loss : 1.801e-04
iter: 1564, loss : 3.997e-04
iter: 1565, loss : 8.775e-04
iter: 1566, loss : 5.801e-04
iter: 1567, loss : 1.780e-04
iter: 1568, loss : 2.316e-03
iter: 1569, loss : 2.617e-03
iter: 1570, loss : 1.154e-03
iter: 1571, loss : 1.709e-04
iter: 1572, loss : 5.752e-04
iter: 1573, loss : 1.038e-04
iter: 1574, loss : 2.185e-03
iter: 1575, loss : 2.670e-04
iter: 1576, loss : 3.966e-04
iter: 1577, loss : 4.575e-04
iter: 1578, loss : 3.994e-04
iter: 1579, loss : 3.231e-04
iter: 1580, loss : 7.919e-04
iter: 1581, loss : 1.877e-04
iter: 1582, loss : 2.448e-04
iter: 1583, loss : 1.596e-04
iter: 1584, loss : 2.419e-04
iter: 1585, loss : 1.688e-04
iter: 1586, loss : 1.004e-04
iter: 1587, loss : 1.689e-04
iter: 1588, loss : 1.809e-04
iter: 1589, loss : 1.940e-04
iter: 1590, loss : 6.479e-05
iter: 1591, loss : 8.007e-04
iter: 1592, loss : 1.693e-04
iter: 1593, loss : 7.736e-05
iter: 1594, loss : 1.678e-04
iter: 1595, loss : 1.724e-04
iter: 1596, loss : 1.123e-04
iter: 1597, loss : 2.068e-03
iter: 1598, loss : 2.130e-04
iter: 1599, loss : 9.843e-05
iter: 1600, loss : 2.698e-04
iter: 1601, loss : 8.279e-05
iter: 1602, loss : 7.377e-04
iter: 1603, loss : 2.542e-04
iter: 1604, loss : 9.560e-04
iter: 1605, loss : 1.263e-04
iter: 1606, loss : 7.992e-05
iter: 1607, loss : 7.071e-04
iter: 1608, loss : 1.328e-04
iter: 1609, loss : 2.119e-04
iter: 1610, loss : 6.604e-04
iter: 1611, loss : 1.809e-04
iter: 1612, loss : 3.674e-03
iter: 1613, loss : 1.140e-03
iter: 1614, loss : 1.131e-04
iter: 1615, loss : 5.005e-04
iter: 1616, loss : 3.105e-04
iter: 1617, loss : 1.937e-03
iter: 1618, loss : 3.259e-04
iter: 1619, loss : 2.844e-03
iter: 1620, loss : 4.006e-04
iter: 1621, loss : 3.717e-04
iter: 1622, loss : 4.055e-04
iter: 1623, loss : 2.945e-04
iter: 1624, loss : 2.471e-04
iter: 1625, loss : 6.725e-04
iter: 1626, loss : 1.104e-04
iter: 1627, loss : 2.382e-04
iter: 1628, loss : 7.056e-05
iter: 1629, loss : 2.925e-04
iter: 1630, loss : 3.612e-04
iter: 1631, loss : 6.823e-04
iter: 1632, loss : 1.879e-03
iter: 1633, loss : 1.714e-04
iter: 1634, loss : 1.360e-04
iter: 1635, loss : 4.617e-04
iter: 1636, loss : 1.237e-04
iter: 1637, loss : 3.662e-04
iter: 1638, loss : 9.813e-05
iter: 1639, loss : 3.052e-04
iter: 1640, loss : 2.034e-04
iter: 1641, loss : 2.884e-04
iter: 1642, loss : 6.939e-05
iter: 1643, loss : 2.469e-03
iter: 1644, loss : 1.795e-03
iter: 1645, loss : 4.103e-04
iter: 1646, loss : 1.576e-04
iter: 1647, loss : 1.783e-04
iter: 1648, loss : 5.075e-04
iter: 1649, loss : 6.653e-05
iter: 1650, loss : 3.002e-04
iter: 1651, loss : 4.183e-03
iter: 1652, loss : 2.344e-04
iter: 1653, loss : 1.991e-03
iter: 1654, loss : 3.546e-04
iter: 1655, loss : 5.883e-04
iter: 1656, loss : 2.958e-04
iter: 1657, loss : 1.951e-03
iter: 1658, loss : 7.029e-04
iter: 1659, loss : 2.642e-04
iter: 1660, loss : 6.556e-04
iter: 1661, loss : 4.642e-04
iter: 1662, loss : 1.372e-04
iter: 1663, loss : 3.882e-04
iter: 1664, loss : 1.002e-03
iter: 1665, loss : 5.666e-04
iter: 1666, loss : 1.386e-04
iter: 1667, loss : 2.455e-04
iter: 1668, loss : 1.762e-04
iter: 1669, loss : 1.804e-04
iter: 1670, loss : 3.153e-04
iter: 1671, loss : 1.080e-04
iter: 1672, loss : 1.335e-04
iter: 1673, loss : 1.841e-04
iter: 1674, loss : 1.921e-04
iter: 1675, loss : 1.491e-03
iter: 1676, loss : 9.225e-04
iter: 1677, loss : 3.251e-04
iter: 1678, loss : 2.898e-04
iter: 1679, loss : 1.589e-04
iter: 1680, loss : 1.908e-03
iter: 1681, loss : 9.640e-04
iter: 1682, loss : 1.497e-04
iter: 1683, loss : 3.480e-04
iter: 1684, loss : 3.046e-04
iter: 1685, loss : 1.353e-04
iter: 1686, loss : 2.980e-04
iter: 1687, loss : 1.707e-03
iter: 1688, loss : 2.521e-04
iter: 1689, loss : 2.084e-04
iter: 1690, loss : 1.647e-03
iter: 1691, loss : 4.680e-04
iter: 1692, loss : 1.070e-03
iter: 1693, loss : 5.378e-04
iter: 1694, loss : 1.003e-04
iter: 1695, loss : 6.137e-04
iter: 1696, loss : 2.001e-04
iter: 1697, loss : 1.349e-04
iter: 1698, loss : 1.778e-04
iter: 1699, loss : 6.960e-04
iter: 1700, loss : 2.219e-04
iter: 1701, loss : 6.206e-04
iter: 1702, loss : 1.915e-04
iter: 1703, loss : 8.825e-05
iter: 1704, loss : 1.203e-03
iter: 1705, loss : 2.769e-04
iter: 1706, loss : 3.127e-03
iter: 1707, loss : 2.393e-04
iter: 1708, loss : 1.099e-03
iter: 1709, loss : 1.207e-03
iter: 1710, loss : 1.952e-04
iter: 1711, loss : 4.041e-04
iter: 1712, loss : 4.043e-03
iter: 1713, loss : 5.591e-04
iter: 1714, loss : 5.863e-04
iter: 1715, loss : 6.058e-04
iter: 1716, loss : 2.024e-03
iter: 1717, loss : 3.273e-04
iter: 1718, loss : 3.518e-04
iter: 1719, loss : 7.159e-04
iter: 1720, loss : 1.019e-03
iter: 1721, loss : 2.018e-04
iter: 1722, loss : 2.796e-04
iter: 1723, loss : 1.234e-03
iter: 1724, loss : 5.485e-04
iter: 1725, loss : 5.183e-04
iter: 1726, loss : 5.045e-04
iter: 1727, loss : 3.061e-04
iter: 1728, loss : 1.725e-04
iter: 1729, loss : 9.432e-05
iter: 1730, loss : 1.632e-03
iter: 1731, loss : 1.809e-03
iter: 1732, loss : 2.658e-04
iter: 1733, loss : 9.192e-04
iter: 1734, loss : 2.375e-04
iter: 1735, loss : 3.825e-04
iter: 1736, loss : 1.008e-04
iter: 1737, loss : 2.504e-03
iter: 1738, loss : 7.288e-04
iter: 1739, loss : 1.396e-04
iter: 1740, loss : 7.853e-04
iter: 1741, loss : 3.579e-04
iter: 1742, loss : 1.064e-03
iter: 1743, loss : 2.707e-03
iter: 1744, loss : 3.231e-04
iter: 1745, loss : 6.929e-04
iter: 1746, loss : 2.103e-04
iter: 1747, loss : 2.014e-04
iter: 1748, loss : 2.990e-04
iter: 1749, loss : 2.347e-03
iter: 1750, loss : 1.154e-03
iter: 1751, loss : 5.709e-04
iter: 1752, loss : 1.187e-03
iter: 1753, loss : 1.830e-04
iter: 1754, loss : 3.840e-04
iter: 1755, loss : 2.843e-04
iter: 1756, loss : 2.813e-04
iter: 1757, loss : 8.223e-04
iter: 1758, loss : 2.612e-04
iter: 1759, loss : 1.716e-04
iter: 1760, loss : 1.169e-04
iter: 1761, loss : 4.739e-04
iter: 1762, loss : 3.541e-03
iter: 1763, loss : 2.957e-04
iter: 1764, loss : 3.159e-04
iter: 1765, loss : 1.210e-04
iter: 1766, loss : 3.108e-04
iter: 1767, loss : 3.183e-04
iter: 1768, loss : 1.601e-03
iter: 1769, loss : 2.793e-04
iter: 1770, loss : 7.488e-05
iter: 1771, loss : 1.494e-04
iter: 1772, loss : 2.137e-04
iter: 1773, loss : 7.930e-04
iter: 1774, loss : 1.705e-03
iter: 1775, loss : 2.025e-04
iter: 1776, loss : 4.882e-04
iter: 1777, loss : 8.930e-04
iter: 1778, loss : 1.888e-04
iter: 1779, loss : 2.283e-04
iter: 1780, loss : 5.754e-05
iter: 1781, loss : 1.413e-04
iter: 1782, loss : 2.889e-04
iter: 1783, loss : 9.065e-04
iter: 1784, loss : 3.089e-04
iter: 1785, loss : 2.088e-04
iter: 1786, loss : 4.091e-03
iter: 1787, loss : 3.676e-04
iter: 1788, loss : 2.707e-04
iter: 1789, loss : 3.144e-04
iter: 1790, loss : 1.862e-04
iter: 1791, loss : 1.414e-04
iter: 1792, loss : 6.292e-04
iter: 1793, loss : 1.318e-03
iter: 1794, loss : 2.070e-04
iter: 1795, loss : 4.347e-04
iter: 1796, loss : 2.534e-04
iter: 1797, loss : 1.518e-04
iter: 1798, loss : 2.209e-04
iter: 1799, loss : 9.158e-05
iter: 1800, loss : 1.544e-04
iter: 1801, loss : 2.008e-04
iter: 1802, loss : 1.457e-04
iter: 1803, loss : 2.760e-04
iter: 1804, loss : 4.264e-03
iter: 1805, loss : 2.759e-04
iter: 1806, loss : 4.853e-04
iter: 1807, loss : 8.811e-04
iter: 1808, loss : 2.973e-04
iter: 1809, loss : 2.582e-04
iter: 1810, loss : 7.336e-04
iter: 1811, loss : 2.458e-04
iter: 1812, loss : 6.499e-04
iter: 1813, loss : 1.560e-04
iter: 1814, loss : 1.746e-04
iter: 1815, loss : 1.472e-03
iter: 1816, loss : 3.566e-04
iter: 1817, loss : 8.655e-04
iter: 1818, loss : 1.481e-04
iter: 1819, loss : 2.850e-04
iter: 1820, loss : 6.697e-04
iter: 1821, loss : 4.038e-04
iter: 1822, loss : 3.862e-04
iter: 1823, loss : 3.644e-04
iter: 1824, loss : 5.987e-04
iter: 1825, loss : 1.125e-04
iter: 1826, loss : 1.506e-04
iter: 1827, loss : 1.763e-03
iter: 1828, loss : 2.128e-04
iter: 1829, loss : 3.814e-04
iter: 1830, loss : 1.214e-04
iter: 1831, loss : 4.030e-04
iter: 1832, loss : 1.536e-04
iter: 1833, loss : 1.110e-04
iter: 1834, loss : 4.876e-04
iter: 1835, loss : 2.617e-04
iter: 1836, loss : 1.641e-03
iter: 1837, loss : 2.867e-03
iter: 1838, loss : 5.693e-04
iter: 1839, loss : 2.060e-04
iter: 1840, loss : 2.286e-04
iter: 1841, loss : 8.898e-05
iter: 1842, loss : 6.351e-04
iter: 1843, loss : 9.441e-05
iter: 1844, loss : 1.898e-04
iter: 1845, loss : 1.150e-04
iter: 1846, loss : 1.246e-04
iter: 1847, loss : 8.535e-05
iter: 1848, loss : 1.930e-04
iter: 1849, loss : 1.374e-03
iter: 1850, loss : 3.121e-04
iter: 1851, loss : 3.025e-03
iter: 1852, loss : 1.643e-04
iter: 1853, loss : 1.154e-04
iter: 1854, loss : 3.780e-03
iter: 1855, loss : 3.695e-04
iter: 1856, loss : 2.514e-04
iter: 1857, loss : 2.556e-03
iter: 1858, loss : 1.855e-04
iter: 1859, loss : 1.358e-04
iter: 1860, loss : 3.656e-04
iter: 1861, loss : 9.262e-04
iter: 1862, loss : 2.367e-04
iter: 1863, loss : 5.346e-05
iter: 1864, loss : 3.730e-03
iter: 1865, loss : 3.643e-04
iter: 1866, loss : 2.740e-04
iter: 1867, loss : 1.522e-04
iter: 1868, loss : 2.905e-04
iter: 1869, loss : 2.031e-04
iter: 1870, loss : 1.323e-04
iter: 1871, loss : 1.445e-03
iter: 1872, loss : 8.996e-05
iter: 1873, loss : 4.300e-04
iter: 1874, loss : 2.856e-04
iter: 1875, loss : 1.265e-04
iter: 1876, loss : 1.062e-04
iter: 1877, loss : 2.850e-03
iter: 1878, loss : 1.408e-04
iter: 1879, loss : 9.324e-04
iter: 1880, loss : 7.898e-04
iter: 1881, loss : 1.250e-03
iter: 1882, loss : 3.987e-04
iter: 1883, loss : 1.867e-03
iter: 1884, loss : 1.883e-04
iter: 1885, loss : 3.257e-04
iter: 1886, loss : 6.745e-05
iter: 1887, loss : 1.439e-04
iter: 1888, loss : 1.795e-04
iter: 1889, loss : 1.404e-04
iter: 1890, loss : 2.018e-04
iter: 1891, loss : 7.898e-04
iter: 1892, loss : 5.882e-04
iter: 1893, loss : 8.819e-05
iter: 1894, loss : 1.974e-03
iter: 1895, loss : 1.565e-04
iter: 1896, loss : 5.230e-04
iter: 1897, loss : 3.840e-04
iter: 1898, loss : 7.805e-04
iter: 1899, loss : 1.621e-04
iter: 1900, loss : 1.370e-04
iter: 1901, loss : 3.610e-03
iter: 1902, loss : 2.301e-04
iter: 1903, loss : 7.341e-05
iter: 1904, loss : 1.208e-03
iter: 1905, loss : 1.622e-04
iter: 1906, loss : 4.934e-04
iter: 1907, loss : 1.094e-04
iter: 1908, loss : 3.567e-03
iter: 1909, loss : 1.391e-04
iter: 1910, loss : 1.307e-04
iter: 1911, loss : 6.187e-04
iter: 1912, loss : 2.606e-04
iter: 1913, loss : 6.382e-05
iter: 1914, loss : 1.329e-03
iter: 1915, loss : 2.115e-04
iter: 1916, loss : 3.388e-04
iter: 1917, loss : 1.175e-04
iter: 1918, loss : 2.408e-04
iter: 1919, loss : 5.554e-05
iter: 1920, loss : 1.995e-04
iter: 1921, loss : 6.372e-04
iter: 1922, loss : 1.025e-04
iter: 1923, loss : 9.200e-05
iter: 1924, loss : 1.185e-04
iter: 1925, loss : 4.086e-03
iter: 1926, loss : 5.213e-04
iter: 1927, loss : 1.685e-04
iter: 1928, loss : 8.458e-04
iter: 1929, loss : 3.765e-04
iter: 1930, loss : 1.122e-03
iter: 1931, loss : 1.986e-04
iter: 1932, loss : 2.058e-04
iter: 1933, loss : 2.104e-04
iter: 1934, loss : 8.453e-04
iter: 1935, loss : 8.585e-04
iter: 1936, loss : 2.450e-04
iter: 1937, loss : 6.456e-04
iter: 1938, loss : 2.988e-04
iter: 1939, loss : 3.181e-03
iter: 1940, loss : 4.870e-04
iter: 1941, loss : 1.182e-03
iter: 1942, loss : 4.932e-04
iter: 1943, loss : 3.321e-04
iter: 1944, loss : 1.016e-04
iter: 1945, loss : 9.989e-04
iter: 1946, loss : 3.910e-04
iter: 1947, loss : 3.086e-03
iter: 1948, loss : 2.884e-04
iter: 1949, loss : 9.854e-05
iter: 1950, loss : 4.198e-04
iter: 1951, loss : 1.912e-04
iter: 1952, loss : 4.449e-04
iter: 1953, loss : 1.962e-04
iter: 1954, loss : 1.289e-04
iter: 1955, loss : 4.431e-05
iter: 1956, loss : 2.880e-04
iter: 1957, loss : 2.672e-04
iter: 1958, loss : 2.004e-04
iter: 1959, loss : 2.722e-03
iter: 1960, loss : 2.315e-04
iter: 1961, loss : 1.570e-04
iter: 1962, loss : 1.294e-04
iter: 1963, loss : 1.124e-04
iter: 1964, loss : 3.036e-03
iter: 1965, loss : 2.046e-04
iter: 1966, loss : 3.140e-04
iter: 1967, loss : 1.092e-03
iter: 1968, loss : 3.701e-03
iter: 1969, loss : 1.988e-03
iter: 1970, loss : 5.199e-04
iter: 1971, loss : 8.679e-04
iter: 1972, loss : 3.076e-04
iter: 1973, loss : 2.633e-04
iter: 1974, loss : 2.587e-04
iter: 1975, loss : 3.434e-04
iter: 1976, loss : 1.296e-04
iter: 1977, loss : 4.150e-04
iter: 1978, loss : 9.704e-05
iter: 1979, loss : 1.166e-04
iter: 1980, loss : 4.529e-04
iter: 1981, loss : 2.333e-04
iter: 1982, loss : 1.579e-04
iter: 1983, loss : 2.345e-04
iter: 1984, loss : 1.463e-03
iter: 1985, loss : 3.397e-04
iter: 1986, loss : 2.519e-04
iter: 1987, loss : 1.884e-03
iter: 1988, loss : 3.611e-04
iter: 1989, loss : 2.011e-04
iter: 1990, loss : 9.908e-04
iter: 1991, loss : 5.079e-04
iter: 1992, loss : 4.831e-04
iter: 1993, loss : 3.389e-03
iter: 1994, loss : 1.679e-04
iter: 1995, loss : 2.817e-04
iter: 1996, loss : 1.471e-04
iter: 1997, loss : 8.058e-05
iter: 1998, loss : 8.390e-05
iter: 1999, loss : 2.134e-04
iter: 2000, loss : 2.897e-04
iter: 2001, loss : 5.728e-04
iter: 2002, loss : 4.235e-03
iter: 2003, loss : 5.141e-04
iter: 2004, loss : 8.143e-04
iter: 2005, loss : 4.285e-04
iter: 2006, loss : 3.784e-04
iter: 2007, loss : 1.997e-04
iter: 2008, loss : 2.991e-04
iter: 2009, loss : 2.719e-04
iter: 2010, loss : 1.853e-04
iter: 2011, loss : 1.957e-04
iter: 2012, loss : 1.917e-04
iter: 2013, loss : 1.310e-04
iter: 2014, loss : 1.619e-04
iter: 2015, loss : 1.694e-04
iter: 2016, loss : 1.626e-04
iter: 2017, loss : 3.759e-04
iter: 2018, loss : 1.733e-04
iter: 2019, loss : 2.777e-04
iter: 2020, loss : 1.708e-03
iter: 2021, loss : 1.453e-04
iter: 2022, loss : 2.748e-04
iter: 2023, loss : 2.694e-04
iter: 2024, loss : 2.288e-04
iter: 2025, loss : 1.687e-04
iter: 2026, loss : 5.637e-04
iter: 2027, loss : 2.966e-04
iter: 2028, loss : 1.443e-03
iter: 2029, loss : 5.910e-04
iter: 2030, loss : 1.050e-03
iter: 2031, loss : 3.083e-04
iter: 2032, loss : 8.923e-05
iter: 2033, loss : 4.674e-04
iter: 2034, loss : 9.848e-04
iter: 2035, loss : 1.137e-04
iter: 2036, loss : 1.451e-04
iter: 2037, loss : 1.430e-04
iter: 2038, loss : 2.529e-04
iter: 2039, loss : 2.039e-04
iter: 2040, loss : 3.283e-04
iter: 2041, loss : 1.937e-03
iter: 2042, loss : 2.388e-04
iter: 2043, loss : 7.242e-04
iter: 2044, loss : 6.972e-04
iter: 2045, loss : 2.392e-04
iter: 2046, loss : 1.086e-04
iter: 2047, loss : 2.459e-04
iter: 2048, loss : 1.366e-04
iter: 2049, loss : 6.995e-05
iter: 2050, loss : 1.628e-04
iter: 2051, loss : 1.696e-04
iter: 2052, loss : 1.261e-04
iter: 2053, loss : 1.964e-04
iter: 2054, loss : 1.201e-03
iter: 2055, loss : 4.173e-04
iter: 2056, loss : 2.518e-04
iter: 2057, loss : 1.086e-03
iter: 2058, loss : 4.728e-04
iter: 2059, loss : 1.427e-04
iter: 2060, loss : 2.516e-04
iter: 2061, loss : 3.507e-03
iter: 2062, loss : 1.806e-04
iter: 2063, loss : 5.694e-04
iter: 2064, loss : 1.608e-03
iter: 2065, loss : 7.862e-05
iter: 2066, loss : 3.034e-04
iter: 2067, loss : 8.884e-04
iter: 2068, loss : 1.657e-03
iter: 2069, loss : 1.789e-04
iter: 2070, loss : 9.273e-04
iter: 2071, loss : 2.813e-04
iter: 2072, loss : 4.777e-04
iter: 2073, loss : 9.032e-05
iter: 2074, loss : 4.861e-04
iter: 2075, loss : 4.177e-04
iter: 2076, loss : 1.642e-04
iter: 2077, loss : 6.171e-05
iter: 2078, loss : 5.614e-05
iter: 2079, loss : 8.683e-05
iter: 2080, loss : 2.286e-04
iter: 2081, loss : 1.613e-04
iter: 2082, loss : 1.348e-04
iter: 2083, loss : 2.270e-04
iter: 2084, loss : 1.328e-04
iter: 2085, loss : 4.142e-04
iter: 2086, loss : 1.686e-04
iter: 2087, loss : 4.751e-04
iter: 2088, loss : 6.385e-05
iter: 2089, loss : 1.698e-04
iter: 2090, loss : 1.143e-04
iter: 2091, loss : 9.876e-05
iter: 2092, loss : 2.292e-04
iter: 2093, loss : 6.619e-05
iter: 2094, loss : 1.185e-04
iter: 2095, loss : 1.189e-04
iter: 2096, loss : 1.118e-04
iter: 2097, loss : 5.352e-04
iter: 2098, loss : 2.346e-04
iter: 2099, loss : 1.537e-04
iter: 2100, loss : 7.025e-05
iter: 2101, loss : 7.869e-04
iter: 2102, loss : 2.011e-03
iter: 2103, loss : 2.176e-04
iter: 2104, loss : 1.142e-04
iter: 2105, loss : 9.109e-05
iter: 2106, loss : 5.974e-04
iter: 2107, loss : 2.093e-04
iter: 2108, loss : 6.764e-05
iter: 2109, loss : 6.428e-04
iter: 2110, loss : 6.301e-05
iter: 2111, loss : 1.227e-04
iter: 2112, loss : 4.943e-05
iter: 2113, loss : 6.357e-04
iter: 2114, loss : 7.348e-04
iter: 2115, loss : 5.746e-05
iter: 2116, loss : 3.566e-05
iter: 2117, loss : 1.013e-03
iter: 2118, loss : 1.579e-04
iter: 2119, loss : 5.288e-04
iter: 2120, loss : 8.689e-04
iter: 2121, loss : 2.417e-04
iter: 2122, loss : 1.590e-04
iter: 2123, loss : 2.973e-03
iter: 2124, loss : 1.414e-04
iter: 2125, loss : 4.780e-04
iter: 2126, loss : 1.695e-04
iter: 2127, loss : 6.999e-05
iter: 2128, loss : 6.827e-05
iter: 2129, loss : 8.718e-05
iter: 2130, loss : 6.363e-04
iter: 2131, loss : 1.816e-04
iter: 2132, loss : 1.462e-04
iter: 2133, loss : 9.479e-05
iter: 2134, loss : 1.842e-03
iter: 2135, loss : 2.998e-04
iter: 2136, loss : 4.849e-04
iter: 2137, loss : 3.219e-04
iter: 2138, loss : 1.667e-04
iter: 2139, loss : 2.017e-04
iter: 2140, loss : 1.367e-03
iter: 2141, loss : 2.534e-04
iter: 2142, loss : 1.957e-04
iter: 2143, loss : 1.095e-04
iter: 2144, loss : 3.016e-04
iter: 2145, loss : 5.406e-04
iter: 2146, loss : 1.449e-04
iter: 2147, loss : 1.400e-04
iter: 2148, loss : 1.398e-04
iter: 2149, loss : 1.956e-04
iter: 2150, loss : 7.096e-05
iter: 2151, loss : 1.240e-04
iter: 2152, loss : 2.875e-04
iter: 2153, loss : 1.369e-03
iter: 2154, loss : 2.570e-04
iter: 2155, loss : 1.007e-04
iter: 2156, loss : 1.099e-04
iter: 2157, loss : 7.020e-05
iter: 2158, loss : 9.289e-05
iter: 2159, loss : 1.034e-03
iter: 2160, loss : 1.977e-04
iter: 2161, loss : 1.691e-04
iter: 2162, loss : 6.408e-04
iter: 2163, loss : 7.898e-05
iter: 2164, loss : 4.369e-04
iter: 2165, loss : 7.068e-04
iter: 2166, loss : 2.155e-04
iter: 2167, loss : 6.837e-05
iter: 2168, loss : 1.195e-04
iter: 2169, loss : 1.559e-04
iter: 2170, loss : 5.335e-04
iter: 2171, loss : 1.912e-04
iter: 2172, loss : 1.691e-04
iter: 2173, loss : 3.007e-04
iter: 2174, loss : 2.714e-04
iter: 2175, loss : 5.116e-04
iter: 2176, loss : 1.319e-04
iter: 2177, loss : 4.927e-05
iter: 2178, loss : 8.512e-05
iter: 2179, loss : 1.014e-04
iter: 2180, loss : 7.084e-05
iter: 2181, loss : 6.117e-05
iter: 2182, loss : 1.183e-03
iter: 2183, loss : 4.380e-04
iter: 2184, loss : 2.834e-04
iter: 2185, loss : 1.925e-04
iter: 2186, loss : 3.801e-04
iter: 2187, loss : 1.180e-04
iter: 2188, loss : 5.685e-04
iter: 2189, loss : 1.358e-04
iter: 2190, loss : 9.071e-05
iter: 2191, loss : 1.283e-03
iter: 2192, loss : 7.567e-04
iter: 2193, loss : 5.292e-05
iter: 2194, loss : 2.184e-04
iter: 2195, loss : 2.133e-04
iter: 2196, loss : 1.629e-04
iter: 2197, loss : 9.884e-05
iter: 2198, loss : 3.940e-04
iter: 2199, loss : 6.223e-04
iter: 2200, loss : 7.075e-05
iter: 2201, loss : 6.208e-05
iter: 2202, loss : 1.003e-04
iter: 2203, loss : 3.315e-04
iter: 2204, loss : 3.915e-03
iter: 2205, loss : 2.646e-04
iter: 2206, loss : 1.072e-04
iter: 2207, loss : 1.460e-04
iter: 2208, loss : 9.623e-05
iter: 2209, loss : 1.136e-04
iter: 2210, loss : 1.077e-04
iter: 2211, loss : 4.430e-05
iter: 2212, loss : 8.251e-05
iter: 2213, loss : 1.585e-03
iter: 2214, loss : 1.887e-04
iter: 2215, loss : 1.256e-03
iter: 2216, loss : 3.972e-04
iter: 2217, loss : 2.632e-04
iter: 2218, loss : 1.168e-04
iter: 2219, loss : 1.825e-04
iter: 2220, loss : 9.301e-05
iter: 2221, loss : 1.390e-03
iter: 2222, loss : 1.638e-04
iter: 2223, loss : 3.720e-04
iter: 2224, loss : 1.636e-04
iter: 2225, loss : 1.286e-03
iter: 2226, loss : 1.566e-03
iter: 2227, loss : 6.498e-04
iter: 2228, loss : 2.435e-04
iter: 2229, loss : 2.249e-03
iter: 2230, loss : 5.862e-04
iter: 2231, loss : 2.956e-04
iter: 2232, loss : 3.387e-04
iter: 2233, loss : 1.540e-03
iter: 2234, loss : 3.777e-04
iter: 2235, loss : 7.605e-04
iter: 2236, loss : 2.060e-04
iter: 2237, loss : 6.092e-04
iter: 2238, loss : 1.208e-03
iter: 2239, loss : 6.693e-04
iter: 2240, loss : 1.316e-04
iter: 2241, loss : 1.096e-03
iter: 2242, loss : 4.735e-04
iter: 2243, loss : 4.243e-04
iter: 2244, loss : 8.545e-05
iter: 2245, loss : 2.103e-04
iter: 2246, loss : 6.149e-04
iter: 2247, loss : 3.144e-04
iter: 2248, loss : 1.566e-04
iter: 2249, loss : 7.283e-04
iter: 2250, loss : 1.672e-04
iter: 2251, loss : 1.449e-04
iter: 2252, loss : 7.199e-04
iter: 2253, loss : 1.538e-04
iter: 2254, loss : 1.140e-03
iter: 2255, loss : 9.255e-05
iter: 2256, loss : 2.199e-04
iter: 2257, loss : 3.249e-03
iter: 2258, loss : 6.774e-04
iter: 2259, loss : 3.756e-04
iter: 2260, loss : 2.119e-04
iter: 2261, loss : 6.633e-04
iter: 2262, loss : 2.541e-04
iter: 2263, loss : 1.511e-04
iter: 2264, loss : 1.185e-04
iter: 2265, loss : 1.352e-04
iter: 2266, loss : 1.733e-04
iter: 2267, loss : 6.088e-04
iter: 2268, loss : 1.146e-04
iter: 2269, loss : 1.905e-04
iter: 2270, loss : 1.588e-04
iter: 2271, loss : 9.897e-05
iter: 2272, loss : 1.603e-04
iter: 2273, loss : 6.911e-05
iter: 2274, loss : 5.513e-04
iter: 2275, loss : 3.113e-03
iter: 2276, loss : 2.697e-04
iter: 2277, loss : 1.458e-03
iter: 2278, loss : 6.417e-04
iter: 2279, loss : 6.996e-05
iter: 2280, loss : 6.822e-04
iter: 2281, loss : 2.543e-04
iter: 2282, loss : 2.629e-04
iter: 2283, loss : 6.416e-05
iter: 2284, loss : 1.576e-03
iter: 2285, loss : 2.344e-04
iter: 2286, loss : 1.905e-04
iter: 2287, loss : 5.257e-04
iter: 2288, loss : 7.689e-05
iter: 2289, loss : 4.078e-03
iter: 2290, loss : 6.426e-04
iter: 2291, loss : 2.066e-04
iter: 2292, loss : 3.325e-03
iter: 2293, loss : 7.695e-04
iter: 2294, loss : 5.510e-04
iter: 2295, loss : 6.867e-04
iter: 2296, loss : 6.721e-04
iter: 2297, loss : 1.313e-04
iter: 2298, loss : 1.903e-04
iter: 2299, loss : 1.966e-04
iter: 2300, loss : 1.710e-03
iter: 2301, loss : 1.229e-04
iter: 2302, loss : 2.641e-04
iter: 2303, loss : 2.204e-04
iter: 2304, loss : 1.706e-04
iter: 2305, loss : 9.575e-05
iter: 2306, loss : 1.282e-03
iter: 2307, loss : 7.283e-04
iter: 2308, loss : 1.368e-04
iter: 2309, loss : 7.745e-04
iter: 2310, loss : 9.650e-04
iter: 2311, loss : 9.183e-04
iter: 2312, loss : 2.912e-04
iter: 2313, loss : 1.148e-04
iter: 2314, loss : 3.031e-04
iter: 2315, loss : 1.298e-04
iter: 2316, loss : 1.709e-04
iter: 2317, loss : 1.097e-03
iter: 2318, loss : 2.206e-04
iter: 2319, loss : 3.168e-04
iter: 2320, loss : 1.399e-04
iter: 2321, loss : 1.070e-04
iter: 2322, loss : 6.337e-05
iter: 2323, loss : 1.200e-04
iter: 2324, loss : 1.244e-04
iter: 2325, loss : 1.628e-04
iter: 2326, loss : 1.121e-04
iter: 2327, loss : 1.155e-04
iter: 2328, loss : 7.245e-05
iter: 2329, loss : 2.366e-04
iter: 2330, loss : 1.262e-03
iter: 2331, loss : 2.648e-04
iter: 2332, loss : 2.041e-04
iter: 2333, loss : 1.096e-04
iter: 2334, loss : 4.077e-04
iter: 2335, loss : 4.904e-04
iter: 2336, loss : 1.188e-03
iter: 2337, loss : 2.272e-04
iter: 2338, loss : 6.418e-05
iter: 2339, loss : 2.420e-04
iter: 2340, loss : 1.904e-04
iter: 2341, loss : 4.140e-04
iter: 2342, loss : 1.002e-04
iter: 2343, loss : 4.757e-04
iter: 2344, loss : 1.215e-04
iter: 2345, loss : 2.068e-04
iter: 2346, loss : 5.541e-04
iter: 2347, loss : 1.229e-04
iter: 2348, loss : 2.884e-04
iter: 2349, loss : 1.231e-03
iter: 2350, loss : 6.722e-04
iter: 2351, loss : 1.872e-04
iter: 2352, loss : 1.798e-04
iter: 2353, loss : 1.625e-04
iter: 2354, loss : 5.539e-04
iter: 2355, loss : 3.194e-04
iter: 2356, loss : 1.040e-04
iter: 2357, loss : 3.272e-03
iter: 2358, loss : 3.530e-04
iter: 2359, loss : 2.526e-04
iter: 2360, loss : 1.012e-03
iter: 2361, loss : 1.265e-04
iter: 2362, loss : 1.877e-04
iter: 2363, loss : 3.888e-04
iter: 2364, loss : 2.891e-04
iter: 2365, loss : 4.231e-04
iter: 2366, loss : 2.041e-04
iter: 2367, loss : 3.490e-04
iter: 2368, loss : 1.623e-04
iter: 2369, loss : 1.410e-04
iter: 2370, loss : 1.133e-04
iter: 2371, loss : 9.855e-05
iter: 2372, loss : 4.387e-05
iter: 2373, loss : 6.560e-05
iter: 2374, loss : 1.160e-03
iter: 2375, loss : 1.299e-03
iter: 2376, loss : 9.151e-04
iter: 2377, loss : 1.443e-04
iter: 2378, loss : 3.652e-04
iter: 2379, loss : 4.396e-04
iter: 2380, loss : 1.001e-03
iter: 2381, loss : 2.560e-04
iter: 2382, loss : 2.315e-04
iter: 2383, loss : 1.033e-03
iter: 2384, loss : 2.468e-04
iter: 2385, loss : 3.141e-03
iter: 2386, loss : 2.519e-04
iter: 2387, loss : 6.054e-04
iter: 2388, loss : 6.126e-04
iter: 2389, loss : 1.238e-04
iter: 2390, loss : 5.270e-04
iter: 2391, loss : 1.506e-03
iter: 2392, loss : 2.863e-04
iter: 2393, loss : 3.010e-04
iter: 2394, loss : 2.820e-04
iter: 2395, loss : 3.427e-04
iter: 2396, loss : 2.247e-04
iter: 2397, loss : 7.853e-05
iter: 2398, loss : 1.810e-04
iter: 2399, loss : 7.766e-04
iter: 2400, loss : 3.604e-04
iter: 2401, loss : 2.912e-04
iter: 2402, loss : 1.553e-04
iter: 2403, loss : 1.148e-04
iter: 2404, loss : 4.674e-04
iter: 2405, loss : 4.176e-04
iter: 2406, loss : 1.156e-04
iter: 2407, loss : 5.379e-05
iter: 2408, loss : 2.100e-04
iter: 2409, loss : 1.136e-04
iter: 2410, loss : 4.918e-04
iter: 2411, loss : 4.800e-04
iter: 2412, loss : 5.871e-04
iter: 2413, loss : 8.147e-04
iter: 2414, loss : 3.342e-04
iter: 2415, loss : 1.404e-04
iter: 2416, loss : 1.319e-04
iter: 2417, loss : 5.732e-04
iter: 2418, loss : 5.519e-04
iter: 2419, loss : 1.586e-04
iter: 2420, loss : 9.089e-04
iter: 2421, loss : 2.262e-04
iter: 2422, loss : 8.840e-05
iter: 2423, loss : 3.717e-03
iter: 2424, loss : 5.067e-04
iter: 2425, loss : 3.377e-04
iter: 2426, loss : 5.033e-04
iter: 2427, loss : 1.681e-04
iter: 2428, loss : 4.411e-04
iter: 2429, loss : 3.023e-04
iter: 2430, loss : 1.584e-04
iter: 2431, loss : 4.101e-04
iter: 2432, loss : 5.192e-04
iter: 2433, loss : 2.738e-04
iter: 2434, loss : 1.002e-04
iter: 2435, loss : 4.499e-04
iter: 2436, loss : 2.414e-03
iter: 2437, loss : 4.904e-04
iter: 2438, loss : 1.254e-04
iter: 2439, loss : 4.887e-04
iter: 2440, loss : 8.654e-05
iter: 2441, loss : 9.660e-05
iter: 2442, loss : 1.476e-04
iter: 2443, loss : 5.572e-04
iter: 2444, loss : 2.185e-03
iter: 2445, loss : 1.522e-04
iter: 2446, loss : 5.357e-04
iter: 2447, loss : 6.810e-05
iter: 2448, loss : 2.978e-03
iter: 2449, loss : 2.023e-04
iter: 2450, loss : 6.205e-04
iter: 2451, loss : 7.842e-04
iter: 2452, loss : 4.258e-04
iter: 2453, loss : 3.818e-04
iter: 2454, loss : 4.100e-04
iter: 2455, loss : 3.039e-04
iter: 2456, loss : 2.903e-04
iter: 2457, loss : 2.385e-04
iter: 2458, loss : 7.044e-05
iter: 2459, loss : 2.817e-04
iter: 2460, loss : 2.222e-04
iter: 2461, loss : 6.034e-05
iter: 2462, loss : 1.211e-04
iter: 2463, loss : 9.464e-05
iter: 2464, loss : 1.697e-04
iter: 2465, loss : 6.541e-04
iter: 2466, loss : 5.291e-05
iter: 2467, loss : 1.263e-04
iter: 2468, loss : 2.180e-04
iter: 2469, loss : 2.488e-04
iter: 2470, loss : 1.962e-04
iter: 2471, loss : 5.393e-05
iter: 2472, loss : 2.864e-04
iter: 2473, loss : 1.053e-04
iter: 2474, loss : 1.527e-04
iter: 2475, loss : 4.983e-04
iter: 2476, loss : 1.826e-04
iter: 2477, loss : 4.859e-05
iter: 2478, loss : 4.361e-05
iter: 2479, loss : 3.848e-05
iter: 2480, loss : 7.254e-05
iter: 2481, loss : 9.900e-05
iter: 2482, loss : 4.320e-05
iter: 2483, loss : 9.881e-05
iter: 2484, loss : 1.742e-04
iter: 2485, loss : 3.819e-03
iter: 2486, loss : 4.878e-04
iter: 2487, loss : 1.195e-04
iter: 2488, loss : 1.380e-04
iter: 2489, loss : 6.254e-04
iter: 2490, loss : 2.570e-04
iter: 2491, loss : 5.898e-05
iter: 2492, loss : 2.461e-04
iter: 2493, loss : 9.130e-05
iter: 2494, loss : 1.510e-03
iter: 2495, loss : 3.878e-04
iter: 2496, loss : 5.743e-04
iter: 2497, loss : 7.023e-04
iter: 2498, loss : 1.438e-04
iter: 2499, loss : 1.534e-04
iter: 2500, loss : 2.183e-04
iter: 2501, loss : 2.990e-04
iter: 2502, loss : 3.788e-05
iter: 2503, loss : 5.018e-04
iter: 2504, loss : 1.947e-04
iter: 2505, loss : 1.597e-04
iter: 2506, loss : 4.191e-04
iter: 2507, loss : 1.929e-04
iter: 2508, loss : 3.421e-03
iter: 2509, loss : 1.122e-04
iter: 2510, loss : 3.328e-03
iter: 2511, loss : 4.087e-04
iter: 2512, loss : 2.094e-04
iter: 2513, loss : 8.789e-04
iter: 2514, loss : 1.288e-04
iter: 2515, loss : 4.641e-04
iter: 2516, loss : 2.797e-04
iter: 2517, loss : 1.967e-04
iter: 2518, loss : 2.394e-04
iter: 2519, loss : 8.783e-05
iter: 2520, loss : 1.303e-04
iter: 2521, loss : 2.725e-03
iter: 2522, loss : 5.065e-04
iter: 2523, loss : 1.924e-04
iter: 2524, loss : 4.128e-04
iter: 2525, loss : 3.996e-04
iter: 2526, loss : 2.001e-04
iter: 2527, loss : 7.257e-05
iter: 2528, loss : 1.047e-04
iter: 2529, loss : 3.869e-04
iter: 2530, loss : 7.556e-04
iter: 2531, loss : 6.625e-04
iter: 2532, loss : 2.059e-04
iter: 2533, loss : 2.116e-04
iter: 2534, loss : 9.403e-05
iter: 2535, loss : 1.205e-04
iter: 2536, loss : 9.464e-05
iter: 2537, loss : 1.529e-04
iter: 2538, loss : 4.252e-04
iter: 2539, loss : 5.124e-05
iter: 2540, loss : 2.195e-04
iter: 2541, loss : 1.071e-04
iter: 2542, loss : 3.109e-03
iter: 2543, loss : 6.665e-05
iter: 2544, loss : 2.367e-04
iter: 2545, loss : 2.352e-04
iter: 2546, loss : 1.462e-04
iter: 2547, loss : 1.566e-04
iter: 2548, loss : 6.536e-04
iter: 2549, loss : 1.495e-04
iter: 2550, loss : 3.565e-04
iter: 2551, loss : 4.612e-04
iter: 2552, loss : 4.569e-04
iter: 2553, loss : 1.549e-04
iter: 2554, loss : 2.208e-04
iter: 2555, loss : 3.889e-05
iter: 2556, loss : 4.727e-04
iter: 2557, loss : 3.323e-04
iter: 2558, loss : 1.177e-04
iter: 2559, loss : 8.940e-05
iter: 2560, loss : 4.091e-04
iter: 2561, loss : 1.551e-04
iter: 2562, loss : 5.552e-04
iter: 2563, loss : 4.078e-04
iter: 2564, loss : 2.828e-04
iter: 2565, loss : 5.002e-04
iter: 2566, loss : 9.442e-05
iter: 2567, loss : 2.151e-04
iter: 2568, loss : 5.074e-05
iter: 2569, loss : 5.671e-05
iter: 2570, loss : 7.080e-05
iter: 2571, loss : 5.013e-04
iter: 2572, loss : 2.950e-05
iter: 2573, loss : 1.784e-04
iter: 2574, loss : 1.731e-04
iter: 2575, loss : 5.089e-04
iter: 2576, loss : 3.269e-04
iter: 2577, loss : 1.498e-04
iter: 2578, loss : 1.000e-04
iter: 2579, loss : 1.837e-03
iter: 2580, loss : 3.065e-04
iter: 2581, loss : 2.308e-04
iter: 2582, loss : 1.453e-04
iter: 2583, loss : 1.010e-04
iter: 2584, loss : 1.991e-04
iter: 2585, loss : 6.968e-05
iter: 2586, loss : 7.316e-05
iter: 2587, loss : 5.167e-05
iter: 2588, loss : 2.075e-04
iter: 2589, loss : 1.003e-04
iter: 2590, loss : 3.376e-05
iter: 2591, loss : 6.246e-05
iter: 2592, loss : 1.176e-04
iter: 2593, loss : 4.730e-04
iter: 2594, loss : 7.519e-04
iter: 2595, loss : 1.171e-04
iter: 2596, loss : 6.607e-05
iter: 2597, loss : 6.238e-04
iter: 2598, loss : 2.491e-04
iter: 2599, loss : 9.584e-04
iter: 2600, loss : 5.999e-05
iter: 2601, loss : 1.788e-04
iter: 2602, loss : 2.387e-03
iter: 2603, loss : 2.512e-04
iter: 2604, loss : 1.202e-04
iter: 2605, loss : 1.069e-03
iter: 2606, loss : 2.814e-04
iter: 2607, loss : 1.337e-04
iter: 2608, loss : 5.709e-04
iter: 2609, loss : 4.071e-04
iter: 2610, loss : 5.590e-05
iter: 2611, loss : 4.027e-04
iter: 2612, loss : 1.544e-04
iter: 2613, loss : 4.542e-04
iter: 2614, loss : 8.674e-04
iter: 2615, loss : 8.229e-04
iter: 2616, loss : 2.143e-04
iter: 2617, loss : 3.415e-04
iter: 2618, loss : 1.262e-04
iter: 2619, loss : 2.487e-04
iter: 2620, loss : 2.520e-04
iter: 2621, loss : 8.089e-05
iter: 2622, loss : 3.483e-04
iter: 2623, loss : 5.801e-05
iter: 2624, loss : 8.356e-05
iter: 2625, loss : 1.145e-04
iter: 2626, loss : 1.533e-04
iter: 2627, loss : 6.969e-05
iter: 2628, loss : 2.623e-03
iter: 2629, loss : 1.952e-03
iter: 2630, loss : 1.104e-04
iter: 2631, loss : 5.134e-04
iter: 2632, loss : 2.799e-04
iter: 2633, loss : 9.801e-05
iter: 2634, loss : 1.139e-04
iter: 2635, loss : 2.215e-03
iter: 2636, loss : 2.232e-04
iter: 2637, loss : 1.119e-03
iter: 2638, loss : 9.224e-04
iter: 2639, loss : 7.411e-04
iter: 2640, loss : 1.351e-04
iter: 2641, loss : 4.978e-04
iter: 2642, loss : 6.709e-05
iter: 2643, loss : 9.708e-05
iter: 2644, loss : 2.193e-04
iter: 2645, loss : 5.515e-04
iter: 2646, loss : 2.572e-04
iter: 2647, loss : 1.209e-04
iter: 2648, loss : 1.270e-03
iter: 2649, loss : 9.835e-04
iter: 2650, loss : 5.712e-04
iter: 2651, loss : 7.138e-05
iter: 2652, loss : 1.773e-04
iter: 2653, loss : 4.763e-04
iter: 2654, loss : 5.698e-04
iter: 2655, loss : 1.529e-04
iter: 2656, loss : 4.202e-04
iter: 2657, loss : 4.806e-04
iter: 2658, loss : 1.522e-04
iter: 2659, loss : 6.178e-04
iter: 2660, loss : 2.832e-04
iter: 2661, loss : 1.626e-04
iter: 2662, loss : 4.132e-04
iter: 2663, loss : 2.938e-04
iter: 2664, loss : 1.127e-03
iter: 2665, loss : 1.340e-04
iter: 2666, loss : 2.562e-04
iter: 2667, loss : 2.680e-04
iter: 2668, loss : 8.649e-05
iter: 2669, loss : 1.099e-04
iter: 2670, loss : 1.411e-04
iter: 2671, loss : 1.713e-04
iter: 2672, loss : 6.094e-05
iter: 2673, loss : 4.204e-04
iter: 2674, loss : 1.024e-04
iter: 2675, loss : 1.135e-04
iter: 2676, loss : 4.390e-04
iter: 2677, loss : 5.807e-04
iter: 2678, loss : 4.598e-04
iter: 2679, loss : 1.520e-04
iter: 2680, loss : 1.008e-04
iter: 2681, loss : 5.982e-05
iter: 2682, loss : 6.358e-05
iter: 2683, loss : 7.945e-05
iter: 2684, loss : 4.005e-04
iter: 2685, loss : 1.704e-04
iter: 2686, loss : 5.600e-05
iter: 2687, loss : 1.657e-04
iter: 2688, loss : 1.276e-04
iter: 2689, loss : 1.248e-04
iter: 2690, loss : 1.511e-04
iter: 2691, loss : 5.709e-05
iter: 2692, loss : 1.407e-03
iter: 2693, loss : 1.066e-03
iter: 2694, loss : 2.464e-04
iter: 2695, loss : 2.154e-04
iter: 2696, loss : 5.223e-04
iter: 2697, loss : 2.046e-04
iter: 2698, loss : 6.047e-04
iter: 2699, loss : 1.164e-04
iter: 2700, loss : 1.188e-04
iter: 2701, loss : 3.234e-03
iter: 2702, loss : 1.889e-03
iter: 2703, loss : 1.217e-03
iter: 2704, loss : 3.623e-04
iter: 2705, loss : 2.303e-04
iter: 2706, loss : 6.147e-04
iter: 2707, loss : 1.169e-04
iter: 2708, loss : 1.361e-04
iter: 2709, loss : 4.181e-04
iter: 2710, loss : 2.900e-05
iter: 2711, loss : 2.216e-04
iter: 2712, loss : 1.382e-03
iter: 2713, loss : 4.160e-04
iter: 2714, loss : 2.090e-04
iter: 2715, loss : 3.420e-04
iter: 2716, loss : 4.450e-04
iter: 2717, loss : 8.938e-05
iter: 2718, loss : 7.348e-05
iter: 2719, loss : 1.378e-04
iter: 2720, loss : 2.071e-04
iter: 2721, loss : 9.539e-04
iter: 2722, loss : 2.664e-03
iter: 2723, loss : 4.542e-04
iter: 2724, loss : 5.557e-04
iter: 2725, loss : 9.328e-05
iter: 2726, loss : 2.600e-03
iter: 2727, loss : 2.310e-04
iter: 2728, loss : 1.698e-04
iter: 2729, loss : 1.211e-04
iter: 2730, loss : 1.030e-04
iter: 2731, loss : 3.381e-04
iter: 2732, loss : 8.592e-05
iter: 2733, loss : 9.695e-05
iter: 2734, loss : 1.588e-04
iter: 2735, loss : 7.331e-04
iter: 2736, loss : 4.581e-04
iter: 2737, loss : 1.702e-04
iter: 2738, loss : 1.698e-04
iter: 2739, loss : 2.170e-04
iter: 2740, loss : 6.125e-05
iter: 2741, loss : 9.376e-04
iter: 2742, loss : 1.911e-04
iter: 2743, loss : 5.523e-04
iter: 2744, loss : 7.663e-05
iter: 2745, loss : 4.827e-04
iter: 2746, loss : 1.261e-04
iter: 2747, loss : 1.505e-04
iter: 2748, loss : 8.487e-05
iter: 2749, loss : 9.003e-04
iter: 2750, loss : 5.111e-04
iter: 2751, loss : 3.361e-04
iter: 2752, loss : 2.299e-04
iter: 2753, loss : 5.961e-05
iter: 2754, loss : 1.076e-04
iter: 2755, loss : 1.844e-04
iter: 2756, loss : 3.843e-04
iter: 2757, loss : 8.533e-05
iter: 2758, loss : 7.229e-05
iter: 2759, loss : 4.266e-04
iter: 2760, loss : 1.111e-04
iter: 2761, loss : 4.940e-04
iter: 2762, loss : 1.049e-04
iter: 2763, loss : 1.119e-04
iter: 2764, loss : 2.430e-04
iter: 2765, loss : 9.820e-05
iter: 2766, loss : 2.590e-04
iter: 2767, loss : 8.254e-05
iter: 2768, loss : 1.455e-03
iter: 2769, loss : 1.424e-04
iter: 2770, loss : 5.157e-04
iter: 2771, loss : 6.782e-05
iter: 2772, loss : 1.543e-04
iter: 2773, loss : 2.706e-04
iter: 2774, loss : 5.993e-05
iter: 2775, loss : 6.092e-04
iter: 2776, loss : 1.046e-04
iter: 2777, loss : 9.749e-05
iter: 2778, loss : 5.582e-05
iter: 2779, loss : 1.007e-04
iter: 2780, loss : 5.545e-05
iter: 2781, loss : 3.014e-04
iter: 2782, loss : 2.674e-04
iter: 2783, loss : 7.561e-05
iter: 2784, loss : 1.646e-04
iter: 2785, loss : 9.947e-05
iter: 2786, loss : 6.761e-05
iter: 2787, loss : 2.138e-05
iter: 2788, loss : 1.315e-04
iter: 2789, loss : 3.226e-05
iter: 2790, loss : 9.139e-05
iter: 2791, loss : 1.170e-03
iter: 2792, loss : 6.279e-05
iter: 2793, loss : 8.628e-05
iter: 2794, loss : 8.613e-04
iter: 2795, loss : 2.037e-04
iter: 2796, loss : 1.209e-04
iter: 2797, loss : 1.929e-04
iter: 2798, loss : 3.783e-04
iter: 2799, loss : 2.931e-04
iter: 2800, loss : 1.022e-04
iter: 2801, loss : 1.713e-04
iter: 2802, loss : 9.757e-05
iter: 2803, loss : 4.947e-04
iter: 2804, loss : 6.830e-05
iter: 2805, loss : 5.703e-05
iter: 2806, loss : 7.934e-05
iter: 2807, loss : 2.783e-04
iter: 2808, loss : 3.517e-04
iter: 2809, loss : 1.905e-04
iter: 2810, loss : 2.403e-04
iter: 2811, loss : 5.300e-04
iter: 2812, loss : 6.446e-05
iter: 2813, loss : 9.936e-05
iter: 2814, loss : 2.887e-05
iter: 2815, loss : 7.686e-05
iter: 2816, loss : 9.050e-05
iter: 2817, loss : 5.157e-05
iter: 2818, loss : 1.401e-04
iter: 2819, loss : 7.043e-05
iter: 2820, loss : 9.983e-04
iter: 2821, loss : 8.418e-04
iter: 2822, loss : 8.744e-05
iter: 2823, loss : 4.571e-05
iter: 2824, loss : 6.970e-04
iter: 2825, loss : 1.003e-04
iter: 2826, loss : 4.385e-04
iter: 2827, loss : 1.475e-04
iter: 2828, loss : 3.370e-03
iter: 2829, loss : 9.795e-04
iter: 2830, loss : 1.360e-04
iter: 2831, loss : 7.778e-04
iter: 2832, loss : 1.148e-03
iter: 2833, loss : 3.738e-04
iter: 2834, loss : 4.731e-04
iter: 2835, loss : 8.193e-05
iter: 2836, loss : 2.597e-04
iter: 2837, loss : 4.364e-04
iter: 2838, loss : 1.767e-04
iter: 2839, loss : 8.483e-05
iter: 2840, loss : 7.446e-04
iter: 2841, loss : 4.420e-04
iter: 2842, loss : 2.444e-04
iter: 2843, loss : 3.698e-04
iter: 2844, loss : 1.694e-04
iter: 2845, loss : 1.297e-04
iter: 2846, loss : 6.014e-04
iter: 2847, loss : 6.157e-04
iter: 2848, loss : 1.402e-04
iter: 2849, loss : 3.921e-04
iter: 2850, loss : 3.398e-04
iter: 2851, loss : 1.217e-04
iter: 2852, loss : 1.377e-04
iter: 2853, loss : 1.545e-04
iter: 2854, loss : 2.138e-04
iter: 2855, loss : 1.315e-04
iter: 2856, loss : 6.155e-05
iter: 2857, loss : 1.138e-04
iter: 2858, loss : 1.352e-04
iter: 2859, loss : 4.619e-05
iter: 2860, loss : 1.192e-04
iter: 2861, loss : 1.447e-04
iter: 2862, loss : 6.186e-05
iter: 2863, loss : 1.076e-04
iter: 2864, loss : 1.358e-04
iter: 2865, loss : 4.484e-04
iter: 2866, loss : 1.139e-04
iter: 2867, loss : 5.278e-04
iter: 2868, loss : 6.813e-04
iter: 2869, loss : 6.747e-05
iter: 2870, loss : 3.273e-04
iter: 2871, loss : 9.572e-05
iter: 2872, loss : 4.988e-04
iter: 2873, loss : 7.372e-05
iter: 2874, loss : 1.455e-04
iter: 2875, loss : 1.668e-04
iter: 2876, loss : 6.928e-04
iter: 2877, loss : 5.359e-04
iter: 2878, loss : 9.092e-05
iter: 2879, loss : 1.071e-04
iter: 2880, loss : 1.866e-04
iter: 2881, loss : 3.534e-05
iter: 2882, loss : 8.948e-05
iter: 2883, loss : 1.681e-04
iter: 2884, loss : 1.198e-04
iter: 2885, loss : 5.505e-04
iter: 2886, loss : 9.127e-05
iter: 2887, loss : 1.366e-04
iter: 2888, loss : 1.206e-04
iter: 2889, loss : 1.046e-04
iter: 2890, loss : 4.669e-04
iter: 2891, loss : 5.143e-05
iter: 2892, loss : 1.382e-04
iter: 2893, loss : 7.470e-05
iter: 2894, loss : 1.928e-03
iter: 2895, loss : 2.031e-03
iter: 2896, loss : 1.728e-03
iter: 2897, loss : 2.573e-04
iter: 2898, loss : 4.694e-04
iter: 2899, loss : 2.360e-04
iter: 2900, loss : 9.569e-04
iter: 2901, loss : 5.638e-04
iter: 2902, loss : 5.394e-04
iter: 2903, loss : 1.189e-03
iter: 2904, loss : 9.556e-05
iter: 2905, loss : 1.307e-04
iter: 2906, loss : 1.549e-04
iter: 2907, loss : 7.354e-04
iter: 2908, loss : 1.240e-04
iter: 2909, loss : 1.920e-04
iter: 2910, loss : 1.675e-04
iter: 2911, loss : 2.211e-04
iter: 2912, loss : 1.268e-04
iter: 2913, loss : 2.634e-03
iter: 2914, loss : 7.005e-04
iter: 2915, loss : 1.702e-04
iter: 2916, loss : 8.933e-05
iter: 2917, loss : 8.416e-05
iter: 2918, loss : 5.462e-05
iter: 2919, loss : 3.057e-04
iter: 2920, loss : 2.655e-03
iter: 2921, loss : 2.403e-04
iter: 2922, loss : 1.242e-04
iter: 2923, loss : 1.418e-04
iter: 2924, loss : 1.784e-04
iter: 2925, loss : 1.145e-04
iter: 2926, loss : 8.109e-05
iter: 2927, loss : 8.478e-04
iter: 2928, loss : 8.747e-05
iter: 2929, loss : 1.172e-04
iter: 2930, loss : 2.032e-04
iter: 2931, loss : 3.192e-04
iter: 2932, loss : 3.329e-04
iter: 2933, loss : 2.528e-03
iter: 2934, loss : 5.222e-04
iter: 2935, loss : 5.333e-04
iter: 2936, loss : 3.016e-04
iter: 2937, loss : 1.802e-03
iter: 2938, loss : 4.062e-04
iter: 2939, loss : 5.978e-04
iter: 2940, loss : 3.898e-04
iter: 2941, loss : 6.231e-04
iter: 2942, loss : 1.864e-04
iter: 2943, loss : 5.682e-04
iter: 2944, loss : 2.641e-04
iter: 2945, loss : 2.915e-04
iter: 2946, loss : 2.254e-04
iter: 2947, loss : 3.626e-04
iter: 2948, loss : 2.702e-04
iter: 2949, loss : 5.269e-05
iter: 2950, loss : 7.683e-05
iter: 2951, loss : 7.668e-05
iter: 2952, loss : 1.938e-03
iter: 2953, loss : 2.718e-04
iter: 2954, loss : 1.923e-04
iter: 2955, loss : 1.145e-04
iter: 2956, loss : 8.269e-05
iter: 2957, loss : 2.044e-04
iter: 2958, loss : 1.182e-04
iter: 2959, loss : 1.131e-03
iter: 2960, loss : 1.203e-03
iter: 2961, loss : 1.586e-04
iter: 2962, loss : 5.544e-04
iter: 2963, loss : 6.995e-04
iter: 2964, loss : 3.673e-04
iter: 2965, loss : 2.260e-04
iter: 2966, loss : 1.639e-04
iter: 2967, loss : 1.712e-04
iter: 2968, loss : 7.187e-05
iter: 2969, loss : 1.639e-04
iter: 2970, loss : 1.660e-04
iter: 2971, loss : 9.117e-05
iter: 2972, loss : 2.295e-03
iter: 2973, loss : 2.296e-03
iter: 2974, loss : 2.968e-04
iter: 2975, loss : 2.789e-04
iter: 2976, loss : 1.111e-04
iter: 2977, loss : 6.964e-04
iter: 2978, loss : 2.182e-04
iter: 2979, loss : 2.169e-04
iter: 2980, loss : 1.884e-04
iter: 2981, loss : 4.850e-04
iter: 2982, loss : 2.683e-04
iter: 2983, loss : 1.323e-04
iter: 2984, loss : 2.524e-04
iter: 2985, loss : 1.786e-04
iter: 2986, loss : 1.046e-04
iter: 2987, loss : 2.811e-03
iter: 2988, loss : 8.076e-04
iter: 2989, loss : 8.237e-05
iter: 2990, loss : 2.239e-04
iter: 2991, loss : 1.743e-03
iter: 2992, loss : 2.789e-04
iter: 2993, loss : 9.438e-05
iter: 2994, loss : 1.132e-04
iter: 2995, loss : 9.374e-04
iter: 2996, loss : 4.850e-05
iter: 2997, loss : 1.840e-04
iter: 2998, loss : 7.015e-04
iter: 2999, loss : 9.236e-05
iter: 2999, loss : 9.236e-05
--- 15.394429206848145 seconds --- to perform 3000 epochs
NOTE: For comparison we also called the torcheqdiff demo using the following command line ‘python ode_demo.py –niters 3000’. This triggers a comparable number of gradient evaluations on CPU. The iterations done in torcheqdiff complete in 348 seconds showing how on common CPUs, the use of heyoka may offer a significant speedup for NeuralODE research.
We are now ready to visualize the result of the learned neural ODE:
plt.plot(gt[:, 0], gt[:, 1])
ta_var.time = precision(0)
ta_var.state[:] = list(ic_var)
ta_var.state[:2] = ic
ta_var.pars[:] = np.array(nn_wb, dtype=np.single)
sol = ta_var.propagate_grid(np.array(t_grid, dtype=precision))[5]
plt.plot(sol[:, 0], sol[:, 1], ".")
[<matplotlib.lines.Line2D at 0x7fddd5b88aa0>]
It is also of interest to visualize the predictions of the trained system over a new random batch, as opposed to the predictions of the entire trajectory from the first observation:
batch_ic, batch_t_grid, batch_y = get_batch(t_grid, gt, batch_size_ic, batch_size_time)
plt.plot(gt[:, 0], gt[:, 1])
for i in range(batch_size_ic):
plt.plot(batch_y[:, i, 0], batch_y[:, i, 1], "k.")
start_time = time.time()
for batch in batched(batch_ic, batch_size_simd):
ta_var_b.set_time(precision(0.0))
ta_var_b.state[:, :] = deepcopy(ic_var_b)
ta_var_b.state[:dim, :] = np.array(batch).T
ta_var_b.pars[:] = np.tile(nn_wb, (batch_size_simd, 1)).T
sol_b = ta_var_b.propagate_grid(np.array(t_grid_b, dtype=precision))[1]
for sol in [sol_b[:, :, i] for i in range(batch_size_simd)]:
plt.plot(sol[:, 0], sol[:, 1], ".")
print(
"--- %s seconds --- to compute prediction over the whole batch"
% (time.time() - start_time)
)
--- 0.008916378021240234 seconds --- to compute prediction over the whole batch
