Neural ODEs - II#
We here implement, in heyoka, a standard traning pipeline for NeuralODE replicating this torcheqdiff demo.
We will be using the very same parameters and dynamics, of the original demo but only use a simple fixed learning rate stochastic gradient descent. Note that it is also possible to use more advanced optimizers such as those implemented in pytorch while computing the gradients using heyoka, but in this tutorial we keep it simple and just update the gradient naively.
See also:
# The main imports
import numpy as np
import time
from itertools import batched
from copy import deepcopy
# Import Heyoka
import heyoka as hy
%matplotlib inline
import matplotlib.pyplot as plt
Generating the ground truth#
The dynamics we consider is a simple third order system:
\[\begin{split}
\left\{
\begin{array}{l}
\dot x = -0.1 x^3 - 2y^3 \\
\dot y = 2 x^3 - 0.1y^3
\end{array}
\right.
\end{split}\]
which we implement in heyoka as follows:
# The symolic variables representing the system state.
x, y = hy.make_vars("x", "y")
# We now assemble the dynamics.
A = np.array([[-0.1, 2.0], [-2.0, -0.1]])
tmp = np.array([[x * x * x, y * y * y]]) @ A
dyn = [(x, tmp[0][0]), (y, tmp[0][1])]
# For convenience we store here the dimension of the state.
dim = len(dyn)
To generate groundtruth observations from this system we will use the following parameters:
# This is the number of observations (i.e. the ground truth).
data_size = 1000
# This is the initial condition used to generate the observations (i.e. the ground truth).
ic = [2.0, 0.0]
# This is the time grid used to generate the observations (i.e. the ground truth).
t_grid = np.linspace(0.0, 25, data_size)
We start by instantiating a heyoka.taylor_adaptive object that will generate our observations. These will be considered as ground truth. Note that in other settings this data may be observed from a real phenomena, rather than generated numerically. In that case there will be noise which is here not modelled.
start_time = time.time()
ta = hy.taylor_adaptive(
# The ODEs.
dyn,
# The initial conditions.
ic,
# Do not operate in compact mode.
compact_mode=False,
# Define the tolerance
tol=1e-18,
)
print(
"--- %s seconds --- to build (jit) the Taylor integrator"
% (time.time() - start_time)
)
--- 0.06826591491699219 seconds --- to build (jit) the Taylor integrator
We are thus ready to generate the the ground truth using the method propagate_grid method of the Taylor adaptive integrator.
# It is good practice to always set initial conditions explicitly
ta.time = 0
ta.state[:] = ic
# This is our ground truth (gt)
gt = ta.propagate_grid(t_grid)[5]
Let us have a look at the generated observations.
plt.plot(gt[:, 0], gt[:, 1], "k.")
[<matplotlib.lines.Line2D at 0x7fa15c1cbda0>]
Instantiating the neural ODE#
So far we have only generated the data we want to teach our Neural ODE to replicate. We now need to instantiate the Neural ODE system so that we can make predictions and compute the gradient of our loss.
Let us start with instantiating a feed-forward neural network using heyoka.models.ffnn.
# We define as nonlinearity a simple linear layer
linear = lambda inp: inp
# We call the factory to construct a FFNN. Following the torcheqdiff example here we put cubes as inputs:
ffnn = hy.model.ffnn(
inputs=[x * x * x, y * y * y],
nn_hidden=[50],
n_out=2,
activations=[hy.tanh, linear],
)
We are now able to build a taylor_adaptive_batch numerical integrator which will propagate the dynamics in batches as to leverage SIMD instructions:
\[\begin{split}
\left\{
\begin{array}{l}
\dot x = \mathcal N_{\theta, x}(x^3, y^3) \\
\dot y = \mathcal N_{\theta, y}(x^3, y^3) \\
\end{array}
\right.
\end{split}\]
And since we will want to be able to compute the sensitivities of this ODE we are first going to augment such dynamics with the first order variational equations:
# We decide on whether to use single or double precision. In this example it may
# Not be too important, but in other settings it can give a speedup when connected with batch and ensamble propagation.
# The single precision does effect, though, the quality of thegradient direction.
precision = np.double # np.single
batch_size_simd = hy.recommended_simd_size(fp_type=precision)
print("Batch size for SIMD use in integration: ", batch_size_simd)
# We thus define the neural dynamics
dyn_n = [(x, ffnn[0]), (y, ffnn[1])]
# We augment it with the variational one (only order 1 - i.e. gradient)
var_dyn_n = hy.var_ode_sys(dyn_n, args=hy.var_args.params, order=1)
Batch size for SIMD use in integration: 4
… and instantiate the variational integrator. For convenience we also instantiate the non batched version
# This is the batched version, which we will use for gradient computation
ta_var_b = hy.taylor_adaptive_batch(
# The ODEs.
var_dyn_n,
# The initial conditions.
np.ones((dim, batch_size_simd), dtype=precision),
# Operate in compact mode.
compact_mode=True,
# Define the tolerance (low tolerance is enough here)
tol=precision(1e-4),
# Single precision is enough here
fp_type=precision,
)
# This is the normal version used for wuick plotting and visualization of a single solution
ta_var = hy.taylor_adaptive(
# The ODEs.
var_dyn_n,
# The initial conditions.
ic,
# Operate in compact mode.
compact_mode=True,
# Define the tolerance (low tolerance is enough here)
tol=precision(1e-4),
# Single precision is enough here
fp_type=precision,
)
# We store for convenience the initial conditions in a numpy array (including the variational state)
ic_var_b = np.array(ta_var_b.state)
ic_var = np.array(ta_var.state)
Before starting the training, we define the details on the batch we are going to use in the generic step of stochastic gradient descent:
# This is the number of initial conditions to use in one batch (M)
batch_size_ic = 20
# This is the number of observations to predict from each sampled initial condition (T)
batch_size_time = 10
# We check that the observation batch size is a multiple of the simd batch size
print("This must be an integer: ", batch_size_ic / batch_size_simd)
This must be an integer: 5.0
This helper function creates a randomly generated batch from the available observation:
def get_batch(t_grid, gt, batch_size_ic=20, batch_size_time=10):
# We select the initial conditions from which generate predictions
s = np.random.choice(
np.arange(gt.shape[0] - batch_size_time, dtype=np.int64),
batch_size_ic,
replace=False,
)
batch_ic = gt[s, :] # (M, D)
# Assuming uniform grid and a non-autonomous system, all predictions will be made on the same time grid
batch_t_grid = t_grid[:batch_size_time] # (T)
batch_y = np.stack([gt[s + i] for i in range(batch_size_time)]) # (T,M,D)
return batch_ic, batch_t_grid, batch_y
Let us visualize what a batch looks like. First we generate one:
batch_ic, batch_t_grid, batch_y = get_batch(t_grid, gt, batch_size_ic, batch_size_time)
# We need the tgrid repeated as per heyoka API of the taylor adaptive batch
t_grid_b = np.repeat(batch_t_grid, batch_size_simd).reshape(-1, batch_size_simd)
Then we plot it overlapped to all the observations:
plt.plot(gt[:, 0], gt[:, 1])
for i in range(batch_size_ic):
plt.plot(batch_y[:, i, 0], batch_y[:, i, 1], "k.")
We initialize the network weights and biases.
# We compute initialization values for weigths / biases
n_pars = len(ta_var_b.pars)
nn_wb = np.random.normal(loc=0, scale=0.1, size=(n_pars,))
And visualize the prediction of the starting random network.
plt.plot(gt[:, 0], gt[:, 1])
for i in range(batch_size_ic):
plt.plot(batch_y[:, i, 0], batch_y[:, i, 1], "k.")
start_time = time.time()
for batch in batched(batch_ic, batch_size_simd):
ta_var_b.set_time(precision(0.0))
ta_var_b.state[:, :] = deepcopy(ic_var_b)
ta_var_b.state[:dim, :] = np.array(batch).T
ta_var_b.pars[:] = np.tile(nn_wb, (batch_size_simd, 1)).T
sol_b = ta_var_b.propagate_grid(np.array(t_grid_b, dtype=precision))[1]
for sol in [sol_b[:, :, i] for i in range(batch_size_simd)]:
plt.plot(sol[:, 0], sol[:, 1], ".")
print(
"--- %s seconds --- to compute prediction over the whole batch"
% (time.time() - start_time)
)
--- 0.006998777389526367 seconds --- to compute prediction over the whole batch
Training#
To train we need to first compute the gradient of the loss over the batch (M initial conditions and T times). Formally
\[
\mathcal L_{\theta} = \sum_{i=0}^{M}\sum_{j=0}^T (\pmb y_{ij} - \hat {\pmb y}_{ij})\cdot (\pmb y_{ij} - \hat {\pmb y}_{ij})
\]
\[
\frac{\partial L_\theta}{\partial \theta} = 2 \sum_{i=0}^{M}\sum_{j=0}^T (\pmb y_{ij} - \hat {\pmb y}_{ij})\cdot \frac{\partial \pmb y_{ij}}{\partial \theta}
\]
This helper function computes the loss and the gradient over one batch. The gradient is computed rearranging the ODE sensitivities computed propagating the variational dynamics:
def loss_and_gradient(nn_wb, batch_ic, batch_y):
# We reset the batch_loss value
batch_loss = 0
# We reset the gradient loss
grad = np.array([0.0] * ta_var_b.pars.shape[0])
counter = 0
# We process the batch (of the dataset) in batches for SIMD efficient processing
for batch_simd in batched(batch_ic, batch_size_simd):
ta_var_b.set_time(precision(0.0))
ta_var_b.state[:, :] = deepcopy(ic_var_b)
ta_var_b.state[:dim, :] = np.array(batch_simd).T
ta_var_b.pars[:] = np.tile(nn_wb, (batch_size_simd, 1)).T
sol_b = ta_var_b.propagate_grid(np.array(t_grid_b, dtype=precision))[1]
# The numerical integration on the simd_batch is done, we process its result
# accumulating loss and gradient on the predictions
for sol in [sol_b[:, :, i] for i in range(batch_size_simd)]:
# Here is the term (y-y_hat)
diff = sol[:, :dim] - batch_y[:, counter, :]
counter += 1
# Which is then summed sum sum (y-y_hat).(y-y_hat)
batch_loss += np.sum(diff**2)
# And the gradient computed as 2 sum (y-y_hat).dy
for dy in sol[:, dim:].reshape(batch_y.shape[0], dim, -1):
grad += 2 * np.sum(diff @ dy, axis=0)
# We then take the mean over the points used
batch_loss /= batch_size_ic * batch_size_time
grad /= batch_size_ic * batch_size_time
return batch_loss, grad
Using the various blocks defined above the training loop takes the form:
# We reset initialization values for weigths / biases
n_pars = len(ta_var_b.pars)
nn_wb = np.random.normal(loc=0, scale=0.1, size=(n_pars,))
# We loop over the epochs
n_epochs = 3000
start_time = time.time()
for i in range(n_epochs):
# 1 - generate a batch
batch_ic, batch_t_grid, batch_y = get_batch(
t_grid, gt, batch_size_ic=batch_size_ic, batch_size_time=batch_size_time
)
# 2 - compute the loss and gradient on the batch
batch_loss, grad = loss_and_gradient(nn_wb, batch_ic, batch_y)
print(f"iter: {i}, loss : {batch_loss:.3e}", end="\r")
# 3 - update the weights and biases (learning rate here is fixed, we use a naive update rule which in this case works)
nn_wb = nn_wb - 0.1 * grad
print(f"iter: {i}, loss : {batch_loss:.3e}", end="\n")
print("--- %s seconds --- to perform %i epochs" % (time.time() - start_time, n_epochs))
iter: 0, loss : 4.776e-02
iter: 1, loss : 1.638e-02
iter: 2, loss : 9.544e-02
iter: 3, loss : 1.895e-01
iter: 4, loss : 1.485e-02
iter: 5, loss : 2.097e-02
iter: 6, loss : 1.046e-01
iter: 7, loss : 3.599e-02
iter: 8, loss : 2.973e-01
iter: 9, loss : 1.989e-01
iter: 10, loss : 3.829e-02
iter: 11, loss : 6.157e-02
iter: 12, loss : 4.516e-02
iter: 13, loss : 8.587e-02
iter: 14, loss : 4.544e-02
iter: 15, loss : 1.849e-02
iter: 16, loss : 4.297e-02
iter: 17, loss : 2.527e-02
iter: 18, loss : 1.614e-01
iter: 19, loss : 2.306e-02
iter: 20, loss : 1.078e-02
iter: 21, loss : 4.993e-02
iter: 22, loss : 3.139e-02
iter: 23, loss : 1.414e-01
iter: 24, loss : 1.614e-02
iter: 25, loss : 3.248e-02
iter: 26, loss : 1.489e-02
iter: 27, loss : 7.074e-02
iter: 28, loss : 9.363e-02
iter: 29, loss : 6.774e-02
iter: 30, loss : 2.395e-02
iter: 31, loss : 1.307e-02
iter: 32, loss : 1.043e-02
iter: 33, loss : 6.348e-02
iter: 34, loss : 3.917e-03
iter: 35, loss : 4.816e-02
iter: 36, loss : 6.040e-02
iter: 37, loss : 3.614e-02
iter: 38, loss : 7.419e-02
iter: 39, loss : 3.443e-02
iter: 40, loss : 2.996e-02
iter: 41, loss : 3.151e-02
iter: 42, loss : 7.759e-02
iter: 43, loss : 7.372e-03
iter: 44, loss : 5.348e-02
iter: 45, loss : 8.601e-03
iter: 46, loss : 9.775e-03
iter: 47, loss : 3.389e-02
iter: 48, loss : 1.927e-02
iter: 49, loss : 3.765e-03
iter: 50, loss : 2.771e-03
iter: 51, loss : 2.027e-02
iter: 52, loss : 4.086e-02
iter: 53, loss : 1.738e-03
iter: 54, loss : 1.309e-02
iter: 55, loss : 2.674e-02
iter: 56, loss : 5.129e-02
iter: 57, loss : 3.592e-02
iter: 58, loss : 6.711e-03
iter: 59, loss : 2.074e-02
iter: 60, loss : 7.103e-03
iter: 61, loss : 4.428e-04
iter: 62, loss : 7.562e-03
iter: 63, loss : 4.172e-04
iter: 64, loss : 6.384e-02
iter: 65, loss : 6.839e-03
iter: 66, loss : 6.451e-03
iter: 67, loss : 2.022e-03
iter: 68, loss : 2.623e-04
iter: 69, loss : 5.102e-04
iter: 70, loss : 1.489e-02
iter: 71, loss : 2.223e-04
iter: 72, loss : 4.701e-03
iter: 73, loss : 6.618e-03
iter: 74, loss : 5.311e-03
iter: 75, loss : 4.763e-03
iter: 76, loss : 3.638e-03
iter: 77, loss : 7.889e-03
iter: 78, loss : 8.618e-03
iter: 79, loss : 2.300e-03
iter: 80, loss : 2.891e-03
iter: 81, loss : 8.911e-03
iter: 82, loss : 8.831e-04
iter: 83, loss : 8.799e-03
iter: 84, loss : 4.634e-03
iter: 85, loss : 4.675e-04
iter: 86, loss : 1.556e-03
iter: 87, loss : 2.444e-02
iter: 88, loss : 7.658e-04
iter: 89, loss : 1.833e-02
iter: 90, loss : 6.797e-05
iter: 91, loss : 2.557e-02
iter: 92, loss : 2.026e-02
iter: 93, loss : 1.548e-02
iter: 94, loss : 1.086e-03
iter: 95, loss : 1.354e-02
iter: 96, loss : 2.453e-04
iter: 97, loss : 1.142e-03
iter: 98, loss : 8.279e-04
iter: 99, loss : 6.788e-03
iter: 100, loss : 4.290e-03
iter: 101, loss : 1.661e-03
iter: 102, loss : 2.594e-03
iter: 103, loss : 5.253e-03
iter: 104, loss : 5.220e-02
iter: 105, loss : 3.054e-03
iter: 106, loss : 3.057e-03
iter: 107, loss : 3.994e-04
iter: 108, loss : 5.443e-04
iter: 109, loss : 6.877e-04
iter: 110, loss : 8.447e-04
iter: 111, loss : 1.571e-02
iter: 112, loss : 1.036e-03
iter: 113, loss : 1.129e-03
iter: 114, loss : 3.505e-04
iter: 115, loss : 5.631e-04
iter: 116, loss : 6.630e-03
iter: 117, loss : 3.944e-03
iter: 118, loss : 5.259e-04
iter: 119, loss : 2.632e-04
iter: 120, loss : 8.512e-04
iter: 121, loss : 5.134e-04
iter: 122, loss : 3.193e-03
iter: 123, loss : 5.191e-04
iter: 124, loss : 4.990e-04
iter: 125, loss : 2.988e-03
iter: 126, loss : 4.301e-04
iter: 127, loss : 6.381e-04
iter: 128, loss : 3.596e-04
iter: 129, loss : 1.973e-02
iter: 130, loss : 7.693e-04
iter: 131, loss : 9.089e-04
iter: 132, loss : 1.463e-02
iter: 133, loss : 1.243e-02
iter: 134, loss : 8.253e-04
iter: 135, loss : 3.655e-03
iter: 136, loss : 1.315e-02
iter: 137, loss : 8.821e-04
iter: 138, loss : 8.808e-03
iter: 139, loss : 7.896e-04
iter: 140, loss : 2.199e-03
iter: 141, loss : 6.497e-04
iter: 142, loss : 1.409e-03
iter: 143, loss : 7.488e-04
iter: 144, loss : 2.076e-03
iter: 145, loss : 2.453e-02
iter: 146, loss : 1.305e-02
iter: 147, loss : 1.024e-03
iter: 148, loss : 5.892e-04
iter: 149, loss : 1.016e-03
iter: 150, loss : 6.693e-03
iter: 151, loss : 1.006e-03
iter: 152, loss : 1.107e-03
iter: 153, loss : 1.341e-02
iter: 154, loss : 1.098e-03
iter: 155, loss : 8.306e-04
iter: 156, loss : 1.480e-03
iter: 157, loss : 1.231e-03
iter: 158, loss : 1.045e-02
iter: 159, loss : 6.646e-04
iter: 160, loss : 1.653e-03
iter: 161, loss : 5.090e-04
iter: 162, loss : 1.456e-02
iter: 163, loss : 9.717e-04
iter: 164, loss : 6.657e-04
iter: 165, loss : 3.391e-03
iter: 166, loss : 8.104e-04
iter: 167, loss : 1.310e-03
iter: 168, loss : 9.515e-03
iter: 169, loss : 1.134e-03
iter: 170, loss : 1.205e-03
iter: 171, loss : 1.134e-02
iter: 172, loss : 9.423e-03
iter: 173, loss : 1.579e-02
iter: 174, loss : 9.895e-04
iter: 175, loss : 2.207e-03
iter: 176, loss : 3.014e-03
iter: 177, loss : 1.332e-03
iter: 178, loss : 1.388e-03
iter: 179, loss : 1.766e-03
iter: 180, loss : 7.678e-03
iter: 181, loss : 1.209e-03
iter: 182, loss : 2.876e-03
iter: 183, loss : 9.958e-04
iter: 184, loss : 4.525e-03
iter: 185, loss : 9.355e-04
iter: 186, loss : 8.510e-04
iter: 187, loss : 3.708e-03
iter: 188, loss : 2.540e-03
iter: 189, loss : 1.540e-03
iter: 190, loss : 8.604e-04
iter: 191, loss : 5.234e-04
iter: 192, loss : 4.095e-04
iter: 193, loss : 2.420e-03
iter: 194, loss : 1.020e-02
iter: 195, loss : 1.574e-03
iter: 196, loss : 8.792e-04
iter: 197, loss : 3.871e-03
iter: 198, loss : 8.832e-04
iter: 199, loss : 7.760e-03
iter: 200, loss : 1.202e-03
iter: 201, loss : 7.980e-04
iter: 202, loss : 7.652e-04
iter: 203, loss : 2.312e-03
iter: 204, loss : 7.485e-03
iter: 205, loss : 7.974e-03
iter: 206, loss : 1.494e-03
iter: 207, loss : 3.451e-03
iter: 208, loss : 1.041e-02
iter: 209, loss : 1.467e-03
iter: 210, loss : 1.552e-03
iter: 211, loss : 1.174e-03
iter: 212, loss : 6.523e-03
iter: 213, loss : 1.683e-03
iter: 214, loss : 7.697e-04
iter: 215, loss : 7.688e-04
iter: 216, loss : 1.223e-03
iter: 217, loss : 9.177e-04
iter: 218, loss : 1.085e-03
iter: 219, loss : 7.639e-03
iter: 220, loss : 1.967e-03
iter: 221, loss : 1.071e-03
iter: 222, loss : 8.055e-04
iter: 223, loss : 7.285e-04
iter: 224, loss : 5.508e-04
iter: 225, loss : 4.615e-03
iter: 226, loss : 7.224e-04
iter: 227, loss : 1.318e-03
iter: 228, loss : 1.645e-03
iter: 229, loss : 9.089e-04
iter: 230, loss : 1.445e-02
iter: 231, loss : 1.610e-03
iter: 232, loss : 6.971e-04
iter: 233, loss : 1.419e-02
iter: 234, loss : 1.214e-03
iter: 235, loss : 1.002e-03
iter: 236, loss : 1.048e-02
iter: 237, loss : 5.636e-03
iter: 238, loss : 7.899e-04
iter: 239, loss : 2.152e-03
iter: 240, loss : 5.278e-04
iter: 241, loss : 3.445e-03
iter: 242, loss : 1.482e-03
iter: 243, loss : 8.127e-03
iter: 244, loss : 2.138e-03
iter: 245, loss : 5.141e-04
iter: 246, loss : 3.219e-04
iter: 247, loss : 1.008e-03
iter: 248, loss : 6.865e-04
iter: 249, loss : 8.908e-04
iter: 250, loss : 9.021e-04
iter: 251, loss : 6.502e-03
iter: 252, loss : 8.320e-03
iter: 253, loss : 3.107e-03
iter: 254, loss : 5.813e-03
iter: 255, loss : 6.804e-03
iter: 256, loss : 1.085e-03
iter: 257, loss : 7.049e-04
iter: 258, loss : 1.035e-02
iter: 259, loss : 2.203e-03
iter: 260, loss : 1.660e-03
iter: 261, loss : 9.198e-04
iter: 262, loss : 2.820e-03
iter: 263, loss : 5.772e-04
iter: 264, loss : 9.693e-04
iter: 265, loss : 7.068e-04
iter: 266, loss : 8.975e-04
iter: 267, loss : 1.126e-03
iter: 268, loss : 2.821e-03
iter: 269, loss : 1.243e-03
iter: 270, loss : 5.468e-04
iter: 271, loss : 2.034e-03
iter: 272, loss : 3.833e-03
iter: 273, loss : 8.926e-04
iter: 274, loss : 1.236e-03
iter: 275, loss : 8.843e-03
iter: 276, loss : 1.215e-02
iter: 277, loss : 1.647e-03
iter: 278, loss : 1.969e-03
iter: 279, loss : 2.389e-03
iter: 280, loss : 6.720e-04
iter: 281, loss : 7.939e-03
iter: 282, loss : 6.501e-04
iter: 283, loss : 1.095e-03
iter: 284, loss : 1.938e-03
iter: 285, loss : 9.009e-04
iter: 286, loss : 9.381e-04
iter: 287, loss : 5.013e-04
iter: 288, loss : 1.613e-03
iter: 289, loss : 9.771e-04
iter: 290, loss : 7.003e-03
iter: 291, loss : 8.502e-04
iter: 292, loss : 1.097e-03
iter: 293, loss : 1.189e-03
iter: 294, loss : 8.243e-04
iter: 295, loss : 9.132e-04
iter: 296, loss : 6.145e-03
iter: 297, loss : 1.217e-02
iter: 298, loss : 7.058e-04
iter: 299, loss : 7.689e-04
iter: 300, loss : 6.637e-04
iter: 301, loss : 1.226e-03
iter: 302, loss : 4.095e-04
iter: 303, loss : 3.344e-03
iter: 304, loss : 4.526e-03
iter: 305, loss : 2.453e-03
iter: 306, loss : 9.893e-04
iter: 307, loss : 3.456e-04
iter: 308, loss : 4.810e-04
iter: 309, loss : 4.921e-04
iter: 310, loss : 7.527e-04
iter: 311, loss : 1.084e-02
iter: 312, loss : 1.160e-03
iter: 313, loss : 6.643e-03
iter: 314, loss : 1.279e-03
iter: 315, loss : 4.543e-04
iter: 316, loss : 7.592e-03
iter: 317, loss : 1.042e-03
iter: 318, loss : 6.752e-04
iter: 319, loss : 7.967e-04
iter: 320, loss : 4.478e-04
iter: 321, loss : 6.628e-03
iter: 322, loss : 8.395e-04
iter: 323, loss : 5.532e-03
iter: 324, loss : 9.138e-04
iter: 325, loss : 5.842e-04
iter: 326, loss : 7.227e-04
iter: 327, loss : 8.579e-03
iter: 328, loss : 9.578e-04
iter: 329, loss : 5.846e-04
iter: 330, loss : 6.279e-04
iter: 331, loss : 4.950e-04
iter: 332, loss : 3.848e-04
iter: 333, loss : 6.338e-03
iter: 334, loss : 5.420e-04
iter: 335, loss : 6.284e-04
iter: 336, loss : 8.176e-04
iter: 337, loss : 1.231e-03
iter: 338, loss : 7.630e-04
iter: 339, loss : 1.103e-02
iter: 340, loss : 6.642e-04
iter: 341, loss : 3.771e-04
iter: 342, loss : 1.499e-03
iter: 343, loss : 8.618e-04
iter: 344, loss : 9.864e-04
iter: 345, loss : 1.353e-03
iter: 346, loss : 3.882e-04
iter: 347, loss : 4.132e-04
iter: 348, loss : 1.363e-03
iter: 349, loss : 9.525e-04
iter: 350, loss : 2.495e-04
iter: 351, loss : 4.378e-04
iter: 352, loss : 5.110e-04
iter: 353, loss : 5.473e-04
iter: 354, loss : 2.316e-04
iter: 355, loss : 3.778e-04
iter: 356, loss : 2.626e-04
iter: 357, loss : 1.628e-03
iter: 358, loss : 5.312e-04
iter: 359, loss : 4.344e-04
iter: 360, loss : 4.156e-04
iter: 361, loss : 3.957e-04
iter: 362, loss : 3.323e-04
iter: 363, loss : 4.749e-03
iter: 364, loss : 4.292e-04
iter: 365, loss : 5.193e-04
iter: 366, loss : 1.508e-03
iter: 367, loss : 4.817e-04
iter: 368, loss : 4.911e-04
iter: 369, loss : 2.770e-03
iter: 370, loss : 4.385e-03
iter: 371, loss : 4.215e-04
iter: 372, loss : 1.438e-02
iter: 373, loss : 4.758e-04
iter: 374, loss : 6.059e-04
iter: 375, loss : 6.814e-04
iter: 376, loss : 1.104e-02
iter: 377, loss : 1.048e-03
iter: 378, loss : 4.257e-04
iter: 379, loss : 7.317e-04
iter: 380, loss : 1.956e-04
iter: 381, loss : 7.611e-04
iter: 382, loss : 4.251e-04
iter: 383, loss : 4.492e-04
iter: 384, loss : 3.864e-04
iter: 385, loss : 1.030e-03
iter: 386, loss : 5.647e-04
iter: 387, loss : 5.327e-04
iter: 388, loss : 5.673e-04
iter: 389, loss : 1.093e-03
iter: 390, loss : 5.709e-03
iter: 391, loss : 3.619e-03
iter: 392, loss : 8.785e-04
iter: 393, loss : 5.786e-03
iter: 394, loss : 3.542e-04
iter: 395, loss : 1.883e-03
iter: 396, loss : 4.644e-04
iter: 397, loss : 5.273e-03
iter: 398, loss : 4.227e-04
iter: 399, loss : 2.138e-03
iter: 400, loss : 1.047e-03
iter: 401, loss : 4.037e-04
iter: 402, loss : 7.276e-03
iter: 403, loss : 9.144e-04
iter: 404, loss : 1.778e-03
iter: 405, loss : 1.143e-03
iter: 406, loss : 8.469e-04
iter: 407, loss : 3.277e-04
iter: 408, loss : 1.847e-03
iter: 409, loss : 5.001e-04
iter: 410, loss : 4.165e-04
iter: 411, loss : 5.757e-04
iter: 412, loss : 6.248e-03
iter: 413, loss : 2.533e-04
iter: 414, loss : 1.977e-03
iter: 415, loss : 8.605e-03
iter: 416, loss : 1.139e-03
iter: 417, loss : 4.072e-04
iter: 418, loss : 7.482e-04
iter: 419, loss : 1.028e-02
iter: 420, loss : 6.287e-04
iter: 421, loss : 9.759e-04
iter: 422, loss : 6.472e-03
iter: 423, loss : 4.172e-04
iter: 424, loss : 8.145e-04
iter: 425, loss : 3.889e-04
iter: 426, loss : 2.974e-03
iter: 427, loss : 7.413e-04
iter: 428, loss : 9.928e-04
iter: 429, loss : 3.543e-03
iter: 430, loss : 1.094e-03
iter: 431, loss : 1.025e-03
iter: 432, loss : 2.704e-04
iter: 433, loss : 9.616e-04
iter: 434, loss : 7.361e-04
iter: 435, loss : 3.479e-04
iter: 436, loss : 8.757e-03
iter: 437, loss : 3.677e-04
iter: 438, loss : 4.927e-03
iter: 439, loss : 4.432e-03
iter: 440, loss : 5.940e-04
iter: 441, loss : 5.503e-03
iter: 442, loss : 5.201e-04
iter: 443, loss : 7.550e-04
iter: 444, loss : 1.832e-03
iter: 445, loss : 3.774e-04
iter: 446, loss : 2.744e-04
iter: 447, loss : 4.894e-03
iter: 448, loss : 4.116e-03
iter: 449, loss : 6.491e-04
iter: 450, loss : 1.458e-03
iter: 451, loss : 2.901e-04
iter: 452, loss : 7.405e-04
iter: 453, loss : 4.265e-03
iter: 454, loss : 6.055e-04
iter: 455, loss : 2.836e-03
iter: 456, loss : 1.215e-03
iter: 457, loss : 6.020e-04
iter: 458, loss : 4.531e-04
iter: 459, loss : 2.733e-03
iter: 460, loss : 5.083e-04
iter: 461, loss : 6.284e-04
iter: 462, loss : 3.634e-04
iter: 463, loss : 5.532e-03
iter: 464, loss : 2.328e-03
iter: 465, loss : 3.594e-04
iter: 466, loss : 5.024e-04
iter: 467, loss : 5.815e-04
iter: 468, loss : 6.674e-04
iter: 469, loss : 3.731e-03
iter: 470, loss : 2.347e-03
iter: 471, loss : 7.273e-03
iter: 472, loss : 8.987e-04
iter: 473, loss : 9.112e-04
iter: 474, loss : 3.647e-04
iter: 475, loss : 1.204e-03
iter: 476, loss : 9.791e-03
iter: 477, loss : 2.460e-03
iter: 478, loss : 6.011e-04
iter: 479, loss : 1.129e-03
iter: 480, loss : 2.531e-03
iter: 481, loss : 7.139e-04
iter: 482, loss : 4.044e-04
iter: 483, loss : 4.209e-04
iter: 484, loss : 4.183e-04
iter: 485, loss : 2.682e-04
iter: 486, loss : 3.655e-04
iter: 487, loss : 9.745e-03
iter: 488, loss : 2.417e-03
iter: 489, loss : 5.370e-04
iter: 490, loss : 5.409e-03
iter: 491, loss : 1.867e-03
iter: 492, loss : 5.170e-04
iter: 493, loss : 6.065e-04
iter: 494, loss : 9.542e-03
iter: 495, loss : 3.161e-04
iter: 496, loss : 8.560e-04
iter: 497, loss : 1.158e-02
iter: 498, loss : 8.756e-04
iter: 499, loss : 4.165e-03
iter: 500, loss : 6.261e-04
iter: 501, loss : 8.305e-03
iter: 502, loss : 4.237e-04
iter: 503, loss : 4.620e-04
iter: 504, loss : 6.278e-04
iter: 505, loss : 4.543e-04
iter: 506, loss : 6.139e-04
iter: 507, loss : 6.833e-04
iter: 508, loss : 3.888e-04
iter: 509, loss : 4.211e-04
iter: 510, loss : 8.077e-03
iter: 511, loss : 3.497e-04
iter: 512, loss : 2.815e-04
iter: 513, loss : 6.325e-04
iter: 514, loss : 8.717e-04
iter: 515, loss : 5.884e-04
iter: 516, loss : 6.528e-04
iter: 517, loss : 4.525e-04
iter: 518, loss : 1.312e-02
iter: 519, loss : 7.997e-04
iter: 520, loss : 6.376e-04
iter: 521, loss : 5.468e-03
iter: 522, loss : 3.651e-04
iter: 523, loss : 8.245e-04
iter: 524, loss : 4.802e-03
iter: 525, loss : 7.688e-04
iter: 526, loss : 3.540e-04
iter: 527, loss : 1.099e-03
iter: 528, loss : 5.643e-03
iter: 529, loss : 1.001e-03
iter: 530, loss : 2.820e-03
iter: 531, loss : 4.762e-04
iter: 532, loss : 1.369e-03
iter: 533, loss : 4.872e-04
iter: 534, loss : 4.921e-03
iter: 535, loss : 7.312e-04
iter: 536, loss : 6.484e-04
iter: 537, loss : 4.054e-03
iter: 538, loss : 2.045e-04
iter: 539, loss : 8.371e-04
iter: 540, loss : 6.444e-04
iter: 541, loss : 5.381e-04
iter: 542, loss : 2.943e-04
iter: 543, loss : 6.673e-04
iter: 544, loss : 4.916e-03
iter: 545, loss : 4.628e-03
iter: 546, loss : 6.054e-04
iter: 547, loss : 2.685e-03
iter: 548, loss : 5.442e-04
iter: 549, loss : 3.688e-04
iter: 550, loss : 1.245e-03
iter: 551, loss : 2.055e-03
iter: 552, loss : 4.088e-03
iter: 553, loss : 3.384e-04
iter: 554, loss : 7.373e-04
iter: 555, loss : 7.309e-04
iter: 556, loss : 1.500e-03
iter: 557, loss : 2.358e-03
iter: 558, loss : 2.692e-03
iter: 559, loss : 7.147e-04
iter: 560, loss : 5.671e-04
iter: 561, loss : 7.500e-04
iter: 562, loss : 3.627e-04
iter: 563, loss : 9.754e-04
iter: 564, loss : 6.201e-03
iter: 565, loss : 2.568e-03
iter: 566, loss : 3.579e-04
iter: 567, loss : 2.428e-04
iter: 568, loss : 7.188e-04
iter: 569, loss : 4.979e-03
iter: 570, loss : 6.535e-04
iter: 571, loss : 1.873e-04
iter: 572, loss : 4.043e-04
iter: 573, loss : 1.196e-03
iter: 574, loss : 2.742e-04
iter: 575, loss : 3.057e-03
iter: 576, loss : 4.870e-04
iter: 577, loss : 1.175e-03
iter: 578, loss : 2.106e-04
iter: 579, loss : 7.537e-04
iter: 580, loss : 4.407e-04
iter: 581, loss : 2.618e-04
iter: 582, loss : 3.173e-04
iter: 583, loss : 5.206e-04
iter: 584, loss : 5.028e-04
iter: 585, loss : 8.525e-03
iter: 586, loss : 1.088e-03
iter: 587, loss : 7.632e-04
iter: 588, loss : 5.137e-04
iter: 589, loss : 1.345e-03
iter: 590, loss : 1.270e-03
iter: 591, loss : 7.187e-04
iter: 592, loss : 7.681e-04
iter: 593, loss : 4.529e-04
iter: 594, loss : 2.700e-04
iter: 595, loss : 4.037e-03
iter: 596, loss : 1.163e-03
iter: 597, loss : 6.446e-04
iter: 598, loss : 6.785e-04
iter: 599, loss : 2.989e-04
iter: 600, loss : 4.630e-04
iter: 601, loss : 3.852e-03
iter: 602, loss : 1.851e-03
iter: 603, loss : 3.205e-04
iter: 604, loss : 4.370e-04
iter: 605, loss : 3.638e-04
iter: 606, loss : 6.450e-04
iter: 607, loss : 8.030e-04
iter: 608, loss : 4.303e-04
iter: 609, loss : 2.050e-04
iter: 610, loss : 6.789e-04
iter: 611, loss : 4.607e-04
iter: 612, loss : 1.537e-03
iter: 613, loss : 1.434e-03
iter: 614, loss : 2.649e-04
iter: 615, loss : 3.197e-03
iter: 616, loss : 4.021e-04
iter: 617, loss : 4.235e-04
iter: 618, loss : 1.146e-03
iter: 619, loss : 3.678e-04
iter: 620, loss : 2.019e-03
iter: 621, loss : 3.476e-03
iter: 622, loss : 3.031e-04
iter: 623, loss : 1.140e-03
iter: 624, loss : 1.194e-03
iter: 625, loss : 4.746e-04
iter: 626, loss : 4.896e-03
iter: 627, loss : 7.215e-04
iter: 628, loss : 2.094e-03
iter: 629, loss : 3.497e-04
iter: 630, loss : 1.473e-03
iter: 631, loss : 1.707e-03
iter: 632, loss : 5.561e-04
iter: 633, loss : 6.403e-04
iter: 634, loss : 1.395e-03
iter: 635, loss : 1.150e-03
iter: 636, loss : 3.569e-03
iter: 637, loss : 5.806e-04
iter: 638, loss : 8.503e-04
iter: 639, loss : 1.741e-04
iter: 640, loss : 4.342e-04
iter: 641, loss : 7.432e-04
iter: 642, loss : 1.545e-04
iter: 643, loss : 1.608e-03
iter: 644, loss : 5.141e-04
iter: 645, loss : 2.602e-04
iter: 646, loss : 4.258e-04
iter: 647, loss : 1.182e-04
iter: 648, loss : 6.029e-04
iter: 649, loss : 3.427e-04
iter: 650, loss : 2.079e-03
iter: 651, loss : 5.000e-03
iter: 652, loss : 2.593e-04
iter: 653, loss : 1.261e-04
iter: 654, loss : 2.597e-04
iter: 655, loss : 1.146e-04
iter: 656, loss : 8.627e-04
iter: 657, loss : 2.834e-03
iter: 658, loss : 4.832e-04
iter: 659, loss : 3.908e-04
iter: 660, loss : 9.217e-04
iter: 661, loss : 9.764e-04
iter: 662, loss : 4.596e-04
iter: 663, loss : 5.052e-04
iter: 664, loss : 3.675e-03
iter: 665, loss : 6.916e-04
iter: 666, loss : 3.344e-03
iter: 667, loss : 5.122e-04
iter: 668, loss : 5.460e-04
iter: 669, loss : 1.687e-03
iter: 670, loss : 4.780e-04
iter: 671, loss : 4.319e-04
iter: 672, loss : 2.814e-03
iter: 673, loss : 4.087e-04
iter: 674, loss : 5.581e-04
iter: 675, loss : 4.706e-04
iter: 676, loss : 2.933e-03
iter: 677, loss : 3.570e-04
iter: 678, loss : 6.087e-04
iter: 679, loss : 7.788e-04
iter: 680, loss : 1.891e-04
iter: 681, loss : 3.625e-04
iter: 682, loss : 3.802e-04
iter: 683, loss : 8.265e-04
iter: 684, loss : 2.662e-04
iter: 685, loss : 1.274e-03
iter: 686, loss : 3.182e-04
iter: 687, loss : 1.634e-04
iter: 688, loss : 1.596e-03
iter: 689, loss : 7.163e-04
iter: 690, loss : 1.548e-04
iter: 691, loss : 1.019e-03
iter: 692, loss : 2.341e-04
iter: 693, loss : 2.677e-04
iter: 694, loss : 3.434e-04
iter: 695, loss : 3.071e-04
iter: 696, loss : 1.599e-04
iter: 697, loss : 1.750e-04
iter: 698, loss : 1.753e-04
iter: 699, loss : 2.686e-04
iter: 700, loss : 1.179e-03
iter: 701, loss : 8.726e-03
iter: 702, loss : 9.107e-04
iter: 703, loss : 2.770e-04
iter: 704, loss : 9.158e-04
iter: 705, loss : 3.379e-04
iter: 706, loss : 1.935e-04
iter: 707, loss : 2.507e-04
iter: 708, loss : 2.527e-04
iter: 709, loss : 2.876e-04
iter: 710, loss : 3.972e-04
iter: 711, loss : 1.516e-04
iter: 712, loss : 3.712e-04
iter: 713, loss : 3.424e-04
iter: 714, loss : 1.721e-03
iter: 715, loss : 2.107e-04
iter: 716, loss : 3.850e-03
iter: 717, loss : 4.121e-04
iter: 718, loss : 3.092e-04
iter: 719, loss : 7.082e-04
iter: 720, loss : 4.695e-04
iter: 721, loss : 1.110e-03
iter: 722, loss : 9.850e-04
iter: 723, loss : 3.009e-04
iter: 724, loss : 4.104e-04
iter: 725, loss : 3.786e-04
iter: 726, loss : 8.842e-04
iter: 727, loss : 7.321e-05
iter: 728, loss : 4.904e-04
iter: 729, loss : 3.639e-04
iter: 730, loss : 7.710e-03
iter: 731, loss : 7.833e-04
iter: 732, loss : 5.188e-04
iter: 733, loss : 1.160e-04
iter: 734, loss : 2.298e-04
iter: 735, loss : 3.335e-03
iter: 736, loss : 3.394e-04
iter: 737, loss : 8.495e-04
iter: 738, loss : 3.075e-04
iter: 739, loss : 8.122e-04
iter: 740, loss : 2.404e-04
iter: 741, loss : 2.440e-04
iter: 742, loss : 3.540e-04
iter: 743, loss : 2.464e-04
iter: 744, loss : 2.707e-03
iter: 745, loss : 1.506e-03
iter: 746, loss : 1.684e-03
iter: 747, loss : 3.863e-04
iter: 748, loss : 5.493e-04
iter: 749, loss : 2.915e-04
iter: 750, loss : 2.351e-04
iter: 751, loss : 1.727e-04
iter: 752, loss : 1.002e-03
iter: 753, loss : 7.256e-04
iter: 754, loss : 1.085e-04
iter: 755, loss : 2.676e-04
iter: 756, loss : 2.175e-04
iter: 757, loss : 3.500e-04
iter: 758, loss : 2.095e-04
iter: 759, loss : 9.606e-05
iter: 760, loss : 6.009e-05
iter: 761, loss : 2.827e-04
iter: 762, loss : 3.881e-04
iter: 763, loss : 8.835e-04
iter: 764, loss : 3.705e-03
iter: 765, loss : 1.684e-04
iter: 766, loss : 1.933e-04
iter: 767, loss : 3.423e-04
iter: 768, loss : 1.488e-04
iter: 769, loss : 4.754e-04
iter: 770, loss : 2.202e-04
iter: 771, loss : 1.224e-04
iter: 772, loss : 1.625e-04
iter: 773, loss : 2.148e-04
iter: 774, loss : 1.079e-03
iter: 775, loss : 2.320e-04
iter: 776, loss : 2.678e-04
iter: 777, loss : 3.828e-04
iter: 778, loss : 3.021e-04
iter: 779, loss : 1.558e-04
iter: 780, loss : 1.100e-04
iter: 781, loss : 1.980e-03
iter: 782, loss : 2.225e-04
iter: 783, loss : 2.788e-04
iter: 784, loss : 7.934e-04
iter: 785, loss : 3.823e-04
iter: 786, loss : 1.882e-03
iter: 787, loss : 1.296e-04
iter: 788, loss : 2.398e-04
iter: 789, loss : 6.937e-05
iter: 790, loss : 7.206e-03
iter: 791, loss : 3.668e-03
iter: 792, loss : 2.196e-04
iter: 793, loss : 4.682e-03
iter: 794, loss : 3.396e-04
iter: 795, loss : 7.303e-03
iter: 796, loss : 4.723e-04
iter: 797, loss : 6.309e-04
iter: 798, loss : 3.040e-04
iter: 799, loss : 1.434e-04
iter: 800, loss : 1.644e-03
iter: 801, loss : 3.029e-04
iter: 802, loss : 6.522e-04
iter: 803, loss : 6.828e-03
iter: 804, loss : 5.562e-04
iter: 805, loss : 2.695e-04
iter: 806, loss : 2.920e-03
iter: 807, loss : 2.427e-04
iter: 808, loss : 3.152e-04
iter: 809, loss : 1.930e-03
iter: 810, loss : 3.154e-04
iter: 811, loss : 2.661e-04
iter: 812, loss : 1.987e-03
iter: 813, loss : 1.919e-03
iter: 814, loss : 3.797e-03
iter: 815, loss : 3.807e-04
iter: 816, loss : 6.663e-04
iter: 817, loss : 6.341e-04
iter: 818, loss : 1.019e-03
iter: 819, loss : 5.318e-04
iter: 820, loss : 4.380e-04
iter: 821, loss : 1.775e-04
iter: 822, loss : 2.990e-04
iter: 823, loss : 5.797e-04
iter: 824, loss : 4.265e-04
iter: 825, loss : 1.722e-04
iter: 826, loss : 3.880e-04
iter: 827, loss : 2.836e-04
iter: 828, loss : 8.319e-04
iter: 829, loss : 1.492e-04
iter: 830, loss : 3.040e-04
iter: 831, loss : 1.487e-04
iter: 832, loss : 1.874e-04
iter: 833, loss : 1.926e-03
iter: 834, loss : 3.162e-04
iter: 835, loss : 1.251e-03
iter: 836, loss : 2.348e-04
iter: 837, loss : 6.486e-04
iter: 838, loss : 2.906e-04
iter: 839, loss : 4.305e-04
iter: 840, loss : 7.858e-04
iter: 841, loss : 2.428e-04
iter: 842, loss : 7.664e-04
iter: 843, loss : 3.015e-04
iter: 844, loss : 2.835e-03
iter: 845, loss : 7.613e-04
iter: 846, loss : 1.938e-04
iter: 847, loss : 5.704e-03
iter: 848, loss : 4.795e-04
iter: 849, loss : 3.199e-04
iter: 850, loss : 1.836e-04
iter: 851, loss : 3.357e-04
iter: 852, loss : 5.544e-04
iter: 853, loss : 2.680e-04
iter: 854, loss : 2.315e-04
iter: 855, loss : 2.982e-04
iter: 856, loss : 8.604e-04
iter: 857, loss : 2.839e-04
iter: 858, loss : 3.060e-04
iter: 859, loss : 5.065e-04
iter: 860, loss : 1.153e-03
iter: 861, loss : 3.849e-04
iter: 862, loss : 8.960e-04
iter: 863, loss : 3.378e-03
iter: 864, loss : 7.305e-03
iter: 865, loss : 9.632e-04
iter: 866, loss : 6.136e-04
iter: 867, loss : 2.766e-03
iter: 868, loss : 6.240e-04
iter: 869, loss : 2.468e-04
iter: 870, loss : 3.779e-03
iter: 871, loss : 6.716e-03
iter: 872, loss : 1.328e-03
iter: 873, loss : 2.065e-04
iter: 874, loss : 1.143e-03
iter: 875, loss : 8.230e-04
iter: 876, loss : 6.068e-04
iter: 877, loss : 4.784e-04
iter: 878, loss : 4.316e-04
iter: 879, loss : 3.334e-04
iter: 880, loss : 9.412e-04
iter: 881, loss : 1.317e-04
iter: 882, loss : 4.223e-04
iter: 883, loss : 7.040e-03
iter: 884, loss : 2.351e-04
iter: 885, loss : 6.345e-04
iter: 886, loss : 2.237e-04
iter: 887, loss : 1.679e-04
iter: 888, loss : 2.525e-04
iter: 889, loss : 2.083e-04
iter: 890, loss : 2.624e-04
iter: 891, loss : 1.129e-03
iter: 892, loss : 3.237e-03
iter: 893, loss : 3.096e-04
iter: 894, loss : 6.143e-04
iter: 895, loss : 2.895e-04
iter: 896, loss : 3.446e-04
iter: 897, loss : 1.460e-04
iter: 898, loss : 2.451e-04
iter: 899, loss : 1.508e-04
iter: 900, loss : 9.094e-05
iter: 901, loss : 1.038e-04
iter: 902, loss : 3.906e-03
iter: 903, loss : 2.131e-03
iter: 904, loss : 9.416e-04
iter: 905, loss : 1.166e-03
iter: 906, loss : 3.980e-04
iter: 907, loss : 3.805e-04
iter: 908, loss : 1.113e-04
iter: 909, loss : 1.547e-03
iter: 910, loss : 5.364e-04
iter: 911, loss : 8.838e-05
iter: 912, loss : 5.491e-04
iter: 913, loss : 2.890e-04
iter: 914, loss : 5.070e-03
iter: 915, loss : 5.313e-04
iter: 916, loss : 5.417e-04
iter: 917, loss : 1.571e-03
iter: 918, loss : 1.454e-03
iter: 919, loss : 5.020e-04
iter: 920, loss : 1.822e-04
iter: 921, loss : 3.444e-04
iter: 922, loss : 2.588e-03
iter: 923, loss : 1.015e-03
iter: 924, loss : 4.158e-04
iter: 925, loss : 6.292e-04
iter: 926, loss : 3.429e-04
iter: 927, loss : 2.092e-04
iter: 928, loss : 2.306e-04
iter: 929, loss : 2.074e-04
iter: 930, loss : 1.751e-04
iter: 931, loss : 2.399e-04
iter: 932, loss : 7.531e-04
iter: 933, loss : 2.532e-04
iter: 934, loss : 4.208e-04
iter: 935, loss : 7.371e-03
iter: 936, loss : 2.274e-04
iter: 937, loss : 6.345e-03
iter: 938, loss : 6.502e-04
iter: 939, loss : 8.303e-04
iter: 940, loss : 4.027e-04
iter: 941, loss : 4.850e-03
iter: 942, loss : 2.127e-04
iter: 943, loss : 1.568e-04
iter: 944, loss : 4.312e-04
iter: 945, loss : 1.105e-03
iter: 946, loss : 1.142e-04
iter: 947, loss : 3.569e-04
iter: 948, loss : 1.677e-04
iter: 949, loss : 1.262e-03
iter: 950, loss : 2.534e-04
iter: 951, loss : 1.572e-03
iter: 952, loss : 5.984e-05
iter: 953, loss : 2.075e-04
iter: 954, loss : 3.933e-04
iter: 955, loss : 3.220e-04
iter: 956, loss : 2.170e-04
iter: 957, loss : 1.036e-04
iter: 958, loss : 2.751e-04
iter: 959, loss : 1.536e-03
iter: 960, loss : 1.522e-04
iter: 961, loss : 2.851e-04
iter: 962, loss : 2.727e-04
iter: 963, loss : 1.262e-04
iter: 964, loss : 1.985e-03
iter: 965, loss : 8.664e-04
iter: 966, loss : 6.152e-04
iter: 967, loss : 1.105e-04
iter: 968, loss : 1.832e-04
iter: 969, loss : 2.156e-03
iter: 970, loss : 3.922e-04
iter: 971, loss : 7.633e-04
iter: 972, loss : 4.445e-04
iter: 973, loss : 5.644e-04
iter: 974, loss : 3.024e-04
iter: 975, loss : 2.010e-03
iter: 976, loss : 2.524e-04
iter: 977, loss : 2.990e-04
iter: 978, loss : 2.792e-04
iter: 979, loss : 4.823e-04
iter: 980, loss : 1.538e-04
iter: 981, loss : 1.671e-04
iter: 982, loss : 5.459e-05
iter: 983, loss : 2.977e-04
iter: 984, loss : 3.438e-04
iter: 985, loss : 2.733e-04
iter: 986, loss : 1.665e-04
iter: 987, loss : 1.503e-04
iter: 988, loss : 2.110e-03
iter: 989, loss : 6.678e-03
iter: 990, loss : 5.984e-04
iter: 991, loss : 5.060e-05
iter: 992, loss : 2.755e-04
iter: 993, loss : 1.250e-04
iter: 994, loss : 1.588e-04
iter: 995, loss : 9.610e-04
iter: 996, loss : 3.704e-04
iter: 997, loss : 3.028e-04
iter: 998, loss : 1.936e-04
iter: 999, loss : 1.771e-04
iter: 1000, loss : 2.846e-03
iter: 1001, loss : 2.580e-03
iter: 1002, loss : 4.360e-04
iter: 1003, loss : 1.417e-03
iter: 1004, loss : 3.211e-04
iter: 1005, loss : 2.780e-04
iter: 1006, loss : 1.429e-04
iter: 1007, loss : 3.705e-03
iter: 1008, loss : 2.833e-03
iter: 1009, loss : 4.806e-04
iter: 1010, loss : 3.783e-04
iter: 1011, loss : 3.449e-04
iter: 1012, loss : 2.968e-04
iter: 1013, loss : 4.450e-04
iter: 1014, loss : 1.745e-03
iter: 1015, loss : 2.626e-04
iter: 1016, loss : 5.588e-04
iter: 1017, loss : 9.678e-04
iter: 1018, loss : 6.682e-04
iter: 1019, loss : 5.680e-04
iter: 1020, loss : 3.710e-04
iter: 1021, loss : 1.143e-04
iter: 1022, loss : 2.338e-03
iter: 1023, loss : 5.147e-04
iter: 1024, loss : 4.751e-04
iter: 1025, loss : 5.164e-04
iter: 1026, loss : 2.137e-04
iter: 1027, loss : 3.781e-03
iter: 1028, loss : 1.818e-03
iter: 1029, loss : 1.424e-03
iter: 1030, loss : 1.281e-03
iter: 1031, loss : 1.004e-03
iter: 1032, loss : 6.161e-04
iter: 1033, loss : 4.195e-04
iter: 1034, loss : 4.408e-04
iter: 1035, loss : 1.724e-03
iter: 1036, loss : 7.636e-04
iter: 1037, loss : 6.628e-03
iter: 1038, loss : 6.286e-04
iter: 1039, loss : 2.215e-03
iter: 1040, loss : 2.287e-04
iter: 1041, loss : 1.351e-03
iter: 1042, loss : 4.941e-04
iter: 1043, loss : 2.437e-04
iter: 1044, loss : 1.760e-04
iter: 1045, loss : 3.348e-03
iter: 1046, loss : 1.904e-04
iter: 1047, loss : 3.867e-04
iter: 1048, loss : 3.083e-04
iter: 1049, loss : 5.707e-04
iter: 1050, loss : 4.505e-04
iter: 1051, loss : 9.630e-04
iter: 1052, loss : 1.494e-03
iter: 1053, loss : 1.729e-03
iter: 1054, loss : 1.820e-03
iter: 1055, loss : 1.342e-04
iter: 1056, loss : 9.679e-04
iter: 1057, loss : 4.164e-04
iter: 1058, loss : 7.302e-04
iter: 1059, loss : 7.116e-04
iter: 1060, loss : 4.401e-04
iter: 1061, loss : 1.769e-04
iter: 1062, loss : 6.119e-04
iter: 1063, loss : 3.059e-04
iter: 1064, loss : 2.115e-04
iter: 1065, loss : 3.468e-04
iter: 1066, loss : 6.173e-04
iter: 1067, loss : 2.058e-04
iter: 1068, loss : 7.490e-04
iter: 1069, loss : 8.434e-04
iter: 1070, loss : 2.315e-04
iter: 1071, loss : 4.096e-04
iter: 1072, loss : 2.779e-04
iter: 1073, loss : 4.839e-03
iter: 1074, loss : 1.498e-04
iter: 1075, loss : 1.716e-04
iter: 1076, loss : 2.134e-04
iter: 1077, loss : 3.430e-04
iter: 1078, loss : 2.578e-04
iter: 1079, loss : 4.124e-04
iter: 1080, loss : 1.039e-04
iter: 1081, loss : 1.613e-03
iter: 1082, loss : 4.649e-03
iter: 1083, loss : 5.330e-04
iter: 1084, loss : 1.407e-04
iter: 1085, loss : 4.867e-04
iter: 1086, loss : 2.381e-04
iter: 1087, loss : 1.437e-04
iter: 1088, loss : 2.468e-04
iter: 1089, loss : 1.895e-04
iter: 1090, loss : 1.652e-04
iter: 1091, loss : 3.485e-04
iter: 1092, loss : 9.536e-04
iter: 1093, loss : 2.382e-03
iter: 1094, loss : 1.876e-04
iter: 1095, loss : 2.072e-03
iter: 1096, loss : 7.128e-04
iter: 1097, loss : 6.698e-04
iter: 1098, loss : 1.467e-03
iter: 1099, loss : 3.091e-04
iter: 1100, loss : 2.759e-04
iter: 1101, loss : 6.590e-04
iter: 1102, loss : 1.700e-04
iter: 1103, loss : 6.264e-04
iter: 1104, loss : 2.526e-04
iter: 1105, loss : 3.116e-04
iter: 1106, loss : 2.128e-04
iter: 1107, loss : 2.230e-04
iter: 1108, loss : 7.100e-03
iter: 1109, loss : 4.616e-04
iter: 1110, loss : 4.763e-04
iter: 1111, loss : 3.243e-03
iter: 1112, loss : 1.814e-03
iter: 1113, loss : 4.009e-04
iter: 1114, loss : 4.386e-04
iter: 1115, loss : 8.146e-04
iter: 1116, loss : 3.140e-03
iter: 1117, loss : 2.337e-03
iter: 1118, loss : 6.678e-04
iter: 1119, loss : 1.482e-03
iter: 1120, loss : 7.661e-04
iter: 1121, loss : 2.251e-04
iter: 1122, loss : 5.306e-04
iter: 1123, loss : 1.880e-04
iter: 1124, loss : 3.526e-04
iter: 1125, loss : 4.467e-04
iter: 1126, loss : 5.433e-04
iter: 1127, loss : 4.939e-04
iter: 1128, loss : 6.987e-04
iter: 1129, loss : 2.691e-04
iter: 1130, loss : 2.174e-04
iter: 1131, loss : 1.672e-04
iter: 1132, loss : 1.150e-04
iter: 1133, loss : 1.811e-04
iter: 1134, loss : 2.848e-04
iter: 1135, loss : 5.258e-04
iter: 1136, loss : 9.746e-05
iter: 1137, loss : 3.523e-03
iter: 1138, loss : 1.431e-04
iter: 1139, loss : 2.499e-04
iter: 1140, loss : 2.832e-04
iter: 1141, loss : 1.830e-04
iter: 1142, loss : 3.543e-04
iter: 1143, loss : 1.001e-03
iter: 1144, loss : 1.356e-04
iter: 1145, loss : 2.032e-04
iter: 1146, loss : 1.141e-04
iter: 1147, loss : 6.498e-04
iter: 1148, loss : 3.551e-03
iter: 1149, loss : 2.513e-04
iter: 1150, loss : 5.692e-04
iter: 1151, loss : 1.093e-03
iter: 1152, loss : 2.244e-03
iter: 1153, loss : 3.988e-04
iter: 1154, loss : 5.043e-04
iter: 1155, loss : 4.512e-04
iter: 1156, loss : 3.427e-04
iter: 1157, loss : 2.788e-03
iter: 1158, loss : 1.454e-03
iter: 1159, loss : 4.945e-04
iter: 1160, loss : 3.434e-04
iter: 1161, loss : 3.260e-04
iter: 1162, loss : 5.690e-04
iter: 1163, loss : 2.518e-04
iter: 1164, loss : 1.902e-04
iter: 1165, loss : 4.887e-04
iter: 1166, loss : 1.110e-04
iter: 1167, loss : 1.671e-04
iter: 1168, loss : 1.979e-04
iter: 1169, loss : 2.432e-04
iter: 1170, loss : 2.625e-03
iter: 1171, loss : 1.827e-04
iter: 1172, loss : 5.657e-03
iter: 1173, loss : 4.730e-04
iter: 1174, loss : 4.285e-04
iter: 1175, loss : 1.748e-04
iter: 1176, loss : 7.058e-04
iter: 1177, loss : 1.607e-04
iter: 1178, loss : 3.155e-04
iter: 1179, loss : 4.892e-04
iter: 1180, loss : 1.831e-03
iter: 1181, loss : 2.240e-04
iter: 1182, loss : 2.104e-03
iter: 1183, loss : 3.457e-04
iter: 1184, loss : 3.645e-04
iter: 1185, loss : 9.712e-05
iter: 1186, loss : 3.496e-04
iter: 1187, loss : 1.407e-03
iter: 1188, loss : 7.191e-04
iter: 1189, loss : 4.736e-04
iter: 1190, loss : 4.588e-04
iter: 1191, loss : 7.482e-04
iter: 1192, loss : 2.375e-04
iter: 1193, loss : 8.339e-04
iter: 1194, loss : 1.109e-04
iter: 1195, loss : 2.378e-04
iter: 1196, loss : 3.342e-04
iter: 1197, loss : 3.925e-04
iter: 1198, loss : 2.661e-04
iter: 1199, loss : 3.544e-04
iter: 1200, loss : 3.361e-04
iter: 1201, loss : 1.908e-03
iter: 1202, loss : 1.173e-04
iter: 1203, loss : 2.759e-04
iter: 1204, loss : 2.176e-04
iter: 1205, loss : 3.921e-04
iter: 1206, loss : 1.847e-03
iter: 1207, loss : 3.179e-04
iter: 1208, loss : 4.263e-04
iter: 1209, loss : 3.665e-04
iter: 1210, loss : 1.827e-04
iter: 1211, loss : 4.664e-04
iter: 1212, loss : 3.538e-04
iter: 1213, loss : 1.143e-04
iter: 1214, loss : 2.860e-04
iter: 1215, loss : 2.352e-04
iter: 1216, loss : 1.308e-03
iter: 1217, loss : 1.545e-04
iter: 1218, loss : 2.091e-04
iter: 1219, loss : 1.427e-03
iter: 1220, loss : 7.839e-04
iter: 1221, loss : 4.533e-04
iter: 1222, loss : 2.659e-04
iter: 1223, loss : 2.330e-03
iter: 1224, loss : 2.206e-04
iter: 1225, loss : 2.088e-04
iter: 1226, loss : 2.058e-03
iter: 1227, loss : 4.225e-04
iter: 1228, loss : 2.725e-04
iter: 1229, loss : 9.738e-04
iter: 1230, loss : 5.720e-04
iter: 1231, loss : 2.962e-04
iter: 1232, loss : 2.160e-04
iter: 1233, loss : 2.533e-04
iter: 1234, loss : 2.703e-04
iter: 1235, loss : 6.056e-04
iter: 1236, loss : 1.120e-04
iter: 1237, loss : 1.300e-04
iter: 1238, loss : 1.964e-04
iter: 1239, loss : 2.089e-04
iter: 1240, loss : 1.414e-04
iter: 1241, loss : 2.190e-04
iter: 1242, loss : 6.085e-05
iter: 1243, loss : 1.254e-04
iter: 1244, loss : 1.621e-04
iter: 1245, loss : 1.282e-04
iter: 1246, loss : 1.524e-03
iter: 1247, loss : 1.634e-04
iter: 1248, loss : 1.774e-04
iter: 1249, loss : 5.138e-04
iter: 1250, loss : 7.527e-05
iter: 1251, loss : 1.038e-04
iter: 1252, loss : 1.529e-04
iter: 1253, loss : 1.357e-03
iter: 1254, loss : 2.176e-04
iter: 1255, loss : 1.067e-04
iter: 1256, loss : 1.371e-04
iter: 1257, loss : 1.602e-04
iter: 1258, loss : 1.921e-03
iter: 1259, loss : 4.812e-03
iter: 1260, loss : 2.279e-04
iter: 1261, loss : 8.220e-04
iter: 1262, loss : 1.030e-03
iter: 1263, loss : 2.484e-04
iter: 1264, loss : 1.976e-04
iter: 1265, loss : 1.248e-04
iter: 1266, loss : 2.603e-03
iter: 1267, loss : 7.866e-04
iter: 1268, loss : 4.790e-04
iter: 1269, loss : 3.927e-04
iter: 1270, loss : 9.280e-05
iter: 1271, loss : 4.602e-03
iter: 1272, loss : 2.332e-04
iter: 1273, loss : 1.592e-04
iter: 1274, loss : 3.383e-04
iter: 1275, loss : 4.812e-04
iter: 1276, loss : 1.319e-04
iter: 1277, loss : 1.376e-04
iter: 1278, loss : 1.617e-04
iter: 1279, loss : 1.197e-04
iter: 1280, loss : 2.463e-04
iter: 1281, loss : 9.918e-05
iter: 1282, loss : 1.698e-04
iter: 1283, loss : 3.801e-04
iter: 1284, loss : 1.656e-04
iter: 1285, loss : 2.349e-03
iter: 1286, loss : 3.775e-04
iter: 1287, loss : 1.839e-04
iter: 1288, loss : 3.045e-03
iter: 1289, loss : 2.395e-04
iter: 1290, loss : 1.880e-04
iter: 1291, loss : 2.280e-03
iter: 1292, loss : 3.030e-04
iter: 1293, loss : 2.387e-04
iter: 1294, loss : 1.300e-04
iter: 1295, loss : 1.945e-03
iter: 1296, loss : 1.263e-04
iter: 1297, loss : 1.113e-03
iter: 1298, loss : 5.424e-04
iter: 1299, loss : 2.214e-04
iter: 1300, loss : 3.442e-04
iter: 1301, loss : 3.048e-04
iter: 1302, loss : 2.909e-04
iter: 1303, loss : 4.480e-04
iter: 1304, loss : 1.149e-03
iter: 1305, loss : 2.892e-04
iter: 1306, loss : 1.977e-03
iter: 1307, loss : 3.589e-04
iter: 1308, loss : 1.993e-04
iter: 1309, loss : 2.586e-04
iter: 1310, loss : 6.881e-03
iter: 1311, loss : 3.092e-04
iter: 1312, loss : 7.662e-04
iter: 1313, loss : 2.456e-04
iter: 1314, loss : 7.099e-04
iter: 1315, loss : 2.068e-04
iter: 1316, loss : 3.016e-04
iter: 1317, loss : 3.192e-04
iter: 1318, loss : 1.483e-04
iter: 1319, loss : 1.957e-04
iter: 1320, loss : 1.403e-04
iter: 1321, loss : 2.670e-04
iter: 1322, loss : 7.569e-04
iter: 1323, loss : 1.195e-04
iter: 1324, loss : 1.304e-04
iter: 1325, loss : 1.015e-03
iter: 1326, loss : 1.506e-03
iter: 1327, loss : 2.190e-04
iter: 1328, loss : 2.077e-04
iter: 1329, loss : 1.111e-04
iter: 1330, loss : 2.477e-04
iter: 1331, loss : 7.414e-04
iter: 1332, loss : 1.334e-03
iter: 1333, loss : 1.396e-04
iter: 1334, loss : 2.190e-03
iter: 1335, loss : 7.947e-04
iter: 1336, loss : 2.468e-04
iter: 1337, loss : 5.460e-03
iter: 1338, loss : 2.500e-04
iter: 1339, loss : 3.593e-04
iter: 1340, loss : 2.491e-04
iter: 1341, loss : 3.255e-04
iter: 1342, loss : 4.278e-04
iter: 1343, loss : 5.918e-05
iter: 1344, loss : 1.475e-04
iter: 1345, loss : 1.548e-04
iter: 1346, loss : 4.431e-03
iter: 1347, loss : 1.465e-04
iter: 1348, loss : 1.810e-04
iter: 1349, loss : 5.275e-04
iter: 1350, loss : 3.834e-04
iter: 1351, loss : 3.656e-04
iter: 1352, loss : 3.325e-04
iter: 1353, loss : 2.983e-04
iter: 1354, loss : 1.043e-04
iter: 1355, loss : 1.892e-04
iter: 1356, loss : 1.878e-04
iter: 1357, loss : 1.489e-03
iter: 1358, loss : 1.350e-04
iter: 1359, loss : 4.372e-04
iter: 1360, loss : 8.746e-05
iter: 1361, loss : 2.086e-04
iter: 1362, loss : 1.178e-04
iter: 1363, loss : 1.308e-04
iter: 1364, loss : 3.170e-03
iter: 1365, loss : 8.352e-04
iter: 1366, loss : 1.279e-03
iter: 1367, loss : 4.970e-04
iter: 1368, loss : 2.766e-04
iter: 1369, loss : 3.232e-04
iter: 1370, loss : 2.954e-04
iter: 1371, loss : 1.406e-04
iter: 1372, loss : 1.710e-04
iter: 1373, loss : 1.606e-04
iter: 1374, loss : 2.999e-04
iter: 1375, loss : 3.873e-04
iter: 1376, loss : 1.115e-03
iter: 1377, loss : 1.612e-04
iter: 1378, loss : 1.746e-04
iter: 1379, loss : 6.240e-04
iter: 1380, loss : 9.748e-04
iter: 1381, loss : 1.694e-04
iter: 1382, loss : 1.829e-04
iter: 1383, loss : 1.991e-04
iter: 1384, loss : 6.935e-04
iter: 1385, loss : 5.907e-04
iter: 1386, loss : 1.222e-03
iter: 1387, loss : 2.789e-04
iter: 1388, loss : 6.954e-03
iter: 1389, loss : 3.611e-04
iter: 1390, loss : 2.602e-04
iter: 1391, loss : 3.109e-04
iter: 1392, loss : 2.768e-04
iter: 1393, loss : 2.183e-04
iter: 1394, loss : 1.161e-03
iter: 1395, loss : 4.357e-04
iter: 1396, loss : 1.456e-04
iter: 1397, loss : 7.409e-04
iter: 1398, loss : 1.742e-04
iter: 1399, loss : 1.322e-04
iter: 1400, loss : 3.108e-04
iter: 1401, loss : 1.825e-04
iter: 1402, loss : 5.665e-04
iter: 1403, loss : 4.031e-04
iter: 1404, loss : 3.323e-04
iter: 1405, loss : 3.091e-04
iter: 1406, loss : 3.062e-04
iter: 1407, loss : 1.863e-04
iter: 1408, loss : 2.759e-04
iter: 1409, loss : 1.406e-04
iter: 1410, loss : 4.259e-04
iter: 1411, loss : 1.550e-04
iter: 1412, loss : 1.358e-04
iter: 1413, loss : 2.674e-03
iter: 1414, loss : 2.055e-04
iter: 1415, loss : 2.207e-04
iter: 1416, loss : 7.314e-04
iter: 1417, loss : 3.940e-04
iter: 1418, loss : 1.588e-03
iter: 1419, loss : 1.562e-04
iter: 1420, loss : 3.743e-03
iter: 1421, loss : 4.914e-04
iter: 1422, loss : 2.297e-04
iter: 1423, loss : 5.849e-04
iter: 1424, loss : 9.911e-04
iter: 1425, loss : 3.950e-04
iter: 1426, loss : 8.653e-05
iter: 1427, loss : 2.118e-04
iter: 1428, loss : 1.330e-03
iter: 1429, loss : 9.255e-05
iter: 1430, loss : 2.175e-04
iter: 1431, loss : 1.679e-04
iter: 1432, loss : 1.592e-04
iter: 1433, loss : 3.518e-04
iter: 1434, loss : 1.896e-04
iter: 1435, loss : 7.262e-04
iter: 1436, loss : 8.533e-04
iter: 1437, loss : 1.863e-04
iter: 1438, loss : 1.454e-04
iter: 1439, loss : 5.727e-04
iter: 1440, loss : 1.223e-04
iter: 1441, loss : 3.742e-04
iter: 1442, loss : 3.963e-04
iter: 1443, loss : 1.927e-03
iter: 1444, loss : 2.044e-04
iter: 1445, loss : 1.118e-04
iter: 1446, loss : 3.111e-04
iter: 1447, loss : 1.094e-04
iter: 1448, loss : 2.207e-04
iter: 1449, loss : 2.966e-04
iter: 1450, loss : 2.295e-04
iter: 1451, loss : 1.246e-03
iter: 1452, loss : 2.244e-04
iter: 1453, loss : 2.219e-04
iter: 1454, loss : 2.357e-03
iter: 1455, loss : 5.314e-03
iter: 1456, loss : 9.541e-04
iter: 1457, loss : 2.378e-04
iter: 1458, loss : 4.006e-04
iter: 1459, loss : 3.695e-04
iter: 1460, loss : 1.765e-04
iter: 1461, loss : 8.591e-05
iter: 1462, loss : 2.624e-04
iter: 1463, loss : 1.011e-03
iter: 1464, loss : 1.025e-04
iter: 1465, loss : 3.871e-04
iter: 1466, loss : 1.542e-04
iter: 1467, loss : 1.121e-04
iter: 1468, loss : 1.520e-04
iter: 1469, loss : 1.898e-04
iter: 1470, loss : 2.440e-03
iter: 1471, loss : 2.593e-04
iter: 1472, loss : 2.809e-04
iter: 1473, loss : 1.429e-03
iter: 1474, loss : 2.713e-04
iter: 1475, loss : 1.077e-03
iter: 1476, loss : 1.091e-03
iter: 1477, loss : 6.132e-04
iter: 1478, loss : 1.573e-04
iter: 1479, loss : 2.168e-03
iter: 1480, loss : 1.433e-04
iter: 1481, loss : 5.347e-04
iter: 1482, loss : 3.213e-04
iter: 1483, loss : 1.256e-04
iter: 1484, loss : 3.285e-04
iter: 1485, loss : 2.423e-04
iter: 1486, loss : 2.195e-03
iter: 1487, loss : 1.906e-04
iter: 1488, loss : 1.356e-04
iter: 1489, loss : 1.855e-04
iter: 1490, loss : 1.853e-04
iter: 1491, loss : 4.330e-05
iter: 1492, loss : 2.737e-04
iter: 1493, loss : 5.279e-04
iter: 1494, loss : 1.964e-04
iter: 1495, loss : 9.607e-05
iter: 1496, loss : 1.459e-04
iter: 1497, loss : 1.733e-04
iter: 1498, loss : 9.014e-05
iter: 1499, loss : 1.135e-04
iter: 1500, loss : 6.242e-04
iter: 1501, loss : 2.135e-04
iter: 1502, loss : 2.198e-03
iter: 1503, loss : 3.539e-04
iter: 1504, loss : 1.705e-04
iter: 1505, loss : 1.615e-04
iter: 1506, loss : 5.461e-04
iter: 1507, loss : 8.949e-05
iter: 1508, loss : 5.279e-04
iter: 1509, loss : 1.446e-04
iter: 1510, loss : 1.867e-04
iter: 1511, loss : 1.157e-04
iter: 1512, loss : 7.988e-05
iter: 1513, loss : 2.678e-04
iter: 1514, loss : 6.547e-04
iter: 1515, loss : 3.111e-04
iter: 1516, loss : 1.024e-04
iter: 1517, loss : 2.192e-04
iter: 1518, loss : 3.183e-03
iter: 1519, loss : 2.371e-04
iter: 1520, loss : 1.152e-04
iter: 1521, loss : 2.906e-04
iter: 1522, loss : 1.457e-04
iter: 1523, loss : 6.059e-04
iter: 1524, loss : 2.508e-04
iter: 1525, loss : 1.103e-04
iter: 1526, loss : 1.936e-04
iter: 1527, loss : 5.901e-05
iter: 1528, loss : 2.701e-03
iter: 1529, loss : 2.993e-04
iter: 1530, loss : 2.199e-04
iter: 1531, loss : 1.592e-04
iter: 1532, loss : 7.177e-04
iter: 1533, loss : 2.354e-04
iter: 1534, loss : 1.780e-04
iter: 1535, loss : 2.296e-04
iter: 1536, loss : 6.470e-04
iter: 1537, loss : 1.465e-04
iter: 1538, loss : 2.635e-04
iter: 1539, loss : 1.452e-04
iter: 1540, loss : 3.098e-03
iter: 1541, loss : 1.048e-04
iter: 1542, loss : 7.069e-04
iter: 1543, loss : 1.937e-03
iter: 1544, loss : 3.804e-04
iter: 1545, loss : 2.863e-04
iter: 1546, loss : 2.119e-04
iter: 1547, loss : 1.078e-04
iter: 1548, loss : 3.016e-04
iter: 1549, loss : 1.061e-04
iter: 1550, loss : 3.893e-03
iter: 1551, loss : 1.385e-04
iter: 1552, loss : 2.055e-04
iter: 1553, loss : 2.377e-04
iter: 1554, loss : 6.886e-04
iter: 1555, loss : 2.279e-04
iter: 1556, loss : 4.988e-04
iter: 1557, loss : 4.392e-04
iter: 1558, loss : 2.002e-04
iter: 1559, loss : 1.341e-04
iter: 1560, loss : 1.240e-04
iter: 1561, loss : 1.218e-04
iter: 1562, loss : 6.548e-04
iter: 1563, loss : 2.100e-04
iter: 1564, loss : 1.571e-04
iter: 1565, loss : 1.043e-04
iter: 1566, loss : 1.732e-04
iter: 1567, loss : 2.819e-04
iter: 1568, loss : 1.223e-04
iter: 1569, loss : 5.361e-03
iter: 1570, loss : 5.510e-05
iter: 1571, loss : 1.625e-04
iter: 1572, loss : 1.191e-04
iter: 1573, loss : 2.102e-04
iter: 1574, loss : 2.270e-04
iter: 1575, loss : 6.856e-05
iter: 1576, loss : 5.454e-03
iter: 1577, loss : 2.484e-04
iter: 1578, loss : 1.656e-03
iter: 1579, loss : 6.519e-04
iter: 1580, loss : 2.728e-04
iter: 1581, loss : 4.912e-05
iter: 1582, loss : 1.618e-04
iter: 1583, loss : 6.675e-04
iter: 1584, loss : 1.576e-04
iter: 1585, loss : 2.442e-04
iter: 1586, loss : 3.540e-04
iter: 1587, loss : 1.442e-04
iter: 1588, loss : 1.637e-04
iter: 1589, loss : 1.097e-04
iter: 1590, loss : 2.402e-04
iter: 1591, loss : 2.108e-04
iter: 1592, loss : 1.669e-04
iter: 1593, loss : 2.803e-03
iter: 1594, loss : 1.345e-03
iter: 1595, loss : 1.074e-04
iter: 1596, loss : 3.609e-04
iter: 1597, loss : 6.440e-04
iter: 1598, loss : 1.865e-04
iter: 1599, loss : 4.173e-04
iter: 1600, loss : 1.744e-04
iter: 1601, loss : 1.971e-04
iter: 1602, loss : 1.470e-04
iter: 1603, loss : 5.270e-03
iter: 1604, loss : 7.056e-04
iter: 1605, loss : 1.014e-04
iter: 1606, loss : 2.253e-04
iter: 1607, loss : 1.732e-04
iter: 1608, loss : 1.894e-04
iter: 1609, loss : 4.186e-04
iter: 1610, loss : 2.402e-04
iter: 1611, loss : 1.493e-04
iter: 1612, loss : 1.023e-03
iter: 1613, loss : 1.237e-04
iter: 1614, loss : 3.460e-03
iter: 1615, loss : 3.190e-04
iter: 1616, loss : 2.815e-04
iter: 1617, loss : 7.289e-04
iter: 1618, loss : 3.093e-04
iter: 1619, loss : 9.099e-04
iter: 1620, loss : 8.360e-04
iter: 1621, loss : 6.854e-05
iter: 1622, loss : 2.274e-04
iter: 1623, loss : 3.142e-04
iter: 1624, loss : 2.664e-04
iter: 1625, loss : 5.524e-04
iter: 1626, loss : 2.329e-04
iter: 1627, loss : 1.839e-04
iter: 1628, loss : 2.718e-03
iter: 1629, loss : 4.786e-04
iter: 1630, loss : 3.377e-04
iter: 1631, loss : 4.577e-03
iter: 1632, loss : 3.605e-04
iter: 1633, loss : 3.582e-04
iter: 1634, loss : 2.421e-04
iter: 1635, loss : 7.080e-04
iter: 1636, loss : 1.944e-04
iter: 1637, loss : 2.744e-04
iter: 1638, loss : 2.918e-04
iter: 1639, loss : 9.841e-04
iter: 1640, loss : 2.597e-04
iter: 1641, loss : 1.358e-03
iter: 1642, loss : 2.187e-04
iter: 1643, loss : 3.633e-04
iter: 1644, loss : 1.014e-04
iter: 1645, loss : 2.276e-04
iter: 1646, loss : 5.343e-04
iter: 1647, loss : 1.780e-04
iter: 1648, loss : 2.984e-04
iter: 1649, loss : 1.325e-04
iter: 1650, loss : 8.157e-05
iter: 1651, loss : 1.388e-03
iter: 1652, loss : 1.881e-04
iter: 1653, loss : 1.501e-04
iter: 1654, loss : 4.565e-03
iter: 1655, loss : 9.846e-04
iter: 1656, loss : 1.082e-03
iter: 1657, loss : 1.422e-03
iter: 1658, loss : 7.072e-04
iter: 1659, loss : 1.332e-04
iter: 1660, loss : 4.240e-04
iter: 1661, loss : 3.463e-04
iter: 1662, loss : 5.890e-04
iter: 1663, loss : 4.873e-04
iter: 1664, loss : 6.140e-05
iter: 1665, loss : 2.416e-04
iter: 1666, loss : 1.061e-04
iter: 1667, loss : 1.704e-04
iter: 1668, loss : 9.312e-05
iter: 1669, loss : 8.823e-05
iter: 1670, loss : 1.436e-04
iter: 1671, loss : 6.629e-04
iter: 1672, loss : 1.455e-04
iter: 1673, loss : 6.245e-04
iter: 1674, loss : 2.378e-04
iter: 1675, loss : 1.950e-04
iter: 1676, loss : 1.497e-03
iter: 1677, loss : 9.086e-05
iter: 1678, loss : 1.834e-03
iter: 1679, loss : 4.153e-03
iter: 1680, loss : 2.498e-04
iter: 1681, loss : 4.541e-04
iter: 1682, loss : 3.671e-04
iter: 1683, loss : 1.551e-03
iter: 1684, loss : 1.252e-04
iter: 1685, loss : 3.214e-04
iter: 1686, loss : 2.626e-04
iter: 1687, loss : 4.375e-04
iter: 1688, loss : 3.669e-04
iter: 1689, loss : 4.072e-04
iter: 1690, loss : 3.771e-04
iter: 1691, loss : 3.627e-04
iter: 1692, loss : 2.273e-03
iter: 1693, loss : 1.912e-04
iter: 1694, loss : 2.392e-04
iter: 1695, loss : 1.234e-04
iter: 1696, loss : 9.538e-04
iter: 1697, loss : 7.717e-04
iter: 1698, loss : 5.650e-04
iter: 1699, loss : 1.944e-04
iter: 1700, loss : 5.485e-04
iter: 1701, loss : 4.417e-04
iter: 1702, loss : 7.331e-04
iter: 1703, loss : 2.048e-04
iter: 1704, loss : 9.038e-04
iter: 1705, loss : 5.088e-04
iter: 1706, loss : 2.281e-04
iter: 1707, loss : 3.755e-03
iter: 1708, loss : 1.302e-04
iter: 1709, loss : 2.033e-04
iter: 1710, loss : 2.338e-04
iter: 1711, loss : 3.760e-04
iter: 1712, loss : 8.539e-05
iter: 1713, loss : 2.042e-04
iter: 1714, loss : 2.306e-04
iter: 1715, loss : 2.866e-04
iter: 1716, loss : 1.259e-04
iter: 1717, loss : 2.359e-04
iter: 1718, loss : 1.990e-04
iter: 1719, loss : 8.912e-05
iter: 1720, loss : 3.113e-04
iter: 1721, loss : 1.678e-04
iter: 1722, loss : 1.753e-04
iter: 1723, loss : 1.786e-04
iter: 1724, loss : 1.535e-03
iter: 1725, loss : 1.070e-04
iter: 1726, loss : 3.702e-03
iter: 1727, loss : 1.933e-04
iter: 1728, loss : 4.564e-04
iter: 1729, loss : 7.100e-04
iter: 1730, loss : 1.920e-04
iter: 1731, loss : 3.185e-04
iter: 1732, loss : 2.677e-04
iter: 1733, loss : 1.582e-04
iter: 1734, loss : 3.230e-03
iter: 1735, loss : 3.293e-04
iter: 1736, loss : 2.888e-04
iter: 1737, loss : 2.180e-03
iter: 1738, loss : 1.985e-04
iter: 1739, loss : 1.818e-04
iter: 1740, loss : 2.365e-04
iter: 1741, loss : 2.743e-04
iter: 1742, loss : 2.301e-04
iter: 1743, loss : 2.048e-04
iter: 1744, loss : 9.052e-04
iter: 1745, loss : 2.332e-04
iter: 1746, loss : 4.667e-04
iter: 1747, loss : 4.810e-04
iter: 1748, loss : 7.778e-04
iter: 1749, loss : 3.492e-04
iter: 1750, loss : 4.192e-04
iter: 1751, loss : 9.439e-04
iter: 1752, loss : 2.563e-04
iter: 1753, loss : 3.718e-04
iter: 1754, loss : 1.630e-03
iter: 1755, loss : 5.066e-04
iter: 1756, loss : 4.099e-04
iter: 1757, loss : 1.434e-04
iter: 1758, loss : 2.569e-04
iter: 1759, loss : 6.386e-04
iter: 1760, loss : 2.479e-03
iter: 1761, loss : 3.243e-04
iter: 1762, loss : 1.397e-04
iter: 1763, loss : 1.760e-04
iter: 1764, loss : 3.413e-04
iter: 1765, loss : 3.848e-04
iter: 1766, loss : 1.811e-03
iter: 1767, loss : 3.080e-03
iter: 1768, loss : 8.673e-04
iter: 1769, loss : 5.298e-04
iter: 1770, loss : 9.980e-04
iter: 1771, loss : 4.124e-04
iter: 1772, loss : 9.309e-05
iter: 1773, loss : 1.598e-04
iter: 1774, loss : 2.831e-04
iter: 1775, loss : 2.052e-04
iter: 1776, loss : 1.522e-04
iter: 1777, loss : 2.496e-04
iter: 1778, loss : 8.148e-05
iter: 1779, loss : 1.714e-04
iter: 1780, loss : 1.090e-04
iter: 1781, loss : 7.386e-04
iter: 1782, loss : 3.374e-04
iter: 1783, loss : 2.347e-04
iter: 1784, loss : 1.956e-04
iter: 1785, loss : 3.229e-03
iter: 1786, loss : 2.235e-04
iter: 1787, loss : 3.226e-03
iter: 1788, loss : 4.116e-04
iter: 1789, loss : 2.039e-04
iter: 1790, loss : 2.299e-04
iter: 1791, loss : 4.890e-04
iter: 1792, loss : 2.545e-04
iter: 1793, loss : 1.748e-04
iter: 1794, loss : 1.916e-04
iter: 1795, loss : 1.804e-04
iter: 1796, loss : 7.103e-04
iter: 1797, loss : 1.361e-04
iter: 1798, loss : 2.955e-04
iter: 1799, loss : 1.353e-03
iter: 1800, loss : 7.655e-05
iter: 1801, loss : 4.631e-04
iter: 1802, loss : 7.035e-04
iter: 1803, loss : 1.866e-04
iter: 1804, loss : 6.012e-04
iter: 1805, loss : 8.753e-05
iter: 1806, loss : 1.763e-04
iter: 1807, loss : 2.168e-04
iter: 1808, loss : 1.579e-04
iter: 1809, loss : 5.490e-05
iter: 1810, loss : 2.067e-04
iter: 1811, loss : 1.474e-04
iter: 1812, loss : 2.050e-04
iter: 1813, loss : 4.743e-04
iter: 1814, loss : 3.343e-04
iter: 1815, loss : 4.742e-04
iter: 1816, loss : 3.304e-04
iter: 1817, loss : 4.123e-05
iter: 1818, loss : 1.592e-04
iter: 1819, loss : 6.927e-04
iter: 1820, loss : 1.264e-04
iter: 1821, loss : 2.356e-04
iter: 1822, loss : 6.519e-05
iter: 1823, loss : 3.001e-04
iter: 1824, loss : 7.055e-04
iter: 1825, loss : 7.281e-04
iter: 1826, loss : 1.562e-04
iter: 1827, loss : 2.674e-04
iter: 1828, loss : 2.219e-04
iter: 1829, loss : 1.314e-03
iter: 1830, loss : 1.850e-04
iter: 1831, loss : 1.000e-04
iter: 1832, loss : 8.543e-04
iter: 1833, loss : 1.909e-04
iter: 1834, loss : 1.022e-03
iter: 1835, loss : 1.119e-04
iter: 1836, loss : 2.156e-04
iter: 1837, loss : 4.353e-04
iter: 1838, loss : 1.101e-04
iter: 1839, loss : 1.888e-04
iter: 1840, loss : 1.721e-03
iter: 1841, loss : 1.306e-03
iter: 1842, loss : 7.233e-04
iter: 1843, loss : 2.250e-04
iter: 1844, loss : 4.127e-04
iter: 1845, loss : 6.628e-04
iter: 1846, loss : 1.108e-04
iter: 1847, loss : 1.130e-04
iter: 1848, loss : 7.070e-05
iter: 1849, loss : 2.438e-04
iter: 1850, loss : 2.683e-04
iter: 1851, loss : 2.293e-04
iter: 1852, loss : 6.791e-05
iter: 1853, loss : 6.412e-05
iter: 1854, loss : 7.632e-04
iter: 1855, loss : 7.759e-05
iter: 1856, loss : 1.814e-04
iter: 1857, loss : 1.185e-04
iter: 1858, loss : 9.312e-05
iter: 1859, loss : 1.524e-03
iter: 1860, loss : 1.939e-04
iter: 1861, loss : 2.855e-03
iter: 1862, loss : 3.785e-04
iter: 1863, loss : 1.187e-04
iter: 1864, loss : 2.722e-04
iter: 1865, loss : 1.150e-04
iter: 1866, loss : 1.596e-03
iter: 1867, loss : 2.488e-03
iter: 1868, loss : 3.980e-04
iter: 1869, loss : 3.858e-04
iter: 1870, loss : 8.275e-04
iter: 1871, loss : 4.587e-04
iter: 1872, loss : 1.563e-03
iter: 1873, loss : 9.229e-04
iter: 1874, loss : 2.148e-04
iter: 1875, loss : 1.378e-03
iter: 1876, loss : 2.379e-03
iter: 1877, loss : 2.147e-04
iter: 1878, loss : 1.057e-03
iter: 1879, loss : 3.233e-04
iter: 1880, loss : 4.676e-04
iter: 1881, loss : 2.300e-04
iter: 1882, loss : 1.232e-03
iter: 1883, loss : 1.245e-03
iter: 1884, loss : 6.664e-04
iter: 1885, loss : 7.918e-04
iter: 1886, loss : 4.191e-04
iter: 1887, loss : 3.530e-04
iter: 1888, loss : 5.578e-04
iter: 1889, loss : 3.257e-04
iter: 1890, loss : 3.570e-04
iter: 1891, loss : 1.265e-04
iter: 1892, loss : 6.065e-04
iter: 1893, loss : 7.003e-04
iter: 1894, loss : 1.066e-04
iter: 1895, loss : 4.243e-04
iter: 1896, loss : 1.347e-04
iter: 1897, loss : 2.483e-04
iter: 1898, loss : 1.296e-04
iter: 1899, loss : 1.436e-04
iter: 1900, loss : 2.764e-04
iter: 1901, loss : 1.508e-03
iter: 1902, loss : 1.739e-04
iter: 1903, loss : 1.229e-04
iter: 1904, loss : 1.280e-04
iter: 1905, loss : 8.872e-04
iter: 1906, loss : 3.289e-04
iter: 1907, loss : 1.576e-04
iter: 1908, loss : 1.506e-03
iter: 1909, loss : 3.877e-04
iter: 1910, loss : 1.266e-03
iter: 1911, loss : 7.470e-04
iter: 1912, loss : 4.273e-05
iter: 1913, loss : 2.152e-04
iter: 1914, loss : 3.547e-04
iter: 1915, loss : 4.147e-03
iter: 1916, loss : 8.661e-04
iter: 1917, loss : 1.767e-04
iter: 1918, loss : 4.614e-04
iter: 1919, loss : 2.774e-03
iter: 1920, loss : 4.974e-04
iter: 1921, loss : 1.929e-04
iter: 1922, loss : 9.603e-04
iter: 1923, loss : 1.763e-04
iter: 1924, loss : 4.264e-04
iter: 1925, loss : 1.588e-04
iter: 1926, loss : 3.750e-04
iter: 1927, loss : 3.712e-04
iter: 1928, loss : 2.862e-04
iter: 1929, loss : 1.406e-04
iter: 1930, loss : 1.340e-04
iter: 1931, loss : 1.780e-04
iter: 1932, loss : 2.115e-04
iter: 1933, loss : 1.789e-04
iter: 1934, loss : 1.467e-04
iter: 1935, loss : 1.486e-04
iter: 1936, loss : 2.346e-04
iter: 1937, loss : 1.709e-04
iter: 1938, loss : 1.226e-04
iter: 1939, loss : 1.979e-04
iter: 1940, loss : 1.292e-04
iter: 1941, loss : 1.077e-04
iter: 1942, loss : 5.279e-04
iter: 1943, loss : 7.402e-05
iter: 1944, loss : 1.358e-04
iter: 1945, loss : 7.233e-05
iter: 1946, loss : 1.110e-04
iter: 1947, loss : 5.202e-05
iter: 1948, loss : 8.091e-05
iter: 1949, loss : 1.215e-04
iter: 1950, loss : 6.365e-05
iter: 1951, loss : 6.052e-05
iter: 1952, loss : 5.548e-05
iter: 1953, loss : 3.647e-04
iter: 1954, loss : 5.511e-04
iter: 1955, loss : 1.121e-04
iter: 1956, loss : 7.176e-05
iter: 1957, loss : 2.174e-04
iter: 1958, loss : 1.894e-04
iter: 1959, loss : 3.335e-04
iter: 1960, loss : 1.090e-03
iter: 1961, loss : 1.205e-03
iter: 1962, loss : 1.007e-03
iter: 1963, loss : 2.655e-04
iter: 1964, loss : 1.672e-04
iter: 1965, loss : 9.554e-04
iter: 1966, loss : 1.873e-04
iter: 1967, loss : 1.006e-03
iter: 1968, loss : 1.865e-03
iter: 1969, loss : 8.717e-04
iter: 1970, loss : 9.263e-05
iter: 1971, loss : 7.071e-04
iter: 1972, loss : 3.100e-03
iter: 1973, loss : 4.500e-04
iter: 1974, loss : 2.675e-04
iter: 1975, loss : 1.247e-04
iter: 1976, loss : 8.047e-04
iter: 1977, loss : 6.425e-05
iter: 1978, loss : 7.719e-05
iter: 1979, loss : 1.041e-03
iter: 1980, loss : 1.761e-04
iter: 1981, loss : 1.209e-04
iter: 1982, loss : 1.722e-04
iter: 1983, loss : 2.303e-04
iter: 1984, loss : 4.890e-03
iter: 1985, loss : 1.447e-04
iter: 1986, loss : 7.329e-04
iter: 1987, loss : 2.172e-04
iter: 1988, loss : 2.422e-04
iter: 1989, loss : 2.821e-04
iter: 1990, loss : 1.448e-04
iter: 1991, loss : 7.155e-04
iter: 1992, loss : 1.515e-04
iter: 1993, loss : 6.453e-04
iter: 1994, loss : 5.750e-04
iter: 1995, loss : 4.249e-04
iter: 1996, loss : 2.129e-04
iter: 1997, loss : 1.568e-04
iter: 1998, loss : 8.718e-04
iter: 1999, loss : 2.828e-04
iter: 2000, loss : 6.052e-05
iter: 2001, loss : 5.683e-05
iter: 2002, loss : 1.462e-04
iter: 2003, loss : 1.763e-04
iter: 2004, loss : 6.656e-05
iter: 2005, loss : 1.243e-04
iter: 2006, loss : 1.498e-04
iter: 2007, loss : 1.418e-03
iter: 2008, loss : 4.417e-04
iter: 2009, loss : 5.289e-04
iter: 2010, loss : 1.206e-04
iter: 2011, loss : 1.118e-04
iter: 2012, loss : 2.716e-04
iter: 2013, loss : 5.617e-04
iter: 2014, loss : 9.157e-05
iter: 2015, loss : 2.305e-04
iter: 2016, loss : 3.303e-03
iter: 2017, loss : 7.841e-04
iter: 2018, loss : 2.124e-04
iter: 2019, loss : 2.376e-03
iter: 2020, loss : 2.900e-04
iter: 2021, loss : 1.848e-04
iter: 2022, loss : 7.836e-05
iter: 2023, loss : 1.160e-04
iter: 2024, loss : 1.440e-04
iter: 2025, loss : 6.251e-05
iter: 2026, loss : 3.980e-03
iter: 2027, loss : 1.903e-04
iter: 2028, loss : 5.187e-04
iter: 2029, loss : 1.634e-04
iter: 2030, loss : 3.074e-04
iter: 2031, loss : 2.136e-04
iter: 2032, loss : 1.106e-04
iter: 2033, loss : 9.511e-04
iter: 2034, loss : 2.782e-03
iter: 2035, loss : 1.685e-03
iter: 2036, loss : 3.172e-04
iter: 2037, loss : 1.925e-04
iter: 2038, loss : 5.983e-04
iter: 2039, loss : 5.500e-04
iter: 2040, loss : 3.017e-04
iter: 2041, loss : 2.041e-04
iter: 2042, loss : 3.070e-04
iter: 2043, loss : 3.486e-03
iter: 2044, loss : 3.972e-04
iter: 2045, loss : 6.966e-04
iter: 2046, loss : 3.937e-04
iter: 2047, loss : 2.238e-04
iter: 2048, loss : 1.037e-04
iter: 2049, loss : 1.245e-04
iter: 2050, loss : 2.734e-04
iter: 2051, loss : 7.090e-04
iter: 2052, loss : 2.772e-04
iter: 2053, loss : 2.277e-04
iter: 2054, loss : 1.394e-04
iter: 2055, loss : 3.355e-04
iter: 2056, loss : 1.696e-04
iter: 2057, loss : 2.113e-04
iter: 2058, loss : 2.380e-04
iter: 2059, loss : 1.374e-04
iter: 2060, loss : 7.891e-05
iter: 2061, loss : 1.079e-04
iter: 2062, loss : 1.326e-04
iter: 2063, loss : 2.602e-04
iter: 2064, loss : 1.119e-04
iter: 2065, loss : 2.021e-03
iter: 2066, loss : 7.098e-04
iter: 2067, loss : 8.572e-04
iter: 2068, loss : 4.267e-04
iter: 2069, loss : 2.891e-04
iter: 2070, loss : 1.894e-04
iter: 2071, loss : 1.958e-04
iter: 2072, loss : 1.093e-04
iter: 2073, loss : 2.155e-04
iter: 2074, loss : 1.011e-04
iter: 2075, loss : 1.675e-04
iter: 2076, loss : 2.189e-04
iter: 2077, loss : 3.162e-04
iter: 2078, loss : 8.539e-05
iter: 2079, loss : 6.051e-05
iter: 2080, loss : 1.086e-04
iter: 2081, loss : 2.247e-04
iter: 2082, loss : 1.845e-04
iter: 2083, loss : 9.823e-05
iter: 2084, loss : 5.723e-05
iter: 2085, loss : 7.068e-05
iter: 2086, loss : 1.736e-04
iter: 2087, loss : 2.967e-04
iter: 2088, loss : 4.678e-04
iter: 2089, loss : 7.060e-05
iter: 2090, loss : 1.709e-03
iter: 2091, loss : 1.369e-04
iter: 2092, loss : 1.020e-04
iter: 2093, loss : 2.258e-04
iter: 2094, loss : 1.864e-04
iter: 2095, loss : 6.982e-05
iter: 2096, loss : 2.082e-04
iter: 2097, loss : 2.263e-04
iter: 2098, loss : 1.881e-03
iter: 2099, loss : 1.993e-04
iter: 2100, loss : 2.922e-03
iter: 2101, loss : 5.399e-04
iter: 2102, loss : 1.072e-04
iter: 2103, loss : 3.641e-04
iter: 2104, loss : 1.650e-04
iter: 2105, loss : 1.171e-04
iter: 2106, loss : 3.789e-04
iter: 2107, loss : 4.021e-04
iter: 2108, loss : 5.883e-04
iter: 2109, loss : 1.561e-03
iter: 2110, loss : 4.009e-04
iter: 2111, loss : 1.119e-03
iter: 2112, loss : 1.363e-04
iter: 2113, loss : 2.808e-04
iter: 2114, loss : 1.967e-04
iter: 2115, loss : 2.030e-04
iter: 2116, loss : 1.158e-03
iter: 2117, loss : 5.564e-04
iter: 2118, loss : 1.595e-04
iter: 2119, loss : 2.915e-04
iter: 2120, loss : 2.344e-04
iter: 2121, loss : 2.393e-04
iter: 2122, loss : 2.548e-04
iter: 2123, loss : 2.268e-04
iter: 2124, loss : 4.804e-04
iter: 2125, loss : 1.353e-04
iter: 2126, loss : 1.107e-04
iter: 2127, loss : 1.100e-04
iter: 2128, loss : 2.839e-04
iter: 2129, loss : 6.149e-05
iter: 2130, loss : 1.861e-04
iter: 2131, loss : 1.119e-04
iter: 2132, loss : 4.120e-04
iter: 2133, loss : 2.184e-04
iter: 2134, loss : 4.729e-04
iter: 2135, loss : 2.581e-03
iter: 2136, loss : 5.239e-04
iter: 2137, loss : 3.983e-04
iter: 2138, loss : 7.385e-04
iter: 2139, loss : 2.636e-04
iter: 2140, loss : 1.185e-04
iter: 2141, loss : 1.294e-04
iter: 2142, loss : 1.612e-04
iter: 2143, loss : 3.802e-04
iter: 2144, loss : 2.012e-04
iter: 2145, loss : 2.976e-04
iter: 2146, loss : 1.768e-04
iter: 2147, loss : 2.260e-04
iter: 2148, loss : 5.128e-04
iter: 2149, loss : 1.364e-04
iter: 2150, loss : 1.498e-04
iter: 2151, loss : 1.671e-04
iter: 2152, loss : 3.184e-03
iter: 2153, loss : 1.479e-04
iter: 2154, loss : 2.281e-04
iter: 2155, loss : 1.519e-04
iter: 2156, loss : 4.218e-03
iter: 2157, loss : 1.112e-03
iter: 2158, loss : 3.363e-04
iter: 2159, loss : 4.861e-04
iter: 2160, loss : 5.931e-04
iter: 2161, loss : 1.541e-04
iter: 2162, loss : 7.202e-05
iter: 2163, loss : 1.771e-04
iter: 2164, loss : 6.624e-04
iter: 2165, loss : 3.135e-04
iter: 2166, loss : 1.100e-04
iter: 2167, loss : 2.174e-04
iter: 2168, loss : 3.292e-03
iter: 2169, loss : 4.057e-04
iter: 2170, loss : 4.540e-04
iter: 2171, loss : 6.549e-04
iter: 2172, loss : 5.360e-04
iter: 2173, loss : 5.051e-04
iter: 2174, loss : 6.583e-04
iter: 2175, loss : 1.887e-03
iter: 2176, loss : 5.453e-04
iter: 2177, loss : 2.189e-04
iter: 2178, loss : 3.953e-03
iter: 2179, loss : 5.693e-04
iter: 2180, loss : 2.621e-04
iter: 2181, loss : 3.379e-03
iter: 2182, loss : 3.546e-04
iter: 2183, loss : 1.255e-04
iter: 2184, loss : 3.042e-04
iter: 2185, loss : 3.339e-04
iter: 2186, loss : 6.018e-04
iter: 2187, loss : 2.373e-05
iter: 2188, loss : 4.017e-04
iter: 2189, loss : 2.451e-04
iter: 2190, loss : 2.523e-04
iter: 2191, loss : 4.130e-04
iter: 2192, loss : 1.204e-04
iter: 2193, loss : 2.645e-04
iter: 2194, loss : 1.082e-04
iter: 2195, loss : 1.796e-04
iter: 2196, loss : 1.546e-04
iter: 2197, loss : 4.940e-05
iter: 2198, loss : 1.576e-03
iter: 2199, loss : 4.947e-04
iter: 2200, loss : 4.940e-04
iter: 2201, loss : 1.933e-04
iter: 2202, loss : 1.454e-04
iter: 2203, loss : 1.661e-04
iter: 2204, loss : 1.252e-04
iter: 2205, loss : 4.588e-05
iter: 2206, loss : 1.181e-04
iter: 2207, loss : 1.472e-03
iter: 2208, loss : 2.053e-04
iter: 2209, loss : 1.197e-03
iter: 2210, loss : 2.552e-04
iter: 2211, loss : 1.564e-04
iter: 2212, loss : 9.889e-05
iter: 2213, loss : 1.082e-04
iter: 2214, loss : 1.626e-04
iter: 2215, loss : 5.798e-05
iter: 2216, loss : 9.984e-05
iter: 2217, loss : 8.238e-04
iter: 2218, loss : 3.339e-04
iter: 2219, loss : 1.251e-04
iter: 2220, loss : 1.107e-04
iter: 2221, loss : 1.044e-03
iter: 2222, loss : 2.555e-04
iter: 2223, loss : 5.355e-04
iter: 2224, loss : 1.771e-04
iter: 2225, loss : 8.321e-04
iter: 2226, loss : 6.080e-04
iter: 2227, loss : 1.103e-04
iter: 2228, loss : 3.327e-03
iter: 2229, loss : 2.400e-04
iter: 2230, loss : 3.291e-04
iter: 2231, loss : 8.253e-05
iter: 2232, loss : 5.040e-04
iter: 2233, loss : 5.004e-04
iter: 2234, loss : 6.447e-04
iter: 2235, loss : 2.323e-04
iter: 2236, loss : 2.063e-04
iter: 2237, loss : 4.889e-04
iter: 2238, loss : 2.073e-04
iter: 2239, loss : 8.461e-04
iter: 2240, loss : 1.852e-04
iter: 2241, loss : 1.015e-03
iter: 2242, loss : 2.874e-04
iter: 2243, loss : 2.671e-04
iter: 2244, loss : 4.231e-04
iter: 2245, loss : 1.081e-04
iter: 2246, loss : 1.361e-04
iter: 2247, loss : 3.130e-04
iter: 2248, loss : 7.575e-05
iter: 2249, loss : 1.776e-04
iter: 2250, loss : 1.435e-04
iter: 2251, loss : 1.539e-04
iter: 2252, loss : 1.036e-03
iter: 2253, loss : 6.343e-04
iter: 2254, loss : 2.292e-04
iter: 2255, loss : 5.578e-04
iter: 2256, loss : 1.375e-03
iter: 2257, loss : 7.037e-04
iter: 2258, loss : 2.573e-04
iter: 2259, loss : 9.742e-05
iter: 2260, loss : 7.552e-05
iter: 2261, loss : 5.630e-04
iter: 2262, loss : 2.482e-04
iter: 2263, loss : 3.852e-04
iter: 2264, loss : 1.727e-04
iter: 2265, loss : 1.190e-04
iter: 2266, loss : 1.418e-04
iter: 2267, loss : 9.153e-05
iter: 2268, loss : 2.068e-03
iter: 2269, loss : 8.414e-05
iter: 2270, loss : 9.679e-05
iter: 2271, loss : 1.849e-03
iter: 2272, loss : 1.988e-04
iter: 2273, loss : 1.758e-04
iter: 2274, loss : 3.586e-03
iter: 2275, loss : 2.838e-04
iter: 2276, loss : 3.119e-04
iter: 2277, loss : 2.590e-04
iter: 2278, loss : 1.567e-04
iter: 2279, loss : 4.728e-04
iter: 2280, loss : 2.510e-04
iter: 2281, loss : 2.079e-04
iter: 2282, loss : 2.170e-03
iter: 2283, loss : 1.308e-04
iter: 2284, loss : 2.584e-04
iter: 2285, loss : 1.365e-04
iter: 2286, loss : 1.103e-04
iter: 2287, loss : 1.284e-04
iter: 2288, loss : 1.873e-04
iter: 2289, loss : 5.365e-04
iter: 2290, loss : 1.001e-04
iter: 2291, loss : 1.261e-04
iter: 2292, loss : 1.210e-04
iter: 2293, loss : 1.152e-03
iter: 2294, loss : 6.961e-04
iter: 2295, loss : 5.055e-04
iter: 2296, loss : 5.571e-05
iter: 2297, loss : 1.128e-04
iter: 2298, loss : 2.788e-04
iter: 2299, loss : 6.246e-05
iter: 2300, loss : 7.338e-05
iter: 2301, loss : 1.627e-04
iter: 2302, loss : 2.356e-04
iter: 2303, loss : 1.338e-03
iter: 2304, loss : 1.451e-04
iter: 2305, loss : 2.343e-05
iter: 2306, loss : 1.433e-04
iter: 2307, loss : 8.928e-04
iter: 2308, loss : 8.460e-04
iter: 2309, loss : 9.500e-04
iter: 2310, loss : 1.372e-04
iter: 2311, loss : 1.549e-03
iter: 2312, loss : 6.305e-04
iter: 2313, loss : 2.515e-04
iter: 2314, loss : 3.524e-04
iter: 2315, loss : 2.711e-04
iter: 2316, loss : 3.725e-04
iter: 2317, loss : 1.845e-04
iter: 2318, loss : 8.989e-05
iter: 2319, loss : 3.478e-04
iter: 2320, loss : 1.088e-04
iter: 2321, loss : 8.140e-05
iter: 2322, loss : 8.118e-05
iter: 2323, loss : 5.283e-04
iter: 2324, loss : 9.099e-05
iter: 2325, loss : 1.331e-04
iter: 2326, loss : 1.839e-04
iter: 2327, loss : 8.779e-05
iter: 2328, loss : 1.484e-04
iter: 2329, loss : 4.295e-03
iter: 2330, loss : 6.939e-04
iter: 2331, loss : 1.200e-04
iter: 2332, loss : 4.012e-04
iter: 2333, loss : 2.614e-04
iter: 2334, loss : 1.155e-04
iter: 2335, loss : 3.210e-04
iter: 2336, loss : 1.989e-04
iter: 2337, loss : 2.728e-04
iter: 2338, loss : 6.995e-05
iter: 2339, loss : 8.638e-04
iter: 2340, loss : 1.711e-04
iter: 2341, loss : 5.415e-04
iter: 2342, loss : 5.211e-05
iter: 2343, loss : 1.270e-04
iter: 2344, loss : 9.661e-04
iter: 2345, loss : 5.894e-04
iter: 2346, loss : 1.567e-04
iter: 2347, loss : 1.453e-03
iter: 2348, loss : 2.567e-04
iter: 2349, loss : 9.318e-05
iter: 2350, loss : 3.519e-03
iter: 2351, loss : 2.278e-04
iter: 2352, loss : 1.775e-04
iter: 2353, loss : 1.943e-04
iter: 2354, loss : 5.037e-04
iter: 2355, loss : 2.188e-03
iter: 2356, loss : 3.876e-04
iter: 2357, loss : 3.062e-04
iter: 2358, loss : 4.588e-04
iter: 2359, loss : 2.675e-03
iter: 2360, loss : 2.246e-04
iter: 2361, loss : 2.756e-04
iter: 2362, loss : 1.465e-03
iter: 2363, loss : 2.406e-04
iter: 2364, loss : 1.193e-03
iter: 2365, loss : 1.214e-04
iter: 2366, loss : 3.982e-04
iter: 2367, loss : 8.202e-05
iter: 2368, loss : 6.432e-04
iter: 2369, loss : 6.268e-04
iter: 2370, loss : 2.004e-04
iter: 2371, loss : 2.843e-04
iter: 2372, loss : 1.377e-04
iter: 2373, loss : 4.611e-04
iter: 2374, loss : 1.634e-04
iter: 2375, loss : 2.466e-04
iter: 2376, loss : 2.908e-04
iter: 2377, loss : 1.932e-04
iter: 2378, loss : 2.469e-04
iter: 2379, loss : 1.523e-04
iter: 2380, loss : 1.601e-04
iter: 2381, loss : 9.581e-05
iter: 2382, loss : 2.513e-04
iter: 2383, loss : 1.242e-03
iter: 2384, loss : 2.954e-04
iter: 2385, loss : 2.238e-04
iter: 2386, loss : 4.065e-04
iter: 2387, loss : 4.310e-04
iter: 2388, loss : 1.616e-03
iter: 2389, loss : 3.008e-03
iter: 2390, loss : 4.394e-04
iter: 2391, loss : 2.900e-04
iter: 2392, loss : 3.367e-04
iter: 2393, loss : 3.334e-04
iter: 2394, loss : 1.369e-04
iter: 2395, loss : 5.861e-04
iter: 2396, loss : 4.800e-04
iter: 2397, loss : 3.845e-04
iter: 2398, loss : 1.627e-03
iter: 2399, loss : 1.226e-04
iter: 2400, loss : 2.454e-04
iter: 2401, loss : 3.537e-04
iter: 2402, loss : 1.948e-04
iter: 2403, loss : 1.567e-04
iter: 2404, loss : 2.382e-03
iter: 2405, loss : 6.825e-04
iter: 2406, loss : 6.596e-05
iter: 2407, loss : 3.312e-04
iter: 2408, loss : 1.896e-04
iter: 2409, loss : 7.591e-04
iter: 2410, loss : 5.730e-04
iter: 2411, loss : 1.823e-04
iter: 2412, loss : 1.381e-03
iter: 2413, loss : 3.684e-04
iter: 2414, loss : 4.863e-04
iter: 2415, loss : 5.895e-05
iter: 2416, loss : 1.169e-04
iter: 2417, loss : 1.953e-04
iter: 2418, loss : 7.589e-05
iter: 2419, loss : 8.888e-05
iter: 2420, loss : 1.186e-04
iter: 2421, loss : 8.177e-05
iter: 2422, loss : 1.526e-04
iter: 2423, loss : 4.586e-04
iter: 2424, loss : 3.680e-04
iter: 2425, loss : 9.495e-05
iter: 2426, loss : 2.070e-04
iter: 2427, loss : 2.229e-04
iter: 2428, loss : 2.345e-03
iter: 2429, loss : 1.449e-04
iter: 2430, loss : 8.790e-04
iter: 2431, loss : 4.016e-04
iter: 2432, loss : 7.098e-05
iter: 2433, loss : 9.529e-04
iter: 2434, loss : 1.591e-04
iter: 2435, loss : 3.094e-03
iter: 2436, loss : 2.411e-04
iter: 2437, loss : 2.432e-03
iter: 2438, loss : 3.244e-04
iter: 2439, loss : 2.643e-04
iter: 2440, loss : 4.524e-04
iter: 2441, loss : 6.585e-04
iter: 2442, loss : 2.975e-04
iter: 2443, loss : 1.875e-04
iter: 2444, loss : 2.660e-04
iter: 2445, loss : 9.933e-04
iter: 2446, loss : 3.962e-04
iter: 2447, loss : 1.131e-04
iter: 2448, loss : 6.505e-04
iter: 2449, loss : 1.481e-04
iter: 2450, loss : 1.205e-04
iter: 2451, loss : 1.518e-04
iter: 2452, loss : 2.384e-04
iter: 2453, loss : 2.642e-04
iter: 2454, loss : 8.939e-05
iter: 2455, loss : 2.070e-04
iter: 2456, loss : 9.246e-05
iter: 2457, loss : 2.551e-04
iter: 2458, loss : 1.921e-04
iter: 2459, loss : 9.175e-05
iter: 2460, loss : 5.135e-05
iter: 2461, loss : 1.142e-04
iter: 2462, loss : 8.715e-04
iter: 2463, loss : 1.327e-04
iter: 2464, loss : 3.342e-04
iter: 2465, loss : 7.785e-05
iter: 2466, loss : 3.576e-04
iter: 2467, loss : 1.392e-04
iter: 2468, loss : 3.181e-04
iter: 2469, loss : 1.464e-04
iter: 2470, loss : 2.305e-04
iter: 2471, loss : 2.449e-04
iter: 2472, loss : 1.833e-04
iter: 2473, loss : 1.441e-04
iter: 2474, loss : 2.081e-04
iter: 2475, loss : 1.101e-04
iter: 2476, loss : 4.728e-05
iter: 2477, loss : 1.647e-04
iter: 2478, loss : 1.120e-04
iter: 2479, loss : 8.037e-05
iter: 2480, loss : 4.985e-04
iter: 2481, loss : 3.747e-03
iter: 2482, loss : 3.636e-04
iter: 2483, loss : 2.202e-04
iter: 2484, loss : 2.418e-04
iter: 2485, loss : 8.651e-05
iter: 2486, loss : 2.064e-04
iter: 2487, loss : 1.969e-04
iter: 2488, loss : 1.484e-04
iter: 2489, loss : 6.862e-05
iter: 2490, loss : 2.090e-04
iter: 2491, loss : 1.225e-04
iter: 2492, loss : 1.375e-04
iter: 2493, loss : 1.256e-04
iter: 2494, loss : 1.346e-04
iter: 2495, loss : 5.494e-05
iter: 2496, loss : 7.036e-04
iter: 2497, loss : 1.625e-04
iter: 2498, loss : 6.939e-05
iter: 2499, loss : 8.763e-05
iter: 2500, loss : 1.204e-04
iter: 2501, loss : 1.388e-04
iter: 2502, loss : 1.436e-04
iter: 2503, loss : 1.573e-04
iter: 2504, loss : 4.460e-04
iter: 2505, loss : 1.326e-04
iter: 2506, loss : 6.412e-05
iter: 2507, loss : 8.371e-05
iter: 2508, loss : 3.732e-05
iter: 2509, loss : 5.754e-05
iter: 2510, loss : 1.404e-04
iter: 2511, loss : 8.554e-05
iter: 2512, loss : 2.111e-03
iter: 2513, loss : 1.399e-04
iter: 2514, loss : 1.436e-04
iter: 2515, loss : 1.154e-04
iter: 2516, loss : 1.629e-04
iter: 2517, loss : 1.090e-04
iter: 2518, loss : 2.689e-04
iter: 2519, loss : 2.489e-05
iter: 2520, loss : 1.056e-04
iter: 2521, loss : 7.833e-05
iter: 2522, loss : 4.092e-05
iter: 2523, loss : 5.491e-04
iter: 2524, loss : 7.966e-04
iter: 2525, loss : 3.986e-03
iter: 2526, loss : 6.714e-05
iter: 2527, loss : 3.305e-04
iter: 2528, loss : 1.385e-04
iter: 2529, loss : 1.141e-04
iter: 2530, loss : 2.190e-04
iter: 2531, loss : 8.834e-05
iter: 2532, loss : 1.935e-04
iter: 2533, loss : 4.962e-04
iter: 2534, loss : 2.053e-04
iter: 2535, loss : 6.089e-05
iter: 2536, loss : 9.277e-05
iter: 2537, loss : 4.595e-04
iter: 2538, loss : 1.167e-04
iter: 2539, loss : 1.858e-04
iter: 2540, loss : 1.466e-04
iter: 2541, loss : 7.392e-05
iter: 2542, loss : 5.592e-04
iter: 2543, loss : 1.817e-03
iter: 2544, loss : 4.578e-04
iter: 2545, loss : 2.730e-04
iter: 2546, loss : 4.706e-04
iter: 2547, loss : 1.025e-04
iter: 2548, loss : 5.847e-05
iter: 2549, loss : 2.435e-04
iter: 2550, loss : 1.752e-04
iter: 2551, loss : 1.316e-04
iter: 2552, loss : 7.168e-04
iter: 2553, loss : 4.562e-05
iter: 2554, loss : 1.742e-04
iter: 2555, loss : 1.902e-04
iter: 2556, loss : 5.263e-04
iter: 2557, loss : 1.134e-03
iter: 2558, loss : 2.006e-04
iter: 2559, loss : 2.311e-04
iter: 2560, loss : 1.401e-04
iter: 2561, loss : 1.660e-04
iter: 2562, loss : 2.253e-04
iter: 2563, loss : 5.306e-04
iter: 2564, loss : 1.182e-04
iter: 2565, loss : 3.805e-04
iter: 2566, loss : 1.485e-04
iter: 2567, loss : 9.447e-05
iter: 2568, loss : 1.630e-04
iter: 2569, loss : 6.980e-04
iter: 2570, loss : 8.262e-05
iter: 2571, loss : 1.767e-04
iter: 2572, loss : 6.255e-04
iter: 2573, loss : 8.497e-05
iter: 2574, loss : 1.198e-04
iter: 2575, loss : 6.355e-05
iter: 2576, loss : 5.765e-05
iter: 2577, loss : 2.709e-04
iter: 2578, loss : 3.815e-04
iter: 2579, loss : 2.239e-03
iter: 2580, loss : 3.461e-04
iter: 2581, loss : 1.165e-04
iter: 2582, loss : 4.773e-05
iter: 2583, loss : 1.697e-04
iter: 2584, loss : 3.287e-04
iter: 2585, loss : 2.574e-04
iter: 2586, loss : 1.006e-04
iter: 2587, loss : 1.442e-04
iter: 2588, loss : 3.905e-05
iter: 2589, loss : 6.350e-05
iter: 2590, loss : 5.106e-04
iter: 2591, loss : 1.921e-04
iter: 2592, loss : 4.321e-04
iter: 2593, loss : 1.088e-04
iter: 2594, loss : 4.752e-04
iter: 2595, loss : 1.892e-05
iter: 2596, loss : 1.813e-03
iter: 2597, loss : 1.265e-04
iter: 2598, loss : 1.900e-04
iter: 2599, loss : 1.197e-04
iter: 2600, loss : 6.122e-04
iter: 2601, loss : 9.027e-05
iter: 2602, loss : 1.802e-04
iter: 2603, loss : 1.625e-04
iter: 2604, loss : 1.710e-03
iter: 2605, loss : 9.929e-05
iter: 2606, loss : 5.550e-04
iter: 2607, loss : 2.622e-04
iter: 2608, loss : 2.502e-04
iter: 2609, loss : 1.346e-04
iter: 2610, loss : 2.422e-04
iter: 2611, loss : 1.411e-04
iter: 2612, loss : 1.106e-03
iter: 2613, loss : 7.247e-05
iter: 2614, loss : 3.820e-04
iter: 2615, loss : 1.945e-04
iter: 2616, loss : 1.505e-04
iter: 2617, loss : 1.864e-04
iter: 2618, loss : 1.112e-04
iter: 2619, loss : 5.732e-05
iter: 2620, loss : 2.795e-05
iter: 2621, loss : 2.237e-04
iter: 2622, loss : 1.951e-03
iter: 2623, loss : 4.539e-04
iter: 2624, loss : 1.085e-04
iter: 2625, loss : 1.903e-04
iter: 2626, loss : 2.314e-04
iter: 2627, loss : 3.389e-03
iter: 2628, loss : 3.043e-04
iter: 2629, loss : 1.182e-04
iter: 2630, loss : 8.758e-05
iter: 2631, loss : 1.656e-03
iter: 2632, loss : 2.019e-04
iter: 2633, loss : 1.470e-04
iter: 2634, loss : 1.330e-04
iter: 2635, loss : 4.556e-04
iter: 2636, loss : 2.187e-04
iter: 2637, loss : 3.797e-04
iter: 2638, loss : 8.708e-05
iter: 2639, loss : 8.727e-05
iter: 2640, loss : 4.243e-04
iter: 2641, loss : 1.588e-03
iter: 2642, loss : 3.169e-04
iter: 2643, loss : 1.835e-04
iter: 2644, loss : 2.639e-04
iter: 2645, loss : 7.182e-05
iter: 2646, loss : 4.758e-04
iter: 2647, loss : 3.028e-03
iter: 2648, loss : 1.045e-04
iter: 2649, loss : 7.842e-05
iter: 2650, loss : 6.281e-04
iter: 2651, loss : 8.881e-05
iter: 2652, loss : 4.296e-04
iter: 2653, loss : 1.669e-04
iter: 2654, loss : 3.088e-04
iter: 2655, loss : 1.894e-04
iter: 2656, loss : 1.280e-04
iter: 2657, loss : 2.047e-03
iter: 2658, loss : 5.529e-04
iter: 2659, loss : 3.741e-04
iter: 2660, loss : 1.114e-03
iter: 2661, loss : 2.616e-04
iter: 2662, loss : 3.223e-04
iter: 2663, loss : 3.525e-04
iter: 2664, loss : 4.235e-05
iter: 2665, loss : 1.195e-04
iter: 2666, loss : 1.482e-04
iter: 2667, loss : 3.741e-04
iter: 2668, loss : 5.380e-04
iter: 2669, loss : 1.222e-04
iter: 2670, loss : 2.731e-04
iter: 2671, loss : 1.411e-04
iter: 2672, loss : 1.915e-04
iter: 2673, loss : 1.445e-04
iter: 2674, loss : 5.579e-05
iter: 2675, loss : 1.799e-04
iter: 2676, loss : 2.581e-04
iter: 2677, loss : 7.284e-05
iter: 2678, loss : 5.450e-05
iter: 2679, loss : 3.684e-04
iter: 2680, loss : 8.842e-05
iter: 2681, loss : 8.282e-05
iter: 2682, loss : 1.029e-04
iter: 2683, loss : 5.303e-04
iter: 2684, loss : 7.773e-05
iter: 2685, loss : 1.257e-04
iter: 2686, loss : 2.620e-03
iter: 2687, loss : 1.641e-04
iter: 2688, loss : 1.451e-04
iter: 2689, loss : 3.994e-04
iter: 2690, loss : 1.005e-04
iter: 2691, loss : 5.618e-05
iter: 2692, loss : 6.256e-04
iter: 2693, loss : 2.011e-04
iter: 2694, loss : 4.012e-05
iter: 2695, loss : 1.864e-03
iter: 2696, loss : 3.135e-04
iter: 2697, loss : 9.818e-05
iter: 2698, loss : 1.442e-04
iter: 2699, loss : 1.498e-04
iter: 2700, loss : 1.599e-03
iter: 2701, loss : 2.167e-04
iter: 2702, loss : 1.345e-04
iter: 2703, loss : 6.476e-05
iter: 2704, loss : 2.336e-04
iter: 2705, loss : 4.582e-04
iter: 2706, loss : 9.620e-05
iter: 2707, loss : 2.327e-03
iter: 2708, loss : 1.107e-03
iter: 2709, loss : 4.409e-04
iter: 2710, loss : 1.346e-04
iter: 2711, loss : 1.028e-04
iter: 2712, loss : 1.190e-04
iter: 2713, loss : 8.518e-04
iter: 2714, loss : 1.302e-04
iter: 2715, loss : 4.363e-05
iter: 2716, loss : 3.962e-04
iter: 2717, loss : 2.472e-04
iter: 2718, loss : 3.271e-04
iter: 2719, loss : 1.405e-04
iter: 2720, loss : 2.654e-04
iter: 2721, loss : 9.985e-05
iter: 2722, loss : 7.107e-04
iter: 2723, loss : 1.595e-04
iter: 2724, loss : 3.964e-04
iter: 2725, loss : 1.957e-04
iter: 2726, loss : 2.351e-04
iter: 2727, loss : 9.214e-05
iter: 2728, loss : 1.044e-04
iter: 2729, loss : 1.572e-04
iter: 2730, loss : 1.948e-04
iter: 2731, loss : 1.100e-04
iter: 2732, loss : 2.680e-04
iter: 2733, loss : 1.392e-04
iter: 2734, loss : 1.536e-04
iter: 2735, loss : 3.296e-03
iter: 2736, loss : 1.957e-04
iter: 2737, loss : 2.082e-04
iter: 2738, loss : 4.391e-04
iter: 2739, loss : 1.516e-03
iter: 2740, loss : 1.908e-04
iter: 2741, loss : 3.002e-04
iter: 2742, loss : 5.190e-05
iter: 2743, loss : 1.925e-04
iter: 2744, loss : 9.759e-04
iter: 2745, loss : 1.024e-03
iter: 2746, loss : 2.975e-04
iter: 2747, loss : 1.508e-04
iter: 2748, loss : 1.077e-03
iter: 2749, loss : 4.161e-04
iter: 2750, loss : 2.720e-04
iter: 2751, loss : 7.553e-04
iter: 2752, loss : 2.773e-04
iter: 2753, loss : 3.595e-04
iter: 2754, loss : 3.208e-04
iter: 2755, loss : 6.930e-04
iter: 2756, loss : 1.633e-04
iter: 2757, loss : 2.036e-04
iter: 2758, loss : 1.350e-04
iter: 2759, loss : 3.364e-04
iter: 2760, loss : 8.576e-05
iter: 2761, loss : 1.036e-04
iter: 2762, loss : 2.370e-04
iter: 2763, loss : 1.445e-04
iter: 2764, loss : 4.777e-04
iter: 2765, loss : 1.480e-04
iter: 2766, loss : 1.728e-03
iter: 2767, loss : 4.298e-04
iter: 2768, loss : 3.124e-04
iter: 2769, loss : 3.491e-04
iter: 2770, loss : 1.186e-04
iter: 2771, loss : 2.539e-04
iter: 2772, loss : 4.897e-05
iter: 2773, loss : 1.217e-04
iter: 2774, loss : 1.389e-04
iter: 2775, loss : 1.075e-04
iter: 2776, loss : 1.278e-04
iter: 2777, loss : 3.122e-03
iter: 2778, loss : 1.128e-04
iter: 2779, loss : 1.235e-04
iter: 2780, loss : 1.120e-04
iter: 2781, loss : 8.112e-05
iter: 2782, loss : 7.434e-05
iter: 2783, loss : 3.586e-04
iter: 2784, loss : 1.003e-04
iter: 2785, loss : 1.335e-04
iter: 2786, loss : 8.556e-04
iter: 2787, loss : 2.178e-03
iter: 2788, loss : 6.148e-05
iter: 2789, loss : 3.995e-04
iter: 2790, loss : 2.939e-04
iter: 2791, loss : 6.576e-05
iter: 2792, loss : 4.216e-03
iter: 2793, loss : 2.041e-04
iter: 2794, loss : 4.886e-04
iter: 2795, loss : 1.812e-03
iter: 2796, loss : 6.687e-04
iter: 2797, loss : 4.370e-04
iter: 2798, loss : 2.067e-04
iter: 2799, loss : 1.427e-04
iter: 2800, loss : 1.552e-04
iter: 2801, loss : 1.178e-04
iter: 2802, loss : 1.290e-04
iter: 2803, loss : 2.204e-04
iter: 2804, loss : 4.555e-04
iter: 2805, loss : 1.218e-04
iter: 2806, loss : 2.024e-04
iter: 2807, loss : 9.459e-04
iter: 2808, loss : 3.154e-04
iter: 2809, loss : 4.801e-04
iter: 2810, loss : 1.059e-04
iter: 2811, loss : 6.514e-05
iter: 2812, loss : 2.227e-04
iter: 2813, loss : 2.306e-04
iter: 2814, loss : 4.157e-04
iter: 2815, loss : 1.800e-04
iter: 2816, loss : 1.138e-04
iter: 2817, loss : 6.779e-05
iter: 2818, loss : 1.618e-04
iter: 2819, loss : 6.999e-04
iter: 2820, loss : 1.948e-04
iter: 2821, loss : 2.595e-04
iter: 2822, loss : 6.036e-05
iter: 2823, loss : 6.665e-05
iter: 2824, loss : 4.636e-05
iter: 2825, loss : 1.093e-04
iter: 2826, loss : 4.363e-04
iter: 2827, loss : 1.719e-04
iter: 2828, loss : 4.849e-05
iter: 2829, loss : 1.071e-03
iter: 2830, loss : 1.404e-04
iter: 2831, loss : 5.783e-04
iter: 2832, loss : 1.510e-04
iter: 2833, loss : 2.576e-04
iter: 2834, loss : 7.530e-05
iter: 2835, loss : 5.357e-04
iter: 2836, loss : 4.342e-05
iter: 2837, loss : 5.523e-04
iter: 2838, loss : 2.034e-03
iter: 2839, loss : 1.976e-03
iter: 2840, loss : 5.764e-04
iter: 2841, loss : 2.286e-03
iter: 2842, loss : 6.637e-04
iter: 2843, loss : 1.773e-03
iter: 2844, loss : 2.417e-04
iter: 2845, loss : 1.698e-04
iter: 2846, loss : 6.197e-05
iter: 2847, loss : 2.666e-04
iter: 2848, loss : 1.700e-03
iter: 2849, loss : 2.553e-04
iter: 2850, loss : 7.237e-04
iter: 2851, loss : 4.717e-04
iter: 2852, loss : 3.664e-04
iter: 2853, loss : 4.481e-04
iter: 2854, loss : 4.082e-04
iter: 2855, loss : 3.598e-04
iter: 2856, loss : 1.308e-04
iter: 2857, loss : 8.831e-05
iter: 2858, loss : 1.377e-04
iter: 2859, loss : 2.567e-03
iter: 2860, loss : 8.347e-04
iter: 2861, loss : 1.478e-04
iter: 2862, loss : 1.098e-04
iter: 2863, loss : 2.976e-04
iter: 2864, loss : 3.872e-04
iter: 2865, loss : 2.815e-04
iter: 2866, loss : 4.318e-04
iter: 2867, loss : 1.115e-04
iter: 2868, loss : 3.171e-04
iter: 2869, loss : 3.341e-04
iter: 2870, loss : 2.114e-04
iter: 2871, loss : 1.760e-04
iter: 2872, loss : 1.727e-04
iter: 2873, loss : 2.254e-04
iter: 2874, loss : 1.632e-03
iter: 2875, loss : 6.376e-05
iter: 2876, loss : 2.787e-04
iter: 2877, loss : 1.608e-04
iter: 2878, loss : 3.875e-04
iter: 2879, loss : 1.235e-04
iter: 2880, loss : 1.359e-04
iter: 2881, loss : 1.112e-04
iter: 2882, loss : 1.516e-04
iter: 2883, loss : 2.762e-04
iter: 2884, loss : 1.107e-04
iter: 2885, loss : 1.419e-04
iter: 2886, loss : 1.029e-04
iter: 2887, loss : 3.342e-04
iter: 2888, loss : 3.415e-03
iter: 2889, loss : 1.219e-04
iter: 2890, loss : 1.426e-04
iter: 2891, loss : 1.503e-04
iter: 2892, loss : 7.012e-05
iter: 2893, loss : 1.242e-04
iter: 2894, loss : 2.981e-04
iter: 2895, loss : 1.473e-04
iter: 2896, loss : 3.588e-04
iter: 2897, loss : 1.403e-04
iter: 2898, loss : 1.362e-04
iter: 2899, loss : 1.111e-04
iter: 2900, loss : 9.127e-05
iter: 2901, loss : 1.099e-04
iter: 2902, loss : 6.554e-05
iter: 2903, loss : 5.258e-05
iter: 2904, loss : 2.910e-04
iter: 2905, loss : 3.987e-04
iter: 2906, loss : 6.150e-05
iter: 2907, loss : 1.112e-04
iter: 2908, loss : 4.391e-04
iter: 2909, loss : 5.337e-04
iter: 2910, loss : 1.503e-04
iter: 2911, loss : 1.579e-04
iter: 2912, loss : 4.372e-04
iter: 2913, loss : 1.816e-04
iter: 2914, loss : 7.906e-05
iter: 2915, loss : 1.858e-04
iter: 2916, loss : 1.523e-04
iter: 2917, loss : 1.180e-04
iter: 2918, loss : 2.545e-04
iter: 2919, loss : 1.561e-04
iter: 2920, loss : 9.729e-05
iter: 2921, loss : 6.036e-05
iter: 2922, loss : 9.813e-05
iter: 2923, loss : 2.340e-03
iter: 2924, loss : 1.509e-04
iter: 2925, loss : 2.205e-04
iter: 2926, loss : 2.593e-04
iter: 2927, loss : 9.872e-05
iter: 2928, loss : 7.105e-05
iter: 2929, loss : 4.927e-05
iter: 2930, loss : 1.722e-04
iter: 2931, loss : 1.357e-04
iter: 2932, loss : 1.521e-04
iter: 2933, loss : 4.114e-05
iter: 2934, loss : 5.714e-04
iter: 2935, loss : 4.118e-04
iter: 2936, loss : 1.061e-04
iter: 2937, loss : 2.031e-04
iter: 2938, loss : 4.316e-05
iter: 2939, loss : 1.166e-03
iter: 2940, loss : 4.753e-04
iter: 2941, loss : 1.365e-04
iter: 2942, loss : 1.524e-04
iter: 2943, loss : 1.541e-03
iter: 2944, loss : 1.742e-04
iter: 2945, loss : 3.698e-04
iter: 2946, loss : 1.179e-04
iter: 2947, loss : 3.082e-04
iter: 2948, loss : 1.618e-04
iter: 2949, loss : 1.048e-04
iter: 2950, loss : 5.371e-05
iter: 2951, loss : 1.984e-04
iter: 2952, loss : 2.239e-04
iter: 2953, loss : 6.397e-05
iter: 2954, loss : 2.457e-04
iter: 2955, loss : 3.950e-05
iter: 2956, loss : 1.949e-04
iter: 2957, loss : 6.696e-04
iter: 2958, loss : 1.360e-04
iter: 2959, loss : 8.556e-04
iter: 2960, loss : 2.232e-04
iter: 2961, loss : 6.521e-04
iter: 2962, loss : 4.953e-04
iter: 2963, loss : 2.328e-04
iter: 2964, loss : 2.264e-04
iter: 2965, loss : 2.346e-04
iter: 2966, loss : 1.203e-04
iter: 2967, loss : 9.906e-04
iter: 2968, loss : 3.352e-04
iter: 2969, loss : 3.108e-03
iter: 2970, loss : 8.628e-05
iter: 2971, loss : 3.901e-04
iter: 2972, loss : 1.239e-04
iter: 2973, loss : 2.143e-04
iter: 2974, loss : 8.336e-05
iter: 2975, loss : 1.936e-04
iter: 2976, loss : 1.055e-04
iter: 2977, loss : 1.477e-04
iter: 2978, loss : 1.127e-03
iter: 2979, loss : 4.960e-04
iter: 2980, loss : 3.343e-04
iter: 2981, loss : 1.406e-04
iter: 2982, loss : 3.257e-03
iter: 2983, loss : 8.445e-05
iter: 2984, loss : 1.917e-04
iter: 2985, loss : 2.872e-04
iter: 2986, loss : 1.928e-04
iter: 2987, loss : 2.811e-04
iter: 2988, loss : 4.442e-04
iter: 2989, loss : 2.876e-04
iter: 2990, loss : 2.900e-04
iter: 2991, loss : 9.634e-05
iter: 2992, loss : 1.396e-04
iter: 2993, loss : 7.790e-05
iter: 2994, loss : 1.062e-04
iter: 2995, loss : 1.464e-04
iter: 2996, loss : 1.078e-03
iter: 2997, loss : 1.818e-04
iter: 2998, loss : 1.204e-04
iter: 2999, loss : 9.441e-05
iter: 2999, loss : 9.441e-05
--- 15.24744963645935 seconds --- to perform 3000 epochs
NOTE: For comparison we also called the torcheqdiff demo using the following command line ‘python ode_demo.py –niters 3000’. This triggers a comparable number of gradient evaluations on CPU. The iterations done in torcheqdiff complete in 348 seconds showing how on common CPUs, the use of heyoka may offer a significant speedup for NeuralODE research.
We are now ready to visualize the result of the learned neural ODE:
plt.plot(gt[:, 0], gt[:, 1])
ta_var.time = precision(0)
ta_var.state[:] = list(ic_var)
ta_var.state[:2] = ic
ta_var.pars[:] = np.array(nn_wb, dtype=np.single)
sol = ta_var.propagate_grid(np.array(t_grid, dtype=precision))[5]
plt.plot(sol[:, 0], sol[:, 1], ".")
[<matplotlib.lines.Line2D at 0x7fa152f30260>]
It is also of interest to visualize the predictions of the trained system over a new random batch, as opposed to the predictions of the entire trajectory from the first observation:
batch_ic, batch_t_grid, batch_y = get_batch(t_grid, gt, batch_size_ic, batch_size_time)
plt.plot(gt[:, 0], gt[:, 1])
for i in range(batch_size_ic):
plt.plot(batch_y[:, i, 0], batch_y[:, i, 1], "k.")
start_time = time.time()
for batch in batched(batch_ic, batch_size_simd):
ta_var_b.set_time(precision(0.0))
ta_var_b.state[:, :] = deepcopy(ic_var_b)
ta_var_b.state[:dim, :] = np.array(batch).T
ta_var_b.pars[:] = np.tile(nn_wb, (batch_size_simd, 1)).T
sol_b = ta_var_b.propagate_grid(np.array(t_grid_b, dtype=precision))[1]
for sol in [sol_b[:, :, i] for i in range(batch_size_simd)]:
plt.plot(sol[:, 0], sol[:, 1], ".")
print(
"--- %s seconds --- to compute prediction over the whole batch"
% (time.time() - start_time)
)
--- 0.008260488510131836 seconds --- to compute prediction over the whole batch
