Neural ODEs - II#
We here implement, in heyoka, a standard traning pipeline for NeuralODE replicating this torcheqdiff demo.
We will be using the very same parameters and dynamics, of the original demo but only use a simple fixed learning rate stochastic gradient descent. Note that it is also possible to use more advanced optimizers such as those implemented in pytorch while computing the gradients using heyoka, but in this tutorial we keep it simple and just update the gradient naively.
See also:
# The main imports
import numpy as np
import time
from itertools import batched
from copy import deepcopy
# Import Heyoka
import heyoka as hy
%matplotlib inline
import matplotlib.pyplot as plt
Generating the ground truth#
The dynamics we consider is a simple third order system:
\[\begin{split}
\left\{
\begin{array}{l}
\dot x = -0.1 x^3 - 2y^3 \\
\dot y = 2 x^3 - 0.1y^3
\end{array}
\right.
\end{split}\]
which we implement in heyoka as follows:
# The symolic variables representing the system state.
x, y = hy.make_vars("x", "y")
# We now assemble the dynamics.
A = np.array([[-0.1, 2.0], [-2.0, -0.1]])
tmp = np.array([[x * x * x, y * y * y]]) @ A
dyn = [(x, tmp[0][0]), (y, tmp[0][1])]
# For convenience we store here the dimension of the state.
dim = len(dyn)
To generate groundtruth observations from this system we will use the following parameters:
# This is the number of observations (i.e. the ground truth).
data_size = 1000
# This is the initial condition used to generate the observations (i.e. the ground truth).
ic = [2.0, 0.0]
# This is the time grid used to generate the observations (i.e. the ground truth).
t_grid = np.linspace(0.0, 25, data_size)
We start by instantiating a heyoka.taylor_adaptive object that will generate our observations. These will be considered as ground truth. Note that in other settings this data may be observed from a real phenomena, rather than generated numerically. In that case there will be noise which is here not modelled.
start_time = time.time()
ta = hy.taylor_adaptive(
# The ODEs.
dyn,
# The initial conditions.
ic,
# Do not operate in compact mode.
compact_mode=False,
# Define the tolerance
tol=1e-18,
)
print(
"--- %s seconds --- to build (jit) the Taylor integrator"
% (time.time() - start_time)
)
--- 0.07192206382751465 seconds --- to build (jit) the Taylor integrator
We are thus ready to generate the the ground truth using the method propagate_grid method of the Taylor adaptive integrator.
# It is good practice to always set initial conditions explicitly
ta.time = 0
ta.state[:] = ic
# This is our ground truth (gt)
gt = ta.propagate_grid(t_grid)[5]
Let us have a look at the generated observations.
plt.plot(gt[:, 0], gt[:, 1], "k.")
[<matplotlib.lines.Line2D at 0x7f144f0a68d0>]
Instantiating the neural ODE#
So far we have only generated the data we want to teach our Neural ODE to replicate. We now need to instantiate the Neural ODE system so that we can make predictions and compute the gradient of our loss.
Let us start with instantiating a feed-forward neural network using heyoka.models.ffnn.
# We define as nonlinearity a simple linear layer
linear = lambda inp: inp
# We call the factory to construct a FFNN. Following the torcheqdiff example here we put cubes as inputs:
ffnn = hy.model.ffnn(
inputs=[x * x * x, y * y * y],
nn_hidden=[50],
n_out=2,
activations=[hy.tanh, linear],
)
We are now able to build a taylor_adaptive_batch numerical integrator which will propagate the dynamics in batches as to leverage SIMD instructions:
\[\begin{split}
\left\{
\begin{array}{l}
\dot x = \mathcal N_{\theta, x}(x^3, y^3) \\
\dot y = \mathcal N_{\theta, y}(x^3, y^3) \\
\end{array}
\right.
\end{split}\]
And since we will want to be able to compute the sensitivities of this ODE we are first going to augment such dynamics with the first order variational equations:
# We decide on whether to use single or double precision. In this example it may
# Not be too important, but in other settings it can give a speedup when connected with batch and ensamble propagation.
# The single precision does effect, though, the quality of thegradient direction.
precision = np.double # np.single
batch_size_simd = hy.recommended_simd_size(fp_type=precision)
print("Batch size for SIMD use in integration: ", batch_size_simd)
# We thus define the neural dynamics
dyn_n = [(x, ffnn[0]), (y, ffnn[1])]
# We augment it with the variational one (only order 1 - i.e. gradient)
var_dyn_n = hy.var_ode_sys(dyn_n, args=hy.var_args.params, order=1)
Batch size for SIMD use in integration: 4
… and instantiate the variational integrator. For convenience we also instantiate the non batched version
# This is the batched version, which we will use for gradient computation
ta_var_b = hy.taylor_adaptive_batch(
# The ODEs.
var_dyn_n,
# The initial conditions.
np.ones((dim, batch_size_simd), dtype=precision),
# Operate in compact mode.
compact_mode=True,
# Define the tolerance (low tolerance is enough here)
tol=precision(1e-4),
# Single precision is enough here
fp_type=precision,
)
# This is the normal version used for wuick plotting and visualization of a single solution
ta_var = hy.taylor_adaptive(
# The ODEs.
var_dyn_n,
# The initial conditions.
ic,
# Operate in compact mode.
compact_mode=True,
# Define the tolerance (low tolerance is enough here)
tol=precision(1e-4),
# Single precision is enough here
fp_type=precision,
)
# We store for convenience the initial conditions in a numpy array (including the variational state)
ic_var_b = np.array(ta_var_b.state)
ic_var = np.array(ta_var.state)
Before starting the training, we define the details on the batch we are going to use in the generic step of stochastic gradient descent:
# This is the number of initial conditions to use in one batch (M)
batch_size_ic = 20
# This is the number of observations to predict from each sampled initial condition (T)
batch_size_time = 10
# We check that the observation batch size is a multiple of the simd batch size
print("This must be an integer: ", batch_size_ic / batch_size_simd)
This must be an integer: 5.0
This helper function creates a randomly generated batch from the available observation:
def get_batch(t_grid, gt, batch_size_ic=20, batch_size_time=10):
# We select the initial conditions from which generate predictions
s = np.random.choice(
np.arange(gt.shape[0] - batch_size_time, dtype=np.int64),
batch_size_ic,
replace=False,
)
batch_ic = gt[s, :] # (M, D)
# Assuming uniform grid and a non-autonomous system, all predictions will be made on the same time grid
batch_t_grid = t_grid[:batch_size_time] # (T)
batch_y = np.stack([gt[s + i] for i in range(batch_size_time)]) # (T,M,D)
return batch_ic, batch_t_grid, batch_y
Let us visualize what a batch looks like. First we generate one:
batch_ic, batch_t_grid, batch_y = get_batch(t_grid, gt, batch_size_ic, batch_size_time)
# We need the tgrid repeated as per heyoka API of the taylor adaptive batch
t_grid_b = np.repeat(batch_t_grid, batch_size_simd).reshape(-1, batch_size_simd)
Then we plot it overlapped to all the observations:
plt.plot(gt[:, 0], gt[:, 1])
for i in range(batch_size_ic):
plt.plot(batch_y[:, i, 0], batch_y[:, i, 1], "k.")
We initialize the network weights and biases.
# We compute initialization values for weigths / biases
n_pars = len(ta_var_b.pars)
nn_wb = np.random.normal(loc=0, scale=0.1, size=(n_pars,))
And visualize the prediction of the starting random network.
plt.plot(gt[:, 0], gt[:, 1])
for i in range(batch_size_ic):
plt.plot(batch_y[:, i, 0], batch_y[:, i, 1], "k.")
start_time = time.time()
for batch in batched(batch_ic, batch_size_simd):
ta_var_b.set_time(precision(0.0))
ta_var_b.state[:, :] = deepcopy(ic_var_b)
ta_var_b.state[:dim, :] = np.array(batch).T
ta_var_b.pars[:] = np.tile(nn_wb, (batch_size_simd, 1)).T
sol_b = ta_var_b.propagate_grid(np.array(t_grid_b, dtype=precision))[1]
for sol in [sol_b[:, :, i] for i in range(batch_size_simd)]:
plt.plot(sol[:, 0], sol[:, 1], ".")
print(
"--- %s seconds --- to compute prediction over the whole batch"
% (time.time() - start_time)
)
--- 0.007146596908569336 seconds --- to compute prediction over the whole batch
Training#
To train we need to first compute the gradient of the loss over the batch (M initial conditions and T times). Formally
\[
\mathcal L_{\theta} = \sum_{i=0}^{M}\sum_{j=0}^T (\pmb y_{ij} - \hat {\pmb y}_{ij})\cdot (\pmb y_{ij} - \hat {\pmb y}_{ij})
\]
\[
\frac{\partial L_\theta}{\partial \theta} = 2 \sum_{i=0}^{M}\sum_{j=0}^T (\pmb y_{ij} - \hat {\pmb y}_{ij})\cdot \frac{\partial \pmb y_{ij}}{\partial \theta}
\]
This helper function computes the loss and the gradient over one batch. The gradient is computed rearranging the ODE sensitivities computed propagating the variational dynamics:
def loss_and_gradient(nn_wb, batch_ic, batch_y):
# We reset the batch_loss value
batch_loss = 0
# We reset the gradient loss
grad = np.array([0.0] * ta_var_b.pars.shape[0])
counter = 0
# We process the batch (of the dataset) in batches for SIMD efficient processing
for batch_simd in batched(batch_ic, batch_size_simd):
ta_var_b.set_time(precision(0.0))
ta_var_b.state[:, :] = deepcopy(ic_var_b)
ta_var_b.state[:dim, :] = np.array(batch_simd).T
ta_var_b.pars[:] = np.tile(nn_wb, (batch_size_simd, 1)).T
sol_b = ta_var_b.propagate_grid(np.array(t_grid_b, dtype=precision))[1]
# The numerical integration on the simd_batch is done, we process its result
# accumulating loss and gradient on the predictions
for sol in [sol_b[:, :, i] for i in range(batch_size_simd)]:
# Here is the term (y-y_hat)
diff = sol[:, :dim] - batch_y[:, counter, :]
counter += 1
# Which is then summed sum sum (y-y_hat).(y-y_hat)
batch_loss += np.sum(diff**2)
# And the gradient computed as 2 sum (y-y_hat).dy
for dy in sol[:, dim:].reshape(batch_y.shape[0], dim, -1):
grad += 2 * np.sum(diff @ dy, axis=0)
# We then take the mean over the points used
batch_loss /= batch_size_ic * batch_size_time
grad /= batch_size_ic * batch_size_time
return batch_loss, grad
Using the various blocks defined above the training loop takes the form:
# We reset initialization values for weigths / biases
n_pars = len(ta_var_b.pars)
nn_wb = np.random.normal(loc=0, scale=0.1, size=(n_pars,))
# We loop over the epochs
n_epochs = 3000
start_time = time.time()
for i in range(n_epochs):
# 1 - generate a batch
batch_ic, batch_t_grid, batch_y = get_batch(
t_grid, gt, batch_size_ic=batch_size_ic, batch_size_time=batch_size_time
)
# 2 - compute the loss and gradient on the batch
batch_loss, grad = loss_and_gradient(nn_wb, batch_ic, batch_y)
print(f"iter: {i}, loss : {batch_loss:.3e}", end="\r")
# 3 - update the weights and biases (learning rate here is fixed, we use a naive update rule which in this case works)
nn_wb = nn_wb - 0.1 * grad
print(f"iter: {i}, loss : {batch_loss:.3e}", end="\n")
print("--- %s seconds --- to perform %i epochs" % (time.time() - start_time, n_epochs))
iter: 0, loss : 1.522e-02
iter: 1, loss : 1.752e-02
iter: 2, loss : 6.587e-02
iter: 3, loss : 5.735e-02
iter: 4, loss : 6.526e-02
iter: 5, loss : 3.101e-02
iter: 6, loss : 9.960e-02
iter: 7, loss : 3.839e-02
iter: 8, loss : 4.778e-02
iter: 9, loss : 3.129e-02
iter: 10, loss : 1.156e-01
iter: 11, loss : 1.324e-02
iter: 12, loss : 3.057e-02
iter: 13, loss : 6.824e-02
iter: 14, loss : 7.627e-02
iter: 15, loss : 3.567e-02
iter: 16, loss : 1.574e-01
iter: 17, loss : 8.066e-02
iter: 18, loss : 3.568e-02
iter: 19, loss : 1.306e-01
iter: 20, loss : 4.350e-02
iter: 21, loss : 2.531e-02
iter: 22, loss : 3.422e-02
iter: 23, loss : 1.152e-02
iter: 24, loss : 2.575e-02
iter: 25, loss : 6.065e-02
iter: 26, loss : 8.935e-03
iter: 27, loss : 1.522e-02
iter: 28, loss : 1.755e-02
iter: 29, loss : 1.092e-02
iter: 30, loss : 9.360e-03
iter: 31, loss : 5.935e-03
iter: 32, loss : 1.037e-02
iter: 33, loss : 2.046e-02
iter: 34, loss : 8.765e-03
iter: 35, loss : 1.023e-02
iter: 36, loss : 5.479e-02
iter: 37, loss : 5.183e-02
iter: 38, loss : 1.075e-01
iter: 39, loss : 2.096e-02
iter: 40, loss : 4.377e-02
iter: 41, loss : 3.420e-02
iter: 42, loss : 4.578e-02
iter: 43, loss : 3.843e-03
iter: 44, loss : 3.366e-02
iter: 45, loss : 1.900e-02
iter: 46, loss : 5.518e-02
iter: 47, loss : 3.470e-03
iter: 48, loss : 3.911e-03
iter: 49, loss : 1.699e-03
iter: 50, loss : 7.079e-03
iter: 51, loss : 4.733e-02
iter: 52, loss : 2.371e-04
iter: 53, loss : 1.037e-02
iter: 54, loss : 2.870e-03
iter: 55, loss : 1.564e-02
iter: 56, loss : 2.011e-03
iter: 57, loss : 3.774e-02
iter: 58, loss : 4.685e-03
iter: 59, loss : 1.295e-02
iter: 60, loss : 2.115e-03
iter: 61, loss : 4.458e-02
iter: 62, loss : 3.591e-02
iter: 63, loss : 1.123e-03
iter: 64, loss : 1.959e-02
iter: 65, loss : 1.211e-03
iter: 66, loss : 1.084e-02
iter: 67, loss : 1.413e-02
iter: 68, loss : 1.413e-03
iter: 69, loss : 1.283e-03
iter: 70, loss : 4.513e-04
iter: 71, loss : 2.251e-04
iter: 72, loss : 3.608e-02
iter: 73, loss : 3.678e-04
iter: 74, loss : 5.628e-03
iter: 75, loss : 3.579e-03
iter: 76, loss : 3.153e-03
iter: 77, loss : 2.191e-02
iter: 78, loss : 4.129e-05
iter: 79, loss : 2.391e-03
iter: 80, loss : 6.197e-03
iter: 81, loss : 1.566e-04
iter: 82, loss : 2.857e-03
iter: 83, loss : 2.956e-03
iter: 84, loss : 3.476e-05
iter: 85, loss : 1.986e-03
iter: 86, loss : 5.653e-04
iter: 87, loss : 1.084e-03
iter: 88, loss : 2.949e-03
iter: 89, loss : 3.095e-02
iter: 90, loss : 2.334e-02
iter: 91, loss : 7.686e-04
iter: 92, loss : 1.383e-02
iter: 93, loss : 1.088e-03
iter: 94, loss : 2.296e-02
iter: 95, loss : 6.781e-04
iter: 96, loss : 8.076e-04
iter: 97, loss : 9.386e-03
iter: 98, loss : 4.312e-04
iter: 99, loss : 3.613e-04
iter: 100, loss : 1.299e-03
iter: 101, loss : 5.698e-03
iter: 102, loss : 4.709e-04
iter: 103, loss : 7.853e-03
iter: 104, loss : 2.422e-03
iter: 105, loss : 3.042e-04
iter: 106, loss : 7.017e-04
iter: 107, loss : 7.817e-04
iter: 108, loss : 3.572e-04
iter: 109, loss : 4.004e-04
iter: 110, loss : 4.990e-04
iter: 111, loss : 3.298e-03
iter: 112, loss : 3.255e-02
iter: 113, loss : 9.219e-04
iter: 114, loss : 5.943e-03
iter: 115, loss : 6.894e-04
iter: 116, loss : 7.189e-04
iter: 117, loss : 1.064e-03
iter: 118, loss : 1.261e-03
iter: 119, loss : 4.987e-03
iter: 120, loss : 3.183e-03
iter: 121, loss : 6.616e-04
iter: 122, loss : 1.117e-02
iter: 123, loss : 4.086e-04
iter: 124, loss : 1.635e-03
iter: 125, loss : 4.851e-03
iter: 126, loss : 2.178e-02
iter: 127, loss : 4.388e-04
iter: 128, loss : 6.890e-03
iter: 129, loss : 1.322e-03
iter: 130, loss : 9.149e-04
iter: 131, loss : 1.189e-03
iter: 132, loss : 1.082e-03
iter: 133, loss : 5.536e-03
iter: 134, loss : 1.165e-03
iter: 135, loss : 7.232e-04
iter: 136, loss : 8.663e-03
iter: 137, loss : 4.715e-04
iter: 138, loss : 5.551e-04
iter: 139, loss : 5.639e-04
iter: 140, loss : 3.429e-03
iter: 141, loss : 4.540e-04
iter: 142, loss : 4.743e-03
iter: 143, loss : 5.495e-04
iter: 144, loss : 7.274e-04
iter: 145, loss : 5.221e-04
iter: 146, loss : 7.734e-03
iter: 147, loss : 1.244e-03
iter: 148, loss : 5.258e-04
iter: 149, loss : 3.815e-03
iter: 150, loss : 5.634e-04
iter: 151, loss : 7.229e-04
iter: 152, loss : 7.892e-04
iter: 153, loss : 7.445e-04
iter: 154, loss : 6.791e-04
iter: 155, loss : 4.217e-04
iter: 156, loss : 1.327e-03
iter: 157, loss : 2.991e-04
iter: 158, loss : 9.099e-04
iter: 159, loss : 6.855e-04
iter: 160, loss : 2.062e-04
iter: 161, loss : 4.068e-04
iter: 162, loss : 4.856e-03
iter: 163, loss : 5.003e-04
iter: 164, loss : 8.807e-04
iter: 165, loss : 9.254e-04
iter: 166, loss : 4.239e-04
iter: 167, loss : 2.449e-04
iter: 168, loss : 3.707e-03
iter: 169, loss : 5.890e-03
iter: 170, loss : 5.898e-03
iter: 171, loss : 5.131e-04
iter: 172, loss : 4.758e-04
iter: 173, loss : 4.810e-04
iter: 174, loss : 5.436e-04
iter: 175, loss : 4.823e-04
iter: 176, loss : 2.187e-04
iter: 177, loss : 1.011e-03
iter: 178, loss : 2.669e-04
iter: 179, loss : 4.056e-04
iter: 180, loss : 7.824e-03
iter: 181, loss : 1.102e-02
iter: 182, loss : 4.185e-04
iter: 183, loss : 6.316e-03
iter: 184, loss : 5.570e-04
iter: 185, loss : 7.334e-04
iter: 186, loss : 6.172e-03
iter: 187, loss : 6.705e-04
iter: 188, loss : 9.788e-03
iter: 189, loss : 1.233e-02
iter: 190, loss : 1.396e-03
iter: 191, loss : 1.340e-02
iter: 192, loss : 1.048e-02
iter: 193, loss : 9.009e-04
iter: 194, loss : 8.662e-03
iter: 195, loss : 1.536e-03
iter: 196, loss : 8.261e-03
iter: 197, loss : 6.260e-03
iter: 198, loss : 6.484e-03
iter: 199, loss : 7.478e-04
iter: 200, loss : 1.886e-03
iter: 201, loss : 1.163e-03
iter: 202, loss : 7.785e-04
iter: 203, loss : 2.859e-04
iter: 204, loss : 4.077e-04
iter: 205, loss : 2.633e-04
iter: 206, loss : 1.283e-03
iter: 207, loss : 5.171e-04
iter: 208, loss : 3.441e-04
iter: 209, loss : 6.296e-03
iter: 210, loss : 1.666e-03
iter: 211, loss : 3.105e-04
iter: 212, loss : 3.040e-03
iter: 213, loss : 4.878e-03
iter: 214, loss : 6.176e-04
iter: 215, loss : 3.999e-03
iter: 216, loss : 1.382e-03
iter: 217, loss : 1.048e-02
iter: 218, loss : 1.297e-02
iter: 219, loss : 1.355e-03
iter: 220, loss : 1.154e-03
iter: 221, loss : 4.810e-04
iter: 222, loss : 1.398e-03
iter: 223, loss : 7.907e-03
iter: 224, loss : 9.047e-04
iter: 225, loss : 3.059e-03
iter: 226, loss : 5.181e-04
iter: 227, loss : 7.901e-03
iter: 228, loss : 3.034e-03
iter: 229, loss : 1.098e-03
iter: 230, loss : 1.536e-03
iter: 231, loss : 2.131e-04
iter: 232, loss : 1.751e-03
iter: 233, loss : 5.789e-04
iter: 234, loss : 9.932e-04
iter: 235, loss : 8.259e-04
iter: 236, loss : 1.102e-03
iter: 237, loss : 1.279e-03
iter: 238, loss : 7.247e-04
iter: 239, loss : 6.564e-04
iter: 240, loss : 6.280e-03
iter: 241, loss : 5.765e-04
iter: 242, loss : 6.717e-03
iter: 243, loss : 9.348e-04
iter: 244, loss : 7.578e-04
iter: 245, loss : 4.612e-03
iter: 246, loss : 2.782e-03
iter: 247, loss : 7.150e-04
iter: 248, loss : 1.489e-03
iter: 249, loss : 1.275e-02
iter: 250, loss : 1.430e-03
iter: 251, loss : 4.022e-03
iter: 252, loss : 2.767e-03
iter: 253, loss : 4.264e-04
iter: 254, loss : 2.883e-04
iter: 255, loss : 1.720e-03
iter: 256, loss : 5.909e-04
iter: 257, loss : 6.030e-04
iter: 258, loss : 3.819e-04
iter: 259, loss : 4.016e-04
iter: 260, loss : 2.443e-03
iter: 261, loss : 1.056e-03
iter: 262, loss : 4.299e-04
iter: 263, loss : 2.697e-03
iter: 264, loss : 4.340e-03
iter: 265, loss : 8.603e-04
iter: 266, loss : 2.623e-03
iter: 267, loss : 2.757e-03
iter: 268, loss : 9.056e-04
iter: 269, loss : 1.986e-03
iter: 270, loss : 7.009e-04
iter: 271, loss : 1.128e-03
iter: 272, loss : 6.728e-04
iter: 273, loss : 3.725e-03
iter: 274, loss : 8.121e-04
iter: 275, loss : 4.113e-04
iter: 276, loss : 7.628e-04
iter: 277, loss : 5.220e-04
iter: 278, loss : 1.949e-03
iter: 279, loss : 4.085e-03
iter: 280, loss : 6.260e-04
iter: 281, loss : 7.889e-04
iter: 282, loss : 4.786e-04
iter: 283, loss : 8.651e-04
iter: 284, loss : 8.218e-04
iter: 285, loss : 1.243e-03
iter: 286, loss : 2.784e-03
iter: 287, loss : 1.973e-02
iter: 288, loss : 1.613e-03
iter: 289, loss : 7.812e-04
iter: 290, loss : 8.959e-04
iter: 291, loss : 6.946e-04
iter: 292, loss : 5.246e-04
iter: 293, loss : 1.282e-02
iter: 294, loss : 7.288e-03
iter: 295, loss : 6.426e-03
iter: 296, loss : 9.524e-04
iter: 297, loss : 9.138e-04
iter: 298, loss : 8.520e-04
iter: 299, loss : 1.400e-03
iter: 300, loss : 7.436e-04
iter: 301, loss : 5.664e-04
iter: 302, loss : 8.500e-04
iter: 303, loss : 9.129e-04
iter: 304, loss : 2.752e-04
iter: 305, loss : 3.593e-04
iter: 306, loss : 3.048e-03
iter: 307, loss : 1.272e-03
iter: 308, loss : 4.947e-03
iter: 309, loss : 6.771e-03
iter: 310, loss : 1.018e-03
iter: 311, loss : 8.631e-03
iter: 312, loss : 2.891e-03
iter: 313, loss : 7.683e-04
iter: 314, loss : 3.458e-03
iter: 315, loss : 4.093e-03
iter: 316, loss : 2.005e-03
iter: 317, loss : 8.154e-04
iter: 318, loss : 1.017e-02
iter: 319, loss : 1.482e-03
iter: 320, loss : 6.095e-04
iter: 321, loss : 1.072e-03
iter: 322, loss : 9.736e-04
iter: 323, loss : 3.900e-04
iter: 324, loss : 5.734e-04
iter: 325, loss : 1.630e-03
iter: 326, loss : 9.138e-04
iter: 327, loss : 7.615e-04
iter: 328, loss : 5.464e-04
iter: 329, loss : 7.380e-04
iter: 330, loss : 2.312e-03
iter: 331, loss : 7.487e-04
iter: 332, loss : 6.265e-04
iter: 333, loss : 2.744e-03
iter: 334, loss : 8.739e-04
iter: 335, loss : 2.916e-03
iter: 336, loss : 3.322e-03
iter: 337, loss : 5.578e-04
iter: 338, loss : 7.034e-04
iter: 339, loss : 8.350e-04
iter: 340, loss : 2.008e-03
iter: 341, loss : 5.673e-04
iter: 342, loss : 6.594e-03
iter: 343, loss : 8.330e-04
iter: 344, loss : 2.892e-04
iter: 345, loss : 1.108e-03
iter: 346, loss : 1.278e-03
iter: 347, loss : 9.214e-04
iter: 348, loss : 2.307e-03
iter: 349, loss : 5.390e-04
iter: 350, loss : 6.799e-04
iter: 351, loss : 5.389e-04
iter: 352, loss : 3.036e-04
iter: 353, loss : 1.495e-02
iter: 354, loss : 9.318e-04
iter: 355, loss : 6.415e-03
iter: 356, loss : 5.249e-04
iter: 357, loss : 6.538e-04
iter: 358, loss : 8.857e-04
iter: 359, loss : 4.986e-03
iter: 360, loss : 7.621e-04
iter: 361, loss : 3.715e-03
iter: 362, loss : 6.500e-04
iter: 363, loss : 4.749e-04
iter: 364, loss : 4.709e-04
iter: 365, loss : 7.651e-04
iter: 366, loss : 3.858e-04
iter: 367, loss : 3.767e-04
iter: 368, loss : 1.668e-04
iter: 369, loss : 8.004e-04
iter: 370, loss : 9.453e-04
iter: 371, loss : 1.430e-03
iter: 372, loss : 8.683e-03
iter: 373, loss : 3.632e-03
iter: 374, loss : 6.426e-03
iter: 375, loss : 2.906e-03
iter: 376, loss : 2.201e-03
iter: 377, loss : 2.045e-03
iter: 378, loss : 8.129e-04
iter: 379, loss : 4.240e-04
iter: 380, loss : 9.374e-04
iter: 381, loss : 4.165e-04
iter: 382, loss : 5.240e-04
iter: 383, loss : 1.120e-03
iter: 384, loss : 5.805e-04
iter: 385, loss : 8.539e-04
iter: 386, loss : 1.052e-02
iter: 387, loss : 1.164e-02
iter: 388, loss : 7.961e-03
iter: 389, loss : 3.724e-03
iter: 390, loss : 1.833e-03
iter: 391, loss : 1.646e-03
iter: 392, loss : 5.761e-04
iter: 393, loss : 5.507e-04
iter: 394, loss : 2.091e-04
iter: 395, loss : 9.212e-03
iter: 396, loss : 1.351e-02
iter: 397, loss : 3.173e-04
iter: 398, loss : 1.597e-03
iter: 399, loss : 2.383e-03
iter: 400, loss : 7.691e-04
iter: 401, loss : 3.820e-03
iter: 402, loss : 1.720e-03
iter: 403, loss : 1.439e-03
iter: 404, loss : 8.950e-04
iter: 405, loss : 3.659e-04
iter: 406, loss : 8.422e-04
iter: 407, loss : 7.441e-04
iter: 408, loss : 8.005e-04
iter: 409, loss : 6.344e-04
iter: 410, loss : 8.061e-04
iter: 411, loss : 5.535e-04
iter: 412, loss : 4.173e-04
iter: 413, loss : 5.275e-04
iter: 414, loss : 1.844e-03
iter: 415, loss : 5.928e-04
iter: 416, loss : 2.534e-04
iter: 417, loss : 2.252e-04
iter: 418, loss : 4.862e-04
iter: 419, loss : 1.454e-03
iter: 420, loss : 5.330e-04
iter: 421, loss : 2.846e-04
iter: 422, loss : 5.030e-04
iter: 423, loss : 1.970e-03
iter: 424, loss : 5.679e-03
iter: 425, loss : 5.540e-04
iter: 426, loss : 2.773e-04
iter: 427, loss : 1.814e-03
iter: 428, loss : 3.123e-04
iter: 429, loss : 2.893e-04
iter: 430, loss : 4.239e-04
iter: 431, loss : 3.975e-04
iter: 432, loss : 6.812e-04
iter: 433, loss : 4.000e-04
iter: 434, loss : 3.657e-04
iter: 435, loss : 2.788e-04
iter: 436, loss : 6.282e-04
iter: 437, loss : 4.622e-03
iter: 438, loss : 3.433e-03
iter: 439, loss : 5.692e-04
iter: 440, loss : 4.369e-04
iter: 441, loss : 3.595e-03
iter: 442, loss : 2.905e-04
iter: 443, loss : 4.805e-04
iter: 444, loss : 7.244e-04
iter: 445, loss : 5.699e-04
iter: 446, loss : 5.202e-04
iter: 447, loss : 2.355e-03
iter: 448, loss : 4.105e-03
iter: 449, loss : 2.486e-03
iter: 450, loss : 4.545e-04
iter: 451, loss : 2.709e-03
iter: 452, loss : 3.958e-04
iter: 453, loss : 3.587e-04
iter: 454, loss : 4.573e-04
iter: 455, loss : 3.033e-04
iter: 456, loss : 4.288e-04
iter: 457, loss : 5.758e-04
iter: 458, loss : 4.793e-04
iter: 459, loss : 4.621e-04
iter: 460, loss : 1.993e-03
iter: 461, loss : 4.752e-04
iter: 462, loss : 4.032e-03
iter: 463, loss : 4.278e-04
iter: 464, loss : 4.134e-04
iter: 465, loss : 2.506e-04
iter: 466, loss : 3.399e-04
iter: 467, loss : 1.474e-04
iter: 468, loss : 2.429e-04
iter: 469, loss : 5.434e-03
iter: 470, loss : 4.729e-03
iter: 471, loss : 3.711e-04
iter: 472, loss : 6.891e-04
iter: 473, loss : 1.766e-03
iter: 474, loss : 9.565e-03
iter: 475, loss : 5.504e-04
iter: 476, loss : 3.606e-04
iter: 477, loss : 3.141e-04
iter: 478, loss : 6.041e-04
iter: 479, loss : 1.365e-04
iter: 480, loss : 2.394e-03
iter: 481, loss : 7.559e-03
iter: 482, loss : 4.461e-04
iter: 483, loss : 1.501e-04
iter: 484, loss : 5.123e-04
iter: 485, loss : 5.616e-04
iter: 486, loss : 8.722e-03
iter: 487, loss : 5.391e-03
iter: 488, loss : 6.842e-04
iter: 489, loss : 5.717e-04
iter: 490, loss : 1.958e-03
iter: 491, loss : 4.029e-04
iter: 492, loss : 3.194e-04
iter: 493, loss : 3.114e-04
iter: 494, loss : 3.682e-03
iter: 495, loss : 1.771e-03
iter: 496, loss : 2.818e-03
iter: 497, loss : 8.544e-04
iter: 498, loss : 4.788e-04
iter: 499, loss : 3.367e-04
iter: 500, loss : 1.173e-03
iter: 501, loss : 2.412e-03
iter: 502, loss : 1.294e-03
iter: 503, loss : 5.056e-04
iter: 504, loss : 3.782e-04
iter: 505, loss : 3.057e-04
iter: 506, loss : 2.009e-04
iter: 507, loss : 4.627e-04
iter: 508, loss : 7.196e-03
iter: 509, loss : 3.115e-03
iter: 510, loss : 4.076e-04
iter: 511, loss : 3.690e-03
iter: 512, loss : 5.161e-04
iter: 513, loss : 1.339e-03
iter: 514, loss : 8.973e-04
iter: 515, loss : 3.266e-04
iter: 516, loss : 2.782e-04
iter: 517, loss : 1.341e-02
iter: 518, loss : 3.819e-03
iter: 519, loss : 8.228e-04
iter: 520, loss : 9.706e-04
iter: 521, loss : 4.331e-03
iter: 522, loss : 1.425e-03
iter: 523, loss : 1.355e-03
iter: 524, loss : 7.800e-04
iter: 525, loss : 3.535e-03
iter: 526, loss : 9.577e-04
iter: 527, loss : 8.335e-04
iter: 528, loss : 4.438e-04
iter: 529, loss : 3.831e-04
iter: 530, loss : 5.519e-04
iter: 531, loss : 5.185e-04
iter: 532, loss : 1.403e-03
iter: 533, loss : 8.287e-04
iter: 534, loss : 4.291e-04
iter: 535, loss : 6.223e-04
iter: 536, loss : 5.932e-04
iter: 537, loss : 1.575e-02
iter: 538, loss : 4.033e-04
iter: 539, loss : 2.974e-04
iter: 540, loss : 3.595e-04
iter: 541, loss : 3.376e-04
iter: 542, loss : 1.020e-03
iter: 543, loss : 4.299e-04
iter: 544, loss : 2.528e-04
iter: 545, loss : 4.067e-03
iter: 546, loss : 3.773e-04
iter: 547, loss : 1.928e-03
iter: 548, loss : 2.585e-04
iter: 549, loss : 2.554e-04
iter: 550, loss : 4.129e-04
iter: 551, loss : 5.025e-03
iter: 552, loss : 9.991e-03
iter: 553, loss : 1.748e-03
iter: 554, loss : 6.864e-04
iter: 555, loss : 5.343e-04
iter: 556, loss : 1.066e-03
iter: 557, loss : 3.384e-04
iter: 558, loss : 3.808e-04
iter: 559, loss : 1.501e-03
iter: 560, loss : 4.342e-03
iter: 561, loss : 2.347e-04
iter: 562, loss : 1.661e-03
iter: 563, loss : 2.432e-04
iter: 564, loss : 2.091e-03
iter: 565, loss : 5.593e-03
iter: 566, loss : 7.055e-04
iter: 567, loss : 5.491e-03
iter: 568, loss : 1.209e-03
iter: 569, loss : 3.200e-03
iter: 570, loss : 2.044e-03
iter: 571, loss : 4.301e-04
iter: 572, loss : 1.479e-03
iter: 573, loss : 9.599e-04
iter: 574, loss : 1.591e-03
iter: 575, loss : 4.122e-03
iter: 576, loss : 1.255e-03
iter: 577, loss : 5.243e-04
iter: 578, loss : 6.613e-03
iter: 579, loss : 5.670e-04
iter: 580, loss : 9.286e-04
iter: 581, loss : 1.083e-03
iter: 582, loss : 7.211e-04
iter: 583, loss : 3.484e-04
iter: 584, loss : 1.023e-03
iter: 585, loss : 2.785e-03
iter: 586, loss : 7.039e-04
iter: 587, loss : 5.570e-04
iter: 588, loss : 8.149e-04
iter: 589, loss : 2.843e-03
iter: 590, loss : 1.321e-03
iter: 591, loss : 5.329e-04
iter: 592, loss : 9.202e-04
iter: 593, loss : 3.881e-04
iter: 594, loss : 1.458e-03
iter: 595, loss : 1.053e-03
iter: 596, loss : 9.035e-03
iter: 597, loss : 5.058e-04
iter: 598, loss : 5.918e-04
iter: 599, loss : 2.978e-03
iter: 600, loss : 1.501e-03
iter: 601, loss : 8.087e-04
iter: 602, loss : 1.534e-03
iter: 603, loss : 4.022e-04
iter: 604, loss : 3.016e-03
iter: 605, loss : 4.061e-04
iter: 606, loss : 5.147e-04
iter: 607, loss : 7.079e-04
iter: 608, loss : 2.993e-04
iter: 609, loss : 7.613e-04
iter: 610, loss : 3.207e-04
iter: 611, loss : 4.391e-04
iter: 612, loss : 3.606e-04
iter: 613, loss : 5.470e-04
iter: 614, loss : 5.452e-04
iter: 615, loss : 5.304e-04
iter: 616, loss : 7.547e-04
iter: 617, loss : 4.251e-04
iter: 618, loss : 4.538e-04
iter: 619, loss : 1.565e-04
iter: 620, loss : 2.288e-04
iter: 621, loss : 3.761e-04
iter: 622, loss : 1.965e-04
iter: 623, loss : 1.844e-04
iter: 624, loss : 8.747e-04
iter: 625, loss : 2.156e-04
iter: 626, loss : 4.181e-04
iter: 627, loss : 4.500e-03
iter: 628, loss : 5.972e-03
iter: 629, loss : 3.507e-03
iter: 630, loss : 2.786e-04
iter: 631, loss : 5.552e-03
iter: 632, loss : 6.717e-03
iter: 633, loss : 1.735e-03
iter: 634, loss : 7.429e-04
iter: 635, loss : 5.029e-04
iter: 636, loss : 3.574e-04
iter: 637, loss : 6.866e-04
iter: 638, loss : 8.474e-03
iter: 639, loss : 1.341e-04
iter: 640, loss : 1.073e-03
iter: 641, loss : 5.689e-04
iter: 642, loss : 4.681e-04
iter: 643, loss : 7.338e-03
iter: 644, loss : 1.428e-03
iter: 645, loss : 6.415e-03
iter: 646, loss : 2.779e-04
iter: 647, loss : 5.277e-04
iter: 648, loss : 5.350e-04
iter: 649, loss : 5.491e-04
iter: 650, loss : 8.124e-04
iter: 651, loss : 7.659e-04
iter: 652, loss : 3.747e-04
iter: 653, loss : 4.764e-04
iter: 654, loss : 2.613e-04
iter: 655, loss : 3.476e-03
iter: 656, loss : 1.269e-03
iter: 657, loss : 1.751e-04
iter: 658, loss : 4.837e-04
iter: 659, loss : 4.013e-04
iter: 660, loss : 4.184e-04
iter: 661, loss : 6.493e-04
iter: 662, loss : 3.388e-04
iter: 663, loss : 2.525e-04
iter: 664, loss : 3.232e-03
iter: 665, loss : 6.443e-04
iter: 666, loss : 3.113e-04
iter: 667, loss : 2.594e-03
iter: 668, loss : 4.219e-04
iter: 669, loss : 3.575e-04
iter: 670, loss : 2.046e-03
iter: 671, loss : 6.318e-04
iter: 672, loss : 2.722e-04
iter: 673, loss : 2.433e-04
iter: 674, loss : 3.518e-04
iter: 675, loss : 1.994e-04
iter: 676, loss : 2.659e-04
iter: 677, loss : 5.277e-03
iter: 678, loss : 6.318e-04
iter: 679, loss : 1.762e-03
iter: 680, loss : 6.716e-03
iter: 681, loss : 5.807e-04
iter: 682, loss : 3.041e-04
iter: 683, loss : 2.398e-04
iter: 684, loss : 1.817e-03
iter: 685, loss : 2.326e-04
iter: 686, loss : 2.885e-04
iter: 687, loss : 2.303e-04
iter: 688, loss : 2.035e-04
iter: 689, loss : 4.366e-03
iter: 690, loss : 1.545e-03
iter: 691, loss : 2.954e-04
iter: 692, loss : 2.803e-04
iter: 693, loss : 2.994e-03
iter: 694, loss : 4.273e-04
iter: 695, loss : 6.345e-04
iter: 696, loss : 2.952e-04
iter: 697, loss : 7.134e-04
iter: 698, loss : 2.942e-04
iter: 699, loss : 2.827e-04
iter: 700, loss : 4.870e-04
iter: 701, loss : 2.585e-04
iter: 702, loss : 2.829e-04
iter: 703, loss : 3.148e-03
iter: 704, loss : 1.397e-03
iter: 705, loss : 3.718e-04
iter: 706, loss : 4.446e-04
iter: 707, loss : 3.362e-04
iter: 708, loss : 3.941e-04
iter: 709, loss : 2.607e-04
iter: 710, loss : 3.468e-04
iter: 711, loss : 2.961e-04
iter: 712, loss : 2.755e-04
iter: 713, loss : 1.545e-04
iter: 714, loss : 5.279e-05
iter: 715, loss : 2.547e-04
iter: 716, loss : 1.418e-03
iter: 717, loss : 6.935e-03
iter: 718, loss : 6.282e-04
iter: 719, loss : 1.449e-03
iter: 720, loss : 1.942e-04
iter: 721, loss : 1.712e-03
iter: 722, loss : 1.114e-04
iter: 723, loss : 8.597e-03
iter: 724, loss : 1.318e-03
iter: 725, loss : 1.551e-02
iter: 726, loss : 6.308e-04
iter: 727, loss : 3.204e-04
iter: 728, loss : 2.903e-04
iter: 729, loss : 1.972e-03
iter: 730, loss : 3.721e-04
iter: 731, loss : 5.366e-04
iter: 732, loss : 2.518e-04
iter: 733, loss : 1.002e-03
iter: 734, loss : 7.422e-04
iter: 735, loss : 3.100e-03
iter: 736, loss : 7.370e-03
iter: 737, loss : 5.534e-04
iter: 738, loss : 2.704e-04
iter: 739, loss : 4.577e-04
iter: 740, loss : 5.520e-04
iter: 741, loss : 9.594e-03
iter: 742, loss : 2.443e-03
iter: 743, loss : 6.069e-04
iter: 744, loss : 3.539e-03
iter: 745, loss : 8.106e-04
iter: 746, loss : 2.947e-04
iter: 747, loss : 4.025e-04
iter: 748, loss : 4.982e-04
iter: 749, loss : 1.478e-04
iter: 750, loss : 1.084e-03
iter: 751, loss : 4.058e-04
iter: 752, loss : 3.502e-04
iter: 753, loss : 3.509e-04
iter: 754, loss : 3.908e-04
iter: 755, loss : 1.447e-03
iter: 756, loss : 3.583e-04
iter: 757, loss : 3.577e-04
iter: 758, loss : 5.299e-04
iter: 759, loss : 1.193e-03
iter: 760, loss : 1.101e-03
iter: 761, loss : 9.417e-04
iter: 762, loss : 4.061e-04
iter: 763, loss : 5.070e-04
iter: 764, loss : 4.992e-04
iter: 765, loss : 2.354e-04
iter: 766, loss : 5.986e-03
iter: 767, loss : 3.485e-04
iter: 768, loss : 2.302e-03
iter: 769, loss : 4.681e-04
iter: 770, loss : 3.932e-04
iter: 771, loss : 1.894e-03
iter: 772, loss : 4.456e-04
iter: 773, loss : 1.458e-03
iter: 774, loss : 2.865e-03
iter: 775, loss : 3.749e-04
iter: 776, loss : 2.829e-04
iter: 777, loss : 4.447e-04
iter: 778, loss : 4.785e-04
iter: 779, loss : 4.479e-04
iter: 780, loss : 1.667e-04
iter: 781, loss : 2.218e-04
iter: 782, loss : 2.436e-04
iter: 783, loss : 1.877e-04
iter: 784, loss : 3.536e-04
iter: 785, loss : 6.055e-03
iter: 786, loss : 4.877e-04
iter: 787, loss : 1.412e-03
iter: 788, loss : 2.020e-04
iter: 789, loss : 3.532e-04
iter: 790, loss : 1.667e-03
iter: 791, loss : 1.204e-03
iter: 792, loss : 2.781e-04
iter: 793, loss : 4.007e-04
iter: 794, loss : 2.384e-04
iter: 795, loss : 6.376e-04
iter: 796, loss : 2.910e-04
iter: 797, loss : 1.102e-03
iter: 798, loss : 1.620e-04
iter: 799, loss : 3.914e-03
iter: 800, loss : 1.428e-03
iter: 801, loss : 2.250e-04
iter: 802, loss : 9.120e-04
iter: 803, loss : 2.327e-04
iter: 804, loss : 1.985e-04
iter: 805, loss : 4.760e-04
iter: 806, loss : 5.600e-04
iter: 807, loss : 2.189e-03
iter: 808, loss : 1.245e-03
iter: 809, loss : 3.678e-04
iter: 810, loss : 4.418e-04
iter: 811, loss : 2.600e-04
iter: 812, loss : 3.500e-04
iter: 813, loss : 4.190e-04
iter: 814, loss : 1.113e-03
iter: 815, loss : 2.681e-04
iter: 816, loss : 3.449e-04
iter: 817, loss : 2.443e-04
iter: 818, loss : 5.877e-04
iter: 819, loss : 1.033e-03
iter: 820, loss : 4.051e-04
iter: 821, loss : 7.733e-05
iter: 822, loss : 1.394e-04
iter: 823, loss : 4.335e-04
iter: 824, loss : 1.614e-04
iter: 825, loss : 1.360e-03
iter: 826, loss : 2.220e-04
iter: 827, loss : 3.500e-03
iter: 828, loss : 3.706e-04
iter: 829, loss : 2.534e-04
iter: 830, loss : 1.760e-03
iter: 831, loss : 2.832e-04
iter: 832, loss : 1.088e-04
iter: 833, loss : 2.446e-04
iter: 834, loss : 2.818e-04
iter: 835, loss : 3.886e-03
iter: 836, loss : 3.268e-04
iter: 837, loss : 1.033e-04
iter: 838, loss : 2.947e-04
iter: 839, loss : 4.257e-04
iter: 840, loss : 1.234e-03
iter: 841, loss : 2.860e-04
iter: 842, loss : 3.531e-04
iter: 843, loss : 9.144e-05
iter: 844, loss : 4.088e-04
iter: 845, loss : 1.777e-04
iter: 846, loss : 1.258e-04
iter: 847, loss : 3.523e-03
iter: 848, loss : 1.003e-03
iter: 849, loss : 2.432e-04
iter: 850, loss : 1.681e-04
iter: 851, loss : 8.341e-04
iter: 852, loss : 3.136e-04
iter: 853, loss : 4.147e-04
iter: 854, loss : 3.095e-04
iter: 855, loss : 1.685e-03
iter: 856, loss : 6.183e-05
iter: 857, loss : 1.950e-04
iter: 858, loss : 1.144e-04
iter: 859, loss : 1.732e-04
iter: 860, loss : 4.017e-04
iter: 861, loss : 1.149e-04
iter: 862, loss : 3.002e-03
iter: 863, loss : 1.215e-04
iter: 864, loss : 1.882e-04
iter: 865, loss : 6.006e-03
iter: 866, loss : 7.555e-04
iter: 867, loss : 3.707e-04
iter: 868, loss : 4.461e-04
iter: 869, loss : 2.043e-04
iter: 870, loss : 2.828e-04
iter: 871, loss : 1.376e-04
iter: 872, loss : 3.609e-03
iter: 873, loss : 2.103e-03
iter: 874, loss : 2.513e-04
iter: 875, loss : 6.900e-04
iter: 876, loss : 3.345e-04
iter: 877, loss : 2.813e-04
iter: 878, loss : 1.187e-03
iter: 879, loss : 1.791e-03
iter: 880, loss : 1.225e-03
iter: 881, loss : 5.373e-04
iter: 882, loss : 3.202e-03
iter: 883, loss : 1.355e-04
iter: 884, loss : 1.324e-03
iter: 885, loss : 2.571e-04
iter: 886, loss : 6.395e-03
iter: 887, loss : 3.460e-04
iter: 888, loss : 1.155e-03
iter: 889, loss : 1.910e-03
iter: 890, loss : 7.525e-03
iter: 891, loss : 6.255e-03
iter: 892, loss : 2.540e-03
iter: 893, loss : 3.831e-04
iter: 894, loss : 7.464e-04
iter: 895, loss : 3.747e-04
iter: 896, loss : 1.323e-03
iter: 897, loss : 7.486e-04
iter: 898, loss : 7.069e-04
iter: 899, loss : 4.624e-04
iter: 900, loss : 2.989e-04
iter: 901, loss : 1.654e-04
iter: 902, loss : 5.440e-04
iter: 903, loss : 1.063e-03
iter: 904, loss : 2.719e-04
iter: 905, loss : 1.951e-03
iter: 906, loss : 4.564e-04
iter: 907, loss : 1.433e-04
iter: 908, loss : 7.920e-04
iter: 909, loss : 5.209e-04
iter: 910, loss : 1.071e-03
iter: 911, loss : 2.449e-03
iter: 912, loss : 3.545e-04
iter: 913, loss : 9.221e-04
iter: 914, loss : 1.662e-04
iter: 915, loss : 3.122e-04
iter: 916, loss : 5.055e-03
iter: 917, loss : 3.407e-04
iter: 918, loss : 9.955e-04
iter: 919, loss : 1.193e-03
iter: 920, loss : 5.280e-04
iter: 921, loss : 1.036e-03
iter: 922, loss : 5.724e-04
iter: 923, loss : 3.003e-04
iter: 924, loss : 2.681e-04
iter: 925, loss : 3.990e-03
iter: 926, loss : 5.385e-04
iter: 927, loss : 2.897e-03
iter: 928, loss : 9.012e-04
iter: 929, loss : 6.989e-04
iter: 930, loss : 4.000e-03
iter: 931, loss : 3.738e-04
iter: 932, loss : 4.667e-03
iter: 933, loss : 1.406e-03
iter: 934, loss : 1.163e-03
iter: 935, loss : 2.901e-04
iter: 936, loss : 3.599e-04
iter: 937, loss : 1.560e-03
iter: 938, loss : 5.116e-04
iter: 939, loss : 1.966e-03
iter: 940, loss : 5.863e-04
iter: 941, loss : 2.656e-03
iter: 942, loss : 7.088e-04
iter: 943, loss : 1.668e-03
iter: 944, loss : 4.427e-04
iter: 945, loss : 2.725e-04
iter: 946, loss : 2.537e-03
iter: 947, loss : 1.099e-03
iter: 948, loss : 4.643e-03
iter: 949, loss : 2.482e-04
iter: 950, loss : 3.255e-04
iter: 951, loss : 9.218e-04
iter: 952, loss : 5.172e-04
iter: 953, loss : 1.253e-03
iter: 954, loss : 3.627e-04
iter: 955, loss : 1.032e-03
iter: 956, loss : 1.262e-03
iter: 957, loss : 3.779e-04
iter: 958, loss : 9.648e-04
iter: 959, loss : 4.888e-03
iter: 960, loss : 1.541e-03
iter: 961, loss : 1.568e-03
iter: 962, loss : 4.403e-04
iter: 963, loss : 1.341e-03
iter: 964, loss : 1.564e-03
iter: 965, loss : 4.268e-04
iter: 966, loss : 3.076e-04
iter: 967, loss : 4.382e-04
iter: 968, loss : 3.647e-03
iter: 969, loss : 1.128e-03
iter: 970, loss : 7.030e-04
iter: 971, loss : 1.551e-04
iter: 972, loss : 4.941e-04
iter: 973, loss : 2.814e-04
iter: 974, loss : 6.298e-03
iter: 975, loss : 4.526e-03
iter: 976, loss : 1.301e-03
iter: 977, loss : 2.610e-04
iter: 978, loss : 5.362e-04
iter: 979, loss : 5.783e-03
iter: 980, loss : 6.812e-04
iter: 981, loss : 8.511e-04
iter: 982, loss : 4.176e-04
iter: 983, loss : 5.306e-04
iter: 984, loss : 1.150e-03
iter: 985, loss : 3.240e-04
iter: 986, loss : 6.160e-04
iter: 987, loss : 2.498e-04
iter: 988, loss : 3.301e-04
iter: 989, loss : 3.896e-04
iter: 990, loss : 1.134e-03
iter: 991, loss : 1.965e-04
iter: 992, loss : 3.744e-03
iter: 993, loss : 3.274e-04
iter: 994, loss : 2.724e-03
iter: 995, loss : 1.886e-04
iter: 996, loss : 4.269e-04
iter: 997, loss : 2.435e-04
iter: 998, loss : 3.553e-03
iter: 999, loss : 3.281e-04
iter: 1000, loss : 3.215e-03
iter: 1001, loss : 7.229e-04
iter: 1002, loss : 3.980e-04
iter: 1003, loss : 2.367e-04
iter: 1004, loss : 4.427e-04
iter: 1005, loss : 2.402e-03
iter: 1006, loss : 2.840e-04
iter: 1007, loss : 5.793e-03
iter: 1008, loss : 2.948e-04
iter: 1009, loss : 3.525e-04
iter: 1010, loss : 5.634e-04
iter: 1011, loss : 2.588e-03
iter: 1012, loss : 2.792e-04
iter: 1013, loss : 1.428e-04
iter: 1014, loss : 4.381e-03
iter: 1015, loss : 1.265e-03
iter: 1016, loss : 1.059e-03
iter: 1017, loss : 4.550e-04
iter: 1018, loss : 2.390e-04
iter: 1019, loss : 1.979e-03
iter: 1020, loss : 3.532e-04
iter: 1021, loss : 3.474e-03
iter: 1022, loss : 4.537e-04
iter: 1023, loss : 1.992e-04
iter: 1024, loss : 2.918e-04
iter: 1025, loss : 1.018e-03
iter: 1026, loss : 4.274e-04
iter: 1027, loss : 1.595e-04
iter: 1028, loss : 1.983e-04
iter: 1029, loss : 3.874e-04
iter: 1030, loss : 9.176e-04
iter: 1031, loss : 2.899e-04
iter: 1032, loss : 3.894e-04
iter: 1033, loss : 4.379e-04
iter: 1034, loss : 2.844e-04
iter: 1035, loss : 1.409e-04
iter: 1036, loss : 1.411e-03
iter: 1037, loss : 3.444e-03
iter: 1038, loss : 1.956e-03
iter: 1039, loss : 1.205e-03
iter: 1040, loss : 9.893e-04
iter: 1041, loss : 9.330e-04
iter: 1042, loss : 1.797e-04
iter: 1043, loss : 5.369e-03
iter: 1044, loss : 6.727e-04
iter: 1045, loss : 1.230e-04
iter: 1046, loss : 5.756e-04
iter: 1047, loss : 2.346e-04
iter: 1048, loss : 3.353e-04
iter: 1049, loss : 5.566e-04
iter: 1050, loss : 3.286e-04
iter: 1051, loss : 2.945e-04
iter: 1052, loss : 2.727e-04
iter: 1053, loss : 8.140e-04
iter: 1054, loss : 1.020e-03
iter: 1055, loss : 1.453e-04
iter: 1056, loss : 4.197e-04
iter: 1057, loss : 2.466e-04
iter: 1058, loss : 7.022e-04
iter: 1059, loss : 2.763e-04
iter: 1060, loss : 2.423e-04
iter: 1061, loss : 1.507e-04
iter: 1062, loss : 1.409e-03
iter: 1063, loss : 3.207e-04
iter: 1064, loss : 2.768e-03
iter: 1065, loss : 5.957e-03
iter: 1066, loss : 2.930e-03
iter: 1067, loss : 1.096e-03
iter: 1068, loss : 7.779e-04
iter: 1069, loss : 3.219e-04
iter: 1070, loss : 2.941e-04
iter: 1071, loss : 2.596e-04
iter: 1072, loss : 1.683e-04
iter: 1073, loss : 2.251e-03
iter: 1074, loss : 7.445e-04
iter: 1075, loss : 3.170e-04
iter: 1076, loss : 5.027e-04
iter: 1077, loss : 3.679e-04
iter: 1078, loss : 5.630e-04
iter: 1079, loss : 8.990e-05
iter: 1080, loss : 7.638e-05
iter: 1081, loss : 2.387e-04
iter: 1082, loss : 1.396e-04
iter: 1083, loss : 1.138e-04
iter: 1084, loss : 3.767e-04
iter: 1085, loss : 1.137e-04
iter: 1086, loss : 1.679e-04
iter: 1087, loss : 1.757e-03
iter: 1088, loss : 1.941e-04
iter: 1089, loss : 3.136e-04
iter: 1090, loss : 1.799e-03
iter: 1091, loss : 1.266e-03
iter: 1092, loss : 1.209e-04
iter: 1093, loss : 2.415e-04
iter: 1094, loss : 1.656e-04
iter: 1095, loss : 1.263e-03
iter: 1096, loss : 8.587e-04
iter: 1097, loss : 2.503e-04
iter: 1098, loss : 1.467e-04
iter: 1099, loss : 1.512e-04
iter: 1100, loss : 2.050e-03
iter: 1101, loss : 3.822e-04
iter: 1102, loss : 3.028e-04
iter: 1103, loss : 4.105e-04
iter: 1104, loss : 3.278e-04
iter: 1105, loss : 1.152e-04
iter: 1106, loss : 3.704e-04
iter: 1107, loss : 1.830e-03
iter: 1108, loss : 1.222e-04
iter: 1109, loss : 6.407e-04
iter: 1110, loss : 2.994e-04
iter: 1111, loss : 1.507e-04
iter: 1112, loss : 3.960e-04
iter: 1113, loss : 2.136e-04
iter: 1114, loss : 1.692e-03
iter: 1115, loss : 1.285e-03
iter: 1116, loss : 3.693e-04
iter: 1117, loss : 1.002e-04
iter: 1118, loss : 1.278e-04
iter: 1119, loss : 2.307e-04
iter: 1120, loss : 1.410e-04
iter: 1121, loss : 2.846e-04
iter: 1122, loss : 1.865e-04
iter: 1123, loss : 9.062e-04
iter: 1124, loss : 3.902e-03
iter: 1125, loss : 1.922e-04
iter: 1126, loss : 3.621e-03
iter: 1127, loss : 3.373e-04
iter: 1128, loss : 9.583e-04
iter: 1129, loss : 2.787e-04
iter: 1130, loss : 8.962e-04
iter: 1131, loss : 2.105e-04
iter: 1132, loss : 8.435e-04
iter: 1133, loss : 1.118e-04
iter: 1134, loss : 2.630e-04
iter: 1135, loss : 2.544e-04
iter: 1136, loss : 1.096e-04
iter: 1137, loss : 1.773e-04
iter: 1138, loss : 1.386e-04
iter: 1139, loss : 2.144e-03
iter: 1140, loss : 1.352e-04
iter: 1141, loss : 9.678e-05
iter: 1142, loss : 2.887e-04
iter: 1143, loss : 1.998e-04
iter: 1144, loss : 2.189e-04
iter: 1145, loss : 1.116e-04
iter: 1146, loss : 9.492e-05
iter: 1147, loss : 3.731e-04
iter: 1148, loss : 9.696e-04
iter: 1149, loss : 1.003e-03
iter: 1150, loss : 7.097e-04
iter: 1151, loss : 2.373e-04
iter: 1152, loss : 2.459e-04
iter: 1153, loss : 1.575e-04
iter: 1154, loss : 1.102e-03
iter: 1155, loss : 1.273e-04
iter: 1156, loss : 2.230e-04
iter: 1157, loss : 2.095e-04
iter: 1158, loss : 2.230e-04
iter: 1159, loss : 2.126e-03
iter: 1160, loss : 3.321e-03
iter: 1161, loss : 3.539e-04
iter: 1162, loss : 3.153e-03
iter: 1163, loss : 5.541e-04
iter: 1164, loss : 1.786e-04
iter: 1165, loss : 4.195e-04
iter: 1166, loss : 9.730e-04
iter: 1167, loss : 1.202e-03
iter: 1168, loss : 4.179e-04
iter: 1169, loss : 1.021e-04
iter: 1170, loss : 2.383e-03
iter: 1171, loss : 1.269e-04
iter: 1172, loss : 5.858e-04
iter: 1173, loss : 3.498e-04
iter: 1174, loss : 3.969e-04
iter: 1175, loss : 9.527e-04
iter: 1176, loss : 1.922e-03
iter: 1177, loss : 3.151e-04
iter: 1178, loss : 2.270e-04
iter: 1179, loss : 2.523e-03
iter: 1180, loss : 3.033e-04
iter: 1181, loss : 1.902e-04
iter: 1182, loss : 9.929e-04
iter: 1183, loss : 8.550e-05
iter: 1184, loss : 1.984e-04
iter: 1185, loss : 7.359e-04
iter: 1186, loss : 1.021e-03
iter: 1187, loss : 1.224e-03
iter: 1188, loss : 5.466e-03
iter: 1189, loss : 4.152e-04
iter: 1190, loss : 4.065e-04
iter: 1191, loss : 3.215e-04
iter: 1192, loss : 5.540e-04
iter: 1193, loss : 5.951e-03
iter: 1194, loss : 1.843e-03
iter: 1195, loss : 5.369e-04
iter: 1196, loss : 4.343e-04
iter: 1197, loss : 3.833e-04
iter: 1198, loss : 3.295e-04
iter: 1199, loss : 1.325e-03
iter: 1200, loss : 2.974e-04
iter: 1201, loss : 2.526e-04
iter: 1202, loss : 1.557e-03
iter: 1203, loss : 1.816e-04
iter: 1204, loss : 1.005e-03
iter: 1205, loss : 3.224e-04
iter: 1206, loss : 3.255e-04
iter: 1207, loss : 5.388e-04
iter: 1208, loss : 3.103e-04
iter: 1209, loss : 1.203e-04
iter: 1210, loss : 8.934e-04
iter: 1211, loss : 3.459e-04
iter: 1212, loss : 2.267e-04
iter: 1213, loss : 1.393e-03
iter: 1214, loss : 1.347e-03
iter: 1215, loss : 2.567e-04
iter: 1216, loss : 1.677e-04
iter: 1217, loss : 1.182e-04
iter: 1218, loss : 1.280e-03
iter: 1219, loss : 4.281e-04
iter: 1220, loss : 2.327e-04
iter: 1221, loss : 8.542e-04
iter: 1222, loss : 2.762e-04
iter: 1223, loss : 1.075e-03
iter: 1224, loss : 5.029e-04
iter: 1225, loss : 2.775e-04
iter: 1226, loss : 5.343e-04
iter: 1227, loss : 2.600e-04
iter: 1228, loss : 4.213e-04
iter: 1229, loss : 1.486e-04
iter: 1230, loss : 1.138e-04
iter: 1231, loss : 4.881e-03
iter: 1232, loss : 3.140e-04
iter: 1233, loss : 1.853e-04
iter: 1234, loss : 2.326e-04
iter: 1235, loss : 1.739e-03
iter: 1236, loss : 1.819e-04
iter: 1237, loss : 1.398e-04
iter: 1238, loss : 1.618e-04
iter: 1239, loss : 1.482e-03
iter: 1240, loss : 5.669e-04
iter: 1241, loss : 1.830e-04
iter: 1242, loss : 2.122e-04
iter: 1243, loss : 6.299e-03
iter: 1244, loss : 1.917e-03
iter: 1245, loss : 3.507e-04
iter: 1246, loss : 4.172e-04
iter: 1247, loss : 5.640e-04
iter: 1248, loss : 2.948e-03
iter: 1249, loss : 4.095e-04
iter: 1250, loss : 8.823e-04
iter: 1251, loss : 1.126e-03
iter: 1252, loss : 2.834e-04
iter: 1253, loss : 2.394e-04
iter: 1254, loss : 7.692e-04
iter: 1255, loss : 2.816e-04
iter: 1256, loss : 2.576e-04
iter: 1257, loss : 1.819e-04
iter: 1258, loss : 2.358e-03
iter: 1259, loss : 1.843e-04
iter: 1260, loss : 6.038e-04
iter: 1261, loss : 2.203e-04
iter: 1262, loss : 2.503e-04
iter: 1263, loss : 2.176e-04
iter: 1264, loss : 1.602e-04
iter: 1265, loss : 3.370e-04
iter: 1266, loss : 1.043e-04
iter: 1267, loss : 1.236e-03
iter: 1268, loss : 2.665e-04
iter: 1269, loss : 9.556e-05
iter: 1270, loss : 1.890e-04
iter: 1271, loss : 1.018e-04
iter: 1272, loss : 1.374e-03
iter: 1273, loss : 1.193e-04
iter: 1274, loss : 8.229e-04
iter: 1275, loss : 1.475e-04
iter: 1276, loss : 2.182e-03
iter: 1277, loss : 9.421e-04
iter: 1278, loss : 8.849e-04
iter: 1279, loss : 8.420e-04
iter: 1280, loss : 1.889e-04
iter: 1281, loss : 8.410e-04
iter: 1282, loss : 2.126e-04
iter: 1283, loss : 2.367e-04
iter: 1284, loss : 1.873e-04
iter: 1285, loss : 5.130e-04
iter: 1286, loss : 9.168e-05
iter: 1287, loss : 2.421e-04
iter: 1288, loss : 1.079e-04
iter: 1289, loss : 4.309e-03
iter: 1290, loss : 7.513e-04
iter: 1291, loss : 7.192e-04
iter: 1292, loss : 3.349e-04
iter: 1293, loss : 1.443e-04
iter: 1294, loss : 9.020e-04
iter: 1295, loss : 1.443e-04
iter: 1296, loss : 7.633e-04
iter: 1297, loss : 4.164e-03
iter: 1298, loss : 6.576e-04
iter: 1299, loss : 2.614e-04
iter: 1300, loss : 2.067e-04
iter: 1301, loss : 7.160e-04
iter: 1302, loss : 3.917e-04
iter: 1303, loss : 3.192e-04
iter: 1304, loss : 1.068e-03
iter: 1305, loss : 2.457e-04
iter: 1306, loss : 2.279e-04
iter: 1307, loss : 3.228e-04
iter: 1308, loss : 1.827e-04
iter: 1309, loss : 7.120e-04
iter: 1310, loss : 3.174e-04
iter: 1311, loss : 2.020e-04
iter: 1312, loss : 8.126e-04
iter: 1313, loss : 1.603e-04
iter: 1314, loss : 1.264e-04
iter: 1315, loss : 7.073e-04
iter: 1316, loss : 1.744e-04
iter: 1317, loss : 3.051e-04
iter: 1318, loss : 6.312e-05
iter: 1319, loss : 1.845e-04
iter: 1320, loss : 1.818e-03
iter: 1321, loss : 2.014e-04
iter: 1322, loss : 1.970e-04
iter: 1323, loss : 9.630e-04
iter: 1324, loss : 3.377e-03
iter: 1325, loss : 4.627e-04
iter: 1326, loss : 3.654e-04
iter: 1327, loss : 7.259e-05
iter: 1328, loss : 3.778e-04
iter: 1329, loss : 2.085e-04
iter: 1330, loss : 1.072e-04
iter: 1331, loss : 1.759e-04
iter: 1332, loss : 1.853e-04
iter: 1333, loss : 6.612e-04
iter: 1334, loss : 5.772e-04
iter: 1335, loss : 2.388e-04
iter: 1336, loss : 1.064e-03
iter: 1337, loss : 3.877e-03
iter: 1338, loss : 5.417e-04
iter: 1339, loss : 3.604e-03
iter: 1340, loss : 2.306e-04
iter: 1341, loss : 4.757e-04
iter: 1342, loss : 2.811e-03
iter: 1343, loss : 4.839e-04
iter: 1344, loss : 8.235e-04
iter: 1345, loss : 1.322e-04
iter: 1346, loss : 5.531e-04
iter: 1347, loss : 3.609e-04
iter: 1348, loss : 3.836e-04
iter: 1349, loss : 4.135e-04
iter: 1350, loss : 3.144e-04
iter: 1351, loss : 1.106e-03
iter: 1352, loss : 1.377e-03
iter: 1353, loss : 5.297e-04
iter: 1354, loss : 2.075e-03
iter: 1355, loss : 5.786e-04
iter: 1356, loss : 3.492e-04
iter: 1357, loss : 4.186e-04
iter: 1358, loss : 2.077e-04
iter: 1359, loss : 3.981e-04
iter: 1360, loss : 5.033e-04
iter: 1361, loss : 4.989e-04
iter: 1362, loss : 1.115e-03
iter: 1363, loss : 6.645e-04
iter: 1364, loss : 2.827e-04
iter: 1365, loss : 3.611e-03
iter: 1366, loss : 7.383e-04
iter: 1367, loss : 2.001e-04
iter: 1368, loss : 1.816e-03
iter: 1369, loss : 4.460e-04
iter: 1370, loss : 6.692e-04
iter: 1371, loss : 3.085e-04
iter: 1372, loss : 9.584e-04
iter: 1373, loss : 3.750e-04
iter: 1374, loss : 2.360e-04
iter: 1375, loss : 1.463e-03
iter: 1376, loss : 1.564e-04
iter: 1377, loss : 7.335e-05
iter: 1378, loss : 3.826e-04
iter: 1379, loss : 7.617e-04
iter: 1380, loss : 9.738e-04
iter: 1381, loss : 7.007e-04
iter: 1382, loss : 1.234e-04
iter: 1383, loss : 1.635e-03
iter: 1384, loss : 8.958e-04
iter: 1385, loss : 2.300e-04
iter: 1386, loss : 6.826e-04
iter: 1387, loss : 4.678e-04
iter: 1388, loss : 7.050e-04
iter: 1389, loss : 3.289e-04
iter: 1390, loss : 4.013e-04
iter: 1391, loss : 8.050e-05
iter: 1392, loss : 2.832e-04
iter: 1393, loss : 4.157e-04
iter: 1394, loss : 2.164e-04
iter: 1395, loss : 1.803e-04
iter: 1396, loss : 1.814e-04
iter: 1397, loss : 4.137e-03
iter: 1398, loss : 2.019e-04
iter: 1399, loss : 3.314e-04
iter: 1400, loss : 1.481e-04
iter: 1401, loss : 2.015e-03
iter: 1402, loss : 6.765e-04
iter: 1403, loss : 2.856e-03
iter: 1404, loss : 8.441e-04
iter: 1405, loss : 1.693e-04
iter: 1406, loss : 1.936e-04
iter: 1407, loss : 8.608e-04
iter: 1408, loss : 4.276e-04
iter: 1409, loss : 2.368e-04
iter: 1410, loss : 2.836e-04
iter: 1411, loss : 1.219e-03
iter: 1412, loss : 1.570e-03
iter: 1413, loss : 1.314e-03
iter: 1414, loss : 6.493e-04
iter: 1415, loss : 1.698e-04
iter: 1416, loss : 1.839e-04
iter: 1417, loss : 7.387e-04
iter: 1418, loss : 1.480e-04
iter: 1419, loss : 5.949e-04
iter: 1420, loss : 3.318e-04
iter: 1421, loss : 1.851e-04
iter: 1422, loss : 2.257e-04
iter: 1423, loss : 1.223e-04
iter: 1424, loss : 1.420e-04
iter: 1425, loss : 9.054e-05
iter: 1426, loss : 9.926e-04
iter: 1427, loss : 2.533e-04
iter: 1428, loss : 1.653e-04
iter: 1429, loss : 1.019e-04
iter: 1430, loss : 1.271e-04
iter: 1431, loss : 1.485e-04
iter: 1432, loss : 1.043e-04
iter: 1433, loss : 1.543e-04
iter: 1434, loss : 8.030e-05
iter: 1435, loss : 7.119e-05
iter: 1436, loss : 6.980e-05
iter: 1437, loss : 1.752e-04
iter: 1438, loss : 5.210e-04
iter: 1439, loss : 1.375e-04
iter: 1440, loss : 1.245e-03
iter: 1441, loss : 1.668e-04
iter: 1442, loss : 7.853e-05
iter: 1443, loss : 5.238e-03
iter: 1444, loss : 3.660e-04
iter: 1445, loss : 3.469e-04
iter: 1446, loss : 2.532e-04
iter: 1447, loss : 4.907e-04
iter: 1448, loss : 4.494e-05
iter: 1449, loss : 7.062e-05
iter: 1450, loss : 2.244e-03
iter: 1451, loss : 7.285e-04
iter: 1452, loss : 2.868e-04
iter: 1453, loss : 6.028e-04
iter: 1454, loss : 1.987e-04
iter: 1455, loss : 7.068e-04
iter: 1456, loss : 1.006e-03
iter: 1457, loss : 2.579e-04
iter: 1458, loss : 1.886e-03
iter: 1459, loss : 3.331e-03
iter: 1460, loss : 5.713e-04
iter: 1461, loss : 4.774e-04
iter: 1462, loss : 2.655e-04
iter: 1463, loss : 2.101e-04
iter: 1464, loss : 1.390e-03
iter: 1465, loss : 3.435e-04
iter: 1466, loss : 1.293e-04
iter: 1467, loss : 1.838e-04
iter: 1468, loss : 1.613e-04
iter: 1469, loss : 1.855e-04
iter: 1470, loss : 1.943e-04
iter: 1471, loss : 2.062e-04
iter: 1472, loss : 1.659e-03
iter: 1473, loss : 9.196e-04
iter: 1474, loss : 2.570e-03
iter: 1475, loss : 3.650e-04
iter: 1476, loss : 7.551e-04
iter: 1477, loss : 3.202e-04
iter: 1478, loss : 1.107e-03
iter: 1479, loss : 1.887e-03
iter: 1480, loss : 5.298e-04
iter: 1481, loss : 3.147e-04
iter: 1482, loss : 2.647e-04
iter: 1483, loss : 1.383e-04
iter: 1484, loss : 2.151e-03
iter: 1485, loss : 1.596e-04
iter: 1486, loss : 2.819e-04
iter: 1487, loss : 8.060e-03
iter: 1488, loss : 7.831e-04
iter: 1489, loss : 1.124e-03
iter: 1490, loss : 2.978e-04
iter: 1491, loss : 1.860e-04
iter: 1492, loss : 1.150e-03
iter: 1493, loss : 5.744e-04
iter: 1494, loss : 1.917e-04
iter: 1495, loss : 9.892e-04
iter: 1496, loss : 4.995e-04
iter: 1497, loss : 1.375e-04
iter: 1498, loss : 1.801e-04
iter: 1499, loss : 7.043e-04
iter: 1500, loss : 9.586e-04
iter: 1501, loss : 3.855e-04
iter: 1502, loss : 4.113e-04
iter: 1503, loss : 3.437e-04
iter: 1504, loss : 1.147e-03
iter: 1505, loss : 2.535e-03
iter: 1506, loss : 2.707e-04
iter: 1507, loss : 2.480e-04
iter: 1508, loss : 6.627e-04
iter: 1509, loss : 3.291e-04
iter: 1510, loss : 3.669e-04
iter: 1511, loss : 2.714e-04
iter: 1512, loss : 3.477e-04
iter: 1513, loss : 7.477e-04
iter: 1514, loss : 2.136e-04
iter: 1515, loss : 2.018e-04
iter: 1516, loss : 2.034e-04
iter: 1517, loss : 2.733e-04
iter: 1518, loss : 1.739e-03
iter: 1519, loss : 1.624e-04
iter: 1520, loss : 2.538e-04
iter: 1521, loss : 4.017e-04
iter: 1522, loss : 2.249e-04
iter: 1523, loss : 1.360e-04
iter: 1524, loss : 9.278e-04
iter: 1525, loss : 3.386e-04
iter: 1526, loss : 2.374e-04
iter: 1527, loss : 7.280e-04
iter: 1528, loss : 1.190e-04
iter: 1529, loss : 2.715e-04
iter: 1530, loss : 2.261e-04
iter: 1531, loss : 1.993e-04
iter: 1532, loss : 3.670e-04
iter: 1533, loss : 2.170e-04
iter: 1534, loss : 9.839e-04
iter: 1535, loss : 1.635e-04
iter: 1536, loss : 6.624e-05
iter: 1537, loss : 2.224e-04
iter: 1538, loss : 1.708e-04
iter: 1539, loss : 1.241e-04
iter: 1540, loss : 6.691e-04
iter: 1541, loss : 2.047e-04
iter: 1542, loss : 5.263e-05
iter: 1543, loss : 7.136e-04
iter: 1544, loss : 2.419e-04
iter: 1545, loss : 1.057e-04
iter: 1546, loss : 1.204e-03
iter: 1547, loss : 7.679e-04
iter: 1548, loss : 3.072e-04
iter: 1549, loss : 1.482e-04
iter: 1550, loss : 1.078e-04
iter: 1551, loss : 2.509e-04
iter: 1552, loss : 5.759e-03
iter: 1553, loss : 2.633e-04
iter: 1554, loss : 1.776e-04
iter: 1555, loss : 1.987e-03
iter: 1556, loss : 1.267e-03
iter: 1557, loss : 2.275e-04
iter: 1558, loss : 4.438e-03
iter: 1559, loss : 3.779e-04
iter: 1560, loss : 2.830e-04
iter: 1561, loss : 2.139e-04
iter: 1562, loss : 1.064e-04
iter: 1563, loss : 6.589e-04
iter: 1564, loss : 2.663e-04
iter: 1565, loss : 5.838e-04
iter: 1566, loss : 6.590e-05
iter: 1567, loss : 1.771e-03
iter: 1568, loss : 4.380e-04
iter: 1569, loss : 1.467e-04
iter: 1570, loss : 1.585e-04
iter: 1571, loss : 7.776e-04
iter: 1572, loss : 2.724e-04
iter: 1573, loss : 1.351e-04
iter: 1574, loss : 9.330e-04
iter: 1575, loss : 2.575e-04
iter: 1576, loss : 1.600e-04
iter: 1577, loss : 8.204e-05
iter: 1578, loss : 1.339e-04
iter: 1579, loss : 1.001e-04
iter: 1580, loss : 1.556e-04
iter: 1581, loss : 7.390e-04
iter: 1582, loss : 1.360e-03
iter: 1583, loss : 1.351e-03
iter: 1584, loss : 9.426e-05
iter: 1585, loss : 9.947e-04
iter: 1586, loss : 1.089e-03
iter: 1587, loss : 7.350e-04
iter: 1588, loss : 1.273e-04
iter: 1589, loss : 6.108e-04
iter: 1590, loss : 1.473e-04
iter: 1591, loss : 5.342e-04
iter: 1592, loss : 5.793e-04
iter: 1593, loss : 4.315e-04
iter: 1594, loss : 8.508e-05
iter: 1595, loss : 7.337e-04
iter: 1596, loss : 1.503e-03
iter: 1597, loss : 9.049e-04
iter: 1598, loss : 3.694e-04
iter: 1599, loss : 1.988e-04
iter: 1600, loss : 1.814e-04
iter: 1601, loss : 1.983e-04
iter: 1602, loss : 2.507e-04
iter: 1603, loss : 3.100e-04
iter: 1604, loss : 5.926e-04
iter: 1605, loss : 1.803e-04
iter: 1606, loss : 3.152e-04
iter: 1607, loss : 1.393e-04
iter: 1608, loss : 2.048e-04
iter: 1609, loss : 1.300e-04
iter: 1610, loss : 1.631e-04
iter: 1611, loss : 5.752e-04
iter: 1612, loss : 8.574e-04
iter: 1613, loss : 4.137e-04
iter: 1614, loss : 3.229e-04
iter: 1615, loss : 3.225e-04
iter: 1616, loss : 1.258e-04
iter: 1617, loss : 1.447e-04
iter: 1618, loss : 2.137e-04
iter: 1619, loss : 1.324e-04
iter: 1620, loss : 1.327e-04
iter: 1621, loss : 7.361e-05
iter: 1622, loss : 6.287e-05
iter: 1623, loss : 6.843e-04
iter: 1624, loss : 7.300e-05
iter: 1625, loss : 1.240e-04
iter: 1626, loss : 1.324e-04
iter: 1627, loss : 6.031e-04
iter: 1628, loss : 1.040e-03
iter: 1629, loss : 9.291e-05
iter: 1630, loss : 1.706e-04
iter: 1631, loss : 1.441e-04
iter: 1632, loss : 6.685e-03
iter: 1633, loss : 4.665e-04
iter: 1634, loss : 3.018e-04
iter: 1635, loss : 5.701e-04
iter: 1636, loss : 2.784e-04
iter: 1637, loss : 1.691e-04
iter: 1638, loss : 2.643e-04
iter: 1639, loss : 5.174e-04
iter: 1640, loss : 3.255e-04
iter: 1641, loss : 8.458e-05
iter: 1642, loss : 3.254e-05
iter: 1643, loss : 7.605e-04
iter: 1644, loss : 1.446e-04
iter: 1645, loss : 2.711e-04
iter: 1646, loss : 1.643e-04
iter: 1647, loss : 1.967e-04
iter: 1648, loss : 2.468e-04
iter: 1649, loss : 8.544e-05
iter: 1650, loss : 5.851e-05
iter: 1651, loss : 2.090e-04
iter: 1652, loss : 3.328e-04
iter: 1653, loss : 6.182e-04
iter: 1654, loss : 1.396e-04
iter: 1655, loss : 2.290e-04
iter: 1656, loss : 1.507e-04
iter: 1657, loss : 2.803e-03
iter: 1658, loss : 1.590e-04
iter: 1659, loss : 2.571e-04
iter: 1660, loss : 6.718e-04
iter: 1661, loss : 2.063e-03
iter: 1662, loss : 3.131e-04
iter: 1663, loss : 2.328e-04
iter: 1664, loss : 1.554e-04
iter: 1665, loss : 6.059e-04
iter: 1666, loss : 4.658e-04
iter: 1667, loss : 1.311e-04
iter: 1668, loss : 1.251e-04
iter: 1669, loss : 7.952e-05
iter: 1670, loss : 1.265e-04
iter: 1671, loss : 2.497e-04
iter: 1672, loss : 2.627e-04
iter: 1673, loss : 2.996e-04
iter: 1674, loss : 9.676e-05
iter: 1675, loss : 7.731e-04
iter: 1676, loss : 7.896e-05
iter: 1677, loss : 9.540e-04
iter: 1678, loss : 1.279e-04
iter: 1679, loss : 2.403e-04
iter: 1680, loss : 7.221e-05
iter: 1681, loss : 2.540e-04
iter: 1682, loss : 8.119e-05
iter: 1683, loss : 3.512e-03
iter: 1684, loss : 1.202e-03
iter: 1685, loss : 7.656e-04
iter: 1686, loss : 1.272e-03
iter: 1687, loss : 3.164e-04
iter: 1688, loss : 8.187e-04
iter: 1689, loss : 1.281e-04
iter: 1690, loss : 1.771e-04
iter: 1691, loss : 3.314e-04
iter: 1692, loss : 1.410e-04
iter: 1693, loss : 3.737e-04
iter: 1694, loss : 8.704e-05
iter: 1695, loss : 1.097e-04
iter: 1696, loss : 1.081e-04
iter: 1697, loss : 8.563e-05
iter: 1698, loss : 3.227e-04
iter: 1699, loss : 1.879e-04
iter: 1700, loss : 2.104e-03
iter: 1701, loss : 2.028e-04
iter: 1702, loss : 2.469e-03
iter: 1703, loss : 1.317e-04
iter: 1704, loss : 1.569e-04
iter: 1705, loss : 2.236e-04
iter: 1706, loss : 8.927e-04
iter: 1707, loss : 2.980e-04
iter: 1708, loss : 1.070e-03
iter: 1709, loss : 1.755e-04
iter: 1710, loss : 7.412e-04
iter: 1711, loss : 9.139e-04
iter: 1712, loss : 9.324e-04
iter: 1713, loss : 1.439e-04
iter: 1714, loss : 2.364e-04
iter: 1715, loss : 3.014e-04
iter: 1716, loss : 6.129e-04
iter: 1717, loss : 2.692e-04
iter: 1718, loss : 6.803e-04
iter: 1719, loss : 4.781e-04
iter: 1720, loss : 1.610e-04
iter: 1721, loss : 7.241e-04
iter: 1722, loss : 3.380e-04
iter: 1723, loss : 3.186e-04
iter: 1724, loss : 1.058e-03
iter: 1725, loss : 2.444e-04
iter: 1726, loss : 8.638e-04
iter: 1727, loss : 1.610e-04
iter: 1728, loss : 2.961e-04
iter: 1729, loss : 3.213e-04
iter: 1730, loss : 1.609e-04
iter: 1731, loss : 1.956e-04
iter: 1732, loss : 1.027e-04
iter: 1733, loss : 4.499e-04
iter: 1734, loss : 4.033e-03
iter: 1735, loss : 4.878e-03
iter: 1736, loss : 3.629e-04
iter: 1737, loss : 2.689e-04
iter: 1738, loss : 1.829e-04
iter: 1739, loss : 4.419e-04
iter: 1740, loss : 2.105e-04
iter: 1741, loss : 2.391e-04
iter: 1742, loss : 1.388e-04
iter: 1743, loss : 2.294e-04
iter: 1744, loss : 1.420e-04
iter: 1745, loss : 1.109e-04
iter: 1746, loss : 2.364e-04
iter: 1747, loss : 2.179e-04
iter: 1748, loss : 1.063e-04
iter: 1749, loss : 8.051e-05
iter: 1750, loss : 1.341e-04
iter: 1751, loss : 3.506e-03
iter: 1752, loss : 1.988e-04
iter: 1753, loss : 1.374e-04
iter: 1754, loss : 2.036e-04
iter: 1755, loss : 1.055e-04
iter: 1756, loss : 1.823e-04
iter: 1757, loss : 1.052e-04
iter: 1758, loss : 2.818e-04
iter: 1759, loss : 2.076e-04
iter: 1760, loss : 1.121e-04
iter: 1761, loss : 7.347e-05
iter: 1762, loss : 1.053e-04
iter: 1763, loss : 6.389e-05
iter: 1764, loss : 1.523e-03
iter: 1765, loss : 1.276e-04
iter: 1766, loss : 2.232e-04
iter: 1767, loss : 7.516e-04
iter: 1768, loss : 2.016e-04
iter: 1769, loss : 6.261e-04
iter: 1770, loss : 1.189e-04
iter: 1771, loss : 5.720e-04
iter: 1772, loss : 1.271e-04
iter: 1773, loss : 6.685e-04
iter: 1774, loss : 2.342e-04
iter: 1775, loss : 2.892e-04
iter: 1776, loss : 1.795e-04
iter: 1777, loss : 1.290e-03
iter: 1778, loss : 1.616e-04
iter: 1779, loss : 2.815e-04
iter: 1780, loss : 1.443e-04
iter: 1781, loss : 2.570e-04
iter: 1782, loss : 1.701e-04
iter: 1783, loss : 4.546e-05
iter: 1784, loss : 1.087e-04
iter: 1785, loss : 1.618e-03
iter: 1786, loss : 1.568e-04
iter: 1787, loss : 6.381e-04
iter: 1788, loss : 2.449e-04
iter: 1789, loss : 2.130e-04
iter: 1790, loss : 3.393e-04
iter: 1791, loss : 6.215e-05
iter: 1792, loss : 1.071e-04
iter: 1793, loss : 1.007e-03
iter: 1794, loss : 2.968e-04
iter: 1795, loss : 3.299e-04
iter: 1796, loss : 9.909e-05
iter: 1797, loss : 9.424e-05
iter: 1798, loss : 1.304e-04
iter: 1799, loss : 8.286e-05
iter: 1800, loss : 1.153e-04
iter: 1801, loss : 2.622e-04
iter: 1802, loss : 3.803e-03
iter: 1803, loss : 2.156e-04
iter: 1804, loss : 1.746e-04
iter: 1805, loss : 1.316e-04
iter: 1806, loss : 8.272e-04
iter: 1807, loss : 2.234e-04
iter: 1808, loss : 2.322e-04
iter: 1809, loss : 4.775e-05
iter: 1810, loss : 1.098e-04
iter: 1811, loss : 1.153e-03
iter: 1812, loss : 4.196e-04
iter: 1813, loss : 8.816e-05
iter: 1814, loss : 2.656e-03
iter: 1815, loss : 2.371e-03
iter: 1816, loss : 3.393e-04
iter: 1817, loss : 3.703e-04
iter: 1818, loss : 1.707e-03
iter: 1819, loss : 1.502e-04
iter: 1820, loss : 2.426e-04
iter: 1821, loss : 1.723e-03
iter: 1822, loss : 1.018e-04
iter: 1823, loss : 4.063e-03
iter: 1824, loss : 4.002e-03
iter: 1825, loss : 5.000e-04
iter: 1826, loss : 2.321e-03
iter: 1827, loss : 2.968e-04
iter: 1828, loss : 4.431e-04
iter: 1829, loss : 7.240e-04
iter: 1830, loss : 5.739e-04
iter: 1831, loss : 1.664e-04
iter: 1832, loss : 5.647e-04
iter: 1833, loss : 1.320e-04
iter: 1834, loss : 1.803e-04
iter: 1835, loss : 1.652e-04
iter: 1836, loss : 2.972e-04
iter: 1837, loss : 1.834e-04
iter: 1838, loss : 2.114e-03
iter: 1839, loss : 4.133e-04
iter: 1840, loss : 3.273e-04
iter: 1841, loss : 1.957e-03
iter: 1842, loss : 3.692e-04
iter: 1843, loss : 1.029e-04
iter: 1844, loss : 1.816e-04
iter: 1845, loss : 2.197e-04
iter: 1846, loss : 2.370e-04
iter: 1847, loss : 1.861e-04
iter: 1848, loss : 2.091e-04
iter: 1849, loss : 2.176e-04
iter: 1850, loss : 1.191e-04
iter: 1851, loss : 1.888e-04
iter: 1852, loss : 1.313e-03
iter: 1853, loss : 2.478e-04
iter: 1854, loss : 6.287e-05
iter: 1855, loss : 1.189e-04
iter: 1856, loss : 3.080e-03
iter: 1857, loss : 2.943e-04
iter: 1858, loss : 2.903e-04
iter: 1859, loss : 3.076e-04
iter: 1860, loss : 2.981e-04
iter: 1861, loss : 2.362e-03
iter: 1862, loss : 1.429e-04
iter: 1863, loss : 2.157e-04
iter: 1864, loss : 2.883e-04
iter: 1865, loss : 8.856e-04
iter: 1866, loss : 6.709e-04
iter: 1867, loss : 1.543e-04
iter: 1868, loss : 2.195e-04
iter: 1869, loss : 8.687e-05
iter: 1870, loss : 4.528e-03
iter: 1871, loss : 1.801e-04
iter: 1872, loss : 3.909e-04
iter: 1873, loss : 7.108e-05
iter: 1874, loss : 1.172e-03
iter: 1875, loss : 1.698e-03
iter: 1876, loss : 5.378e-04
iter: 1877, loss : 1.356e-04
iter: 1878, loss : 8.557e-04
iter: 1879, loss : 3.949e-04
iter: 1880, loss : 3.277e-04
iter: 1881, loss : 1.819e-04
iter: 1882, loss : 1.325e-04
iter: 1883, loss : 1.066e-03
iter: 1884, loss : 8.145e-04
iter: 1885, loss : 2.865e-03
iter: 1886, loss : 3.357e-04
iter: 1887, loss : 3.935e-04
iter: 1888, loss : 2.496e-04
iter: 1889, loss : 5.033e-04
iter: 1890, loss : 3.860e-04
iter: 1891, loss : 1.270e-04
iter: 1892, loss : 1.837e-04
iter: 1893, loss : 4.731e-04
iter: 1894, loss : 2.829e-04
iter: 1895, loss : 1.115e-04
iter: 1896, loss : 1.022e-04
iter: 1897, loss : 2.529e-04
iter: 1898, loss : 2.137e-04
iter: 1899, loss : 4.420e-04
iter: 1900, loss : 1.619e-04
iter: 1901, loss : 1.623e-03
iter: 1902, loss : 3.402e-04
iter: 1903, loss : 5.604e-04
iter: 1904, loss : 1.933e-04
iter: 1905, loss : 2.541e-04
iter: 1906, loss : 1.963e-04
iter: 1907, loss : 2.854e-04
iter: 1908, loss : 9.387e-04
iter: 1909, loss : 2.067e-04
iter: 1910, loss : 4.584e-03
iter: 1911, loss : 2.345e-04
iter: 1912, loss : 3.415e-04
iter: 1913, loss : 1.399e-04
iter: 1914, loss : 5.943e-04
iter: 1915, loss : 1.537e-04
iter: 1916, loss : 1.222e-04
iter: 1917, loss : 2.296e-04
iter: 1918, loss : 6.809e-05
iter: 1919, loss : 1.365e-04
iter: 1920, loss : 1.398e-03
iter: 1921, loss : 2.770e-04
iter: 1922, loss : 2.013e-04
iter: 1923, loss : 2.075e-04
iter: 1924, loss : 6.931e-05
iter: 1925, loss : 1.957e-04
iter: 1926, loss : 9.324e-05
iter: 1927, loss : 1.085e-04
iter: 1928, loss : 7.263e-04
iter: 1929, loss : 6.792e-04
iter: 1930, loss : 1.683e-03
iter: 1931, loss : 1.771e-04
iter: 1932, loss : 5.664e-04
iter: 1933, loss : 4.806e-04
iter: 1934, loss : 8.106e-05
iter: 1935, loss : 1.809e-04
iter: 1936, loss : 1.787e-04
iter: 1937, loss : 2.506e-04
iter: 1938, loss : 2.825e-04
iter: 1939, loss : 6.189e-05
iter: 1940, loss : 1.329e-04
iter: 1941, loss : 7.924e-05
iter: 1942, loss : 1.455e-04
iter: 1943, loss : 2.149e-04
iter: 1944, loss : 8.106e-05
iter: 1945, loss : 1.572e-03
iter: 1946, loss : 1.982e-04
iter: 1947, loss : 1.152e-04
iter: 1948, loss : 4.077e-05
iter: 1949, loss : 8.340e-05
iter: 1950, loss : 6.634e-04
iter: 1951, loss : 1.314e-04
iter: 1952, loss : 1.043e-04
iter: 1953, loss : 7.853e-04
iter: 1954, loss : 1.533e-04
iter: 1955, loss : 5.419e-04
iter: 1956, loss : 1.242e-04
iter: 1957, loss : 5.372e-04
iter: 1958, loss : 1.364e-04
iter: 1959, loss : 1.128e-03
iter: 1960, loss : 1.366e-04
iter: 1961, loss : 1.198e-03
iter: 1962, loss : 2.007e-04
iter: 1963, loss : 1.276e-04
iter: 1964, loss : 6.533e-04
iter: 1965, loss : 1.383e-04
iter: 1966, loss : 4.141e-04
iter: 1967, loss : 2.761e-04
iter: 1968, loss : 1.947e-04
iter: 1969, loss : 9.843e-05
iter: 1970, loss : 1.600e-04
iter: 1971, loss : 2.314e-04
iter: 1972, loss : 1.064e-03
iter: 1973, loss : 1.173e-04
iter: 1974, loss : 4.329e-04
iter: 1975, loss : 2.393e-04
iter: 1976, loss : 1.589e-04
iter: 1977, loss : 3.129e-03
iter: 1978, loss : 5.676e-04
iter: 1979, loss : 2.055e-04
iter: 1980, loss : 8.443e-05
iter: 1981, loss : 5.402e-04
iter: 1982, loss : 9.269e-05
iter: 1983, loss : 1.590e-04
iter: 1984, loss : 2.615e-04
iter: 1985, loss : 8.351e-04
iter: 1986, loss : 1.358e-04
iter: 1987, loss : 1.177e-04
iter: 1988, loss : 2.539e-04
iter: 1989, loss : 1.386e-04
iter: 1990, loss : 1.189e-04
iter: 1991, loss : 3.386e-04
iter: 1992, loss : 1.024e-04
iter: 1993, loss : 1.260e-04
iter: 1994, loss : 6.379e-05
iter: 1995, loss : 2.420e-04
iter: 1996, loss : 6.021e-05
iter: 1997, loss : 1.074e-04
iter: 1998, loss : 5.791e-04
iter: 1999, loss : 1.536e-04
iter: 2000, loss : 8.115e-05
iter: 2001, loss : 8.054e-05
iter: 2002, loss : 6.889e-04
iter: 2003, loss : 5.502e-04
iter: 2004, loss : 8.538e-04
iter: 2005, loss : 9.228e-05
iter: 2006, loss : 1.125e-04
iter: 2007, loss : 1.911e-04
iter: 2008, loss : 1.183e-04
iter: 2009, loss : 8.065e-05
iter: 2010, loss : 2.125e-05
iter: 2011, loss : 3.173e-04
iter: 2012, loss : 1.369e-04
iter: 2013, loss : 1.365e-04
iter: 2014, loss : 4.560e-03
iter: 2015, loss : 1.245e-04
iter: 2016, loss : 2.128e-04
iter: 2017, loss : 1.827e-04
iter: 2018, loss : 2.366e-04
iter: 2019, loss : 1.354e-03
iter: 2020, loss : 1.345e-04
iter: 2021, loss : 3.039e-04
iter: 2022, loss : 4.917e-04
iter: 2023, loss : 1.584e-04
iter: 2024, loss : 8.849e-05
iter: 2025, loss : 4.783e-05
iter: 2026, loss : 1.063e-04
iter: 2027, loss : 4.609e-03
iter: 2028, loss : 2.558e-04
iter: 2029, loss : 6.630e-04
iter: 2030, loss : 3.004e-04
iter: 2031, loss : 1.244e-03
iter: 2032, loss : 1.037e-04
iter: 2033, loss : 3.815e-04
iter: 2034, loss : 1.043e-04
iter: 2035, loss : 2.508e-04
iter: 2036, loss : 9.161e-04
iter: 2037, loss : 3.350e-04
iter: 2038, loss : 1.686e-04
iter: 2039, loss : 1.260e-03
iter: 2040, loss : 4.023e-04
iter: 2041, loss : 7.943e-04
iter: 2042, loss : 5.561e-04
iter: 2043, loss : 1.901e-04
iter: 2044, loss : 8.188e-04
iter: 2045, loss : 8.421e-04
iter: 2046, loss : 1.963e-04
iter: 2047, loss : 3.724e-04
iter: 2048, loss : 1.737e-04
iter: 2049, loss : 2.873e-04
iter: 2050, loss : 3.485e-04
iter: 2051, loss : 3.382e-04
iter: 2052, loss : 1.229e-03
iter: 2053, loss : 2.383e-04
iter: 2054, loss : 2.160e-04
iter: 2055, loss : 2.191e-04
iter: 2056, loss : 1.125e-04
iter: 2057, loss : 1.702e-04
iter: 2058, loss : 1.292e-04
iter: 2059, loss : 2.323e-03
iter: 2060, loss : 9.603e-05
iter: 2061, loss : 8.169e-05
iter: 2062, loss : 2.369e-04
iter: 2063, loss : 2.879e-04
iter: 2064, loss : 1.161e-04
iter: 2065, loss : 9.565e-04
iter: 2066, loss : 3.582e-04
iter: 2067, loss : 8.673e-05
iter: 2068, loss : 8.040e-04
iter: 2069, loss : 2.605e-04
iter: 2070, loss : 6.785e-05
iter: 2071, loss : 1.990e-04
iter: 2072, loss : 1.134e-04
iter: 2073, loss : 1.938e-04
iter: 2074, loss : 9.599e-05
iter: 2075, loss : 4.737e-04
iter: 2076, loss : 6.866e-05
iter: 2077, loss : 5.896e-05
iter: 2078, loss : 7.717e-05
iter: 2079, loss : 7.072e-05
iter: 2080, loss : 1.277e-04
iter: 2081, loss : 5.216e-04
iter: 2082, loss : 1.726e-04
iter: 2083, loss : 2.490e-03
iter: 2084, loss : 5.727e-04
iter: 2085, loss : 1.733e-04
iter: 2086, loss : 5.621e-04
iter: 2087, loss : 2.653e-04
iter: 2088, loss : 5.208e-04
iter: 2089, loss : 3.918e-03
iter: 2090, loss : 1.077e-04
iter: 2091, loss : 1.756e-04
iter: 2092, loss : 1.893e-04
iter: 2093, loss : 1.124e-04
iter: 2094, loss : 2.019e-03
iter: 2095, loss : 1.570e-04
iter: 2096, loss : 2.105e-04
iter: 2097, loss : 1.450e-04
iter: 2098, loss : 2.604e-03
iter: 2099, loss : 9.119e-04
iter: 2100, loss : 1.069e-03
iter: 2101, loss : 4.853e-04
iter: 2102, loss : 6.847e-04
iter: 2103, loss : 5.840e-04
iter: 2104, loss : 2.026e-04
iter: 2105, loss : 2.340e-04
iter: 2106, loss : 4.187e-04
iter: 2107, loss : 3.429e-04
iter: 2108, loss : 4.199e-05
iter: 2109, loss : 2.238e-04
iter: 2110, loss : 8.835e-04
iter: 2111, loss : 1.976e-04
iter: 2112, loss : 3.438e-04
iter: 2113, loss : 4.798e-04
iter: 2114, loss : 2.677e-03
iter: 2115, loss : 2.436e-04
iter: 2116, loss : 1.931e-04
iter: 2117, loss : 2.312e-04
iter: 2118, loss : 1.803e-04
iter: 2119, loss : 9.519e-05
iter: 2120, loss : 1.370e-04
iter: 2121, loss : 1.300e-04
iter: 2122, loss : 5.684e-04
iter: 2123, loss : 7.249e-04
iter: 2124, loss : 9.775e-04
iter: 2125, loss : 1.122e-03
iter: 2126, loss : 5.465e-04
iter: 2127, loss : 5.917e-04
iter: 2128, loss : 5.498e-04
iter: 2129, loss : 4.309e-04
iter: 2130, loss : 5.486e-04
iter: 2131, loss : 2.445e-03
iter: 2132, loss : 3.857e-04
iter: 2133, loss : 1.920e-04
iter: 2134, loss : 3.828e-04
iter: 2135, loss : 3.706e-04
iter: 2136, loss : 2.360e-04
iter: 2137, loss : 5.610e-04
iter: 2138, loss : 2.475e-04
iter: 2139, loss : 6.111e-04
iter: 2140, loss : 2.616e-04
iter: 2141, loss : 7.607e-05
iter: 2142, loss : 3.512e-04
iter: 2143, loss : 1.788e-04
iter: 2144, loss : 2.237e-04
iter: 2145, loss : 8.045e-04
iter: 2146, loss : 1.850e-04
iter: 2147, loss : 1.218e-03
iter: 2148, loss : 1.025e-04
iter: 2149, loss : 2.855e-04
iter: 2150, loss : 4.123e-03
iter: 2151, loss : 6.237e-04
iter: 2152, loss : 3.044e-04
iter: 2153, loss : 3.317e-04
iter: 2154, loss : 4.383e-04
iter: 2155, loss : 3.192e-04
iter: 2156, loss : 1.164e-04
iter: 2157, loss : 1.363e-04
iter: 2158, loss : 3.636e-03
iter: 2159, loss : 1.865e-03
iter: 2160, loss : 3.418e-04
iter: 2161, loss : 8.229e-04
iter: 2162, loss : 3.903e-04
iter: 2163, loss : 1.128e-03
iter: 2164, loss : 1.979e-04
iter: 2165, loss : 2.914e-04
iter: 2166, loss : 1.942e-04
iter: 2167, loss : 1.206e-04
iter: 2168, loss : 5.109e-04
iter: 2169, loss : 6.350e-04
iter: 2170, loss : 5.032e-04
iter: 2171, loss : 9.946e-04
iter: 2172, loss : 2.677e-04
iter: 2173, loss : 1.840e-04
iter: 2174, loss : 1.712e-04
iter: 2175, loss : 1.979e-04
iter: 2176, loss : 2.437e-04
iter: 2177, loss : 1.008e-04
iter: 2178, loss : 3.394e-04
iter: 2179, loss : 7.679e-05
iter: 2180, loss : 4.240e-04
iter: 2181, loss : 7.819e-04
iter: 2182, loss : 1.189e-04
iter: 2183, loss : 2.606e-04
iter: 2184, loss : 3.881e-04
iter: 2185, loss : 1.695e-04
iter: 2186, loss : 3.502e-04
iter: 2187, loss : 3.344e-04
iter: 2188, loss : 1.378e-03
iter: 2189, loss : 1.396e-04
iter: 2190, loss : 1.994e-04
iter: 2191, loss : 1.203e-03
iter: 2192, loss : 9.972e-04
iter: 2193, loss : 2.688e-04
iter: 2194, loss : 1.556e-04
iter: 2195, loss : 1.352e-03
iter: 2196, loss : 3.916e-04
iter: 2197, loss : 7.583e-05
iter: 2198, loss : 1.982e-04
iter: 2199, loss : 1.783e-03
iter: 2200, loss : 2.010e-04
iter: 2201, loss : 2.649e-04
iter: 2202, loss : 2.394e-04
iter: 2203, loss : 6.794e-04
iter: 2204, loss : 1.569e-04
iter: 2205, loss : 2.506e-04
iter: 2206, loss : 8.734e-05
iter: 2207, loss : 1.007e-04
iter: 2208, loss : 1.521e-04
iter: 2209, loss : 7.283e-04
iter: 2210, loss : 2.445e-04
iter: 2211, loss : 2.794e-04
iter: 2212, loss : 2.081e-04
iter: 2213, loss : 1.046e-04
iter: 2214, loss : 2.683e-04
iter: 2215, loss : 1.056e-04
iter: 2216, loss : 5.765e-04
iter: 2217, loss : 7.572e-05
iter: 2218, loss : 1.289e-04
iter: 2219, loss : 3.383e-03
iter: 2220, loss : 5.427e-04
iter: 2221, loss : 2.229e-04
iter: 2222, loss : 8.999e-04
iter: 2223, loss : 5.810e-04
iter: 2224, loss : 1.953e-04
iter: 2225, loss : 2.829e-04
iter: 2226, loss : 1.071e-04
iter: 2227, loss : 9.371e-05
iter: 2228, loss : 6.921e-05
iter: 2229, loss : 1.487e-04
iter: 2230, loss : 3.601e-04
iter: 2231, loss : 8.654e-05
iter: 2232, loss : 1.601e-04
iter: 2233, loss : 1.839e-04
iter: 2234, loss : 1.504e-04
iter: 2235, loss : 2.568e-04
iter: 2236, loss : 2.352e-04
iter: 2237, loss : 1.792e-04
iter: 2238, loss : 1.927e-04
iter: 2239, loss : 2.160e-03
iter: 2240, loss : 2.700e-04
iter: 2241, loss : 1.258e-04
iter: 2242, loss : 9.146e-04
iter: 2243, loss : 2.466e-03
iter: 2244, loss : 1.256e-04
iter: 2245, loss : 5.055e-04
iter: 2246, loss : 4.547e-04
iter: 2247, loss : 3.958e-04
iter: 2248, loss : 5.023e-04
iter: 2249, loss : 1.985e-04
iter: 2250, loss : 2.961e-04
iter: 2251, loss : 5.303e-04
iter: 2252, loss : 1.909e-03
iter: 2253, loss : 1.569e-04
iter: 2254, loss : 7.367e-05
iter: 2255, loss : 2.979e-04
iter: 2256, loss : 1.047e-04
iter: 2257, loss : 2.876e-04
iter: 2258, loss : 6.343e-05
iter: 2259, loss : 1.866e-04
iter: 2260, loss : 1.538e-04
iter: 2261, loss : 1.316e-04
iter: 2262, loss : 1.723e-04
iter: 2263, loss : 1.787e-03
iter: 2264, loss : 1.504e-04
iter: 2265, loss : 5.045e-04
iter: 2266, loss : 4.430e-04
iter: 2267, loss : 1.038e-04
iter: 2268, loss : 1.541e-04
iter: 2269, loss : 6.257e-05
iter: 2270, loss : 1.667e-04
iter: 2271, loss : 8.602e-05
iter: 2272, loss : 2.376e-04
iter: 2273, loss : 2.084e-04
iter: 2274, loss : 1.240e-04
iter: 2275, loss : 1.508e-04
iter: 2276, loss : 7.136e-05
iter: 2277, loss : 3.756e-05
iter: 2278, loss : 2.621e-04
iter: 2279, loss : 9.450e-05
iter: 2280, loss : 1.215e-04
iter: 2281, loss : 1.109e-03
iter: 2282, loss : 1.877e-04
iter: 2283, loss : 9.907e-05
iter: 2284, loss : 2.864e-03
iter: 2285, loss : 1.216e-04
iter: 2286, loss : 3.196e-04
iter: 2287, loss : 3.486e-03
iter: 2288, loss : 1.195e-03
iter: 2289, loss : 3.609e-04
iter: 2290, loss : 2.947e-04
iter: 2291, loss : 4.049e-04
iter: 2292, loss : 9.160e-05
iter: 2293, loss : 2.236e-04
iter: 2294, loss : 1.457e-04
iter: 2295, loss : 1.660e-04
iter: 2296, loss : 1.476e-04
iter: 2297, loss : 7.543e-04
iter: 2298, loss : 1.173e-04
iter: 2299, loss : 1.051e-04
iter: 2300, loss : 3.511e-03
iter: 2301, loss : 3.653e-04
iter: 2302, loss : 1.877e-03
iter: 2303, loss : 1.415e-03
iter: 2304, loss : 3.181e-04
iter: 2305, loss : 1.164e-04
iter: 2306, loss : 2.375e-04
iter: 2307, loss : 1.083e-04
iter: 2308, loss : 1.503e-04
iter: 2309, loss : 1.647e-04
iter: 2310, loss : 2.786e-04
iter: 2311, loss : 2.331e-03
iter: 2312, loss : 1.414e-04
iter: 2313, loss : 8.445e-05
iter: 2314, loss : 2.389e-04
iter: 2315, loss : 1.270e-04
iter: 2316, loss : 1.397e-04
iter: 2317, loss : 9.038e-05
iter: 2318, loss : 4.819e-05
iter: 2319, loss : 5.237e-04
iter: 2320, loss : 7.121e-04
iter: 2321, loss : 1.083e-04
iter: 2322, loss : 4.017e-04
iter: 2323, loss : 1.898e-04
iter: 2324, loss : 1.890e-04
iter: 2325, loss : 1.240e-04
iter: 2326, loss : 1.215e-04
iter: 2327, loss : 1.319e-04
iter: 2328, loss : 3.741e-03
iter: 2329, loss : 1.428e-04
iter: 2330, loss : 2.956e-04
iter: 2331, loss : 2.246e-04
iter: 2332, loss : 1.848e-04
iter: 2333, loss : 1.103e-04
iter: 2334, loss : 1.159e-04
iter: 2335, loss : 2.128e-04
iter: 2336, loss : 4.475e-04
iter: 2337, loss : 3.023e-04
iter: 2338, loss : 1.182e-03
iter: 2339, loss : 1.695e-04
iter: 2340, loss : 2.248e-04
iter: 2341, loss : 7.591e-04
iter: 2342, loss : 2.466e-04
iter: 2343, loss : 2.847e-04
iter: 2344, loss : 1.728e-04
iter: 2345, loss : 1.099e-04
iter: 2346, loss : 9.405e-05
iter: 2347, loss : 9.529e-05
iter: 2348, loss : 4.524e-05
iter: 2349, loss : 2.524e-04
iter: 2350, loss : 2.000e-04
iter: 2351, loss : 1.359e-04
iter: 2352, loss : 1.350e-03
iter: 2353, loss : 1.280e-04
iter: 2354, loss : 2.394e-04
iter: 2355, loss : 8.979e-05
iter: 2356, loss : 9.917e-05
iter: 2357, loss : 3.823e-03
iter: 2358, loss : 2.171e-03
iter: 2359, loss : 2.211e-04
iter: 2360, loss : 2.999e-03
iter: 2361, loss : 6.367e-04
iter: 2362, loss : 6.157e-04
iter: 2363, loss : 4.495e-04
iter: 2364, loss : 4.963e-04
iter: 2365, loss : 3.454e-04
iter: 2366, loss : 2.348e-04
iter: 2367, loss : 4.097e-04
iter: 2368, loss : 1.112e-03
iter: 2369, loss : 3.336e-04
iter: 2370, loss : 7.019e-04
iter: 2371, loss : 1.830e-03
iter: 2372, loss : 6.573e-04
iter: 2373, loss : 4.939e-04
iter: 2374, loss : 4.040e-04
iter: 2375, loss : 2.758e-04
iter: 2376, loss : 1.237e-03
iter: 2377, loss : 7.804e-04
iter: 2378, loss : 6.316e-04
iter: 2379, loss : 3.057e-04
iter: 2380, loss : 3.365e-04
iter: 2381, loss : 3.905e-04
iter: 2382, loss : 1.972e-04
iter: 2383, loss : 4.393e-04
iter: 2384, loss : 3.785e-04
iter: 2385, loss : 2.632e-04
iter: 2386, loss : 3.855e-04
iter: 2387, loss : 3.193e-04
iter: 2388, loss : 2.678e-04
iter: 2389, loss : 6.356e-05
iter: 2390, loss : 3.118e-04
iter: 2391, loss : 1.428e-03
iter: 2392, loss : 1.209e-04
iter: 2393, loss : 8.258e-05
iter: 2394, loss : 9.528e-05
iter: 2395, loss : 4.644e-04
iter: 2396, loss : 1.080e-04
iter: 2397, loss : 1.572e-04
iter: 2398, loss : 1.119e-04
iter: 2399, loss : 1.007e-03
iter: 2400, loss : 5.228e-04
iter: 2401, loss : 1.805e-04
iter: 2402, loss : 1.186e-04
iter: 2403, loss : 2.321e-04
iter: 2404, loss : 4.293e-04
iter: 2405, loss : 1.292e-04
iter: 2406, loss : 1.025e-04
iter: 2407, loss : 1.192e-03
iter: 2408, loss : 2.399e-04
iter: 2409, loss : 2.832e-04
iter: 2410, loss : 6.075e-04
iter: 2411, loss : 3.615e-04
iter: 2412, loss : 1.856e-04
iter: 2413, loss : 4.339e-05
iter: 2414, loss : 1.262e-04
iter: 2415, loss : 9.044e-05
iter: 2416, loss : 1.064e-04
iter: 2417, loss : 4.616e-05
iter: 2418, loss : 1.298e-04
iter: 2419, loss : 4.434e-03
iter: 2420, loss : 2.107e-04
iter: 2421, loss : 1.209e-04
iter: 2422, loss : 1.528e-03
iter: 2423, loss : 9.629e-04
iter: 2424, loss : 2.006e-04
iter: 2425, loss : 6.887e-04
iter: 2426, loss : 6.448e-04
iter: 2427, loss : 2.833e-04
iter: 2428, loss : 1.933e-04
iter: 2429, loss : 1.848e-04
iter: 2430, loss : 4.346e-04
iter: 2431, loss : 1.501e-04
iter: 2432, loss : 1.689e-03
iter: 2433, loss : 2.317e-03
iter: 2434, loss : 5.000e-04
iter: 2435, loss : 1.558e-03
iter: 2436, loss : 2.075e-04
iter: 2437, loss : 6.872e-04
iter: 2438, loss : 5.672e-04
iter: 2439, loss : 2.011e-03
iter: 2440, loss : 4.449e-04
iter: 2441, loss : 2.985e-04
iter: 2442, loss : 3.813e-04
iter: 2443, loss : 6.543e-04
iter: 2444, loss : 3.333e-04
iter: 2445, loss : 3.972e-04
iter: 2446, loss : 1.098e-04
iter: 2447, loss : 1.481e-03
iter: 2448, loss : 3.232e-04
iter: 2449, loss : 3.061e-04
iter: 2450, loss : 2.211e-04
iter: 2451, loss : 8.224e-05
iter: 2452, loss : 9.334e-05
iter: 2453, loss : 2.398e-04
iter: 2454, loss : 4.447e-04
iter: 2455, loss : 4.218e-04
iter: 2456, loss : 7.343e-05
iter: 2457, loss : 1.520e-04
iter: 2458, loss : 1.672e-04
iter: 2459, loss : 3.916e-04
iter: 2460, loss : 3.293e-04
iter: 2461, loss : 2.011e-04
iter: 2462, loss : 6.069e-04
iter: 2463, loss : 1.589e-04
iter: 2464, loss : 1.676e-04
iter: 2465, loss : 9.057e-05
iter: 2466, loss : 9.928e-04
iter: 2467, loss : 1.716e-04
iter: 2468, loss : 2.485e-04
iter: 2469, loss : 1.564e-03
iter: 2470, loss : 7.103e-04
iter: 2471, loss : 1.808e-04
iter: 2472, loss : 3.366e-04
iter: 2473, loss : 1.008e-04
iter: 2474, loss : 1.105e-03
iter: 2475, loss : 2.885e-04
iter: 2476, loss : 5.485e-04
iter: 2477, loss : 1.434e-04
iter: 2478, loss : 3.749e-04
iter: 2479, loss : 1.948e-04
iter: 2480, loss : 4.653e-04
iter: 2481, loss : 1.972e-04
iter: 2482, loss : 2.642e-04
iter: 2483, loss : 1.270e-04
iter: 2484, loss : 1.160e-04
iter: 2485, loss : 1.090e-03
iter: 2486, loss : 6.964e-05
iter: 2487, loss : 3.687e-04
iter: 2488, loss : 3.066e-04
iter: 2489, loss : 9.071e-04
iter: 2490, loss : 2.832e-04
iter: 2491, loss : 2.595e-04
iter: 2492, loss : 4.800e-04
iter: 2493, loss : 1.463e-04
iter: 2494, loss : 3.541e-04
iter: 2495, loss : 1.007e-03
iter: 2496, loss : 5.215e-04
iter: 2497, loss : 2.777e-04
iter: 2498, loss : 1.839e-04
iter: 2499, loss : 1.258e-03
iter: 2500, loss : 3.320e-04
iter: 2501, loss : 5.140e-04
iter: 2502, loss : 1.376e-03
iter: 2503, loss : 2.925e-04
iter: 2504, loss : 5.192e-04
iter: 2505, loss : 1.304e-04
iter: 2506, loss : 1.633e-04
iter: 2507, loss : 1.385e-04
iter: 2508, loss : 1.237e-04
iter: 2509, loss : 1.099e-03
iter: 2510, loss : 3.330e-04
iter: 2511, loss : 1.557e-04
iter: 2512, loss : 4.958e-04
iter: 2513, loss : 1.605e-04
iter: 2514, loss : 1.293e-03
iter: 2515, loss : 4.815e-04
iter: 2516, loss : 3.509e-04
iter: 2517, loss : 1.867e-04
iter: 2518, loss : 4.088e-03
iter: 2519, loss : 1.388e-04
iter: 2520, loss : 1.904e-04
iter: 2521, loss : 4.031e-04
iter: 2522, loss : 2.668e-03
iter: 2523, loss : 5.880e-04
iter: 2524, loss : 2.930e-03
iter: 2525, loss : 9.907e-04
iter: 2526, loss : 2.005e-04
iter: 2527, loss : 3.043e-04
iter: 2528, loss : 2.101e-04
iter: 2529, loss : 4.034e-04
iter: 2530, loss : 1.212e-04
iter: 2531, loss : 4.575e-04
iter: 2532, loss : 4.031e-04
iter: 2533, loss : 2.062e-03
iter: 2534, loss : 2.580e-04
iter: 2535, loss : 1.180e-04
iter: 2536, loss : 4.835e-04
iter: 2537, loss : 2.773e-04
iter: 2538, loss : 3.031e-04
iter: 2539, loss : 2.601e-04
iter: 2540, loss : 1.429e-03
iter: 2541, loss : 1.008e-04
iter: 2542, loss : 2.530e-03
iter: 2543, loss : 2.877e-04
iter: 2544, loss : 1.661e-04
iter: 2545, loss : 7.250e-04
iter: 2546, loss : 4.915e-04
iter: 2547, loss : 2.330e-04
iter: 2548, loss : 2.927e-04
iter: 2549, loss : 1.334e-04
iter: 2550, loss : 1.281e-04
iter: 2551, loss : 9.033e-04
iter: 2552, loss : 4.203e-04
iter: 2553, loss : 1.291e-04
iter: 2554, loss : 5.119e-04
iter: 2555, loss : 1.764e-04
iter: 2556, loss : 1.448e-04
iter: 2557, loss : 1.558e-03
iter: 2558, loss : 2.782e-04
iter: 2559, loss : 3.266e-03
iter: 2560, loss : 4.390e-04
iter: 2561, loss : 2.686e-04
iter: 2562, loss : 3.076e-04
iter: 2563, loss : 2.210e-04
iter: 2564, loss : 3.866e-04
iter: 2565, loss : 2.008e-03
iter: 2566, loss : 2.122e-04
iter: 2567, loss : 2.414e-04
iter: 2568, loss : 5.423e-04
iter: 2569, loss : 1.563e-04
iter: 2570, loss : 1.756e-04
iter: 2571, loss : 2.018e-04
iter: 2572, loss : 1.163e-04
iter: 2573, loss : 1.956e-04
iter: 2574, loss : 6.117e-04
iter: 2575, loss : 3.516e-04
iter: 2576, loss : 2.861e-04
iter: 2577, loss : 2.815e-04
iter: 2578, loss : 1.802e-04
iter: 2579, loss : 3.687e-03
iter: 2580, loss : 3.235e-04
iter: 2581, loss : 2.442e-04
iter: 2582, loss : 8.210e-05
iter: 2583, loss : 5.524e-04
iter: 2584, loss : 2.723e-04
iter: 2585, loss : 3.452e-04
iter: 2586, loss : 1.084e-04
iter: 2587, loss : 1.140e-04
iter: 2588, loss : 2.387e-04
iter: 2589, loss : 7.150e-04
iter: 2590, loss : 2.121e-04
iter: 2591, loss : 2.587e-04
iter: 2592, loss : 3.057e-03
iter: 2593, loss : 1.907e-04
iter: 2594, loss : 1.112e-04
iter: 2595, loss : 2.027e-04
iter: 2596, loss : 7.655e-04
iter: 2597, loss : 4.457e-04
iter: 2598, loss : 1.291e-04
iter: 2599, loss : 3.104e-04
iter: 2600, loss : 1.120e-04
iter: 2601, loss : 9.424e-05
iter: 2602, loss : 2.047e-04
iter: 2603, loss : 1.660e-04
iter: 2604, loss : 1.024e-04
iter: 2605, loss : 4.258e-04
iter: 2606, loss : 1.829e-04
iter: 2607, loss : 1.214e-04
iter: 2608, loss : 1.095e-04
iter: 2609, loss : 2.071e-03
iter: 2610, loss : 2.491e-03
iter: 2611, loss : 2.788e-04
iter: 2612, loss : 6.547e-04
iter: 2613, loss : 2.217e-04
iter: 2614, loss : 3.356e-04
iter: 2615, loss : 2.544e-04
iter: 2616, loss : 3.802e-04
iter: 2617, loss : 1.007e-03
iter: 2618, loss : 5.928e-04
iter: 2619, loss : 8.291e-04
iter: 2620, loss : 7.187e-04
iter: 2621, loss : 2.251e-04
iter: 2622, loss : 5.410e-04
iter: 2623, loss : 4.036e-04
iter: 2624, loss : 2.694e-04
iter: 2625, loss : 7.420e-05
iter: 2626, loss : 1.143e-03
iter: 2627, loss : 5.390e-04
iter: 2628, loss : 7.012e-04
iter: 2629, loss : 2.316e-04
iter: 2630, loss : 1.208e-04
iter: 2631, loss : 6.087e-04
iter: 2632, loss : 1.470e-04
iter: 2633, loss : 2.824e-04
iter: 2634, loss : 3.198e-03
iter: 2635, loss : 2.115e-04
iter: 2636, loss : 2.669e-04
iter: 2637, loss : 5.005e-04
iter: 2638, loss : 4.157e-04
iter: 2639, loss : 1.164e-04
iter: 2640, loss : 7.036e-05
iter: 2641, loss : 2.375e-04
iter: 2642, loss : 2.086e-04
iter: 2643, loss : 1.010e-04
iter: 2644, loss : 1.845e-04
iter: 2645, loss : 1.062e-04
iter: 2646, loss : 1.052e-04
iter: 2647, loss : 2.236e-04
iter: 2648, loss : 1.320e-04
iter: 2649, loss : 9.332e-05
iter: 2650, loss : 9.927e-05
iter: 2651, loss : 1.246e-04
iter: 2652, loss : 4.286e-04
iter: 2653, loss : 1.188e-04
iter: 2654, loss : 1.019e-04
iter: 2655, loss : 6.357e-05
iter: 2656, loss : 1.604e-04
iter: 2657, loss : 1.639e-04
iter: 2658, loss : 1.398e-04
iter: 2659, loss : 1.361e-04
iter: 2660, loss : 9.441e-05
iter: 2661, loss : 2.527e-05
iter: 2662, loss : 6.425e-05
iter: 2663, loss : 4.531e-05
iter: 2664, loss : 1.244e-03
iter: 2665, loss : 1.545e-04
iter: 2666, loss : 9.673e-05
iter: 2667, loss : 1.136e-04
iter: 2668, loss : 7.066e-05
iter: 2669, loss : 1.645e-03
iter: 2670, loss : 2.053e-04
iter: 2671, loss : 1.156e-04
iter: 2672, loss : 1.035e-04
iter: 2673, loss : 4.687e-04
iter: 2674, loss : 2.221e-04
iter: 2675, loss : 1.686e-04
iter: 2676, loss : 8.695e-05
iter: 2677, loss : 1.294e-04
iter: 2678, loss : 1.051e-04
iter: 2679, loss : 4.123e-05
iter: 2680, loss : 8.150e-04
iter: 2681, loss : 1.184e-04
iter: 2682, loss : 3.310e-04
iter: 2683, loss : 6.219e-05
iter: 2684, loss : 1.911e-04
iter: 2685, loss : 1.275e-04
iter: 2686, loss : 2.036e-04
iter: 2687, loss : 1.941e-04
iter: 2688, loss : 6.186e-04
iter: 2689, loss : 6.606e-04
iter: 2690, loss : 1.154e-04
iter: 2691, loss : 1.185e-04
iter: 2692, loss : 1.574e-04
iter: 2693, loss : 3.105e-03
iter: 2694, loss : 1.622e-04
iter: 2695, loss : 1.166e-03
iter: 2696, loss : 1.665e-04
iter: 2697, loss : 1.429e-04
iter: 2698, loss : 1.243e-04
iter: 2699, loss : 1.653e-04
iter: 2700, loss : 2.725e-04
iter: 2701, loss : 1.265e-03
iter: 2702, loss : 2.687e-04
iter: 2703, loss : 1.713e-04
iter: 2704, loss : 2.298e-04
iter: 2705, loss : 2.375e-04
iter: 2706, loss : 2.082e-04
iter: 2707, loss : 2.011e-04
iter: 2708, loss : 1.143e-03
iter: 2709, loss : 1.666e-04
iter: 2710, loss : 1.807e-04
iter: 2711, loss : 1.346e-04
iter: 2712, loss : 1.649e-04
iter: 2713, loss : 1.975e-04
iter: 2714, loss : 4.752e-05
iter: 2715, loss : 1.479e-03
iter: 2716, loss : 3.935e-04
iter: 2717, loss : 2.595e-04
iter: 2718, loss : 1.108e-04
iter: 2719, loss : 4.792e-04
iter: 2720, loss : 5.541e-04
iter: 2721, loss : 1.744e-04
iter: 2722, loss : 2.521e-04
iter: 2723, loss : 5.673e-04
iter: 2724, loss : 3.414e-04
iter: 2725, loss : 1.289e-04
iter: 2726, loss : 9.295e-05
iter: 2727, loss : 1.111e-04
iter: 2728, loss : 3.665e-05
iter: 2729, loss : 7.523e-05
iter: 2730, loss : 1.065e-04
iter: 2731, loss : 7.293e-04
iter: 2732, loss : 1.874e-04
iter: 2733, loss : 2.709e-04
iter: 2734, loss : 1.955e-04
iter: 2735, loss : 2.035e-04
iter: 2736, loss : 1.847e-04
iter: 2737, loss : 5.741e-05
iter: 2738, loss : 2.348e-04
iter: 2739, loss : 3.652e-04
iter: 2740, loss : 7.013e-04
iter: 2741, loss : 1.412e-04
iter: 2742, loss : 1.302e-04
iter: 2743, loss : 1.049e-04
iter: 2744, loss : 4.592e-04
iter: 2745, loss : 8.674e-05
iter: 2746, loss : 1.524e-04
iter: 2747, loss : 7.213e-04
iter: 2748, loss : 1.548e-04
iter: 2749, loss : 2.758e-04
iter: 2750, loss : 3.187e-03
iter: 2751, loss : 4.041e-04
iter: 2752, loss : 4.083e-05
iter: 2753, loss : 2.851e-04
iter: 2754, loss : 1.146e-03
iter: 2755, loss : 4.057e-04
iter: 2756, loss : 9.591e-05
iter: 2757, loss : 1.872e-04
iter: 2758, loss : 8.939e-04
iter: 2759, loss : 2.753e-04
iter: 2760, loss : 1.294e-04
iter: 2761, loss : 5.359e-04
iter: 2762, loss : 4.309e-04
iter: 2763, loss : 8.559e-05
iter: 2764, loss : 1.083e-03
iter: 2765, loss : 1.416e-04
iter: 2766, loss : 1.068e-04
iter: 2767, loss : 3.967e-04
iter: 2768, loss : 5.853e-04
iter: 2769, loss : 3.135e-04
iter: 2770, loss : 3.588e-04
iter: 2771, loss : 3.180e-04
iter: 2772, loss : 1.254e-04
iter: 2773, loss : 1.148e-04
iter: 2774, loss : 1.247e-04
iter: 2775, loss : 1.030e-04
iter: 2776, loss : 8.237e-05
iter: 2777, loss : 2.266e-03
iter: 2778, loss : 4.499e-04
iter: 2779, loss : 1.015e-04
iter: 2780, loss : 6.943e-04
iter: 2781, loss : 3.640e-04
iter: 2782, loss : 2.568e-04
iter: 2783, loss : 2.108e-04
iter: 2784, loss : 4.464e-04
iter: 2785, loss : 8.958e-05
iter: 2786, loss : 9.015e-05
iter: 2787, loss : 1.934e-04
iter: 2788, loss : 1.035e-03
iter: 2789, loss : 1.092e-04
iter: 2790, loss : 1.791e-04
iter: 2791, loss : 2.406e-04
iter: 2792, loss : 4.156e-04
iter: 2793, loss : 4.102e-04
iter: 2794, loss : 2.756e-04
iter: 2795, loss : 1.043e-04
iter: 2796, loss : 1.282e-04
iter: 2797, loss : 1.974e-04
iter: 2798, loss : 1.180e-04
iter: 2799, loss : 2.564e-05
iter: 2800, loss : 9.628e-04
iter: 2801, loss : 1.378e-04
iter: 2802, loss : 4.205e-04
iter: 2803, loss : 2.864e-04
iter: 2804, loss : 2.496e-04
iter: 2805, loss : 1.840e-04
iter: 2806, loss : 4.791e-04
iter: 2807, loss : 8.445e-05
iter: 2808, loss : 1.395e-04
iter: 2809, loss : 1.399e-04
iter: 2810, loss : 4.150e-04
iter: 2811, loss : 1.109e-03
iter: 2812, loss : 2.659e-04
iter: 2813, loss : 3.805e-04
iter: 2814, loss : 7.500e-05
iter: 2815, loss : 6.213e-04
iter: 2816, loss : 2.145e-04
iter: 2817, loss : 6.919e-05
iter: 2818, loss : 3.824e-04
iter: 2819, loss : 4.562e-05
iter: 2820, loss : 3.615e-05
iter: 2821, loss : 7.926e-04
iter: 2822, loss : 3.085e-04
iter: 2823, loss : 4.114e-04
iter: 2824, loss : 1.445e-04
iter: 2825, loss : 1.001e-03
iter: 2826, loss : 8.470e-04
iter: 2827, loss : 2.572e-04
iter: 2828, loss : 6.614e-04
iter: 2829, loss : 5.762e-04
iter: 2830, loss : 1.242e-03
iter: 2831, loss : 1.664e-04
iter: 2832, loss : 4.286e-04
iter: 2833, loss : 1.940e-04
iter: 2834, loss : 3.063e-04
iter: 2835, loss : 1.696e-04
iter: 2836, loss : 2.706e-04
iter: 2837, loss : 2.428e-04
iter: 2838, loss : 1.167e-04
iter: 2839, loss : 4.563e-04
iter: 2840, loss : 2.006e-04
iter: 2841, loss : 3.086e-04
iter: 2842, loss : 3.139e-04
iter: 2843, loss : 1.201e-04
iter: 2844, loss : 4.177e-04
iter: 2845, loss : 1.107e-04
iter: 2846, loss : 2.964e-04
iter: 2847, loss : 1.919e-04
iter: 2848, loss : 2.150e-04
iter: 2849, loss : 2.955e-04
iter: 2850, loss : 2.688e-04
iter: 2851, loss : 2.232e-04
iter: 2852, loss : 8.343e-05
iter: 2853, loss : 5.905e-05
iter: 2854, loss : 1.549e-04
iter: 2855, loss : 2.489e-04
iter: 2856, loss : 1.783e-04
iter: 2857, loss : 8.409e-05
iter: 2858, loss : 8.648e-05
iter: 2859, loss : 1.024e-03
iter: 2860, loss : 1.037e-04
iter: 2861, loss : 2.116e-03
iter: 2862, loss : 7.866e-04
iter: 2863, loss : 1.380e-04
iter: 2864, loss : 2.509e-04
iter: 2865, loss : 7.645e-04
iter: 2866, loss : 1.098e-04
iter: 2867, loss : 1.348e-04
iter: 2868, loss : 1.638e-04
iter: 2869, loss : 3.447e-04
iter: 2870, loss : 1.495e-04
iter: 2871, loss : 6.229e-05
iter: 2872, loss : 4.322e-04
iter: 2873, loss : 1.686e-04
iter: 2874, loss : 1.112e-04
iter: 2875, loss : 3.385e-04
iter: 2876, loss : 2.280e-04
iter: 2877, loss : 1.071e-04
iter: 2878, loss : 1.093e-03
iter: 2879, loss : 6.462e-04
iter: 2880, loss : 2.450e-04
iter: 2881, loss : 1.760e-04
iter: 2882, loss : 1.642e-04
iter: 2883, loss : 2.232e-04
iter: 2884, loss : 1.738e-04
iter: 2885, loss : 2.052e-04
iter: 2886, loss : 1.708e-04
iter: 2887, loss : 1.381e-04
iter: 2888, loss : 2.057e-04
iter: 2889, loss : 1.285e-04
iter: 2890, loss : 4.775e-04
iter: 2891, loss : 1.517e-04
iter: 2892, loss : 1.060e-04
iter: 2893, loss : 2.246e-04
iter: 2894, loss : 6.988e-05
iter: 2895, loss : 4.913e-05
iter: 2896, loss : 1.269e-04
iter: 2897, loss : 1.245e-03
iter: 2898, loss : 1.165e-04
iter: 2899, loss : 1.813e-04
iter: 2900, loss : 5.951e-04
iter: 2901, loss : 1.793e-04
iter: 2902, loss : 5.267e-05
iter: 2903, loss : 3.323e-04
iter: 2904, loss : 3.361e-04
iter: 2905, loss : 1.196e-04
iter: 2906, loss : 1.891e-04
iter: 2907, loss : 3.159e-04
iter: 2908, loss : 6.985e-04
iter: 2909, loss : 1.357e-04
iter: 2910, loss : 3.164e-04
iter: 2911, loss : 3.155e-04
iter: 2912, loss : 2.033e-04
iter: 2913, loss : 1.557e-04
iter: 2914, loss : 1.839e-04
iter: 2915, loss : 1.704e-04
iter: 2916, loss : 1.870e-04
iter: 2917, loss : 4.786e-04
iter: 2918, loss : 8.792e-05
iter: 2919, loss : 1.165e-04
iter: 2920, loss : 7.773e-05
iter: 2921, loss : 5.245e-05
iter: 2922, loss : 3.896e-04
iter: 2923, loss : 1.138e-04
iter: 2924, loss : 1.941e-04
iter: 2925, loss : 1.505e-04
iter: 2926, loss : 3.985e-04
iter: 2927, loss : 2.960e-04
iter: 2928, loss : 7.995e-05
iter: 2929, loss : 4.977e-04
iter: 2930, loss : 7.111e-05
iter: 2931, loss : 9.700e-04
iter: 2932, loss : 3.013e-04
iter: 2933, loss : 3.298e-03
iter: 2934, loss : 7.007e-04
iter: 2935, loss : 3.717e-04
iter: 2936, loss : 3.095e-04
iter: 2937, loss : 2.084e-04
iter: 2938, loss : 1.695e-04
iter: 2939, loss : 3.961e-04
iter: 2940, loss : 1.014e-04
iter: 2941, loss : 8.509e-05
iter: 2942, loss : 6.080e-05
iter: 2943, loss : 1.846e-04
iter: 2944, loss : 8.557e-04
iter: 2945, loss : 1.024e-04
iter: 2946, loss : 1.881e-04
iter: 2947, loss : 2.835e-04
iter: 2948, loss : 1.231e-04
iter: 2949, loss : 2.955e-04
iter: 2950, loss : 1.081e-04
iter: 2951, loss : 2.558e-04
iter: 2952, loss : 2.051e-04
iter: 2953, loss : 1.267e-04
iter: 2954, loss : 1.386e-04
iter: 2955, loss : 1.526e-04
iter: 2956, loss : 3.689e-04
iter: 2957, loss : 1.755e-04
iter: 2958, loss : 9.800e-05
iter: 2959, loss : 5.963e-05
iter: 2960, loss : 1.314e-04
iter: 2961, loss : 3.253e-03
iter: 2962, loss : 1.357e-03
iter: 2963, loss : 7.956e-05
iter: 2964, loss : 2.741e-04
iter: 2965, loss : 2.002e-03
iter: 2966, loss : 1.235e-04
iter: 2967, loss : 2.553e-04
iter: 2968, loss : 1.643e-04
iter: 2969, loss : 8.445e-05
iter: 2970, loss : 1.721e-04
iter: 2971, loss : 1.179e-04
iter: 2972, loss : 7.132e-04
iter: 2973, loss : 1.383e-03
iter: 2974, loss : 4.193e-04
iter: 2975, loss : 2.497e-03
iter: 2976, loss : 3.926e-04
iter: 2977, loss : 3.758e-04
iter: 2978, loss : 2.458e-03
iter: 2979, loss : 5.377e-04
iter: 2980, loss : 3.246e-04
iter: 2981, loss : 1.240e-04
iter: 2982, loss : 2.383e-04
iter: 2983, loss : 4.733e-05
iter: 2984, loss : 9.350e-05
iter: 2985, loss : 3.848e-04
iter: 2986, loss : 8.385e-04
iter: 2987, loss : 8.238e-05
iter: 2988, loss : 3.504e-04
iter: 2989, loss : 1.029e-04
iter: 2990, loss : 7.616e-05
iter: 2991, loss : 1.158e-04
iter: 2992, loss : 9.387e-05
iter: 2993, loss : 1.317e-04
iter: 2994, loss : 1.957e-04
iter: 2995, loss : 2.249e-03
iter: 2996, loss : 1.832e-04
iter: 2997, loss : 2.562e-03
iter: 2998, loss : 1.938e-04
iter: 2999, loss : 2.307e-04
iter: 2999, loss : 2.307e-04
--- 14.955312490463257 seconds --- to perform 3000 epochs
NOTE: For comparison we also called the torcheqdiff demo using the following command line ‘python ode_demo.py –niters 3000’. This triggers a comparable number of gradient evaluations on CPU. The iterations done in torcheqdiff complete in 348 seconds showing how on common CPUs, the use of heyoka may offer a significant speedup for NeuralODE research.
We are now ready to visualize the result of the learned neural ODE:
plt.plot(gt[:, 0], gt[:, 1])
ta_var.time = precision(0)
ta_var.state[:] = list(ic_var)
ta_var.state[:2] = ic
ta_var.pars[:] = np.array(nn_wb, dtype=np.single)
sol = ta_var.propagate_grid(np.array(t_grid, dtype=precision))[5]
plt.plot(sol[:, 0], sol[:, 1], ".")
[<matplotlib.lines.Line2D at 0x7f144633e360>]
It is also of interest to visualize the predictions of the trained system over a new random batch, as opposed to the predictions of the entire trajectory from the first observation:
batch_ic, batch_t_grid, batch_y = get_batch(t_grid, gt, batch_size_ic, batch_size_time)
plt.plot(gt[:, 0], gt[:, 1])
for i in range(batch_size_ic):
plt.plot(batch_y[:, i, 0], batch_y[:, i, 1], "k.")
start_time = time.time()
for batch in batched(batch_ic, batch_size_simd):
ta_var_b.set_time(precision(0.0))
ta_var_b.state[:, :] = deepcopy(ic_var_b)
ta_var_b.state[:dim, :] = np.array(batch).T
ta_var_b.pars[:] = np.tile(nn_wb, (batch_size_simd, 1)).T
sol_b = ta_var_b.propagate_grid(np.array(t_grid_b, dtype=precision))[1]
for sol in [sol_b[:, :, i] for i in range(batch_size_simd)]:
plt.plot(sol[:, 0], sol[:, 1], ".")
print(
"--- %s seconds --- to compute prediction over the whole batch"
% (time.time() - start_time)
)
--- 0.007606029510498047 seconds --- to compute prediction over the whole batch
