Neural ODEs - II#
We here implement, in heyoka, a standard traning pipeline for NeuralODE replicating this torcheqdiff demo.
We will be using the very same parameters and dynamics, of the original demo but only use a simple fixed learning rate stochastic gradient descent. Note that it is also possible to use more advanced optimizers such as those implemented in pytorch while computing the gradients using heyoka, but in this tutorial we keep it simple and just update the gradient naively.
See also:
# The main imports
import numpy as np
import time
from itertools import batched
from copy import deepcopy
# Import Heyoka
import heyoka as hy
%matplotlib inline
import matplotlib.pyplot as plt
Generating the ground truth#
The dynamics we consider is a simple third order system:
\[\begin{split}
\left\{
\begin{array}{l}
\dot x = -0.1 x^3 - 2y^3 \\
\dot y = 2 x^3 - 0.1y^3
\end{array}
\right.
\end{split}\]
which we implement in heyoka as follows:
# The symolic variables representing the system state.
x, y = hy.make_vars("x", "y")
# We now assemble the dynamics.
A = np.array([[-0.1, 2.0], [-2.0, -0.1]])
tmp = np.array([[x*x*x, y*y*y]]) @ A
dyn = [(x, tmp[0][0]), (y, tmp[0][1])]
# For convenience we store here the dimension of the state.
dim = len(dyn)
To generate groundtruth observations from this system we will use the following parameters:
# This is the number of observations (i.e. the ground truth).
data_size = 1000
# This is the initial condition used to generate the observations (i.e. the ground truth).
ic = [2.0, 0.0]
# This is the time grid used to generate the observations (i.e. the ground truth).
t_grid = np.linspace(0.0, 25, data_size)
We start by instantiating a heyoka.taylor_adaptive object that will generate our observations. These will be considered as ground truth. Note that in other settings this data may be observed from a real phenomena, rather than generated numerically. In that case there will be noise which is here not modelled.
start_time = time.time()
ta = hy.taylor_adaptive(
# The ODEs.
dyn,
# The initial conditions.
ic,
# Do not operate in compact mode.
compact_mode=False,
# Define the tolerance
tol=1e-18,
)
print(
"--- %s seconds --- to build (jit) the Taylor integrator"
% (time.time() - start_time)
)
[2024-12-17 22:48:12.574] [heyoka] [info] heyoka logger initialised
--- 0.07188987731933594 seconds --- to build (jit) the Taylor integrator
We are thus ready to generate the the ground truth using the method propagate_grid method of the Taylor adaptive integrator.
# It is good practice to always set initial conditions explicitly
ta.time = 0
ta.state[:] = ic
# This is our ground truth (gt)
gt = ta.propagate_grid(t_grid)[5]
Let us have a look at the generated observations.
plt.plot(gt[:, 0], gt[:, 1], 'k.')
[<matplotlib.lines.Line2D at 0x7f1fd444a000>]
Instantiating the neural ODE#
So far we have only generated the data we want to teach our Neural ODE to replicate. We now need to instantiate the Neural ODE system so that we can make predictions and compute the gradient of our loss.
Let us start with instantiating a feed-forward neural network using heyoka.models.ffnn.
# We define as nonlinearity a simple linear layer
linear = lambda inp: inp
# We call the factory to construct a FFNN. Following the torcheqdiff example here we put cubes as inputs:
ffnn = hy.model.ffnn(
inputs=[x*x*x, y*y*y],
nn_hidden=[50],
n_out=2,
activations=[hy.tanh, linear],
)
We are now able to build a taylor_adaptive_batch numerical integrator which will propagate the dynamics in batches as to leverage SIMD instructions:
\[\begin{split}
\left\{
\begin{array}{l}
\dot x = \mathcal N_{\theta, x}(x^3, y^3) \\
\dot y = \mathcal N_{\theta, y}(x^3, y^3) \\
\end{array}
\right.
\end{split}\]
And since we will want to be able to compute the sensitivities of this ODE we are first going to augment such dynamics with the first order variational equations:
# We decide on whether to use single or double precision. In this example it may
# Not be too important, but in other settings it can give a speedup when connected with batch and ensamble propagation.
# The single precision does effect, though, the quality of thegradient direction.
precision = np.double # np.single
batch_size_simd = hy.recommended_simd_size(fp_type=precision)
print("Batch size for SIMD use in integration: ", batch_size_simd)
# We thus define the neural dynamics
dyn_n = [(x, ffnn[0]), (y, ffnn[1])]
# We augment it with the variational one (only order 1 - i.e. gradient)
var_dyn_n = hy.var_ode_sys(dyn_n, args=hy.var_args.params, order=1)
Batch size for SIMD use in integration: 4
… and instantiate the variational integrator. For convenience we also instantiate the non batched version
# This is the batched version, which we will use for gradient computation
ta_var_b = hy.taylor_adaptive_batch(
# The ODEs.
var_dyn_n,
# The initial conditions.
np.ones((dim, batch_size_simd), dtype=precision),
# Operate in compact mode.
compact_mode=True,
# Define the tolerance (low tolerance is enough here)
tol=precision(1e-4),
# Single precision is enough here
fp_type=precision,
)
# This is the normal version used for wuick plotting and visualization of a single solution
ta_var = hy.taylor_adaptive(
# The ODEs.
var_dyn_n,
# The initial conditions.
ic,
# Operate in compact mode.
compact_mode=True,
# Define the tolerance (low tolerance is enough here)
tol=precision(1e-4),
# Single precision is enough here
fp_type=precision,
)
# We store for convenience the initial conditions in a numpy array (including the variational state)
ic_var_b = np.array(ta_var_b.state)
ic_var = np.array(ta_var.state)
Before starting the training, we define the details on the batch we are going to use in the generic step of stochastic gradient descent:
# This is the number of initial conditions to use in one batch (M)
batch_size_ic = 20
# This is the number of observations to predict from each sampled initial condition (T)
batch_size_time = 10
# We check that the observation batch size is a multiple of the simd batch size
print("This must be an integer: ", batch_size_ic/batch_size_simd)
This must be an integer: 5.0
This helper function creates a randomly generated batch from the available observation:
def get_batch(t_grid, gt, batch_size_ic=20, batch_size_time=10):
# We select the initial conditions from which generate predictions
s = np.random.choice(
np.arange(gt.shape[0] - batch_size_time, dtype=np.int64),
batch_size_ic,
replace=False,
)
batch_ic = gt[s, :] # (M, D)
# Assuming uniform grid and a non-autonomous system, all predictions will be made on the same time grid
batch_t_grid = t_grid[:batch_size_time] # (T)
batch_y = np.stack([gt[s + i] for i in range(batch_size_time)]) # (T,M,D)
return batch_ic, batch_t_grid, batch_y
Let us visualize what a batch looks like. First we generate one:
batch_ic, batch_t_grid, batch_y = get_batch(t_grid, gt, batch_size_ic, batch_size_time)
# We need the tgrid repeated as per heyoka API of the taylor adaptive batch
t_grid_b = np.repeat(batch_t_grid, batch_size_simd).reshape(-1, batch_size_simd)
Then we plot it overlapped to all the observations:
plt.plot(gt[:, 0], gt[:, 1])
for i in range(batch_size_ic):
plt.plot(batch_y[:, i, 0], batch_y[:, i, 1], "k.")
We initialize the network weights and biases.
# We compute initialization values for weigths / biases
n_pars = len(ta_var_b.pars)
nn_wb = np.random.normal(loc=0, scale=0.1, size=(n_pars,))
And visualize the prediction of the starting random network.
plt.plot(gt[:, 0], gt[:, 1])
for i in range(batch_size_ic):
plt.plot(batch_y[:, i, 0], batch_y[:, i, 1], "k.")
start_time = time.time()
for batch in batched(batch_ic, batch_size_simd):
ta_var_b.set_time(precision(0.))
ta_var_b.state[:,:] = deepcopy(ic_var_b)
ta_var_b.state[:dim,:] = np.array(batch).T
ta_var_b.pars[:] = np.tile(nn_wb, (batch_size_simd,1)).T
sol_b = ta_var_b.propagate_grid(np.array(t_grid_b, dtype=precision))[1]
for sol in [sol_b[:,:,i] for i in range(batch_size_simd)]:
plt.plot(sol[:, 0], sol[:, 1], ".")
print(
"--- %s seconds --- to compute prediction over the whole batch"
% (time.time() - start_time)
)
--- 0.006775617599487305 seconds --- to compute prediction over the whole batch
Training#
To train we need to first compute the gradient of the loss over the batch (M initial conditions and T times). Formally
\[
\mathcal L_{\theta} = \sum_{i=0}^{M}\sum_{j=0}^T (\pmb y_{ij} - \hat {\pmb y}_{ij})\cdot (\pmb y_{ij} - \hat {\pmb y}_{ij})
\]
\[
\frac{\partial L_\theta}{\partial \theta} = 2 \sum_{i=0}^{M}\sum_{j=0}^T (\pmb y_{ij} - \hat {\pmb y}_{ij})\cdot \frac{\partial \pmb y_{ij}}{\partial \theta}
\]
This helper function computes the loss and the gradient over one batch. The gradient is computed rearranging the ODE sensitivities computed propagating the variational dynamics:
def loss_and_gradient(nn_wb, batch_ic, batch_y):
# We reset the batch_loss value
batch_loss = 0
# We reset the gradient loss
grad = np.array([0.0] * ta_var_b.pars.shape[0])
counter=0
# We process the batch (of the dataset) in batches for SIMD efficient processing
for batch_simd in batched(batch_ic, batch_size_simd):
ta_var_b.set_time(precision(0.))
ta_var_b.state[:,:] = deepcopy(ic_var_b)
ta_var_b.state[:dim,:] = np.array(batch_simd).T
ta_var_b.pars[:] = np.tile(nn_wb, (batch_size_simd,1)).T
sol_b = ta_var_b.propagate_grid(np.array(t_grid_b, dtype=precision))[1]
# The numerical integration on the simd_batch is done, we process its result
# accumulating loss and gradient on the predictions
for sol in [sol_b[:,:,i] for i in range(batch_size_simd)]:
# Here is the term (y-y_hat)
diff = sol[:, :dim] - batch_y[:, counter, :]
counter+=1
# Which is then summed sum sum (y-y_hat).(y-y_hat)
batch_loss += np.sum(diff**2)
# And the gradient computed as 2 sum (y-y_hat).dy
for dy in sol[:, dim:].reshape(batch_y.shape[0], dim, -1):
grad += 2 * np.sum(diff @ dy, axis=0)
# We then take the mean over the points used
batch_loss /= batch_size_ic * batch_size_time
grad /= batch_size_ic * batch_size_time
return batch_loss, grad
Using the various blocks defined above the training loop takes the form:
# We reset initialization values for weigths / biases
n_pars = len(ta_var_b.pars)
nn_wb = np.random.normal(loc=0, scale=0.1, size=(n_pars,))
# We loop over the epochs
n_epochs = 3000
start_time = time.time()
for i in range(n_epochs):
# 1 - generate a batch
batch_ic, batch_t_grid, batch_y = get_batch(
t_grid, gt, batch_size_ic=batch_size_ic, batch_size_time=batch_size_time
)
# 2 - compute the loss and gradient on the batch
batch_loss, grad = loss_and_gradient(nn_wb, batch_ic, batch_y)
print(f"iter: {i}, loss : {batch_loss:.3e}", end="\r")
# 3 - update the weights and biases (learning rate here is fixed, we use a naive update rule which in this case works)
nn_wb = nn_wb - 0.1 * grad
print(f"iter: {i}, loss : {batch_loss:.3e}", end="\n")
print(
"--- %s seconds --- to perform %i epochs"
% (time.time() - start_time, n_epochs)
)
iter: 0, loss : 3.955e-02
iter: 1, loss : 5.897e-02
iter: 2, loss : 1.393e-01
iter: 3, loss : 1.170e-01
iter: 4, loss : 9.009e-02
iter: 5, loss : 3.262e-02
iter: 6, loss : 9.069e-02
iter: 7, loss : 3.560e-02
iter: 8, loss : 9.208e-02
iter: 9, loss : 6.473e-02
iter: 10, loss : 1.942e-01
iter: 11, loss : 6.008e-03
iter: 12, loss : 1.323e-01
iter: 13, loss : 9.296e-02
iter: 14, loss : 7.767e-02
iter: 15, loss : 4.100e-02
iter: 16, loss : 4.030e-02
iter: 17, loss : 6.469e-02
iter: 18, loss : 2.371e-02
iter: 19, loss : 1.506e-01
iter: 20, loss : 6.414e-02
iter: 21, loss : 7.207e-02
iter: 22, loss : 2.297e-03
iter: 23, loss : 4.617e-02
iter: 24, loss : 8.140e-02
iter: 25, loss : 1.050e-02
iter: 26, loss : 9.198e-02
iter: 27, loss : 1.822e-02
iter: 28, loss : 2.421e-02
iter: 29, loss : 3.560e-02
iter: 30, loss : 3.605e-03
iter: 31, loss : 9.588e-03
iter: 32, loss : 1.129e-02
iter: 33, loss : 8.705e-02
iter: 34, loss : 1.278e-01
iter: 35, loss : 4.628e-02
iter: 36, loss : 9.683e-03
iter: 37, loss : 6.095e-02
iter: 38, loss : 6.852e-02
iter: 39, loss : 8.911e-03
iter: 40, loss : 9.392e-03
iter: 41, loss : 5.221e-02
iter: 42, loss : 5.952e-02
iter: 43, loss : 8.742e-02
iter: 44, loss : 2.595e-02
iter: 45, loss : 4.053e-03
iter: 46, loss : 4.904e-02
iter: 47, loss : 1.587e-02
iter: 48, loss : 1.675e-02
iter: 49, loss : 4.514e-02
iter: 50, loss : 1.768e-03
iter: 51, loss : 5.563e-03
iter: 52, loss : 1.190e-02
iter: 53, loss : 3.979e-03
iter: 54, loss : 3.128e-02
iter: 55, loss : 3.593e-03
iter: 56, loss : 2.229e-02
iter: 57, loss : 2.746e-02
iter: 58, loss : 8.145e-03
iter: 59, loss : 1.436e-02
iter: 60, loss : 1.213e-02
iter: 61, loss : 6.665e-02
iter: 62, loss : 2.557e-02
iter: 63, loss : 3.301e-02
iter: 64, loss : 1.137e-02
iter: 65, loss : 7.407e-03
iter: 66, loss : 2.291e-02
iter: 67, loss : 9.424e-03
iter: 68, loss : 3.920e-02
iter: 69, loss : 5.518e-03
iter: 70, loss : 1.515e-03
iter: 71, loss : 7.204e-04
iter: 72, loss : 9.377e-03
iter: 73, loss : 3.478e-02
iter: 74, loss : 2.873e-02
iter: 75, loss : 6.951e-03
iter: 76, loss : 8.060e-04
iter: 77, loss : 2.610e-03
iter: 78, loss : 4.035e-04
iter: 79, loss : 3.531e-03
iter: 80, loss : 4.973e-03
iter: 81, loss : 4.849e-04
iter: 82, loss : 2.676e-02
iter: 83, loss : 2.257e-02
iter: 84, loss : 3.118e-04
iter: 85, loss : 7.005e-03
iter: 86, loss : 1.014e-04
iter: 87, loss : 1.589e-04
iter: 88, loss : 1.534e-02
iter: 89, loss : 1.948e-02
iter: 90, loss : 2.859e-04
iter: 91, loss : 2.004e-02
iter: 92, loss : 7.364e-03
iter: 93, loss : 2.088e-02
iter: 94, loss : 2.100e-04
iter: 95, loss : 1.298e-03
iter: 96, loss : 1.904e-02
iter: 97, loss : 2.645e-04
iter: 98, loss : 2.412e-04
iter: 99, loss : 1.119e-03
iter: 100, loss : 4.425e-04
iter: 101, loss : 1.871e-02
iter: 102, loss : 2.072e-04
iter: 103, loss : 4.972e-04
iter: 104, loss : 5.148e-03
iter: 105, loss : 5.589e-04
iter: 106, loss : 1.593e-04
iter: 107, loss : 9.212e-04
iter: 108, loss : 1.568e-03
iter: 109, loss : 1.358e-03
iter: 110, loss : 1.189e-03
iter: 111, loss : 3.561e-04
iter: 112, loss : 2.803e-03
iter: 113, loss : 2.969e-04
iter: 114, loss : 2.312e-03
iter: 115, loss : 2.993e-04
iter: 116, loss : 2.259e-04
iter: 117, loss : 7.187e-03
iter: 118, loss : 4.408e-04
iter: 119, loss : 2.088e-03
iter: 120, loss : 3.771e-04
iter: 121, loss : 3.881e-04
iter: 122, loss : 1.573e-02
iter: 123, loss : 1.206e-04
iter: 124, loss : 2.624e-02
iter: 125, loss : 5.283e-03
iter: 126, loss : 4.437e-04
iter: 127, loss : 7.204e-04
iter: 128, loss : 1.384e-02
iter: 129, loss : 2.519e-03
iter: 130, loss : 7.428e-04
iter: 131, loss : 8.316e-04
iter: 132, loss : 1.143e-03
iter: 133, loss : 6.995e-04
iter: 134, loss : 3.315e-03
iter: 135, loss : 7.879e-04
iter: 136, loss : 8.413e-03
iter: 137, loss : 3.493e-04
iter: 138, loss : 4.716e-04
iter: 139, loss : 5.499e-04
iter: 140, loss : 5.723e-04
iter: 141, loss : 3.502e-04
iter: 142, loss : 3.439e-04
iter: 143, loss : 1.068e-02
iter: 144, loss : 3.860e-04
iter: 145, loss : 1.081e-03
iter: 146, loss : 7.668e-03
iter: 147, loss : 4.599e-04
iter: 148, loss : 2.576e-04
iter: 149, loss : 4.054e-03
iter: 150, loss : 1.176e-02
iter: 151, loss : 8.798e-04
iter: 152, loss : 6.842e-04
iter: 153, loss : 5.134e-04
iter: 154, loss : 1.482e-03
iter: 155, loss : 6.578e-04
iter: 156, loss : 1.099e-02
iter: 157, loss : 3.395e-04
iter: 158, loss : 3.384e-03
iter: 159, loss : 1.047e-02
iter: 160, loss : 1.310e-02
iter: 161, loss : 6.841e-04
iter: 162, loss : 3.888e-03
iter: 163, loss : 1.104e-02
iter: 164, loss : 8.940e-03
iter: 165, loss : 7.522e-03
iter: 166, loss : 6.700e-04
iter: 167, loss : 1.886e-03
iter: 168, loss : 2.208e-03
iter: 169, loss : 8.156e-04
iter: 170, loss : 7.082e-04
iter: 171, loss : 6.785e-03
iter: 172, loss : 1.013e-03
iter: 173, loss : 7.295e-03
iter: 174, loss : 1.144e-02
iter: 175, loss : 7.672e-04
iter: 176, loss : 1.209e-03
iter: 177, loss : 1.049e-03
iter: 178, loss : 1.097e-03
iter: 179, loss : 2.611e-03
iter: 180, loss : 1.091e-03
iter: 181, loss : 4.759e-03
iter: 182, loss : 1.631e-03
iter: 183, loss : 3.258e-04
iter: 184, loss : 5.070e-04
iter: 185, loss : 5.386e-04
iter: 186, loss : 7.154e-04
iter: 187, loss : 4.601e-04
iter: 188, loss : 9.865e-03
iter: 189, loss : 3.433e-03
iter: 190, loss : 1.198e-03
iter: 191, loss : 8.688e-04
iter: 192, loss : 6.917e-03
iter: 193, loss : 9.089e-03
iter: 194, loss : 1.810e-03
iter: 195, loss : 1.776e-03
iter: 196, loss : 1.352e-03
iter: 197, loss : 2.980e-03
iter: 198, loss : 1.414e-02
iter: 199, loss : 1.514e-03
iter: 200, loss : 6.966e-04
iter: 201, loss : 6.520e-03
iter: 202, loss : 7.035e-03
iter: 203, loss : 1.163e-02
iter: 204, loss : 2.474e-04
iter: 205, loss : 7.791e-03
iter: 206, loss : 1.731e-02
iter: 207, loss : 1.484e-02
iter: 208, loss : 2.751e-03
iter: 209, loss : 2.496e-03
iter: 210, loss : 6.801e-03
iter: 211, loss : 1.884e-03
iter: 212, loss : 6.058e-03
iter: 213, loss : 1.213e-03
iter: 214, loss : 9.718e-04
iter: 215, loss : 9.607e-03
iter: 216, loss : 8.777e-03
iter: 217, loss : 1.545e-03
iter: 218, loss : 1.579e-03
iter: 219, loss : 1.600e-03
iter: 220, loss : 4.395e-04
iter: 221, loss : 1.559e-02
iter: 222, loss : 5.137e-03
iter: 223, loss : 6.755e-03
iter: 224, loss : 1.789e-03
iter: 225, loss : 1.449e-03
iter: 226, loss : 1.369e-03
iter: 227, loss : 6.670e-03
iter: 228, loss : 1.855e-03
iter: 229, loss : 5.643e-03
iter: 230, loss : 4.037e-03
iter: 231, loss : 3.612e-03
iter: 232, loss : 1.977e-03
iter: 233, loss : 3.647e-03
iter: 234, loss : 1.147e-03
iter: 235, loss : 1.311e-03
iter: 236, loss : 6.051e-03
iter: 237, loss : 9.093e-04
iter: 238, loss : 1.035e-03
iter: 239, loss : 1.260e-03
iter: 240, loss : 7.835e-04
iter: 241, loss : 4.891e-04
iter: 242, loss : 8.729e-04
iter: 243, loss : 3.341e-04
iter: 244, loss : 1.545e-03
iter: 245, loss : 6.697e-03
iter: 246, loss : 6.089e-04
iter: 247, loss : 6.690e-03
iter: 248, loss : 4.433e-03
iter: 249, loss : 9.242e-04
iter: 250, loss : 1.832e-03
iter: 251, loss : 8.544e-04
iter: 252, loss : 8.557e-03
iter: 253, loss : 3.278e-03
iter: 254, loss : 1.126e-02
iter: 255, loss : 3.808e-04
iter: 256, loss : 3.984e-04
iter: 257, loss : 1.169e-03
iter: 258, loss : 1.234e-03
iter: 259, loss : 1.226e-03
iter: 260, loss : 1.079e-03
iter: 261, loss : 8.334e-04
iter: 262, loss : 8.680e-04
iter: 263, loss : 2.476e-03
iter: 264, loss : 6.226e-03
iter: 265, loss : 6.341e-04
iter: 266, loss : 1.031e-03
iter: 267, loss : 5.365e-04
iter: 268, loss : 8.801e-03
iter: 269, loss : 9.176e-04
iter: 270, loss : 7.622e-04
iter: 271, loss : 4.735e-03
iter: 272, loss : 1.900e-03
iter: 273, loss : 1.507e-03
iter: 274, loss : 5.565e-03
iter: 275, loss : 9.921e-04
iter: 276, loss : 8.265e-03
iter: 277, loss : 6.483e-03
iter: 278, loss : 6.894e-04
iter: 279, loss : 7.889e-03
iter: 280, loss : 3.595e-03
iter: 281, loss : 4.650e-04
iter: 282, loss : 1.003e-03
iter: 283, loss : 8.985e-04
iter: 284, loss : 2.536e-03
iter: 285, loss : 4.460e-04
iter: 286, loss : 5.389e-04
iter: 287, loss : 5.760e-03
iter: 288, loss : 1.078e-03
iter: 289, loss : 9.077e-04
iter: 290, loss : 1.640e-03
iter: 291, loss : 1.079e-03
iter: 292, loss : 1.187e-03
iter: 293, loss : 6.871e-04
iter: 294, loss : 9.093e-03
iter: 295, loss : 9.813e-04
iter: 296, loss : 6.375e-04
iter: 297, loss : 1.086e-03
iter: 298, loss : 1.233e-03
iter: 299, loss : 4.039e-03
iter: 300, loss : 4.666e-03
iter: 301, loss : 1.374e-03
iter: 302, loss : 5.740e-03
iter: 303, loss : 4.094e-04
iter: 304, loss : 4.811e-03
iter: 305, loss : 1.298e-03
iter: 306, loss : 4.216e-04
iter: 307, loss : 1.003e-03
iter: 308, loss : 6.812e-04
iter: 309, loss : 3.040e-03
iter: 310, loss : 1.008e-03
iter: 311, loss : 5.023e-04
iter: 312, loss : 5.564e-04
iter: 313, loss : 4.735e-03
iter: 314, loss : 6.831e-03
iter: 315, loss : 7.226e-04
iter: 316, loss : 6.688e-04
iter: 317, loss : 6.583e-04
iter: 318, loss : 5.128e-03
iter: 319, loss : 6.437e-03
iter: 320, loss : 8.829e-04
iter: 321, loss : 5.220e-03
iter: 322, loss : 1.111e-03
iter: 323, loss : 7.075e-04
iter: 324, loss : 3.115e-03
iter: 325, loss : 2.468e-03
iter: 326, loss : 1.106e-03
iter: 327, loss : 5.017e-04
iter: 328, loss : 4.693e-03
iter: 329, loss : 1.091e-03
iter: 330, loss : 1.112e-03
iter: 331, loss : 2.617e-03
iter: 332, loss : 5.529e-03
iter: 333, loss : 8.930e-04
iter: 334, loss : 9.089e-04
iter: 335, loss : 7.490e-04
iter: 336, loss : 2.938e-03
iter: 337, loss : 3.179e-03
iter: 338, loss : 8.819e-04
iter: 339, loss : 2.586e-03
iter: 340, loss : 2.037e-03
iter: 341, loss : 1.394e-03
iter: 342, loss : 2.802e-03
iter: 343, loss : 7.543e-04
iter: 344, loss : 4.488e-03
iter: 345, loss : 5.794e-04
iter: 346, loss : 9.182e-04
iter: 347, loss : 1.659e-03
iter: 348, loss : 6.734e-04
iter: 349, loss : 2.301e-04
iter: 350, loss : 1.011e-03
iter: 351, loss : 1.958e-03
iter: 352, loss : 2.876e-04
iter: 353, loss : 4.110e-03
iter: 354, loss : 5.663e-04
iter: 355, loss : 6.914e-04
iter: 356, loss : 1.107e-03
iter: 357, loss : 7.670e-04
iter: 358, loss : 3.246e-03
iter: 359, loss : 3.949e-04
iter: 360, loss : 3.701e-04
iter: 361, loss : 4.721e-04
iter: 362, loss : 6.131e-04
iter: 363, loss : 8.352e-04
iter: 364, loss : 1.444e-03
iter: 365, loss : 5.598e-04
iter: 366, loss : 3.491e-04
iter: 367, loss : 5.871e-04
iter: 368, loss : 4.813e-04
iter: 369, loss : 3.300e-03
iter: 370, loss : 5.743e-03
iter: 371, loss : 3.391e-03
iter: 372, loss : 1.843e-03
iter: 373, loss : 7.609e-04
iter: 374, loss : 1.964e-03
iter: 375, loss : 5.291e-03
iter: 376, loss : 2.669e-04
iter: 377, loss : 5.291e-03
iter: 378, loss : 9.874e-04
iter: 379, loss : 7.200e-04
iter: 380, loss : 9.040e-03
iter: 381, loss : 7.980e-04
iter: 382, loss : 6.012e-04
iter: 383, loss : 7.842e-03
iter: 384, loss : 8.528e-04
iter: 385, loss : 2.015e-03
iter: 386, loss : 8.057e-04
iter: 387, loss : 5.133e-04
iter: 388, loss : 5.945e-04
iter: 389, loss : 6.109e-04
iter: 390, loss : 1.818e-03
iter: 391, loss : 5.058e-03
iter: 392, loss : 6.640e-04
iter: 393, loss : 4.581e-04
iter: 394, loss : 3.798e-03
iter: 395, loss : 5.780e-04
iter: 396, loss : 2.416e-04
iter: 397, loss : 2.877e-04
iter: 398, loss : 6.219e-04
iter: 399, loss : 3.300e-03
iter: 400, loss : 5.975e-04
iter: 401, loss : 8.345e-04
iter: 402, loss : 6.811e-04
iter: 403, loss : 5.632e-04
iter: 404, loss : 6.113e-04
iter: 405, loss : 3.982e-04
iter: 406, loss : 2.737e-03
iter: 407, loss : 4.168e-03
iter: 408, loss : 4.297e-04
iter: 409, loss : 1.018e-03
iter: 410, loss : 5.610e-04
iter: 411, loss : 4.597e-04
iter: 412, loss : 1.020e-02
iter: 413, loss : 9.829e-03
iter: 414, loss : 7.891e-04
iter: 415, loss : 6.356e-04
iter: 416, loss : 5.601e-04
iter: 417, loss : 7.576e-04
iter: 418, loss : 4.666e-03
iter: 419, loss : 4.776e-04
iter: 420, loss : 1.055e-02
iter: 421, loss : 1.242e-03
iter: 422, loss : 2.827e-03
iter: 423, loss : 2.111e-03
iter: 424, loss : 4.379e-04
iter: 425, loss : 1.897e-03
iter: 426, loss : 7.178e-03
iter: 427, loss : 1.589e-03
iter: 428, loss : 4.332e-04
iter: 429, loss : 1.222e-03
iter: 430, loss : 2.031e-03
iter: 431, loss : 2.642e-03
iter: 432, loss : 7.939e-04
iter: 433, loss : 1.074e-03
iter: 434, loss : 3.043e-03
iter: 435, loss : 3.837e-03
iter: 436, loss : 3.082e-03
iter: 437, loss : 1.538e-03
iter: 438, loss : 5.943e-04
iter: 439, loss : 8.819e-04
iter: 440, loss : 6.733e-04
iter: 441, loss : 1.922e-03
iter: 442, loss : 8.299e-04
iter: 443, loss : 7.887e-03
iter: 444, loss : 3.884e-04
iter: 445, loss : 1.510e-03
iter: 446, loss : 1.244e-03
iter: 447, loss : 5.434e-04
iter: 448, loss : 6.462e-04
iter: 449, loss : 3.794e-03
iter: 450, loss : 7.467e-04
iter: 451, loss : 4.259e-04
iter: 452, loss : 7.119e-04
iter: 453, loss : 2.790e-04
iter: 454, loss : 1.130e-03
iter: 455, loss : 1.735e-03
iter: 456, loss : 2.507e-03
iter: 457, loss : 1.288e-03
iter: 458, loss : 7.969e-04
iter: 459, loss : 1.202e-03
iter: 460, loss : 7.282e-03
iter: 461, loss : 7.466e-04
iter: 462, loss : 2.447e-04
iter: 463, loss : 7.161e-04
iter: 464, loss : 3.788e-04
iter: 465, loss : 2.575e-03
iter: 466, loss : 6.844e-04
iter: 467, loss : 5.590e-04
iter: 468, loss : 3.152e-04
iter: 469, loss : 4.114e-03
iter: 470, loss : 9.685e-04
iter: 471, loss : 5.872e-04
iter: 472, loss : 3.970e-03
iter: 473, loss : 8.639e-04
iter: 474, loss : 4.421e-03
iter: 475, loss : 6.070e-04
iter: 476, loss : 2.968e-04
iter: 477, loss : 7.182e-04
iter: 478, loss : 7.240e-03
iter: 479, loss : 6.106e-04
iter: 480, loss : 1.623e-03
iter: 481, loss : 5.150e-04
iter: 482, loss : 1.983e-03
iter: 483, loss : 3.039e-04
iter: 484, loss : 3.440e-03
iter: 485, loss : 2.988e-03
iter: 486, loss : 7.683e-04
iter: 487, loss : 5.605e-04
iter: 488, loss : 9.302e-04
iter: 489, loss : 3.151e-04
iter: 490, loss : 2.472e-03
iter: 491, loss : 1.175e-04
iter: 492, loss : 3.469e-04
iter: 493, loss : 3.811e-04
iter: 494, loss : 2.333e-03
iter: 495, loss : 5.491e-04
iter: 496, loss : 3.387e-04
iter: 497, loss : 1.406e-04
iter: 498, loss : 7.134e-03
iter: 499, loss : 6.115e-04
iter: 500, loss : 6.315e-04
iter: 501, loss : 8.382e-04
iter: 502, loss : 2.569e-03
iter: 503, loss : 1.167e-03
iter: 504, loss : 4.223e-04
iter: 505, loss : 5.380e-04
iter: 506, loss : 7.169e-04
iter: 507, loss : 6.411e-04
iter: 508, loss : 7.033e-04
iter: 509, loss : 2.692e-03
iter: 510, loss : 7.047e-04
iter: 511, loss : 2.016e-04
iter: 512, loss : 1.782e-04
iter: 513, loss : 6.239e-03
iter: 514, loss : 4.590e-04
iter: 515, loss : 3.927e-04
iter: 516, loss : 1.061e-03
iter: 517, loss : 7.493e-03
iter: 518, loss : 5.505e-04
iter: 519, loss : 5.389e-04
iter: 520, loss : 3.023e-04
iter: 521, loss : 2.096e-03
iter: 522, loss : 6.246e-04
iter: 523, loss : 5.095e-04
iter: 524, loss : 8.918e-03
iter: 525, loss : 4.442e-04
iter: 526, loss : 2.340e-04
iter: 527, loss : 2.045e-03
iter: 528, loss : 2.702e-04
iter: 529, loss : 2.844e-04
iter: 530, loss : 1.599e-04
iter: 531, loss : 1.835e-03
iter: 532, loss : 2.582e-04
iter: 533, loss : 7.636e-04
iter: 534, loss : 5.188e-03
iter: 535, loss : 9.716e-03
iter: 536, loss : 1.608e-03
iter: 537, loss : 1.539e-03
iter: 538, loss : 5.511e-04
iter: 539, loss : 1.593e-03
iter: 540, loss : 5.860e-03
iter: 541, loss : 6.030e-03
iter: 542, loss : 7.542e-04
iter: 543, loss : 5.313e-04
iter: 544, loss : 1.033e-03
iter: 545, loss : 3.203e-03
iter: 546, loss : 1.500e-03
iter: 547, loss : 4.419e-04
iter: 548, loss : 1.929e-03
iter: 549, loss : 3.780e-04
iter: 550, loss : 4.808e-04
iter: 551, loss : 3.740e-04
iter: 552, loss : 3.233e-04
iter: 553, loss : 1.288e-03
iter: 554, loss : 8.549e-04
iter: 555, loss : 1.744e-04
iter: 556, loss : 4.422e-04
iter: 557, loss : 1.833e-03
iter: 558, loss : 7.777e-04
iter: 559, loss : 3.794e-03
iter: 560, loss : 3.791e-04
iter: 561, loss : 7.142e-04
iter: 562, loss : 2.620e-03
iter: 563, loss : 6.881e-04
iter: 564, loss : 5.349e-04
iter: 565, loss : 7.881e-04
iter: 566, loss : 6.061e-04
iter: 567, loss : 2.771e-04
iter: 568, loss : 3.680e-04
iter: 569, loss : 4.721e-04
iter: 570, loss : 2.104e-03
iter: 571, loss : 2.055e-04
iter: 572, loss : 2.436e-04
iter: 573, loss : 4.684e-04
iter: 574, loss : 2.290e-04
iter: 575, loss : 2.996e-04
iter: 576, loss : 6.635e-03
iter: 577, loss : 2.638e-04
iter: 578, loss : 5.066e-03
iter: 579, loss : 3.854e-03
iter: 580, loss : 3.602e-04
iter: 581, loss : 4.563e-03
iter: 582, loss : 4.588e-04
iter: 583, loss : 2.648e-03
iter: 584, loss : 4.720e-04
iter: 585, loss : 2.574e-04
iter: 586, loss : 2.165e-03
iter: 587, loss : 1.745e-03
iter: 588, loss : 5.942e-04
iter: 589, loss : 2.504e-03
iter: 590, loss : 1.769e-03
iter: 591, loss : 2.417e-03
iter: 592, loss : 2.727e-03
iter: 593, loss : 8.413e-04
iter: 594, loss : 2.469e-03
iter: 595, loss : 6.251e-04
iter: 596, loss : 5.160e-04
iter: 597, loss : 1.123e-03
iter: 598, loss : 1.823e-04
iter: 599, loss : 3.998e-03
iter: 600, loss : 4.534e-04
iter: 601, loss : 3.794e-03
iter: 602, loss : 4.844e-04
iter: 603, loss : 7.729e-04
iter: 604, loss : 5.942e-04
iter: 605, loss : 7.004e-04
iter: 606, loss : 8.169e-04
iter: 607, loss : 2.384e-03
iter: 608, loss : 4.547e-04
iter: 609, loss : 4.559e-04
iter: 610, loss : 2.074e-03
iter: 611, loss : 2.531e-03
iter: 612, loss : 1.300e-03
iter: 613, loss : 3.997e-04
iter: 614, loss : 3.853e-04
iter: 615, loss : 6.797e-04
iter: 616, loss : 2.440e-04
iter: 617, loss : 2.663e-04
iter: 618, loss : 5.685e-04
iter: 619, loss : 4.581e-04
iter: 620, loss : 1.013e-03
iter: 621, loss : 2.721e-04
iter: 622, loss : 1.454e-03
iter: 623, loss : 3.240e-04
iter: 624, loss : 2.415e-03
iter: 625, loss : 1.574e-04
iter: 626, loss : 1.655e-04
iter: 627, loss : 2.312e-03
iter: 628, loss : 1.270e-03
iter: 629, loss : 6.070e-04
iter: 630, loss : 1.658e-03
iter: 631, loss : 2.414e-04
iter: 632, loss : 3.105e-04
iter: 633, loss : 5.265e-04
iter: 634, loss : 2.302e-04
iter: 635, loss : 2.165e-03
iter: 636, loss : 1.719e-03
iter: 637, loss : 7.577e-03
iter: 638, loss : 1.039e-03
iter: 639, loss : 4.940e-04
iter: 640, loss : 8.923e-05
iter: 641, loss : 1.362e-04
iter: 642, loss : 1.659e-04
iter: 643, loss : 3.086e-04
iter: 644, loss : 4.064e-03
iter: 645, loss : 6.395e-05
iter: 646, loss : 9.206e-04
iter: 647, loss : 5.255e-04
iter: 648, loss : 2.326e-04
iter: 649, loss : 1.282e-03
iter: 650, loss : 4.190e-04
iter: 651, loss : 3.112e-04
iter: 652, loss : 3.952e-04
iter: 653, loss : 3.822e-04
iter: 654, loss : 4.025e-04
iter: 655, loss : 1.858e-04
iter: 656, loss : 4.391e-04
iter: 657, loss : 1.328e-04
iter: 658, loss : 3.857e-04
iter: 659, loss : 5.227e-03
iter: 660, loss : 3.929e-04
iter: 661, loss : 2.921e-04
iter: 662, loss : 3.518e-03
iter: 663, loss : 2.446e-03
iter: 664, loss : 3.918e-04
iter: 665, loss : 3.887e-04
iter: 666, loss : 1.027e-03
iter: 667, loss : 3.642e-04
iter: 668, loss : 7.703e-03
iter: 669, loss : 2.245e-04
iter: 670, loss : 4.874e-03
iter: 671, loss : 4.378e-04
iter: 672, loss : 1.743e-04
iter: 673, loss : 6.040e-04
iter: 674, loss : 6.844e-03
iter: 675, loss : 1.930e-03
iter: 676, loss : 2.159e-04
iter: 677, loss : 4.525e-04
iter: 678, loss : 1.023e-03
iter: 679, loss : 6.958e-04
iter: 680, loss : 1.433e-04
iter: 681, loss : 4.271e-04
iter: 682, loss : 1.675e-04
iter: 683, loss : 3.014e-04
iter: 684, loss : 3.009e-04
iter: 685, loss : 2.071e-03
iter: 686, loss : 5.559e-03
iter: 687, loss : 4.536e-04
iter: 688, loss : 3.734e-04
iter: 689, loss : 3.748e-04
iter: 690, loss : 3.159e-04
iter: 691, loss : 5.119e-04
iter: 692, loss : 3.277e-04
iter: 693, loss : 1.783e-03
iter: 694, loss : 2.227e-04
iter: 695, loss : 1.966e-04
iter: 696, loss : 2.836e-03
iter: 697, loss : 2.378e-04
iter: 698, loss : 2.807e-03
iter: 699, loss : 4.974e-04
iter: 700, loss : 2.821e-03
iter: 701, loss : 4.183e-04
iter: 702, loss : 3.288e-04
iter: 703, loss : 2.320e-03
iter: 704, loss : 2.207e-03
iter: 705, loss : 4.195e-04
iter: 706, loss : 3.274e-04
iter: 707, loss : 1.033e-03
iter: 708, loss : 5.896e-04
iter: 709, loss : 2.885e-04
iter: 710, loss : 7.317e-03
iter: 711, loss : 5.484e-04
iter: 712, loss : 5.438e-04
iter: 713, loss : 5.780e-04
iter: 714, loss : 3.037e-03
iter: 715, loss : 2.806e-04
iter: 716, loss : 2.474e-04
iter: 717, loss : 6.030e-04
iter: 718, loss : 1.075e-03
iter: 719, loss : 1.313e-03
iter: 720, loss : 4.515e-03
iter: 721, loss : 2.882e-04
iter: 722, loss : 2.892e-03
iter: 723, loss : 8.428e-04
iter: 724, loss : 4.481e-04
iter: 725, loss : 2.758e-03
iter: 726, loss : 1.480e-03
iter: 727, loss : 1.010e-03
iter: 728, loss : 1.138e-03
iter: 729, loss : 8.272e-04
iter: 730, loss : 6.735e-04
iter: 731, loss : 4.525e-04
iter: 732, loss : 2.120e-04
iter: 733, loss : 3.284e-04
iter: 734, loss : 1.537e-03
iter: 735, loss : 4.795e-03
iter: 736, loss : 3.184e-03
iter: 737, loss : 2.057e-04
iter: 738, loss : 5.171e-03
iter: 739, loss : 1.064e-03
iter: 740, loss : 3.272e-04
iter: 741, loss : 1.665e-03
iter: 742, loss : 1.245e-03
iter: 743, loss : 7.514e-04
iter: 744, loss : 1.563e-04
iter: 745, loss : 2.504e-04
iter: 746, loss : 6.363e-04
iter: 747, loss : 3.525e-04
iter: 748, loss : 4.242e-04
iter: 749, loss : 1.081e-03
iter: 750, loss : 1.537e-03
iter: 751, loss : 2.291e-04
iter: 752, loss : 5.303e-04
iter: 753, loss : 1.452e-04
iter: 754, loss : 5.552e-04
iter: 755, loss : 1.389e-03
iter: 756, loss : 2.553e-04
iter: 757, loss : 1.005e-03
iter: 758, loss : 3.838e-04
iter: 759, loss : 2.251e-04
iter: 760, loss : 1.144e-04
iter: 761, loss : 1.350e-04
iter: 762, loss : 1.353e-03
iter: 763, loss : 1.404e-04
iter: 764, loss : 2.440e-03
iter: 765, loss : 3.487e-04
iter: 766, loss : 4.304e-03
iter: 767, loss : 7.814e-04
iter: 768, loss : 3.275e-03
iter: 769, loss : 2.437e-04
iter: 770, loss : 2.035e-04
iter: 771, loss : 2.470e-04
iter: 772, loss : 2.023e-03
iter: 773, loss : 7.462e-04
iter: 774, loss : 1.149e-03
iter: 775, loss : 1.711e-03
iter: 776, loss : 2.144e-04
iter: 777, loss : 2.039e-04
iter: 778, loss : 1.187e-03
iter: 779, loss : 4.315e-03
iter: 780, loss : 1.467e-03
iter: 781, loss : 1.620e-03
iter: 782, loss : 1.519e-03
iter: 783, loss : 8.544e-04
iter: 784, loss : 4.484e-04
iter: 785, loss : 1.651e-04
iter: 786, loss : 3.450e-04
iter: 787, loss : 3.670e-04
iter: 788, loss : 6.467e-04
iter: 789, loss : 1.162e-03
iter: 790, loss : 7.349e-04
iter: 791, loss : 1.053e-03
iter: 792, loss : 3.461e-04
iter: 793, loss : 2.972e-03
iter: 794, loss : 2.661e-04
iter: 795, loss : 1.558e-03
iter: 796, loss : 2.778e-04
iter: 797, loss : 3.341e-04
iter: 798, loss : 2.642e-04
iter: 799, loss : 1.533e-04
iter: 800, loss : 2.459e-04
iter: 801, loss : 3.225e-04
iter: 802, loss : 1.775e-04
iter: 803, loss : 2.356e-04
iter: 804, loss : 3.472e-04
iter: 805, loss : 7.321e-04
iter: 806, loss : 3.828e-04
iter: 807, loss : 3.475e-04
iter: 808, loss : 3.642e-04
iter: 809, loss : 9.042e-05
iter: 810, loss : 4.317e-04
iter: 811, loss : 1.124e-03
iter: 812, loss : 1.410e-04
iter: 813, loss : 7.693e-04
iter: 814, loss : 1.998e-04
iter: 815, loss : 4.315e-03
iter: 816, loss : 2.971e-04
iter: 817, loss : 2.982e-03
iter: 818, loss : 1.948e-03
iter: 819, loss : 2.753e-03
iter: 820, loss : 1.560e-03
iter: 821, loss : 5.464e-04
iter: 822, loss : 8.197e-04
iter: 823, loss : 4.982e-04
iter: 824, loss : 5.183e-04
iter: 825, loss : 3.836e-04
iter: 826, loss : 4.607e-04
iter: 827, loss : 1.095e-03
iter: 828, loss : 4.300e-04
iter: 829, loss : 6.842e-03
iter: 830, loss : 9.821e-04
iter: 831, loss : 4.679e-04
iter: 832, loss : 4.289e-04
iter: 833, loss : 4.079e-04
iter: 834, loss : 4.813e-04
iter: 835, loss : 3.851e-04
iter: 836, loss : 5.185e-04
iter: 837, loss : 2.947e-04
iter: 838, loss : 3.832e-04
iter: 839, loss : 1.825e-04
iter: 840, loss : 9.080e-04
iter: 841, loss : 1.748e-04
iter: 842, loss : 1.934e-03
iter: 843, loss : 2.183e-04
iter: 844, loss : 2.072e-04
iter: 845, loss : 1.366e-04
iter: 846, loss : 3.538e-04
iter: 847, loss : 9.339e-04
iter: 848, loss : 1.276e-03
iter: 849, loss : 3.919e-04
iter: 850, loss : 1.928e-04
iter: 851, loss : 4.845e-04
iter: 852, loss : 2.709e-04
iter: 853, loss : 3.189e-04
iter: 854, loss : 4.684e-04
iter: 855, loss : 5.055e-04
iter: 856, loss : 9.798e-05
iter: 857, loss : 1.636e-04
iter: 858, loss : 2.259e-04
iter: 859, loss : 2.757e-04
iter: 860, loss : 6.680e-04
iter: 861, loss : 3.931e-04
iter: 862, loss : 1.644e-04
iter: 863, loss : 1.034e-03
iter: 864, loss : 1.409e-03
iter: 865, loss : 1.923e-04
iter: 866, loss : 4.981e-03
iter: 867, loss : 2.305e-04
iter: 868, loss : 4.588e-04
iter: 869, loss : 9.577e-04
iter: 870, loss : 2.182e-04
iter: 871, loss : 9.961e-04
iter: 872, loss : 5.854e-04
iter: 873, loss : 1.432e-03
iter: 874, loss : 6.355e-04
iter: 875, loss : 4.287e-03
iter: 876, loss : 1.129e-03
iter: 877, loss : 4.173e-04
iter: 878, loss : 3.922e-04
iter: 879, loss : 2.490e-03
iter: 880, loss : 2.892e-04
iter: 881, loss : 1.475e-03
iter: 882, loss : 9.164e-04
iter: 883, loss : 2.972e-04
iter: 884, loss : 4.632e-04
iter: 885, loss : 3.376e-04
iter: 886, loss : 8.280e-05
iter: 887, loss : 2.071e-03
iter: 888, loss : 3.401e-03
iter: 889, loss : 2.596e-03
iter: 890, loss : 6.026e-04
iter: 891, loss : 6.381e-04
iter: 892, loss : 7.022e-04
iter: 893, loss : 1.769e-03
iter: 894, loss : 8.260e-04
iter: 895, loss : 4.935e-04
iter: 896, loss : 4.354e-04
iter: 897, loss : 9.102e-04
iter: 898, loss : 3.127e-04
iter: 899, loss : 1.579e-04
iter: 900, loss : 5.909e-03
iter: 901, loss : 8.360e-04
iter: 902, loss : 8.640e-04
iter: 903, loss : 2.523e-04
iter: 904, loss : 4.226e-04
iter: 905, loss : 2.626e-03
iter: 906, loss : 1.924e-03
iter: 907, loss : 1.756e-03
iter: 908, loss : 3.807e-04
iter: 909, loss : 3.056e-04
iter: 910, loss : 4.847e-04
iter: 911, loss : 5.421e-04
iter: 912, loss : 2.378e-03
iter: 913, loss : 6.878e-04
iter: 914, loss : 3.650e-04
iter: 915, loss : 3.763e-04
iter: 916, loss : 1.036e-03
iter: 917, loss : 4.126e-04
iter: 918, loss : 1.037e-03
iter: 919, loss : 6.434e-04
iter: 920, loss : 2.085e-04
iter: 921, loss : 9.175e-04
iter: 922, loss : 4.165e-04
iter: 923, loss : 7.722e-04
iter: 924, loss : 4.907e-04
iter: 925, loss : 2.183e-04
iter: 926, loss : 9.735e-04
iter: 927, loss : 2.969e-04
iter: 928, loss : 6.476e-03
iter: 929, loss : 7.760e-04
iter: 930, loss : 2.905e-04
iter: 931, loss : 1.627e-03
iter: 932, loss : 2.926e-03
iter: 933, loss : 3.110e-04
iter: 934, loss : 9.114e-04
iter: 935, loss : 2.396e-04
iter: 936, loss : 7.556e-04
iter: 937, loss : 2.491e-04
iter: 938, loss : 1.753e-03
iter: 939, loss : 3.914e-04
iter: 940, loss : 3.907e-04
iter: 941, loss : 1.898e-04
iter: 942, loss : 2.433e-04
iter: 943, loss : 4.089e-04
iter: 944, loss : 9.145e-04
iter: 945, loss : 2.437e-04
iter: 946, loss : 3.442e-04
iter: 947, loss : 2.571e-04
iter: 948, loss : 4.873e-04
iter: 949, loss : 1.504e-03
iter: 950, loss : 2.476e-03
iter: 951, loss : 2.788e-04
iter: 952, loss : 3.790e-04
iter: 953, loss : 3.660e-04
iter: 954, loss : 1.241e-03
iter: 955, loss : 5.263e-04
iter: 956, loss : 2.019e-04
iter: 957, loss : 9.254e-04
iter: 958, loss : 6.731e-03
iter: 959, loss : 2.740e-04
iter: 960, loss : 2.360e-04
iter: 961, loss : 1.938e-04
iter: 962, loss : 5.645e-04
iter: 963, loss : 3.836e-04
iter: 964, loss : 6.955e-04
iter: 965, loss : 1.119e-03
iter: 966, loss : 2.776e-03
iter: 967, loss : 2.956e-04
iter: 968, loss : 2.934e-04
iter: 969, loss : 3.234e-04
iter: 970, loss : 2.984e-04
iter: 971, loss : 2.862e-04
iter: 972, loss : 4.632e-04
iter: 973, loss : 4.383e-03
iter: 974, loss : 2.843e-04
iter: 975, loss : 2.113e-04
iter: 976, loss : 3.041e-04
iter: 977, loss : 2.796e-03
iter: 978, loss : 3.932e-04
iter: 979, loss : 2.804e-04
iter: 980, loss : 1.963e-04
iter: 981, loss : 3.505e-04
iter: 982, loss : 3.854e-03
iter: 983, loss : 2.772e-03
iter: 984, loss : 9.725e-04
iter: 985, loss : 1.562e-03
iter: 986, loss : 2.737e-03
iter: 987, loss : 8.331e-04
iter: 988, loss : 2.157e-03
iter: 989, loss : 3.804e-04
iter: 990, loss : 2.279e-04
iter: 991, loss : 3.845e-04
iter: 992, loss : 7.747e-04
iter: 993, loss : 2.935e-04
iter: 994, loss : 2.281e-04
iter: 995, loss : 5.613e-03
iter: 996, loss : 2.079e-04
iter: 997, loss : 3.618e-04
iter: 998, loss : 4.123e-04
iter: 999, loss : 2.346e-03
iter: 1000, loss : 1.384e-04
iter: 1001, loss : 6.200e-04
iter: 1002, loss : 3.061e-04
iter: 1003, loss : 4.595e-04
iter: 1004, loss : 4.523e-04
iter: 1005, loss : 4.640e-04
iter: 1006, loss : 6.824e-03
iter: 1007, loss : 2.642e-04
iter: 1008, loss : 2.129e-04
iter: 1009, loss : 3.502e-03
iter: 1010, loss : 8.384e-04
iter: 1011, loss : 1.130e-03
iter: 1012, loss : 4.980e-04
iter: 1013, loss : 1.704e-04
iter: 1014, loss : 4.505e-04
iter: 1015, loss : 3.626e-04
iter: 1016, loss : 1.082e-03
iter: 1017, loss : 1.887e-03
iter: 1018, loss : 1.287e-03
iter: 1019, loss : 1.418e-04
iter: 1020, loss : 5.089e-03
iter: 1021, loss : 5.506e-04
iter: 1022, loss : 7.684e-04
iter: 1023, loss : 3.121e-04
iter: 1024, loss : 1.763e-04
iter: 1025, loss : 4.192e-04
iter: 1026, loss : 2.482e-04
iter: 1027, loss : 7.991e-04
iter: 1028, loss : 3.796e-04
iter: 1029, loss : 3.582e-04
iter: 1030, loss : 4.105e-04
iter: 1031, loss : 1.968e-04
iter: 1032, loss : 2.290e-04
iter: 1033, loss : 5.286e-03
iter: 1034, loss : 3.929e-04
iter: 1035, loss : 1.262e-03
iter: 1036, loss : 1.218e-03
iter: 1037, loss : 2.412e-04
iter: 1038, loss : 4.364e-04
iter: 1039, loss : 1.307e-03
iter: 1040, loss : 1.512e-04
iter: 1041, loss : 4.019e-04
iter: 1042, loss : 4.268e-04
iter: 1043, loss : 2.173e-04
iter: 1044, loss : 7.230e-03
iter: 1045, loss : 2.838e-04
iter: 1046, loss : 2.967e-04
iter: 1047, loss : 2.506e-04
iter: 1048, loss : 6.975e-04
iter: 1049, loss : 1.825e-04
iter: 1050, loss : 4.984e-04
iter: 1051, loss : 3.374e-04
iter: 1052, loss : 3.027e-04
iter: 1053, loss : 1.270e-03
iter: 1054, loss : 1.903e-03
iter: 1055, loss : 3.872e-04
iter: 1056, loss : 1.039e-03
iter: 1057, loss : 5.447e-04
iter: 1058, loss : 2.632e-03
iter: 1059, loss : 3.238e-04
iter: 1060, loss : 1.036e-04
iter: 1061, loss : 4.734e-04
iter: 1062, loss : 4.115e-04
iter: 1063, loss : 3.011e-04
iter: 1064, loss : 3.367e-04
iter: 1065, loss : 1.350e-04
iter: 1066, loss : 2.450e-04
iter: 1067, loss : 1.938e-04
iter: 1068, loss : 1.766e-04
iter: 1069, loss : 6.235e-05
iter: 1070, loss : 1.835e-04
iter: 1071, loss : 1.505e-04
iter: 1072, loss : 6.308e-04
iter: 1073, loss : 1.609e-03
iter: 1074, loss : 3.019e-04
iter: 1075, loss : 1.980e-04
iter: 1076, loss : 2.700e-04
iter: 1077, loss : 2.318e-04
iter: 1078, loss : 1.279e-04
iter: 1079, loss : 9.971e-04
iter: 1080, loss : 8.017e-04
iter: 1081, loss : 4.950e-04
iter: 1082, loss : 3.077e-04
iter: 1083, loss : 2.533e-04
iter: 1084, loss : 9.430e-05
iter: 1085, loss : 4.741e-04
iter: 1086, loss : 3.414e-04
iter: 1087, loss : 2.491e-04
iter: 1088, loss : 1.671e-04
iter: 1089, loss : 1.655e-03
iter: 1090, loss : 4.471e-04
iter: 1091, loss : 2.746e-04
iter: 1092, loss : 1.563e-04
iter: 1093, loss : 9.392e-05
iter: 1094, loss : 1.192e-04
iter: 1095, loss : 2.980e-03
iter: 1096, loss : 1.804e-03
iter: 1097, loss : 3.039e-04
iter: 1098, loss : 2.762e-04
iter: 1099, loss : 2.181e-04
iter: 1100, loss : 1.088e-03
iter: 1101, loss : 2.111e-04
iter: 1102, loss : 8.595e-04
iter: 1103, loss : 1.408e-04
iter: 1104, loss : 1.801e-03
iter: 1105, loss : 1.888e-03
iter: 1106, loss : 1.495e-04
iter: 1107, loss : 2.645e-04
iter: 1108, loss : 2.902e-04
iter: 1109, loss : 3.302e-04
iter: 1110, loss : 1.420e-04
iter: 1111, loss : 3.202e-04
iter: 1112, loss : 4.302e-04
iter: 1113, loss : 2.335e-04
iter: 1114, loss : 2.107e-04
iter: 1115, loss : 2.187e-04
iter: 1116, loss : 8.585e-04
iter: 1117, loss : 1.415e-04
iter: 1118, loss : 1.802e-04
iter: 1119, loss : 6.944e-05
iter: 1120, loss : 1.036e-04
iter: 1121, loss : 2.067e-04
iter: 1122, loss : 7.393e-05
iter: 1123, loss : 8.331e-05
iter: 1124, loss : 6.522e-05
iter: 1125, loss : 1.797e-04
iter: 1126, loss : 1.409e-04
iter: 1127, loss : 1.359e-04
iter: 1128, loss : 1.891e-04
iter: 1129, loss : 1.109e-03
iter: 1130, loss : 5.270e-03
iter: 1131, loss : 2.799e-04
iter: 1132, loss : 1.219e-03
iter: 1133, loss : 1.382e-04
iter: 1134, loss : 1.461e-04
iter: 1135, loss : 5.850e-03
iter: 1136, loss : 8.773e-04
iter: 1137, loss : 1.657e-04
iter: 1138, loss : 7.381e-05
iter: 1139, loss : 6.073e-04
iter: 1140, loss : 3.910e-04
iter: 1141, loss : 8.719e-04
iter: 1142, loss : 4.762e-04
iter: 1143, loss : 2.467e-04
iter: 1144, loss : 3.094e-04
iter: 1145, loss : 2.175e-04
iter: 1146, loss : 1.775e-04
iter: 1147, loss : 1.299e-04
iter: 1148, loss : 3.360e-03
iter: 1149, loss : 2.055e-04
iter: 1150, loss : 2.007e-03
iter: 1151, loss : 1.720e-04
iter: 1152, loss : 1.161e-03
iter: 1153, loss : 2.308e-04
iter: 1154, loss : 5.623e-03
iter: 1155, loss : 3.256e-04
iter: 1156, loss : 3.161e-04
iter: 1157, loss : 3.014e-04
iter: 1158, loss : 4.800e-04
iter: 1159, loss : 3.055e-04
iter: 1160, loss : 4.036e-04
iter: 1161, loss : 3.598e-04
iter: 1162, loss : 2.094e-04
iter: 1163, loss : 1.696e-03
iter: 1164, loss : 5.395e-04
iter: 1165, loss : 2.604e-04
iter: 1166, loss : 3.534e-03
iter: 1167, loss : 2.625e-04
iter: 1168, loss : 9.301e-04
iter: 1169, loss : 1.324e-04
iter: 1170, loss : 2.677e-04
iter: 1171, loss : 2.303e-03
iter: 1172, loss : 5.214e-04
iter: 1173, loss : 2.994e-04
iter: 1174, loss : 1.617e-03
iter: 1175, loss : 5.663e-03
iter: 1176, loss : 9.801e-04
iter: 1177, loss : 7.500e-04
iter: 1178, loss : 2.673e-03
iter: 1179, loss : 4.410e-04
iter: 1180, loss : 3.205e-04
iter: 1181, loss : 1.665e-04
iter: 1182, loss : 2.830e-04
iter: 1183, loss : 3.011e-04
iter: 1184, loss : 1.466e-04
iter: 1185, loss : 1.949e-03
iter: 1186, loss : 2.556e-04
iter: 1187, loss : 1.161e-03
iter: 1188, loss : 1.620e-03
iter: 1189, loss : 4.137e-04
iter: 1190, loss : 1.559e-03
iter: 1191, loss : 3.077e-04
iter: 1192, loss : 1.187e-03
iter: 1193, loss : 1.601e-04
iter: 1194, loss : 1.532e-03
iter: 1195, loss : 2.279e-04
iter: 1196, loss : 1.798e-03
iter: 1197, loss : 2.032e-04
iter: 1198, loss : 1.269e-03
iter: 1199, loss : 6.011e-04
iter: 1200, loss : 1.880e-04
iter: 1201, loss : 6.116e-04
iter: 1202, loss : 2.964e-04
iter: 1203, loss : 1.519e-04
iter: 1204, loss : 1.422e-03
iter: 1205, loss : 1.201e-04
iter: 1206, loss : 7.509e-05
iter: 1207, loss : 5.041e-03
iter: 1208, loss : 4.858e-04
iter: 1209, loss : 2.537e-04
iter: 1210, loss : 4.075e-04
iter: 1211, loss : 2.285e-04
iter: 1212, loss : 2.036e-04
iter: 1213, loss : 1.016e-03
iter: 1214, loss : 1.779e-04
iter: 1215, loss : 3.564e-04
iter: 1216, loss : 2.598e-04
iter: 1217, loss : 1.252e-04
iter: 1218, loss : 8.597e-04
iter: 1219, loss : 1.721e-04
iter: 1220, loss : 9.464e-04
iter: 1221, loss : 3.698e-04
iter: 1222, loss : 1.754e-04
iter: 1223, loss : 1.568e-04
iter: 1224, loss : 8.890e-04
iter: 1225, loss : 2.033e-04
iter: 1226, loss : 2.490e-04
iter: 1227, loss : 8.809e-04
iter: 1228, loss : 2.858e-04
iter: 1229, loss : 2.020e-04
iter: 1230, loss : 4.475e-03
iter: 1231, loss : 2.032e-04
iter: 1232, loss : 3.265e-04
iter: 1233, loss : 4.253e-03
iter: 1234, loss : 2.188e-04
iter: 1235, loss : 4.131e-04
iter: 1236, loss : 2.499e-04
iter: 1237, loss : 3.606e-04
iter: 1238, loss : 1.127e-03
iter: 1239, loss : 9.592e-04
iter: 1240, loss : 7.479e-04
iter: 1241, loss : 5.801e-04
iter: 1242, loss : 1.609e-04
iter: 1243, loss : 3.150e-04
iter: 1244, loss : 3.939e-03
iter: 1245, loss : 1.080e-03
iter: 1246, loss : 2.176e-04
iter: 1247, loss : 2.869e-03
iter: 1248, loss : 4.226e-03
iter: 1249, loss : 2.717e-04
iter: 1250, loss : 2.074e-03
iter: 1251, loss : 4.257e-03
iter: 1252, loss : 5.402e-04
iter: 1253, loss : 3.187e-04
iter: 1254, loss : 3.705e-04
iter: 1255, loss : 4.404e-04
iter: 1256, loss : 3.496e-04
iter: 1257, loss : 2.619e-04
iter: 1258, loss : 8.649e-04
iter: 1259, loss : 1.025e-03
iter: 1260, loss : 1.660e-03
iter: 1261, loss : 3.470e-04
iter: 1262, loss : 5.356e-04
iter: 1263, loss : 1.560e-03
iter: 1264, loss : 2.120e-04
iter: 1265, loss : 4.543e-04
iter: 1266, loss : 5.373e-04
iter: 1267, loss : 1.389e-04
iter: 1268, loss : 2.005e-03
iter: 1269, loss : 4.380e-04
iter: 1270, loss : 1.770e-04
iter: 1271, loss : 1.821e-03
iter: 1272, loss : 5.507e-04
iter: 1273, loss : 3.467e-04
iter: 1274, loss : 4.309e-04
iter: 1275, loss : 2.142e-03
iter: 1276, loss : 3.362e-04
iter: 1277, loss : 3.967e-04
iter: 1278, loss : 6.180e-04
iter: 1279, loss : 6.073e-04
iter: 1280, loss : 1.614e-04
iter: 1281, loss : 1.580e-04
iter: 1282, loss : 4.626e-04
iter: 1283, loss : 2.535e-04
iter: 1284, loss : 2.522e-04
iter: 1285, loss : 2.069e-04
iter: 1286, loss : 1.309e-04
iter: 1287, loss : 2.360e-03
iter: 1288, loss : 1.798e-04
iter: 1289, loss : 2.973e-04
iter: 1290, loss : 8.711e-05
iter: 1291, loss : 1.012e-04
iter: 1292, loss : 1.002e-03
iter: 1293, loss : 1.752e-04
iter: 1294, loss : 2.100e-04
iter: 1295, loss : 2.628e-04
iter: 1296, loss : 2.225e-04
iter: 1297, loss : 3.222e-04
iter: 1298, loss : 1.385e-04
iter: 1299, loss : 3.817e-04
iter: 1300, loss : 1.698e-04
iter: 1301, loss : 1.620e-04
iter: 1302, loss : 8.729e-04
iter: 1303, loss : 5.129e-04
iter: 1304, loss : 9.465e-05
iter: 1305, loss : 1.340e-03
iter: 1306, loss : 6.643e-04
iter: 1307, loss : 3.071e-04
iter: 1308, loss : 1.293e-04
iter: 1309, loss : 1.293e-04
iter: 1310, loss : 8.739e-04
iter: 1311, loss : 2.334e-04
iter: 1312, loss : 2.438e-04
iter: 1313, loss : 1.542e-04
iter: 1314, loss : 1.560e-04
iter: 1315, loss : 7.331e-05
iter: 1316, loss : 2.519e-03
iter: 1317, loss : 1.679e-04
iter: 1318, loss : 3.730e-04
iter: 1319, loss : 2.186e-04
iter: 1320, loss : 9.159e-05
iter: 1321, loss : 8.406e-04
iter: 1322, loss : 3.090e-04
iter: 1323, loss : 1.995e-04
iter: 1324, loss : 2.894e-04
iter: 1325, loss : 1.365e-04
iter: 1326, loss : 2.479e-04
iter: 1327, loss : 6.861e-05
iter: 1328, loss : 2.005e-04
iter: 1329, loss : 1.206e-03
iter: 1330, loss : 1.743e-04
iter: 1331, loss : 2.235e-03
iter: 1332, loss : 4.540e-04
iter: 1333, loss : 1.231e-04
iter: 1334, loss : 4.086e-04
iter: 1335, loss : 8.706e-05
iter: 1336, loss : 8.707e-04
iter: 1337, loss : 2.745e-04
iter: 1338, loss : 1.428e-04
iter: 1339, loss : 2.176e-04
iter: 1340, loss : 7.572e-05
iter: 1341, loss : 1.490e-04
iter: 1342, loss : 1.158e-03
iter: 1343, loss : 1.200e-04
iter: 1344, loss : 9.017e-04
iter: 1345, loss : 2.147e-04
iter: 1346, loss : 6.407e-03
iter: 1347, loss : 1.440e-04
iter: 1348, loss : 1.853e-04
iter: 1349, loss : 5.124e-03
iter: 1350, loss : 1.013e-04
iter: 1351, loss : 5.682e-04
iter: 1352, loss : 1.726e-04
iter: 1353, loss : 1.375e-04
iter: 1354, loss : 1.449e-04
iter: 1355, loss : 2.163e-04
iter: 1356, loss : 4.241e-04
iter: 1357, loss : 1.334e-04
iter: 1358, loss : 1.761e-04
iter: 1359, loss : 1.931e-03
iter: 1360, loss : 3.916e-04
iter: 1361, loss : 3.023e-04
iter: 1362, loss : 1.155e-04
iter: 1363, loss : 1.104e-04
iter: 1364, loss : 1.637e-04
iter: 1365, loss : 1.813e-04
iter: 1366, loss : 1.649e-04
iter: 1367, loss : 1.258e-04
iter: 1368, loss : 9.442e-05
iter: 1369, loss : 1.234e-04
iter: 1370, loss : 1.685e-04
iter: 1371, loss : 9.241e-04
iter: 1372, loss : 1.260e-04
iter: 1373, loss : 1.593e-04
iter: 1374, loss : 1.084e-04
iter: 1375, loss : 2.634e-03
iter: 1376, loss : 4.334e-04
iter: 1377, loss : 2.108e-04
iter: 1378, loss : 2.231e-04
iter: 1379, loss : 2.145e-03
iter: 1380, loss : 1.174e-04
iter: 1381, loss : 1.078e-04
iter: 1382, loss : 3.328e-04
iter: 1383, loss : 1.513e-04
iter: 1384, loss : 3.696e-03
iter: 1385, loss : 1.043e-03
iter: 1386, loss : 3.965e-04
iter: 1387, loss : 1.136e-03
iter: 1388, loss : 1.652e-04
iter: 1389, loss : 1.206e-03
iter: 1390, loss : 1.059e-03
iter: 1391, loss : 1.032e-03
iter: 1392, loss : 2.505e-03
iter: 1393, loss : 2.188e-04
iter: 1394, loss : 1.370e-03
iter: 1395, loss : 1.852e-04
iter: 1396, loss : 3.246e-04
iter: 1397, loss : 1.101e-03
iter: 1398, loss : 2.402e-04
iter: 1399, loss : 4.488e-04
iter: 1400, loss : 1.776e-03
iter: 1401, loss : 1.918e-03
iter: 1402, loss : 3.958e-04
iter: 1403, loss : 3.243e-04
iter: 1404, loss : 3.141e-04
iter: 1405, loss : 1.926e-04
iter: 1406, loss : 1.452e-04
iter: 1407, loss : 4.169e-03
iter: 1408, loss : 2.430e-04
iter: 1409, loss : 2.045e-03
iter: 1410, loss : 2.535e-04
iter: 1411, loss : 3.225e-04
iter: 1412, loss : 1.058e-04
iter: 1413, loss : 8.794e-04
iter: 1414, loss : 2.798e-04
iter: 1415, loss : 6.915e-05
iter: 1416, loss : 7.104e-04
iter: 1417, loss : 6.776e-04
iter: 1418, loss : 2.599e-04
iter: 1419, loss : 1.070e-04
iter: 1420, loss : 4.071e-04
iter: 1421, loss : 3.646e-04
iter: 1422, loss : 2.606e-03
iter: 1423, loss : 9.962e-04
iter: 1424, loss : 3.554e-04
iter: 1425, loss : 7.545e-05
iter: 1426, loss : 3.114e-03
iter: 1427, loss : 1.120e-03
iter: 1428, loss : 4.366e-03
iter: 1429, loss : 4.077e-04
iter: 1430, loss : 1.735e-04
iter: 1431, loss : 2.631e-04
iter: 1432, loss : 7.215e-04
iter: 1433, loss : 2.616e-04
iter: 1434, loss : 3.052e-03
iter: 1435, loss : 1.015e-03
iter: 1436, loss : 3.665e-04
iter: 1437, loss : 5.795e-04
iter: 1438, loss : 3.154e-04
iter: 1439, loss : 7.825e-04
iter: 1440, loss : 6.611e-03
iter: 1441, loss : 5.001e-04
iter: 1442, loss : 2.763e-04
iter: 1443, loss : 1.009e-03
iter: 1444, loss : 1.364e-04
iter: 1445, loss : 3.084e-04
iter: 1446, loss : 3.229e-03
iter: 1447, loss : 1.670e-04
iter: 1448, loss : 2.881e-04
iter: 1449, loss : 3.344e-04
iter: 1450, loss : 6.929e-04
iter: 1451, loss : 1.126e-04
iter: 1452, loss : 3.842e-04
iter: 1453, loss : 2.561e-04
iter: 1454, loss : 2.165e-04
iter: 1455, loss : 3.804e-04
iter: 1456, loss : 7.477e-04
iter: 1457, loss : 2.832e-04
iter: 1458, loss : 1.280e-04
iter: 1459, loss : 2.339e-04
iter: 1460, loss : 2.086e-04
iter: 1461, loss : 7.778e-04
iter: 1462, loss : 8.303e-04
iter: 1463, loss : 7.841e-04
iter: 1464, loss : 7.439e-04
iter: 1465, loss : 3.542e-04
iter: 1466, loss : 1.532e-04
iter: 1467, loss : 5.519e-04
iter: 1468, loss : 2.005e-04
iter: 1469, loss : 1.529e-03
iter: 1470, loss : 9.311e-04
iter: 1471, loss : 1.028e-03
iter: 1472, loss : 4.909e-04
iter: 1473, loss : 7.709e-04
iter: 1474, loss : 1.210e-03
iter: 1475, loss : 6.790e-04
iter: 1476, loss : 2.841e-04
iter: 1477, loss : 1.229e-04
iter: 1478, loss : 1.959e-04
iter: 1479, loss : 1.543e-04
iter: 1480, loss : 9.887e-04
iter: 1481, loss : 6.643e-04
iter: 1482, loss : 8.557e-04
iter: 1483, loss : 6.201e-04
iter: 1484, loss : 3.037e-03
iter: 1485, loss : 2.192e-04
iter: 1486, loss : 4.628e-04
iter: 1487, loss : 6.958e-05
iter: 1488, loss : 5.131e-04
iter: 1489, loss : 2.590e-04
iter: 1490, loss : 1.201e-04
iter: 1491, loss : 3.858e-04
iter: 1492, loss : 7.483e-04
iter: 1493, loss : 2.389e-04
iter: 1494, loss : 6.883e-04
iter: 1495, loss : 2.035e-04
iter: 1496, loss : 1.060e-03
iter: 1497, loss : 9.436e-04
iter: 1498, loss : 1.556e-03
iter: 1499, loss : 2.817e-04
iter: 1500, loss : 1.631e-04
iter: 1501, loss : 1.244e-03
iter: 1502, loss : 2.382e-04
iter: 1503, loss : 2.907e-04
iter: 1504, loss : 6.084e-04
iter: 1505, loss : 2.595e-04
iter: 1506, loss : 2.170e-03
iter: 1507, loss : 2.484e-04
iter: 1508, loss : 2.894e-04
iter: 1509, loss : 4.136e-04
iter: 1510, loss : 6.364e-04
iter: 1511, loss : 2.072e-04
iter: 1512, loss : 2.058e-04
iter: 1513, loss : 2.217e-04
iter: 1514, loss : 2.314e-04
iter: 1515, loss : 3.065e-04
iter: 1516, loss : 1.023e-04
iter: 1517, loss : 4.955e-04
iter: 1518, loss : 1.016e-03
iter: 1519, loss : 1.012e-04
iter: 1520, loss : 2.219e-04
iter: 1521, loss : 1.098e-04
iter: 1522, loss : 1.919e-04
iter: 1523, loss : 1.946e-04
iter: 1524, loss : 5.035e-04
iter: 1525, loss : 1.065e-04
iter: 1526, loss : 3.178e-04
iter: 1527, loss : 1.238e-04
iter: 1528, loss : 2.554e-04
iter: 1529, loss : 2.537e-04
iter: 1530, loss : 1.314e-04
iter: 1531, loss : 1.345e-04
iter: 1532, loss : 8.996e-05
iter: 1533, loss : 4.839e-05
iter: 1534, loss : 1.061e-04
iter: 1535, loss : 9.471e-05
iter: 1536, loss : 1.588e-04
iter: 1537, loss : 7.535e-05
iter: 1538, loss : 1.210e-04
iter: 1539, loss : 8.542e-04
iter: 1540, loss : 9.883e-05
iter: 1541, loss : 9.588e-05
iter: 1542, loss : 6.983e-05
iter: 1543, loss : 8.841e-05
iter: 1544, loss : 9.102e-05
iter: 1545, loss : 7.190e-04
iter: 1546, loss : 1.342e-03
iter: 1547, loss : 7.306e-04
iter: 1548, loss : 6.386e-04
iter: 1549, loss : 1.900e-04
iter: 1550, loss : 7.146e-04
iter: 1551, loss : 2.505e-04
iter: 1552, loss : 1.840e-04
iter: 1553, loss : 8.827e-04
iter: 1554, loss : 1.511e-04
iter: 1555, loss : 7.695e-04
iter: 1556, loss : 3.807e-04
iter: 1557, loss : 3.250e-04
iter: 1558, loss : 1.962e-04
iter: 1559, loss : 1.408e-04
iter: 1560, loss : 5.016e-03
iter: 1561, loss : 5.291e-04
iter: 1562, loss : 5.429e-03
iter: 1563, loss : 1.363e-04
iter: 1564, loss : 4.735e-04
iter: 1565, loss : 3.776e-04
iter: 1566, loss : 4.099e-04
iter: 1567, loss : 2.014e-04
iter: 1568, loss : 3.644e-04
iter: 1569, loss : 1.357e-04
iter: 1570, loss : 2.547e-04
iter: 1571, loss : 3.346e-03
iter: 1572, loss : 5.366e-04
iter: 1573, loss : 3.250e-03
iter: 1574, loss : 7.021e-04
iter: 1575, loss : 7.093e-04
iter: 1576, loss : 1.479e-04
iter: 1577, loss : 6.965e-04
iter: 1578, loss : 3.189e-04
iter: 1579, loss : 3.024e-04
iter: 1580, loss : 2.065e-04
iter: 1581, loss : 1.121e-04
iter: 1582, loss : 2.127e-04
iter: 1583, loss : 1.450e-04
iter: 1584, loss : 2.490e-03
iter: 1585, loss : 1.140e-03
iter: 1586, loss : 1.982e-04
iter: 1587, loss : 2.804e-04
iter: 1588, loss : 1.219e-03
iter: 1589, loss : 3.718e-04
iter: 1590, loss : 1.501e-03
iter: 1591, loss : 2.264e-04
iter: 1592, loss : 3.514e-03
iter: 1593, loss : 1.024e-03
iter: 1594, loss : 3.468e-04
iter: 1595, loss : 6.457e-04
iter: 1596, loss : 2.397e-04
iter: 1597, loss : 3.008e-04
iter: 1598, loss : 2.255e-04
iter: 1599, loss : 1.557e-03
iter: 1600, loss : 6.655e-04
iter: 1601, loss : 1.060e-03
iter: 1602, loss : 1.791e-04
iter: 1603, loss : 2.626e-04
iter: 1604, loss : 3.685e-04
iter: 1605, loss : 1.035e-03
iter: 1606, loss : 9.104e-04
iter: 1607, loss : 2.806e-04
iter: 1608, loss : 5.817e-04
iter: 1609, loss : 3.801e-04
iter: 1610, loss : 2.190e-04
iter: 1611, loss : 2.063e-04
iter: 1612, loss : 3.702e-04
iter: 1613, loss : 8.177e-04
iter: 1614, loss : 1.390e-03
iter: 1615, loss : 2.554e-04
iter: 1616, loss : 1.363e-04
iter: 1617, loss : 4.898e-04
iter: 1618, loss : 2.055e-04
iter: 1619, loss : 1.366e-04
iter: 1620, loss : 4.806e-04
iter: 1621, loss : 4.183e-05
iter: 1622, loss : 9.615e-05
iter: 1623, loss : 2.256e-04
iter: 1624, loss : 2.024e-04
iter: 1625, loss : 1.165e-03
iter: 1626, loss : 2.020e-04
iter: 1627, loss : 2.774e-04
iter: 1628, loss : 7.265e-04
iter: 1629, loss : 1.372e-04
iter: 1630, loss : 1.498e-03
iter: 1631, loss : 1.359e-04
iter: 1632, loss : 1.184e-04
iter: 1633, loss : 3.661e-04
iter: 1634, loss : 9.935e-05
iter: 1635, loss : 1.215e-04
iter: 1636, loss : 2.585e-04
iter: 1637, loss : 5.466e-04
iter: 1638, loss : 6.653e-04
iter: 1639, loss : 2.550e-04
iter: 1640, loss : 6.645e-05
iter: 1641, loss : 3.107e-04
iter: 1642, loss : 2.028e-04
iter: 1643, loss : 1.515e-04
iter: 1644, loss : 3.688e-03
iter: 1645, loss : 3.079e-03
iter: 1646, loss : 9.245e-04
iter: 1647, loss : 2.735e-04
iter: 1648, loss : 2.827e-04
iter: 1649, loss : 3.052e-04
iter: 1650, loss : 6.316e-04
iter: 1651, loss : 3.051e-03
iter: 1652, loss : 2.111e-04
iter: 1653, loss : 8.104e-04
iter: 1654, loss : 1.771e-04
iter: 1655, loss : 8.877e-05
iter: 1656, loss : 1.453e-04
iter: 1657, loss : 1.653e-04
iter: 1658, loss : 6.676e-04
iter: 1659, loss : 1.707e-04
iter: 1660, loss : 1.543e-04
iter: 1661, loss : 1.093e-03
iter: 1662, loss : 1.023e-03
iter: 1663, loss : 8.507e-04
iter: 1664, loss : 1.841e-03
iter: 1665, loss : 9.460e-04
iter: 1666, loss : 2.178e-04
iter: 1667, loss : 5.349e-03
iter: 1668, loss : 1.083e-03
iter: 1669, loss : 5.757e-04
iter: 1670, loss : 3.123e-04
iter: 1671, loss : 1.731e-04
iter: 1672, loss : 6.380e-04
iter: 1673, loss : 4.873e-04
iter: 1674, loss : 6.535e-04
iter: 1675, loss : 2.087e-03
iter: 1676, loss : 9.902e-05
iter: 1677, loss : 1.753e-04
iter: 1678, loss : 2.574e-04
iter: 1679, loss : 3.097e-04
iter: 1680, loss : 2.141e-04
iter: 1681, loss : 2.375e-04
iter: 1682, loss : 1.344e-04
iter: 1683, loss : 1.106e-04
iter: 1684, loss : 7.714e-04
iter: 1685, loss : 1.940e-04
iter: 1686, loss : 2.365e-04
iter: 1687, loss : 1.019e-04
iter: 1688, loss : 1.313e-03
iter: 1689, loss : 1.307e-04
iter: 1690, loss : 4.011e-04
iter: 1691, loss : 2.034e-04
iter: 1692, loss : 2.150e-04
iter: 1693, loss : 1.069e-04
iter: 1694, loss : 7.098e-05
iter: 1695, loss : 1.145e-04
iter: 1696, loss : 8.708e-04
iter: 1697, loss : 7.509e-05
iter: 1698, loss : 1.110e-04
iter: 1699, loss : 1.690e-04
iter: 1700, loss : 2.012e-04
iter: 1701, loss : 4.887e-04
iter: 1702, loss : 5.380e-04
iter: 1703, loss : 1.233e-04
iter: 1704, loss : 1.980e-04
iter: 1705, loss : 1.092e-04
iter: 1706, loss : 2.862e-04
iter: 1707, loss : 1.324e-04
iter: 1708, loss : 6.434e-05
iter: 1709, loss : 1.430e-04
iter: 1710, loss : 1.636e-03
iter: 1711, loss : 3.725e-04
iter: 1712, loss : 1.140e-04
iter: 1713, loss : 1.613e-04
iter: 1714, loss : 2.173e-04
iter: 1715, loss : 1.838e-04
iter: 1716, loss : 6.395e-04
iter: 1717, loss : 8.289e-05
iter: 1718, loss : 6.467e-05
iter: 1719, loss : 1.267e-04
iter: 1720, loss : 6.040e-04
iter: 1721, loss : 1.226e-04
iter: 1722, loss : 1.512e-04
iter: 1723, loss : 1.969e-04
iter: 1724, loss : 1.616e-04
iter: 1725, loss : 6.347e-04
iter: 1726, loss : 4.363e-03
iter: 1727, loss : 2.802e-04
iter: 1728, loss : 1.599e-04
iter: 1729, loss : 1.042e-04
iter: 1730, loss : 2.125e-04
iter: 1731, loss : 2.164e-04
iter: 1732, loss : 1.645e-04
iter: 1733, loss : 3.141e-04
iter: 1734, loss : 5.182e-04
iter: 1735, loss : 3.140e-04
iter: 1736, loss : 5.060e-05
iter: 1737, loss : 1.576e-04
iter: 1738, loss : 2.441e-04
iter: 1739, loss : 2.868e-04
iter: 1740, loss : 1.157e-04
iter: 1741, loss : 5.212e-05
iter: 1742, loss : 1.061e-04
iter: 1743, loss : 2.056e-04
iter: 1744, loss : 2.186e-03
iter: 1745, loss : 8.123e-05
iter: 1746, loss : 1.926e-04
iter: 1747, loss : 1.661e-04
iter: 1748, loss : 7.601e-04
iter: 1749, loss : 2.166e-04
iter: 1750, loss : 1.857e-04
iter: 1751, loss : 1.386e-04
iter: 1752, loss : 1.518e-04
iter: 1753, loss : 1.451e-04
iter: 1754, loss : 8.317e-05
iter: 1755, loss : 1.652e-04
iter: 1756, loss : 1.686e-04
iter: 1757, loss : 1.005e-04
iter: 1758, loss : 9.070e-05
iter: 1759, loss : 6.252e-05
iter: 1760, loss : 7.173e-04
iter: 1761, loss : 6.977e-05
iter: 1762, loss : 1.142e-04
iter: 1763, loss : 4.454e-03
iter: 1764, loss : 1.137e-04
iter: 1765, loss : 1.076e-04
iter: 1766, loss : 9.040e-04
iter: 1767, loss : 3.031e-04
iter: 1768, loss : 8.066e-04
iter: 1769, loss : 3.734e-04
iter: 1770, loss : 4.865e-03
iter: 1771, loss : 3.302e-04
iter: 1772, loss : 3.903e-04
iter: 1773, loss : 3.539e-04
iter: 1774, loss : 1.116e-04
iter: 1775, loss : 1.783e-03
iter: 1776, loss : 1.486e-04
iter: 1777, loss : 3.410e-04
iter: 1778, loss : 1.884e-04
iter: 1779, loss : 1.324e-04
iter: 1780, loss : 2.480e-04
iter: 1781, loss : 8.056e-05
iter: 1782, loss : 2.536e-04
iter: 1783, loss : 1.019e-04
iter: 1784, loss : 1.644e-04
iter: 1785, loss : 2.075e-04
iter: 1786, loss : 2.866e-04
iter: 1787, loss : 1.714e-04
iter: 1788, loss : 4.368e-04
iter: 1789, loss : 2.034e-04
iter: 1790, loss : 1.612e-04
iter: 1791, loss : 9.637e-04
iter: 1792, loss : 2.399e-04
iter: 1793, loss : 1.083e-04
iter: 1794, loss : 1.132e-03
iter: 1795, loss : 1.808e-04
iter: 1796, loss : 3.366e-03
iter: 1797, loss : 1.261e-03
iter: 1798, loss : 4.302e-04
iter: 1799, loss : 3.299e-04
iter: 1800, loss : 1.072e-04
iter: 1801, loss : 1.599e-04
iter: 1802, loss : 5.365e-04
iter: 1803, loss : 6.305e-04
iter: 1804, loss : 2.418e-04
iter: 1805, loss : 1.569e-04
iter: 1806, loss : 2.892e-03
iter: 1807, loss : 1.178e-04
iter: 1808, loss : 6.565e-04
iter: 1809, loss : 3.038e-04
iter: 1810, loss : 2.731e-04
iter: 1811, loss : 2.690e-04
iter: 1812, loss : 2.311e-04
iter: 1813, loss : 2.502e-04
iter: 1814, loss : 2.140e-04
iter: 1815, loss : 1.824e-03
iter: 1816, loss : 1.173e-04
iter: 1817, loss : 1.479e-04
iter: 1818, loss : 1.979e-04
iter: 1819, loss : 1.162e-04
iter: 1820, loss : 3.895e-04
iter: 1821, loss : 1.526e-04
iter: 1822, loss : 5.766e-04
iter: 1823, loss : 5.471e-05
iter: 1824, loss : 1.667e-04
iter: 1825, loss : 3.716e-04
iter: 1826, loss : 6.478e-05
iter: 1827, loss : 6.005e-04
iter: 1828, loss : 4.028e-04
iter: 1829, loss : 2.882e-04
iter: 1830, loss : 1.095e-03
iter: 1831, loss : 2.105e-04
iter: 1832, loss : 4.560e-04
iter: 1833, loss : 1.093e-04
iter: 1834, loss : 2.147e-04
iter: 1835, loss : 1.205e-04
iter: 1836, loss : 9.562e-04
iter: 1837, loss : 1.965e-04
iter: 1838, loss : 5.206e-04
iter: 1839, loss : 1.064e-03
iter: 1840, loss : 1.797e-03
iter: 1841, loss : 8.372e-04
iter: 1842, loss : 2.277e-04
iter: 1843, loss : 4.464e-04
iter: 1844, loss : 7.473e-04
iter: 1845, loss : 1.660e-04
iter: 1846, loss : 5.232e-04
iter: 1847, loss : 1.694e-04
iter: 1848, loss : 1.473e-04
iter: 1849, loss : 1.781e-04
iter: 1850, loss : 1.910e-04
iter: 1851, loss : 1.518e-04
iter: 1852, loss : 9.191e-05
iter: 1853, loss : 8.598e-05
iter: 1854, loss : 3.452e-04
iter: 1855, loss : 3.181e-03
iter: 1856, loss : 2.868e-03
iter: 1857, loss : 6.111e-04
iter: 1858, loss : 4.168e-04
iter: 1859, loss : 2.091e-04
iter: 1860, loss : 1.253e-04
iter: 1861, loss : 5.641e-04
iter: 1862, loss : 2.262e-04
iter: 1863, loss : 1.420e-03
iter: 1864, loss : 6.770e-05
iter: 1865, loss : 1.962e-04
iter: 1866, loss : 2.453e-04
iter: 1867, loss : 6.621e-04
iter: 1868, loss : 2.672e-04
iter: 1869, loss : 5.519e-04
iter: 1870, loss : 1.333e-04
iter: 1871, loss : 7.462e-05
iter: 1872, loss : 9.668e-04
iter: 1873, loss : 1.969e-03
iter: 1874, loss : 7.893e-04
iter: 1875, loss : 2.427e-04
iter: 1876, loss : 6.151e-04
iter: 1877, loss : 1.331e-04
iter: 1878, loss : 1.481e-04
iter: 1879, loss : 1.638e-04
iter: 1880, loss : 1.596e-04
iter: 1881, loss : 4.025e-04
iter: 1882, loss : 1.183e-04
iter: 1883, loss : 3.985e-03
iter: 1884, loss : 1.536e-04
iter: 1885, loss : 1.198e-04
iter: 1886, loss : 6.404e-05
iter: 1887, loss : 1.590e-04
iter: 1888, loss : 2.824e-04
iter: 1889, loss : 2.004e-04
iter: 1890, loss : 2.412e-04
iter: 1891, loss : 1.657e-04
iter: 1892, loss : 1.560e-04
iter: 1893, loss : 5.788e-04
iter: 1894, loss : 9.986e-05
iter: 1895, loss : 1.553e-04
iter: 1896, loss : 9.311e-04
iter: 1897, loss : 4.506e-04
iter: 1898, loss : 2.027e-04
iter: 1899, loss : 1.374e-04
iter: 1900, loss : 1.737e-04
iter: 1901, loss : 1.924e-04
iter: 1902, loss : 1.900e-04
iter: 1903, loss : 1.261e-04
iter: 1904, loss : 2.234e-03
iter: 1905, loss : 1.440e-03
iter: 1906, loss : 3.977e-04
iter: 1907, loss : 1.118e-04
iter: 1908, loss : 4.040e-04
iter: 1909, loss : 1.545e-04
iter: 1910, loss : 2.074e-04
iter: 1911, loss : 5.858e-04
iter: 1912, loss : 7.139e-04
iter: 1913, loss : 1.958e-04
iter: 1914, loss : 2.747e-04
iter: 1915, loss : 2.416e-04
iter: 1916, loss : 2.764e-03
iter: 1917, loss : 2.273e-04
iter: 1918, loss : 4.147e-04
iter: 1919, loss : 6.218e-04
iter: 1920, loss : 2.601e-03
iter: 1921, loss : 5.129e-04
iter: 1922, loss : 1.004e-04
iter: 1923, loss : 3.277e-04
iter: 1924, loss : 1.181e-03
iter: 1925, loss : 1.902e-04
iter: 1926, loss : 1.852e-04
iter: 1927, loss : 4.638e-04
iter: 1928, loss : 1.326e-04
iter: 1929, loss : 2.305e-04
iter: 1930, loss : 1.589e-04
iter: 1931, loss : 1.488e-04
iter: 1932, loss : 2.053e-04
iter: 1933, loss : 2.114e-03
iter: 1934, loss : 8.085e-05
iter: 1935, loss : 9.356e-04
iter: 1936, loss : 3.414e-04
iter: 1937, loss : 2.547e-04
iter: 1938, loss : 1.560e-04
iter: 1939, loss : 2.754e-04
iter: 1940, loss : 3.058e-04
iter: 1941, loss : 4.408e-04
iter: 1942, loss : 2.147e-03
iter: 1943, loss : 1.026e-03
iter: 1944, loss : 4.594e-04
iter: 1945, loss : 5.611e-04
iter: 1946, loss : 3.869e-04
iter: 1947, loss : 1.477e-04
iter: 1948, loss : 4.308e-04
iter: 1949, loss : 9.020e-05
iter: 1950, loss : 1.988e-04
iter: 1951, loss : 1.465e-04
iter: 1952, loss : 1.071e-04
iter: 1953, loss : 1.301e-04
iter: 1954, loss : 7.432e-05
iter: 1955, loss : 1.742e-04
iter: 1956, loss : 8.679e-05
iter: 1957, loss : 1.825e-04
iter: 1958, loss : 6.890e-05
iter: 1959, loss : 5.414e-04
iter: 1960, loss : 1.352e-04
iter: 1961, loss : 1.217e-03
iter: 1962, loss : 9.400e-04
iter: 1963, loss : 2.306e-04
iter: 1964, loss : 2.779e-04
iter: 1965, loss : 3.354e-04
iter: 1966, loss : 2.563e-04
iter: 1967, loss : 2.268e-04
iter: 1968, loss : 5.530e-04
iter: 1969, loss : 4.501e-04
iter: 1970, loss : 2.672e-04
iter: 1971, loss : 2.016e-04
iter: 1972, loss : 5.711e-04
iter: 1973, loss : 3.115e-04
iter: 1974, loss : 1.756e-04
iter: 1975, loss : 9.954e-04
iter: 1976, loss : 1.091e-04
iter: 1977, loss : 1.021e-04
iter: 1978, loss : 1.604e-04
iter: 1979, loss : 3.429e-04
iter: 1980, loss : 6.281e-04
iter: 1981, loss : 2.062e-04
iter: 1982, loss : 1.258e-04
iter: 1983, loss : 1.188e-04
iter: 1984, loss : 1.774e-04
iter: 1985, loss : 1.401e-03
iter: 1986, loss : 5.263e-04
iter: 1987, loss : 3.160e-04
iter: 1988, loss : 1.791e-04
iter: 1989, loss : 9.084e-05
iter: 1990, loss : 1.988e-04
iter: 1991, loss : 7.695e-05
iter: 1992, loss : 1.280e-04
iter: 1993, loss : 1.151e-03
iter: 1994, loss : 1.596e-04
iter: 1995, loss : 6.311e-04
iter: 1996, loss : 5.039e-04
iter: 1997, loss : 1.071e-04
iter: 1998, loss : 9.595e-05
iter: 1999, loss : 3.793e-04
iter: 2000, loss : 1.047e-03
iter: 2001, loss : 3.604e-04
iter: 2002, loss : 1.363e-03
iter: 2003, loss : 1.072e-03
iter: 2004, loss : 1.879e-03
iter: 2005, loss : 2.271e-03
iter: 2006, loss : 8.025e-04
iter: 2007, loss : 5.456e-04
iter: 2008, loss : 1.233e-04
iter: 2009, loss : 4.371e-04
iter: 2010, loss : 3.367e-04
iter: 2011, loss : 1.963e-04
iter: 2012, loss : 5.554e-04
iter: 2013, loss : 4.301e-04
iter: 2014, loss : 8.450e-05
iter: 2015, loss : 1.603e-04
iter: 2016, loss : 4.537e-04
iter: 2017, loss : 1.231e-04
iter: 2018, loss : 3.782e-03
iter: 2019, loss : 7.643e-04
iter: 2020, loss : 7.758e-05
iter: 2021, loss : 9.908e-05
iter: 2022, loss : 1.105e-03
iter: 2023, loss : 1.426e-04
iter: 2024, loss : 6.391e-04
iter: 2025, loss : 2.481e-04
iter: 2026, loss : 1.406e-04
iter: 2027, loss : 2.323e-04
iter: 2028, loss : 1.798e-03
iter: 2029, loss : 1.275e-04
iter: 2030, loss : 2.100e-04
iter: 2031, loss : 2.058e-04
iter: 2032, loss : 1.742e-03
iter: 2033, loss : 5.682e-04
iter: 2034, loss : 5.018e-04
iter: 2035, loss : 1.416e-04
iter: 2036, loss : 1.232e-03
iter: 2037, loss : 6.606e-04
iter: 2038, loss : 8.086e-05
iter: 2039, loss : 3.466e-04
iter: 2040, loss : 2.639e-04
iter: 2041, loss : 6.308e-05
iter: 2042, loss : 1.712e-04
iter: 2043, loss : 4.188e-03
iter: 2044, loss : 5.409e-04
iter: 2045, loss : 1.590e-04
iter: 2046, loss : 2.882e-04
iter: 2047, loss : 3.457e-03
iter: 2048, loss : 3.661e-04
iter: 2049, loss : 3.436e-04
iter: 2050, loss : 1.824e-04
iter: 2051, loss : 4.777e-04
iter: 2052, loss : 1.873e-04
iter: 2053, loss : 2.355e-04
iter: 2054, loss : 2.142e-03
iter: 2055, loss : 3.623e-04
iter: 2056, loss : 3.598e-04
iter: 2057, loss : 2.148e-04
iter: 2058, loss : 2.527e-03
iter: 2059, loss : 2.746e-04
iter: 2060, loss : 1.560e-04
iter: 2061, loss : 3.792e-04
iter: 2062, loss : 6.427e-04
iter: 2063, loss : 1.229e-04
iter: 2064, loss : 1.711e-04
iter: 2065, loss : 2.809e-04
iter: 2066, loss : 7.094e-04
iter: 2067, loss : 6.645e-04
iter: 2068, loss : 2.945e-04
iter: 2069, loss : 2.431e-04
iter: 2070, loss : 1.162e-03
iter: 2071, loss : 2.619e-04
iter: 2072, loss : 3.309e-04
iter: 2073, loss : 1.019e-03
iter: 2074, loss : 3.496e-04
iter: 2075, loss : 1.088e-04
iter: 2076, loss : 1.417e-04
iter: 2077, loss : 2.729e-04
iter: 2078, loss : 6.444e-04
iter: 2079, loss : 1.532e-03
iter: 2080, loss : 3.079e-04
iter: 2081, loss : 2.390e-04
iter: 2082, loss : 3.898e-04
iter: 2083, loss : 1.432e-04
iter: 2084, loss : 3.082e-04
iter: 2085, loss : 2.676e-04
iter: 2086, loss : 4.125e-04
iter: 2087, loss : 1.880e-04
iter: 2088, loss : 1.440e-04
iter: 2089, loss : 2.896e-04
iter: 2090, loss : 6.309e-04
iter: 2091, loss : 1.844e-04
iter: 2092, loss : 1.676e-04
iter: 2093, loss : 8.983e-05
iter: 2094, loss : 2.714e-03
iter: 2095, loss : 2.054e-04
iter: 2096, loss : 1.168e-04
iter: 2097, loss : 4.963e-04
iter: 2098, loss : 4.113e-04
iter: 2099, loss : 4.453e-04
iter: 2100, loss : 2.981e-04
iter: 2101, loss : 1.742e-04
iter: 2102, loss : 1.192e-04
iter: 2103, loss : 1.222e-03
iter: 2104, loss : 1.634e-03
iter: 2105, loss : 5.330e-04
iter: 2106, loss : 1.657e-04
iter: 2107, loss : 2.195e-04
iter: 2108, loss : 5.242e-04
iter: 2109, loss : 3.452e-04
iter: 2110, loss : 5.172e-04
iter: 2111, loss : 6.316e-05
iter: 2112, loss : 2.519e-03
iter: 2113, loss : 1.482e-04
iter: 2114, loss : 2.864e-04
iter: 2115, loss : 2.024e-04
iter: 2116, loss : 1.449e-04
iter: 2117, loss : 9.602e-05
iter: 2118, loss : 2.655e-04
iter: 2119, loss : 3.014e-04
iter: 2120, loss : 3.600e-04
iter: 2121, loss : 1.848e-03
iter: 2122, loss : 1.852e-04
iter: 2123, loss : 2.209e-03
iter: 2124, loss : 2.227e-03
iter: 2125, loss : 6.120e-04
iter: 2126, loss : 3.383e-03
iter: 2127, loss : 5.985e-04
iter: 2128, loss : 5.133e-04
iter: 2129, loss : 4.528e-04
iter: 2130, loss : 6.803e-04
iter: 2131, loss : 4.000e-04
iter: 2132, loss : 1.728e-04
iter: 2133, loss : 3.508e-04
iter: 2134, loss : 9.970e-05
iter: 2135, loss : 1.405e-03
iter: 2136, loss : 5.873e-04
iter: 2137, loss : 2.743e-04
iter: 2138, loss : 1.695e-04
iter: 2139, loss : 3.279e-04
iter: 2140, loss : 2.764e-04
iter: 2141, loss : 2.915e-04
iter: 2142, loss : 2.423e-04
iter: 2143, loss : 3.452e-04
iter: 2144, loss : 3.479e-04
iter: 2145, loss : 3.083e-04
iter: 2146, loss : 7.170e-04
iter: 2147, loss : 1.044e-04
iter: 2148, loss : 2.045e-04
iter: 2149, loss : 4.340e-04
iter: 2150, loss : 2.380e-04
iter: 2151, loss : 1.980e-03
iter: 2152, loss : 1.996e-04
iter: 2153, loss : 4.111e-04
iter: 2154, loss : 3.869e-04
iter: 2155, loss : 2.646e-04
iter: 2156, loss : 2.336e-04
iter: 2157, loss : 1.615e-04
iter: 2158, loss : 1.162e-03
iter: 2159, loss : 5.251e-04
iter: 2160, loss : 1.765e-04
iter: 2161, loss : 1.947e-04
iter: 2162, loss : 1.551e-04
iter: 2163, loss : 1.491e-04
iter: 2164, loss : 5.153e-04
iter: 2165, loss : 5.153e-04
iter: 2166, loss : 1.441e-04
iter: 2167, loss : 2.172e-04
iter: 2168, loss : 2.196e-04
iter: 2169, loss : 8.688e-05
iter: 2170, loss : 1.142e-04
iter: 2171, loss : 9.367e-04
iter: 2172, loss : 1.341e-04
iter: 2173, loss : 5.981e-04
iter: 2174, loss : 4.638e-04
iter: 2175, loss : 2.284e-04
iter: 2176, loss : 2.450e-04
iter: 2177, loss : 2.870e-04
iter: 2178, loss : 2.331e-04
iter: 2179, loss : 2.008e-03
iter: 2180, loss : 1.604e-04
iter: 2181, loss : 1.260e-04
iter: 2182, loss : 7.174e-05
iter: 2183, loss : 9.541e-05
iter: 2184, loss : 6.687e-04
iter: 2185, loss : 3.085e-04
iter: 2186, loss : 1.371e-04
iter: 2187, loss : 1.594e-03
iter: 2188, loss : 3.401e-04
iter: 2189, loss : 1.070e-04
iter: 2190, loss : 1.215e-03
iter: 2191, loss : 2.146e-04
iter: 2192, loss : 1.233e-04
iter: 2193, loss : 2.538e-04
iter: 2194, loss : 5.617e-04
iter: 2195, loss : 2.365e-04
iter: 2196, loss : 4.790e-04
iter: 2197, loss : 1.367e-04
iter: 2198, loss : 1.366e-04
iter: 2199, loss : 8.640e-05
iter: 2200, loss : 5.664e-05
iter: 2201, loss : 7.899e-05
iter: 2202, loss : 5.982e-04
iter: 2203, loss : 1.246e-04
iter: 2204, loss : 1.396e-03
iter: 2205, loss : 4.403e-04
iter: 2206, loss : 1.162e-04
iter: 2207, loss : 1.140e-04
iter: 2208, loss : 9.838e-05
iter: 2209, loss : 1.320e-04
iter: 2210, loss : 8.544e-05
iter: 2211, loss : 1.900e-04
iter: 2212, loss : 1.397e-04
iter: 2213, loss : 1.205e-03
iter: 2214, loss : 6.766e-04
iter: 2215, loss : 2.347e-04
iter: 2216, loss : 9.242e-05
iter: 2217, loss : 1.633e-04
iter: 2218, loss : 1.777e-04
iter: 2219, loss : 9.603e-04
iter: 2220, loss : 6.262e-04
iter: 2221, loss : 5.028e-04
iter: 2222, loss : 5.345e-04
iter: 2223, loss : 3.653e-04
iter: 2224, loss : 4.657e-04
iter: 2225, loss : 1.270e-04
iter: 2226, loss : 4.316e-04
iter: 2227, loss : 1.545e-04
iter: 2228, loss : 2.555e-04
iter: 2229, loss : 2.827e-03
iter: 2230, loss : 9.307e-05
iter: 2231, loss : 1.785e-04
iter: 2232, loss : 1.880e-04
iter: 2233, loss : 1.523e-04
iter: 2234, loss : 2.148e-04
iter: 2235, loss : 1.427e-04
iter: 2236, loss : 1.689e-04
iter: 2237, loss : 1.198e-04
iter: 2238, loss : 1.008e-04
iter: 2239, loss : 2.399e-04
iter: 2240, loss : 1.791e-04
iter: 2241, loss : 9.906e-04
iter: 2242, loss : 5.247e-04
iter: 2243, loss : 3.054e-04
iter: 2244, loss : 1.883e-04
iter: 2245, loss : 7.010e-05
iter: 2246, loss : 4.388e-04
iter: 2247, loss : 2.426e-04
iter: 2248, loss : 8.784e-05
iter: 2249, loss : 4.098e-04
iter: 2250, loss : 6.434e-04
iter: 2251, loss : 1.567e-04
iter: 2252, loss : 1.559e-04
iter: 2253, loss : 2.637e-04
iter: 2254, loss : 1.051e-04
iter: 2255, loss : 3.232e-03
iter: 2256, loss : 9.804e-05
iter: 2257, loss : 3.483e-03
iter: 2258, loss : 1.726e-04
iter: 2259, loss : 1.094e-04
iter: 2260, loss : 3.663e-04
iter: 2261, loss : 2.472e-04
iter: 2262, loss : 1.470e-04
iter: 2263, loss : 1.584e-04
iter: 2264, loss : 5.638e-05
iter: 2265, loss : 8.824e-05
iter: 2266, loss : 1.033e-04
iter: 2267, loss : 1.839e-03
iter: 2268, loss : 2.004e-04
iter: 2269, loss : 5.646e-04
iter: 2270, loss : 9.817e-04
iter: 2271, loss : 2.268e-03
iter: 2272, loss : 6.150e-04
iter: 2273, loss : 5.038e-04
iter: 2274, loss : 2.262e-04
iter: 2275, loss : 1.982e-04
iter: 2276, loss : 5.431e-04
iter: 2277, loss : 1.402e-04
iter: 2278, loss : 4.197e-04
iter: 2279, loss : 1.154e-04
iter: 2280, loss : 1.851e-04
iter: 2281, loss : 1.896e-04
iter: 2282, loss : 3.264e-04
iter: 2283, loss : 1.464e-03
iter: 2284, loss : 7.412e-04
iter: 2285, loss : 3.114e-04
iter: 2286, loss : 9.688e-05
iter: 2287, loss : 3.381e-04
iter: 2288, loss : 3.969e-04
iter: 2289, loss : 4.765e-04
iter: 2290, loss : 2.248e-04
iter: 2291, loss : 2.908e-03
iter: 2292, loss : 1.435e-03
iter: 2293, loss : 5.200e-04
iter: 2294, loss : 8.943e-05
iter: 2295, loss : 1.846e-04
iter: 2296, loss : 1.909e-04
iter: 2297, loss : 1.994e-04
iter: 2298, loss : 6.462e-04
iter: 2299, loss : 3.047e-03
iter: 2300, loss : 4.482e-04
iter: 2301, loss : 2.357e-04
iter: 2302, loss : 2.842e-04
iter: 2303, loss : 3.248e-04
iter: 2304, loss : 6.548e-04
iter: 2305, loss : 3.074e-04
iter: 2306, loss : 2.843e-04
iter: 2307, loss : 1.867e-04
iter: 2308, loss : 3.329e-04
iter: 2309, loss : 1.566e-04
iter: 2310, loss : 1.132e-04
iter: 2311, loss : 1.152e-04
iter: 2312, loss : 1.215e-04
iter: 2313, loss : 3.770e-04
iter: 2314, loss : 1.741e-04
iter: 2315, loss : 1.289e-04
iter: 2316, loss : 1.843e-04
iter: 2317, loss : 8.960e-05
iter: 2318, loss : 6.419e-04
iter: 2319, loss : 1.615e-04
iter: 2320, loss : 5.221e-04
iter: 2321, loss : 1.981e-04
iter: 2322, loss : 1.307e-04
iter: 2323, loss : 1.007e-04
iter: 2324, loss : 3.461e-03
iter: 2325, loss : 1.888e-04
iter: 2326, loss : 7.092e-04
iter: 2327, loss : 3.467e-04
iter: 2328, loss : 2.027e-04
iter: 2329, loss : 1.591e-04
iter: 2330, loss : 1.032e-03
iter: 2331, loss : 9.781e-04
iter: 2332, loss : 3.837e-04
iter: 2333, loss : 7.085e-04
iter: 2334, loss : 8.670e-04
iter: 2335, loss : 1.544e-04
iter: 2336, loss : 2.237e-04
iter: 2337, loss : 4.462e-04
iter: 2338, loss : 1.523e-04
iter: 2339, loss : 2.893e-04
iter: 2340, loss : 8.797e-05
iter: 2341, loss : 1.806e-04
iter: 2342, loss : 1.485e-04
iter: 2343, loss : 6.563e-05
iter: 2344, loss : 1.003e-03
iter: 2345, loss : 2.878e-04
iter: 2346, loss : 5.610e-04
iter: 2347, loss : 1.336e-04
iter: 2348, loss : 2.654e-04
iter: 2349, loss : 7.433e-04
iter: 2350, loss : 5.286e-04
iter: 2351, loss : 4.712e-04
iter: 2352, loss : 1.157e-04
iter: 2353, loss : 5.078e-04
iter: 2354, loss : 1.370e-04
iter: 2355, loss : 2.832e-04
iter: 2356, loss : 3.333e-04
iter: 2357, loss : 1.809e-04
iter: 2358, loss : 1.492e-04
iter: 2359, loss : 4.763e-04
iter: 2360, loss : 7.543e-04
iter: 2361, loss : 4.222e-04
iter: 2362, loss : 1.052e-04
iter: 2363, loss : 1.287e-04
iter: 2364, loss : 2.834e-04
iter: 2365, loss : 4.158e-04
iter: 2366, loss : 6.258e-05
iter: 2367, loss : 1.011e-04
iter: 2368, loss : 1.440e-04
iter: 2369, loss : 1.405e-03
iter: 2370, loss : 5.965e-04
iter: 2371, loss : 4.189e-04
iter: 2372, loss : 1.225e-03
iter: 2373, loss : 4.582e-04
iter: 2374, loss : 6.385e-04
iter: 2375, loss : 2.025e-04
iter: 2376, loss : 2.850e-04
iter: 2377, loss : 2.029e-04
iter: 2378, loss : 5.594e-04
iter: 2379, loss : 6.366e-04
iter: 2380, loss : 2.814e-04
iter: 2381, loss : 2.046e-04
iter: 2382, loss : 2.668e-04
iter: 2383, loss : 4.707e-04
iter: 2384, loss : 7.445e-04
iter: 2385, loss : 5.197e-04
iter: 2386, loss : 2.442e-04
iter: 2387, loss : 8.323e-05
iter: 2388, loss : 3.638e-03
iter: 2389, loss : 2.101e-03
iter: 2390, loss : 3.361e-04
iter: 2391, loss : 8.651e-04
iter: 2392, loss : 2.869e-04
iter: 2393, loss : 3.116e-04
iter: 2394, loss : 2.803e-04
iter: 2395, loss : 3.019e-03
iter: 2396, loss : 3.119e-04
iter: 2397, loss : 4.509e-04
iter: 2398, loss : 1.569e-04
iter: 2399, loss : 4.301e-04
iter: 2400, loss : 1.156e-04
iter: 2401, loss : 1.303e-04
iter: 2402, loss : 8.834e-05
iter: 2403, loss : 1.734e-04
iter: 2404, loss : 3.190e-03
iter: 2405, loss : 3.833e-03
iter: 2406, loss : 3.014e-04
iter: 2407, loss : 1.784e-03
iter: 2408, loss : 4.591e-04
iter: 2409, loss : 3.652e-04
iter: 2410, loss : 7.565e-05
iter: 2411, loss : 1.867e-03
iter: 2412, loss : 4.631e-04
iter: 2413, loss : 4.791e-04
iter: 2414, loss : 7.059e-04
iter: 2415, loss : 1.022e-03
iter: 2416, loss : 4.332e-04
iter: 2417, loss : 6.187e-05
iter: 2418, loss : 4.855e-04
iter: 2419, loss : 2.122e-04
iter: 2420, loss : 3.465e-04
iter: 2421, loss : 2.205e-03
iter: 2422, loss : 3.682e-04
iter: 2423, loss : 2.038e-04
iter: 2424, loss : 4.993e-04
iter: 2425, loss : 1.812e-04
iter: 2426, loss : 2.051e-04
iter: 2427, loss : 9.197e-04
iter: 2428, loss : 1.458e-04
iter: 2429, loss : 2.347e-04
iter: 2430, loss : 3.898e-04
iter: 2431, loss : 1.578e-04
iter: 2432, loss : 9.069e-04
iter: 2433, loss : 5.030e-04
iter: 2434, loss : 2.730e-04
iter: 2435, loss : 5.599e-04
iter: 2436, loss : 2.785e-04
iter: 2437, loss : 9.186e-05
iter: 2438, loss : 8.258e-04
iter: 2439, loss : 5.833e-05
iter: 2440, loss : 5.111e-04
iter: 2441, loss : 3.472e-04
iter: 2442, loss : 1.700e-03
iter: 2443, loss : 8.318e-04
iter: 2444, loss : 2.022e-04
iter: 2445, loss : 3.592e-04
iter: 2446, loss : 2.166e-04
iter: 2447, loss : 1.555e-04
iter: 2448, loss : 3.119e-04
iter: 2449, loss : 2.798e-04
iter: 2450, loss : 5.915e-04
iter: 2451, loss : 1.201e-04
iter: 2452, loss : 2.746e-04
iter: 2453, loss : 1.622e-04
iter: 2454, loss : 3.186e-04
iter: 2455, loss : 6.111e-04
iter: 2456, loss : 1.031e-04
iter: 2457, loss : 1.373e-04
iter: 2458, loss : 2.493e-04
iter: 2459, loss : 3.502e-04
iter: 2460, loss : 7.617e-05
iter: 2461, loss : 4.370e-04
iter: 2462, loss : 1.179e-04
iter: 2463, loss : 1.725e-04
iter: 2464, loss : 5.833e-05
iter: 2465, loss : 3.443e-05
iter: 2466, loss : 1.418e-04
iter: 2467, loss : 7.234e-05
iter: 2468, loss : 4.043e-05
iter: 2469, loss : 1.638e-04
iter: 2470, loss : 5.406e-04
iter: 2471, loss : 8.658e-05
iter: 2472, loss : 7.393e-05
iter: 2473, loss : 5.131e-04
iter: 2474, loss : 1.305e-04
iter: 2475, loss : 5.830e-04
iter: 2476, loss : 7.255e-04
iter: 2477, loss : 2.629e-04
iter: 2478, loss : 1.257e-04
iter: 2479, loss : 3.812e-04
iter: 2480, loss : 2.431e-04
iter: 2481, loss : 4.073e-04
iter: 2482, loss : 1.350e-04
iter: 2483, loss : 4.421e-04
iter: 2484, loss : 1.129e-04
iter: 2485, loss : 1.369e-03
iter: 2486, loss : 5.269e-04
iter: 2487, loss : 6.605e-05
iter: 2488, loss : 2.794e-04
iter: 2489, loss : 1.094e-04
iter: 2490, loss : 1.551e-04
iter: 2491, loss : 2.646e-03
iter: 2492, loss : 1.965e-04
iter: 2493, loss : 3.600e-04
iter: 2494, loss : 7.699e-05
iter: 2495, loss : 8.489e-04
iter: 2496, loss : 1.765e-04
iter: 2497, loss : 4.594e-04
iter: 2498, loss : 5.717e-05
iter: 2499, loss : 1.538e-04
iter: 2500, loss : 1.845e-04
iter: 2501, loss : 2.711e-04
iter: 2502, loss : 7.059e-05
iter: 2503, loss : 1.201e-04
iter: 2504, loss : 7.111e-05
iter: 2505, loss : 8.501e-04
iter: 2506, loss : 2.234e-04
iter: 2507, loss : 2.152e-04
iter: 2508, loss : 1.247e-04
iter: 2509, loss : 3.225e-04
iter: 2510, loss : 1.726e-04
iter: 2511, loss : 8.985e-05
iter: 2512, loss : 1.685e-04
iter: 2513, loss : 1.017e-04
iter: 2514, loss : 1.877e-04
iter: 2515, loss : 1.415e-04
iter: 2516, loss : 1.577e-04
iter: 2517, loss : 1.742e-03
iter: 2518, loss : 1.409e-04
iter: 2519, loss : 2.444e-04
iter: 2520, loss : 4.794e-05
iter: 2521, loss : 1.002e-04
iter: 2522, loss : 7.324e-05
iter: 2523, loss : 8.215e-04
iter: 2524, loss : 2.550e-04
iter: 2525, loss : 5.001e-04
iter: 2526, loss : 1.710e-04
iter: 2527, loss : 2.602e-04
iter: 2528, loss : 1.266e-04
iter: 2529, loss : 4.539e-04
iter: 2530, loss : 1.162e-04
iter: 2531, loss : 1.463e-04
iter: 2532, loss : 2.208e-04
iter: 2533, loss : 3.273e-03
iter: 2534, loss : 3.932e-04
iter: 2535, loss : 1.627e-03
iter: 2536, loss : 5.845e-04
iter: 2537, loss : 6.010e-04
iter: 2538, loss : 3.732e-04
iter: 2539, loss : 3.254e-04
iter: 2540, loss : 1.658e-04
iter: 2541, loss : 2.633e-04
iter: 2542, loss : 7.959e-04
iter: 2543, loss : 6.314e-04
iter: 2544, loss : 2.239e-04
iter: 2545, loss : 2.521e-04
iter: 2546, loss : 1.635e-03
iter: 2547, loss : 2.578e-03
iter: 2548, loss : 5.293e-04
iter: 2549, loss : 4.320e-04
iter: 2550, loss : 1.923e-04
iter: 2551, loss : 5.232e-04
iter: 2552, loss : 2.203e-04
iter: 2553, loss : 7.077e-04
iter: 2554, loss : 1.852e-04
iter: 2555, loss : 7.410e-04
iter: 2556, loss : 4.311e-04
iter: 2557, loss : 1.713e-04
iter: 2558, loss : 3.021e-04
iter: 2559, loss : 3.744e-04
iter: 2560, loss : 1.358e-04
iter: 2561, loss : 1.845e-04
iter: 2562, loss : 5.922e-04
iter: 2563, loss : 1.745e-04
iter: 2564, loss : 2.929e-03
iter: 2565, loss : 2.574e-04
iter: 2566, loss : 6.347e-04
iter: 2567, loss : 4.798e-04
iter: 2568, loss : 2.945e-04
iter: 2569, loss : 6.097e-04
iter: 2570, loss : 2.858e-04
iter: 2571, loss : 5.213e-04
iter: 2572, loss : 3.505e-04
iter: 2573, loss : 1.121e-04
iter: 2574, loss : 2.087e-04
iter: 2575, loss : 7.082e-04
iter: 2576, loss : 2.036e-04
iter: 2577, loss : 1.112e-03
iter: 2578, loss : 4.360e-04
iter: 2579, loss : 1.121e-04
iter: 2580, loss : 4.701e-04
iter: 2581, loss : 3.265e-03
iter: 2582, loss : 2.280e-04
iter: 2583, loss : 1.351e-04
iter: 2584, loss : 4.703e-04
iter: 2585, loss : 2.417e-04
iter: 2586, loss : 2.135e-04
iter: 2587, loss : 7.136e-04
iter: 2588, loss : 1.575e-04
iter: 2589, loss : 1.645e-04
iter: 2590, loss : 2.064e-04
iter: 2591, loss : 2.070e-04
iter: 2592, loss : 1.063e-03
iter: 2593, loss : 1.902e-04
iter: 2594, loss : 4.821e-04
iter: 2595, loss : 2.379e-04
iter: 2596, loss : 1.005e-04
iter: 2597, loss : 9.462e-05
iter: 2598, loss : 1.039e-03
iter: 2599, loss : 8.684e-05
iter: 2600, loss : 1.626e-04
iter: 2601, loss : 8.250e-04
iter: 2602, loss : 1.095e-04
iter: 2603, loss : 5.052e-04
iter: 2604, loss : 3.237e-04
iter: 2605, loss : 1.669e-04
iter: 2606, loss : 4.012e-04
iter: 2607, loss : 2.947e-04
iter: 2608, loss : 1.885e-03
iter: 2609, loss : 1.335e-04
iter: 2610, loss : 4.735e-04
iter: 2611, loss : 2.916e-04
iter: 2612, loss : 2.834e-04
iter: 2613, loss : 8.170e-04
iter: 2614, loss : 1.541e-04
iter: 2615, loss : 8.995e-05
iter: 2616, loss : 4.565e-04
iter: 2617, loss : 1.013e-04
iter: 2618, loss : 1.212e-04
iter: 2619, loss : 1.450e-04
iter: 2620, loss : 2.162e-04
iter: 2621, loss : 1.261e-04
iter: 2622, loss : 1.630e-04
iter: 2623, loss : 6.386e-05
iter: 2624, loss : 4.646e-04
iter: 2625, loss : 1.539e-04
iter: 2626, loss : 9.983e-05
iter: 2627, loss : 1.009e-04
iter: 2628, loss : 1.563e-04
iter: 2629, loss : 6.928e-05
iter: 2630, loss : 5.353e-04
iter: 2631, loss : 6.817e-05
iter: 2632, loss : 1.479e-04
iter: 2633, loss : 1.015e-04
iter: 2634, loss : 1.749e-04
iter: 2635, loss : 4.274e-03
iter: 2636, loss : 1.829e-04
iter: 2637, loss : 2.282e-04
iter: 2638, loss : 5.771e-04
iter: 2639, loss : 2.223e-04
iter: 2640, loss : 4.077e-04
iter: 2641, loss : 1.413e-04
iter: 2642, loss : 2.285e-04
iter: 2643, loss : 7.502e-05
iter: 2644, loss : 3.149e-03
iter: 2645, loss : 1.116e-03
iter: 2646, loss : 2.596e-03
iter: 2647, loss : 4.225e-04
iter: 2648, loss : 2.854e-04
iter: 2649, loss : 8.340e-05
iter: 2650, loss : 2.427e-04
iter: 2651, loss : 3.560e-04
iter: 2652, loss : 5.065e-05
iter: 2653, loss : 1.079e-03
iter: 2654, loss : 1.253e-04
iter: 2655, loss : 4.234e-04
iter: 2656, loss : 2.569e-04
iter: 2657, loss : 1.307e-03
iter: 2658, loss : 6.612e-04
iter: 2659, loss : 1.756e-04
iter: 2660, loss : 5.577e-04
iter: 2661, loss : 2.104e-04
iter: 2662, loss : 1.089e-04
iter: 2663, loss : 7.928e-05
iter: 2664, loss : 6.738e-05
iter: 2665, loss : 1.524e-04
iter: 2666, loss : 1.590e-03
iter: 2667, loss : 2.598e-04
iter: 2668, loss : 2.267e-04
iter: 2669, loss : 1.395e-04
iter: 2670, loss : 5.519e-05
iter: 2671, loss : 2.420e-03
iter: 2672, loss : 1.435e-04
iter: 2673, loss : 1.620e-04
iter: 2674, loss : 3.544e-04
iter: 2675, loss : 1.710e-04
iter: 2676, loss : 2.214e-04
iter: 2677, loss : 3.672e-04
iter: 2678, loss : 4.512e-04
iter: 2679, loss : 4.002e-04
iter: 2680, loss : 1.192e-04
iter: 2681, loss : 8.226e-05
iter: 2682, loss : 4.121e-04
iter: 2683, loss : 1.074e-03
iter: 2684, loss : 1.453e-04
iter: 2685, loss : 8.969e-05
iter: 2686, loss : 1.053e-04
iter: 2687, loss : 2.334e-04
iter: 2688, loss : 8.851e-05
iter: 2689, loss : 5.708e-05
iter: 2690, loss : 1.208e-04
iter: 2691, loss : 4.610e-04
iter: 2692, loss : 6.255e-04
iter: 2693, loss : 5.406e-05
iter: 2694, loss : 1.058e-04
iter: 2695, loss : 1.487e-04
iter: 2696, loss : 1.399e-04
iter: 2697, loss : 1.432e-04
iter: 2698, loss : 2.459e-04
iter: 2699, loss : 3.062e-04
iter: 2700, loss : 1.346e-04
iter: 2701, loss : 2.018e-04
iter: 2702, loss : 1.681e-04
iter: 2703, loss : 1.529e-04
iter: 2704, loss : 7.982e-05
iter: 2705, loss : 8.463e-05
iter: 2706, loss : 5.375e-05
iter: 2707, loss : 1.131e-04
iter: 2708, loss : 1.428e-04
iter: 2709, loss : 5.894e-05
iter: 2710, loss : 1.046e-03
iter: 2711, loss : 2.341e-03
iter: 2712, loss : 1.264e-04
iter: 2713, loss : 1.142e-04
iter: 2714, loss : 9.099e-04
iter: 2715, loss : 1.005e-04
iter: 2716, loss : 1.164e-04
iter: 2717, loss : 4.108e-04
iter: 2718, loss : 2.333e-04
iter: 2719, loss : 7.730e-05
iter: 2720, loss : 8.962e-05
iter: 2721, loss : 8.245e-05
iter: 2722, loss : 1.571e-04
iter: 2723, loss : 3.275e-03
iter: 2724, loss : 1.332e-04
iter: 2725, loss : 1.058e-03
iter: 2726, loss : 1.200e-04
iter: 2727, loss : 2.215e-04
iter: 2728, loss : 3.943e-03
iter: 2729, loss : 1.716e-04
iter: 2730, loss : 8.877e-04
iter: 2731, loss : 9.259e-04
iter: 2732, loss : 1.107e-04
iter: 2733, loss : 1.407e-04
iter: 2734, loss : 4.410e-04
iter: 2735, loss : 5.201e-04
iter: 2736, loss : 1.772e-03
iter: 2737, loss : 2.937e-03
iter: 2738, loss : 2.352e-04
iter: 2739, loss : 2.948e-04
iter: 2740, loss : 3.437e-04
iter: 2741, loss : 2.424e-04
iter: 2742, loss : 7.719e-05
iter: 2743, loss : 1.410e-04
iter: 2744, loss : 1.332e-04
iter: 2745, loss : 9.492e-05
iter: 2746, loss : 3.978e-05
iter: 2747, loss : 2.380e-04
iter: 2748, loss : 5.075e-04
iter: 2749, loss : 2.105e-04
iter: 2750, loss : 2.007e-04
iter: 2751, loss : 7.047e-05
iter: 2752, loss : 2.341e-04
iter: 2753, loss : 5.519e-05
iter: 2754, loss : 1.053e-04
iter: 2755, loss : 1.631e-04
iter: 2756, loss : 1.975e-04
iter: 2757, loss : 3.921e-04
iter: 2758, loss : 5.272e-05
iter: 2759, loss : 4.068e-04
iter: 2760, loss : 4.241e-04
iter: 2761, loss : 8.016e-05
iter: 2762, loss : 3.728e-04
iter: 2763, loss : 2.638e-03
iter: 2764, loss : 7.506e-04
iter: 2765, loss : 1.064e-04
iter: 2766, loss : 3.383e-04
iter: 2767, loss : 6.763e-05
iter: 2768, loss : 6.581e-05
iter: 2769, loss : 1.546e-04
iter: 2770, loss : 1.102e-04
iter: 2771, loss : 3.426e-05
iter: 2772, loss : 2.588e-03
iter: 2773, loss : 5.113e-04
iter: 2774, loss : 3.175e-04
iter: 2775, loss : 1.239e-04
iter: 2776, loss : 1.332e-04
iter: 2777, loss : 7.595e-04
iter: 2778, loss : 1.528e-04
iter: 2779, loss : 2.308e-04
iter: 2780, loss : 3.519e-04
iter: 2781, loss : 4.737e-04
iter: 2782, loss : 1.901e-04
iter: 2783, loss : 2.431e-03
iter: 2784, loss : 2.686e-04
iter: 2785, loss : 1.733e-04
iter: 2786, loss : 1.484e-04
iter: 2787, loss : 2.179e-04
iter: 2788, loss : 5.283e-04
iter: 2789, loss : 1.400e-04
iter: 2790, loss : 7.027e-04
iter: 2791, loss : 1.996e-04
iter: 2792, loss : 1.995e-04
iter: 2793, loss : 1.180e-04
iter: 2794, loss : 9.740e-05
iter: 2795, loss : 7.253e-05
iter: 2796, loss : 1.579e-04
iter: 2797, loss : 8.637e-05
iter: 2798, loss : 1.031e-04
iter: 2799, loss : 1.858e-04
iter: 2800, loss : 1.491e-04
iter: 2801, loss : 1.002e-04
iter: 2802, loss : 4.240e-04
iter: 2803, loss : 4.567e-05
iter: 2804, loss : 1.580e-04
iter: 2805, loss : 8.014e-05
iter: 2806, loss : 1.419e-04
iter: 2807, loss : 4.018e-05
iter: 2808, loss : 9.364e-05
iter: 2809, loss : 8.390e-05
iter: 2810, loss : 5.157e-05
iter: 2811, loss : 1.465e-04
iter: 2812, loss : 2.871e-04
iter: 2813, loss : 1.210e-03
iter: 2814, loss : 3.622e-04
iter: 2815, loss : 9.099e-05
iter: 2816, loss : 2.550e-03
iter: 2817, loss : 2.187e-04
iter: 2818, loss : 3.538e-04
iter: 2819, loss : 1.416e-04
iter: 2820, loss : 2.352e-04
iter: 2821, loss : 3.131e-04
iter: 2822, loss : 1.366e-04
iter: 2823, loss : 3.337e-04
iter: 2824, loss : 1.840e-04
iter: 2825, loss : 9.531e-05
iter: 2826, loss : 8.654e-05
iter: 2827, loss : 4.668e-04
iter: 2828, loss : 9.774e-05
iter: 2829, loss : 4.912e-04
iter: 2830, loss : 4.399e-04
iter: 2831, loss : 9.473e-05
iter: 2832, loss : 1.498e-04
iter: 2833, loss : 1.363e-04
iter: 2834, loss : 7.076e-05
iter: 2835, loss : 4.720e-04
iter: 2836, loss : 4.581e-04
iter: 2837, loss : 8.526e-04
iter: 2838, loss : 4.645e-05
iter: 2839, loss : 7.439e-05
iter: 2840, loss : 9.207e-04
iter: 2841, loss : 1.067e-04
iter: 2842, loss : 1.073e-04
iter: 2843, loss : 5.942e-05
iter: 2844, loss : 5.510e-05
iter: 2845, loss : 1.804e-04
iter: 2846, loss : 1.407e-04
iter: 2847, loss : 9.082e-05
iter: 2848, loss : 9.837e-05
iter: 2849, loss : 1.335e-04
iter: 2850, loss : 1.554e-04
iter: 2851, loss : 6.263e-05
iter: 2852, loss : 1.423e-04
iter: 2853, loss : 5.099e-04
iter: 2854, loss : 5.764e-05
iter: 2855, loss : 5.417e-05
iter: 2856, loss : 1.444e-04
iter: 2857, loss : 1.075e-04
iter: 2858, loss : 5.238e-04
iter: 2859, loss : 9.266e-05
iter: 2860, loss : 1.485e-03
iter: 2861, loss : 1.339e-04
iter: 2862, loss : 2.539e-04
iter: 2863, loss : 1.594e-04
iter: 2864, loss : 1.633e-03
iter: 2865, loss : 3.007e-05
iter: 2866, loss : 1.346e-04
iter: 2867, loss : 1.912e-04
iter: 2868, loss : 5.182e-05
iter: 2869, loss : 1.301e-03
iter: 2870, loss : 1.939e-04
iter: 2871, loss : 8.911e-05
iter: 2872, loss : 1.605e-04
iter: 2873, loss : 2.803e-04
iter: 2874, loss : 1.316e-04
iter: 2875, loss : 6.185e-04
iter: 2876, loss : 7.339e-04
iter: 2877, loss : 4.638e-04
iter: 2878, loss : 1.847e-04
iter: 2879, loss : 2.091e-04
iter: 2880, loss : 3.632e-04
iter: 2881, loss : 1.142e-04
iter: 2882, loss : 4.083e-05
iter: 2883, loss : 5.656e-05
iter: 2884, loss : 1.475e-04
iter: 2885, loss : 8.463e-05
iter: 2886, loss : 1.101e-04
iter: 2887, loss : 2.675e-03
iter: 2888, loss : 1.672e-04
iter: 2889, loss : 1.644e-04
iter: 2890, loss : 1.523e-04
iter: 2891, loss : 1.574e-04
iter: 2892, loss : 1.240e-03
iter: 2893, loss : 3.788e-04
iter: 2894, loss : 1.761e-04
iter: 2895, loss : 2.807e-04
iter: 2896, loss : 6.945e-05
iter: 2897, loss : 1.075e-04
iter: 2898, loss : 1.831e-04
iter: 2899, loss : 5.483e-05
iter: 2900, loss : 1.294e-04
iter: 2901, loss : 1.270e-04
iter: 2902, loss : 5.340e-05
iter: 2903, loss : 1.421e-03
iter: 2904, loss : 2.074e-04
iter: 2905, loss : 2.077e-04
iter: 2906, loss : 1.774e-04
iter: 2907, loss : 7.799e-05
iter: 2908, loss : 1.758e-04
iter: 2909, loss : 1.739e-04
iter: 2910, loss : 4.883e-05
iter: 2911, loss : 2.816e-03
iter: 2912, loss : 1.122e-04
iter: 2913, loss : 1.429e-04
iter: 2914, loss : 1.066e-04
iter: 2915, loss : 1.196e-04
iter: 2916, loss : 2.157e-04
iter: 2917, loss : 1.088e-04
iter: 2918, loss : 5.924e-04
iter: 2919, loss : 1.981e-04
iter: 2920, loss : 7.231e-05
iter: 2921, loss : 5.670e-04
iter: 2922, loss : 6.829e-05
iter: 2923, loss : 1.392e-04
iter: 2924, loss : 1.272e-04
iter: 2925, loss : 4.715e-05
iter: 2926, loss : 1.281e-04
iter: 2927, loss : 2.456e-04
iter: 2928, loss : 1.309e-04
iter: 2929, loss : 6.852e-05
iter: 2930, loss : 4.880e-05
iter: 2931, loss : 7.297e-05
iter: 2932, loss : 5.344e-05
iter: 2933, loss : 9.416e-04
iter: 2934, loss : 8.894e-05
iter: 2935, loss : 1.048e-04
iter: 2936, loss : 1.717e-04
iter: 2937, loss : 5.009e-04
iter: 2938, loss : 6.158e-05
iter: 2939, loss : 5.505e-05
iter: 2940, loss : 3.535e-05
iter: 2941, loss : 5.274e-04
iter: 2942, loss : 3.699e-05
iter: 2943, loss : 7.082e-05
iter: 2944, loss : 2.905e-05
iter: 2945, loss : 1.269e-04
iter: 2946, loss : 3.133e-03
iter: 2947, loss : 3.454e-04
iter: 2948, loss : 1.489e-04
iter: 2949, loss : 1.770e-04
iter: 2950, loss : 2.324e-04
iter: 2951, loss : 2.613e-03
iter: 2952, loss : 3.881e-04
iter: 2953, loss : 7.472e-05
iter: 2954, loss : 1.203e-04
iter: 2955, loss : 2.173e-04
iter: 2956, loss : 1.114e-04
iter: 2957, loss : 9.150e-05
iter: 2958, loss : 1.956e-04
iter: 2959, loss : 1.439e-04
iter: 2960, loss : 6.706e-04
iter: 2961, loss : 2.436e-04
iter: 2962, loss : 1.372e-04
iter: 2963, loss : 3.178e-04
iter: 2964, loss : 4.062e-04
iter: 2965, loss : 2.443e-04
iter: 2966, loss : 3.344e-04
iter: 2967, loss : 9.786e-05
iter: 2968, loss : 1.642e-04
iter: 2969, loss : 6.192e-05
iter: 2970, loss : 1.062e-04
iter: 2971, loss : 6.092e-05
iter: 2972, loss : 4.903e-04
iter: 2973, loss : 5.755e-05
iter: 2974, loss : 3.953e-04
iter: 2975, loss : 1.290e-04
iter: 2976, loss : 4.434e-04
iter: 2977, loss : 6.484e-05
iter: 2978, loss : 8.355e-05
iter: 2979, loss : 1.630e-04
iter: 2980, loss : 8.576e-05
iter: 2981, loss : 1.172e-04
iter: 2982, loss : 4.607e-04
iter: 2983, loss : 2.801e-05
iter: 2984, loss : 1.612e-04
iter: 2985, loss : 9.030e-05
iter: 2986, loss : 3.044e-04
iter: 2987, loss : 2.865e-04
iter: 2988, loss : 2.227e-03
iter: 2989, loss : 9.667e-05
iter: 2990, loss : 1.410e-04
iter: 2991, loss : 1.925e-04
iter: 2992, loss : 7.198e-04
iter: 2993, loss : 1.925e-04
iter: 2994, loss : 2.480e-03
iter: 2995, loss : 1.704e-04
iter: 2996, loss : 1.282e-04
iter: 2997, loss : 5.033e-04
iter: 2998, loss : 8.455e-05
iter: 2999, loss : 1.056e-04
iter: 2999, loss : 1.056e-04
--- 15.687845945358276 seconds --- to perform 3000 epochs
NOTE: For comparison we also called the torcheqdiff demo using the following command line ‘python ode_demo.py –niters 3000’. This triggers a comparable number of gradient evaluations on CPU. The iterations done in torcheqdiff complete in 348 seconds showing how on common CPUs, the use of heyoka may offer a significant speedup for NeuralODE research.
We are now ready to visualize the result of the learned neural ODE:
plt.plot(gt[:, 0], gt[:, 1])
ta_var.time = precision(0)
ta_var.state[:] = list(ic_var)
ta_var.state[:2] = ic
ta_var.pars[:] = np.array(nn_wb, dtype=np.single)
sol = ta_var.propagate_grid(np.array(t_grid, dtype=precision))[5]
plt.plot(sol[:, 0], sol[:, 1], ".")
[<matplotlib.lines.Line2D at 0x7f1f83754e00>]
It is also of interest to visualize the predictions of the trained system over a new random batch, as opposed to the predictions of the entire trajectory from the first observation:
batch_ic, batch_t_grid, batch_y = get_batch(t_grid, gt, batch_size_ic, batch_size_time)
plt.plot(gt[:, 0], gt[:, 1])
for i in range(batch_size_ic):
plt.plot(batch_y[:, i, 0], batch_y[:, i, 1], "k.")
start_time = time.time()
for batch in batched(batch_ic, batch_size_simd):
ta_var_b.set_time(precision(0.))
ta_var_b.state[:,:] = deepcopy(ic_var_b)
ta_var_b.state[:dim,:] = np.array(batch).T
ta_var_b.pars[:] = np.tile(nn_wb, (batch_size_simd,1)).T
sol_b = ta_var_b.propagate_grid(np.array(t_grid_b, dtype=precision))[1]
for sol in [sol_b[:,:,i] for i in range(batch_size_simd)]:
plt.plot(sol[:, 0], sol[:, 1], ".")
print(
"--- %s seconds --- to compute prediction over the whole batch"
% (time.time() - start_time)
)
--- 0.008912801742553711 seconds --- to compute prediction over the whole batch
