Parameters

Tensorforce distinguishes between agent/module arguments (primitive types: bool/int/float) which either specify part of the TensorFlow model architecture, like the layer size, or a value within the architecture, like the learning rate. Whereas the former are statically defined as part of the agent initialization, the latter can be dynamically adjusted afterwards. These dynamic hyperparameter are indicated by parameter as part of their argument type specification in the documentation, and can alternatively be assigned a parameter module instead of a constant value, for instance, to specify a decaying learning rate.

Default parameter: Constant, so a bool/int/float value is a short-form specification of a constant (dynamic) parameter:

Agent.create(
    ...
    exploration=0.1,
    ...
)

Example of how to specify an exponentially decaying learning rate:

Agent.create(
    ...
    optimizer=dict(optimizer='adam', learning_rate=dict(
        type='exponential', unit='timesteps', num_steps=1000,
        initial_value=0.01, decay_rate=0.5
    )),
    ...
)

Example of how to specify a linearly increasing reward horizon:

Agent.create(
    ...
    reward_estimation=dict(horizon=dict(
        type='linear', unit='episodes', num_steps=1000,
        initial_value=10, final_value=50
    )),
    ...
)
class tensorforce.core.parameters.Constant(value, *, name=None, dtype=None, min_value=None, max_value=None)

Constant hyperparameter (specification key: constant).

Parameters:
  • value (float | int | bool) – Constant hyperparameter value (required).
  • name (string) – internal use.
  • dtype (type) – internal use.
  • min_value (dtype-compatible value) – internal use.
  • max_value (dtype-compatible value) – internal use.
class tensorforce.core.parameters.Linear(*, unit, num_steps, initial_value, final_value, name=None, dtype=None, min_value=None, max_value=None)

Linear hyperparameter (specification key: linear).

Parameters:
  • unit ("timesteps" | "episodes" | "updates") – Unit of decay schedule (required).
  • num_steps (int) – Number of decay steps (required).
  • initial_value (float) – Initial value (required).
  • final_value (float) – Final value (required).
  • name (string) – internal use.
  • dtype (type) – internal use.
  • min_value (dtype-compatible value) – internal use.
  • max_value (dtype-compatible value) – internal use.
class tensorforce.core.parameters.PiecewiseConstant(*, unit, boundaries, values, name=None, dtype=None, min_value=None, max_value=None)

Piecewise-constant hyperparameter (specification key: piecewise_constant).

Parameters:
  • unit ("timesteps" | "episodes" | "updates") – Unit of interval boundaries (required).
  • boundaries (iter[long]) – Strictly increasing interval boundaries for constant segments (required).
  • values (iter[dtype-dependent]) – Interval values of constant segments, one more than (required).
  • name (string) – internal use.
  • dtype (type) – internal use.
  • min_value (dtype-compatible value) – internal use.
  • max_value (dtype-compatible value) – internal use.
class tensorforce.core.parameters.Exponential(*, unit, num_steps, initial_value, decay_rate, staircase=False, name=None, dtype=None, min_value=None, max_value=None, **kwargs)

Exponentially decaying hyperparameter (specification key: exponential).

Parameters:
  • unit ("timesteps" | "episodes" | "updates") – Unit of decay schedule (required).
  • num_steps (int) – Number of decay steps (required).
  • initial_value (float) – Initial value (required).
  • decay_rate (float) – Decay rate (required).
  • staircase (bool) – Whether to apply decay in a discrete staircase, as opposed to continuous, fashion (default: false).
  • name (string) – internal use.
  • dtype (type) – internal use.
  • min_value (dtype-compatible value) – internal use.
  • max_value (dtype-compatible value) – internal use.
class tensorforce.core.parameters.Decaying(*, decay, unit, num_steps, initial_value, increasing=False, inverse=False, scale=1.0, name=None, dtype=None, min_value=None, max_value=None, **kwargs)

Decaying hyperparameter (specification key: decaying, linear, exponential, polynomial, inverse_time, cosine, cosine_restarts, linear_cosine, linear_cosine_noisy).

Parameters:
  • decay ("linear" | "exponential" | "polynomial" | "inverse_time" | "cosine" | "cosine_restarts" | "linear_cosine" | "linear_cosine_noisy") – Decay type, see also TensorFlow docs (required).
  • unit ("timesteps" | "episodes" | "updates") – Unit of decay schedule (required).
  • num_steps (int) – Number of decay steps (required).
  • initial_value (float | int) – Initial value (required).
  • increasing (bool) – Whether to subtract the decayed value from 1.0 (default: false).
  • inverse (bool) – Whether to take the inverse of the decayed value (default: false).
  • scale (float) – Scaling factor for (inverse) decayed value (default: 1.0).
  • kwargs – Additional arguments depend on decay mechanism.
    Linear decay:
    • final_value (float | int) – Final value (required).
    Exponential decay:
    • decay_rate (float) – Decay rate (required).
    • staircase (bool) – Whether to apply decay in a discrete staircase, as opposed to continuous, fashion. (default: false).
    Polynomial decay:
    • final_value (float | int) – Final value (required).
    • power (float | int) – Power of polynomial (default: 1, thus linear).
    • cycle (bool) – Whether to cycle beyond num_steps (default: false).
    Inverse time decay:
    • decay_rate (float) – Decay rate (required).
    • staircase (bool) – Whether to apply decay in a discrete staircase, as opposed to continuous, fashion. (default: false).
    Cosine decay:
    • alpha (float) – Minimum learning rate value as a fraction of learning_rate (default: 0.0).
    Cosine decay with restarts:
    • t_mul (float) – Used to derive the number of iterations in the i-th period (default: 2.0).
    • m_mul (float) – Used to derive the initial learning rate of the i-th period (default: 1.0).
    • alpha (float) – Minimum learning rate value as a fraction of the learning_rate (default: 0.0).
    Linear cosine decay:
    • num_periods (float) – Number of periods in the cosine part of the decay (default: 0.5).
    • alpha (float) – Alpha value (default: 0.0).
    • beta (float) – Beta value (default: 0.001).
    Noisy linear cosine decay:
    • initial_variance (float) – Initial variance for the noise (default: 1.0).
    • variance_decay (float) – Decay for the noise's variance (default: 0.55).
    • num_periods (float) – Number of periods in the cosine part of the decay (default: 0.5).
    • alpha (float) – Alpha value (default: 0.0).
    • beta (float) – Beta value (default: 0.001).
  • name (string) – internal use.
  • dtype (type) – internal use.
  • min_value (dtype-compatible value) – internal use.
  • max_value (dtype-compatible value) – internal use.
class tensorforce.core.parameters.OrnsteinUhlenbeck(*, theta=0.15, sigma=0.3, mu=0.0, absolute=False, name=None, dtype=None, min_value=None, max_value=None)

Ornstein-Uhlenbeck process (specification key: ornstein_uhlenbeck).

Parameters:
  • theta (float > 0.0) – Theta value (default: 0.15).
  • sigma (float > 0.0) – Sigma value (default: 0.3).
  • mu (float) – Mu value (default: 0.0).
  • absolute (bool) – Absolute value (default: false).
  • name (string) – internal use.
  • dtype (type) – internal use.
  • min_value (dtype-compatible value) – internal use.
  • max_value (dtype-compatible value) – internal use.
class tensorforce.core.parameters.Random(*, distribution, name=None, dtype=None, shape=(), min_value=None, max_value=None, **kwargs)

Random hyperparameter (specification key: random).

Parameters:
  • distribution ("normal" | "uniform") – Distribution type for random hyperparameter value (required).
  • kwargs – Additional arguments dependent on distribution type.
    Normal distribution:
    • mean (float) – Mean (default: 0.0).
    • stddev (float > 0.0) – Standard deviation (default: 1.0).
    Uniform distribution:
    • minval (int / float) – Lower bound (default: 0 / 0.0).
    • maxval (float > minval) – Upper bound (default: 1.0 for float, required for int).
  • name (string) – internal use.
  • dtype (type) – internal use.
  • shape (iter[int > 0]) – internal use.
  • min_value (dtype-compatible value) – internal use.
  • max_value (dtype-compatible value) – internal use.