tensorforce.core.optimizers package

Submodules

tensorforce.core.optimizers.clipped_step module

class tensorforce.core.optimizers.clipped_step.ClippedStep(optimizer, clipping_value, scope='clipped-step', summary_labels=())

Bases: tensorforce.core.optimizers.meta_optimizer.MetaOptimizer

The clipped-shep meta optimizer clips the values of the optimization step proposed by another optimizer.

__init__(optimizer, clipping_value, scope='clipped-step', summary_labels=())

Creates a new multi-step meta optimizer instance.

Parameters:
  • optimizer – The optimizer which is modified by this meta optimizer.
  • clipping_value – Clip deltas at this value.
apply_step(variables, deltas)

Applies step deltas to variable values.

Parameters:
  • variables – List of variables.
  • deltas – List of deltas of same length.
Returns:

The step-applied operation.
from_spec(spec, kwargs=None)

Creates an optimizer from a specification dict.

get_summaries()

Returns the TensorFlow summaries reported by the optimizer.

Returns:List of summaries.
get_variables()
minimize(time, variables, **kwargs)

Performs an optimization step.

Parameters:
  • time – Time tensor.
  • variables – List of variables to optimize.
  • **kwargs –

    Additional optimizer-specific arguments. The following arguments are used by some optimizers:

  • arguments (-) – Dict of arguments for callables, like fn_loss.
  • fn_loss (-) – A callable returning the loss of the current model.
  • fn_reference (-) – A callable returning the reference values, in case of a comparative loss.
  • fn_kl_divergence (-) – A callable returning the KL-divergence relative to the current model.
  • return_estimated_improvement (-) – Returns the estimated improvement resulting from the natural gradient calculation if true.
  • source_variables (-) – List of source variables to synchronize with.
  • global_variables (-) – List of global variables to apply the proposed optimization step to.
Returns:

The optimization operation.
tf_step(time, variables, **kwargs)

Creates the TensorFlow operations for performing an optimization step.

Parameters:
  • time – Time tensor.
  • variables – List of variables to optimize.
  • **kwargs –

    Additional arguments passed on to the internal optimizer.
Returns:

List of delta tensors corresponding to the updates for each optimized variable.

tensorforce.core.optimizers.evolutionary module

class tensorforce.core.optimizers.evolutionary.Evolutionary(learning_rate, num_samples=1, unroll_loop=False, scope='evolutionary', summary_labels=())

Bases: tensorforce.core.optimizers.optimizer.Optimizer

Evolutionary optimizer which samples random perturbations and applies them either positively or negatively, depending on their improvement of the loss.

__init__(learning_rate, num_samples=1, unroll_loop=False, scope='evolutionary', summary_labels=())

Creates a new evolutionary optimizer instance.

Parameters:
  • learning_rate – Learning rate.
  • num_samples – Number of sampled perturbations.
apply_step(variables, deltas)

Applies step deltas to variable values.

Parameters:
  • variables – List of variables.
  • deltas – List of deltas of same length.
Returns:

The step-applied operation.
from_spec(spec, kwargs=None)

Creates an optimizer from a specification dict.

get_summaries()

Returns the TensorFlow summaries reported by the optimizer.

Returns:List of summaries.
get_variables()

Returns the TensorFlow variables used by the optimizer.

Returns:List of variables.
minimize(time, variables, **kwargs)

Performs an optimization step.

Parameters:
  • time – Time tensor.
  • variables – List of variables to optimize.
  • **kwargs –

    Additional optimizer-specific arguments. The following arguments are used by some optimizers:

  • arguments (-) – Dict of arguments for callables, like fn_loss.
  • fn_loss (-) – A callable returning the loss of the current model.
  • fn_reference (-) – A callable returning the reference values, in case of a comparative loss.
  • fn_kl_divergence (-) – A callable returning the KL-divergence relative to the current model.
  • return_estimated_improvement (-) – Returns the estimated improvement resulting from the natural gradient calculation if true.
  • source_variables (-) – List of source variables to synchronize with.
  • global_variables (-) – List of global variables to apply the proposed optimization step to.
Returns:

The optimization operation.
tf_step(time, variables, arguments, fn_loss, **kwargs)

Creates the TensorFlow operations for performing an optimization step.

Parameters:
  • time – Time tensor.
  • variables – List of variables to optimize.
  • arguments – Dict of arguments for callables, like fn_loss.
  • fn_loss – A callable returning the loss of the current model.
  • **kwargs –

    Additional arguments, not used.
Returns:

List of delta tensors corresponding to the updates for each optimized variable.

tensorforce.core.optimizers.global_optimizer module

class tensorforce.core.optimizers.global_optimizer.GlobalOptimizer(optimizer, scope='global-optimizer', summary_labels=())

Bases: tensorforce.core.optimizers.meta_optimizer.MetaOptimizer

The global optimizer applies an optimizer to the local variables. In addition, it also applies the update to a corresponding set of global variables and subsequently updates the local variables to the value of these global variables. Note: This is used for the current distributed mode, and will likely change with the next major version update.

__init__(optimizer, scope='global-optimizer', summary_labels=())

Creates a new global optimizer instance.

Parameters:optimizer – The optimizer which is modified by this meta optimizer.
apply_step(variables, deltas)

Applies step deltas to variable values.

Parameters:
  • variables – List of variables.
  • deltas – List of deltas of same length.
Returns:

The step-applied operation.
from_spec(spec, kwargs=None)

Creates an optimizer from a specification dict.

get_summaries()

Returns the TensorFlow summaries reported by the optimizer.

Returns:List of summaries.
get_variables()
minimize(time, variables, **kwargs)

Performs an optimization step.

Parameters:
  • time – Time tensor.
  • variables – List of variables to optimize.
  • **kwargs –

    Additional optimizer-specific arguments. The following arguments are used by some optimizers:

  • arguments (-) – Dict of arguments for callables, like fn_loss.
  • fn_loss (-) – A callable returning the loss of the current model.
  • fn_reference (-) – A callable returning the reference values, in case of a comparative loss.
  • fn_kl_divergence (-) – A callable returning the KL-divergence relative to the current model.
  • return_estimated_improvement (-) – Returns the estimated improvement resulting from the natural gradient calculation if true.
  • source_variables (-) – List of source variables to synchronize with.
  • global_variables (-) – List of global variables to apply the proposed optimization step to.
Returns:

The optimization operation.
tf_step(time, variables, global_variables, **kwargs)

Creates the TensorFlow operations for performing an optimization step.

Parameters:
  • time – Time tensor.
  • variables – List of variables to optimize.
  • global_variables – List of global variables to apply the proposed optimization step to.
  • **kwargs –

    ??? coming soon
Returns:

List of delta tensors corresponding to the updates for each optimized variable.

tensorforce.core.optimizers.meta_optimizer module

class tensorforce.core.optimizers.meta_optimizer.MetaOptimizer(optimizer, scope='meta-optimizer', summary_labels=(), **kwargs)

Bases: tensorforce.core.optimizers.optimizer.Optimizer

A meta optimizer takes the optimization implemented by another optimizer and modifies/optimizes its proposed result. For example, line search might be applied to find a more optimal step size.

__init__(optimizer, scope='meta-optimizer', summary_labels=(), **kwargs)

Creates a new meta optimizer instance.

Parameters:optimizer – The optimizer which is modified by this meta optimizer.
apply_step(variables, deltas)

Applies step deltas to variable values.

Parameters:
  • variables – List of variables.
  • deltas – List of deltas of same length.
Returns:

The step-applied operation.
from_spec(spec, kwargs=None)

Creates an optimizer from a specification dict.

get_summaries()

Returns the TensorFlow summaries reported by the optimizer.

Returns:List of summaries.
get_variables()
minimize(time, variables, **kwargs)

Performs an optimization step.

Parameters:
  • time – Time tensor.
  • variables – List of variables to optimize.
  • **kwargs –

    Additional optimizer-specific arguments. The following arguments are used by some optimizers:

  • arguments (-) – Dict of arguments for callables, like fn_loss.
  • fn_loss (-) – A callable returning the loss of the current model.
  • fn_reference (-) – A callable returning the reference values, in case of a comparative loss.
  • fn_kl_divergence (-) – A callable returning the KL-divergence relative to the current model.
  • return_estimated_improvement (-) – Returns the estimated improvement resulting from the natural gradient calculation if true.
  • source_variables (-) – List of source variables to synchronize with.
  • global_variables (-) – List of global variables to apply the proposed optimization step to.
Returns:

The optimization operation.
tf_step(time, variables, **kwargs)

Creates the TensorFlow operations for performing an optimization step.

Parameters:
  • time – Time tensor.
  • variables – List of variables to optimize.
  • **kwargs –

    Additional arguments depending on the specific optimizer implementation. For instance, often includes fn_loss if a loss function is optimized.
Returns:

List of delta tensors corresponding to the updates for each optimized variable.

tensorforce.core.optimizers.multi_step module

class tensorforce.core.optimizers.multi_step.MultiStep(optimizer, num_steps=10, unroll_loop=False, scope='multi-step', summary_labels=())

Bases: tensorforce.core.optimizers.meta_optimizer.MetaOptimizer

The multi-step meta optimizer repeatedly applies the optimization step proposed by another optimizer a number of times.

__init__(optimizer, num_steps=10, unroll_loop=False, scope='multi-step', summary_labels=())

Creates a new multi-step meta optimizer instance.

Parameters:
  • optimizer – The optimizer which is modified by this meta optimizer.
  • num_steps – Number of optimization steps to perform.
apply_step(variables, deltas)

Applies step deltas to variable values.

Parameters:
  • variables – List of variables.
  • deltas – List of deltas of same length.
Returns:

The step-applied operation.
from_spec(spec, kwargs=None)

Creates an optimizer from a specification dict.

get_summaries()

Returns the TensorFlow summaries reported by the optimizer.

Returns:List of summaries.
get_variables()
minimize(time, variables, **kwargs)

Performs an optimization step.

Parameters:
  • time – Time tensor.
  • variables – List of variables to optimize.
  • **kwargs –

    Additional optimizer-specific arguments. The following arguments are used by some optimizers:

  • arguments (-) – Dict of arguments for callables, like fn_loss.
  • fn_loss (-) – A callable returning the loss of the current model.
  • fn_reference (-) – A callable returning the reference values, in case of a comparative loss.
  • fn_kl_divergence (-) – A callable returning the KL-divergence relative to the current model.
  • return_estimated_improvement (-) – Returns the estimated improvement resulting from the natural gradient calculation if true.
  • source_variables (-) – List of source variables to synchronize with.
  • global_variables (-) – List of global variables to apply the proposed optimization step to.
Returns:

The optimization operation.
tf_step(time, variables, arguments, fn_reference=None, **kwargs)

Creates the TensorFlow operations for performing an optimization step.

Parameters:
  • time – Time tensor.
  • variables – List of variables to optimize.
  • arguments – Dict of arguments for callables, like fn_loss.
  • fn_reference – A callable returning the reference values, in case of a comparative loss.
  • **kwargs –

    Additional arguments passed on to the internal optimizer.
Returns:

List of delta tensors corresponding to the updates for each optimized variable.

tensorforce.core.optimizers.natural_gradient module

class tensorforce.core.optimizers.natural_gradient.NaturalGradient(learning_rate, cg_max_iterations=20, cg_damping=0.001, cg_unroll_loop=False, scope='natural-gradient', summary_labels=())

Bases: tensorforce.core.optimizers.optimizer.Optimizer

Natural gradient optimizer.

__init__(learning_rate, cg_max_iterations=20, cg_damping=0.001, cg_unroll_loop=False, scope='natural-gradient', summary_labels=())

Creates a new natural gradient optimizer instance.

Parameters:
  • learning_rate – Learning rate, i.e. KL-divergence of distributions between optimization steps.
  • cg_max_iterations – Conjugate gradient solver max iterations.
  • cg_damping – Conjugate gradient solver damping factor.
  • cg_unroll_loop – Unroll conjugate gradient loop if true.
apply_step(variables, deltas)

Applies step deltas to variable values.

Parameters:
  • variables – List of variables.
  • deltas – List of deltas of same length.
Returns:

The step-applied operation.
from_spec(spec, kwargs=None)

Creates an optimizer from a specification dict.

get_summaries()

Returns the TensorFlow summaries reported by the optimizer.

Returns:List of summaries.
get_variables()

Returns the TensorFlow variables used by the optimizer.

Returns:List of variables.
minimize(time, variables, **kwargs)

Performs an optimization step.

Parameters:
  • time – Time tensor.
  • variables – List of variables to optimize.
  • **kwargs –

    Additional optimizer-specific arguments. The following arguments are used by some optimizers:

  • arguments (-) – Dict of arguments for callables, like fn_loss.
  • fn_loss (-) – A callable returning the loss of the current model.
  • fn_reference (-) – A callable returning the reference values, in case of a comparative loss.
  • fn_kl_divergence (-) – A callable returning the KL-divergence relative to the current model.
  • return_estimated_improvement (-) – Returns the estimated improvement resulting from the natural gradient calculation if true.
  • source_variables (-) – List of source variables to synchronize with.
  • global_variables (-) – List of global variables to apply the proposed optimization step to.
Returns:

The optimization operation.
tf_step(time, variables, arguments, fn_loss, fn_kl_divergence, return_estimated_improvement=False, **kwargs)

Creates the TensorFlow operations for performing an optimization step.

Parameters:
  • time – Time tensor.
  • variables – List of variables to optimize.
  • arguments – Dict of arguments for callables, like fn_loss.
  • fn_loss – A callable returning the loss of the current model.
  • fn_kl_divergence – A callable returning the KL-divergence relative to the current model.
  • return_estimated_improvement – Returns the estimated improvement resulting from the natural gradient calculation if true.
  • **kwargs –

    Additional arguments, not used.
Returns:

List of delta tensors corresponding to the updates for each optimized variable.

tensorforce.core.optimizers.optimized_step module

class tensorforce.core.optimizers.optimized_step.OptimizedStep(optimizer, ls_max_iterations=10, ls_accept_ratio=0.9, ls_mode='exponential', ls_parameter=0.5, ls_unroll_loop=False, scope='optimized-step', summary_labels=())

Bases: tensorforce.core.optimizers.meta_optimizer.MetaOptimizer

The optimized-step meta optimizer applies line search to the proposed optimization step of another optimizer to find a more optimal step size.

__init__(optimizer, ls_max_iterations=10, ls_accept_ratio=0.9, ls_mode='exponential', ls_parameter=0.5, ls_unroll_loop=False, scope='optimized-step', summary_labels=())

Creates a new optimized step meta optimizer instance.

Parameters:
  • optimizer – The optimizer which is modified by this meta optimizer.
  • ls_max_iterations – Maximum number of line search iterations.
  • ls_accept_ratio – Line search acceptance ratio.
  • ls_mode – Line search mode, see LineSearch solver.
  • ls_parameter – Line search parameter, see LineSearch solver.
  • ls_unroll_loop – Unroll line search loop if true.
apply_step(variables, deltas)

Applies step deltas to variable values.

Parameters:
  • variables – List of variables.
  • deltas – List of deltas of same length.
Returns:

The step-applied operation.
from_spec(spec, kwargs=None)

Creates an optimizer from a specification dict.

get_summaries()

Returns the TensorFlow summaries reported by the optimizer.

Returns:List of summaries.
get_variables()
minimize(time, variables, **kwargs)

Performs an optimization step.

Parameters:
  • time – Time tensor.
  • variables – List of variables to optimize.
  • **kwargs –

    Additional optimizer-specific arguments. The following arguments are used by some optimizers:

  • arguments (-) – Dict of arguments for callables, like fn_loss.
  • fn_loss (-) – A callable returning the loss of the current model.
  • fn_reference (-) – A callable returning the reference values, in case of a comparative loss.
  • fn_kl_divergence (-) – A callable returning the KL-divergence relative to the current model.
  • return_estimated_improvement (-) – Returns the estimated improvement resulting from the natural gradient calculation if true.
  • source_variables (-) – List of source variables to synchronize with.
  • global_variables (-) – List of global variables to apply the proposed optimization step to.
Returns:

The optimization operation.
tf_step(time, variables, arguments, fn_loss, fn_reference, **kwargs)

Creates the TensorFlow operations for performing an optimization step.

Parameters:
  • time – Time tensor.
  • variables – List of variables to optimize.
  • arguments – Dict of arguments for callables, like fn_loss.
  • fn_loss – A callable returning the loss of the current model.
  • fn_reference – A callable returning the reference values, in case of a comparative loss.
  • **kwargs –

    Additional arguments passed on to the internal optimizer.
Returns:

List of delta tensors corresponding to the updates for each optimized variable.

tensorforce.core.optimizers.optimizer module

class tensorforce.core.optimizers.optimizer.Optimizer(scope='optimizer', summary_labels=None)

Bases: object

Base class for optimizers which minimize a not yet further specified expression, usually some kind of loss function. More generally, an optimizer can be considered as some method of updating a set of variables.

__init__(scope='optimizer', summary_labels=None)

Creates a new optimizer instance.

apply_step(variables, deltas)

Applies step deltas to variable values.

Parameters:
  • variables – List of variables.
  • deltas – List of deltas of same length.
Returns:

The step-applied operation.
static from_spec(spec, kwargs=None)

Creates an optimizer from a specification dict.

get_summaries()

Returns the TensorFlow summaries reported by the optimizer.

Returns:List of summaries.
get_variables()

Returns the TensorFlow variables used by the optimizer.

Returns:List of variables.
minimize(time, variables, **kwargs)

Performs an optimization step.

Parameters:
  • time – Time tensor.
  • variables – List of variables to optimize.
  • **kwargs –

    Additional optimizer-specific arguments. The following arguments are used by some optimizers:

  • arguments (-) – Dict of arguments for callables, like fn_loss.
  • fn_loss (-) – A callable returning the loss of the current model.
  • fn_reference (-) – A callable returning the reference values, in case of a comparative loss.
  • fn_kl_divergence (-) – A callable returning the KL-divergence relative to the current model.
  • return_estimated_improvement (-) – Returns the estimated improvement resulting from the natural gradient calculation if true.
  • source_variables (-) – List of source variables to synchronize with.
  • global_variables (-) – List of global variables to apply the proposed optimization step to.
Returns:

The optimization operation.
tf_step(time, variables, **kwargs)

Creates the TensorFlow operations for performing an optimization step.

Parameters:
  • time – Time tensor.
  • variables – List of variables to optimize.
  • **kwargs –

    Additional arguments depending on the specific optimizer implementation. For instance, often includes fn_loss if a loss function is optimized.
Returns:

List of delta tensors corresponding to the updates for each optimized variable.

tensorforce.core.optimizers.synchronization module

class tensorforce.core.optimizers.synchronization.Synchronization(sync_frequency=1, update_weight=1.0, scope='synchronization', summary_labels=())

Bases: tensorforce.core.optimizers.optimizer.Optimizer

The synchronization optimizer updates variables periodically to the value of a corresponding set of source variables.

__init__(sync_frequency=1, update_weight=1.0, scope='synchronization', summary_labels=())

Creates a new synchronization optimizer instance.

Parameters:
  • sync_frequency – The interval between optimization calls actually performing a
  • step. (synchronization) –
  • update_weight – The update weight, 1.0 meaning a full assignment of the source
  • values. (variables) –
apply_step(variables, deltas)

Applies step deltas to variable values.

Parameters:
  • variables – List of variables.
  • deltas – List of deltas of same length.
Returns:

The step-applied operation.
from_spec(spec, kwargs=None)

Creates an optimizer from a specification dict.

get_summaries()

Returns the TensorFlow summaries reported by the optimizer.

Returns:List of summaries.
get_variables()

Returns the TensorFlow variables used by the optimizer.

Returns:List of variables.
minimize(time, variables, **kwargs)

Performs an optimization step.

Parameters:
  • time – Time tensor.
  • variables – List of variables to optimize.
  • **kwargs –

    Additional optimizer-specific arguments. The following arguments are used by some optimizers:

  • arguments (-) – Dict of arguments for callables, like fn_loss.
  • fn_loss (-) – A callable returning the loss of the current model.
  • fn_reference (-) – A callable returning the reference values, in case of a comparative loss.
  • fn_kl_divergence (-) – A callable returning the KL-divergence relative to the current model.
  • return_estimated_improvement (-) – Returns the estimated improvement resulting from the natural gradient calculation if true.
  • source_variables (-) – List of source variables to synchronize with.
  • global_variables (-) – List of global variables to apply the proposed optimization step to.
Returns:

The optimization operation.
tf_step(time, variables, source_variables, **kwargs)

Creates the TensorFlow operations for performing an optimization step.

Parameters:
  • time – Time tensor.
  • variables – List of variables to optimize.
  • source_variables – List of source variables to synchronize with.
  • **kwargs –

    Additional arguments, not used.
Returns:

List of delta tensors corresponding to the updates for each optimized variable.

tensorforce.core.optimizers.tf_optimizer module

class tensorforce.core.optimizers.tf_optimizer.TFOptimizer(optimizer, scope=None, summary_labels=(), **kwargs)

Bases: tensorforce.core.optimizers.optimizer.Optimizer

Wrapper class for TensorFlow optimizers.

__init__(optimizer, scope=None, summary_labels=(), **kwargs)

Creates a new optimizer instance of a TensorFlow optimizer.

Parameters:
  • optimizer – The name of the optimizer, one of ‘adadelta’, ‘adagrad’, ‘adam’, ‘nadam’,
  • 'momentum', 'rmsprop'. ('gradient_descent',) –
  • **kwargs –

    Additional arguments passed on to the TensorFlow optimizer constructor.
apply_step(variables, deltas)

Applies step deltas to variable values.

Parameters:
  • variables – List of variables.
  • deltas – List of deltas of same length.
Returns:

The step-applied operation.
from_spec(spec, kwargs=None)

Creates an optimizer from a specification dict.

get_summaries()

Returns the TensorFlow summaries reported by the optimizer.

Returns:List of summaries.
get_variables()
static get_wrapper(optimizer)

Returns a TFOptimizer constructor callable for the given optimizer name.

Parameters:
  • optimizer – The name of the optimizer, one of ‘adadelta’, ‘adagrad’, ‘adam’, ‘nadam’,
  • 'momentum', 'rmsprop'. ('gradient_descent',) –
Returns:

The TFOptimizer constructor callable.
minimize(time, variables, **kwargs)

Performs an optimization step.

Parameters:
  • time – Time tensor.
  • variables – List of variables to optimize.
  • **kwargs –

    Additional optimizer-specific arguments. The following arguments are used by some optimizers:

  • arguments (-) – Dict of arguments for callables, like fn_loss.
  • fn_loss (-) – A callable returning the loss of the current model.
  • fn_reference (-) – A callable returning the reference values, in case of a comparative loss.
  • fn_kl_divergence (-) – A callable returning the KL-divergence relative to the current model.
  • return_estimated_improvement (-) – Returns the estimated improvement resulting from the natural gradient calculation if true.
  • source_variables (-) – List of source variables to synchronize with.
  • global_variables (-) – List of global variables to apply the proposed optimization step to.
Returns:

The optimization operation.
tf_optimizers = {'nadam': <sphinx.ext.autodoc._MockObject object>, 'adam': <sphinx.ext.autodoc._MockObject object>, 'adadelta': <sphinx.ext.autodoc._MockObject object>, 'rmsprop': <sphinx.ext.autodoc._MockObject object>, 'adagrad': <sphinx.ext.autodoc._MockObject object>, 'momentum': <sphinx.ext.autodoc._MockObject object>, 'gradient_descent': <sphinx.ext.autodoc._MockObject object>}
tf_step(time, variables, arguments, fn_loss, **kwargs)

Creates the TensorFlow operations for performing an optimization step.

Parameters:
  • time – Time tensor.
  • variables – List of variables to optimize.
  • arguments – Dict of arguments for callables, like fn_loss.
  • fn_loss – A callable returning the loss of the current model.
  • **kwargs –

    Additional arguments, not used.
Returns:

List of delta tensors corresponding to the updates for each optimized variable.

Module contents

class tensorforce.core.optimizers.Optimizer(scope='optimizer', summary_labels=None)

Bases: object

Base class for optimizers which minimize a not yet further specified expression, usually some kind of loss function. More generally, an optimizer can be considered as some method of updating a set of variables.

__init__(scope='optimizer', summary_labels=None)

Creates a new optimizer instance.

apply_step(variables, deltas)

Applies step deltas to variable values.

Parameters:
  • variables – List of variables.
  • deltas – List of deltas of same length.
Returns:

The step-applied operation.
static from_spec(spec, kwargs=None)

Creates an optimizer from a specification dict.

get_summaries()

Returns the TensorFlow summaries reported by the optimizer.

Returns:List of summaries.
get_variables()

Returns the TensorFlow variables used by the optimizer.

Returns:List of variables.
minimize(time, variables, **kwargs)

Performs an optimization step.

Parameters:
  • time – Time tensor.
  • variables – List of variables to optimize.
  • **kwargs –

    Additional optimizer-specific arguments. The following arguments are used by some optimizers:

  • arguments (-) – Dict of arguments for callables, like fn_loss.
  • fn_loss (-) – A callable returning the loss of the current model.
  • fn_reference (-) – A callable returning the reference values, in case of a comparative loss.
  • fn_kl_divergence (-) – A callable returning the KL-divergence relative to the current model.
  • return_estimated_improvement (-) – Returns the estimated improvement resulting from the natural gradient calculation if true.
  • source_variables (-) – List of source variables to synchronize with.
  • global_variables (-) – List of global variables to apply the proposed optimization step to.
Returns:

The optimization operation.
tf_step(time, variables, **kwargs)

Creates the TensorFlow operations for performing an optimization step.

Parameters:
  • time – Time tensor.
  • variables – List of variables to optimize.
  • **kwargs –

    Additional arguments depending on the specific optimizer implementation. For instance, often includes fn_loss if a loss function is optimized.
Returns:

List of delta tensors corresponding to the updates for each optimized variable.
class tensorforce.core.optimizers.MetaOptimizer(optimizer, scope='meta-optimizer', summary_labels=(), **kwargs)

Bases: tensorforce.core.optimizers.optimizer.Optimizer

A meta optimizer takes the optimization implemented by another optimizer and modifies/optimizes its proposed result. For example, line search might be applied to find a more optimal step size.

__init__(optimizer, scope='meta-optimizer', summary_labels=(), **kwargs)

Creates a new meta optimizer instance.

Parameters:optimizer – The optimizer which is modified by this meta optimizer.
apply_step(variables, deltas)

Applies step deltas to variable values.

Parameters:
  • variables – List of variables.
  • deltas – List of deltas of same length.
Returns:

The step-applied operation.
from_spec(spec, kwargs=None)

Creates an optimizer from a specification dict.

get_summaries()

Returns the TensorFlow summaries reported by the optimizer.

Returns:List of summaries.
get_variables()
minimize(time, variables, **kwargs)

Performs an optimization step.

Parameters:
  • time – Time tensor.
  • variables – List of variables to optimize.
  • **kwargs –

    Additional optimizer-specific arguments. The following arguments are used by some optimizers:

  • arguments (-) – Dict of arguments for callables, like fn_loss.
  • fn_loss (-) – A callable returning the loss of the current model.
  • fn_reference (-) – A callable returning the reference values, in case of a comparative loss.
  • fn_kl_divergence (-) – A callable returning the KL-divergence relative to the current model.
  • return_estimated_improvement (-) – Returns the estimated improvement resulting from the natural gradient calculation if true.
  • source_variables (-) – List of source variables to synchronize with.
  • global_variables (-) – List of global variables to apply the proposed optimization step to.
Returns:

The optimization operation.
tf_step(time, variables, **kwargs)

Creates the TensorFlow operations for performing an optimization step.

Parameters:
  • time – Time tensor.
  • variables – List of variables to optimize.
  • **kwargs –

    Additional arguments depending on the specific optimizer implementation. For instance, often includes fn_loss if a loss function is optimized.
Returns:

List of delta tensors corresponding to the updates for each optimized variable.
class tensorforce.core.optimizers.GlobalOptimizer(optimizer, scope='global-optimizer', summary_labels=())

Bases: tensorforce.core.optimizers.meta_optimizer.MetaOptimizer

The global optimizer applies an optimizer to the local variables. In addition, it also applies the update to a corresponding set of global variables and subsequently updates the local variables to the value of these global variables. Note: This is used for the current distributed mode, and will likely change with the next major version update.

__init__(optimizer, scope='global-optimizer', summary_labels=())

Creates a new global optimizer instance.

Parameters:optimizer – The optimizer which is modified by this meta optimizer.
apply_step(variables, deltas)

Applies step deltas to variable values.

Parameters:
  • variables – List of variables.
  • deltas – List of deltas of same length.
Returns:

The step-applied operation.
from_spec(spec, kwargs=None)

Creates an optimizer from a specification dict.

get_summaries()

Returns the TensorFlow summaries reported by the optimizer.

Returns:List of summaries.
get_variables()
minimize(time, variables, **kwargs)

Performs an optimization step.

Parameters:
  • time – Time tensor.
  • variables – List of variables to optimize.
  • **kwargs –

    Additional optimizer-specific arguments. The following arguments are used by some optimizers:

  • arguments (-) – Dict of arguments for callables, like fn_loss.
  • fn_loss (-) – A callable returning the loss of the current model.
  • fn_reference (-) – A callable returning the reference values, in case of a comparative loss.
  • fn_kl_divergence (-) – A callable returning the KL-divergence relative to the current model.
  • return_estimated_improvement (-) – Returns the estimated improvement resulting from the natural gradient calculation if true.
  • source_variables (-) – List of source variables to synchronize with.
  • global_variables (-) – List of global variables to apply the proposed optimization step to.
Returns:

The optimization operation.
tf_step(time, variables, global_variables, **kwargs)

Creates the TensorFlow operations for performing an optimization step.

Parameters:
  • time – Time tensor.
  • variables – List of variables to optimize.
  • global_variables – List of global variables to apply the proposed optimization step to.
  • **kwargs –

    ??? coming soon
Returns:

List of delta tensors corresponding to the updates for each optimized variable.
class tensorforce.core.optimizers.TFOptimizer(optimizer, scope=None, summary_labels=(), **kwargs)

Bases: tensorforce.core.optimizers.optimizer.Optimizer

Wrapper class for TensorFlow optimizers.

__init__(optimizer, scope=None, summary_labels=(), **kwargs)

Creates a new optimizer instance of a TensorFlow optimizer.

Parameters:
  • optimizer – The name of the optimizer, one of ‘adadelta’, ‘adagrad’, ‘adam’, ‘nadam’,
  • 'momentum', 'rmsprop'. ('gradient_descent',) –
  • **kwargs –

    Additional arguments passed on to the TensorFlow optimizer constructor.
apply_step(variables, deltas)

Applies step deltas to variable values.

Parameters:
  • variables – List of variables.
  • deltas – List of deltas of same length.
Returns:

The step-applied operation.
from_spec(spec, kwargs=None)

Creates an optimizer from a specification dict.

get_summaries()

Returns the TensorFlow summaries reported by the optimizer.

Returns:List of summaries.
get_variables()
static get_wrapper(optimizer)

Returns a TFOptimizer constructor callable for the given optimizer name.

Parameters:
  • optimizer – The name of the optimizer, one of ‘adadelta’, ‘adagrad’, ‘adam’, ‘nadam’,
  • 'momentum', 'rmsprop'. ('gradient_descent',) –
Returns:

The TFOptimizer constructor callable.
minimize(time, variables, **kwargs)

Performs an optimization step.

Parameters:
  • time – Time tensor.
  • variables – List of variables to optimize.
  • **kwargs –

    Additional optimizer-specific arguments. The following arguments are used by some optimizers:

  • arguments (-) – Dict of arguments for callables, like fn_loss.
  • fn_loss (-) – A callable returning the loss of the current model.
  • fn_reference (-) – A callable returning the reference values, in case of a comparative loss.
  • fn_kl_divergence (-) – A callable returning the KL-divergence relative to the current model.
  • return_estimated_improvement (-) – Returns the estimated improvement resulting from the natural gradient calculation if true.
  • source_variables (-) – List of source variables to synchronize with.
  • global_variables (-) – List of global variables to apply the proposed optimization step to.
Returns:

The optimization operation.
tf_optimizers = {'nadam': <sphinx.ext.autodoc._MockObject object>, 'adam': <sphinx.ext.autodoc._MockObject object>, 'adadelta': <sphinx.ext.autodoc._MockObject object>, 'rmsprop': <sphinx.ext.autodoc._MockObject object>, 'adagrad': <sphinx.ext.autodoc._MockObject object>, 'momentum': <sphinx.ext.autodoc._MockObject object>, 'gradient_descent': <sphinx.ext.autodoc._MockObject object>}
tf_step(time, variables, arguments, fn_loss, **kwargs)

Creates the TensorFlow operations for performing an optimization step.

Parameters:
  • time – Time tensor.
  • variables – List of variables to optimize.
  • arguments – Dict of arguments for callables, like fn_loss.
  • fn_loss – A callable returning the loss of the current model.
  • **kwargs –

    Additional arguments, not used.
Returns:

List of delta tensors corresponding to the updates for each optimized variable.
class tensorforce.core.optimizers.Evolutionary(learning_rate, num_samples=1, unroll_loop=False, scope='evolutionary', summary_labels=())

Bases: tensorforce.core.optimizers.optimizer.Optimizer

Evolutionary optimizer which samples random perturbations and applies them either positively or negatively, depending on their improvement of the loss.

__init__(learning_rate, num_samples=1, unroll_loop=False, scope='evolutionary', summary_labels=())

Creates a new evolutionary optimizer instance.

Parameters:
  • learning_rate – Learning rate.
  • num_samples – Number of sampled perturbations.
apply_step(variables, deltas)

Applies step deltas to variable values.

Parameters:
  • variables – List of variables.
  • deltas – List of deltas of same length.
Returns:

The step-applied operation.
from_spec(spec, kwargs=None)

Creates an optimizer from a specification dict.

get_summaries()

Returns the TensorFlow summaries reported by the optimizer.

Returns:List of summaries.
get_variables()

Returns the TensorFlow variables used by the optimizer.

Returns:List of variables.
minimize(time, variables, **kwargs)

Performs an optimization step.

Parameters:
  • time – Time tensor.
  • variables – List of variables to optimize.
  • **kwargs –

    Additional optimizer-specific arguments. The following arguments are used by some optimizers:

  • arguments (-) – Dict of arguments for callables, like fn_loss.
  • fn_loss (-) – A callable returning the loss of the current model.
  • fn_reference (-) – A callable returning the reference values, in case of a comparative loss.
  • fn_kl_divergence (-) – A callable returning the KL-divergence relative to the current model.
  • return_estimated_improvement (-) – Returns the estimated improvement resulting from the natural gradient calculation if true.
  • source_variables (-) – List of source variables to synchronize with.
  • global_variables (-) – List of global variables to apply the proposed optimization step to.
Returns:

The optimization operation.
tf_step(time, variables, arguments, fn_loss, **kwargs)

Creates the TensorFlow operations for performing an optimization step.

Parameters:
  • time – Time tensor.
  • variables – List of variables to optimize.
  • arguments – Dict of arguments for callables, like fn_loss.
  • fn_loss – A callable returning the loss of the current model.
  • **kwargs –

    Additional arguments, not used.
Returns:

List of delta tensors corresponding to the updates for each optimized variable.
class tensorforce.core.optimizers.NaturalGradient(learning_rate, cg_max_iterations=20, cg_damping=0.001, cg_unroll_loop=False, scope='natural-gradient', summary_labels=())

Bases: tensorforce.core.optimizers.optimizer.Optimizer

Natural gradient optimizer.

__init__(learning_rate, cg_max_iterations=20, cg_damping=0.001, cg_unroll_loop=False, scope='natural-gradient', summary_labels=())

Creates a new natural gradient optimizer instance.

Parameters:
  • learning_rate – Learning rate, i.e. KL-divergence of distributions between optimization steps.
  • cg_max_iterations – Conjugate gradient solver max iterations.
  • cg_damping – Conjugate gradient solver damping factor.
  • cg_unroll_loop – Unroll conjugate gradient loop if true.
apply_step(variables, deltas)

Applies step deltas to variable values.

Parameters:
  • variables – List of variables.
  • deltas – List of deltas of same length.
Returns:

The step-applied operation.
from_spec(spec, kwargs=None)

Creates an optimizer from a specification dict.

get_summaries()

Returns the TensorFlow summaries reported by the optimizer.

Returns:List of summaries.
get_variables()

Returns the TensorFlow variables used by the optimizer.

Returns:List of variables.
minimize(time, variables, **kwargs)

Performs an optimization step.

Parameters:
  • time – Time tensor.
  • variables – List of variables to optimize.
  • **kwargs –

    Additional optimizer-specific arguments. The following arguments are used by some optimizers:

  • arguments (-) – Dict of arguments for callables, like fn_loss.
  • fn_loss (-) – A callable returning the loss of the current model.
  • fn_reference (-) – A callable returning the reference values, in case of a comparative loss.
  • fn_kl_divergence (-) – A callable returning the KL-divergence relative to the current model.
  • return_estimated_improvement (-) – Returns the estimated improvement resulting from the natural gradient calculation if true.
  • source_variables (-) – List of source variables to synchronize with.
  • global_variables (-) – List of global variables to apply the proposed optimization step to.
Returns:

The optimization operation.
tf_step(time, variables, arguments, fn_loss, fn_kl_divergence, return_estimated_improvement=False, **kwargs)

Creates the TensorFlow operations for performing an optimization step.

Parameters:
  • time – Time tensor.
  • variables – List of variables to optimize.
  • arguments – Dict of arguments for callables, like fn_loss.
  • fn_loss – A callable returning the loss of the current model.
  • fn_kl_divergence – A callable returning the KL-divergence relative to the current model.
  • return_estimated_improvement – Returns the estimated improvement resulting from the natural gradient calculation if true.
  • **kwargs –

    Additional arguments, not used.
Returns:

List of delta tensors corresponding to the updates for each optimized variable.
class tensorforce.core.optimizers.ClippedStep(optimizer, clipping_value, scope='clipped-step', summary_labels=())

Bases: tensorforce.core.optimizers.meta_optimizer.MetaOptimizer

The clipped-shep meta optimizer clips the values of the optimization step proposed by another optimizer.

__init__(optimizer, clipping_value, scope='clipped-step', summary_labels=())

Creates a new multi-step meta optimizer instance.

Parameters:
  • optimizer – The optimizer which is modified by this meta optimizer.
  • clipping_value – Clip deltas at this value.
apply_step(variables, deltas)

Applies step deltas to variable values.

Parameters:
  • variables – List of variables.
  • deltas – List of deltas of same length.
Returns:

The step-applied operation.
from_spec(spec, kwargs=None)

Creates an optimizer from a specification dict.

get_summaries()

Returns the TensorFlow summaries reported by the optimizer.

Returns:List of summaries.
get_variables()
minimize(time, variables, **kwargs)

Performs an optimization step.

Parameters:
  • time – Time tensor.
  • variables – List of variables to optimize.
  • **kwargs –

    Additional optimizer-specific arguments. The following arguments are used by some optimizers:

  • arguments (-) – Dict of arguments for callables, like fn_loss.
  • fn_loss (-) – A callable returning the loss of the current model.
  • fn_reference (-) – A callable returning the reference values, in case of a comparative loss.
  • fn_kl_divergence (-) – A callable returning the KL-divergence relative to the current model.
  • return_estimated_improvement (-) – Returns the estimated improvement resulting from the natural gradient calculation if true.
  • source_variables (-) – List of source variables to synchronize with.
  • global_variables (-) – List of global variables to apply the proposed optimization step to.
Returns:

The optimization operation.
tf_step(time, variables, **kwargs)

Creates the TensorFlow operations for performing an optimization step.

Parameters:
  • time – Time tensor.
  • variables – List of variables to optimize.
  • **kwargs –

    Additional arguments passed on to the internal optimizer.
Returns:

List of delta tensors corresponding to the updates for each optimized variable.
class tensorforce.core.optimizers.MultiStep(optimizer, num_steps=10, unroll_loop=False, scope='multi-step', summary_labels=())

Bases: tensorforce.core.optimizers.meta_optimizer.MetaOptimizer

The multi-step meta optimizer repeatedly applies the optimization step proposed by another optimizer a number of times.

__init__(optimizer, num_steps=10, unroll_loop=False, scope='multi-step', summary_labels=())

Creates a new multi-step meta optimizer instance.

Parameters:
  • optimizer – The optimizer which is modified by this meta optimizer.
  • num_steps – Number of optimization steps to perform.
apply_step(variables, deltas)

Applies step deltas to variable values.

Parameters:
  • variables – List of variables.
  • deltas – List of deltas of same length.
Returns:

The step-applied operation.
from_spec(spec, kwargs=None)

Creates an optimizer from a specification dict.

get_summaries()

Returns the TensorFlow summaries reported by the optimizer.

Returns:List of summaries.
get_variables()
minimize(time, variables, **kwargs)

Performs an optimization step.

Parameters:
  • time – Time tensor.
  • variables – List of variables to optimize.
  • **kwargs –

    Additional optimizer-specific arguments. The following arguments are used by some optimizers:

  • arguments (-) – Dict of arguments for callables, like fn_loss.
  • fn_loss (-) – A callable returning the loss of the current model.
  • fn_reference (-) – A callable returning the reference values, in case of a comparative loss.
  • fn_kl_divergence (-) – A callable returning the KL-divergence relative to the current model.
  • return_estimated_improvement (-) – Returns the estimated improvement resulting from the natural gradient calculation if true.
  • source_variables (-) – List of source variables to synchronize with.
  • global_variables (-) – List of global variables to apply the proposed optimization step to.
Returns:

The optimization operation.
tf_step(time, variables, arguments, fn_reference=None, **kwargs)

Creates the TensorFlow operations for performing an optimization step.

Parameters:
  • time – Time tensor.
  • variables – List of variables to optimize.
  • arguments – Dict of arguments for callables, like fn_loss.
  • fn_reference – A callable returning the reference values, in case of a comparative loss.
  • **kwargs –

    Additional arguments passed on to the internal optimizer.
Returns:

List of delta tensors corresponding to the updates for each optimized variable.
class tensorforce.core.optimizers.OptimizedStep(optimizer, ls_max_iterations=10, ls_accept_ratio=0.9, ls_mode='exponential', ls_parameter=0.5, ls_unroll_loop=False, scope='optimized-step', summary_labels=())

Bases: tensorforce.core.optimizers.meta_optimizer.MetaOptimizer

The optimized-step meta optimizer applies line search to the proposed optimization step of another optimizer to find a more optimal step size.

__init__(optimizer, ls_max_iterations=10, ls_accept_ratio=0.9, ls_mode='exponential', ls_parameter=0.5, ls_unroll_loop=False, scope='optimized-step', summary_labels=())

Creates a new optimized step meta optimizer instance.

Parameters:
  • optimizer – The optimizer which is modified by this meta optimizer.
  • ls_max_iterations – Maximum number of line search iterations.
  • ls_accept_ratio – Line search acceptance ratio.
  • ls_mode – Line search mode, see LineSearch solver.
  • ls_parameter – Line search parameter, see LineSearch solver.
  • ls_unroll_loop – Unroll line search loop if true.
apply_step(variables, deltas)

Applies step deltas to variable values.

Parameters:
  • variables – List of variables.
  • deltas – List of deltas of same length.
Returns:

The step-applied operation.
from_spec(spec, kwargs=None)

Creates an optimizer from a specification dict.

get_summaries()

Returns the TensorFlow summaries reported by the optimizer.

Returns:List of summaries.
get_variables()
minimize(time, variables, **kwargs)

Performs an optimization step.

Parameters:
  • time – Time tensor.
  • variables – List of variables to optimize.
  • **kwargs –

    Additional optimizer-specific arguments. The following arguments are used by some optimizers:

  • arguments (-) – Dict of arguments for callables, like fn_loss.
  • fn_loss (-) – A callable returning the loss of the current model.
  • fn_reference (-) – A callable returning the reference values, in case of a comparative loss.
  • fn_kl_divergence (-) – A callable returning the KL-divergence relative to the current model.
  • return_estimated_improvement (-) – Returns the estimated improvement resulting from the natural gradient calculation if true.
  • source_variables (-) – List of source variables to synchronize with.
  • global_variables (-) – List of global variables to apply the proposed optimization step to.
Returns:

The optimization operation.
tf_step(time, variables, arguments, fn_loss, fn_reference, **kwargs)

Creates the TensorFlow operations for performing an optimization step.

Parameters:
  • time – Time tensor.
  • variables – List of variables to optimize.
  • arguments – Dict of arguments for callables, like fn_loss.
  • fn_loss – A callable returning the loss of the current model.
  • fn_reference – A callable returning the reference values, in case of a comparative loss.
  • **kwargs –

    Additional arguments passed on to the internal optimizer.
Returns:

List of delta tensors corresponding to the updates for each optimized variable.
class tensorforce.core.optimizers.SubsamplingStep(optimizer, fraction=0.1, scope='subsampling-step', summary_labels=())

Bases: tensorforce.core.optimizers.meta_optimizer.MetaOptimizer

The subsampling-step meta optimizer randomly samples a subset of batch instances to calculate the optimization step of another optimizer.

__init__(optimizer, fraction=0.1, scope='subsampling-step', summary_labels=())

Creates a new subsampling-step meta optimizer instance.

Parameters:
  • optimizer – The optimizer which is modified by this meta optimizer.
  • fraction – The fraction of instances of the batch to subsample.
apply_step(variables, deltas)

Applies step deltas to variable values.

Parameters:
  • variables – List of variables.
  • deltas – List of deltas of same length.
Returns:

The step-applied operation.
from_spec(spec, kwargs=None)

Creates an optimizer from a specification dict.

get_summaries()

Returns the TensorFlow summaries reported by the optimizer.

Returns:List of summaries.
get_variables()
minimize(time, variables, **kwargs)

Performs an optimization step.

Parameters:
  • time – Time tensor.
  • variables – List of variables to optimize.
  • **kwargs –

    Additional optimizer-specific arguments. The following arguments are used by some optimizers:

  • arguments (-) – Dict of arguments for callables, like fn_loss.
  • fn_loss (-) – A callable returning the loss of the current model.
  • fn_reference (-) – A callable returning the reference values, in case of a comparative loss.
  • fn_kl_divergence (-) – A callable returning the KL-divergence relative to the current model.
  • return_estimated_improvement (-) – Returns the estimated improvement resulting from the natural gradient calculation if true.
  • source_variables (-) – List of source variables to synchronize with.
  • global_variables (-) – List of global variables to apply the proposed optimization step to.
Returns:

The optimization operation.
tf_step(time, variables, arguments, **kwargs)

Creates the TensorFlow operations for performing an optimization step.

Parameters:
  • time – Time tensor.
  • variables – List of variables to optimize.
  • arguments – Dict of arguments for callables, like fn_loss.
  • **kwargs –

    Additional arguments passed on to the internal optimizer.
Returns:

List of delta tensors corresponding to the updates for each optimized variable.
class tensorforce.core.optimizers.Synchronization(sync_frequency=1, update_weight=1.0, scope='synchronization', summary_labels=())

Bases: tensorforce.core.optimizers.optimizer.Optimizer

The synchronization optimizer updates variables periodically to the value of a corresponding set of source variables.

__init__(sync_frequency=1, update_weight=1.0, scope='synchronization', summary_labels=())

Creates a new synchronization optimizer instance.

Parameters:
  • sync_frequency – The interval between optimization calls actually performing a
  • step. (synchronization) –
  • update_weight – The update weight, 1.0 meaning a full assignment of the source
  • values. (variables) –
apply_step(variables, deltas)

Applies step deltas to variable values.

Parameters:
  • variables – List of variables.
  • deltas – List of deltas of same length.
Returns:

The step-applied operation.
from_spec(spec, kwargs=None)

Creates an optimizer from a specification dict.

get_summaries()

Returns the TensorFlow summaries reported by the optimizer.

Returns:List of summaries.
get_variables()

Returns the TensorFlow variables used by the optimizer.

Returns:List of variables.
minimize(time, variables, **kwargs)

Performs an optimization step.

Parameters:
  • time – Time tensor.
  • variables – List of variables to optimize.
  • **kwargs –

    Additional optimizer-specific arguments. The following arguments are used by some optimizers:

  • arguments (-) – Dict of arguments for callables, like fn_loss.
  • fn_loss (-) – A callable returning the loss of the current model.
  • fn_reference (-) – A callable returning the reference values, in case of a comparative loss.
  • fn_kl_divergence (-) – A callable returning the KL-divergence relative to the current model.
  • return_estimated_improvement (-) – Returns the estimated improvement resulting from the natural gradient calculation if true.
  • source_variables (-) – List of source variables to synchronize with.
  • global_variables (-) – List of global variables to apply the proposed optimization step to.
Returns:

The optimization operation.
tf_step(time, variables, source_variables, **kwargs)

Creates the TensorFlow operations for performing an optimization step.

Parameters:
  • time – Time tensor.
  • variables – List of variables to optimize.
  • source_variables – List of source variables to synchronize with.
  • **kwargs –

    Additional arguments, not used.
Returns:

List of delta tensors corresponding to the updates for each optimized variable.