tensorforce.core.optimizers package

Submodules

tensorforce.core.optimizers.clipped_step module

class tensorforce.core.optimizers.clipped_step.ClippedStep(optimizer, clipping_value, summaries=None, summary_labels=None)

Bases: tensorforce.core.optimizers.meta_optimizer.MetaOptimizer

The multi-shep meta optimizer repeatedly applies the optimization step proposed by another optimizer a number of times.

tf_step(time, variables, **kwargs)

Creates the TensorFlow operations for performing an optimization step.

Parameters:
  • time – Time tensor.
  • variables – List of variables to optimize.
  • **kwargs –

    Additional arguments passed on to the internal optimizer.
Returns:

List of delta tensors corresponding to the updates for each optimized variable.

tensorforce.core.optimizers.evolutionary module

class tensorforce.core.optimizers.evolutionary.Evolutionary(learning_rate, num_samples=1, summaries=None, summary_labels=None)

Bases: tensorforce.core.optimizers.optimizer.Optimizer

Evolutionary optimizer which samples random perturbations and applies them either positively or negatively, depending on their improvement of the loss.

tf_step(time, variables, fn_loss, **kwargs)

Creates the TensorFlow operations for performing an optimization step.

Parameters:
  • time – Time tensor.
  • variables – List of variables to optimize.
  • fn_loss – A callable returning the loss of the current model.
  • **kwargs –

    Additional arguments, not used.
Returns:

List of delta tensors corresponding to the updates for each optimized variable.

tensorforce.core.optimizers.global_optimizer module

class tensorforce.core.optimizers.global_optimizer.GlobalOptimizer(optimizer, summaries=None, summary_labels=None)

Bases: tensorforce.core.optimizers.meta_optimizer.MetaOptimizer

The global optimizer applies an optimizer to the local variables. In addition, it also applies the update to a corresponding set of global variables and subsequently updates the local variables to the value of these global variables. Note: This is used for the current distributed mode, and will likely change with the next major version update.

tf_step(time, variables, global_variables, **kwargs)

Creates the TensorFlow operations for performing an optimization step.

Parameters:
  • time – Time tensor.
  • variables – List of variables to optimize.
  • global_variables – List of global variables to apply the proposed optimization step to.
  • **kwargs –

    ??? coming soon
Returns:

List of delta tensors corresponding to the updates for each optimized variable.

tensorforce.core.optimizers.meta_optimizer module

class tensorforce.core.optimizers.meta_optimizer.MetaOptimizer(optimizer, **kwargs)

Bases: tensorforce.core.optimizers.optimizer.Optimizer

A meta optimizer takes the optimization implemented by another optimizer and modifies/optimizes its proposed result. For example, line search might be applied to find a more optimal step size.

get_variables()

tensorforce.core.optimizers.multi_step module

class tensorforce.core.optimizers.multi_step.MultiStep(optimizer, num_steps=5, summaries=None, summary_labels=None)

Bases: tensorforce.core.optimizers.meta_optimizer.MetaOptimizer

The multi-step meta optimizer repeatedly applies the optimization step proposed by another optimizer a number of times.

tf_step(time, variables, **kwargs)

Creates the TensorFlow operations for performing an optimization step.

Parameters:
  • time – Time tensor.
  • variables – List of variables to optimize.
  • **kwargs –

    Additional arguments passed on to the internal optimizer.
Returns:

List of delta tensors corresponding to the updates for each optimized variable.

tensorforce.core.optimizers.natural_gradient module

class tensorforce.core.optimizers.natural_gradient.NaturalGradient(learning_rate, cg_max_iterations=20, cg_damping=0.001, cg_unroll_loop=False, summaries=None, summary_labels=None)

Bases: tensorforce.core.optimizers.optimizer.Optimizer

Natural gradient optimizer.

tf_step(time, variables, fn_loss, fn_kl_divergence, return_estimated_improvement=False, **kwargs)

Creates the TensorFlow operations for performing an optimization step.

Parameters:
  • time – Time tensor.
  • variables – List of variables to optimize.
  • fn_loss – A callable returning the loss of the current model.
  • fn_kl_divergence – A callable returning the KL-divergence relative to the current model.
  • return_estimated_improvement – Returns the estimated improvement resulting from the natural gradient calculation if true.
  • **kwargs –

    Additional arguments, not used.
Returns:

List of delta tensors corresponding to the updates for each optimized variable.

tensorforce.core.optimizers.optimized_step module

class tensorforce.core.optimizers.optimized_step.OptimizedStep(optimizer, ls_max_iterations=10, ls_accept_ratio=0.9, ls_mode='exponential', ls_parameter=0.5, ls_unroll_loop=False, summaries=None, summary_labels=None)

Bases: tensorforce.core.optimizers.meta_optimizer.MetaOptimizer

The optimized-step meta optimizer applies line search to the proposed optimization step of another optimizer to find a more optimal step size.

tf_step(time, variables, fn_loss, fn_reference=None, fn_compare=None, **kwargs)

Creates the TensorFlow operations for performing an optimization step.

Parameters:
  • time – Time tensor.
  • variables – List of variables to optimize.
  • fn_loss – A callable returning the loss of the current model.
  • fn_reference – A callable returning the reference values necessary for comparison.
  • fn_compare – A callable comparing the current model to the reference model given by its values.
  • **kwargs –

    Additional arguments passed on to the internal optimizer.
Returns:

List of delta tensors corresponding to the updates for each optimized variable.

tensorforce.core.optimizers.optimizer module

class tensorforce.core.optimizers.optimizer.Optimizer(summaries=None, summary_labels=None)

Bases: object

Generic TensorFlow optimizer which minimizes a not yet further specified expression, usually some kind of loss function. More generally, an optimizer can be considered as some method of updating a set of variables.

apply_step(variables, deltas)

Applies step deltas to variable values.

Parameters:
  • variables – List of variables.
  • deltas – List of deltas of same length.
Returns:

The step-applied operation.
static from_spec(spec, kwargs=None)

Creates an optimizer from a specification dict.

get_variables()

Returns the TensorFlow variables used by the optimizer.

Returns:List of variables.
minimize(time, variables, **kwargs)

Performs an optimization step.

Parameters:
  • time – Time tensor.
  • variables – List of variables to optimize.
  • **kwargs –

    Additional optimizer-specific arguments. The following arguments are used by some optimizers:

  • fn_loss (-) – A callable returning the loss of the current model.
  • fn_kl_divergence (-) – A callable returning the KL-divergence relative to the current model.
  • return_estimated_improvement (-) – Returns the estimated improvement resulting from the natural gradient calculation if true.
  • fn_reference (-) – A callable returning the reference values necessary for comparison.
  • fn_compare (-) – A callable comparing the current model to the reference model given by its values.
  • source_variables (-) – List of source variables to synchronize with.
  • global_variables (-) – List of global variables to apply the proposed optimization step to.
Returns:

The optimization operation.
tf_step(time, variables, **kwargs)

Creates the TensorFlow operations for performing an optimization step.

Parameters:
  • time – Time tensor.
  • variables – List of variables to optimize.
  • **kwargs –

    Additional arguments depending on the specific optimizer implementation. For instance, often includes fn_loss if a loss function is optimized.
Returns:

List of delta tensors corresponding to the updates for each optimized variable.

tensorforce.core.optimizers.synchronization module

class tensorforce.core.optimizers.synchronization.Synchronization(sync_frequency=1, update_weight=1.0)

Bases: tensorforce.core.optimizers.optimizer.Optimizer

The synchronization optimizer updates variables periodically to the value of a corresponding set of source variables.

get_variables()
tf_step(time, variables, source_variables, **kwargs)

Creates the TensorFlow operations for performing an optimization step.

Parameters:
  • time – Time tensor.
  • variables – List of variables to optimize.
  • source_variables – List of source variables to synchronize with.
  • **kwargs –

    Additional arguments, not used.
Returns:

List of delta tensors corresponding to the updates for each optimized variable.

tensorforce.core.optimizers.tf_optimizer module

class tensorforce.core.optimizers.tf_optimizer.TFOptimizer(optimizer, summaries=None, summary_labels=None, **kwargs)

Bases: tensorforce.core.optimizers.optimizer.Optimizer

Wrapper class for TensorFlow optimizers.

get_variables()
static get_wrapper(optimizer)

Returns a TFOptimizer constructor callable for the given optimizer name.

Parameters:
  • optimizer – The name of the optimizer, one of ‘adadelta’, ‘adagrad’, ‘adam’, ‘nadam’,
  • 'momentum', 'rmsprop'. ('gradient_descent',) –
Returns:

The TFOptimizer constructor callable.
tf_optimizers = {'nadam': <sphinx.ext.autodoc._MockObject object>, 'adam': <sphinx.ext.autodoc._MockObject object>, 'adadelta': <sphinx.ext.autodoc._MockObject object>, 'rmsprop': <sphinx.ext.autodoc._MockObject object>, 'adagrad': <sphinx.ext.autodoc._MockObject object>, 'momentum': <sphinx.ext.autodoc._MockObject object>, 'gradient_descent': <sphinx.ext.autodoc._MockObject object>}
tf_step(time, variables, fn_loss, **kwargs)

Creates the TensorFlow operations for performing an optimization step.

Parameters:
  • time – Time tensor.
  • variables – List of variables to optimize.
  • fn_loss – A callable returning the loss of the current model.
  • **kwargs –

    Additional arguments, not used.
Returns:

List of delta tensors corresponding to the updates for each optimized variable.

Module contents

class tensorforce.core.optimizers.Optimizer(summaries=None, summary_labels=None)

Bases: object

Generic TensorFlow optimizer which minimizes a not yet further specified expression, usually some kind of loss function. More generally, an optimizer can be considered as some method of updating a set of variables.

apply_step(variables, deltas)

Applies step deltas to variable values.

Parameters:
  • variables – List of variables.
  • deltas – List of deltas of same length.
Returns:

The step-applied operation.
static from_spec(spec, kwargs=None)

Creates an optimizer from a specification dict.

get_variables()

Returns the TensorFlow variables used by the optimizer.

Returns:List of variables.
minimize(time, variables, **kwargs)

Performs an optimization step.

Parameters:
  • time – Time tensor.
  • variables – List of variables to optimize.
  • **kwargs –

    Additional optimizer-specific arguments. The following arguments are used by some optimizers:

  • fn_loss (-) – A callable returning the loss of the current model.
  • fn_kl_divergence (-) – A callable returning the KL-divergence relative to the current model.
  • return_estimated_improvement (-) – Returns the estimated improvement resulting from the natural gradient calculation if true.
  • fn_reference (-) – A callable returning the reference values necessary for comparison.
  • fn_compare (-) – A callable comparing the current model to the reference model given by its values.
  • source_variables (-) – List of source variables to synchronize with.
  • global_variables (-) – List of global variables to apply the proposed optimization step to.
Returns:

The optimization operation.
tf_step(time, variables, **kwargs)

Creates the TensorFlow operations for performing an optimization step.

Parameters:
  • time – Time tensor.
  • variables – List of variables to optimize.
  • **kwargs –

    Additional arguments depending on the specific optimizer implementation. For instance, often includes fn_loss if a loss function is optimized.
Returns:

List of delta tensors corresponding to the updates for each optimized variable.
class tensorforce.core.optimizers.MetaOptimizer(optimizer, **kwargs)

Bases: tensorforce.core.optimizers.optimizer.Optimizer

A meta optimizer takes the optimization implemented by another optimizer and modifies/optimizes its proposed result. For example, line search might be applied to find a more optimal step size.

get_variables()
class tensorforce.core.optimizers.TFOptimizer(optimizer, summaries=None, summary_labels=None, **kwargs)

Bases: tensorforce.core.optimizers.optimizer.Optimizer

Wrapper class for TensorFlow optimizers.

get_variables()
static get_wrapper(optimizer)

Returns a TFOptimizer constructor callable for the given optimizer name.

Parameters:
  • optimizer – The name of the optimizer, one of ‘adadelta’, ‘adagrad’, ‘adam’, ‘nadam’,
  • 'momentum', 'rmsprop'. ('gradient_descent',) –
Returns:

The TFOptimizer constructor callable.
tf_optimizers = {'nadam': <sphinx.ext.autodoc._MockObject object>, 'adam': <sphinx.ext.autodoc._MockObject object>, 'adadelta': <sphinx.ext.autodoc._MockObject object>, 'rmsprop': <sphinx.ext.autodoc._MockObject object>, 'adagrad': <sphinx.ext.autodoc._MockObject object>, 'momentum': <sphinx.ext.autodoc._MockObject object>, 'gradient_descent': <sphinx.ext.autodoc._MockObject object>}
tf_step(time, variables, fn_loss, **kwargs)

Creates the TensorFlow operations for performing an optimization step.

Parameters:
  • time – Time tensor.
  • variables – List of variables to optimize.
  • fn_loss – A callable returning the loss of the current model.
  • **kwargs –

    Additional arguments, not used.
Returns:

List of delta tensors corresponding to the updates for each optimized variable.
class tensorforce.core.optimizers.Evolutionary(learning_rate, num_samples=1, summaries=None, summary_labels=None)

Bases: tensorforce.core.optimizers.optimizer.Optimizer

Evolutionary optimizer which samples random perturbations and applies them either positively or negatively, depending on their improvement of the loss.

tf_step(time, variables, fn_loss, **kwargs)

Creates the TensorFlow operations for performing an optimization step.

Parameters:
  • time – Time tensor.
  • variables – List of variables to optimize.
  • fn_loss – A callable returning the loss of the current model.
  • **kwargs –

    Additional arguments, not used.
Returns:

List of delta tensors corresponding to the updates for each optimized variable.
class tensorforce.core.optimizers.NaturalGradient(learning_rate, cg_max_iterations=20, cg_damping=0.001, cg_unroll_loop=False, summaries=None, summary_labels=None)

Bases: tensorforce.core.optimizers.optimizer.Optimizer

Natural gradient optimizer.

tf_step(time, variables, fn_loss, fn_kl_divergence, return_estimated_improvement=False, **kwargs)

Creates the TensorFlow operations for performing an optimization step.

Parameters:
  • time – Time tensor.
  • variables – List of variables to optimize.
  • fn_loss – A callable returning the loss of the current model.
  • fn_kl_divergence – A callable returning the KL-divergence relative to the current model.
  • return_estimated_improvement – Returns the estimated improvement resulting from the natural gradient calculation if true.
  • **kwargs –

    Additional arguments, not used.
Returns:

List of delta tensors corresponding to the updates for each optimized variable.
class tensorforce.core.optimizers.MultiStep(optimizer, num_steps=5, summaries=None, summary_labels=None)

Bases: tensorforce.core.optimizers.meta_optimizer.MetaOptimizer

The multi-step meta optimizer repeatedly applies the optimization step proposed by another optimizer a number of times.

tf_step(time, variables, **kwargs)

Creates the TensorFlow operations for performing an optimization step.

Parameters:
  • time – Time tensor.
  • variables – List of variables to optimize.
  • **kwargs –

    Additional arguments passed on to the internal optimizer.
Returns:

List of delta tensors corresponding to the updates for each optimized variable.
class tensorforce.core.optimizers.OptimizedStep(optimizer, ls_max_iterations=10, ls_accept_ratio=0.9, ls_mode='exponential', ls_parameter=0.5, ls_unroll_loop=False, summaries=None, summary_labels=None)

Bases: tensorforce.core.optimizers.meta_optimizer.MetaOptimizer

The optimized-step meta optimizer applies line search to the proposed optimization step of another optimizer to find a more optimal step size.

tf_step(time, variables, fn_loss, fn_reference=None, fn_compare=None, **kwargs)

Creates the TensorFlow operations for performing an optimization step.

Parameters:
  • time – Time tensor.
  • variables – List of variables to optimize.
  • fn_loss – A callable returning the loss of the current model.
  • fn_reference – A callable returning the reference values necessary for comparison.
  • fn_compare – A callable comparing the current model to the reference model given by its values.
  • **kwargs –

    Additional arguments passed on to the internal optimizer.
Returns:

List of delta tensors corresponding to the updates for each optimized variable.
class tensorforce.core.optimizers.Synchronization(sync_frequency=1, update_weight=1.0)

Bases: tensorforce.core.optimizers.optimizer.Optimizer

The synchronization optimizer updates variables periodically to the value of a corresponding set of source variables.

get_variables()
tf_step(time, variables, source_variables, **kwargs)

Creates the TensorFlow operations for performing an optimization step.

Parameters:
  • time – Time tensor.
  • variables – List of variables to optimize.
  • source_variables – List of source variables to synchronize with.
  • **kwargs –

    Additional arguments, not used.
Returns:

List of delta tensors corresponding to the updates for each optimized variable.
class tensorforce.core.optimizers.ClippedStep(optimizer, clipping_value, summaries=None, summary_labels=None)

Bases: tensorforce.core.optimizers.meta_optimizer.MetaOptimizer

The multi-shep meta optimizer repeatedly applies the optimization step proposed by another optimizer a number of times.

tf_step(time, variables, **kwargs)

Creates the TensorFlow operations for performing an optimization step.

Parameters:
  • time – Time tensor.
  • variables – List of variables to optimize.
  • **kwargs –

    Additional arguments passed on to the internal optimizer.
Returns:

List of delta tensors corresponding to the updates for each optimized variable.
class tensorforce.core.optimizers.GlobalOptimizer(optimizer, summaries=None, summary_labels=None)

Bases: tensorforce.core.optimizers.meta_optimizer.MetaOptimizer

The global optimizer applies an optimizer to the local variables. In addition, it also applies the update to a corresponding set of global variables and subsequently updates the local variables to the value of these global variables. Note: This is used for the current distributed mode, and will likely change with the next major version update.

tf_step(time, variables, global_variables, **kwargs)

Creates the TensorFlow operations for performing an optimization step.

Parameters:
  • time – Time tensor.
  • variables – List of variables to optimize.
  • global_variables – List of global variables to apply the proposed optimization step to.
  • **kwargs –

    ??? coming soon
Returns:

List of delta tensors corresponding to the updates for each optimized variable.