tensorforce.core.optimizers.solvers package

Submodules

tensorforce.core.optimizers.solvers.conjugate_gradient module

class tensorforce.core.optimizers.solvers.conjugate_gradient.ConjugateGradient(max_iterations, damping, unroll_loop=False)

Bases: tensorforce.core.optimizers.solvers.iterative.Iterative

Conjugate gradient algorithm which iteratively finds a solution $x$ for a system of linear equations of the form $A x = b$, where $A x$ could be, for instance, a locally linear approximation of a high-dimensional function.

See below pseudo-code taken from Wikipedia:

def conjgrad(A, b, x_0):
    r_0 := b - A * x_0
    c_0 := r_0
    r_0^2 := r^T * r

    for t in 0, ..., max_iterations - 1:
        Ac := A * c_t
        cAc := c_t^T * Ac
        lpha := r_t^2 / cAc
        x_{t+1} := x_t + lpha * c_t
        r_{t+1} := r_t - lpha * Ac
        r_{t+1}^2 := r_{t+1}^T * r_{t+1}
        if r_{t+1} < \epsilon:
            break
        eta = r_{t+1}^2 / r_t^2
        c_{t+1} := r_{t+1} + eta * c_t

    return x_{t+1}
tf_initialize(x_init, b)

Initialization step preparing the arguments for the first iteration of the loop body: $x_0, 0, p_0, r_0, r_0^2$.

Parameters:
  • x_init – Initial solution guess $x_0$, zero vector if None.
  • b – The right-hand side $b$ of the system of linear equations.
Returns:

Initial arguments for tf_step.
tf_next_step(x, iteration, conjugate, residual, squared_residual)

Termination condition: max number of iterations, or residual sufficiently small.

Parameters:
  • x – Current solution estimate $x_t$.
  • iteration – Current iteration counter $t$.
  • conjugate – Current conjugate $c_t$.
  • residual – Current residual $r_t$.
  • squared_residual – Current squared residual $r_t^2$.
Returns:

True if another iteration should be performed.
tf_solve(fn_x, x_init, b)

Iteratively solves the system of linear equations $A x = b$.

Parameters:
  • fn_x – A callable returning the left-hand side $A x$ of the system of linear equations.
  • x_init – Initial solution guess $x_0$, zero vector if None.
  • b – The right-hand side $b$ of the system of linear equations.
Returns:

A solution $x$ to the problem as given by the solver.
tf_step(x, iteration, conjugate, residual, squared_residual)

Iteration loop body of the conjugate gradient algorithm.

Parameters:
  • x – Current solution estimate $x_t$.
  • iteration – Current iteration counter $t$.
  • conjugate – Current conjugate $c_t$.
  • residual – Current residual $r_t$.
  • squared_residual – Current squared residual $r_t^2$.
Returns:

Updated arguments for next iteration.

tensorforce.core.optimizers.solvers.iterative module

class tensorforce.core.optimizers.solvers.iterative.Iterative(max_iterations, unroll_loop=False)

Bases: tensorforce.core.optimizers.solvers.solver.Solver

Generic solver which iteratively solves an equation/optimization problem. Involves an initialization step, the iteration loop body and the termination condition.

tf_initialize(x_init, *args)

Initialization step preparing the arguments for the first iteration of the loop body (default: initial solution guess and iteration counter).

Parameters:
  • x_init – Initial solution guess $x_0$.
  • *args –

    Additional solver-specific arguments.
Returns:

Initial arguments for tf_step.
tf_next_step(x, iteration, *args)

Termination condition (default: max number of iterations).

Parameters:
  • x – Current solution estimate.
  • iteration – Current iteration counter.
  • *args –

    Additional solver-specific arguments.
Returns:

True if another iteration should be performed.
tf_solve(fn_x, x_init, *args)

Iteratively solves an equation/optimization for $x$ involving an expression $f(x)$.

Parameters:
  • fn_x – A callable returning an expression $f(x)$ given $x$.
  • x_init – Initial solution guess $x_0$.
  • *args –

    Additional solver-specific arguments.
Returns:

A solution $x$ to the problem as given by the solver.
tf_step(x, iteration, *args)

Iteration loop body of the iterative solver (default: increment iteration step). The first two loop arguments have to be the current solution estimate and the iteration step.

Parameters:
  • x – Current solution estimate.
  • iteration – Current iteration counter.
  • *args –

    Additional solver-specific arguments.
Returns:

Updated arguments for next iteration.

tensorforce.core.optimizers.solvers.solver module

class tensorforce.core.optimizers.solvers.solver.Solver

Bases: object

Generic TensorFlow-based solver which solves a not yet further specified equation/optimization problem.

static from_config(config, kwargs=None)

Creates a solver from a specification dict.

tf_solve(fn_x, *args)

Solves an equation/optimization for $x$ involving an expression $f(x)$.

Parameters:
  • fn_x – A callable returning an expression $f(x)$ given $x$.
  • *args –

    Additional solver-specific arguments.
Returns:

A solution $x$ to the problem as given by the solver.

Module contents

class tensorforce.core.optimizers.solvers.Solver

Bases: object

Generic TensorFlow-based solver which solves a not yet further specified equation/optimization problem.

static from_config(config, kwargs=None)

Creates a solver from a specification dict.

tf_solve(fn_x, *args)

Solves an equation/optimization for $x$ involving an expression $f(x)$.

Parameters:
  • fn_x – A callable returning an expression $f(x)$ given $x$.
  • *args –

    Additional solver-specific arguments.
Returns:

A solution $x$ to the problem as given by the solver.
class tensorforce.core.optimizers.solvers.Iterative(max_iterations, unroll_loop=False)

Bases: tensorforce.core.optimizers.solvers.solver.Solver

Generic solver which iteratively solves an equation/optimization problem. Involves an initialization step, the iteration loop body and the termination condition.

tf_initialize(x_init, *args)

Initialization step preparing the arguments for the first iteration of the loop body (default: initial solution guess and iteration counter).

Parameters:
  • x_init – Initial solution guess $x_0$.
  • *args –

    Additional solver-specific arguments.
Returns:

Initial arguments for tf_step.
tf_next_step(x, iteration, *args)

Termination condition (default: max number of iterations).

Parameters:
  • x – Current solution estimate.
  • iteration – Current iteration counter.
  • *args –

    Additional solver-specific arguments.
Returns:

True if another iteration should be performed.
tf_solve(fn_x, x_init, *args)

Iteratively solves an equation/optimization for $x$ involving an expression $f(x)$.

Parameters:
  • fn_x – A callable returning an expression $f(x)$ given $x$.
  • x_init – Initial solution guess $x_0$.
  • *args –

    Additional solver-specific arguments.
Returns:

A solution $x$ to the problem as given by the solver.
tf_step(x, iteration, *args)

Iteration loop body of the iterative solver (default: increment iteration step). The first two loop arguments have to be the current solution estimate and the iteration step.

Parameters:
  • x – Current solution estimate.
  • iteration – Current iteration counter.
  • *args –

    Additional solver-specific arguments.
Returns:

Updated arguments for next iteration.
class tensorforce.core.optimizers.solvers.ConjugateGradient(max_iterations, damping, unroll_loop=False)

Bases: tensorforce.core.optimizers.solvers.iterative.Iterative

Conjugate gradient algorithm which iteratively finds a solution $x$ for a system of linear equations of the form $A x = b$, where $A x$ could be, for instance, a locally linear approximation of a high-dimensional function.

See below pseudo-code taken from Wikipedia:

def conjgrad(A, b, x_0):
    r_0 := b - A * x_0
    c_0 := r_0
    r_0^2 := r^T * r

    for t in 0, ..., max_iterations - 1:
        Ac := A * c_t
        cAc := c_t^T * Ac
        lpha := r_t^2 / cAc
        x_{t+1} := x_t + lpha * c_t
        r_{t+1} := r_t - lpha * Ac
        r_{t+1}^2 := r_{t+1}^T * r_{t+1}
        if r_{t+1} < \epsilon:
            break
        eta = r_{t+1}^2 / r_t^2
        c_{t+1} := r_{t+1} + eta * c_t

    return x_{t+1}
tf_initialize(x_init, b)

Initialization step preparing the arguments for the first iteration of the loop body: $x_0, 0, p_0, r_0, r_0^2$.

Parameters:
  • x_init – Initial solution guess $x_0$, zero vector if None.
  • b – The right-hand side $b$ of the system of linear equations.
Returns:

Initial arguments for tf_step.
tf_next_step(x, iteration, conjugate, residual, squared_residual)

Termination condition: max number of iterations, or residual sufficiently small.

Parameters:
  • x – Current solution estimate $x_t$.
  • iteration – Current iteration counter $t$.
  • conjugate – Current conjugate $c_t$.
  • residual – Current residual $r_t$.
  • squared_residual – Current squared residual $r_t^2$.
Returns:

True if another iteration should be performed.
tf_solve(fn_x, x_init, b)

Iteratively solves the system of linear equations $A x = b$.

Parameters:
  • fn_x – A callable returning the left-hand side $A x$ of the system of linear equations.
  • x_init – Initial solution guess $x_0$, zero vector if None.
  • b – The right-hand side $b$ of the system of linear equations.
Returns:

A solution $x$ to the problem as given by the solver.
tf_step(x, iteration, conjugate, residual, squared_residual)

Iteration loop body of the conjugate gradient algorithm.

Parameters:
  • x – Current solution estimate $x_t$.
  • iteration – Current iteration counter $t$.
  • conjugate – Current conjugate $c_t$.
  • residual – Current residual $r_t$.
  • squared_residual – Current squared residual $r_t^2$.
Returns:

Updated arguments for next iteration.
class tensorforce.core.optimizers.solvers.LineSearch(max_iterations, accept_ratio, mode, parameter, unroll_loop=False)

Bases: tensorforce.core.optimizers.solvers.iterative.Iterative

Line search algorithm which iteratively optimizes the value $f(x)$ for $x$ on the line between $x’$ and $x_0$ by optimistically taking the first acceptable $x$ starting from $x_0$ and moving towards $x’$.

tf_initialize(x_init, base_value, target_value, estimated_improvement)

Initialization step preparing the arguments for the first iteration of the loop body.

Parameters:
  • x_init – Initial solution guess $x_0$.
  • base_value – Value $f(x’)$ at $x = x’$.
  • target_value – Value $f(x_0)$ at $x = x_0$.
  • estimated_improvement – Estimated value at $x = x_0$, $f(x’)$ if None.
Returns:

Initial arguments for tf_step.
tf_next_step(x, iteration, deltas, improvement, last_improvement, estimated_improvement)

Termination condition: max number of iterations, or no improvement for last step, or improvement less than acceptable ratio, or estimated value not positive.

Parameters:
  • x – Current solution estimate $x_t$.
  • iteration – Current iteration counter $t$.
  • deltas – Current difference $x_t - x’$.
  • improvement – Current improvement $(f(x_t) - f(x’)) / v’$.
  • last*improvement –

    Last improvement $(f(x*{t-1}) - f(x’)) / v’$.

  • estimated_improvement – Current estimated value $v’$.
Returns:

True if another iteration should be performed.
tf_solve(fn_x, x_init, base_value, target_value, estimated_improvement=None)

Iteratively optimizes $f(x)$ for $x$ on the line between $x’$ and $x_0$.

Parameters:
  • fn_x – A callable returning the value $f(x)$ at $x$.
  • x_init – Initial solution guess $x_0$.
  • base_value – Value $f(x’)$ at $x = x’$.
  • target_value – Value $f(x_0)$ at $x = x_0$.
  • estimated_improvement – Estimated improvement for $x = x_0$, $f(x’)$ if None.
Returns:

A solution $x$ to the problem as given by the solver.
tf_step(x, iteration, deltas, improvement, last_improvement, estimated_improvement)

Iteration loop body of the line search algorithm.

Parameters:
  • x – Current solution estimate $x_t$.
  • iteration – Current iteration counter $t$.
  • deltas – Current difference $x_t - x’$.
  • improvement – Current improvement $(f(x_t) - f(x’)) / v’$.
  • last*improvement –

    Last improvement $(f(x*{t-1}) - f(x’)) / v’$.

  • estimated_improvement – Current estimated value $v’$.
Returns:

Updated arguments for next iteration.