TensorForce: Details for "summary_spec" agent parameters

Docs Gitter Build Status License

summary_spec

TensorForce has the ability to record summary data for use with TensorBoard as well STDIO and file export. This is accomplished through dictionary parameter called "summary_spec" passed to the agent on initialization.

"summary_spec" supports the following optional dictionary entries:

Key Value
directory (str) Path to storage for TensorBoard summary data
steps (int) Frequency in steps between storage of summary data
seconds (int) Frequency in seconds to store summary data
labels (list) Requested Export, See "LABELS" section
meta_dict (dict) For used with label "configuration"

LABELS

Entry Data produced
losses Training total-loss and "loss-without-regularization"
total-loss Final calculated loss value
variables Network variables
inputs Equivalent to: ['states', 'actions', 'rewards']
states Histogram of input state space
actions Histogram of input action space
rewards Histogram of input reward space
gradients Histogram and scalar gradients
gradients_histogram Variable gradients as histograms
gradients_scalar Variable Mean/Variance of gradients as scalar
regularization Regularization values
configuration See Configuration Export for more detail
configuration Export configuration to "TEXT" tab in TensorBoard
print_configuration Prints configuration to STDOUT
from tensorforce.agents import PPOAgent

# Create a Proximal Policy Optimization agent
agent = PPOAgent(
    states_spec=...,
    actions_spec=...,
    network_spec=...,
    summary_spec=dict(directory="./board/", 
                        steps=50,
                        labels=['configuration',
                            'gradients_scalar',
                            'regularization',
                            'inputs',
                            'losses',
                            'variables']
                    ),      
    ...
)

Configuration Export

Adding the "configuration" label will create a "TEXT" tab in TensorBoard that contains all the parameters passed to the Agent. By using the additional "summary_spec" dictionary key "meta_dict", custom keys and values can be added to the data export. The user may want to pass "Description", "Experiement #", "InputDataSet", etc.

If a key is already in use within TensorForce an error will be raised to notify you to change the key value. To use the custom feature, create a dictionary with keys to export:

from tensorforce.agents import PPOAgent

metaparams['MyDescription'] = "This experiment covers the first test ...."
metaparams['My2D'] = np.ones((9,9))   # 9x9 matrix of  1.0's
metaparams['My1D'] = np.ones((9))   # Column of 9   1.0's

# Create a Proximal Policy Optimization agent
agent = PPOAgent(
    states_spec=...,
    actions_spec=...,
    network_spec=...,
    summary_spec=dict(directory="./board/", 
                        steps=50,
                        meta_dict=metaparams,  #Add custom keys to export
                        labels=['configuration',
                            'gradients_scalar',
                            'regularization',
                            'inputs',
                            'losses',
                            'variables']
                    ),      
    ...
)

Use the "print_configuration" label to export the configuration data to the command line's STDOUT.