Architecture overview

In this section, we will take a brief look at the internal organization of the Machin library, to better understand the functionality of every module.

Env

Currently machin.env has two sub modules: machin.env.utils and machin.env.wrappers.

The submodule machin.env.utils of the environment module provides some convenient utility functions you might will need in your own application, such as disabling the rendering window while keeping the rendered result in OpenAI gym.

The submodule machin.env.wrappers provides process-level parallel environment wrappers for different environments.

Framework

machin.frame is the most important core part of the Machin library, the framework module constitutes of:

  1. machin.frame.algorithms : RL algorithm implementations.

  2. machin.frame.buffers : Replay buffer implementations.

  3. machin.frame.helpers : Utility to help you initialize a framework.

  4. machin.frame.noise : Action space & parameter space noise generators.

Model

machin.model is a collection of popular network models you might will use in your own program, for example, ResNet.

Model module also contains the basis of all network modules: NeuralNetworkModule, this wrapper is built upon regular torch.nn.Module, and allows users to specify input/output sub module.

Parallel

machin.parallel is the second core part of the Machin library, the parallel module is a collection of refined implementations including:

  1. machin.parallel.thread : Thread (With exception catching).

  2. machin.parallel.process : Process (With remote exception catching).

  3. machin.parallel.queues : Queues. (Used in pools).

  4. machin.parallel.pool : Process pools (allow local functions,
    customize serialization policy), thread pools, pools with contexts, etc.

  5. machin.parallel.assigner : Heuristic based model-device assignment.

  6. machin.parallel.server : Implementations of different servers used in distributed
    algorithms such as A3C, DQNApex, DDPGApex and IMPALA.
  7. machin.parallel.distributed : A naive implementation of a part of
    RFC #41546

Utils

machin.utils is a messy hotchpotch of various tools, it is very hard to categorize them, but they could be helpful sometimes, so we left them here:

  1. machin.utils.checker : A checker implementation, using forward & backward hooks
    provided by pytorch to check the input/ouput, input gradient of models. Supports user
    defined checkers and tensorboard.
  2. machin.utils.conf : Functions designed to load/save a json configuration file, as
    well as loading parametrs from commandline.
  3. machin.utils.helper_classes : Various helper classes, such as Timer, Counter, etc.
  4. machin.utils.learning_rate : Functions used in learning rate schedulers. Useful
    if you would like to have finer control over the learning rate.

  5. machin.utils.loading : Logging utility module.

  6. machin.utils.media : Media writing utility, mainly images and videos, useful if you would
    like to log rendered environments.
  7. machin.utils.prepare : Functions used to create directories, loading models (take care of
    devices automatically), for preparing a training session.
  8. machin.utils.save_env : A standard reinforcement training environment creator, will create
    unique directories by time for you.
  9. machin.utils.visualize : Visualize your model, currently only contains some simple functions
    for gradient flow checking.

  10. machin.utils.tensorboard: A simple tensorboard wrapper.