gluetool.glue module

class gluetool.glue.ArgumentParser(prog=None, usage=None, description=None, epilog=None, version=None, parents=[], formatter_class=<class 'argparse.HelpFormatter'>, prefix_chars='-', fromfile_prefix_chars=None, argument_default=None, conflict_handler='error', add_help=True)

Bases: argparse.ArgumentParser

Pretty much the argparse.ArgumentParser, it overrides just the argparse.ArgumentParser.error() method, to catch errors and to wrap them into nice and common GlueError instances.

The original prints (for us) useless message, including the program name, and raises SystemExit exception. Such action does not provide necessary information when encountered in Sentry, for example.

error(message)

Must not return - raising an exception is a good way to “not return”.

Raises:gluetool.glue.GlueError – When argument parser encounters an error.

class gluetool.glue.Configurable

Bases: object

Base class of two main gluetool classes - gluetool.glue.Glue and gluetool.glue.Module. Gives them the ability to use options, settable from configuration files and/or command-line arguments.

Variables:_config (dict) – internal configuration store. Values of all options are stored here, regardless of them being set on command-line or by the configuration file.

classmethod _create_args_parser(**kwargs)

Create an argument parser. Used by Sphinx to document “command-line” options of the module - which are, by the way, the module options as well.

Parameters:kwargs (dict) – Additional arguments passed to argparse.ArgumentParser.

_dryrun_allows(threshold, msg)

Check whether current dry-run level allows an action. If the current dry-run level is equal of higher than threshold, then the action is not allowed.

E.g. when action’s threshold is DryRunLevels.ISOLATED, and the current level is DryRunLevels.DRY, the action is allowed.

Parameters:

• threshold (DryRunLevels) – Dry-run level the action is not allowed.
• msg (str) – Message logged (as a warning) when the action is deemed not allowed.

Returns:

True when action is allowed, False otherwise.

static _for_each_option(callback, options)

Given dictionary defining options, call a callback for each of them.

Parameters:

• options (dict) – Dictionary of options, in a form option-name: option-params.
• callback (callable) – Must accept at least 3 parameters: option name (str), all option names (short and long ones) (tuple(str, str)), and option params (dict).

static _for_each_option_group(callback, options)

Given set of options, call a callback for each option group.

Parameters:

• options – List of option groups, or a dict listing options directly.
• callback (callable) – Must accept at least 2 parameters: options (dict), listing options in the group, and keyword parameter group_name (str), which is set to group name when the options defines an option group.

_parse_args(args, **kwargs)

Parse command-line arguments. Uses argparse for the actual parsing. Updates module’s configuration store with values returned by parser.

Parameters:args (list) – arguments passed to this module. Similar to what sys.argv provides on program level.

_parse_config(paths)

Parse configuration files. Uses ConfigParser for the actual parsing. Updates module’s configuration store with values found returned by the parser.

Parameters:paths (list) – List of paths to possible configuration files.

check_dryrun()

Checks whether this object supports current dry-run level.

check_required_options()

dryrun_allows(msg)

Checks whether current dry-run level allows an action which is disallowed on DryRunLevels.DRY level.

See Configurable._dryrun_allows() for detailed description.

dryrun_enabled

True if dry-run level is enabled, on any level.

dryrun_level

Return current dry-run level. This must be implemented by class descendants because each one finds the necessary information in different places.

eval_context

Return “evaluation context” - a dictionary of variable names (usually in uppercase) and their values, which is supposed to be used in various “evaluate this” operations like rendering of templates.

To provide nice and readable documentation of variables, returned by a module’s eval_context property, assign a dictionary, describing these variables, to a local variable named __content__:

...

@property
def eval_context(self):
    __content__ = {
        'FOO': 'This is an important variable, extracted from clouds.'
     }

     return {
         'FOO': 42
     }

gluetool core will extract this information and will use it to generate different help texts like your module’s help or a list of all known context variables.

Return type:dict

isolatedrun_allows(msg)

Checks whether current dry-run level allows an action which is disallowed on DryRunLevels.ISOLATED level.

option(*names)

Return values of given options from module’s configuration store.

Parameters:names (str) – names of requested options. Returns:either a value or None if such option does not exist. When multiple options are requested, a tuple of their values is returned, for a single option its value is not wrapped by a tuple.

options = {}

The options variable defines options accepted by module, and their properties:

options = {
    <option name>: {
        <option properties>
    },
    ...
}

where

• <option name> is used to name the option in the parser, and two formats are accepted (don’t add any leading dashes (- nor --):
  • <long name>
  • tuple(<short name>, <long name #1>, <long name #2>, ...)
• the first of long names (long name #1) is used to identify the option - other long names are understood by argument parser but their values are stored under long name #1 option.
• dictionary <option properties> is passed to argparse.ArgumentParser.add_argument() as keyword arguments when the option is being added to the parser, therefore any arguments recognized by argparse can be used.

It is also possible to use groups:

options = [
    (<group name>,  <group options>),
    ...
]

where

• <group name> is the name of the group, e.g. Debugging options
• <group options> is the dict with all group options, as described above.

This way, you can split pile of options into conceptualy closer groups of options. A single dict you would have is split into multiple smaller dictionaries, and each one is coupled with the group name in a tuple.

options_note = None

If set, it will be printed after all options as a help’s epilog.

parse_args(args)

Public entry point to argument parsing. Child classes must implement this method, e.g. by calling gluetool.glue.Configurable._parse_args() which makes use of additional argparse.ArgumentParser options.

parse_config()

Public entry point to configuration parsing. Child classes must implement this method, e.g. by calling gluetool.glue.Configurable._parse_config() which requires list of paths.

required_options = []

Iterable of names of required options.

supported_dryrun_level = 0

Highest supported level of dry-run.

unique_name = None

Unque name of this instance. Used by modules, has no meaning elsewhere, but since dry-run checks are done on this level, it must be declared here to make pylint happy :/

class gluetool.glue.DryRunLevels

Bases: enum.IntEnum

Dry-run levels.

Variables:

• DEFAULT (int) – Default level - everything is allowed.
• DRY (int) – Well-known “dry-run” - no changes to the outside world are allowed.
• ISOLATED (int) – No interaction with the outside world is allowed (networks connections, reading files, etc.)

DEFAULT = 0

DRY = 1

ISOLATED = 2

_member_map_ = OrderedDict([('DEFAULT', <DryRunLevels.DEFAULT: 0>), ('DRY', <DryRunLevels.DRY: 1>), ('ISOLATED', <DryRunLevels.ISOLATED: 2>)])

_member_names_ = ['DEFAULT', 'DRY', 'ISOLATED']

_member_type_

alias of int

_value2member_map_ = {0: <DryRunLevels.DEFAULT: 0>, 1: <DryRunLevels.DRY: 1>, 2: <DryRunLevels.ISOLATED: 2>}

class gluetool.glue.Failure(module, exc_info)

Bases: object

Bundles exception related info. Used to inform modules in their destroy() phase that gluetool session was killed because of exception raised by one of modules.

Parameters:

• module (gluetool.glue.Module) – module in which the error happened, or None.
• exc_info (tuple) – Exception information as returned by sys.exc_info().

Variables:

• module (gluetool.glue.Module) – module in which the error happened, or None.
• exception (Exception) – Shortcut to exc_info[1], if available, or None.
• exc_info (tuple) – Exception information as returned by sys.exc_info().
• sentry_event_id (str) – If set, the failure was reported to the Sentry under this ID.

class gluetool.glue.Glue(tool=None, sentry=None)

Bases: gluetool.glue.Configurable

Main workhorse of the gluetool. Manages modules, their instances and runs them as requested.

Parameters:

• tool (gluetool.tool.Tool) – If set, it’s an gluetool-like tool that created this instance. Some functionality may need it to gain access to bits like its command-name.
• sentry (gluetool.sentry.Sentry) – If set, it provides interface to Sentry.

_add_shared(funcname, module, func)

Register a shared function. Overwrite previously registered function with the same name, if there was any such.

This is a helper method for easier testability. It is not a part of public API of this class.

Parameters:

• funcname (str) – Name of the shared function.
• module (gluetool.glue.Module) – Module instance providing the shared function.
• func (callable) – Shared function.

_check_module_file(mfile)

Make sure the file looks like a gluetool module:

• can be processed by Python parser,
• imports gluetool.glue.Glue and gluetool.glue.Module,
• contains child class of gluetool.glue.Module.

Parameters:mfile (str) – path to a file. Returns:True if file contains gluetool module, False otherwise. Raises:gluetool.glue.GlueError – when it’s not possible to finish the check.

_eval_context()

Gather contexts of all modules in a pipeline and merge them together.

Always returns a unique dictionary object, therefore it is safe for caller to update it. The return value is not cached in any way, therefore the change if its content won’t affect future callers.

Provided as a shared function, registered by the Glue instance itself.

Return type:dict

_eval_context_module_caller()

Infere module instance calling the eval context shared function.

Return type:gluetool.glue.Module

_for_each_module(modules, callback, *args, **kwargs)

_import_module(import_name, filename)

Attempt to import a Python module from a file.

Parameters:

• import_name (str) – name assigned to the imported module.
• filepath (str) – path to a file.

Returns:

imported Python module.

Raises:

gluetool.glue.GlueError – when import failed.

_load_gluetool_modules(group, module_name, filepath)

Load gluetool modules from a file. Method attempts to import the file as a Python module, and then checks its content and adds all gluetool modules to internal module registry.

Parameters:

• group (str) – module group.
• module_name (str) – name assigned to the imported Python module.
• filepath (str) – path to a file.

Return type:

[(module_group, module_class), ..]

Returns:

list of loaded gluetool modules

_load_module_path(ppath)

Search and load gluetool modules from a directory.

In essence, it scans every file with .py suffix, and searches for classes derived from gluetool.glue.Module.

Parameters:ppath (str) – directory to search for gluetool modules.

_load_python_module(group, module_name, filepath)

Load Python module from a file, if it contains gluetool modules. If the file does not look like it contains gluetool modules, or when it’s not possible to import the Python module successfully, method simply warns user and ignores the file.

Parameters:

• import_name (str) – name assigned to the imported module.
• filepath (str) – path to a file.

Returns:

loaded Python module.

Raises:

gluetool.glue.GlueError – when import failed.

add_shared(funcname, module)

Register a shared function. Overwrite previously registered function with the same name, if there was any such.

Parameters:

• funcname (str) – Name of the shared function.
• module (gluetool.glue.Module) – Module instance providing the shared function.

del_shared(funcname)

destroy_modules(failure=None)

dryrun_level

eval_context

Returns “global” evaluation context - some variables that are nice to have in all contexts.

get_shared(funcname)

has_shared(funcname)

init_module(module_name, actual_module_name=None)

Given a name of the module, create its instance and give it a name.

Parameters:

• module_name (str) – Name under which will be the module instance known.
• actual_module_name (str) – Name of the module to instantiate. It does not have to match module_name - actual_module_name refers to the list of known gluetool modules while module_name is basically an arbitrary name new instance calls itself. If it’s not set, which is the most common situation, it defaults to module_name.

Returns:

A Module instance.

load_modules()

Load all available gluetool modules.

module_config_paths

List of paths in which module config files reside.

module_data_paths

List of paths in which module data files reside.

module_group_list()

Returns a dictionary of groups of modules with description

module_list()

module_list_usage(groups)

Returns a string with modules description

module_paths

List of paths in which modules reside.

name = 'gluetool core'

options = [('Global options', {('V', 'version'): {'action': 'store_true', 'help': 'Print version'}, ('E', 'list-eval-context'): {'action': 'store_true', 'default': False, 'help': 'List all available variables provided by modules in their evaluation contexts.'}, ('L', 'list-shared'): {'action': 'store_true', 'default': False, 'help': 'List all available shared functions.'}, ('l', 'list-modules'): {'action': 'append', 'const': True, 'nargs': '?', 'help': 'List all available modules. If a GROUP is set, limits list to the given module group.', 'metavar': 'GROUP'}, 'no-sentry-exceptions': {'action': 'append', 'default': [], 'help': 'List of exception names, which are not reported to Sentry (Default: none)'}, ('r', 'retries'): {'default': 0, 'type': <type 'int'>, 'help': 'Number of retries'}}), ('Output control', {('j', 'json-file'): {'default': None, 'help': '\n If set, all log messages (including ``VERBOSE``) are stored in this file\n in a form of JSON structures (default: %(default)s).\n '}, 'verbose-file': {'default': None, 'help': 'Log messages with ``VERBOSE`` level sent to this file.'}, ('v', 'verbose'): {'action': 'store_true', 'help': 'Log **all** messages to the terminal (WARNING: even more verbose than ``-d``!).'}, ('p', 'pid'): {'action': 'store_true', 'help': 'Log PID of gluetool process'}, ('c', 'colors'): {'action': 'store_true', 'help': 'Colorize logging on the terminal'}, 'show-traceback': {'action': 'store_true', 'default': False, 'help': '\n Display exception tracebacks on terminal (besides the debug file when ``--debug-file``\n is used) (default: %(default)s).\n '}, ('q', 'quiet'): {'action': 'store_true', 'help': 'Silence info messages'}, ('i', 'info'): {'action': 'store_true', 'help': 'Print command-line that would re-run the gluetool session'}, ('o', 'debug-file', 'output'): {'help': 'Log messages with at least ``DEBUG`` level are sent to this file.'}, ('d', 'debug'): {'action': 'store_true', 'help': 'Log debugging messages to the terminal (WARNING: very verbose!).'}}), ('Directories', {'module-path': {'action': 'append', 'default': [], 'help': 'Specify directory with modules.', 'metavar': 'DIR'}, 'module-config-path': {'action': 'append', 'default': [], 'help': 'Specify directory with module configuration files.', 'metavar': 'DIR'}, 'module-data-path': {'action': 'append', 'default': [], 'help': 'Specify directory with module data files.', 'metavar': 'DIR'}}), ('Dry run options', {'isolated-run': {'action': 'store_true', 'default': False, 'help': 'Modules that support this option will not interact with the outside world.'}, 'dry-run': {'action': 'store_true', 'default': False, 'help': 'Modules that support this option will make no changes to the outside world.'}}), {'pipeline': {'raw': True, 'nargs': '...', 'help': 'List of modules and their options, passed after gluetool options.'}}]

parse_args(args)

parse_config(paths)

require_shared(*names, **kwargs)

run_module(module_name, module_argv=None, actual_module_name=None, register=False)

Syntax sugar for run_modules(), in the case you want to run just a one-shot module.

Parameters:

• module_name (str) – Name under which will be the module instance known.
• module_argv (list(str)) – Arguments of the module.
• actual_module_name (str) – Name of the module to instantiate. It does not have to match module_name - actual_module_name refers to the list of known gluetool modules while module_name is basically an arbitrary name new instance calls itself. If it’s not set, which is the most common situation, it defaults to module_name.
• register (bool) – If True, module instance is added to a list of modules in this Glue instance, and it will be collected when destroy_modules() gets called.

run_modules(pipeline_desc, register=False)

Run a pipeline, consisting of multiple modules.

Parameters:

• pipeline_desc (list(PipelineStep)) – List of pipeline steps.
• register (bool) – If True, module instance is added to a list of modules in this Glue instance, and it will be collected when destroy_modules() gets called.

sentry_submit_exception(*args, **kwargs)

Submits exceptions to the Sentry server. Does nothing by default, unless this instance is initialized with a gluetool.sentry.Sentry instance which actually does the job.

See gluetool.sentry.Sentry.submit_exception().

sentry_submit_warning(*args, **kwargs)

Submits warnings to the Sentry server. Does nothing by default, unless this instance is initialized with a gluetool.sentry.Sentry instance which actually does the job.

See gluetool.sentry.Sentry.submit_warning().

shared(funcname, *args, **kwargs)

shared_functions = None

Shared function registry. funcname: (module, fn)

exception gluetool.glue.GlueCommandError(cmd, output, **kwargs)

Bases: gluetool.glue.GlueError

Exception raised when external command failes.

Parameters:

• cmd (list) – Command as passed to gluetool.utils.run_command helper.
• output (gluetool.utils.ProcessOutput) – Process output data.

Variables:

• cmd (list) – Command as passed to gluetool.utils.run_command helper.
• output (gluetool.utils.ProcessOutput) – Process output data.

exception gluetool.glue.GlueError(message, caused_by=None, **kwargs)

Bases: exceptions.Exception

Generic gluetool exception.

Parameters:

• message (str) – Exception message, describing what happened.
• caused_by (tuple) – If set, contains tuple as returned by sys.exc_info(), describing the exception that caused this one to be born. If not set, constructor will try to auto-detect this information, and if there’s no such information, instance property caused_by will be set to None.

Variables:

• message (str) – Exception message, describing what happened.
• caused_by (tuple) – If set, contains tuple as returned by sys.exc_info(), describing the exception that caused this one to be born. None otherwise.

no_sentry_exceptions = []

sentry_fingerprint(current)

Default grouping of events into issues might be too general for some cases. This method gives users a chance to provide custom fingerprint Sentry could use to group events in a more suitable way.

E.g. user might be interested in some sort of connection issues but they would like to have them grouped not by a traceback (which is the default method) but per remote host IP. For that, the Sentry integration code will call sentry_fingerprint method of a raised exception, and the method should return new fingerprint, let’s say [<exception class name>, <remote IP>], and Sentry will group events using this fingerprint.

Parameters:current (list(str)) – current fingerprint. Usually ['{{ default }}'] telling Sentry to use its default method, but it could already be more specific. Return type:list(str) Returns:new fingerprint, e.g. ['FailedToConnectToAPI', '10.20.30.40']

sentry_tags(current)

Add, modify or remove tags attached to a Sentry event, reported when the exception was raised.

Most common usage would be an addition of tags, e.g. remote-host to allow search for events related to the same remote address.

Parameters:str) current (dict(str,) – current set of tags and their values. Return type:dict(str, str) Returns:new set of tags. It is possible to add tags directly into current and then return it.

submit_to_sentry

Decide whether the exception should be submitted to Sentry or not. By default, all exceptions are submitted. Exception listed in no_sentry_exceptions are not submitted.

Return type:bool Returns:True when the exception should be submitted to Sentry, False otherwise.

exception gluetool.glue.GlueRetryError(message, caused_by=None, **kwargs)

Bases: gluetool.glue.GlueError

Retry gluetool exception

class gluetool.glue.Module(glue, name)

Bases: gluetool.glue.Configurable

Base class of all gluetool modules.

Parameters:

glue (gluetool.glue.Glue) – Glue instance owning the module.

Variables:

• glue (gluetool.glue.Glue) – Glue instance owning the module.
• _config (dict) – internal configuration store. Values of all module options are stored here, regardless of them being set on command-line or in the configuration file.
• _overloaded_shared_functions (dict) – If a shared function added by this module overloades an older function of the same name, registered by a previous module, the overloaded one is added into this dictionary. The module can then call this saved function - using overloaded_shared() - to implement a “chain” of shared functions, when one calls another, implementing the same operation.

_generate_shared_functions_help()

Generate help for shared functions provided by the module.

Returns:Formatted help, describing module’s shared functions.

_paths_with_module(roots)

Return paths cretaed by joining roots with module’s unique name.

Parameters:roots (list(str)) – List of root directories.

add_shared()

Register module’s shared functions with Glue, to allow other modules to use them.

del_shared(funcname)

description = None

Short module description, displayed in gluetool‘s module listing.

destroy(failure=None)

Here should go any code that needs to be run on exit, like job cleanup etc.

Parameters:failure (gluetool.glue.Failure) – if set, carries information about failure that made gluetool to destroy the whole session. Modules might want to take actions based on provided information, e.g. send different notifications.

dryrun_level

execute()

In this method, modules can perform any work they deemed necessary for completing their purpose. E.g. if the module promises to run some tests, this is the place where the code belongs to.

By default, this method does nothing. Reimplement as needed.

get_shared(funcname)

has_shared(funcname)

name = None

Module name. Usually matches the name of the source file, no suffix.

overloaded_shared(funcname, *args, **kwargs)

Call a shared function overloaded by the one provided by this module. This way, a module can give chance to other implementations of its action, e.g. to publish messages on a different message bus.

parse_args(args)

parse_config()

require_shared(*names, **kwargs)

run_module(module, args=None)

sanity()

In this method, modules can define additional checks before execution.

Some examples:

• Advanced checks on passed options
• Check for additional requirements (tools, data, etc.)

By default, this method does nothing. Reimplement as needed.

shared(*args, **kwargs)

shared_functions = []

Iterable of names of shared functions exported by the module.

class gluetool.glue.PipelineStep(module, actual_module=None, argv=None)

Bases: object

Step of gluetool‘s pipeline - which is basically just a list of steps.

Parameters:

• module (str) – name to give to the module instance. This name is used e.g. in logging or when searching for module’s config file.
• actual_module (str) – The actual module class the step uses. Usually it is same as module but may differ, module is then a mere “alias”. actual_module is used to locate a module class, whose instance is then given name module.
• argv (list(str)) – list of options to be given to the module, in a form similar to sys.argv.

module_designation

exception gluetool.glue.SoftGlueError(message, caused_by=None, **kwargs)

Bases: gluetool.glue.GlueError

Soft errors are errors Glue Ops and/or developers shouldn’t be bothered with, things that are up to the user to fix, e.g. empty set of tests. Hard errors are supposed to warn Ops/Devel teams about important infrastructure issues, code deficiencies, bugs and other issues that are fixable only by actions of Glue staff.

However, we still must provide notification to user(s), and since we expect them to fix the issues that led to raising the soft error, we must provide them with as much information as possible. Therefore modules dealing with notifications are expected to give these exceptions a chance to influence the outgoing messages, e.g. by letting them provide an e-mail body template.

gluetool.glue.retry(*args)

Retry decorator This decorator catches given exceptions and returns libRetryError exception instead.

usage: @retry(exception1, exception2, ..)