gluetool package

Subpackages

• gluetool.pylint package
  • Submodules
    • gluetool.pylint.option_default module
    • gluetool.pylint.shared_defined module
    • gluetool.pylint.unknown_option module
  • Module contents
• gluetool.tests package
  • Submodules
    • gluetool.tests.conftest module
    • gluetool.tests.test_core module
    • gluetool.tests.test_error module
    • gluetool.tests.test_eval_context module
    • gluetool.tests.test_help module
    • gluetool.tests.test_json module
    • gluetool.tests.test_load_yaml module
    • gluetool.tests.test_log_exception module
    • gluetool.tests.test_logging module
    • gluetool.tests.test_module_discovery module
    • gluetool.tests.test_new_xml_element module
    • gluetool.tests.test_normalize_option module
    • gluetool.tests.test_option module
    • gluetool.tests.test_pipeline_step module
    • gluetool.tests.test_render_template module
    • gluetool.tests.test_requests module
    • gluetool.tests.test_result module
    • gluetool.tests.test_run_command module
    • gluetool.tests.test_run_modules module
    • gluetool.tests.test_treat_url module
    • gluetool.tests.test_utils module
    • gluetool.tests.test_wait module
  • Module contents

Submodules

• gluetool.action module
• gluetool.color module
• gluetool.glue module
• gluetool.help module
• gluetool.html_log module
• gluetool.log module
• gluetool.proxy module
• gluetool.result module
• gluetool.sentry module
• gluetool.tool module
• gluetool.utils module
• gluetool.version module

Module contents

class gluetool.Glue(tool=None, sentry=None)

Bases: gluetool.glue.Configurable

Main workhorse of the gluetool. Manages modules, their instances and runs them as requested.

Parameters:

• tool (gluetool.tool.Tool) – If set, it’s an gluetool-like tool that created this instance. Some functionality may need it to gain access to bits like its command-name.
• sentry (gluetool.sentry.Sentry) – If set, it provides interface to Sentry.

_check_pm_file(filepath)

Make sure a file looks like a gluetool module:

• can be processed by Python parser,
• imports gluetool.glue.Glue and gluetool.glue.Module,
• contains child class of gluetool.glue.Module.

Parameters:filepath (str) – path to a file. Returns:True if file contains gluetool module, False otherwise. Raises:gluetool.glue.GlueError – when it’s not possible to finish the check.

_discover_gm_in_dir(dirpath, registry, pm_prefix)

Discover gluetool modules in a directory tree.

In essence, it scans directory and its subdirectories for files with .py suffix, and searches for classes derived from gluetool.glue.Module in these files.

Parameters:

• dirpath (str) – path to a directory.
• DiscoveredModule) registry (dict(str,) – registry of modules to which new ones would be added.
• pm_prefix (str) – a string used to prefix all imported Python module names.

_discover_gm_in_entry_point(entry_point, registry)

_discover_gm_in_file(registry, filepath, pm_name, group_name)

Discover gluetool modules in a file.

Attempts to import the file as a Python module, and then checks its content and looks for gluetool modules.

Parameters:

• DiscoveredModule) registry (dict(str,) – registry of modules to which new ones would be added.
• filepath (str) – path to a file.
• pm_name (str) – a Python module name to use for the imported module.
• group_name (str) – a gluetool module group name assigned to gluetool modules discovered in the file.

_do_import_pm(filepath, pm_name)

Attempt to import a file as a Python module.

Parameters:

• filepath (str) – a file to import.
• pm_name (str) – name assigned to the imported module.

Returns:

imported Python module.

Raises:

gluetool.glue.GlueError – when import failed.

_eval_context()

Gather contexts of all modules in a pipeline and merge them together.

Always returns a unique dictionary object, therefore it is safe for caller to update it. The return value is not cached in any way, therefore the change if its content won’t affect future callers.

Provided as a shared function, registered by the Glue instance itself.

Return type:dict

_eval_context_module_caller()

Infere module instance calling the eval context shared function.

Return type:gluetool.glue.Module

_import_pm(filepath, pm_name)

If a file contains gluetool modules, import the file as a Python module. If the file does not look like it contains gluetool modules, or when it’s not possible to import the Python module successfully, method simply warns user and ignores the file.

Parameters:

• filepath (str) – file to load.
• pm_name (str) – Python module name for the loaded module.

Returns:

loaded Python module.

Raises:

gluetool.glue.GlueError – when import failed.

_register_module(registry, group_name, klass, filepath)

Register one discovered gluetool module.

Parameters:

• DiscoveredModule) registry (dict(str,) – module registry to add module to.
• group_name (str) – group the module belongs to.
• klass (Module) – module class.
• filepath (str) – path to a file module comes from.

add_shared(funcname, module)

Add a shared function. Overwrites previously registered shared function of the same name.

Parameters:

• funcname (str) – name of the shared function.
• module (Module) – module providing the shared function.

current_module

current_pipeline

discover_modules(entry_points=None, paths=None)

Discover and load all accessible modules.

Two sources are examined:

1. entry points, handled by setuptools, to which Python packages can attach gluetool modules they provide,
2. directory trees.

Parameters:

• entry_points (list(str)) – list of entry point names to which gluetool modules are attached. If not set, entry points set byt the configuration (--module-entry-point option) are used.
• paths (list(str)) – list of directories to search for gluetool modules. If not set, paths set by the configuration (--module-path option) are used.

Return type:

dict(str, DiscoveredModule)

Returns:

mapping between module names and DiscoveredModule instances, describing each module.

dryrun_level

eval_context

Returns “global” evaluation context - some variables that are nice to have in all contexts.

get_shared(funcname)

Return a shared function.

Parameters:funcname (str) – name of the shared function. Returns:a callable (shared function), or None if no such shared function exists.

has_shared(funcname)

Check whether a shared function of a given name exists.

Parameters:funcname (str) – name of the shared function. Return type:bool

init_module(module_name, actual_module_name=None)

Given a name of the module, create its instance and give it a name.

Parameters:

• module_name (str) – Name under which will be the module instance known.
• actual_module_name (str) – Name of the module to instantiate. It does not have to match module_name - actual_module_name refers to the list of known gluetool modules while module_name is basically an arbitrary name new instance calls itself. If it’s not set, which is the most common situation, it defaults to module_name.

Returns:

A Module instance.

module_config_paths

List of paths in which module config files reside.

module_data_paths

List of paths in which module data files reside.

module_entry_points

List of setuptools entry points to which modules are attached.

module_paths

List of paths in which modules reside.

modules_as_groups(modules=None)

Gathers modules by their groups.

Return type:dict(str, dict(str, DiscoveredModule)) Returns:dictonary where keys represent module groups, and values are mappings between module names and the corresponding modules.

modules_descriptions(modules=None, groups=None)

Returns a string with modules and their descriptions.

Parameters:

• DiscoveredModule) modules (dict(str,) – mapping with modules. If not set, current known modules are used.
• groups (list(str)) – if set, limit descriptions to modules belinging to the given groups.

Return type:

str

name = 'gluetool core'

options = [('Global options', {('V', 'version'): {'action': 'store_true', 'help': 'Print version'}, ('E', 'list-eval-context'): {'action': 'store_true', 'default': False, 'help': 'List all available variables provided by modules in their evaluation contexts.'}, ('L', 'list-shared'): {'action': 'store_true', 'default': False, 'help': 'List all available shared functions.'}, ('l', 'list-modules'): {'action': 'append', 'const': True, 'nargs': '?', 'help': 'List all available modules. If a GROUP is set, limits list to the given module group.', 'metavar': 'GROUP'}, 'no-sentry-exceptions': {'action': 'append', 'default': [], 'help': 'List of exception names, which are not reported to Sentry (Default: none)'}, ('r', 'retries'): {'default': 0, 'type': <type 'int'>, 'help': 'Number of retries'}}), ('Output control', {('j', 'json-file'): {'default': None, 'help': '\n If set, all log messages (including ``VERBOSE``) are stored in this file\n in a form of JSON structures (default: %(default)s).\n '}, 'verbose-file': {'default': None, 'help': 'Log messages with ``VERBOSE`` level sent to this file.'}, ('v', 'verbose'): {'action': 'store_true', 'help': 'Log **all** messages to the terminal (WARNING: even more verbose than ``-d``!).'}, ('i', 'info'): {'action': 'store_true', 'help': 'Print command-line that would re-run the gluetool session'}, ('d', 'debug'): {'action': 'store_true', 'help': 'Log debugging messages to the terminal (WARNING: very verbose!).'}, ('p', 'pid'): {'action': 'store_true', 'help': 'Log PID of gluetool process'}, ('o', 'debug-file', 'output'): {'help': 'Log messages with at least ``DEBUG`` level are sent to this file.'}, ('c', 'colors'): {'action': 'store_true', 'help': 'Colorize logging on the terminal'}, 'show-traceback': {'action': 'store_true', 'default': False, 'help': '\n Display exception tracebacks on terminal (besides the debug file when ``--debug-file``\n is used) (default: %(default)s).\n '}, 'json-file-pretty': {'action': 'store_true', 'help': '\n When JSON output to file is enable (via ``-j/--json-file``), instead of emmiting one log\n entry per line, nicely format the entries for better readability.\n '}, ('J', 'json-output'): {'action': 'store_true', 'help': '\n If set, all log messages sent to the terminal are emitted as JSON structures.\n '}, ('q', 'quiet'): {'action': 'store_true', 'help': 'Silence info messages'}, 'json-output-pretty': {'action': 'store_true', 'help': '\n When JSON output is enable (via ``-J/--json-output``), instead of emmiting one log\n entry per line, nicely format the entries for better readability.\n '}}), ('Directories', {'module-entry-point': {'action': 'append', 'default': [], 'help': 'Specify setuptools entry point for modules.', 'metavar': 'ENTRY-POINT'}, 'module-path': {'action': 'append', 'default': [], 'help': 'Specify directory with modules.', 'metavar': 'DIR'}, 'module-config-path': {'action': 'append', 'default': [], 'help': 'Specify directory with module configuration files.', 'metavar': 'DIR'}, 'module-data-path': {'action': 'append', 'default': [], 'help': 'Specify directory with module data files.', 'metavar': 'DIR'}}), ('Dry run options', {'isolated-run': {'action': 'store_true', 'default': False, 'help': 'Modules that support this option will not interact with the outside world.'}, 'dry-run': {'action': 'store_true', 'default': False, 'help': 'Modules that support this option will make no changes to the outside world.'}}), {'pipeline': {'raw': True, 'nargs': '...', 'help': 'List of modules and their options, passed after gluetool options.'}}]

parse_args(args)

Public entry point to argument parsing. Child classes must implement this method, e.g. by calling gluetool.glue.Configurable._parse_args() which makes use of additional argparse.ArgumentParser options.

parse_config(paths)

Public entry point to configuration parsing. Child classes must implement this method, e.g. by calling gluetool.glue.Configurable._parse_config() which requires list of paths.

require_shared(*names, **kwargs)

Make sure given shared functions exist.

Parameters:

• names (tuple(str)) – iterable of shared function names.
• warn_only (bool) – if set, only warning is emitted. Otherwise, when any required shared function didn’t exist, an exception is raised.

run_module(module_name, module_argv=None, actual_module_name=None)

Syntax sugar for run_modules(), in the case you want to run just a one-shot module.

Parameters:

• module_name (str) – Name under which will be the module instance known.
• module_argv (list(str)) – Arguments of the module.
• actual_module_name (str) – Name of the module to instantiate. It does not have to match module_name - actual_module_name refers to the list of known gluetool modules while module_name is basically an arbitrary name new instance calls itself. If it’s not set, which is the most common situation, it defaults to module_name.

run_modules(steps)

run_pipeline(pipeline)

sentry_submit_exception(*args, **kwargs)

Submits exceptions to the Sentry server. Does nothing by default, unless this instance is initialized with a gluetool.sentry.Sentry instance which actually does the job.

See gluetool.sentry.Sentry.submit_exception().

sentry_submit_message(*args, **kwargs)

Submits message to the Sentry server. Does nothing by default, unless this instance is initialized with a gluetool.sentry.Sentry instance which actually does the job.

See gluetool.sentry.Sentry.submit_warning().

shared(funcname, *args, **kwargs)

Call a shared function, passing it all positional and keyword arguments.

class gluetool.Module(glue, name)

Bases: gluetool.glue.Configurable

Base class of all gluetool modules.

Parameters:

glue (gluetool.glue.Glue) – Glue instance owning the module.

Variables:

• glue (gluetool.glue.Glue) – Glue instance owning the module.
• _config (dict) – internal configuration store. Values of all module options are stored here, regardless of them being set on command-line or in the configuration file.
• _overloaded_shared_functions (dict) – If a shared function added by this module overloades an older function of the same name, registered by a previous module, the overloaded one is added into this dictionary. The module can then call this saved function - using overloaded_shared() - to implement a “chain” of shared functions, when one calls another, implementing the same operation.

_generate_shared_functions_help()

Generate help for shared functions provided by the module.

Returns:Formatted help, describing module’s shared functions.

_paths_with_module(roots)

Return paths cretaed by joining roots with module’s unique name.

Parameters:roots (list(str)) – List of root directories.

add_shared()

Add all shared functions declared by the module.

description = None

Short module description, displayed in gluetool’s module listing.

destroy(failure=None)

Here should go any code that needs to be run on exit, like job cleanup etc.

Parameters:failure (gluetool.glue.Failure) – if set, carries information about failure that made gluetool to destroy the whole session. Modules might want to take actions based on provided information, e.g. send different notifications.

dryrun_level

execute()

In this method, modules can perform any work they deemed necessary for completing their purpose. E.g. if the module promises to run some tests, this is the place where the code belongs to.

By default, this method does nothing. Reimplement as needed.

get_shared(funcname)

Return a shared function.

Parameters:funcname (str) – name of the shared function. Returns:a callable (shared function), or None if no such shared function exists.

has_shared(funcname)

Check whether a shared function of a given name exists.

Parameters:funcname (str) – name of the shared function. Return type:bool

overloaded_shared(funcname, *args, **kwargs)

Call a shared function overloaded by the one provided by this module. This way, a module can give chance to other implementations of the same name, e.g. function named publish_message, working with message bus A, would call previously shared function holding of this name, registered by a module earlier in the pipeline, which works with message bus B.

parse_args(args)

Public entry point to argument parsing. Child classes must implement this method, e.g. by calling gluetool.glue.Configurable._parse_args() which makes use of additional argparse.ArgumentParser options.

parse_config()

Public entry point to configuration parsing. Child classes must implement this method, e.g. by calling gluetool.glue.Configurable._parse_config() which requires list of paths.

require_shared(*names, **kwargs)

Make sure given shared functions exist.

Parameters:

• names (tuple(str)) – iterable of shared function names.
• warn_only (bool) – if set, only warning is emitted. Otherwise, when any required shared function didn’t exist, an exception is raised.

run_module(module, args=None)

sanity()

In this method, modules can define additional checks before execution.

Some examples:

• Advanced checks on passed options
• Check for additional requirements (tools, data, etc.)

By default, this method does nothing. Reimplement as needed.

shared(funcname, *args, **kwargs)

Call a shared function, passing it all positional and keyword arguments.

shared_functions = []

Iterable of names of shared functions exported by the module.

exception gluetool.GlueError(message, caused_by=None, sentry_fingerprint=None, sentry_tags=None, **kwargs)

Bases: exceptions.Exception

Generic gluetool exception.

Parameters:

• message (str) – Exception message, describing what happened.
• caused_by (tuple) – If set, contains tuple as returned by sys.exc_info(), describing the exception that caused this one to be born. If not set, constructor will try to auto-detect this information, and if there’s no such information, instance property caused_by will be set to None.
• sentry_fingerprint (list(str)) – if set, it is used as a Sentry fingerprint of the exception. See sentry_fingerprint() for more details.
• str) sentry_tags (dict(str,) – if set, it is merged with other tags when submitting the exception. See sentry_tags() for more details.

Variables:

• message (str) – Exception message, describing what happened.
• caused_by (tuple) – If set, contains tuple as returned by sys.exc_info(), describing the exception that caused this one to be born. None otherwise.

no_sentry_exceptions = []

sentry_fingerprint(current)

Default grouping of events into issues might be too general for some cases. This method gives users a chance to provide custom fingerprint Sentry could use to group events in a more suitable way.

E.g. user might be interested in some sort of connection issues but they would like to have them grouped not by a traceback (which is the default method) but per remote host IP. For that, the Sentry integration code will call sentry_fingerprint method of a raised exception, and the method should return new fingerprint, let’s say [<exception class name>, <remote IP>], and Sentry will group events using this fingerprint.

If the exception was raised with sentry_fingerprint parameter set, it is returned instead of current, after prefixing the list of tags with a name of the exception’s class.

Parameters:current (list(str)) – current fingerprint. Usually ['{{ default }}'] telling Sentry to use its default method, but it could already be more specific. Return type:list(str) Returns:new fingerprint, e.g. ['FailedToConnectToAPI', '10.20.30.40']

sentry_tags(current)

Add, modify or remove tags attached to a Sentry event, reported when the exception was raised.

Most common usage would be an addition of tags, e.g. remote-host to allow search for events related to the same remote address.

If the exception was raised with sentry_tags parameter set, its value is injected to current before returning it.

Parameters:str) current (dict(str,) – current set of tags and their values. Return type:dict(str, str) Returns:new set of tags. It is possible to add tags directly into current and then return it.

submit_to_sentry

Decide whether the exception should be submitted to Sentry or not. By default, all exceptions are submitted. Exception listed in no_sentry_exceptions are not submitted.

Return type:bool Returns:True when the exception should be submitted to Sentry, False otherwise.

exception gluetool.SoftGlueError(message, caused_by=None, sentry_fingerprint=None, sentry_tags=None, **kwargs)

Bases: gluetool.glue.GlueError

Soft errors are errors Glue Ops and/or developers shouldn’t be bothered with, things that are up to the user to fix, e.g. empty set of tests. Hard errors are supposed to warn Ops/Devel teams about important infrastructure issues, code deficiencies, bugs and other issues that are fixable only by actions of Glue staff.

However, we still must provide notification to user(s), and since we expect them to fix the issues that led to raising the soft error, we must provide them with as much information as possible. Therefore modules dealing with notifications are expected to give these exceptions a chance to influence the outgoing messages, e.g. by letting them provide an e-mail body template.

exception gluetool.GlueRetryError(message, caused_by=None, sentry_fingerprint=None, sentry_tags=None, **kwargs)

Bases: gluetool.glue.GlueError

Retry gluetool exception

exception gluetool.GlueCommandError(cmd, output, **kwargs)

Bases: gluetool.glue.GlueError

Exception raised when external command failes.

Parameters:

• cmd (list) – Command as passed to gluetool.utils.run_command helper.
• output (gluetool.utils.ProcessOutput) – Process output data.

Variables:

• cmd (list) – Command as passed to gluetool.utils.run_command helper.
• output (gluetool.utils.ProcessOutput) – Process output data.

class gluetool.Failure(module, exc_info)

Bases: object

Bundles exception related info. Used to inform modules in their destroy() phase that gluetool session was killed because of exception raised by one of modules.

Parameters:

• module (gluetool.glue.Module) – module in which the error happened, or None.
• exc_info (tuple) – Exception information as returned by sys.exc_info().

Variables:

• module (gluetool.glue.Module) – module in which the error happened, or None.
• exception (Exception) – Shortcut to exc_info[1], if available, or None.
• exc_info (tuple) – Exception information as returned by sys.exc_info().
• sentry_event_id (str) – If set, the failure was reported to the Sentry under this ID.

class gluetool.Result(_is_ok, _value, _force=False)

Bases: typing.Generic

A simple Result type inspired by Rust.

A Result can be either Ok(value) - valid result, contains a meaningful value - or Error(error) which represents an error, carrying error’s description.

Parameters:

• _is_ok (bool) – True when the _value is OK-ish.
• _value – the value carried by the result.
• _force (bool) – guards against accidental direct use.

classmethod Error(error)

classmethod Ok(value)

_abc_cache = <_weakrefset.WeakSet object>

_abc_generic_negative_cache = <_weakrefset.WeakSet object>

_abc_generic_negative_cache_version = 31

_abc_registry = <_weakrefset.WeakSet object>

_gorg

alias of Result

error

Return the result value - error - if it is invalid. Otherwise, None is returned.

expect(message)

Return the result value if it is valid. Otherwise, an exception is raised.

expect_error(message)

Return the result value if it is invalid. Otherwise, an exception is raised.

is_error

Returns True if the result value is invalid.

is_ok

ok

Return the result value - valid - if it is valid. Otherwise, None is returned.

unwrap()

Return the result value if it is valid. Otherwise, an exception is rised.

unwrap_error()

Return the error value if the result is invalid. Othwerise, an exception is raised.

unwrap_or(default)

Return the result value if it is valid. Otherwise, default is returned.

value

Return the result value. It will be either one of valid and error types.

gluetool.Ok(value)

Shortcut function to create a new valid Result.

gluetool.Error(error)

Shortcut function to create a new error Result.