gluetool.glue module¶
-
class
gluetool.glue.
ArgumentParser
(prog=None, usage=None, description=None, epilog=None, version=None, parents=[], formatter_class=<class 'argparse.HelpFormatter'>, prefix_chars='-', fromfile_prefix_chars=None, argument_default=None, conflict_handler='error', add_help=True)[source]¶ Bases:
argparse.ArgumentParser
Pretty much the
argparse.ArgumentParser
, it overrides just theargparse.ArgumentParser.error()
method, to catch errors and to wrap them into nice and commonGlueError
instances.The original prints (for us) useless message, including the program name, and raises
SystemExit
exception. Such action does not provide necessary information when encountered in Sentry, for example.-
error
(message)[source]¶ Must not return - raising an exception is a good way to “not return”.
Raises: gluetool.glue.GlueError – When argument parser encounters an error.
-
-
class
gluetool.glue.
CallbackModule
(name, glue, callback, *args, **kwargs)[source]¶ Bases:
mock.MagicMock
Stand-in replacement for common :py:`Module` instances which does not represent any real module. We need it only to simplify code pipeline code - it can keep working with
Module
-like instances, since this class mocks each and every method, but calls givencallback
in itsexecute
method.Parameters: - name (str) – name of the pseudo-module.
- glue (Glue) –
Glue
instance governing the pipeline this module is part of. - callback (callable) – called in the
execute
method. Its arguments will beglue
, followed by remaining positional and keyword arguments (args
andkwargs
). - args (tuple) – passed to
callback
. - kwargs (dict) – passed to
callback
.
-
_get_child_mock
(**kw)[source]¶ Create the child mocks for attributes and return value. By default child mocks will be the same type as the parent. Subclasses of Mock may want to override this to customize the way child mocks are made.
For non-callable mocks the callable variant will be used (rather than any custom subclass).
-
class
gluetool.glue.
Configurable
(logger)[source]¶ Bases:
gluetool.log.LoggerMixin
,object
Base class of two main
gluetool
classes -gluetool.glue.Glue
andgluetool.glue.Module
. Gives them the ability to use options, settable from configuration files and/or command-line arguments.Variables: _config (dict) – internal configuration store. Values of all options are stored here, regardless of them being set on command-line or by the configuration file. -
classmethod
_create_args_parser
(**kwargs)[source]¶ Create an argument parser. Used by Sphinx to document “command-line” options of the module - which are, by the way, the module options as well.
Parameters: kwargs (dict) – Additional arguments passed to argparse.ArgumentParser
.
-
_dryrun_allows
(threshold, msg)[source]¶ Check whether current dry-run level allows an action. If the current dry-run level is equal of higher than
threshold
, then the action is not allowed.E.g. when action’s
threshold
isDryRunLevels.ISOLATED
, and the current level isDryRunLevels.DRY
, the action is allowed.Parameters: - threshold (DryRunLevels) – Dry-run level the action is not allowed.
- msg (str) – Message logged (as a warning) when the action is deemed not allowed.
Returns: True
when action is allowed,False
otherwise.
-
static
_for_each_option
(callback, options)[source]¶ Given dictionary defining options, call a callback for each of them.
Parameters: - options (dict) – Dictionary of options, in a form
option-name: option-params
. - callback (callable) – Must accept at least 3 parameters: option name (
str
), all option names (short and long ones) (tuple(str, str)
), and option params (dict
).
- options (dict) – Dictionary of options, in a form
-
static
_for_each_option_group
(callback, options)[source]¶ Given set of options, call a callback for each option group.
Parameters: - options – List of option groups, or a dict listing options directly.
- callback (callable) – Must accept at least 2 parameters:
options
(dict
), listing options in the group, and keyword parametergroup_name
(str
), which is set to group name when theoptions
defines an option group.
-
_parse_args
(args, **kwargs)[source]¶ Parse command-line arguments. Uses
argparse
for the actual parsing. Updates module’s configuration store with values returned by parser.Parameters: args (list) – arguments passed to this module. Similar to what sys.argv
provides on program level.
-
_parse_config
(paths)[source]¶ Parse configuration files. Uses
ConfigParser
for the actual parsing. Updates module’s configuration store with values found returned by the parser.Parameters: paths (list) – List of paths to possible configuration files.
-
dryrun_allows
(msg)[source]¶ Checks whether current dry-run level allows an action which is disallowed on
DryRunLevels.DRY
level.See
Configurable._dryrun_allows()
for detailed description.
-
dryrun_enabled
¶ True
if dry-run level is enabled, on any level.
-
dryrun_level
¶ Return current dry-run level. This must be implemented by class descendants because each one finds the necessary information in different places.
-
eval_context
¶ Return “evaluation context” - a dictionary of variable names (usually in uppercase) and their values, which is supposed to be used in various “evaluate this” operations like rendering of templates.
To provide nice and readable documentation of variables, returned by a module’s
eval_context
property, assign a dictionary, describing these variables, to a local variable named__content__
:... @property def eval_context(self): __content__ = { 'FOO': 'This is an important variable, extracted from clouds.' } return { 'FOO': 42 }
gluetool
core will extract this information and will use it to generate different help texts like your module’s help or a list of all known context variables.Return type: dict
-
isolatedrun_allows
(msg)[source]¶ Checks whether current dry-run level allows an action which is disallowed on
DryRunLevels.ISOLATED
level.
-
name
= None¶ Module name. Usually matches the name of the source file, no suffix.
-
option
(*names)[source]¶ Return values of given options from module’s configuration store.
Parameters: names (str) – names of requested options. Returns: either a value or None
if such option does not exist. When multiple options are requested, a tuple of their values is returned, for a single option its value is not wrapped by a tuple.
-
options
= {}¶ The
options
variable defines options accepted by module, and their properties:options = { <option name>: { <option properties> }, ... }
where
<option name>
is used to name the option in the parser, and two formats are accepted (don’t add any leading dashes (-
nor--
):<long name>
tuple(<short name>, <long name #1>, <long name #2>, ...)
- the first of long names (
long name #1
) is used to identify the option - other long names are understood by argument parser but their values are stored underlong name #1
option. - dictionary
<option properties>
is passed toargparse.ArgumentParser.add_argument()
as keyword arguments when the option is being added to the parser, therefore any arguments recognized byargparse
can be used.
It is also possible to use groups:
options = [ (<group name>, <group options>), ... ]
where
<group name>
is the name of the group, e.g.Debugging options
<group options>
is thedict
with all group options, as described above.
This way, you can split pile of options into conceptualy closer groups of options. A single
dict
you would have is split into multiple smaller dictionaries, and each one is coupled with the group name in atuple
.
-
options_note
= None¶ If set, it will be printed after all options as a help’s epilog.
-
parse_args
(args)[source]¶ Public entry point to argument parsing. Child classes must implement this method, e.g. by calling
gluetool.glue.Configurable._parse_args()
which makes use of additionalargparse.ArgumentParser
options.
-
parse_config
()[source]¶ Public entry point to configuration parsing. Child classes must implement this method, e.g. by calling
gluetool.glue.Configurable._parse_config()
which requires list of paths.
-
required_options
= []¶ Iterable of names of required options.
-
supported_dryrun_level
= 0¶ Highest supported level of dry-run.
-
unique_name
= None¶ Unque name of this (module) instance.
Used by modules, has no meaning elsewhere, but since dry-run checks are done on this level, it must be declared here to make pylint happy :/
-
classmethod
-
class
gluetool.glue.
DiscoveredModule
(klass, group)¶ Bases:
tuple
Describes one discovered
gluetool
module.Variables: -
_asdict
()¶ Return a new OrderedDict which maps field names to their values
-
_field_types
= {'group': <type 'str'>, 'klass': typing.Type[gluetool.glue.Module]}¶
-
_fields
= ('klass', 'group')¶
-
classmethod
_make
(iterable, new=<built-in method __new__ of type object>, len=<built-in function len>)¶ Make a new DiscoveredModule object from a sequence or iterable
-
_replace
(**kwds)¶ Return a new DiscoveredModule object replacing specified fields with new values
-
group
¶ Alias for field number 1
-
klass
¶ Alias for field number 0
-
-
class
gluetool.glue.
DryRunLevels
[source]¶ Bases:
enum.IntEnum
Dry-run levels.
Variables: -
DEFAULT
= 0¶
-
DRY
= 1¶
-
ISOLATED
= 2¶
-
-
class
gluetool.glue.
Failure
(module, exc_info)[source]¶ Bases:
object
Bundles exception related info. Used to inform modules in their
destroy()
phase thatgluetool
session was killed because of exception raised by one of modules.Parameters: - module (gluetool.glue.Module) – module in which the error happened, or
None
. - exc_info (tuple) – Exception information as returned by
sys.exc_info()
.
Variables: - module (gluetool.glue.Module) – module in which the error happened, or
None
. - exception (Exception) – Shortcut to
exc_info[1]
, if available, orNone
. - exc_info (tuple) – Exception information as returned by
sys.exc_info()
. - sentry_event_id (str) – If set, the failure was reported to the Sentry under this ID.
- module (gluetool.glue.Module) – module in which the error happened, or
-
class
gluetool.glue.
Glue
(tool=None, sentry=None)[source]¶ Bases:
gluetool.glue.Configurable
Main workhorse of the
gluetool
. Manages modules, their instances and runs them as requested.Parameters: - tool (gluetool.tool.Tool) – If set, it’s an
gluetool
-like tool that created this instance. Some functionality may need it to gain access to bits like its command-name. - sentry (gluetool.sentry.Sentry) – If set, it provides interface to Sentry.
-
_check_pm_file
(filepath)[source]¶ Make sure a file looks like a
gluetool
module:- can be processed by Python parser,
- imports
gluetool.glue.Glue
andgluetool.glue.Module
, - contains child class of
gluetool.glue.Module
.
Parameters: filepath (str) – path to a file. Returns: True
if file containsgluetool
module,False
otherwise.Raises: gluetool.glue.GlueError – when it’s not possible to finish the check.
-
_discover_gm_in_dir
(dirpath, registry, pm_prefix)[source]¶ Discover
gluetool
modules in a directory tree.In essence, it scans directory and its subdirectories for files with
.py
suffix, and searches for classes derived fromgluetool.glue.Module
in these files.Parameters:
-
_discover_gm_in_file
(registry, filepath, pm_name, group_name)[source]¶ Discover
gluetool
modules in a file.Attempts to import the file as a Python module, and then checks its content and looks for
gluetool
modules.Parameters:
-
_do_import_pm
(filepath, pm_name)[source]¶ Attempt to import a file as a Python module.
Parameters: Returns: imported Python module.
Raises: gluetool.glue.GlueError – when import failed.
-
_eval_context
()[source]¶ Gather contexts of all modules in a pipeline and merge them together.
Always returns a unique dictionary object, therefore it is safe for caller to update it. The return value is not cached in any way, therefore the change if its content won’t affect future callers.
Provided as a shared function, registered by the Glue instance itself.
Return type: dict
-
_eval_context_module_caller
()[source]¶ Infere module instance calling the eval context shared function.
Return type: gluetool.glue.Module
-
_import_pm
(filepath, pm_name)[source]¶ If a file contains
gluetool
modules, import the file as a Python module. If the file does not look like it containsgluetool
modules, or when it’s not possible to import the Python module successfully, method simply warns user and ignores the file.Parameters: Returns: loaded Python module.
Raises: gluetool.glue.GlueError – when import failed.
-
_register_module
(registry, group_name, klass, filepath)[source]¶ Register one discovered
gluetool
module.Parameters:
Add a shared function. Overwrites previously registered shared function of the same name.
Parameters:
-
current_module
¶
-
current_pipeline
¶
-
discover_modules
(entry_points=None, paths=None)[source]¶ Discover and load all accessible modules.
Two sources are examined:
- entry points, handled by setuptools, to which Python packages can attach
gluetool
modules they provide, - directory trees.
Parameters: - entry_points (list(str)) – list of entry point names to which
gluetool
modules are attached. If not set, entry points set byt the configuration (--module-entry-point
option) are used. - paths (list(str)) – list of directories to search for
gluetool
modules. If not set, paths set by the configuration (--module-path
option) are used.
Return type: Returns: mapping between module names and
DiscoveredModule
instances, describing each module.- entry points, handled by setuptools, to which Python packages can attach
-
dryrun_level
¶
-
eval_context
¶ Returns “global” evaluation context - some variables that are nice to have in all contexts.
Return a shared function.
Parameters: funcname (str) – name of the shared function. Returns: a callable (shared function), or None
if no such shared function exists.
Check whether a shared function of a given name exists.
Parameters: funcname (str) – name of the shared function. Return type: bool
-
init_module
(module_name, actual_module_name=None)[source]¶ Given a name of the module, create its instance and give it a name.
Parameters: - module_name (str) – Name under which will be the module instance known.
- actual_module_name (str) – Name of the module to instantiate. It does not have to match
module_name
-actual_module_name
refers to the list of knowngluetool
modules whilemodule_name
is basically an arbitrary name new instance calls itself. If it’s not set, which is the most common situation, it defaults tomodule_name
.
Returns: A
Module
instance.
-
module_config_paths
¶ List of paths in which module config files reside.
-
module_data_paths
¶ List of paths in which module data files reside.
-
module_entry_points
¶ List of setuptools entry points to which modules are attached.
-
module_paths
¶ List of paths in which modules reside.
-
modules_as_groups
(modules=None)[source]¶ Gathers modules by their groups.
Return type: dict(str, dict(str, DiscoveredModule)) Returns: dictonary where keys represent module groups, and values are mappings between module names and the corresponding modules.
-
modules_descriptions
(modules=None, groups=None)[source]¶ Returns a string with modules and their descriptions.
Parameters: Return type:
-
name
= 'gluetool core'¶
-
options
= [('Global options', {('V', 'version'): {'action': 'store_true', 'help': 'Print version'}, ('E', 'list-eval-context'): {'action': 'store_true', 'default': False, 'help': 'List all available variables provided by modules in their evaluation contexts.'}, ('L', 'list-shared'): {'action': 'store_true', 'default': False, 'help': 'List all available shared functions.'}, ('l', 'list-modules'): {'action': 'append', 'const': True, 'nargs': '?', 'help': 'List all available modules. If a GROUP is set, limits list to the given module group.', 'metavar': 'GROUP'}, 'no-sentry-exceptions': {'action': 'append', 'default': [], 'help': 'List of exception names, which are not reported to Sentry (Default: none)'}, ('r', 'retries'): {'default': 0, 'type': <type 'int'>, 'help': 'Number of retries'}}), ('Output control', {('j', 'json-file'): {'default': None, 'help': '\n If set, all log messages (including ``VERBOSE``) are stored in this file\n in a form of JSON structures (default: %(default)s).\n '}, 'verbose-file': {'default': None, 'help': 'Log messages with ``VERBOSE`` level sent to this file.'}, ('v', 'verbose'): {'action': 'store_true', 'help': 'Log **all** messages to the terminal (WARNING: even more verbose than ``-d``!).'}, ('i', 'info'): {'action': 'store_true', 'help': 'Print command-line that would re-run the gluetool session'}, ('d', 'debug'): {'action': 'store_true', 'help': 'Log debugging messages to the terminal (WARNING: very verbose!).'}, ('p', 'pid'): {'action': 'store_true', 'help': 'Log PID of gluetool process'}, ('o', 'debug-file', 'output'): {'help': 'Log messages with at least ``DEBUG`` level are sent to this file.'}, ('c', 'colors'): {'action': 'store_true', 'help': 'Colorize logging on the terminal'}, 'show-traceback': {'action': 'store_true', 'default': False, 'help': '\n Display exception tracebacks on terminal (besides the debug file when ``--debug-file``\n is used) (default: %(default)s).\n '}, 'json-file-pretty': {'action': 'store_true', 'help': '\n When JSON output to file is enable (via ``-j/--json-file``), instead of emmiting one log\n entry per line, nicely format the entries for better readability.\n '}, ('J', 'json-output'): {'action': 'store_true', 'help': '\n If set, all log messages sent to the terminal are emitted as JSON structures.\n '}, ('q', 'quiet'): {'action': 'store_true', 'help': 'Silence info messages'}, 'json-output-pretty': {'action': 'store_true', 'help': '\n When JSON output is enable (via ``-J/--json-output``), instead of emmiting one log\n entry per line, nicely format the entries for better readability.\n '}}), ('Directories', {'module-entry-point': {'action': 'append', 'default': [], 'help': 'Specify setuptools entry point for modules.', 'metavar': 'ENTRY-POINT'}, 'module-path': {'action': 'append', 'default': [], 'help': 'Specify directory with modules.', 'metavar': 'DIR'}, 'module-config-path': {'action': 'append', 'default': [], 'help': 'Specify directory with module configuration files.', 'metavar': 'DIR'}, 'module-data-path': {'action': 'append', 'default': [], 'help': 'Specify directory with module data files.', 'metavar': 'DIR'}}), ('Dry run options', {'isolated-run': {'action': 'store_true', 'default': False, 'help': 'Modules that support this option will not interact with the outside world.'}, 'dry-run': {'action': 'store_true', 'default': False, 'help': 'Modules that support this option will make no changes to the outside world.'}}), {'pipeline': {'raw': True, 'nargs': '...', 'help': 'List of modules and their options, passed after gluetool options.'}}]¶
-
parse_args
(args)[source]¶ Public entry point to argument parsing. Child classes must implement this method, e.g. by calling
gluetool.glue.Configurable._parse_args()
which makes use of additionalargparse.ArgumentParser
options.
-
parse_config
(paths)[source]¶ Public entry point to configuration parsing. Child classes must implement this method, e.g. by calling
gluetool.glue.Configurable._parse_config()
which requires list of paths.
Make sure given shared functions exist.
Parameters:
-
run_module
(module_name, module_argv=None, actual_module_name=None)[source]¶ Syntax sugar for
run_modules()
, in the case you want to run just a one-shot module.Parameters: - module_name (str) – Name under which will be the module instance known.
- module_argv (list(str)) – Arguments of the module.
- actual_module_name (str) – Name of the module to instantiate. It does not have to match
module_name
-actual_module_name
refers to the list of knowngluetool
modules whilemodule_name
is basically an arbitrary name new instance calls itself. If it’s not set, which is the most common situation, it defaults tomodule_name
.
-
sentry_submit_exception
(*args, **kwargs)[source]¶ Submits exceptions to the Sentry server. Does nothing by default, unless this instance is initialized with a
gluetool.sentry.Sentry
instance which actually does the job.
-
sentry_submit_message
(*args, **kwargs)[source]¶ Submits message to the Sentry server. Does nothing by default, unless this instance is initialized with a
gluetool.sentry.Sentry
instance which actually does the job.See
gluetool.sentry.Sentry.submit_warning()
.
Call a shared function, passing it all positional and keyword arguments.
- tool (gluetool.tool.Tool) – If set, it’s an
-
exception
gluetool.glue.
GlueCommandError
(cmd, output, **kwargs)[source]¶ Bases:
gluetool.glue.GlueError
Exception raised when external command failes.
Parameters: - cmd (list) – Command as passed to gluetool.utils.run_command helper.
- output (gluetool.utils.ProcessOutput) – Process output data.
Variables: - cmd (list) – Command as passed to gluetool.utils.run_command helper.
- output (gluetool.utils.ProcessOutput) – Process output data.
-
exception
gluetool.glue.
GlueError
(message, caused_by=None, sentry_fingerprint=None, sentry_tags=None, **kwargs)[source]¶ Bases:
exceptions.Exception
Generic
gluetool
exception.Parameters: - message (str) – Exception message, describing what happened.
- caused_by (tuple) – If set, contains tuple as returned by
sys.exc_info()
, describing the exception that caused this one to be born. If not set, constructor will try to auto-detect this information, and if there’s no such information, instance propertycaused_by
will be set toNone
. - sentry_fingerprint (list(str)) – if set, it is used as a Sentry fingerprint of the exception. See
sentry_fingerprint()
for more details. - str) sentry_tags (dict(str,) – if set, it is merged with other tags when submitting the exception.
See
sentry_tags()
for more details.
Variables: - message (str) – Exception message, describing what happened.
- caused_by (tuple) – If set, contains tuple as returned by
sys.exc_info()
, describing the exception that caused this one to be born.None
otherwise.
-
no_sentry_exceptions
= []¶
-
sentry_fingerprint
(current)[source]¶ Default grouping of events into issues might be too general for some cases. This method gives users a chance to provide custom fingerprint Sentry could use to group events in a more suitable way.
E.g. user might be interested in some sort of connection issues but they would like to have them grouped not by a traceback (which is the default method) but per remote host IP. For that, the Sentry integration code will call
sentry_fingerprint
method of a raised exception, and the method should return new fingerprint, let’s say[<exception class name>, <remote IP>]
, and Sentry will group events using this fingerprint.If the exception was raised with
sentry_fingerprint
parameter set, it is returned instead ofcurrent
, after prefixing the list of tags with a name of the exception’s class.Parameters: current (list(str)) – current fingerprint. Usually ['{{ default }}']
telling Sentry to use its default method, but it could already be more specific.Return type: list(str) Returns: new fingerprint, e.g. ['FailedToConnectToAPI', '10.20.30.40']
Add, modify or remove tags attached to a Sentry event, reported when the exception was raised.
Most common usage would be an addition of tags, e.g.
remote-host
to allow search for events related to the same remote address.If the exception was raised with
sentry_tags
parameter set, its value is injected tocurrent
before returning it.Parameters: str) current (dict(str,) – current set of tags and their values. Return type: dict(str, str) Returns: new set of tags. It is possible to add tags directly into current
and then return it.
-
exception
gluetool.glue.
GlueRetryError
(message, caused_by=None, sentry_fingerprint=None, sentry_tags=None, **kwargs)[source]¶ Bases:
gluetool.glue.GlueError
Retry gluetool exception
-
class
gluetool.glue.
Module
(glue, name)[source]¶ Bases:
gluetool.glue.Configurable
Base class of all
gluetool
modules.Parameters: glue (gluetool.glue.Glue) –
Glue
instance owning the module.Variables: - glue (gluetool.glue.Glue) –
Glue
instance owning the module. - _config (dict) – internal configuration store. Values of all module options are stored here, regardless of them being set on command-line or in the configuration file.
- _overloaded_shared_functions (dict) – If a shared function added by this module
overloades an older function of the same name, registered by a previous module,
the overloaded one is added into this dictionary. The module can then call this
saved function - using
overloaded_shared()
- to implement a “chain” of shared functions, when one calls another, implementing the same operation.
Generate help for shared functions provided by the module.
Returns: Formatted help, describing module’s shared functions.
-
_paths_with_module
(roots)[source]¶ Return paths cretaed by joining roots with module’s unique name.
Parameters: roots (list(str)) – List of root directories.
Add all shared functions declared by the module.
-
description
= None¶ Short module description, displayed in
gluetool
’s module listing.
-
destroy
(failure=None)[source]¶ Here should go any code that needs to be run on exit, like job cleanup etc.
Parameters: failure (gluetool.glue.Failure) – if set, carries information about failure that made gluetool
to destroy the whole session. Modules might want to take actions based on provided information, e.g. send different notifications.
-
dryrun_level
¶
-
execute
()[source]¶ In this method, modules can perform any work they deemed necessary for completing their purpose. E.g. if the module promises to run some tests, this is the place where the code belongs to.
By default, this method does nothing. Reimplement as needed.
Return a shared function.
Parameters: funcname (str) – name of the shared function. Returns: a callable (shared function), or None
if no such shared function exists.
Check whether a shared function of a given name exists.
Parameters: funcname (str) – name of the shared function. Return type: bool
Call a shared function overloaded by the one provided by this module. This way, a module can give chance to other implementations of the same name, e.g. function named
publish_message
, working with message bus A, would call previously shared function holding of this name, registered by a module earlier in the pipeline, which works with message bus B.
-
parse_args
(args)[source]¶ Public entry point to argument parsing. Child classes must implement this method, e.g. by calling
gluetool.glue.Configurable._parse_args()
which makes use of additionalargparse.ArgumentParser
options.
-
parse_config
()[source]¶ Public entry point to configuration parsing. Child classes must implement this method, e.g. by calling
gluetool.glue.Configurable._parse_config()
which requires list of paths.
Make sure given shared functions exist.
Parameters:
-
sanity
()[source]¶ In this method, modules can define additional checks before execution.
Some examples:
- Advanced checks on passed options
- Check for additional requirements (tools, data, etc.)
By default, this method does nothing. Reimplement as needed.
Call a shared function, passing it all positional and keyword arguments.
Iterable of names of shared functions exported by the module.
- glue (gluetool.glue.Glue) –
-
gluetool.glue.
ModuleRegistryType
¶ alias of
typing.Dict
-
class
gluetool.glue.
NamedPipeline
(glue, name, steps)[source]¶ Bases:
gluetool.glue.Pipeline
Pipeline with a name. The name is recorded in log messages emitted by the pipeline itself.
Parameters: - glue (Glue) –
Glue
instance, taking care of this pipeline. - name (str) – name of the pipeline.
- steps (list(PipelineStep)) – modules to run and their options.
Variables: - modules (list(Module)) – list of instantiated modules forming the pipeline.
- current_module (Module) – if set, it is the module which is currently being executed.
- glue (Glue) –
-
class
gluetool.glue.
Pipeline
(glue, steps, logger=None)[source]¶ Bases:
gluetool.log.LoggerMixin
,object
Pipeline of
gluetool
modules. Defined by its steps, takes care of registering their shared functions, running modules and destroying the pipeline.To simplify the workflow, 2 primitives are defined:
_safe_call()
- calls a given callback, returns its return value. Any exception raised by the callback is wrapped byFailure
instance and returned instead of what callback would return._for_each_module()
- loop over given list of modules, calling given callback for each of the modules._safe_call()
is used for the call, making sure we always have a return value and no exceptions.
Coupled with the following rules, things clear up a bit:
Callbacks are allowed to return either
None
orFailure
instance. * This rule matches_safe_call
behavior. There’s no need to worry about the return values, return valueof a callback can be immediately passed through
_safe_call
.- There are no other possible return values - return value is always either nothing or failure.
There are no “naked” exceptions, all are catched by
_safe_call
and converted to a common return value.Users of
_safe_call
and_for_each_module
also return eitherNone
orFailure
instance, therefore it is very easy to end method by areturn self._for_each_method(...)
call - return types of callback,_safe_call
,_for_each_module
and our method match, no need to translate them between these method.
Parameters: - glue (Glue) –
Glue
instance, taking care of this pipeline. - steps (list(PipelineStep)) – modules to run and their options.
Variables: - modules (list(Module)) – list of instantiated modules forming the pipeline.
- current_module (Module) – if set, it is the module which is currently being executed.
Add a shared function. Overwrites previously registered shared function of the same name.
Private part of API, for easier testing.
Parameters:
-
_destroy
(failure=None)[source]¶ “Destroy” the pipeline - call each module’s
destroy
method, reversing the order of modules. If adestroy
method raises an exception, loop ends and the failure is returned.Parameters: failure (Failure) – if set, it represents a failure that caused pipeline to stop, which was followed by a call to currently running _destroy
. It is passed to modules’destroy
methods.Returns: None
if everything went well, or aFailure
instance if anydestroy
method raised an exception.
-
_for_each_module
(modules, callback, *args, **kwargs)[source]¶ For each module in a list, call a given function with module as its first argument. If the call returns anything but
None
, the value is returned by this function as well, ending the loop.Parameters: Returns: value returned by
callback
, orNone
when loop finished.
-
_log_failure
(module, failure, label=None)[source]¶ Log a failure, and submit it to Sentry.
Parameters: - module (Module) – module to use for logging - apparently, the failure appeared when this module was running.
- failure (Failure) – failure to log.
- label (str) – label for logging purposes. If it’s set and exception exists, exception message is appended. If it’s not set and exception exists, exception message is used. If failure has no exception, a generic message is the final choice.
-
_safe_call
(callback, *args, **kwargs)[source]¶ “Safe” call a function with given arguments, converting raised exceptions to
Failure
instance.Parameters: callback (callable) – callable to call. Must return either None
orFailure
instance, although it can freely raise exceptions.Returns: value returned by callback
, orFailure
instance wrapping exception raise bycallback
.
Add a shared function. Overwrites previously registered shared function of the same name.
Parameters:
Return a shared function.
Parameters: funcname (str) – name of the shared function. Returns: a callable (shared function), or None
if no such shared function exists.
Check whether a shared function of a given name exists.
Parameters: funcname (str) – name of the shared function. Return type: bool
-
run
()[source]¶ Run a pipeline - instantiate modules, prepare and execute each of them. When done, destroy all modules.
Returns: tuple of two items, each of them either None
or aFailure
instance. The first item represents the output of the pipeline, the second item represents the output of the destroy chain. If the item isNone
, the stage finished without any issues, if it’s aFailure
instance, then an exception was raised during the stage, andFailure
wraps it.
Shared function registry. funcname: (module, fn)
-
class
gluetool.glue.
PipelineAdapter
(logger, pipeline_name)[source]¶ Bases:
gluetool.log.ContextAdapter
Custom logger adapter, adding pipeline name as a context.
Parameters: logger (logging.Logger) – parent logger this adapter modifies.
-
class
gluetool.glue.
PipelineStep
[source]¶ Bases:
object
Step of
gluetool
’s pipeline - which is basically just a list of steps.
-
class
gluetool.glue.
PipelineStepCallback
(name, callback, *args, **kwargs)[source]¶ Bases:
gluetool.glue.PipelineStep
Step of
gluetool
’s pipeline backed by callable.Parameters: - name (str) – name to give to the module instance. This name is used e.g. in logging or when searching for module’s config file.
- callback (callable) – a callable to execute.
-
class
gluetool.glue.
PipelineStepModule
(module, actual_module=None, argv=None)[source]¶ Bases:
gluetool.glue.PipelineStep
Step of
gluetool
’s pipeline backed by a module.Parameters: - module (str) – name to give to the module instance. This name is used e.g. in logging or when searching for module’s config file.
- actual_module (str) – The actual module class the step uses. Usually it is same as
module
but may differ,module
is then a mere “alias”.actual_module
is used to locate a module class, whose instance is then given namemodule
. - argv (list(str)) – list of options to be given to the module, in a form similar
to
sys.argv
.
-
module_designation
¶
-
exception
gluetool.glue.
SoftGlueError
(message, caused_by=None, sentry_fingerprint=None, sentry_tags=None, **kwargs)[source]¶ Bases:
gluetool.glue.GlueError
Soft errors are errors Glue Ops and/or developers shouldn’t be bothered with, things that are up to the user to fix, e.g. empty set of tests. Hard errors are supposed to warn Ops/Devel teams about important infrastructure issues, code deficiencies, bugs and other issues that are fixable only by actions of Glue staff.
However, we still must provide notification to user(s), and since we expect them to fix the issues that led to raising the soft error, we must provide them with as much information as possible. Therefore modules dealing with notifications are expected to give these exceptions a chance to influence the outgoing messages, e.g. by letting them provide an e-mail body template.