gluetool package¶
Subpackages¶
- gluetool.pylint package
- gluetool.tests package
- Submodules
- gluetool.tests.conftest module
- gluetool.tests.test_core module
- gluetool.tests.test_error module
- gluetool.tests.test_eval_context module
- gluetool.tests.test_help module
- gluetool.tests.test_json module
- gluetool.tests.test_load_yaml module
- gluetool.tests.test_log_exception module
- gluetool.tests.test_logging module
- gluetool.tests.test_new_xml_element module
- gluetool.tests.test_normalize_option module
- gluetool.tests.test_option module
- gluetool.tests.test_pipeline_step module
- gluetool.tests.test_render_template module
- gluetool.tests.test_requests module
- gluetool.tests.test_run_command module
- gluetool.tests.test_run_modules module
- gluetool.tests.test_shared module
- gluetool.tests.test_treat_url module
- gluetool.tests.test_utils module
- gluetool.tests.test_wait module
- Module contents
- Submodules
Submodules¶
Module contents¶
-
class
gluetool.
Glue
(tool=None, sentry=None)[source]¶ Bases:
gluetool.glue.Configurable
Main workhorse of the
gluetool
. Manages modules, their instances and runs them as requested.Parameters: - tool (gluetool.tool.Tool) – If set, it’s an
gluetool
-like tool that created this instance. Some functionality may need it to gain access to bits like its command-name. - sentry (gluetool.sentry.Sentry) – If set, it provides interface to Sentry.
Register a shared function. Overwrite previously registered function with the same name, if there was any such.
This is a helper method for easier testability. It is not a part of public API of this class.
Parameters: - funcname (str) – Name of the shared function.
- module (gluetool.glue.Module) – Module instance providing the shared function.
- func (callable) – Shared function.
-
_check_module_file
(mfile)[source]¶ Make sure the file looks like a
gluetool
module:- can be processed by Python parser,
- imports
gluetool.glue.Glue
andgluetool.glue.Module
, - contains child class of
gluetool.glue.Module
.
Parameters: mfile (str) – path to a file. Returns: True
if file containsgluetool
module,False
otherwise.Raises: gluetool.glue.GlueError – when it’s not possible to finish the check.
-
_eval_context
()[source]¶ Gather contexts of all modules in a pipeline and merge them together.
Always returns a unique dictionary object, therefore it is safe for caller to update it. The return value is not cached in any way, therefore the change if its content won’t affect future callers.
Provided as a shared function, registered by the Glue instance itself.
Return type: dict
-
_eval_context_module_caller
()[source]¶ Infere module instance calling the eval context shared function.
Return type: gluetool.glue.Module
-
_import_module
(import_name, filename)[source]¶ Attempt to import a Python module from a file.
Parameters: Returns: imported Python module.
Raises: gluetool.glue.GlueError – when import failed.
-
_load_gluetool_modules
(group, module_name, filepath)[source]¶ Load
gluetool
modules from a file. Method attempts to import the file as a Python module, and then checks its content and adds all gluetool modules to internal module registry.Parameters: Return type: [(module_group, module_class), ..]
Returns: list of loaded
gluetool
modules
-
_load_module_path
(ppath)[source]¶ Search and load
gluetool
modules from a directory.In essence, it scans every file with
.py
suffix, and searches for classes derived fromgluetool.glue.Module
.Parameters: ppath (str) – directory to search for gluetool modules.
-
_load_python_module
(group, module_name, filepath)[source]¶ Load Python module from a file, if it contains
gluetool
modules. If the file does not look like it containsgluetool
modules, or when it’s not possible to import the Python module successfully, method simply warns user and ignores the file.Parameters: Returns: loaded Python module.
Raises: gluetool.glue.GlueError – when import failed.
Register a shared function. Overwrite previously registered function with the same name, if there was any such.
Parameters: - funcname (str) – Name of the shared function.
- module (gluetool.glue.Module) – Module instance providing the shared function.
-
dryrun_level
¶
-
eval_context
¶ Returns “global” evaluation context - some variables that are nice to have in all contexts.
-
init_module
(module_name, actual_module_name=None)[source]¶ Given a name of the module, create its instance and give it a name.
Parameters: - module_name (str) – Name under which will be the module instance known.
- actual_module_name (str) – Name of the module to instantiate. It does not have to match
module_name
-actual_module_name
refers to the list of knowngluetool
modules whilemodule_name
is basically an arbitrary name new instance calls itself. If it’s not set, which is the most common situation, it defaults tomodule_name
.
Returns: A
Module
instance.
-
module_config_paths
¶ List of paths in which module config files reside.
-
module_data_paths
¶ List of paths in which module data files reside.
-
module_paths
¶ List of paths in which modules reside.
-
name
= 'gluetool core'¶
-
options
= [('Global options', {('V', 'version'): {'action': 'store_true', 'help': 'Print version'}, ('E', 'list-eval-context'): {'action': 'store_true', 'default': False, 'help': 'List all available variables provided by modules in their evaluation contexts.'}, ('L', 'list-shared'): {'action': 'store_true', 'default': False, 'help': 'List all available shared functions.'}, ('l', 'list-modules'): {'action': 'append', 'const': True, 'nargs': '?', 'help': 'List all available modules. If a GROUP is set, limits list to the given module group.', 'metavar': 'GROUP'}, 'no-sentry-exceptions': {'action': 'append', 'default': [], 'help': 'List of exception names, which are not reported to Sentry (Default: none)'}, ('r', 'retries'): {'default': 0, 'type': <type 'int'>, 'help': 'Number of retries'}}), ('Output control', {('j', 'json-file'): {'default': None, 'help': '\n If set, all log messages (including ``VERBOSE``) are stored in this file\n in a form of JSON structures (default: %(default)s).\n '}, 'verbose-file': {'default': None, 'help': 'Log messages with ``VERBOSE`` level sent to this file.'}, ('v', 'verbose'): {'action': 'store_true', 'help': 'Log **all** messages to the terminal (WARNING: even more verbose than ``-d``!).'}, ('p', 'pid'): {'action': 'store_true', 'help': 'Log PID of gluetool process'}, ('c', 'colors'): {'action': 'store_true', 'help': 'Colorize logging on the terminal'}, 'show-traceback': {'action': 'store_true', 'default': False, 'help': '\n Display exception tracebacks on terminal (besides the debug file when ``--debug-file``\n is used) (default: %(default)s).\n '}, ('q', 'quiet'): {'action': 'store_true', 'help': 'Silence info messages'}, ('i', 'info'): {'action': 'store_true', 'help': 'Print command-line that would re-run the gluetool session'}, ('o', 'debug-file', 'output'): {'help': 'Log messages with at least ``DEBUG`` level are sent to this file.'}, ('d', 'debug'): {'action': 'store_true', 'help': 'Log debugging messages to the terminal (WARNING: very verbose!).'}}), ('Directories', {'module-path': {'action': 'append', 'default': [], 'help': 'Specify directory with modules.', 'metavar': 'DIR'}, 'module-config-path': {'action': 'append', 'default': [], 'help': 'Specify directory with module configuration files.', 'metavar': 'DIR'}, 'module-data-path': {'action': 'append', 'default': [], 'help': 'Specify directory with module data files.', 'metavar': 'DIR'}}), ('Dry run options', {'isolated-run': {'action': 'store_true', 'default': False, 'help': 'Modules that support this option will not interact with the outside world.'}, 'dry-run': {'action': 'store_true', 'default': False, 'help': 'Modules that support this option will make no changes to the outside world.'}}), {'pipeline': {'raw': True, 'nargs': '...', 'help': 'List of modules and their options, passed after gluetool options.'}}]¶
-
run_module
(module_name, module_argv=None, actual_module_name=None, register=False)[source]¶ Syntax sugar for
run_modules()
, in the case you want to run just a one-shot module.Parameters: - module_name (str) – Name under which will be the module instance known.
- module_argv (list(str)) – Arguments of the module.
- actual_module_name (str) – Name of the module to instantiate. It does not have to match
module_name
-actual_module_name
refers to the list of knowngluetool
modules whilemodule_name
is basically an arbitrary name new instance calls itself. If it’s not set, which is the most common situation, it defaults tomodule_name
. - register (bool) – If
True
, module instance is added to a list of modules in thisGlue
instance, and it will be collected whendestroy_modules()
gets called.
-
run_modules
(pipeline_desc, register=False)[source]¶ Run a pipeline, consisting of multiple modules.
Parameters: - pipeline_desc (list(PipelineStep)) – List of pipeline steps.
- register (bool) – If
True
, module instance is added to a list of modules in thisGlue
instance, and it will be collected whendestroy_modules()
gets called.
-
sentry_submit_exception
(*args, **kwargs)[source]¶ Submits exceptions to the Sentry server. Does nothing by default, unless this instance is initialized with a
gluetool.sentry.Sentry
instance which actually does the job.
-
sentry_submit_warning
(*args, **kwargs)[source]¶ Submits warnings to the Sentry server. Does nothing by default, unless this instance is initialized with a
gluetool.sentry.Sentry
instance which actually does the job.
- tool (gluetool.tool.Tool) – If set, it’s an
-
class
gluetool.
Module
(glue, name)[source]¶ Bases:
gluetool.glue.Configurable
Base class of all
gluetool
modules.Parameters: glue (gluetool.glue.Glue) –
Glue
instance owning the module.Variables: - glue (gluetool.glue.Glue) –
Glue
instance owning the module. - _config (dict) – internal configuration store. Values of all module options are stored here, regardless of them being set on command-line or in the configuration file.
- _overloaded_shared_functions (dict) – If a shared function added by this module
overloades an older function of the same name, registered by a previous module,
the overloaded one is added into this dictionary. The module can then call this
saved function - using
overloaded_shared()
- to implement a “chain” of shared functions, when one calls another, implementing the same operation.
Generate help for shared functions provided by the module.
Returns: Formatted help, describing module’s shared functions.
-
_paths_with_module
(roots)[source]¶ Return paths cretaed by joining roots with module’s unique name.
Parameters: roots (list(str)) – List of root directories.
Register module’s shared functions with Glue, to allow other modules to use them.
-
description
= None¶
-
destroy
(failure=None)[source]¶ Here should go any code that needs to be run on exit, like job cleanup etc.
Parameters: failure (gluetool.glue.Failure) – if set, carries information about failure that made gluetool
to destroy the whole session. Modules might want to take actions based on provided information, e.g. send different notifications.
-
dryrun_level
¶
-
execute
()[source]¶ In this method, modules can perform any work they deemed necessary for completing their purpose. E.g. if the module promises to run some tests, this is the place where the code belongs to.
By default, this method does nothing. Reimplement as needed.
-
name
= None¶
Call a shared function overloaded by the one provided by this module. This way, a module can give chance to other implementations of its action, e.g. to publish messages on a different message bus.
-
sanity
()[source]¶ In this method, modules can define additional checks before execution.
Some examples:
- Advanced checks on passed options
- Check for additional requirements (tools, data, etc.)
By default, this method does nothing. Reimplement as needed.
- glue (gluetool.glue.Glue) –
-
exception
gluetool.
GlueError
(message, caused_by=None, **kwargs)[source]¶ Bases:
exceptions.Exception
Generic
gluetool
exception.Parameters: - message (str) – Exception message, describing what happened.
- caused_by (tuple) – If set, contains tuple as returned by
sys.exc_info()
, describing the exception that caused this one to be born. If not set, constructor will try to auto-detect this information, and if there’s no such information, instance propertycaused_by
will be set toNone
.
Variables: - message (str) – Exception message, describing what happened.
- caused_by (tuple) – If set, contains tuple as returned by
sys.exc_info()
, describing the exception that caused this one to be born.None
otherwise.
-
no_sentry_exceptions
= []¶
-
sentry_fingerprint
(current)[source]¶ Default grouping of events into issues might be too general for some cases. This method gives users a chance to provide custom fingerprint Sentry could use to group events in a more suitable way.
E.g. user might be interested in some sort of connection issues but they would like to have them grouped not by a traceback (which is the default method) but per remote host IP. For that, the Sentry integration code will call
sentry_fingerprint
method of a raised exception, and the method should return new fingerprint, let’s say[<exception class name>, <remote IP>]
, and Sentry will group events using this fingerprint.Parameters: current (list(str)) – current fingerprint. Usually ['{{ default }}']
telling Sentry to use its default method, but it could already be more specific.Return type: list(str) Returns: new fingerprint, e.g. ['FailedToConnectToAPI', '10.20.30.40']
Add, modify or remove tags attached to a Sentry event, reported when the exception was raised.
Most common usage would be an addition of tags, e.g.
remote-host
to allow search for events related to the same remote address.Parameters: str) current (dict(str,) – current set of tags and their values. Return type: dict(str, str) Returns: new set of tags. It is possible to add tags directly into current
and then return it.
-
exception
gluetool.
SoftGlueError
(message, caused_by=None, **kwargs)[source]¶ Bases:
gluetool.glue.GlueError
Soft errors are errors Glue Ops and/or developers shouldn’t be bothered with, things that are up to the user to fix, e.g. empty set of tests. Hard errors are supposed to warn Ops/Devel teams about important infrastructure issues, code deficiencies, bugs and other issues that are fixable only by actions of Glue staff.
However, we still must provide notification to user(s), and since we expect them to fix the issues that led to raising the soft error, we must provide them with as much information as possible. Therefore modules dealing with notifications are expected to give these exceptions a chance to influence the outgoing messages, e.g. by letting them provide an e-mail body template.
-
exception
gluetool.
GlueRetryError
(message, caused_by=None, **kwargs)[source]¶ Bases:
gluetool.glue.GlueError
Retry gluetool exception
-
exception
gluetool.
GlueCommandError
(cmd, output, **kwargs)[source]¶ Bases:
gluetool.glue.GlueError
Exception raised when external command failes.
Parameters: - cmd (list) – Command as passed to gluetool.utils.run_command helper.
- output (gluetool.utils.ProcessOutput) – Process output data.
Variables: - cmd (list) – Command as passed to gluetool.utils.run_command helper.
- output (gluetool.utils.ProcessOutput) – Process output data.
-
class
gluetool.
Failure
(module, exc_info)[source]¶ Bases:
object
Bundles exception related info. Used to inform modules in their
destroy()
phase thatgluetool
session was killed because of exception raised by one of modules.Parameters: - module (gluetool.glue.Module) – module in which the error happened, or
None
. - exc_info (tuple) – Exception information as returned by
sys.exc_info()
.
Variables: - module (gluetool.glue.Module) – module in which the error happened, or
None
. - exception (Exception) – Shortcut to
exc_info[1]
, if available, orNone
. - exc_info (tuple) – Exception information as returned by
sys.exc_info()
. - sentry_event_id (str) – If set, the failure was reported to the Sentry under this ID.
- module (gluetool.glue.Module) – module in which the error happened, or