gluetool.utils module

Various helpers.

class gluetool.utils.Bunch(*args, **kwargs)

Bases: object

class gluetool.utils.Command(executable, options=None, logger=None)

Bases: gluetool.log.LoggerMixin, object

Wrap an external command, its options and other information, necessary for running the command.

The main purpose is to gather all relevant pieces into a single space, call subprocess.Popen, and log everything.

By default, both standard output and error output are of the process are captured and returned back to caller. Under some conditions, caller might want to see the output in “real-time”. For that purpose, they can pass callable via inspect_callback parameter - such callable will be called for every received bit of input on both standard and error outputs. E.g.

def foo(stream, s, flush=False):
    if s is not None and 'a' in s:
        print s

Command(['/bin/foo']).run(inspect=foo)

This example will print all substrings containing letter a. Strings passed to foo may be of arbitrary lengths, and may change between subsequent use of Command class.

Parameters:

• executable (list) – Executable to run. Feel free to use the whole command, including its options, if you have no intention to modify them before running the command.
• options (list) – If set, it’s a list of options to pass to the executable. Options are specified in a separate list to allow modifications of executable and options before actually running the command.
• logger (gluetool.log.ContextAdapter) – Parent logger whose methods will be used for logging.

New in version 1.1.

_apply_quotes()

Return options to pass to Popen. Applies quotes as necessary.

_communicate_batch()

_communicate_inspect(inspect_callback)

_construct_output()

run(inspect=False, inspect_callback=None, **kwargs)

Run the command, wait for it to finish and return the output.

Parameters:

• inspect (bool) – If set, inspect_callback will receive the output of command in “real-time”.
• inspect_callback (callable) – callable that will receive command output. If not set, default “write to sys.stdout” is used.

Return type:

gluetool.utils.ProcessOutput instance

Returns:

gluetool.utils.ProcessOutput instance whose attributes contain data returned by the child process.

Raises:

• gluetool.glue.GlueError – When somethign went wrong.
• gluetool.glue.GlueCommandError – When command exited with non-zero exit code.

exception gluetool.utils.IncompatibleOptionsError(message, caused_by=None, sentry_fingerprint=None, sentry_tags=None, **kwargs)

Bases: gluetool.glue.SoftGlueError

class gluetool.utils.PatternMap(filepath, spices=None, logger=None, allow_variables=False)

Bases: gluetool.log.LoggerMixin, object

Pattern map is a list of <pattern>: <converter> pairs. Pattern is a regular expression used to match a string, converter is a function that transforms a string into another one, accepting the pattern and the string as arguments.

It is defined in a YAML file:

---
- 'foo-(\d+)': 'bar-\1'
- 'baz-(\d+)': 'baz, find_the_most_recent, append_dot'
- 'bar-(\d+)':
  - 'bar, find_the_most_recent, append_dot'
  - 'bar, find_the_oldest, append_dot'

Patterns are the keys in each pair, while converter is a string (or list of strings), consisting of multiple items, separated by comma. The first item is always a string, let’s call it R. R, given input string S1 and the pattern, is used to transform S1 to a new string, S2, by calling pattern.sub(R, S1). R can make use of anything re.sub() supports, including capturing groups.

If there are other items in the converter string, they are names of spices, additional functions that will be called with pattern and the output of the previous spicing function, starting with S2 in the case of the first spice.

To allow spicing, user of PatternMap class must provide spice makers - mapping between spice names and functions that generate spicing functions. E.g.:

def create_spice_append_dot(previous_spice):
    def _spice(pattern, s):
        s = previous_spice(pattern, s)
        return s + '.'
    return _spice

create_spice_append_dot is a spice maker, used during creation of a pattern map after its definition is read, _spice is the actual spicing function used during the transformation process.

There can be multiple converters for a single pattern, resulting in multiple values returned when the input string matches the corresponding pattern.

Parameters:

• filepath (text) – Path to a YAML file with map definition.
• spices (dict) – apping between spices and their makers.
• logger (gluetool.log.ContextLogger) – Logger used for logging.
• allow_variables (bool) – if set, both patterns and converters are first treated as templates, and as such are rendered before doing anything else. Map may contain special comments, # !include <path>, where path refers to a YAML file providing the necessary variables.

match(s, multiple=False)

Try to match s by the map. If the match is found - the first one wins - then its conversions are applied to the s.

There can be multiple conversions for a pattern, by default only the product of the first one is returned. If multiple is set to True, list of all products is returned instead.

Return type:text Returns:if matched, output of the corresponding transformation.

class gluetool.utils.ProcessOutput(cmd, exit_code, stdout, stderr, kwargs)

Bases: object

Result of external process.

log(logger)

log_stream(stream, logger)

class gluetool.utils.SimplePatternMap(filepath, logger=None, allow_variables=False)

Bases: gluetool.log.LoggerMixin, object

Pattern map is a list of <pattern>: <result> pairs. Pattern is a regular expression used to match a string, result is what the matching string maps to.

Basically an ordered dictionary with regexp matching of keys, backed by an YAML file.

Parameters:

• filepath (str) – Path to a YAML file with map definition.
• logger (gluetool.log.ContextLogger) – Logger used for logging.
• allow_variables (bool) – if set, both patterns and converters are first treated as templates, and as such are rendered before doing anything else. Map may contain special comments, # !include <path>, where path refers to a YAML file providing the necessary variables.

match(s)

Try to match s by the map. If the match is found - the first one wins - then its transformation is applied to the s.

Return type:text Returns:if matched, output of the corresponding transformation.

class gluetool.utils.StreamReader(stream, name=None, block=16)

Bases: object

content

name

read()

wait()

class gluetool.utils.ThreadAdapter(logger, thread)

Bases: gluetool.log.ContextAdapter

Custom logger adapter, adding thread name as a context.

Parameters:

• logger (gluetool.log.ContextAdapter) – parent logger whose methods will be used for logging.
• thread (threading.Thread) – thread whose name will be added.

gluetool.utils.WaitCheckType

Generic type for callbacks used by wait() function. Accepts no arguments, returns an instance of gluetool.result.Result.

alias of Callable

class gluetool.utils.WorkerThread(logger, fn, fn_args=None, fn_kwargs=None, **kwargs)

Bases: gluetool.log.LoggerMixin, threading.Thread

Worker threads gets a job to do, and returns a result. It gets a callable, fn, which will be called in thread’s run() method, and thread’s result property will be the result - value returned by fn, or exception raised during the runtime of fn.

Parameters:

• logger (gluetool.log.ContextAdapter) – logger to use for logging.
• fn – thread will start fn to do the job.
• fn_args – arguments for fn
• fn_kwargs – keyword arguments for fn

run()

gluetool.utils.YAML()

Provides YAML read/write interface with common settings.

Return type:ruamel.yaml.YAML

gluetool.utils._json_byteify(data, ignore_dicts=False)

gluetool.utils._load_yaml_variables(data, enabled=True, logger=None)

Load all variables from files referenced by a YAML, and return function to render a string as a template using these variables. The files containing variables are mentioned in comments, in a form # !include <filepath> form.

Parameters:

• data – data loaded from a YAML file.
• enabled (bool) – when set to False, variables are not loaded and a simple no-op function is returned.
• logger (gluetool.log.ContextLogger) – Logger used for logging.

Returns:

Function accepting a string and returning a rendered template.

class gluetool.utils.cached_property(method)

Bases: object

property-like decorator - at first access, it calls decorated method to acquire the real value, and then replaces itself with this value, making it effectively “cached”. Useful for properties whose value does not change over time, and where getting the real value could penalize execution with unnecessary (network, memory) overhead.

Delete attribute to clear the cached value - on next access, decorated method will be called again, to acquire the real value.

Of possible options, only read-only instance attribute access is supported so far.

gluetool.utils.check_for_commands(cmds)

Checks if all commands in list cmds are valid

gluetool.utils.deprecated(func)

This is a decorator which can be used to mark functions as deprecated. It will result in a warning being emitted when the function is used.

gluetool.utils.dict_update(dst, *args)

Python’s dict.update does not return the dictionary just updated but a None. This function is a helper that does updates the dictionary and returns it. So, instead of:

d.update(other)
return d

you can use:

return dict_update(d, other)

Parameters:

• dst (dict) – dictionary to be updated.
• args – dictionaries to update dst with.

gluetool.utils.dump_yaml(data, filepath, logger=None)

Save data stored in variable to YAML file.

Parameters:

• data (object) – Data to store in YAML file
• filepath (text) – Path to an output file.

Raises:

gluetool.glue.GlueError – if it was not possible to successfully save data to file.

gluetool.utils.fetch_url(*args, **kwargs)

“Get me content of this URL” helper.

Very thin wrapper around urllib. Added value is logging, and converting possible errors to gluetool.glue.GlueError exception.

Parameters:

• url – URL to get.
• logger (gluetool.log.ContextLogger) – Logger used for logging.
• success_codes (tuple) – tuple of HTTP response codes representing successfull request.

Returns:

tuple (response, content) where response is what requests.get() returns, and content is the payload of the response.

gluetool.utils.format_command_line(cmdline)

Return formatted command-line.

All but the first line are indented by 4 spaces.

Parameters:cmdline (list) – list of iterables, representing command-line split to multiple lines.

gluetool.utils.from_json(json_string)

Convert JSON in a string into Python data structures.

Similar to json.loads() but uses special object hook to avoid unicode strings in the output..

gluetool.utils.from_yaml(yaml_string)

Convert YAML in a string into Python data structures.

Uses internal YAML parser to produce result. Paired with load_yaml() and their JSON siblings to provide unified access to JSON and YAML.

gluetool.utils.load_json(filepath, logger=None)

Load data stored in JSON file, and return their Python representation.

Parameters:

• filepath (text) – Path to a file. ~ or ~<username> are expanded before using.
• logger (gluetool.log.ContextLogger) – Logger used for logging.

Return type:

object

Returns:

structures representing data in the file.

Raises:

gluetool.glue.GlueError – if it was not possible to successfully load content of the file.

gluetool.utils.load_yaml(filepath, logger=None)

Load data stored in YAML file, and return their Python representation.

Parameters:

• filepath (text) – Path to a file. ~ or ~<username> are expanded before using.
• logger (gluetool.log.ContextLogger) – Logger used for logging.

Return type:

object

Returns:

structures representing data in the file.

Raises:

gluetool.glue.GlueError – if it was not possible to successfully load content of the file.

gluetool.utils.new_xml_element(tag_name, _parent=None, **attrs)

Create new XML element.

Parameters:

• tag_name (text) – Name of the element.
• _parent (element) – If set, the newly created element will be appended to this element.
• attrs (dict) – Attributes to set on the newly created element.

Returns:

Newly created XML element.

gluetool.utils.normalize_bool_option(option_value)

Convert option value to Python’s boolean.

option_value is what all those internal option processing return, which may be a default value set for an option, or what user passed in.

As switches, options with values can be used:

--foo=yes|no
--foo=true|false
--foo=1|0
--foo=Y|N
--foo=on|off

With combination of store_true/store_false and a default value module developer sets for the option, simple form without value is evaluated as easily. With store_true and False default, following option turn the feature foo on:

--enable-foo

With store_false and True default, following simple option turn the feature foo off:

--disable-foo

gluetool.utils.normalize_multistring_option(option_value, separator=', ')

Reduce string, representing comma-separated list of items, or possibly a list of such strings, to a simple list of items. Strips away the whitespace wrapping such items.

foo --option value1 --option value2, value3
foo --option value1,value2,value3

Or, when option is set by a config file:

option = value1
option = value1, value2,                  value3

After processing, different variants can be found when option('option') is called, ['value1', 'value2,value3'], ['value1,value2,value3'], 'value1' and value1, value2, value3.

To reduce the necessary work, use this helper function to treat such option’s value, and get simple ['value1', 'value2', 'value3'] structure.

gluetool.utils.normalize_path(path)

Apply common treatments on a given path:

• replace home directory reference (~ and similar), and
• convert path to a normalized absolutized version of the pathname.

gluetool.utils.normalize_path_option(option_value, separator=', ')

Reduce many ways how list of paths is specified by user, to a simple list of paths. See normalize_multistring_option() for more details.

gluetool.utils.normalize_shell_option(option_value)

Reduce string, using a shell-like syntax, or possibly a list of such strings, to a simple list of items. Strips away the whitespace wrapping such items.

foo --option value1 --option value2\ value3 --option "value4 value5"

Or, when option is set by a config file:

option = value1 value2\ value3 "value4 value5"

After processing, different variants can be found when option('option') is called, ['value1', 'value2,value3'], ['value1,value2,value3'], 'value1' and value1, value2, value3.

To reduce the necessary work, use this helper function to treat such option’s value, and get simple ['value1', 'value2 value3', 'value4 value5'] structure.

gluetool.utils.render_template(template, logger=None, **kwargs)

Render Jinja2 template. Logs errors, and raises an exception when it’s not possible to correctly render the template.

Parameters:

• template – Template to render. It can be either jinja2.environment.Template instance, or a string.
• kwargs (dict) – Keyword arguments passed to render process.

Returns:

Rendered template.

Raises:

gluetool.glue.GlueError – when the rednering failed.

gluetool.utils.requests(*args, **kwds)

Wrap requests with few layers providing us with the logging and better insight into what has been happening when requests did their job.

Used as a context manager, yields a patched requests module. As long as inside the context, detailed information about HTTP traffic are logged via given logger.

Note

The original requests library is returned, with slight modifications for better integration with gluetool logging facilities. Each and every requests API feature is available and , hopefully, enhancements applied by this wrapper wouldn’t interact with requests functionality.

with gluetool.utils.requests() as R:
    R.get(...).json()
    ...

    r = R.post(...)
    assert r.code == 404
    ...

Parameters:logger – used for logging. Returns:requests module.

gluetool.utils.run_command(*args, **kwargs)

Wrapper for ``Command(...).run().

Provided for backward compatibility.

Deprecated since version 1.1: Use gluetool.utils.Command instead.

gluetool.utils.treat_url(url, logger=None)

Remove “weird” artifacts from the given URL. Collapse adjacent ‘.’s, apply ‘..’, etc.

Parameters:

• url (text) – URL to clear.
• logger (gluetool.log.ContextAdapter) – logger to use for logging.

Return type:

text

Raises:

gluetool.glue.GlueError: if URL is invalid

Returns:

Treated URL.

gluetool.utils.wait(label, check, timeout=None, tick=30, logger=None)

Wait for a condition to be true.

Parameters:

• label (text) – printable label used for logging.
• check (callable) – called to test the condition. It must be of type WaitCheckType: takes no arguments, must return instance of gluetool.Result. If the result is valid, the condition is assumed to pass the check and waiting ends.
• timeout (int) – fail after this many seconds. None means test forever.
• tick (int) – test condition every tick seconds.
• logger (gluetool.log.ContextAdapter) – parent logger whose methods will be used for logging.

Raises:

gluetool.glue.GlueError – when timeout elapses while condition did not pass the check.

Returns:

if the condition became true, the value returned by the check function is returned. It is unpacked from the Result returned by check.