gluetool.utils module

Various helpers.

class gluetool.utils.Bunch(**kwargs)

Bases: object

exception gluetool.utils.IncompatibleOptionsError(message, caused_by=None, **kwargs)

Bases: gluetool.glue.SoftGlueError

class gluetool.utils.PatternMap(filepath, spices=None, logger=None)

Bases: object

Pattern map is a list of <pattern>: <converter> pairs. Pattern is a regular expression used to match a string, converter is a function that transforms a string into another one, accepting the pattern and the string as arguments.

It is defined in a YAML file:

---
- 'foo-(\d+)': 'bar-\1'
- 'baz-(\d+)': 'baz, find_the_most_recent, append_dot'
- 'bar-(\d+)':
  - 'bar, find_the_most_recent, append_dot'
  - 'bar, find_the_oldest, append_dot'

Patterns are the keys in each pair, while converter is a string (or list of strings), consisting of multiple items, separated by comma. The first item is always a string, let’s call it R. R, given input string S1 and the pattern, is used to transform S1 to a new string, S2, by calling pattern.sub(R, S1). R can make use of anything re.sub() supports, including capturing groups.

If there are other items in the converter string, they are names of spices, additional functions that will be called with pattern and the output of the previous spicing function, starting with S2 in the case of the first spice.

To allow spicing, user of PatternMap class must provide spice makers - mapping between spice names and functions that generate spicing functions. E.g.:

def create_spice_append_dot(previous_spice):
    def _spice(pattern, s):
        s = previous_spice(pattern, s)
        return s + '.'
    return _spice

create_spice_append_dot is a spice maker, used during creation of a pattern map after its definition is read, _spice is the actual spicing function used during the transformation process.

There can be multiple converters for a single pattern, resulting in multiple values returned when the input string matches the corresponding pattern.

Parameters:

• filepath (str) – Path to a YAML file with map definition.
• spices (dict) – apping between spices and their makers.
• logger (gluetool.log.ContextLogger) – Logger used for logging.

match(s, multiple=False)

Try to match s by the map. If the match is found - the first one wins - then its conversions are applied to the s.

There can be multiple conversions for a pattern, by default only the product of the first one is returned. If multiple is set to True, list of all products is returned instead.

Return type:str Returns:if matched, output of the corresponding transformation.

class gluetool.utils.ProcessOutput(cmd, exit_code, stdout, stderr, kwargs)

Bases: object

Result of external process.

log(logger)

log_stream(stream, logger)

class gluetool.utils.SimplePatternMap(filepath, logger=None)

Bases: object

Pattern map is a list of <pattern>: <result> pairs. Pattern is a regular expression used to match a string, result is what the matching string maps to.

Basically an ordered dictionary with regexp matching of keys, backed by an YAML file.

Parameters:

• filepath (str) – Path to a YAML file with map definition.
• logger (gluetool.log.ContextLogger) – Logger used for logging.

match(s)

Try to match s by the map. If the match is found - the first one wins - then its transformation is applied to the s.

Return type:str Returns:if matched, output of the corresponding transformation.

class gluetool.utils.StreamReader(stream, name=None, block=16)

Bases: object

content

name

read()

wait()

class gluetool.utils.ThreadAdapter(logger, thread)

Bases: gluetool.log.ContextAdapter

Custom logger adapter, adding thread name as a context.

Parameters:

• logger (gluetool.log.ContextAdapter) – parent logger whose methods will be used for logging.
• thread (threading.Thread) – thread whose name will be added.

class gluetool.utils.WorkerThread(logger, fn, fn_args=None, fn_kwargs=None, **kwargs)

Bases: threading.Thread

Worker threads gets a job to do, and returns a result. It gets a callable, fn, which will be called in thread’s run() method, and thread’s result property will be the result - value returned by fn, or exception raised during the runtime of fn.

Parameters:

• logger (gluetool.log.ContextAdapter) – logger to use for logging.
• fn – thread will start fn to do the job.
• fn_args – arguments for fn
• fn_kwargs – keyword arguments for fn

run()

gluetool.utils._json_byteify(data, ignore_dicts=False)

class gluetool.utils.cached_property(method)

Bases: object

property-like decorator - at first access, it calls decorated method to acquire the real value, and then replaces itself with this value, making it effectively “cached”. Useful for properties whose value does not change over time, and where getting the real value could penalize execution with unnecessary (network, memory) overhead.

Delete attribute to clear the cached value - on next access, decorated method will be called again, to acquire the real value.

Of possible options, only read-only instance attribute access is supported so far.

gluetool.utils.check_for_commands(cmds)

Checks if all commands in list cmds are valid

gluetool.utils.dict_update(dst, *args)

Python’s dict.update does not return the dictionary just updated but a None. This function is a helper that does updates the dictionary and returns it. So, instead of:

d.update(other)
return d

you can use:

return dict_update(d, other)

Parameters:

• dst (dict) – dictionary to be updated.
• args – dictionaries to update dst with.

gluetool.utils.dump_yaml(data, filepath, logger=None)

Save data stored in variable to YAML file.

Parameters:

• data (object) – Data to store in YAML file
• filepath (str) – Path to an output file.

Raises:

gluetool.glue.GlueError – if it was not possible to successfully save data to file.

gluetool.utils.fetch_url(url, logger=None, success_codes=(200, ))

“Get me content of this URL” helper.

Very thin wrapper around urllib. Added value is logging, and converting possible errors to gluetool.glue.GlueError exception.

Parameters:

• url (str) – URL to get.
• logger (gluetool.log.ContextLogger) – Logger used for logging.
• success_codes (tuple) – tuple of HTTP response codes representing successfull request.

Returns:

tuple (response, content) where response is what urllib2.urlopen() returns, and content is the payload of the response.

gluetool.utils.format_command_line(cmdline)

Return formatted command-line.

All but the first line are indented by 4 spaces.

Parameters:cmdline (list) – list of iterables, representing command-line split to multiple lines.

gluetool.utils.from_json(json_string)

Convert JSON in a string into Python data structures.

Similar to json.loads() but uses special object hook to avoid unicode strings in the output..

gluetool.utils.from_yaml(yaml_string)

Convert YAML in a string into Python data structures.

Uses internal YAML parser to produce result. Paired with load_yaml() and their JSON siblings to provide unified access to JSON and YAML.

gluetool.utils.load_json(filepath, logger=None)

Load data stored in JSON file, and return their Python representation.

Parameters:

• filepath (str) – Path to a file. ~ or ~<username> are expanded before using.
• logger (gluetool.log.ContextLogger) – Logger used for logging.

Return type:

object

Returns:

structures representing data in the file.

Raises:

gluetool.glue.GlueError – if it was not possible to successfully load content of the file.

gluetool.utils.load_yaml(filepath, logger=None)

Load data stored in YAML file, and return their Python representation.

Parameters:

• filepath (str) – Path to a file. ~ or ~<username> are expanded before using.
• logger (gluetool.log.ContextLogger) – Logger used for logging.

Return type:

object

Returns:

structures representing data in the file.

Raises:

gluetool.glue.GlueError – if it was not possible to successfully load content of the file.

gluetool.utils.new_xml_element(tag_name, _parent=None, **attrs)

Create new XML element.

Parameters:

• tag_name (str) – Name of the element.
• _parent (element) – If set, the newly created element will be appended to this element.
• attrs (dict) – Attributes to set on the newly created element.

Returns:

Newly created XML element.

gluetool.utils.normalize_bool_option(option_value)

Convert option value to Python’s boolean.

option_value is what all those internal option processing return, which may be a default value set for an option, or what user passed in.

As switches, options with values can be used:

--foo=yes|no
--foo=true|false
--foo=1|0
--foo=Y|N
--foo=on|off

With combination of store_true/store_false and a default value module developer sets for the option, simple form without value is evaluated as easily. With store_true and False default, following option turn the feature foo on:

--enable-foo

With store_false and True default, following simple option turn the feature foo off:

--disable-foo

gluetool.utils.normalize_multistring_option(option_value, separator=', ')

Reduce string, representing comma-separated list of items, or possibly a list of such strings, to a simple list of items. Strips away the whitespace wrapping such items.

foo --option value1 --option value2, value3
foo --option value1,value2,value3

Or, when option is set by a config file:

option = value1
option = value1, value2,                  value3

After processing, different variants can be found when option('option') is called, ['value1', 'value2,value3'], ['value1,value2,value3'], 'value1' and value1, value2, value3.

To reduce the necessary work, use this helper function to treat such option’s value, and get simple ['value1', 'value2', 'value3'] structure.

gluetool.utils.normalize_path(path)

Apply common treatments on a given path:

• replace home directory reference (~ and similar), and
• convert path to a normalized absolutized version of the pathname.

gluetool.utils.normalize_path_option(option_value, separator=', ')

Reduce many ways how list of paths is specified by user, to a simple list of paths. See normalize_multistring_option() for more details.

gluetool.utils.render_template(template, logger=None, **kwargs)

Render Jinja2 template. Logs errors, and raises an exception when it’s not possible to correctly render the template.

Parameters:

• template – Template to render. It can be either jinja2.environment.Template instance, or a string.
• kwargs (dict) – Keyword arguments passed to render process.

Return type:

str

Returns:

Rendered template.

Raises:

gluetool.glue.GlueError – when the rednering failed.

gluetool.utils.run_command(cmd, logger=None, inspect=False, inspect_callback=None, **kwargs)

Run external command, and return it’s exit code and output.

This is a very thin and simple wrapper above subprocess.Popen, and its main purpose is to log everything that happens before and after execution. All additional arguments are passed directly to Popen constructor.

If stdout or stderr keyword arguments are not specified, function will set them to subprocess.PIPE, to capture both output streams in separate strings.

By default, output of the process is captured for both stdout and stderr, and returned back to the caller. Under some conditions, caller might want to see the output in “real-time”. For that purpose, it can pass callable via inspect_callback parameter - such callable will be called for every received bit of input on both stdout and stderr. E.g.

def foo(stream, s, flush=False):
  if s is not None and 'a' in s:
    print s

run_command(['/bin/foo'], inspect=foo)

This example will print all substrings containing letter a. Strings passed to foo may be of arbitrary lengths, and may change between subsequent calls of run_command.

Parameters:

• cmd (list) – command to execute.
• logger (gluetool.log.ContextAdapter) – parent logger whose methods will be used for logging.
• inspect (bool) – if set, inspect_callback will receive the output of command in “real-time”.
• inspect_callback (callable) – callable that will receive command output. If not set, default “write to sys.stdout” is used.

Return type:

gluetool.utils.ProcessOutput instance

Returns:

gluetool.utils.ProcessOutput instance whose attributes contain data returned by the process.

Raises:

• gluetool.glue.GlueError – when command was not found.
• gluetool.glue.GlueCommandError – when command exited with non-zero exit code.
• Exception – when anything else breaks.

gluetool.utils.treat_url(url, logger=None)

Remove “weird” artifacts from the given URL. Collapse adjacent ‘.’s, apply ‘..’, etc.

Parameters:

• url (str) – URL to clear.
• logger (gluetool.log.ContextAdapter) – logger to use for logging.

Return type:

str

Returns:

Treated URL.

gluetool.utils.wait(label, check, timeout=None, tick=30, logger=None)

Wait for a condition to be true.

Parameters:

• label (str) – printable label used for logging.
• check (callable) – called to test the condition. If its return value evaluates as True, the condition is assumed to pass the test and waiting ends.
• timeout (int) – fail after this many seconds. None means test forever.
• tick (int) – test condition every tick seconds.
• logger (gluetool.log.ContextAdapter) – parent logger whose methods will be used for logging.

Raises:

gluetool.glue.GlueError – when timeout elapses while condition did not pass the check.