gluetool.log module

Logging support.

Sets up logging environment for use by gluetool and modules. Based on standard library’s logging module, augmented a bit to support features loke colorized messages and stackable context information.

Example usage:

# initialize logger as soon as possible
Logging.setup_logger()
logger = Logging.get_logger()

# now it's possible to use it for logging:
logger.debug('foo!')

# find out what your logging should look like, e.g. by parsing command-line options
...

# tell logger about the final setup
logger = Logging.setup_logger(output_file='/tmp/foo.log', level=...)

# when you want to make logging methods easily accessible in your class, throw in
# LoggerMixin:
class Foo(LoggerMixin, ...):
  def __init__(self, some_logger):
      super(Foo, self).__init__(logger)

      self.debug('foo!')

class gluetool.log.BlobLogger(intro, outro=None, on_finally=None, writer=None)

Bases: object

Context manager to help with “real time” logging - some code may produce output continuously, e.g. when running a command and streaming its output to our stdout, and yet we still want to wrap it with boundaries and add a header.

This code:

with BlobLogger('ls of root', outro='end of ls'):
    subprocess.call(['ls', '/'])

will lead to the output similar to this:

[20:30:50] [+] ---v---v---v---v---v--- ls of root
bin  boot  data  dev ...
[20:30:50] [+] ---^---^---^---^---^--- end of ls

Note

When you already hold the data you wish to log, please use gluetool.log.log_blob() or gluetool.log.log_dict(). The example above could be rewritten using log_blob by using subprocess.check_output() and passing its return value to log_blob. BlobLogger is designed to wrap output whose creation caller don’t want to (or cannot) control.

Parameters:

• intro (str) – Label to show what is the meaning of the logged data.
• outro (str) – Label to show by the final boundary to mark the end of logging.
• on_finally (callable) – When set, it will be called in __exit__ method. User of this context manager might need to flush used streams or close resources even in case the exception was raised while inside the context manager. on_finally is called with all arguments the __exit__ was called, and its return value is returned by __exit__ itself, therefore it can examine possible exceptions, and override them.
• writer (callable) – A function which is used to actually log the text. Usually a one of some logger methods.

class gluetool.log.ContextAdapter(logger, extra=None)

Bases: logging.LoggerAdapter

Generic logger adapter that collects “contexts”, and prepends them to the message.

“context” is any key in extra dictionary starting with ctx_, whose value is expected to be tuple of (priority, value). Contexts are then sorted by their priorities before inserting them into the message (lower priority means context will be placed closer to the beggining of the line - highest priority comes last.

ContextAdapter is a corner stone of our logging infrastructure, everything is supposed to, one way or another, to use this class (or one of its children) for logging. We should avoid using bare Logger instances because they lack context propagation, message routing to debug and verbose files, verbose method & sentry parameter.

Parent class, logging.LoggerAdapter, provides all common logging methods. We overload them with our own implementations because we want to 1) handle type annotations on our side, 2) let users submit messages to Sentry.

Parameters:

• logger – parent logger this adapter modifies.
• extras (dict) – additional extra keys passed to the parent class. The dictionary is then used to update messages’ extra key with the information about context.

addHandler(*args, **kwargs)

debug(msg, exc_info=None, extra=None, sentry=False)

error(msg, exc_info=None, extra=None, sentry=False)

exception(msg, exc_info=None, extra=None, sentry=False)

info(msg, exc_info=None, extra=None, sentry=False)

isEnabledFor(level)

log(level, msg, exc_info=None, extra=None, sentry=False)

process(msg, kwargs)

Original process overwrites kwargs['extra'] which doesn’t work for us - we want to chain adapters, getting more and more contexts on the way. Therefore update instead of assignment.

removeHandler(*args, **kwargs)

verbose(msg, exc_info=None, extra=None, sentry=False)

warn(msg, exc_info=None, extra=None, sentry=False)

warning(msg, exc_info=None, extra=None, sentry=False)

class gluetool.log.JSONLoggingFormatter(**kwargs)

Bases: logging.Formatter

Custom logging formatter producing a JSON dictionary describing the log record.

static _format_exception_chain(serialized, exc_info)

“Format” exception chain - transform it into a bunch of JSON structures describing exceptions, stack frames, local variables and so on.

Serves the same purpose as LoggingFormatter._format_exception_chain but that one produces a string, textual representation suitable for printing. This method produces JSON structures, suitable for, hm, JSON log.

format(record)

class gluetool.log.LoggerMixin(logger, *args, **kwargs)

Bases: object

Use as a parent class (or one of them) when you want to “attach” methods of a given logger to class’ instances.

Parameters:logger (ContextAdapter) – logger to propagate.

attach_logger(logger)

Initialize this object’s logging methods to those provided by a given logger.

class gluetool.log.Logging

Bases: object

Container wrapping configuration and access to logging infrastructure gluetool uses for logging.

OUR_LOGGERS = (<logging.Logger object>, <logging.Logger object>, <logging.Logger object>)

static _setup_log_file(filepath, level, limit_level=False, formatter_class=<class 'gluetool.log.LoggingFormatter'>)

adapted_logger = None

Logger singleton - if anyone asks for a logger, they will get this one. Needs to be properly initialized by calling setup_logger().

static configure_logger(logger)

Configure given logger to conform with Gluetool’s idea of logging. The logger is set to VERBOSE level, shared stderr handler is added, and Sentry integration status is propagated as well.

After this method, the logger will behave like Gluetool’s main logger.

debug_file_handler = None

static enable_debug_file(logger)

static enable_json_file(logger)

static enable_logger_sentry(logger)

static enable_verbose_file(logger)

static get_logger()

Returns a logger-like object suitable for logging stuff.

Return type:ContextAdapter Returns:an instance of ContextAdapter wrapping root logger, or None when there’s no logger yet.

json_file_handler = None

logger = None

Bare root logger we’re hiding inside adapted_logger

sentry = None

static setup_logger(level=20, debug_file=None, verbose_file=None, json_file=None, sentry=None, show_traceback=False)

Create and setup logger.

This method is called at least twice:

• when gluetool.glue.Glue is instantiated: only a stderr handler is set up, with loglevel being INFO;
• when all arguments and options are processed, and Glue instance can determine desired log level, whether it’s expected to stream debugging messages into a file, etc. This time, method only modifies propagates necessary updates to already existing logger.

Parameters:

• debug_file (str) – if set, new handler will be attached to the logger, streaming messages of at least DEBUG level into this this file.
• verbose_file (str) – if set, new handler will be attached to the logger, streaming messages of VERBOSE log levels into this this file.
• json_file (str) – if set, all logging messages are sent to this file in a form of JSON structures.
• level (int) – desired log level. One of constants defined in logging module, e.g. logging.DEBUG or logging.ERROR.
• sentry (bool) – if set, logger will be augmented to send every log message to the Sentry server.
• show_traceback (bool) – if set, exception tracebacks would be sent to stderr handler as well as to the debug file.

Return type:

ContextAdapter

Returns:

a ContextAdapter instance, set up for logging.

stderr_handler = None

Stream handler printing out to stderr.

verbose_file_handler = None

class gluetool.log.LoggingFormatter(colors=True, log_tracebacks=False)

Bases: logging.Formatter

Custom log record formatter. Produces output in form of:

[stamp] [level] [ctx1] [ctx2] ... message

Parameters:

• colors (bool) – if set, colorize output. Enabled by default but when used with file-backed destinations, colors are disabled by logging subsystem.
• log_tracebacks (bool) – if set, add tracebacks to the message. By default, we don’t need tracebacks on the terminal, unless its loglevel is verbose enough, but we want them in the debugging file.

static _format_exception_chain(exc_info)

Format exception chain. Start with the one we’re given, and follow its caused_by property until we ran out of exceptions to format.

_level_color = {40: <function <lambda>>, 50: <function <lambda>>, 20: <function <lambda>>, 30: <function <lambda>>}

Colorizers assigned to loglevels

_level_tags = {5: 'V', 40: 'E', 10: 'D', 50: 'C', 20: '+', 30: 'W'}

Tags used to express loglevel.

format(record)

Format a logging record. It puts together pieces like time stamp, log level, possibly also different contexts if there are any stored in the record, and finally applies colors if asked to do so.

Parameters:record (logging.LogRecord) – record describing the event. Return type:str Returns:string representation of the event record.

class gluetool.log.ModuleAdapter(logger, module)

Bases: gluetool.log.ContextAdapter

Custom logger adapter, adding module name as a context.

Parameters:

• logger – parent logger this adapter modifies.
• module (gluetool.glue.Module) – module whose name is added as a context.

class gluetool.log.PackageAdapter(logger, name)

Bases: gluetool.log.ContextAdapter

Custom logger dapter, adding a package name as a context. Intended to taint log records produced by a 3rd party packages.

Parameters:

• logger – parent logger this adapter modifies.
• name (str) – name of the library.

class gluetool.log.SingleLogLevelFileHandler(level, *args, **kwargs)

Bases: logging.FileHandler

emit(record)

class gluetool.log.StreamToLogger(log_fn)

Bases: object

Fake file-like stream object that redirects writes to a given logging method.

write(buf)

gluetool.log._extract_stack(tb)

Construct a “stack” by merging two sources of data:

1. what’s provided by traceback.extract_tb(), i.e. (filename, lineno, fnname, text) tuple for each frame;
2. stack frame objects, hidden inside traceback object and available via following links from one frame to another.

Return type:list(list(str, int, str, str, frame))

gluetool.log.format_blob(blob)

Format a blob of text for printing. Wraps the text with boundaries to mark its borders.

gluetool.log.format_dict(dictionary)

Format a Python data structure for printing. Uses json.dumps() formatting capabilities to present readable representation of a given structure.

gluetool.log.format_table(table, **kwargs)

Format a table, represented by an iterable of rows, represented by iterables.

Internally, tabulate is used to do the formatting. All keyword arguments are passed to tabulate call.

Parameters:table (list(list())) – table to format. Returns:formatted table.

gluetool.log.format_xml(element)

Format an XML element, e.g. Beaker job description, for printing.

Parameters:element – XML element to format.

gluetool.log.log_blob(writer, intro, blob)

Log “blob” of characters of unknown structure, e.g. output of a command or response of a HTTP request. The blob is preceded by a header and followed by a footer to mark exactly the blob boundaries.

Note

For logging structured data, e.g. JSON or Python structures, use gluetool.log.log_dict(). It will make structure of the data more visible, resulting in better readability of the log.

Parameters:

• writer (callable) – A function which is used to actually log the text. Usually a one of some logger methods.
• intro (str) – Label to show what is the meaning of the logged blob.
• blob (str) – The actual blob of text.

gluetool.log.log_dict(writer, intro, data)

Log structured data, e.g. JSON responses or a Python list.

Note

For logging unstructured “blobs” of text, use gluetool.log.log_blob(). It does not attempt to format the output, and wraps it by header and footer to mark its boundaries.

Note

Using gluetool.log.format_dict() directly might be shorter, depending on your your code. For example, this code:

self.debug('Some data:\n{}'.format(format_dict(data)))

is equivalent to:

log_dict(self.debug, 'Some data', data)

If you need more formatting, or you wish to fit more information into a single message, using logger methods with format_dict is a way to go, while for logging a single structure log_dict is more suitable.

Parameters:

• writer (callable) – A function which is used to actually log the text. Usually a one of some logger methods.
• intro (str) – Label to show what is the meaning of the logged structure.
• blob (str) – The actual data to log.

gluetool.log.log_table(writer, intro, table, **kwargs)

Log a formatted table.

All keyword arguments are passed to format_table() call which does the actual formatting.

Parameters:

• writer (callable) – A function which is used to actually log the text. Usually a one of some logger methods.
• intro (str) – Label to show what is the meaning of the logged table.
• table (list(list())) – table to format.

gluetool.log.log_xml(writer, intro, element)

Log an XML element, e.g. Beaker job description.

Parameters:

• writer (callable) – A function which is used to actually log the text. Usually a one of some logger methods.
• intro (str) – Label to show what is the meaning of the logged blob.
• element – XML element to log.

gluetool.log.print_wrapper(*args, **kwds)

While active, replaces sys.stdout and sys.stderr streams with fake file-like streams which send all outgoing data to a given logging method.

Parameters:

• log_fn (call) – A callback that is called for every line, produced by print. If not set, a info method of gluetool main logger is used.
• label (text) – short description, presented in the intro and outro headers, wrapping captured output.