gluetool.log module

Logging support.

Sets up logging environment for use by gluetool and modules. Based on standard library’s logging module, augmented a bit to support features loke colorized messages and stackable context information.

Example usage:

# initialize logger as soon as possible
logger = Logging.create_logger()

# now it's possible to use it for logging:
logger.debug('foo!')

# or connect it with current instance (if you're doing all this
# inside some class' constructor):
logger.connect(self)

# now you can access logger's methods directly:
self.debug('foo once again!')

# find out what your logging should look like, e.g. by parsing command-line options
...

# tell logger about the final setup
logger = Logging.create_logger(output_file='/tmp/foo.log', level=...)

# and, finally, create a root context logger - when we create another loggers during
# the code flow, this context logger will be in the root of this tree of loggers.
logger = ContextAdapter(logger)

# don't forget to re-connect with the context logger if you connected your instance
# with previous logger, to make sure helpers are set correctly
logger.connect(self)

class gluetool.log.BlobLogger(intro, outro=None, on_finally=None, writer=None)

Bases: object

Context manager to help with “real time” logging - some code may produce output continuously, e.g. when running a command and streaming its output to our stdout, and yet we still want to wrap it with boundaries and add a header.

This code:

with BlobLogger('ls of root', outro='end of ls'):
    subprocess.call(['ls', '/'])

will lead to the output similar to this:

[20:30:50] [+] ---v---v---v---v---v--- ls of root
bin  boot  data  dev ...
[20:30:50] [+] ---^---^---^---^---^--- end of ls

Note

When you already hold the data you wish to log, please use gluetool.log.log_blob() or gluetool.log.log_dict(). The example above could be rewritten using log_blob by using subprocess.check_output() and passing its return value to log_blob. BlobLogger is designed to wrap output whose creation caller don’t want to (or cannot) control.

Parameters:

• intro (str) – Label to show what is the meaning of the logged data.
• outro (str) – Label to show by the final boundary to mark the end of logging.
• on_finally (callable) – When set, it will be called in __exit__ method. User of this context manager might need to flush used streams or close resources even in case the exception was raised while inside the context manager. on_finally is called with all arguments the __exit__ was called, and its return value is returned by __exit__ itself, therefore it can examine possible exceptions, and override them.
• writer (callable) – A function which is used to actually log the text. Usually a one of some logger methods.

class gluetool.log.ContextAdapter(logger, extra=None)

Bases: logging.LoggerAdapter

Generic logger adapter that collects “contexts”, and prepends them to the message.

“context” is any key in extra dictionary starting with ctx_, whose value is expected to be tuple of (priority, value). Contexts are then sorted by their priorities before inserting them into the message (lower priority means context will be placed closer to the beggining of the line - highest priority comes last.

Parameters:

• logger (logging.Logger) – parent logger this adapter modifies.
• extras (dict) – additional extra keys passed to the parent class. The dictionary is then used to update messages’ extra key with the information about context.

connect(parent)

Create helper methods in parent, by assigning adapter’s methods to its attributes. One can then call parent.debug and so on, instead of less readable parent.logger.debug.

Simply instantiate adapter and call its connect with an object as a parent argument, and the object will be enhanced with all these logging helpers.

Parameters:parent – object to enhance with logging helpers.

process(msg, kwargs)

Original process overwrites kwargs['extra'] which doesn’t work for us - we want to chain adapters, getting more and more contexts on the way. Therefore update instead of assignment.

class gluetool.log.JSONLoggingFormatter(**kwargs)

Bases: logging.Formatter

Custom logging formatter producing a JSON dictionary describing the log record.

static _format_exception_chain(serialized, exc_info)

“Format” exception chain - transform it into a bunch of JSON structures describing exceptions, stack frames, local variables and so on.

Serves the same purpose as LoggingFormatter._format_exception_chain but that one produces a string, textual representation suitable for printing. This method produces JSON structures, suitable for, hm, JSON log.

format(record)

class gluetool.log.Logging

Bases: object

Container wrapping configuration and access to logging infrastructure gluetool uses for logging.

OUR_LOGGERS = (<logging.Logger object>, <logging.Logger object>)

static _setup_log_file(filepath, level, limit_level=False, formatter=<class 'gluetool.log.LoggingFormatter'>)

static configure_logger(logger)

Configure given logger to conform with Gluetool’s idea of logging. The logger is set to VERBOSE level, shared stderr handler is added, and Sentry integration status is propagated as well.

After this method, the logger will behave like Gluetool’s main logger.

static create_logger(level=20, debug_file=None, verbose_file=None, json_file=None, sentry=None, sentry_submit_warning=None, show_traceback=False)

Create and setup logger.

This method is called at least twice:

• when gluetool.glue.Glue is instantiated: only a stderr handler is set up, with loglevel being INFO;
• when all arguments and options are processed, and Glue instance can determine desired log level, whether it’s expected to stream debugging messages into a file, etc. This time, method only modifies propagates necessary updates to already existing logger.

Parameters:

• debug_file (str) – if set, new handler will be attached to the logger, streaming messages of at least DEBUG level into this this file.
• verbose_file (str) – if set, new handler will be attached to the logger, streaming messages of VERBOSE log levels into this this file.
• json_file (str) – if set, all logging messages are sent to this file in a form of JSON structures.
• level (int) – desired log level. One of constants defined in logging module, e.g. logging.DEBUG or logging.ERROR.
• sentry (bool) – if set, logger will be augmented to send every log message to the Sentry server.
• sentry_submit_warning (callable) – if set, it is used by warning methods of derived loggers to submit warning to the Sentry server, if asked by a caller to do so.
• show_traceback (bool) – if set, exception tracebacks would be sent to stderr handler as well as to the debug file.

Return type:

logging.Logger

Returns:

a logging.Logger instance, set up for logging.

debug_file_handler = None

static enable_debug_file(logger)

static enable_json_file(logger)

static enable_logger_sentry(logger)

static enable_verbose_file(logger)

static get_logger()

Returns a logger instance.

Expects there was a call to create_logger() method before calling this method that would actually create and set up the logger.

Return type:logging.Logger Returns:a logging.Logger instance, set up for logging, or None when there’s no logger yet.

json_file_handler = None

logger = None

Logger singleton - if anyone asks for a logger, they will get this one. Needs to be properly initialized by calling create_logger().

stderr_handler = None

Stream handler printing out to stderr.

verbose_file_handler = None

class gluetool.log.LoggingFormatter(colors=True, log_tracebacks=False)

Bases: logging.Formatter

Custom log record formatter. Produces output in form of:

[stamp] [level] [ctx1] [ctx2] ... message

Parameters:

• colors (bool) – if set, colorize output. Enabled by default but when used with file-backed destinations, colors are disabled by logging subsystem.
• log_tracebacks (bool) – if set, add tracebacks to the message. By default, we don’t need tracebacks on the terminal, unless its loglevel is verbose enough, but we want them in the debugging file.

static _format_exception_chain(exc_info)

Format exception chain. Start with the one we’re given, and follow its caused_by property until we ran out of exceptions to format.

_level_color = {40: <function <lambda>>, 50: <function <lambda>>, 20: <function <lambda>>, 30: <function <lambda>>}

Colorizers assigned to loglevels

_level_tags = {5: 'V', 40: 'E', 10: 'D', 50: 'C', 20: '+', 30: 'W'}

Tags used to express loglevel.

format(record)

Format a logging record. It puts together pieces like time stamp, log level, possibly also different contexts if there are any stored in the record, and finally applies colors if asked to do so.

Parameters:record (logging.LogRecord) – record describing the event. Return type:str Returns:string representation of the event record.

class gluetool.log.ModuleAdapter(logger, module)

Bases: gluetool.log.ContextAdapter

Custom logger adapter, adding module name as a context.

Parameters:

• logger (logging.Logger) – parent logger this adapter modifies.
• module (gluetool.glue.Module) – module whose name is added as a context.

class gluetool.log.PackageAdapter(logger, name)

Bases: gluetool.log.ContextAdapter

Custom logger dapter, adding a package name as a context. Intended to taint log records produced by a 3rd party packages.

Logging.Logger logger:  parent logger this adapter modifies. Parameters:name (str) – name of the library.

class gluetool.log.SingleLogLevelFileHandler(level, *args, **kwargs)

Bases: logging.FileHandler

emit(record)

class gluetool.log.StreamToLogger(log_fn)

Bases: object

Fake file-like stream object that redirects writes to a given logging method.

write(buf)

gluetool.log._extract_stack(tb)

Construct a “stack” by merging two sources of data:

1. what’s provided by traceback.extract_tb(), i.e. (filename, lineno, fnname, text) tuple for each frame;
2. stack frame objects, hidden inside traceback object and available via following links from one frame to another.

Return type:list(list(str, int, str, str, frame))

gluetool.log.format_blob(blob)

Format a blob of text for printing. Wraps the text with boundaries to mark its borders.

gluetool.log.format_dict(dictionary)

Format a Python data structure for printing. Uses json.dumps() formatting capabilities to present readable representation of a given structure.

gluetool.log.format_xml(element)

Format an XML element, e.g. Beaker job description, for printing.

Parameters:element – XML element to format.

gluetool.log.log_blob(writer, intro, blob)

Log “blob” of characters of unknown structure, e.g. output of a command or response of a HTTP request. The blob is preceded by a header and followed by a footer to mark exactly the blob boundaries.

Note

For logging structured data, e.g. JSON or Python structures, use gluetool.log.log_dict(). It will make structure of the data more visible, resulting in better readability of the log.

Parameters:

• writer (callable) – A function which is used to actually log the text. Usually a one of some logger methods.
• intro (str) – Label to show what is the meaning of the logged blob.
• blob (str) – The actual blob of text.

gluetool.log.log_dict(writer, intro, data)

Log structured data, e.g. JSON responses or a Python list.

Note

For logging unstructured “blobs” of text, use gluetool.log.log_blob(). It does not attempt to format the output, and wraps it by header and footer to mark its boundaries.

Note

Using gluetool.log.format_dict() directly might be shorter, depending on your your code. For example, this code:

self.debug('Some data:\n{}'.format(format_dict(data)))

is equivalent to:

log_dict(self.debug, 'Some data', data)

If you need more formatting, or you wish to fit more information into a single message, using logger methods with format_dict is a way to go, while for logging a single structure log_dict is more suitable.

Parameters:

• writer (callable) – A function which is used to actually log the text. Usually a one of some logger methods.
• intro (str) – Label to show what is the meaning of the logged structure.
• blob (str) – The actual data to log.

gluetool.log.log_xml(writer, intro, element)

Log an XML element, e.g. Beaker job description.

Parameters:

• writer (callable) – A function which is used to actually log the text. Usually a one of some logger methods.
• intro (str) – Label to show what is the meaning of the logged blob.
• element – XML element to log.

gluetool.log.print_wrapper(*args, **kwds)

While active, replaces sys.stdout and sys.stderr streams with fake file-like streams which send all outgoing data to a given logging method.

Parameters:

• log_fn (call) – A callback that is called for every line, produced by print. If not set, a info method of gluetool main logger is used.
• label (str) – short description, presented in the intro and outro headers, wrapping captured output.

gluetool.log.verbose_adapter(self, message, *args, **kwargs)

gluetool.log.verbose_logger(self, message, *args, **kwargs)

gluetool.log.warn_sentry(self, message, *args, **kwargs)

Beside calling the original the warning method (stored as self.orig_warning), this one also submits warning to the Sentry server when asked to do so by a keyword argument sentry set to True.