Contents | Previous (8.1 Testing) | Next (8.3 Debugging)

8.2 Logging

This section briefly introduces the logging module.

logging Module

The logging module is a standard library module for recording diagnostic information. It's also a very large module with a lot of sophisticated functionality. We will show a simple example to illustrate its usefulness.

Exceptions Revisited

In the exercises, we wrote a function parse() that looked something like this:

# fileparse.pydefparse(f, types=None, names=None, delimiter=None):
records= []
forlineinf:
line=line.strip()
ifnotline: continuetry:
records.append(split(line,types,names,delimiter))
exceptValueErrorase:
print("Couldn't parse :", line)
print("Reason :", e)
returnrecords

Focus on the try-except statement. What should you do in the except block?

Should you print a warning message?

try:
records.append(split(line,types,names,delimiter))
exceptValueErrorase:
print("Couldn't parse :", line)
print("Reason :", e)

Or do you silently ignore it?

try:
records.append(split(line,types,names,delimiter))
exceptValueErrorase:
pass

Neither solution is satisfactory because you often want both behaviors (user selectable).

Using logging

The logging module can address this.

# fileparse.pyimportlogginglog=logging.getLogger(__name__)
defparse(f,types=None,names=None,delimiter=None):
...
try:
records.append(split(line,types,names,delimiter))
exceptValueErrorase:
log.warning("Couldn't parse : %s", line)
log.debug("Reason : %s", e)

The code is modified to issue warning messages or a special Logger object. The one created with logging.getLogger(__name__).

Logging Basics

Create a logger object.

log=logging.getLogger(name) # name is a string

Issuing log messages.

log.critical(message [, args])
log.error(message [, args])
log.warning(message [, args])
log.info(message [, args])
log.debug(message [, args])

Each method represents a different level of severity.

All of them create a formatted log message. args is used with the % operator to create the message.

logmsg=message%args# Written to the log

Logging Configuration

The logging behavior is configured separately.

# main.py
...
if__name__=='__main__':
importlogginglogging.basicConfig(
filename='app.log', # Log output filelevel=logging.INFO, # Output level
)

Typically, this is a one-time configuration at program startup. The configuration is separate from the code that makes the logging calls.

Comments

Logging is highly configurable. You can adjust every aspect of it: output files, levels, message formats, etc. However, the code that uses logging doesn't have to worry about that.

Exercises

Exercise 8.2: Adding logging to a module

In fileparse.py, there is some error handling related to exceptions caused by bad input. It looks like this:

# fileparse.pyimportcsvdefparse_csv(lines, select=None, types=None, has_headers=True, delimiter=',', silence_errors=False):
''' Parse a CSV file into a list of records with type conversion. '''ifselectandnothas_headers:
raiseRuntimeError('select requires column headers')
rows=csv.reader(lines, delimiter=delimiter)
# Read the file headers (if any)headers=next(rows) ifhas_headerselse []
# If specific columns have been selected, make indices for filtering and set output columnsifselect:
indices= [ headers.index(colname) forcolnameinselect ]
headers=selectrecords= []
forrowno, rowinenumerate(rows, 1):
ifnotrow: # Skip rows with no datacontinue# If specific column indices are selected, pick them outifselect:
row= [ row[index] forindexinindices]
# Apply type conversion to the rowiftypes:
try:
row= [func(val) forfunc, valinzip(types, row)]
exceptValueErrorase:
ifnotsilence_errors:
print(f"Row {rowno}: Couldn't convert {row}")
print(f"Row {rowno}: Reason {e}")
continue# Make a dictionary or a tupleifheaders:
record=dict(zip(headers, row))
else:
record=tuple(row)
records.append(record)
returnrecords

Notice the print statements that issue diagnostic messages. Replacing those prints with logging operations is relatively simple. Change the code like this:

# fileparse.pyimportcsvimportlogginglog=logging.getLogger(__name__)
defparse_csv(lines, select=None, types=None, has_headers=True, delimiter=',', silence_errors=False):
''' Parse a CSV file into a list of records with type conversion. '''ifselectandnothas_headers:
raiseRuntimeError('select requires column headers')
rows=csv.reader(lines, delimiter=delimiter)
# Read the file headers (if any)headers=next(rows) ifhas_headerselse []
# If specific columns have been selected, make indices for filtering and set output columnsifselect:
indices= [ headers.index(colname) forcolnameinselect ]
headers=selectrecords= []
forrowno, rowinenumerate(rows, 1):
ifnotrow: # Skip rows with no datacontinue# If specific column indices are selected, pick them outifselect:
row= [ row[index] forindexinindices]
# Apply type conversion to the rowiftypes:
try:
row= [func(val) forfunc, valinzip(types, row)]
exceptValueErrorase:
ifnotsilence_errors:
log.warning("Row %d: Couldn't convert %s", rowno, row)
log.debug("Row %d: Reason %s", rowno, e)
continue# Make a dictionary or a tupleifheaders:
record=dict(zip(headers, row))
else:
record=tuple(row)
records.append(record)
returnrecords

Now that you've made these changes, try using some of your code on bad data.

>>>importreport>>>a=report.read_portfolio('Data/missing.csv')
Row4: Badrow: ['MSFT', '', '51.23']
Row7: Badrow: ['IBM', '', '70.44']
>>>

If you do nothing, you'll only get logging messages for the WARNING level and above. The output will look like simple print statements. However, if you configure the logging module, you'll get additional information about the logging levels, module, and more. Type these steps to see that:

>>>importlogging>>>logging.basicConfig()
>>>a=report.read_portfolio('Data/missing.csv')
WARNING:fileparse:Row4: Badrow: ['MSFT', '', '51.23']
WARNING:fileparse:Row7: Badrow: ['IBM', '', '70.44']
>>>

You will notice that you don't see the output from the log.debug() operation. Type this to change the level.

>>> logging.getLogger('fileparse').level = logging.DEBUG
>>> a = report.read_portfolio('Data/missing.csv')
WARNING:fileparse:Row 4: Bad row: ['MSFT', '', '51.23']
DEBUG:fileparse:Row 4: Reason: invalid literal for int() with base 10: ''
WARNING:fileparse:Row 7: Bad row: ['IBM', '', '70.44']
DEBUG:fileparse:Row 7: Reason: invalid literal for int() with base 10: ''
>>>

Turn off all, but the most critical logging messages:

>>> logging.getLogger('fileparse').level=logging.CRITICAL
>>> a = report.read_portfolio('Data/missing.csv')
>>>

Exercise 8.3: Adding Logging to a Program

To add logging to an application, you need to have some mechanism to initialize the logging module in the main module. One way to do this is to include some setup code that looks like this:

# This file sets up basic configuration of the logging module.
# Change settings here to adjust logging output as needed.
import logging
logging.basicConfig(
filename = 'app.log', # Name of the log file (omit to use stderr)
filemode = 'w', # File mode (use 'a' to append)
level = logging.WARNING, # Logging level (DEBUG, INFO, WARNING, ERROR, or CRITICAL)
)

Again, you'd need to put this someplace in the startup steps of your program. For example, where would you put this in your report.py program?