Suppose you are maintaining a module, package, or data structure that has an API with many methods and you want to add the ability to log every API call. Furthermore, you need to enable or disable the logging feature using a single flag. Do you manually add a block of logging code to every method definition? Do you write a wrapper API? How do you ensure the logging feature is modular in its implementation, remains compatible as the API evolves, and is easy to maintain?
Python's concrete syntax includes support for something called decorators. This is a syntactic feature that acts as a convenient and concise tool for modifying, analyzing, or associating functions and methods at the point at which they are defined. This can reduce redundancy and clutter in code. Importantly, it leverages Python's native support for a functional programming paradigm to offer you a different kind of modularity and composability that can help you add the logging feature in a relatively quick, concise, and elegant way.
To understand decorators, it is necessary to at least be aware of higher-order functions. In Python, functions can be defined and used just like values. This is a characteristic feature of the functional programming paradigm, which Python supports. Consider the following example in which a function f is defined and then passed as an argument to another function twice. Note that when twice receives the argument f it is assigned to the local variable g. Then this yields g(g(y)) = f(f(y)) = f(y + y) = (y + y) + (y + y). If you assign 2 to y, this will yield g(g(2)) = f(f(2)) = f(2 + 2) = (2 + 2) + (2 + 2) = 8.
def f(x):
return x + x
def twice(g, y):
return g(g(y))
twice(f, 2)
Just as functions can be arguments, they can also be results. In the example below, a different variant of twice takes a function as its sole input and returns a new function that behaves like its input function but is applied twice.
def f(x):
return x + x
def twice(g):
# Define a new function locally.
def h(y):
return g(g(y))
# Return the local function as the result.
return h
twice(f)
Because twice(f) is a function, you can apply it to an argument and it will return a result.
twice(f)(2)
As a slightly simpler variant of the motivating example described in the introduction, suppose you want to modify some existing functions so that they display their results using print (in addition to returning their results as they normally would). To do this in a reusable way, you can write a higher-order function that takes the original function as an input and returns a new function that also prints the result.
def displays(f):
# This is the new variant of the function `f`.
def f_displays(x):
r = f(x)
print('The result is:', r)
return r
return f_displays
This transformer function displays can then be applied to any existing function to give you a new version that also prints its result.
def double(x):
return x + x
double = displays(double)
double(2)
Python's concrete syntax lets you do exactly the same thing using a more concise notation: prepend the @ symbol before your higher-order function and place it immediately before the definition of the function you want to transform.
@displays
def triple(x):
return x + x + x
In the example above, the variable triple (after the decorated definition is executed) refers to the transformed version of the function in the definition.
triple(2)
Because the higher-order function used as a decorator is itself just a function, it can also be the result of a function. Thus, you can create a function that creates decorators! Below, the function displays_with returns a decorator that prints a custom message rather than the hard-coded one in the examples above.
def displays_with(message):
# Create the function that converts a function
# into a function that display (i.e., our old
# decorator).
def displays(f):
# This is the new variant of the function `f`.
def f_displays(x):
r = f(x)
print(message, r)
return r
return f_displays
# Return the function created above.
return displays
The decorator syntax allows you to supply the argument to the function that creates a decorator.
@displays_with('The function returned:')
def triple(x):
return x + x + x
triple(2)
To clarify what is happening, a code block that is functionally equivalent to code above is presented below.
def triple(x):
return x + x + x
triple = displays_with('The function returned:')(triple)
triple(2)
Decorators can also be stacked. Suppose you create a decorator that also adds the decorated function to a running list of functions.
functions = []
def function(f):
functions.append(f)
You can now decorate a function with both decorators.
@displays_with('The function returned:')
@function
def triple(x):
return x + x + x
functions
Note that order does matter: if you place the function decorator above the displays_with, the function added to the list using function will be the one already modified by displays_with. Thus, when multiple decorators are present they are applied from the bottom up (or, in other words, decorators are right-associative). In the example below, the transformed function triple is added to the list of functions.
functions = []
@function
@displays_with('The function returned:')
def triple(x):
return x + x + x
functions[0](2)
In the example below, on the other hand, the original version of triple is added to the list of functions.
functions = []
@displays_with('The function returned:')
@function
def triple(x):
return x + x + x
functions[0](2)
Just as functions in Python can be used as values, so can classes. This article will not go into much depth on this subject. It is enough to see that the same examples presented involving function definitions have corresponding examples involving class definitions.
First, note that you can define a function that takes a class (not an object of the class, but the class itself) as an input. The function check below takes a class as an input and checks if it has a method called method.
class C:
def __init__(self, attr):
self.attr = attr
class D:
def method(self):
pass
def check(cls):
return callable(getattr(cls, 'method', None))
check(C), check(D)
Decorators can be added to a class definition in the same way that they can be added to a function definition. In the examples below, the decorator displayable checks whether a class definition includes a method called display and raises an exception if it does not.
def displayable(cls):
if not callable(getattr(cls, 'display', None)):
raise Exception('objects of this class are not displayable')
@displayable
class C:
def display(self):
return 'C'
The expected behavior can be seen in the example below.
try:
@displayable
class D:
def method(self):
pass
except Exception as e:
print(e)
As illustrated above, decorators are a reusable way to analyze, log, associate, or transform a function, method, or class by adding just one line to their definition. The original motivating use case, as well as a few others, are reviewed below.
Logging is a compelling use case for decorators because it illustrates how they can save significant time and effort. To review: you have a large API and need to log the inputs and outputs of every API call. The simple API below can act as a placeholder or this example.
class Database:
def __init__(self):
self.data = []
def insert(self, entry):
data.append(entry)
return True
def find(self, entry):
return entry in self.data
One approach you can take is to implement a single decorator that you will reuse for every method in the API implementation.
log = []
def logged(f):
def logged_f(db, inp):
outp = f(db, inp)
log.append({'method':f.__name__, 'in':inp, 'out':outp})
return outp
return logged_f
You can add the decorator before every public method definition.
class Database:
def __init__(self):
self.data = []
@logged
def insert(self, entry):
self.data.append(entry)
return True
@logged
def find(self, entry):
return entry in self.data
Below is what you might see in the log after a few API calls are made.
db = Database()
db.insert('alice')
db.find('alice')
db.find('bob')
log
As an exercise, you may want to try writing a single decorator for an entire class definition in order to avoid adding a decorator to every function. You may find the built-in function dir useful for this purpose.
Decorators are used in a number of popular libraries, such as Flask. In the example below drawn from the Flask documentation, Flask is used to set up an HTTP server with a single route. The function that handles processing of a requests and the construction of a response is associated with that event using a decorator provided by the Flask API.
from flask import Flask
app = Flask(__name__)
@app.route('/')
def hello_world():
return 'Hello, World!'
Note that the decorator is itself a method, and also that it is technically higher-order (in that it takes an argument consisting of the route path and returns a decorator that is then applied to the function being defined).
As illustrated in the example with the class decorator displayable, decorators can be used to implement static or dynamic analyses of functions, methods, and classes. A static analysis might only examine its input function/class (or the code inside it) at the time of the definition without actually running the code or modifying the function/class itself. A dynamic analysis might run the code itself or it might modify the code to measure its own operation in some way. Because you have already seen an example of the former, examples of the latter are presented below.
For the first example, suppose you want to test that a method always returns positive outputs in a range of inputs. The decorator definition below illustrates one way that this can be accomplished.
def check(f):
# Run some tests on `f`.
for x in range(-10,11):
if f(x) < 0:
raise ValueError('incorrect negative output')
# Do not modify the original function.
return f
@check
def square(x):
return x * x
For the second example, consider a situation in which you want to check the running time of various methods. You can use decorators in conjunction with the built-in time package.
def timed(f):
from time import time
def time_f():
ts = time()
result = f()
te = time()
print(f.__name__ + ':', te-ts, 'seconds')
return result
return time_f
The example below demonstrates how the timed decorator defined above might be used.
@timed
def work():
from time import sleep
sleep(1)
work()
One issue that might arise when an analyzing functions in this way, especially when you are stacking decorators, is that the decorated function will not preserve the original function's metadata.
work.__name__
def decorated(f):
from functools import wraps
@wraps(f)
def decorated_f():
return f()
return decorated_f
Using wraps as in the above example ensures the metadata of the original function is preserved.
@decorated
def f():
pass
f.__name__
Hopefully, this article leaves you with a better understanding of how decorators are a syntactically convenient way to use higher-order functions and helps you recognize some of the situations for which they may be well-suited in your own work. There are many other compelling use cases for both higher-order functions and decorators, some of which may be covered in future articles. For a more comprehensive resource on decorators, you may want to look at the Python Wiki. To learn more about the history of the feature, you can review the Python Enhancement Proposal for this feature.