Article 3 of 3 in the series Python's super() explained
By Austin Bingham

In the previous articles in this series [1] we uncovered a small mystery regarding how Python's super() works, and we looked at some of the underlying mechanics of how super() really works. In this article we'll see how those details work together to resolve the mystery.

The mystery revisited

As you'll recall, the mystery we uncovered has to do with how a single use of super() seemed to invoke two separate implementations of a method in our SortedIntList example. As a reminder, our class diagram looks like this:

Inheritance graph

SimpleList defines a simplified list API, and IntList and SortedList each enforce specific constraints on the contents of the list. The class SortedIntList inherits from both IntList and SortedList and enforces the constraints of both base classes. Interestingly, though, SortedIntList has a trivial implementation:

class SortedIntList(IntList, SortedList):
    pass

How does SortedIntList do what it does? From the code it seems that SortedIntList does no coordination between its base classes, and certainly the base classes don't know anything about each other. Yet somehow SortedIntList manages to invoke both implementations of add(), thereby enforcing all of the necessary constraints.

super() with no arguments

We've already looked at method resolution order, C3 linearization, and the general behavior of super instances. The final bit of information we need in order to resolve the mystery is how super() behaves when called with no arguments. [2] Both IntList and SortedList use super() this way, in both their initializers and in add().

For instance methods such as these, calling super() with no arguments is the same as calling super() with the method's class as the first argument and self as the second. In other words, it constructs a super proxy that uses the MRO from self starting at the class implementing the method. Knowing this, it's easy to see how, in simple cases, using super() is equivalent to "calling the base class implementation". In these cases, type(self) is the same as the class implementing the method, so the method is resolved using everything in that class's MRO except the class itself. The next entry in the MRO will, of course, be the class's first base class, so simple uses of super() are equivalent to invoking a method on the first base class.

A key point to notice here is that type(self) will not always be the same as the class implementing a specific method. If you invoke super() in a method on a class with a subclass, then type(self) may well be that subclass. You can see this in a trivial example:

>>> class Base:
...     def f(self):
...         print('Type of self in Base.f() =', type(self))
...
>>> class Sub(Base):
...     pass
...
>>> Sub().f()
Type of self in Base.f() = <class '__main__.Sub'>

Understanding this point is the final key to seeing how SortedIntList works. If type(self) in a base class is not necessarily the type of the class implementing the method, then the MRO that gets used by super() is not necessarily the MRO of the class implementing the method...it may be that of the subclass. Since the entries in type(self).mro() may include entries that are not in the MRO for the implementing class, calls to super() in a base class may resolve to implementations that are not in the base class's own MRO. In other words, Python's method resolution is - as you might have guessed - extremely dynamic and depends not just on a class's base classes but its subclasses as well.

Method resolution order to the rescue

With that in mind, let's finally see how all of the elements - MRO, no-argument super(), and multiple inheritance - coordinate to make SortedIntList work. As we've just seen, when super() is used with no arguments in an instance method, the proxy uses the MRO of self which, of course, will be that of SortedIntList in our example. So the first thing to look at is the MRO of SortedIntList itself:

>>> SortedIntList.mro()
[<class 'SortedIntList'>,
<class 'IntList'>,
<class 'SortedList'>,
<class 'SimpleList'>,
<class 'object'>]

A critical point here is that the MRO contains IntList and SortedList, meaning that both of them can contribute implementations when super() is used.

Next, let's examine the behavior of SortedIntList when we call its add() method. [3] Because IntList is the first class in the MRO which implements add(), a call to add() on a SortedIntList resolves to IntList.add():

class IntList(SimpleList):
    # ...

    def add(self, item):
        self._validate(item)
        super().add(item)

And this is where things get interesting!

In IntList.add() there is a call to super().add(item), and because of how no-argument super() calls work, this is equivalent to super(IntList, self).add(item). Since type(self) == SortedIntList, this call to super() uses the MRO for SortedIntList and not just IntList. As a result, even though IntList doesn't really "know" anything about SortedList, it can access SortedList methods via a subclass's MRO.

In the end, the call to super().add(item) in IntList.add() takes the MRO of SortedIntList, find's IntList in that MRO, and uses everything after IntList in the MRO to resolve the method invocation. Since that MRO-tail looks like this:

[<class 'SortedList'>, <class 'SimpleList'>, <class 'object'>]

and since method resolution uses the first class in an MRO that implements a method, the call resolves to SortedList.add() which, of course, enforces the sorting constraint.

So by including both of its base classes in its MRO [4] - and because IntList and SortedList use super() in a cooperative way - SortedIntList ensures that both the sorting and type constraint are enforced.

No more mystery

We've seen that a subclass can leverage MRO and super() to do some pretty interesting things. It can create entirely new method resolutions for its base classes, resolutions that aren't apparent in the base class definitions and are entirely dependent on runtime context.

Used properly, this can lead to some really powerful designs. Our SortedIntList example is just one instance of what can be done. At the same time, if used naively, super() can have some surprising and unexpected effects, so it pays to think deeply about the consequences of super() when you use it. For example, if you really do want to just call a specific base class implementation, you might be better off calling it directly rather than leaving the resolution open to the whims of subclass developers. It may be cliche, but it's true: with great power comes great responsibility.

For more information on this topic, you can always see the Inheritance and Subtype Polymorphism module of our Python: Beyond the Basics course on PluralSight. [5]

[1]Part 1 and Part 2
[2]Note that calling super() with no arguments is only supported in Python 3. Python 2 users will need to use the longer explicit form.
[3]The same logic applies to it's __init__() which also involves calls to super().
[4]Thanks, of course, to how C3 works.
[5]Python: Beyond the Basics on PluralSight
Category: misc