Understanding of metaclasses in python

Posted by Varma69 on Tue, 07 Jan 2020 15:25:06 +0100

1. Class is also an object

In most programming languages, a class is a set of code snippets that describe how to generate an object. This is still true in Python:

>>> class ObjectCreator(object):
...       pass
...
>>> my_object = ObjectCreator()
>>> print my_object
<__main__.ObjectCreator object at 0x8974f2c>

However, classes in Python go far beyond that. Class is also an object. Yes, that's right. It's the object. As long as you use the keyword class, the Python interpreter creates an object when it executes.

The following code snippet:

>>> class ObjectCreator(object):
...       pass
...

An object will be created in memory with the name ObjectCreator. This object (class object ObjectCreator) has the ability to create objects (instance objects). However, its essence is still an object, so you can operate it as follows:

  • You can assign it to a variable
  • You can copy it
  • You can add attributes to it
  • You can pass it as a function parameter

Here is an example:

'''
No one answers the questions? Xiaobian created a Python learning exchange QQ group: 579817333 
Looking for like-minded small partners, mutual help, there are good video learning tutorials and PDF ebooks in the group!
'''
>>>Print objectcreator - you can print a class because it is also an object
<class '__main__.ObjectCreator'>
>>> def echo(o):
...       print o
...
>>>Echo (objectcreator) - you can pass a class as an argument to a function
<class '__main__.ObjectCreator'>
>>> print hasattr(ObjectCreator, 'new_attribute')
Fasle
 >>>Objectcreator. New? Attribute = 'foo'? You can add attributes to a class
>>> print hasattr(ObjectCreator, 'new_attribute')
True
>>> print ObjectCreator.new_attribute
foo
 >>>Objectcreationmirror = objectcreator you can assign a class to a variable
>>> print ObjectCreatorMirror()
<__main__.ObjectCreator object at 0x8997b4c>

2. Create classes dynamically

Because classes are also objects, you can create them dynamically at run time, just like any other object. First, you can create a class in the function, using the class keyword.

>>> def choose_class(name):
...       if name == 'foo':
...           class Foo(object):
...               pass
...           return Foo     # Class is returned, not an instance of class
...       else:
...           class Bar(object):
...               pass
...           return Bar
...
>>> MyClass = choose_class('foo')
>>> print MyClass              # Function returns a class, not an instance of a class
<class '__main__'.Foo>
>>> print MyClass()            # You can use this class to create class instances, that is, objects
<__main__.Foo object at 0x89c6d4c>

But it's not dynamic enough, because you still need to write your own code for the entire class. Because classes are also objects, they must be generated by something. When you use the class keyword, the Python interpreter automatically creates the object. But like most things in Python, python still provides you with a way to handle it manually.

Remember the built-in function type? This ancient but powerful function can let you know what the type of an object is, like this:

'''
No one answers the questions? Xiaobian created a Python learning exchange QQ group: 579817333 
Looking for like-minded small partners, mutual help, there are good video learning tutorials and PDF ebooks in the group!
'''
>>>Print type (1) × type of value
<type 'int'>
>>>Print type ("1") × type of string
<type 'str'>
>>>Print type (objectcreator()) ා type of instance object
<class '__main__.ObjectCreator'>
>>>Print type (objectcreator) ා type of class
<type 'type'>

After careful observation of the above running results, it is found that using type to view the type of ObjectCreator is that the answer is type, isn't it a little surprising... Look at the bottom.

3. Use type to create a class

type also has a completely different function of dynamically creating classes.

type can take the description of a class as a parameter and then return a class. (it's silly to know that the same function has two completely different uses depending on the arguments passed in, but this is for backward compatibility in Python.)

type works like this:

Type (class name, tuple composed of parent class name (can be empty for inheritance), dictionary containing attributes (name and value))

For example, the following code:

In [2]: class Test: #A Test class is defined
   ...:     pass
   ...:
In [3]: Test() #Created an instance object of Test class
Out[3]: <__main__.Test at 0x10d3f8438>

You can create it manually like this:

Test2 = type("Test2",(),{}) #Set a Test2 class
In [5]: Test2() #Created an instance object of Test2 class
Out[5]: <__main__.Test2 at 0x10d406b38>

We use "Test2" as the class name, and we can also use it as a variable as a reference to the class. Classes and variables are different. There's no reason to complicate things here. That is, the first argument in the type function can also be called another name, which means the name of the class

'''
No one answers the questions? Xiaobian created a Python learning exchange QQ group: 579817333 
Looking for like-minded small partners, mutual help, there are good video learning tutorials and PDF ebooks in the group!
'''
In [23]: MyDogClass = type('MyDog', (), {})

In [24]: print MyDogClass
<class '__main__.MyDog'>

Use help to test these two classes

In [10]: help(Test) #Use help to view Test class

Help on class Test in module __main__:

class Test(builtins.object)
 |  Data descriptors defined here:
 |
 |  __dict__
 |      dictionary for instance variables (if defined)
 |
 |  __weakref__
 |      list of weak references to the object (if defined)
In [8]: help(Test2) #Use help to view Test2 class

Help on class Test2 in module __main__:

class Test2(builtins.object)
 |  Data descriptors defined here:
 |
 |  __dict__
 |      dictionary for instance variables (if defined)
 |
 |  __weakref__
 |      list of weak references to the object (if defined)

4. Use type to create a class with attributes

type takes a dictionary to define attributes for the class, so

>>> Foo = type('Foo', (), {'bar':True})

It can be translated into:

'''
No one answers the questions? Xiaobian created a Python learning exchange QQ group: 579817333 
Looking for like-minded small partners, mutual help, there are good video learning tutorials and PDF ebooks in the group!
'''
>>> class Foo(object):
...       bar = True

Foo can also be used as a common class:

>>> print Foo
<class '__main__.Foo'>
>>> print Foo.bar
True
>>> f = Foo()
>>> print f
<__main__.Foo object at 0x8a9b84c>
>>> print f.bar
True

Of course, you can inherit from this class, so the following code:

>>> class FooChild(Foo):
...       pass

It can be written as:

>>> FooChild = type('FooChild', (Foo,),{})
>>> print FooChild
<class '__main__.FooChild'>
>>> print FooChild.bar   # The bar attribute is inherited by Foo
True

Be careful:

The second parameter of type, in tuple is the name of the parent class, not the string
The added property is a class property, not an instance property

5. Use type to create a class with methods

Eventually you will want to add methods to your classes. Just define a function with the appropriate signature and assign it as an attribute.

Add instance method

'''
//No one answers the questions? Xiaobian created a Python learning exchange QQ group: 579817333 
//Looking for like-minded small partners, mutual help, there are good video learning tutorials and PDF ebooks in the group!
'''
In [46]: def echo_bar(self): #A common function is defined
    ...:     print(self.bar)
    ...:

In [47]: FooChild = type('FooChild', (Foo,), {'echo_bar': echo_bar}) #Let the echo bar attribute in the FooChild class point to the function defined above

In [48]: hasattr(Foo, 'echo_bar') #Determine whether there is echo bar in Foo class
Out[48]: False

In [49]:

In [49]: hasattr(FooChild, 'echo_bar') #Determine whether there is echo bar in the FooChild class
Out[49]: True

In [50]: my_foo = FooChild()

In [51]: my_foo.echo_bar()
True

Add static method

In [36]: @staticmethod
    ...: def testStatic():
    ...:     print("static method ....")
    ...:

In [37]: Foochild = type('Foochild', (Foo,), {"echo_bar":echo_bar, "testStatic":
    ...: testStatic})

In [38]: fooclid = Foochild()

In [39]: fooclid.testStatic
Out[39]: <function __main__.testStatic>

In [40]: fooclid.testStatic()
static method ....

In [41]: fooclid.echo_bar()
True

Add class method

'''
No one answers the questions? Xiaobian created a Python learning exchange QQ group: 579817333 
Looking for like-minded small partners, mutual help, there are good video learning tutorials and PDF ebooks in the group!
'''
In [42]: @classmethod
    ...: def testClass(cls):
    ...:     print(cls.bar)
    ...:

In [43]:

In [43]: Foochild = type('Foochild', (Foo,), {"echo_bar":echo_bar, "testStatic":
    ...: testStatic, "testClass":testClass})

In [44]:

In [44]: fooclid = Foochild()

In [45]: fooclid.testClass()
True

You can see that in Python, classes are also objects, and you can create classes dynamically. This is what Python does behind the scenes when you use the keyword class, which is implemented through metaclasses.

6. What is metaclass (finally to the topic)

Metaclasses are "things" used to create classes. You create classes to create instance objects of classes, don't you? But we have learned that classes in Python are also objects.

Metaclasses are used to create these classes (objects). Metaclasses are classes of classes. You can understand them as follows:

MyClass = MetaClass() #Use metaclasses to create an object called a "class"
MyObject = MyClass() #Use class to create an instance object

You've seen that type lets you do things like this:

MyClass = type('MyClass', (), {})

This is because the function type is actually a metaclass. Type is the metaclass Python uses to create all classes. Now you want to know why all types are lowercase instead of type?

Well, I guess it's for consistency with str, which is the class used to create string objects, and int is the class used to create integer objects. Type is the class that creates the class object. You can see this by checking the class attribute. Everything in Python, notice, I mean everything - objects. This includes integers, strings, functions, and classes. They are all objects, and they are all created from a class called type.

'''
No one answers the questions? Xiaobian created a Python learning exchange QQ group: 579817333 
Looking for like-minded small partners, mutual help, there are good video learning tutorials and PDF ebooks in the group!
'''
>>> age = 35
>>> age.__class__
<type 'int'>
>>> name = 'bob'
>>> name.__class__
<type 'str'>
>>> def foo(): pass
>>>foo.__class__
<type 'function'>
>>> class Bar(object): pass
>>> b = Bar()
>>> b.__class__
<class '__main__.Bar'>

Now, what is the class attribute of any class?

>>> a.__class__.__class__
<type 'type'>
>>> age.__class__.__class__
<type 'type'>
>>> foo.__class__.__class__
<type 'type'>
>>> b.__class__.__class__
<type 'type'>

So metaclasses are the things that create objects like classes. type is Python's built-in metaclass. Of course, you can also create your own metaclass.

7. Metadata attribute

You can add the metaclass attribute to a class when you define it.

class Foo(object):
    __metaclass__ = something...
    ...ellipsis...

If you do, python uses metaclasses to create the Foo class. Be careful. There are some tricks in it. You write down the class Foo(object) first, but the class Foo has not been created in memory. Python will look for the metaclass attribute in the class definition. If it is found, python will use it to create the class Foo. If it is not found, it will use the built-in type to create the class. Read the following passage over and over again. When you write the following code:

class Foo(Bar):
    pass

Python does the following:

Is there a metadata attribute in Foo? If so, Python creates a class (object) named Foo through metaclass
If Python does not find the metaclass, it continues to look for the metaclass attribute in Bar (the parent class) and tries to do the same as before.

If Python can't find the metaclass in any of its parent classes, it will look for the metaclass at the module level and try to do the same.

If you still can't find metaclass,Python creates the class object with the built-in type.

The question now is, what code can you put in metaclass? The answer is: you can create something of a class. So what can be used to create a class? Type, or anything that uses type or subclass type.

8. Custom metaclass

The primary purpose of metaclasses is to automatically change classes when they are created. Usually, you do this for the API, and you want to be able to create classes that match the current context.

Imagine a silly example where you decide that the properties of all classes in your module should be in uppercase. There are several ways to do this, but one is by setting up metaclass at the module level. In this way, all classes in this module will be created through this metaclass. All we need to do is tell the metaclass to change all properties to uppercase.

Fortunately, metaclass can actually be called at will, and it doesn't need to be a formal class. So let's start with a simple function.

python2 medium

#-*- coding:utf-8 -*-
def upper_attr(future_class_name, future_class_parents, future_class_attr):

    #Traverse the property dictionary to capitalize property names that do not begin with \
    newAttr = {}
    for name,value in future_class_attr.items():
        if not name.startswith("__"):
            newAttr[name.upper()] = value

    #Call type to create a class
    return type(future_class_name, future_class_parents, newAttr)

class Foo(object):
    __metaclass__ = upper_attr #Set the metaclass of Foo class to upper "attr"
    bar = 'bip'

print(hasattr(Foo, 'bar'))
print(hasattr(Foo, 'BAR'))

f = Foo()
print(f.BAR)

python3 medium

'''
//No one answers the questions? Xiaobian created a Python learning exchange QQ group: 579817333 
//Looking for like-minded small partners, mutual help, there are good video learning tutorials and PDF ebooks in the group!
'''
#-*- coding:utf-8 -*-
def upper_attr(future_class_name, future_class_parents, future_class_attr):

    #Traverse the property dictionary to capitalize property names that do not begin with \
    newAttr = {}
    for name,value in future_class_attr.items():
        if not name.startswith("__"):
            newAttr[name.upper()] = value

    #Call type to create a class
    return type(future_class_name, future_class_parents, newAttr)

class Foo(object, metaclass=upper_attr):
    bar = 'bip'

print(hasattr(Foo, 'bar'))
print(hasattr(Foo, 'BAR'))

f = Foo()
print(f.BAR)

Now let's do it again, this time with a real class as the metaclass.

#coding=utf-8

class UpperAttrMetaClass(type):
    # __new is a special method called before init
    # __new? Is the method used to create an object and return it
    # And init is only used to initialize the passed in parameters to the object
    # You rarely use new unless you want to be able to control the creation of objects
    # Here, the created object is a class, and we want to be able to customize it, so we rewrite \__
    # You can also do something in init if you want
    # There are also some advanced uses that involve overriding the special method, but we don't use it here
    def __new__(cls, future_class_name, future_class_parents, future_class_attr):
        #Traverse the property dictionary to capitalize property names that do not begin with \
        newAttr = {}
        for name,value in future_class_attr.items():
            if not name.startswith("__"):
                newAttr[name.upper()] = value

        # Method 1: create class object by 'type'
        # return type(future_class_name, future_class_parents, newAttr)

        # Method 2: reuse the type. New method
        # That's basic OOP programming. No magic
        # return type.__new__(cls, future_class_name, future_class_parents, newAttr)

        # Method 3: use super method
        return super(UpperAttrMetaClass, cls).__new__(cls, future_class_name, future_class_parents, newAttr)

#The usage of python2
class Foo(object):
    __metaclass__ = UpperAttrMetaClass
    bar = 'bip'

# Usage of Python 3
# class Foo(object, metaclass = UpperAttrMetaClass):
#     bar = 'bip'

print(hasattr(Foo, 'bar'))
# Output: False
print(hasattr(Foo, 'BAR'))
# Output: True

f = Foo()
print(f.BAR)
# Output:'bip'

That's it. Besides that, there's really nothing else to say about metaclasses. But in terms of metaclasses themselves, they are very simple:

  • Creation of interception class
  • Modification class
  • Return the modified class
  • Why use metaclasses?

Now back to our main topic, why do you use such an error prone and obscure feature? Well, in general, you don't use it at all:

"Metaclass is the magic of depth, 99% of users should not worry about it at all. If you want to know if you need a metaclass, you don't need it. Those who actually use metaclasses know exactly what they need to do, and they don't need to explain why they use metaclasses at all. " ——Tim Peters, the leader of Python

Topics: Python Attribute Programming