__new__ & __init__: Zero to Expert

Dive deeper into __new__ and __init__

---

Deep Dive into Class and Instance in Python

In Python, you can create an object from the class to represent specific data and perform several actions. The data is presented as variables, and the actions are methods.

The simple User class has two variables and two different methods. It has a name and age as data. Also, it can perform “introduce” and “hello” with its own data.

class User:
    def __init__(self, name, age):
        self.name = name
        self.age = age

    def introduce(self):
        return f"My name is {self.name} and I am {self.age} years old"

    def hello(self):
        return f"Hello, I am {self.name}"

You should know that this is just a template about the User.

The actual user should be created as an object with actual data. We usually call this an instance. An instance is an object made from the class with the actual data. Its shape is literally the same as the User class, but the difference is that it has the actual data. The methods are still tied to the class and are not copied to the instance.

Let’s create some instances from the User class. I made two instances with a different variable name and age values.

daniel = User("Daniel", 35)
emily = User("Emily", 20)

daniel.introduce()
emily.introduce()

# Result
--------------------------------
'My name is Daniel and I am 35 years old'
'My name is Emily and I am 20 years old'

Now, let’s check how the introduce function works internally. If you print the function without calling it, you can see which function is bound to it. As you can see from the result below, it is tied to the method User.introduce, and the object is 0x103be9eb0, which is the id of the Daniel object.

This means that I can call the function like this.

See? The same result came out.

What would happen if I changed the method in the class after creating the instance?

Let’s change the introduce method to return a different string. As you might guess, the instance method has been changed as well. This is simply because the instance method is actually bound to the class. A class function is called with the instance as a function parameter.

This is why we add self to the instance method. Internally, calling the instance method is the same as calling a class function with an instance as a parameter.

daniel.introduce() == User.introduce(daniel)

What is the __new__ method?

The object is made based on the class definition when you create the instance. Some might have learned that the __init__ method is used to create an instance, but it’s wrong. It is just the initialization method that is called after the instance is created. The actual method to create an instance is the __new__ method.

Let’s create the instance using the __new__ method.

class User:
    def __init__(self, name, age):
        self.name = name
        self.age = age

    def introduce(self):
        return f"My name is {self.name} and I am {self.age} years old"

    def hello(self):
        return f"Hello!!"

mark = User.__new__(User)
print(mark.__dict__)
print(type(mark))
print(isinstance(mark, User))
print(mark.hello())


# Result
--------------------------------
{}
<class '__main__.User'>
True
Hello!!

The instance named mark was created successfully but has no data inside the __dict__ variable. However, the hello function is available because it is an instance of the User class. If you want to give it a name and age, you should call the __init__ method. This is how Python creates the instance and initializes the data of that instance.

How does __new__ and __init__ works internally

When you create a class, it actually inherits the object type. Object type is the base class that CPython manages. So, when you make a new type by making the class, it inherits an object and calls it an object.__new__ method.

First, when you create the class, it actually calls the type() function internally. The type function is used to create a custom type like a class. If the type is well created, you create an instance with that type, which is the class here.

In CPython, type_call is the function called when you create an instance with a class type like User(). In this function, type→tp_new is called. After that, type→tp_init is called, and the object is passed as a parameter.

static PyObject *
type_call(PyObject *self, PyObject *args, PyObject *kwds)
{
    PyTypeObject *type = (PyTypeObject *)self;
    PyObject *obj;

    //(... skipped ...) 

    obj = type->tp_new(type, args, kwds);
    obj = _Py_CheckFunctionResult(tstate, (PyObject*)type, obj, NULL);
    if (obj == NULL)
        return NULL;

    //(... skipped ...) 

    type = Py_TYPE(obj);
    if (type->tp_init != NULL) {
        int res = type->tp_init(obj, args, kwds);
        
        //(... skipped ...) 
    }
    return obj;
}

You can ensure that the tp_init is called after creating the object with the tp_new function. Now, we need to find out the actual code of tp_new for the object type.

Here’s the CPython code that defines the object type. The tp_new function is mapped with the object_new function.

PyTypeObject PyBaseObject_Type = {
    PyVarObject_HEAD_INIT(&PyType_Type, 0)
    "object",                                   /* tp_name */
    sizeof(PyObject),                           /* tp_basicsize */
    // ( ... skip ... )
    object_init,                                /* tp_init */
    object_new,                                 /* tp_new */
};

Let’s examine the object_new function to discover what happens inside the __new__ method.

static PyObject *
object_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
{
    // (...skipped...)

    PyObject *obj = type->tp_alloc(type, 0);
    if (obj == NULL) {
        return NULL;
    }
    return obj;
}

“It just allocates the object's memory.”

Now, you can assume that no matter what type your class type inherits, you should make it call the object.__new__ method to allocate the memory for the class. If you do not call the object.__new__ method, then it does not call __init__ method because type_call checks the return object type.

You can test this by customizing the __new__ method for the base type. Here’s an example. I made two classes and inherited A from B. When super().__new__(cls) line is executed, it calls the __new__ method in class A. But the __new__ method of class A returns the string type. So, the __init__ method is not called.

Summary

Here’s the overall process for creating the class and the instance. Remember that the __init__ method is just for initializing the instance. The __new__ method creates the instance itself in memory.