Refined Object Model and Python Bindings

May 11, 2020

Hi everyone,

In the past two months, as I was trying to analyze and share information about the state of the current pandemic, I've started to (re-)learn two well-known scientific visualization libraries: d3 in Javascript, and matplotlib in Python.

Learning new libraries is always a good source of inspiration when you're writing a library yourself, and it prompted me to finally take the time to properly fix an annoying problem I was having with the Python bindings of VGC: memory management of tree data structures.

Description of the change (a bit technical)

In programming terminology, a "tree" simply refers to a hierarchy of objects, like an organization of folders in your hard drive, or a family tree. There is a "root" object (the unique ancestor of all objects in the tree), and this root can have one or more children, which can themselves have one or more children, etc.

None of this is really complicated to implement in C++ alone, or in Python alone, but it is a bit tricky in our case where the implementation is in C++ but users can create, access, and modify the tree in Python too. The reason it is tricky is that Python and C++ have fundamentally different ways to manage memory: Python has what is called a "garbage collector", while C++ has not. A garbage collector is a mechanism which automatically releases the memory taken by an object once it isn't accessible anymore, that is, when no variables point to the object anymore.

The problem in our case is that because the data structure is implemented in C++, this means that Python isn't aware of the internal references of the tree, which doesn't play well with its garbage collector. The previous solution in VGC was to give Python "shared ownership" of all objects in the tree, which is quite a standard solution when writing Python bindings of C++ code, but this had the side effects that the C++ parent of each object isn't anymore its "unique owner", which creates all sorts of problems, such as making it harder or impossible to enforce that some desirable properties stay true (= "invariants" in programming terminology).

The new solution which I have just finished to implement is a more refined memory ownership model, where Python is only given a "weak reference" to objects in the tree, which doesn't guarantee that the referenced object itself stays alive, but guarantees instead that the root of this object stays alive. If a Python script attempts to access an object which doesn't exist anymore, then a Python exception is raised.

This was quite a significant change, which took a long time to design, and around 1,500 lines of code were added or modified. It doesn't change anything to the user interface, but it was one of the few important pieces of architecture that was yet to be done, and I'm glad this is over.

What's next?

Up to May 31, I will actually spend 100% of my time making a short video explaining in an intuitive way some of the technology behind VPaint/VGC, which should be featured in the end credits of the Hand Drawn Documentary (if all goes well). Hopefully, this might help a few animators to be aware of this project :-)

I should then go back to developing VGC in June.

As always, thanks a lot for your support!

Boris