Hybrid C++ Pybind11/Panel integration Destructor issues.

[tstrutz]

I am experiencing an issue with a hybrid C++ and Python integrated app using Panel. More information below.

ALL software version info

VCPKG
GCC 11.4.0 (C++ 17 Requirement)
Ninja
CMake 3.22.1
Python 3.10
Numpy 1.24.4
Panel 1.4.2
Bokeh 3.4.1
Param 2.1.0
Pybind11 C++ (Latest)
WSL Ubuntu 22.04

Please also set VCPKG_ROOT to the location of your installed VCPKG to test this. We use that in a CMakePresets file.

Description of expected behavior and the observed behavior

Application works as expected, but cleanup DOES not. I create a C++ object in C++ using python bindings generated using pybind11. This object is passed to the my Panel Dashboard, and it's state is used to populate several features of the dashboard. This dashboard is a real-time visualization tool for an algorithm written in C++.

Further testing has indicated that destructors on several of my Panel objects are not getting called. My Dashboard view contains a tab that the destructor never gets called on. The full version contains several tabs, plots, a controller, and a data model populated by the C++ object. All of this is stripped out for simplicity.

There is some cleanup such as closing output files that often occurs when the destructors get called in the full application, but this is not happening due to this bug.

Complete, minimal, self-contained example code that reproduces the issue

Basic Dashboard and C++ Application Launcher

main.py

"""This module contains an abstract class to act as a template for creating new pages
"""
import abc
import panel as pn
import param
import json
import os
import webbrowser
import holoviews as hv
import cpp_app_test
from bokeh.server.server import Server

pn.config.notifications = True
pn.config.theme = 'dark'

# Allow main page to resize adaptively
pn.extension("tabulator", "plotly", sizing_mode="stretch_width", notifications=True)
hv.extension("bokeh")
pn.extension(nthreads=8)

class BasePage_Meta(type(abc.ABC), type(param.Parameterized)):
    """Metaclass that inherits from the ABC and Parameterized metaclass.
    """

class BasePage(abc.ABC, param.Parameterized, metaclass=BasePage_Meta):
    """Abstract class for all classes pertaining to creating pages for the
    Dashboards. This requires each child class to instantiate their own version
    of the `update()` and `view()` methods.

    Parameters
    ----------

    controller : Controller
        See `_controller` in Attributes.
    parent : PanelPageBase
        Reference to this parents panel.

    Attributes
    ----------

    _controller : Controller
        Reference to the application controller.

    """
    def __init__(self, parent=None, **params) -> None:
        super().__init__(**params)
        self._parent = parent

    @abc.abstractmethod
    def view(self) -> pn.Column | pn.Row:
        """Returns a panel view containing the content to be displayed on this page.

        Returns
        ---------------

        pn.Column
            Panel column containing the content to be displayed on this page.
        """

class MainPanel(BasePage):
    """Main Panel for dashboard.

    Attributes
    ----------
    _sidebar : Sidebar
        Collapsable sidebar panel containing plotting controls.

    Parameters
    ----------
    app : App
        Reference to Python-Bound C++ app.
    """
    def __init__(self, app, **params):
        super().__init__(**params)

        self._app = app

    def __del__(self):
        print("Delete main page")

    def serve(self) -> pn.Template:
        """ Starts and makes available the panel as a web application.

        Returns
        -------
        Template
            Panel template representing main panel view.
        """
        template = pn.template.FastListTemplate(
            busy_indicator=None,
            collapsed_sidebar=False,
            header_background="#222829",
            main = self.view(),
            sidebar = None,
            sidebar_width=250,
            theme_toggle=True,
            title="Test Dashboard"
        )

        return template.servable()

    def view(self) -> pn.Tabs:
        """ Render panel layout.

        Returns
        -------
        tabs : pn.Tabs
            Panel tabs object.
        """
        tabs = pn.Tabs(dynamic=True)
        # Removed previously added tabs here.
        tabs.extend([
            ("Home", Home(self._app).view()),
        ])

        return tabs

class Home(BasePage):
    """Class that contains the contents and layout for the dashboard.

    Parameters:
    -----------

    controller : `Controller`
        Application controller (Replaced with app to test issue)

    Attributes:
    -----------

    open_doc_button : pn.widgets.Button
        Button to open documentation in separate tab.
    """

    def __init__(self, app, **params) -> None:
        super().__init__(**params)
        self._app = app
        self.__create_widgets()

    def __del__(self):
        print("Delete home page")

    def __create_widgets(self):
        """ Creates Page Widgets."""
        self.open_doc_button = pn.widgets.Button(
            name="Open User Manual",
            button_type="primary",
            on_click=self.open_documentation,
            width=250
            )

    def open_documentation(self, event):
        """Function callback that activates once the 'Open Documentation' button is clicked.
            Upon click, user documentation is launched in the browser.

        Parameters
        ----------

        event
            Signals that button click has occurred prompting app to open the user documentation
            page.
        """
        file = os.path.abspath("../ReferenceDocs/doc_page.html")
        return webbrowser.open(f"file://{file}")

    def view(self) -> pn.Column:
        """View function that houses and displays the components of the dashboard
        homepage.

        Returns
        --------

        pn.Column
            Object encapsulates all the pages' components to be rendered on the server.
        """
        header = pn.pane.Markdown(
            """
            ## Home
            Documentation of what this dashboard does here.
            """
        )

        pages_paragraph = pn.pane.Markdown(
            """
            ## Pages

            * __Home__: Serves as homepage and offers quick reference for documentation.
            * __Plots__: Displays plots generated by the sidebar controls.
            """
        )

        open_doc_row = pn.Column(pn.Row(self.open_doc_button))

        return pn.Column(
            header,
            open_doc_row,
            pages_paragraph
        )

class AppLauncher:
    """This class creates an application launcher that will create the Panel
    application as well as manage running any given dashboard.

    Parameters
    ----------
    app : cpp_app_test.Application
        See `_app` in Attributes.

    Attributes
    ----------
    _app : cpp_app_test.Application
        Python-Bound C++ Application to run.
    _main_panel : MainPanel
        Main class for real-time Dashboard.
    _dashboard : pn.viewable.Viewable
        Viewable representation of the dashboard.
    _server : bokeh.server.server.Server
        Bokeh dashboard server.
    """
    def __init__(self, app: cpp_app_test.Application) -> None:
        self._app: cpp_app_test.Application = app

        self._main_panel: MainPanel = MainPanel(self._app)
        self._dashboard: pn.viewable.Viewable = pn.panel(self._main_panel.serve(), width=1920)
        self._server: Server = None

    def run(self):
        """Starts the C++ app (normally) and Python dashboard. When the C++
        application exits, the dashboard will automatically stop.

        Parameters
        ----------
        args : argparse.Namespace
            Argparse container containing command-line parsed arguments.
        """
        # Server is running in a separate thread.
        self._server = self._dashboard.show(open=False, threaded=True, port=5000, address="0.0.0.0")

        # Wait here for application to exit.
        self._app.run()

        # Stop server when C++ exits via signal handler.
        self._server.stop()

if __name__ == "__main__":
    print("Creating C++ application")

    # Pybinding object.
    app = cpp_app_test.Application()

    print("C++ application created.")

    launcher = AppLauncher(app)
    launcher.run()

    print("Exiting interpreter. Destructors should be called after this point!")

Stack traceback and/or browser JavaScript console output

Creating C++ application
Constructed application.
C++ application created.
Launching server at http://0.0.0.0:5000
^CCaught signal 2, shutting down...
Exiting interpreter. Destructors should be called after this point!

C++ Code
app.hpp

#include <atomic>

class Application
{
    public:
        Application();

        virtual ~Application();

        void stop();

        void run();

    private:

        std::atomic<bool> m_running{false};
};

app.cpp

#include <app.hpp>

#include <functional>
#include <csignal>
#include <iostream>

// Put signal handling in unnamed namespace to make it local
namespace
{
    // The signal callback implementation
    std::function<void(int)> shutdownHandler;

    /**
     * @brief The signal callback to shutdown
     *
     * @param signal The signal number
     */
    void signalHandler(int signal)
    {
        // If the function was assigned at this point, then call it
        if (shutdownHandler)
        {
            shutdownHandler(signal);
        }
    }
} // namespace

Application::Application()
{
    std::cout << "Constructed application." << std::endl;
}

Application::~Application()
{
    std::cout << "Destroyed application" << std::endl;
}

void Application::stop()
{
    m_running.store(false);
}

void Application::run()
{
    // Setup signal handler for interrupt.

    // Set signal handling callback to capture Application object using this.
    shutdownHandler = [this](int signal)
    {
        std::cout << "Caught signal " << signal << ", shutting down..." << std::endl;
        try
        {
            stop();
        }
        catch(std::runtime_error & e)
        {
            std::cout << "Exception thrown in signal handler " << e.what() << std::endl;
        }
    };
    // Setup signal handling only once application is initialized. Otherwise, application will stall.
    std::signal(SIGINT, signalHandler); // For CTRL + C
    std::signal(SIGTERM, signalHandler);
    std::signal(SIGKILL, signalHandler);

    // Start an infinite loop. Will be interrupted when user interrupts the application and exit the app and the
    // dashboard server on python side.
    m_running.store(true);

    while (m_running.load())
    {

    }
}

Python binding code
app_pybind.cpp

#include <iostream>
#include <csignal>
#include <functional>
#include <atomic>

#include <pybind11/pybind11.h>
#include <pybind11/stl.h>

#include <app.hpp>

namespace py = pybind11;

PYBIND11_MODULE(cpp_app_test, m)
{
    // Same binding code
    py::class_<Application> app(m, "Application");
    app.def(py::init<>());
    app.def("stop", &Application::stop);
    app.def("run", &Application::run, py::call_guard<py::gil_scoped_release>());
}

Build Stuff

CMakePresets.json

{
    "version": 3,
    "configurePresets": [
        {
            "name": "default",
            "inherits": "online",
            "generator": "Ninja Multi-Config"
        },
        {
            "name": "online",
            "binaryDir": "${sourceDir}/build",
            "toolchainFile": "$env{VCPKG_ROOT}/scripts/buildsystems/vcpkg.cmake"
        }
    ],
    "buildPresets": [
        {
            "name": "Release",
            "jobs": 32,
            "configurePreset": "default",
            "configuration": "Release",
            "targets": ["install"]
        },
        {
            "name": "Debug",
            "configurePreset": "default",
            "configuration": "Debug",
            "targets": ["install"]
        },
        {
            "name": "RelWithDebInfo",
            "configurePreset": "default",
            "configuration": "RelWithDebInfo",
            "targets": ["install"]
        }
    ]
}

vcpkg.json

{
    "name": "panel_test",
    "version": "1.0.0",
    "description": "Test for Panel Bug",
    "homepage": "",
    "dependencies": [
      "nlohmann-json",
      "pybind11",
      {
        "name": "vcpkg-cmake",
        "host": true
      },
      {
        "name": "vcpkg-cmake-config",
        "host": true
      },
      {
        "name": "vcpkg-tool-ninja"
      }
    ]
  }

CMakeLists.txt

cmake_minimum_required(VERSION 3.22)

project(panel_test VERSION 0.0.0)

set(CXX_STANDARD_REQUIRED 17)

find_package(Python REQUIRED COMPONENTS Interpreter Development)
find_package(pybind11 CONFIG REQUIRED)

# Python method:
pybind11_add_module(cpp_app_test SHARED THIN_LTO OPT_SIZE app_pybind.cpp app.cpp)

target_include_directories(cpp_app_test PUBLIC ${CMAKE_CURRENT_SOURCE_DIR})

# Just put binaries in the same folder as the python module.
set (CMAKE_INSTALL_PREFIX ${CMAKE_SOURCE_DIR})

install(TARGETS cpp_app_test
        RUNTIME DESTINATION .
        ARCHIVE DESTINATION .
        LIBRARY DESTINATION .)

How to build the minimum viable code

Create and place all files within the same folder. I can upload a zip if that makes it easier.

cmake --preset default
cmake --build --preset Release

To run

Run the following in the same folder the files are in.

python main.py 

Notice, how python exits and the application does NOT call the C++ destructor.

Screenshots or screencasts of the bug in action

• I may be interested in making a pull request to address this

[tstrutz]

I may have figured out what was causing this. It looks like Panel is holding onto Panel resources even after the Python interpreter exits. Maybe a leak of some kind?

Anyways, I found that running pn.state.kill_all_servers() as the last line in main.py eliminates all of the problems here. I also moved all of the variables to a def main function so that there are zero global variables other than the app object. The weird thing with this is I am stopping the server. It shouldn't be running, unless the state is still left behind?

Why doesn't panel automatically do this clean up operation when Python exits? It's probably a good idea.

def main(app):
    # Pybinding object.
    launcher = AppLauncher(app)
    launcher.run()

if __name__ == "__main__":
    print("Creating C++ application")

    app = cpp_app_test.Application()

    print("C++ application created.")

    main(app)

    print("Exiting interpreter. Destructors should be called after this point!")

    pn.state.kill_all_servers()

produces the output:

Creating C++ application
Constructed application.
C++ application created.
Launching server at http://0.0.0.0:5000
^CCaught signal 2, shutting down...
Delete main page
Exiting interpreter. Destructors should be called after this point!
Delete home page
Destroyed application

I've noticed that the full application which contains a Sidebar and a Controller still doesn't work though. I may just reduce the number of references to self_app to fix this.