Missing log messages when closing the program

[sealor]

Description

The Producer and Consumer can use Python logging via the logger configuration option. This works well as long as the program is running. Unfortunately, the last log messages get lost when the program exits. Those messages can only be seen if the logger feature is not used.

How to reproduce

logger.py:

import logging
import sys
from confluent_kafka import Producer, Consumer
import confluent_kafka

logger = logging.getLogger("kafka")
logger.setLevel(logging.DEBUG)
logger.addHandler(logging.StreamHandler())

c = Consumer({
    "bootstrap.servers": "not-available-broker",
    "group.id": "my-group",
    "debug": "msg",
    "logger": logger if sys.argv[-1] == "logger" else None,
})
c.poll(0)
c.close()
print("version:", confluent_kafka.version(), "lib-version:", confluent_kafka.libversion())

output with logger:

$ ./logger.py logger
INIT [rdkafka#consumer-1] [thrd:app]: librdkafka v2.3.0 (0x20300ff) rdkafka#consumer-1 initialized (builtin.features gzip,snappy,ssl,sasl,regex,lz4,sasl_plain,sasl_scram,plugins,zstd,sasl_oauthbearer,http,oidc, STRIP STATIC_LINKING GCC GXX PKGCONFIG INSTALL GNULD LIBDL PLUGINS ZLIB SSL ZSTD CURL HDRHISTOGRAM SYSLOG SNAPPY SOCKEM SASL_SCRAM SASL_OAUTHBEARER OAUTHBEARER_OIDC CRC32C_HW, debug 0x40)
version: ('2.3.0', 33751040) lib-version: ('2.3.0', 33751295)

output without logger:

$ ./logger.py
%7|1709646389.068|INIT|rdkafka#consumer-1| [thrd:app]: librdkafka v2.3.0 (0x20300ff) rdkafka#consumer-1 initialized (builtin.features gzip,snappy,ssl,sasl,regex,lz4,sasl_plain,sasl_scram,plugins,zstd,sasl_oauthbearer,http,oidc, STRIP STATIC_LINKING GCC GXX PKGCONFIG INSTALL GNULD LIBDL PLUGINS ZLIB SSL ZSTD CURL HDRHISTOGRAM SYSLOG SNAPPY SOCKEM SASL_SCRAM SASL_OAUTHBEARER OAUTHBEARER_OIDC CRC32C_HW, debug 0x40)
%7|1709646389.068|DESTROY|rdkafka#consumer-1| [thrd:app]: Terminating instance (destroy flags none (0x0))
%7|1709646389.068|DESTROY|rdkafka#consumer-1| [thrd:main]: Destroy internal
%7|1709646389.068|DESTROY|rdkafka#consumer-1| [thrd:main]: Removing all topics
%3|1709646389.114|FAIL|rdkafka#consumer-1| [thrd:not-available-broker:9092/bootstrap]: not-available-broker:9092/bootstrap: Failed to resolve 'not-available-broker:9092': Temporary failure in name resolution (after 46ms in state CONNECT)
version: ('2.3.0', 33751040) lib-version: ('2.3.0', 33751295)

Expected behaviour

Both outputs should be similar.

Consumer.close() should really terminate the consumer. The Producer should provide a close() method to terminate it cleanly.

Checklist

Please provide the following information:

• confluent-kafka-python and librdkafka version (confluent_kafka.version() and confluent_kafka.libversion()):
• Apache Kafka broker version: not relevant
• Client configuration: {...}
• Operating system: Ubuntu 22.04
• Provide client logs (with 'debug': '..' as necessary)
• Provide broker log excerpts
• Critical issue

[pranavrth]

When the consumer is closed, internally librdkafka closes the consumer first and then destroys the client. The logs related to consumer close method is being logged properly and logs related to destroy method are not logged. The logs are added to the queue in destroy as well but not getting logged properly. We will need to debug this more.