Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Expose partitioners to python #1759

Open
6 tasks
pavelschon opened this issue Jun 3, 2024 · 0 comments
Open
6 tasks

Expose partitioners to python #1759

pavelschon opened this issue Jun 3, 2024 · 0 comments

Comments

@pavelschon
Copy link

Description

We have use case where we need to manually calculate partition ID using the murmur2 partitioner from the librdkafka.
Rationale: we need to verify that messages received belongs to the right partition.

Since the desired function is not exposed to python via confluent_kafka, we have a workaround with using ctypes, see below.
But we still think the right approach is to use desired function (if available) via confluent_kafka module.

How to reproduce

librdkafka = ctypes.cdll.LoadLibrary("librdkafka.so.1")

assert librdkafka.rd_kafka_msg_partitioner_murmur2(None, key, len(key), num_partitions, None, None) == expected_partition

Checklist

Please provide the following information:

  • confluent-kafka-python and librdkafka version (confluent_kafka.version() and confluent_kafka.libversion()): 2.3.0
  • Apache Kafka broker version: ?
  • Operating system: -
  • Provide client logs (with 'debug': '..' as necessary) -
  • Provide broker log excerpts -
  • Critical issue -
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants