Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Different per-topic timeout configuration #1671

Open
2 of 7 tasks
swehner opened this issue Nov 13, 2023 · 1 comment · May be fixed by #1695
Open
2 of 7 tasks

Different per-topic timeout configuration #1671

swehner opened this issue Nov 13, 2023 · 1 comment · May be fixed by #1695

Comments

@swehner
Copy link

swehner commented Nov 13, 2023

Description

I'm looking to use a single Kafka producer for two slightly differnt use cases/message types. One that is just fire-and-forget and that will simply enqueue a message without wait for any acknowledgment.
For the second use case I'd like to make sure that the message was written to broker before returning a response - similar to what's described in #1174

It seems that the way to achieve this would be to set a low message timeout and wait for poll() to acknowledge message delivery via the callback.

Having a low global message timeout wouldn't be very good for the fire-and-forget use cause, so it seems that the best option would be to use the per-topic configuration.
Was there any progress on implementing this since #328 (comment) ? It looks like none of the APIs are exposed through the python client.

If there's still no way to provide per-topic config I'd be open to work on it and make a pull request, but would like to understand what the preferred approach is. maybe exposing the topic creation methods on the produce with a config dict would be enough?

Thanks in advance

Checklist

Please provide the following information:

  • confluent-kafka-python==2.20 librdkafka==2.2.0
  • Apache Kafka broker version: 3.4.0
  • Client configuration: {...}
  • Operating system:
  • Provide client logs (with 'debug': '..' as necessary)
  • Provide broker log excerpts
  • Critical issue
@swehner swehner changed the title Different topic timeout configration Different per-topic timeout configuration Nov 13, 2023
@swehner
Copy link
Author

swehner commented Dec 26, 2023

I've created a pull-request for exposing the topic_new to be able to set a custom configuration here: #1695

Let me know if that approach works for you

@swehner swehner linked a pull request Jan 2, 2024 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant