Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multiple Consumers/Readers for different topics #1301

Open
avngzy opened this issue Jun 18, 2024 · 1 comment
Open

Multiple Consumers/Readers for different topics #1301

avngzy opened this issue Jun 18, 2024 · 1 comment

Comments

@avngzy
Copy link

avngzy commented Jun 18, 2024

Hi there, wondering if it is possible to create multiple readers/consumers?
From this issue #1200 it looks like only 1 reader is possible, but I couldn't find where in the documentation this was stated. Edit: Looks like I misread and 2 readers are allowed, but I'm not sure what I'm doing wrong

I'm trying to have 2 different consumer groups consuming from 2 different topics. I've tried creating 2 readers but keep getting connection issues(Connection refused) to the Broker(Amazon MSK) for the 2nd reader I'm creating. Previously when we only had a single consumer/reader, we were able to connect with no issues. Any references to existing documentations on suggestions on how to do so is appreciated!

Kafka Version

What version of kafka-go are you using?
0.4.40

To Reproduce

Resources to reproduce the behavior:

How I'm initialising reader

func (kReader *SegmentioKafkaReader) NewReader(brokers []string, topic, groupID string) error {
	// Check if its a valid reader config
	rc := kafkaGo.ReaderConfig{
		Brokers:  brokers,
		GroupID:  groupID,
		Topic:    topic,
		Dialer:   kafkaGo.DefaultDialer,
		MaxBytes: 10e6, // 10MB
	}
	if err := rc.Validate(); err != nil {
		errStr := fmt.Sprintf("kafka.NewConsumer(): invalid kafka reader config!, err %v", err)
		log.Printf(errStr)
		return err
	}
	kReader.r = kafkaGo.NewReader(rc)
	return nil
}

Errors I am getting

 Error connecting to kafka broker abc.kafka.us-west-2.amazonaws.com:9092. Error: failed to dial: failed to open connection to abc.kafka.us-west-2.amazonaws.com:9092: dial tcp: lookup abc.kafka.us-west-2.amazonaws.com on 10.3.0.10:53: dial tcp 10.3.0.10:53: connect: connection refused.

Expected Behavior

I should be able to create 2 readers with different consumer groups consuming from different topics

@avngzy
Copy link
Author

avngzy commented Jun 18, 2024

@petedannemann Not sure if you're still active but tagging in hopes you are and know the answer

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant