kafka-deb-packaging

Simple debian packaging for Apache Kafka and Zookeeper

Changelog

• 2015-Nov-18 : Updated for latest Kafka 0.8.2.2 and adapt for Debian
• 2015-Mar-18 : Updated for latest Kafka 0.8.2.1 and use sbt in system path.
• 2015-Mar-30 : Final Test, ready for git pull, packaged Zookeeper init.d!

Pre-requisites - Install fpm

$ sudo apt-get update
$ sudo apt-get install ruby-dev build-essential
$ sudo gem install fpm

Options

You can change the scala version and the kafka version in dist_kafka on new versions of both.

Usage

$ ./dist_kakfa.sh

Sample Output

~/git/kafka-deb-packaging$ ./dist_kafka.sh

--2015-03-30 17:39:01--  http://mirror.sdunix.com/apache/kafka/0.8.2.1/kafka_2.10-0.8.2.1.tgz
Resolving mirror.sdunix.com (mirror.sdunix.com)... 74.206.97.82
Connecting to mirror.sdunix.com (mirror.sdunix.com)|74.206.97.82|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 16162559 (15M) [application/x-gzip]
Saving to: ‘kafka_2.10-0.8.2.1.tgz’

100%[========================>] 16,162,559   340KB/s   in 44s

2015-03-30 17:39:45 (361 KB/s) - ‘kafka_2.10-0.8.2.1.tgz’ saved [16162559/16162559]

~/git/kafka-deb-packaging/tmp ~/git/kafka-deb-packaging
Created package {:path=>"kafka_0.8.2.1-5_all.deb"}
~/git/kafka-deb-packaging

Installation

$ dpkg -i kafka_0.8.2.1-10_all.deb

or if you have your own repo:

$ ~/gpg-agent-headless.sh
$ reprepro -b /var/repositories/ includedeb trusty $@
$ apt-get install kafka

Note: Installs and runs as user 'kafka'/'zookeeper'. Easy to change for your needs.

Post install

$ sudo update-rc.d zookeeper defaults 20

$ sudo update-rc.d kafka defaults 25

 Adding system startup for /etc/init.d/kafka ...
   /etc/rc0.d/K25kafka -> ../init.d/kafka
   /etc/rc1.d/K25kafka -> ../init.d/kafka
   /etc/rc6.d/K25kafka -> ../init.d/kafka
   /etc/rc2.d/S25kafka -> ../init.d/kafka
   /etc/rc3.d/S25kafka -> ../init.d/kafka
   /etc/rc4.d/S25kafka -> ../init.d/kafka
   /etc/rc5.d/S25kafka -> ../init.d/kafka

System Check

$ /usr/local/kafka/bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic installtest --partitions 1 --replication-factor 1

Created topic "installtest".

Tested Platforms

• Ubuntu 14.04LTS, Amazon EC2
• Debian Wheezy

---

Package info

Debian pkg: kafka_0.8.2.2-10_all.deb Version :

• kafka 8.2.2
• scala 2.10

Init scripts:

• /etc/init.d/zookeeper
• /etc/init.d/kafka

Configuration:

• /etc/kafka
  • consumer.properties
  • log4j.properties
  • producer.properties
  • server.properties
  • test-log4j.properties
  • tools-log4j.properties
  • zookeeper.properties
• /etc/default/kafka

Logs:

• /var/log/zookeeper-server.log
• /var/log/kafka/

Binaries:

• /usr/local/kafka/

Data:

• /usr/local/kafka/kafka-logs
• /usr/local/zookeeper/data

Network ports:

• Zookeeper : 2181
• Kafka : 9092

Users:

• Kafka : kafka
• Zookeeper : zookeeper

---

Kafka tools (Optional)

Install KafkaT (Simplified command-line administration for Kafka brokers.)

sudo aptitude install ruby ruby-dev build-essential

sudo gem install kafkat --source https://rubygems.org --no-ri --no-rdoc

Create a new configuration file to match your deployment.

vim /etc/kafkatcfg

This is a configuration file which KafkaT uses to determine the installation and log directories of your Kafka server. It should also point KafkaT to your ZooKeeper instance. Accordingly, add the following lines to it:

{
"kafka_path": "/usr/local/kafka",
"log_path": "/usr/local/kafka/kafka-logs",
"zk_path": "localhost:2181"
}

~# kafkat
kafkat 0.0.10: Simplified command-line administration for Kafka brokers
usage: kafkat [command] [options]

Here's a list of supported commands:

  brokers                                                             Print available brokers from Zookeeper.
  clean-indexes                                                       Delete untruncated Kafka log indexes from the filesystem.
  controller                                                          Print the current controller.
  elect-leaders [topic]                                               Begin election of the preferred leaders.
  partitions [topic]                                                  Print partitions by topic.
  partitions [topic] --under-replicated                               Print partitions by topic (only under-replicated).
  partitions [topic] --unavailable                                    Print partitions by topic (only unavailable).
  reassign [topic] [--brokers <ids>] [--replicas <n>]                 Begin reassignment of partitions.
  resign-rewrite <broker id>                                          Forcibly rewrite leaderships to exclude a broker.
  resign-rewrite <broker id> --force                                  Same as above but proceed if there are no available ISRs.
  set-replication-factor [topic] [--newrf <n>] [--brokers id[,id]]    Set the replication factor of
  shutdown <broker id>                                                Gracefully remove leaderships from a broker (requires JMX).
  topics                                                              Print all topics.