1. 介绍

虽然kafka提供了一些工具,但是都比较简陋。近期有个需求就是希望动态调整某个topic的replicas数量。虽然网上有一些方法,但是都比较繁琐。今天就推荐一款linkedin开源的工具kafka-tools

2. 安装

准备工作:
In addition, you will need to run it on a host that has the following:

A copy of the Kafka admin tools (including kafka-reassign-partitions.sh)
Access to the Zookeeper ensemble for the cluster
SSH access to the Kafka brokers (with credentials preferably loaded into ssh-agent)

# 安装python,如果有了就不必安装
yum install python
# 安装pip(下载安装脚本执行即可),教程官网:https://pip.pypa.io/en/stable/installing/
wget https://bootstrap.pypa.io/get-pip.py
# 执行安装脚本
python get-pip.py
# 安装依赖
yum install openssl-devel
yum install libssl-dev
yum install gcc
yum install python-devel
yum install libffi-devel
# 安装依赖的python模块,可以用豆瓣源
pip install  -i https://pypi.douban.com/simple paramiko
pip install  -i https://pypi.douban.com/simple kazoo
pip install  -i https://pypi.douban.com/simple kafka-tools

#更新下setuptools
pip install --upgrade setuptools

3. 使用

修改某个topic的replications

# 动态调整replicas
kafka-assigner -z localhost:2181 -e set-replication-factor --topic __consumer_offsets --replication-factor 3
# 动态平衡所有分区(根据数量和大小)
kafka-assigner -z zook.example.com:2181 -e balance --types count size

运行命令效果如下: