1. 介绍

之前在kafka manager使用教程这篇文章介绍了如何安装使用kafka manager。然而,在实际使用过程中,kafka manager提供的监控内容还不够满足生产环境下所需的监控要求。至于原因,大家可以参考我下面这篇重要参考资料(强烈推荐看一下,因为是从linkedin工程师那里翻译过来的,很有参考价值):

重要参考资料:LinkedIn开源Kafka Monitor

2. 安装

2.1 安装gradle和jdk

,jdk安装这里不废话了。安装gradle可以去官网下载安装文件。下载好后到服务器上解压,并且配置环境变量,并且用source命令生效。

# gradle
export GRADLE_HOME=/root/Downloads/gradle-3.1
PATH=$GRADLE_HOME/bin:$PATH

gradle -v来验证下

2.2 构建kafka-monitor工程

到$GRADLE_HOME下执行gradle jar来构建工程

2.3 新建topic

新建一个kafka-monitor需要使用的topic,名字为kmf-topic

可以用kafka manger来新建topic也可以用命令:

bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic kmf-topic

2.4 修改配置文件

vi $GRADLE_HOME/config/kafka-monitor.properties

主要修改下zk地址和broker地址

{
  "end-to-end": {
    "class.name": "com.linkedin.kmf.tests.BasicEndToEndTest",
    "topic": "kmf-topic",
    "zookeeper.connect": "localhost:2181",
    "bootstrap.servers": "localhost:9092",
    "produce.record.delay.ms": 100,
    "produce.producer.props": {
      "client.id": "kmf-client-id"
    },

    "consume.latency.sla.ms": "20000",
    "consume.consumer.props": {

    }
  },

  "reporter-service": {
    "class.name": "com.linkedin.kmf.services.DefaultMetricsReporterService",
    "report.interval.sec": 1,
    "report.metrics.list": [
      "kmf.services:type=produce-service,name=*:produce-availability-avg",
      "kmf.services:type=consume-service,name=*:consume-availability-avg",
      "kmf.services:type=produce-service,name=*:records-produced-total",
      "kmf.services:type=consume-service,name=*:records-consumed-total",
      "kmf.services:type=consume-service,name=*:records-lost-total",
      "kmf.services:type=consume-service,name=*:records-duplicated-total",
      "kmf.services:type=consume-service,name=*:records-delay-ms-avg",
      "kmf.services:type=produce-service,name=*:produce-error-rate",
      "kmf.services:type=consume-service,name=*:consume-error-rate"
    ]
  },

  "jetty-service": {
    "class.name": "com.linkedin.kmf.services.JettyService",
    "jetty.port": 8000
  },

  "jolokia-service": {
    "class.name": "com.linkedin.kmf.services.JolokiaService"
  }
}

2.5 启动kafka-monitor