1.首先安装zookeeper 启动zookeeper
bin/zkServer.sh start

2.kafka集群部署

# 修改配置文件
vi server.properties  

broker.id=1  
delete.topic.enable=true  
log.dirs=/bigdata/kafka_2.11-0.10.2.1/data
zookeeper.connect=node-1.xiaoniu.com:2181,node-2.xiaoniu.com:2181,node-3.xiaoniu.com:2181

分发带三台机器
scp -r kafka/ hadoop02:$PWD
scp -r kafka/ hadoop02:$PWD

修改另外两台其中的配置
broker.id=2
broker.id=3

**3.启动kafka **

bin/kafka-server-start.sh -daemon /root/apps/kafka_2.11-0.11.0.3/config/server.properties

4.停止kafka

/bigdata/kafka_2.11-0.10.2.1/bin/kafka-server-stop.sh 

5.创建topic

/bigdata/kafka_2.11-0.10.2.1/bin/kafka-topics.sh --create --zookeeper node-1.xiaoniu.com:2181,node-2.xiaoniu.com:2181,node-3.xiaoniu.com:2181 --replication-factor 3 --partitions 3 --topic my-topic

6.列出所有topic

bin/kafka-topics.sh --list --zookeeper hadoop6:2181,hadoop7:2181,hadoop8:2181

7.查看某个topic信息

/bigdata/kafka_2.11-0.10.2.1/bin/kafka-topics.sh --describe --zookeeper node-1.xiaoniu.com:2181,node-2.xiaoniu.com:2181,node-3.xiaoniu.com:2181 --topic my-topic

8.启动一个命令行的生产者

/bigdata/kafka_2.11-0.10.2.1/bin/kafka-console-producer.sh --broker-list node-1.xiaoniu.com:9092,node-1.xiaoniu.xom:9092,node-3.xiaoniu.com:9092 --topic xiaoniu

9.启动一个命令行的消费者

/bigdata/kafka_2.11-0.10.2.1/bin/kafka-console-consumer.sh --zookeeper node-1.xiaoniu.com:2181,node-2.xiaoniu.com:2181,node-3.xiaoniu.com:2181 --topic my-topic --from-beginning

消费者连接到borker的地址

/bigdata/kafka_2.11-0.10.2.1/bin/kafka-console-consumer.sh --bootstrap-server node-1.xiaoniu.com:9092,node-2.xiaoniu.com:9092,node-3.xiaoniu.com:9092 --topic xiaoniu --from-beginning 

Kafka Connect:
https://kafka.apache.org/documentation/#connect
http://docs.confluent.io/2.0.0/connect/connect-jdbc/docs/index.html

Kafka Stream:
https://kafka.apache.org/documentation/streams
https://spark.apache.org/docs/1.6.1/streaming-kafka-integration.html

kafka monitor:
https://kafka.apache.org/documentation/#monitoring
https://github.com/quantifind/KafkaOffsetMonitor
https://github.com/yahoo/kafka-manager

kafka生态圈：
https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem