在MacOS上搭建Kafka

Kafka是流式计算中重要的数据源，我分享一下在本机MacOS上搭建Kafka的经验。

使用的是 Homebrew (https://brew.sh/)，一条指令就可以进行安装、卸载、更新、查看、搜索等功能。

一、直接使用brew安装kafka

命令如下，

brew install kafka

注：M1 Mac 需使用命令 brew install -s kafka 安装

运行过程显示为：

Updating Homebrew...
==> Auto-updated Homebrew!
Updated 1 tap (homebrew/core).
==> Updated Formulae
exploitdb                               frpc                                    frps                                    rav1e
==> Installing dependencies for kafka: zookeeper
==> Installing kafka dependency: zookeeper
==> Downloading https://homebrew.bintray.com/bottles/zookeeper-3.4.14.mojave.bottle.tar.gz
==> Downloading from https://akamai.bintray.com/e4/e4cc87d3dc3d2e406fbc262b0b98bea4b8ab2464ca17c24b98abc92a055a4454?__gda__=exp=1578065413~hmac=3e478264b3eda3
######################################################################## 100.0%
==> Pouring zookeeper-3.4.14.mojave.bottle.tar.gz
Error: The `brew link` step did not complete successfully
The formula built, but is not symlinked into /usr/local
Could not symlink include/zookeeper
/usr/local/include is not writable.
You can try again using:
  brew link zookeeper
==> Caveats
To have launchd start zookeeper now and restart at login:
  brew services start zookeeper
Or, if you don't want/need a background service you can just run:
  zkServer start
==> Summary
🍺  /usr/local/Cellar/zookeeper/3.4.14: 430 files, 36.5MB
==> Installing kafka
==> Downloading https://homebrew.bintray.com/bottles/kafka-2.3.1.mojave.bottle.tar.gz
==> Downloading from https://akamai.bintray.com/0d/0d3bdac93bd2602f0a7ee444fa77081362da52ea54e3d66488904a67128b7afd?__gda__=exp=1578065677~hmac=e5d9ede6b9dac6
######################################################################## 100.0%
==> Pouring kafka-2.3.1.mojave.bottle.tar.gz
==> Caveats
To have launchd start kafka now and restart at login:
  brew services start kafka
Or, if you don't want/need a background service you can just run:
  zookeeper-server-start /usr/local/etc/kafka/zookeeper.properties & kafka-server-start /usr/local/etc/kafka/server.properties
==> Summary
🍺  /usr/local/Cellar/kafka/2.3.1: 171 files, 59.5MB
==> Caveats
==> zookeeper
To have launchd start zookeeper now and restart at login:
  brew services start zookeeper
Or, if you don't want/need a background service you can just run:
  zkServer start
==> kafka
To have launchd start kafka now and restart at login:
  brew services start kafka
Or, if you don't want/need a background service you can just run:
zookeeper-server-start /usr/local/etc/kafka/zookeeper.properties & kafka-server-start /usr/local/etc/kafka/server.properties

可以看到，安装过程会自动安装zookeeper，并给出zookeeper和kafka的路径、文件个数及存储大小：

/usr/local/Cellar/zookeeper/3.4.14: 430 files, 36.5MB

/usr/local/Cellar/kafka/2.3.1: 171 files, 59.5MB

二、启动kafka服务

我们需要先后启动zookeeper和kafka服务。

它们都需要进入 /usr/local/Cellar/kafka/2.3.1目录，然后再启动相应的命令。

cd /usr/local/Cellar/kafka/2.3.1

启动zookeeper服务，运行命令：

zookeeper-server-start /usr/local/etc/kafka/zookeeper.properties

启动kafka服务，运行命令：

kafka-server-start /usr/local/etc/kafka/server.properties

三、创建Topic，显示数据

Kafka中创建一个Topic，名称为iris

kafka-topics --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic iris

创建成功后，可以使用如下命令，显示所有Topic的列表：

kafka-topics --list --zookeeper localhost:2181

显示结果为

iris

然后，我们通过Alink的 Kafka SinkStreamOp可以将iris数据集写入该Topic。这里不详细展开，有兴趣的读者可以参阅如下的文章。

Alink品数：Alink连接Kafka数据源（Python版本）

Alink品数：Alink连接Kafka数据源（Java版本）

使用如下命令，读取（消费）topic iris中的数据：

kafka-console-consumer --bootstrap-server 127.0.0.1:9092 --topic iris --from-beginning

显示结果如下，略去了中间的大部分数据：

{"sepal_width":3.4,"petal_width":0.2,"sepal_length":4.8,"category":"Iris-setosa","petal_length":1.6}
{"sepal_width":4.1,"petal_width":0.1,"sepal_length":5.2,"category":"Iris-setosa","petal_length":1.5}
{"sepal_width":2.8,"petal_width":1.5,"sepal_length":6.5,"category":"Iris-versicolor","petal_length":4.6}
{"sepal_width":3.0,"petal_width":1.8,"sepal_length":6.1,"category":"Iris-virginica","petal_length":4.9}
{"sepal_width":2.9,"petal_width":1.8,"sepal_length":7.3,"category":"Iris-virginica","petal_length":6.3}
...........
{"sepal_width":2.2,"petal_width":1.0,"sepal_length":6.0,"category":"Iris-versicolor","petal_length":4.0}
{"sepal_width":2.4,"petal_width":1.0,"sepal_length":5.5,"category":"Iris-versicolor","petal_length":3.7}
{"sepal_width":3.1,"petal_width":0.2,"sepal_length":4.6,"category":"Iris-setosa","petal_length":1.5}
{"sepal_width":3.4,"petal_width":0.2,"sepal_length":4.8,"category":"Iris-setosa","petal_length":1.9}
{"sepal_width":2.9,"petal_width":1.4,"sepal_length":6.1,"category":"Iris-versicolor","petal_length":4.7}