前言

ZooKeeper是一个分布式的,开放源码的分布式应用程序协调服务,是Google的Chubby一个开源的实现,是Hadoop和Hbase的重要组件。它是一个为分布式应用提供一致性服务的软件,提供的功能包括:配置维护、域名服务、分布式同步、组服务等。

下面我们通过k8s构建zookeeper镜像,上传私有仓库,创建PV/PVC,最后yaml部署3个节点的zookeeper集群。
集群需要奇数位,本文创建了3个pv,3个pvc,3个service,3个pod。



zookeeper镜像

准备文件,包括安装包,启动脚本,配置,秘钥等:

ls zookeeper/

# 目录,包括查看zk状态脚本
bin  
# 目录,包括zoo配置和属性
conf  
# 构建文件
Dockerfile  
# 启动脚本,动态生成zoo id
entrypoint.sh  
# 秘钥文件
KEYS  
# 镜像源
repositories  
# zk安装包
zookeeper-3.4.14.tar.gz  
# 安装包验证
zookeeper-3.4.14.tar.gz.asc

构建Dockerfile镜像内容:

vim Dockerfile

# 基于官网精简版java8
FROM elevy/slim_java:8

ENV ZK_VERSION 3.4.14
ADD repositories /etc/apk/repositories 
# 拷贝zookeeper程序
COPY zookeeper-3.4.14.tar.gz /tmp/zk.tgz
COPY zookeeper-3.4.14.tar.gz.asc /tmp/zk.tgz.asc
COPY KEYS /tmp/KEYS
RUN apk add --no-cache --virtual .build-deps \
...
...

# 复制配置和启动文件,一定要加上可执行权限。
COPY conf /zookeeper/conf/
COPY bin/zkReady.sh /zookeeper/bin/
RUN chmod +x /zookeeper/bin/zkReady.sh
COPY entrypoint.sh /
RUN chmod +x /entrypoint.sh

# 环境变量
ENV PATH=/zookeeper/bin:${PATH} \
    ZOO_LOG_DIR=/zookeeper/log \
    ZOO_LOG4J_PROP="INFO, CONSOLE, ROLLINGFILE" \
    JMXPORT=9010

# 启动脚本ENTRYPOINT 参数为CMD 
ENTRYPOINT [ "/entrypoint.sh" ]
CMD [ "zkServer.sh", "start-foreground" ]

# 暴露服务端口
EXPOSE 2181 2888 3888 9010

构建镜像,上传镜像:

nerdctl build -t easzlab.io.local:5000/myhub/zookeeper:v1 .
nerdctl push --insecure-registry easzlab.io.local:5000/myhub/zookeeper:v1

创建NFS共享目录

在NFS服务器创建3个共享目录:

# 创建3个共享目录,属主属组选nfsnobody。
mkdir -p /data/k8sdata/myserver/zookeeper-datadir-1
mkdir -p /data/k8sdata/myserver/zookeeper-datadir-2
mkdir -p /data/k8sdata/myserver/zookeeper-datadir-3
chown nfsnobody.nfsnobody /data/k8sdata/myserver/zookeeper-datadir-1
chown nfsnobody.nfsnobody /data/k8sdata/myserver/zookeeper-datadir-2
chown nfsnobody.nfsnobody /data/k8sdata/myserver/zookeeper-datadir-3


# 提供共享服务,可共享的IP段为物理机IP,rw可读写,all_squash远程用户连到nfs服务器都压榨成nfsnobody用户。
cat << EOF >> /etc/exports
/data/k8sdata/myserver/zookeeper-datadir-1 192.168.100.0/24(rw,all_squash)
/data/k8sdata/myserver/zookeeper-datadir-2 192.168.100.0/24(rw,all_squash)
/data/k8sdata/myserver/zookeeper-datadir-3 192.168.100.0/24(rw,all_squash)
EOF

# 重新加载,查看挂载情况
exportfs -r
exportfs -v

# nfs涉及端口较多,测试环境可关闭防火墙
systemctl disable --now firewalld

创建PV

创建3个pv,分配的3个pv总和可以比NFS硬盘空间大,但不建议。

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: zookeeper-datadir-pv-1
spec:
  capacity:
    storage: 20Gi
  accessModes:
    - ReadWriteOnce 
  nfs:
    server: 192.168.100.155
    path: /data/k8sdata/myserver/zookeeper-datadir-1 

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: zookeeper-datadir-pv-2
spec:
  capacity:
    storage: 20Gi
  accessModes:
    - ReadWriteOnce
  nfs:
    server: 192.168.100.155 
    path: /data/k8sdata/myserver/zookeeper-datadir-2 

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: zookeeper-datadir-pv-3
spec:
  capacity:
    storage: 20Gi
  accessModes:
    - ReadWriteOnce
  nfs:
    server: 192.168.100.155  
    path: /data/k8sdata/myserver/zookeeper-datadir-3 

查看创建的3个pv:

sudo kubectl get pv
NAME                     CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS   REASON   AGE
zookeeper-datadir-pv-1   20Gi       RWO            Retain           Available                                   7s
zookeeper-datadir-pv-2   20Gi       RWO            Retain           Available                                   7s
zookeeper-datadir-pv-3   20Gi       RWO            Retain           Available                                   7s

创建PVC

创建3个pvc,pvc容量不能大于对应的pv容量:

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: zookeeper-datadir-pvc-1
  namespace: myserver
spec:
  accessModes:
    - ReadWriteOnce
  volumeName: zookeeper-datadir-pv-1
  resources:
    requests:
      storage: 10Gi
      
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: zookeeper-datadir-pvc-2
  namespace: myserver
spec:
  accessModes:
    - ReadWriteOnce
  volumeName: zookeeper-datadir-pv-2
  resources:
    requests:
      storage: 10Gi
      
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: zookeeper-datadir-pvc-3
  namespace: myserver
spec:
  accessModes:
    - ReadWriteOnce
  volumeName: zookeeper-datadir-pv-3
  resources:
    requests:
      storage: 10Gi

查看3个pvc:

sudo kubectl get pvc -n myserver 
NAME                      STATUS   VOLUME                   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
zookeeper-datadir-pvc-1   Bound    zookeeper-datadir-pv-1   20Gi       RWO                           47s
zookeeper-datadir-pvc-2   Bound    zookeeper-datadir-pv-2   20Gi       RWO                           47s
zookeeper-datadir-pvc-3   Bound    zookeeper-datadir-pv-3   20Gi       RWO                           46s

部署zookeeper集群

创建服务:zookeeper,zookeeper1,zookeeper2,zookeeper3。
创建Pod:zookeeper1,zookeeper2,zookeeper3。
服务端口:2181,2888,3888。

apiVersion: v1
kind: Service
metadata:
  name: zookeeper
  namespace: myserver
spec:
  ports:
    - name: client
      port: 2181
  selector:
    app: zookeeper
---
apiVersion: v1
kind: Service
metadata:
  name: zookeeper1
  namespace: myserver
spec:
  type: NodePort        
  ports:
    - name: client
      port: 2181
      nodePort: 32181
    - name: followers
      port: 2888
    - name: election
      port: 3888
  selector:
    app: zookeeper
    server-id: "1"
---
apiVersion: v1
kind: Service
metadata:
  name: zookeeper2
  namespace: myserver
spec:
  type: NodePort        
  ports:
    - name: client
      port: 2181
      nodePort: 32182
    - name: followers
      port: 2888
    - name: election
      port: 3888
  selector:
    app: zookeeper
    server-id: "2"
---
apiVersion: v1
kind: Service
metadata:
  name: zookeeper3
  namespace: myserver
spec:
  type: NodePort        
  ports:
    - name: client
      port: 2181
      nodePort: 32183
    - name: followers
      port: 2888
    - name: election
      port: 3888
  selector:
    app: zookeeper
    server-id: "3"


---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: zookeeper1
  namespace: myserver
spec:
  replicas: 1
  selector:
    matchLabels:
      app: zookeeper
  template:
    metadata:
      labels:
        app: zookeeper
        server-id: "1"
    spec:
      volumes:
        - name: data
          emptyDir: {}
        - name: wal
          emptyDir:
            medium: Memory
      containers:
        - name: server
          image: easzlab.io.local:5000/myhub/zookeeper:v1 
          imagePullPolicy: Always
          env:
            - name: MYID
              value: "1"
            - name: SERVERS
              value: "zookeeper1,zookeeper2,zookeeper3"
            - name: JVMFLAGS
              value: "-Xmx2G"
          ports:
            - containerPort: 2181
            - containerPort: 2888
            - containerPort: 3888
          volumeMounts:
          - mountPath: "/zookeeper/data"
            name: zookeeper-datadir-pvc-1 
      volumes:
        - name: zookeeper-datadir-pvc-1 
          persistentVolumeClaim:
            claimName: zookeeper-datadir-pvc-1
---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: zookeeper2
  namespace: myserver
spec:
  replicas: 1
  selector:
    matchLabels:
      app: zookeeper
  template:
    metadata:
      labels:
        app: zookeeper
        server-id: "2"
    spec:
      volumes:
        - name: data
          emptyDir: {}
        - name: wal
          emptyDir:
            medium: Memory
      containers:
        - name: server
          image: easzlab.io.local:5000/myhub/zookeeper:v1 
          imagePullPolicy: Always
          env:
            - name: MYID
              value: "2"
            - name: SERVERS
              value: "zookeeper1,zookeeper2,zookeeper3"
            - name: JVMFLAGS
              value: "-Xmx2G"
          ports:
            - containerPort: 2181
            - containerPort: 2888
            - containerPort: 3888
          volumeMounts:
          - mountPath: "/zookeeper/data"
            name: zookeeper-datadir-pvc-2 
      volumes:
        - name: zookeeper-datadir-pvc-2
          persistentVolumeClaim:
            claimName: zookeeper-datadir-pvc-2
---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: zookeeper3
  namespace: myserver
spec:
  replicas: 1
  selector:
    matchLabels:
      app: zookeeper
  template:
    metadata:
      labels:
        app: zookeeper
        server-id: "3"
    spec:
      volumes:
        - name: data
          emptyDir: {}
        - name: wal
          emptyDir:
            medium: Memory
      containers:
        - name: server
          image: easzlab.io.local:5000/myhub/zookeeper:v1 
          imagePullPolicy: Always
          env:
            - name: MYID
              value: "3"
            - name: SERVERS
              value: "zookeeper1,zookeeper2,zookeeper3"
            - name: JVMFLAGS
              value: "-Xmx2G"
          ports:
            - containerPort: 2181
            - containerPort: 2888
            - containerPort: 3888
          volumeMounts:
          - mountPath: "/zookeeper/data"
            name: zookeeper-datadir-pvc-3
      volumes:
        - name: zookeeper-datadir-pvc-3
          persistentVolumeClaim:
           claimName: zookeeper-datadir-pvc-3

检查集群

容器启动后检查状态,3个节点一主两从为正常状态:

sudo kubectl get pod -n myserver
zookeeper1-68ff8f4f7d-kc86m                       1/1     Running   0          17m
zookeeper2-5bcfcd94df-8qjmw                       1/1     Running   0          17m
zookeeper3-7d66b87c88-v6x2j                       1/1     Running   0          17m


sudo kubectl exec zookeeper1-68ff8f4f7d-kc86m  -n myserver -- /zookeeper/bin/zkServer.sh status | grep Mode
Mode: follower

sudo kubectl exec zookeeper2-5bcfcd94df-8qjmw  -n myserver -- /zookeeper/bin/zkServer.sh status | grep Mode
Mode: follower

sudo kubectl exec zookeeper3-7d66b87c88-v6x2j  -n myserver -- /zookeeper/bin/zkServer.sh status | grep Mode
Mode: leader


更多推荐