我的第一个开源项目，ansible批量管理dockerfile集群架构
·
文章目录
Ansible批量管理Docker集群架构
1. 集群项目架构
2.环境准备
2.1 环境列表
| 角色 | 主机名 | 公网IP | 内网IP |
|---|---|---|---|
| 负载均衡 | lb01 | 47.112.125.253 | 172.27.164.1 |
| web服务器 | web05 | 47.127.162.125 | 172.27.164.2 |
| web服务器 | web06 | 47.146.142.156 | 172.27.164.3 |
| web服务器 | web07 | 47.119.120.119 | 172.27.164.4 |
| web服务器 | web08 | 112.74.95.149 | 172.27.164.5 |
| 数据库与缓存 | db01 | 8.129.101.143 | 172.27.164.6 |
| 储存 | nfs01 | 47.112.176.242 | 172.27.164.7 |
| 私有仓库 | reg | 119.43.67.33 | 172.27.164.8 |
| 批量管理机器 | devops | 118.192.54.16 | 172.27.164.9 |
2.2 配置主机域名解析
cat >/etc/hosts<<EOF
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
172.27.164.1 lb01
172.27.164.2 web05
172.27.164.3 web06
172.27.164.4 web07
172.27.164.5 web08
172.27.164.6 db01
172.27.164.7 nfs01
172.27.164.8 reg reg.buffes.cn
172.27.164.9 devops devops.buffes.cn
EOF
2.3 管理机环境准备
1)python3及python3-pip
[root@devops ~]# yum install -y python3-pip rust
2)升级pip与配置pip源 (pypi)
pip3 install -i https://pypi.tuna.tsinghua.edu.cn/simple -- upgrade pip
pip3 config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple
3)安装ansible与docker
pip3 install setuptools_rust
pip3 install ansible
pip3 install docker == 2.6.1
4)安装常用工具
yum install tree nmap dos2unix lrzsz nc lsof wget tcpdump htop iftop iotop sysstat nethogs -y
yum install psmisc net-tools bash-completion vim-enhanced ntpdate -y
2.4 密钥认证
[root@devops ~]# cat /server/scripts/fenfa.sh
#iuthor: buffes
#desc 一键自动化创建和分发公钥
ssh_root_pass="ho07pZVggjvx!"
echo '--------------------------------------------'
echo '1.创建key'
echo '--------------------------------------------'
ssh-keygen -f ~/.ssh/id_rsa -P ''
echo '--------------------------------------------'
echo '2. 分发 pub key'
echo '--------------------------------------------'
for ip in 172.27.164{1..9}
do
sshpass -p$ssh_root_pass ssh-copy-id -i ~/.ssh/id_rsa.pub -o StrictHostKeyChecking=no root@$ip
done
2.5 ansible主机清单与环境变量配置
[root@devops ~]# mkdir -p /server/ansible/
[root@devops ansible]# cat hosts
[lb]
172.27.164.1 hostname=lb01
[web]
172.27.164.2 hostname=web05
172.27.164.3 hostname=web06
172.27.164.4 hostname=web07
172.27.164.5 hostname=web08
[db]
172.27.164.6 hostname=db01
[nfs]
172.27.164.7 hostname=nfs01
[reg]
172.27.164.8 hostname=reg
[root@devops ansible]# grep ^inventory /etc/ansible/ansible.cfg
inventory = ./hosts
[root@devops ansible]# echo 'export ANSIBLE_INVENTORY=./hosts' >> /etc/profile
[root@devops ansible]# source /etc/profile
[root@devops ansible]# ansible all -m ping
3.服务部署
3.1 基础环境优化
[root@devops ansible]# mkdir -p sys_basic/{files,templates,tasks,handlers} group_vars/all/
[root@devops ansible]# cat sys_basic/tasks/main.yml
- name: 1. yum安装常用工具
yum:
name:
- tree
- vim
- wget
- bash-completion
- bash-completion-extras
- lrzsz
- net-tools
- sysstat
- iotop
- iftop
- htop
- unzip
- nc
- nmap
- telnet
- bc
- psmisc
- httpd-tools
- bind-utils
- nethogs
- expect
- python3-pip
- name: 1. 分发hosts文件
copy:
src: hosts
dest: /etc/hosts
backup: yes
- name: 3. 安装python3的 docker依赖
command: pip3 install docker==2.6.1
- name: 4. 开启内核转发
shell:
cmd: |
echo 'net.ipv4.ip_forward = 1' >>/etc/sysctl.conf
sysctl -p
- name: 5. 安装docker-ce,docker-compose 源
shell:
cmd: |
yum install -y yum-utils device-mapper-persistent-data lvm2
yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
sed -i 's+download.docker.com+mirrors.aliyun.com/docker-ce+' /etc/yum.repos.d/docker-ce.repo
- name: 5. 安装docker-ce,docker-compose 源
yum:
name:
- docker-ce
- docker-compose
- name: 6. pre mkdir
file:
path: /etc/docker
state: directory
- name: 6. 配置docker加速和取消docker https
copy:
src: daemon.json
dest: /etc/docker/daemon.json
backup: yes
- name: 7. startup docker-ce
systemd:
name: docker
enabled: yes
state: started
- name: 8. hostname
hostname:
name: "{{hostname}}"
[root@devops ansible]# cat sys_basic/files/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
172.27.164.1 lb01
172.27.164.2 web05
172.27.164.3 web06
172.27.164.4 web07
172.27.164.5 web08
172.27.164.6 db01
172.27.164.7 nfs01
172.27.164.8 reg reg.buffes.cn
172.27.164.9 devops devops.buffes.cn
[root@devops ansible]# cat sys_basic/files/daemon.json
{
"registry-mirrors": ["https://bjjtv7cs.mirror.aliyuncs.com"],
"insecure-registries": ["reg.buffes.cn:5000"]
}
[root@devops ansible]# cat group_vars/all/main.yml
#compose dir
cp_dir: /server/compose
[root@devops ansible]# cat top.yml
#- hosts: all
# vars:
# ansible_python_interpreter: /usr/bin/python2
# roles:
# - role: sys_basic
#- hosts: db
# vars:
# ansible_python_interpreter: /usr/bin/python3
# roles:
# - role: db
#- hosts: nfs
# vars:
# ansible_python_interpreter: /usr/bin/python3
# roles:
# - role: nfs
#- hosts: reg
# vars:
# ansible_python_interpreter: /usr/bin/python3
# roles:
# - role: reg
- hosts: web
vars:
ansible_python_interpreter: /usr/bin/python3
roles:
- role: web
- hosts: lb
vars:
ansible_python_interpreter: /usr/bin/python3
roles:
- role: lb
3.2 部署服务-db-redis
[root@devops ansible]# mkdir -p db/{files,templates,tasks,handlers}
[root@devops ansible]# db/tasks/main.yml
- name: config-mysql5.7 repo
copy:
src: mysql.repo
dest: /etc/yum.repos.d/mysql.repo
backup: yes
- name: 安装db
yum:
name:
- mysql-community-server
state: installed
- name: 安装数据库模块
pip:
name: PyMySQL
executable: pip3
- name: 配置db(留空)
shell: echo
- name: 启动db
systemd:
name: mysqld
enabled: yes
state: started
- name: db default_password
shell: awk '/temporary password/{print $NF}' /var/log/mysqld.log
register: db_root_pass
- name: print pass
debug:
msg: "{{ db_root_pass.stdout }}"
- name: 设置root密码
shell:
cmd: mysql -uroot -p"{{ db_root_pass.stdout }}" -e "alter user root@localhost identified by 'koboid&kod129000!!';flush privileges;" --connect-expired-password
ignore_errors: yes
# mysql_user:
# name: root
# host: localhost
# password: "koboid&kod129000!!"
# login_user: root
# login_password: "{{ db_root_pass.stdout }}"
# state: present
# update_password: always
- name: 删除用户
mysql_user:
name: ""
host: "{{ item }}"
login_user: root
login_password: "koboid&kod129000!!"
state: absent
ignore_errors: yes
with_items:
- "{{ ansible_hostname }}"
- localhost
- name: 创建库
mysql_db:
name: "{{item}}"
login_user: root
login_password: "koboid&kod129000!!!"
state: present
loop:
- blog
- kodbox
- name: 创建用户
mysql_user:
name: "{{item}}"
host: 172.%
password: "{{ item }}"
priv: "{{ item }}.*:ALL"
login_user: root
login_password: "koboid&kod129000!!"
state: present
loop:
- blog
- kodbox
- name: redis
yum:
name: redis
- name: conf
lineinfile:
path: /etc/redis.conf
regexp: '^bind'
line: "bind {{ansible_default_ipv4.address}} 127.0.0.1 "
- name: start redis
systemd:
name: redis
enabled: yes
state: started
[root@devops ansible]# cat db/files/mysql.repo
[mysql-connectors-community]
name=MySQL Connectors Community
baseurl=https://mirrors.tuna.tsinghua.edu.cn/mysql/yum/mysql-connectors-community-el7-$basearch/
enabled=0
gpgcheck=0
gpgkey=https://repo.mysql.com/RPM-GPG-KEY-mysql
[mysql-tools-community]
name=MySQL Tools Community
baseurl=https://mirrors.tuna.tsinghua.edu.cn/mysql/yum/mysql-tools-community-el7-$basearch/
enabled=0
gpgcheck=0
gpgkey=https://repo.mysql.com/RPM-GPG-KEY-mysql
[mysql-5.6-community]
name=MySQL 5.6 Community Server
baseurl=https://mirrors.tuna.tsinghua.edu.cn/mysql/yum/mysql-5.6-community-el7-$basearch/
enabled=0
gpgcheck=0
gpgkey=https://repo.mysql.com/RPM-GPG-KEY-mysql
[mysql-5.7-community]
name=MySQL 5.7 Community Server
baseurl=https://mirrors.tuna.tsinghua.edu.cn/mysql/yum/mysql-5.7-community-el7-$basearch/
enabled=1
gpgcheck=0
gpgkey=https://repo.mysql.com/RPM-GPG-KEY-mysql
[mysql-8.0-community]
name=MySQL 8.0 Community Server
baseurl=https://mirrors.tuna.tsinghua.edu.cn/mysql/yum/mysql-8.0-community-el7-$basearch/
enabled=0
gpgcheck=0
gpgkey=https://repo.mysql.com/RPM-GPG-KEY-mysql
3.2 nfs-储存服务
[root@devops ansible]# mkdir -p nfs/{files,templates,tasks,handlers}
[root@devops ansible]# cat nfs/tasks/main.yml
- name: 安装rpcbind,nfs-utils
yum:
name:
- rpcbind
- nfs-utils
state: installed
- name: 启动rpcbind,nfs
systemd:
name: "{{ item }}"
enabled: yes
state: started
with_items:
- rpcbind
- nfs
- name: 创建共享目录
file:
path: "{{ item }}"
owner: nfsnobody
group: nfsnobody
state: directory
loop:
- /data/wordpress
- /data/kodbox
- name: 配置nfs服务端
shell:
cmd: echo "{{ item }} {{ ansible_default_ipv4.network }}/24(rw,all_squash)" >>/etc/exports
loop:
- /data/wordpress
- /data/kodbox
- name: 重启nfs
systemd:
name: nfs
state: reloaded
3.3 镜像仓库
[root@devops ansible]# mkdir -p reg/{files,templates,tasks,handlers}
[root@devops ansible]# cat reg/tasks/main.yml
- name: 0. mkdir
file:
path: "{{ item }}"
state: directory
loop:
- "{{ cp_dir }}"
- /server/dockerfile
- name: 1. 分发registry compose文件
copy:
src: reg-docker-compose.yml
dest: "{{ cp_dir }}/reg-docker-compose.yml"
backup: yes
- name: 2. 运行compose创建私有仓库
docker_compose:
project_src: /server/compose/
files: reg-docker-compose.yml
state: present
- name: 3. 发送dockerfile及依赖文件配置....
copy:
src: "{{ item }}"
dest: "/server/dockerfile/{{ item }}"
loop:
- nginx
- php
- name: 4. 构建镜像及上传到私有仓库
docker_image:
build:
path: "/server/dockerfile/{{ item }}"
name: "reg.buffes.cn:5000/web/{{ item }}"
tag: v3.0_final
push: yes
source: build
loop:
- nginx
- php
[root@devops ansible]# cat reg/files/php/reg-docker-compose.yml
version: "3.3"
services:
registry_image:
image: "registry:latest"
ports:
- "5000:5000"
volumes:
- "/data/reg-images:/var/lib/registry"
restart: always
[root@devops ansible]# cat reg/files/nginx/dockerfile
FROM nginx:1.20.2-alpine
LABEL maintainer="buffes koboid@qq.com" author="buffes"
COPY nginx.conf /etc/nginx/nginx.conf
COPY *.buffes.cn.conf /etc/nginx/conf.d/
ENV Kod_Code=kodbox.tar.gz \
Wp_Code=wordpress.tar.gz
ADD ${Kod_Code} /app/code/kodbox/
ADD ${Wp_Code} /app/code/wp/
RUN chown -R nginx.nginx /app/code/*
ENTRYPOINT ["/docker-entrypoint.sh"]
EXPOSE 80
STOPSIGNAL SIGQUIT
CMD ["nginx", "-g", "daemon off;"]
[root@devops ansible]# cat reg/files/nginx/kodbox.buffes.cn.conf
server {
listen 80;
server_name kodbox.buffes.cn;
root /app/code/kodbox;
location / {
index index.php;
}
location ~ \.php$ {
fastcgi_pass php:9000;
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
include fastcgi_params;
}
}
[root@devops ansible]# cat reg/files/nginx/nginx.conf
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log notice;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
#tcp_nopush on;
keepalive_timeout 65;
##
#gzip on;
include /etc/nginx/conf.d/*.conf;
}
[root@devops ansible]# cat reg/files/nginx/wp.buffes.cn.conf
server {
listen 80;
server_name wp.buffes.cn;
root /app/code/wp;
location / {
index index.php;
}
location ~ \.php$ {
fastcgi_pass php:9000;
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
include fastcgi_params;
}
}
3.4 lb负载均衡
[root@devops ansible]# mkdir -p lb/{files,templates,tasks,handlers}
[root@devops ansible]# cat lb/files/kodbox.buffes.cn.conf
upstream kodbox_web_pools {
server web05:80;
# server web05:81;
# server web06:80;
# server web06:81;
}
server {
listen 80;
server_name kodbox.buffes.cn;
location / {
proxy_pass http://kodbox_web_pools;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Real-Ip $remote_addr;
}
}
[root@devops ansible]# cat lb/files/wp.buffes.cn.conf
upstream wp_web_pools {
server web05:80;
# server web05:81;
# server web06:80;
# server web06:81;
}
server {
listen 80;
server_name wp.buffes.cn;
location / {
proxy_pass http://wp_web_pools;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Real-Ip $remote_addr;
}
}
[root@devops ansible]# cat lb/handlers/main.yml
- name: restart_nginx
systemd:
name: nginx
enabled: yes
state: restarted
[root@devops ansible]# cat lb/tasks/main.yml
- name: 1. 安装并启动ngx
yum:
name: nginx
- name: 2. 分发配置文件 handler
copy:
src: "{{ item }}"
dest: /etc/nginx/conf.d/
loop:
- wp.buffes.cn.conf
- kodbox.buffes.cn.conf
notify:
- restart_nginx
3.4 web服务
[root@devops ansible]# mkdir -p web/{files,templates,tasks,handlers}
[root@devops ansible]# cat web/tasks/main.yml
- name: 1.创建dk_compose目录
file:
path: "{{ item }}"
state: directory
loop:
- "{{ cp_dir }}"
- /app/code/wp
- /app/code/kodbox
- name: 2. unarchive
unarchive:
src: "{{ item.src }}"
dest: "{{ item.dest }}"
mode: 0777
loop:
- { src: "./kodbox.tar.gz", dest: "/app/code/kodbox/" }
- { src: "./wordpress.tar.gz", dest: "/app/code/wp/" }
- name: 2.分发文件
copy:
src: web-docker-compose.yml
dest: "{{ cp_dir }}/"
- name: 3.运行docker-compose
docker_compose:
project_src: "{{ cp_dir }}"
files: web-docker-compose.yml
state: present
- name: nfs
yum:
name: nfs-utils
state: installed
- name: mkdir
file:
path: "{{ item }}"
state: directory
loop:
- "/app/code/wp/wp-content/uploads/"
- "/app/code/kodbox/data/files/"
- name: mount
mount:
src: "{{ item.s }}"
path: "{{ item.d }}"
fstype: nfs
state: mounted
loop:
- { s: "nfs01:/data/wordpress/", d: "/app/code/wp/wp-content/uploads/" }
- { s: "nfs01:/data/kodbox/", d: "/app/code/kodbox/data/files/" }
[root@devops ansible]# cat web/files/web-docker-compose.yml
version: "3.3"
services:
web_ngx_a:
image: "reg.buffes.cn:5000/web/nginx:v3.0_final"
ports:
- "80:80"
volumes:
- "/app/code/wp:/app/code/wp/"
- "/app/code/kodbox:/app/code/kodbox"
links:
- "web_php:php"
restart: always
web_ngx_b:
image: "reg.buffes.cn:5000/web/nginx:v3.0_final"
ports:
- "81:80"
volumes:
- "/app/code/wp:/app/code/wp/"
- "/app/code/kodbox:/app/code/kodbox"
links:
- "web_php:php"
restart: always
web_php:
image: "reg.buffes.cn:5000/web/php:v3.0_final"
volumes:
- "/app/code/wp:/app/code/wp/"
- "/app/code/kodbox:/app/code/kodbox"
restart: always
整体目录文件结构
[root@devops ansible]# ll
total 8
drwxr-xr-x 6 root root 65 Aug 6 09:40 db
drwxr-xr-x 3 root root 17 Aug 6 09:40 group_vars
-rw-r--r-- 1 root root 241 May 30 2022 hosts
drwxr-xr-x 6 root root 65 Aug 6 09:40 lb
drwxr-xr-x 6 root root 65 Aug 6 09:40 nfs
drwxr-xr-x 6 root root 65 Aug 6 09:40 reg
drwxr-xr-x 6 root root 78 Aug 6 09:40 sys_basic
-rw-r--r-- 1 root root 598 May 30 2022 top.yml
drwxr-xr-x 6 root root 65 Aug 6 09:40 web
[root@devops /server/ansible]# tree -F
.
├── db/
│ ├── files/
│ │ └── mysql.repo
│ ├── handlers/
│ ├── tasks/
│ │ └── main.yml
│ └── templates/
├── group_vars/
│ └── all/
│ └── main.yml
├── hosts
├── lb/
│ ├── files/
│ │ ├── kodbox.buffes.cn.conf
│ │ └── wp.buffes.cn.conf
│ ├── handlers/
│ │ └── main.yml
│ ├── tasks/
│ │ └── main.yml
│ └── templates/
├── nfs/
│ ├── files/
│ ├── handlers/
│ ├── tasks/
│ │ └── main.yml
│ └── templates/
├── reg/
│ ├── files/
│ │ ├── nginx/
│ │ │ ├── Dockerfile
│ │ │ ├── kodbox.buffes.cn.conf
│ │ │ ├── kodbox.tar.gz
│ │ │ ├── nginx.conf
│ │ │ ├── wordpress.tar.gz
│ │ │ └── wp.buffes.cn.conf
│ │ ├── php/
│ │ │ ├── Dockerfile
│ │ │ ├── kodbox.tar.gz
│ │ │ ├── wordpress.tar.gz
│ │ │ └── www.conf
│ │ └── reg-docker-compose.yml
│ ├── handlers/
│ ├── tasks/
│ │ └── main.yml
│ └── templates/
├── sys_basic/
│ ├── files/
│ │ ├── daemon.json
│ │ └── hosts
│ ├── handlers/
│ ├── hosts
│ ├── tasks/
│ │ └── main.yml
│ └── templates/
├── top.yml
└── web/
├── files/
│ ├── kodbox.tar.gz
│ ├── web-docker-compose.yml
│ └── wordpress.tar.gz
├── handlers/
├── tasks/
│ └── main.yml
└── templates/
34 directories, 30 files
总结:
本项目采用 Ansible 自动化工具，实现了 Docker 容器集群的批量一键部署，极大提升了环境一致性、部署效率和可维护性。整体架构分为四层：控制层：通过 DevOps 节点运行 Ansible Playbook，统一控制所有节点；负载层：使用 Nginx 或 HAProxy 作为负载均衡，转发前端流量；应用层：将 Kodbox 和 WordPress 服务容器化，分布部署在不同 web 节点，按模块解耦；数据层：服务访问通过 Redis 缓存与数据库分离，加快响应速度，同时通过 NFS 提供共享存储支持
更多推荐
31
0- 0
已为社区贡献1条内容
所有评论(0)