1. 环境准备

1.1 创建VM

使用vagrant创建实验环境共5个VM

1
2
3
4
5
192.168.33.106 node06
192.168.33.107 node07
192.168.33.108 node08
192.168.33.109 node09
192.168.33.110 node10

Vagrantfile 如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
λ cat Vagrantfile
# -*- mode: ruby -*-
# vi: set ft=ruby :

hosts = {
"node06" => "192.168.33.106",
"node07" => "192.168.33.107",
"node08" => "192.168.33.108",
"node09" => "192.168.33.109",
"node10" => "192.168.33.110"
}

Vagrant.configure("2") do |config|
hosts.each do |name, ip|
config.vm.define name do |machine|
machine.vm.box = "bento/centos-7.2"
machine.vm.box_check_update = false
machine.ssh.insert_key = false
machine.vm.hostname = "%s.cloud.priv" % name
machine.vm.network :private_network, ip: ip
machine.vm.synced_folder ".", "/vagrant", disabled: true
machine.vm.provider "virtualbox" do |v|
v.name = name
v.customize ["modifyvm", :id, "--memory", 512]
end
end
end
end

1.2 node基本配置

在所有节点上配置 hostname, hosts文件.

1
2
3
4
5
6
7
8
9
10
11
12
[vagrant@localhost ~]$ sudo su -
[root@localhost ~]# hostname node06
[root@localhost ~]# vi /etc/hostname
[root@node06 dockercoins]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.33.106 node06
192.168.33.107 node07
192.168.33.108 node08
192.168.33.109 node09
192.168.33.110 node10
[root@localhost ~]# yum update -y

2. 安装 docker-engine 1.12

在所有节点上安装 docker engine 1.12

1
2
3
4
5
6
7
8
9
10
11
12
13
# 添加测试repo
[root@localhost]# tee /etc/yum.repos.d/docker.repo <<-'EOF'
[dockerrepo]
name=Docker Repository
baseurl=https://yum.dockerproject.org/repo/experimental/centos/7/
enabled=1
gpgcheck=1
gpgkey=https://yum.dockerproject.org/gpg
EOF

[root@localhost]# yum install docker-engine -y
[root@localhost]# systemctl enable docker
[root@node06]# systemctl start docker

3. 安装 docker-compose

在 node06上安装 docker-compose

1
2
3
4
[root@node06 ~]# curl -L https://github.com/docker/compose/releases/download/1.8.0-rc2/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose
[root@node06 ~]# chmod +x /usr/local/bin/docker-compose
[root@node06 ~]# docker-compose -v
docker-compose version 1.8.0-rc2, build c72c966

4. Docker 1.12 swarm 集群创建

Docker engine 1.12 集成了 swarmkit 用于管理和创建docker swarm集群:

  • docker swarm enable Swarm mode; join a Swarm; adjust cluster parameters

  • docker node view nodes; promote/demote managers; manage nodes

  • docker service create and manage services

4.1 初始化swarm集群, 只在第一个节点上执行

1
2
3
4
5
6
7
8
9
10
11
12
[root@node06 dockercoins]# docker swarm init
Swarm initialized: current node (385kvuhbzaegte9z64pm84n59) is now a manager.
## 检查swarm集群状态
[root@node06 dockercoins]# docker info
...
Swarm: active
NodeID: 385kvuhbzaegte9z64pm84n59
IsManager: Yes
Managers: 1
Nodes: 1
CACertHash: sha256:7f844fafc637ccbcc3168b528642bbc2d0910760919cc2394d66a8f077c326f7
...

4.2 向swarm集群中添加node

查询swarm集群node

1
2
3
[root@node06 dockercoins]# docker node ls
ID HOSTNAME MEMBERSHIP STATUS AVAILABILITY MANAGER STATUS
385kvuhbzaegte9z64pm84n59 * node06 Accepted Ready Active Leader

方法一使用ssh添加node

1
2
3
4
5
6
7
8
9
10
11
12
[root@node06 dockercoins]# ssh node07 docker swarm join node06:2377
The authenticity of host 'node07 (192.168.33.107)' can't be established.
ECDSA key fingerprint is 3b:6f:e4:72:5e:72:f3:76:b9:d0:7e:0e:dd:83:9f:5a.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'node07,192.168.33.107' (ECDSA) to the list of known hosts.
root@node07's password:
This node joined a Swarm as a worker.

[root@node06 dockercoins]# docker node ls
ID HOSTNAME MEMBERSHIP STATUS AVAILABILITY MANAGER STATUS
385kvuhbzaegte9z64pm84n59 * node06 Accepted Ready Active Leader
5bgs2uqz2mouv9ehxuk1jdayx node07 Accepted Ready Active

方法二使用 docker API添加 node

1
2
3
4
5
6
# Set DOCKER_HOST and add node3 to the Swarm:

DOCKER_HOST=tcp://node3:55555 docker swarm join node1:2377
# Check that the node is here:

docker node ls

4.3 swarm集群node权限控制

配置node加入集群需要密码授权

1
2
[root@node06 dockercoins]# docker swarm update --secret passwordforswarm
Swarm updated.

查询swarm集群参数, 你会看到policies下保存了 secret的hash密码.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
[root@node06 dockercoins]# docker swarm inspect
[
{
...
"Policies": [
{
"Role": "worker",
"Autoaccept": true,
"Secret": "$2a$10$pk/JS733l9njZFwGWYXFKuEccshMYDMuRCjt.nSUWXpUaq3ZEdTTO"
},
{
"Role": "manager",
"Autoaccept": false,
"Secret": "$2a$10$pk/JS733l9njZFwGWYXFKuEccshMYDMuRCjt.nSUWXpUaq3ZEdTTO"
}
]
},
...
]

加入一个需要密码的swarm集群

1
2
3
4
5
6
7
8
[root@node06 dockercoins]# ssh node08 docker swarm join node06:2377 --secret passwordforswarm
root@node08's password:
This node joined a Swarm as a worker.
[root@node06 dockercoins]# docker node ls
ID HOSTNAME MEMBERSHIP STATUS AVAILABILITY MANAGER STATUS
385kvuhbzaegte9z64pm84n59 * node06 Accepted Ready Active Leader
5bgs2uqz2mouv9ehxuk1jdayx node07 Accepted Ready Active
9nzqe5g66kxtpldneykll8mg1 node08 Accepted Ready Active

4.4 使用手工授权方式管理swarm集群node

关闭 swarm 集群 auto-accept 模式, 清除集群密码(设置密码为””空字符,将会关闭集群密码认证)

1
2
[root@node06 dockercoins]# docker swarm update --auto-accept none --secret ""
Swarm updated.

手工接受node加入 swarm 集群

1
2
3
4
[root@node06 dockercoins]# ssh node09 docker swarm join node06:2377
root@node09's password:
Error response from daemon: Your node is in the process of joining the cluster but needs to be accepted by existing cluster member.
To accept this node into cluster run "docker node accept 9wrzob2xaptfny05fbmbfu875" in an existing cluster manager. Use "docker info" command to see the current Swarm status of your node.

会提示需要accept node加入swarm集群

1
2
3
4
5
6
7
8
9
10
11
12
13
## node ls 会看到 node的状态是pending
[root@node06 dockercoins]# docker node ls
ID HOSTNAME MEMBERSHIP STATUS AVAILABILITY MANAGER STATUS
9wrzob2xaptfny05fbmbfu875 Pending Unknown Active
## 执行 docker node accpet命令 接受node(ID可以简写前面几位)
[root@node06 dockercoins]# docker node accept 9wrzob2xaptfny05fbmbfu875
Node 9wrzob2xaptfny05fbmbfu875 accepted in the swarm.
[root@node06 dockercoins]# docker node ls
ID HOSTNAME MEMBERSHIP STATUS AVAILABILITY MANAGER STATUS
385kvuhbzaegte9z64pm84n59 * node06 Accepted Ready Active Leader
5bgs2uqz2mouv9ehxuk1jdayx node07 Accepted Ready Active
9nzqe5g66kxtpldneykll8mg1 node08 Accepted Ready Active
9wrzob2xaptfny05fbmbfu875 node09 Accepted Ready Active

底层发生了什么?

  1. 当我们用docker swarm init 命令初始化swarm集群的时候, 一个TLS的根CA会被自动创建, 然后签发一个密钥给第一个node.
  2. 当后面的node加入swarm集群的时候,他们发布自己的密钥给根CA签名,并接受根CA的公钥和证书.
  3. swarm集群中所有的通信都是TLS协议加密的.
  4. 所有node的证书会在一段时间内自动更新(默认为90天, 可以使用 docker swarm update 配置)

4.5 提升一个node变为管理节点

现在我们只有一个管理节点(node06), 如果node06挂掉,我们的集群也就挂掉了,为了swarm集群高可用我们可以创建多个管理节点.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
# 查询节点状态
[root@node06 dockercoins]# docker node ls
ID HOSTNAME MEMBERSHIP STATUS AVAILABILITY MANAGER STATUS
385kvuhbzaegte9z64pm84n59 * node06 Accepted Ready Active Leader
5bgs2uqz2mouv9ehxuk1jdayx node07 Accepted Ready Active
9nzqe5g66kxtpldneykll8mg1 node08 Accepted Ready Active
9wrzob2xaptfny05fbmbfu875 node09 Accepted Ready Active
d565ghfnd8m2hgemmv4hevx2b node10 Accepted Ready Active
# 将node07 node08 提升为管理节点
[root@node06 dockercoins]# docker node promote node07 node08
Node node07 promoted to a manager in the swarm.
Node node08 promoted to a manager in the swarm.
# 验证节点状态
[root@node06 dockercoins]# docker node ls

5. 运行swarm service

如何运行一个 swarm service?
很简单 把单机docker下面的 docker run 替换成 docker service create

1
2
3
4
5
6
7
8
9
10
11
#创建一个 service 容器 ping google的DNS
[root@node06 dockercoins]# docker service create alpine ping 8.8.8.8
8q5jjgzp5zztbbxrpdh3zu3km
#查询service列表
[root@node06 dockercoins]# docker service list
ID NAME REPLICAS IMAGE COMMAND
8q5jjgzp5zzt determined_mcclintock 0/1 alpine ping 8.8.8.8
#使用<serviceID> 查询service运行在那个node上
[root@node06 dockercoins]# docker service tasks 8q5jj
ID NAME SERVICE IMAGE LAST STATE DESIRED STATE NODE
59a9c0mkr7anykdra5z9di64x determined_mcclintock.1 determined_mcclintock alpine Running 2 minutes Running node06

检查service运行情况,我们的service运行在node06上, 如果在别的node上可以ssh node0X docker ps

1
2
3
4
5
6
7
8
[root@node06 dockercoins]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
d6155498b874 alpine:latest "ping 8.8.8.8" 8 minutes ago Up 8 minutes determined_mcclintock.1.59a9c0mkr7anykdra5z9di64x
[root@node06 dockercoins]# docker logs d6155498b874
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=1 ttl=52 time=83.599 ms
64 bytes from 8.8.8.8: seq=8 ttl=52 time=137.677 ms
64 bytes from 8.8.8.8: seq=12 ttl=52 time=79.402 ms

5.1 扩展(scale)service

使用 docker service update <serviceID> 命令拓展service

1
2
3
4
5
6
7
8
9
10
11
12
13
[root@node06 dockercoins]# docker service list
ID NAME REPLICAS IMAGE COMMAND
8q5jjgzp5zzt determined_mcclintock 1/1 alpine ping 8.8.8.8
# scale up 10个 service(每个node 2个)
[root@node06 dockercoins]# docker service update 8q5jj --replicas 10
8q5jj
[root@node06 dockercoins]# docker service list
ID NAME REPLICAS IMAGE COMMAND
8q5jjgzp5zzt determined_mcclintock 10/10 alpine ping 8.8.8.8
[root@node06 dockercoins]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
4237eaf4f253 alpine:latest "ping 8.8.8.8" About a minute ago Up About a minute determined_mcclintock.7.8ejd9wgyqvvh0up508q6ewt1s
d6155498b874 alpine:latest "ping 8.8.8.8" 14 minutes ago Up 14 minutes determined_mcclintock.1.59a9c0mkr7anykdra5z9di64x

5.2 暴露service端口

使用docker service create -p 暴露service端口

1
2
[root@node06 dockercoins]# docker service create --name search --publish 9200:9200 --replicas 7 elasticsearch
[root@node06 dockercoins]# watch docker service tasks search

测试端口

1
[root@node06 dockercoins]# curl localhost:9200

5.3 任务(Tasks)的生命周期

使用如下命令查看tasks 状态

1
docker service tasks <serviceID or name>

如果你足够快, 可以看见 tasks 有几种状态:

  1. accepted 任务已经被分配到一个node
  2. preparing 准备,多数时候意味着在下载image
  3. running

如果一个任务被终止(stoped, killed…),他不会再被重启了,会新建一个替代任务.

5.4 删除所有service

移除swarm集群内所有service

1
2
3
4
5
6
7
8
9
[root@node06 dockercoins]# docker service list
ID NAME REPLICAS IMAGE COMMAND
6kp7y61tz5bz search 6/7 elasticsearch
8q5jjgzp5zzt determined_mcclintock 5/5 alpine ping 8.8.8.8

[root@node06 dockercoins]# docker service ls -q | xargs docker service rm

[root@node06 dockercoins]# docker service list
ID NAME REPLICAS IMAGE COMMAND