1. 环境准备

1.1 创建VM

使用vagrant创建实验环境共5个VM

1
2
3
4
5
192.168.33.106 node06
192.168.33.107 node07
192.168.33.108 node08
192.168.33.109 node09
192.168.33.110 node10

Vagrantfile 如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
λ cat Vagrantfile
# -*- mode: ruby -*-
# vi: set ft=ruby :

hosts = {
  "node06" => "192.168.33.106",
  "node07" => "192.168.33.107",
  "node08" => "192.168.33.108",
  "node09" => "192.168.33.109",
  "node10" => "192.168.33.110"
}

Vagrant.configure("2") do |config|
  hosts.each do |name, ip|
    config.vm.define name do |machine|
      machine.vm.box = "bento/centos-7.2"
      machine.vm.box_check_update = false
      machine.ssh.insert_key = false
      machine.vm.hostname = "%s.cloud.priv" % name
      machine.vm.network :private_network, ip: ip
      machine.vm.synced_folder ".", "/vagrant", disabled: true
      machine.vm.provider "virtualbox" do |v|
          v.name = name
          v.customize ["modifyvm", :id, "--memory", 512]
      end
    end
  end
end

1.2 node基本配置

在所有节点上配置 hostname, hosts文件.

1
2
3
4
5
6
7
8
9
10
11
12
[vagrant@localhost ~]$ sudo su -
[root@localhost ~]# hostname node06
[root@localhost ~]# vi /etc/hostname
[root@node06 dockercoins]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.33.106 node06
192.168.33.107 node07
192.168.33.108 node08
192.168.33.109 node09
192.168.33.110 node10
[root@localhost ~]# yum update -y

2. 安装 docker-engine 1.12

在所有节点上安装 docker engine 1.12

1
2
3
4
5
6
7
8
9
10
11
12
13
# 添加测试repo
[root@localhost]# tee /etc/yum.repos.d/docker.repo <<-'EOF'
[dockerrepo]
name=Docker Repository
baseurl=https://yum.dockerproject.org/repo/experimental/centos/7/
enabled=1
gpgcheck=1
gpgkey=https://yum.dockerproject.org/gpg
EOF

[root@localhost]# yum install docker-engine -y
[root@localhost]# systemctl enable docker
[root@node06]# systemctl start docker

3. 安装 docker-compose

在 node06上安装 docker-compose

1
2
3
4
[root@node06 ~]# curl -L https://github.com/docker/compose/releases/download/1.8.0-rc2/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose
[root@node06 ~]# chmod +x /usr/local/bin/docker-compose
[root@node06 ~]# docker-compose -v
docker-compose version 1.8.0-rc2, build c72c966

4. Docker 1.12 swarm 集群创建

Docker engine 1.12 集成了 swarmkit 用于管理和创建docker swarm集群:

  • docker swarm enable Swarm mode; join a Swarm; adjust cluster parameters

  • docker node view nodes; promote/demote managers; manage nodes

  • docker service create and manage services

4.1 初始化swarm集群, 只在第一个节点上执行

1
2
3
4
5
6
7
8
9
10
11
12
[root@node06 dockercoins]# docker swarm init
Swarm initialized: current node (385kvuhbzaegte9z64pm84n59) is now a manager.
## 检查swarm集群状态
[root@node06 dockercoins]# docker info
...
Swarm: active
 NodeID: 385kvuhbzaegte9z64pm84n59
 IsManager: Yes
 Managers: 1
 Nodes: 1
 CACertHash: sha256:7f844fafc637ccbcc3168b528642bbc2d0910760919cc2394d66a8f077c326f7
...

4.2 向swarm集群中添加node

查询swarm集群node

1
2
3
[root@node06 dockercoins]# docker node ls
ID                           HOSTNAME  MEMBERSHIP  STATUS  AVAILABILITY  MANAGER STATUS
385kvuhbzaegte9z64pm84n59 *  node06    Accepted    Ready   Active        Leader

方法一使用ssh添加node

1
2
3
4
5
6
7
8
9
10
11
12
[root@node06 dockercoins]# ssh node07 docker swarm join node06:2377
The authenticity of host 'node07 (192.168.33.107)' can't be established.
ECDSA key fingerprint is 3b:6f:e4:72:5e:72:f3:76:b9:d0:7e:0e:dd:83:9f:5a.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'node07,192.168.33.107' (ECDSA) to the list of known hosts.
root@node07's password:
This node joined a Swarm as a worker.

[root@node06 dockercoins]# docker node ls
ID                           HOSTNAME  MEMBERSHIP  STATUS  AVAILABILITY  MANAGER STATUS
385kvuhbzaegte9z64pm84n59 *  node06    Accepted    Ready   Active        Leader
5bgs2uqz2mouv9ehxuk1jdayx    node07    Accepted    Ready   Active

方法二使用 docker API添加 node

1
2
3
4
5
6
# Set DOCKER_HOST and add node3 to the Swarm:

DOCKER_HOST=tcp://node3:55555 docker swarm join node1:2377
# Check that the node is here:

docker node ls

4.3 swarm集群node权限控制

配置node加入集群需要密码授权

1
2
[root@node06 dockercoins]# docker swarm update --secret passwordforswarm
Swarm updated.

查询swarm集群参数, 你会看到policies下保存了 secret的hash密码.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
[root@node06 dockercoins]# docker swarm inspect
[
    {
...
                "Policies": [
                    {
                        "Role": "worker",
                        "Autoaccept": true,
                        "Secret": "$2a$10$pk/JS733l9njZFwGWYXFKuEccshMYDMuRCjt.nSUWXpUaq3ZEdTTO"
                    },
                    {
                        "Role": "manager",
                        "Autoaccept": false,
                        "Secret": "$2a$10$pk/JS733l9njZFwGWYXFKuEccshMYDMuRCjt.nSUWXpUaq3ZEdTTO"
                    }
                ]
            },
...
]

加入一个需要密码的swarm集群

1
2
3
4
5
6
7
8
[root@node06 dockercoins]# ssh node08 docker swarm join node06:2377 --secret passwordforswarm
root@node08's password:
This node joined a Swarm as a worker.
[root@node06 dockercoins]# docker node ls
ID                           HOSTNAME  MEMBERSHIP  STATUS  AVAILABILITY  MANAGER STATUS
385kvuhbzaegte9z64pm84n59 *  node06    Accepted    Ready   Active        Leader
5bgs2uqz2mouv9ehxuk1jdayx    node07    Accepted    Ready   Active        
9nzqe5g66kxtpldneykll8mg1    node08    Accepted    Ready   Active

4.4 使用手工授权方式管理swarm集群node

关闭 swarm 集群 auto-accept 模式, 清除集群密码(设置密码为””空字符,将会关闭集群密码认证)

1
2
[root@node06 dockercoins]# docker swarm update --auto-accept none --secret ""
Swarm updated.

手工接受node加入 swarm 集群

1
2
3
4
[root@node06 dockercoins]# ssh node09 docker swarm join node06:2377
root@node09's password:
Error response from daemon: Your node is in the process of joining the cluster but needs to be accepted by existing cluster member.
To accept this node into cluster run "docker node accept 9wrzob2xaptfny05fbmbfu875" in an existing cluster manager. Use "docker info" command to see the current Swarm status of your node.

会提示需要accept node加入swarm集群

1
2
3
4
5
6
7
8
9
10
11
12
13
## node ls 会看到 node的状态是pending
[root@node06 dockercoins]# docker node ls
ID                           HOSTNAME  MEMBERSHIP  STATUS   AVAILABILITY  MANAGER STATUS       
9wrzob2xaptfny05fbmbfu875              Pending     Unknown  Active  
## 执行 docker node accpet命令 接受node(ID可以简写前面几位)
[root@node06 dockercoins]# docker node accept 9wrzob2xaptfny05fbmbfu875
Node 9wrzob2xaptfny05fbmbfu875 accepted in the swarm.
[root@node06 dockercoins]# docker node ls
ID                           HOSTNAME  MEMBERSHIP  STATUS   AVAILABILITY  MANAGER STATUS
385kvuhbzaegte9z64pm84n59 *  node06    Accepted    Ready    Active        Leader
5bgs2uqz2mouv9ehxuk1jdayx    node07    Accepted    Ready    Active        
9nzqe5g66kxtpldneykll8mg1    node08    Accepted    Ready    Active        
9wrzob2xaptfny05fbmbfu875    node09    Accepted    Ready    Active

底层发生了什么?

  1. 当我们用docker swarm init 命令初始化swarm集群的时候, 一个TLS的根CA会被自动创建, 然后签发一个密钥给第一个node.
  2. 当后面的node加入swarm集群的时候,他们发布自己的密钥给根CA签名,并接受根CA的公钥和证书.
  3. swarm集群中所有的通信都是TLS协议加密的.
  4. 所有node的证书会在一段时间内自动更新(默认为90天, 可以使用 docker swarm update 配置)

4.5 提升一个node变为管理节点

现在我们只有一个管理节点(node06), 如果node06挂掉,我们的集群也就挂掉了,为了swarm集群高可用我们可以创建多个管理节点.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
# 查询节点状态
[root@node06 dockercoins]# docker node ls
ID                           HOSTNAME  MEMBERSHIP  STATUS  AVAILABILITY  MANAGER STATUS
385kvuhbzaegte9z64pm84n59 *  node06    Accepted    Ready   Active        Leader
5bgs2uqz2mouv9ehxuk1jdayx    node07    Accepted    Ready   Active        
9nzqe5g66kxtpldneykll8mg1    node08    Accepted    Ready   Active        
9wrzob2xaptfny05fbmbfu875    node09    Accepted    Ready   Active        
d565ghfnd8m2hgemmv4hevx2b    node10    Accepted    Ready   Active  
# 将node07 node08 提升为管理节点
[root@node06 dockercoins]# docker node promote node07 node08
Node node07 promoted to a manager in the swarm.
Node node08 promoted to a manager in the swarm.
# 验证节点状态
[root@node06 dockercoins]# docker node ls

5. 运行swarm service

如何运行一个 swarm service?
很简单 把单机docker下面的 docker run 替换成 docker service create

1
2
3
4
5
6
7
8
9
10
11
#创建一个 service 容器 ping google的DNS
[root@node06 dockercoins]# docker service create alpine ping 8.8.8.8
8q5jjgzp5zztbbxrpdh3zu3km
#查询service列表
[root@node06 dockercoins]# docker service list
ID            NAME                   REPLICAS  IMAGE   COMMAND
8q5jjgzp5zzt  determined_mcclintock  0/1       alpine  ping 8.8.8.8
#使用<serviceID> 查询service运行在那个node上
[root@node06 dockercoins]# docker service tasks 8q5jj
ID                         NAME                     SERVICE                IMAGE   LAST STATE         DESIRED STATE  NODE
59a9c0mkr7anykdra5z9di64x  determined_mcclintock.1  determined_mcclintock  alpine  Running 2 minutes  Running        node06

检查service运行情况,我们的service运行在node06上, 如果在别的node上可以ssh node0X docker ps

1
2
3
4
5
6
7
8
[root@node06 dockercoins]# docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
d6155498b874        alpine:latest       "ping 8.8.8.8"      8 minutes ago       Up 8 minutes                            determined_mcclintock.1.59a9c0mkr7anykdra5z9di64x
[root@node06 dockercoins]# docker logs d6155498b874
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=1 ttl=52 time=83.599 ms
64 bytes from 8.8.8.8: seq=8 ttl=52 time=137.677 ms
64 bytes from 8.8.8.8: seq=12 ttl=52 time=79.402 ms

5.1 扩展(scale)service

使用 docker service update <serviceID> 命令拓展service

1
2
3
4
5
6
7
8
9
10
11
12
13
[root@node06 dockercoins]# docker service list
ID            NAME                   REPLICAS  IMAGE   COMMAND
8q5jjgzp5zzt  determined_mcclintock  1/1       alpine  ping 8.8.8.8
# scale up 10个 service(每个node 2个)
[root@node06 dockercoins]# docker service update 8q5jj --replicas 10
8q5jj
[root@node06 dockercoins]# docker service list
ID            NAME                   REPLICAS  IMAGE   COMMAND
8q5jjgzp5zzt  determined_mcclintock  10/10     alpine  ping 8.8.8.8
[root@node06 dockercoins]# docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED              STATUS              PORTS               NAMES
4237eaf4f253        alpine:latest       "ping 8.8.8.8"      About a minute ago   Up About a minute                       determined_mcclintock.7.8ejd9wgyqvvh0up508q6ewt1s
d6155498b874        alpine:latest       "ping 8.8.8.8"      14 minutes ago       Up 14 minutes                           determined_mcclintock.1.59a9c0mkr7anykdra5z9di64x

5.2 暴露service端口

使用docker service create -p 暴露service端口

1
2
[root@node06 dockercoins]# docker service create --name search --publish 9200:9200 --replicas 7 elasticsearch
[root@node06 dockercoins]# watch docker service tasks search

测试端口

1
[root@node06 dockercoins]# curl localhost:9200

5.3 任务(Tasks)的生命周期

使用如下命令查看tasks 状态

1
docker service tasks <serviceID or name>

如果你足够快, 可以看见 tasks 有几种状态:

  1. accepted 任务已经被分配到一个node
  2. preparing 准备,多数时候意味着在下载image
  3. running

如果一个任务被终止(stoped, killed…),他不会再被重启了,会新建一个替代任务.

5.4 删除所有service

移除swarm集群内所有service

1
2
3
4
5
6
7
8
9
[root@node06 dockercoins]# docker service list
ID            NAME                   REPLICAS  IMAGE          COMMAND
6kp7y61tz5bz  search                 6/7       elasticsearch  
8q5jjgzp5zzt  determined_mcclintock  5/5       alpine         ping 8.8.8.8

[root@node06 dockercoins]# docker service ls -q | xargs docker service rm

[root@node06 dockercoins]# docker service list
ID            NAME                   REPLICAS  IMAGE          COMMAND