使用Ansible + kubeadm部署K8S集群
纯属练手，这几天看了Ansible相关的资料，练习一下，加深了解！
1、环境清单
1.1、系统清单
| IP | Role | OS |
|---|---|---|
| 192.168.119.160 | k8s-master | CentOS 7 |
| 192.168.119.161 | K8s-node-1 | CentOS 7 |
| 192.168.119.162 | K8s-node-2 | CentOS 7 |
| 192.168.119.169 | ansible-manage | CentOS 7 |
1.2、软件清单
1.2.1、Ansible
[chenlei@node-ansible ~]$ ansible --version
ansible 2.5.0 (devel d1f19125a5) last updated 2018/02/08 21:21:11 (GMT +800)
config file = None
configured module search path = [u'/home/chenlei/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
ansible python module location = /home/chenlei/ansible/lib/ansible
executable location = /home/chenlei/ansible/bin/ansible
python version = 2.7.5 (default, Nov 6 2016, 00:28:07) [GCC 4.8.5 20150623 (Red Hat 4.8.5-11)]1.2.2、Kubernetes
参见上一篇文章内容，本文是上一篇kubeadm的ansible版本。
1.3、镜像清单
参见上一篇文章内容，本文是上一篇kubeadm的ansible版本。
2、部署说明
2.1、安装Ansible
### 安装git ###
[chenlei@node-ansible ~]$ sudo yum install -y git
### 从GitHub上下载源代码 ###
[chenlei@node-ansible ~]$ git clone https://github.com/ansible/ansible.git --recursive
正克隆到 'ansible'...
remote: Counting objects: 302399, done.
remote: Compressing objects: 100% (46/46), done.
remote: Total 302399 (delta 11), reused 9 (delta 0), pack-reused 302352
接收对象中: 100% (302399/302399), 109.50 MiB | 16.00 KiB/s, done.
处理 delta 中: 100% (190336/190336), done.
[chenlei@node-ansible ~]$ ll
drwxrwxr-x. 14 chenlei chenlei 4096 2月 28 10:48 ansible
### 安装 ###
[chenlei@node-ansible ~]$ source ansible/hacking/env-setup
Ansible now needs setuptools in order to build. Install it using your package manager (usually python-setuptools) or via pip (pip install setuptools).
Setting up Ansible to run out of checkout...
PATH=/home/chenlei/ansible/bin:/home/chenlei/ansible/test/runner:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/chenlei/.local/bin:/home/chenlei/bin
PYTHONPATH=/home/chenlei/ansible/lib:
MANPATH=/home/chenlei/ansible/docs/man:
Remember, you may wish to specify your host file with -i
Done!这里直接以源码方式运行Ansible、从github上下载源码之后，根据源码内的脚本设置好环境变量即可
这里的安装只是设置环境变量，系统重启后即失效
2.2、安装setuptools
### 下载setuptools ###
[chenlei@node-ansible ~]$ wget https://pypi.python.org/packages/6c/54/f7e9cea6897636a04e74c3954f0d8335cc38f7d01e27eec98026b049a300/setuptools-38.5.1.zip#md5=1705ae74b04d1637f604c336bb565720
--2018-02-28 11:01:29-- https://pypi.python.org/packages/6c/54/f7e9cea6897636a04e74c3954f0d8335cc38f7d01e27eec98026b049a300/setuptools-38.5.1.zip
正在解析主机 pypi.python.org (pypi.python.org)... 151.101.228.223, 2a04:4e42:36::223
正在连接 pypi.python.org (pypi.python.org)|151.101.228.223|:443... 已连接。
已发出 HTTP 请求，正在等待回应... 200 OK
长度：737995 (721K) [binary/octet-stream]
正在保存至: “setuptools-38.5.1.zip”
100%[====================================================================>] 737,995 378KB/s 用时 1.9s
2018-02-28 11:01:32 (378 KB/s) - 已保存 “setuptools-38.5.1.zip” [737995/737995])
[chenlei@node-ansible ~]$ ll
drwxrwxr-x. 14 chenlei chenlei 4096 2月 28 10:48 ansible
-rw-rw-r--. 1 chenlei chenlei 737995 2月 6 18:59 setuptools-38.5.1.zip
### 安装setuptools ###
[chenlei@node-ansible ~]$ unzip setuptools-38.5.1.zip
[chenlei@node-ansible ~]$ cd setuptools-38.5.1
[chenlei@node-ansible setuptools-38.5.1]$ sudo python setup.py install
### 安装pip ###
[chenlei@node-ansible ~]$ sudo easy_install pipsetuptools即easy_install，用于安装pip，安装pip是为了安装ansible的依赖
2.3、安装ansible的依赖
### 安装Jinja2 ###
[chenlei@node-ansible ~]$ sudo pip install Jinja2
### 安装PyYAML ###
[chenlei@node-ansible ~]$ sudo pip install PyYAML2.4、测试ansible
1、在ansible-master上生成RSA密钥对，用于在SSH链接时免密登录
### 生成RSA密钥对 ###
[chenlei@node-ansible ~]$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/chenlei/.ssh/id_rsa):
Created directory '/home/chenlei/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/chenlei/.ssh/id_rsa.
Your public key has been saved in /home/chenlei/.ssh/id_rsa.pub.
The key fingerprint is:
b1:ed:58:8f:81:1f:de:6f:54:35:43:bb:36:d0:fa:73 chenlei@node-ansible
The key's randomart image is:
+--[ RSA 2048]----+
| .. |
| .oo|
| . . o+|
| = o o|
| S = . = |
| * * + .|
| . = o. oE|
| .. o|
| .. |
+-----------------+
### 将公钥复制到其他被管节点上 ###
[chenlei@node-ansible ~]$ ssh-copy-id root@192.168.119.160
[chenlei@node-ansible ~]$ ssh-copy-id root@192.168.119.161
[chenlei@node-ansible ~]$ ssh-copy-id root@192.168.119.162
### 测试ssh连接是否需要密码 ###
[chenlei@node-ansible ~]$ ssh root@192.168.119.160
[chenlei@node-ansible ~]$ ssh root@192.168.119.161
[chenlei@node-ansible ~]$ ssh root@192.168.119.1622、创建ansible默认hosts文件
[chenlei@node-ansible ~]$ vi /etc/ansible/hosts
192.168.119.160
192.168.119.161
192.168.119.162
### 测试 ###
[chenlei@node-ansible ~]$ ansible all -m ping -u root
192.168.119.160 | SUCCESS => {
"changed": false,
"failed": false,
"ping": "pong"
}
192.168.119.162 | SUCCESS => {
"changed": false,
"failed": false,
"ping": "pong"
}
192.168.119.161 | SUCCESS => {
"changed": false,
"failed": false,
"ping": "pong"
}all表示全部主机，-m用来指定使用的module，-u用来指定远端用户
2.5、编写playbook
2.5.1、目录结构
[chenlei@node-ansible test-01]$ pwd
/home/chenlei/playbooks/test-01
[chenlei@node-ansible test-01]$ tree .
.
├── application.yml # 应用部署playbook文件
├── group_vars/ # group变量存放路径
│ └── master # master组变量
├── host_vars/ # host变量存放目录，目前没有使用
├── kubernetes.yml # 集群部署playbook文件
├── master.yml # 部署master的playbook文件
├── node.yml # 部署node的playbook文件
├── production # 生产环境的inventory文件，目前没有使用
├── reset.yml # 集群重置的playbook文件
├── roles/ # 存放role的文件夹，role本质上就是一个固定目录结构
│ ├── common/ # common role，包含公共预处理task和文件
│ │ ├── files/
│ │ │ └── k8s.conf
│ │ └── tasks/
│ │ └── main.yml
│ ├── dashboard/ # dashboard role，包含dashboard部署task
│ │ └── tasks/
│ │ └── main.yml
│ ├── docker/ # docker role，包括docker安装启动task和镜像导入task
│ │ ├── defaults/
│ │ ├── files/
│ │ ├── handlers/
│ │ ├── meta/
│ │ ├── tasks/
│ │ │ └── main.yml
│ │ ├── templates/
│ │ └── vars/
│ ├── kubelet/ # kubelet role，包含kubelet安装启动task
│ │ └── tasks/
│ │ └── main.yml
│ ├── master/ # master role，包含master初始化和设置集群加入脚本fact
│ │ └── tasks/
│ │ └── main.yml
│ └── node/ # node role，包含node加入集群的task
│ └── tasks/
│ └── main.yml
└── stage # 演示环境的inventory文件
22 directories, 15 files为了学习和了解ansible，上述很多目录列举出来，但是没有使用，可以删除。
2.5.2、文件内容
1、主playbook文件：kubernetes.yml
[chenlei@node-ansible test-01]$ vi kubernetes.yml
--- - import_playbook: master.yml - import_playbook: node.yml - import_playbook: application.yml
[chenlei@node-ansible test-01]$ vi master.yml
--- - hosts: master roles:
- common - docker - kubelet - master
[chenlei@node-ansible test-01]$ vi node.yml
---
- hosts: node roles:
- common - docker - kubelet - node
[chenlei@node-ansible test-01]$ vi application.yml
--- - hosts: master roles:
- dashboardAnsible使用这种文件拆分和引用的方式来更好的重用和管理配置信息
2、集群重置playbook文件：reset.yml
[chenlei@node-ansible test-01]$ vi reset.yml
---
- hosts: master,node tasks:
- name: Reset node shell: kubeadm reset3、Inventory文件内容
[chenlei@node-ansible test-01]$ vi stage
[master]
192.168.119.160
[node]
192.168.119.161
192.168.119.162本文使用stage清单文件，production文件为空。标准情况下，stage用于配置演示或测试环境，production存放生产环境。
中括号表示group
4、角色配置文件
- common role
[chenlei@node-ansible roles]$ vi common/files/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
[chenlei@node-ansible roles]$ vi common/tasks/main.yml
---
- name: Be sure swap is off
command: /usr/sbin/swapoff -a
- name: Be sure selinux is off
command: /usr/sbin/setenforce 0
- name: Stop and disable firewall
systemd:
name: firewalld
state: stopped
enabled: no
- name: Set sysctl file
copy:
src: k8s.conf
dest: /etc/sysctl.d/k8s.conf
- name: Make up sysctl file
shell: sysctl --system- docker role
[chenlei@node-ansible roles]$ vi docker/tasks/main.yml
---
- name: Be sure docker is installed
yum:
pkg: docker
state: installed
tags: docker
- name: Be sure docker is running and enable
systemd:
name: docker
enabled: yes
state: started
daemon_reload: yes
- name: Send docker images
copy: src=~/kubernetes/images/ dest=~/kubernetes/images/
- name: Load docker images
shell: for file in ~/kubernetes/images/*; do docker load -i $file; done;- kubelet role
[chenlei@node-ansible roles]$ vi kubelet/tasks/main.yml
---
- name: Send rpm packages
copy: src=~/kubernetes/rpm/ dest=~/kubernetes/rpm/
- name: Be sure kubelet is installed
yum:
name: /root/kubernetes/rpm/kubeadm-1.9.2-0.x86_64.rpm,/root/kubernetes/rpm/kubectl-1.9.2-0.x86_64.rpm,/root/kubernetes/rpm/kubelet-1.9.2-0.x86_64.rpm,/root/kubernetes/rpm/kubernetes-cni-0.6.0-0.x86_64.rpm,/root/kubernetes/rpm/socat-1.7.3.2-2.el7.x86_64.rpm
state: present
- name: Be sure kubelet is enabled
systemd:
name: kubelet
enabled: yes- master role
[chenlei@node-ansible roles]$ vi master/tasks/main.yml
---
- name: Exec kubeadm init shell: kubeadm init --kubernetes-version=v1.9.2 --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address={{ apiserver_address }}
- name: Set cluster config shell: HOME=~; sudo cp /etc/kubernetes/admin.conf $HOME/.kube/config; sudo chown $(id -u):$(id -g) $HOME/.kube/config
- name: Generate cluster join script shell: kubeadm token create --print-join-command
register: cluster_join
- set_fact: cluster_join_script={{ cluster_join.stdout }}
- name: send yaml file copy:
src=~/kubernetes/yml/
dest=~/kubernetes/yml/
- name: set up cni shell: kubectl apply -f ~/kubernetes/yml/kube-flannel.yml这里设置了cluster_join_script，用于在node中执行kubeadm join
- node role
[chenlei@node-ansible roles]$ vi node/tasks/main.yml --- - set_fact: cluster_join_script={{ hostvars[groups['master'][0]]['cluster_join_script'] }} - name: Join kubernetes cluster shell: echo '' | {{ cluster_join_script }} when: cluster_join_script is definedcluster_join_script取自master组中的第一个主机的cluster_join_script，也就是上面设置的内容。hostvars是ansible内置变量。
- dashboard role
[chenlei@node-ansible roles]$ vi dashboard/tasks/main.yml
---
- name: Deploy kubernetes dashboard shell: kubectl apply -f ~/kubernetes/yml/kubernetes-dashboard.yml
- name: Create dashboard account shell: kubectl apply -f ~/kubernetes/yml/kubernetes-dashboard-rbac-admin.yml2.6、运行playbook
[chenlei@node-ansible test-01]$ ansible-playbook -i stage -u root kubernetes.yml
### 重置 ###
[chenlei@node-ansible test-01]$ ansible-playbook -i stage -u root reset.yml2.7、检查k8s集群状态
[root@k8s-master ~]# kubectl get cs
NAME STATUS MESSAGE ERROR
controller-manager Healthy ok
scheduler Healthy ok
etcd-0 Healthy {"health": "true"}
[root@k8s-master ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master Ready master 4m v1.9.2
k8s-node-1 Ready <none> 3m v1.9.2
k8s-node-2 Ready <none> 3m v1.9.2
[root@k8s-master ~]# kubectl get pods --all-namespaces -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE
kube-system etcd-k8s-master 1/1 Running 0 2m 192.168.119.160 k8s-master
kube-system kube-apiserver-k8s-master 1/1 Running 0 2m 192.168.119.160 k8s-master
kube-system kube-controller-manager-k8s-master 1/1 Running 0 2m 192.168.119.160 k8s-master
kube-system kube-dns-6f4fd4bdf-mwddx 3/3 Running 0 3m 10.244.0.19 k8s-master
kube-system kube-flannel-ds-nwxcl 1/1 Running 0 2m 192.168.119.162 k8s-node-2
kube-system kube-flannel-ds-pdxhs 1/1 Running 0 2m 192.168.119.161 k8s-node-1
kube-system kube-flannel-ds-qrkwd 1/1 Running 0 3m 192.168.119.160 k8s-master
kube-system kube-proxy-7lsqw 1/1 Running 0 2m 192.168.119.161 k8s-node-1
kube-system kube-proxy-f9kps 1/1 Running 0 2m 192.168.119.162 k8s-node-2
kube-system kube-proxy-g47nx 1/1 Running 0 3m 192.168.119.160 k8s-master
kube-system kube-scheduler-k8s-master 1/1 Running 0 2m 192.168.119.160 k8s-master
kube-system kubernetes-dashboard-845747bdd4-gcj47 1/1 Running 0 2m 10.244.0.20 k8s-master
[root@k8s-master ~]# kubectl get svc --all-namespaces
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 3m
kube-system kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP 3m
kube-system kubernetes-dashboard NodePort 10.111.158.221 <none> 443:32286/TCP 2m2.8、访问dashboard
浏览器访问：https://192.168.119.160:32286
3、参考资料
http://www.ansible.com.cn/index.html
http://docs.ansible.com/ansible/latest/index.html
官方文档的Module index中有大量的使用范例可供参考