cordon节点,drain驱逐节点,delete 节点

时间:2022-11-05 18:04:39

服务器版本 docker软件版本 Kubernetes(k8s)集群版本 CPU架构
CentOS Linux release 7.4.1708 (Core) Docker version 20.10.12 v1.21.9 x86_64

Kubernetes集群架构:k8scloude1作为master节点,k8scloude2,k8scloude3作为worker节点

服务器 操作系统版本 CPU架构 进程 功能描述
k8scloude1/192.168.110.130 CentOS Linux release 7.4.1708 (Core) x86_64 docker,kube-apiserver,etcd,kube-scheduler,kube-controller-manager,kubelet,kube-proxy,coredns,calico k8s master节点
k8scloude2/192.168.110.129 CentOS Linux release 7.4.1708 (Core) x86_64 docker,kubelet,kube-proxy,calico k8s worker节点
k8scloude3/192.168.110.128 CentOS Linux release 7.4.1708 (Core) x86_64 docker,kubelet,kube-proxy,calico k8s worker节点

二.前言

本文介绍cordon节点,drain驱逐节点,delete 节点,在对k8s集群节点执行维护(例如内核升级、硬件维护等)时候会用到。

cordon节点,drain驱逐节点,delete 节点的前提是已经有一套可以正常运行的Kubernetes集群,关于Kubernetes(k8s)集群的安装部署,可以查看博客《Centos7 安装部署Kubernetes(k8s)集群》https://www.cnblogs.com/renshengdezheli/p/16686769.html

三.cordon节点

3.1 cordon节点概览

cordon 节点会使其停止调度,会将node状态调为SchedulingDisabled,之后再创建新pod,新pod不会被调度到该节点,原有的pod不会受到影响,仍正常对外提供服务。

3.2 cordon节点

创建目录存放yaml文件

[root@k8scloude1 ~]# mkdir deploy

[root@k8scloude1 ~]# cd deploy/

使用--dry-run生成deploy配置文件

[root@k8scloude1 deploy]# kubectl create deploy nginx --image=nginx --dry-run=client -o yaml >nginx.yaml

[root@k8scloude1 deploy]# cat nginx.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    app: nginx
  name: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: nginx
    spec:
      containers:
      - image: nginx
        name: nginx
        resources: {}
status: {}

修改deploy配置文件,replicas: 5表示副本数为 5,deploy将创建5个pod

[root@k8scloude1 deploy]# vim nginx.yaml 

#修改配置文件:
# replicas: 5  副本数修改为5
#terminationGracePeriodSeconds: 0  宽限期修改为0
# imagePullPolicy: IfNotPresent  镜像下载策略为存在镜像就不下载
[root@k8scloude1 deploy]# cat nginx.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    app: nginx
  name: nginx
spec:
  replicas: 5
  selector:
    matchLabels:
      app: nginx
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: nginx
    spec:
      terminationGracePeriodSeconds: 0
      containers:
      - image: nginx
        name: nginx
        imagePullPolicy: IfNotPresent
        resources: {}
status: {}

创建deploy和使用pod yaml文件创建pod

[root@k8scloude1 deploy]# cat pod.yaml 
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: pod1
  name: pod1
spec:
  terminationGracePeriodSeconds: 0
  containers:
  - image: nginx
    imagePullPolicy: IfNotPresent
    name: n1
    resources: {}
  dnsPolicy: ClusterFirst
  restartPolicy: Always
status: {}

[root@k8scloude1 deploy]# kubectl apply -f pod.yaml 
pod/pod1 created

[root@k8scloude1 deploy]# kubectl apply -f nginx.yaml 
deployment.apps/nginx created

查看pod,可以看到deploy生成5个pod(nginx-6cf858f6cf-XXXXXXX),还有一个pod1。

[root@k8scloude1 deploy]# kubectl get pods -o wide
NAME                     READY   STATUS    RESTARTS   AGE   IP               NODE         NOMINATED NODE   READINESS GATES
nginx-6cf858f6cf-fwhmh   1/1     Running   0          52s   10.244.251.217   k8scloude3   <none>           <none>
nginx-6cf858f6cf-hr6bn   1/1     Running   0          52s   10.244.251.218   k8scloude3   <none>           <none>
nginx-6cf858f6cf-j2ccs   1/1     Running   0          52s   10.244.112.161   k8scloude2   <none>           <none>
nginx-6cf858f6cf-l7n4w   1/1     Running   0          52s   10.244.112.162   k8scloude2   <none>           <none>
nginx-6cf858f6cf-t6qxq   1/1     Running   0          52s   10.244.112.163   k8scloude2   <none>           <none>
pod1                     1/1     Running   0          60s   10.244.251.216   k8scloude3   <none>           <none>

假设某天要对k8scloude2进行维护测试,不希望k8scloude2节点上被分配新的pod可以对某个节点执行cordon之后,此节点就不会再调度新的pod了

cordon k8scloude2节点,k8scloude2节点变为SchedulingDisabled状态

[root@k8scloude1 deploy]# kubectl cordon k8scloude2
node/k8scloude2 cordoned

[root@k8scloude1 deploy]# kubectl get nodes
NAME         STATUS                     ROLES                  AGE     VERSION
k8scloude1   Ready                      control-plane,master   8d      v1.21.0
k8scloude2   Ready,SchedulingDisabled   <none>                 7d23h   v1.21.0
k8scloude3   Ready                      <none>                 7d23h   v1.21.0

kubectl scale deploy命令使nginx deploy的副本数扩展为10个

[root@k8scloude1 deploy]# kubectl scale deploy nginx --replicas=10
deployment.apps/nginx scaled

查看pod,可以发现新生成的pod都被调度到到k8scloude3上,某个节点被cordon之后,新的pod将不被调度到该节点,原先的pod不变

[root@k8scloude1 deploy]# kubectl get pod -o wide
NAME                     READY   STATUS    RESTARTS   AGE     IP               NODE         NOMINATED NODE   READINESS GATES
nginx-6cf858f6cf-7fdnr   1/1     Running   0          4s      10.244.251.221   k8scloude3   <none>           <none>
nginx-6cf858f6cf-fwhmh   1/1     Running   0          9m9s    10.244.251.217   k8scloude3   <none>           <none>
nginx-6cf858f6cf-g92ls   1/1     Running   0          4s      10.244.251.219   k8scloude3   <none>           <none>
nginx-6cf858f6cf-hr6bn   1/1     Running   0          9m9s    10.244.251.218   k8scloude3   <none>           <none>
nginx-6cf858f6cf-j2ccs   1/1     Running   0          9m9s    10.244.112.161   k8scloude2   <none>           <none>
nginx-6cf858f6cf-l7n4w   1/1     Running   0          9m9s    10.244.112.162   k8scloude2   <none>           <none>
nginx-6cf858f6cf-lsvsg   1/1     Running   0          4s      10.244.251.223   k8scloude3   <none>           <none>
nginx-6cf858f6cf-mpwjl   1/1     Running   0          4s      10.244.251.222   k8scloude3   <none>           <none>
nginx-6cf858f6cf-s8x6b   1/1     Running   0          4s      10.244.251.220   k8scloude3   <none>           <none>
nginx-6cf858f6cf-t6qxq   1/1     Running   0          9m9s    10.244.112.163   k8scloude2   <none>           <none>
pod1                     1/1     Running   0          9m17s   10.244.251.216   k8scloude3   <none>           <none>

来个极端的例子,先把deploy的副本数变为0,再变为10,此时所有的pod都运行在k8scloude3节点了。

[root@k8scloude1 deploy]# kubectl scale deploy nginx --replicas=0
deployment.apps/nginx scaled

[root@k8scloude1 deploy]# kubectl get pod -o wide
NAME   READY   STATUS    RESTARTS   AGE   IP               NODE         NOMINATED NODE   READINESS GATES
pod1   1/1     Running   0          10m   10.244.251.216   k8scloude3   <none>           <none>

[root@k8scloude1 deploy]# kubectl scale deploy nginx --replicas=10
deployment.apps/nginx scaled

[root@k8scloude1 deploy]# kubectl get pod -o wide
NAME                     READY   STATUS    RESTARTS   AGE   IP               NODE         NOMINATED NODE   READINESS GATES
nginx-6cf858f6cf-5cx9s   1/1     Running   0          8s    10.244.251.231   k8scloude3   <none>           <none>
nginx-6cf858f6cf-6cblj   1/1     Running   0          8s    10.244.251.228   k8scloude3   <none>           <none>
nginx-6cf858f6cf-827cz   1/1     Running   0          8s    10.244.251.233   k8scloude3   <none>           <none>
nginx-6cf858f6cf-b989n   1/1     Running   0          8s    10.244.251.229   k8scloude3   <none>           <none>
nginx-6cf858f6cf-kwxhn   1/1     Running   0          8s    10.244.251.224   k8scloude3   <none>           <none>
nginx-6cf858f6cf-ljjxz   1/1     Running   0          8s    10.244.251.225   k8scloude3   <none>           <none>
nginx-6cf858f6cf-ltrpr   1/1     Running   0          8s    10.244.251.227   k8scloude3   <none>           <none>
nginx-6cf858f6cf-lwf7g   1/1     Running   0          8s    10.244.251.230   k8scloude3   <none>           <none>
nginx-6cf858f6cf-xw84l   1/1     Running   0          8s    10.244.251.226   k8scloude3   <none>           <none>
nginx-6cf858f6cf-zpwhq   1/1     Running   0          8s    10.244.251.232   k8scloude3   <none>           <none>
pod1                     1/1     Running   0          11m   10.244.251.216   k8scloude3   <none>           <none>

3.3 uncordon节点

要让节点恢复调度pod,uncordon即可。

uncordon k8scloude2节点,k8scloude2节点状态变为Ready,恢复调度。

#需要uncordon
[root@k8scloude1 deploy]# kubectl uncordon k8scloude2
node/k8scloude2 uncordoned

[root@k8scloude1 deploy]# kubectl get nodes
NAME         STATUS   ROLES                  AGE   VERSION
k8scloude1   Ready    control-plane,master   8d    v1.21.0
k8scloude2   Ready    <none>                 8d    v1.21.0
k8scloude3   Ready    <none>                 8d    v1.21.0

四.drain节点

4.1 drain节点概览

在对节点执行维护(例如内核升级、硬件维护等)之前, 可以使用 kubectl drain 从节点安全地逐出所有 Pods。 安全的驱逐过程允许 Pod 的容器 体面地终止, 并确保满足指定的 PodDisruptionBudgets,PodDisruptionBudget 是一个对象,用于定义可能对一组 Pod 造成的最大干扰。。
说明: 默认情况下, kubectl drain 将忽略节点上不能杀死的特定系统 Pod; 'drain' 驱逐或删除除镜像 pod 之外的所有 pod(不能通过 API 服务器删除)。如果有 daemon set-managed pods,drain 不会在没有 --ignore-daemonsets 的情况下继续进行,并且无论如何它都不会删除任何 daemon set-managed pods,因为这些 pods 将立即被 daemon set 控制器替换,它会忽略不可调度的标记。如果有任何 pod 既不是镜像 pod,也不是由复制控制器、副本集、守护程序集、有状态集或作业管理的,那么除非您使用 --force,否则 drain 不会删除任何 pod。如果一个或多个 pod 的管理资源丢失, --force 也将允许继续删除。

kubectl drain 的成功返回,表明所有的 Pods(除了上一段中描述的被排除的那些), 已经被安全地逐出(考虑到期望的终止宽限期和你定义的 PodDisruptionBudget)。 然后就可以安全地关闭节点, 比如关闭物理机器的电源,如果它运行在云平台上,则删除它的虚拟机。

4.2 drain 节点

查看node状态和pod

[root@k8scloude1 deploy]# kubectl get nodes
NAME         STATUS   ROLES                  AGE   VERSION
k8scloude1   Ready    control-plane,master   8d    v1.21.0
k8scloude2   Ready    <none>                 8d    v1.21.0
k8scloude3   Ready    <none>                 8d    v1.21.0

[root@k8scloude1 deploy]# kubectl get pod -o wide
NAME                     READY   STATUS    RESTARTS   AGE   IP               NODE         NOMINATED NODE   READINESS GATES
nginx-6cf858f6cf-58wnd   1/1     Running   0          65s   10.244.112.167   k8scloude2   <none>           <none>
nginx-6cf858f6cf-5rrk4   1/1     Running   0          65s   10.244.112.164   k8scloude2   <none>           <none>
nginx-6cf858f6cf-86wxr   1/1     Running   0          65s   10.244.251.237   k8scloude3   <none>           <none>
nginx-6cf858f6cf-89wj9   1/1     Running   0          65s   10.244.112.168   k8scloude2   <none>           <none>
nginx-6cf858f6cf-9njrj   1/1     Running   0          65s   10.244.251.236   k8scloude3   <none>           <none>
nginx-6cf858f6cf-hchtb   1/1     Running   0          65s   10.244.251.234   k8scloude3   <none>           <none>
nginx-6cf858f6cf-mb2ft   1/1     Running   0          65s   10.244.112.166   k8scloude2   <none>           <none>
nginx-6cf858f6cf-nq6zv   1/1     Running   0          65s   10.244.112.169   k8scloude2   <none>           <none>
nginx-6cf858f6cf-pl7ww   1/1     Running   0          65s   10.244.251.235   k8scloude3   <none>           <none>
nginx-6cf858f6cf-sf2w6   1/1     Running   0          65s   10.244.112.165   k8scloude2   <none>           <none>
pod1                     1/1     Running   0          36m   10.244.251.216   k8scloude3   <none>           <none>

drain驱逐节点:drain=cordon+evicted

drain k8scloude2节点,--delete-emptydir-data删除数据,--ignore-daemonsets忽略daemonsets

[root@k8scloude1 deploy]# kubectl drain k8scloude2
node/k8scloude2 cordoned
error: unable to drain node "k8scloude2", aborting command...

There are pending nodes to be drained:
 k8scloude2
cannot delete Pods with local storage (use --delete-emptydir-data to override): kube-system/metrics-server-bcfb98c76-k5dmj
cannot delete DaemonSet-managed Pods (use --ignore-daemonsets to ignore): kube-system/calico-node-nsbfs, kube-system/kube-proxy-lpj8z

[root@k8scloude1 deploy]# kubectl get node
NAME         STATUS                     ROLES                  AGE   VERSION
k8scloude1   Ready                      control-plane,master   8d    v1.21.0
k8scloude2   Ready,SchedulingDisabled   <none>                 8d    v1.21.0
k8scloude3   Ready                      <none>                 8d    v1.21.0

[root@k8scloude1 deploy]# kubectl drain k8scloude2 --ignore-daemonsets
node/k8scloude2 already cordoned
error: unable to drain node "k8scloude2", aborting command...

There are pending nodes to be drained:
 k8scloude2
error: cannot delete Pods with local storage (use --delete-emptydir-data to override): kube-system/metrics-server-bcfb98c76-k5dmj

[root@k8scloude1 deploy]# kubectl drain k8scloude2 --ignore-daemonsets --force --delete-emptydir-data
node/k8scloude2 already cordoned
WARNING: ignoring DaemonSet-managed Pods: kube-system/calico-node-nsbfs, kube-system/kube-proxy-lpj8z
evicting pod pod/nginx-6cf858f6cf-sf2w6
evicting pod pod/nginx-6cf858f6cf-5rrk4
evicting pod kube-system/metrics-server-bcfb98c76-k5dmj
evicting pod pod/nginx-6cf858f6cf-58wnd
evicting pod pod/nginx-6cf858f6cf-mb2ft
evicting pod pod/nginx-6cf858f6cf-89wj9
evicting pod pod/nginx-6cf858f6cf-nq6zv
pod/nginx-6cf858f6cf-5rrk4 evicted
pod/nginx-6cf858f6cf-mb2ft evicted
pod/nginx-6cf858f6cf-sf2w6 evicted
pod/nginx-6cf858f6cf-58wnd evicted
pod/nginx-6cf858f6cf-nq6zv evicted
pod/nginx-6cf858f6cf-89wj9 evicted
pod/metrics-server-bcfb98c76-k5dmj evicted
node/k8scloude2 evicted

查看pod,k8scloude2节点被drain之后,pod都调度到了k8scloude3节点。

节点被drain驱逐的本质就是删除节点上的pod,k8scloude2节点被drain驱逐之后,k8scloude2上运行的pod会被删除。

deploy是一个控制器,会监控pod的副本数,当k8scloude2上的pod被驱逐之后,副本数少于10,于是在可调度的节点创建pod,补足副本数。

单独的pod不具备再生性,删除之后就真删除了,如果k8scloude3被驱逐,则pod pod1会被删除,其他可调度节点也不会再生一个pod1。

[root@k8scloude1 deploy]# kubectl get pod -o wide
NAME                     READY   STATUS    RESTARTS   AGE     IP               NODE         NOMINATED NODE   READINESS GATES
nginx-6cf858f6cf-7gh4z   1/1     Running   0          84s     10.244.251.240   k8scloude3   <none>           <none>
nginx-6cf858f6cf-7lmfd   1/1     Running   0          85s     10.244.251.238   k8scloude3   <none>           <none>
nginx-6cf858f6cf-86wxr   1/1     Running   0          6m14s   10.244.251.237   k8scloude3   <none>           <none>
nginx-6cf858f6cf-9bn2b   1/1     Running   0          85s     10.244.251.243   k8scloude3   <none>           <none>
nginx-6cf858f6cf-9njrj   1/1     Running   0          6m14s   10.244.251.236   k8scloude3   <none>           <none>
nginx-6cf858f6cf-bqk2w   1/1     Running   0          84s     10.244.251.241   k8scloude3   <none>           <none>
nginx-6cf858f6cf-hchtb   1/1     Running   0          6m14s   10.244.251.234   k8scloude3   <none>           <none>
nginx-6cf858f6cf-hjddp   1/1     Running   0          84s     10.244.251.244   k8scloude3   <none>           <none>
nginx-6cf858f6cf-pl7ww   1/1     Running   0          6m14s   10.244.251.235   k8scloude3   <none>           <none>
nginx-6cf858f6cf-sgxfg   1/1     Running   0          84s     10.244.251.242   k8scloude3   <none>           <none>
pod1                     1/1     Running   0          41m     10.244.251.216   k8scloude3   <none>           <none>

查看node节点状态

[root@k8scloude1 deploy]# kubectl get nodes
NAME         STATUS                     ROLES                  AGE   VERSION
k8scloude1   Ready                      control-plane,master   8d    v1.21.0
k8scloude2   Ready,SchedulingDisabled   <none>                 8d    v1.21.0
k8scloude3   Ready                      <none>                 8d    v1.21.0

4.3 uncordon节点

要取消drain某个节点,直接uncordon即可,没有undrain操作。

[root@k8scloude1 deploy]# kubectl undrain k8scloude2
Error: unknown command "undrain" for "kubectl"

Did you mean this?
        drain

Run 'kubectl --help' for usage.

uncordon k8scloude2节点,节点恢复调度

[root@k8scloude1 deploy]# kubectl uncordon k8scloude2
node/k8scloude2 uncordoned

[root@k8scloude1 deploy]# kubectl get nodes
NAME         STATUS   ROLES                  AGE   VERSION
k8scloude1   Ready    control-plane,master   8d    v1.21.0
k8scloude2   Ready    <none>                 8d    v1.21.0
k8scloude3   Ready    <none>                 8d    v1.21.0

把deploy副本数变为0,再变为10,再观察pod分布

[root@k8scloude1 deploy]# kubectl scale deploy nginx --replicas=0
deployment.apps/nginx scaled

[root@k8scloude1 deploy]# kubectl get pods -o wide
NAME   READY   STATUS    RESTARTS   AGE   IP               NODE         NOMINATED NODE   READINESS GATES
pod1   1/1     Running   0          52m   10.244.251.216   k8scloude3   <none>           <none>

[root@k8scloude1 deploy]# kubectl scale deploy nginx --replicas=10
deployment.apps/nginx scaled

k8scloude2节点恢复可调度pod状态

[root@k8scloude1 deploy]# kubectl get pods -o wide
NAME                     READY   STATUS    RESTARTS   AGE   IP               NODE         NOMINATED NODE   READINESS GATES
nginx-6cf858f6cf-4sqj8   1/1     Running   0          6s    10.244.112.172   k8scloude2   <none>           <none>
nginx-6cf858f6cf-cjqxv   1/1     Running   0          6s    10.244.112.176   k8scloude2   <none>           <none>
nginx-6cf858f6cf-fk69r   1/1     Running   0          6s    10.244.112.175   k8scloude2   <none>           <none>
nginx-6cf858f6cf-ghznd   1/1     Running   0          6s    10.244.112.173   k8scloude2   <none>           <none>
nginx-6cf858f6cf-hnxzs   1/1     Running   0          6s    10.244.251.246   k8scloude3   <none>           <none>
nginx-6cf858f6cf-hshnm   1/1     Running   0          6s    10.244.112.171   k8scloude2   <none>           <none>
nginx-6cf858f6cf-jb5sh   1/1     Running   0          6s    10.244.112.170   k8scloude2   <none>           <none>
nginx-6cf858f6cf-l9xlm   1/1     Running   0          6s    10.244.112.174   k8scloude2   <none>           <none>
nginx-6cf858f6cf-pgjlb   1/1     Running   0          6s    10.244.251.247   k8scloude3   <none>           <none>
nginx-6cf858f6cf-rlnh6   1/1     Running   0          6s    10.244.251.245   k8scloude3   <none>           <none>
pod1                     1/1     Running   0          52m   10.244.251.216   k8scloude3   <none>           <none>

删除deploy,删除pod。

[root@k8scloude1 deploy]# kubectl delete -f nginx.yaml 
deployment.apps "nginx" deleted

[root@k8scloude1 deploy]# kubectl delete pod pod1 --force
warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
pod "pod1" force deleted

[root@k8scloude1 deploy]# kubectl get pods -o wide
No resources found in pod namespace.

五.delete 节点

5.1 delete节点概览

delete 删除节点就直接把一个节点就k8s集群中删除了,delete 节点之前需要先drain 节点。

关于delete节点以及重装节点的详细内容,请查看博客《模拟重装Kubernetes(k8s)集群:删除k8s集群然后重装》https://www.cnblogs.com/renshengdezheli/p/16686997.html

5.2 delete节点

kubectl drain 安全驱逐节点上面所有的 pod,--ignore-daemonsets往往需要指定的,这是因为deamonset会忽略SchedulingDisabled标签(使用kubectl drain时会自动给节点打上不可调度SchedulingDisabled标签),因此deamonset控制器控制的pod被删除后,可能马上又在此节点上启动起来,这样就会成为死循环。因此这里忽略daemonset。

[root@k8scloude1 ~]# kubectl drain k8scloude3 --ignore-daemonsets 
node/k8scloude3 cordoned
WARNING: ignoring DaemonSet-managed Pods: kube-system/calico-node-wmz4r, kube-system/kube-proxy-84gcx
evicting pod kube-system/calico-kube-controllers-6b9fbfff44-rl2mh
pod/calico-kube-controllers-6b9fbfff44-rl2mh evicted
node/k8scloude3 evicted

k8scloude3变为SchedulingDisabled

[root@k8scloude1 ~]# kubectl get nodes 
NAME         STATUS                     ROLES                  AGE   VERSION
k8scloude1   Ready                      control-plane,master   64m   v1.21.0
k8scloude2   Ready                      <none>                 56m   v1.21.0
k8scloude3   Ready,SchedulingDisabled   <none>                 56m   v1.21.0

删除节点k8scloude3

[root@k8scloude1 ~]# kubectl delete nodes k8scloude3
node "k8scloude3" deleted

[root@k8scloude1 ~]# kubectl get nodes 
NAME         STATUS   ROLES                  AGE   VERSION
k8scloude1   Ready    control-plane,master   65m   v1.21.0
k8scloude2   Ready    <none>                 57m   v1.21.0