K8S从入门到放弃系列-(5)kubernetes集群之kube-apiserver部署

时间:2022-05-08 20:31:47
摘要:
1、kube-apiserver为是整个k8s集群中的数据总线和数据中心,提供了对集群的增删改查及watch等HTTP Rest接口
2、kube-apiserver是无状态的,虽然客户端如kubelet可通过启动参数"--api-servers"指定多个api-server,但只有第一个生效,并不能达到高可用的效果,关于kube-apiserver高可用方案,我们在后面介绍,本章,之介绍如何安装。
创建k8s集群各组件运行用户

安全性考虑,我们创建单独的用户运行k8s中各组件
[root@k8s-master01 ~]# ansible k8s-master -m group -a 'name=kube'
[root@k8s-master01 ~]# ansible k8s-master -m user -a 'name=kube group=kube comment="Kubernetes user" shell=/sbin/nologin createhome=no'
1)创建kube-apiserver证书请求文件
apiserver TLS 认证端口需要的证书
[root@k8s-master01 ~]# vim /opt/k8s/certs/kube-apiserver-csr.json
{
"CN": "kubernetes",
"hosts": [
"127.0.0.1",
"10.10.0.18",
"10.10.0.19",
"10.10.0.20",
"10.254.0.1",
"localhost",
"kubernetes",
"kubernetes.default",
"kubernetes.default.svc",
"kubernetes.default.svc.cluster",
"kubernetes.default.svc.cluster.local"
],
"key": {
"algo": "rsa",
"size":
},
"names": [
{
"C": "CN",
"ST": "ShangHai",
"L": "ShangHai",
"O": "k8s",
"OU": "System"
}
]
}
hosts字段列表中,指定了master节点ip,本地ip,10.254.0.1为集群service ip一般为设置的网络段中第一个ip
2)生成 kubernetes 证书和私钥

[root@k8s-master01 certs]# cfssl gencert -ca=/etc/kubernetes/ssl/ca.pem \
-ca-key=/etc/kubernetes/ssl/ca-key.pem \
-config=/opt/k8s/certs/ca-config.json \
-profile=kubernetes kube-apiserver-csr.json | cfssljson -bare kube-apiserver
// :: [INFO] generate received request
// :: [INFO] received CSR
// :: [INFO] generating key: rsa-
// :: [INFO] encoded CSR
// :: [INFO] signed certificate with serial number
// :: [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1., from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2. ("Information Requirements").
3)查看证书生成

[root@k8s-master01 certs]# ll kube-apiserver*
-rw-r--r-- root root Apr : kube-apiserver.csr
-rw-r--r-- root root Apr : kube-apiserver-csr.json
-rw------- root root Apr : kube-apiserver-key.pem
-rw-r--r-- root root Apr : kube-apiserver.pem
4)证书分发

[root@k8s-master01 ~]# ansible k8s-master -m copy -a 'src=/opt/k8s/certs/kube-apiserver.pem dest=/etc/kubernetes/ssl'
[root@k8s-master01 ~]# ansible k8s-master -m copy -a 'src=/opt/k8s/certs/kube-apiserver-key.pem dest=/etc/kubernetes/ssl'
5)配置kube-apiserver客户端使用的token文件

kubelet 启动时向 kube-apiserver发送注册信息,在双向的TLS加密通信环境中需要认证,手工为kubelet生成证书/私钥在node节点较少且数量固定时可行,采用TLS Bootstrapping 机制,可使大量的node节点自动完成向kube-apiserver的注册请求。
原理:kubelet 首次启动时向 kube-apiserver 发送 TLS Bootstrapping 请求,kube-apiserver 验证 kubelet 请求中的 token 是否与它配置的 token.csv 一致,如果一致则自动为 kubelet生成证书和秘钥。
[root@k8s-master01 ~]# head -c  /dev/urandom | od -An -t x | tr -d ' '
fb8f04963e38858eab0867e8d2296d6b
[root@k8s-master01 ~]# vim /opt/k8s/cfg/bootstrap-token.csv
fb8f04963e38858eab0867e8d2296d6b,kubelet-bootstrap,,"system:kubelet-bootstrap"
##分发token文件
[root@k8s-master01 ~]# ansible k8s-master -m copy -a 'src=/opt/k8s/cfg/bootstrap-token.csv dest=/etc/kubernetes/config/'
6)生成 apiserver RBAC 审计配置文件

[root@k8s-master01 ~]# vim /opt/k8s/cfg/audit-policy.yaml
# Log all requests at the Metadata level.
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
- level: Metadata
##分发审计文件
[root@k8s-master01 ~]# ansible k8s-master -m copy -a 'src=/opt/k8s/cfg/audit-policy.yaml dest=/etc/kubernetes/config/'
7)编辑kube-apiserver核心文件

apiserver 启动参数配置文件,注意创建参数中涉及的日志目录,并授权kube用户访问
[root@k8s-master01 ~]# vim /opt/k8s/cfg/kube-apiserver.conf
###
# kubernetes system config
#
# The following values are used to configure the kube-apiserver
#

# The address on the local server to listen to.
KUBE_API_ADDRESS="--advertise-address=10.10.0.18 --bind-address=0.0.0.0"

# The port on the local server to listen on.
KUBE_API_PORT="--secure-port=6443"

# Port minions listen on
# KUBELET_PORT="--kubelet-port=10250"

# Comma separated list of nodes in the etcd cluster
KUBE_ETCD_SERVERS="--etcd-servers=https://10.10.0.18:2379,https://10.10.0.19:2379,https://10.10.0.20:2379"

# Address range to use for services
KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range=10.254.0.0/16"

# default admission control policies
KUBE_ADMISSION_CONTROL="--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,Priority,ResourceQuota"

# Add your own!
KUBE_API_ARGS=" --allow-privileged=true \
                --anonymous-auth=false \
                --alsologtostderr \
                --apiserver-count=3 \
                --audit-log-maxage=30 \
                --audit-log-maxbackup=3 \
                --audit-log-maxsize=100 \
                --audit-log-path=/var/log/kube-audit/audit.log \
                --audit-policy-file=/etc/kubernetes/config/audit-policy.yaml \
                --authorization-mode=Node,RBAC \
                --client-ca-file=/etc/kubernetes/ssl/ca.pem \
                --token-auth-file=/etc/kubernetes/config/bootstrap-token.csv \
                --enable-bootstrap-token-auth \
                --enable-garbage-collector \
                --enable-logs-handler \
                --endpoint-reconciler-type=lease \
                --etcd-cafile=/etc/kubernetes/ssl/ca.pem \
                --etcd-certfile=/etc/kubernetes/ssl/etcd.pem \
                --etcd-keyfile=/etc/kubernetes/ssl/etcd-key.pem \
                --etcd-compaction-interval=0s \
                --event-ttl=168h0m0s \
                --kubelet-https=true \
                --kubelet-certificate-authority=/etc/kubernetes/ssl/ca.pem \
                --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem \
                --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem \
                --kubelet-timeout=3s \
                --runtime-config=api/all=true \
                --service-node-port-range=30000-50000 \
                --service-account-key-file=/etc/kubernetes/ssl/ca-key.pem \
                --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem \
                --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem \
                --v=2"
##分发参数配置文件,同时把参数中出现的IP修改为对应的本机IP
[root@k8s-master01 ~]# ansible k8s-master -m copy -a 'src=/opt/k8s/cfg/kube-apiserver.conf dest=/etc/kubernetes/config/'
##创建日志目录并授权
[root@k8s-master01 ~]# ansible k8s-master -m file -a 'path=/var/log/kube-audit state=directory owner=kube group=kube'
个别参数解释:
  1. KUBE_API_ADDRESS:向集群成员通知apiserver消息的IP地址。这个地址必须能够被集群中其他成员访问。如果IP地址为空,将会使用--bind-address,如果未指定--bind-address,将会使用主机的默认接口地址
  2. KUBE_API_PORT:用于监听具有认证授权功能的HTTPS协议的端口。如果为0,则不会监听HTTPS协议。 (默认值6443)
  3. KUBE_ETCD_SERVERS:连接的etcd服务器列表
  4. KUBE_ADMISSION_CONTROL:控制资源进入集群的准入控制插件的顺序列表
  5. apiserver-count:集群中apiserver数量
  6. KUBE_SERVICE_ADDRESSES: CIDR IP范围,用于分配service 集群IP。不能与分配给节点pod的任何IP范围重叠

kube-apiserver启动脚本配置文件kube-apiserver.service
[root@k8s-master01 ~]# vim /opt/k8s/unit/kube-apiserver.service
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target
After=etcd.service [Service]
EnvironmentFile=-/etc/kubernetes/config/kube-apiserver.conf
User=kube
ExecStart=/usr/local/bin/kube-apiserver \
$KUBE_LOGTOSTDERR \
$KUBE_LOG_LEVEL \
$KUBE_ETCD_SERVERS \
$KUBE_API_ADDRESS \
$KUBE_API_PORT \
$KUBELET_PORT \
$KUBE_ALLOW_PRIV \
$KUBE_SERVICE_ADDRESSES \
$KUBE_ADMISSION_CONTROL \
$KUBE_API_ARGS
Restart=on-failure
Type=notify
LimitNOFILE= [Install]
WantedBy=multi-user.target
## 分发apiserver启动脚本文件
[root@k8s-master01 ~]# ansible k8s-master -m copy -a 'src=/opt/k8s/unit/kube-apiserver.service dest=/usr/lib/systemd/system/'
8)启动kube-apiserver 服务

[root@k8s-master01 ~]# ansible k8s-master -m shell -a 'systemctl daemon-reload'
[root@k8s-master01 ~]# ansible k8s-master -m shell -a 'systemctl enable kube-apiserver'
[root@k8s-master01 ~]# ansible k8s-master -m shell -a 'systemctl start kube-apiserver'

9)授予 kubernetes 证书访问 kubelet API 的权限

[root@k8s-master01 ~]# kubectl create clusterrolebinding kube-apiserver:kubelet-apis --clusterrole=system:kubelet-api-admin --user kubernetes
后面部署好集群,在执行 kubectl exec、run、logs 等命令时,apiserver 会转发到 kubelet。这里定义 RBAC 规则,授权 apiserver 调用 kubelet API,否则会报类似以下错误:
Error from server (Forbidden): Forbidden (user=kubernetes, verb=get, resource=nodes, subresource=proxy) ( pods/log nginx-8477bdff5d-2lf7k)