Kubernetes 跨集群流量调度实战 :访问控制

时间:2022-12-29 14:07:03

Kubernetes 跨集群流量调度实战 :访问控制

背景

众所周知,Flomesh 的服务网格产品 osm-edge 是基于 SMI(Service Mesh Interface,服务网格接口) 标准的实现。SMI 定义了流量标识、访问控制、遥测和管理的规范。在 上一篇 中,我们体验过了多集群服务(Multi-Cluster Service,MCS),以及流量在多集群中的调度策略,属于最基本最基础的能力,这篇会带大家体验一下基于 SMI 实现的跨集群访问控制。

在开始之前,先回顾一下 SMI 访问控制的规范。在 osm-edge 中的 流量策略有两种形式宽松模式流量策略模式。前者允许网格中的服务互相访问,后者需要提供相应的流量策略才可以访问。

SMI 访问控制策略

在流量策略模式下,SMI 通过 CRD TrafficTarget 来定义基于 ServiceAccount 的访问控制,其中定义了流量源(sources)、目标(destinations)以及规则(rules)。表达的是使用 sources 中指定 ServiceAccount 的应用可以访问 destinations 指定 ServiceAccount 的应用,可访问的流量由 rules 指定。

比如下面的示例,表示的是使用 ServiceAccount promethues 运行的负载,发送 GET 请求到使用 ServiceAccount service-a 运行的负载的 /metrics 端点。HTTPRouteGroup 定义了流量的标识:即访问端点 /metricsGET 请求。

kind: HTTPRouteGroup
metadata:
  name: the-routes
spec:
  matches:
  - name: metrics
    pathRegex: "/metrics"
    methods:
    - GET
---
kind: TrafficTarget
metadata:
  name: path-specific
  namespace: default
spec:
  destination:
    kind: ServiceAccount
    name: service-a
    namespace: default
  rules:
  - kind: HTTPRouteGroup
    name: the-routes
    matches:
    - metrics
  sources:
  - kind: ServiceAccount
    name: prometheus
    namespace: default

那么在多集群中,访问控制的表现如何呢?

FSM 的 ServiceExport

FSM 的 ServiceExport 用于导出服务到其他集群,也就是服务注册的过程。ServiceExport 的字段 spec.serviceAccountName 可以用来指定服务负载使用的 ServiceAccount

apiVersion: flomesh.io/v1alpha1
kind: ServiceExport
metadata:
  namespace: httpbin
  name: httpbin
spec:
  serviceAccountName: "*"
  rules:
    - portNumber: 8080
      path: "/cluster-1/httpbin-mesh"
      pathType: Prefix

接下来,我们在上次的环境基础上开始演示,没有看过上一篇的小伙伴可以参考 上一篇 来搭建环境。

部署应用

Kubernetes 跨集群流量调度实战 :访问控制

部署示例应用

在集群 cluster-1cluster-3httpbin 命名空间(由网格管理,会注入 sidecar)下,部署 httpbin 应用。这里我们指定 ServiceAccounthttpbin

export NAMESPACE=httpbin
for CLUSTER_NAME in cluster-1 cluster-3
do
  kubectx k3d-${CLUSTER_NAME}
  kubectl create namespace ${NAMESPACE}
  osm namespace add ${NAMESPACE}
  kubectl apply -n ${NAMESPACE} -f - <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: httpbin
---  
apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpbin
  labels:
    app: pipy
spec:
  replicas: 1
  selector:
    matchLabels:
      app: pipy
  template:
    metadata:
      labels:
        app: pipy
    spec:
      serviceAccountName: httpbin
      containers:
        - name: pipy
          image: flomesh/pipy:latest
          ports:
            - containerPort: 8080
          command:
            - pipy
            - -e
            - |
              pipy()
              .listen(8080)
              .serveHTTP(new Message('Hi, I am from ${CLUSTER_NAME} and controlled by mesh!\n'))
---
apiVersion: v1
kind: Service
metadata:
  name: httpbin
spec:
  ports:
    - port: 8080
      targetPort: 8080
      protocol: TCP
  selector:
    app: pipy
---
apiVersion: v1
kind: Service
metadata:
  name: httpbin-${CLUSTER_NAME}
spec:
  ports:
    - port: 8080
      targetPort: 8080
      protocol: TCP
  selector:
    app: pipy
EOF

  sleep 3
  kubectl wait --for=condition=ready pod -n ${NAMESPACE} --all --timeout=60s
done  

cluster-2 的命名空间 httpbin 下 部署 httpbin 应用,但不指定 ServiceAccount,使用默认的 ServiceAccount default

export NAMESPACE=httpbin
export CLUSTER_NAME=cluster-2

kubectx k3d-${CLUSTER_NAME}
kubectl create namespace ${NAMESPACE}
osm namespace add ${NAMESPACE}
kubectl apply -n ${NAMESPACE} -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpbin
  labels:
    app: pipy
spec:
  replicas: 1
  selector:
    matchLabels:
      app: pipy
  template:
    metadata:
      labels:
        app: pipy
    spec:
      containers:
        - name: pipy
          image: flomesh/pipy:latest
          ports:
            - containerPort: 8080
          command:
            - pipy
            - -e
            - |
              pipy()
              .listen(8080)
              .serveHTTP(new Message('Hi, I am from ${CLUSTER_NAME}! and controlled by mesh!\n'))
---
apiVersion: v1
kind: Service
metadata:
  name: httpbin
spec:
  ports:
    - port: 8080
      targetPort: 8080
      protocol: TCP
  selector:
    app: pipy
---
apiVersion: v1
kind: Service
metadata:
  name: httpbin-${CLUSTER_NAME}
spec:
  ports:
    - port: 8080
      targetPort: 8080
      protocol: TCP
  selector:
    app: pipy
EOF

sleep 3
kubectl wait --for=condition=ready pod -n ${NAMESPACE} --all --timeout=60s

在集群 cluster-2 的命名空间 curl 下部署 curl 应用,这个命名空间是被网格管理的,注入的 sidecar 会完全流量的跨集群调度。这里指定使用 ServiceAccout curl

export NAMESPACE=curl
kubectx k3d-cluster-2
kubectl create namespace ${NAMESPACE}
osm namespace add ${NAMESPACE}
kubectl apply -n ${NAMESPACE} -f - <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: curl
---
apiVersion: v1
kind: Service
metadata:
  name: curl
  labels:
    app: curl
    service: curl
spec:
  ports:
    - name: http
      port: 80
  selector:
    app: curl
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: curl
spec:
  replicas: 1
  selector:
    matchLabels:
      app: curl
  template:
    metadata:
      labels:
        app: curl
    spec:
      serviceAccountName: curl
      containers:
      - image: curlimages/curl
        imagePullPolicy: IfNotPresent
        name: curl
        command: ["sleep", "365d"]
EOF

sleep 3
kubectl wait --for=condition=ready pod -n ${NAMESPACE} --all --timeout=60s

导出服务

export NAMESPACE_MESH=httpbin
for CLUSTER_NAME in cluster-1 cluster-3
do
  kubectx k3d-${CLUSTER_NAME}
  kubectl apply -f - <<EOF
apiVersion: flomesh.io/v1alpha1
kind: ServiceExport
metadata:
  namespace: ${NAMESPACE_MESH}
  name: httpbin
spec:
  serviceAccountName: "httpbin"
  rules:
    - portNumber: 8080
      path: "/${CLUSTER_NAME}/httpbin-mesh"
      pathType: Prefix
---
apiVersion: flomesh.io/v1alpha1
kind: ServiceExport
metadata:
  namespace: ${NAMESPACE_MESH}
  name: httpbin-${CLUSTER_NAME}
spec:
  serviceAccountName: "httpbin"
  rules:
    - portNumber: 8080
      path: "/${CLUSTER_NAME}/httpbin-mesh-${CLUSTER_NAME}"
      pathType: Prefix
EOF
sleep 1
done

测试

我们切换回集群 cluster-2

kubectx k3d-cluster-2

默认路由类型是 Locality ,我们需要创建一个 ActiveActive 策略,来允许使用其他集群的服务实例来处理请求。

kubectl apply -n httpbin -f  - <<EOF
apiVersion: flomesh.io/v1alpha1
kind: GlobalTrafficPolicy
metadata:
  name: httpbin
spec:
  lbType: ActiveActive
  targets:
    - clusterKey: default/default/default/cluster-1
    - clusterKey: default/default/default/cluster-3
EOF

cluster-2 集群的 curl 应用中, 我们向 httpbin.httpbin 发送请求。

curl_client="$(kubectl get pod -n curl -l app=curl -o jsonpath='{.items[0].metadata.name}')"

kubectl exec "${curl_client}" -n curl -c curl -- curl -s http://httpbin.httpbin:8080/

多请求几次会看到如下的响应:

Hi, I am from cluster-1 and controlled by mesh!
Hi, I am from cluster-2 and controlled by mesh!
Hi, I am from cluster-3 and controlled by mesh!
Hi, I am from cluster-1 and controlled by mesh!
Hi, I am from cluster-2 and controlled by mesh!
Hi, I am from cluster-3 and controlled by mesh!

演示

调整流量策略模式

我们先将集群 cluster-2 的流量策略模式调整下,这样才能应用流量策略:

kubectx k3d-cluster-2
export osm_namespace=osm-system
kubectl patch meshconfig osm-mesh-config -n "$osm_namespace" -p '{"spec":{"traffic":{"enablePermissiveTrafficPolicyMode":false}}}' --type=merge

此时,再尝试发送请求,会发现请求失败。因为在流量策略模式下,没有配置策略的话应用间的互访是被禁止的。

kubectl exec "${curl_client}" -n curl -c curl -- curl -s http://httpbin.httpbin:8080/
command terminated with exit code 52

应用访问控制策略

文章开头介绍 SMI 的访问控制策略时有提到是基于 ServiceAccount 的,这就是为什么我们部署的 httpbin 服务在集群 cluster-1cluster-3 与集群 cluster-2 中使用的 ServiceAccount 不同:

  • cluster-1:httpbin
  • cluster-2:default
  • clsuter-3:httpbin

接下来,我们会为集群内和集群外的服务,分别设置不同的访问控制策略 TrafficTarget,通过在 TrafficTarget 中目标负载的 ServiceAccount 来进行区分。

Kubernetes 跨集群流量调度实战 :访问控制

执行下面的命令,创建流量策略 curl-to-httpbin,允许 curl 访问命名空间 httpbin 下使用 ServiceAccount default 的负载。

kubectl apply -n httpbin -f - <<EOF
apiVersion: specs.smi-spec.io/v1alpha4
kind: HTTPRouteGroup
metadata:
  name: httpbin-route
spec:
  matches:
  - name: all
    pathRegex: "/"
    methods:
    - GET
---
kind: TrafficTarget
apiVersion: access.smi-spec.io/v1alpha3
metadata:
  name: curl-to-httpbin
spec:
  destination:
    kind: ServiceAccount
    name: default
    namespace: httpbin
  rules:
  - kind: HTTPRouteGroup
    name: httpbin-route
    matches:
    - all
  sources:
  - kind: ServiceAccount
    name: curl
    namespace: curl
EOF

多次发送请求尝试,集群 cluster-2 的服务都会做出响应,而集群 cluster-1cluster-3 不会参与服务。

Hi, I am from cluster-2 and controlled by mesh!
Hi, I am from cluster-2 and controlled by mesh!
Hi, I am from cluster-2 and controlled by mesh!

执行下面命令检查 ServiceImports,可以看到 cluster-1cluster-3 导出的服务使用的是 ServiceAccount httpbin

kubectl get serviceimports httpbin -n httpbin -o jsonpath='{.spec}' | jq

{
  "ports": [
    {
      "endpoints": [
        {
          "clusterKey": "default/default/default/cluster-1",
          "target": {
            "host": "192.168.1.110",
            "ip": "192.168.1.110",
            "path": "/cluster-1/httpbin-mesh",
            "port": 81
          }
        },
        {
          "clusterKey": "default/default/default/cluster-3",
          "target": {
            "host": "192.168.1.110",
            "ip": "192.168.1.110",
            "path": "/cluster-3/httpbin-mesh",
            "port": 83
          }
        }
      ],
      "port": 8080,
      "protocol": "TCP"
    }
  ],
  "serviceAccountName": "httpbin",
  "type": "ClusterSetIP"
}

因此,我们创建另一个 TrafficTarget curl-to-ext-httpbin,允许 curl 访问使用 ServiceAccount httpbin 的负载。

kubectl apply -n httpbin -f - <<EOF
kind: TrafficTarget
apiVersion: access.smi-spec.io/v1alpha3
metadata:
  name: curl-to-ext-httpbin
spec:
  destination:
    kind: ServiceAccount
    name: httpbin
    namespace: httpbin
  rules:
  - kind: HTTPRouteGroup
    name: httpbin-route
    matches:
    - all
  sources:
  - kind: ServiceAccount
    name: curl
    namespace: curl
EOF

应用策略后,再测试一下,所有请求都成功。

Hi, I am from cluster-2 and controlled by mesh!
Hi, I am from cluster-1 and controlled by mesh!
Hi, I am from cluster-3 and controlled by mesh!

总结

MCS(Multi-cluster Service)API 的实现目标是能够像 Service 一样,使用其他集群的服务。虽然在 Kubernetes 本身实现还有漫长的路要走,但作为 SMI 标准实现的服务网格来说,可以将其视作 Service 一样来进行操作。

就如本篇的访问控制一样,SMI 的流量拆分一样也可以支持多集群的服务。预告一下,下一篇就为大家介绍多集群服务的流量拆分。