OpenShift DNS的机制

时间:2023-03-08 18:24:52

为什么不直接用kube-dns?

为什么不直接用kube-dns?

为什么不直接用kube-dns?

感谢各位前辈的专研,在下午有限的时间里把Openshift DNS的机制理了一下。更详细的材料大家可以参考

https://blog.cloudtechgroup.cn/Blog/2018/07/23/ocp-2018-07-23/

https://www.redhat.com/en/blog/red-hat-openshift-container-platform-dns-deep-dive-dns-changes-red-hat-openshift-container-platform-36

https://www.cnblogs.com/sammyliu/p/10056035.html

本篇主要是基于3.11版本

1.DNS架构

OpenShift DNS的机制

也就是说所有容器的dns寻址都是通过外部的dnsmasq以及SkyDNS来进行的,不是走容器内部网络去找kube-dns或者类似的Pod

可以通过命令查看一下

  • 在master节点
[root@master dnsmasq.d]# netstat -tunlp|grep
tcp 0.0.0.0: 0.0.0.0:* LISTEN /openshift
tcp 127.0.0.1: 0.0.0.0:* LISTEN /openshift
tcp 10.128.0.1: 0.0.0.0:* LISTEN /dnsmasq
tcp 172.17.0.1: 0.0.0.0:* LISTEN /dnsmasq
tcp 192.168.56.113: 0.0.0.0:* LISTEN /dnsmasq

查看具体的进程,注意进程号

# ps -ef|grep openshift

root           : ?        :: openshift start network --config=/etc/origin/node/node-config.yaml --kubeconfig=/tmp/kubeconfig --loglevel=
root : ? :: openshift start master api --config=/etc/origin/master/master-config.yaml --loglevel=
root : ? :: openshift start master controllers --config=/etc/origin/master/master-config.yaml --listen=https://0.0.0.0:8444 --loglevel=2

查看路由信息

[root@master dnsmasq.d]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.56.1 0.0.0.0 UG enp0s3
10.128.0.0 0.0.0.0 255.252.0.0 U tun0
172.17.0.0 0.0.0.0 255.255.0.0 U docker0
172.30.0.0 0.0.0.0 255.255.0.0 U tun0
192.168.56.0 0.0.0.0 255.255.255.0 U enp0s3

可以看到10.128.0.0是pod网段,172.17.0.0是SVC网段,192.169.56.0是宿主机网段,每个网段都启动了一个dnsmasq,监听在53端口

  • 在node节点
[root@node1 node]# netstat -tunlp|grep
tcp 127.0.0.1: 0.0.0.0:* LISTEN /openshift
tcp 10.131.0.1: 0.0.0.0:* LISTEN /dnsmasq
tcp 172.17.0.1: 0.0.0.0:* LISTEN /dnsmasq
tcp 192.168.56.104: 0.0.0.0:* LISTEN /dnsmasq
ps -ef|grep openshift

root           : ?        :: openshift start network --config=/etc/origin/node/node-config.yaml --kubeconfig=/tmp/kubeconfig --loglevel=

2.DNS的配置信息

Pod中的dns配置会指向Pod所在宿主机IP,配置为

$ cat /etc/resolv.conf
nameserver 192.168.56.105
search myproject.svc.cluster.local svc.cluster.local cluster.local redhat.com example.com
options ndots:

192.168.56.105是pod所在的宿主机

宿主机的resolv.conf文件

[root@node1 node]# cat /etc/resolv.conf
# nameserver updated by /etc/NetworkManager/dispatcher.d/99-origin-dns.sh
# Generated by NetworkManager
search cluster.local cluster.local example.com
nameserver 192.168.56.104

在部署环境时,会在每个节点上部署 /etc/NetworkManager/dispatcher.d/99-origin-dns.sh 文件。每当节点上的 NetworkManager 服务启动时,该文件会被运行。它的任务包括:

  • 创建 dnsmasq 配置文件 :
    • node-dnsmasq.conf (没有)
    • origin-dns.conf
    • origin-upstream-dns.conf(没有)
  • 当 NetworkManager 服务启动时启动 dnsmasq 服务
  • 设置宿主机的所有默认路由 IP 为 Dnsmasq 的侦听IP
  • 修改 /etc/resolv.conf,设置搜索域,以及将宿主机的默认 IP 作为 nameserver
  • 创建 /etc/origin/node/resolv.conf

origin-dns.conf的配置目录在/etc/dnsmasq.d/,内容如下

[root@node1 dnsmasq.d]# cat origin-dns.conf
no-resolv
domain-needed
no-negcache
max-cache-ttl=
enable-dbus
dns-forward-max=
cache-size=
bind-dynamic
min-port=
except-interface=lo
# End of config

如果有文件origin-upstream-dns.conf ,中定义了上游(upstream) DNS 名字服务器,如果没有可以手工创建.

[root@node2 dnsmasq.d]# cat origin-upstream-dns.conf
server=10.72.17.5
server=10.68.5.26
server=202.96.134.33
server=202.96.128.86

如果需要解析外部域名,是需要在pod运行的宿主机节点上进行创建的。

node-dnsmasq.conf的内容是

server=/in-addr.arpa/127.0.0.1
server=/cluster.local/127.0.0.1

根据前辈SammyTalksAboutCloud的研究,这个已经写到程序里面去了。

可以通过journalctl -u dnsmasq去查看日志

[root@node2 dnsmasq.d]# journalctl -u dnsmasq
-- Logs begin at Fri -- :: CST, end at Thu -- :: CST. --
Dec :: node2.example.com systemd[]: Started DNS caching server..
Dec :: node2.example.com dnsmasq[]: started, version 2.76 cachesize
Dec :: node2.example.com dnsmasq[]: compile time options: IPv6 GNU-getopt DBus no-i18n IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth no-D
Dec :: node2.example.com dnsmasq[]: DBus support enabled: connected to system bus
Dec :: node2.example.com dnsmasq[]: warning: no upstream servers configured
Dec :: node2.example.com dnsmasq[]: read /etc/hosts - addresses
Dec :: node2.example.com dnsmasq[]: setting upstream servers from DBus
Dec :: node2.example.com dnsmasq[]: using nameserver 127.0.0.1# for domain in-addr.arpa
Dec :: node2.example.com dnsmasq[]: using nameserver 127.0.0.1# for domain cluster.local

根据日志信息,知道dnsmasq实际把请求转发给了监听在127.0.0.1:53上的skyDNS

skyDNS并不是作为一个单独的进程启动,而是在启动网络

openshift start network --config=/etc/origin/node/node-config.yaml --kubeconfig=/tmp/kubeconfig --loglevel=2

中启动,SkyDNS 调用 OpenShift API 服务来获取主机名、IP地址等信息,然后封装成标准 DNS 记录并返回给查询客户端。

3.DNS在openshift中的配置

  • master
cat /etc/origin/master/master-config.yaml
dnsConfig:
bindAddress: 0.0.0.0:
bindNetwork: tcp4

bind在每个ip的8053端口

  • node
cat /etc/origin/node/node-config.yaml
dnsBindAddress: 127.0.0.1:
dnsDomain: cluster.local
dnsIP: 0.0.0.0
dnsNameservers: null
dnsRecursiveResolvConf: /etc/origin/node/resolv.conf

根据这种机制,service在宿主机范围内(不仅只是在容器中)能够解析

[root@node2 dnsmasq.d]# dig tomcat.myproject.svc.cluster.local

; <<>> DiG 9.9.-RedHat-9.9.-.el7 <<>> tomcat.myproject.svc.cluster.local
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id:
;; flags: qr aa rd ra; QUERY: , ANSWER: , AUTHORITY: , ADDITIONAL: ;; QUESTION SECTION:
;tomcat.myproject.svc.cluster.local. IN A ;; ANSWER SECTION:
tomcat.myproject.svc.cluster.local. IN A 172.30.16.194 ;; Query time: msec
;; SERVER: 10.0.3.15#(10.0.3.15)
;; WHEN: Thu Jan :: CST
;; MSG SIZE rcvd:

能够访问

[root@node2 dnsmasq.d]# curl  tomcat.myproject.svc:

<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<title>Apache Tomcat/8.5.</title>
<link href="favicon.ico" rel="icon" type="image/x-icon" />
<link href="favicon.ico" rel="shortcut icon" type="image/x-icon" />
<link href="tomcat.css" rel="stylesheet" type="text/css" />
</head> <body>

4.查询流程图

OpenShift DNS的机制

查看dnsmasq更详细日志

vi /etc/dnsmasq.conf 

# For debugging purposes, log each DNS query as it passes through
# dnsmasq.
log-queries
systemctl restart dnsmasq
[root@node2 dnsmasq.d]# journalctl -f -u dnsmasq
-- Logs begin at Fri -- :: CST. --
Jan :: node2.example.com dnsmasq[]: using nameserver 10.72.17.5#
Jan :: node2.example.com dnsmasq[]: using nameserver 127.0.0.1# for domain in-addr.arpa
Jan :: node2.example.com dnsmasq[]: using nameserver 127.0.0.1# for domain cluster.local
Jan :: node2.example.com dnsmasq[]: setting upstream servers from DBus
Jan :: node2.example.com dnsmasq[]: using nameserver 202.96.128.86#
Jan :: node2.example.com dnsmasq[]: using nameserver 202.96.134.33#
Jan :: node2.example.com dnsmasq[]: using nameserver 10.68.5.26#
Jan :: node2.example.com dnsmasq[]: using nameserver 10.72.17.5#
Jan :: node2.example.com dnsmasq[]: using nameserver 127.0.0.1# for domain in-addr.arpa
Jan :: node2.example.com dnsmasq[]: using nameserver 127.0.0.1# for domain cluster.local
Jan :: node2.example.com dnsmasq[]: query[A] www.baidu.com from 10.0.3.15
Jan :: node2.example.com dnsmasq[]: forwarded www.baidu.com to 202.96.128.86
Jan :: node2.example.com dnsmasq[]: forwarded www.baidu.com to 202.96.134.33
Jan :: node2.example.com dnsmasq[]: forwarded www.baidu.com to 10.68.5.26
Jan :: node2.example.com dnsmasq[]: forwarded www.baidu.com to 10.72.17.5
Jan :: node2.example.com dnsmasq[]: query[AAAA] www.baidu.com from 10.0.3.15
Jan :: node2.example.com dnsmasq[]: forwarded www.baidu.com to 202.96.128.86
Jan :: node2.example.com dnsmasq[]: forwarded www.baidu.com to 202.96.134.33
Jan :: node2.example.com dnsmasq[]: forwarded www.baidu.com to 10.68.5.26
Jan :: node2.example.com dnsmasq[]: forwarded www.baidu.com to 10.72.17.5
Jan :: node2.example.com dnsmasq[]: reply www.baidu.com is <CNAME>
Jan :: node2.example.com dnsmasq[]: reply www.a.shifen.com is 14.215.177.38
Jan :: node2.example.com dnsmasq[]: reply www.a.shifen.com is 14.215.177.39
Jan :: node2.example.com dnsmasq[]: reply www.baidu.com is <CNAME>