为什么不直接用kube-dns?
为什么不直接用kube-dns?
为什么不直接用kube-dns?
感谢各位前辈的专研，在下午有限的时间里把Openshift DNS的机制理了一下。更详细的材料大家可以参考
https://blog.cloudtechgroup.cn/Blog/2018/07/23/ocp-2018-07-23/
https://www.redhat.com/en/blog/red-hat-openshift-container-platform-dns-deep-dive-dns-changes-red-hat-openshift-container-platform-36
https://www.cnblogs.com/sammyliu/p/10056035.html
本篇主要是基于3.11版本
1.DNS架构
也就是说所有容器的dns寻址都是通过外部的dnsmasq以及SkyDNS来进行的，不是走容器内部网络去找kube-dns或者类似的Pod
可以通过命令查看一下
- 在master节点
[root@master dnsmasq.d]# netstat -tunlp|grep
tcp 0.0.0.0: 0.0.0.0:* LISTEN /openshift
tcp 127.0.0.1: 0.0.0.0:* LISTEN /openshift
tcp 10.128.0.1: 0.0.0.0:* LISTEN /dnsmasq
tcp 172.17.0.1: 0.0.0.0:* LISTEN /dnsmasq
tcp 192.168.56.113: 0.0.0.0:* LISTEN /dnsmasq
查看具体的进程，注意进程号
# ps -ef|grep openshift root : ? :: openshift start network --config=/etc/origin/node/node-config.yaml --kubeconfig=/tmp/kubeconfig --loglevel=
root : ? :: openshift start master api --config=/etc/origin/master/master-config.yaml --loglevel=
root : ? :: openshift start master controllers --config=/etc/origin/master/master-config.yaml --listen=https://0.0.0.0:8444 --loglevel=2
查看路由信息
[root@master dnsmasq.d]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.56.1 0.0.0.0 UG enp0s3
10.128.0.0 0.0.0.0 255.252.0.0 U tun0
172.17.0.0 0.0.0.0 255.255.0.0 U docker0
172.30.0.0 0.0.0.0 255.255.0.0 U tun0
192.168.56.0 0.0.0.0 255.255.255.0 U enp0s3
可以看到10.128.0.0是pod网段，172.17.0.0是SVC网段，192.169.56.0是宿主机网段，每个网段都启动了一个dnsmasq,监听在53端口
- 在node节点
[root@node1 node]# netstat -tunlp|grep
tcp 127.0.0.1: 0.0.0.0:* LISTEN /openshift
tcp 10.131.0.1: 0.0.0.0:* LISTEN /dnsmasq
tcp 172.17.0.1: 0.0.0.0:* LISTEN /dnsmasq
tcp 192.168.56.104: 0.0.0.0:* LISTEN /dnsmasq
ps -ef|grep openshift root : ? :: openshift start network --config=/etc/origin/node/node-config.yaml --kubeconfig=/tmp/kubeconfig --loglevel=
2.DNS的配置信息
Pod中的dns配置会指向Pod所在宿主机IP，配置为
$ cat /etc/resolv.conf
nameserver 192.168.56.105
search myproject.svc.cluster.local svc.cluster.local cluster.local redhat.com example.com
options ndots:
192.168.56.105是pod所在的宿主机
宿主机的resolv.conf文件
[root@node1 node]# cat /etc/resolv.conf
# nameserver updated by /etc/NetworkManager/dispatcher.d/99-origin-dns.sh
# Generated by NetworkManager
search cluster.local cluster.local example.com
nameserver 192.168.56.104
在部署环境时，会在每个节点上部署 /etc/NetworkManager/dispatcher.d/99-origin-dns.sh 文件。每当节点上的 NetworkManager 服务启动时，该文件会被运行。它的任务包括：
- 创建 dnsmasq 配置文件 :
- node-dnsmasq.conf （没有）
- origin-dns.conf
- origin-upstream-dns.conf(没有)
- 当 NetworkManager 服务启动时启动 dnsmasq 服务
- 设置宿主机的所有默认路由 IP 为 Dnsmasq 的侦听IP
- 修改 /etc/resolv.conf，设置搜索域，以及将宿主机的默认 IP 作为 nameserver
- 创建 /etc/origin/node/resolv.conf
origin-dns.conf的配置目录在/etc/dnsmasq.d/,内容如下
[root@node1 dnsmasq.d]# cat origin-dns.conf
no-resolv
domain-needed
no-negcache
max-cache-ttl=
enable-dbus
dns-forward-max=
cache-size=
bind-dynamic
min-port=
except-interface=lo
# End of config
如果有文件origin-upstream-dns.conf ，中定义了上游（upstream） DNS 名字服务器，如果没有可以手工创建.
[root@node2 dnsmasq.d]# cat origin-upstream-dns.conf
server=10.72.17.5
server=10.68.5.26
server=202.96.134.33
server=202.96.128.86
如果需要解析外部域名，是需要在pod运行的宿主机节点上进行创建的。
node-dnsmasq.conf的内容是
server=/in-addr.arpa/127.0.0.1
server=/cluster.local/127.0.0.1
根据前辈SammyTalksAboutCloud的研究，这个已经写到程序里面去了。
可以通过journalctl -u dnsmasq去查看日志
[root@node2 dnsmasq.d]# journalctl -u dnsmasq
-- Logs begin at Fri -- :: CST, end at Thu -- :: CST. --
Dec :: node2.example.com systemd[]: Started DNS caching server..
Dec :: node2.example.com dnsmasq[]: started, version 2.76 cachesize
Dec :: node2.example.com dnsmasq[]: compile time options: IPv6 GNU-getopt DBus no-i18n IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth no-D
Dec :: node2.example.com dnsmasq[]: DBus support enabled: connected to system bus
Dec :: node2.example.com dnsmasq[]: warning: no upstream servers configured
Dec :: node2.example.com dnsmasq[]: read /etc/hosts - addresses
Dec :: node2.example.com dnsmasq[]: setting upstream servers from DBus
Dec :: node2.example.com dnsmasq[]: using nameserver 127.0.0.1# for domain in-addr.arpa
Dec :: node2.example.com dnsmasq[]: using nameserver 127.0.0.1# for domain cluster.local
根据日志信息，知道dnsmasq实际把请求转发给了监听在127.0.0.1：53上的skyDNS
skyDNS并不是作为一个单独的进程启动，而是在启动网络
openshift start network --config=/etc/origin/node/node-config.yaml --kubeconfig=/tmp/kubeconfig --loglevel=2
中启动,SkyDNS 调用 OpenShift API 服务来获取主机名、IP地址等信息，然后封装成标准 DNS 记录并返回给查询客户端。
3.DNS在openshift中的配置
- master
cat /etc/origin/master/master-config.yaml
dnsConfig:
bindAddress: 0.0.0.0:
bindNetwork: tcp4
bind在每个ip的8053端口
- node
cat /etc/origin/node/node-config.yaml
dnsBindAddress: 127.0.0.1:
dnsDomain: cluster.local
dnsIP: 0.0.0.0
dnsNameservers: null
dnsRecursiveResolvConf: /etc/origin/node/resolv.conf
根据这种机制，service在宿主机范围内（不仅只是在容器中)能够解析
[root@node2 dnsmasq.d]# dig tomcat.myproject.svc.cluster.local ; <<>> DiG 9.9.-RedHat-9.9.-.el7 <<>> tomcat.myproject.svc.cluster.local
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id:
;; flags: qr aa rd ra; QUERY: , ANSWER: , AUTHORITY: , ADDITIONAL: ;; QUESTION SECTION:
;tomcat.myproject.svc.cluster.local. IN A ;; ANSWER SECTION:
tomcat.myproject.svc.cluster.local. IN A 172.30.16.194 ;; Query time: msec
;; SERVER: 10.0.3.15#(10.0.3.15)
;; WHEN: Thu Jan :: CST
;; MSG SIZE rcvd:
能够访问
[root@node2 dnsmasq.d]# curl tomcat.myproject.svc: <!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<title>Apache Tomcat/8.5.</title>
<link href="favicon.ico" rel="icon" type="image/x-icon" />
<link href="favicon.ico" rel="shortcut icon" type="image/x-icon" />
<link href="tomcat.css" rel="stylesheet" type="text/css" />
</head> <body>
4.查询流程图
查看dnsmasq更详细日志
vi /etc/dnsmasq.conf # For debugging purposes, log each DNS query as it passes through
# dnsmasq.
log-queries
systemctl restart dnsmasq
[root@node2 dnsmasq.d]# journalctl -f -u dnsmasq
-- Logs begin at Fri -- :: CST. --
Jan :: node2.example.com dnsmasq[]: using nameserver 10.72.17.5#
Jan :: node2.example.com dnsmasq[]: using nameserver 127.0.0.1# for domain in-addr.arpa
Jan :: node2.example.com dnsmasq[]: using nameserver 127.0.0.1# for domain cluster.local
Jan :: node2.example.com dnsmasq[]: setting upstream servers from DBus
Jan :: node2.example.com dnsmasq[]: using nameserver 202.96.128.86#
Jan :: node2.example.com dnsmasq[]: using nameserver 202.96.134.33#
Jan :: node2.example.com dnsmasq[]: using nameserver 10.68.5.26#
Jan :: node2.example.com dnsmasq[]: using nameserver 10.72.17.5#
Jan :: node2.example.com dnsmasq[]: using nameserver 127.0.0.1# for domain in-addr.arpa
Jan :: node2.example.com dnsmasq[]: using nameserver 127.0.0.1# for domain cluster.local
Jan :: node2.example.com dnsmasq[]: query[A] www.baidu.com from 10.0.3.15
Jan :: node2.example.com dnsmasq[]: forwarded www.baidu.com to 202.96.128.86
Jan :: node2.example.com dnsmasq[]: forwarded www.baidu.com to 202.96.134.33
Jan :: node2.example.com dnsmasq[]: forwarded www.baidu.com to 10.68.5.26
Jan :: node2.example.com dnsmasq[]: forwarded www.baidu.com to 10.72.17.5
Jan :: node2.example.com dnsmasq[]: query[AAAA] www.baidu.com from 10.0.3.15
Jan :: node2.example.com dnsmasq[]: forwarded www.baidu.com to 202.96.128.86
Jan :: node2.example.com dnsmasq[]: forwarded www.baidu.com to 202.96.134.33
Jan :: node2.example.com dnsmasq[]: forwarded www.baidu.com to 10.68.5.26
Jan :: node2.example.com dnsmasq[]: forwarded www.baidu.com to 10.72.17.5
Jan :: node2.example.com dnsmasq[]: reply www.baidu.com is <CNAME>
Jan :: node2.example.com dnsmasq[]: reply www.a.shifen.com is 14.215.177.38
Jan :: node2.example.com dnsmasq[]: reply www.a.shifen.com is 14.215.177.39
Jan :: node2.example.com dnsmasq[]: reply www.baidu.com is <CNAME>