一、heapster简介
什么是Heapster？
Heapster是容器集群监控和性能分析工具，天然的支持Kubernetes和CoreOS。
Kubernetes有个出名的监控agent---cAdvisor。在每个kubernetes Node上都会运行cAdvisor，它会收集本机以及容器的监控数据(cpu,memory,filesystem,network,uptime)。
在较新的版本中，K8S已经将cAdvisor功能集成到kubelet组件中。每个Node节点可以直接进行web访问。
cAdvisor web界面访问： http://< Node-IP >:4194
cAdvisor也提供Restful API: https://github.com/google/cadv ... pi.md
Heapster是一个收集者，将每个Node上的cAdvisor的数据进行汇总，然后导到第三方工具(如InfluxDB)。
二、heapster调用kubelet源码分析
1、整体源码
heapster整体的源码分析可以参考文章：https://segmentfault.com/a/1190000008863353#articleHeader2，
上面链接中的文章中会对heapster从启动开始进行分析，主要讲述了下面内容：
- main（）
- 创建数据源
- 创建后端服务
- 创建数据processors
- 获取源数据并存储
- heapster API创建
给上述文章的作者点个赞，思路真的非常清晰。不过笔者最关注的点是heapster如何进行的kubelet调用。
2、调用kubelet源码分析
从上述文章提到的 NewKubeletProvider 函数中，我们找到了创建kubeClient和kubeletClient的地方，见下面代码的飘黄部分：
func NewKubeletProvider(uri *url.URL) (MetricsSourceProvider, error) {
// 创建kubernetes master及kubelet client相关的配置
kubeConfig, kubeletConfig, err := GetKubeConfigs(uri)
if err != nil {
return nil, err
}
// 创建kubeClient及kubeletClient
kubeClient := kube_client.NewOrDie(kubeConfig)
kubeletClient, err := NewKubeletClient(kubeletConfig)
if err != nil {
return nil, err
}
// 获取下所有的Nodes，测试下创建的client是否能正常通讯
if _, err := kubeClient.Nodes().List(kube_api.ListOptions{
LabelSelector: labels.Everything(),
FieldSelector: fields.Everything()}); err != nil {
glog.Errorf("Failed to load nodes: %v", err)
}
// 监控k8s的nodes变更
// 这里会创建协程进行watch，便于后面调用nodeLister.List()列出所有的nodes。
// 该Watch的实现，需要看下apiServer中的实现，后面会进行讲解
lw := cache.NewListWatchFromClient(kubeClient, "nodes", kube_api.NamespaceAll, fields.Everything())
nodeLister := &cache.StoreToNodeLister{Store: cache.NewStore(cache.MetaNamespaceKeyFunc)}
reflector := cache.NewReflector(lw, &kube_api.Node{}, nodeLister.Store, time.Hour)
reflector.Run()
// 结构在前面介绍过
return &kubeletProvider{
nodeLister: nodeLister,
reflector: reflector,
kubeletClient: kubeletClient,
}, nil
}
1）heapster源码目录
我们发现这个函数所在的文件为kubelet.go，所在的包为metrics/sources/kubelet
我们可以看到，kubelet调用相关的代码都在这个里面：
2）heapster调用kubelet的API
我们进入到NewKubeletClient函数看一下：
// 传入的是KubeletClientConfig，通过读取相关配置，来初始化client
func NewKubeletClient(kubeletConfig *kubelet_client.KubeletClientConfig) (*KubeletClient, error) {
transport, err := kubelet_client.MakeTransport(kubeletConfig)
if err != nil {
return nil, err
}
c := &http.Client{ // 此处可以看到，是http调用
Transport: transport,
Timeout: kubeletConfig.HTTPTimeout,
}
return &KubeletClient{
config: kubeletConfig,
client: c,
}, nil
}
接下来我们看一下kubelet_client.go中KubeletClient有哪些方法：
(i)获取所有的containersInfo
第一个，获取所有的container统计信息，如果对cAdvisor比较了解的话，可以进入到该方法的返回值的ContainerInfo里面看下，是非常全面的容器监控信息（后续会有cAdvisor的源码分析，敬请期待）
func (self *KubeletClient) getAllContainers(url string, start, end time.Time) ([]cadvisor.ContainerInfo, error) {
// Request data from all subcontainers.此处是构造requestBody
request := statsRequest{
ContainerName: "/",
NumStats: 1,
Start: start, // 2017-11-10T06:46:17Z 这种utc格式的时间戳
End: end,
Subcontainers: true,
}
body, err := json.Marshal(request)
if err != nil {
return nil, err
}
req, err := http.NewRequest("POST", url, bytes.NewBuffer(body))
if err != nil {
return nil, err
}
// 设置请求头
req.Header.Set("Content-Type", "application/json")
var containers map[string]cadvisor.ContainerInfo
client := self.client
if client == nil {
client = http.DefaultClient
}
err = self.postRequestAndGetValue(client, req, &containers)
if err != nil {
return nil, fmt.Errorf("failed to get all container stats from Kubelet URL %q: %v", url, err)
}
result := make([]cadvisor.ContainerInfo, 0, len(containers))
for _, containerInfo := range containers {
cont := self.parseStat(&containerInfo)
if cont != nil {
result = append(result, *cont)
}
}
return result, nil
}
我们可以看到上面的源码其实是构造了一个http的post请求。那么我们在本地通过restClient模拟一下：
http://ip:10255/stats/container/（为什么用10255端口呢？在上面的源码中初始化NewKubeletClient的地方，我们进入到GetKubeClientConfig看一下就会发现，用的是kubelet的默认开放端口，在configs.go文件中）
返回值部分摘录如下（内容太多，折叠一下）：
"/": {
"name": "/",
"subcontainers": [
{
"name": "/docker"
},
{
"name": "/init.scope"
},
{
"name": "/kube-proxy"
},
{
"name": "/kubepods"
},
{
"name": "/system.slice"
},
{
"name": "/user.slice"
}
],
"spec": {
"creation_time": "2017-10-14T17:16:59.086488213+08:00",
"has_cpu": true,
"cpu": {
"limit": 1024,
"max_limit": 0,
"mask": "0-3",
"period": 100000
},
"has_memory": true,
"memory": {
"limit": 33604153344,
"reservation": 9223372036854771712
},
"has_network": true,
"has_filesystem": true,
"has_diskio": true,
"has_custom_metrics": false
},
"stats": [
{
"timestamp": "2017-11-10T14:29:25.105117933+08:00",
"cpu": {
"usage": {
"total": 916886132160774,
"per_cpu_usage": [
225713500943530,
230866928282148,
229896967301966,
230408735633130
],
"user": 443962220000000,
"system": 438380440000000
},
"cfs": {
"periods": 0,
"throttled_periods": 0,
"throttled_time": 0
},
"load_average": 0
},
"diskio": {
"io_service_bytes": [
{
"major": 7,
"minor": 4,
"stats": {
"Async": 0,
"Read": 0,
"Sync": 0,
"Total": 0,
"Write": 0
}
},
{
"major": 1,
"minor": 15,
"stats": {
"Async": 12288,
"Read": 12288,
"Sync": 0,
"Total": 12288,
"Write": 0
}
},
{
"major": 1,
"minor": 14,
"stats": {
"Async": 12288,
"Read": 12288,
"Sync": 0,
"Total": 12288,
"Write": 0
}
},
{
"major": 1,
"minor": 12,
"stats": {
"Async": 12288,
"Read": 12288,
"Sync": 0,
"Total": 12288,
"Write": 0
}
},
{
"major": 1,
"minor": 0,
"stats": {
"Async": 12288,
"Read": 12288,
"Sync": 0,
"Total": 12288,
"Write": 0
}
},
{
"major": 7,
"minor": 0,
"stats": {
"Async": 0,
"Read": 0,
"Sync": 0,
"Total": 0,
"Write": 0
}
},
{
"major": 1,
"minor": 11,
"stats": {
"Async": 12288,
"Read": 12288,
"Sync": 0,
"Total": 12288,
"Write": 0
}
},
{
"major": 1,
"minor": 4,
"stats": {
"Async": 12288,
"Read": 12288,
"Sync": 0,
"Total": 12288,
"Write": 0
}
},
{
"major": 253,
"minor": 0,
"stats": {
"Async": 61989669888,
"Read": 23649280,
"Sync": 17028616192,
"Total": 79018286080,
"Write": 78994636800
}
},
{
"major": 7,
"minor": 7,
"stats": {
"Async": 0,
"Read": 0,
"Sync": 0,
"Total": 0,
"Write": 0
}
},
{
"major": 9,
"minor": 0,
"stats": {
"Async": 0,
"Read": 0,
"Sync": 0,
"Total": 0,
"Write": 0
}
},
{
"major": 1,
"minor": 13,
"stats": {
"Async": 12288,
"Read": 12288,
"Sync": 0,
"Total": 12288,
"Write": 0
}
},
{
"major": 1,
"minor": 6,
"stats": {
"Async": 12288,
"Read": 12288,
"Sync": 0,
"Total": 12288,
"Write": 0
}
},
{
"major": 7,
"minor": 6,
"stats": {
"Async": 0,
"Read": 0,
"Sync": 0,
"Total": 0,
"Write": 0
}
},
{
"major": 7,
"minor": 5,
"stats": {
"Async": 0,
"Read": 0,
"Sync": 0,
"Total": 0,
"Write": 0
}
},
{
"major": 1,
"minor": 5,
"stats": {
"Async": 12288,
"Read": 12288,
"Sync": 0,
"Total": 12288,
"Write": 0
}
},
{
"major": 1,
"minor": 3,
"stats": {
"Async": 12288,
"Read": 12288,
"Sync": 0,
"Total": 12288,
"Write": 0
}
},
{
"major": 1,
"minor": 7,
"stats": {
"Async": 12288,
"Read": 12288,
"Sync": 0,
"Total": 12288,
"Write": 0
}
},
{
"major": 253,
"minor": 16,
"stats": {
"Async": 258048,
"Read": 21504,
"Sync": 4773888,
"Total": 5031936,
"Write": 5010432
}
},
{
"major": 7,
"minor": 3,
"stats": {
"Async": 0,
"Read": 0,
"Sync": 0,
"Total": 0,
"Write": 0
}
},
{
"major": 7,
"minor": 2,
"stats": {
"Async": 0,
"Read": 0,
"Sync": 0,
"Total": 0,
"Write": 0
}
},
{
"major": 7,
"minor": 1,
"stats": {
"Async": 0,
"Read": 0,
"Sync": 0,
"Total": 0,
"Write": 0
}
},
{
"major": 1,
"minor": 10,
"stats": {
"Async": 12288,
"Read": 12288,
"Sync": 0,
"Total": 12288,
"Write": 0
}
},
{
"major": 1,
"minor": 9,
"stats": {
"Async": 12288,
"Read": 12288,
"Sync": 0,
"Total": 12288,
"Write": 0
}
},
{
"major": 1,
"minor": 8,
"stats": {
"Async": 12288,
"Read": 12288,
"Sync": 0,
"Total": 12288,
"Write": 0
}
},
{
"major": 1,
"minor": 2,
"stats": {
"Async": 12288,
"Read": 12288,
"Sync": 0,
"Total": 12288,
"Write": 0
}
},
{
"major": 1,
"minor": 1,
"stats": {
"Async": 12288,
"Read": 12288,
"Sync": 0,
"Total": 12288,
"Write": 0
}
}
],
"io_serviced": [
{
"major": 7,
"minor": 4,
"stats": {
"Async": 0,
"Read": 0,
"Sync": 0,
"Total": 0,
"Write": 0
}
},
{
"major": 1,
"minor": 9,
"stats": {
"Async": 3,
"Read": 3,
"Sync": 0,
"Total": 3,
"Write": 0
}
},
{
"major": 1,
"minor": 8,
"stats": {
"Async": 3,
"Read": 3,
"Sync": 0,
"Total": 3,
"Write": 0
}
},
{
"major": 9,
"minor": 0,
"stats": {
"Async": 0,
"Read": 0,
"Sync": 0,
"Total": 0,
"Write": 0
}
},
{
"major": 1,
"minor": 1,
"stats": {
"Async": 3,
"Read": 3,
"Sync": 0,
"Total": 3,
"Write": 0
}
},
{
"major": 1,
"minor": 15,
"stats": {
"Async": 3,
"Read": 3,
"Sync": 0,
"Total": 3,
"Write": 0
}
},
{
"major": 7,
"minor": 7,
"stats": {
"Async": 0,
"Read": 0,
"Sync": 0,
"Total": 0,
"Write": 0
}
},
{
"major": 7,
"minor": 1,
"stats": {
"Async": 0,
"Read": 0,
"Sync": 0,
"Total": 0,
"Write": 0
}
},
{
"major": 1,
"minor": 4,
"stats": {
"Async": 3,
"Read": 3,
"Sync": 0,
"Total": 3,
"Write": 0
}
},
{
"major": 1,
"minor": 0,
"stats": {
"Async": 3,
"Read": 3,
"Sync": 0,
"Total": 3,
"Write": 0
}
},
{
"major": 253,
"minor": 0,
"stats": {
"Async": 2738092,
"Read": 2965,
"Sync": 4157380,
"Total": 6895472,
"Write": 6892507
}
},
{
"major": 1,
"minor": 13,
"stats": {
"Async": 3,
"Read": 3,
"Sync": 0,
"Total": 3,
"Write": 0
}
},
{
"major": 7,
"minor": 5,
"stats": {
"Async": 0,
"Read": 0,
"Sync": 0,
"Total": 0,
"Write": 0
}
},
{
"major": 1,
"minor": 14,
"stats": {
"Async": 3,
"Read": 3,
"Sync": 0,
"Total": 3,
"Write": 0
}
},
{
"major": 1,
"minor": 5,
"stats": {
"Async": 3,
"Read": 3,
"Sync": 0,
"Total": 3,
"Write": 0
}
},
{
"major": 253,
"minor": 16,
"stats": {
"Async": 57,
"Read": 7,
"Sync": 32,
"Total": 89,
"Write": 82
}
},
{
"major": 1,
"minor": 3,
"stats": {
"Async": 3,
"Read": 3,
"Sync": 0,
"Total": 3,
"Write": 0
}
},
{
"major": 1,
"minor": 2,
"stats": {
"Async": 3,
"Read": 3,
"Sync": 0,
"Total": 3,
"Write": 0
}
},
{
"major": 7,
"minor": 6,
"stats": {
"Async": 0,
"Read": 0,
"Sync": 0,
"Total": 0,
"Write": 0
}
},
{
"major": 7,
"minor": 2,
"stats": {
"Async": 0,
"Read": 0,
"Sync": 0,
"Total": 0,
"Write": 0
}
},
{
"major": 1,
"minor": 11,
"stats": {
"Async": 3,
"Read": 3,
"Sync": 0,
"Total": 3,
"Write": 0
}
},
{
"major": 1,
"minor": 10,
"stats": {
"Async": 3,
"Read": 3,
"Sync": 0,
"Total": 3,
"Write": 0
}
},
{
"major": 1,
"minor": 6,
"stats": {
"Async": 3,
"Read": 3,
"Sync": 0,
"Total": 3,
"Write": 0
}
},
{
"major": 1,
"minor": 12,
"stats": {
"Async": 3,
"Read": 3,
"Sync": 0,
"Total": 3,
"Write": 0
}
},
{
"major": 1,
"minor": 7,
"stats": {
"Async": 3,
"Read": 3,
"Sync": 0,
"Total": 3,
"Write": 0
}
},
{
"major": 7,
"minor": 3,
"stats": {
"Async": 0,
"Read": 0,
"Sync": 0,
"Total": 0,
"Write": 0
}
},
{
"major": 7,
"minor": 0,
"stats": {
"Async": 0,
"Read": 0,
"Sync": 0,
"Total": 0,
"Write": 0
}
}
]
},
"memory": {
"usage": 12189630464,
"cache": 288108544,
"rss": 3631652864,
"swap": 0,
"working_set": 7756312576,
"failcnt": 0,
"container_data": {
"pgfault": 8947531,
"pgmajfault": 87
},
"hierarchical_data": {
"pgfault": 8947531,
"pgmajfault": 87
}
},
"network": {
"name": "cbr0",
"rx_bytes": 1885776219,
"rx_packets": 8199780,
"rx_errors": 0,
"rx_dropped": 0,
"tx_bytes": 1854690043,
"tx_packets": 8765796,
"tx_errors": 0,
"tx_dropped": 0,
"interfaces": [
{
"name": "cbr0",
"rx_bytes": 1885776219,
"rx_packets": 8199780,
"rx_errors": 0,
"rx_dropped": 0,
"tx_bytes": 1854690043,
"tx_packets": 8765796,
"tx_errors": 0,
"tx_dropped": 0
},
{
"name": "ens3",
"rx_bytes": 25108832715,
"rx_packets": 74352449,
"rx_errors": 0,
"rx_dropped": 0,
"tx_bytes": 93157559917,
"tx_packets": 75983196,
"tx_errors": 0,
"tx_dropped": 0
}
],
"tcp": {
"Established": 0,
"SynSent": 0,
"SynRecv": 0,
"FinWait1": 0,
"FinWait2": 0,
"TimeWait": 0,
"Close": 0,
"CloseWait": 0,
"LastAck": 0,
"Listen": 0,
"Closing": 0
},
"tcp6": {
"Established": 0,
"SynSent": 0,
"SynRecv": 0,
"FinWait1": 0,
"FinWait2": 0,
"TimeWait": 0,
"Close": 0,
"CloseWait": 0,
"LastAck": 0,
"Listen": 0,
"Closing": 0
},
"udp": {
"Listen": 0,
"Dropped": 0,
"RxQueued": 0,
"TxQueued": 0
},
"udp6": {
"Listen": 0,
"Dropped": 0,
"RxQueued": 0,
"TxQueued": 0
}
},
"filesystem": [
{
"device": "/dev/root",
"type": "vfs",
"capacity": 20749852672,
"usage": 9800732672,
"base_usage": 0,
"available": 10932342784,
"has_inodes": true,
"inodes": 2560000,
"inodes_free": 2340243,
"reads_completed": 0,
"reads_merged": 0,
"sectors_read": 0,
"read_time": 0,
"writes_completed": 0,
"writes_merged": 0,
"sectors_written": 0,
"write_time": 0,
"io_in_progress": 0,
"io_time": 0,
"weighted_io_time": 0
},
{
"device": "tmpfs",
"type": "vfs",
"capacity": 16802074624,
"usage": 2224128,
"base_usage": 0,
"available": 16799850496,
"has_inodes": true,
"inodes": 4102069,
"inodes_free": 4101999,
"reads_completed": 0,
"reads_merged": 0,
"sectors_read": 0,
"read_time": 0,
"writes_completed": 0,
"writes_merged": 0,
"sectors_written": 0,
"write_time": 0,
"io_in_progress": 0,
"io_time": 0,
"weighted_io_time": 0
},
{
"device": "shm",
"type": "vfs",
"capacity": 67108864,
"usage": 0,
"base_usage": 0,
"available": 67108864,
"has_inodes": true,
"inodes": 4102069,
"inodes_free": 4102068,
"reads_completed": 0,
"reads_merged": 0,
"sectors_read": 0,
"read_time": 0,
"writes_completed": 0,
"writes_merged": 0,
"sectors_written": 0,
"write_time": 0,
"io_in_progress": 0,
"io_time": 0,
"weighted_io_time": 0
},
{
"device": "/dev/vdb",
"type": "vfs",
"capacity": 53660876800,
"usage": 35852288,
"base_usage": 0,
"available": 53625024512,
"has_inodes": true,
"inodes": 26214400,
"inodes_free": 26214393,
"reads_completed": 457,
"reads_merged": 0,
"sectors_read": 13162,
"read_time": 3428,
"writes_completed": 231,
"writes_merged": 20,
"sectors_written": 72963,
"write_time": 1756,
"io_in_progress": 0,
"io_time": 3584,
"weighted_io_time": 5184
}
],
"task_stats": {
"nr_sleeping": 0,
"nr_running": 0,
"nr_stopped": 0,
"nr_uninterruptible": 0,
"nr_io_wait": 0
}
},
{
"timestamp": "2017-11-10T14:29:36.487311961+08:00",
"cpu": {
"usage": {
"total": 916888682985895,
"per_cpu_usage": [
225714036142385,
230867670611465,
229897633996290,
230409342235755
],
"user": 443963990000000,
"system": 438381050000000
},
"cfs": {
"periods": 0,
"throttled_periods": 0,
"throttled_time": 0
},
"load_average": 0
},
"diskio": {
"io_service_bytes": [
{
"major": 9,
"minor": 0,
"stats": {
"Async": 0,
"Read": 0,
"Sync": 0,
"Total": 0,
"Write": 0
}
},
{
"major": 253,
"minor": 0,
"stats": {
"Async": 61989710848,
"Read": 23649280,
"Sync": 17028677632,
"Total": 79018388480,
"Write": 78994739200
}
},
{
"major": 7,
"minor": 6,
"stats": {
"Async": 0,
"Read": 0,
"Sync": 0,
"Total": 0,
"Write": 0
}
},
{
"major": 7,
"minor": 1,
"stats": {
"Async": 0,
"Read": 0,
"Sync": 0,
"Total": 0,
"Write": 0
}
},
{
"major": 1,
"minor": 10,
"stats": {
"Async": 12288,
"Read": 12288,
"Sync": 0,
"Total": 12288,
"Write": 0
}
},
{
"major": 1,
"minor": 8,
"stats": {
"Async": 12288,
"Read": 12288,
"Sync": 0,
"Total": 12288,
"Write": 0
}
},
{
"major": 1,
"minor": 4,
"stats": {
"Async": 12288,
"Read": 12288,
"Sync": 0,
"Total": 12288,
"Write": 0
}
},
{
"major": 1,
"minor": 2,
"stats": {
"Async": 12288,
"Read": 12288,
"Sync": 0,
"Total": 12288,
"Write": 0
}
},
{
"major": 1,
"minor": 13,
"stats": {
"Async": 12288,
"Read": 12288,
"Sync": 0,
"Total": 12288,
"Write": 0
}
},
{
"major": 1,
"minor": 1,
"stats": {
"Async": 12288,
"Read": 12288,
"Sync": 0,
"Total": 12288,
"Write": 0
}
},
{
"major": 1,
"minor": 7,
"stats": {
"Async": 12288,
"Read": 12288,
"Sync": 0,
"Total": 12288,
"Write": 0
}
},
{
"major": 1,
"minor": 0,
"stats": {
"Async": 12288,
"Read": 12288,
"Sync": 0,
"Total": 12288,
"Write": 0
}
},
{
"major": 253,
"minor": 16,
"stats": {
"Async": 258048,
"Read": 21504,
"Sync": 4773888,
"Total": 5031936,
"Write": 5010432
}
},
{
"major": 1,
"minor": 14,
"stats": {
"Async": 12288,
"Read": 12288,
"Sync": 0,
"Total": 12288,
"Write": 0
}
},
{
"major": 1,
"minor": 12,
"stats": {
"Async": 12288,
"Read": 12288,
"Sync": 0,
"Total": 12288,
"Write": 0
}
},
{
"major": 1,
"minor": 9,
"stats": {
"Async": 12288,
"Read": 12288,
"Sync": 0,
"Total": 12288,
"Write": 0
}
},
{
"major": 1,
"minor": 6,
"stats": {
"Async": 12288,
"Read": 12288,
"Sync": 0,
"Total": 12288,
"Write": 0
}
},
{
"major": 7,
"minor": 7,
"stats": {
"Async": 0,
"Read": 0,
"Sync": 0,
"Total": 0,
"Write": 0
}
},
{
"major": 1,
"minor": 15,
"stats": {
"Async": 12288,
"Read": 12288,
"Sync": 0,
"Total": 12288,
"Write": 0
}
},
{
"major": 1,
"minor": 5,
"stats": {
"Async": 12288,
"Read": 12288,
"Sync": 0,
"Total": 12288,
"Write": 0
}
},
{
"major": 7,
"minor": 5,
"stats": {
"Async": 0,
"Read": 0,
"Sync": 0,
"Total": 0,
"Write": 0
}
},
{
"major": 7,
"minor": 4,
"stats": {
"Async": 0,
"Read": 0,
"Sync": 0,
"Total": 0,
"Write": 0
}
},
{
"major": 7,
"minor": 3,
"stats": {
"Async": 0,
"Read": 0,
"Sync": 0,
"Total": 0,
"Write": 0
}
},
{
"major": 7,
"minor": 2,
"stats": {
"Async": 0,
"Read": 0,
"Sync": 0,
"Total": 0,
"Write": 0
}
},
{
"major": 1,
"minor": 3,
"stats": {
"Async": 12288,
"Read": 12288,
"Sync": 0,
"Total": 12288,
"Write": 0
}
},
{
"major": 7,
"minor": 0,
"stats": {
"Async": 0,
"Read": 0,
"Sync": 0,
"Total": 0,
"Write": 0
}
},
{
"major": 1,
"minor": 11,
"stats": {
"Async": 12288,
"Read": 12288,
"Sync": 0,
"Total": 12288,
"Write": 0
}
}
],
"io_serviced": [
{
"major": 1,
"minor": 11,
"stats": {
"Async": 3,
"Read": 3,
"Sync": 0,
"Total": 3,
"Write": 0
}
},
{
"major": 1,
"minor": 9,
"stats": {
"Async": 3,
"Read": 3,
"Sync": 0,
"Total": 3,
"Write": 0
}
},
{
"major": 7,
"minor": 4,
"stats": {
"Async": 0,
"Read": 0,
"Sync": 0,
"Total": 0,
"Write": 0
}
},
{
"major": 7,
"minor": 3,
"stats": {
"Async": 0,
"Read": 0,
"Sync": 0,
"Total": 0,
"Write": 0
}
},
{
"major": 1,
"minor": 14,
"stats": {
"Async": 3,
"Read": 3,
"Sync": 0,
"Total": 3,
"Write": 0
}
},
{
"major": 1,
"minor": 10,
"stats": {
"Async": 3,
"Read": 3,
"Sync": 0,
"Total": 3,
"Write": 0
}
},
{
"major": 1,
"minor": 5,
"stats": {
"Async": 3,
"Read": 3,
"Sync": 0,
"Total": 3,
"Write": 0
}
},
{
"major": 1,
"minor": 2,
"stats": {
"Async": 3,
"Read": 3,
"Sync": 0,
"Total": 3,
"Write": 0
}
},
{
"major": 1,
"minor": 1,
"stats": {
"Async": 3,
"Read": 3,
"Sync": 0,
"Total": 3,
"Write": 0
}
},
{
"major": 1,
"minor": 0,
"stats": {
"Async": 3,
"Read": 3,
"Sync": 0,
"Total": 3,
"Write": 0
}
},
{
"major": 9,
"minor": 0,
"stats": {
"Async": 0,
"Read": 0,
"Sync": 0,
"Total": 0,
"Write": 0
}
},
{
"major": 253,
"minor": 0,
"stats": {
"Async": 2738102,
"Read": 2965,
"Sync": 4157395,
"Total": 6895497,
"Write": 6892532
}
},
{
"major": 7,
"minor": 1,
"stats": {
"Async": 0,
"Read": 0,
"Sync": 0,
"Total": 0,
"Write": 0
}
},
{
"major": 1,
"minor": 15,
"stats": {
"Async": 3,
"Read": 3,
"Sync": 0,
"Total": 3,
"Write": 0
}
},
{
"major": 1,
"minor": 3,
"stats": {
"Async": 3,
"Read": 3,
"Sync": 0,
"Total": 3,
"Write": 0
}
},
{
"major": 7,
"minor": 5,
"stats": {
"Async": 0,
"Read": 0,
"Sync": 0,
"Total": 0,
"Write": 0
}
},
{
"major": 1,
"minor": 13,
"stats": {
"Async": 3,
"Read": 3,
"Sync": 0,
"Total": 3,
"Write": 0
}
},
{
"major": 1,
"minor": 12,
"stats": {
"Async": 3,
"Read": 3,
"Sync": 0,
"Total": 3,
"Write": 0
}
},
{
"major": 1,
"minor": 6,
"stats": {
"Async": 3,
"Read": 3,
"Sync": 0,
"Total": 3,
"Write": 0
}
},
{
"major": 1,
"minor": 4,
"stats": {
"Async": 3,
"Read": 3,
"Sync": 0,
"Total": 3,
"Write": 0
}
},
{
"major": 253,
"minor": 16,
"stats": {
"Async": 57,
"Read": 7,
"Sync": 32,
"Total": 89,
"Write": 82
}
},
{
"major": 7,
"minor": 6,
"stats": {
"Async": 0,
"Read": 0,
"Sync": 0,
"Total": 0,
"Write": 0
}
},
{
"major": 7,
"minor": 2,
"stats": {
"Async": 0,
"Read": 0,
"Sync": 0,
"Total": 0,
"Write": 0
}
},
{
"major": 7,
"minor": 0,
"stats": {
"Async": 0,
"Read": 0,
"Sync": 0,
"Total": 0,
"Write": 0
}
},
{
"major": 1,
"minor": 7,
"stats": {
"Async": 3,
"Read": 3,
"Sync": 0,
"Total": 3,
"Write": 0
}
},
{
"major": 7,
"minor": 7,
"stats": {
"Async": 0,
"Read": 0,
"Sync": 0,
"Total": 0,
"Write": 0
}
},
{
"major": 1,
"minor": 8,
"stats": {
"Async": 3,
"Read": 3,
"Sync": 0,
"Total": 3,
"Write": 0
}
}
]
},
"memory": {
"usage": 12189343744,
"cache": 288108544,
"rss": 3631366144,
"swap": 0,
"working_set": 7756025856,
"failcnt": 0,
"container_data": {
"pgfault": 8947531,
"pgmajfault": 87
},
"hierarchical_data": {
"pgfault": 8947531,
"pgmajfault": 87
}
},
"network": {
"name": "cbr0",
"rx_bytes": 1885783888,
"rx_packets": 8199819,
"rx_errors": 0,
"rx_dropped": 0,
"tx_bytes": 1854698778,
"tx_packets": 8765835,
"tx_errors": 0,
"tx_dropped": 0,
"interfaces": [
{
"name": "cbr0",
"rx_bytes": 1885783888,
"rx_packets": 8199819,
"rx_errors": 0,
"rx_dropped": 0,
"tx_bytes": 1854698778,
"tx_packets": 8765835,
"tx_errors": 0,
"tx_dropped": 0
},
{
"name": "ens3",
"rx_bytes": 25108971486,
"rx_packets": 74353959,
"rx_errors": 0,
"rx_dropped": 0,
"tx_bytes": 93160391886,
"tx_packets": 75985129,
"tx_errors": 0,
"tx_dropped": 0
}
],
"tcp": {
"Established": 0,
"SynSent": 0,
"SynRecv": 0,
"FinWait1": 0,
"FinWait2": 0,
"TimeWait": 0,
"Close": 0,
"CloseWait": 0,
"LastAck": 0,
"Listen": 0,
"Closing": 0
},
"tcp6": {
"Established": 0,
"SynSent": 0,
"SynRecv": 0,
"FinWait1": 0,
"FinWait2": 0,
"TimeWait": 0,
"Close": 0,
"CloseWait": 0,
"LastAck": 0,
"Listen": 0,
"Closing": 0
},
"udp": {
"Listen": 0,
"Dropped": 0,
"RxQueued": 0,
"TxQueued": 0
},
"udp6": {
"Listen": 0,
"Dropped": 0,
"RxQueued": 0,
"TxQueued": 0
}
},
"filesystem": [
{
"device": "/dev/root",
"type": "vfs",
"capacity": 20749852672,
"usage": 9800732672,
"base_usage": 0,
"available": 10932342784,
"has_inodes": true,
"inodes": 2560000,
"inodes_free": 2340243,
"reads_completed": 0,
"reads_merged": 0,
"sectors_read": 0,
"read_time": 0,
"writes_completed": 0,
"writes_merged": 0,
"sectors_written": 0,
"write_time": 0,
"io_in_progress": 0,
"io_time": 0,
"weighted_io_time": 0
},
{
"device": "tmpfs",
"type": "vfs",
"capacity": 16802074624,
"usage": 2224128,
"base_usage": 0,
"available": 16799850496,
"has_inodes": true,
"inodes": 4102069,
"inodes_free": 4101999,
"reads_completed": 0,
"reads_merged": 0,
"sectors_read": 0,
"read_time": 0,
"writes_completed": 0,
"writes_merged": 0,
"sectors_written": 0,
"write_time": 0,
"io_in_progress": 0,
"io_time": 0,
"weighted_io_time": 0
},
{
"device": "shm",
"type": "vfs",
"capacity": 67108864,
"usage": 0,
"base_usage": 0,
"available": 67108864,
"has_inodes": true,
"inodes": 4102069,
"inodes_free": 4102068,
"reads_completed": 0,
"reads_merged": 0,
"sectors_read": 0,
"read_time": 0,
"writes_completed": 0,
"writes_merged": 0,
"sectors_written": 0,
"write_time": 0,
"io_in_progress": 0,
"io_time": 0,
"weighted_io_time": 0
},
{
"device": "/dev/vdb",
"type": "vfs",
"capacity": 53660876800,
"usage": 35852288,
"base_usage": 0,
"available": 53625024512,
"has_inodes": true,
"inodes": 26214400,
"inodes_free": 26214393,
"reads_completed": 457,
"reads_merged": 0,
"sectors_read": 13162,
"read_time": 3428,
"writes_completed": 231,
"writes_merged": 20,
"sectors_written": 72963,
"write_time": 1756,
"io_in_progress": 0,
"io_time": 3584,
"weighted_io_time": 5184
}
],
"task_stats": {
"nr_sleeping": 0,
"nr_running": 0,
"nr_stopped": 0,
"nr_uninterruptible": 0,
"nr_io_wait": 0
}
}
]
}
我们可以看到这一段json数据主要分了4部分：
- reference——container本身的引用
- subcontainers——子容器的信息
- spec——说明信息
- stats——统计信息
与cAdvisor提供的统计数据一致：
type ContainerInfo struct {
ContainerReference
// The direct subcontainers of the current container.
Subcontainers []ContainerReference `json:"subcontainers,omitempty"`
// The isolation used in the container.
Spec ContainerSpec `json:"spec,omitempty"`
// Historical statistics gathered from the container.
Stats []*ContainerStats `json:"stats,omitempty"`
}
太棒了。
那么我就可以知道，获取allcontainers的逻辑，其实就是向kubelet发送了一个http的post请求，然后获取到返回的json数据。
(ii) 获取pod相关的summary
在kubelet_client.go文件中还有一个获取summary的方法：
func (self *KubeletClient) GetSummary(host Host) (*stats.Summary, error) {
url := url.URL{
Scheme: "http",
Host: host.String(),
Path: "/stats/summary/",
}
if self.config != nil && self.config.EnableHttps {
url.Scheme = "https"
}
req, err := http.NewRequest("GET", url.String(), nil)
if err != nil {
return nil, err
}
summary := &stats.Summary{}
client := self.client
if client == nil {
client = http.DefaultClient
}
err = self.postRequestAndGetValue(client, req, summary)
return summary, err
}
恩，分析一下，很简单，就是一个http get方法的调用。那我们再来模拟一下：
直接调用这个url即可，返回值太多，就不一一分析了。返回的数据结构如下：
type Summary struct {
// Overall node stats.
Node NodeStats `json:"node"`
// Per-pod stats.
Pods []PodStats `json:"pods"`
}
主要就是当前node的统计信息，还有pod的聚合信息。可以自己追进NodeStats和PodeStats的源码中看一下结构，在此就不列了。
值得注意的是，summary里面的stats和ContainerInfo是有些不一样的。比如ContainerInfo里面会有DiskInfo相关的信息。
3）小结
heapster调用kubelet，其实就是发送了两个http请求，一个是get方法，获取所有pod的summary信息，另一个是post方法，获取cAdvisor提供的所有ContainerInfo。然后在pod_aggregator.go方法中进行的数据相关的聚合。那我们对kubelet调用的源码分析就到此为止了。接下来是笔者自己做的一个小实验。
三、实验
1、禁掉节点的4194端口
禁掉node的4194端口后，就无法通过ip：4194获取到cAdvisor的stats数据了。那我们调用一下kubelet的api：调用kubelet的api：http://ip:10255/stats/container/
我们会发现，返回的数据项里缺少了stats这一项，而这一项恰恰是cAdvisor提供的最重要的监控统计数据。
2、重新开启节点的4194端口
重新开启node的4194端口，再重新调用一下kubelet的api，这下返回的结果里面又重新出现了stats这一项。
3、总结
结合kubelet的源码可以发现，在kubelet启动的时候会去启动cAdvisor的4194端口，获取基础的统计数据。