阿里云k8s应用最新日志采集不到的问题

时间:2022-08-30 15:58:08

问题描述: 阿里云k8s应用日志之前一直都是可以正常的采集, 先出现一问题, 通过kibana 和阿里云的日志服务都没法展示最新的k8s应用的日志, 部分应用的最新日志有被采集到,但大部分应用日志没有做采集到。

通过 命令 curl '17*****0:9200/_cat/indices?v'   查看 

部分应用没有建立今天的日志索引

  

删除了elasticsearch组件, 再重新安装以后, 问题依然存在

  

通过
kubectl delete DaemonSet log-pilot -n kube-system
kubectl delete DaemonSet logtail-ds -n kube-system 删除了日志采集工具 log-pilot, logtail 后再尝试重新安装

  

重新安装 log-pilot组件 

kubectl apply -f https://acs-logging.oss-cn-hangzhou.aliyuncs.com/log-pilot.yml
daemonset.extensions/log-pilot created

  

kubectl get DaemonSet -n kube-system
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
cloud-controller-manager 3 3 3 3 3 node-role.kubernetes.io/master= 76d
flexvolume 6 6 6 6 6 <none> 76d
kube-flannel-ds 6 6 6 6 6 beta.kubernetes.io/arch=amd64 76d
kube-proxy-master 3 3 3 3 3 node-role.kubernetes.io/master= 76d
kube-proxy-worker 3 3 3 3 3 <none> 76d
log-pilot 6 6 6 6 6 <none> 3s

  

再次通过命令查看日志索引 , 已经可以看到最新的应用日志索引

 curl '172.21.7.210:9200/_cat/indices?v'
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open audit-c48ec13c7dbc94c6fba92748baaf296fc-2019.03.13 G7yQFpNVQfa-xh18CmVCvQ 5 1 0 0 1.1kb 486b
yellow open d******-ui-access-2019.03.13 oZ9qhpYzQQeRGVtAFWHEaQ 5 1 0 0 486b 486b
green open im*****ess-2019.03.13 Lq6meYJcQZGK0MdXgnN7bw 5 1 6 0 167kb 83.5kb
green open of******ess-2019.03.13 PaeAsTGMTEKflSMwaq-V0A 5 1 4 0 83.2kb 41.6kb
yellow open da*****rror-2019.03.13 hoO90XVTTiOr8OExHZOoKQ 5 1 0 0 1.1kb 648b
yellow open d******o-2019.03.13 eWFqQEBFT92x1sQy4Vp2rg 5 1 0 0 486b 486b

  

重新安装 logtail  组件 

helm del --purge alibaba-log-controller
release "alibaba-log-controller" deleted [root@k8****03 ~]# wget http://logtail-release-cn-hangzhou.oss-cn-hangzhou.aliyuncs.com/kubernetes/alicloud-log-k8s-install.sh -O alicloud-log-k8s-install.sh; chmod 744 ./alicloud-log-k8s-install.sh; sh ./alicloud-log-k8s-install.sh c48ec13c7d****296fc
--2019-03-13 14:30:38-- http://logtail-release-cn-hangzhou.oss-cn-hangzhou.aliyuncs.com/kubernetes/alicloud-log-k8s-install.sh
Resolving logtail-release-cn-hangzhou.oss-cn-hangzhou.aliyuncs.com (logtail-release-cn-hangzhou.oss-cn-hangzhou.aliyuncs.com)... 47.110.177.92
Connecting to logtail-release-cn-hangzhou.oss-cn-hangzhou.aliyuncs.com (logtail-release-cn-hangzhou.oss-cn-hangzhou.aliyuncs.com)|47.110.177.92|:80... connected.
HTTP request sent, awaiting response... 200 OK kubectl get DaemonSet -n kube-system
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE log-pilot 6 6 6 6 6 <none> 6m
logtail-ds 6 6 6 6 6 <none> 10s