logstash5.5 数据采入elasticsearch5.5(基于x-pack)
以采集redis日志信息为例
redis日志文件、MQ(redis)、logstash主机:10.10.83.162
logstash、elasticsearch、kibana:10.10.83.163
1、redis日志文件主机(10.10.83.162)部署logstash,目录/etc/logstash/conf.d
自定义文件名redis_log.conf
input{
file{
path => ["/home/redis-3.2.1/log/redis_16379.log"]
type => "redis-log"
start_position => "beginning"
}
} filter{
} output{
redis {
db => 1
host => ["10.10.83.162"]
port => 16379
key =>"logstash-list"
timeout => 30
data_type => "list"
}
}
通过MQ(redis)缓存日志数据
2、10.10.83.163主机部署logstash采集redis中数据,放入elasticsearch
目录/etc/logstash/conf.d
自定义文件名redis_log.conf
input{
redis {
db => 1
host => ["10.10.83.162"]
port => 16379
key =>"logstash-list"
timeout => 30
data_type => "list"
}
} output{
elasticsearch {
hosts => ["10.10.83.163:9200"]
index => "redis_log-%{+YYYY.MM.dd}"
user => "elastic"
password => "qwerasdf"
}
}
备注:outpu中elasticsearch的user、password是在开启x-pack使用到的。
重要提示:如果/etc/elasticsearch/elasticsearch.yml配置文件用使用到action.auto_create_index,需要将你自定义的index加入,否则ES无法出现数据。
action.auto_create_index: .security,.monitoring*,.watches,.triggered_watches,.watcher-history*,.ml*,redis_log*
https://www.elastic.co/guide/en/elasticsearch/reference/current/installing-xpack-es.html
我之前根据官方文档在elasticsearch安装x-pack,配置了action.auto_create_index,没有加入自定义的redis_log-%{+YYYY.MM.dd},数据一直没入ES,也没有报错信息。
多主机配置ES集群参考http://www.cnblogs.com/shhnwangjian/p/7156066.html