elk 平台搭建: ELK平台搭建 系统环境 System: Centos release 6.7 (Final) ElasticSearch: 2.1.0 Logstash: 2.1.1 Kibana: 4.3.0 Java: openjdk version "1.8.0_65" 注：由于Logstash的运行依赖于Java环境， 而Logstash 1.5以上版本不低于java 1.7，因此推荐使用最新版本的Java。因为我们只需要Java的运行环境，所以可以只安装JRE，不过这里我依然使用JDK，请自行搜索安装。 配置ElasticSearch： tar -zxvf elasticsearch-2.1.0.tar.gz
cd elasticsearch-2.1.0 zjtest7-redis:/usr/local/elasticsearch-2.3.4# ./bin/plugin install mobz/elasticsearch-head
-> Installing mobz/elasticsearch-head...
Plugins directory [/usr/local/elasticsearch-2.3.4/plugins] does not exist. Creating...
Trying https://github.com/mobz/elasticsearch-head/archive/master.zip ...
Downloading .....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................DONE
Verifying https://github.com/mobz/elasticsearch-head/archive/master.zip checksums if available ...
NOTE: Unable to verify checksum for downloaded plugin (unable to find .sha1 or .md5 file to verify)
Installed head into /usr/local/elasticsearch-2.3.4/plugins/head 然后编辑ES的配置文件： vi config/elasticsearch.yml 添加配置:
cluster.name=es_cluster
node.name=node0
path.data=/tmp/elasticsearch/data
path.logs=/tmp/elasticsearch/logs
#当前hostname或IP，我这里是centos2
network.host=192.168.32.80
network.port=9200 启动报错:
zjtest7-redis:/usr/local/elasticsearch-2.3.4# ./bin/elasticsearch
Exception in thread "main" SettingsException[Failed to load settings from [elasticsearch.yml]]; nested: ElasticsearchParseException[malformed, expected settings to start with 'object', instead was [VALUE_STRING]];
Likely root cause: ElasticsearchParseException[malformed, expected settings to start with 'object', instead was [VALUE_STRING]]
at org.elasticsearch.common.settings.loader.XContentSettingsLoader.load(XContentSettingsLoader.java:65)
at org.elasticsearch.common.settings.loader.XContentSettingsLoader.load(XContentSettingsLoader.java:45)
at org.elasticsearch.common.settings.loader.YamlSettingsLoader.load(YamlSettingsLoader.java:46)
at org.elasticsearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1080)
at org.elasticsearch.common.settings.Settings$Builder.loadFromPath(Settings.java:1067)
at org.elasticsearch.node.internal.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.java:88)
at org.elasticsearch.common.cli.CliTool.<init>(CliTool.java:107)
at org.elasticsearch.common.cli.CliTool.<init>(CliTool.java:100)
at org.elasticsearch.bootstrap.BootstrapCLIParser.<init>(BootstrapCLIParser.java:48)
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:226)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)
Refer to the log for complete error details. 原因 需要这么写:
cluster.name: es_cluster
node.name: node01
path.data: /tmp/elasticsearch/data
path.logs: /tmp/elasticsearch/logs
network.host: 192.168.32.80
network.port: 9200 zjtest7-redis:/usr/local/elasticsearch-2.3.4# ./bin/elasticsearch
Exception in thread "main" java.lang.RuntimeException: don't run elasticsearch as root.
at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:93)
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:144)
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:270)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)
Refer to the log for complete error details. 创建 elk 用户和group: [elk@zjtest7-redis local]$ cd elasticsearch-2.3.4
[elk@zjtest7-redis elasticsearch-2.3.4]$ ./bin/elasticsearch
[2016-07-22 14:03:22,797][WARN ][bootstrap ] unable to install syscall filter: seccomp unavailable: requires kernel 3.5+ with CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER compiled in
[2016-07-22 14:03:23,728][INFO ][node ] [node01] version[2.3.4], pid[1756], build[e455fd0/2016-06-30T11:24:31Z]
[2016-07-22 14:03:23,728][INFO ][node ] [node01] initializing ...
[2016-07-22 14:03:25,787][INFO ][plugins ] [node01] modules [reindex, lang-expression, lang-groovy], plugins [head], sites [head]
[2016-07-22 14:03:25,878][INFO ][env ] [node01] using [1] data paths, mounts [[/ (/dev/mapper/vg00-lv_root)]], net usable_space [86.7gb], net total_space [96.2gb], spins? [possibly], types [ext4]
[2016-07-22 14:03:25,882][INFO ][env ] [node01] heap size [1015.6mb], compressed ordinary object pointers [true]
[2016-07-22 14:03:25,882][WARN ][env ] [node01] max file descriptors [4096] for elasticsearch process likely too low, consider increasing to at least [65536]
[2016-07-22 14:03:33,299][INFO ][node ] [node01] initialized
[2016-07-22 14:03:33,302][INFO ][node ] [node01] starting ...
[2016-07-22 14:03:33,561][INFO ][transport ] [node01] publish_address {192.168.32.80:9300}, bound_addresses {192.168.32.80:9300}
[2016-07-22 14:03:33,576][INFO ][discovery ] [node01] es_cluster/gYNVInstR16CujdeJ1T6YQ
[2016-07-22 14:03:36,717][INFO ][cluster.service ] [node01] new_master {node01}{gYNVInstR16CujdeJ1T6YQ}{192.168.32.80}{192.168.32.80:9300}, reason: zen-disco-join(elected_as_master, [0] joins received)
[2016-07-22 14:03:36,759][INFO ][http ] [node01] publish_address {192.168.32.80:9200}, bound_addresses {192.168.32.80:9200}
[2016-07-22 14:03:36,759][INFO ][node ] [node01] started
[2016-07-22 14:03:36,867][INFO ][gateway ] [node01] recovered [0] indices into cluster_state 可以看到，它跟其他的节点的传输端口为9300，接受HTTP请求的端口为9200。 使用ctrl+C停止。当然，也可以使用后台进程的方式启动ES： ./bin/elasticsearch &
然后可以打开页面localhost:9200，将会看到以下内容： http://192.168.32.80:9200/ 返回； {
"name" : "node01",
"cluster_name" : "es_cluster",
"version" : {
"number" : "2.3.4",
"build_hash" : "e455fd0c13dceca8dbbdbb1665d068ae55dabe3f",
"build_timestamp" : "2016-06-30T11:24:31Z",
"build_snapshot" : false,
"lucene_version" : "5.5.0"
},
"tagline" : "You Know, for Search"
} 返回展示了配置的cluster_name和name，以及安装的ES的版本等信息。 刚刚安装的head插件，它是一个用浏览器跟ES集群交互的插件，可以查看集群状态、集群的doc内容、执行搜索和普通的Rest请求等。 现在也可以使用它打开localhost:9200/_plugin/head页面来查看ES集群状态： Logstash Logstash的功能如下： [elk@zjtest7-redis logstash-2.3.4]$ ./bin/logstash
No command given Usage: logstash <command> [command args]
Run a command with the --help flag to see the arguments.
For example: logstash agent --help Available commands:
agent - runs the logstash agent
version - emits version info about this logstash [elk@zjtest7-redis logstash-2.3.4]$ ./bin/logstash agent -f config/log4j_to_es.conf
Settings: Default pipeline workers: 1
log4j:WARN No appenders could be found for logger (org.apache.http.client.protocol.RequestAuthCache).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Pipeline main started [elk@zjtest7-redis logstash-2.3.4]$ ./bin/logstash agent -f config/log4j_to_es.conf & 输入和输出日志: 代码如下： input {
file {
type => "nginx_access"
path => ["/usr/share/nginx/logs/test.access.log"]
}
}
output {
redis {
host => "localhost"
data_type => "list"
key => "logstash:redis"
}
} output {
# For detail config for elasticsearch as output,
# See: https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html
elasticsearch {
action => "index" #The operation on ES
hosts => "192.168.32.80:9200" #ElasticSearch host, can be array.
index => "applog" #The index to write data to.
}
} logstash,elasticsearch,kibana 怎么进行nginx的日志分析呢？首先，架构方面，nginx是有日志文件的，它的每个请求的状态等都有日志文件进行记录。其次，需要有个队列，redis的list结构正好可以作为队列使用。然后分析使用elasticsearch就可以进行分析和查询了。 我们需要的是一个分布式的，日志收集和分析系统。logstash有agent和indexer两个角色。对于agent角色，放在单独的web机器上面，然后这个agent不断地读取nginx的日志文件，每当它读到新的日志信息以后，就将日志传送到网络上的一台redis队列上。对于队列上的这些未处理的日志，有不同的几台logstash indexer进行接收和分析。分析之后存储到elasticsearch进行搜索分析。再由统一的kibana进行日志web界面的展示。 启动/usr/local/elasticsearch-2.3.4# ./bin/elasticsearch [elk@zjtest7-redis logstash-2.3.4]$ ./bin/logstash agent -f config/log4j_to_es.conf logstash日志收集分发到elastic集群， elasticsearch进行数据索引， kibana进行结构化查询展示，redis做缓存队列。 装： logstash，新版下载即可用，需加配置文件agent.conf和index.conf，当然最好途径是官网sample即得即用。 /usr/local/redis/bin/redis-server *:6379 input {
file {
type => "nginx_access"
path => ["/usr/local/nginx/logs/test.access.log"]
}
}
output {
redis {
host => "localhost"
data_type => "list"
key => "logstash:redis"
port=>6379
password => "1234567"
}
}
~
~ zjtest7-redis:/usr/local/logstash-2.3.4# ./bin/logstash -f /usr/local/logstash-2.3.4/config/logstash_agent.conf
Settings: Default pipeline workers: 1
Pipeline main started 日志格式: log_format logstash '$http_host $server_addr $remote_addr [$time_local] "$request" '
'$request_body $status $body_bytes_sent "$http_referer" "$http_user_agent" '
'$request_time $upstream_response_time'; redis : 127.0.0.1:6379> llen "logstash:redis"
(integer) 328 到这里数据已经存放到redis队列: logstash 数据写入到redis 队列: zjtest7-redis:/usr/local/logstash-2.3.4# ./bin/logstash -f /usr/local/logstash-2.3.4/config/logstash_agent.conf ### 存数据 zjtest7-redis:/usr/local/logstash-2.3.4# ./bin/logstash -f /usr/local/logstash-2.3.4/config/logstash_indexer.conf logstash分为 index和aget ，agent负责监控、过滤日志， index负责收集日志并将日志交给ElasticSearch 做搜索 Shipper：发送事件（events）至LogStash； 通常，远程代理端（agent）只需要运行这个组件即可；
Broker and Indexer：接收并索引化事件； Search and Storage：允许对事件进行搜索和存储；Web Interface： 基于Web的展示界面
192.168.32.162为日志查看服务器，该机器需要安装redis, elasticserch, logstatsh, kibana这四个应用程序。 192.168.32.161?为应用程序nginx应用程序，我们这次只收集他的日志进行分析。 这个配置文件,是读取nginx日志写入到redis zjtest7-redis:/usr/local/logstash-2.3.4/config# cat logstash_agent.conf
input {
file {
type => "nginx_access"
path => ["/usr/local/nginx/logs/test.access.log"]
}
}
output {
redis {
host => "localhost"
data_type => "list"
key => "logstash:redis"
port=>"6379"
password => "1234567"
}
} 这个配置文件是读取本地的redis数据,交给elasticsearch
zjtest7-redis:/usr/local/logstash-2.3.4/config# cat logstash_indexer.conf
input {
redis {
host => "localhost"
data_type => "list"
key => "logstash:redis"
type => "redis-input"
password => "1234567"
port =>"6379"
}
}
output {
elasticsearch {
embedded => false
protocol => "http"
host => "localhost"
port => "9200"
index => "access-%{+YYYY.MM.dd}"
document_type="access"
}
stdout {
codec => rubydebug
}
} Kibana 配置Kibana: tar -zxvf kibana-4.3.0-linux-x86.tar.gz
cd kibana-4.3.0-linux-x86
vi config/kibana.yml server.port: 5601
server.host: "192.168.32.80"
elasticsearch.url: http://192.168.32.80:9200
kibana.index:".kibana'