流利的不会将日志写入亚马逊s3

时间:2021-12-22 23:06:05

I am trying to test the fluentd-s3-plugin yet at the moment it is not posting my logs into s3 bucket. I am running everything on ubuntu xenial, having installed fluentd with td-agent. The following is the td-agent log file located at /var/log/td-agent/td-agent.log

我正在尝试测试流利的-s3插件,但目前还没有将我的日志发布到s3存储桶中。我在ubuntu xenial上运行所有东西,已经安装了流畅的td-agent。以下是位于/var/log/td-agent/td-agent.log的td-agent日志文件

 2016-09-23 09:16:18 -0300 [info]: reading config file path="/etc/td-agent/td-agent.conf"
2016-09-23 09:16:18 -0300 [info]: starting fluentd-0.12.26
2016-09-23 09:16:18 -0300 [info]: gem 'fluent-mixin-config-placeholders' version '0.4.0'
2016-09-23 09:16:18 -0300 [info]: gem 'fluent-mixin-plaintextformatter' version '0.2.6'
2016-09-23 09:16:18 -0300 [info]: gem 'fluent-plugin-kinesis' version '1.1.1'
2016-09-23 09:16:18 -0300 [info]: gem 'fluent-plugin-mongo' version '0.7.13'
2016-09-23 09:16:18 -0300 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '1.5.5'
2016-09-23 09:16:18 -0300 [info]: gem 'fluent-plugin-s3' version '0.6.8'
2016-09-23 09:16:18 -0300 [info]: gem 'fluent-plugin-scribe' version '0.10.14'
2016-09-23 09:16:18 -0300 [info]: gem 'fluent-plugin-td' version '0.10.28'
2016-09-23 09:16:18 -0300 [info]: gem 'fluent-plugin-td-monitoring' version '0.2.2'
2016-09-23 09:16:18 -0300 [info]: gem 'fluent-plugin-webhdfs' version '0.4.2'
2016-09-23 09:16:18 -0300 [info]: gem 'fluentd' version '0.12.26'
2016-09-23 09:16:18 -0300 [info]: adding match pattern="nginx.log" type="s3"
2016-09-23 09:16:18 -0300 [info]: adding source type="tail"
2016-09-23 09:16:18 -0300 [info]: using configuration file: <ROOT>
  <source>
    type tail
    format nginx
    path /var/log/nginx/access.log
    pos_file /var/log/td-agent/nginx-access.pos
    tag nginx.log
  </source>
  <match nginx.log>
    @type s3
    s3_bucket kd.creatives
    aws_key_id xxxxxx
    aws_sec_key xxxxxx
    s3_region us-west-2
    path logs/
    buffer_path /var/log/td-agent/s3
    time_slice_format %Y%m%d%H%M
    utc 
    format_json true
    include_time_key true
    buffer_chunk_limit 256m
  </match>
</ROOT>
2016-09-23 09:16:18 -0300 [warn]: parameter 'format_json' in <match nginx.log>
  @type s3
  s3_bucket kd.creatives
  aws_key_id xxxxxx
  aws_sec_key xxxxxx
  s3_region us-west-2
  path logs/
  buffer_path /var/log/td-agent/s3
  time_slice_format %Y%m%d%H%M
  utc 
  format_json true
  include_time_key true
  buffer_chunk_limit 256m
</match> is not used.
2016-09-23 09:16:20 -0300 [info]: following tail of /var/log/nginx/access.log
2016-09-23 09:16:25 -0300 [warn]: pattern not match: "172.17.0.1 - - [23/Sep/2016:12:16:25 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.89 Safari/537.36\" \"-\""
2016-09-23 09:16:25 -0300 [warn]: pattern not match: "172.17.0.1 - - [23/Sep/2016:12:16:25 +0000] \"GET /favicon.ico HTTP/1.1\" 404 571 \"http://localhost:10241/\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.89 Safari/537.36\" \"-\""
2016-09-23 09:16:26 -0300 [warn]: pattern not match: "172.17.0.1 - - [23/Sep/2016:12:16:26 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.89 Safari/537.36\" \"-\""
2016-09-23 09:16:26 -0300 [warn]: pattern not match: "172.17.0.1 - - [23/Sep/2016:12:16:26 +0000] \"GET /favicon.ico HTTP/1.1\" 404 571 \"http://localhost:10241/\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.89 Safari/537.36\" \"-\""
2016-09-23 09:16:26 -0300 [warn]: pattern not match: "172.17.0.1 - - [23/Sep/2016:12:16:26 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.89 Safari/537.36\" \"-\""
2016-09-23 09:16:26 -0300 [warn]: pattern not match: "172.17.0.1 - - [23/Sep/2016:12:16:26 +0000] \"GET /favicon.ico HTTP/1.1\" 404 571 \"http://localhost:10241/\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.89 Safari/537.36\" \"-\""
2016-09-23 09:16:27 -0300 [warn]: pattern not match: "172.17.0.1 - - [23/Sep/2016:12:16:27 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.89 Safari/537.36\" \"-\""
2016-09-23 09:16:27 -0300 [warn]: pattern not match: "172.17.0.1 - - [23/Sep/2016:12:16:27 +0000] \"GET /favicon.ico HTTP/1.1\" 404 571 \"http://localhost:10241/\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.89 Safari/537.36\" \"-\""
2016-09-23 09:16:27 -0300 [warn]: pattern not match: "172.17.0.1 - - [23/Sep/2016:12:16:27 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.89 Safari/537.36\" \"-\""
2016-09-23 09:16:27 -0300 [warn]: pattern not match: "172.17.0.1 - - [23/Sep/2016:12:16:27 +0000] \"GET /favicon.ico HTTP/1.1\" 404 571 \"http://localhost:10241/\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.89 Safari/537.36\" \"-\""
2016-09-23 09:16:28 -0300 [warn]: pattern not match: "172.17.0.1 - - [23/Sep/2016:12:16:28 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.89 Safari/537.36\" \"-\""
2016-09-23 09:16:28 -0300 [warn]: pattern not match: "172.17.0.1 - - [23/Sep/2016:12:16:28 +0000] \"GET /favicon.ico HTTP/1.1\" 404 571 \"http://localhost:10241/\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.89 Safari/537.36\" \"-\""
2016-09-23 09:16:28 -0300 [warn]: pattern not match: "172.17.0.1 - - [23/Sep/2016:12:16:28 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.89 Safari/537.36\" \"-\""
2016-09-23 09:16:28 -0300 [warn]: pattern not match: "172.17.0.1 - - [23/Sep/2016:12:16:28 +0000] \"GET /favicon.ico HTTP/1.1\" 404 571 \"http://localhost:10241/\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.89 Safari/537.36\" \"-\""
2016-09-23 09:16:28 -0300 [warn]: pattern not match: "172.17.0.1 - - [23/Sep/2016:12:16:28 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.89 Safari/537.36\" \"-\""
2016-09-23 09:16:28 -0300 [warn]: pattern not match: "172.17.0.1 - - [23/Sep/2016:12:16:28 +0000] \"GET /favicon.ico HTTP/1.1\" 404 571 \"http://localhost:10241/\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.89 Safari/537.36\" \"-\""
2016-09-23 09:16:28 -0300 [warn]: pattern not match: "172.17.0.1 - - [23/Sep/2016:12:16:28 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.89 Safari/537.36\" \"-\""
2016-09-23 09:16:28 -0300 [warn]: pattern not match: "172.17.0.1 - - [23/Sep/2016:12:16:28 +0000] \"GET /favicon.ico HTTP/1.1\" 404 571 \"http://localhost:10241/\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.89 Safari/537.36\" \"-\""

Also here is my td-agent.conf file

这里还有我的td-agent.conf文件

    # Centralized nginx logs into Amazon s3
<source>
    type tail
    format nginx
    path /var/log/nginx/access.log
    pos_file /var/log/td-agent/nginx-access.pos
    tag nginx.log
</source>

<match nginx.log>
    @type s3
    s3_bucket xxxxxxx
    aws_key_id xxxxxxxxxxxxxxxx
    aws_sec_key xxxxxxxxxxxxxxxxxxxxxxxxxx
    s3_region us-west-2

    path logs/
    buffer_path /var/log/td-agent/s3
    time_slice_format %Y%m%d%H%M

    utc

    format_json true
    include_time_key true
    buffer_chunk_limit 256m
</match>

I would really appreciate any answer on why fluentd is not posting the logs into s3 bucket.

我真的很感激为什么流利的人没有将日志发布到s3桶中的任何答案。

1 个解决方案

#1


1  

It looks in_tail failed to parse your nginx log. So, fluentd didn't send all log to s3. format nginx treats Combined Log Format as default.

看起来in_tail无法解析你的nginx日志。所以,流利的并没有将所有日志发送到s3。 format nginx将组合日志格式视为默认值。

I think you need to use custom format to parse your nginx log.

我认为您需要使用自定义格式来解析您的nginx日志。

<source>
    type tail
    format /^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)" "(?<forwarder>[^\"]*)")?/
    time_format %d/%b/%Y:%H:%M:%S %z
    path /var/log/nginx/access.log
    pos_file /var/log/td-agent/nginx-access.pos
    tag nginx.log
</source>

Ref. http://docs.fluentd.org/articles/in_tail

参考。 http://docs.fluentd.org/articles/in_tail

You can test your own regex by fluentd-ui. http://docs.fluentd.org/articles/fluentd-ui

您可以通过流利的ui测试自己的正则表达式。 http://docs.fluentd.org/articles/fluentd-ui

#1


1  

It looks in_tail failed to parse your nginx log. So, fluentd didn't send all log to s3. format nginx treats Combined Log Format as default.

看起来in_tail无法解析你的nginx日志。所以,流利的并没有将所有日志发送到s3。 format nginx将组合日志格式视为默认值。

I think you need to use custom format to parse your nginx log.

我认为您需要使用自定义格式来解析您的nginx日志。

<source>
    type tail
    format /^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)" "(?<forwarder>[^\"]*)")?/
    time_format %d/%b/%Y:%H:%M:%S %z
    path /var/log/nginx/access.log
    pos_file /var/log/td-agent/nginx-access.pos
    tag nginx.log
</source>

Ref. http://docs.fluentd.org/articles/in_tail

参考。 http://docs.fluentd.org/articles/in_tail

You can test your own regex by fluentd-ui. http://docs.fluentd.org/articles/fluentd-ui

您可以通过流利的ui测试自己的正则表达式。 http://docs.fluentd.org/articles/fluentd-ui