Flume性能测试报告

时间:2022-09-15 00:23:06

1. 测试环境

1.1 硬件

  • CPU:Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz(8核)
  • 内存:16G

1.2 软件

  • Flume:1.6.0
  • Hadoop:2.6.0-cdh5.5.0
  • Kfaka:2.11-0.9.0.1
  • JDK:1.8.0_91-b14 64位

1.3 测试文件

文件大小:107M ,共490010条记录

1.4 Flume配置

(1)Source配置 
Flume Source采用spooldir的方式,直接读取预先准备好的测试文件。

agent.sources.source1.type = spooldir
agent.sources.source1.spoolDir=/data/flume/dir
  • 1
  • 2
  • 1
  • 2

(2)MemoryChannel配置

agent.channels.memoryChannel.capacity = 1000000
agent.channels.memoryChannel.transactionCapacity = 1000000
agent.channels.memoryChannel.type=memory
  • 1
  • 2
  • 3
  • 1
  • 2
  • 3

(3)FileChannel配置

agent.channels.fileChannel.type = file
agent.channels.fileChannel.checkpointDir = /data/flume/checkpoint
agent.channels.fileChannel.dataDirs = /data/flume/data
  • 1
  • 2
  • 3
  • 1
  • 2
  • 3

(4)JVM配置

JAVA_OPTS="-Xms256m -Xmx256m -Xss256k -Xmn128m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:-UseGCOverheadLimit"
   
   
  • 1
  • 1

2. 写入文件性能

Flume Config Time(s) Throughput(events/s)
MemoryChannel+FileSink 51 9608
FileChannel+FileSink 250 1960

3. 写入Kafka性能

Flume Config Time(s) Throughput(events/s)
KafkaSink+MemoryChannel 57 8597
KafkaChannel 50 9800
KafkaSink+FileChannel 830 590

4. 写入HDFS性能

Flume Config Time(s) Throughput(events/s)
FileChannel+HdfsSink 148 3311