报错背景：
CDH集群中，将kafka和Flume整合，将kafka的数据发送给Flume消费。
启动kafka的时候正常，但是启动Flume的时候出现了报错现象。
报错现象：
DH-5.15.-.cdh5.15.1.p0./lib/hadoop/lib/native:/opt/cloudera/parcels/CDH-5.15.-.cdh5.15.1.p0./lib/hbase/bin/../lib/native/Linux-amd64-
// :: INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
// :: INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
// :: INFO zookeeper.ZooKeeper: Client environment:os.name=Linux
// :: INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64
// :: INFO zookeeper.ZooKeeper: Client environment:os.version=3.10.-.el7.x86_64
// :: INFO zookeeper.ZooKeeper: Client environment:user.name=root
// :: INFO zookeeper.ZooKeeper: Client environment:user.home=/root
// :: INFO zookeeper.ZooKeeper: Client environment:user.dir=/opt/cloudera/parcels/CDH-5.15.-.cdh5.15.1.p0./etc/flume-ng/conf.empty
// :: INFO zookeeper.ZooKeeper: Initiating client connection, connectString=192.168.52.26: sessionTimeout= watcher=org.I0Itec.zkclient.ZkClient@2ad96f71
// :: INFO zkclient.ZkClient: Waiting for keeper state SyncConnected
// :: INFO zookeeper.ClientCnxn: Opening socket connection to server master/192.168.52.26:2181. Will not attempt to authenticate using SASL (unknown error)
// :: INFO zookeeper.ClientCnxn: Socket connection established, initiating session, client: /192.168.52.26:, server: master/192.168.52.26:
// :: INFO zookeeper.ClientCnxn: Session establishment complete on server master/192.168.52.26:, sessionid = 0x36ab52d38c20b13, negotiated timeout =
// :: INFO zkclient.ZkClient: zookeeper state changed (SyncConnected)
// :: INFO zkclient.ZkEventThread: Terminate ZkClient event thread.
// :: INFO zookeeper.ZooKeeper: Session: 0x36ab52d38c20b13 closed
// :: INFO kafka.KafkaSource: Group ID was not specified. Using flume as the group id.
// :: INFO zookeeper.ClientCnxn: EventThread shut down
// :: INFO sink.DefaultSinkFactory: Creating instance of sink: hdfsSink, type: hdfs
// :: INFO node.AbstractConfigurationProvider: Channel memoryChannel connected to [kafkaSource, hdfsSink]
// :: INFO node.Application: Starting new configuration:{ sourceRunners:{kafkaSource=PollableSourceRunner: { source:org.apache.flume.source.kafka.KafkaSource{name:kafkaSource,state:IDLE} counterGroup:{ name:null counters:{} } }} sinkRunners:{hdfsSink=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@414aa552 counterGroup:{ name:null counters:{} } }} channels:{memoryChannel=org.apache.flume.channel.MemoryChannel{name: memoryChannel}} }
// :: INFO node.Application: Starting Channel memoryChannel
// :: INFO node.Application: Waiting for channel: memoryChannel to start. Sleeping for ms
// :: INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: CHANNEL, name: memoryChannel: Successfully registered new MBean.
// :: INFO instrumentation.MonitoredCounterGroup: Component type: CHANNEL, name: memoryChannel started
// :: INFO node.Application: Starting Sink hdfsSink
// :: INFO node.Application: Starting Source kafkaSource
// :: INFO kafka.KafkaSource: Starting org.apache.flume.source.kafka.KafkaSource{name:kafkaSource,state:IDLE}...
// :: INFO zookeeper.ZooKeeper: Initiating client connection, connectString=192.168.52.26: sessionTimeout= watcher=org.I0Itec.zkclient.ZkClient@6bbdc6fd
// :: INFO zkclient.ZkEventThread: Starting ZkClient event thread.
// :: INFO zkclient.ZkClient: Waiting for keeper state SyncConnected
// :: INFO zookeeper.ClientCnxn: Opening socket connection to server master/192.168.52.26:. Will not attempt to authenticate using SASL (unknown error)
// :: INFO zookeeper.ClientCnxn: Socket connection established, initiating session, client: /192.168.52.26:, server: master/192.168.52.26:
// :: INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: SINK, name: hdfsSink: Successfully registered new MBean.
// :: INFO instrumentation.MonitoredCounterGroup: Component type: SINK, name: hdfsSink started
// :: INFO zookeeper.ClientCnxn: Session establishment complete on server master/192.168.52.26:, sessionid = 0x36ab52d38c20b14, negotiated timeout =
// :: INFO zkclient.ZkClient: zookeeper state changed (SyncConnected)
// :: INFO consumer.ConsumerConfig: ConsumerConfig values:
auto.commit.interval.ms =
auto.offset.reset = latest
bootstrap.servers = [master:, worker1:, worker2:]
check.crcs = true
client.id =
connections.max.idle.ms =
enable.auto.commit = false
exclude.internal.topics = true
fetch.max.bytes =
fetch.max.wait.ms =
fetch.min.bytes =
group.id = flume
heartbeat.interval.ms =
interceptor.classes = null
internal.leave.group.on.close = true
key.deserializer = class org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes =
max.poll.interval.ms =
max.poll.records =
metadata.max.age.ms =
metric.reporters = []
metrics.num.samples =
metrics.recording.level = INFO
metrics.sample.window.ms =
partition.assignment.strategy = [class org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes =
reconnect.backoff.ms =
request.timeout.ms =
retry.backoff.ms =
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin =
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
send.buffer.bytes =
session.timeout.ms =
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1., TLSv1., TLSv1]
ssl.endpoint.identification.algorithm = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLS
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer // :: WARN consumer.ConsumerConfig: The configuration 'timeout.ms' was supplied but isn't a known config.
// :: INFO utils.AppInfoParser: Kafka version : 0.10.-kafka-2.2.
// :: INFO utils.AppInfoParser: Kafka commitId : unknown
报错原因：
这个报错并不是Flume的原因，而是kafka的锅。kafka由于某些原因报错，导致Flume连接kafka的时候报错。解决报错的时候需要去定位并解决kafka的报错。
报错解决：
未解决。。。