依赖冲突的表现
ClassNotFound
// class 名字会随着冲突的不同而不同
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.tracing.SpanReceiverHost
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 21 more
NoSuchMethod
java.lang.NoSuchMethodError: org.elasticsearch.client.RestHighLevelClient.bulkAsync(Lorg/elasticsearch/action/bulk/BulkRequest;Lorg/elasticsearch/client/RequestOptions;Lorg/elasticsearch/action/ActionListener;)Lorg/elasticsearch/client/Cancellable;
at org.apache.flink.streaming.connectors.elasticsearch7.Elasticsearch7ApiCallBridge.lambda$createBulkProcessorBuilder$0(Elasticsearch7ApiCallBridge.java:82)
at org.elasticsearch.action.bulk.Retry$RetryHandler.execute(Retry.java:202)
at org.elasticsearch.action.bulk.Retry.withBackoff(Retry.java:62)
如何查找冲突
查找依赖冲突的来源
在解决冲突之前，我们需要知道冲突的 Jar 是从哪里引入的
如果在提交作业时，提示「可能有冲突」的 jar，则需要排查项目中哪里依赖了对应的 jar 包
如果是 NoSuchMethod 或者 ClassNotFound 等异常，则根据报错的 class 来查找对应的 jar 包（比如上文中 ClassNotFound 示例中需要查找 SpanReceiverHost 从哪个 jar 中引入，NoSuchMethod 异常示例中，则需要查找 从哪里引入-- 从依赖栈我们知道是 elasticserach7 引入而来）
首先到项目的目录执行如下命令
mvn dependency:tree -Dverbose
会得到类似如下的结果
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building Flink-Demo 1.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @ Flink-Demo ---
[INFO] org.example:Flink-Demo:jar:1.0-SNAPSHOT
[INFO] +- org.apache.flink:flink-core:jar:1.7-tdflink-0.2.40:compile
[INFO] | +- org.apache.flink:flink-annotations:jar:1.7-tdflink-0.2.40:compile
[INFO] | | +- (org.slf4j:slf4j-api:jar:1.7.15:compile - omitted for duplicate)
[INFO] | | +- (com.google.code.findbugs:jsr305:jar:1.3.9:compile - omitted for duplicate)
[INFO] | | \- (org.apache.flink:force-shading:jar:1.7-tdflink-0.2.40:compile - omitted for duplicate)
[INFO] | +- org.apache.flink:flink-metrics-core:jar:1.7-tdflink-0.2.40:compile
[INFO] | | \- (org.apache.flink:force-shading:jar:1.7-tdflink-0.2.40:compile - omitted for duplicate)
[INFO] | +- org.apache.flink:flink-shaded-asm:jar:5.0.4-5.0:compile
[INFO] | +- org.apache.commons:commons-lang3:jar:3.3.2:compile
[INFO] | +- com.esotericsoftware.kryo:kryo:jar:2.24.0:compile
[INFO] | | +- com.esotericsoftware.minlog:minlog:jar:1.2:compile
[INFO] | | \- org.objenesis:objenesis:jar:2.1:compile
[INFO] | +- commons-collections:commons-collections:jar:3.2.2:compile
[INFO] | +- org.apache.commons:commons-compress:jar:1.21:compile
[INFO] | +- org.slf4j:slf4j-api:jar:1.7.15:compile
[INFO] | +- com.google.code.findbugs:jsr305:jar:1.3.9:compile
[INFO] | \- org.apache.flink:force-shading:jar:1.7-tdflink-0.2.40:compile
[INFO] +- org.apache.flink:flink-runtime_2.11:jar:1.7-tdflink-0.2.40:compile
[INFO] | +- (org.apache.flink:flink-core:jar:1.7-tdflink-0.2.40:compile - omitted for duplicate)
[INFO] | +- org.apache.flink:flink-java:jar:1.7-tdflink-0.2.40:compile
[INFO] | | +- (org.apache.flink:flink-core:jar:1.7-tdflink-0.2.40:compile - omitted for duplicate)
[INFO] | | +- (org.apache.flink:flink-shaded-asm:jar:5.0.4-5.0:compile - omitted for duplicate)
[INFO] | | +- (org.apache.commons:commons-lang3:jar:3.3.2:compile - omitted for duplicate)
[INFO] | | +- (org.apache.commons:commons-math3:jar:3.5:compile - omitted for duplicate)
[INFO] | | +- (org.slf4j:slf4j-api:jar:1.7.15:compile - omitted for duplicate)
[INFO] | | +- (com.google.code.findbugs:jsr305:jar:1.3.9:compile - omitted for duplicate)
[INFO] | | \- (org.apache.flink:force-shading:jar:1.7-tdflink-0.2.40:compile - omitted for duplicate)
[INFO] | +- org.apache.flink:flink-queryable-state-client-java_2.11:jar:1.7-tdflink-0.2.40:compile
[INFO] | | +- (org.apache.flink:flink-shaded-netty:jar:4.1.24.Final-5.0:compile - omitted for duplicate)
[INFO] | | +- (org.apache.flink:flink-shaded-guava:jar:18.0-5.0:compile - omitted for duplicate)
[INFO] | | \- (org.apache.flink:force-shading:jar:1.7-tdflink-0.2.40:compile - omitted for duplicate)
[INFO] | +- org.apache.flink:flink-hadoop-fs:jar:1.7-tdflink-0.2.40:compile
[INFO] | | +- (org.apache.flink:flink-core:jar:1.7-tdflink-0.2.40:compile - omitted for duplicate)
[INFO] | | \- (org.apache.flink:force-shading:jar:1.7-tdflink-0.2.40:compile - omitted for duplicate)
[INFO] | +- commons-io:commons-io:jar:2.4:compile
[INFO] | +- org.apache.flink:flink-shaded-netty:jar:4.1.24.Final-5.0:compile
[INFO] | +- org.apache.flink:flink-shaded-guava:jar:18.0-5.0:compile
[INFO] | +- (org.apache.flink:flink-shaded-asm:jar:5.0.4-5.0:compile - omitted for duplicate)
[INFO] | +- org.apache.flink:flink-shaded-jackson:jar:2.7.9-5.0:compile
[INFO] | +- (org.apache.commons:commons-lang3:jar:3.3.2:compile - omitted for duplicate)
[INFO] | +- commons-cli:commons-cli:jar:1.3.1:compile
[INFO] | +- org.javassist:javassist:jar:3.19.0-GA:compile
[INFO] | +- org.scala-lang:scala-library:jar:2.11.12:compile
[INFO] | +- com.typesafe.akka:akka-actor_2.11:jar:2.4.20:compile
[INFO] | | +- com.typesafe:config:jar:1.3.0:compile
[INFO] | | \- org.scala-lang.modules:scala-java8-compat_2.11:jar:0.7.0:compile
[INFO] | +- com.typesafe.akka:akka-stream_2.11:jar:2.4.20:compile
[INFO] | | +- (com.typesafe.akka:akka-actor_2.11:jar:2.4.20:compile - omitted for duplicate)
[INFO] | | +- org.reactivestreams:reactive-streams:jar:1.0.0:compile
[INFO] | | \- com.typesafe:ssl-config-core_2.11:jar:0.2.1:compile
[INFO] | | \- org.scala-lang.modules:scala-parser-combinators_2.11:jar:1.0.4:compile
[INFO] | +- com.typesafe.akka:akka-protobuf_2.11:jar:2.4.20:compile
[INFO] | +- com.typesafe.akka:akka-slf4j_2.11:jar:2.4.20:compile
[INFO] | | +- (com.typesafe.akka:akka-actor_2.11:jar:2.4.20:compile - omitted for duplicate)
[INFO] | | \- (org.slf4j:slf4j-api:jar:1.7.16:compile - omitted for conflict with 1.7.15)
[INFO] | +- org.clapper:grizzled-slf4j_2.11:jar:1.3.2:compile
[INFO] | | \- (org.slf4j:slf4j-api:jar:1.7.9:compile - omitted for conflict with 1.7.15)
[INFO] | +- com.github.scopt:scopt_2.11:jar:3.5.0:compile
[INFO] | +- org.xerial.snappy:snappy-java:jar:1.1.4:compile
[INFO] | +- com.twitter:chill_2.11:jar:0.7.6:compile
[INFO] | | \- com.twitter:chill-java:jar:0.7.6:compile
[INFO] | +- com.github.oshi:oshi-core:jar:3.4.0:compile
[INFO] | | +- net.java.dev.jna:jna-platform:jar:4.2.2:compile
[INFO] | | | \- net.java.dev.jna:jna:jar:4.2.2:compile
[INFO] | | \- (org.slf4j:slf4j-api:jar:1.7.22:compile - omitted for conflict with 1.7.15)
[INFO] | +- (org.slf4j:slf4j-api:jar:1.7.15:compile - omitted for duplicate)
[INFO] | +- (com.google.code.findbugs:jsr305:jar:1.3.9:compile - omitted for duplicate)
[INFO] | \- (org.apache.flink:force-shading:jar:1.7-tdflink-0.2.40:compile - omitted for duplicate)
[INFO] +- org.apache.flink:flink-streaming-java_2.11:jar:1.7-tdflink-0.2.40:compile
[INFO] | +- (org.apache.flink:flink-runtime_2.11:jar:1.7-tdflink-0.2.40:compile - omitted for duplicate)
[INFO] | +- org.apache.flink:flink-clients_2.11:jar:1.7-tdflink-0.2.40:compile
[INFO] | | +- (org.apache.flink:flink-core:jar:1.7-tdflink-0.2.40:compile - omitted for duplicate)
[INFO] | | +- (org.apache.flink:flink-runtime_2.11:jar:1.7-tdflink-0.2.40:compile - omitted for duplicate)
[INFO] | | +- org.apache.flink:flink-optimizer_2.11:jar:1.7-tdflink-0.2.40:compile
[INFO] | | | +- (org.apache.flink:flink-core:jar:1.7-tdflink-0.2.40:compile - omitted for duplicate)
[INFO] | | | +- (org.apache.flink:flink-runtime_2.11:jar:1.7-tdflink-0.2.40:compile - omitted for duplicate)
[INFO] | | | +- (org.apache.flink:flink-java:jar:1.7-tdflink-0.2.40:compile - omitted for duplicate)
[INFO] | | | +- (org.apache.flink:flink-shaded-guava:jar:18.0-5.0:compile - omitted for duplicate)
[INFO] | | | +- (org.apache.flink:flink-shaded-jackson:jar:2.7.9-5.0:compile - omitted for duplicate)
[INFO] | | | +- (org.slf4j:slf4j-api:jar:1.7.15:compile - omitted for duplicate)
[INFO] | | | +- (com.google.code.findbugs:jsr305:jar:1.3.9:compile - omitted for duplicate)
[INFO] | | | \- (org.apache.flink:force-shading:jar:1.7-tdflink-0.2.40:compile - omitted for duplicate)
[INFO] | | +- (org.apache.flink:flink-java:jar:1.7-tdflink-0.2.40:compile - omitted for duplicate)
[INFO] | | +- (commons-cli:commons-cli:jar:1.3.1:compile - omitted for duplicate)
[INFO] | | +- (org.slf4j:slf4j-api:jar:1.7.15:compile - omitted for duplicate)
[INFO] | | +- (com.google.code.findbugs:jsr305:jar:1.3.9:compile - omitted for duplicate)
[INFO] | | \- (org.apache.flink:force-shading:jar:1.7-tdflink-0.2.40:compile - omitted for duplicate)
[INFO] | +- (org.apache.flink:flink-shaded-guava:jar:18.0-5.0:compile - omitted for duplicate)
[INFO] | +- org.apache.commons:commons-math3:jar:3.5:compile
[INFO] | +- (org.slf4j:slf4j-api:jar:1.7.15:compile - omitted for duplicate)
[INFO] | +- (com.google.code.findbugs:jsr305:jar:1.3.9:compile - omitted for duplicate)
[INFO] | \- (org.apache.flink:force-shading:jar:1.7-tdflink-0.2.40:compile - omitted for duplicate)
[INFO] +- org.slf4j:slf4j-log4j12:jar:1.7.14:compile
[INFO] | +- (org.slf4j:slf4j-api:jar:1.7.14:compile - omitted for conflict with 1.7.15)
[INFO] | \- (log4j:log4j:jar:1.2.17:compile - omitted for duplicate)
[INFO] \- log4j:log4j:jar:1.2.17:compile
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1.437 s
[INFO] Finished at: 2021-11-25T11:22:25+08:00
[INFO] Final Memory: 13M/309M
[INFO] ------------------------------------------------------------------------
在上述输出的日志中，打印出项目的依赖树以及版本冲突情况，其中
version managed from xxx, omitted for duplicated：这个依赖和另一个依赖版本冲突了，使用xxx为最终的版本，并忽略这个依赖
omitted for duplicated：这个依赖已经存在了，忽略这个依赖
omitted for conflict with xxx：这个依赖和另一个依赖版本冲突了，使用xxx为最终的版本，并忽略这个依赖
如何解决依赖冲突
将依赖的 scope 改成 provided
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-core</artifactId>
<version>${flink.version}</version>
<scope>provided</scope> # 将 :flink-core 对应的 jar 改成 provided 的
</dependency>
显示的 exclusive 掉相关依赖
<dependency>
<groupId>com.alibaba.occea</groupId>
<artifactId>TDB</artifactId>
<version>1.2.14</version>
<exclusions>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
</exclusion>
</exclusions>
</dependency>
对 class 进行 shade 操作
<build>
<plugins>
...
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.1.1</version>
<executions>
<execution>
<id>shade-plugin</id>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<shadedArtifactAttached>false</shadedArtifactAttached>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>*.jar</exclude>
<exclude>META-INF/maven/**</exclude>
</excludes>
</filter>
</filters>
<artifactSet>
<includes>
<include>org.json4s:json4s-ast_${scala.binary.version}</include>
<include>org.json4s:json4s-core_${scala.binary.version}</include>
<include>org.json4s:json4s-jackson_${scala.binary.version}</include>
<include>org.json4s:json4s-scalap_${scala.binary.version}</include>
</includes>
</artifactSet>
<relocations>
<relocation>
<pattern>org.json4s</pattern>
<shadedPattern>org.apache.livy.shaded.json4s</shadedPattern>
</relocation>
</relocations>
</configuration>
</execution>
</executions>
<configuration>
<relocations>
<relocation>
<pattern>org.yaml</pattern>
<shadedPattern>test</shadedPattern>
</relocation>
</relocations>
</configuration>
</plugin>
</plugins>
</build>