UnsupportedFileSystemSchemeException: Hadoop is not in the classpath/dependencies.

时间:2024-03-13 19:11:37

flink程序在idea上运行正常,打成jar包上传服务器运行,各种报错。下面这个错误印象最为深刻

Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not find a file system implementation for scheme ‘hdfs’. The scheme is not directly supported by Flink and no Hadoop file system to support this scheme could be loaded.
at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:405)
at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:320)
at org.apache.flink.core.fs.Path.getFileSystem(Path.java:293)
at org.apache.flink.api.common.io.FileInputFormat.createInputSplits(FileInputFormat.java:472)
at org.apache.flink.api.common.io.FileInputFormat.createInputSplits(FileInputFormat.java:62)
at org.apache.flink.runtime.executiongraph.ExecutionJobVertex.(ExecutionJobVertex.java:248)
… 22 more
Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Cannot support file system for ‘hdfs’ via Hadoop, because Hadoop is not in the classpath, or some classes are missing from the classpath.
at org.apache.flink.runtime.fs.hdfs.HadoopFsFactory.create(HadoopFsFactory.java:179)
at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:401)
… 27 more
Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.hdfs.HdfsConfiguration
at org.apache.flink.runtime.util.HadoopUtils.getHadoopConfiguration(HadoopUtils.java:49)
at org.apache.flink.runtime.fs.hdfs.HadoopFsFactory.create(HadoopFsFactory.java:85)
… 28 more

上网查了各种资料,比如说下面的两步很多教程都说了。
解决方法:
1.环境变量加入以下配置(别忘了source /etc/profile 后重启Flink)
HADOOP_HOME=xxx
export HADOOP_HOME
export HADOOP_CLASSPATH=hadoop classpath
2.下载一个flink集成hadoop的jar包放在Flink的lib目录下
下载地址:https://repo.maven.apache.org/maven2/org/apache/flink/flink-shaded-hadoop-2-uber/

hadoop的版本是2.6.0,flink的版本是1.10.0
UnsupportedFileSystemSchemeException: Hadoop is not in the classpath/dependencies.

UnsupportedFileSystemSchemeException: Hadoop is not in the classpath/dependencies.
但是上面两步操作后,重启flink后还是报相同的错误。

最后发现重启linux系统就好了。。。。。。