Beam is using both Google's auto/value and auto/service tools.
Beam正在使用Google的自动/价值和自动/服务工具。
I want to run a pipeline with Dataflow runner and data is stored on Google Cloud Storage.
我想运行带有Dataflow流水线的管道,数据存储在Google云端存储上。
I've added a dependencies:
我添加了一个依赖项:
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-runners-google-cloud-dataflow-java</artifactId>
<version>2.0.0</version>
</dependency>
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-sdks-java-extensions-google-cloud-platform-core</artifactId>
<version>2.0.0</version>
</dependency>
I'm able to start the pipeline from the IntelliJ. But when the jar is compiled through a mvn package
and run with java -jar
it throws an error:
我能够从IntelliJ启动管道。但是当jar通过mvn包编译并使用java -jar运行时会抛出一个错误:
java.lang.IllegalStateException: Unable to find registrar for gs
The fatjar is package with maven-assembly-plugin
. GcsFileSystemRegistrar
class is in the jar.
fatjar是包含maven-assembly-plugin的包。 GcsFileSystemRegistrar类在jar中。
2 个解决方案
#1
7
The issue is in the way that you are building your fatjar. The maven-assembly-plugin
is not handling files associated with ServiceLoader
correctly. ServiceLoader
relies on entries being listed within META-INF/services/org.apache.beam.sdk.io.FileSystemRegistrar
for each implementation so that Java knows how to find them.
问题在于你正在构建你的fatjar。 maven-assembly-plugin没有正确处理与ServiceLoader关联的文件。 ServiceLoader依赖于META-INF / services / org.apache.beam.sdk.io.FileSystemRegistrar中列出的每个实现的条目,以便Java知道如何找到它们。
The contents of the META-INF/services/org.apache.beam.sdk.io.FileSystemRegistrar
in your fatjar is likely only:
你的fatjar中的META-INF / services / org.apache.beam.sdk.io.FileSystemRegistrar的内容可能只是:
org.apache.beam.sdk.io.LocalFileSystemRegistrar
You need to have it list (and any other implementations that you want):
您需要拥有它(以及您想要的任何其他实现):
org.apache.beam.sdk.io.LocalFileSystemRegistrar
org.apache.beam.sdk.extensions.gcp.storage.GcsFileSystemRegistrar
Your best bet is to use a tool which understands these ServiceLoader
requirements like the maven-shade-plugin
when configured to use the ServicesResourceTransformer to build your fatjar.
最好的办法是在配置为使用ServicesResourceTransformer来构建你的fatjar时,使用一个了解这些ServiceLoader要求的工具,如maven-shade-plugin。
#2
2
This looks like a problem with assembly strategy, you should accumulate/merge the services for org.apache.beam.sdk.io.FileSystemRegistrar
. More on similar problem here.
这看起来像汇编策略的问题,您应该为org.apache.beam.sdk.io.FileSystemRegistrar积累/合并服务。更多类似的问题在这里。
#1
7
The issue is in the way that you are building your fatjar. The maven-assembly-plugin
is not handling files associated with ServiceLoader
correctly. ServiceLoader
relies on entries being listed within META-INF/services/org.apache.beam.sdk.io.FileSystemRegistrar
for each implementation so that Java knows how to find them.
问题在于你正在构建你的fatjar。 maven-assembly-plugin没有正确处理与ServiceLoader关联的文件。 ServiceLoader依赖于META-INF / services / org.apache.beam.sdk.io.FileSystemRegistrar中列出的每个实现的条目,以便Java知道如何找到它们。
The contents of the META-INF/services/org.apache.beam.sdk.io.FileSystemRegistrar
in your fatjar is likely only:
你的fatjar中的META-INF / services / org.apache.beam.sdk.io.FileSystemRegistrar的内容可能只是:
org.apache.beam.sdk.io.LocalFileSystemRegistrar
You need to have it list (and any other implementations that you want):
您需要拥有它(以及您想要的任何其他实现):
org.apache.beam.sdk.io.LocalFileSystemRegistrar
org.apache.beam.sdk.extensions.gcp.storage.GcsFileSystemRegistrar
Your best bet is to use a tool which understands these ServiceLoader
requirements like the maven-shade-plugin
when configured to use the ServicesResourceTransformer to build your fatjar.
最好的办法是在配置为使用ServicesResourceTransformer来构建你的fatjar时,使用一个了解这些ServiceLoader要求的工具,如maven-shade-plugin。
#2
2
This looks like a problem with assembly strategy, you should accumulate/merge the services for org.apache.beam.sdk.io.FileSystemRegistrar
. More on similar problem here.
这看起来像汇编策略的问题,您应该为org.apache.beam.sdk.io.FileSystemRegistrar积累/合并服务。更多类似的问题在这里。