在Cloud Dataflow中使用Beam SDK

时间:2023-01-11 23:13:40

We are currently using Google's Cloud Dataflow SDK (1.6.0) to run dataflow jobs in GCP, however, we are considering moving to the Apache Beam SDK (0.1.0). We will still be running our jobs in GCP using the dataflow service. Has anyone gone through this transition and have advice? Are there any compatibility issues here and is this move encouraged by GCP?

我们目前正在使用Google的Cloud Dataflow SDK(1.6.0)在GCP中运行数据流作业,但是,我们正在考虑转向Apache Beam SDK(0.1.0)。我们仍将使用数据流服务在GCP中运行我们的工作。有没有人经历过这种转变并有建议?这里是否存在任何兼容性问题,GCP是否鼓励此举?

1 个解决方案

#1


2  

Formally Beam is not yet supported on Dataflow (although that is certainly what we are working towards). We recommend staying with the Dataflow SDK, especially if SLA or support are important to you. that said, our tests show that Beam runs on Dataflow, and although that may break at any time, you are certainly welcome to attempt at your own risk.

Dataflow尚不支持Formally(虽然这肯定是我们正在努力的方向)。我们建议您继续使用Dataflow SDK,尤其是在SLA或支持对您很重要的情况下。也就是说,我们的测试显示Beam在Dataflow上运行,虽然可能随时都会中断,但您肯定会自担风险。

Update: The Dataflow SDKs are now based on Beam as of the release of Dataflow SDK 2.0 (https://cloud.google.com/dataflow/release-notes/release-notes-java-2). Both Beam and the Dataflow SDKs are currently supported on Cloud Dataflow.

更新:自Dataflow SDK 2.0发布以来,Dataflow SDK现在基于Beam(https://cloud.google.com/dataflow/release-notes/release-notes-java-2)。 Cloud Dataflow目前支持Beam和Dataflow SDK。

#1


2  

Formally Beam is not yet supported on Dataflow (although that is certainly what we are working towards). We recommend staying with the Dataflow SDK, especially if SLA or support are important to you. that said, our tests show that Beam runs on Dataflow, and although that may break at any time, you are certainly welcome to attempt at your own risk.

Dataflow尚不支持Formally(虽然这肯定是我们正在努力的方向)。我们建议您继续使用Dataflow SDK,尤其是在SLA或支持对您很重要的情况下。也就是说,我们的测试显示Beam在Dataflow上运行,虽然可能随时都会中断,但您肯定会自担风险。

Update: The Dataflow SDKs are now based on Beam as of the release of Dataflow SDK 2.0 (https://cloud.google.com/dataflow/release-notes/release-notes-java-2). Both Beam and the Dataflow SDKs are currently supported on Cloud Dataflow.

更新:自Dataflow SDK 2.0发布以来,Dataflow SDK现在基于Beam(https://cloud.google.com/dataflow/release-notes/release-notes-java-2)。 Cloud Dataflow目前支持Beam和Dataflow SDK。