切换到Firebase Firestore作为管道源后,“数据流似乎停留在使用Apache Beam 2.1.1的Cloud Dataflow”

时间:2021-08-22 15:29:47

I am struggling with this, and initially thought it could be the result of switching the pipeline data source from Cloud Datastore to Firebase Firestore, which required a new project. But I've since found the same error in separate pipelines. All pipelines run successfully on the local DirectRunner and the permissions appear to be the same as the old project.

我正在努力解决这个问题,最初认为这可能是将管道数据源从Cloud Datastore切换到Firebase Firestore的结果,Firebase Firestore需要一个新项目。但我已经在单独的管道中发现了同样的错误。所有管道都在本地DirectRunner上成功运行,并且权限看起来与旧项目相同。

It looks like none of the VMs are booting and the pipeline never scales above 0 workers. "The Dataflow appears to be stuck" is the only error message I could find and there is nothing in StackDriver. Tried every dependency management variation I could find in the docs but it doesn't seem to be the problem.

看起来没有任何VM正在启动,并且管道从不扩展到超过0个工作者。 “数据流似乎被卡住”是我能找到的唯一错误消息,StackDriver中没有任何内容。尝试了我可以在文档中找到的每个依赖管理变体,但它似乎不是问题。

My last Dataflow job-id is 2017-10-11_11_12_01-15165703816317931044.

我的上一个Dataflow工作ID是2017-10-11_11_12_01-15165703816317931044。

Tried elevating the access roles of all services accounts and still no luck.

尝试提升所有服务帐户的访问角色,但仍然没有运气。

1 个解决方案

#1


1  

Without any logging information, it's hard to pinpoint. But this can happen if you have changed the permissions or roles of the Dataflow service account or the Compute Engine service account so that the service account does not have enough permissions to get the images for the Dataflow workers.

没有任何记录信息,很难确定。但是,如果您更改了Dataflow服务帐户或Compute Engine服务帐户的权限或角色,以致服务帐户没有足够的权限来获取Dataflow工作者的映像,则会发生这种情况。

#1


1  

Without any logging information, it's hard to pinpoint. But this can happen if you have changed the permissions or roles of the Dataflow service account or the Compute Engine service account so that the service account does not have enough permissions to get the images for the Dataflow workers.

没有任何记录信息,很难确定。但是,如果您更改了Dataflow服务帐户或Compute Engine服务帐户的权限或角色,以致服务帐户没有足够的权限来获取Dataflow工作者的映像,则会发生这种情况。