从BigQuery到Cloud Bigtable的Google Cloud Dataflow管道中的例外情况

Executing DataFlow pipelines, every once in a while we see those Exceptions. Is there anything we can do about them? We have a quite simple flow that reads data from a BigQuery query and populate data in BigTable.

执行DataFlow管道，我们偶尔会看到这些异常。我们能对他们做些什么吗？我们有一个非常简单的流程，它从BigQuery查询中读取数据并在BigTable中填充数据。

Also what happens to data inside the pipeline? Is it reprocessed? Or is it lost in transit to BigTable?

管道内的数据会发生什么？它被重新加工了吗？或者它在转移到BigTable时丢失了？

 CloudBigtableIO.initializeForWrite(p);
     p.apply(BigQueryIO.Read.fromQuery(getQuery()))
     .apply(ParDo.of(new DoFn<TableRow, Mutation>() {
           public void processElement(ProcessContext c) {
             Mutation output = convertDataToRow(c.element());
             c.output(output);
           }

           }))
         .apply(CloudBigtableIO.writeToTable(config));


private static Mutation convertDataToRow(TableRow element) {
     LOG.info("element: "+ element);
     LOG.info("BASM_segment_id: "+ element.get("BASM_segment_id"));
     if(element.get("BASM_AID") != null){
         Put obj = new Put(getRowKey(element).getBytes()).addColumn(SEGMENT_FAMILY, SEGMENT_COLUMN_NAME, ((String)element.get("BAS_category")).getBytes() );
                obj.addColumn(USER_FAMILY, "AID".getBytes(), ((String)element.get("BASM_AID")).getBytes());
         if(element.get("BASM_segment_id") != null){
                obj.addColumn(SEGMENT_FAMILY, "segment_id".getBytes(), ((String)element.get("BASM_segment_id")).getBytes());
         }
         if(element.get("BAS_sub_category") != null){
                obj.addColumn(SEGMENT_FAMILY, "sub_category".getBytes(), ((String)element.get("BAS_sub_category")).getBytes());
         }
         if(element.get("BAS_name") != null){
                obj.addColumn(SEGMENT_FAMILY, "name".getBytes(), ((String)element.get("BAS_name")).getBytes());
         }
         if(element.get("BAS_description") != null){
                obj.addColumn(SEGMENT_FAMILY, "description".getBytes(), ((String)element.get("BAS_description")).getBytes());
         }
         if(element.get("BAS_last_compute_day") != null){obj.addColumn(USER_FAMILY, "Krux_User_id".getBytes(), ((String)element.get("BASM_krux_user_id")).getBytes());
                obj.addColumn(SEGMENT_FAMILY, "last_compute_day".getBytes(), ((String)element.get("BAS_last_compute_day")).getBytes());
         }
         if(element.get("BAS_type") != null){
                obj.addColumn(SEGMENT_FAMILY, "type".getBytes(), ((String)element.get("BAS_type")).getBytes());
         }      
         if(element.get("BASM_REGID") != null){
                obj.addColumn(USER_FAMILY, "REGID".getBytes(), ((String)element.get("BASM_REGID")).getBytes() );
         }
        return obj;
     }else{
         return null;
     }
    }

Following is the exception which we are getting:

以下是我们获得的例外情况：

2016-08-22T21:47:33.469Z: Error: (84707221e08b977b): java.lang.RuntimeException: com.google.cloud.dataflow.sdk.util.UserCodeExc ption: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1 action: StatusRuntimeException: 1 time, at com.google.cloud.dataflow.sdk.runners.worker.SimpleParDoFn$1.output(SimpleParDoFn.java:162) at com.google.cloud.dataflow.sdk.util.DoFnRunnerBase$DoFnContext.outputWindowedValue(DoFnRunnerBase.java:287) at com.google.cloud.dataflow.sdk.util.DoFnRunnerBase$DoFnProcessContext.output(DoFnRunnerBase.java:449) at com.nytimes.adtech.dataflow.pipelines.BigTableSegmentData$2.processElement(BigTableSegmentData.java:70) Caused by: com.google.cloud.dataflow.sdk.util.UserCodeException: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsExcept on: Failed 1 action: StatusRuntimeException: 1 time, at com.google.cloud.dataflow.sdk.util.UserCodeException.wrap(UserCodeException.java:35) at com.google.cloud.dataflow.sdk.util.UserCodeException.wrapIf(UserCodeException.java:40) at com.google.cloud.dataflow.sdk.util.DoFnRunnerBase.wrapUserCodeException(DoFnRunnerBase.java:368) at com.google.cloud.dataflow.sdk.util.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:51) at com.google.cloud.dataflow.sdk.util.DoFnRunnerBase.processElement(DoFnRunnerBase.java:138) at com.google.cloud.dataflow.sdk.runners.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:190) at com.google.cloud.dataflow.sdk.runners.worker.ForwardingParDoFn.processElement(ForwardingParDoFn.java:42) at com.google.cloud.dataflow.sdk.runners.worker.DataflowWorkerLoggingParDoFn.processElement(DataflowWorkerLoggingParDoFn. ava:47) at com.google.cloud.dataflow.sdk.util.common.worker.ParDoOperation.process(ParDoOperation.java:53) at com.google.cloud.dataflow.sdk.util.common.worker.OutputReceiver.process(OutputReceiver.java:52) at com.google.cloud.dataflow.sdk.runners.worker.SimpleParDoFn$1.output(SimpleParDoFn.java:160) at com.google.cloud.dataflow.sdk.util.DoFnRunnerBase$DoFnContext.outputWindowedValue(DoFnRunnerBase.java:287) at com.google.cloud.dataflow.sdk.util.DoFnRunnerBase$DoFnProcessContext.output(DoFnRunnerBase.java:449) at com.nytimes.adtech.dataflow.pipelines.BigTableSegmentData$2.processElement(BigTableSegmentData.java:70) at com.google.cloud.dataflow.sdk.util.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:49) at com.google.cloud.dataflow.sdk.util.DoFnRunnerBase.processElement(DoFnRunnerBase.java:138) at com.google.cloud.dataflow.sdk.runners.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:190) at com.google.cloud.dataflow.sdk.runners.worker.ForwardingParDoFn.processElement(ForwardingParDoFn.java:42) at com.google.cloud.dataflow.sdk.runners.worker.DataflowWorkerLoggingParDoFn.processElement(DataflowWorkerLoggingParDoFn. ava:47) at com.google.cloud.dataflow.sdk.util.common.worker.ParDoOperation.process(ParDoOperation.java:53) at com.google.cloud.dataflow.sdk.util.common.worker.OutputReceiver.process(OutputReceiver.java:52) at com.google.cloud.dataflow.sdk.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:226) at com.google.cloud.dataflow.sdk.util.common.worker.ReadOperation.start(ReadOperation.java:167) at com.google.cloud.dataflow.sdk.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:71) at com.google.cloud.dataflow.sdk.runners.worker.DataflowWorker.executeWork(DataflowWorker.java:288) at com.google.cloud.dataflow.sdk.runners.worker.DataflowWorker.doWork(DataflowWorker.java:221) at com.google.cloud.dataflow.sdk.runners.worker.DataflowWorker.getAndPerformWork(DataflowWorker.java:173) at com.google.cloud.dataflow.sdk.runners.worker.DataflowWorkerHarness$WorkerThread.doWork(DataflowWorkerHarness.java:193) at com.google.cloud.dataflow.sdk.runners.worker.DataflowWorkerHarness$WorkerThread.call(DataflowWorkerHarness.java:173) at com.google.cloud.dataflow.sdk.runners.worker.DataflowWorkerHarness$WorkerThread.call(DataflowWorkerHarness.java:160) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

2016-08-22T21：47：33.469Z：错误：（84707221e08b977b）：java.lang.RuntimeException：com.google.cloud.dataflow.sdk.util.UserCodeExc ption：org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException：失败1操作：StatusRuntimeException：1次，位于com.google.cloud.dataflow.sdk.util.DoFnRunnerBase $的com.google.cloud.dataflow.sdk.runners.worker.SimpleParDoFn $ 1.output（SimpleParDoFn.java:162） DoFnContext.outputWindowedValue（DoFnRunnerBase.java:287）位于com.google.cloud.dataflow.sdk.util.DoFnRunnerBase $ DoFnProcessContext.output（DoFnRunnerBase.java:449）com.nytimes.adtech.dataflow.pipelines.BigTableSegmentData $ 2.processElement （BigTableSegmentData.java:70）引起：com.google.cloud.dataflow.sdk.util.UserCodeException：org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsExcept on：失败1动作：StatusRuntimeException：1次，在com.google com.google.cloud.dataflow.sdk.util.UserCodeException.wrap中的.cloud.dataflow.sdk.util.UserCodeException.wrap（UserCodeException.java：35）如果（UserCodeException.java:40）com.google.cloud.dataflow.sdk.util.DoFnRunnerBase.wrapUserCodeException（DoFnRunnerBase.java:368）位于com.google.cloud.dataflow.sdk.util.SimpleDoFnRunner.invokeProcessElement（SimpleDoFnRunner。 java：51）com.google.cloud.dataflow.sdk.util.DoFnRunnerBase.processElement（DoFnRunnerBase.java:138）com.google.cloud.dataflow.sdk.runners.worker.SimpleParDoFn.processElement（SimpleParDoFn.java： 190）com.google.cloud.dataflow.sdk.runners.worker.ForwardingParDoFn.processElement（ForwardingParDoFn.java:42）at com.google.cloud.dataflow.sdk.runners.worker.DataflowWorkerLoggingParDoFn.processElement（DataflowWorkerLoggingParDoFn。 ava：47）位于com.google.cloud.dataflow.sdk.udo.com。工作.OutputReceiver的com.google.cloud.dataflow.sdk.util.common.worker.ParDoOperation.process(ParDoOperation.java:53）。 com.google.cloud.dataflow.sdk.udu.DoFnRunnerBase $ doFnContext中com.google.cloud.dataflow.sdk.runners.worker.SimpleParDoFn $ 1.output（SimpleParDoFn.java:160）上的进程（OutputReceiver.java:52） .outputWindowedValue（DoFnRunnerBase.java:287）位于com.google.cloud.dataflow.sdk.util.DoFnRunnerBase $ DoFnProcessContext.output（DoFnRunnerBase.java:449）com.nytimes.adtech.dataflow.pipelines.BigTableSegmentData $ 2.processElement（ BigTableSegmentData.java:70）at com.google.cloud.dataflow.sdk.util.SumpleDoFnRunner.invokeProcessElement（SimpleDoFnRunner.java:49）com.google.cloud.dataflow.sdk.util.DoFnRunnerBase.processElement（DoFnRunnerBase.java： 138）com.google.cloud.dataflow.sdk.runners.worker.SimpleParDoFn.processElement（SimpleParDoFn.java:190）com.google.cloud.dataflow.sdk.runners.worker.ForwardingParDoFn.processElement（F） orwardingParDoFn.java:42）在com.google.cloud.dataflow.sdk.runners.worker.DataflowWorkerLoggingParDoFn.processElement（DataflowWorkerLoggingParDoFn。 ava：47）位于com.google.cloud.dataflow.sdk.udo.com。工作.OutputReceiver的com.google.cloud.dataflow.sdk.util.common.worker.ParDoOperation.process(ParDoOperation.java:53）。处理（OutputReceiver.java:52）com.google.cloud.dataflow.sdk.util.common.worker.ReadOperation.runReadLoop（ReadOperation.java:226）com.google.cloud.dataflow.sdk.util.common。在com.google.cloud.dataflow.sdk的com.google.cloud.dataflow.sdk.util.common.worker.MapTaskExecutor.execute（MapTaskExecutor.java:71）上的worker.ReadOperation.start（ReadOperation.java:167）。在com.google.cloud.dataflow.sdk的com.google.cloud.dataflow.sdk.runners.worker.DataflowWorker.doWork(DataflowWorker.java:221）上运行runners.worker.DataflowWorker.executeWork（DataflowWorker.java:288）。 runners.worker.DataflowWorker.getAndPerformWork（DataflowWorker.java:173）位于com.google.cloud.dataflow的com.google.cloud.dataflow.sdk.runners.worker.DataflowWorkerHarness $ WorkerThread.doWork（DataflowWorkerHarness.java:193）。 sdk.runners.worker.DataflowWorkerHarness $禾rkerThread.call（DataflowWorkerHarness.java:173）位于java.util.concurrent.FutureTask.run（FutureTask。）的com.google.cloud.dataflow.sdk.runners.worker.DataflowWorkerHarness $ WorkerThread.call（DataflowWorkerHarness.java:160）。 java：266）java.util.concurrent.ThreadPoolExecutor.runWorker（ThreadPoolExecutor.java:1142）at java.util.concurrent.ThreadPoolExecutor $ Worker.run（ThreadPoolExecutor.java:617）at java.lang.Thread.run（Thread的.java：745）

Caused by:

引起：

org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1 action: StatusRuntimeException: 1 time, at com.google.cloud.bigtable.hbase.BigtableBufferedMutator.handleExceptions(BigtableBufferedMutator.java:389) at com.google.cloud.bigtable.hbase.BigtableBufferedMutator.mutate(BigtableBufferedMutator.java:274) at com.google.cloud.bigtable.dataflow.CloudBigtableIO$CloudBigtableSingleTableBufferedWriteFn.processElement(CloudBigtabl IO.java:966)

org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException：失败1动作：StatusRuntimeException：1次，位于com.google.cloud.bigtable的com.google.cloud.bigtable.hbase.BigtableBufferedMutator.handleExceptions（BigtableBufferedMutator.java:389） .hbase.BigtableBufferedMutator.mutate（BigtableBufferedMutator.java:274）位于com.google.cloud.bigtable.dataflow.CloudBigtableIO $ CloudBigtableSingleTableBufferedWriteFn.processElement（CloudBigtabl IO.java:966）

Exception copied from Dataflow console

从Dataflow控制台复制的异常

(7e75740160102c05): java.lang.RuntimeException: com.google.cloud.dataflow.sdk.util.UserCodeException: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1 action: StatusRuntimeException: 1 time, at com.google.cloud.dataflow.sdk.runners.worker.SimpleParDoFn$1.output(SimpleParDoFn.java:162) at com.google.cloud.dataflow.sdk.util.DoFnRunnerBase$DoFnContext.outputWindowedValue(DoFnRunnerBase.java:287) at com.google.cloud.dataflow.sdk.util.DoFnRunnerBase$DoFnProcessContext.output(DoFnRunnerBase.java:449) at com.nytimes.adtech.dataflow.pipelines.BigTableSegmentData$2.processElement(BigTableSegmentData.java:70) Caused by: com.google.cloud.dataflow.sdk.util.UserCodeException: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1 action: StatusRuntimeException: 1 time, at com.google.cloud.dataflow.sdk.util.UserCodeException.wrap(UserCodeException.java:35) at com.google.cloud.dataflow.sdk.util.UserCodeException.wrapIf(UserCodeException.java:40) at com.google.cloud.dataflow.sdk.util.DoFnRunnerBase.wrapUserCodeException(DoFnRunnerBase.java:368) at com.google.cloud.dataflow.sdk.util.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:51) at com.google.cloud.dataflow.sdk.util.DoFnRunnerBase.processElement(DoFnRunnerBase.java:138) at com.google.cloud.dataflow.sdk.runners.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:190) at com.google.cloud.dataflow.sdk.runners.worker.ForwardingParDoFn.processElement(ForwardingParDoFn.java:42) at com.google.cloud.dataflow.sdk.runners.worker.DataflowWorkerLoggingParDoFn.processElement(DataflowWorkerLoggingParDoFn.java:47) at com.google.cloud.dataflow.sdk.util.common.worker.ParDoOperation.process(ParDoOperation.java:53) at com.google.cloud.dataflow.sdk.util.common.worker.OutputReceiver.process(OutputReceiver.java:52) at com.google.cloud.dataflow.sdk.runners.worker.SimpleParDoFn$1.output(SimpleParDoFn.java:160) at com.google.cloud.dataflow.sdk.util.DoFnRunnerBase$DoFnContext.outputWindowedValue(DoFnRunnerBase.java:287) at com.google.cloud.dataflow.sdk.util.DoFnRunnerBase$DoFnProcessContext.output(DoFnRunnerBase.java:449) at com.nytimes.adtech.dataflow.pipelines.BigTableSegmentData$2.processElement(BigTableSegmentData.java:70) at com.google.cloud.dataflow.sdk.util.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:49) at com.google.cloud.dataflow.sdk.util.DoFnRunnerBase.processElement(DoFnRunnerBase.java:138) at com.google.cloud.dataflow.sdk.runners.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:190) at com.google.cloud.dataflow.sdk.runners.worker.ForwardingParDoFn.processElement(ForwardingParDoFn.java:42) at com.google.cloud.dataflow.sdk.runners.worker.DataflowWorkerLoggingParDoFn.processElement(DataflowWorkerLoggingParDoFn.java:47) at com.google.cloud.dataflow.sdk.util.common.worker.ParDoOperation.process(ParDoOperation.java:53) at com.google.cloud.dataflow.sdk.util.common.worker.OutputReceiver.process(OutputReceiver.java:52) at com.google.cloud.dataflow.sdk.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:226) at com.google.cloud.dataflow.sdk.util.common.worker.ReadOperation.start(ReadOperation.java:167) at com.google.cloud.dataflow.sdk.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:71) at com.google.cloud.dataflow.sdk.runners.worker.DataflowWorker.executeWork(DataflowWorker.java:288) at com.google.cloud.dataflow.sdk.runners.worker.DataflowWorker.doWork(DataflowWorker.java:221) at com.google.cloud.dataflow.sdk.runners.worker.DataflowWorker.getAndPerformWork(DataflowWorker.java:173) at com.google.cloud.dataflow.sdk.runners.worker.DataflowWorkerHarness$WorkerThread.doWork(DataflowWorkerHarness.java:193) at com.google.cloud.dataflow.sdk.runners.worker.DataflowWorkerHarness$WorkerThread.call(DataflowWorkerHarness.java:173) at com.google.cloud.dataflow.sdk.runners.worker.DataflowWorkerHarness$WorkerThread.call(DataflowWorkerHarness.java:160) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1 action: StatusRuntimeException: 1 time, at com.google.cloud.bigtable.hbase.BigtableBufferedMutator.handleExceptions(BigtableBufferedMutator.java:389) at com.google.cloud.bigtable.hbase.BigtableBufferedMutator.mutate(BigtableBufferedMutator.java:274) at com.google.cloud.bigtable.dataflow.CloudBigtableIO$CloudBigtableSingleTableBufferedWriteFn.processElement(CloudBigtableIO.java:966)

2016-08-23 (13:17:54) java.lang.RuntimeException: com.google.cloud.dataflow.sdk.util.UserCodeException: org.apache.hadoop.... 2016-08-23 (13:17:54) java.lang.RuntimeException: com.google.cloud.dataflow.sdk.util.UserCodeException: org.apache.hadoop.... 2016-08-23 (13:17:54) java.lang.RuntimeException: com.google.cloud.dataflow.sdk.util.UserCodeException: org.apache.hadoop.... 2016-08-23 (13:17:54) java.lang.RuntimeException: com.google.cloud.dataflow.sdk.util.UserCodeException: org.apache.hadoop.... 2016-08-23 (13:17:54) java.lang.RuntimeException: com.google.cloud.dataflow.sdk.util.UserCodeException: org.apache.hadoop.... 2016-08-23 (13:17:54) java.lang.RuntimeException: com.google.cloud.dataflow.sdk.util.UserCodeException: org.apache.hadoop.... 2016-08-23 (13:17:54) java.lang.RuntimeException: com.google.cloud.dataflow.sdk.util.UserCodeException: org.apache.hadoop.... 2016-08-23 (13:17:54) java.lang.RuntimeException: com.google.cloud.dataflow.sdk.util.UserCodeException: org.apache.hadoop.... 2016-08-23 (13:17:54) java.lang.RuntimeException: com.google.cloud.dataflow.sdk.util.UserCodeException: org.apache.hadoop.... 2016-08-23 (13:17:54) java.lang.RuntimeException: com.google.cloud.dataflow.sdk.util.UserCodeException: org.apache.hadoop.... 2016-08-23 (13:17:54) java.lang.RuntimeException: com.google.cloud.dataflow.sdk.util.UserCodeException: org.apache.hadoop.... 2016-08-23 (13:17:54) java.lang.RuntimeException: com.google.cloud.dataflow.sdk.util.UserCodeException: org.apache.hadoop.... 2016-08-23 (13:17:54) java.lang.RuntimeException: com.google.cloud.dataflow.sdk.util.UserCodeException: org.apache.hadoop.... 2016-08-23 (13:17:54) java.lang.RuntimeException: com.google.cloud.dataflow.sdk.util.UserCodeException: org.apache.hadoop.... 2016-08-23 (13:17:54) java.lang.RuntimeException: com.google.cloud.dataflow.sdk.util.UserCodeException: org.apache.hadoop.... 2016-08-23 (13:17:54) java.lang.RuntimeException: com.google.cloud.dataflow.sdk.util.UserCodeException: org.apache.hadoop.... 2016-08-23 (13:17:54) java.lang.RuntimeException: com.google.cloud.dataflow.sdk.util.UserCodeException: org.apache.hadoop.... 2016-08-23 (13:17:54) java.lang.RuntimeException: com.google.cloud.dataflow.sdk.util.UserCodeException: org.apache.hadoop.... 2016-08-23 (13:17:54) java.lang.RuntimeException: com.google.cloud.dataflow.sdk.util.UserCodeException: org.apache.hadoop.... 2016-08-23 (13:17:54) java.lang.RuntimeException: com.google.cloud.dataflow.sdk.util.UserCodeException: org.apache.hadoop.... 2016-08-23 (13:17:54) java.lang.RuntimeException: com.google.cloud.dataflow.sdk.util.UserCodeException: org.apache.hadoop.... 2016-08-23 (13:17:54) java.lang.RuntimeException: com.google.cloud.dataflow.sdk.util.UserCodeException: org.apache.hadoop.... 2016-08-23 (13:17:54) java.lang.RuntimeException: com.google.cloud.dataflow.sdk.util.UserCodeException: org.apache.hadoop.... 2016-08-23 (13:17:54) java.lang.RuntimeException: com.google.cloud.dataflow.sdk.util.UserCodeException: org.apache.hadoop.... 2016-08-23 (13:17:54) java.lang.RuntimeException: com.google.cloud.dataflow.sdk.util.UserCodeException: org.apache.hadoop.... 2016-08-23 (13:17:54) java.lang.RuntimeException:

2016-08-23（13:17:54）java.lang.RuntimeException：com.google.cloud.dataflow.sdk.util.UserCodeException：org.apache.hadoop .... 2016-08-23（13:17）：54）java.lang.RuntimeException：com.google.cloud.dataflow.sdk.util.UserCodeException：org.apache.hadoop .... 2016-08-23（13:17:54）java.lang.RuntimeException： com.google.cloud.dataflow.sdk.util.UserCodeException：org.apache.hadoop .... 2016-08-23（13:17:54）java.lang.RuntimeException：com.google.cloud.dataflow.sdk .util.UserCodeException：org.apache.hadoop .... 2016-08-23（13:17:54）java.lang.RuntimeException：com.google.cloud.dataflow.sdk.util.UserCodeException：org.apache。 hadoop .... 2016-08-23（13:17:54）java.lang.RuntimeException：com.google.cloud.dataflow.sdk.util.UserCodeException：org.apache.hadoop .... 2016-08- 23（13:17:54）java.lang.RuntimeException：com.google.cloud.dataflow.sdk.util.UserCodeException：org.apache.hadoop .... 2016-08-23（13:17:54）java .lang.RuntimeException：com.google.cloud.dataflow.sdk.util.UserCodeException：org.apache.had oop .... 2016-08-23（13:17:54）java.lang.RuntimeException：com.google.cloud.dataflow.sdk.util.UserCodeException：org.apache.hadoop .... 2016-08- 23（13:17:54）java.lang.RuntimeException：com.google.cloud.dataflow.sdk.util.UserCodeException：org.apache.hadoop .... 2016-08-23（13:17:54）java .lang.RuntimeException：com.google.cloud.dataflow.sdk.util.UserCodeException：org.apache.hadoop .... 2016-08-23（13:17:54）java.lang.RuntimeException：com.google。 cloud.dataflow.sdk.util.UserCodeException：org.apache.hadoop .... 2016-08-23（13:17:54）java.lang.RuntimeException：com.google.cloud.dataflow.sdk.util.UserCodeException ：org.apache.hadoop .... 2016-08-23（13:17:54）java.lang.RuntimeException：com.google.cloud.dataflow.sdk.util.UserCodeException：org.apache.hadoop ... 。2016-08-23（13:17:54）java.lang.RuntimeException：com.google.cloud.dataflow.sdk.util.UserCodeException：org.apache.hadoop .... 2016-08-23（13： 17:54）java.lang.RuntimeException：com.google.cloud.dataflow.sdk.util.UserCodeException：org.ap ache.hadoop .... 2016-08-23（13:17:54）java.lang.RuntimeException：com.google.cloud.dataflow.sdk.util.UserCodeException：org.apache.hadoop .... 2016- 08-23（13:17:54）java.lang.RuntimeException：com.google.cloud.dataflow.sdk.util.UserCodeException：org.apache.hadoop .... 2016-08-23（13:17:54 ）java.lang.RuntimeException：com.google.cloud.dataflow.sdk.util.UserCodeException：org.apache.hadoop .... 2016-08-23（13:17:54）java.lang.RuntimeException：com。 google.cloud.dataflow.sdk.util.UserCodeException：org.apache.hadoop .... 2016-08-23（13:17:54）java.lang.RuntimeException：com.google.cloud.dataflow.sdk.util .UserCodeException：org.apache.hadoop .... 2016-08-23（13:17:54）java.lang.RuntimeException：com.google.cloud.dataflow.sdk.util.UserCodeException：org.apache.hadoop。 ... 2016-08-23（13:17:54）java.lang.RuntimeException：com.google.cloud.dataflow.sdk.util.UserCodeException：org.apache.hadoop .... 2016-08-23（ 13:17:54）java.lang.RuntimeException：com.google.cloud.dataflow.sdk.util.UserCodeException ：org.apache.hadoop .... 2016-08-23（13:17:54）java.lang.RuntimeException：com.google.cloud.dataflow.sdk.util.UserCodeException：org.apache.hadoop ... .2016-08-23（13:17:54）java.lang.RuntimeException：

Thanks in Advance

提前致谢

1 个解决方案

#1

We spoke offline. The problem here is that you have too many Dataflow workers compared to the number of Cloud Bigtable nodes in your cluster. You need to change that ratio by either reducing Dataflow workers or contacting our team to increase your Cloud Bigtable resources.

我们离线了。这里的问题是，与集群中的Cloud Bigtable节点数相比，您有太多的Dataflow工作者。您需要通过减少Dataflow工作人员或联系我们的团队来增加Cloud Bigtable资源来更改该比率。

Bigtable was performing admirably relative to the amount of Cloud Bigtable Nodes you had, but the load from Dataflow was too high to reliably handle.

Bigtable相对于您拥有的Cloud Bigtable节点的数量表现令人钦佩，但Dataflow的负载太高而无法可靠地处理。

You can view your usage in the "CPU Usage" graph in the Google Cloud console. Anything over 80% of your capacity is likely to cause problems. If you get more Bigtable Quota, you can increase the number of nodes you have before you run the Dataflow job, and reduce it after the job is done. For example, Scio does that.

您可以在Google Cloud控制台的“CPU使用率”图表中查看您的使用情况。超过80％的容量可能会导致问题。如果获得更多Bigtable Quota，则可以在运行Dataflow作业之前增加节点数，并在作业完成后减少它。例如，Scio就是这么做的。

Regarding "Also what happens to data inside the pipeline? Is it reprocessed? Or is it lost in transit to BigTable?":

关于“管道内的数据会发生什么？它是重新处理的吗？还是在传输到BigTable时丢失了？”：

Dataflow tries to send the data to Bigtable again. In those cases, Dataflow's retry mechanism will correct for temporary issues.

Dataflow再次尝试将数据发送到Bigtable。在这些情况下，Dataflow的重试机制将纠正临时问题。

Unfortunately, when the problem turns out to be Cloud Bigtable overload, the retries compound the problem by sending more traffic to Bigtable, thereby exacerbating the problem.

不幸的是，当问题变成Cloud Bigtable过载时，重试会通过向Bigtable发送更多流量来加剧问题，从而加剧了问题。

#1