Presto集群不能对hive定义的表运行查询——“没有可用的节点可以运行查询”

时间:2021-12-11 01:37:47

I am running a 2 node presto cluster pointed to hive on EMR, which is configured with data on S3.

我正在运行一个指向EMR上的hive的2节点presto集群,它配置了S3上的数据。

The hive metadata is visible; in the CLI I can describe (table) claim1, and see metadata about it.

可见的hive元数据;在CLI中,我可以描述(表)claim1,并查看关于它的元数据。

Both nodes show up as active in the sys.node table.

这两个节点在sys中都显示为活动的。节点表。

When I run a query (select count(*) from claim1 where col1='M'), I see a lot of logging on the coordinator node, finishing with:

当我运行claim1中的查询(选择count(*),其中col1='M')时,我在协调器节点上看到很多日志记录,最后是:

2014-05-19T22:19:32.760+0000     INFO   HiveHdfsWalker-144  stdout  22:19:32.760 [HiveHdfsWalker-144] DEBUG c.a.s.s.m.t.XmlResponsesSaxParser - Examining listing for bucket: unzippeddata

2014-05-19T22:19:32.763+0000     INFO   HiveHdfsWalker-144  stdout  22:19:32.763 [HiveHdfsWalker-144] DEBUG com.amazonaws.request - Received successful response: 200, AWS Request ID: 9B844EEC8586FF3B

2014-05-19T22:19:32.766+0000    DEBUG   query-scheduler-8   com.facebook.presto.execution.SqlStageExecution Stage 20140519_221932_00005_mfhtx.1 is FAILED
2014-05-19T22:19:32.766+0000    DEBUG   query-scheduler-6   com.facebook.presto.execution.SqlStageExecution Stage 20140519_221932_00005_mfhtx.0 is FAILED
2014-05-19T22:19:32.768+0000    DEBUG   query-scheduler-7   com.facebook.presto.execution.QueryStateMachine Query 20140519_221932_00005_mfhtx is FAILED
2014-05-19T22:19:32.770+0000    ERROR   Stage-20140519_221932_00005_mfhtx.1-126 com.facebook.presto.execution.SqlStageExecution Error while starting stage 20140519_221932_00005_mfhtx.1
com.facebook.presto.spi.PrestoException: No nodes available to run query
    at com.facebook.presto.util.Failures.checkCondition(Failures.java:79) ~[presto-main-0.68.jar:0.68]
    at com.facebook.presto.util.Failures.checkCondition(Failures.java:73) ~[presto-main-0.68.jar:0.68]
    at com.facebook.presto.execution.NodeScheduler$NodeSelector.computeAssignments(NodeScheduler.java:184) ~[presto-main-0.68.jar:0.68]
    at com.facebook.presto.execution.SqlStageExecution.scheduleSourcePartitionedNodes(SqlStageExecution.java:631) [presto-main-0.68.jar:0.68]
    at com.facebook.presto.execution.SqlStageExecution.startTasks(SqlStageExecution.java:549) [presto-main-0.68.jar:0.68]
    at com.facebook.presto.execution.SqlStageExecution.access$200(SqlStageExecution.java:91) [presto-main-0.68.jar:0.68]
    at com.facebook.presto.execution.SqlStageExecution$4.run(SqlStageExecution.java:521) [presto-main-0.68.jar:0.68]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_51]
    at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_51]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_51]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_51]
    at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51]
2014-05-19T22:19:32.775+0000     INFO   query-scheduler-7   com.facebook.presto.event.query.QueryMonitor    TIMELINE: Query 20140519_221932_00005_mfhtx :: elapsed 483.00ms :: planning 74.12ms :: scheduling 409.00ms :: running 0.00ms :: finishing 409.00ms :: begin 2014-05-19T22:19:32.285Z :: end 2014-05-19T22:19:32.768Z
2014-05-19T22:19:32.872+0000    DEBUG   task-notification-0 com.facebook.presto.execution.TaskStateMachine  Task 20140519_221932_00005_mfhtx.0.0 is CANCELED
2014-05-19T22:19:32.880+0000    DEBUG   20140519_221932_00005_mfhtx.0.0-0-56    com.facebook.presto.execution.TaskExecutor  Split 20140519_221932_00005_mfhtx.0.0-0 (start = 1400537972443, wall = 437 ms, cpu = 3 ms, calls = 2) is finished

...or alternately:

…或交替:

2014-05-19T22:22:43.972+0000     INFO   HiveHdfsWalker-170  stdout  22:22:43.972 [HiveHdfsWalker-170] DEBUG c.a.s.s.m.t.XmlResponsesSaxParser - Parsing XML response document with handler: class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler

2014-05-19T22:22:43.972+0000     INFO   HiveHdfsWalker-170  stdout  22:22:43.972 [HiveHdfsWalker-170] DEBUG c.a.s.s.m.t.XmlResponsesSaxParser - Examining listing for bucket: unzippeddata

2014-05-19T22:22:43.976+0000     INFO   HiveHdfsWalker-170  stdout  22:22:43.976 [HiveHdfsWalker-170] DEBUG com.amazonaws.request - Received successful response: 200, AWS Request ID: 476D4ABA552DAA66

2014-05-19T22:22:43.979+0000    DEBUG   query-scheduler-17  com.facebook.presto.execution.SqlStageExecution Stage 20140519_222243_00007_mfhtx.1 is FAILED
2014-05-19T22:22:43.979+0000    ERROR   Stage-20140519_222243_00007_mfhtx.1-160 com.facebook.presto.execution.SqlStageExecution Error while starting stage 20140519_222243_00007_mfhtx.1
com.facebook.presto.spi.PrestoException: No nodes available to run query
    at com.facebook.presto.util.Failures.checkCondition(Failures.java:79) ~[presto-main-0.68.jar:0.68]
    at com.facebook.presto.util.Failures.checkCondition(Failures.java:73) ~[presto-main-0.68.jar:0.68]
    at com.facebook.presto.execution.NodeScheduler$NodeSelector.computeAssignments(NodeScheduler.java:184) ~[presto-main-0.68.jar:0.68]
    at com.facebook.presto.execution.SqlStageExecution.scheduleSourcePartitionedNodes(SqlStageExecution.java:631) [presto-main-0.68.jar:0.68]
    at com.facebook.presto.execution.SqlStageExecution.startTasks(SqlStageExecution.java:549) [presto-main-0.68.jar:0.68]
    at com.facebook.presto.execution.SqlStageExecution.access$200(SqlStageExecution.java:91) [presto-main-0.68.jar:0.68]
    at com.facebook.presto.execution.SqlStageExecution$4.run(SqlStageExecution.java:521) [presto-main-0.68.jar:0.68]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_51]
    at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_51]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_51]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_51]
    at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51]
2014-05-19T22:22:43.983+0000    DEBUG   query-scheduler-13  com.facebook.presto.execution.SqlStageExecution Stage 20140519_222243_00007_mfhtx.0 is FAILED
2014-05-19T22:22:43.984+0000    DEBUG   query-scheduler-14  com.facebook.presto.execution.QueryStateMachine Query 20140519_222243_00007_mfhtx is FAILED
2014-05-19T22:22:43.990+0000     INFO   query-scheduler-14  com.facebook.presto.event.query.QueryMonitor    TIMELINE: Query 20140519_222243_00007_mfhtx :: elapsed 233.00ms :: planning 60.01ms :: scheduling 173.00ms :: running 0.00ms :: finishing 173.00ms :: begin 2014-05-19T22:22:43.751Z :: end 2014-05-19T22:22:43.984Z
2014-05-19T22:22:44.088+0000    DEBUG   task-notification-3 com.facebook.presto.execution.TaskStateMachine  Task 20140519_222243_00007_mfhtx.0.0 is CANCELED
2014-05-19T22:22:44.102+0000    DEBUG   20140519_222243_00007_mfhtx.0.0-0-50    com.facebook.presto.execution.TaskExecutor  Split 20140519_222243_00007_mfhtx.0.0-0 (start = 1400538163839, wall = 259 ms, cpu = 0 ms, calls = 2) is finished

The non-coordinator node sometimes (but not consistently) gets a couple lines in its logs:

非协调节点有时(但不一致)在其日志中获取几行:

2014-05-19T22:19:32.340+0000    DEBUG   task-notification-10    com.facebook.presto.execution.TaskStateMachine  Task 20140519_222103_00006_mfhtx.0.0 is CANCELED
2014-05-19T22:19:32.352+0000    DEBUG   20140519_222103_00006_mfhtx.0.0-0-48    com.facebook.presto.execution.TaskExecutor  Split 20140519_222103_00006_mfhtx.0.0-0 (start = 1400538063009, wall = 343 ms, cpu = 0 ms, calls = 2) is finished

1 个解决方案

#1


3  

This problem used to be caused by forgetting to configure datasources on the workers, but this is no longer a problem because Presto no longer requires explicit configuration for this (and the config property will likely be removed in a future release).

这个问题以前是由于忘记在worker上配置数据源而导致的,但这不再是问题,因为Presto不再需要为此进行显式配置(配置属性很可能在将来的版本中被删除)。

#1


3  

This problem used to be caused by forgetting to configure datasources on the workers, but this is no longer a problem because Presto no longer requires explicit configuration for this (and the config property will likely be removed in a future release).

这个问题以前是由于忘记在worker上配置数据源而导致的,但这不再是问题,因为Presto不再需要为此进行显式配置(配置属性很可能在将来的版本中被删除)。