在节点中迭代大量的异步调用/结果。js(使用ES6 / async / bluebird / generator)?

I am writing a utility in node.js that has to process and concatenate a large number of files every night. In synchronous pseudocode it would look like that (omitting try / catch for clarity):

我正在node中编写一个实用程序。它必须每天晚上处理和连接大量的文件。在同步伪代码中，它看起来是这样的(为了清晰起见，省略了try / catch):

while (true) {
    var next = db.popNext();
    if (!next) return;

    out.append(next);
}

However, in the library I am using popNext() is actually a node-style asynchronous method and rather looks like this: popNext(callback).

但是，在我正在使用的库中，popNext()实际上是一个节点样式的异步方法，它看起来是这样的:popNext(回调)。

Since I am writing the middleware from scratch I could use --harmony (e.g., generators), async or bluebird.

因为我正在从头开始编写中间件，所以我可以使用harmony(例如，generator)、async或bluebird。

Ideally I would prefer something like:

理想情况下，我更喜欢以下内容:

forEachOrdered(db.popNext, (error, next, ok, fail) => {
   if(error) return; // skip

   // If there was an internal error, terminate the whole loop.
   if(out.append(next)) ok();
   else fail();
}).then(() => {
   // All went fine.
}).catch(e => {
   // Fail was called.
});

However, I am open to other 'standard' solutions. I was wondering what would be the most concise solution to this problem?

然而，我对其他“标准”解决方案持开放态度。我想知道最简洁的解决方法是什么?

Edit Just spawning all (in a regular for loop) at the same time would probably not solve my problem since we're talking about 100k's and for every item I have to open and read a file, so I would probably run out of file descriptors.

同时编辑所有(在一个常规的for循环中)可能不能解决我的问题，因为我们讨论的是100k，对于每个我必须打开和读取文件的项，我可能会用完文件描述符。

1 个解决方案

#1

Here is a solution using bluebird coroutines using your "ideal" code:

下面是使用“理想”代码的bluebird coroutines的解决方案:

var db = Promise.promisifyAll(db);

var processAll = Promise.coroutine(function*(){
  while(true){
    var next = yield db.popNextAsync(); // promisify gives Async suffix
    if(!next) return;
    out.append(next); // some processing
  }       
});

In ES2016 (ES7) this becomes:

2016年ES7 (ES7):

var db = Promise.promisifyAll(db); // still need to promisify

async function processAll(){
  let next;
  while(next = await db.popNextAsync()){
     // whatever
     out.append(next);
  }
}

Although, I'd argue the output collection should be an iterable (and lazy) too, so using ES2016 async iterators:

尽管如此，我认为输出集合也应该是可迭代的(而且是惰性的)，所以使用ES2016异步迭代器:

var db = Promise.promisifyAll(db);
async function* process(){
    while(true){
       var val = await db.popNextAsync();
       if(!val) return;
       // process val;
       yield process(val); // yield it forward
    }
}

Although if we really want to go all out here, after converting db.popNext into an async iterator this becomes in ES2016 async for notation:

但是如果我们真的想在这里，在转换db之后。将popNext插入到异步迭代器中，这在ES2016异步中表示符号:

async function* processAll(){
    for async(let next of db.asAsyncIterator()){ // need to write this like above
       yield process(next); // do some processing
    }
}

Leveraging the whole ES2016 async iteration API. If you can't, or don't want to use generators you can always convert while loops to recursion:

利用整个ES2016异步迭代API。如果您不能，或者不想使用生成器，您可以在循环到递归时进行转换:

function processAll(){ // works on netscape 7
   return db.popNextAsync().then(function next(value){
      if(!value) return;
      out.push(process(value));
      return db.popNextAsync().then(next); // after bluebird promisify
   });
}

#1