PHP中的并行处理——你是怎么做的?

I am currently trying to implement a job queue in php. The queue will then be processed as a batch job and should be able to process some jobs in parallel.

我目前正在尝试在php中实现一个作业队列。然后队列将被作为批处理作业处理，并且应该能够并行地处理一些作业。

I already did some research and found several ways to implement it, but I am not really aware of their advantages and disadvantages.

我已经做了一些研究，找到了一些方法来实现它，但是我并没有真正意识到它们的优缺点。

E.g. doing the parallel processing by calling a script several times through fsockopen like explained here:
Easy parallel processing in PHP

例如，通过fsockopen多次调用脚本进行并行处理，如这里所述:PHP中的简单并行处理

Another way I found was using the curl_multi functions.
curl_multi_exec PHP docs

另一种方法是使用curl_multifunctions。curl_multi_exec PHP文档

But I think those 2 ways will add pretty much overhead for creating batch processing on a queue taht should mainly run on the background?

但是我认为这两种方法会增加在队列taht主要在后台运行时创建批处理的开销吗?

I also read about pcntl_fork wich also seems to be a way to handle the problem. But that looks like it can get really messy if you don't really know what you are doing (like me at the moment ;)).

我也读了关于pcntl_fork的文章，似乎也是处理这个问题的一种方法。但如果你真的不知道自己在做什么(就像现在的我一样;)，事情就会变得一团糟。

I also had a look at Gearman, but there I would also need to spawn the worker threads dynamically as needed and not just run a few and let the gearman job server then sent it to the free workers. Especially because the threads should be exit cleanly after one job has been executed, to not run into eventual memory leaks (code may not be perfect in that issue).
Gearman Getting Started

我还看了Gearman，但是我还需要根据需要动态地生成工作线程，而不是只运行一些，然后让Gearman作业服务器将它发送给*工作者。特别是因为线程应该在执行了一个作业之后干净地退出，以免最终导致内存泄漏(在这个问题中，代码可能并不完美)。Gearman开始

So my question is, how do you handle parallel processing in PHP? And why do you choose your method, which advantages/disadvantages may the different methods have?

我的问题是，如何在PHP中处理并行处理?你为什么选择你的方法，不同的方法有哪些优点和缺点?

Thanks for any input.

谢谢你的任何输入。

7 个解决方案

#1

i use exec(). Its easy and clean. You basically need to build a thread manager, and thread scripts, that will do what you need.

我使用exec()。其简单和干净。您基本上需要构建一个线程管理器和线程脚本，它们将完成所需的工作。

I dont like fsockopen() because it will open a server connection, that will build up and may hit the apache's connection limit

我不喜欢fsockopen()，因为它会打开一个服务器连接，这会建立起来，可能会达到apache的连接限制

I dont like curl functions for the same reason

我不喜欢旋度函数是因为同样的原因

I dont like pnctl because it needs the pnctl extension available, and you have to keep track of parent/child relations.

我不喜欢pnctl，因为它需要pnctl扩展，而且你必须跟踪父母/孩子的关系。

never played with gearman...

从来没有玩过gearman……

#2

Well I guess we have 3 options there:

我想我们有三个选择

A. Multi-Thread:

多线程:

PHP does not support multithread natively. But there is one PHP extension (experimental) called pthreads (https://github.com/krakjoe/pthreads) that allows you to do just that.

PHP本身不支持多线程。但是有一个PHP扩展(实验性的)叫做pthreads (https://github.com/krakjoe/pthreads)，允许您这样做。

B. Multi-Process:

b多进程:

This can be done in 3 ways:

这可以通过三种方式实现:

• Forking
• 分叉
• Executing Commands
• 执行命令
• Piping
• 管道

C. Distributed Parallel Processing:

c .分布式并行处理:

How it works:

它是如何工作的:

1. The Client App sends data (AKA message) “can be JSON formatted” to the Engine (MQ Engine) “can be local or external a web service”
2. 客户端应用程序向引擎(MQ引擎)发送数据(即消息)“可以是JSON格式的”，“可以是本地的或外部的web服务”。
3. The MQ Engine stores the data “mostly in Memory and optionally in Database” inside a queues (you can define the queue name)
4. MQ引擎将数据“主要存储在内存中，可选地存储在数据库中”存储在队列中(您可以定义队列名称)
5. The Client App asks the MQ Engine for a data (message) to be processed them in order (FIFO or based on priority) “you can also request data from specific queue".
6. 客户端应用程序要求MQ引擎按顺序(FIFO或基于优先级)处理数据(消息)，“您还可以从特定队列请求数据”。

Some MQ Engines:

一些MQ引擎:

• ZeroMQ (good option, hard to use) a message orientated IPC Library, is a Message Queue Server in Erlang, stores jobs in memory. It is a socket library that acts as a concurrency framework. Faster than TCP for clustered products and supercomputing.
• ZeroMQ(好选项，难用)面向消息的IPC库，是Erlang中的消息队列服务器，将作业存储在内存中。它是一个作为并发框架的套接字库。在集群产品和超级计算上比TCP快。
• RabbitMQ (good option, easy to use) self hosted, Enterprise Message Queues, Not really a work queue - but rather a message queue that can be used as a work queue but requires additional semantics.
• RabbitMQ(良好的选项，易于使用)是自托管的企业消息队列，不是真正的工作队列——而是可以用作工作队列但需要附加语义的消息队列。
• Beanstalkd (best option, easy to use) (Laravel built in support, built by facebook, for work queue) - has a "Beanstalkd console" tool which is very nice
• Beanstalkd(最好的选择，易于使用)(Laravel内置支持，由facebook构建，用于工作队列)——有一个“Beanstalkd控制台”工具，非常不错
• Gearman (problem: centralized broker system for distributed processing)
• Gearman(问题:分布式处理的集中式代理系统)
• Apache ActiveMQ the most popular open source message broker in Java, (problem: lot of bugs and problems)
• Apache ActiveMQ是Java中最流行的开源消息代理(问题:很多bug和问题)
• Amazon SQS (Laravel built in support, Hosted - so no administration is required. Not really a work queue thus will require extra work to handle semantics such as burying a job)
• Amazon SQS (Laravel内置支持，托管——因此不需要管理。并不是真正的工作队列，因此需要额外的工作来处理语义，比如埋葬一个作业)
• IronMQ (Laravel built in support, Written in Go, Available both as cloud version and on-premise)
• IronMQ (Laravel内置支持，用Go编写，可作为云版本和本地版本)
• Redis (Laravel built in support, not that fast as its not designed for that)
• Redis (Laravel内置了支持功能，但没有设计得那么快)
• Sparrow (written in Ruby that based on memcache)
• 麻雀(基于memcache的Ruby语言)
• Starling (written in Ruby that based on memcache, built in twitter)
• Starling(用Ruby编写，基于memcache，在twitter上构建)
• Kestrel (just another QM)
• 红隼(另一个QM)
• Kafka (Written at LinkedIn in Scala)
• 卡夫卡(在斯卡拉的LinkedIn上)
• EagleMQ open source, high-performance and lightweight queue manager (Written in C)
• EagleMQ开源、高性能和轻量级队列管理器(用C编写)

More of them can be foun here: http://queues.io

在这里可以看到更多的内容:http://queues.io

#3

If your application is going to run under a unix/linux enviroment I would suggest you go with the forking option. It's basically childs play to get it working. I have used it for a Cron manager and had code for it to revert to a Windows friendly codepath if forking was not an option.

如果您的应用程序将在unix/linux环境下运行，我建议您使用forking选项。这基本上是孩子们的游戏。我曾将它用于Cron管理器，并为它编写代码，以便在不允许分叉的情况下恢复到Windows友好的代码depath。

The options of running the entire script several times do, as you state, add quite a bit of overhead. If your script is small it might not be a problem. But you will probably get used to doing parallel processing in PHP by the way you choose to go. And next time when you have a job that uses 200mb of data it might very well be a problem. So you'd be better of learning a way that you can stick with.

在您声明时，多次运行整个脚本的选项会增加相当大的开销。如果您的脚本很小，这可能不是问题。但是您可能会习惯使用PHP进行并行处理。下次当你有一份使用200mb数据的工作时，这很可能是个问题。所以你最好学习一种你可以坚持的方法。

I have also tested Gearman and I like it a lot. There are a few thing to think about but as a whole it offers a very good way to distribute works to different servers running different applications written in different languages. Besides setting it up, actually using it from within PHP, or any other language for that matter, is... once again... childs play.

我也测试过Gearman，我很喜欢它。有一些东西需要考虑，但是作为一个整体，它提供了一种很好的方式来将作品分发给运行不同语言的不同应用程序的不同服务器。除了设置它，实际上从PHP或其他语言中使用它是……再一次…孩子玩。

It could very well be overkill for what you need to do. But it will open your eyes to new possibilities when it comes to handling data and jobs, so I would recommend you to try Gearman for that fact alone.

对于你需要做的事情来说，这很可能会有点过头。但是，当涉及到处理数据和作业时，它将使您看到新的可能性，所以我建议您单独尝试Gearman。

#4

I prefer exec() and gearman. exec() is easy and no connection and less memory consuming. gearman should need a socket connection and the worker should take some memory. But gearman is more flexible and faster than exec(). And the most important is that it can deploy the worker in other server. If the work is time and resource consuming. I'm using gearman in my current project.

我更喜欢exec()和gearman。exec()非常简单，没有连接，占用的内存也更少。gearman需要一个套接字连接，工作人员需要一些内存。但是gearman比exec()更灵活、更快。最重要的是，它可以将worker部署到其他服务器上。如果工作是时间和资源消耗。我现在用的是gearman。

#5

I use PHP's pnctl - it is good as long as you know what you do. I understand you situation but I don't think it's something difficult to understand our code, we just have to be little more conscious than ever when implementing JOB queue or Parallel process.

我使用PHP的pnctl—只要您知道自己在做什么，它就很好。我理解您的情况，但我不认为理解我们的代码有什么困难，我们只需要在实现作业队列或并行进程时比以前更有意识一点。

I feel as long as you code it perfectly and make sure the flow is perfect off-course you should keep PARALLEL PROCESS in mind when you implement.

我觉得只要你把它写得完美，确保流程是完美的，你就应该在实现的时候记住并行过程。

Where you could do mistakes:

你可以犯错误的地方:

1. Loops - should be able to handle by GLOBAL vars.
2. 循环-应该能够由全球的vars处理。
3. Processing some set of transactions - again as long as you define the sets proper, you should be able to get it done.
4. 处理一些事务集——同样，只要定义了正确的事务集，就应该能够完成它。

Take a look at this example - https://github.com/rakesh-sankar/Tools/blob/master/PHP/fork-parallel-process.php.

看一下这个例子——https://github.com/rakesh-sankar/Tools/blob/master/PHP/fork-parallel-process.php。

Hope it helps.

希望它可以帮助。

#6

The method described in 'Easy parallel processing in PHP' is downright scary - the principle is OK - but the implementation??? As you've already pointed out the curl_multi_ fns provide a much better way of implementing this approach.

“PHP中简单的并行处理”中描述的方法非常可怕——原理还可以——但是实现呢?正如您已经指出的，curl_multi_fns提供了一种更好的实现这种方法的方法。

But I think those 2 ways will add pretty much overhead

但我认为这两种方法会增加很多开销

Yes, you probably don't need a client and server HTTP stack for handing off the job - but unless you're working for Google, your development time is much more expensive than your hardware costs - and there are plenty of tools for managing HTTP/analysing performance - and there is a defined standard covering stuff such as status notifications and authentication.

是的,你可能不需要客户端和服务器的HTTP栈递给了工作——但除非你为谷歌工作,你的开发时间是昂贵得多比你的硬件成本,有大量的工具来管理HTTP /分析性能,有一个定义的标准涵盖的东西比如状态通知和身份验证。

A lot of how you implement the solution depends on the level transactional integrity you require and whether you require in-order processing.

实现解决方案的方式很大程度上取决于所需的事务完整性以及是否需要按顺序处理。

Out of the approaches you mention I'd recommend focussing on the HTTP request method using curl_multi_ . But if you need good transactional control / in order delivery then you should definitely run a broker daemon between the source of the messages and the processing agents (there is a well written single threaded server suitable for use as a framework for the broker here). Note that the processing agents should process a single message at a time.

除了您提到的方法之外，我建议使用curl_multi_关注HTTP请求方法。但是，如果您需要好的事务控制/订单交付，那么您肯定应该在消息源和处理代理之间运行代理守护进程(这里有一个编写良好的单线程服务器，适合用作代理的框架)。注意，处理代理每次应该处理一个消息。

If you need a highly scalable solution, then take a look at a proper message queuing system such as RabbitMQ.

如果您需要高度可伸缩的解决方案，那么请查看正确的消息队列系统，如RabbitMQ。

HTH

HTH

C.

C。

#7

Here's a summary of a few options for parallel processing in PHP.

下面是PHP中一些并行处理选项的总结。

AMP

Checkout Amp - Asynchronous concurrency made simple - this looks to be the most mature PHP library I've seen for parallel processing.

签出Amp—异步并发变得简单—这看起来是我所见过的最成熟的用于并行处理的PHP库。

Peec's Process Class

This class was posted in the comments of PHP's exec() function and provides a real simple starting point for forking new processes and keeping track of them.

这个类被发布在PHP exec()函数的注释中，并为创建新进程和跟踪它们提供了一个非常简单的起点。

Example:

例子:

// You may use status(), start(), and stop(). notice that start() method gets called automatically one time.
$process = new Process('ls -al');

// or if you got the pid, however here only the status() metod will work.
$process = new Process();
$process.setPid(my_pid);

// Then you can start/stop/check status of the job.
$process.stop();
$process.start();
if ($process.status()) {
    echo "The process is currently running";
} else {
    echo "The process is not running.";
}

Other Options Compared

There's also a great article Async processing or multitasking in PHP that explains the pros and cons of various approaches:

PHP中有一篇很棒的文章，介绍了异步处理或多任务处理的优缺点:

• pthreads extension (see also this SitePoint article)
• pthreads扩展(参见这个SitePoint文章)
• Amp\Thread Library
• Amp \线程库
• hack's async (requires running Facebook's HHVM)
• hack's async(需要运行Facebook的HHVM)
• pcntl_fork
• pcntl_fork
• popen
• popen
• fopen/curl/fsockopen
• fopen /卷/ fsockopen

Doorman

Then, there's also this simple tutorial which was wrapped up into a little library called Doorman.

然后，还有一个简单的教程，它被包裹在一个叫做看门人的小图书馆里。

Hope these links provide a useful starting point for more research.

希望这些链接能为更多的研究提供一个有用的起点。