We run a meta search engine (price comparison) where each search spawns a number of real-time searches on various websites and presents the merged and sorted search results on our site.
We use PHP/MySQL/Apache on a Linux Debian server in a quite straight-forward setup, but the background processing of searching multiple sites in parallel is handled by Tomcat and a Java servlet. However, because of some problems with this setup (Tomcat), I am investigation new approaches to the background searching.
One way that looks promising is also very simple: From the main search page, each individual search script is run as PHP CLI using exec():
exec("nohup /usr/bin/php search.php '$params' &> /dev/null &");
Using the ampersand in the end sends the PHP CLI scripts directly to the background and the main page can continue without waiting. There are about 20 scripts running for each search.
It seems to perform well enough, but when I run a stress test using Apache Benchmark with concurrent requests, problems start to show up. What happens is that MySQL reports this error:
Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (11)
It seems to be a variation of "Too many connections" but it does not help to raise the limits in my.cnf. But what really puzzles me is that these errors are present even if I call a small dummy script that does nothing, and certainly does not access MySQL. So, it looks like just running PHP CLI puts load on MySQL even if it is not used.
There are at most about 700 instances of PHP running during my tests, and the CPU and memory load is lower than when using Tomcat for calling the PHP scripts, so from that point of view it looks feasable.
Does someone have a clue about the MySQL problem? And of course, I am open for new ideas of how to handling the background tasks!
Regards,
Martin
No comments:
Post a Comment