Thursday, November 16, 2017

Wordpress and Apache crashing MySql (CLOSE_WAIT)

Last days mysql server on my vps start crashing, sometimes right after restart.

I found that problem in wordpress website. After service mysql start many apache2 processes overload memory and cpu. This is screenshot of top.



enter image description here



I didn't find anything usefull from apache logs, mysql error log show messages about lack of memory:



140922 12:24:43 [Note] Plugin 'FEDERATED' is disabled.
140922 12:24:43 InnoDB: The InnoDB memory heap is disabled
140922 12:24:43 InnoDB: Mutexes and rw_locks use GCC atomic builtins
140922 12:24:43 InnoDB: Compressed tables use zlib 1.2.8

140922 12:24:43 InnoDB: Using Linux native AIO
140922 12:24:43 InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137363456 bytes) failed; errno 12
140922 12:24:43 InnoDB: Completed initialization of buffer pool
140922 12:24:43 InnoDB: Fatal error: cannot allocate memory for the buffer pool
140922 12:24:43 [ERROR] Plugin 'InnoDB' init function returned error.
140922 12:24:43 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
140922 12:24:43 [ERROR] Unknown/unsupported storage engine: InnoDB
140922 12:24:43 [ERROR] Aborting


140922 12:24:43 [Note] /usr/sbin/mysqld: Shutdown complete


First I thought maybe problem in db and checked database with mysqlcheck --all-databases but all ok, I tried drop and create new database, no luck.



Also I tried to delete wordpress plugins folder. Wordpress is last version, comments disabled, 3 users, little amount of visitors (corporate blog).



Now I don't know where to look and how to diagnose problem, mysql lives no more than 10-20 sec if wordpress config enabled in apache.



UPDATE: I'm deleted WP folder and download new WP without any config. If wp site is enabled, vps starts overloading, apache can't answer to requests. Without WP enabled all works smooth. I don't destroy this droplet yet because I want found the cause of the problem. Maybe this is some exploit in vps?




UPDATE2: after searching in logs and netstat I found that problem appears when:




  1. wordpress site was enabled and apache with mysql works

  2. Some request from dummy bot for rpc.php or something like this set apache connection in CLOSE_WAIT, after several request apache had many workers in CLOSE_WAIT condition, as result many workers and luck of memory (yes I should reduce max, but anyway site stopped work when all workers waiting)

  3. MySQL falling with luck of memory (it's not the problem as suggested in answer it's consequence of another problems, and upgrade VM not solve the issue)

  4. After mysql falling overload stops. (this led me to think that perhaps the database is corrupted, but after checking db not errors were found)




What reason may enter apache in CLOSE_WAIT state after dummy request for missing file?

No comments:

Post a Comment

linux - How to SSH to ec2 instance in VPC private subnet via NAT server

I have created a VPC in aws with a public subnet and a private subnet. The private subnet does not have direct access to external network. S...