I am using godaddy's dedicate server to host my web application, it's centOS 5(I have no idea why they are still using this.), I installed mysql in that server, everything was running OK since one day I got an report saying that the web application cannot connect to mysql, I checked and found out that mysql crashed, after restarting mysql, everything worked again. But the weird thing is that it will crash randomly, sometimes once in a month, sometimes once or twice in couple months. Please help me figure out what the problem is. The following is the most recent crash log from mysql. Thank you very much for your time.
121015 08:59:35 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
121015 8:59:36 [Note] Plugin 'FEDERATED' is disabled.
121015 8:59:36 InnoDB: The InnoDB memory heap is disabled
121015 8:59:36 InnoDB: Mutexes and rw_locks use InnoDB's own implementation
121015 8:59:36 InnoDB: Compressed tables use zlib 1.2.3
121015 8:59:36 InnoDB: Using Linux native AIO
121015 8:59:36 InnoDB: Initializing buffer pool, size = 128.0M
121015 8:59:36 InnoDB: Completed initialization of buffer pool
121015 8:59:36 InnoDB: highest supported file format is Barracuda.
121015 8:59:38 InnoDB: Waiting for the background threads to start
121015 8:59:39 InnoDB: 1.1.8 started; log sequence number 3584614
121015 8:59:39 [Note] Server hostname (bind-address): '0.0.0.0'; port: 3306
121015 8:59:39 [Note] - '0.0.0.0' resolves to '0.0.0.0';
121015 8:59:39 [Note] Server socket created on IP: '0.0.0.0'.
121015 8:59:40 [Note] Event Scheduler: Loaded 2 events
121015 8:59:40 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.27' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Server (GPL) by Remi
121019 23:42:58 [Warning] IP address '118.186.38.55' could not be resolved: Name or service not known
121024 05:58:39 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
121024 5:58:41 [Note] Plugin 'FEDERATED' is disabled.
121024 5:58:41 InnoDB: The InnoDB memory heap is disabled
121024 5:58:41 InnoDB: Mutexes and rw_locks use InnoDB's own implementation
121024 5:58:41 InnoDB: Compressed tables use zlib 1.2.3
121024 5:58:41 InnoDB: Using Linux native AIO
121024 5:58:41 InnoDB: Initializing buffer pool, size = 128.0M
121024 5:58:41 InnoDB: Completed initialization of buffer pool
121024 5:58:41 InnoDB: highest supported file format is Barracuda.
InnoDB: The log sequence number in ibdata files does not match
nnoDB: the log sequence number in the ib_logfiles!
121024 5:58:41 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
121024 5:58:43 InnoDB: Waiting for the background threads to start
121024 5:58:44 InnoDB: 1.1.8 started; log sequence number 3584963
121024 5:58:44 [Note] Server hostname (bind-address): '0.0.0.0'; port: 3306
121024 5:58:44 [Note] - '0.0.0.0' resolves to '0.0.0.0';
121024 5:58:44 [Note] Server socket created on IP: '0.0.0.0'.
121024 5:58:45 [Note] Event Scheduler: Loaded 2 events
121024 5:58:45 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.27' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Server (GPL) by Remi
121024 6:06:32 [Note] /usr/libexec/mysqld: Normal shutdown
121024 6:06:32 [Note] Event Scheduler: Purging the queue. 2 events
121024 6:06:34 [Warning] /usr/libexec/mysqld: Forcing close of thread 3 user: 'root'
121024 6:06:34 [Warning] /usr/libexec/mysqld: Forcing close of thread 2 user: 'root'
121024 6:06:34 InnoDB: Starting shutdown...
121024 6:06:37 InnoDB: Shutdown completed; log sequence number 3584963
121024 6:06:37 [Note] /usr/libexec/mysqld: Shutdown complete
121024 06:06:37 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
121024 06:08:23 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
121024 6:08:23 [Note] Plugin 'FEDERATED' is disabled.
121024 6:08:23 InnoDB: The InnoDB memory heap is disabled
121024 6:08:23 InnoDB: Mutexes and rw_locks use InnoDB's own implementation
121024 6:08:23 InnoDB: Compressed tables use zlib 1.2.3
121024 6:08:23 InnoDB: Using Linux native AIO
121024 6:08:23 InnoDB: Initializing buffer pool, size = 128.0M
121024 6:08:23 InnoDB: Completed initialization of buffer pool
121024 6:08:23 InnoDB: highest supported file format is Barracuda.
121024 6:08:23 InnoDB: Waiting for the background threads to start
121024 6:08:24 InnoDB: 1.1.8 started; log sequence number 3584963
121024 6:08:24 [Note] Server hostname (bind-address): '0.0.0.0'; port: 3306
121024 6:08:24 [Note] - '0.0.0.0' resolves to '0.0.0.0';
121024 6:08:24 [Note] Server socket created on IP: '0.0.0.0'.
121024 6:08:24 [Note] Event Scheduler: Loaded 2 events
121024 6:08:24 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.27' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Server (GPL) by Remi
121026 10:49:09 [Warning] IP address '69.46.72.78' could not be resolved: Temporary failure in name resolution
121027 7:00:57 [Warning] IP address '119.1.96.27' could not be resolved: Name or service not known
121028 1:42:49 [Warning] IP address '211.152.54.123' could not be resolved: Name or service not known
121029 11:48:07 [Warning] IP address '58.16.63.214' could not be resolved: Name or service not known
121031 10:52:46 [Note] Event Scheduler: scheduler thread started with id 99
121105 14:04:01 [Warning] IP address '61.134.25.4' could not be resolved: Name or service not known
121107 09:18:43 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
121107 9:18:44 [Note] Plugin 'FEDERATED' is disabled.
121107 9:18:44 InnoDB: The InnoDB memory heap is disabled
121107 9:18:44 InnoDB: Mutexes and rw_locks use InnoDB's own implementation
121107 9:18:44 InnoDB: Compressed tables use zlib 1.2.3
121107 9:18:44 InnoDB: Using Linux native AIO
121107 9:18:44 InnoDB: Initializing buffer pool, size = 128.0M
121107 9:18:44 InnoDB: Completed initialization of buffer pool
121107 9:18:44 InnoDB: highest supported file format is Barracuda.
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
121107 9:18:44 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
121107 9:18:48 InnoDB: Waiting for the background threads to start
121107 9:18:49 InnoDB: 1.1.8 started; log sequence number 3609300
121107 9:18:49 [Note] Server hostname (bind-address): '0.0.0.0'; port: 3306
121107 9:18:49 [Note] - '0.0.0.0' resolves to '0.0.0.0';
121107 9:18:49 [Note] Server socket created on IP: '0.0.0.0'.
121107 9:18:50 [Note] Event Scheduler: Loaded 2 events
121107 9:18:50 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.27' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Server (GPL) by Remi
121107 9:23:34 [Note] Event Scheduler: scheduler thread started with id 5
Answer
It looks like there is an issues with InnoDB corruption. If you can get it up and running, you can try running "mysqlcheck --all-databases --quick --fast --auto-repair --force ". If that will not work, you can try to dump the content and then import everything again as this will re-create the ibdata file. If you cannot run either of these before the server restarts again, you may need to look into using the Percona Data Recovery Tool for InnoDB.
No comments:
Post a Comment