Out of memory kill on Centos 6

For the last few days I am having regular out of memory problems on my Centos 6 server.

Aug  1 03:40:42 myserver xinetd[2304]: START: smtp pid=11222 from=87.109.36.82

Aug  1 03:40:46 myserver xinetd[2304]: EXIT: smtp status=1 pid=11222 duration=4(sec)
Aug  1 03:41:08 myserver xinetd[2304]: START: smtp pid=11224 from=189.174.169.28
Aug  1 03:41:19 myserver xinetd[2304]: EXIT: smtp status=0 pid=11224 duration=11(sec)
Aug  1 03:44:40 myserver xinetd[2304]: START: smtp pid=11240 from=109.230.244.236
Aug  1 03:44:40 myserver xinetd[2304]: EXIT: smtp status=0 pid=11240 duration=0(sec)
Aug  1 03:44:41 myserver xinetd[2304]: START: smtp pid=11241 from=109.230.244.236
Aug  1 03:44:41 myserver xinetd[2304]: EXIT: smtp status=0 pid=11241 duration=0(sec)
Aug  1 03:44:41 myserver xinetd[2304]: START: smtp pid=11242 from=109.230.244.236
Aug  1 03:44:41 myserver xinetd[2304]: EXIT: smtp status=0 pid=11242 duration=0(sec)
Aug  1 03:44:51 myserver xinetd[2304]: START: smtp pid=11243 from=113.22.231.187

Aug  1 03:44:56 myserver xinetd[2304]: EXIT: smtp status=1 pid=11243 duration=5(sec)
Aug  1 03:51:36 myserver kernel: postmaster invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
Aug  1 03:51:36 myserver kernel:
Aug  1 03:51:36 myserver kernel: Call Trace:
Aug  1 03:51:36 myserver kernel:  [] out_of_memory+0x8e/0x2f5
Aug  1 03:51:36 myserver kernel:  [] __alloc_pages+0x245/0x2ce
Aug  1 03:51:36 myserver kernel:  [] __do_page_cache_readahead+0x95/0x1d9
Aug  1 03:51:36 myserver kernel:  [] __wait_on_bit_lock+0x5b/0x66
Aug  1 03:51:36 myserver kernel:  [] __lock_page+0x5e/0x64
Aug  1 03:51:36 myserver kernel:  [] filemap_nopage+0x148/0x322

Aug  1 03:51:36 myserver kernel:  [] __handle_mm_fault+0x1f8/0xe23
Aug  1 03:51:36 myserver kernel:  [] do_page_fault+0x4cb/0x830
Aug  1 03:51:36 myserver kernel:  [] free_pages_and_swap_cache+0x73/0x8f
Aug  1 03:51:36 myserver kernel:  [] error_exit+0x0/0x84
Aug  1 03:51:36 myserver kernel:
Aug  1 03:51:36 myserver kernel: Mem-info:
Aug  1 03:51:36 myserver kernel: Node 0 DMA per-cpu:
Aug  1 03:51:36 myserver kernel: cpu 0 hot: high 0, batch 1 used:0
Aug  1 03:51:36 myserver kernel: cpu 0 cold: high 0, batch 1 used:0
Aug  1 03:51:36 myserver kernel: cpu 1 hot: high 0, batch 1 used:0

Another example:

Aug  1 03:55:17 myserver xinetd[2304]: EXIT: smtp status=0 pid=11289 duration=36(sec)
Aug  1 03:55:21 myserver xinetd[2304]: START: smtp pid=11293 from=109.230.244.236
Aug  1 03:59:28 myserver kernel: qmail-remote invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
Aug  1 03:59:28 myserver kernel:
Aug  1 03:59:28 myserver kernel: Call Trace:
Aug  1 03:59:28 myserver kernel:  [] out_of_memory+0x8e/0x2f5

Aug  1 03:59:28 myserver kernel:  [] __alloc_pages+0x245/0x2ce

The regular server memory use is about 30-40%. The server has been very stable and "trouble-free" for years...What could cause that memory spike? Does it have something to do with xinetd?

Over the last few days I updated clamav, installed Comodo mod_security riules and maldet. I also installed clamd as a new application.

Any suggestions? How can I narrow the problem?

Thanks

Answer

As mentioned in above reply..checking updated package is the first debugging step as based on the log trace we can see kernel is doing it job i.e out of memory function is called when all the system memory is occupied including(RAM+Swap) and system will not resume normal operation till memory is freed.

Other option you have is to configure kdump along with vm.panic_on_oom = 1(/etc/sysctl.conf),this will generate vmcore.You can find more info about it here http://people.redhat.com/anderson/crash_whitepaper/