linux - Apache2 quietly breaking, won't make apache2.pid

When I run
# /etc/init.d/apache2 restart
It happily claims to have started the program, but hasn't actually done so. The output goes a little something like this.

www ~ # /etc/init.d/apache2 restart
* Stopping apache2 ...
[Sun Oct 25 23:50:38 2009] [warn] Useless use of AllowOverride in line 63 of /etc/apache2/vhosts.d/30_subversion_ssl_vhost.conf.
[Sun Oct 25 23:50:38 2009] [warn] NameVirtualHost --removed--:443 has no VirtualHosts
[Sun Oct 25 23:50:38 2009] [warn] NameVirtualHost --removed1--:8443 has no VirtualHosts
httpd (no pid file) not running                                           [ ok ]
 * Starting apache2 ...
[Sun Oct 25 23:50:38 2009] [warn] Useless use of AllowOverride in line 63 of /et
c/apache2/vhosts.d/30_subversion_ssl_vhost.conf.
[Sun Oct 25 23:50:38 2009] [warn] NameVirtualHost --removed--:443 has no VirtualHosts
[Sun Oct 25 23:50:38 2009] [warn] NameVirtualHost --removed1--:8443 has no VirtualHosts

[ ok ]

As you can see, no httpd process was ever running, despite claiming to have started before, as it does once again here. Verifying by hand, /var/run/apache2/apache2.pid doesn't exist. The run/apache2/ directory wasn't even there before I added and chowned it by hand. Nothing is appended to /var/log/apache2/error_log, nor to any of the domain specific logs for any of my vhosts.

However, running with the --verbose and --debug options, it does seem to get hung up on a for loop involving the location /var/run/apache2/apache2.pid

+ i=0
+ '[' '!' -e /var/run/apache2.pid ']'
+ '[' 0 -lt 10 ']'
+ sleep 1
++ expr 0 + 1
+ i=1
+ '[' '!' -e /var/run/apache2.pid ']'
+ '[' 1 -lt 10 ']'
+ sleep 1
++ expr 1 + 1
...

I'm running apache 2.2 on the latest build of Gentoo Linux. The vhosts.d config files have been imported from a previous setup, fyi, but the httpd.conf and associated files are all brand new.

edit:
on further investigation, the apache2 init script calls

/usr/sbin/apache2 -D DEFAULT_VHOST -D INFO -D SSL -D SSL_DEFAULT_VHOST \
-D LANGUAGE -d /usr/lib/apache2 -f /etc/apache2/httpd.conf -k start

If I remove the -D SSL part, the pid creates itself fine... however, I would like ssl... any ideas?

edit2:

start() {
        checkconfig || return 1


        [ -f /var/log/apache2/ssl_scache ] && rm /var/log/apache2/ssl_scache

        ebegin "Starting ${SVCNAME}"
        ${APACHE2} ${APACHE2_OPTS} -k start

        i=0
        while [ ! -e "${PIDFILE}" ] && [ $i -lt ${TIMEOUT} ]; do
                sleep 1 && i=$(expr $i + 1)
        done


        test $i -le ${TIMEOUT}
        eend $?
}

If this provides any further insight, let me know

edit3:

if, in my ssl vhosts, i comment out SSLEngine on, apache successfully starts...

edit4:

Just a look at my apache build options

www ~ # emerge -av apache

These are the packages that would be merged, in order:


Calculating dependencies |... done!
[ebuild   R   ] www-servers/apache-2.2.11-r2  USE="sni ssl -debug -doc -ldap (-s
elinux) -static -suexec -threads" APACHE2_MODULES="actions alias asis auth_diges
t authz_host autoindex cache dav deflate dir env expires filter headers include
info log_config logio mime negotiation proxy rewrite setenvif speling status use
rdir usertrack -auth_basic -authn_alias -authn_anon -authn_dbd -authn_dbm -authn
_default -authn_file -authz_dbm -authz_default -authz_groupfile -authz_owner -au
thz_user -cern_meta -charset_lite -dav_fs -dav_lock -dbd -disk_cache -dumpio -ex
t_filter -file_cache -ident -imagemap -log_forensic -mem_cache -mime_magic -prox

y_ajp -proxy_balancer -proxy_connect -proxy_ftp -proxy_http -substitute -unique_
id -version -vhost_alias" APACHE2_MPMS="-event -itk -peruser -prefork -worker" 0
 kB

Total: 1 package (1 reinstall), Size of downloads: 0 kB

Would you like to merge these packages? [Yes/No]

And also verifying checksums

www ~ # equery check apache
[ Checking www-servers/apache-2.2.11-r2 ]
!!! /etc/apache2/vhosts.d/00_default_vhost.conf has incorrect md5sum
!!! /etc/conf.d/apache2 has incorrect md5sum
!!! /etc/apache2/vhosts.d/00_default_ssl_vhost.conf has incorrect md5sum
 * 403 out of 406 files good

edit5:

Woo! Edit 5!

Startup output after changing the corresponding line in /etc/conf.d/apache2 to
APACHE2_OPTS="-X -e debug -D DEFAULT_VHOST -D INFO -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE -D PHP5

 * Starting apache2 ...
[Tue Oct 27 23:15:19 2009] [debug] mod_so.c(246): loaded module actions_module
[Tue Oct 27 23:15:19 2009] [debug] mod_so.c(246): loaded module alias_module
[Tue Oct 27 23:15:19 2009] [debug] mod_so.c(246): loaded module asis_module

[Tue Oct 27 23:15:19 2009] [debug] mod_so.c(246): loaded module authz_host_modul
e
[Tue Oct 27 23:15:19 2009] [debug] mod_so.c(246): loaded module autoindex_module

[Tue Oct 27 23:15:19 2009] [debug] mod_so.c(246): loaded module cgi_module
[Tue Oct 27 23:15:19 2009] [debug] mod_so.c(246): loaded module deflate_module
[Tue Oct 27 23:15:19 2009] [debug] mod_so.c(246): loaded module dir_module
[Tue Oct 27 23:15:19 2009] [debug] mod_so.c(246): loaded module env_module
[Tue Oct 27 23:15:19 2009] [debug] mod_so.c(246): loaded module expires_module
[Tue Oct 27 23:15:19 2009] [debug] mod_so.c(246): loaded module filter_module

[Tue Oct 27 23:15:19 2009] [debug] mod_so.c(246): loaded module headers_module
[Tue Oct 27 23:15:19 2009] [debug] mod_so.c(246): loaded module include_module
[Tue Oct 27 23:15:19 2009] [debug] mod_so.c(246): loaded module info_module
[Tue Oct 27 23:15:19 2009] [debug] mod_so.c(246): loaded module log_config_modul
e
[Tue Oct 27 23:15:19 2009] [debug] mod_so.c(246): loaded module logio_module
[Tue Oct 27 23:15:19 2009] [debug] mod_so.c(246): loaded module mime_module
[Tue Oct 27 23:15:19 2009] [debug] mod_so.c(246): loaded module negotiation_modu
le
[Tue Oct 27 23:15:19 2009] [debug] mod_so.c(246): loaded module rewrite_module

[Tue Oct 27 23:15:19 2009] [debug] mod_so.c(246): loaded module setenvif_module
[Tue Oct 27 23:15:19 2009] [debug] mod_so.c(246): loaded module speling_module
[Tue Oct 27 23:15:19 2009] [debug] mod_so.c(246): loaded module ssl_module
[Tue Oct 27 23:15:19 2009] [debug] mod_so.c(246): loaded module usertrack_module

[Tue Oct 27 23:15:19 2009] [debug] mod_so.c(246): loaded module php5_mod  [ ok ]

No hints yet. Also, not sure how to verify ssl config, if that is, indeed, the problem

Answer

Hmm, I am pretty suspicious of the config file

/etc/apache2/vhosts.d/30_subversion_ssl_vhost.conf

When you remove the '-D SSL' you will cause all parts of the configuration files that are enclosed in ... to be skipped. The default SSL vhost file on my Gentoo box is wrapped in that tag so I wonder if, by removing the '-D SSL', you are preventing the config in 30_subversion_ssl_vhost.conf from being run at all and if that is what is allowing Apache to start.

If you temporarily remove the file 30_subversion_ssl_vhost.conf from /etc/apache2/vhosts.d does Apache run? Are there any other SSL related vhost.conf files in vhosts.d? My reasonably fresh/unused Apache install's vhosts.d directory looks like this:

# pwd && ls

/etc/apache2/vhosts.d
00_default_ssl_vhost.conf  00_default_vhost.conf  default_vhost.include

edit 1:

So much for that theory :) I am now wondering if the problem is with the Apache SSL setup itself. I apologize if I am covering ground you have already checked but I am wondering if you could do the following to help verify your Apache install.

On my Apache install with working SSL the use flags are as follows:

# emerge -av apache

These are the packages that would be merged, in order:

Calculating dependencies... done!
[ebuild     U ] www-servers/apache-2.2.11-r2 [2.2.11] USE="ssl -debug -doc -ldap (-selinux) -sni -static -suexec -threads" APACHE2_MODULES="actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias -asis -auth_digest -authn_dbd -cern_meta -charset_lite -dbd -dumpio -ident -imagemap -log_forensic -proxy -proxy_ajp -proxy_balancer -proxy_connect -proxy_ftp -proxy_http -substitute -version" APACHE2_MPMS="-event -itk -peruser -prefork -worker" 64 kB

In particular do you have the 'ssl' USE flag set?

Also, could you use equery to verify the integrity of your apache2 install? If you do not have the equery command you can install it by running 'emerge -av gentoolkit'. The following command should verify the integrity of your apache install:

equery check apache

On my server the above command gives the following output:

[ Checking www-servers/apache-2.2.11 ]
!!! /etc/apache2/vhosts.d/00_default_ssl_vhost.conf has wrong mtime (is 1256620928, should be 1246793824)
!!! /etc/apache2/modules.d/00_default_settings.conf has wrong mtime (is 1246796304, should be 1246793824)

!!! /etc/conf.d/apache2 has incorrect md5sum
 * 429 out of 432 files good

edit 2:

Well the install looks good to me, so much for theory 2. I am wondering if we can coax Apache into giving some more information on startup. In /etc/conf.d/apache2 if you change your APACHE2_OPTS line from:

APACHE2_OPTS="-D DEFAULT_VHOST -D INFO -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE"

to

APACHE2_OPTS="-X -e debug -D DEFAULT_VHOST -D INFO -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE"

and then start Apache (/etc/init.d/apache2 start) the daemon should stay in the foreground (the -X flag) and output debuging messages as it starts (the -e debug option). Maybe this will give a clue as to why it is dying on startup.