apache 2.2 - TCP 443 Connection goes into server but never reaches Apache2 (Passenger)

We have some weird problems, when external servers try to connect to our Website on Port 443(TCP) which is an Apache2+Passenger and a Rails Application.
The problem only happens rarely. I started to investigate that by enabling iptables logging using iptables -I INPUT -m state --state NEW,INVALID -j LOG --log-prefix "iptables [INPUT]: " (did this also for FORWARD (normally not used) and OUTPUT as well).

Now every new connection is logged into syslog. Now ive seen normal Connections on that syslog and afterwars the connection to the Vhost (normal working request) but sometimes (in about 1 of 100 connections) i only see the new connection in the iptables log but the connection seems to not reach the apache2 (or the passenger instances) (No apache Log entry and no result from the connection). When doing this with curl, we get a timeout or an empty string back. We just sent over a debug-enabled curl request code snippet for enhanced logging what the curl error looks like.

We`re using Debian Squeeze (stable) running in a VMWare ESXi environment with bridged ethernet with E1000 NICs (virtualized). But as the packets reaching the VM-OS itself i dont think that anything of that causes the issue.

I also checked ip_conntrack which has about 40 to 50 connections and Squeezes Limit is about 65k, so thats not the problem. Also we dont have any Dropped Packages on the NIC`s itself.

Kernel-Version: Linux servername 2.6.32-5-amd64 #1 SMP Sun Sep 23 10:07:46 UTC 2012 x86_64 GNU/Linux
Apache2: Server version: Apache/2.2.16 (Debian) - Server built: Nov 30 2012 08:58:38
Passenger: Phusion Passenger version 3.0.12
Ruby: RVM 1.17.9 + Ruby 1.9.3-p362

iptables -L

Chain INPUT (policy ACCEPT)
target prot opt source destination
LOG all -- anywhere anywhere state INVALID,NEW LOG level warning prefix `iptables [INPUT]: '

Chain FORWARD (policy ACCEPT)
target prot opt source destination
LOG all -- anywhere anywhere state INVALID,NEW LOG level warning prefix `iptables [FORWARD]: '

Chain OUTPUT (policy ACCEPT)

target prot opt source destination
LOG all -- anywhere anywhere state INVALID,NEW LOG level warning prefix iptables [OUTPUT]: '

Answer

I have no really 'Answer' on that, but maybe a way you could trace it a little more.
The problem seems to happen in the kernel space, right after the iptables chaining or in the user space, when or before apache is accepting the connection.

The kernel can drop packets itself AFTER the nic and iptables, because of the load that packet Capturing costs. Did you ever build a test environment without iptables and a defined number of accesses that can be counted in the apache log?

If you have some C experience you should give libpcap a try, which is a wonderful lib to do bad and good things. Here is a nice how-to-start article on this: http://recursos.aldabaknocking.com/libpcapHakin9LuisMartinGarcia.pdf

This might show you differences in the packets going thru and that which will be dropped.