tcp - FreeBSD slow transfers - RFC 1323 scaling issue?

I think I may be having an issue with window scaling (RFC 1323) and am hoping that someone can enlighten me on what's going on.

• Server: FreeBSD 9, apache22, serving a static 100MB zip file. 192.168.18.30


• Client: Mac OS X 10.6, Firefox 192.168.17.47



• Network: Only a switch between them - the subnet is 192.168.16/22 (In this test, I also have dummynet filtering simulating an 80ms ping time on all IP traffic. I've seen nearly identical traces with a "real" setup, with real internet traffic/latency also)

Questions:

• Does this look normal?


• Is packet #2 specifying a window size of 65535 and a scale of 512?


• Is packet #5 then shrinking the window size so it can use the 512 scale and still keep the overall calculated window size near 64K?


• Why is the window scale so high?

Here are the first 6 packets from wireshark. For packets 5 and 6 I've included the details showing the window size and scaling factor being used for the data transfer.

No. Time Source Destination Protocol Length Info

108 6.699922 192.168.17.47 192.168.18.30 TCP 78 49190 > http [SYN] Seq=0 Win=65535 Len=0 MSS=1460 WS=8 TSval=945617489 TSecr=0 SACK_PERM=1

115 6.781971 192.168.18.30 192.168.17.47 TCP 74 http > 49190 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1460 WS=512 SACK_PERM=1 TSval=2617517338 TSecr=945617489


116 6.782218 192.168.17.47 192.168.18.30 TCP 66 49190 > http [ACK] Seq=1 Ack=1 Win=524280 Len=0 TSval=945617490 TSecr=2617517338

117 6.782220 192.168.17.47 192.168.18.30 HTTP 490 GET /utils/speedtest/large.file.zip HTTP/1.1

118 6.867070 192.168.18.30 192.168.17.47 TCP 375 [TCP segment of a reassembled PDU]

Details:

Transmission Control Protocol, Src Port: http (80), Dst Port: 49190 (49190), Seq: 1, Ack: 425, Len: 309

Source port: http (80)
Destination port: 49190 (49190)
[Stream index: 4]
Sequence number: 1 (relative sequence number)
[Next sequence number: 310 (relative sequence number)]
Acknowledgement number: 425 (relative ack number)
Header length: 32 bytes
Flags: 0x018 (PSH, ACK)
Window size value: 130
[Calculated window size: 66560]

[Window size scaling factor: 512]
Checksum: 0xd182 [validation disabled]
Options: (12 bytes)
No-Operation (NOP)
No-Operation (NOP)
Timestamps: TSval 2617517423, TSecr 945617490
[SEQ/ACK analysis]
TCP segment data (309 bytes)

Answer

I got word from the sysadmin team that this issue was being caused by some issue with the VMWare net driver not respecting / playing nice the sysctl tunables. Throughput with the same setup on physical hardware has throughput with a reasonable percentage for the pipe rather than the 1/10th or less we saw with VMware.