virtualization - Explain this strange environment-related Ruby performance problem

We're diagnosing Ruby performance problems on our application servers which
we've managed to reduce to a simple test case.

We compare performance on a machine in our development cluster to a machine in
our production data centre.

We used this simple Ruby oneliner:

5000000.times { a = []; a << 1; a.length }

And we benchmarked it as being consistently 55% slower on the production machine.

Things it obviously could be and why we think it's not:

1. Different software - dev and production machines are installed from the same ubuntu os, ubuntu installation scripts, package repositories, and we use puppet to keep the configurations consistent.


2. Different hardware - possibly, but see below.


3. Different load - neither dev nor production machines are significantly loaded, and again see below.

Why don't we think it's load or hardware?

First, they have similar loads and hardware configurations.
Second, we wrote a python test script:

n = 10000000
while n > 1:
  n = n - 1

  a = []
  a.append(4)
  len(a)

and this is consistently 10% faster on production than development, which is what we would expect. If the problem was load or hardware, wouldn't Python be slower on production as well?

Briefly both machines are virtualized using ESXi

• development vm has 4GB RAM and hosted on a machine with dual quad-core AMD Opteron 2376 @ 2.294Ghz 32GB providing one virtual core to the vm




• production vm has 4GB RAM and hosted on a machine with dual quad-core AMD Opteron 2354 @ 2.211Ghz 32GB providing four virtual cores to the vm (update: we have now tried with one virtual core on all the vms and it made no difference)

The operating system is Ubuntu Hardy 64bit. Our Ruby interpreter is:

ruby 1.8.6 (2008-08-11 patchlevel 287) [x86_64-linux]

and our python interpreter is

Python 2.5.2 (r252:60911, Jul 31 2008, 17:31:22)

NB. We have also tried this with Ruby Enterprise Edition, and the results are the same.