virtualization - XenServer Performance

we've got 2 x HP DL360 G5s with Quad Quad Xeons 2.6GHz and 32GB of memory each running XenServer 5.5 and they access an OpenFiler box (with 8 x 320GB SAS 10K drives) via copper CAT5 (1GB) for the storage.

We've used this setup for testing a lot of stuff which has worked out perfectly, but now we are moving to use this setup in production and are experiencing performance issues. There are currently 27 VMs split across the two servers which are all in use (albeit not doing a lot of work) but they seem "slow", especially our employee thin clients - they always complain logging in times and accessing files via the network are slow.

Personally, I think it's a throughput issue and we should go SCSI or FC for our storage but I need some evidence to back my theory up and I'm quite new to Xen (it was setup by a previous employee).

My questions: from the info I've gave would it be possible that the storage box is overloaded, trying to squeeze too much over that one cable;? how do I monitor network access in real-time from the XenServers themselves?

Thanks :-)

Answer

I have seen this issue many times.
I really love xenserver, however, its like an unpolished gem...

you should check with ifconfig -a (on dom0, xenserver console )
and look for dropped packets

you can use:
ifconfig -a | grep dropped | awk {'print $3'} | grep -v ":0"

if you see dropped packets , you should do:

1. On the Virtual Machines, Click Start, click Run, type regedit, and then click OK.


2. Locate and then click the following registry subkey:
   HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters


3. In the right pane, make sure that the DisableTaskOffload registry entry exists. If this entry does not exist, follow these steps to add the entry:
   a. On the Edit menu, point to New, and then click DWORD Value.
   b. Type DisableTaskOffload, and then press ENTER.



4. Click DisableTaskOffload.


5. On the Edit menu, click Modify.


6. Type 1 in the Value data box, and then press ENTER.


7. Exit Registry Editor.


8. Restart all Virtual Machines

and on server xenserver console:

Get the UUID of the physical interface:

xe pif-list host-name-label=XEN1

Disable checksum on the interfaces:

xe pif-param-set other-config:ethtool-tx="off" uuid=3281b044-2a93-2f1b-e8e1-eaf0faccbd1f;
xe pif-param-set other-config:ethtool-rx="off" uuid=3281b044-2a93-2f1b-e8e1-eaf0faccbd1f