Wednesday, October 5, 2016

ZFS SLOG IOPS closer to random IOPS than sequential

I have a Solaris 11.3 install with a ZFS pool consisting of a 5-vdev-stripe each consisting of three 10K SAS disks. I've also configured a SLOG containing mirrored vdevs. I've set my ZFS tuning parameters to the following:




zil_slog_limit: 1073741824
zfs_txg_timeout:256
zfs_vdev_max_pending: 256
zfs_immediate_write_sz: 67108864


I'm experiencing much slower write performance than expected when writing to a file system with async=always (I am trying to determine the best performance I can expect when this file system is mounted via NFS with sync on for VM disk images). When I run



time dd if=/dev/urandom of=testfile bs=512 count=10000



I get around 100 IOPS per vdev (2-disk mirrored 10K SAS disks) in the SLOG (so about 100 seconds to run with one vdev in the SLOG and 50 seconds with two in the SLOG). I don't have extra drives to try in the array, but this behaviour for one vs. two devices suggests the SLOG vdevs are working. I have also used zpool iostat -v 5 to verify they are the only devices that get writes to them while I am running the test (aside from the data disks when the SLOG gets flushed). The writes per second from zpool iostat approximately matches the IOPS I've calculated I'm getting from timing dd.



I am under the impression that SLOGs are supposed to be written almost entirely sequentially; 100 IOPS is what I would expect for random writes.



Edit: I tried a similar thing with a spare machine running FreeNAS. The machine has two drives in a mirror + an unmirrored SLOG. IOPS is random-I/O slow, not sequential-slow, on the pool if using a HDD for the SLOG and >10k IOPS using an SSD.



Is there something missing in my understanding and/or some parameters I need to change? Thanks in advance!

No comments:

Post a Comment

linux - How to SSH to ec2 instance in VPC private subnet via NAT server

I have created a VPC in aws with a public subnet and a private subnet. The private subnet does not have direct access to external network. S...