Monday, May 7, 2018

dd using all memory and swap



I have a dedicated Centos 6.9 linux server with a company that crashed due to hardware failure and has been restored with the old HDD's on a usb caddy. I am trying to move lvm partitions over to the new internal hard disks. I have created a new volume group and same size logical volumes. The partitions are currently being used by kvm virtual machines. I have taken a snaphot of the lv. Then I am using something like the following to copy the partitions.



dd if=/dev/OldVolGroup/lv__snap of=/dev/NewVolGroup/lv bs=1M


This seems to work well with the following speed:




8192+0 records in
8192+0 records out
8589934592 bytes (8.6 GB) copied, 264.61 s, 32.5 MB/s


However, the ram and then swap slowly starts to get swallowed up. The box has 24GB of RAM and around 4GB spare when I start. It has a 2GB lv for swap.



I was able to turn some kvm machines off to free more memory, around 7-8GB and it just managed to finish the 8GB transfer without using all resources.



Can anyone tell me why it uses up so much memory and then swap? Is this normal?



Answer



It's because dd reads into cache before writing. If you have top running you'll see that it's cache that is being used up before memory. All you can really do is try a larger block size. It's not a good thing to be doing on a running server but obviously it's what you're working with.


No comments:

Post a Comment

linux - How to SSH to ec2 instance in VPC private subnet via NAT server

I have created a VPC in aws with a public subnet and a private subnet. The private subnet does not have direct access to external network. S...