Friday, April 24, 2015

compression - compress big files on linux



I got a several of big files using command



mysql> select * into outfile ...



for example, dump_news.sql, dump_weibo.sql, dump_report.sql ...



The total size is about 7.6G, I use command



tar -zcvf dump.tar.gz dump_*


on linux, to compress the above files, however it takes about 4 hours. That's to long. I tested the tool of bzip2, it takes a longer time, but the size of the compressed file is smaller. So i want to know if there is a tool for me to finish the compressing more quickly.



Answer



A few months ago I had to compress my entire home directory, so I made a bash script that ran tar in multiple processes and kept track of the number of processes so that there were always 6 going at once (I have 4 cores, so the 6 threads was so that it could compress while waiting for the iowait since I was putting it over usb onto my hard drive).



My script is here: https://gist.github.com/3989700



If change the output folder to your desired output destination, put all the sql files in the same folder as the script, give it executable permissions, and run it from the terminal, it should compress your files.



You may also want to change the number of threads it runs to suit your own processor.


No comments:

Post a Comment

linux - How to SSH to ec2 instance in VPC private subnet via NAT server

I have created a VPC in aws with a public subnet and a private subnet. The private subnet does not have direct access to external network. S...