I got a several of big files using command
mysql> select * into outfile ...
for example, dump_news.sql, dump_weibo.sql, dump_report.sql ...
The total size is about 7.6G, I use command
tar -zcvf dump.tar.gz dump_*
on linux, to compress the above files, however it takes about 4 hours. That's to long. I tested the tool of bzip2, it takes a longer time, but the size of the compressed file is smaller. So i want to know if there is a tool for me to finish the compressing more quickly.
Answer
A few months ago I had to compress my entire home directory, so I made a bash script that ran tar in multiple processes and kept track of the number of processes so that there were always 6 going at once (I have 4 cores, so the 6 threads was so that it could compress while waiting for the iowait since I was putting it over usb onto my hard drive).
My script is here: https://gist.github.com/3989700
If change the output folder to your desired output destination, put all the sql files in the same folder as the script, give it executable permissions, and run it from the terminal, it should compress your files.
You may also want to change the number of threads it runs to suit your own processor.
No comments:
Post a Comment