NOTE: A newer version of this page is at Linux compression from the command line, Any updates will be made to that page

Compression Under Linux
Why this article
There are many compress decompress utilities in Linux, But most people run linux from the command line remotely (Welcome microsoft who are now doing the same thing with Windows Sever 2008).

The point is, we all know how to use TAR to make tarballs, we all know how to bzip2 -z -k myfile to compress that file, but i have 8CPUs on one of my servers and i really really want to use them all when compressing a file... why wait so long

So, to utilize a multi processor computer, i need a utility that is multi threaded, multi threading is all about using more processes to accomplish 1 task (To state things simply), there are such utilities for zip, 7zip and others, but i use bzip2 and here is how you do it !

You get a program called pbzip2 and run a command like

pbzip2 -p8v somefile

Where 8 is the number of processors, so if your computer is core 2 duo you replace the 8 with a 2, and if it is core 2 quad you replace the 8 with a 4, my computer is dual xeon quad and that gives the 8 is you are wondering.

The following is the help for the utility, just like bzip2 k means keep original file, d is decompress v is tell me what is happening and all the other things are shown here

-----------------

Usage: pbzip2 [-1 .. -9] [-b#cdfklp#qrtV] <filename> <filename2> <filenameN>
-b# : where # is the file block size in 100k (default 9 = 900k)
-c : output to standard out (stdout)
-d : decompress file
-f : force, overwrite existing output file
-k : keep input file, don't delete
-l : load average determines max number processors to use
-p# : where # is the number of processors (default: autodetect)
-r : read entire input file into RAM and split between processors
-t : test compressed file integrity
-v : verbose mode
-V : display version info for pbzip2 then exit
-1 .. -9 : set BWT block size to 100k .. 900k (default 900k)

Example: pbzip2 -b15vk myfile.tar
Example: pbzip2 -p4 -r -5 myfile.tar second*.txt
Example: pbzip2 -d myfile.tar.bz2

------------------------

See how to 7ZIP under linux click here