I wasn't aware of pbzip2, which is very useful for multicore machines.
PBZIP2 is a parallel implementation of the bzip2 block-sorting file compressor that uses pthreads and achieves near-linear speedup on SMP machines. The output of this version is fully compatible with bzip2 v1.0.2 or newer (ie: anything compressed with pbzip2 can be decompressed with bzip2). PBZIP2 should work on any system that has a pthreads compatible C++ compiler (such as gcc). It has been tested on: Linux, Windows (cygwin & MinGW), Solaris, Tru64/OSF1, HP-UX, and Irix.
This is my first test (With no I/O, files were cached in memory). Here I am compressing a GIT repository with some files already compressed.
$ time bzip2 linux-2.6.20.tar real 1m28.765s user 1m27.397s sys 0m1.180s
$ time bzip2 -d linux-2.6.20.tar.bz2 real 0m46.583s user 0m45.019s sys 0m1.236s $ time pbzip2 -p2 linux-2.6.20.tar real 0m53.963s user 1m44.963s sys 0m1.944s
About 40% of wall clock time is saved with this compression.
DiscussionLast update: 2008-12-22 (Rev 15017)