Let Ideas Compete via Compfight

We’re continuing our Features In Depth series of posts with an overview of usenet header compression. We’ve long offered header compression capabilities but we’ve never really advertised it or talked much about it.

Understanding the Basics

Header compression is a method for vastly improving the download time for usenet headers. The are hundreds of thousands of newsgroups available on usenet, and some of those groups have millions of articles – which means millions of headers. To be able to access those articles you’ll have to download the millions of headers, which may end up being several gigabytes of data. It takes a long time to download and consumes a lot of your available bandwidth.

How It Works

So how exactly does header compression work? The are several data fields, such as poster and subject that are common to all or many headers. When redundant content is detected, it is removed and only unique data is needed by the newsreader. This means much less header data is needed to represent the entirety of the articles available.

The Results of Header Compression

Using header compression can bring benefits of 6X – 10X speed improvements. For example, if you are downloading headers for a group that has 10 gigabytes of headers you can expect header compression to reduce that to anywhere from 1 gigabyte to 1.6 gigabytes. That’s quite a lot of bandwidth saved, and download time as well.

One Last Thing

Usenet header compression will only work in newsreaders that support the feature. Also, you can’t compress articles, only headers.