NZB Files & Indexing

How people find things on Usenet without downloading millions of headers themselves. This section covers the NZB file format, indexers and their APIs, header downloading, subject parsing, and the difference between an indexer's retention and a provider's. If a download fails before it starts, the cause is usually defined somewhere in here.

28 terms in this category.

NZB: An XML file listing the Message-IDs of every article that makes up a file set, along with the newsgroups those articles live in and each segment's size and number. Devised around 2002 by an indexing site of the era, it is a shopping list rather than content: a newsreader reads it, requests the named articles from a provider, and reassembles the original files.
NZB File Format: Defines the XML structure NZB files follow: a root <nzb> element containing one <file> element per posted file, each holding a <groups> list and a <segments> list of Message-IDs with byte sizes and part numbers. An optional <head> section carries <meta> tags for title, category and password. The format is simple enough that scripts commonly generate or edit NZBs with any XML library.
Indexer: A site that scans newsgroup headers server-side, parses subject lines to work out what each posting is, and publishes the results as searchable entries with downloadable NZB files. Indexers spare users from pulling gigabytes of headers themselves. Most expose both a web search page and an API, meter usage in searches and downloads, and cover only the groups and time range their own database reaches.
Public Indexer: An indexer open to anyone who registers, usually with a free tier capped at a few dozen searches and downloads per day and a paid tier lifting those limits. Open sites carry heavy traffic, so coverage, speed and uptime vary considerably between them. Registration typically requires only an email address, and the account's API key authenticates any query made against it.
Header Downloading: The act of pulling the subject, poster, date and size lines for articles in a group rather than the article bodies. Newsreaders do this with the XOVER or OVER command, often thousands of records per request. A busy binary group can produce several gigabytes of headers, which is why header downloading has largely given way to querying an indexer for NZB files instead.
Header Index: The searchable store built from downloaded headers, mapping subject lines to the article numbers and Message-IDs that make up each posting. Indexers keep one server-side and share it with every user; desktop newsreaders build a private copy on disk. Either way the index is what turns a raw group of millions of articles into a list of named files that can be searched.
Local Header Index: A header index stored on the user's own machine rather than on an indexer. Desktop newsreaders still work this way: they download headers for a group, cache them in a database file, and search offline. The approach needs no third-party site and no API key, but consumes gigabytes of disk for large binary groups and goes stale unless headers are refreshed regularly.
API Key: A long random string tied to an indexer account that authenticates API requests in place of a username and password. It appears as the apikey parameter in every query, so it travels in URLs and server logs and should be treated as a credential. Regenerating the key on the indexer's profile page immediately invalidates the old one and requires updating every tool that stored it.
API Hits: One of the two quotas indexers meter, counting searches rather than downloads. Every query an automation tool sends consumes a hit, so a setup polling a dozen indexers every fifteen minutes burns through an allowance quickly. Free accounts are often limited to a few dozen or a few hundred hits a day; exceeding the cap returns an error until the counter resets.
Grab: The second quota indexers meter: downloading an NZB file, as opposed to searching for one. Grabs are counted per NZB retrieved, whether by clicking a link on the site or by an automation tool fetching it over the API. Daily grab limits are usually much lower than search limits, since each grab represents a file set someone intends to actually download.
Indexer Retention: How far back an indexer's own database reaches, measured in days from the oldest entry it still lists. This is separate from a provider's article retention and usually much shorter: a provider may hold articles for over 5,000 days while an indexer only indexed the last 2,000. An entry missing from the indexer can still exist on the server, and vice versa.
Indexer Category: The classification an indexer assigns each entry, drawn from a numeric scheme common across most sites: 2000 for movies, 5000 for television, 3000 for audio, 7000 for other, with sub-codes beneath each. Searches filter on these numbers rather than on text, so a miscategorized entry can be invisible to a query that asks for only one range.
Subject Parsing: How an indexer turns raw Usenet subject lines into usable entries, applying regular expressions to strip part counters, yEnc markers and filenames until a release name remains. Posters vary their subject formats constantly and some deliberately obfuscate them, so parsing rules need frequent updating. A failed parse leaves a posting either unlisted or listed under a garbled name that no search will match.
Release Name: The standardized string identifying a release, typically combining title, year or season and episode, resolution, source and codec, joined by periods. Indexers reconstruct it from subject lines and parse it again to work out what a posting contains. Because the format is conventional rather than specified, small variations regularly defeat both indexer parsers and the matching rules built on top of them.
Indexer RSS Feed: A continuously updated feed of an indexer's newest entries, filterable by category and search term through URL parameters. Automation tools poll it every few minutes instead of running repeated searches, which catches new postings within minutes of indexing and costs fewer API hits. The feed carries the same fields as a search response, including links to the corresponding NZB files.
NZB Segment List: The <segments> block inside each <file> element of an NZB, holding one <segment> entry per article with its byte size, sequence number and Message-ID. Order matters only through the number attribute, since the client requests segments in whatever order it likes and reassembles by index. A missing or wrong Message-ID here produces a failed segment that PAR2 files may be able to repair.
NZB Groups Element: The <groups> block in an NZB file element, listing the newsgroups where that file's articles were posted. A client uses it to issue a GROUP command before requesting articles, and multiple entries appear when a posting was crossposted. If a provider does not carry any of the listed groups, the download fails even though the Message-IDs themselves might be retrievable elsewhere.
NZB Meta Tag: An optional <meta> element inside the NZB's <head> section, carrying a type attribute and a value. Common types are title, category and password. Download clients read the password meta to unlock encrypted RAR archives without prompting, and use title for the job name in the queue. Meta tags are advisory: many NZBs omit the head section entirely and clients fall back on the filename.
Failed Grab: An NZB download from an indexer that does not complete, either because the indexer returned an error page instead of XML or because the quota for the day was already spent. Automation tools log it and move to the next indexer in priority order. A grab that succeeds but yields articles the provider no longer holds is a different problem, showing up later as missing segments.
Backfill: The process of scanning older headers to extend an indexer's coverage backwards in time, as opposed to routine scanning of new articles. Operators run it group by group, often in chunks of a few days at a time, because pulling years of headers is slow and storage-hungry. Backfilling is how an indexer's retention grows beyond the date it was first set up.
Indexer Coverage: Which groups an indexer scans and how completely, distinct from how far back its database reaches. A site tracking 200 binary groups will miss postings made only to a group outside that list, and gaps also appear when scanning breaks for a period. Comparing coverage between sites is why some setups query several indexers for the same search.
Usenet Search: Finding postings without downloading a group's headers, done by querying an indexer's web page or API. The search runs against the indexer's parsed release names rather than against Usenet itself, so results depend entirely on what that site has scanned and how well it parsed the subjects. Searching several indexers at once, usually through an aggregator, compensates for the gaps in any single database.
NZB ID: The identifier an indexer assigns each entry, used in the URL that returns the NZB file and in the guid field of API responses. IDs are local to one indexer, so the same release carries different IDs on different sites. Download tools store them to recognize items already handled and to avoid grabbing the same entry twice from the same source.
NZB Age: How many days have passed since the articles an NZB points to were posted, shown on indexer listings and in API results. Age matters because it is compared against a provider's retention: an NZB older than the retention window points at articles that no longer exist. Older postings are also likelier to be incomplete, since propagation gaps and removals accumulate over time.
Indexer Uptime: The share of time an indexer is reachable and answering queries. Many are run by small volunteer teams on modest hardware, so outages of hours or days happen, and an indexer that is down blocks every automated search pointed at it. Setups that care about continuity configure several indexers with a priority order so a failure falls through to the next one.
Indexer API Limit: The daily ceiling an indexer places on searches and NZB downloads for an account, enforced per API key and reset on a fixed schedule. Free tiers commonly allow a few dozen of each; paid tiers raise the numbers substantially. Once the cap is reached the API returns an error rather than results, which client software reports as the indexer being unavailable.
NZB Import: Handing an NZB file to a download client, either by opening it directly, dropping it into a watched folder, or posting it to the client's API. The client reads the segment lists, queues every Message-ID and begins requesting articles. Import is also how manually saved NZBs enter an otherwise automated setup, bypassing the indexer connection the tools would normally use.
Group Scanning: The routine work an indexer does: connecting to a news server, pulling new headers for each group it tracks with OVER or XOVER, parsing the subjects and writing entries to its database. Scanning runs continuously, often every few minutes per group, and its speed determines how soon after posting an item becomes searchable. Backfill is the same operation aimed at older articles.

See the full Usenet glossary.

NZB Files & Indexing

Put it into practice with the world's best usenet service