Protocol & Server Infrastructure

NNTP itself, plus the machinery behind a news server. This section covers the protocol commands a newsreader sends, the three-digit response codes it gets back, the ports and encryption involved, and how spools, feeds, and peering arrangements are built. Useful for anyone configuring a client or trying to understand what a provider is actually operating.

59 terms in this category.

NNTP: Short for Network News Transfer Protocol, the TCP-based protocol that moves Usenet articles between servers and between a server and a newsreader. Defined in RFC 977 in 1986 and rewritten as RFC 3977 in 2006, it uses plain text commands such as GROUP, ARTICLE and POST, each answered by a three-digit response code. Every newsreader and download client speaks it.
NNTPS: NNTP wrapped in a TLS session from the moment the connection opens, conventionally on port 563. The encryption hides article contents, group names and login credentials from an ISP or anyone else watching the wire, at a small CPU cost on both ends. Commercial providers offer it on every account, and most newsreaders default to it rather than plaintext port 119.
RFC 3977: The 2006 specification that defines NNTP as it is used today, replacing the 1986 RFC 977. It formalizes capability negotiation, the OVER and HDR commands, streaming transfer with CHECK and TAKETHIS, and the meaning of each response code class. Server authors treat it as the reference text; most of what a modern newsreader sends over the wire is described in it.
RFC 977: The original NNTP specification, published in 1986 by Brian Kantor and Phil Lapsley, which replaced the older practice of shipping news over UUCP batch transfers. It defined the command and response structure still recognizable today, including GROUP, ARTICLE, POST and IHAVE. Obsoleted by RFC 3977 in 2006, it survives mainly as historical context and in the occasional very old server.
RFC 5536: The 2009 document that specifies the netnews article format: which headers an article may carry, how Message-ID, Newsgroups, From and References are constructed, and what counts as a valid body. It split the old RFC 1036 into format and architecture halves, the architecture half becoming RFC 5537. Articles that violate it may be rejected or silently dropped by strict servers.
RFC 5537: Covers netnews architecture rather than article syntax: how servers relay articles to peers, how the Path header records the route, how control messages such as newgroup and cancel are meant to be handled, and what injecting agents must do. Published in 2009 alongside RFC 5536, the pair replaced RFC 1036 from 1987. It describes propagation rules that news administrators still follow.
RFC 1036: The 1987 standard for the format of Usenet articles, itself a netnews adaptation of the mail format in RFC 822. For two decades it was the document everything from newsreaders to injection scripts was written against. Split and obsoleted in 2009 by RFC 5536 for article format and RFC 5537 for architecture, though old software and documentation still cite it.
Port 119: The registered TCP port for plaintext NNTP, in use since the protocol's first specification. Traffic on it is unencrypted, so an ISP can see which groups are fetched and, without STARTTLS, the account password as well. Some ISPs block or throttle 119 outright. Most providers still accept connections there for compatibility, but encrypted port 563 has largely replaced it.
Port 563: The registered TCP port for NNTP over TLS, assigned under the name nntps. A client opening a connection here negotiates encryption before sending any command, so credentials never cross the network in the clear. It is the default in most newsreaders and download clients, and the setting providers publish first. Its plaintext counterpart is port 119.
Alternate Ports: Ports other than 119 and 563 that a provider accepts NNTP on, commonly 80, 443 and 8080. They exist because some ISPs, workplaces and hotel networks block the standard news ports while leaving web ports open. Traffic on 443 in particular looks like ordinary HTTPS to simple filtering. Providers usually publish both a plaintext and an encrypted alternate port.
SSL/TLS: The cryptographic layer that encrypts an NNTP connection, TLS being the current protocol and SSL its deprecated predecessor. On a news connection it protects the password sent by AUTHINFO, the group names requested and the article data returned. Modern servers accept TLS 1.2 and 1.3 and refuse older versions. Enabling it costs a few percent of CPU and no meaningful download speed.
STARTTLS: A command that upgrades an already open plaintext NNTP connection to an encrypted one, without changing ports. The client connects on 119, issues STARTTLS, and the TLS handshake follows before any password is sent. It suits servers that cannot dedicate a second port, but implicit TLS on 563 is more common with commercial providers and avoids the risk of a downgrade.
AUTHINFO: The NNTP command pair that logs a client into a server: AUTHINFO USER followed by the account name, then AUTHINFO PASS followed by the password. The server answers 381 after the username and 281 once the password is accepted. It is near universal on commercial providers, which meter connections per account. Sent over an unencrypted link, both values travel in the clear.
CAPABILITIES: The command a client sends to ask what a server supports before doing anything else. The reply is a list of labels such as READER, POST, OVER, HDR, STARTTLS and AUTHINFO, one per line. Introduced in RFC 3977 to replace guesswork and the older LIST EXTENSIONS, it lets a newsreader decide whether to use OVER or fall back to XOVER.
MODE READER: Sent by a client to tell a server it wants reading commands rather than a peer-to-peer article feed. On INN the server may hand the connection to a separate reader process and return a fresh greeting, which is why some clients issue it immediately after connecting. Pure reader servers accept it as a no-op. Feeding software never sends it.
GROUP Command: Selects a newsgroup for the rest of the session. The server replies with the estimated article count, the lowest and highest article numbers it holds, and the group name, then sets the current article pointer to the first article. Almost every reading operation depends on it, since ARTICLE, HEAD and OVER take article numbers that only mean something inside a selected group.
LIST Command: Returns server-wide information, most often the full list of carried newsgroups with their high and low water marks and posting status. Variants include LIST ACTIVE.TIMES for creation dates, LIST NEWSGROUPS for one-line descriptions and LIST OVERVIEW.FMT for the overview field order. On a server carrying over a hundred thousand groups the plain listing runs to several megabytes, so clients cache it.
LISTGROUP Command: Returns the article numbers actually present in a group, one per line, rather than the range summary GROUP gives. The difference matters because expiry and cancellation leave gaps: a group reporting 1 to 900000 may hold far fewer articles. Clients use it to avoid requesting numbers that no longer exist. RFC 3977 also allows a range argument to limit the output.
ARTICLE Command: Fetches a complete article, headers and body together, identified either by its number in the currently selected group or by its Message-ID. The server answers 220 and then sends the text, terminated by a line containing a single dot. Download clients use it for every segment of a binary; a 430 reply means the article is not in the spool.
HEAD Command: Retrieves only the headers of an article, leaving the body on the server. Header-downloading newsreaders once used it heavily to build local indexes, since pulling headers for a whole binary group is far cheaper than pulling the articles. OVER has largely displaced it for bulk work because it returns several articles' worth of fields in one response, but HEAD remains useful for inspecting one article.
BODY Command: Retrieves an article's body without its headers. Download clients that already hold header data from an NZB or an overview fetch use it to save the few hundred bytes of headers on every segment, which adds up across the thousands of articles in a large release. The reply is 222 followed by the body, ended by a lone dot.
STAT Command: Asks whether an article exists without transferring it. The server replies with the article number and Message-ID if present, or 430 if not. Because nothing is downloaded, it is the cheap way to test whether a provider still holds a release before starting a large job, and connection-test tools and fill-server logic rely on it.
NEXT and LAST Commands: Move the current article pointer forward or backward one article within the selected group, returning the new number and Message-ID but no content. They date from the era of reading a group sequentially at a terminal, and text newsreaders still use them to walk through unread articles. Binary download clients ignore them entirely, addressing articles by Message-ID from an NZB.
POST Command: Submits a new article to a server. The client sends POST, gets a 340 invitation, sends headers and body ending with a single dot on its own line, and receives 240 on success or 441 on rejection. Servers that do not accept posting from an account answer 440. Dot-stuffing protects any body line that itself begins with a period.
IHAVE Command: The older way one server offers an article to another: it sends IHAVE with a Message-ID, waits for 335 or 435, and only then transfers the text. The wait after every offer makes it slow over long-latency links, since nothing else can be sent meanwhile. Modern feeds use the streaming pair CHECK and TAKETHIS instead, though IHAVE remains in the specification.
CHECK and TAKETHIS: The streaming transfer pair. CHECK asks a peer whether it wants an article by Message-ID; TAKETHIS sends the article immediately without waiting for an answer, letting the receiver reject it afterwards. Because a feeder can pipeline hundreds of these down one connection, throughput no longer depends on round-trip time. Together they carry most inter-server traffic on the modern backbone.
Streaming Mode: A way of feeding articles between servers that removes the lockstep wait of IHAVE. The sender pipelines CHECK and TAKETHIS commands continuously and processes the replies as they arrive, keeping the link full even on a transatlantic path where each round trip costs 80 milliseconds or more. Every high-volume transit server negotiates it, usually via the MODE STREAM command.
XOVER: The non-standard command that made header browsing practical: one request returns subject, author, date, Message-ID, references, byte count and line count for a whole range of articles. Newsreaders used it to build group listings without fetching a single article. RFC 3977 standardized the same function as OVER; servers keep XOVER as an alias, and older clients still send it.
OVER Command: The standardized form of XOVER, defined in RFC 3977. Given a range of article numbers or a Message-ID, it returns one tab-separated line per article containing the overview fields the server maintains. Indexers and header-downloading clients depend on it: fetching overview lines for a binary group costs a fraction of what fetching the articles themselves would.
XHDR: Returns the value of one named header across a range of articles, for example every Subject line between two article numbers. Cheaper than OVER when only a single field is wanted, and able to reach headers the overview database does not store. RFC 3977 standardized the same capability as HDR, with LIST HEADERS reporting which fields a server will serve.
XZVER: A non-standard extension that returns the same overview data as XOVER, compressed with deflate before transmission. Header text compresses well, so a group whose overview runs to hundreds of megabytes may transfer in a fifth of that. It matters chiefly to indexers and to newsreaders building a local header index; support varies by provider and by server software.
Header Compression: Compressing overview data before sending it, through non-standard extensions such as XZVER or XFEATURE COMPRESS GZIP. Because overview lines are repetitive text, compression ratios of three to five to one are ordinary, cutting the time to build a local header index for a busy binary group from hours to minutes. Only useful to clients that download headers in bulk.
NEWNEWS: Asks a server for the Message-IDs of every article received in a set of groups since a given date and time. Useful in principle for catching up or for a small site pulling a feed, but answering it can require scanning the entire history file, so most public servers disable it. Clients that need new articles track high water marks instead.
NEWGROUPS Command: Lists newsgroups created on a server since a given date, letting a newsreader show what is new without downloading the entire active file again. Servers derive the answer from creation timestamps, so a group that was carried late may appear long after it was actually created elsewhere. Newsreaders typically run it at startup and offer any new groups for subscription.
NNTP Response Code: The three-digit number opening every server reply. The first digit gives the class: 1xx informational, 2xx the command succeeded, 3xx the server wants more data such as an article body, 4xx the command was understood but failed, 5xx the command was not recognized or not permitted. The second and third digits narrow it down, as with 430 for a missing article.
Spool: The on-disk store where a news server keeps article text. Its size relative to the daily incoming feed sets retention: a backbone taking well over a hundred terabytes a day needs enormous storage to hold years of binaries. Articles are written by the feed process and read out on demand by ARTICLE and BODY, using a storage method such as CNFS or tradspool.
Active File: The server's master list of carried newsgroups. Each line holds the group name, the highest and lowest article numbers present, and a flag showing whether posting is allowed, whether the group is moderated, or whether articles are aliased elsewhere. The LIST command serves it to clients, and a group missing from it is not carried, whatever exists on other servers.
History File: An index of every Message-ID the server has seen recently, with the article's arrival time and storage location. Its purpose is duplicate rejection: when a peer offers an article by CHECK or IHAVE, the server looks the ID up here and declines anything already held. It also drives expiry. On a busy transit server it holds tens of millions of entries.
INN: InterNetNews, the most widely deployed open-source news server, written by Rich Salz in 1991 and maintained since by the Internet Systems Consortium. It handles reading and transit in one package, supports CNFS and tradspool storage, and hosts filter hooks that Cleanfeed plugs into. Most university and hobbyist news sites that still exist run it, as do parts of some commercial backends.
Diablo: An open-source news server built for transit rather than reading, originally by Matt Dillon in the 1990s. It concentrates on accepting and forwarding enormous article volumes with modest hardware, splitting feeding and reader duties across separate processes and machines. Several commercial backbones grew up around it. Its reader side exists but is less capable than the transit side it was designed for.
Cleanfeed: A spam filter for INN, written in Perl and hooked into the server's article acceptance path. It scores incoming articles on excessive crossposting, binaries in text groups, forged headers and known spam signatures, then rejects or quietly drops what fails. Deployed widely from the late 1990s, it cut Usenet spam volumes sharply and is still run by administrators who carry text hierarchies.
Transit Server: A news server whose job is moving articles between peers rather than serving readers. It accepts feeds by CHECK and TAKETHIS, deduplicates against its history file, and forwards to downstream sites, often keeping articles only long enough to pass them on. Providers run transit machines separately from reader machines so that a burst of incoming volume cannot slow customer downloads.
Reader Server: The machine customers actually connect to, running the commands a newsreader sends: GROUP, OVER, ARTICLE, BODY and AUTHINFO. It authenticates accounts, enforces the connection limit, and pulls article text from the spool or from a storage tier behind it. Providers scale readers horizontally behind round-robin DNS, keeping them separate from the transit servers that take the incoming feed.
Feeder: The process or machine that pushes articles out to peers. It reads a queue of newly accepted Message-IDs, opens streaming connections to each downstream site, and offers articles with CHECK, sending the ones wanted by TAKETHIS. Feeder configuration decides which hierarchies each peer receives, and a backlogged feeder is the usual cause of articles arriving late at a downstream server.
Peer: Another news server that exchanges articles with this one. Peering is arranged bilaterally: each side agrees which hierarchies to send and how many concurrent streams to allow, then both run feeders pointed at each other. A well-connected server keeps several peers so that one outage does not create gaps, and articles reach it by whichever path arrives first.
Backbone: The set of large, densely peered servers that carry the full article feed and move it worldwide within seconds. There are only a handful of independent ones, mostly in Europe and North America; many providers are resellers of one of them. Which backbone an account sits on determines its retention, completion and the practical value of adding a second provider.
Connection: A single TCP session between a client and a news server, carrying one NNTP command stream. Because each connection fetches articles serially, download speed scales with how many run at once, up to the account's allowed limit and the available bandwidth. Clients open them at startup, authenticate each with AUTHINFO, and reuse them across thousands of article requests.
Concurrent Connections: The number of simultaneous NNTP sessions a client keeps open to one server. Providers sell accounts with a fixed ceiling, commonly between 8 and 60. More connections raise throughput on high-latency links up to a point, after which the bottleneck moves to bandwidth or disk; exceeding the ceiling produces a 502 refusal on the extra sessions rather than slower transfers.
Persistent Connection: A connection held open across many article fetches instead of being torn down after each one. Setting up a TCP session and a TLS handshake costs a round trip or two, which would dominate the time spent on small segments, so download clients keep sessions alive and reuse them. Servers close idle ones after a timeout, typically a few minutes.
Round-Robin DNS: Publishing several server addresses under one hostname so that successive lookups return different machines. Providers use it to spread customer connections across a reader cluster and to route around a failed node. One consequence is that a client opening twenty connections may land on several different servers, which is harmless for reading but can make troubleshooting a single bad node confusing.
CNFS: Cyclic News File System, a storage method that writes articles into fixed-size buffers laid out in advance and overwrites the oldest data when a buffer fills. Nothing is deleted and no expiry run is needed, so disk use is constant and predictable. It is the standard choice for high-volume binary spools, where per-article filesystem overhead would otherwise be crippling.
Tradspool: The traditional storage method: one file per article, in a directory tree that mirrors the group name, so alt.test article 42 lands in alt/test/42. Easy to inspect with ordinary shell tools and convenient for small text sites, but the filesystem overhead of millions of tiny files makes it unworkable for binary volumes, where CNFS is used instead.
Timehash: A storage method that files articles into directories derived from their arrival time rather than their newsgroup. Because everything written in the same period sits together, expiring old content becomes a matter of removing whole directories, which is far cheaper than tracking articles individually. It sits between tradspool and CNFS in practice, used where per-article files are wanted but expiry must stay fast.
Storage Method: How a news server lays articles out on disk. INN offers tradspool, timehash and CNFS among others, and an administrator can assign different methods to different hierarchies, keeping text groups in individual files while binaries go into cyclic buffers. The choice sets how expiry works, how much overhead each article carries, and how large a spool the hardware can sustain.
Article Numbering: Each server assigns its own sequential number to an article within each group it appears in, starting from 1 and never reused. The numbers are local: article 500000 on one server is a different article on another, which is why Message-IDs, not numbers, identify articles globally. Gaps appear as articles expire, so a range from GROUP overstates what is actually held.
TAKETHIS Response: What a receiving server returns after an article has already been sent by TAKETHIS. 239 means the article was accepted; 439 means it was rejected, usually as a duplicate, as filtered spam, or because the group is not carried. Since the transfer happened before the answer, a 439 wastes bandwidth, which is why feeders send CHECK first when the link is expensive.
NNTP Session: Everything that happens on one connection from the server's greeting to QUIT. A typical reading session runs: 200 greeting, CAPABILITIES, AUTHINFO USER and PASS, GROUP to select a newsgroup, then a long run of ARTICLE or BODY requests. State such as the selected group and the current article pointer lives only inside the session and is lost when it closes.
News Cluster: A group of machines presented to customers as one hostname, splitting the work of authentication, article retrieval and storage. A large provider may run dozens of reader front ends against a shared spool tier, with round-robin DNS distributing connections. Clustering is what lets a service hold years of retention and answer thousands of simultaneous connections without any single machine doing everything.
Edge Server: A reader machine placed near customers geographically, often in a separate city or continent from the main spool, to cut round-trip time and transit costs. Requests it cannot satisfy locally are pulled from the core storage behind it. For a customer, the practical effect is a lower latency connection and better throughput per connection than reaching across an ocean.

See the full Usenet glossary.

Protocol & Server Infrastructure

Put it into practice with the world's best usenet service