Watching AI model documentation:
what changes, what goes quiet, and what the watcher misses

Setting and data

Each daily run probes every tracked URL (link check), re-fetches a 15 % rotation of documents and compares a text fingerprint that ignores known page furniture such as download counters, sidebars and footers (content check), diffs publisher index pages for new links, and then lets a sandboxed LLM agent propose additions through a deterministic validator. A supervised backfill on 9 August 2026 seeded 196 documents; the snapshot covers 53 runs from 11 August to 29 September 2026, during which 159 more were discovered. The only continuous public log of edits to such documents we know of is the Midas Project’s hand-curated Watchtower (The Midas Project 2026); this study is fingerprint-based and model-centred.

What changes

Of the 239 consecutive version pairs in the database, 32 (13.4%) are the tracker’s own doing, so the diff compares two captures, not two revisions: canonical URLs migrated on 31 August from announcement pages to full PDFs, and first captures of a redirect target after a recorded move. The remaining 207 publisher-side pairs were classified by reading the text diffs (Figure 1a): 58.0% were Hugging Face download counters, evaluation-widget rows, footers or PDF re-extraction artefacts, 29.5% changed the document’s content and 8.7% did so substantially. On 31 August the fingerprint was also recomputed to ignore known furniture. This cut the per-check change-flag rate from 49.2% to 17.6% while the number of substantive pairs found did not fall (23 before, 38 after, over a longer window), so precision per flag rose from about 13.1% to 24.1%. The before side is incomplete: 107 versions that the changelog records as written are no longer in the database, 106 of them from before the recompute, so pairs of every kind are missing there. The remaining leak is evaluation-widget rows that end in a score, which the bullet-anchored patterns do not match. In document terms, 49 of 323 tracked documents (15.2%) changed substantively during the window, an estimated 16.5% of HTML and 15.9% of PDF documents within 30 days of tracking; the two curves in Figure 1b are not distinguishable at this sample size.

The substantive changes are the reason to run a tracker at all (Table 1; Appendix A lists all 61 with URLs). Of the publisher-side pairs, 14 corrected a stated number or fact, 13 touched an evaluation number and 24 touched a safety, risk or mitigation section. Three patterns recur (Appendix A ids in brackets). First, some corrections come with dated change logs, mostly at OpenAI: one corrected number propagated with its change log into 5 OpenAI documents that quote it as a baseline (A4, A11, A14, A19, A22), and most recently the GPT-6 Astra system card gained an appendix covering GPT-6 Sol and Luna and corrected Astra’s HealthBench results after an evaluation misconfiguration, with dated entries (A52); xAI (A12) and Anthropic (A44) each added one dated log. Second, edits without a change log exist: an NVIDIA card revised a tool-calling score downward and dropped its research-only use restriction with no note (A6); OpenAI removed a prompt-response table in which GPT-5.6 Cyber, alone of four configurations, answered a request for a Keychain-bypass and Chrome-cookie-decryption tool (A27), and deleted a named launch partner from an access-programme page (A40). Third, documents change scope: OpenAI rebuilt its Daybreak cyber-access tiers to add GPT-6 Sol, Luna and Astra and to require extra approval for GPT-5.6-Cyber (A54); Anthropic’s cyber-safeguards page now excludes Opus 5.5 and Sonnet 5.5 (A59); NVIDIA’s Cosmos 3 cards were each cut back to a single model (A50, A51); and inclusionAI added a “training content summary” linking a public training-data statement to several of its cards (A53, A55, A56, A57, A58). A caution the other way: the previous edition found that apparent deletions of Anthropic footnotes were extractor artefacts. The publisher had wrapped its footnotes in an element whose class name contains “footer”, which the boilerplate remover discards, while the raw HTML still held every footnote. The extractor has since been fixed and twelve classified versions, the artefacts among them, were deleted from the database; one new footnote deletion in the same position is treated as an artefact until it is checked against the raw capture. A fingerprint tracker cannot tell disclosed from undisclosed edits, or edits from extraction bugs, but it produces the diff that lets a human tell.

What goes quiet

Excluding host-side outage runs and the 86 probes that never reached a server leaves 8,259 valid link checks over 346 documents (33.6 document-years, including documents later removed from the corpus), of which 97.3% succeeded. Hard loss is rare: 2 documents ever returned a 404 (0.06 per document-year). The monitor declared one of them dead after three consecutive 404s, a FAR.AI evaluation of the cyber safeguards of Alibaba’s Qoder agent (Qwen3.8-Max); the other was a slug typo that the publisher fixed with a redirect. Two cases cannot give a reliable rate over 7.3 weeks, but they are not out of line with published baselines for young pages: Pew found 8 % of pages under one year old inaccessible (Pew Research Center 2024), deep links in the Wayback corpus have a median lifetime of 1.3 years (Garg et al. 2024), and 1 to 4 % of web references in scholarly articles from 2012 were already rotten when checked (Klein et al. 2014). Moves are more common: 6 documents moved via permanent redirect (0.18 per document-year: site restructures, a trailing slash, the typo fix and the soft-gone pages below). Bot blocking is the larger nuisance: 16 documents were refused with HTTP 403 at least once, 12 of them OpenAI documents on openai.com. For most of the window the blocks were intermittent (overall a median of 46.1% of later checks), consistent with rate-based bot detection, but in the last week most of the blocked OpenAI pages were refused in most runs; whether that reflects the site or a change on the tracker’s host is not known, and block rates depend on the fetch client’s fingerprint as much as on the site (Gundelach, Mühlhauser, and Herrmann 2026). This matters because OpenAI is also the publisher whose documents carried most of the dated corrections above. The quietest failures were invisible to the link checker: two pages kept returning HTTP 200 while their body became a two-line client-side redirect stub (meta refresh plus a canonical link to the new location), and only the content fingerprint noticed. For at least 17 days the link checker saw nothing, until the publisher replaced the stubs with real permanent redirects on 23 September and the monitor marked them moved (Table 3). This is the reference rot of Zittrain et al. (Zittrain, Albert, and Lessig 2014), the URL resolving while the content is gone, although here it was temporary because the publisher had moved rather than removed the content.

Discovery is fast

For the 80 documents published after the backfill, the median delay from publication to first tracking was 2.3 days and 85.0% were tracked within a week. Leads from index diffs and from the monitor’s own candidate queue were tracked within two days (median 1.3 to 1.5 days); free-form agent search took 4.8 days and manual submissions 14.5 days.

Limitations

The window is 7.3 weeks on one host with one fetch client: enough to measure short-run edit rates, not link rot, whose baselines are annual; block rates in the literature range from under 1 % for browser-like clients to 15 % for headless ones (Gundelach, Mühlhauser, and Herrmann 2026). Coverage was uneven: 41.5% of the daily runs lost their checks to a host-side fault (Appendix B), which stretched the content re-check interval’s tail to 18.0 days at the 90th percentile against an intended 6.7 days, and content changes are observed only when a document’s turn in the 15 % rotation comes up, so change dates lag true edit dates by up to a re-check interval and edits reverted between checks are missed. The instrument changed mid-window (URL migrations and the fingerprint recompute on 31 August). The classification was done by LLM sub-agents with a written taxonomy (189 pairs at high confidence, 48 at medium, 2 at low); it was spot-checked, not double-coded, categories at the boundary of “minor” and “metadata” are soft, and text extraction can drop page elements, so apparent deletions need a check against the raw capture. The corpus is curated by an allow-list and an agent, and the literature baselines cover different populations.

Future work

On the science: a longer horizon allows a hazard model of edit rates by publisher and document type; aligning detected edits with publishers’ change logs and with Watchtower’s disclosed and undisclosed tags would estimate the disclosed fraction; classifying every minted version in the pipeline would turn the appendix into a live feed. On the instrument: re-fetch every document daily instead of a 15 % rotation (the whole corpus is under 400 MB and the link probe already touches every URL), so edits are dated to the day; treat a body that collapses to a redirect stub as a soft 404 and re-point moved documents; keep the new footnote pre-clean under test and diff the raw capture alongside the extracted text so extractor regressions are never read as deletions; extend the furniture patterns to widget rows that end in a score; run on an always-on host.

Every substantive publisher-side change

Table 2 lists all 61 publisher-side version pairs classified as a content change (minor, major, or replaced), in order of detection, with the canonical URL and the stored version ids so the diff can be reproduced from the database (document_versions rows, text under data/text/). Detection dates lag the edit by up to one re-check interval. Table 3 lists the pages that went quiet behind an HTTP 200 and Table 4 the recorded moves.

Tracker reliability

Of the 53 runs, 22 recorded a fetch error for every URL, the longest streak lasting 4 consecutive runs (Figure 2); the last was on 22 September, and none of the 9 runs since then was a full outage. The snapshot does not record why the outages stopped, and the run logs stop on the same date. The systemd journal explains 21 of them: the run started within two minutes of the laptop waking from suspend (median 0 s after resume, versus 38,412 s for healthy runs). The timer is Persistent=true, so a trigger missed while asleep fires at the moment of resume, before Wi-Fi is back, and every probe fails instantly. The run then continues; the agent phase, minutes later, usually has network again (9 outage runs still added documents) and recent runs label the commit as an outage, but that day’s link and content checks are lost. A further two runs lost part of their checks (86 of 287 probes on 18 September, while the network flapped). The curation agent failed in 16 runs, 13 of them outage runs where its credential refresh had no network either; only 3 healthy runs lost their agent, and 19 of the 44 runs with a run log had both phases healthy. Run logs after 22 September were not in the snapshot, so the agent phase of the last 9 runs is unknown (“no log” in Table 5). Outages stretched the tail of the content re-check interval. Table 5 gives every run.