Last Known Copy

PRACTICE

Auditing a link collection for rot: what to check, in what order

Checking whether saved links still work is not one job but four, and the standard advice — use the Wayback Machine — answers only the third of them.

Published 2026-08-06 · Updated — · 3,000 words · 23 sources · Status of facts checked 2026-08-06

A saved link fails in four different ways, and the word “broken” covers all four without separating them. Four failures need four checks, in order, each with its own pass condition:

The advice a reader is usually given — look it up in the Wayback Machine — answers the third. It is offered as an answer to the first two, which it is not, and says nothing about the fourth, the only one a collection’s owner controls.

What a decade does to a collection

Pew Research Center sampled about 91,000 webpages a year from 2013 to 2023 — close to a million in all, drawn from the Common Crawl archive — and re-requested them in October 2023. It found that 38% of the pages that existed in 2013 were no longer accessible, that a quarter of all pages sampled across the eleven years were gone, and that about one-in-five pages from 2021 were already inaccessible two years later.1 Those figures are a floor. The methodology counts a page as inaccessible if it returned one of nine specific codes — 204, 400, 404, 410, 500, 501, 502, 503, 523 — or if its domain was not available in a DNS server, and accessible in every other case.2 The report names a second way a page can be lost — the address still exists but its content has been changed — and says it focused on the first case, pages that no longer exist.1 Everything in stage two below sits inside that exclusion.

Against that, the recovery rate. Sawood Alam, research lead at the Wayback Machine, published a comparison on 2026-04-23 of several link-rot studies against the archive’s holdings. Its table gives, per study, the share of the sample found dead and the share of the whole sample that is both dead and available from a capture: for the full Pew dataset of 5.4 million URLs, 26% dead and 16% rescued; for the one-million-URL general subset of it, 27% dead and 13% rescued; for a 27.3-million-URL Old Dominion University study, 65% dead and 65% rescued.3 The last pair needs reading carefully: those URLs were sampled from the index of the Wayback Machine itself, so a dead URL in that study is one the archive already holds, and 65% against 65% is a fact about the sampling frame rather than a coverage rate any other collection can expect. The Pew rows are the closer analogue to a personal collection, and in both of them fewer than half the dead URLs had a capture behind them. For the 2013 cohort of the general sample Alam gives about 38% of the dead URLs rescued, about 15% of the total.3

Audit the export, not the application

As of 2026-08-06, the steps below operate on a reader’s own export file and their own saved URLs.

Get the collection out of whatever holds it first: an audit needs a list, and the export is also the artefact that outlives the tool. One format recurs. Microsoft’s archived documentation states that the Internet Explorer Favorites file format is exactly the same as the Netscape bookmark file format, and sets out that format: an HTML document opening <!DOCTYPE NETSCAPE-Bookmark-file-1> in which each saved link is an anchor carrying HREF, ADD_DATE, LAST_VISIT and LAST_MODIFIED, nested in a folder tree of <H3> headings.4 linkding’s documentation describes both its export and its import as a single HTML file of bookmarks, and lists what that file leaves behind: not the HTML snapshots, not the favicons, not the URLs of snapshots on the Wayback Machine.5

The field that decides the quality of the audit is the one nobody reads. The same documentation notes that throughout the format definition, {date} — ADD_DATE included — is a decimal integer representing the number of seconds elapsed since midnight, 1 January 1970.4 That integer is the moment the page was saved, and therefore the moment an archive should be asked about later: a 2026 capture of a URL saved in 2011 may be a capture of the domain’s next owner. Keep the original export unmodified and work on a copy of it.

Stage one: resolve

Pass condition: every URL has a recorded response, or a recorded reason there was none.

Record five things per URL: the first status code, the status code after redirects, the final URL, the response body size, and a hash of the body. Four of the five exist to make stage two possible; a run that stores only status codes has to be repeated. Three constraints on the requests, all from the HTTP specifications:

Then run the list again, days later, before recording anything as dead: three of Pew’s nine death codes — 500, 502, 503 — are states a working server enters and leaves.2 A single-pass audit converts an outage into a deletion.

Two distinctions are worth keeping. RFC 9110 §15.5.5 defines 404 as the origin server not finding a current representation, and states that a 404 does not indicate whether the condition is temporary or permanent, while §15.5.11 defines 410, Gone, for the case where the server knows the condition is likely to be permanent.6 A 410 is a statement; a 404 is a shrug. RFC 7725 defines 451 for a resource withheld under a legal demand9: not deleted, and possibly still served elsewhere.

Stage two: classify, because a 200 is not evidence of survival

Pass condition: every 2xx response has been tested against the possibility that it is an error page. Google’s crawling documentation, last updated 2025-12-18, gives the definition:

A soft 404 error is when a URL that returns a page telling the user that the page does not exist and also a 200 (success) status code.10

The same page tells site owners to return 404 or 410 instead.10 Three signals separate a soft 404 from a living page, all computable from the five fields recorded in stage one.

Fingerprint the host’s own error page

For each host in the collection, request one path on it that certainly does not exist — a random string. A 4xx means that host reports a missing page with a status code, and its codes can be taken at face value. A 200 means the host emits soft 404s, and the body just returned is the template: hash it, then compare that hash against the recorded hash of every saved URL on the same host. This is the one signal of the three that yields an exact match rather than a threshold, at one extra request per host.

Read the final URL, not only the code

A redirect chain ending at a host root, a generic landing path, or a different domain is the machine-readable form of “this is gone.” Google’s documentation is explicit that a permanent redirect belongs to the case where a page has moved or has a clear replacement on the same site.10

Compare body size against the host’s median

Where several saved pages share a host, a response far below that host’s median size is a candidate error page. Weak alone; useful for ranking a review queue, not for deciding.

What this looks like on a real list

The 2007 roster of social bookmarking services captured at this address carried fifty names. Checked 2026-08-06: eight still offer somewhere to save a link, one survives read-only, eleven are documented as shut down, and thirty could not be verified.a A status-code-only sweep would report a much healthier roster, because many of those domains answer, cheerfully, with something else (all checked 2026-08-06):

Every one of those addresses answers a request, or redirects to one that does. None of them serves the saved page. This is precisely the class Pew set aside when it counted only pages that no longer exist,1 and it is why stage two exists.

Stage three: find a copy

Pass condition: every URL classified dead has been looked up in an archive index, and the result recorded as a capture timestamp or as an absence. Two Internet Archive endpoints do different jobs, and choosing wrongly multiplies the request count.

The availability JSON endpoint accepts a url and an optional timestamp, and returns the closest capture as an object with available, url, timestamp and status; with nothing to return it emits {"archived_snapshots":{}}.11 Note what available: true asserts: that the index holds a capture near that time, and where it is. It does not assert that the capture is of the content rather than of the site’s own error page — which is why the response carries a status field, and why an audit that ignores that field will file captures of 404 pages as recoveries. The optional timestamp is where the ADD_DATE integer from the export goes.

The CDX Server endpoint is what makes bulk work possible, and the reason is matchType, which accepts exact, prefix, host and domain.12 One request with matchType=domain returns the index for an entire host, to be joined against the saved URLs locally: 10,000 URLs across 2,000 hosts becomes 2,000 requests, not 10,000. The other documented parameters that matter here: fl selects from the fields urlkey, timestamp, original, mimetype, statuscode, digest, length; filter=statuscode:200 drops captures of error pages at the source; collapse=digest removes adjacent captures with identical content hashes; from and to bound the range; and showResumeKey with resumeKey pages through a large host.12 The digest field has a second use: a host serving one boilerplate page at thousands of addresses produces one digest across all of them, which is the stage-two fingerprint computed without touching the dead host. No request rate is published in that documentation, and this publication could not verify one.

Then the case readers keep reporting. On the Hacker News thread discussing the Internet Archive’s own link-rot post (2026-06-30), commenter firefoxd described linking to a Wayback copy after the original returned 404, then finding that copy removed from the archive as well; commenter afpx wrote that many old bookmarked URLs had been removed from the Internet Archive and that only some versions were affected (both comments 2026-07-04).13 Commenter kiicia, on a separate thread, gave the general shape of it: link rot on the one bookmark actually needed is the case that hurts (2025-08-28).14 The mechanism could not be verified here, and the consequence does not depend on it. An archive lookup is a query against another organisation’s index at a point in time; “the archive has it” is a fact about 2026-08-06, not a property of the link. Record the capture timestamp rather than a yes or no, because a timestamp is re-checkable later and a boolean is not.

Other archives are worth a second pass: RFC 7089 specifies Memento, a standard way to ask any conforming archive for a resource’s state at a given datetime.15 Which aggregators were serving Memento requests on 2026-08-06 could not be verified.

Stage four: hold a copy

Pass condition: the pages the collection cannot afford to lose exist as files under its owner’s control. This is where auditing becomes archiving, and it is the only stage that changes the result of the next audit. Documented options, as of 2026-08-06:

On the audit rather than the archiving: as of 2026-08-06 linkding’s options documentation records no built-in broken-link check, and issue #1330, “[FEATURE REQUEST] Broken Link Detection,” has been open since 2026-03-25.20 Whether any other tool named here ships a link-health check could not be verified. Stages one and two must be supplied elsewhere.

The rule, as a rule: a collection that must survive the loss of the source page requires a stored copy held by its owner; a collection that only needs to be re-findable does not. Those are different requirements at different costs, and deciding which applies, per link, is the real output of an audit.

The residue, and what a URL alone cannot answer

An honest audit ends with a list it could not resolve: no live page, no capture, no copy. Keep the row — dead URL, title as saved, ADD_DATE, date checked. A dead URL with a date is a citation; a deleted row is nothing.

Every stage above degrades to nothing at the same point, and the cause is record shape rather than tooling. A Netscape export stores a URL, a title, a folder and a set of timestamps4 and no text from the page. While the page lives that is sufficient, because the URL fetches everything else. Once it is gone the URL is only a key: good for asking an archive, useless for identifying the thing if the archive says no. A title gives a search a chance; extracted readable text, stored at save time, answers the question outright; a stored snapshot answers what the page looked like. What was stored at save time decides how much of a dead link is left.

Where to find it now

Service dates below are the Ledger’s; each row carries its own sources and checked date at /ledger/.

Delicious / del.icio.us
Bought by Pinboard’s Maciej Cegłowski on 2017-06-01 and read-only from 2017-06-15: no new bookmarks, no API, no deadline to migrate.21 Checked 2026-08-06: del.icio.us returns a login form only, delicious.com sits behind Cloudflare and is not an operating bookmarking service, and no export endpoint is reachable without an account. Data on shutdown: login-walled.
Ma.gnolia
Database and filesystem failure 2009-01-30; Larry Halff announced on 2009-02-17 that user data was irretrievable, and what users recovered afterwards came from third-party caches and feeds.22c ma.gnolia.com does not resolve (checked 2026-08-06). Data on shutdown: irretrievable.
Furl, Yahoo! My Web, StumbleUpon, Squidoo
Furl shut down 2009-04-17, bookmarks migrated into Diigo. My Web closed 2009-03-18; where the data went is unknown. StumbleUpon closed 2018-06-30 with accounts migrated to Mix.com. Squidoo’s content migrated to HubPages after the 2014-08-15 announcement; squidoo.com now returns a redirect loop (checked 2026-08-06). Data on shutdown: migrated, except My Web, unknown.
CiteULike
Closure announced February 2019, ceased March 2019. citeulike.org still returns a homepage as of 2026-08-06; whether it is a frozen copy could not be verified. Data on shutdown: unknown.
Pocket
Mozilla announced the shutdown 2025-05-22;23 the service stopped working 2025-07-08, and exports remained available until 2025-10-08, after which accounts and data were deleted. An export taken before that date is what remains. Data on shutdown: deleted.
The thirty unverified roster names
What the Wayback Machine holds for each of these domains can be listed with one CDX request per host using matchType=domain.12 No surviving export format is known for any of them, and where the data went could not be verified.
Storage mode, for the tools named above
SingleFile writes one self-contained HTML file per page to local disk.16 ArchiveBox saves HTML, JS, PDFs and media on disk.17 Karakeep stores a full-page archive via monolith alongside the record.18 linkding stores local snapshots via single-file and, as a background task, requests snapshots on the Wayback Machine — the second being custody by a third party, not by the collection’s owner.19 A linkding export carries neither the local snapshots nor the Wayback URLs.5

References

  1. Chapekis, Athena, Samuel Bestvater, Emma Remy and Gonzalo Rivero. “When Online Content Disappears.” Pew Research Center, 2024-05-17. https://www.pewresearch.org/data-labs/2024/05/17/when-online-content-disappears/ — retrieved 2026-08-06. ↩
  2. Pew Research Center. “Methodology: When Online Content Disappears.” Pew Research Center, 2024-05-17. https://www.pewresearch.org/data-labs/2024/05/17/methodology-link-rot/ — retrieved 2026-08-06. ↩
  3. Alam, Sawood. “Gone but Not Forgotten: Recovering the Dead Web.” Internet Archive Blogs, 2026-04-23. https://blog.archive.org/2026/04/23/gone-but-not-forgotten-recovering-the-dead-web/ — retrieved 2026-08-06. ↩
  4. Microsoft. “Netscape Bookmark File Format (Internet Explorer).” Microsoft Learn, archived documentation, 2017-08-15. https://learn.microsoft.com/en-us/previous-versions/windows/internet-explorer/ie-developer/platform-apis/aa753582(v=vs.85) — retrieved 2026-08-06. ↩
  5. Ißbrücker, Sascha. “linkding documentation: Backups.” linkding.link, n.d. https://linkding.link/backups/ — retrieved 2026-08-06. ↩
  6. Fielding, R., M. Nottingham and J. Reschke, eds. “RFC 9110: HTTP Semantics.” IETF, 2022-06. https://www.rfc-editor.org/rfc/rfc9110.html — retrieved 2026-08-06. ↩
  7. Nottingham, M. and R. Fielding. “RFC 6585: Additional HTTP Status Codes.” IETF, 2012-04. https://www.rfc-editor.org/rfc/rfc6585.html — retrieved 2026-08-06. ↩
  8. Koster, M., G. Illyes, H. Zeller and L. Sassman. “RFC 9309: Robots Exclusion Protocol.” IETF, 2022-09. https://www.rfc-editor.org/rfc/rfc9309.html — retrieved 2026-08-06. ↩
  9. Bray, T. “RFC 7725: An HTTP Status Code to Report Legal Obstacles.” IETF, 2016-02. https://www.rfc-editor.org/rfc/rfc7725.html — retrieved 2026-08-06. ↩
  10. Google. “Troubleshoot crawling errors.” Google Search Central, last updated 2025-12-18. https://developers.google.com/search/docs/crawling-indexing/troubleshoot-crawling-errors — retrieved 2026-08-06. ↩
  11. Internet Archive. “Wayback Machine APIs: Availability JSON API.” archive.org, n.d. https://archive.org/help/wayback_api.php — retrieved 2026-08-06. ↩
  12. Internet Archive. “Wayback CDX Server API.” GitHub, internetarchive/wayback, n.d. https://github.com/internetarchive/wayback/blob/master/wayback-cdx-server/README.md — retrieved 2026-08-06. ↩
  13. Hacker News. “Gone but Not Forgotten: Recovering the Dead Web” (item 48739682), 2026-06-30; comments by firefoxd and afpx, 2026-07-04. https://news.ycombinator.com/item?id=48739682 — retrieved 2026-08-06. ↩
  14. Hacker News. “Bookmarks.txt is a concept of keeping URLs in plain text files” (item 45047572), 2025-08-28; comment by kiicia, 2025-08-28. https://news.ycombinator.com/item?id=45047572 — retrieved 2026-08-06. ↩
  15. Van de Sompel, H., M. Nelson and R. Sanderson. “RFC 7089: HTTP Framework for Time-Based Access to Resource States — Memento.” IETF, 2013-12. https://www.rfc-editor.org/rfc/rfc7089.html — retrieved 2026-08-06. ↩
  16. Lormeau, Gildas. “SingleFile.” GitHub, n.d.; repository metadata (AGPL-3.0, 22,101 stars, last push 2026-08-05) read from the GitHub API. https://github.com/gildas-lormeau/SingleFile — retrieved 2026-08-06. ↩
  17. ArchiveBox. “Release v0.7.4.” GitHub, 2026-05-18; repository description and licence (MIT) read from the GitHub API. https://github.com/ArchiveBox/ArchiveBox/releases/tag/v0.7.4 — retrieved 2026-08-06. ↩
  18. Karakeep. “Karakeep: README.” GitHub, n.d.; release v0.33.1 dated 2026-08-01. https://github.com/karakeep-app/karakeep — retrieved 2026-08-06. ↩
  19. Ißbrücker, Sascha. “linkding documentation: Options.” linkding.link, n.d.; release v1.45.0 dated 2026-01-06. https://linkding.link/options/ — retrieved 2026-08-06. ↩
  20. sissbruecker/linkding. “[FEATURE REQUEST] Broken Link Detection” (issue #1330), 2026-03-25. https://github.com/sissbruecker/linkding/issues/1330 — retrieved 2026-08-06. ↩
  21. Cegłowski, Maciej. “Pinboard Acquires Delicious.” Pinboard Blog, 2017-06-01. https://blog.pinboard.in/2017/06/pinboard_acquires_delicious/ — retrieved 2026-08-06. ↩
  22. Backblaze. “Ma.gnolia Wilts with No Backup.” Backblaze Blog, n.d. https://www.backblaze.com/blog/magnolia-wilts-with-no-backup/ — retrieved 2026-08-06. ↩
  23. 9to5Mac. “Mozilla announces shutdown of Pocket.” 9to5Mac, 2025-05-22. https://9to5mac.com/2025/05/22/mozilla-announces-shutdown-of-pocket/ — retrieved 2026-08-06. ↩

Notes

  1. Founding years for twenty-three names on the roster could not be verified. Earliest Wayback captures were collected for several of them; those reflect domain registration rather than service launch and are not founding dates. ↩
  2. The three are urlex.info, listerlister.com and mybookmarkmanager.com. Checked 2026-08-06, the first serves adult video and the other two serve gambling sites. They are named because a census that omits them is incomplete, and left unlinked because nothing in the argument requires sending a reader to them. ↩
  3. Ma.gnolia’s founding year is given variously as 2005 and 2006 and is not asserted here. ↩