patchwork-turbo

Fix stale pages, lost pages and retry storms found in review

State and fetching:
- Validators, sitemap dates and 404 counts are kept per track, and
  stored only after the track's pages are written. Two tracks reading
  one URL no longer starve each other with 304s, and a failed write is
  redone instead of hidden behind a 304.
- Changing a track's select or ignore converts its pages again.
- Concurrent state saves take turns, each with its own temp file; run
  no longer fails tracks that finish together.
- A failed track waits 15 minutes (or its own interval) instead of
  retrying on every 30-second tick.
- A host that answered 429 or 503 fails at once while it's paused,
  instead of holding one of the four run slots for up to an hour.
- check reads the real state and never saves it.

Pages:
- A 404 or 410 keeps the page; the second in a row removes it.
- A sitemap with no pages on its own host, or none readable, fails and
  keeps the folder. max_pages limits fetching, never which pages stay.
  A page that fails to convert doesn't record its lastmod.
- Feed items are named with their link's #fragment; items that still
  collide get a hash, whatever order the feed lists them in.
- Colliding data paths are sorted and suffixed, so the bytes never
  change between runs and no value is lost.
- A const or var group keeps its name when it grows: adding C to A, B
  is "Changed: const A", not a breaking removal.
- URL slugs drop .html, .php and similar: /a.html is "a".
- DOCX: outline level 9 is body text, text boxes no longer split their
  paragraph, and a truncated or broken document is an error instead of
  a mass removal.

Also: mdtext holds the escapes, tables and whitespace helpers the
converters had copied; sitemaps may be up to 50 MB; go.mod lists x/mod
as direct; README and the example config describe the actual layout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Pablo Murad

Import history from the Internet Archive, add the README

- import-wayback lists a url track's captures (one per day at most, only
  when the bytes changed, the newest ones first), converts each with the
  track's selectors and feeds them to `patchwork snapshot` oldest first.
  The newest copy stays in the output folder, so the first live run's
  change becomes a revision with a note.
- Tracks that follow one thing write out/<name>.md, so their slug is
  the track name.
- README: what each track reads, how notes are built, politeness rules,
  limits.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Pablo Murad

Track Go APIs and dependencies

- gocode: one page per package of a Go module with every exported
  declaration, its doc comment and no function bodies. Files are chosen
  for linux/amd64 so every machine writes the same pages; main packages,
  tests and nested modules are skipped. The note leads with what can
  break importers: "Breaking: changed func New, removed func Old.
  Changed: type Thing. Added: type Option".
- deps: go.mod (with replace lines) and package.json dependencies, one
  per line; the note names what was updated, added and removed.
- summary: the capped "a, b and 3 more" phrases shared by the notes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Pablo Murad

Track data files and office documents

- datamd: JSON, YAML and CSV become one list item per field, sorted, so
  patchwork's diff reads as field-level changes. Lists of objects are
  keyed by id, slug, key, name or title when one is unique, so adding an
  element doesn't renumber the rest. The revision note names the fields:
  "plans[Pro].price 49 → 59; added plans[Max].price".
- officemd: DOCX and ODT headings, paragraphs, lists and tables. DOCX
  headings come from the style's outline level, which works whatever
  language the style names are in.
- Pages can bring their own change summary; data tracks use it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Pablo Murad

Add the skeleton and web tracks

patchwork-turbo turns things that aren't Markdown pages into Markdown
pages, written into a folder that a normal `patchwork serve` watches.
The two never import each other.

- config: turbo.yaml with strict keys and defaults; one source per track.
- out: writes a page only when its content changed, atomically, with a
  revision note that names the time and what changed, so every real
  change becomes a revision and re-fetching costs nothing.
- fetch: polite HTTP (robots.txt, per-host spacing, conditional GET,
  Retry-After, 2 MB cap).
- htmlmd: HTML to Markdown blocks with selectors to pick and to ignore.
- url, sitemap and feed tracks; once, run, check and init commands.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Pablo Murad