patchwork-turbo

# patchwork-turbo

[Patchwork](https://code.pablomurad.com/patchwork.git) publishes revision feeds for Markdown pages. patchwork-turbo brings everything else to it: web pages, whole sites, other people's feeds, JSON and CSV files, Word and LibreOffice documents, the API of a Go module and a project's dependencies.

It turns each thing it tracks into a Markdown file in a folder, and a normal `patchwork serve` watches that folder. The two programs never import each other. You can open any file turbo wrote and read exactly what Patchwork will compare.

```
turbo.yaml ──> patchwork-turbo run ──> out/*.md ──> patchwork serve ──> Atom, JSON Feed, API, text
```

> Status: early development. The JSON contract and the file layout may change before 1.0.

## What it tracks

| Track | Source | One page per | What a revision note says |
| --- | --- | --- | --- |
| `url` | a web page | the page | `Changed on 2026-10-24 14:05 -03: 1 block added.` |
| `sitemap` | a sitemap.xml or sitemap index | listed page | the same, per page |
| `feed` | an RSS or Atom feed | item | the same, per item |
| `data` | JSON, YAML or CSV, local or remote | file | `plans[Pro].price 49 → 59; added plans[Max].price` |
| `doc` | DOCX or ODT, local or remote | document | `2 blocks changed.` |
| `go` | a Go module on disk | package | `Breaking: changed func New, removed func Old. Added: type Option` |
| `deps` | go.mod or package.json | file | `Updated golang.org/x/net v0.60.0 → v0.61.0; added golang.org/x/mod v0.42.0` |

Patchwork publishes a revision when a page's `revision` note changes. Turbo writes a file only when the converted content changed, and gives it a new note naming the time and the change, so every real change becomes one revision and a page that didn't change costs nothing. The first version of a page is Patchwork's silent baseline, as always.

## Install

```sh
git clone https://code.pablomurad.com/patchwork-turbo.git
cd patchwork-turbo
go build -o patchwork-turbo ./cmd/patchwork-turbo
```

One binary, no cgo, about 10 MB stripped.

## Quick start

```sh
patchwork-turbo init        # writes a commented turbo.yaml
patchwork-turbo check       # runs every track, writes nothing, shows what it found
patchwork-turbo once        # runs every track now
patchwork-turbo run         # keeps running each track when it is due
```

Then point Patchwork at turbo's output:

```yaml
# patchwork.yaml
source:
  mode: folder
  path: /data/turbo        # turbo.yaml's `out`
  include: ["**/*.md"]
  slug: path
```

A track that follows one thing (`url`, `data`, `doc`, `deps`) writes `out/<name>.md`, so its slug is the track name. The others write into `out/<name>/`.

## Configuration

```yaml
out: out                   # the folder patchwork watches; keep it on a local disk
timezone: America/Sao_Paulo
user_agent: "patchwork-turbo (+https://example.com/contact)"
host_gap: 5s               # minimum time between two requests to one host

tracks:
  - name: home
    url: https://example.com/
    select: main           # CSS selector for the content (default: main, article, body)
    ignore: ["footer", ".visit-counter", "time"]
    every: 6h

  - name: docs
    sitemap: https://example.com/sitemap.xml
    select: article
    max_pages: 200
    every: 24h

  - name: blog
    feed: https://example.com/feed.xml
    every: 3h

  - name: prices
    data: https://example.com/prices.json

  - name: handbook
    doc: ./handbook.docx

  - name: api
    go: ../my-module

  - name: deps
    deps: ../my-module/go.mod
```

Every track takes `every` (default `6h`, at least `1m`). `TURBO_CONFIG`, `TURBO_OUT` and `TURBO_STATE` override the config path, the output folder and the state file.

## How each converter reads its source

### Web pages

Web pages keep headings, paragraphs, lists, quotes, code blocks, tables, links and images with alt text. Scripts, styles, navigation, sidebars, form controls and hidden elements are dropped, plus whatever `ignore` lists; forms are dropped too (comments, search, newsletter), except one that holds the page itself, with a top-level heading or an article inside, as ASP.NET pages have. Nested lists and quotes indent at most eight levels deep. Whitespace is normalized, so the same text always gives the same bytes. Use `ignore` for parts that change on their own, like dates, visit counters or "related posts".

### Sitemaps

Sitemaps are fetched in full each run (up to 50 MB), but a page is fetched again only when its `<lastmod>` moved. Pages that leave the sitemap leave the folder. A sitemap with no pages on its own host, or one whose pages all fail to load (404s included), is treated as broken: the run fails and the folder stays as it was. When only some pages fail, they keep their last version; if the failure may pass (a timeout, a 5xx) the track is retried 15 minutes later, and if it won't (a 403, a body too large) it waits for its next run. `max_pages` caps how many pages are fetched, never which pages are kept: pages past the cap that are already in the folder stay there, unchanged. A sitemap index is read up to 50 files; past that, nothing is removed, since a missing page may sit in a file that wasn't read. Two URLs that would share a file name, like `/a.html` and `/a`, get two files, and a page keeps its file: a URL that shows up later never takes it. When a URL leaves a complete sitemap, its name is free again, so a page renamed from `/a.html` to `/a` stays `a.md` and its change is a revision.

### Feeds

Feeds become one page per item, named after its link (with the `#fragment`, so `/changelog#v1` and `/changelog#v2` are two pages), or after its id when it has no link. `select` picks part of each item when it matches (an item it misses is converted whole) and `ignore` drops parts of it, as for pages. An item that falls off the feed keeps its last version, since feeds only carry the latest items.

### Data files

Data files become one line per field, sorted: `- plans[Pro].price: 49`. Lists of objects are keyed by `id`, `slug`, `key`, `name` or `title` when one of them is unique, so inserting an element doesn't renumber everything after it. CSV rows are keyed by their first column. Two fields that would share a path (`{"a.b": 1}` next to `{"a": {"b": 2}}`) are both kept, the second as `a.b (2)`, in the same order every run. Remote data files and documents may be up to 20 MB.

### Documents

Documents keep headings (from the style's outline level, so localized style names work), paragraphs, lists and tables. Formatting, images, text boxes, comments and footnotes are left out, and so is deleted text in tracked changes, in both formats. A document that is truncated or doesn't parse is an error, never a page with most of its text gone.

### Go modules

Go modules become one page per package with every exported declaration, its doc comment and no function bodies; git already keeps the bodies. Files are chosen for linux/amd64 so every machine writes the same pages. Main packages, tests, `testdata`, `vendor` and nested modules are skipped. Removing a declaration or changing a function's signature is reported as breaking. Constants and variables are compared name by name, so adding `C` to a group `A, B` reads as one name added; in a group that uses `iota`, moving a name changes its value and is reported.

### Dependencies

Dependencies come from `go.mod` (requires, replaces, the Go version and toolchain) or `package.json` (runtime, dev, peer and optional dependencies).

## Politeness

Turbo identifies itself with `user_agent`, follows robots.txt (the group naming the first word of `user_agent`, else `*`), waits `host_gap` between requests to the same host, sends `If-None-Match` and `If-Modified-Since` so unchanged pages cost a 304, honors `Retry-After` on 429 and 503, and refuses page bodies over 2 MB. When robots.txt can't be read because of a server error, the host is skipped for an hour rather than crawled blind. A track that fails waits 15 minutes (or its own `every`, if shorter) before trying again, and a host that asked to slow down fails at once instead of holding up other tracks.

### When a page goes missing

A 404 or 410 keeps the page; a second one in a row removes it, so one bad answer doesn't delete anything. A page that can't be read for any other reason keeps its last version. Each track keeps its own validators, and stores them only after its pages are written: a failed write is redone on the next run. Each page remembers the `select` and `ignore` it was converted with, so after you change them every page is converted again the next time it is read, even if the server would answer 304; a page that can't be read keeps its old version until it can.

## Importing history from the Internet Archive

A `url` track can start with the page's past instead of today:

```sh
patchwork-turbo import-wayback -track home -limit 20 \
  -patchwork /usr/local/bin/patchwork -patchwork-config patchwork.yaml
```

Turbo lists the archive's captures (at most one per day, only when the content changed, the newest `-limit` of them), converts each with the track's selectors and hands them to `patchwork snapshot`, oldest first. Patchwork can only add history before a page's first version, so run this before the track's first `once` or `run`. Add `-dry-run` to see which copies would go in.

An import that stops halfway, because patchwork refused a copy or the archive was slow, can be run again: it remembers the last copy patchwork took and carries on from there. The progress lives next to turbo's state file, in `wayback/<track>`; delete that folder to start over, for example after resetting patchwork's database. `-dry-run` starts from the same progress, so it lists only what a real run would still import.

Copies that fail the same way every time (a body too large, refused by robots.txt, lost by the archive) are skipped. A copy the archive fails to serve stops the import, so a rerun can get it; add `-skip-failed` to skip copies that keep failing. If the page already has history newer than the archive's copies, the import stops and says so: archived copies can only go before a page's first version.

## Limits

- Pages that build their content with JavaScript come out nearly empty. Turbo has no headless browser on purpose; `check` warns when a page has fewer than 30 words.
- PDF isn't read yet: there's no pure-Go text extractor good enough.
- Code tracking reads Go only.
- No AI anywhere. Notes are counts, field names and declaration names.

## License

MIT. See [LICENSE](LICENSE).