patchwork-turbo

Patchwork publishes revision feeds for Markdown pages. patchwork-turbo brings everything else to it: web pages, whole sites, other people’s feeds, JSON and CSV files, Word and LibreOffice documents, the API of a Go module and a project’s dependencies.

It turns each thing it tracks into a Markdown file in a folder, and a normal patchwork serve watches that folder. The two programs never import each other. You can open any file turbo wrote and read exactly what Patchwork will compare.

turbo.yaml ──> patchwork-turbo run ──> out/*.md ──> patchwork serve ──> Atom, JSON Feed, API, text

Status: early development. The JSON contract and the file layout may change before 1.0.

What it tracks

Track	Source	One page per	What a revision note says
url	a web page	the page	Changed on 2026-10-24 14:05 -03: 1 block added.
sitemap	a sitemap.xml or sitemap index	listed page	the same, per page
feed	an RSS or Atom feed	item	the same, per item
data	JSON, YAML or CSV, local or remote	file	plans[Pro].price 49 → 59; added plans[Max].price
doc	DOCX or ODT, local or remote	document	2 blocks changed.
go	a Go module on disk	package	Breaking: changed func New, removed func Old. Added: type Option
deps	go.mod or package.json	file	Updated golang.org/x/net v0.60.0 → v0.61.0; added golang.org/x/mod v0.42.0

Patchwork publishes a revision when a page’s revision note changes. Turbo writes a file only when the converted content changed, and gives it a new note naming the time and the change, so every real change becomes one revision and a page that didn’t change costs nothing. The first version of a page is Patchwork’s silent baseline, as always.

Install

git clone https://code.pablomurad.com/patchwork-turbo.git
cd patchwork-turbo
go build -o patchwork-turbo ./cmd/patchwork-turbo

One binary, no cgo, about 10 MB stripped.

Quick start

patchwork-turbo init        # writes a commented turbo.yaml
patchwork-turbo check       # runs every track, writes nothing, shows what it found
patchwork-turbo once        # runs every track now
patchwork-turbo run         # keeps running each track when it is due

Then point Patchwork at turbo’s output:

# patchwork.yaml
source:
  mode: folder
  path: /data/turbo        # turbo.yaml's `out`
  include: ["**/*.md"]
  slug: path

A track that follows one thing (url, data, doc, deps) writes out/<name>.md, so its slug is the track name. The others write into out/<name>/.

Configuration

out: out                   # the folder patchwork watches; keep it on a local disk
timezone: America/Sao_Paulo
user_agent: "patchwork-turbo (+https://example.com/contact)"
host_gap: 5s               # minimum time between two requests to one host

tracks:
  - name: home
    url: https://example.com/
    select: main           # CSS selector for the content (default: main, article, body)
    ignore: ["footer", ".visit-counter", "time"]
    every: 6h

  - name: docs
    sitemap: https://example.com/sitemap.xml
    select: article
    max_pages: 200
    every: 24h

  - name: blog
    feed: https://example.com/feed.xml
    every: 3h

  - name: prices
    data: https://example.com/prices.json

  - name: handbook
    doc: ./handbook.docx

  - name: api
    go: ../my-module

  - name: deps
    deps: ../my-module/go.mod

Every track takes every (default 6h, at least 1m). TURBO_CONFIG, TURBO_OUT and TURBO_STATE override the config path, the output folder and the state file.

How each converter reads its source

Web pages

Web pages keep headings, paragraphs, lists, quotes, code blocks, tables, links and images with alt text. Scripts, styles, navigation, forms, sidebars and hidden elements are dropped, plus whatever ignore lists. Whitespace is normalized, so the same text always gives the same bytes. Use ignore for parts that change on their own, like dates, visit counters or “related posts”.

Sitemaps

Sitemaps are fetched in full each run, but a page is fetched again only when its <lastmod> moved. Pages that leave the sitemap leave the folder, and an empty sitemap is treated as an error rather than as a site with no pages.

Feeds

Feeds become one page per item, keyed by the item’s id or link. An item that falls off the feed keeps its last version, since feeds only carry the latest items.

Data files

Data files become one line per field, sorted: - plans[Pro].price: 49. Lists of objects are keyed by id, slug, key, name or title when one of them is unique, so inserting an element doesn’t renumber everything after it. CSV rows are keyed by their first column.

Documents

Documents keep headings (from the style’s outline level, so localized style names work), paragraphs, lists and tables. Formatting, images, comments and footnotes are left out. Deleted text in tracked changes is ignored.

Go modules

Go modules become one page per package with every exported declaration, its doc comment and no function bodies; git already keeps the bodies. Files are chosen for linux/amd64 so every machine writes the same pages. Main packages, tests, testdata, vendor and nested modules are skipped. Removing a declaration or changing a function’s signature is reported as breaking.

Dependencies

Dependencies come from go.mod (requires, replaces, the Go version and toolchain) or package.json (runtime, dev, peer and optional dependencies).

Politeness

Turbo identifies itself with user_agent, follows robots.txt (a group naming patchwork-turbo, else *), waits host_gap between requests to the same host, sends If-None-Match and If-Modified-Since so unchanged pages cost a 304, honors Retry-After on 429 and 503, and refuses bodies over 2 MB. When robots.txt can’t be read because of a server error, the host is skipped for an hour rather than crawled blind.

Importing history from the Internet Archive

A url track can start with the page’s past instead of today:

patchwork-turbo import-wayback -track home -limit 20 \
  -patchwork /usr/local/bin/patchwork -patchwork-config patchwork.yaml

Turbo lists the archive’s captures (at most one per day, only when the content changed, the newest -limit of them), converts each with the track’s selectors and hands them to patchwork snapshot, oldest first. Patchwork can only add history before a page’s first version, so run this before the track’s first once or run. Add -dry-run to see which copies would go in.

Limits

• Pages that build their content with JavaScript come out nearly empty. Turbo has no headless browser on purpose; check warns when a page has fewer than 30 words.
• PDF isn’t read yet: there’s no pure-Go text extractor good enough.
• Code tracking reads Go only.
• No AI anywhere. Notes are counts, field names and declaration names.

License

MIT. See LICENSE.