Feeds patchwork with web pages, sites, other feeds, data files, documents, Go APIs and dependencies, written in Go
Import history from the Internet Archive, add the README
- import-wayback lists a url track's captures (one per day at most, only
when the bytes changed, the newest ones first), converts each with the
track's selectors and feeds them to `patchwork snapshot` oldest first.
The newest copy stays in the output folder, so the first live run's
change becomes a revision with a note.
- Tracks that follow one thing write out/<name>.md, so their slug is
the track name.
- README: what each track reads, how notes are built, politeness rules,
limits.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@@ -0,0 +1,153 @@+# patchwork-turbo
+
+[Patchwork](https://code.pablomurad.com/patchwork.git) publishes revision feeds for Markdown pages. patchwork-turbo brings everything else to it: web pages, whole sites, other people's feeds, JSON and CSV files, Word and LibreOffice documents, the API of a Go module and a project's dependencies.
+
+It turns each thing it tracks into a Markdown file in a folder, and a normal `patchwork serve` watches that folder. The two programs never import each other. You can open any file turbo wrote and read exactly what Patchwork will compare.
+
+```
+turbo.yaml ──> patchwork-turbo run ──> out/*.md ──> patchwork serve ──> Atom, JSON Feed, API, text
+```
+
+> Status: early development. The JSON contract and the file layout may change before 1.0.
+
+## What it tracks
+
+| Track | Source | One page per | What a revision note says |
+| --- | --- | --- | --- |
+| `url` | a web page | the page | `Changed on 2026-10-24 14:05 -03: 1 block added.` |
+| `sitemap` | a sitemap.xml or sitemap index | listed page | the same, per page |
+| `feed` | an RSS or Atom feed | item | the same, per item |
+| `data` | JSON, YAML or CSV, local or remote | file | `plans[Pro].price 49 → 59; added plans[Max].price` |
+| `doc` | DOCX or ODT, local or remote | document | `2 blocks changed.` |
+| `go` | a Go module on disk | package | `Breaking: changed func New, removed func Old. Added: type Option` |
+| `deps` | go.mod or package.json | file | `Updated golang.org/x/net v0.60.0 → v0.61.0; added golang.org/x/mod v0.42.0` |
+
+Patchwork publishes a revision when a page's `revision` note changes. Turbo writes a file only when the converted content changed, and gives it a new note naming the time and the change, so every real change becomes one revision and a page that didn't change costs nothing. The first version of a page is Patchwork's silent baseline, as always.
+
+## Install
+
+```sh
+git clone https://code.pablomurad.com/patchwork-turbo.git
+cd patchwork-turbo
+go build -o patchwork-turbo ./cmd/patchwork-turbo
+```
+
+One binary, no cgo, about 10 MB stripped.
+
+## Quick start
+
+```sh
+patchwork-turbo init # writes a commented turbo.yaml
+patchwork-turbo check # runs every track, writes nothing, shows what it found
+patchwork-turbo once # runs every track now
+patchwork-turbo run # keeps running each track when it is due
+```
+
+Then point Patchwork at turbo's output:
+
+```yaml
+# patchwork.yaml
+source:
+ mode: folder
+ path: /data/turbo # turbo.yaml's `out`
+ include: ["**/*.md"]
+ slug: path
+```
+
+A track that follows one thing (`url`, `data`, `doc`, `deps`) writes `out/<name>.md`, so its slug is the track name. The others write into `out/<name>/`.
+
+## Configuration
+
+```yaml
+out: out # the folder patchwork watches; keep it on a local disk
+timezone: America/Sao_Paulo
+user_agent: "patchwork-turbo (+https://example.com/contact)"
+host_gap: 5s # minimum time between two requests to one host
+
+tracks:
+ - name: home
+ url: https://example.com/
+ select: main # CSS selector for the content (default: main, article, body)
+ ignore: ["footer", ".visit-counter", "time"]
+ every: 6h
+
+ - name: docs
+ sitemap: https://example.com/sitemap.xml
+ select: article
+ max_pages: 200
+ every: 24h
+
+ - name: blog
+ feed: https://example.com/feed.xml
+ every: 3h
+
+ - name: prices
+ data: https://example.com/prices.json
+
+ - name: handbook
+ doc: ./handbook.docx
+
+ - name: api
+ go: ../my-module
+
+ - name: deps
+ deps: ../my-module/go.mod
+```
+
+Every track takes `every` (default `6h`, at least `1m`). `TURBO_CONFIG`, `TURBO_OUT` and `TURBO_STATE` override the config path, the output folder and the state file.
+
+## How each converter reads its source
+
+### Web pages
+
+Web pages keep headings, paragraphs, lists, quotes, code blocks, tables, links and images with alt text. Scripts, styles, navigation, forms, sidebars and hidden elements are dropped, plus whatever `ignore` lists. Whitespace is normalized, so the same text always gives the same bytes. Use `ignore` for parts that change on their own, like dates, visit counters or "related posts".
+
+### Sitemaps
+
+Sitemaps are fetched in full each run, but a page is fetched again only when its `<lastmod>` moved. Pages that leave the sitemap leave the folder, and an empty sitemap is treated as an error rather than as a site with no pages.
+
+### Feeds
+
+Feeds become one page per item, keyed by the item's id or link. An item that falls off the feed keeps its last version, since feeds only carry the latest items.
+
+### Data files
+
+Data files become one line per field, sorted: `- plans[Pro].price: 49`. Lists of objects are keyed by `id`, `slug`, `key`, `name` or `title` when one of them is unique, so inserting an element doesn't renumber everything after it. CSV rows are keyed by their first column.
+
+### Documents
+
+Documents keep headings (from the style's outline level, so localized style names work), paragraphs, lists and tables. Formatting, images, comments and footnotes are left out. Deleted text in tracked changes is ignored.
+
+### Go modules
+
+Go modules become one page per package with every exported declaration, its doc comment and no function bodies; git already keeps the bodies. Files are chosen for linux/amd64 so every machine writes the same pages. Main packages, tests, `testdata`, `vendor` and nested modules are skipped. Removing a declaration or changing a function's signature is reported as breaking.
+
+### Dependencies
+
+Dependencies come from `go.mod` (requires, replaces, the Go version and toolchain) or `package.json` (runtime, dev, peer and optional dependencies).
+
+## Politeness
+
+Turbo identifies itself with `user_agent`, follows robots.txt (a group naming `patchwork-turbo`, else `*`), waits `host_gap` between requests to the same host, sends `If-None-Match` and `If-Modified-Since` so unchanged pages cost a 304, honors `Retry-After` on 429 and 503, and refuses bodies over 2 MB. When robots.txt can't be read because of a server error, the host is skipped for an hour rather than crawled blind.
+
+## Importing history from the Internet Archive
+
+A `url` track can start with the page's past instead of today:
+
+```sh
+patchwork-turbo import-wayback -track home -limit 20 \
+ -patchwork /usr/local/bin/patchwork -patchwork-config patchwork.yaml
+```
+
+Turbo lists the archive's captures (at most one per day, only when the content changed, the newest `-limit` of them), converts each with the track's selectors and hands them to `patchwork snapshot`, oldest first. Patchwork can only add history before a page's first version, so run this before the track's first `once` or `run`. Add `-dry-run` to see which copies would go in.
+
+## Limits
+
+- Pages that build their content with JavaScript come out nearly empty. Turbo has no headless browser on purpose; `check` warns when a page has fewer than 30 words.
+- PDF isn't read yet: there's no pure-Go text extractor good enough.
+- Code tracking reads Go only.
+- No AI anywhere. Notes are counts, field names and declaration names.
+
+## License
+
+MIT. See [LICENSE](LICENSE).
@@ -36,6 +36,8 @@ init write a commented turbo.yaml once run every track (or one, with -track) now
run keep running each track when it is due
check run tracks without writing anything, and report what they find
+ import-wayback
+ build a url track's history from the Internet Archive
version print the version
Commands read turbo.yaml from the current directory, or the file named by
@@ -63,6 +65,8 @@ case "check": err = cmdOnce(ctx, args, true)
case "run":
err = cmdRun(ctx, args)
+ case "import-wayback":
+ err = cmdImportWayback(ctx, args)
case "version", "-version", "--version":
fmt.Fprintln(stdout, "patchwork-turbo", version)
case "help", "-h", "-help", "--help":
@@ -199,11 +203,19 @@ paths = append(paths, p) }
sort.Strings(paths)
for _, p := range paths {
- fmt.Fprintf(stdout, " %s: %s\n", p, rep.Notes[p])
+ name := p
+ if name == out.Single {
+ name = t.Name
+ }
+ fmt.Fprintf(stdout, " %s: %s\n", name, rep.Notes[p])
}
if dryRun {
for _, p := range res.Pages {
- fmt.Fprintf(stdout, " page %s %q %d words\n", p.Path, p.Title, len(strings.Fields(p.Body)))
+ name := p.Path
+ if name == out.Single {
+ name = t.Name
+ }
+ fmt.Fprintf(stdout, " page %s %q %d words\n", name, p.Title, len(strings.Fields(p.Body)))
}
}
for _, w := range res.Warnings {
@@ -34,6 +34,11 @@ // Compare. Summary func(before, after string) (summary, kind string)
}
+// Single is the path of a track's only page, for tracks that follow one
+// thing: it is written as <track>.md next to the other tracks' folders, so
+// its patchwork slug is just the track name.
+const Single = ""
+
// Writer writes the pages of every track under Dir.
type Writer struct {
Dir string
@@ -63,11 +68,11 @@ rep := Report{Notes: map[string]string{}, Unchanged: len(keepPaths)} dir := filepath.Join(w.Dir, track)
keep := map[string]bool{}
for _, p := range keepPaths {
- keep[w.file(track, p)] = true
+ keep[w.File(track, p)] = true
}
written := map[string]bool{}
for _, p := range pages {
- file := w.file(track, p.Path)
+ file := w.File(track, p.Path)
if written[file] {
continue // two URLs that slugify alike: the first one wins
}
@@ -91,22 +96,38 @@ return rep, nil }
n, err := w.removeOthers(dir, keep)
rep.Removed = n
+ if err != nil {
+ return rep, err
+ }
+ if single := w.File(track, Single); !keep[single] && w.Has(track, Single) {
+ rep.Removed++
+ if !w.DryRun {
+ err = os.Remove(single)
+ }
+ }
return rep, err
}
// Has reports whether a page's file exists.
func (w *Writer) Has(track, path string) bool {
- _, err := os.Stat(w.file(track, path))
+ _, err := os.Stat(w.File(track, path))
return err == nil
}
// HasAny reports whether a track has written any page yet.
func (w *Writer) HasAny(track string) bool {
+ if w.Has(track, Single) {
+ return true
+ }
entries, err := os.ReadDir(filepath.Join(w.Dir, track))
return err == nil && len(entries) > 0
}
-func (w *Writer) file(track, path string) string {
+// File is where a page of a track is written.
+func (w *Writer) File(track, path string) string {
+ if path == Single {
+ return filepath.Join(w.Dir, track+".md")
+ }
return filepath.Join(w.Dir, track, filepath.FromSlash(path)+".md")
}
@@ -19,9 +19,9 @@ // thinPage is the word count under which a page probably needs JavaScript // to show its content, or the selector missed it.
const thinPage = 30
-// page tracks one URL. Its file is always named index.
+// page tracks one URL, as the track's single page.
func (r *Runner) page(ctx context.Context, t config.Track) (Result, error) {
- const path = "index"
+ const path = out.Single
var res Result
resp, err := r.Fetch.Get(ctx, t.URL, r.Writer.Has(t.Name, path))
switch {
@@ -0,0 +1,73 @@+// Package wayback lists the Internet Archive's captures of a page, so its
+// history from before turbo ran can be imported.
+package wayback
+
+import (
+ "context"
+ "encoding/json"
+ "fmt"
+ "net/url"
+ "strconv"
+ "time"
+
+ "git.pablomurad.com/pablo/patchwork-turbo/internal/fetch"
+)
+
+// Capture is one archived copy of a page.
+type Capture struct {
+ At time.Time
+ Original string
+ // URL serves the copy exactly as the site sent it, without the
+ // archive's toolbar or rewritten links.
+ URL string
+}
+
+// CDX is the archive's capture index.
+const CDX = "https://web.archive.org/cdx/search/cdx"
+
+// Captures lists successful captures of u, oldest first, at most one per
+// day and only when the content changed since the previous one. With a
+// limit, the newest captures are the ones kept.
+func Captures(ctx context.Context, c *fetch.Client, u string, from int, limit int) ([]Capture, error) {
+ q := url.Values{}
+ q.Set("url", u)
+ q.Set("output", "json")
+ q.Set("fl", "timestamp,original")
+ q.Set("filter", "statuscode:200")
+ q.Add("collapse", "timestamp:8") // one per day
+ q.Add("collapse", "digest") // and only when the bytes changed
+ if from > 0 {
+ q.Set("from", strconv.Itoa(from))
+ }
+ if limit > 0 {
+ q.Set("limit", strconv.Itoa(-limit)) // negative: the last ones
+ }
+ resp, err := c.Get(ctx, CDX+"?"+q.Encode(), false)
+ if err != nil {
+ return nil, err
+ }
+ return parse(resp.Body)
+}
+
+// parse reads the CDX JSON output: a header row, then one row per capture.
+func parse(body []byte) ([]Capture, error) {
+ var rows [][]string
+ if len(body) == 0 {
+ return nil, nil
+ }
+ if err := json.Unmarshal(body, &rows); err != nil {
+ return nil, fmt.Errorf("capture index: %w", err)
+ }
+ var caps []Capture
+ for i, row := range rows {
+ if i == 0 || len(row) < 2 {
+ continue // the header
+ }
+ at, err := time.Parse("20060102150405", row[0])
+ if err != nil {
+ return nil, fmt.Errorf("capture index: timestamp %q: %w", row[0], err)
+ }
+ caps = append(caps, Capture{At: at, Original: row[1], URL: "https://web.archive.org/web/" + row[0] + "id_/" + row[1]})
+ }
+ return caps, nil
+}
