indiecheck

@@ -13,7 +13,9 @@ **IndieWeb.** The representative h-card, who wrote each h-entry (following the authorship rules: the entry, then the parent h-feed, then rel=author), rel=me, Webmention, Micropub, Microsub, IndieAuth and WebSub. Endpoints are read from the HTML and from the HTTP `Link` header, which is where some sites keep them. 
 **Meta tags.** Title, description, lang, charset, viewport, canonical, Open Graph, Twitter cards, `fediverse:creator`, theme-color, icons, the manifest and feeds. Declared feeds get requested, so a feed that answers 401 or comes back as `text/plain` gets flagged.
 
-**Advice.** Errors break parsing, like a `dt-published` that isn't a date or a class called `hcard`. Warnings leave out something tools look for, like an h-entry without an author. Tips are nice to have. Each one says where the problem is and comes with a snippet.
+**The site around the page.** robots.txt (missing, or blocking every crawler), the sitemaps it declares or the one at `/sitemap.xml`, whether the page ends up on https, whether rel=canonical points to another host, and whether both `www.` and the bare domain serve the site without one redirecting to the other. These checks only run on a fetched URL, not on pasted HTML.
+
+**Advice.** Errors break parsing, like a `dt-published` that isn't a date or a class called `hcard`. Warnings leave out something tools look for, like an h-entry without an author. Tips are nice to have. Each one says where the problem is and comes with a snippet. When the page's `generator` meta tag says WordPress, Hugo or Eleventy, the snippets for a missing h-card, rel=me, feed or Webmention endpoint say where the code goes in that tool, or which plugin does it for you.
 
 ## Running it
 
@@ -45,7 +47,30 @@ -d "base=https://example.com/post" \   -d "format=json"
 ```
 
-The canonical mf2 parse is under `mf2`. The rest of the response is what the page shows: `summary`, `indieweb`, `rels`, `metadata`, `feed-checks` and `advice`. A failed fetch answers 422 with `{"error": "..."}`.
+The canonical mf2 parse is under `mf2`. The rest of the response is what the page shows: `summary`, `indieweb`, `rels`, `metadata`, `feed-checks`, `robots`, `domain` and `advice`. `robots` and `domain` are `null` for pasted HTML. A failed fetch answers 422 with `{"error": "..."}`.
+
+## Markdown
+
+`format=markdown` returns the same report as Markdown, served as `text/markdown`: the fetch facts, the counts, a short summary, then every finding grouped into errors, warnings and tips, with its fix in a code block. It's meant for pasting into an issue or keeping next to the site's source.
+
+```bash
+curl "http://127.0.0.1:8123/?url=https://tantek.com/&format=markdown" > report.md
+```
+
+## In CI
+
+Add `fail=errors` and the response comes back as HTTP 412 when the page has any error. `fail=warnings` does the same for errors or warnings. It works with every format, so `curl -f` is enough to break a build:
+
+```bash
+curl -sf "https://indie.pablomurad.com/?url=https://example.com/&fail=errors" -o /dev/null
+```
+
+For finer rules, read the counts from the JSON:
+
+```bash
+curl -s "https://indie.pablomurad.com/?url=https://example.com/&format=json" \
+  | jq -e '.summary.errors == 0 and .summary.warnings <= 2'
+```
 
 ## Fetching
 
@@ -55,13 +80,20 @@ - only http and https, on ports 80, 443, 8080 and 8443 - no private, loopback, link-local or otherwise reserved addresses, checked again after every redirect, and cURL is pinned to the IP that passed the check
 - at most 5 redirects, 2 MB and 20 seconds per page
 - up to 4 feeds per page, with 8 seconds each (only the headers are read)
+- robots.txt at 64 KB and 8 seconds, then up to 2 of the sitemaps it declares, or `/sitemap.xml` when it declares none (headers only)
+- one request to the `www.` or bare-domain twin of the host, when there is one (headers only)
+
+That's at most 10 requests for one check.
 
 A site on your own network can't be fetched. Paste its HTML instead.
 
 ## Where things are
 
-- `public/index.php` takes the request and answers with a page or JSON
+- `public/index.php` takes the request and answers with a page, JSON or Markdown
 - `src/Fetcher.php` does the HTTP requests and the address checks
+- `src/Report.php` runs a check, including the robots.txt, sitemap and domain requests
+- `src/Robots.php` reads robots.txt
+- `src/Markdown.php` writes the Markdown report
 - `src/Mf2Parser.php` extends php-mf2 so that top-level `geo`, `hreview-aggregate` and `hnews` get parsed, and marks items that came from classic markup
 - `src/Microformats.php`, `src/Rels.php` and `src/Metadata.php` read the page
 - `src/Advice.php` has every rule behind the advice list

@@ -2,6 +2,7 @@ <?php 
 use Indiechecker\FetchError;
 use Indiechecker\Fetcher;
+use Indiechecker\Markdown;
 use Indiechecker\Report;
 
 use function Indiechecker\view;
@@ -13,6 +14,7 @@ $fetcher = new Fetcher();
 $isPost = $_SERVER['REQUEST_METHOD'] === 'POST';
 $format = $isPost ? ($_POST['format'] ?? '') : ($_GET['format'] ?? '');
+$fail = $isPost ? ($_POST['fail'] ?? '') : ($_GET['fail'] ?? '');
 $url = trim($_GET['url'] ?? '');
 $html = $isPost ? (string) ($_POST['html'] ?? '') : null;
 $base = $isPost ? (string) ($_POST['base'] ?? '') : '';
@@ -32,6 +34,14 @@ header('X-Content-Type-Options: nosniff');
 header('Referrer-Policy: no-referrer');
 
+$status = match (true) {
+    $error !== null => 422,
+    $report === null => 400,
+    $fail === 'errors' && $report->advice->count('error') > 0 => 412,
+    $fail === 'warnings' && $report->advice->count('error') + $report->advice->count('warning') > 0 => 412,
+    default => 200,
+};
+
 if ($format === 'json') {
     header('Content-Type: application/json; charset=utf-8');
     header('Access-Control-Allow-Origin: *');
@@ -41,16 +51,30 @@ $error !== null => ['error' => $error],         $report !== null => $report->toArray(),
         default => ['usage' => 'GET /?url=https://example.com/&format=json, or POST html (and optionally base) with format=json.'],
     };
-    http_response_code($error !== null ? 422 : ($report === null ? 400 : 200));
+    http_response_code($status);
     echo json_encode($body, JSON_PRETTY_PRINT | JSON_UNESCAPED_SLASHES | JSON_UNESCAPED_UNICODE | JSON_INVALID_UTF8_SUBSTITUTE);
 
     return;
 }
 
+if ($format === 'markdown') {
+    header('Content-Type: text/markdown; charset=utf-8');
+    header('Access-Control-Allow-Origin: *');
+
+    http_response_code($status);
+    echo match (true) {
+        $error !== null => Markdown::error($error),
+        $report !== null => Markdown::render($report),
+        default => "# indiechecker\n\nGET /?url=https://example.com/&format=markdown, or POST html (and optionally base) with format=markdown.\n",
+    };
+
+    return;
+}
+
 header("Content-Security-Policy: default-src 'none'; style-src 'self'; font-src 'self'; img-src * data:; script-src 'self'; form-action 'self'; base-uri 'none'; frame-ancestors 'none'");
 
-if ($error !== null) {
-    http_response_code(422);
+if ($error !== null || $report !== null) {
+    http_response_code($status);
 }
 
 echo view('page', [

@@ -8,6 +8,29 @@ private const LEVELS = ['error' => 0, 'warning' => 1, 'tip' => 2];     private const DURATIONS = ['duration'];
     private const LOOSE_URLS = ['email', 'tel', 'impp', 'key', 'uid'];
 
+    private const GENERATOR_FIXES = [
+        'no-card' => [
+            'wordpress' => "<!-- WordPress: the “IndieWeb” plugin adds an h-card widget and author h-card for you. By hand, in your theme: -->\n%card%",
+            'hugo' => "<!-- Hugo: put this in layouts/partials/h-card.html and include it from your base template with {{ partial \"h-card.html\" . }} -->\n%card%",
+            'eleventy' => "<!-- Eleventy: put this in _includes/h-card.njk and include it from your base layout with {% include \"h-card.njk\" %} -->\n%card%",
+        ],
+        'no-rel-me' => [
+            'wordpress' => "<!-- WordPress: the “IndieWeb” plugin adds rel=me links from your user profile. By hand, in your theme: -->\n<a rel=\"me\" href=\"https://github.com/you\">GitHub</a>\n<a rel=\"me\" href=\"https://mastodon.social/@you\">Mastodon</a>",
+            'hugo' => "<!-- Hugo: list your profiles under [params] in hugo.toml, then in a partial: -->\n{{ range .Site.Params.relme }}<a rel=\"me\" href=\"{{ . }}\">{{ . }}</a>{{ end }}",
+            'eleventy' => "<!-- Eleventy: list your profiles in _data/site.json as \"relme\": [...], then in your layout: -->\n{% for url in site.relme %}<a rel=\"me\" href=\"{{ url }}\">{{ url }}</a>{% endfor %}",
+        ],
+        'no-feed' => [
+            'wordpress' => "<!-- WordPress makes /feed/ for you. If the link is missing, your theme removed it; add this to header.php: -->\n<link rel=\"alternate\" type=\"application/rss+xml\" title=\"RSS\" href=\"<?php bloginfo('rss2_url'); ?>\">",
+            'hugo' => "<!-- Hugo builds /index.xml by default. In layouts/partials/head.html: -->\n{{ with .OutputFormats.Get \"rss\" }}<link rel=\"alternate\" type=\"application/rss+xml\" title=\"{{ $.Site.Title }}\" href=\"{{ .Permalink }}\">{{ end }}",
+            'eleventy' => "<!-- Eleventy: the official @11ty/eleventy-plugin-rss plugin builds the feed. Then in your base layout: -->\n<link rel=\"alternate\" type=\"application/atom+xml\" title=\"Feed\" href=\"/feed.xml\">",
+        ],
+        'no-webmention' => [
+            'wordpress' => "<!-- WordPress: the “Webmention” plugin adds the endpoint and receives mentions on your own site. Or point to a hosted service in header.php: -->\n<link rel=\"webmention\" href=\"https://webmention.io/example.com/webmention\">",
+            'hugo' => "<!-- Hugo: in layouts/partials/head.html -->\n<link rel=\"webmention\" href=\"https://webmention.io/{{ .Site.BaseURL | replaceRE \"^https?://([^/]+).*\" \"$1\" }}/webmention\">",
+            'eleventy' => "<!-- Eleventy: in your base layout -->\n<link rel=\"webmention\" href=\"https://webmention.io/example.com/webmention\">",
+        ],
+    ];
+
     private array $items = [];
 
     public function __construct(
@@ -15,12 +38,16 @@ private Microformats $microformats,         private Rels $rels,
         private Metadata $metadata,
         private array $feedChecks,
+        private ?array $robots,
+        private ?array $domain,
+        private ?string $generator,
         private ?string $pageUrl,
         private ?string $headerCharset,
     ) {
         $this->checkMicroformats();
         $this->checkIndieWeb();
         $this->checkMetadata();
+        $this->checkSite();
     }
 
     public function all(): array
@@ -51,7 +78,7 @@         if ($mf->isEmpty() && !$mf->classic) {
             $this->add('no-mf', 'warning', 'Microformats', 'No microformats on this page',
                 'Nothing here is marked up with microformats, so readers, Webmention receivers and IndieWeb tools can’t tell who wrote it or what it is. An h-card is the usual first step.',
-                $this->cardSnippet());
+                $this->fixFor('no-card', $this->cardSnippet()));
         }
 
         foreach ($mf->classic as $class => $found) {
@@ -259,7 +286,7 @@         if (!$mf->isEmpty() && !$cards) {
             $this->add('no-card', 'warning', 'IndieWeb', 'No h-card on the page',
                 'An h-card says who is behind the site. IndieWeb tools use it to show your name and photo next to replies and mentions.',
-                $this->cardSnippet());
+                $this->fixFor('no-card', $this->cardSnippet()));
         } elseif ($cards && $mf->representativeCard === null && $this->pageUrl !== null) {
             $this->add('no-representative-card', 'warning', 'IndieWeb', 'None of the h-cards represents this page',
                 'Tools look for a representative h-card: one whose u-url is also a rel=me link, or whose u-url and u-uid both equal the page URL. Mark your own h-card that way.',
@@ -269,13 +296,13 @@         if (!$this->rels->has('me')) {
             $this->add('no-rel-me', 'tip', 'IndieWeb', 'No rel=me links',
                 'rel=me links connect this site to your other profiles. They power IndieLogin and the verified links on Mastodon.',
-                "<a rel=\"me\" href=\"https://github.com/you\">GitHub</a>\n<a rel=\"me\" href=\"https://mastodon.social/@you\">Mastodon</a>");
+                $this->fixFor('no-rel-me', "<a rel=\"me\" href=\"https://github.com/you\">GitHub</a>\n<a rel=\"me\" href=\"https://mastodon.social/@you\">Mastodon</a>"));
         }
 
         if (!$this->rels->has('webmention')) {
             $this->add('no-webmention', 'tip', 'IndieWeb', 'No Webmention endpoint',
                 'Without one, other sites can’t notify you when they reply to or mention your posts. A hosted service like webmention.io works well.',
-                '<link rel="webmention" href="https://webmention.io/example.com/webmention">');
+                $this->fixFor('no-webmention', '<link rel="webmention" href="https://webmention.io/example.com/webmention">'));
         }
 
         $authorization = $this->rels->has('authorization_endpoint');
@@ -349,13 +376,77 @@ $this->add('no-touch-icon', 'tip', 'Metadata', 'No apple-touch-icon', 'iOS and some Android launchers use it for home screen shortcuts.', '<link rel="apple-touch-icon" href="/apple-touch-icon.png">');         }
 
         if (!$meta->feeds && !$this->microformats->ofType('h-feed') && !$this->microformats->ofType('h-entry')) {
-            $this->add('no-feed', 'tip', 'Metadata', 'No feed', 'No RSS, Atom, JSON Feed or h-feed found. Readers need one to follow the site.', '<link rel="alternate" type="application/rss+xml" title="RSS" href="/feed.xml">');
+            $this->add('no-feed', 'tip', 'Metadata', 'No feed', 'No RSS, Atom, JSON Feed or h-feed found. Readers need one to follow the site.', $this->fixFor('no-feed', '<link rel="alternate" type="application/rss+xml" title="RSS" href="/feed.xml">'));
         }
         foreach ($this->feedChecks as $check) {
             if (!$check['ok']) {
                 $this->add('broken-feed-' . $check['url'], 'warning', 'Metadata', 'A declared feed has a problem', $check['problem'], null, $check['url']);
             }
         }
+    }
+
+    private function checkSite(): void
+    {
+        if ($this->robots === null && $this->domain === null) {
+            return;
+        }
+
+        $host = $this->pageUrl !== null ? strtolower((string) parse_url($this->pageUrl, PHP_URL_HOST)) : '';
+        $domain = $this->domain;
+
+        if ($domain !== null) {
+            if (!$domain['https']) {
+                $this->add('not-https', 'warning', 'Site', 'Served over plain http',
+                    'The page’s final address is http, so browsers mark it as not secure and anyone on the network can change it in transit. Webmention and IndieAuth services expect https. Let’s Encrypt certificates are free, and most hosts turn them on with one setting.');
+            }
+            if ($domain['canonical-matches'] === false) {
+                $this->add('canonical-elsewhere', 'warning', 'Site', 'canonical points to a different host',
+                    "The canonical URL is on “{$domain['canonical-host']}”, but this page lives on “{$host}”. Search engines will credit the other host, and tools that follow canonical links will leave this one. If this is the real address, point canonical here.",
+                    sprintf('<link rel="canonical" href="%s">', $this->pageUrl));
+            }
+            $variant = $domain['variant'];
+            if ($variant !== null && !$variant['redirects-to-canonical'] && $variant['status'] < 400) {
+                $this->add('www-duplicate', 'warning', 'Site', "Both {$variant['host']} and {$host} serve the site",
+                    "https://{$variant['host']}/ answered HTTP {$variant['status']} without redirecting to {$host}, so the site exists at two addresses. Links, webmentions and search credit get split between them. Redirect one to the other with a 301.");
+            }
+        }
+
+        $robots = $this->robots;
+        if ($robots === null || $robots['unavailable']) {
+            return;
+        }
+
+        if (!$robots['found']) {
+            $this->add('no-robots', 'tip', 'Site', 'No robots.txt',
+                "The site answers HTTP {$robots['status']} for /robots.txt. It isn’t required, but crawlers look there first for your sitemap and your crawling preferences.",
+                sprintf("User-agent: *\nAllow: /\n\nSitemap: https://%s/sitemap.xml", $host ?: 'example.com'), $robots['url']);
+        }
+        if ($robots['blocks-all']) {
+            $this->add('robots-blocks-all', 'warning', 'Site', 'robots.txt blocks every crawler',
+                '“User-agent: *” with “Disallow: /” tells search engines and feed crawlers to stay away from the whole site. If that isn’t on purpose, remove the rule or narrow it to the paths you want hidden.',
+                null, $robots['url']);
+        }
+        foreach ($robots['sitemaps'] as $check) {
+            if (!$check['ok'] && !$check['unavailable']) {
+                $this->add('broken-sitemap-' . $check['url'], 'warning', 'Site', 'A declared sitemap has a problem', $check['problem'], null, $check['url']);
+            }
+        }
+        $fallback = $robots['fallback-sitemap'];
+        if ($robots['sitemaps'] === [] && $fallback !== null && $fallback['missing']) {
+            $this->add('no-sitemap', 'tip', 'Site', 'No sitemap',
+                'robots.txt doesn’t declare a sitemap and /sitemap.xml doesn’t answer. A sitemap lets search engines find every page, including the ones nothing links to yet.',
+                sprintf('Sitemap: https://%s/sitemap.xml', $host ?: 'example.com'));
+        }
+    }
+
+    private function fixFor(string $key, ?string $default): ?string
+    {
+        $fix = self::GENERATOR_FIXES[$key][$this->generator] ?? null;
+        if ($fix === null) {
+            return $default;
+        }
+
+        return str_replace('%card%', $this->cardSnippet(), $fix);
     }
 
     private function cardSnippet(): string

@@ -0,0 +1,126 @@+<?php
+
+namespace Indiechecker;
+
+final class Markdown
+{
+    private const HEADINGS = ['error' => 'Errors', 'warning' => 'Warnings', 'tip' => 'Tips'];
+    private const WHERE_LIMIT = 8;
+
+    public static function render(Report $report): string
+    {
+        $lines = [];
+        $url = $report->url();
+        $response = $report->response;
+
+        $lines[] = '# indiechecker: ' . ($response === null ? 'pasted HTML' : $url);
+        $lines[] = '';
+        if ($response !== null) {
+            $facts = [
+                'HTTP ' . $response->status,
+                number_format(strlen($response->body) / 1024, 1) . ' KB',
+                number_format($response->milliseconds / 1000, 2) . ' s',
+            ];
+            if ($response->redirects) {
+                $count = count($response->redirects);
+                $facts[] = "{$count} redirect" . ($count > 1 ? 's' : '') . " from {$response->requestedUrl}";
+            }
+            $lines[] = implode(' · ', $facts);
+            $lines[] = '';
+        } elseif ($url !== null) {
+            $lines[] = "Pasted HTML, resolved against {$url}";
+            $lines[] = '';
+        }
+
+        $errors = $report->advice->count('error');
+        $warnings = $report->advice->count('warning');
+        $tips = $report->advice->count('tip');
+        $lines[] = sprintf('**%s · %s · %s**', self::plural($errors, 'error'), self::plural($warnings, 'warning'), self::plural($tips, 'tip'));
+        $lines[] = '';
+
+        $lines[] = '## Summary';
+        $lines[] = '';
+        $counts = $report->microformats->counts();
+        if ($counts) {
+            $types = [];
+            foreach ($counts as $type => $count) {
+                $types[] = "{$count} {$type}";
+            }
+            $lines[] = '- Microformats: ' . implode(', ', $types);
+        } else {
+            $lines[] = '- Microformats: none';
+        }
+        if ($report->generator !== null) {
+            $lines[] = '- Built with: ' . $report->metadata->generator;
+        }
+        if ($report->feedChecks) {
+            $ok = count(array_filter($report->feedChecks, fn (array $check) => $check['ok']));
+            $lines[] = sprintf('- Feeds: %d of %d checked load fine', $ok, count($report->feedChecks));
+        }
+        if ($report->robots !== null) {
+            $robots = $report->robots;
+            $state = match (true) {
+                $robots['unavailable'] => 'couldn’t be checked' . ($robots['status'] !== null ? " (HTTP {$robots['status']})" : ''),
+                !$robots['found'] => 'not found',
+                $robots['blocks-all'] => 'found, blocks every crawler',
+                default => 'found',
+            };
+            $sitemaps = count($robots['sitemaps']);
+            $lines[] = "- robots.txt: {$state}" . ($sitemaps ? ', ' . self::plural($sitemaps, 'sitemap') . ' declared' : '');
+        }
+        $lines[] = '';
+
+        $items = $report->advice->all();
+        foreach (self::HEADINGS as $level => $heading) {
+            $group = array_filter($items, fn (array $item) => $item['level'] === $level);
+            if (!$group) {
+                continue;
+            }
+            $lines[] = "## {$heading}";
+            $lines[] = '';
+            foreach ($group as $item) {
+                $lines[] = "### {$item['title']} ({$item['area']})";
+                $lines[] = '';
+                $lines[] = $item['detail'];
+                $lines[] = '';
+                if ($item['where']) {
+                    $lines[] = 'Where:';
+                    $lines[] = '';
+                    foreach (array_slice($item['where'], 0, self::WHERE_LIMIT) as $where) {
+                        $lines[] = '- ' . self::inlineCode($where);
+                    }
+                    $more = count($item['where']) - self::WHERE_LIMIT;
+                    if ($more > 0) {
+                        $lines[] = "- and {$more} more";
+                    }
+                    $lines[] = '';
+                }
+                if ($item['fix'] !== null) {
+                    $lines[] = '```html';
+                    $lines[] = $item['fix'];
+                    $lines[] = '```';
+                    $lines[] = '';
+                }
+            }
+        }
+
+        return implode("\n", $lines);
+    }
+
+    public static function error(string $message): string
+    {
+        return "# indiechecker\n\nThe check failed: {$message}\n";
+    }
+
+    private static function plural(int $count, string $word): string
+    {
+        return "{$count} {$word}" . ($count === 1 ? '' : 's');
+    }
+
+    private static function inlineCode(string $text): string
+    {
+        $text = preg_replace('/\s+/', ' ', $text);
+
+        return str_contains($text, '`') ? "`` {$text} ``" : "`{$text}`";
+    }
+}

@@ -6,13 +6,18 @@ final class Report {
     private const HTML_TYPES = ['text/html', 'application/xhtml+xml'];
     private const MAX_FEED_CHECKS = 4;
+    private const MAX_SITEMAP_CHECKS = 2;
     private const MAX_PASTE_BYTES = 2_000_000;
+    private const ROBOTS_MAX_BYTES = 65_536;
 
     public readonly Document $document;
     public readonly Microformats $microformats;
     public readonly Rels $rels;
     public readonly Metadata $metadata;
     public readonly array $feedChecks;
+    public readonly ?array $robots;
+    public readonly ?array $domain;
+    public readonly ?string $generator;
     public readonly Advice $advice;
 
     private function __construct(public readonly ?Response $response, string $html, ?string $url, Fetcher $fetcher)
@@ -22,7 +27,10 @@ $this->microformats = new Microformats($this->document);         $this->rels = new Rels($this->microformats->parsed, $response);
         $this->metadata = new Metadata($this->document);
         $this->feedChecks = $this->checkFeeds($fetcher);
-        $this->advice = new Advice($this->microformats, $this->rels, $this->metadata, $this->feedChecks, $url, $response?->charset());
+        $this->robots = $this->checkRobots($fetcher);
+        $this->domain = $this->checkDomain($fetcher);
+        $this->generator = detect_generator($this->metadata->generator);
+        $this->advice = new Advice($this->microformats, $this->rels, $this->metadata, $this->feedChecks, $this->robots, $this->domain, $this->generator, $url, $response?->charset());
     }
 
     public static function fromUrl(string $url, Fetcher $fetcher): self
@@ -128,6 +136,7 @@ 'summary' => [                 'types' => $this->microformats->counts() ?: new \stdClass(),
                 'classic' => $this->microformats->classic ?: new \stdClass(),
                 'representative-h-card' => $this->microformats->representativeCard['card']['properties']['url'][0] ?? null,
+                'generator' => $this->generator,
                 'errors' => $this->advice->count('error'),
                 'warnings' => $this->advice->count('warning'),
                 'tips' => $this->advice->count('tip'),
@@ -142,6 +151,8 @@ 'vote-links' => $this->microformats->voteLinks,             ],
             'metadata' => $this->metadata->toArray(),
             'feed-checks' => $this->feedChecks,
+            'robots' => $this->robots,
+            'domain' => $this->domain,
             'advice' => $this->advice->all(),
         ];
     }
@@ -156,6 +167,132 @@ 'ok' => $endpoint !== null,             'detail' => $endpoint ? 'Found in the ' . $endpoint['source'] . '.' : 'None found.',
             'links' => $endpoint ? [$endpoint['url']] : [],
         ];
+    }
+
+    private function checkRobots(Fetcher $fetcher): ?array
+    {
+        if ($this->response === null) {
+            return null;
+        }
+
+        $origin = $this->origin();
+        $robotsUrl = "{$origin}/robots.txt";
+        $check = ['url' => $robotsUrl, 'found' => false, 'unavailable' => false, 'status' => null, 'blocks-all' => false, 'sitemaps' => [], 'fallback-sitemap' => null];
+
+        try {
+            $small = new Fetcher(maxBytes: self::ROBOTS_MAX_BYTES, timeout: 8, maxRedirects: 3);
+            $response = $small->get($robotsUrl, 'text/plain, */*;q=0.5');
+            $check['status'] = $response->status;
+            if ($response->status < 400) {
+                $check['found'] = true;
+                $parsed = Robots::parse($response->body);
+                $check['blocks-all'] = $parsed['blocks-all'];
+                foreach (array_slice($parsed['sitemaps'], 0, self::MAX_SITEMAP_CHECKS) as $sitemap) {
+                    $check['sitemaps'][] = $this->probeSitemap($fetcher, $sitemap);
+                }
+            } elseif (!self::isMissing($response->status)) {
+                $check['unavailable'] = true;
+            }
+        } catch (FetchError) {
+            $check['unavailable'] = true;
+        }
+
+        if ($check['sitemaps'] === [] && !$check['unavailable']) {
+            try {
+                $fallbackUrl = "{$origin}/sitemap.xml";
+                $response = $fetcher->probe($fallbackUrl, 'application/xml, text/xml;q=0.9, */*;q=0.5');
+                $check['fallback-sitemap'] = [
+                    'url' => $fallbackUrl,
+                    'ok' => $response->status < 400,
+                    'missing' => self::isMissing($response->status),
+                    'status' => $response->status,
+                ];
+            } catch (FetchError) {
+            }
+        }
+
+        return $check;
+    }
+
+    /** A 4xx answer means the file isn’t there. 429 and 5xx only mean the server didn’t answer this time. */
+    private static function isMissing(int $status): bool
+    {
+        return $status >= 400 && $status < 500 && $status !== 429;
+    }
+
+    private function probeSitemap(Fetcher $fetcher, string $url): array
+    {
+        try {
+            $response = $fetcher->probe($url, 'application/xml, text/xml;q=0.9, */*;q=0.5');
+        } catch (FetchError $error) {
+            return ['url' => $url, 'ok' => false, 'unavailable' => false, 'problem' => $error->getMessage()];
+        }
+
+        $unavailable = $response->status >= 400 && !self::isMissing($response->status);
+        $problem = $response->status >= 400 && !$unavailable ? "It answered with HTTP {$response->status}." : null;
+
+        return [
+            'url' => $url,
+            'ok' => $response->status < 400,
+            'unavailable' => $unavailable,
+            'problem' => $problem,
+            'status' => $response->status,
+            'content-type' => $response->contentType(),
+        ];
+    }
+
+    private function checkDomain(Fetcher $fetcher): ?array
+    {
+        if ($this->response === null) {
+            return null;
+        }
+
+        $final = parse_url($this->response->url);
+        $host = strtolower($final['host'] ?? '');
+        $canonicalHost = null;
+        if ($this->metadata->canonical !== null) {
+            $canonicalHost = strtolower((string) parse_url($this->metadata->canonical, PHP_URL_HOST)) ?: null;
+        }
+
+        $alternate = match (true) {
+            str_starts_with($host, 'www.') => substr($host, 4),
+            substr_count($host, '.') === 1 => "www.{$host}",
+            default => null,
+        };
+
+        $variant = null;
+        if ($alternate !== null) {
+            try {
+                $response = $fetcher->probe("https://{$alternate}/", 'text/html, */*;q=0.5');
+                $finalHost = strtolower((string) parse_url($response->url, PHP_URL_HOST));
+                $variant = [
+                    'host' => $alternate,
+                    'status' => $response->status,
+                    'final-host' => $finalHost,
+                    'redirects-to-canonical' => $finalHost === $host,
+                ];
+            } catch (FetchError) {
+                // The variant doesn’t resolve or answer; nothing to report.
+            }
+        }
+
+        return [
+            'https' => ($final['scheme'] ?? '') === 'https',
+            'canonical-host' => $canonicalHost,
+            'canonical-matches' => $canonicalHost === null ? null : $canonicalHost === $host,
+            'variant' => $variant,
+        ];
+    }
+
+    private function origin(): string
+    {
+        $parts = parse_url($this->response->url);
+        $origin = "{$parts['scheme']}://{$parts['host']}";
+        if (isset($parts['port'])) {
+            $origin .= ":{$parts['port']}";
+        }
+
+        return $origin;
     }
 
     private function checkFeeds(Fetcher $fetcher): array

@@ -0,0 +1,65 @@+<?php
+
+namespace Indiechecker;
+
+final class Robots
+{
+    /**
+     * Parses a robots.txt body. Deliberately simple: it only answers
+     * whether every crawler is blocked and which sitemaps are declared.
+     *
+     * @return array{blocks-all: bool, sitemaps: string[]}
+     */
+    public static function parse(string $body): array
+    {
+        $sitemaps = [];
+        $groups = [];
+        $agents = [];
+        $collectingAgents = true;
+
+        foreach (preg_split('/\r\n|\r|\n/', $body) as $line) {
+            $line = trim(preg_replace('/#.*/', '', $line));
+            if ($line === '' || !preg_match('/^([a-z-]+)\s*:\s*(.*)$/i', $line, $match)) {
+                continue;
+            }
+            $field = strtolower($match[1]);
+            $value = trim($match[2]);
+
+            if ($field === 'sitemap') {
+                if (is_absolute_url($value) && !in_array($value, $sitemaps, true)) {
+                    $sitemaps[] = $value;
+                }
+                continue;
+            }
+
+            if ($field === 'user-agent') {
+                if (!$collectingAgents) {
+                    $agents = [];
+                    $collectingAgents = true;
+                }
+                $agents[] = strtolower($value);
+                continue;
+            }
+
+            if ($field === 'disallow' || $field === 'allow') {
+                $collectingAgents = false;
+                foreach ($agents as $agent) {
+                    $groups[$agent][] = [$field, $value];
+                }
+            }
+        }
+
+        $blocksAll = false;
+        foreach ($groups['*'] ?? [] as [$field, $value]) {
+            if ($field === 'allow' && $value !== '') {
+                $blocksAll = false;
+                break;
+            }
+            if ($field === 'disallow' && $value === '/') {
+                $blocksAll = true;
+            }
+        }
+
+        return ['blocks-all' => $blocksAll, 'sitemaps' => $sitemaps];
+    }
+}

@@ -45,6 +45,20 @@     return sprintf('<a href="%s" rel="nofollow noopener noreferrer">%s</a>', e($url), e($text ?? $url));
 }
 
+function detect_generator(?string $generator): ?string
+{
+    if ($generator === null) {
+        return null;
+    }
+
+    return match (true) {
+        stripos($generator, 'wordpress') !== false => 'wordpress',
+        stripos($generator, 'hugo') !== false => 'hugo',
+        stripos($generator, 'eleventy') !== false, stripos($generator, '11ty') !== false => 'eleventy',
+        default => null,
+    };
+}
+
 function looks_like_image(string $url): bool
 {
     return (bool) preg_match('~\.(avif|gif|ico|jpe?g|png|svg|webp)([?#].*)?$~i', $url);

@@ -100,6 +100,44 @@ <?php endforeach ?>         </ul>
     <?php endif ?>
 
+    <?php if ($report->robots !== null): $robots = $report->robots; $fallback = $robots['fallback-sitemap']; ?>
+        <h3>robots.txt and sitemap</h3>
+        <ul class="checklist">
+            <li class="<?= !$robots['found'] ? 'is-unknown' : ($robots['blocks-all'] ? 'is-bad' : 'is-ok') ?>">
+                <span class="mark"><svg class="icon"><use href="#<?= !$robots['found'] ? 'i-missing' : ($robots['blocks-all'] ? 'i-error' : 'i-check') ?>"/></svg></span>
+                <div>
+                    <strong>robots.txt</strong>
+                    <span class="detail"><?= link_to($robots['url']) ?></span>
+                    <span class="detail"><?php if ($robots['unavailable']): ?>Couldn’t be checked<?= $robots['status'] !== null ? ': the server answered HTTP ' . e($robots['status']) : '' ?>. Try again later.<?php elseif (!$robots['found']): ?>Not found (HTTP <?= e($robots['status']) ?>).<?php elseif ($robots['blocks-all']): ?>Blocks every crawler from the whole site.<?php else: ?>Found<?= $robots['sitemaps'] ? ', declares ' . count($robots['sitemaps']) . ' sitemap' . (count($robots['sitemaps']) > 1 ? 's' : '') : ', declares no sitemap' ?>.<?php endif ?></span>
+                </div>
+            </li>
+            <?php foreach ($robots['sitemaps'] as $check): ?>
+                <li class="<?= $check['ok'] ? 'is-ok' : ($check['unavailable'] ? 'is-unknown' : 'is-bad') ?>">
+                    <span class="mark"><svg class="icon"><use href="#<?= $check['ok'] ? 'i-check' : ($check['unavailable'] ? 'i-missing' : 'i-error') ?>"/></svg></span>
+                    <div>
+                        <strong>Sitemap</strong> <span class="muted">declared in robots.txt</span>
+                        <span class="detail"><?= link_to($check['url']) ?></span>
+                        <span class="detail"><?php if ($check['ok']): ?>Loads fine, served as <code><?= e($check['content-type']) ?></code>.<?php elseif ($check['unavailable']): ?>Couldn’t be checked: the server answered HTTP <?= e($check['status']) ?>.<?php else: ?><?= e($check['problem']) ?><?php endif ?></span>
+                    </div>
+                </li>
+            <?php endforeach ?>
+            <?php if ($fallback !== null): ?>
+                <li class="<?= $fallback['ok'] ? 'is-ok' : 'is-unknown' ?>">
+                    <span class="mark"><svg class="icon"><use href="#<?= $fallback['ok'] ? 'i-check' : 'i-missing' ?>"/></svg></span>
+                    <div>
+                        <strong>Sitemap</strong> <span class="muted">at the usual address</span>
+                        <span class="detail"><?= link_to($fallback['url']) ?></span>
+                        <span class="detail"><?= match (true) {
+                            $fallback['ok'] => 'Answers HTTP ' . e($fallback['status']) . '.',
+                            $fallback['missing'] => 'Answers HTTP ' . e($fallback['status']) . ', so there’s no sitemap here.',
+                            default => 'Answers HTTP ' . e($fallback['status']) . ', so it couldn’t be checked.',
+                        } ?></span>
+                    </div>
+                </li>
+            <?php endif ?>
+        </ul>
+    <?php endif ?>
+
     <h3>Icons</h3>
     <?php if (!$meta->icons): ?>
         <p class="empty">No icon links. Browsers will try <code>/favicon.ico</code>.</p>

@@ -31,7 +31,11 @@ <span><?= e(number_format($response->milliseconds / 1000, 2)) ?> s</span>             <?php if ($response->redirects): ?>
                 <span><?= count($response->redirects) ?> redirect<?= count($response->redirects) > 1 ? 's' : '' ?> from <?= link_to($response->requestedUrl) ?></span>
             <?php endif ?>
+            <?php if ($report->generator !== null): ?>
+                <span>Built with <?= e($report->metadata->generator) ?></span>
+            <?php endif ?>
             <a href="?url=<?= e(rawurlencode($report->url())) ?>&amp;format=json">JSON</a>
+            <a href="?url=<?= e(rawurlencode($report->url())) ?>&amp;format=markdown">Markdown</a>
         </p>
     <?php endif ?>