I built ardvark, a crawler that finds ARD catalogs on the web.

ARD (Agentic Resource Discovery) is a new spec that lets a website say what AI agents, MCP servers, and skills it offers, in an ai-catalog.json file at /.well-known/. Think robots.txt or sitemap.xml, but for agents. The spec is new and there's no index of who publishes these files, so I wrote a crawler that goes and checks.

Finding Catalogs

The spec defines three ways to publish a catalog: the /.well-known/ai-catalog.json file itself, an Agentmap: directive in robots.txt, or a <link rel="ai-catalog"> tag in the HTML. ardvark checks all three on every host. When it finds a catalog it keeps going: into nested catalogs, into the agent cards and MCP server cards that entries point to, and into registries, which it queries through their search API. Because nobody links to .well-known files, you can't find them by following links. Instead, ardvark fills its queue from Certificate Transparency logs (a public feed of freshly-certified domains), from crt.sh, or from the Tranco top list.

Verifying the Catalogs

Each catalog is validated against the official JSON Schema, plus seven extra checks the schema can't cover: URN grammar, unique identifiers, whether the publisher in the URN matches the domain serving the file, and so on. Every check is recorded as pass or fail, and the catalog gets an overall verdict of valid, valid_with_warnings, or invalid. Broken catalogs are kept in the database and flagged, not thrown away. You can also run the verifier by itself against your own catalog before you publish it; it exits 1 if something is wrong, so it works in CI too.

Everything is stored in SQLite (or MySQL/Postgres if you change one config key), along with a JSONL event log. Raw documents are saved as-is, so you can re-process the data later without crawling again. The queue also lives in the database, so you can stop a crawl and pick it up later.

# Homebrew
brew install helgesverre/tap/ardvark

# or Go
go install github.com/helgesverre/ardvark/cmd/ardvark@latest

Then seed it with some domains and start crawling:

ardvark seed ct --count 1000
ardvark crawl
ardvark stats
ardvark verify https://example.com/.well-known/ai-catalog.json

It's written in Go, MIT licensed, rate-limits itself, and respects robots.txt. The code is at github.com/HelgeSverre/ardvark.