Crawler / bot information

We check shops with our own crawler. This is how to recognise and control it:

User-Agent: ShopsForBotsChecker/0.1 (+https://shopsforbots.com/bot/)
Behaviour: read-only GET requests, limited rate per host (at most 1 per second), roughly 10 requests per shop per check run, backs off on 429 responses, honours robots.txt, no logins, no forms, no purchases
Opt-out: Via robots.txt (User-agent: ShopsForBotsChecker, Disallow: /) or by email: [email protected]

User-agent: ShopsForBotsChecker
Disallow: /

Access for agents

Agents can use the directory without an account or API key: through a REST API, an MCP server (Streamable HTTP) or WebMCP in the browser. All data comes from our own read-only checks of each domain. “MCP declared” only means that the shop’s UCP manifest names an MCP endpoint. We have not called that endpoint and we do not test purchases. Scores are heuristic, unknown values are null.

REST: GET /api/search, GET /api/shop/<domain>, OpenAPI
MCP: POST https://shopsforbots.com/mcp (tools: search_shops, get_shop, list_categories, get_stats)
Discovery: /.well-known/mcp.json, /.well-known/ai-plugin.json, /llms.txt
Fair use: at most 60 requests per minute per IP (RateLimit-* headers; on 429 honour Retry-After). For bulk use, download the full index instead: /api/index.json. Data: fair use for private use; for commercial use get in touch ([email protected]).

REST

# search: parameters q, category, country, platform, min_score, status, protocol, limit (max 100), offset
curl -s 'https://shopsforbots.com/api/search?q=shoes&min_score=60&protocol=ucp,mcp&limit=5'

# full record of one shop
curl -s https://shopsforbots.com/api/shop/allbirds.com

MCP

Configuration for clients that support remote MCP servers (Streamable HTTP):

{
  "mcpServers": {
    "shopsforbots": { "type": "http", "url": "https://shopsforbots.com/mcp" }
  }
}

# or for stdio-only clients, via mcp-remote
{ "mcpServers": { "shopsforbots": { "command": "npx", "args": ["-y", "mcp-remote", "https://shopsforbots.com/mcp"] } } }

# raw JSON-RPC
curl -s https://shopsforbots.com/mcp -H 'Content-Type: application/json'   -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"search_shops","arguments":{"q":"coffee","limit":3}}}'

WebMCP

On the directory and shop pages we register the tools search_shops and get_shop via navigator.modelContext when the browser supports WebMCP. Otherwise nothing happens.