Bot detection

Rybbit checks every incoming tracking request against several detection layers, on every site. The per-site Block Bot Traffic setting decides what happens to a request that a layer flags as a bot.

Block Bot Traffic

Block Bot Traffic is on by default for new sites. To change it:

1. Open your site in the Rybbit dashboard
2. Go to Site Settings
3. Toggle Block Bot Traffic

What Happens to Detected Bots

With blocking on, detected bot requests are not added to your normal analytics data. This keeps dashboard totals, reports, journeys, funnels, session lists, and usage metrics focused on human traffic.

Blocked bot requests also do not count toward billable usage, because usage is counted from your normal analytics events only. With blocking off, detected bots are tracked and billed like any other traffic.

For blocked requests, Rybbit stores a compact bot event record so you can inspect what was filtered. Bot event records include route, device, location, ASN, and which detection layers matched.

Detection Layers

Rybbit runs all detection layers before making a final decision. A request can match multiple layers, and the bot event records every layer that matched.

For sites set up as mobile apps, the user-agent and header layers are skipped, because native app traffic legitimately looks like a non-browser HTTP client. The other layers still run.

Bot Identity

Every detected bot carries the layers that caught it. Bots matched by one of Rybbit's curated user-agent patterns carry three more fields, which is what the Bots dashboard groups by:

A bot matched only by a broad pattern (crawl, spider) is still filtered, but has no published identity — those rows read as Unnamed, which is more useful than a guess.

Purposes

bot_purpose answers a different question from bot_category. Category groups bots into families; purpose says what the operator does with the page, which is the distinction that matters for AI traffic:

Splitting the AI family in three is the point: a training crawler reading 4,000 pages and a person's assistant opening one page are both category: ai, but they mean opposite things for your site.

How Decisions Are Made

Rybbit does not stop at the first matching layer. It runs every layer, records all matches, and then makes one final decision.

A request is marked as bot traffic when at least one blocking layer matches. The resulting bot event includes boolean fields for each layer:

• User-agent pattern
• Header heuristics
• Client signals
• Bot ASN
• Rate anomaly

Because multiple layers can match the same request, per-layer bot counts can add up to more than the total number of bot requests.

Server-Side Tracking

Requests sent to /api/track with Authorization: Bearer <api key> and a valid key for the site with the ingest:write scope (or an unrestricted key) are treated as trusted server-side ingestion. Four of the five layers are skipped: the headers, the client signals, the network address, and the request rate all belong to your server, not to the visitor it is reporting, so scoring them would convict your own backend.

The user-agent layer still runs, on the user_agent you send. Nothing changes for ordinary backend events — a request reporting a real browser's user agent is tracked normally.

Reporting crawlers that never run JavaScript

Rybbit's tracker is JavaScript, so a crawler that fetches your HTML and executes nothing never reaches Rybbit at all. Most AI training crawlers work exactly that way, which is why the Bots dashboard carries a note about its own coverage.

If you want to see them, report them from the edge — middleware, a worker, or a reverse proxy — forwarding the real request's user_agent:

{
  "site_id": "1",
  "hostname": "example.com",
  "pathname": "/pricing",
  "user_agent": "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.2; +https://openai.com/gptbot"
}

With Block Bot Traffic on, a forwarded crawler user agent is classified and lands in your bot events with full identity, rather than in your analytics as a phantom visitor. With blocking off, it is tracked as a normal visit.

Proxies and CDN Setups

If you proxy Rybbit through Cloudflare Workers, AWS CloudFront, Nginx, Caddy, or another reverse proxy, forward the original visitor IP:

X-Forwarded-For: <visitor-ip>
X-Real-IP: <visitor-ip>

Also preserve the original User-Agent, Referer, and Accept-Language headers where possible.

If the proxy IP is sent instead of the visitor IP, traffic may be geolocated to the proxy location, may inherit the proxy provider's ASN, and many visitors can look like one busy address to the rate layer.

Then turn on First-Party Proxy in Site Settings. It tells Rybbit to read the visitor IP from the forwarded headers (X-Real-IP, then X-Forwarded-For) instead of inferring whether a proxy is in front.

See the proxy troubleshooting guide for examples.

What Bot Detection Does Not Guarantee

Bot detection improves analytics quality, but no bot filter is perfect.

Some sophisticated bots can look like normal browsers. Some legitimate users may browse through unusual network paths or constrained browser environments. Rybbit uses multiple layers to reduce both misses and false positives, but you should still interpret bot counts as an operational signal rather than an exact measurement of all automation on your site.