Keeping analytics honest: filtering data-center traffic with a sorted list of IP ranges

One morning our analytics showed a spike of "direct" visitors who stayed two seconds each. Six machines had opened about two thousand job pages in a row. They ran a real headless Chrome, sent an ordinary browser User-Agent, executed our JavaScript and clicked the "apply" links.

Nothing in the request said "bot". Every counter we had, page views, sessions and application clicks, happily counted them. This post is about the one signal that did give them away, and the small module we built around it.

What did not work

The first idea was the User-Agent. Well-behaved crawlers announce themselves, so a block list of names catches them. These machines did not. They sent the same string as a laptop in Lisbon.

The second idea was behaviour: two seconds on page, no scrolling, no mouse. That works in a report written after the fact, but it is useless at request time, and a decent scraper can fake a scroll event.

The third idea was a rate limit per address. Six machines, each making a modest number of requests, stayed well under any threshold we could set without hurting real users behind a shared office address.

The signal that held up

Real job seekers sit at home, in offices, on phones. Their addresses belong to residential and mobile networks. A browser farm runs in a cloud region, and cloud providers publish which address ranges belong to their cloud.

So the question changes from "does this request look like a bot?" to "does this address belong to someone's rented server?" That is a lookup in a static table, and it needs no cooperation from the client.

Most big providers publish their ranges as plain files with no API key. Where a provider has no such file, the routing registry can tell you which networks an autonomous system announces. We use both, in one table of sources:

SOURCES = {
    "gcp":    ("https://www.gstatic.com/ipranges/cloud.json", parse_gcp),
    "aws":    ("https://ip-ranges.amazonaws.com/ip-ranges.json", parse_aws),
    "oracle": ("https://docs.oracle.com/.../public_ip_ranges.json", parse_oracle),
    "alibaba": asn_source(45102),   # announced prefixes via a public registry API
    "tencent": asn_source(132203),
}

Each parser returns a flat list of CIDR strings. Everything after that is provider-agnostic.

Which clouds to include, and which not

This is the part that needs judgement rather than code. We deliberately left out several large hosting companies. Those ranges are full of VPN exit nodes, and a visible share of our real visitors come through a VPN. Blocking the range would mean quietly dropping people we want.

The rule we settled on: include a range only if it is overwhelmingly rented compute that nobody browses from by choice, and add a provider only after we see its traffic in the logs. The list grew in steps, each time after a new wave of fake sessions from a cloud we had not covered.

A lookup in a few microseconds

A provider list has tens of thousands of CIDR blocks, and we check every analytics hit. A loop over networks is out of the question. The trick is to turn all blocks into integer intervals, merge the overlapping ones, and binary-search.

def build(cidrs):
    spans = {4: [], 6: []}
    for cidr in cidrs:
        net = ipaddress.ip_network(cidr.strip(), strict=False)
        spans[net.version].append(
            (int(net.network_address), int(net.broadcast_address)))
    tables = {}
    for version, items in spans.items():
        merged = []
        for start, end in sorted(items):
            if merged and start <= merged[-1][1] + 1:
                merged[-1][1] = max(merged[-1][1], end)
            else:
                merged.append([start, end])
        tables[version] = ([s for s, _ in merged], [e for _, e in merged])
    return tables

IPv4 and IPv6 get separate tables because their integers live in different spaces. Merging also joins adjacent blocks, which shrinks the table a lot, since providers often list many neighbouring /24s.

The check itself is a single bisect:

def is_datacenter(ip):
    addr = ipaddress.ip_address(ip.strip())
    if not addr.is_global:
        return False
    starts, ends = tables[addr.version]
    value = int(addr)
    i = bisect.bisect_right(starts, value) - 1
    return i >= 0 and value <= ends[i]

We benchmarked it on a synthetic table of 100,000 random /24 blocks and 100,000 random addresses. Building the table took about 0.3 seconds. A lookup took about 4 microseconds, including parsing the address string. The real table is smaller than that test, so the cost is invisible next to the request itself.

Fail safe in both directions

Two failure modes matter, and they pull in opposite directions.

If the list file is missing or unreadable, the module returns False for every address. The site behaves exactly as before, with unfiltered analytics. We would rather count a few bots than lose real visitors because a download failed.

If one provider's download fails during a refresh, we keep that provider's previous ranges instead of dropping them:

for name, (url, parse) in SOURCES.items():
    try:
        cidrs = fetch(url, parse)
        if len(cidrs) < 10:
            raise ValueError("suspiciously short list")
        providers[name] = cidrs
    except Exception:
        if old.get(name):
            providers[name] = old[name]

The "fewer than ten ranges" check catches the quiet failure where a provider returns an error page with HTTP 200 and the parser finds nothing. Without it, one bad response would erase a whole cloud from the table.

The new file is written to a temporary file in the same directory and swapped in with os.replace, so a reader never sees a half-written list. The web process compares the file's modification time with the one it loaded and reloads only when it changed.

Where the flag is used

One boolean, three uses:

1. The analytics snippet is not served to flagged addresses, and the property also has an internal-traffic filter, so numbers stay clean even if one slips through.
2. Clicks on outbound "apply" links from a flagged address are not counted as applications.
3. Page-view counters on job pages skip flagged addresses, so employers do not see inflated views.

Real people on a cloud-hosted VPN will occasionally be miscounted. That is an accepted cost: the job page still opens for them normally, only the counters ignore the hit.

What we learned

• Identify bots by where they come from, not by what they claim to be. The address is the one thing a scraper cannot easily fake.
• Be conservative about what goes on the list. A false positive silently hides a real user, while a false negative is just a slightly noisy chart.
• A sorted list of merged integer intervals plus bisect is enough. No database, no radix tree, no extra dependency.
• Design the refresh job so that every failure leaves you with the last good data, never with an empty table.

We use this at SpinHire, a job board for the iGaming industry (https://spinhire.io).