Here are the lessons that cost us the most time.

1. Start with the public ATS APIs

Many companies host their careers page on an applicant tracking system, and the big ones expose public, unauthenticated JSON for job boards:

• Greenhouse: boards-api.greenhouse.io/v1/boards/{board}/jobs?content=true

• Lever: api.lever.co/v0/postings/{site}?mode=json

• SmartRecruiters: api.smartrecruiters.com/v1/companies/{id}/postings, paged 100 at a time, plus one detail call per job

• BambooHR: {account}.bamboohr.com/careers/list and /careers/{id}/detail

These are designed to be consumed, they are stable, and they give you a real external ID. Parsing HTML should be the last resort, not the first.

One quirk worth knowing: Greenhouse returns job descriptions HTML-escaped twice. Unescape once and you get <p> in your text; unescape twice and you get markup.

import html

def clean_description(raw: str) -> str:
    text = html.unescape(html.unescape(raw or ""))
    return text[:20_000]

2. Be boring about scheduling

Our crawler runs in a separate worker process, sequentially, one source after another. A lock prevents two runs from overlapping. Every source gets two attempts, a 25-second timeout and a small delay between requests (0.15–0.6 s). The bot identifies itself with a descriptive User-Agent that links back to the site.

No async, no queue, no fan-out. For a few thousand postings a day, the bottleneck is never throughput. It is correctness, and sequential code is much easier to reason about when something breaks.

The web app and the worker share one SQLite database in WAL mode, so readers never block on the writer. On a single-vCPU server, the thing that actually hurt us was not the crawl but a derived-pages job rebuilding every ten minutes. It pushed the box into swap. Rebuilding hourly fixed it.

3. Locations: parse from the end

Location strings are the worst field in any job dataset. Our rule of thumb: read them right to left, because the most specific signal about the country is usually last.

The bugs we hit:

• Boston, MA was being read as Morocco (MA is Morocco's ISO code).

• Atlanta, Georgia, US was being read as the country Georgia.

• Two-letter codes are ambiguous: MT is Malta or Montana, CA is Canada or California. We use letter case and the tokens around the code to decide.

• Lowercase English words matched country codes: at, is and me turned into Austria, Iceland and Montenegro.

A simplified version:

US_STATES = {"MA", "GA", "CA", "MT", "NJ", "NY", "PA"}  # abbreviated

def country_of(location: str) -> str | None:
    parts = [p.strip() for p in location.split(",") if p.strip()]
    for part in reversed(parts):
        if part in ("US", "USA", "United States"):
            return "US"
        if part.isupper() and part in US_STATES and len(parts) > 1:
            return "US"          # "Boston, MA" is not Morocco
        country = lookup_country(part)  # exact names only, never lowercase 2-letter words
        if country:
            return country
    return None

4. Keyword filters need negative context

We keep only vacancies from one industry, so a lot of the pipeline is keyword classification. Every naive keyword produced a funny false positive:

• "unity" is inside "opportunity", which tagged shop-floor jobs as Unity developer roles. Word boundaries fixed it.

• "slot" in a drone-firmware job description, singular, had nothing to do with slot games.

• "betting on Google Ads" is a figure of speech in marketing copy.

• In Ukrainian, "Procter & Gamble" is transliterated as "Проктер енд Гембл", which contains the Cyrillic stem for "gamble". It needed an explicit exception.

What worked better than longer keyword lists:

1. Check the title before the description. Titles are short and intentional; descriptions are full of boilerplate.

2. Ignore a keyword when it sits in a list next to unrelated industries ("fintech, forex, crypto, gaming").

3. Trust a company once it proves itself: at least two matching roles, and at least a third of its roles matching.

4. Require two independent signals for generic employers such as large IT outsourcers.

5. Salary parsing is a context problem

Numbers are easy to find. Knowing which ones are pay is not. Our parser accepts an amount only if a pay word appears within 140 characters and no perk word ("allowance", "budget", "bonus scheme") is closer to it. Otherwise "€1,000 learning budget" becomes a salary.

When the period is not stated, we infer it from magnitude: a range topping out under 10k is monthly, one starting at 20k+ is yearly, and anything in between is dropped. A missing salary is better than a wrong one. Sanity bounds catch the rest: yearly 8k–2M, monthly 400–100k, hourly 3–500.

Decimal and thousands separators need the same care: 18.25 is a decimal, 85,000 is eighty-five thousand. And a shorthand like €4k–6k is a case we still do not parse, which is a good reminder to measure what your parser misses, not just what it finds.

6. Deduplicate conservatively

The same job often arrives from two or three sources. We use two layers:

• Exact: (source, external_id) is the upsert key.

• Cross-source: a normalized key of title + first 12 characters of the company + first 16 of the location, reduced to lowercase letters and digits. Within a group we keep the posting with the longest description and archive the rest.

No fuzzy matching, no MinHash. It misses some duplicates, but it almost never merges two different jobs, and that asymmetry is what we want for a public index.

7. Only archive what you fully saw

A job that disappears from its source should be archived. But if a source fails halfway, "missing" does not mean "closed". We only archive missing jobs from sources whose fetch succeeded and returned fewer than the per-board cap (300). A truncated response never closes anything. Jobs that reappear are reactivated.

8. Monitor per source, not per run

Our worst incident was silent: a source kept returning +0 new jobs for about a month, and a total that still looked healthy hid it. Now every run stores per-source counts, attempts, timings and errors, with the last 30 runs kept, so a flat line on one source stands out immediately.

Takeaways

• Prefer official, public JSON endpoints over HTML.

• Sequential and boring beats clever for small daily batches.

• Parse locations from the end and distrust two-letter tokens.

• Every keyword needs a negative context rule.

• Drop uncertain salaries instead of guessing.

• Deduplicate conservatively and archive only what you fully observed.

• Alert on each source separately.

We use this pipeline to run SpinHire, an open index of iGaming jobs: spinhire.io