Translating HTML after it renders: one template, eleven languages

Our site started in one language. The templates were written in Russian, the strings lived inline, and nobody had wrapped a single one in a _() call.

Then we needed English. A few months later we needed German, Polish, French, Spanish, Portuguese, Italian, Greek, Romanian, Bulgarian and Ukrainian. Eleven versions on top of the original, with a small team and a codebase that was already in production.

Retrofitting gettext into hundreds of templates and thousands of f-strings was not going to happen. So we did something unusual: we translate the HTML after it is rendered. This post is how that works, where it broke, and the guard rails that keep it honest.

The shape of it

Every request passes through one HTTP middleware. It reads the language from the URL prefix (/de/jobs, /pl/blog/...), strips the prefix, and lets the normal route render the page exactly as it always did, in the base language.

On the way out, the middleware takes the HTML body and rewrites it. Text nodes and a handful of attributes are looked up in a dictionary for the target language. Links get the language prefix added back. Then the <html lang>, og:locale, hreflang and canonical tags are set for that version.

@app.middleware("http")
async def language_layer(request, call_next):
    lang, inner = split_lang_prefix(request.url.path)   # "/de/jobs" -> ("de", "/jobs")
    if lang:
        request.scope["path"] = inner
        request.scope["raw_path"] = inner.encode()
    request.scope.setdefault("state", {})["lang"] = lang or "ru"

    response = await call_next(request)
    if not lang or not response.headers.get("content-type", "").startswith("text/html"):
        return response

    body = b"".join([chunk async for chunk in response.body_iterator])
    html = body.decode("utf-8", "replace")
    html = await run_in_threadpool(translate_html, html, lang)
    html = prefix_links(html, lang)
    return HTMLResponse(html, status_code=response.status_code)

Routes stay language-agnostic. The few that need the language (an API that returns English data, a Markdown mirror) read it from request.state.

Why subdirectories

We briefly ran languages on subdomains. Each one needed its own DNS record, its own certificate, and from a search engine's point of view it was a separate site starting from zero.

Subdirectories need none of that. /de/ inherits the authority of the main domain, and one deploy covers every language. The only wrinkle was Ukrainian: the language code is uk, but /uk/ reads as "United Kingdom" to a Ukrainian visitor. We serve it under /ua/ and keep uk everywhere a standard demands a language code: hreflang, og:locale, dictionary file names. Two tiny functions map one to the other, and old /uk/ URLs get a 301.

The translator itself

translate_html splits the page so that <script>, <style> and <textarea> blocks are left alone. Everything else is scanned with two regular expressions: one for text between tags, one for the attributes that carry visible text (placeholder, aria-label, title, alt, content).

TEXT_RE = re.compile(r">([^<>]+)<")
ATTR_RE = re.compile(r'((?:placeholder|aria-label|title|content|alt)=")([^"]+)(")')

def translate_text(raw, full):
    key = " ".join(raw.replace("&nbsp;", " ").split())
    lead = raw[: len(raw) - len(raw.lstrip())]
    trail = raw[len(raw.rstrip()):]
    if key in full:
        return lead + full[key] + trail
    return (with_numbers(key) or with_slot(key)
            or translate_terms(raw) or raw)

The key is the whitespace-normalized text node. Templates are indented, and the same sentence can arrive with a newline in the middle on one page and on a single line on another. Normalizing first means one dictionary entry covers both. The original leading and trailing whitespace is put back so the layout does not shift.

The   replacement looks trivial. It cost us a hero heading that stayed untranslated for weeks, because the template used a non-breaking space between two words and the dictionary key had a plain one.

Dictionaries, in layers

Each language has around twelve thousand entries spread across several JSON files: interface strings extracted from templates by a script, hand-corrected interface strings, static pages, and article translations. They load in a fixed order so a human fix always overrides a machine one:

files = sorted(os.listdir(I18N_DIR), key=lambda n: (".ui." not in n, n))
for name in files:
    code = name.split(".")[0]
    FULL.setdefault(code, {}).update(json.load(open(I18N_DIR / name)))

The browser gets a subset. Some text is drawn by JavaScript after the page loads, so the client needs the interface dictionary too. With article translations included it would weigh about two megabytes; without them it is around 250 KB, and the article pages arrive from the server already translated anyway.

There used to be a second copy of the dictionary inside the client bundle. The two copies drifted apart, which is how we learned the next lesson.

Strings with numbers and names inside

Exact matching fails the moment a string contains live data. "5240 live jobs" becomes "5241 live jobs" an hour later, and no dictionary can list every count.

So there is a second index where every number is replaced by #. At lookup time the incoming string is masked the same way, and if a template matches, the real numbers are put back in order:

NUM_RE = re.compile(r"\d+(?:[ ¬†вАѓ]\d+)*")

def with_numbers(raw, index):
    numbers = NUM_RE.findall(raw)
    target = index.get(NUM_RE.sub("#", raw)) if numbers else None
    if not target:
        return ""
    parts = target.split("#")
    if len(parts) - 1 != len(numbers):
        return ""
    return parts[0] + "".join(n + t for n, t in zip(numbers, parts[1:]))

The number pattern accepts thin and regular spaces as thousands separators, because Russian typography writes "5 240". A trailing space is not part of the number.

Names work the same way with a single slot: "Acme: jobs at this company" is stored as #: jobs at this company. We split each such key into a head and a tail, and if the text starts with the head and ends with the tail, the middle is the slot. Keys are tried longest-anchor first, and anchors shorter than eight characters are ignored, because a two-character anchor matches almost anything.

Dates got their own mask (@), since a date is also full of numbers and would break the # count.

The one-word entries that broke every page

Some values are ours, not the employer's: country names, work formats, categories, salary periods. These appear inside longer strings ("Nairobi, Kenya" with the country part translated), so they cannot be matched only as whole nodes. They need substring replacement.

At first we dumped every dictionary into the substring pass. Article translations included table cells containing a single word: "in", "and", "of". Those got replaced inside every string on every page. A heading meaning "Jobs in iGaming" came out as the Russian word for "jobs" followed by an English "to". A sentence about "a team of 7,400" kept its Russian noun and gained an English "of".

The fix was to make the substring list closed and explicit: a dedicated module holds only platform vocabulary, and only that module is used inside strings. Everything else matches whole nodes or nothing.

The second bug was word boundaries. \b in Python regex is Unicode-aware, but our client-side copy did not use boundaries at all, and an inflected Russian word with a matching stem became half English, half Russian. We now build the pattern with explicit Cyrillic lookarounds and sort keys longest-first, so "not specified (fem.)" is tried before its shorter masculine form:

CYR = "–Р-–ѓ–∞-—П–Б—С–Ж—Ц–З—Ч–Д—Ф“Р“С"
BOUNDARY = rf"(?<![{CYR}])(?:%s)(?![{CYR}])"

def build_term_re(table):
    keys = sorted(table, key=len, reverse=True)
    return re.compile(BOUNDARY % "|".join(re.escape(k) for k in keys))

And the client copy is gone. The browser fetches its dictionary from the same source as the server.

Structured data is text too

JSON-LD blocks are scripts, so the first version skipped them. Search engines then saw a German page whose headline, FAQ answers and breadcrumbs were in Russian.

Now <script type="application/ld+json"> is parsed, walked recursively, and every string is translated with the same function as visible text. inLanguage is set to the target language, and URL fields get the language prefix so the structured data points at the German page, not the original.

It was blocking the event loop

Translating a large listing page means thousands of regex lookups. Done inside an async middleware, that ran on the event loop, and for those seconds the whole server stopped answering, including static files and robots.txt.

The fix is one line, await run_in_threadpool(translate_html, ...), but it only helps if the thread pool is sized sensibly. We cap it, because forty concurrent heavy translations also means forty copies of a large page in memory.

Not every translation deserves an index

Dictionary translation is good enough for an interface. It is not good enough to present a page as a real German page to a search engine when most of the content is a list of English job ads with a German frame around it.

So the middleware decides per path which language versions are indexable. The homepage, the profession catalogue and blog articles are fully translated and go into hreflang and the sitemap. Listing pages in the smaller languages get noindex,follow.

Blog articles are decided by coverage: we count the paragraphs longer than 40 characters in the original, check how many have an exact dictionary match, and treat the article as translated only at 60% or more. Before this check, fresh untranslated articles were listed in hreflang for eleven languages while still showing the original text, and 13 of 67 articles were in that state when we found it.

Catching leaks

Because translation happens on output, the failure mode is silent: a string missing from the dictionary simply stays in Russian. Two checks catch it.

A unit test asserts that every value the crawler can write to the database (country, format, category) has a translation in every language. An audit script fetches live pages for each language and flags Cyrillic text on Latin-script versions. For Bulgarian and Ukrainian, which are Cyrillic themselves, it flags text nodes identical to the Russian original. It exits non-zero on leaks, so it can gate a deploy.

Would we do it again?

For a new project, no: wrap strings from day one. For an existing site in production, translating rendered HTML got us eleven languages without touching most templates, and every page, including ones written later, is translated automatically.

The price is discipline. Keep substring replacement to a closed list, mask numbers instead of storing them, test the vocabulary the database can produce, and audit the live output, because that is the only place the truth is visible.

We use this at SpinHire, a job board for the iGaming industry.