Training our own legal PII detection model: the full engineering story

This article details the technical work behind our legal document anonymization system. Where that article explains why it's hard, this one explains how we solved it — with the numbers, the rejected hypotheses, and the bugs we had to track down one at a time.

The starting point: a generic model, two known limits

Our pseudonymization pipeline is built on GLiNER, a "zero-shot" entity-detection architecture: you give it a list of categories (person, organization, IBAN, national ID...) and it detects them without needing retraining for every new category. In production we were running a generic pre-trained model (gliner_multi_pii-v1).

Two known limits pushed us to act: recall on individuals (the share of people actually present in a document that the model successfully detects) was too low on our real legal documents, and organization recall had regressed after an earlier change.

Before committing to expensive training, we tested the fastest option: swapping in a different pre-trained, PII-specialized model (fastino/gliner2-privacy-filter-PII-multi). Result: +2.7 points of person recall, but −8.9 points of precision (net F1 score actually worse), and organization detection structurally absent from its label taxonomy — not a tuning gap, an empty slot. That test confirmed our path: train our own model on our own taxonomy, starting from the existing production checkpoint.

Building a real benchmark before touching the model

Before any training, we hardened our internal evaluation methodology: a frozen test set of several hundred documents and several thousand annotated entities, never used for training, with strict acceptance thresholds (person recall ≥ 85%, organizations ≥ 75%, structured data ≥ 90%) and a detailed report broken down by document type and legal domain (business law, criminal, family, housing, labor...).

This step turned out to be decisive: it caught cases where an apparent gain on the aggregate score was hiding a total regression on a specific sub-category — notably real court decisions, where organization recall had sat at 0% for several iterations without anyone seeing it in the aggregate number.

Four iterations, four hypotheses ruled out one by one

The first training run (600–800 documents) failed the acceptance thresholds: person recall at 42.5% (target ≥85%), plus a regression on structured data. Each following iteration tested a different hypothesis:

1. Was the learning rate too conservative? Doubling the encoder learning rate moved recall by only 2 points. Ruled out.
2. Was there not enough training volume? Adding more real court decisions and splitting long documents (beyond the model's technical token limit) gained only one more point of recall. Ruled out — but the investigation surfaced something far more serious: zero real court decisions existed anywhere in the training data. Every real document ever fetched had gone straight into the test set, never into training.
3. Was the annotation quality itself the issue? Digging into the missed detections, we found a bug in our training-data reconstruction: when a person's name was represented as several adjacent fragments in Judilibre's anonymized decisions, our script generated a full random name per fragment instead of one name per person — producing corrupted training data (two full names glued together). Fixed, re-annotated, re-trained. A real improvement, but still not enough.
4. The real problem wasn't training at all. Reading individual sentences where the model missed a person, a pattern emerged: 80% of these cases were partial references (surname alone, no first name) to a person already detected elsewhere in the same document under their full name. This wasn't a model problem — it was a production pipeline problem. A propagation mechanism, added directly to our detection code (not the training run), let the system re-detect these partial references from entities already found.

This last fix pushed the system past the first two acceptance thresholds (person recall 85.5%, organizations 100%) — for the first time in the whole project.

The last hurdle: catastrophic forgetting

One problem remained: structured data (VAT numbers, IBANs, company IDs) had regressed after fine-tuning. Digging further, we found the cause: these categories were 30 times less represented in our training data than individuals. A model trained heavily on one thing gradually "forgets" categories it rarely sees — a well-known machine-learning phenomenon called catastrophic forgetting.

The quick fix would have been to duplicate existing documents containing these rare categories. We rejected it: duplicating a document only adds frequency, not diversity, and risks the model memorizing patterns rather than learning to generalize. We chose the more expensive but sounder path: generating new, diverse synthetic documents specifically for these underrepresented categories, doubling their training volume without ever touching the test set.

One last bug, found by digging rather than retraining

Even after that fix, a gap remained on VAT numbers. Rather than launching another training run, we analyzed each missed case individually — and found a detection-merging bug in production: when a regular expression correctly detects a VAT number, and the model (incorrectly) detects a national ID number at the exact same spot with the same character length, our overlap-merging logic only ever compared length, never entity type or detection source — and sometimes let the wrong label win.

Once fixed (on a length tie, the regex detection — more reliable for structured data — always wins), structured-data recall jumped from 78.0% to 95.7%, without spending a single additional second of compute. The fix even improved the untouched base model's performance, confirming this bug predated the entire project.

Final result

Metric	Result	Acceptance threshold
Person recall	85.5%	≥ 85% ✅
Organization recall	95.7%	≥ 75% ✅
Structured data recall	95.7%	≥ 90% ✅

The final model was trained exclusively on public data (anonymized court decisions, company registries, public procurement notices) and synthetic documents — no real client document was ever used for training. Production inference stays on our existing infrastructure (no architecture change, no new recurring cost); only the one-off training run used temporary external GPU compute.

What we take away from this

Across six training iterations, four failures each revealed a different cause — never the one we'd originally assumed. The one constant: every real breakthrough came from individually analyzing failures, never from a blind adjustment. That's the method we apply to every technical decision at KGM: measure precisely, understand the real cause, fix the right thing — even when the right thing isn't the one you set out to fix.