Model-agnostic PII detection with LLMs

extract

Artificial Intelligence Model-agnostic PII detection with LLMs A configurable, instruction-driven detector that runs on any large language model (LLM) managed on Amazon Bedrock, evaluated on five public PII corpora across nine LLM-based detectors, including the OpenAI PrivacyFilter. Fine-tuning a model on real-world text creates a personally identifiable information (PII) detection problem. Training corpora are full of PII: names, home addresses, email and phone numbers, national-ID and social-security numbers, bank accounts, dates of birth. A model trained on uncleaned text can memorize that data and later reproduce it, leaking a real person’s details through a prompt that was never meant to surface them. In this post, we describe a configurable, model-agnostic detector built on large language models (LLMs), walk through its implementation, benchmark it against an off-the-shelf tool, and show how to run it on your own data. Sample code: The detector described in this post ships as the pii-detector package, available in the sample-llm-pii-detection repository. Every code snippet that follows is drawn from that package, and the Running the detector end to end section walks through installing and running it on your own data. PII rarely sits in a tidy form field. It hides in customer-support transcripts, HR records, chat logs, and the long free-text columns that make up the custom datasets teams fine-tune on. It arrives in messy, multilingual formats that no fixed schema anticipated. The usual tools are bi-directional token-classification models: transformer taggers that label each token with a PII type fixed at training time. A domain-specific identifier like an employee ID or crypto-wallet address is exactly what a custom fine-tuning corpus introduces, and it falls outside that frozen schema. Adding it means relabeling and retraining. And they are locked to one model and one deployment. Large language models reframe the problem. An LLM reads its instructions at inference time, so the entities to detect, the output format, and the deployment backend all become configuration rather than code. One detector can target a new entity type by editing a prompt instead of retraining, run on a managed API or inside your own virtual private cloud (VPC), and reason about context across eight languages without a translation step. The rest of this post describes such a detector, walks through the engineering behind it, and shows how it measures up against existing tools. Solution overview The detector treats the language model as a configurable, swappable component. You wrap the input text in instructions that define the PII entities to detect and the expected output. The model then returns a structured list of detected entities. Two design choices make it model-agnostic: - Instruction-driven detection: The detection logic lives entirely in the instructions and a thin parsing layer. That makes it independent of any single model’s idiosyncrasies. - Configurable backend: The model is reached through a uniform inference interface, the Inferencer. The package ships an adapter for Amazon Bedrock (managed, for example Mistral or OSS-GPT). The same interface accepts a custom adapter for an open model, such as OSS-GPT 20B served on your own infrastructure with a GPU. That covers secure or air-gapped environments that cannot reach Amazon Bedrock. Any object that takes a list of messages and returns the assistant’s text satisfies the interface, so the detector is agnostic to which backend it holds. Customization comes from two independent components. The first is the model, which sets accuracy, latency, and cost: you choose a frontier Amazon Bedrock model or a small open model on a single GPU. The second is the entity set, which defines what counts as PII. To extend it, you add a domain-specific identifier or drop one that you don’t need. Changing the entity set is a one-line edit to the instructions, with no retraining and no redeployment. The LLM’s job is narrow and well-defined. It reads the text, identifies all PII spans, and labels each with an entity type from the schema. It returns those spans as structured JSON, and a post-processing step computes exact character offsets and removes duplicates. To place the approach in context, we evaluate it span-for-span alongside eight other LLM-based detectors, including the OpenAI PrivacyFilter. All are scored on a common ground truth. Technical implementation The detector is built from four parts. A prompt defines the schema, a backend runs the model, a parsing-and-offset layer turns the response into located spans, and a thin call sequence ties them together. This section walks through each part in the order a request flows through the system, pointing to the module in the package repository that implements it. PII schema and detection prompt The schema lives in a single system-prompt template, the heart of the detector: fifteen entity categories each with a one-line definition, a do-not-flag list, optional few-shot examples, and the input text. Because the schema is text, adding or removing a category is a one-line edit. The model is instructed to respond with a JSON list, one object per detected entity carrying the entity type and the exact text value found. It does not return character offsets, which an LLM cannot produce reliably. Those are recovered in post-processing: The full prompt is in pii_detector/templates.py, and the end-to-end walkthrough that follows runs it as-is against a sample string. LLM backend integration Because detection lives in the prompt, the backend is a free choice. In our provided implementation, the detector talks to a small interface, the Inferencer: messages in, text out. The same detector therefore runs against a managed model on Amazon Bedrock or an open model you host yourself on Amazon Elastic Compute Cloud (Amazon EC2). The package ships the Amazon Bedrock adapter (pii_detector/bedrock_inferencer.py), a thin wrapper over the Converse API. The walkthrough that follows runs that path end to end. Post-processing The model’s raw text becomes a clean list of located spans in three steps, all in pii_detector/detector.py: - JSON parsing: The text response is converted into a list of dictionaries, each item corresponding to a detected PII (or an empty list if the record contains none). - Offset calculation: Because the model returns values and not positions, each value is located in the source text with a regular expression. - Hallucinated-label recovery: LLMs routinely emit near-miss labels (DATE for DATES, EMAIL for CONTACT_INFO), so each emitted label is re-homed onto the prompt’s own vocabulary by morphology and a curated alias table. A label that no tier can map is marked UNK rather than force-fitted, so genuine hallucinations stay visible. Running the detector end to end This section walks through running the detector on your own data, from prerequisites to cleanup. Every step uses the pii-detector package referenced at the top of this post. Prerequisites To follow along, you must have the following prerequisites. - Python: Python 3.11 or later. - AWS account with Amazon Bedrock model access: An AWS account whose credentials can call the Amazon Bedrock Converse API. You also need model access enabled in the Amazon Bedrock console for the model that you choose, for example a Mistral or OSS-GPT model. For model availability by AWS Region, see Supported models by AWS Region in Amazon Bedrock. The detector resolves credentials through the standard AWS credential chain, so set AWS_PROFILE (or an IAM role or SSO profile) and AWS_REGION in your environment. - Python dependency: Boto3, the only runtime dependency, installed into a virtual environment (see step 1). The following steps assume you have cloned the pii-detector repository and are working from its root directory. Step 1: Install the package and its dependency Create a virtual environment and install boto3. The package runs from the repository root, so set PYTHONPATH to make the pii_detector module resolve. Step 2: Configure AWS credentials for Amazon Bedrock Point Boto3 at an account with Amazon Bedrock access and select the Region where you enabled model access. If you aren’t using a named profile, Boto3 also supports AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY, but we recommend an AWS Identity and Access Management (IAM) role or SSO profile instead of long-lived static keys. Step 3: Run the bundled example The repository ships a runnable example (examples/detect.py) that detects PII in a sample string. Run it as a module from the repository root. It fails fast with actionable guidance if credentials or model access are missing. Step 4: Call the detector on your own text Construct an Amazon Bedrock inferencer with any Amazon Bedrock Converse model id, wrap it in a PiiDetector, and call the detector on a string. It returns the list of located spans, each with its exact character offsets, ready to feed a downstream redaction step. There are no servers to manage, because Amazon Bedrock is fully managed. The model_id is any Amazon Bedrock Converse model id or inference-profile id, for example amazon.nova-lite-v1:0 or mistral.mistral-large-3-675b-instruct. Switching models is a one-line change. The detector and the call site stay identical. Step 5: Clean up Amazon Bedrock is serverless, so there’s no infrastructure to tear down and you pay only for the tokens you use. To clean up, deactivate the virtual environment (deactivate) and, if you no longer need it, disable the model access you enabled in the Amazon Bedrock console. If you supply your own self-hosted backend instead of Amazon Bedrock, remember to shut down that host yourself, since the detector doesn’t manage backend infrastructure. Benchmarking Evaluation uses five public PII corpora from Hugging Face, each carrying ground-truth spans, sampling roughly 10,000 rows per dataset. Together they cover 49,365 records and 222,114 ground-truth core spans across eight languages (de, en, es, fr, hi, it, nl, te). Their domains run from multilingual synthetic profiles to English HR and customer-service documents, which makes the aggregate a fair stress test. A predicted span is matched to ground truth by exact (start, end, label) overlap (IoU = 1.0) and scored with Precision, Recall, and F1. Comparing detectors across these datasets is harder than it looks, because labels do not line up. Each detector and each dataset uses its own vocabulary: PRIVATE_NAMES compared to NAME, street_address compared to street. To make the comparison fair, every raw label, from detector output and dataset ground truth alike, is mapped onto a single canonical taxonomy of twelve common entities. Each detector is then scored only on the intersection of the label scopes it and the dataset both declare. This way, no detector is penalized for a category it never claimed to support. The canonical core entity taxonomy. These twelve types are common across the datasets and detectors, so they form the basis of the headline comparison. The package repository gives the exact raw-label-to-canonical mapping for each of the five datasets. This taxonomy defines two reporting scopes. Core F1, the fair head-to-head number, covers the twelve common entity types. Extended-entity F1 covers the dataset-specific categories (occupation, company name, crypto-wallet addresses, and similar) that most off-the-shelf detectors have no notion of. We cover that scope under Customization. The headline metric is span-level Core F1. The following table reports it together with estimated per-detection latency across a representative selection of LLM-based detectors. It covers managed models on Amazon Bedrock and open models served on Amazon EC2, including the OpenAI PrivacyFilter. The open models are chosen both smaller and larger than OSS-GPT 20B so the range is visible. Amazon Bedrock is model-agnostic, so the right choice depends on your workload’s accuracy, latency, and cost needs rather than any single ranking. Results vary by model. docs/benchmarks.md gives the complete table with every model we tested. Span-level Core F1 (all five datasets, 49,365 records) and estimated per-detection latency, grouped by backend (Amazon Bedrock, Amazon EC2). In practice detection runs over many records with parallel worker threads. The per-detection figure is the total wall-clock time extrapolated back to a single record, so it is indicative rather than a strict single-call measurement. On the same corpora, Core F1 ranges from 74.9 percent (Nova Lite 2) to 83.1 percent (Mistral Large 3), with PrivacyFilter at 80.7 percent. Mistral Large 3 and OSS-GPT 120B run on Amazon Bedrock, and OSS-GPT 20B (81.6 percent) runs on hardware you control. Latency is driven by the model, not its parameter count. OSS-GPT 20B runs in about 1.2 seconds on Amazon EC2, while the similarly sized Qwen3.6-27B takes about 12.8 seconds, because reasoning verbosity and architecture matter more than raw size. And the backend is a free choice, since OSS-GPT 20B scores within 0.3 points on Amazon EC2 (81.6 percent) and Amazon Bedrock (81.3 percent). Accuracy holds across languages and high-stakes identifiers. On the ai4privacy_500k breakdown (see the package repository), OSS-GPT 20B stays in a tight 83–90 percent Core-F1 band across all eight languages, including non-Latin Hindi and Telugu. It also scores at or above the frontier models on the identifiers that matter most: SSN, financial, and ID numbers all above 95 percent. The shared weak spot is DATE, at about 50 percent, where span boundaries and formats are genuinely ambiguous. Customization The accuracy results so far exercised the model lever, where you trade accuracy against latency and cost. The second lever is the set of entities to detect, defined entirely in the instructions. This is where the approach extends beyond fixed-scope tools, and the clearest evidence is on the rare, domain-specific entities. Each dataset annotates its own beyond-core categories. The nemotron and gretel corpora label occupation, job title, and company name. The isotonic corpus labels crypto-wallet (Bitcoin and Ethereum) addresses, vehicle identifiers, and user-agent strings. The ai4privacy_500k corpus labels sex, gender, and organization. A detector running the base configuration has no notion of these categories and scores near zero on them. Recovering them requires no new model and no retraining, only an instruction change. We call this the Ext (Extended) configuration. It adds each dataset’s extra category definitions and a few worked examples to the prompt. It also removes the do-not-flag lines that would otherwise conflict with them, for instance dropping “business addresses” from the public list once company name becomes a target. The package repository lists the full extra-category definitions. The effect is large, and it holds across all models we tested, frontier and small alike. Extended-entity F1 jumps several-fold while core accuracy is unchanged or slightly better: Base prompt compared to the Extended (Ext) configuration, on the five public datasets, sorted by Extended-entity F1. Adding the extra category definitions lifts extended-entity F1 roughly six-fold and also nudges core F1 up. This works on all tested models, whereas a fixed-scope tagger cannot target these categories without retraining. The same lever generalizes to brand-new entity types. To target one for a specific domain, add its definition and an example to the prompt. There is no model to fine-tune and no pipeline to redeploy. Combined with the freedom to choose the model behind it, this lets one detector adapt to each domain’s vocabulary at the instruction level. Conclusion An LLM-based PII detector turns the hardest constraints of off-the-shelf tools into configuration set by two levers. Those constraints are a fixed entity scope and lock-in to one model and one deployment. The model lever sets accuracy, latency, and cost. On five public corpora spanning eight languages, Core F1 across the nine detectors ranges from 74.9–83.1 percent, with PrivacyFilter at 80.7 percent. The open OSS-GPT 20B (81.6 percent) runs equally well on Amazon Bedrock or your own GPU. The entity lever sets what counts as PII: the Extended configuration lifts extended-entity F1 from about 12 percent to about 73 percent across all tested models, without retraining. Because the detection logic is text rather than weights, the same detector adapts to a new domain or backend without a new model. Next steps, if you want to apply this to your own data, follow the walkthrough in the Running the detector end to end section: - Try the detector: Install the package and run the bundled example against a sample of your own corpus to see what it flags. - Pick a backend: For the highest accuracy with no self-hosting, use a managed model on Amazon Bedrock such as Mistral Large 3 or OSS-GPT 20B. For control over data residency, supply your own adapter for an open model such as OSS-GPT 20B served on your own GPU. - Extend the schema: Add your domain-specific entity definitions to the prompt using the Extended configuration, documented in the package repository, as a template, then re-run on your sample. - Close the loop: Feed the detected spans, which carry exact character offsets, into a redaction step so cleaned text flows into your training pipeline. From there, the natural extensions follow the same pattern. Supporting new entity types or additional languages is an instruction change, not a new model. Resources The full detection system prompt, the per-dataset label mappings, the complete detector benchmark table, and the Extended-configuration category definitions are documented in the package repository. - Sample code: the pii-detector package, including the runnable example and the full prompt and label-mapping documentation. - Benchmarks and label mappings: the full detector benchmark table, per-dataset label mappings, and Extended-configuration category definitions. - Amazon Bedrock console: enable model access and try a model. - Amazon Bedrock service page: overview, supported models, and pricing. - Amazon Bedrock documentation: the Converse API and model-access guides. - Related post: Evaluate large language models for quality and responsibility – evaluating and comparing models on your own criteria. - Related post: Redact sensitive data from streaming data using Amazon Comprehend – applying managed PII detection and redaction to text at scale.