Benchmark-Leading PII Detection
for Enterprise AI

What does a correct detection mean?

PII extraction is structured prediction under an annotation contract. A detector can identify the right person but assign the wrong name role, cover only part of an identifier, or merge several entities into one span. These errors have different consequences for privacy and downstream utility. A single overlap score obscures those distinctions.

1.1 Typed spans and label-conditioned extraction

Let x_i be a document and ℒ its allowed entity labels. Gold and predicted entities are triples (a, b, ℓ): a half-open interval of original character offsets and an entity type. This fixes the unit of comparison before tokenization, normalization, or replacement.

Prior work on label-conditioned extraction, including GLiNER [1], formulates entity detection through contextual spans and entity descriptions. This perspective makes label semantics central to measurement: a complete personal name, a given name, and an organization are different annotation targets. Our evaluation retains each cohort's declared label inventory and states the matching rule explicitly.

1.2 Official overlap and exact-match estimands

PrivacyBench's official scorer [6] uses two directional overlap counts. Let G and N denote total gold and predicted spans. D counts gold spans touched by a same-label prediction; M counts predictions touching a same-label gold span. Write g ∼ p when their labels agree and their character intervals have a nonempty intersection:

Overlap matching is not one-to-one: D and M can differ. Exact scoring instead requires the complete typed triple. The exact metric therefore requires both the correct label and complete character boundaries:

1.3 From entity recognition to masking coverage

For privacy diagnostics, let U_i^G and U_i^P be unions of gold and predicted character intervals, ignoring type. Sensitive-character recall measures what a redaction would cover; excess masking measures the fraction of annotated non-sensitive characters it would remove:

These diagnostics separate missed sensitive content from loss of useful context. They inform acceptance checks independently of leaderboard F1. They are coverage proxies; coherent replacement is a distinct endpoint, defined in Section 6.

1.4 From span recall to complete identity coverage

An identity can recur across many records. Masking nine mentions while leaving a tenth intact can still reveal the person behind the corpus. The Text Anonymization Benchmark makes this distinction explicit by evaluating coverage at the entity level [12]. Let ℰ be the nonempty set of annotated entities, and let M_e be the nonempty set of mentions of entity e. The binary indicator z_j equals one if and only if mention j is completely covered. Complete-entity coverage is:

The product makes coverage conjunctive: every mention must be covered. Let p_j = Pr(z_j = 0), the probability of failing to cover mention j completely. The chance of at least one coverage failure obeys:

These bounds require no independence assumption. Repeated aliases and shared formatting can correlate errors, so multiplying independent success probabilities is generally unjustified. This is a statement about annotated coverage, not a re-identification probability; F1 cannot be substituted for the individual miss probabilities. It explains why Nyne treats identity continuity as a corpus-level problem.

1.5 Preserve context under a coverage requirement

Removing an entire document can maximize masking coverage while destroying its value. A more useful objective is to minimize excess masking subject to a declared missed-character budget ε. For a masking policy T, Equation 4's coverage and excess-masking quantities define:

This separates the privacy requirement from the utility cost instead of hiding both inside one weighted average. Constrained treatment of asymmetric errors has a formal precedent in Neyman-Pearson statistical learning [13]. Here, excess masking is a proxy for context loss; downstream task utility remains a separate question. For a nonempty feasible set with an attained minimum, the optimizer states the preferred policy. These equations describe analytical requirements, not additional measured results or a claim that the optimization has been solved.

1.6 From metric definitions to observed behavior

From candidate spans
to coherent replacement.

Consider “Mara Chen signed for Cedar Labs.” Detecting “Mara” is not equivalent to detecting “Mara Chen,” and replacing either string does not establish that a later email belongs to the same person. The system has to carry a decision through three representations: scores over candidate spans, a set of accepted mentions, and identities that remain consistent after replacement. The formulation below follows that chain.

2.1 Make the complete identifier win

Nyne Blackvault uses a fine-tuned GLiNER-PII detector. Following GLiNER's architecture, contextual word and label representations are projected into a shared space [1][10]. Let h_i and v_ℓ denote those contextual representations, with learned functions producing the span vector s_ij and label vector e_ℓ. Stack n span vectors and k label vectors in a shared dimension d:

“Mara Chen” and “Mara” occupy different rows of S but share the PERSON label vector. Their scores depend on much of the same context and many of the same parameters. The sigmoid acts independently on each pair; a high PERSON score for the fragment does not suppress the full name. Their competition is resolved later.

To see how learning changes that competition, fix the input and candidate inventory and flatten Z in column order into z. Let G_θ = ∂z/∂θ^T be the score Jacobian, and let r = ∇_zJ be the residual of a scoring loss J(z). A local gradient-descent update gives, for sufficiently small η > 0:

Here the score map and loss are twice continuously differentiable locally, with bounded second derivatives; all derivatives on the right are evaluated before the update. Each entry K_ab is the inner product of two candidates' parameter gradients. Its off-diagonal terms describe how changing one candidate affects another. This finite-network tangent-kernel view [20] analyzes the coupling; it does not require materializing a kernel in the detector. The displayed step is a local reference update rather than a specification of the optimizer [16].

The boundary decision depends on a contrast within that coupled system. Let a be the correctly typed full name and b an overlapping fragment. Define v_ab as the vector with +1 at a, −1 at b, and zeros elsewhere. The ranking margin Δ_ab and eligibility margin γ_a are:

For 0 < τ_a < 1, γ_a says whether the full span clears its score cutoff; Δ_ab says whether it outranks this fragment. Although the first-order change in the loss in Equation 9 is nonpositive, the change in this particular ranking margin can have either sign. A better aggregate objective can therefore coexist with a worse boundary decision. Exact-span evaluation tests the decision that survives this competition, including competing candidates beyond this pair.

2.2 Choose which candidates survive

The scores now have to become an admissible set of spans. Map word endpoints back to original character intervals, index typed candidates by a, and put conflicting pairs in ℭ. Under a flat annotation policy, overlap makes a pair incompatible. If u_a records whether candidate a is selected, the admissible set is:

A greedy decoder visits eligible candidates in score order and retains compatible spans. The full name can lose to a fragment even when both clear the cutoff; conversely, raising the cutoff can remove both. This is the connection between the margins above and the sensitive characters left in the output. The constraints specify feasible selections, without asserting that greedy selection solves a global maximum-score problem.

Some decisions depend on the candidate's meaning in context as well as its neural score. The evaluated Blackvault Ensemble includes structured candidate review using TypeSafe's Jev. Jev assesses proposed spans and returns constrained type or non-entity decisions; the surrounding system uses these outputs to retain, reject, or relabel candidates and handle uncertain outcomes. Blackvault performs detection without this review stage. Jev supplies contextual review here, while identity linking and replacement are downstream tasks. [15]

This review can be expressed as a decision over a finite action set 𝒜. Let q_k(x, a) be the posterior probability of true class k for candidate a in context x, and let C(α, k) assign the cost of an action α under that class. Then the minimum-risk action is [17]:

Retaining an entity, rejecting non-entity text, changing a type, and deferring a decision have different consequences. The action set and cost matrix express those differences. This is a decision-theoretic interpretation of the interface; the evaluated review policy is not claimed to solve this optimization. Substituting estimated probabilities requires calibration in the decision domain [19]. The span scores in Equation 8 establish neither that calibration nor a probability of disclosure.

2.3 Keep one identity consistent across records

The accepted spans identify where a transformation may act. They do not yet establish which mentions must share a replacement. Suppose “Mara Chen,” “M. Chen,” and a related email have been resolved to one identity. Represent the assignments of m = 1^Tu accepted mentions to c identity classes by a binary matrix B, with exactly one nonzero entry in each row. Its induced relation H records which mention pairs share an identity:

This representation enforces transitivity: if two mentions share a class with a third, they share a class with each other. Independent pairwise decisions need not satisfy that property [18]. The task is to infer the right partition and then preserve it during replacement; a coherent but incorrect partition would consistently repeat the original resolution error.

Let R assign the c original identity classes to s fictional identities, where s ≥ c. Giving every class one surrogate and preventing collisions between classes yields an exact preservation condition:

The equality follows because the rows of R are distinct one-hot vectors, so RR^T = I_c. It states precisely what “the same fictional person across records” means. A surname and an email can render different attributes of that identity while preserving the relation. If two original classes collide, RR^T gains off-diagonal entries and unrelated mentions merge. If one original class receives several surrogates, a single assignment matrix R no longer describes the transformation and its links can split.

Nyne's evaluated transformation stage uses deterministic identity resolution, surrogate assignment, and contextual rendering. Equation 14 expresses the invariant that this process seeks to preserve. The pairwise measures in Section 6 define how to count departures from it as merges and splits [14].

2.4 Apply the decisions to an entire corpus

The same construction scales from one sentence to a corpus 𝒳 = (x₁, …, x_D). Concatenate only its coordinate inventories: let L be the total number of original character positions and N the total number of typed candidates. Each document has its own contextual span and label matrices; stack their scores and selection vectors in the same document order. A sparse incidence matrix A records which characters each candidate covers. Each column lies entirely inside one document's block. Selection then projects from candidate space back into character space:

The indicator is elementwise: a character is covered if at least one selected candidate contains it. This is the coverage vector underlying Equation 4, now expressed directly as a function of the decisions u. Meanwhile, BR assigns every selected mention a surrogate identity. Together they specify where the renderer acts and which fictional identity supplies each replacement.

For document d, sort its m_d selected spans by original offsets (a_dj, b_dj, ℓ_dj). Let i(d, j) identify the corresponding row of B, and let ψ render the assigned surrogate in the entity type and surface form of that mention. Writing ⊕ for string concatenation and setting b_d0 = 0, the transformed document is:

Each untouched gap is copied from the original document; each selected span is rendered from its shared identity assignment. Replacement strings may have different lengths because every slice still refers to the source coordinates. When a document has no selected spans, the empty concatenation leaves the whole document intact. Applying this operator to every document produces the transformed corpus without treating repeated mentions as independent replacement problems. Coverage specifies the selected characters; rendering must also change their sensitive values appropriately.

2.5 What success requires across records

Preserving the accepted-mention partition cannot recover an identifier omitted before that partition was built. Return to all annotated mentions of an identity e. Let A_e mean that every such mention is correctly localized and typed, B_e that their identity assignments are correct, and C_e that their replacements change the sensitive values coherently. Complete success requires their intersection:

This factorization retains the dependence between stages. A boundary error changes what the resolver receives; a resolution error changes which surrogate the renderer uses. Multiplying their marginal accuracies would erase those interactions. If an earlier conditioning event has probability zero, complete success is zero. Otherwise the conditional factors describe exactly where the remaining risk enters.

The chain connects representation learning to the output requirements: complete spans must survive selection, accepted mentions must form the right identity classes, and replacement must preserve those classes while changing sensitive values. These identity-level events define the analytical target. The following results report the measured detection and transformation endpoints, with their respective populations and denominators.

The benchmark, the metric,
and the comparison field.

Evaluation begins with a defined population, annotation policy, and endpoint. Selection can overfit a noisy metric, as analyzed by Cawley and Talbot [2] and Varma and Simon [3]. We therefore interpret these measurements at their stated scope, distinguish exact localization from overlap, and examine errors beyond the headline score.

3.1 Benchmark populations

We evaluate all five PrivacyBench entity types, then broaden coverage to the 20 native AI4Privacy types and 55 Nemotron-PII types. Each declared cohort retains negative records and every annotated type. The external evaluations use 500 English validation records from AI4Privacy and 500 test records from Nemotron-PII. [8][9]

3.2 Interpreting a reported score

The primary detection results pool counts across records before computing F1. We report 95% bootstrap intervals for Nyne measurements, following the resampling principle of Efron [4]. These quantify sampling uncertainty in the evaluated outputs. Published competitor results are reproduced from their sources; they are not paired reruns on our records. Detailed comparison scope is collected in the source notes.

#1 in exact F1.
The entire field, compared.

Across all nine systems shown, Blackvault Ensemble ranks first in exact-match F1. On 5,198 messages containing 8,805 human-annotated spans, it obtains 94.17 exact F1, a 0.77-point improvement over micro1's published 93.40. Exact precision is 94.44 and recall is 93.90. Blackvault obtains 93.86 exact F1. Blackvault Ensemble's exact-match interval lies above the published reference; Blackvault's higher point estimate has an interval that includes it. These comparisons use reported reference values rather than paired predictions. [7][P1]

Blackvault Ensemble's overlap F1 is 96.14 [95.79, 96.48], numerically above micro1's 96.00; its interval includes that reference. Blackvault ties micro1 at the displayed overlap F1 of 96.00. The difference between the two readings is scientifically useful: exact matching demands type and boundary fidelity, while overlap asks whether entities were touched. The stronger exact result therefore supports a localization claim. It does not isolate which component caused the gain without a controlled ablation.

Equal-export macro F1 changes Blackvault Ensemble's exact score from 94.17 to 94.10 and Blackvault's from 93.86 to 93.83. The small shift shows that the pooled result is not driven solely by weighting larger exports more heavily. Precision, recall, and both F1 readings are retained in the complete table below.

Leading scores.
Two broader label inventories.

PrivacyBench's five labels cannot by themselves establish broad PII coverage. We retain all 20 AI4Privacy and 55 Nemotron-PII native types on fixed 500-record cohorts, including 1,237 and 4,019 annotated spans respectively. Blackvault is evaluated on both.

Results use strict typed exact-span F1: a detection must recover both boundaries and the correct entity type. The complete field includes micro1, NeMo Anonymizer, NVIDIA's published GLiNER-PII model-card result, and a GLiNER-PII control measured on our cohorts. Published figures retain the metric stated by their source.

5.1 Broad PII coverage

Under the strict reading, Blackvault reaches 92.11 [90.39, 93.71] on AI4Privacy and 93.41 [92.42, 94.37] on Nemotron-PII, the highest strict scores in each displayed comparison.

micro1 reports strict-span F1 of 83.74 on AI4Privacy and 88.33 on Nemotron; its NeMo references are 73.08 and 87.84. We preserve that stated metric. The numerical comparison is informative, while the directly measured NVIDIA baseline provides the stronger control on population and scoring. Published rows are not paired reruns. [P2]

Detection and replacement
are coupled objectives.

Replacing each detected string independently can preserve local readability while breaking corpus-level identity relationships. Coherent transformation must keep a person's related names, handles, and email addresses consistent. We evaluate replacement quality separately from detection to distinguish missed entities from failures to transform a detected entity correctly.

6.1 A decomposable endpoint

Tonic's synthesis scorer selects the same-label prediction with the largest intersection for each gold span. Let G be annotated spans, D those detected, and C those changed and judged coherent. Unchanged replacements receive no credit; missing verdicts remain in the denominator. The three reported quantities are:

This factorization distinguishes a missed detection from an incoherent replacement. Equivalently, the combined error can be partitioned into two disjoint contributions:

For the evaluated Blackvault transformation configuration, recall is 98.07% and conditional accuracy is 99.00%. Using the rounded rates, about 1.93 percentage points of annotated spans are missed and 0.98 points are detected without coherent transformation. The residual is about 2.91%, consistent with the measured 97.09% combined result. This endpoint has no precision term because the generated annotation set is not exhaustive.

6.2 Identity links: two distinct failure modes

Local fluency cannot establish relational consistency. On a fixed set of aligned mentions, let A be the set of unordered mention pairs that share an original identity, and let B_T contain pairs assigned the same fictional identity. Pairwise link precision and recall distinguish accidental identity merges from identity splits:

These errors count mention pairs: a merge invents a relationship between different people; a split breaks the connection between two mentions of one person. Pairwise evaluation is established in coreference research [14]. Its pair counts give larger identity clusters more weight; singleton entities and missing mentions need separate coverage accounting. For nonempty pair sets, the equations state a corpus-level consistency objective. The judged transformation scores below measure coherent replacements, not this separate link metric.

6.3 Conditional accuracy and end-to-end success

The evaluated Blackvault transformation configuration records 99.00 [98.76, 99.23] synthesis accuracy and 97.09 [96.68, 97.49] combined (adaptive after two judged looks), alongside micro1's published 98.28 and 95.46. The useful distinction is between conditional replacement quality and total annotated-span success: a coherent replacement cannot recover a span that was never detected.

We use Tonic's synthesis evaluator to assess whether replacements change the source value and remain coherent, with Claude Opus 4.7 providing the stored judgments. Confidence intervals resample document counts while holding those judgments fixed. They describe the reported configuration and do not capture repeated-judge variability. The comparator values are published references, with potentially different evaluation populations and judge configurations.

What the evidence supports.

Three findings emerge. First, exact localization provides the strongest PrivacyBench comparison: Blackvault Ensemble's point estimate and interval exceed the published reference. Second, Blackvault supports broader native-label inventories, with competitive strict scores on AI4Privacy and Nemotron-PII. Third, the transformation evaluation measures detection and coherent replacement together.

Our endpoint is empirical extraction and transformation quality. Differential privacy, for example, constrains output distributions on neighboring datasets, a different mathematical object [5]. The measurements here concern detection, coverage, and coherence; they do not establish such a mechanism-level guarantee.