A Name Isn't an Identity

The DataParade scanner reads a codebase and finds every place a piece of personal data shows up: a parameter, a local variable, an object key, a column name, a form field. On one of our test repositories it finds a few hundred mentions of email addresses. That part was never the hard part. Tree-sitter and a list of patterns will get you there in an afternoon.

The hard part is the question that comes right after. A member's account email, a staff user's email, and the email key in a webhook payload from a payment processor are three different data items. They have different owners, different purposes, and different places they are allowed to go. If you collapse them into one node called "email", every data flow you draw from that node is wrong, and every finding you raise about it is a guess.

The short version: we tried the obvious thing, which is to group mentions by name, and it took our grouping precision on the ghost repository from 86% to 51%. Then we found the same failure described in someone else's project notes, which told us the problem was general. The fix was to join only on evidence the parse tree can show you, and to treat a shared name as a hint, never as proof.

What "grouping precision" means here

We score grouping pairwise. Take any two mentions the scanner has placed in the same data item, and ask whether the labeled corpus agrees that they belong together. Precision is the share of those pairs the labels confirm. Recall is the share of labeled pairs the scanner found. We report them per repository and we do not average them into one number, because the two test repositories fail in different ways and a single score would hide that.

On ghost, grouping by declaration alone sat at 86% precision. That is a low bar in one sense, since it only joins mentions that share a declaration in the same scope. It is a high bar in another, since every one of those joins has a line of code behind it.

Joining on names: 86% to 51%

The first thing we tried to raise recall was to join mentions with the same name inside the same file. It seemed safe. Two variables called userEmail in one file are usually the same thing.

They are not usually the same thing. Precision fell to 51% on ghost, which is close to a coin flip. The failures were what you would expect once you look: a request handler with a local email for the person logging in, and twenty lines later a local email for the person being invited. A test file with the same fixture name reused for three different users. A migration that renames a column and mentions both the old name and the new one.

Name words carry information. They just don't carry identity.

Someone else hit this wall and backed away from it

While this was going on, we were reading the notes for Graphify, a tool that maps a codebase into a graph of files, classes, functions and calls. Their early versions gave each node an ID built from its name. Two local variables called set or user in unrelated files got the same ID and collapsed into one node, which then appeared connected to everything. Their notes call these "phantom god-nodes". Their fix was to stop emitting nodes for local variables at all and keep only module-level declarations and class fields, where names are scoped well enough to be safe.

That is a reasonable trade for a code map. A code map does not need to know what each local variable holds.

We cannot make that trade, because the locals are where the data is. Most of the email mentions in a web application are parameters, locals and object keys. If we drop them, we drop the thing we exist to find. So we had to answer exactly the question Graphify stepped around: when do two mentions refer to the same data item?

Join only on evidence

The rule we landed on is simple to state. Two mentions are joined when the parse tree shows a path between them, and only then. The paths we accept today:

1. A shared declaration. Both mentions resolve to the same variable, parameter or field in the same scope.
2. A field read. user.email and the email field declared on user's type.
3. A value passed into a parameter. The argument at a call site and the parameter it binds to in the callee.
4. A typed receiver. A method call whose receiver has a known type that declares the field.

Everything else is a hint. A shared name, a shared file, a shared database table: each adds a vote toward a candidate join, and a vote can push a borderline case over the line. But two groups with different names are never merged on the strength of a vote alone, and every join we do make carries the line of code that justified it.

With that rule in place, ghost went back to the mid-80s on precision and recall came up from where declaration-only grouping had left it.

Every join carries its evidence

The part that matters for a privacy tool is not the precision number. It is that each link in a data item has a type. Declaration links count as found in the code. Name, file and table votes count as inferred. When a reviewer opens a data item and asks why the scanner thinks these forty mentions are one thing, the answer is a list of specific lines, each with a reason, and the inferred ones are marked as inferred.

That is the difference between a finding and an opinion. A finding can be checked, diffed when the code changes, and corrected when it is wrong. An opinion can only be re-prompted.

What we haven't tested

• Cross-file import resolution. Our engine does not yet follow a value through import and export across files, which is the biggest remaining gap in recall. Graphify's import resolution covers path aliases, workspace packages and re-exports in about 4,000 lines, and its license allows reuse with attribution. We expect to borrow the design.
• Languages other than TypeScript, JavaScript and Ruby. The Ruby support is being tuned on Chatwoot and is behind the others.
• Whether the inferred-link votes are calibrated. We have not yet measured how often a name vote is right when it is the deciding factor.

Lessons

1. A name is a hint, not a key. The moment you use it as a key you inherit every phantom node that Graphify already found and fixed by deletion.
2. Precision has to be measured pairwise. Whole-answer scores hide the fact that one bad merge poisons every flow drawn from the merged node.
3. Keep the evidence on the edge. If the join can't name the line that justified it, it isn't a join, it's a guess with good posture.

The scanner, the fixture ground truth and the pairwise scorer are in the DataParade scanner repository.