Building an Identity Graph for Clickstream Data

Clickstream data is easy to collect and hard to use. Every modern system can emit page views, taps, API calls, and application events with timestamps and attributes. The trouble starts when analysis or downstream services require a notion of “user.”

In most production systems, identity is incomplete by default. Many events arrive without a logged-in account. Cookies reset. Mobile devices are shared. IP addresses rotate. A single person often appears as several disconnected records, while unrelated users occasionally collide on the same attributes.

This article walks through an engineering-first approach to identity resolution using a graph model. The emphasis is not on achieving a perfect user map, which is usually impossible, but on building a system that produces explainable, governable approximations that downstream teams can trust.

The discussion is structured as explicit system layers, mirroring how most engineers reason about production pipelines.

System Overview: Identity as a Layered Architecture

A practical identity graph is not a single database or algorithm. It’s a layered stack, with each layer being responsible for a narrow slice:

1. Ingestion and normalization
2. Graph construction
3. Identity resolution
4. Governance and privacy enforcement
5. Operational integration

With distinct layers, we avoid rule sprawl, make audits much simpler, and can evolve one bit of the system without rewriting everything all at once.

1. Ingestion and Normalization Layer

The ingestion layer is where most identity systems quietly fail. If identifiers are inconsistent here, no amount of matching logic will fix it later.

A typical raw clickstream event includes a mix of stable and unstable attributes:

• Device or browser identifiers
• Application instance IDs
• Session tokens
• IP addresses
• Coarse location signals
• Optional account ID

The goal of normalization is not enrichment. It is consolidation and constraint.

Key decisions at this stage:

• Preserve original identifiers verbatim in immutable storage.
• Normalize formats early (for example, lower-casing device IDs, canonicalizing IPs).
• Reduce precision where possible (store IP blocks instead of full addresses).
• Discard identifiers that cannot be justified operationally.

A simplified normalization step might emit rows like:

(event_id,
 event_ts,
 device_id,
 ip_block,
 location_id,
 account_id)

At this stage, there is no inferred user. The output is intentionally boring. That is a feature.

2. Graph Construction Layer

The graph layer materializes entities and relationships explicitly. This is where identity stops being a collection of columns and becomes a structure.

Schema and Constraints

Using Neo4j as an example, constraints anchor the model and prevent accidental duplication:

CREATE CONSTRAINT event_id_unique IF NOT EXISTS
FOR (e:Event)
REQUIRE e.event_id IS UNIQUE;

CREATE CONSTRAINT device_id_unique IF NOT EXISTS
FOR (d:Device)
REQUIRE d.device_id IS UNIQUE;

CREATE CONSTRAINT account_id_unique IF NOT EXISTS
FOR (a:Account)
REQUIRE a.account_id IS UNIQUE;

Without these constraints, graph growth quickly becomes unmanageable.

Nodes and Edges

A minimal schema typically includes:

Nodes

• Event
• Device
• IPBlock
• Location
• Account

Edges

• (Event)-[:USES_DEVICE]->(Device)
• (Event)-[:FROM_IP_BLOCK]->(IPBlock)
• (Event)-[:NEAR_LOCATION]->(Location)
• (Event)-[:TIED_TO_ACCOUNT]->(Account) when present

Materializing a clickstream row into the graph is mechanical:

MERGE (e:Event {event_id: $event_id})
SET e.ts = $timestamp,
    e.event_type = $event_type;

MERGE (d:Device {device_id: $device_id})
MERGE (e)-[:USES_DEVICE]->(d);

Growth and Pruning

Graphs grow very quickly; we accumulate thousands of events over time, IP blocks connect unrelated random users, and it becomes a more expensive operation to traverse. In practice, systems impose limits:

• Edges timed out (all relationships should expire)
• Event fan-out per device should be limited
• Aggregation nodes for very noisy attributes

One of the most common failures in production systems is forgetting about how the graph will grow.

3. Identity Resolution Layer

This is where the judgment lives, where the “what are the consequences of this judiciously” lives, and actually, where we don’t match is oftentimes more important than aggressively matching.

Edge Weighting

Some relationships are more significant than others, and they themselves carry a lot of institutional knowledge:

EDGE_WEIGHTS = {
    "USES_DEVICE": 0.6,
    "FROM_IP_BLOCK": 0.2,
    "NEAR_LOCATION": 0.1,
    "TIED_TO_ACCOUNT": 1.0
}

These are not universal for all institutions to share; they are specific to your traffic pattern and threat model.

Path-Based Scoring

Using NetworkX for illustration:

def compute_confidence(graph, event_node, account_node):
    score = 0.0
    for path in nx.all_simple_paths(
            graph, event_node, account_node, cutoff=3):
        path_score = 1.0
        for u, v in zip(path[:-1], path[1:]):
            path_score *= EDGE_WEIGHTS.get(
                graph[u][v]["type"], 0
            )
        score += path_score
    return score

Several constraints are deliberate:

• Traversal depth is capped
• Only simple paths are considered
• Multiple weak paths can accumulate evidence

Threshold Enforcement

Identity inference stops explicitly:

if confidence_score >= MATCH_THRESHOLD:
    inferred_user_id = account_id
else:
    inferred_user_id = None

Leaving events unmatched is not a failure. It is often the correct outcome when signals conflict.

False Positives vs. False Negatives

Most teams underestimate the cost of false positives. Incorrectly merging two users contaminates analytics, personalization, and compliance workflows. False negatives merely leave data unused.

Thresholds should be biased toward caution unless there is a compelling business case otherwise.

4. Governance and Privacy Enforcement Layer

Governance is not documentation. It is an executable policy.

Key practices include:

• Attribute minimization: store only what is required by matching.
• Tokenization and hashing of raw identifiers.
• Role-based access to sensitive nodes and edges.
• Audit logging for identity rule execution.

Every inferred mapping should be explainable as a set of graph paths and weights. If you cannot explain a match, you should not ship it.

5. Operational Integration Layer

The identity graph is only useful if it integrates cleanly with the rest of the stack.

Batch Enrichment

A common pattern is a warehouse-friendly mapping table:

enriched_events = (
    events_df
    .join(identity_map_df, "event_id", "left")
    .select("event_id", "inferred_user_id", "confidence_score")
)

enriched_events.write \
    .mode("overwrite") \
    .saveAsTable("event_identity_enriched")

This keeps analytics reproducible and auditable.

Streaming and Services

“Goes from batch to streaming. Being streamed automatically injects more risk. You need your rules to be stable (or else) and latency aware, you need your wider infrastructure to be observability frugal, and the latter is easy to defer if you start with batch and move to streaming only when other teams are explicitly telling you that you have a latency requirement.

Failure Modes and Operational Guardrails

Identity systems fail silently  — how do we surface that they are failing as early as possible?

• limit traversal depth
• version matching rules explicitly
• backtest threshold changes
• sample matches for human review
• maintain rollback paths for rule updates

And remind your team that this is like a production dependency, not an experiment.

Conclusion

Our identity resolution system is not discovering the secret holy grail of “true user” or doing any of that nonsense. It is about constructing a defensible approximation that a downstream system can reason about and that an auditor can dig into.

Where we trace a user is explicit to the user, relationship scoring is explicit to the data engineer, and where our governance must be enforced as a result is also explicit to the governance staff. And when layered, with clear stop points, it is part of our data platform, not a fragile bag of joins.

For the engineering owner of a fragmented clickstream, that change in framing is most of the value.