DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Newsletter
Log In / Join
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

Related

  • Running TypeSafe AI's Jev Inside Neo4j for Confidence-Aware Sentiment Decisions
  • Your AI Coding Assistant Stopped Suggesting and Started Shipping. Now What?
  • Your AI Agent Trusts Every Tool It's Ever Been Introduced To; That's the Whole Problem
  • GraphRAG in Practice Using Spring AI, Neo4j, and Goodreads Data

Trending

  • Build a Voice Assistant With Gemini Live API
  • Prompt Caching for LLMs, Part 2: Semantic Caching in Production
  • Kafka Consumers on EKS: Designing for a Platform That Restarts Your Consumer Group
  • Architecture Review Boards for Generative AI: A Framework for Governance, Risk Management, and Responsible Adoption

Draft This article hasn't been published, so readers can't see it.

In moderation This article is waiting for review, so readers can't see it yet.

Deleted This article has been deleted, so readers can't see it.

  1. DZone
  2. Data Engineering
  3. AI/ML
  4. When Jev Says It's Sure, Is It Right? A Fraud Case Study in Neo4j

When Jev Says It's Sure, Is It Right? A Fraud Case Study in Neo4j

Jev’s confidence exceeds its accuracy, and only its top range holds up, mostly on easy cases. Use confidence for triage and error rates for automation thresholds.

By 
Akmal Chaudhri
DZone Core CORE ·
Nov. 17, 26 · Analysis
Post on X (opens in a new tab) Share on LinkedIn (opens in a new tab)
6 Views

Join the DZone community Save articles, join the discussion, and download Refcards and Trend Reports with a free account.

Join for free Already a member? Log in

In Part 3 of our sentiment series, we ended on a cautious note: a model can be highly confident and still be wrong. This article puts a number on that caution.

We generate 500 synthetic fraud transactions with hidden labels, ask Jev to flag or pass each one, store every decision in Neo4j, and then ask the question that matters for automation: when Jev says it's sure, is it right? Because the decisions live in a graph, the join to the hidden labels, a cross-check of the reliability bins, and the review queues are each a Cypher query, and pandas does the metrics.

The short answer is that Jev is more confident than it should be almost everywhere. Only at the very top of its range do the results hold up and, as we'll see, that's mostly because the cases up there are easy. We'll also show a mistake we made along the way, because it's easy to repeat.

The full source code is available on GitHub.

Why Calibration Matters

A probability only helps if it means what it says. If the decisions Jev scores at 0.95 are right 95% of the time, we can automate those and send the rest to a person. If they're right 80% of the time, the automation quietly lets errors through. A model is calibrated when its stated confidence matches how often it's actually right. Until we test them against known outcomes, Jev's probabilities are the model's own claims, not measured rates.

Fraud review is a good fit for this question. In real life, most transactions are routine, and a few are clearly fraudulent, but a stubborn middle is genuinely hard. Those are the cases where a person's time is best spent and where an overconfident model does the most damage.

What We're Building

Three standalone notebooks that share data through a data folder:

  1. Data and Probe. Generate synthetic transactions with hidden fraud labels and check the shape of Jev's response.
  2. Decisions in Neo4j. Load the transactions, get a decision for each one and store it as a Decision node.
  3. Calibration Analysis. Measure calibration with Cypher and pandas.

The graph model is deliberately small:

Plain Text
 
(:Transaction {txn_id, amount, merchant_category, ..., scenario, true_p, is_fraud})
  -[:HAS_DECISION]->
(:Decision {decision_id, choice, confidence, score, p_flag, p_pass, config_hash})


The scenario, true_p and is_fraud properties live on the Transaction node so the analysis can score each decision. They're never part of a request to Jev.

Prerequisites

  • Neo4j. Aura Free is enough, because the whole study fits in one database.
  • Jupyter in a Python 3.12 virtual environment. Each notebook installs its own pinned libraries.
  • A Jev API key (optional). It's only needed to make live calls. The decisions we analyze here ship with the notebooks, so you can reproduce every number without a key.

Set the following environment variables before launching Jupyter:

Shell
 
export NEO4J_URI="neo4j+s://xxxx.databases.neo4j.io"
export NEO4J_USERNAME="your-username"
export NEO4J_PASSWORD="your-password"
export NEO4J_DATABASE="your-database"
export TYPESAFE_API_KEY="your-api-key"  # only for live calls


Warning: The second notebook resets the database by default, removing all data, relationships, indexes, and constraints. That keeps a free Aura instance from filling up with old runs, but it means you should point it at a database you don't mind wiping clean, or set RESET_DATABASE = False.

The Synthetic Data

Real fraud labels are scarce, slow to arrive, and private. With synthetic data we know the truth for every transaction, so we can measure calibration directly. The price is that "correct" only means it agrees with the generator, so treat the results as a demonstration of the method rather than a verdict on real fraud.

Each transaction belongs to one of seven scenarios. A scenario sets how the visible features look, and a latent probability of fraud, and the label is then drawn from that probability. A clear_fraud transaction is fraud 97% of the time and a traveler transaction 12% of the time.

Scenario Share of data Latent fraud probability Type
routine 30% 0.01 Clear
clear_fraud 12% 0.97 Clear
takeover 8% 0.95 Clear
traveler 12% 0.12 Ambiguous
big_purchase 12% 0.18 Ambiguous
card_testing 13% 0.65 Ambiguous
night_atm 13% 0.40 Ambiguous


Half the data are deliberately ambiguous. In those scenarios, the visible signals only partly predict fraud, which gives the middle of the confidence range something to measure. Without them, we'd only be testing easy cases. The generator produces 500 transactions, 189 of them labeled fraud. That's a fraud rate of 37.8%, far above real-world rates, and Jev doesn't know the generator's base rates for each scenario. Accuracy and error rates here depend on that mix and won't carry over to real data.

Jev sees the eight visible fields as a JSON string and nothing else:

JSON
 
{
  "amount": 2992.06,
  "merchant_category": "travel",
  "distance_from_home_km": 8.1,
  "hour_of_day": 14,
  "account_age_days": 1483,
  "prior_flags": 0,
  "txns_last_24h": 2,
  "typical_monthly_spend": 3230
}


Probing Jev and a Trap

Each request carries two questions, and Jev answers both in a single call. The first is a choice: flag this transaction for review or pass it through. The second is a score: how confident is Jev in its own decision. Jev rates itself against three levels, from not confident to very confident, and returns a continuous score between 0 and 2, the probability-weighted average of the levels. A choice question takes its options as a map and a score question takes its levels as an array. Mixing the two shapes returns a 422 error.

For the first transaction, the choice part of the response looks like this:

JSON
 
{
  "choice": "Pass",
  "confidence": 0.74,
  "probabilities": {
    "Flag": 0.13,
    "Pass": 0.87
  }
}


It's tempting to read confidence: 0.74 as "Jev is 74% sure it's right." That's what we did first and it's wrong. The reliability diagram told us so: the 29 decisions with a confidence of 0.10 or less were right about 45% of the time, nowhere near the 10% or less that reading would imply.

The answer is in the numbers above. Pass is 0.87, and Flag is 0.13, and 0.87 minus 0.13 is 0.74. With two options, confidence is the margin between them. A margin of 0 is a coin flip and a margin of 1 is certainty. The probability that Jev is right is the one attached to the option it chose, which is (1 + confidence) / 2, or 0.87 here. That formula rests on two assumptions: the two probabilities sum to 1, and confidence is the chosen option's probability minus the other's.

We now check both assumptions on every decision. The probabilities sum to 1, the margin matches the gap between them to within the two-decimal rounding in Jev's response, and the probability of the chosen option matches (1 + confidence) / 2 to within half a hundredth. Two decisions, T0066 and T0175, are exact coin flips at 0.50 and 0.50. Jev picked Flag for one and Pass for the other, so a tie can go either way, but their margin is zero and the formula still holds. The lesson is a general one: check what a field means before you build a metric on it!

Storing Decisions in the Graph

The second notebook loads the transactions and stores a decision for each one. Two uniqueness constraints come first, so a re-run can never create duplicates:

Cypher
 
CREATE CONSTRAINT txn_id IF NOT EXISTS
FOR (t:Transaction) REQUIRE t.txn_id IS UNIQUE;

CREATE CONSTRAINT decision_id IF NOT EXISTS
FOR (d:Decision) REQUIRE d.decision_id IS UNIQUE;


The decision_id is the transaction ID joined to a config_hash. The hash is a fingerprint of everything that shapes a decision: the model, the questions, the options, the fields Jev sees, and a version number for how the state is built. Change any of them, and the hash changes, so old decisions are never mistaken for new ones. It also means we can keep several configurations in one graph and choose which one to analyze. The hash identifies a configuration, not a single execution: run the same configuration again and every transaction already has a decision, so nothing new is stored. To compare two live runs of one configuration, export the first and reset the database, or add a run identifier to the decision_id.

Every decision is written with one query:

Cypher
 
UNWIND $rows AS row
MATCH (t:Transaction {txn_id: row.txn_id})
MERGE (d:Decision {decision_id: row.decision_id})
  ON CREATE SET d.created_at = datetime()
SET d += row.props
MERGE (t)-[:HAS_DECISION]->(d)


Repeating a write never adds a second copy. The uniqueness constraint and the MERGE on decision_id mean a repeat updates the same node and the second MERGE does the same for the HAS_DECISION relationship. The created_at timestamp is set with ON CREATE, so it records when the decision was first stored, and a repeat doesn't overwrite it.

There are two ways to fill the graph, and both use that same query, so a decision looks identical whichever way it arrived:

  • Import the shipped decisions. This is the default. It reads decisions.csv into Neo4j and needs no key, so everyone analyzes the same stored run.
  • Call Jev live. The notebook asks Neo4j which transactions have no decision for the current configuration, sends them to Jev with four parallel workers, and writes the results in batches of 25. An interrupted run picks up where it stopped, and the run stops early after five failures in a row instead of burning through the rest. A live run ends by exporting the decisions to a CSV file. If decisions.csv already exists, the export goes to a new timestamped file, so the shipped run is never overwritten.

A quick check confirms the load: 500 Decision nodes, 500 HAS_DECISION relationships, 253 Flags and 247 Passes.

How We Measure Calibration

We compare two signals from the same call:

  • The probability of the chosen option. This is the larger of the two probabilities, between 0.5 and 1.
  • The self-rating. This is Jev's own score, a continuous value between 0 and 2 that averages its three confidence levels. We divide it by 2 to get a value between 0 and 1. It's still an ordinal rating rather than a probability.

Both are scored against a simple definition of correct: Flag when the transaction is fraud and Pass when it isn't. Expected calibration error (ECE) and Brier score are probability metrics, so they suit the first signal. Rescaling an ordinal rating to 0 to 1 doesn't turn it into a probability, so for the self-rating we lean on AUROC (Area Under the Receiver Operating Characteristic) and treat the other two as exploratory. The metrics are:

  • Reliability diagram and ECE. Group decisions by signal strength and compare each group's average signal with its accuracy. ECE is the average gap, weighted by group size. Zero is perfect.
  • Brier score. The mean squared gap between the signal and the outcome. Lower is better.
  • AUROC. How well a signal separates right decisions from wrong ones. 0.5 is chance and 1.0 is perfect. It only uses the ordering of a signal.
  • Threshold sweep. If we automate every decision above a cutoff, how much do we automate and how many errors slip through?
  • Saturation. How often a signal sits at its ceiling, where it can't tell cases apart.

Reliability, Checked Twice

One query joins each decision to its transaction's hidden label, and pandas works out the probability of the chosen option as the larger of p_flag and p_pass. That probability runs from 0.5 to 1, so we use five bins of width 0.1.

Then we count the same bins inside Neo4j, as a cross-check that the pandas pipeline reads the graph the way we think it does. The query takes three parameters: $hash picks the run, $lo is 0.5, the lowest value the probability of the chosen option can take, and $n_bins is 5:

Cypher
 
MATCH (t:Transaction)-[:HAS_DECISION]->(d:Decision {config_hash: $hash})
WHERE d.p_flag IS NOT NULL AND d.p_pass IS NOT NULL
WITH d, t,
     CASE WHEN d.p_flag >= d.p_pass THEN d.p_flag ELSE d.p_pass END AS p_chosen
WITH d, t, p_chosen,
     floor((p_chosen - $lo) * $n_bins / (1.0 - $lo) + 0.000001) AS position
WITH d, t, p_chosen,
     toInteger(CASE WHEN position >= $n_bins THEN $n_bins - 1 ELSE position END) AS bin
WITH bin, p_chosen,
     CASE WHEN (d.choice = 'Flag') = (t.is_fraud = 1) THEN 1.0 ELSE 0.0 END AS correct
RETURN bin,
       count(*)      AS n,
       avg(p_chosen) AS mean_p_chosen,
       avg(correct)  AS accuracy
ORDER BY bin


The two tables agree on every count. Here are the bins:

Probability bin Decisions Stated (mean) Actual accuracy
0.5 to 0.6 62 0.553 0.387
0.6 to 0.7 49 0.642 0.531
0.7 to 0.8 46 0.746 0.630
0.8 to 0.9 86 0.848 0.767
0.9 to 1.0 257 0.965 0.930


In every bin, Jev's stated probability is higher than its accuracy. The gap is widest at the bottom, where Jev says 0.553 and is right 38.7% of the time, and narrows toward the top, where it says 0.965 and is right 93.0% of the time.

Results

Across all 500 decisions, Jev's average stated probability is 0.842, and its accuracy is 0.768, so it's overconfident by about seven points. Because every bin errs in the same direction, the ECE of 0.074 is the same as that overall gap in absolute terms.

Signal Subset Mean signal Accuracy ECE Brier AUROC
Probability of chosen option All scenarios 0.842 0.768 0.074 0.146 0.812
Probability of chosen option Ambiguous only 0.732 0.544 0.188 0.278 0.602
Self-rating (rescaled) All scenarios n/a 0.768 n/a n/a 0.669
Self-rating (rescaled) Ambiguous only n/a 0.544 n/a n/a 0.562


We leave out the mean signal, ECE, and Brier score for the self-rating. It's an ordinal rating built on three levels, and rescaling it to 0 to 1 doesn't make it a probability. The notebook still prints those numbers as exploratory diagnostics. For this signal, AUROC is the figure to read.

In the ambiguous scenarios, the overconfidence is much larger: a stated 0.732 against an accuracy of 0.544, a gap of nineteen points.

The Top of the Range Looks Better

The picture changes at the very top. Of the 72 decisions at 0.99 or above, 71 are right. That's 14.4% of all decisions at an observed error rate of 1.4%. The threshold sweep shows what we get as we lower the cutoff:

Cutoff Decisions automated Error rate among automated
0.50 100.0% 23.2%
0.70 77.8% 14.1%
0.80 68.6% 11.1%
0.90 51.4% 7.0%
0.95 40.4% 4.0%
0.99 14.4% 1.4%


A cutoff of 0.95 automates 40.4% of the decisions at an observed error rate of 4.0%. Whether that's acceptable is a business question, but the curve gives us the trade-off in numbers rather than by feel.

Those are observed rates from one run, and they're not the rates we should expect on new data. At the top, the groups are small. A 95% Wilson interval, a binomial confidence interval that assumes the decisions are independent, puts the accuracy of the 72 decisions at 0.99 or above somewhere between 92.5% and 99.8%, which means an error rate anywhere from 0.2% to 7.5%. For the 202 decisions at 0.95 or above (8 of them wrong), the error rate could plausibly be anywhere from 2.0% to 7.6%. We also chose these cutoffs after looking at the results, which makes the observed numbers a little optimistic. The notebook computes both intervals, so you can check them.

But Most of the Top Is Easy Cases

Before we lean on that, look at who is up there. Of the 72 decisions at 0.99 or above, 47 are routine, 18 are takeover, 6 are clear_fraud and only 1 comes from an ambiguous scenario. These are the cases where nearly any reasonable judge gets the right answer. Splitting the same cutoffs by scenario type shows how much that matters:

Cutoff Clear: decisions Clear: error rate Ambiguous: decisions Ambiguous: error rate
0.90 217 0.9% 40 40.0%
0.95 187 0.5% 15 46.7%
0.99 71 1.4% 1 0.0%


In the clear scenarios, a high cutoff behaves as hoped. In the ambiguous ones, decisions at 0.90 or above are wrong 40% of the time and decisions at 0.95 or above nearly half the time, although those groups hold only 40 and 15 decisions. The 0.90 to 0.99 band as a whole (185 decisions) says 0.954 and is right 90.8% of the time. So a high probability from Jev tells us partly that the case is an easy one and much less that this particular answer is safe.

The one error at 0.99 or above is a takeover transaction that the generator happened to label legitimate, which it does 5% of the time. That error is noise in the labels rather than a failure of judgment. Treat the ambiguous rows as a warning, not a measurement.

Where the Overconfidence Comes From

Averages can hide where the problem is, so we split by scenario:

Scenario Decisions Accuracy Best expected accuracy Mean stated probability
traveler 60 0.333 0.88 0.596
night_atm 65 0.462 0.60 0.696
card_testing 65 0.662 0.65 0.789
big_purchase 60 0.717 0.82 0.847
takeover 40 0.950 0.95 0.975
clear_fraud 60 1.000 0.97 0.918
routine 150 1.000 0.99 0.959


"Best expected accuracy" is the expected accuracy of a judge that knows each scenario's fraud probability. It isn't a ceiling for any single sample. If a scenario is fraud 12% of the time, the best a judge can expect is to call everything legitimate and be right 88% of the time. A few scenarios land at or just above that figure by chance.

The overconfidence is concentrated in the four ambiguous scenarios. In the easy ones, Jev is right nearly every time. takeover is slightly overconfident too (0.975 against 0.950), while clear_fraud and routine are underconfident: Jev says 0.918 and 0.959 and is right 100% of the time.

The best expected accuracy column separates two questions. How accurate Jev can be is limited by the labels: in night_atm, nobody can expect to beat 0.60. Jev gets 0.46 there and in traveler it gets 0.33 against a best expected accuracy of 0.88, so it falls well short of what was achievable. The shortfall in traveler is over-flagging: Jev flags 45 of the 60 transactions and 38 of those flags are false alarms. Whether Jev's confidence is real is a different question and the random labels don't change it. A judge that says 0.6 should be right about 60% of the time even when outcomes are random. Jev says 0.70 in night_atm and is right 46% of the time, so the overconfidence is real.

There's a second, subtler finding. The overall AUROC of 0.81 looks strong, but it comes from telling easy cases from hard ones. Inside a single ambiguous scenario, the probability barely ranks right answers above wrong ones: the AUROC is 0.58 for traveler, 0.49 for night_atm, 0.42 for card_testing and 0.46 for big_purchase. Jev knows which kind of transaction is hard. It's much less able to say which individual case it'll get wrong.

What About the Self-Rating?

Jev's self-rating tells us something, but not as much as we might hope. Its rank correlation with the probability is 0.48, so the two signals overlap only partly, and its AUROC of 0.67 is lower than the probability's 0.81. It also sits in the middle of its scale: no decision reaches 0.8 once rescaled. In the notebook's automation plot, its curve actually runs below the probability's at the far left, because the 110 highest-rated decisions (22% of the total) are all right. But 108 of those are routine transactions, so the self-rating is mostly picking out the plainly legitimate ones. It rates clear_fraud transactions lower than routine ones (0.41 on average against 0.65), although Jev is right about all 60 of them.

At the top of the range, it adds a hint at most. Among the 72 decisions at 0.99 or above, 16 have a self-rating below the "Somewhat confident" level, and 15 of those are right. The other 56 are all right. The single error, a takeover transaction flagged at 1.00, has a self-rating of 0.99, just under that level. That's a point in the self-rating's favor, but one error in a group of 72 is too little to build a rule on.

Review Queues in One Query Each

The point of a calibrated signal is a better queue, and the two kinds of mistakes cost different things. We build one queue for each. The first lists the flagged transactions that Jev is least sure about, so a person looks at the shaky ones first:

Cypher
 
MATCH (t:Transaction)-[:HAS_DECISION]->(d:Decision {config_hash: $hash})
WHERE d.choice = 'Flag' AND d.p_flag IS NOT NULL
RETURN t.txn_id            AS txn_id,
       t.amount            AS amount,
       t.merchant_category AS merchant_category,
       d.p_flag            AS p_chosen,
       d.score             AS score,
       t.is_fraud          AS is_fraud
ORDER BY d.p_flag ASC, d.score ASC
LIMIT 15


Of the 15 least certain flags in our run, 13 are legitimate transactions. Jev's shakiest flags are mostly false alarms, which is exactly the signal a reviewer wants.

This is a false-positive queue, though. It catches legitimate transactions that were flagged by mistake and says nothing about fraud that Jev let through. Missed fraud usually costs more than a needless review, so we want the mirror image as well: the passes Jev is least sure about.

Cypher
 
MATCH (t:Transaction)-[:HAS_DECISION]->(d:Decision {config_hash: $hash})
WHERE d.choice = 'Pass' AND d.p_pass IS NOT NULL
RETURN t.txn_id            AS txn_id,
       t.amount            AS amount,
       t.merchant_category AS merchant_category,
       d.p_pass            AS p_chosen,
       d.score             AS score,
       t.is_fraud          AS is_fraud
ORDER BY d.p_pass ASC, d.score ASC
LIMIT 15


The notebook runs both queries. In our run, 5 of the 15 passes Jev was least sure about turned out to be fraud, so the second queue earns its place. Both queries return is_fraud only so we can evaluate the synthetic experiment. A live review queue wouldn't have that variable, so we'd drop it. A complete review strategy would weigh them together with the amount at stake, and it would set capacity limits for each queue. We only show the mechanics here. Because every Decision hangs off its Transaction, adding the account, the merchant, or a ring check later is one more MATCH.

Speed

The notebook times each call as a client-side round trip. We ran it from a laptop with four calls in flight at a time, so treat the numbers as indicative. Across the 500 decisions, the median was about 220 milliseconds and the slowest was under 400.

Caveats

Keep these in mind before drawing conclusions:

  • The labels are synthetic. "Correct" means agreeing with the generator, not catching real fraud. The method carries over to real data. The numbers don't.
  • The fraud rate is far above real life. About 38% of these transactions are fraud, against a tiny fraction of real transaction volume. Accuracy, error rates, and the share you can automate all depend on that base rate, so don't carry any of them over.
  • Jev doesn't know the generator's base rates. The generator decides how often each scenario is fraud, and Jev sees only the visible fields. It flags 45 of 60 traveler transactions, where 15% are fraud, and 60 of 65 card_testing transactions, where 68% are. In the other direction, it flags just 1 of 60 big_purchase transactions, where 27% are fraud. Across all 500, it makes 90 false positives and misses 26 frauds. Some of what we measured as overconfidence is a mismatch between Jev's assumptions and this generator's, and real data would bring its own.
  • The ambiguous scenarios limit accuracy, not calibration. The generator draws labels at random, so no judge can be right every time, and the best expected accuracy column shows the limit. That limit doesn't excuse overconfidence, because calibration is measured against random outcomes.
  • Observed rates aren't future rates. The error rate at a high cutoff comes from a small group of decisions, and we chose the cutoff after looking at the data.
  • The self-rating is self-reported and ordinal. It's Jev's own description of its confidence, and it takes only three levels. It isn't a probability, so read its AUROC and treat any calibration metric on it as exploratory.
  • The probabilities are coarse. Jev rounds them to two decimals, so many decisions share the same value.
  • The results are for one model version. We used jev-1.13.0. TypeSafe AI has since released jev-1.14.0, so rerun the study before assuming a newer model is calibrated the same way.
  • Results vary between runs. Live runs of the same 500 transactions give slightly different decisions each time, so the number of Flags moves from run to run. The shipped file is one stored run, so your numbers will match ours exactly only if you use it.
  • The sample is small. With a few hundred decisions, the bins in the middle of the range hold few points, so the curve there is noisy. Treat the numbers as an illustration of the method.

Summary

We stored 500 fraud decisions in Neo4j and measured how far Jev's confidence can be trusted. Jev is more confident than it should be, by about seven points overall and nineteen in the ambiguous scenarios. Only at the very top did the results hold up: decisions at 0.99 or above were right 71 times out of 72, but almost all of them were easy cases. In the ambiguous scenarios, decisions at 0.90 or above were right only 60% of the time.

The practical lesson isn't that a particular threshold is safe. On this synthetic dataset, Jev's highest-confidence decisions had strong observed accuracy, mostly because they were easy cases, while the middle of the range was systematically overconfident. A high score tells us partly that a case is easy, so confidence works better for triage than as a guarantee, and thresholds should be validated rather than assumed. Automation policies should rest on measured error rates, on the uncertainty around them and on what each kind of mistake costs. A missed fraud and a needless review aren't the same price.

Neo4j made that measurement cheap. A config_hash kept configurations from mixing and one write query served both the shipped data and live calls. One query joined every decision to its label, the reliability bins could be cross-checked with a single query, and each review queue was a single query. The reliability diagram built from that data also helped us catch our own mistake: confidence is a margin, not a probability.

More Jev examples can be found in the free online books Seven Ways to Use Jev and Lessons Learned Putting Jev Inside Neo4j.

The full source code is available on GitHub.

IT Neo4j AI

Opinions expressed by DZone contributors are their own.

Related

  • Running TypeSafe AI's Jev Inside Neo4j for Confidence-Aware Sentiment Decisions
  • Your AI Coding Assistant Stopped Suggesting and Started Shipping. Now What?
  • Your AI Agent Trusts Every Tool It's Ever Been Introduced To; That's the Whole Problem
  • GraphRAG in Practice Using Spring AI, Neo4j, and Goodreads Data

Partner Resources

×

Comments

Loading comments

The comments didn't load

Please try again in a moment.

Likes

Loading

The next likes didn't load. Please try again.

The likes didn't load

Please try again in a moment.

See who liked this

Log in or join DZone for free to see who liked this article.

Log in Join for free
  • RSS
  • X
  • Facebook

ABOUT US

  • About DZone
  • Support and feedback
  • Community research

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 215
  • Nashville, TN 37211
  • [email protected]

Let's be friends:

  • RSS
  • X
  • Facebook