Data

The Moment It Gets Real At some point in the last year, every data engineer had the same experience. You opened a copilot tool, typed a rough description of what you needed, and watched it generate a working ETL pipeline in about thirty seconds. Not a skeleton. Not pseudocode. Actual, runnable PySpark with joins, transformations, and a DAG scaffold. And for a moment, the question that the industry had been treating as hypothetical became very concrete: if AI can do this, what exactly am I here for? That question deserves a serious answer — not the dismissive "AI is just a tool" reassurance, and not the catastrophist "engineers are obsolete" take. The honest answer is more nuanced, more interesting, and more actionable than either of those. What AI Can Actually Do Today Let's be precise about what has changed, because the hype runs in both directions. AI copilots in 2026 are genuinely impressive at a specific class of data engineering tasks. Give a well-prompted model a schema and a business requirement, and it will produce SQL that would have taken a competent engineer thirty minutes to write. Ask it to scaffold a dbt model with tests and documentation, and it delivers something you can actually work from. Point it at a slow query and ask for optimization suggestions, and it identifies the right indexes and join strategies most of the time. The work that once defined the day-to-day of data engineering — writing transformations, building pipeline boilerplate, generating unit tests, documenting schemas — is now legitimately acceleratable by an order of magnitude. That compression is real. A pipeline that took a week to build from scratch now takes a day. A day's worth of dbt model work now takes a morning. The cycle time has collapsed, and pretending otherwise is not a useful position. But Would You Actually Deploy It? Here is where the honest conversation has to happen. AI generates code that looks production-ready. It compiles. The DAG runs. The transformations return the right rows on the test dataset. And then you look closer. There are no retry semantics. There is no idempotency guarantee — run it twice, and you get duplicates. There are no data quality checks, no row count assertions, no schema drift detection. Observability is absent. The error handling catches exceptions and logs them to nowhere. Governance controls do not exist because the model has no idea what your data classification policies are. The code is impressively correct at the logic layer and completely unprepared for production reality. And that gap — between "AI generated it" and "it is actually deployable" — is not a small gap. It represents most of what makes data engineering genuinely hard. This is not a criticism of AI tooling. It is a precise description of where the boundary currently sits. And that boundary is exactly where the value of a skilled data engineer now concentrates. The Three-Bucket Reality Not all data engineering work is equally automatable, and the honest framework is to split it into three categories based on where AI sits today. What AI handles well. SQL and transformation generation, dbt model scaffolding, unit test generation, schema documentation, query explanation, code refactoring, and first-draft pipeline boilerplate. These tasks are high-volume, pattern-heavy, and well-represented in training data. AI performs them at a level that meets or exceeds what most engineers produce under time pressure. What AI assists but cannot own. Pipeline architecture decisions, root cause analysis on production failures, performance tuning for complex distributed jobs, and data modeling judgment for novel domains. AI is genuinely useful here as a thought partner and accelerant, but the decisions require context, business knowledge, and judgment that models do not reliably carry. What remains fundamentally human. Trade-off evaluation with real organizational constraints, governance and compliance decisions, architecture choices with long-term consequences, and anything requiring accountability. These require not just the right answer but the right answer for this company, this data, this regulatory environment, this team. That is irreducibly human work. The critical observation is that the boundary between these buckets is not static. Tasks that sat in the second bucket eighteen months ago have migrated into the first. The direction of travel is clear. Engineers who have concentrated their value entirely in automatable work are already exposed. Engineers who have built depth in judgment, architecture, and systems thinking are in an increasingly strong position. The Workflow Has Already Changed The before and after is not theoretical. It is visible in how high-performing data engineering teams actually operate today. The traditional workflow moved linearly through extraction, transformation, loading, and serving — each stage measured in hours to days, the full cycle measured in weeks. It was plagued by boilerplate, manual testing, documentation that was always out of date, and context-switching that fragmented deep work. The AI-enhanced workflow runs the same stages but with a fundamentally different time signature. StageTraditionalAI-EnhancedExtractHours — manual SQL, custom connectorsMinutes — AI-generated queries, auto connectorsTransformDays — dbt models, Spark jobsHours — AI-assisted modeling, auto schema detectionLoadHours — DAG authoring, schedulingMinutes — auto DAG generation, smart schedulingServeDays — dashboard building, documentationHours — auto documentation, natural language query The total cycle time compresses from weeks to days. That compression does not come from removing the engineer. It comes from removing the repetitive execution work so the engineer can focus on the decisions that actually require human judgment. What the Collaboration Actually Looks Like The AI-native data engineer workflow is not "prompt and deploy." It is a structured collaboration with a clear division of responsibility. AI accelerates the build. The engineer ensures it is correct, reliable, observable, and production-ready. The accountability for what ships belongs to the engineer, not the model. That accountability is not a burden — it is the source of professional value. The engineers who treat AI output as a draft to be critically evaluated and hardened will consistently outperform those who either ignore the tools entirely or treat generated code as finished work. Both of those failure modes are common. Neither is sustainable. The Skill Set Reorganizes, Not Disappears The skills required to be an excellent data engineer are shifting, but they are not evaporating. They are reorganizing around three pillars. Technical depth now centers on evaluating AI-generated code rather than writing all code from scratch. This requires strong fundamentals — you cannot spot the subtle join fanout in AI-generated SQL if you do not understand join semantics. It also means investing in observability, reliability engineering, and prompt crafting as first-class technical skills. A well-constructed prompt that produces deployable output in one iteration is genuinely more valuable than the ability to write the same code manually from scratch. Systems thinking becomes the primary differentiator. Architecture decisions, data modeling judgment, trade-off evaluation, and problem framing are tasks that compound in value as AI handles more execution work. The engineer who can look at a generated pipeline and immediately identify the three ways it will fail at scale is providing something no current model reliably provides. Engineering leadership expands to include guiding AI usage within a team, establishing review standards for AI-generated code, owning governance controls, and setting the quality bar that separates production-ready from impressive-looking. This is not a soft skill add-on — it is a core engineering responsibility in an environment where the output volume of any individual engineer has increased dramatically. The role is shifting from execution to judgment. That is an upgrade, not a downgrade, for engineers willing to make the transition deliberately. How to Actually Evolve The path forward is concrete, not abstract. Start by integrating AI into your daily work right now — not as an experiment but as a workflow change. Use it for SQL drafting, pipeline scaffolding, and test generation. Build the muscle of critically evaluating what it produces. Develop prompting habits that consistently get you to a usable first draft rather than something you have to rewrite from scratch. Level up by investing deliberately in the areas AI does not cover well. System design. Distributed systems fundamentals. Reliability and observability patterns. Data modeling for complex domains. These skills appreciate in value as AI handles more of the execution layer — the relative scarcity of strong systems thinkers increases as the supply of generated boilerplate becomes effectively infinite. Lead by taking ownership of AI quality standards on your team. Be the person who defines what "production-ready" means for AI-generated pipelines, who establishes review checklists, who sets governance guardrails. This is influence that compounds over time and is not replicable by a model. The Honest Bottom Line AI will not replace data engineers. But data engineers who treat their value as residing primarily in writing code — rather than in the judgment, architecture, and reliability thinking that makes code worth deploying — are taking a position that becomes harder to defend with each model release. The opportunity is real, and it is now. The engineers who learn to work with AI as a genuine collaborator, who develop the critical evaluation skills to close the gap between generated and production-ready, and who invest in the systems thinking that AI cannot replicate — those engineers are not threatened by this transition. They are the ones who define what data engineering looks like on the other side of it. Evolve deliberately. The alternative is not standing still — it is falling behind at an accelerating rate.

By Janani Annur Thiruvengadam CORE

In high-volume data platforms, hardcoding validation logic into individual processing pipelines creates significant operational drag. As an enterprise data asset footprint grows, maintaining manual checks for hundreds of tables inevitably leads to mounting technical debt, silent schema drift, and a fragmented audit trail. To achieve data governance at scale, data architects must decouple validation rules from the execution engine. By utilizing a centralized metadata repository to dynamically generate validation suites, organizations can transform data quality from a reactive, script-based bottleneck into a configuration-driven infrastructure asset. The Metadata-Driven Architecture Instead of embedding validation constraints directly inside an ETL/ELT pipeline, this pattern isolates validation rules inside a centralized relational database schema. The orchestration engine programmatically queries this metadata at runtime, constructs the validation suites on the fly, executes them against target tables, and routes the evaluation metrics to an observability layer. This architecture provides three primary engineering advantages: Decoupled Governance: Data stewards can alter business rules or add expectations via simple DML updates without modifying or redeploying production application code.Schema Drift Resilience: The engine dynamically adapts to structural variations by programmatically evaluating target datasets against rules defined at the column level.Centralized Observability: Every rule execution generates a standardized, traceable metric payload, laying a consistent foundation for real-time data auditing and data lineage maps. 1. Defining the Metadata Schema (DDL) To implement this framework in an enterprise Lakehouse ecosystem, the metadata table must act as an immutable source of truth for constraints. Below is the production DDL required to initialize the control directory in Snowflake or Databricks: SQL CREATE TABLE data_quality_rules ( rule_id INT IDENTITY(1,1), table_name VARCHAR(255) NOT NULL, column_name VARCHAR(255) NOT NULL, expectation_type VARCHAR(255) NOT NULL, expectation_kwargs VARIANT NOT NULL, -- Stored as JSON object is_active BOOLEAN DEFAULT TRUE, updated_at TIMESTAMP_NTZ DEFAULT CURRENT_TIMESTAMP(), CONSTRAINT pk_rule_id PRIMARY KEY (rule_id) ); -- Seed metadata rules for execution tracking INSERT INTO data_quality_rules (table_name, column_name, expectation_type, expectation_kwargs) VALUES ('CUSTOMERS', 'CUST_ID', 'expect_column_values_to_not_be_null', '{}'), ('CUSTOMERS', 'AGE', 'expect_column_values_to_be_between', '{"min_value": 18, "max_value": 60}'), ('ORDERS', 'ORDER_ID', 'expect_column_values_to_not_be_null', '{}'); 2. Implementation: The Programmatic Execution Engine The core execution wrapper leverages Python and Great Expectations (gx) to programmatically turn rows of metadata into active validation suites. This script establishes a secure database connection via SQLAlchemy, harvests active constraints, generates runtime batch requests, and triggers structured checkpoints. Python import os import json import logging from datetime import datetime import pandas as pd from sqlalchemy import create_engine import great_expectations as gx from great_expectations.core.batch import RuntimeBatchRequest # Configure structured logging for production auditing logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s') logger = logging.getLogger(__name__) class MetadataDataQualityEngine: def __init__(self, connection_string: str): self.engine = create_engine(connection_string) # Initialize Great Expectations ephemeral context for programmatic runtime control self.context = gx.get_context(context_root_dir=None) def fetch_active_metadata(self) -> pd.DataFrame: """Harvests active validation configurations from the centralized store.""" query = """ SELECT table_name, column_name, expectation_type, expectation_kwargs FROM data_quality_rules WHERE is_active = TRUE """ try: df = pd.read_sql(query, self.engine) logger.info(f"Successfully harvested {len(df)} active validation rules.") return df except Exception as e: logger.error(f"Failed to query metadata repository: {str(e)}") raise def compile_expectation_suite(self, table_name: str, rules_df: pd.DataFrame): """Assembles validation rules into a Great Expectations suite on the fly.""" suite_name = f"{table_name}_suite" suite = self.context.add_or_update_expectation_suite(expectation_suite_name=suite_name) # Filter metadata constraints for the specific target asset table_rules = rules_df[rules_df['table_name'] == table_name] for _, row in table_rules.iterrows(): # Parse JSON kwargs configuration gracefully kwargs = row['expectation_kwargs'] if isinstance(kwargs, str): kwargs = json.loads(kwargs) kwargs['column'] = row['column_name'] # Programmatically map string values to structured GX expectation objects expectation_config = gx.core.ExpectationConfiguration( expectation_type=row['expectation_type'], kwargs=kwargs, meta={"notes": f"Automated constraint enforcement for column: {row['column_name']}"} ) suite.add_expectation(expectation_config) self.context.add_or_update_expectation_suite(suite=suite) return suite def execute_quality_checkpoint(self, table_name: str, target_df: pd.DataFrame): """Builds a runtime batch request and evaluates data against the generated suite.""" timestamp = datetime.utcnow().strftime("%Y%m%dT%H%M%SZ") suite_name = f"{table_name}_suite" checkpoint_name = f"{table_name}_checkpoint" # Unique runtime composite signature prevents processing trace collisions batch_request = RuntimeBatchRequest( datasource_name="lakehouse_runtime_datasource", data_connector_name="runtime_data_connector", data_asset_name=f"{table_name}_{timestamp}", runtime_parameters={"batch_data": target_df}, batch_identifiers={"table_name": table_name, "execution_timestamp": timestamp} ) # Register and fire a dynamic checkpoint execution self.context.add_or_update_checkpoint( name=checkpoint_name, config_version=1, class_name="SimpleCheckpoint", validations=[{ "batch_request": batch_request, "expectation_suite_name": suite_name }] ) logger.info(f"Launching data quality checkpoint for table: {table_name}") return self.context.run_checkpoint(checkpoint_name=checkpoint_name) # Production Loop Execution Pattern if __name__ == "__main__": SF_CONN = "snowflake://<user>:<pass>@<account>/<db>/<schema>?warehouse=COMPUTE_WH&role=SYSADMIN" dq_engine = MetadataDataQualityEngine(connection_string=SF_CONN) metadata_rules = dq_engine.fetch_active_metadata() distinct_target_tables = metadata_rules['table_name'].unique() for current_table in distinct_target_tables: try: # Stage current batch dataset from target engine raw_data_df = pd.read_sql(f"SELECT * FROM {current_table}", dq_engine.engine) # Step 1: Build suite dynamically from relational rules dq_engine.compile_expectation_suite(table_name=current_table, rules_df=metadata_rules) # Step 2: Validate batch data and extract metrics payload eval_result = dq_engine.execute_quality_checkpoint(table_name=current_table, target_df=raw_data_df) if not eval_result["success"]: logger.warning(f"Data Quality anomalies detected on asset: {current_table}") else: logger.info(f"Asset {current_table} successfully cleared all metadata expectations.") except Exception as err: # Fault isolation ensures an asset failure never crashes cascading pipeline steps logger.error(f"Processing loop interrupted on asset {current_table}: {str(err)}") continue 3. Production-Grade Engineering Guardrails Building a dynamic system requires putting structural guardrails around the execution engine to prevent it from failing under enterprise pressures. Fault Isolation and Pipeline Resilience: Never let a validation failure on an upstream or non-critical business table halt your entire orchestration loop. Wrapping individual target assets in localized try-except blocks ensures that a failure on a secondary table (like CUSTOMERS) does not block downstream transactional tables (like ORDERS) from completing their validation lifecycles.Idempotency and Batching Identifiers: Every unique quality run must be traceable back to a specific moment in time to avoid overwriting or colliding results in your metadata tracking layer. Pair the table_name with an immutable execution_timestamp (such as a UTC ISO string) as a composite batch identifier. This guarantees an explicit audit trail across parallel streaming windows or backfilled data runs.Metadata-as-Code Frameworks: Treat the validation matrix table with the same operational rigor as production application code. Changes, additions, or deprecations of quality thresholds must follow a strict GitOps progression. Use schema migration version control tools (like Flyway or Liquibase) to manage, track, and deploy DML changes safely across staging and production clusters.Proactive Alerting Integration: Local HTML docs are insufficient for zero-downtime platforms. The metadata evaluation output dictionary must be integrated directly into cloud native alerting systems. Configure Webhook integrations or cloud alerting channels to route failed validation metrics directly to Slack channels or PagerDuty schedules. This ensures on-call engineers are proactively notified the moment a metric payload trends outside acceptable operational thresholds. Summary: The Architectural Impact Transitioning to a metadata-driven approach shifts data quality from a reactive "clean-up" task to an integrated, proactive engineering asset. By treating validation criteria as configurable metadata parameters rather than hardcoded script directives, architects eliminate technical debt and bridge the gap between business semantics and computing layers. This decoupled architecture provides the strict governance framework required to support high-stakes analytics and downstream machine learning layers, ensuring that every data element hitting your warehouse is automatically and transparently vetted before reaching production consumers.

Analytics teams do not get too upset about small errors. If a product dashboard is off by half a percent on a Tuesday, nobody files a ticket. If your marketing funnel counts some web sessions twice, the overall trend is still okay. Everyone moves on. I spent a part of my early career in that world. It is a place to learn how to move fast, ship features, and use data to get a general idea. Then I started building pipelines that fed automated billing and revenue recognition systems. The rules changed completely. Financial-grade data is different. When a number goes on a customer invoice, drives a usage-based billing meter, or gets repeated by an executive to the board of directors, "roughly right" becomes a problem. The pipeline is not just informing a business decision - it is the decision. If it fails, someone has to answer for it to an external auditor. That change moving from analytics to shipping numbers people stake their reputations on — made me scrap my old way of doing things and rethink how I design data infrastructure. If you are building lakehouse platforms that have to scale out and remain completely defensible under scrutiny, here is what actually matters. The Reconciliation Gap Nobody Warns You About Here is the first painful lesson: correctness and scale do not work well together, and billing data is right in the middle. Usage-based billing means you are dealing with huge, high-volume event streams, API hits, compute-seconds, database operations, and converting those numbers into actual cash. The volume forces you toward distributed systems. The money demands accuracy. You cannot ship an infrastructure that's very fast but drops some events, and you cannot ship a framework that is perfectly consistent but takes a long time to close out a daily ledger. The place where this trade-off is hardest is late-arriving or out-of-order data. Imagine a streaming meter where an event happens at 11:58 PM. It does not hit your ingestion engine until 12:03 AM the next morning. If your daily aggregation pipeline already completed at midnight, that customer usage falls into the wrong billing month or disappears. Multiply that event by many transactions, and you have a massive reconciliation gap that your finance team will catch. Because of this, my absolute baseline rule for any pipeline touching revenue is that it must be 100% idempotent and completely reprocessable from source. I mean reprocessable in the sense that I can replay a raw event window from three weeks ago and land on the exact same decimal point. To do that, your transformation logic has to be completely deterministic and keyed entirely on business identifiers rather than system arrival times. In production, that usually looks like a merge statement driven by event and entity IDs: SQL MERGE INTO billing_usage_gold AS target USING staged_events AS source ON target.event_id = source.event_id WHEN MATCHED AND source.ingested_at > target.ingested_at THEN UPDATE SET * WHEN NOT MATCHED THEN INSERT * The SQL looks simple. The actual engineering discipline is ensuring that event_id remains stable, unique, and uncorrupted all the way back to the source application code. If you lock down that data contract, your downstream reconciliation nightmares mostly go away. Layering for Defensiveness, Not Aesthetics I am a pragmatist when it comes to the classic layered lakehouse. Many data teams adopt this setup just because it looks tidy in a slide deck. When you are dealing with financial pipelines, those layers serve a functional, defensive purpose. The raw layer needs to be entirely immutable and append-only. Think of it as a ledger of exactly what the world looked like when the event happened, timestamped, raw, and completely untouched. Never let transformation logic touch or rewrite this layer. When an auditor asks, "What exactly did the system report on November 14th?" this table holds the answer. It should not change just because you refactored a downstream SQL model six months later. The refined layer is where you handle the reality of data engineering: deduplication, type casting, schema enforcement, and core business rules. This is also where you have to build structural data-quality checkpoints. For architectures, that means ditching passive logs or soft warnings and leaning into automated testing frameworks like dbt to physically break things when they go wrong. If a data point turns into an invoice line item, a bad value should not log an error; it needs to kill the process. We handle this by setting our dbt data assertions to a hard error severity level: YAML # models/staging/staged_events.yml version: 2 models: - name: billing_usage_silver columns: - name: event_id tests: - unique: config: severity: error - not_null: config: severity: error - name: compute_seconds tests: - dbt_utils.expression_is_true: expression: ">= 0" config: severity: error By explicitly setting severity: error, a single duplicate event ID or a bizarre negative usage value will not just trigger a warning. It will kill the execution DAG instantly. Is it annoying to debug a stopped pipeline at 2:00 AM? Yes. I would much rather explain a delayed operational dashboard to an internal stakeholder than explain a fraudulent or inaccurate charge to a paying enterprise customer. The serving layer is your business-facing interface. It features grains, locked-down definitions, and the exact tables that feed your downstream billing engines, margin tools, and executive reporting. By the time any row hits this layer, it has survived every quality gate you can throw at it. Your analysts and finance partners can build on top of it safely, without rewriting core logic five different ways and coming up with five different answers. If It Isn't Observable, It Isn't Auditable People in data engineering tend to talk about observability like it's a nice-to-have optimization trick or a post-launch polish item. For financial systems, observability is literally the entire game. When you sit down with auditors or finance directors, they do not care if your Apache Spark clusters are running at peak efficiency. They want to know two things: How do you know this final number is correct, and can you prove it to me right now? Answering that honestly requires three things built directly into your infrastructure: Freshness monitoring that actually wakes you up. Silence does not mean everything is working. If a key serving table misses its scheduled data drop, you should not find out because a finance manager pings you on Slack. You need to wire freshness monitoring into a high-priority on-call rotation like PagerDuty. You have to catch the delay before the downstream billing window closes out.Lineage a human can trace. When a revenue metric looks weird on a summary, you need to be able to trace that specific number back through every single SQL transformation, join, and filter to the original raw event in minutes. Relying on "trust me I wrote the code" does not work. Automated, column-level data lineage maps turn an afternoon of code review into a two-minute look.Continuous data quality logging. Treat data quality metrics as a first-class production output. We track row-count variations, null rates, and distribution drifts on every run, logging them out to monitoring tables or platforms like Elementary. If your system ingestion drops out of nowhere, you need to know whether your customers actually stopped using the product or an upstream webhook silently broke. [Raw Event Ingestion] ⬇ Flows into:[Silver Layer] ➡ (Runs dbt Hard Schema & Unique Tests ➡ Fails? HALT & ALERT) ⬇ Flows into:[Gold Serving] ➡ (Triggers Continuous DQ & Freshness Monitoring ➡ PagerDuty / Slack Alerts) Compliance Is Just a Feature Wearing a Suit If you have ever been through a pre-IPO sprint or a standard Sarbanes-Oxley (SOX) audit, you know how exhausting it feels. The biggest mental shift is realizing that compliance guidelines are really just standard system requirements written in legal language. Auditors care about controls, lineage, reproducibility, and separation of duties. If you translate that into engineering terms, it means: your transformation code must be version-controlled and peer-reviewed, production deployments should happen via automated CI/CD pipelines instead of a local laptop terminal, data access needs to be tightly permissioned and logged, and you must be able to reproduce historical numbers on demand. Infrastructure-as-Code (IaC) handles all of this heavy lifting for you. When your cloud environments, access roles, and pipeline configurations live inside a Git repository, the question of "Who changed this permission, and when did they do it?" always has an unalterable answer. Teams that treat compliance as a chore end up panicking every single quarter. Teams that build these automated checks directly into their deployment workflow barely even notice the audit happening. It is the same amount of work either way; doing it continuously is just significantly cheaper. Unlocking Self-Service Without the Chaos The real reward for dealing with all this architecture is that you can finally let other teams get their own data without causing problems. "Self-service analytics" usually gets a bad name because companies often give raw, messy tables to a lot of people. As you would expect, everyone comes up with their own definition of what "gross margin" or "active user" means, and you end up with big arguments inside the company about whose spreadsheet is correct. A controlled and reliable serving layer completely changes this situation. When your definitions are fixed, consistent, and easy to see, your finance team can look at margins by market segment, your marketing teams can build expansion models, and your product managers can look at consumption trends. Everyone is getting their data from the same place. That is the moment your data engineering team stops being a bottleneck for the whole organization. Instead of spending your week answering special requests or running manual data extractions, you get to focus on building infrastructure that can handle a lot of work. Faster decision-making and clear visibility into operations do not come from a magic machine learning model. They happen because your underlying numbers are finally stable enough to act on without needing to check. A Few Things I Wish I Knew Earlier If you are currently moving from building product analytics to managing data that has real financial importance, remember that while your technical skills are still useful, your standards for engineering are not good enough. Design your systems so that you can repeat everything exactly, not just handle a lot of work. Make your data quality tools stop the pipeline if there is a problem instead of just giving a warning. Treat data history, system updates, and automated alerts as parts of your infrastructure rather than things you will do later. And stop thinking of compliance as a rule. A well-built pipeline is already mostly ready for audits anyway. The logic of distributed systems is hard. That is what we all talk about and study. The harder thing is accepting that when your data represents real money, "close enough" is not good enough.

In this blog, you will take a closer look at the different exchange types that can be used in RabbitMQ. All are demonstrated by means of examples in a Spring Boot application. Enjoy! Introduction In the previous blog, you learned the basic concepts of RabbitMQ and how to use it in a Spring Boot application. However, you only scratched the surface of it, so now it is time to dig a bit deeper into the different exchange types. If you are not yet familiar with the basic concepts, it is advised to read the previous blog. The official RabbitMQ documentation also provides detailed information that is worth reading. Sources used in this blog can be found on GitHub. Prerequisites Prerequisites for reading this blog are: Basic knowledge of Java;Basic knowledge of Spring Boot;Basic knowledge of Docker Compose;Basic knowledge of RabbitMQ. Topics The code can be found in the topics module. In the previous blog, you created two consumers A and B. Consumer A was bound to Queue A with routing key event.general.*. Consumer B was bound to Queue B with routing keys event.general.* and event.specific.*. The asterisk (*) wildcard was used and is a substitute for exactly one word. In the examples, the routing keys event.general.message and event.specific.message were used. You can also use the hash (#) wildcard, and this is a substitute for zero or more words. This is visualized in the figure below. In the RabbitMqConfig, you declare queue C and bind it to the TopicExchange with routing key event.general.#. Java public static final String QUEUE_CONSUMER_C = "consumer-c.queue"; public static final String ROUTING_KEY_NESTED_GENERAL_MESSAGE = "event.general.#"; @Bean Binding bindingConsumerBSpecific(Queue queueConsumerB, TopicExchange exchange) { return BindingBuilder.bind(queueConsumerB).to(exchange).with(ROUTING_KEY_SPECIFIC_MESSAGE); } @Bean public Queue queueConsumerC() { return new Queue(QUEUE_CONSUMER_C, false); } @Bean Binding bindingConsumerCNestedGeneral(Queue queueConsumerC, TopicExchange exchange) { return BindingBuilder.bind(queueConsumerC).to(exchange).with(ROUTING_KEY_NESTED_GENERAL_MESSAGE); } In the MessageController, you create an endpoint for sending a message with routing key event.general.message.nested. This routing key will not match the bindings of consumers A and B. Java @RequestMapping( method = RequestMethod.POST, value = "send-nested-general" ) public ResponseEntity<Void> sendNestedGeneralMessage(@RequestBody String message) { messageService.sendMessage("event.general.message.nested", message); return new ResponseEntity<>(HttpStatus.CREATED); } The ReceiverC listens to messages received in queue C and prints a message. Java @Component public class ReceiverC { @RabbitListener(queues = RabbitMqConfig.QUEUE_CONSUMER_C) public void receiveMessage(String message) { System.out.println("Queue Consumer C received <" + message + ">"); } } Start the application from within the topics module. Shell mvn spring-boot:run First, post a general message; this should be received by all consumers. Shell curl -X POST http://localhost:8080/send-general \ -H "Content-Type: text/plain" \ -d "This is a general message" In the application console log, you notice that all consumers receive the message. Plain Text Queue Consumer B received <This is a general message> Queue Consumer A received <This is a general message> Queue Consumer C received <This is a general message> Now, post a nested general message, which should be received only by consumer C. Shell curl -X POST http://localhost:8080/send-nested-general \ -H "Content-Type: text/plain" \ -d "This is a nested general message" In the application console log, you notice that the message is only received by consumer C. Plain Text Queue Consumer C received <This is a nested general message> Work Queues The code can be found in the work module. With work queues, you can publish a message and dispatch it to a pool of consumers. One of the consumers will pick up the message and start processing it. This is especially useful for dispatching long-running tasks. You use the default direct exchange in this case, and the queue name is used as the routing key. No need to use a custom exchange. This is visualized in the figure below. The RabbitMqConfig is quite small; you only define the queue. Java @Configuration public class RabbitMqConfig { public static final String QUEUE_TASK = "task.queue"; @Bean public Queue queueTask() { return new Queue(QUEUE_TASK, false); } } When sending a message via an endpoint, you use the queue name as the routing key. Java @RequestMapping( method = RequestMethod.POST, value = "send-work" ) public ResponseEntity<Void> sendWorkMessage(@RequestBody String message) { messageService.sendMessage(RabbitMqConfig.QUEUE_TASK, message); return new ResponseEntity<>(HttpStatus.CREATED); } Every consumer listens to the queue. Java @Component public class ReceiverA { @RabbitListener(queues = RabbitMqConfig.QUEUE_TASK) public void receiveMessage(String message) { System.out.println("Task picked up by Consumer A <" + message + ">"); } } @Component public class ReceiverB { @RabbitListener(queues = RabbitMqConfig.QUEUE_TASK) public void receiveMessage(String message) { System.out.println("Task picked up by Consumer B <" + message + ">"); } } @Component public class ReceiverC { @RabbitListener(queues = RabbitMqConfig.QUEUE_TASK) public void receiveMessage(String message) { System.out.println("Task picked up by Consumer C <" + message + ">"); } } Start the application from within the work module. Shell mvn spring-boot:run Send a message to the queue. Shell curl -X POST http://localhost:8080/send-work \ -H "Content-Type: text/plain" \ -d "This is a work message" The message is processed by one consumer. Plain Text Task picked up by Consumer A <This is a work message> Fanout The code can be found in the fanout module. With fanout, you want to broadcast messages to all queues. You send messages to the exchange, but there is no need to specify a routing key. You can also ensure that temporary queues are used. When temporary queues are used, the queue name will be generated. In the RabbitMqConfig, you define a FanoutExchange. The queues are defined as an AnonymousQueue. This creates a non-durable, exclusive, auto-delete queue with a generated name. You bind the queues to the exchange. Java @Configuration public class RabbitMqConfig { public static final String FANOUT_EXCHANGE_NAME = "fanout.exchange"; @Bean FanoutExchange fanoutExchange() { return new FanoutExchange(FANOUT_EXCHANGE_NAME); } @Bean public Queue queueConsumerA() { return new AnonymousQueue(); } @Bean Binding bindingConsumerA(Queue queueConsumerA, FanoutExchange exchange) { return BindingBuilder.bind(queueConsumerA).to(exchange); } @Bean public Queue queueConsumerB() { return new AnonymousQueue(); } @Bean Binding bindingConsumerBGeneral(Queue queueConsumerB, FanoutExchange exchange) { return BindingBuilder.bind(queueConsumerB).to(exchange); } @Bean Binding bindingConsumerBSpecific(Queue queueConsumerB, FanoutExchange exchange) { return BindingBuilder.bind(queueConsumerB).to(exchange); } } In order to send messages, you only need to send them to the exchange. This can be seen in the MessageService. Java public void sendMessage(String message) { rabbitTemplate.convertAndSend(RabbitMqConfig.FANOUT_EXCHANGE_NAME, "", message); } On the receiving side, you listen to the generated queue name (thus not a specific one in this case). Java @Component public class ReceiverA { @RabbitListener(queues = "#{queueConsumerA.name}") public void receiveMessage(String message) { System.out.println("Queue Consumer A received <" + message + ">"); } } @Component public class ReceiverB { @RabbitListener(queues = "#{queueConsumerB.name}") public void receiveMessage(String message) { System.out.println("Queue Consumer B received <" + message + ">"); } } Start the application from within the fanout module. Shell mvn spring-boot:run Send a message to the queue. Shell curl -X POST http://localhost:8080/send-to-all \ -H "Content-Type: text/plain" \ -d "This is a fanout message" In the application console log, you notice that the message is consumed by all queues. Plain Text Queue Consumer B received <This is a fanout message> Queue Consumer A received <This is a fanout message> RPC The code can be found in the RPC module. Remote Procedure Call (RPC) can be used when you need to execute a function on a remote application and wait for the result. The event is sent to the queue and is processed by Consumer A. The result is sent to a queue in the replyTo field of the request. The publisher waits for data to be returned on this callback queue. When the message appears, it checks the correlationId. If it matches the value of the request, the response is returned to the publisher. All of this is done automatically by the RabbitTemplate. In the RabbitMqConfig, a DirectExchange is used. With a DirectExchange, you match exactly on events; you cannot use wildcards here, just like a TopicExchange. Java @Configuration public class RabbitMqConfig { public static final String QUEUE_CONSUMER_A = "consumer-a.queue"; public static final String DIRECT_EXCHANGE_NAME = "events.exchange"; public static final String ROUTING_KEY_RPC_MESSAGE = "event.rpc"; @Bean DirectExchange eventsExchange() { return new DirectExchange(DIRECT_EXCHANGE_NAME); } @Bean public Queue queueConsumerA() { return new Queue(QUEUE_CONSUMER_A, false); } @Bean Binding bindingConsumerA(Queue queueConsumerA, DirectExchange exchange) { return BindingBuilder.bind(queueConsumerA).to(exchange).with(ROUTING_KEY_RPC_MESSAGE); } } The MessageController contains an endpoint for sending the event. Java @RequestMapping( method = RequestMethod.POST, value = "send-rpc" ) public ResponseEntity<Void> sendRpcMessage(@RequestBody String message) { messageService.sendMessage(message); return new ResponseEntity<>(HttpStatus.CREATED); } In the MessageService, you use convertSendAndReceive and process the response. Java public void sendMessage(String message) { Object response = rabbitTemplate.convertSendAndReceive(RabbitMqConfig.DIRECT_EXCHANGE_NAME, ROUTING_KEY_RPC_MESSAGE, message); if (response != null) { System.out.println("Sender received response: " + response); } else { System.out.println("No response received"); } } In the receiver, you receive the message and send a response. Do note that some additional processing is added in order to trigger a timeout. More on that in a moment. Java @Component public class ReceiverA { @RabbitListener(queues = RabbitMqConfig.QUEUE_CONSUMER_A) public String receiveMessage(String message) { System.out.println("Queue Consumer A received <" + message + ">"); if (message.equals("This is an rpc message")) { return "success"; } else if (message.equals("This is a timeout message")) { try { Thread.sleep(10000); } catch (InterruptedException e) { throw new RuntimeException(e); } return "success"; } else { return "failure"; } } } Start the application from within the rpc module. Shell mvn spring-boot:run Send a message to the queue. Shell curl -X POST http://localhost:8080/send-rpc \ -H "Content-Type: text/plain" \ -d "This is an rpc message" In the application console log, you notice that the message is consumed by consumer A, and that a successful response is received by the publisher. Plain Text Queue Consumer A received <This is an rpc message> Sender received response: success But what if it takes too long to process the message? In real life, the remote application can be unreachable for one reason or another. Send a timeout message. Shell curl -X POST http://localhost:8080/send-rpc \ -H "Content-Type: text/plain" \ -d "This is a timeout message" In the MessageService, the response will return null, and a timeout exception is raised. Plain Text Queue Consumer A received <This is a timeout message> No response received 2026-04-25T14:50:16.785+02:00 WARN 482297 --- [MySpringRabbitMqPlanet] [pool-2-thread-8] o.s.amqp.rabbit.core.RabbitTemplate : Reply received after timeout for 2 2026-04-25T14:50:16.785+02:00 WARN 482297 --- [MySpringRabbitMqPlanet] [pool-2-thread-8] s.a.r.l.ConditionalRejectingErrorHandler : Execution of Rabbit message listener failed. org.springframework.amqp.rabbit.support.ListenerExecutionFailedException: Listener threw exception at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.wrapToListenerExecutionFailedExceptionIfNeeded(AbstractMessageListenerContainer.java:1795) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:1687) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.actualInvokeListener(AbstractMessageListenerContainer.java:1612) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:1599) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:1590) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.executeListenerAndHandleException(AbstractMessageListenerContainer.java:1539) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.executeListener(AbstractMessageListenerContainer.java:1520) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.DirectMessageListenerContainer$SimpleConsumer.callExecuteListener(DirectMessageListenerContainer.java:1206) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.DirectMessageListenerContainer$SimpleConsumer.handleDelivery(DirectMessageListenerContainer.java:1163) ~[spring-rabbit-4.0.2.jar:4.0.2] at com.rabbitmq.client.impl.ConsumerDispatcher$5.run(ConsumerDispatcher.java:149) ~[amqp-client-5.27.1.jar:5.27.1] at com.rabbitmq.client.impl.ConsumerWorkService$WorkPoolRunnable.run(ConsumerWorkService.java:111) ~[amqp-client-5.27.1.jar:5.27.1] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090) ~[na:na] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614) ~[na:na] at java.base/java.lang.Thread.run(Thread.java:1474) ~[na:na] Caused by: org.springframework.amqp.AmqpRejectAndDontRequeueException: Reply received after timeout at org.springframework.amqp.rabbit.core.RabbitTemplate.onMessage(RabbitTemplate.java:2721) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.DirectReplyToMessageListenerContainer.lambda$setMessageListener$0(DirectReplyToMessageListenerContainer.java:93) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:1683) ~[spring-rabbit-4.0.2.jar:4.0.2] ... 12 common frames omitted 2026-04-25T14:50:16.790+02:00 ERROR 482297 --- [MySpringRabbitMqPlanet] [pool-2-thread-8] .l.DirectReplyToMessageListenerContainer : Failed to invoke listener org.springframework.amqp.rabbit.support.ListenerExecutionFailedException: Listener threw exception at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.wrapToListenerExecutionFailedExceptionIfNeeded(AbstractMessageListenerContainer.java:1795) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:1687) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.actualInvokeListener(AbstractMessageListenerContainer.java:1612) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:1599) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:1590) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.executeListenerAndHandleException(AbstractMessageListenerContainer.java:1539) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.executeListener(AbstractMessageListenerContainer.java:1520) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.DirectMessageListenerContainer$SimpleConsumer.callExecuteListener(DirectMessageListenerContainer.java:1206) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.DirectMessageListenerContainer$SimpleConsumer.handleDelivery(DirectMessageListenerContainer.java:1163) ~[spring-rabbit-4.0.2.jar:4.0.2] at com.rabbitmq.client.impl.ConsumerDispatcher$5.run(ConsumerDispatcher.java:149) ~[amqp-client-5.27.1.jar:5.27.1] at com.rabbitmq.client.impl.ConsumerWorkService$WorkPoolRunnable.run(ConsumerWorkService.java:111) ~[amqp-client-5.27.1.jar:5.27.1] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090) ~[na:na] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614) ~[na:na] at java.base/java.lang.Thread.run(Thread.java:1474) ~[na:na] Caused by: org.springframework.amqp.AmqpRejectAndDontRequeueException: Reply received after timeout at org.springframework.amqp.rabbit.core.RabbitTemplate.onMessage(RabbitTemplate.java:2721) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.DirectReplyToMessageListenerContainer.lambda$setMessageListener$0(DirectReplyToMessageListenerContainer.java:93) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:1683) ~[spring-rabbit-4.0.2.jar:4.0.2] ... 12 common frames omitted How to solve this? In this case, you are better off using the AsyncRabbitTemplate. This template is not automatically autowired, so you have to define it as a bean. Let's do so in the RabbitMqConfig. Java @Bean public AsyncRabbitTemplate asyncRabbitTemplate(RabbitTemplate rabbitTemplate) { return new AsyncRabbitTemplate(rabbitTemplate); } In the MessageController, you define an endpoint to trigger the async template. Java @RequestMapping( method = RequestMethod.POST, value = "send-async" ) public ResponseEntity<Void> sendAsyncMessage(@RequestBody String message) { messageService.sendAsyncMessage(message); return new ResponseEntity<>(HttpStatus.CREATED); } In the MessageService, you autowire the AsyncRabbitTemplate. And because it is an async call, you catch the response by means of a CompletableFuture. Java public void sendAsyncMessage(String message) { CompletableFuture<Object> future = asyncRabbitTemplate.convertSendAndReceive(RabbitMqConfig.DIRECT_EXCHANGE_NAME, ROUTING_KEY_RPC_MESSAGE, message); future.thenAccept(response -> { if (response != null) { System.out.println("Sender received response: " + response); } else { System.out.println("No response received"); } }); } Start the application from within the rpc module. Shell mvn spring-boot:run Send a message to the queue. Shell curl -X POST http://localhost:8080/send-async \ -H "Content-Type: text/plain" \ -d "This is a timeout message" In the application log, you see the same result: the response is null, but no timeout exception anymore. Conclusion In this post, you learned different exchange types. Each serves its own use case. It is up to you to choose the right pattern for your use case.

By Gunter Rotsaert CORE

Columnar engines naturally organize computation around vectors to make effective use of single instruction, multiple data (SIMD) instructions. This makes vectors first-class citizens in such engines. The difficult design question appears when an engine's internal application programming interface (API) must be exposed to users: where should programming happen? A native C API is sufficient for embedding, and many engines stop there. Building a complete analytical database, however, requires a full-featured language for programming on top of the engine. There are well-known options: SQL - solves the relational part of this problem, but a database runtime may also expose direct vector calculations, object creation, user-defined lambdas, OS integration, orchestration, control flow, graph algorithms as well as generic-purpose programming, not only analytic queries. Implementing all of these in SQL would produce a standalone dialect besides the relational language.Embed an established scripting language - it immediately solves lambdas and control flow, but it also introduces a second runtime. Thus the language design problem is larger than just query syntax: "How can a columnar engine expose its native data types and operations without restricting users to C, duplicating those values in another runtime or hiding expressions from the engine optimizer"? Rayfall provides a concrete case study. It is an expression language in which native values, vector operations, user-defined lambdas, and relational queries are the natural language components. Its syntax is well-known S-expressions, which bring powerful mechanisms as well as a dramatically simple parser, which is mandatory in the case of real-time requests that require fast responses thus wasting time in parser would be irrational. Requirements RequirementConsequenceNative valuesLanguage values are engine objects rather than wrappers.Array semanticsOperations work on atoms, vectors, and table columns.General programming The language needs user defined lambdas support, control flow, error handling, debug info, stack unwinding.First class queriesQuery expressions must be fully visible to the optimizer.IPC/SerializationThe language should naturally support remote execution as well as local without introducing separate mechanisms or 3rd party protocols. Architecture Rayfall uses S-expressions: Clojure (+ 2 3) (* (+ 2 3) 4) (sum [10 20 30]) The parser does not produce any AST; rather, it produces an evaluation tree immediately where each object is the same struct with a type tag, reference counter, and payload. (...) - is a List[...] - homogeneous vector{...} - dictionary"..." - string literal[0-9]* - number[a-zA-Z]* - symbol(fn [args] (body)) - lambda A dictionary is just a two array of the same length: keys and values. Thus, a table is just a flipped dict where keys are table column symbols, values are lists of column vectors. This is how, for example, Q language acts. Another advantage of an S-expressions-based language is equivalence of code and data: Clojure (+ 1 2) ; an executable list (quote (+ 1 2)) ; the same structure treated as data [AAPL MSFT NVDA] ; a typed symbol vector {from: trades where: (> price 100)} ; a dictionary containing expressions Unification Let's consider multiplication. Clojure (* 12.5 4) ; scalar call (* [12.5 20.0 8.0] 4) ; vector - scalar (select {from: trades notional: (* price qty)}) ; table columns While the expression surface stays the same, the execution strategy changes depending on argument types. This is the core idea behind Rayfall. Vector operations are not a library lying on top of a low-level API, and query expressions are not strings passed to a separate SQL frontend. They are the same language expressions interpreted in the same context. Queries Consider the following simple table: Clojure ‣ (set trades … (table [sym time side price qty] … (list … [AAPL AAPL MSFT MSFT NVDA AAPL] … [09:30:00.000 09:30:30.000 09:31:00.000 … 09:31:30.000 09:32:00.000 09:32:30.000] … [BUY SELL BUY SELL BUY BUY] … [100.0 101.0 400.0 399.0 170.0 102.0] … [150 50 20 30 100 200]))) ┌──────┬──────────────┬──────┬───────┬─────┐ │ sym │ time │ side │ price │ qty │ │ SYM │ TIME │ SYM │ F64 │ I64 │ ├──────┼──────────────┼──────┼───────┼─────┤ │ AAPL │ 09:30:00.000 │ BUY │ 100.0 │ 150 │ │ AAPL │ 09:30:30.000 │ SELL │ 101.0 │ 50 │ │ MSFT │ 09:31:00.000 │ BUY │ 400.0 │ 20 │ │ MSFT │ 09:31:30.000 │ SELL │ 399.0 │ 30 │ │ NVDA │ 09:32:00.000 │ BUY │ 170.0 │ 100 │ │ AAPL │ 09:32:30.000 │ BUY │ 102.0 │ 200 │ ├──────┴──────────────┴──────┴───────┴─────┤ │ 6 rows (6 shown) 5 columns (5 shown) │ └──────────────────────────────────────────┘ Columns are just regular vectors that can be extracted and passed to ordinary functions: Clojure ‣ (set prices (at trades 'price)) [100.0 101.0 400.0 399.0 170.0 102.0] ‣ (set quantities (at trades 'qty)) [150 50 20 30 100 200] ‣ (* prices quantities) [15000.0 5050.0 8000.0 11970.0 17000.0 20400.0] And the result is another typed vector: Clojure [15000.0 5050.0 8000.0 11970.0 17000.0 20400.0] There is no query yet. Multiplication simply lifts its scalar behavior over two equally sized vectors. Broadcasting is straightforward: Clojure (* prices 1.05) Now place the multiplication inside select: Clojure ‣ (select {from: trades sym: sym price: price qty: qty notional: (* price qty)}) ┌──────┬───────┬─────┬────────────────┐ │ sym │ price │ qty │ notional │ │ SYM │ F64 │ I64 │ F64 │ ├──────┼───────┼─────┼────────────────┤ │ AAPL │ 100.0 │ 150 │ 15000.0 │ │ AAPL │ 101.0 │ 50 │ 5050.0 │ │ MSFT │ 400.0 │ 20 │ 8000.0 │ │ MSFT │ 399.0 │ 30 │ 11970.0 │ │ NVDA │ 170.0 │ 100 │ 17000.0 │ │ AAPL │ 102.0 │ 200 │ 20400.0 │ ├──────┴───────┴─────┴────────────────┤ │ 6 rows (6 shown) 4 columns (4 shown)│ └─────────────────────────────────────┘ Here the expression itself has not changed. What changed is name resolution. Inside query, price, and qty are column names. Before any query evaluation, the engine mounts columns to their names as a regular environment frame, naturally making all existing expressions work the same way as they do in a non-query context! Once a query is passed to a select function, it becomes an operation graph and, when its shape is supported, a fused DAG pipeline. Otherwise, it falls back to a regular operators defined as a language primitives. This opens a door to composing vector operations inside a predicate: Clojure ‣ (select {from: trades … where: (and … (> (* price qty) 10000.0) … (in sym [AAPL MSFT])) … sym: sym-name … notional: (* price qty)}) ┌──────┬──────────────────────────────┐ │ sym │ notional │ │ SYM │ F64 │ ├──────┼──────────────────────────────┤ │ AAPL │ 15000.0 │ │ MSFT │ 11970.0 │ │ AAPL │ 20400.0 │ ├──────┴──────────────────────────────┤ │ 3 rows (3 shown) 2 columns (2 shown)│ └─────────────────────────────────────┘ Here is a short breakdown: `(* price qty)` produces a floating-point vector.`>` converts it into a Boolean vector. `(in sym [AAPL MSFT])` produces another Boolean vector.`and` combines the predicates. The entire expression is reused as a projection. At the language level, this follows the same composition rules as: Clojure (and (> (* prices quantities) 10000.0) (in (at trades 'sym) [AAPL MSFT])) But inside the pipeline, the DAG optimizer can fuse or rewrite expressions without changing semantics. From the user's point of view nothing changed. Such homoiconical behavior allows the use of any vector operations existing in the language inside queries. Consider the following example: Clojure ; xbar rounds values down to a fixed bucket. Applied directly to a time vector ‣ (xbar … [09:30:12.000 09:30:48.000 09:31:05.000] … 60000) [09:30:00.000 09:30:00.000 09:31:00.000] ; The same operation can define a group key ‣ (select {from: trades … by: {minute: (xbar time 60000)} … trades: (count qty) … volume: (sum qty) … vwap: (/ (sum (* price qty)) … (sum qty))}) ┌──────────────┬────────┬────────┬────────┐ │ minute │ trades │ volume │ vwap │ │ TIME │ I64 │ I64 │ F64 │ ├──────────────┼────────┼────────┼────────┤ │ 09:30:00.000 │ 2 │ 200 │ 100.25 │ │ 09:31:00.000 │ 2 │ 50 │ 399.4 │ │ 09:32:00.000 │ 2 │ 300 │ 124.67 │ ├──────────────┴────────┴────────┴────────┤ │ 3 rows (3 shown) 4 columns (4 shown) │ └─────────────────────────────────────────┘ This example reveals several execution levels without changing languages: `xbar` transforms a vector into a grouping key.`*` derives a vector consumed by `sum`.`sum` reduces values per group.`-` folds aggregate results into a scalar per group. In a system with separate array and query languages, these often require different syntax, a user-defined function boundary, or intermediate materialized columns. Here they remain one expression tree. Lambdas The integration becomes even more interesting when the expression is named: Clojure ; define user function: ‣ (set trade-value … (fn [price quantity] … (* price quantity))) lambda ; It can be called with ordinary vectors: ‣ (trade-value … (at trades 'price) … (at trades 'qty)) [15000.0 5050.0 8000.0 11970.0 17000.0 20400.0] ; And the same function can be called with query columns: ‣ (select {from: trades … where: (> (trade-value price qty) 10000.0) … sym: sym-name … value: (trade-value price qty)}) ┌──────┬──────────────────────────────┐ │ sym │ value │ │ SYM │ F64 │ ├──────┼──────────────────────────────┤ │ AAPL │ 15000.0 │ │ MSFT │ 11970.0 │ │ NVDA │ 17000.0 │ │ AAPL │ 20400.0 │ ├──────┴──────────────────────────────┤ │ 4 rows (4 shown) 2 columns (2 shown)│ └─────────────────────────────────────┘ For a lowerable single expression lambda, the query compiler reduces the call into the operation graph. Actual arguments are compiled once and referenced by offsets, so using a formal parameter more than once shares the corresponding subexpression rather than rebuilding it. And this is interesting, because Rayfall is not only giving familiar names to built-in query operations. User-defined lambdas can participate in queries as well as any other language expressions. Layout Internally, a Rayfall builtin is a runtime object. Even more, any datatype, including scalar, vector, dict, table, function, or builtin, is the same ray_t struct with type tag, reference counter, and payload union. This allows implementing a simple and efficient buddy allocator that operates on ray_t blocks; even more on-disk data is exactly the same as in-memory, so the runtime doesn't care about actual object allocation and allows lazy mmaping of huge datasets seamlessly. For standalone vectors, the atomic dispatcher can itself build a small operation graph and execute it via the vector engine. For unsupported shapes, it retains a typed per-operation path. And this happens seamlessly, not exposed to a user Compiler Rayfall has two relevant compilation paths. Ordinary user-defined functions compile lazily into bytecode for a stack VM. The bytecode handles local slots, calls, control flow, recursion, traps, and returns. If compilation can not handle a form, execution can fall back into the recursive evaluator and vice versa. Query expressions take another route. The query layer attempts to lower an expression into a typed operation DAG: Literals become constant nodes.Column names become scan nodes.Arithmetic and comparisons become typed operations.Supported lambdas are beta-reduced.Aggregations become reduction nodes.Structural clauses add filters, groups, projections, sorts, and limits. The DAG then passes through type inference, constant folding, predicate and projection pushdown, filter reordering, partition pruning, and dead-code elimination. The Bottom Line Rayfall can be understood as an attempt to close the space between a low-level C API and a high-level query interface. The resulting language is a LISP-like one, with some extensions like first-class homogeneous vectors, dictionaries and tables. The parser, evaluator, and query compiler discussed here are available in the GitHub repo.

If you have wired an AI agent into a real production workflow, you have probably hit this wall; the agent is genuinely good at the task, but it is expensive to run it every single time, especially when a meaningful chunk of the requests it receives are things it has already solved before. That was exactly the situation I ran into. The setup looked like this; Someone drops a slash command as a GitHub issue comment — something like /collect-data --source=warehouse-a --range=2026-07 A web-hook fires, runs some validation, and triggers a Jenkins job.An AI agent reads a skill definition, does the actual work, and the result gets posted back as another issue comment. It works well. The problem is that a large fraction of these requests are repeats: same source, same range, or a near-miss of something we have already computed. Running a full agent invocation (LLM reasoning + Jenkins pipeline) for a task we have already done is just burning usage credits for no benefit. The fix is not to use a smaller model or prompt more efficiently. It is to stop asking the model in the first place when we already know the answer, and to only ask the part of the question we do not already know. The Core Idea: A Cache-Augmented Agent This is a fairly well-known pattern in retrieval-augmented systems, just applied to task execution instead of document QA. The mental model: Before you reason, look it up. If you find a partial answer, reason about the gap, not the whole thing. Three tiers, cheapest first: TierMechanismCost1. Exact matchSHA-256 hash of normalised task paramsA single indexed DB lookup - no AI2. Semantic matchpgvector cosine similarity within the same task typeA single DB query - no AI3. Agent fallbackFull or scoped agent invocationOnly pay for genuinely new work The key detail that makes this actually save money, rather than just being a fancy cache: tiers 1 and 2 run as plain code in the webhook handler, before the agent is ever invoked. The decision of "do we need the AI here?" is made without AI. Why Hashing Alone Isn't Enough A naive cache would just hash (task_type, params) and check for an exact match. That handles literal repeats — someone re-running the identical command but it misses the far more common case: near-duplicate requests. Think about it from the requester's side. /collect-data --source=warehouse-a --range=2026-07 and /collect-data --source=warehouse-a --range=2026-07 --format=csv are 90% the same task. So are two requests that differ only in a date range that has mostly already been collected. An exact-hash cache treats these as completely unrelated and re-runs the whole thing. That's why there is a second tier: turn the task into a short natural-language description, Plain Text task: collect-data; range=2026-07; source=warehouse-a embed it, and search for the closest prior tasks of the same type using cosine similarity in Postgres with pgvector. If something is very close (above a "full match" threshold), we serve it directly. If it is close but not close enough to the same source, different range, say, we treat it as a partial hit: we know part of the answer, and we hand that to the agent as context so it only has to fill the gap. Schema The whole cache lives in one table, plus an execution log for observability: SQL CREATE EXTENSION IF NOT EXISTS vector; CREATE TABLE task_knowledge ( id BIGSERIAL PRIMARY KEY, task_type TEXT NOT NULL, signature_hash TEXT NOT NULL UNIQUE, -- exact-match lookup params JSONB NOT NULL, description TEXT NOT NULL, -- text fed to the embedding model embedding vector(1024), -- semantic-match lookup result JSONB NOT NULL, covered_scope JSONB NOT NULL DEFAULT '{}'::jsonb, missing_scope JSONB NOT NULL DEFAULT '{}'::jsonb, status TEXT NOT NULL DEFAULT 'complete', -- complete | partial ttl_seconds INT NOT NULL DEFAULT 86400, executed_at TIMESTAMPTZ NOT NULL DEFAULT now() ); CREATE INDEX idx_embedding ON task_knowledge USING ivfflat (embedding vector_cosine_ops) WITH (lists = 100); Two columns do a lot of the conceptual work: covered_scope and missing_scope. Every cached result knows what it actually answers and what it doesn't; this is what lets a partial hit be useful instead of all-or-nothing. The Signature Has to Be Genuinely Deterministic The exact-match tier is only as good as the hash is stable. {"source": "warehouse-a", "range": "2026-07"} and {"range": "2026-07", "Source": "warehouse-a "} need to hash identically, or the cache silently misses on trivial formatting differences. So normalization happens before hashing: Python def normalize_params(params: dict) -> dict: normalized = {} for key, value in params.items(): norm_key = key.strip().lower() if isinstance(value, str): value = value.strip() normalized[norm_key] = value return normalized def build_signature(task_type: str, params: dict) -> str: payload = {"task_type": task_type.strip().lower(), "params": normalize_params(params)} canonical = json.dumps(payload, sort_keys=True, separators=(",", ":")) return hashlib.sha256(canonical.encode()).hexdigest() sort_keys=True matters more than it looks without it; dict key order leaks into the hash, and two functionally identical requests produce different signatures. The Lookup Flow Python async def handle_task(ctx: TaskContext) -> None: task_type, params = ctx.command.task_type, ctx.command.params if not ctx.command.force: exact = await kb_service.exact_lookup(task_type, params) if exact.kind == "exact": return await _serve_cached(ctx, exact.result, "cache-exact") semantic = await kb_service.semantic_lookup(task_type, params) if semantic.kind == "semantic_full": return await _serve_cached(ctx, semantic.result, "cache-semantic") if semantic.kind == "semantic_partial": return await _run_agent_and_finish( ctx, scope="partial", prior_result=semantic.result, missing_scope=semantic.missing_scope, ) # nothing usable in cache, or --force was passed await _run_agent_and_finish(ctx, scope="full", prior_result=None, missing_scope=None) Notice the order: cheapest and most certain first. By the time you are calling the agent, you already know either this is genuinely new or here is what exactly is missing; the agent never has to rediscover context it already had access to in a prior run. Scoping the Agent Call Is the Actual Cost Saver It is tempting to stop at caching the full result and skip the agent on hits. That alone helps, but the bigger win is what happens on a partial hit. Instead of: Do the whole task from scratch the agent gets: Here's what we already know. Here is specifically what is missing. Fill only that. Plain Text payload = { "task_type": task_type, "params": params, "scope": scope, # "full" or "partial" "prior_result": prior_result, # trusted context on a partial run "missing_scope": missing_scope, # exactly what to compute } A well-scoped prompt on a partial hit is dramatically cheaper than a cold-start prompt has less context to establish, less reasoning to redo, and fewer tool calls in many cases. This is the difference between caching the whole answer or not and actually decomposing the task so the agent's effort is proportional to what is genuinely new. Freshness Matters as Much as Matching A cache with no expiry is a correctness bug waiting to happen; data pipelines especially. ttl_seconds is set per task type (data pulls might be valid for a day, static reference lookups for a month), and every lookup checks staleness before it is considered a hit at all: Python def _is_fresh(executed_at: datetime, ttl_seconds: int) -> bool: age = (datetime.now(timezone.utc) - executed_at).total_seconds() return age <= ttl_seconds And because the cache is wrong is always a possibility someone needs to escape from, the slash command supports a --force flag that skips all three tiers and always re-runs the agent; cheap insurance against a bad cache entry blocking someone. What This Actually Buys You For a workflow where a meaningful fraction of requests are repeats or near-repeats: Exact hits cost nothing – a single indexed hash lookup instead of an agent invocation and a Jenkins run.Semantic hits cost nothing – same, just via vector similarity instead of literal equality.Partial hits cost a fraction of a full run – the agent's context and reasoning scope shrink to just the gap.The system gets better over time – every agent run, full or partial, ends with an upsert into the knowledge base, so the next similar request has a better chance of hitting tier 1 or 2. None of this requires touching the AI agent's internals or model choice. It is entirely a decision layer sitting in front of it, which is exactly why it is cheap to build and safe to roll out incrementally: worst case, everything falls through to tier 3 and behaves exactly like the system did before. Where This Pattern Breaks Down Worth being honest about the limits: Highly unique tasks (every request meaningfully different) get no benefit; you are just adding a cache lookup with no hits.Semantic thresholds need real tuning. Too loose, and you serve stale near-misses as if they were exact. Too tight, and tier 2 never fires, and you've built a vector index for nothing. This needs actual production traffic to calibrate, not guesswork.Partial-scope decomposition only works if your agent (or its skill definitions) can meaningfully interpret "do just this part." Some tasks are not decomposable; collecting one row of a dataset is not a well-defined sub-task if the pipeline processes the range as a single unit. In those cases, a partial hit should probably just be a lower similarity threshold for a full re-run, not a scoped one.Correctness > cost. If being wrong is expensive (financial data, compliance), skew every tuning knob toward fewer cache hits, not more. The Broader Point AI agents are excellent at reasoning over genuinely new problems and bad economics for repeated ones. Most production agent workflows I have seen treat every request as novel by default, which is the expensive default. Adding a deterministic lookup layer in front — one that is cheap enough to always check and specific enough to trust — turns running the agent from the default action into the fallback action. That one inversion is where most of the savings come from.

Senior data engineers are trained to be skeptical of proprietary platforms. When I entered a Palantir Foundry training bootcamp, I expected to find a slow, expensive alternative to the mature tools I know on AWS and Azure. What I found instead was a platform built for a radically different user, one who cannot write SQL but needs answers now. I want to write about what I actually observed honestly, including where I think the hype is justified and where I think it is not, because most Foundry content I have seen is either from Palantir's own marketing or from practitioners so embedded in the platform they have forgotten what it was like to come to it fresh. I am writing this while that perspective is still clear. The Speed Thing Is Real The surprise that hit me hardest was not a feature. It was pace. During the bootcamp we worked across a range of tasks: connecting data sources, building transformation pipelines, setting up workflows that business users could interact with directly. To make this concrete: building a pipeline that ingested data from multiple sources, applied transformations, and exposed the output to business users took only hours in Foundry. On a standard AWS or Snowflake stack with dbt and an orchestration layer, a comparable setup typically runs to a full sprint for a small team, not because of any single hard step, but because of the coordination overhead between tools. I want to be careful about what I am and am not claiming here. This was a structured training environment with guided examples, not production infrastructure with real enterprise complexity and legacy constraints. The comparison is not controlled. But the direction of the difference was clear enough that I took notice. Foundry's Pipeline Builder abstracts away a lot of the coordination work that consumes time in a more assembled stack. Whether that advantage holds at full scale is a question I cannot answer from a single bootcamp, but it is worth asking seriously. The honest counter-argument: speed in a training environment does not always translate to speed in production. A well-resourced engineering team that already knows Snowflake deeply can move fast too, without the overhead of learning a new paradigm. If your team is highly capable on your current stack, the productivity gain from switching may not justify the learning curve cost. "Tasks I would have planned for a full day on my normal stack were done in a couple of hours." Who Actually Benefits Most However, raw speed is not the platform's most disruptive feature. The more I used it, the more I realized that the real value of that speed is not for engineers. It is for the people who are usually waiting on us. The more I worked with Foundry during the training, the clearer it became that the people getting the most out of it in the room were not the engineers. They were the non-technical participants, the analysts, the operations people, the business users who in a traditional stack would be waiting for an engineer to build them something before they could interact with data at all. Foundry's ontology model, the way it creates a shared semantic layer that different types of users can navigate without writing code, is differentiated from what I work with on AWS, Azure, and Snowflake. On those platforms, self-service data access for non-engineers is possible, but it takes deliberate, often significant engineering effort to expose data in a way that non-technical people can actually use. In Foundry, it felt closer to the default. If I were advising an organization on whether to consider Foundry, the first question I would ask is: what percentage of the people who need to interact with your data can actually write SQL? In organizations where more than half of business analysts and operational users cannot write code, the engineering burden of building self-service access on a traditional stack becomes a recurring, compounding cost. That is the environment where Foundry's default self-service capabilities start to justify serious evaluation. The counter-argument here is worth stating directly: a strong, well-resourced data engineering team could build a better, more tailored self-service layer on Snowflake in the same time it takes to master Foundry's ontology. If your organization has that team and the patience to build the right abstractions, the open platform may serve you better in the long run. Foundry's self-service advantage is most compelling when you do not have that engineering capacity, or when the number of non-technical users is large enough that a custom-built solution would require constant maintenance. The Cost Reality Palantir does not publish list pricing for Foundry. Everything is negotiated. The platform uses a core-based licensing model, meaning you pay based on the computational capacity (server cores) allocated to the platform rather than by the number of users. Based on publicly available government procurement records, core-based licenses start at roughly 66,000 pounds per server core per year, with no additional per-user fees on top. Solution-based use case licenses, which bundle implementation and support, start at 250,000 pounds at entry level and scale significantly from there depending on data complexity, user base, and operational scope. What this means practically is that Foundry's cost is not a fixed number you can evaluate on a spreadsheet. It is a negotiation. According to procurement advisory analysis of Palantir Foundry negotiations conducted between 2024 and 2025, annual platform fees for comparable mid-size deployments varied by a factor of two to three depending purely on negotiation posture (Redress Compliance, 2025). The leverage comes primarily from having a credible, costed alternative, which for most organizations means Databricks or Snowflake with named engineering owners and a realistic build timeline. Organizations that enter Palantir conversations without that alternative built tend to pay significantly more for the same deployment than organizations that do. "The leverage in the Foundry cost negotiation comes primarily from having a credible, costed alternative built before you walk in." My honest assessment after the bootcamp is that the cost is hard to justify for smaller organizations or simpler use cases. If a well-designed Snowflake environment can meet your data engineering needs with dbt and a standard BI layer on top, Foundry is probably not the right answer, and the delta in platform cost will buy you a lot of engineering time on the stack you already know. The calculus changes for large enterprises with complex, multi-team data environments and a significant population of non-technical users who need meaningful data access. What I Would Tell a Data Engineering Leader A few things I would want another senior data engineer or engineering leader to know before evaluating Foundry: Do not evaluate Foundry on pipeline performance alone. That is not its primary differentiator. Compare it to Snowflake or Databricks on what it does for the non-engineer users in your organization, not on compute efficiency.Build your alternative cost model first. Whatever your current stack is, cost out what it would take to build the data product capabilities Foundry promises on that stack, with your own team. That number is your negotiating anchor.Take the learning curve seriously. Foundry has a broad ecosystem: the ontology model, Pipeline Builder, Code Repositories, AI integrations, and coming to it fresh from a traditional data engineering background takes real adjustment. The training helped, but it is not a platform you pick up in a day.Be specific about who your users are. Foundry earns its cost fastest in environments where non-technical users need to do more with data than your current stack allows. If your users are primarily technical, the value proposition narrows considerably.Negotiate the second contract inside the first. Procurement analysis consistently shows that organizations that lock in phase two pricing before signing the initial contract pay significantly less per added use case than those who do not. Treat the pilot as the deal. The Honest Summary I came to Palantir Foundry expecting to be underwhelmed. I was not. But understanding its value requires a paradigm shift for any engineer raised on AWS or Snowflake. Evaluate Foundry not as a faster pipeline tool, but as a platform for organizational data literacy. For enterprises drowning in data but starved of accessible insights, it is a compelling, if expensive, contender. For everyone else, the tools you already have remain the better investment. The challenge is being honest enough with yourself to know which bucket your organization falls into.

Most conversations about vector databases start and end with embeddings. Discussions typically center around how they're generated, which model produced them, how many dimensions they carry. Embeddings get all the attention, but they aren't what determines whether your AI search, RAG pipeline, or recommendation engine feels instant or painfully slow in production. That comes down to indexing. The exact same set of vectors can return results in single-digit milliseconds or take seconds to resolve, depending entirely on the indexing strategy running underneath the database. This article breaks down how vector indexing actually works, walks through the major indexing types in production use today, and lays out the tradeoffs engineers need to understand before choosing one. Why Indexing Is the Real Bottleneck A vector database's job is to find the nearest neighbors to a query vector inside a massive, high-dimensional space. Doing this exactly — comparing the query against every single stored vector — is mathematically simple but computationally expensive. As the dataset grows into the millions or billions of vectors, brute-force comparison stops being viable for any latency-sensitive application. Indexing strategies exist to solve exactly this problem: they organize vectors ahead of time so that search can skip most of the dataset instead of scanning all of it. The strategy chosen determines the balance a system strikes between four competing constraints: Speed: How fast a query returns results.Accuracy: How close the returned neighbors are to the true nearest neighbors.Memory: How much RAM or disk the index consumes.Mutability: How well the index handles inserts, updates, and deletes after it's built. No indexing method wins on all four. Understanding that tradeoff is the actual skill in building production vector search systems. The Major Indexing Types Flat Index A flat index performs brute-force search: every query is compared against every vector in the dataset. There's no approximation involved, so accuracy is as good as it gets. This is exact nearest-neighbor search. The cost is scalability. Search time grows linearly with the size of the dataset, which makes flat indexes impractical once collections move past a few hundred thousand vectors. They're most useful for small datasets, baseline accuracy testing, or as a ground truth to benchmark approximate methods against. IVF (Inverted File Index) IVF clusters the vector space ahead of time — typically using something like k-means — and assigns every vector to its nearest cluster centroid. At query time, the search only probes the handful of clusters closest to the query vector instead of the entire dataset. This makes IVF significantly faster than a flat index at scale, at the cost of some accuracy, since relevant vectors sitting near a cluster boundary can be missed. It's a solid middle ground for large-scale retrieval where some approximation is acceptable. HNSW (Hierarchical Navigable Small World) HNSW builds a multi-layer graph structure where vectors are connected to their approximate neighbors, and search navigates the graph from a sparse top layer down to denser lower layers to converge on nearest neighbors. HNSW is the reason "approximate nearest neighbor" search became viable at production scale. It delivers a strong balance of speed and accuracy, supports incremental updates far better than most alternatives, and is why it underpins the majority of modern production vector databases and libraries. IVF + PQ (Product Quantization) IVF + PQ combines IVF's clustering with product quantization, which compresses each vector into a compact code rather than storing it at full precision. The result is a dramatic reduction in memory footprint — often an order of magnitude smaller than storing raw vectors. The tradeoff is a further hit to accuracy from the compression itself, on top of IVF's own approximation. This method matters most when the dataset is too large to fit in memory at full precision and memory efficiency becomes the binding constraint. LSH (Locality-Sensitive Hashing) LSH uses hash functions specifically designed so that similar vectors are more likely to land in the same hash bucket. Search then only needs to compare against vectors sharing a bucket with the query. LSH tends to be less accurate and less commonly used in modern production stacks than HNSW or IVF variants, but it remains relevant in niche, low-latency similarity-grouping use cases where its specific properties fit well. Annoy (Approximate Nearest Neighbors Oh Yeah) Annoy builds a forest of random projection trees to partition the vector space, optimized heavily for fast, read-only lookups. It's particularly well suited to recommendation workloads where the index is built once and queried repeatedly with minimal updates. Its main limitation is mutability — Annoy indexes are not designed for frequent inserts or updates, making it a poor fit for datasets that change often. Speed vs. Accuracy vs. Memory: The Core Tradeoff Every indexing method sits somewhere on a triangle between speed, accuracy, and memory usage — and pulling one lever tends to push against the others: IndexSpeedAccuracyMemorySupports UpdatesFlatSlow at scaleExactHighYesIVFFastApproximateModerateLimitedHNSWFastHighModerate–HighYesIVF + PQFastLowerVery lowLimitedLSHFast (niche)LowerLow–ModerateYesAnnoyFast (read-heavy)ApproximateModerateNo (static) This is why approximate search beats exact search in most production systems: once a dataset reaches real scale, the marginal accuracy loss from approximation is a small price for the latency and infrastructure savings it buys. Exact search only makes sense when the dataset is small enough, or the accuracy requirement strict enough, that brute-force comparison remains fast. Why HNSW Dominates Today HNSW's popularity in modern vector databases isn't accidental. It occupies the most favorable point on the tradeoff triangle for the majority of real-world workloads: strong recall, low latency, and, critically, the ability to handle updates without a full rebuild. RAG systems and semantic search applications, where the underlying document set changes constantly, benefit directly from that mutability in a way that static-index methods like Annoy cannot match. That said, "dominates" doesn't mean "always correct choice." Memory-constrained environments still lean on IVF + PQ. Read-heavy, rarely-updated recommendation systems still get real value from Annoy's simplicity and speed. The right index is a function of the workload, not a universal ranking. How Indexing Impacts Cost and Scalability Indexing choice ripples directly into infrastructure economics: Latency: An index poorly matched to dataset size and query volume shows up immediately as slow response times — the difference between a flat index and HNSW at scale can be orders of magnitude.Scalability: Some methods (flat, to a lesser extent IVF) degrade as data grows; others (HNSW, IVF+PQ) are explicitly engineered to hold performance steady as scale increases.Infrastructure cost: Memory-hungry indexes translate directly into more expensive hardware. Compression-based methods like IVF+PQ exist specifically to control that cost at scale.Update patterns: A dataset that changes frequently needs an index that supports mutability without expensive full rebuilds. A mismatch here creates operational overhead that compounds over time. The Real Takeaway There is no universally "best" vector index, only the right tradeoff for a given workload between speed, accuracy, memory usage, and update frequency. Vector databases aren't magic. They're carefully engineered compromises, purpose-built to make searching astronomically large, high-dimensional spaces feel instantaneous. Understanding the indexing layer, not just the embeddings feeding into it, is what separates a vector search system that scales gracefully from one that quietly falls over as data grows.

By Balaji Venkatasubramaniyar CORE

The Uncomfortable Truth You’ve spent days prompt-engineering your LLM. You’ve benchmarked Claude against GPT. You’ve debated whether to use Mixtral. But your RAG pipeline is still returning garbage answers, and you’re blaming the wrong component. The LLM is only as good as the context it receives. Context quality is entirely determined by retrieval. Retrieval quality is entirely determined by your embedding model. Fix the bottom, and the top fixes itself. I ran the same RAG pipeline across four embedding models on a 10,000-document legal corpus Q and A task. Same LLM (Claude Sonnet 4.6), same chunking strategy, same vector store (pgvector), same top-k=5. Only the embedding model changed. Model Retrieval P@5 Faithfulness Dims Cost/1M text-embedding-3-large 0.91 0.88 3072 $0.13 BGE-M3 (local) 0.88 0.85 1024 Free text-embedding-3-small 0.74 0.69 1536 $0.02 all-MiniLM-L6-v2 0.61 0.55 384 Free The gap between all-MiniLM-L6-v2 and text-embedding-3-large is 30 precision points. That’s not a minor tweak. That’s the difference between a product people trust and one they abandon. Your LLM had nothing to do with it. Why Embedding Models Differ So Dramatically An embedding model maps text into a high-dimensional vector space. Two chunks are “similar” if their vectors are close, measured by cosine similarity. The problem: not all models learn the same notion of similarity. A general-purpose model trained on web data will cluster “bank” near both “river” and “finance.” A domain-aware model trained on legal or financial corpora knows context. This distinction cascades into every retrieval decision your system makes. What Embedding Models Actually Learn During training, embedding models are optimized to pull semantically similar sentences closer in vector space and push dissimilar ones apart. The training data, loss function, and model architecture determine what “similar” means. Contrastive learning (SBERT, BGE): Learns from positive/negative sentence pairsMatryoshka Representation Learning (MRL, OpenAI): Encodes quality at multiple scalesLate interaction models (ColBERT): Compares token-level representations at query timeSparse + dense hybrids (BGE-M3): Combines lexical and semantic signals Key Insight: Embedding models encode your domain assumptions. If your model doesn’t understand your domain, no amount of LLM tuning will compensate for what it retrieves. Benchmarking Embedding Models on Your Own Data Don’t trust vendor benchmarks on MTEB. MTEB tests general English retrieval. Your use case is specific. Run this evaluation harness against your own corpus before committing to any embedding model: Python from sentence_transformers import SentenceTransformer from openai import OpenAI import numpy as np from sklearn.metrics.pairwise import cosine_similarity # Ground-truth query -> relevant chunk pairs from YOUR data eval_pairs = [ ("What is the penalty for breach of contract?", "Section 12.3 outlines liquidated damages of 5%..."), ("When does the agreement terminate?", "This agreement expires on December 31st 2026..."), ] def precision_at_k(embed_fn, corpus, queries, relevant_ids, k=5): corpus_embs = embed_fn(corpus) hits = 0 for i, query in enumerate(queries): q_emb = embed_fn([query]) sims = cosine_similarity(q_emb, corpus_embs)[0] top_k = np.argsort(sims)[::-1][:k] if relevant_ids[i] in top_k: hits += 1 return hits / len(queries) # Wrap OpenAI embeddings client = OpenAI() def openai_embed(texts, model="text-embedding-3-large"): resp = client.embeddings.create(input=texts, model=model) return np.array([d.embedding for d in resp.data]) # Wrap local BGE-M3 st_model = SentenceTransformer("BAAI/bge-m3") def bge_embed(texts): return st_model.encode(texts, normalize_embeddings=True) models = { "text-embedding-3-large": openai_embed, "BGE-M3 (local)": bge_embed, } for name, fn in models.items(): score = precision_at_k(fn, corpus, queries, relevant_ids) print(f"{name}: precision@5 = {score:.3f}") Run this before you commit to any embedding model. Twenty minutes of benchmarking here saves weeks of LLM debugging later. Build the evaluation dataset from your domain expert’s known query-answer pairs — even 50 pairs gives a strong signal. Matryoshka Embeddings: Large-Model Quality at Small-Model Cost OpenAI’s text-embedding-3 models support Matryoshka Representation Learning (MRL). The model is trained so that any prefix of the full embedding vector retains useful semantic structure. This means you can truncate a 3072-dimensional vector to 512 dimensions and still retain ~94% of its retrieval quality, at a fraction of the storage cost. Python from openai import OpenAI import numpy as np client = OpenAI() def embed_with_matryoshka(texts: list[str], dimensions: int = 512): """ text-embedding-3-large supports 256 -> 3072 dims. 512 dims = ~83% storage reduction, ~94% benchmark quality retained. """ response = client.embeddings.create( input=texts, model="text-embedding-3-large", dimensions=dimensions ) return np.array([item.embedding for item in response.data]) # Insert into pgvector import psycopg2 conn = psycopg2.connect(DATABASE_URL) cur = conn.cursor() for chunk_id, chunk_text in chunks: emb = embed_with_matryoshka([chunk_text], dimensions=512)[0] cur.execute( "INSERT INTO documents (id, content, embedding) VALUES (%s, %s, %s)", (chunk_id, chunk_text, emb.tolist()) ) conn.commit() With 512 dimensions, you get ~94% of the full model’s retrieval quality at ~17% of the storage and index cost. This is the default I now recommend for most production RAG pipelines. Only go to 3072 if you’re in a domain with extremely dense technical vocabulary. Going Local With BGE-M3 and Hybrid Retrieval API-based embeddings have three costs: latency, money, and privacy. If you’re embedding sensitive documents — legal contracts, medical records, internal financials — sending them to an external API is a risk your legal team will veto. BGE-M3 from BAAI solves all three problems. It runs locally, it’s multilingual (100+ languages), and it uniquely supports three retrieval modes from a single model: dense, sparse (BM25-style), and ColBERT late interaction. The hybrid of dense + sparse consistently outperforms pure dense retrieval by 5-12 points on technical corpora. Python from FlagEmbedding import BGEM3FlagModel import numpy as np # Load once, reuse across requests model = BGEM3FlagModel("BAAI/bge-m3", use_fp16=True) def hybrid_embed(texts: list[str]) -> tuple: """ Returns dense vectors + sparse lexical weights. Combine both in your vector store with Reciprocal Rank Fusion. """ output = model.encode( texts, return_dense=True, return_sparse=True, return_colbert_vecs=False ) return output["dense_vecs"], output["lexical_weights"] def reciprocal_rank_fusion(dense_ranks, sparse_ranks, k=60): """Combine dense and sparse retrieval results with RRF.""" scores = {} for rank, doc_id in enumerate(dense_ranks): scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank + 1) for rank, doc_id in enumerate(sparse_ranks): scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank + 1) return sorted(scores, key=scores.get, reverse=True) # Usage dense_vecs, sparse_weights = hybrid_embed(document_chunks) # Store dense_vecs in pgvector / Qdrant / Weaviate # Store sparse_weights in Elasticsearch / OpenSearch BM25 index # At query time, retrieve from both and fuse with RRF Chunking Strategy Interacts With Your Embedding Model Your embedding model and your chunking strategy are not independent decisions. A model trained on sentence pairs (like SBERT) performs best when chunks are coherent semantic units — not arbitrary 512-token windows. If you’re using semantic chunking, your choice of embedding model even affects how you split the document. Semantic Chunking With Sentence Similarity Python from sentence_transformers import SentenceTransformer import numpy as np model = SentenceTransformer("BAAI/bge-m3") def semantic_chunk(text: str, threshold: float = 0.75) -> list[str]: """ Split text where consecutive sentence similarity drops below threshold. Preserves semantic coherence per chunk. """ sentences = text.split('. ') embeddings = model.encode(sentences, normalize_embeddings=True) chunks, current = [], [sentences[0]] for i in range(1, len(sentences)): sim = np.dot(embeddings[i-1], embeddings[i]) if sim < threshold: chunks.append('. '.join(current)) current = [sentences[i]] else: current.append(sentences[i]) chunks.append('. '.join(current)) return chunks chunks = semantic_chunk(document_text, threshold=0.72) print(f"Created {len(chunks)} semantic chunks") # vs naive: 512-token windows often cut mid-sentence Using semantic chunking with BGE-M3 on the same legal document corpus improved my precision@5 by an additional 6 points over fixed-size chunking with the same model. The embedding model and chunking strategy compound each other. Production Architecture: Putting It Together Here is the full production-grade embedding pipeline I use, combining everything covered above: Matryoshka embeddings for cost efficiency, hybrid retrieval for precision, and a fallback to local BGE-M3 for sensitive data: Python import os from enum import Enum from dataclasses import dataclass from openai import OpenAI from FlagEmbedding import BGEM3FlagModel import numpy as np class EmbedMode(Enum): OPENAI_MRL = 'openai_mrl' # API, cost-efficient via Matryoshka BGE_HYBRID = 'bge_hybrid' # Local, privacy-safe, hybrid retrieval @dataclass class EmbedConfig: mode: EmbedMode dimensions: int = 512 # For OPENAI_MRL fp16: bool = True # For BGE_HYBRID class ProductionEmbedder: def __init__(self, config: EmbedConfig): self.config = config if config.mode == EmbedMode.OPENAI_MRL: self.client = OpenAI() else: self.model = BGEM3FlagModel("BAAI/bge-m3", use_fp16=config.fp16) def embed(self, texts: list[str]) -> dict: if self.config.mode == EmbedMode.OPENAI_MRL: resp = self.client.embeddings.create( input=texts, model="text-embedding-3-large", dimensions=self.config.dimensions ) return {'dense': np.array([d.embedding for d in resp.data])} else: out = self.model.encode( texts, return_dense=True, return_sparse=True ) return {"dense": out["dense_vecs"], "sparse": out["lexical_weights"]} # For most teams: embedder = ProductionEmbedder(EmbedConfig(mode=EmbedMode.OPENAI_MRL, dimensions=512)) # For regulated industries (healthcare, legal, finance): # embedder = ProductionEmbedder(EmbedConfig(mode=EmbedMode.BGE_HYBRID)) The Decision Framework Here is how I choose an embedding model for any new project. Run through these questions in order: Situation Recommended model Why General English, tight budget text-embedding-3-small Best cost/quality ratio for standard RAG Best API quality, flexible cost text-embedding-3-large @ 512d (MRL) 94% quality at 17% storage cost Privacy / regulated industry BGE-M3 local + hybrid On-prem, no data leaves your infra Multilingual corpus BGE-M3 local 100+ languages, best non-English retrieval Edge / mobile / <50ms latency all-MiniLM-L6-v2 Tiny model, still useful for simple domains Unknown domain Benchmark first Run precision@5 eval before committing Common Mistakes to Avoid Using the same embedding model for indexing and a different one at query time: Vectors from different models are not comparable. Always use identical model + dimension settings for both.Not normalizing embeddings before cosine similarity: Always set normalize_embeddings= True or call np.linalg.norm(v) yourself.Choosing dimensionality based on benchmark scores, not your actual index size: At 10M+ documents, the storage cost of 3072d vs 512d is the difference between a $200/month and a $1,200/month vector DB bill.Ignoring embedding model updates: Text-embedding-3 is not the same as text-embedding-ada-002. Re-embed your entire corpus when you upgrade models. Do not mix vectors from different model versions in the same index.Skipping evaluation on your own data: MTEB leaderboard rankings do not predict performance on your specific domain. A model ranked 5th globally may outperform a rank 1 on your data. Conclusion Your LLM is not your bottleneck. Switching from GPT-4 to Claude Sonnet gives you maybe a 5% quality lift on a well-constructed RAG pipeline. Switching from all-MiniLM-L6-v2 to text-embedding-3-large gave me 30 precision points on the same pipeline, with the same LLM and the same prompt. Benchmark your embeddings first. Pick your chunking strategy second. Obsess about your LLM last. The teams winning with production AI are not the ones with the best model subscription. They are the ones who built retrieval pipelines that actually surface the right context, and it starts with the embedding model. As a rule of thumb, spend 20% of your AI engineering time on embedding evaluation. It will return 80% of your RAG quality gains.

Compliance Checkbox vs. Architectural Constraint Most data platforms treat audit-readiness as a downstream concern. The pipelines are built, the warehouse is populated, the dashboards ship, and only then does someone ask how the platform would respond to a regulator's request to reconstruct account balances as of a date eighteen months ago, or to prove that a reported figure hasn't been altered since submission. At that point, the answer is usually assembled after the fact: cross-referencing backups, reconstructing state from scattered logs, or worse, discovering that the required history was never captured at all. This reactive posture is what "compliance checkbox" architecture looks like in practice. The alternative audit-ready by design treats three properties as non-negotiable architectural constraints from the outset, not features added later: lineage, point-in-time reconstruction, and immutability. The distinction matters because a constraint enforced at the architecture level cannot be silently bypassed under deadline pressure the way a bolted-on compliance script can. Three Architectural Constraints, Defined Lineage Every data point must be traceable to its origin, and every transformation it passed through must be recorded as a first-class artifact of the pipeline, not reconstructed later from job logs or tribal knowledge. Lineage that lives only in a wiki page or a data dictionary is documentation, not architecture; lineage that lives in pipeline metadata, enforced by the platform itself, is a constraint. Point-in-Time Reconstruction A regulator's question is rarely "what does the data look like today"; it's "what did the data look like as of a specific past date, and can you prove it?" A platform designed for point-in-time reconstruction can reproduce the exact reported state as of any historical timestamp, not just restore from the nearest backup window. Immutability Once a record has been reported or submitted, it should be architecturally incapable of silent modification. This doesn't mean data can never be corrected; it means corrections are new, versioned, timestamped events layered on top of history, never in-place overwrites of it. The Audit-Readiness by Design (ARD) Maturity Model To evaluate whether a given data platform or pipeline is genuinely audit-ready by design, it helps to score it across the same three dimensions on a four-level maturity scale from bolted-on compliance to constraint-native architecture. This is deliberately structured the same way infrastructure maturity models work: each level represents a materially different failure mode under regulatory scrutiny, not just a stylistic difference. Dimension Level 1: Bolted-On Level 2: Retrofitted Level 3: Designed-In Level 4: Constraint-Native Lineage Manual documentation only; no code-level trace Logging added after pipelines built; partial coverage Lineage captured by pipeline metadata at build time Lineage is a required schema element; pipelines fail to deploy without it Point-in-Time Reconstruction No historical state; only current snapshot exists Periodic backups allow coarse-grained rollback Versioned tables enable reconstruction to any recorded checkpoint Any timestamp is reconstructable to the transaction level, by design Immutability Tables freely overwritten (UPDATE/DELETE in place) Soft-delete flags added; underlying rows still mutable Append-only storage for regulated tables Immutability enforced at the storage layer; mutation is architecturally impossible A platform's ARD maturity is not a single score but a profile across the three rows; it's common to see a platform at Level 3 on Immutability while still at Level 1 on Lineage, and that gap is usually exactly where audit findings originate. The model is most useful as a gap-identification tool during architecture review, applied per regulated data domain rather than to an entire platform at once, since different domains (e.g., transactional reporting vs. internal analytics) typically warrant different target levels. Design Patterns That Support Each Constraint Event sourcing: Storing state as an append-only sequence of events rather than mutable current-state tables gives lineage and immutability simultaneously; the event log is both the audit trail and the source of truth.Table versioning/time-travel storage: Storage formats that retain prior versions of a table as of any commit or timestamp directly support point-in-time reconstruction without requiring separate backup infrastructure.Append-only ledgers for regulated tables: Rather than updating a row, a correction is written as a new row referencing the one it supersedes; the history is never destroyed, only extended.Metadata-driven pipeline orchestration: Lineage capture built into the orchestration layer itself (rather than added as a separate logging step) ensures lineage cannot be skipped, since the pipeline cannot run without emitting it. Common Pitfalls Lineage tracked only in documentation: A data dictionary or architecture diagram is not evidence a regulator can independently verify against the running system.Silent backfills: Correcting historical data by overwriting it in place destroys the very history the platform may later be asked to prove.Soft-delete mistaken for immutability: A boolean "deleted" flag on an otherwise mutable row provides none of the guarantees of true append-only storage.Backup cadence mistaken for point-in-time capability: Nightly backups allow rollback to the nearest backup window, not reconstruction of the exact state as of an arbitrary past timestamp. A Composite Example Consider a generalized (composite, non-attributable) regulatory reporting platform for a financial services organization. An architecture review using the ARD model found the platform at Level 3 on Immutability (append-only storage for core ledger tables) but Level 1 on Lineage transformation logic lived in scheduler scripts with no captured metadata trail. When a regulator later requested a full transformation history for a reported figure, reconstructing it took several weeks of manual log archaeology rather than a direct query. Applying the ARD model earlier in the platform's design would have surfaced this specific gap: a strong Immutability posture masking a materially weaker Lineage posture well before it became a live audit finding. Conclusion Audit-readiness that is designed in behaves fundamentally differently under regulatory pressure than audit-readiness that is bolted on: one is a property of the architecture that cannot be quietly skipped, the other is a checklist item that depends on someone remembering to run it. Treating lineage, point-in-time reconstruction, and immutability as architectural constraints and using a structured model like ARD to find the gaps between them turns audit-readiness from a recurring fire drill into a property the platform simply has.