DZone Spotlight

Modern CI/CD pipelines often execute complete regression suites for every code change, regardless of the actual impact of the modification. While this approach guarantees broad validation coverage, it also introduces unnecessary test execution, slower feedback loops, and increased infrastructure cost. This challenge becomes more visible in microservice-based systems where repositories, services, and automation suites are distributed across multiple projects. A small change in one module can unintentionally trigger an entire regression pipeline containing tests unrelated to the updated code. To explore a lightweight solution for this problem, I built a personal engineering project that performs impact-based test selection using Git diff analysis, Spring Boot, JGit, GitHub Actions, and Karate. The goal of the project was simple: instead of executing the full regression suite for every change, dynamically determine which tests are actually impacted and execute only those tests. The project demonstrates how selective test execution can help reduce unnecessary CI workload while maintaining baseline validation coverage through fallback smoke testing. The Problem With Traditional Regression Execution In many CI/CD pipelines, regression execution is static. Every push event or pull request triggers: Full API regression suitesComplete integration testingBroad validation across unrelated modules Although this guarantees high coverage, it creates several practical problems. Slow Feedback Cycles Developers may wait several minutes or even longer to receive pipeline feedback for small localized changes. For example, updating a single payments API controller might still trigger: transactions teststransfer testsauthentication regression testsunrelated smoke validations As projects scale, this delay affects engineering productivity and release speed. Unnecessary Infrastructure Usage Executing the same large regression suites repeatedly consumes unnecessary CI resources. This becomes more expensive when: Pipelines run in parallelMultiple pull requests are activeCloud runners are billed by execution time Reduced Pipeline Efficiency In many cases, only a small subset of tests is truly relevant to the code change. Running the full suite results in redundant execution and inefficient utilization of CI infrastructure. The objective of this project was to explore whether lightweight impact analysis could reduce unnecessary test execution without introducing complicated tooling or dependency management systems. Project Overview The solution uses Git diff analysis to identify changed files and map those changes to relevant Karate test tags. The architecture consists of two repositories: Plain Text Developer Repository (fintech-impact-services) ↓ Push Event GitHub Repository Dispatch ↓ Automation Repository (karate-change-impact-test) ├── Start Spring Boot Application ├── Perform Git Diff Analysis ├── Generate Impacted Test Tags ├── Execute Targeted Karate Tests └── Generate Execution Metrics The development repository contains the Spring Boot microservice and impact analysis API. The automation repository contains Karate test suites and GitHub Actions workflows responsible for selective test execution. This separation allowed the automation layer to remain reusable and independently managed. Cross-Repository Workflow The workflow begins when code is pushed to the development repository. A GitHub Repository Dispatch event triggers the automation repository pipeline. Example dispatch payload: Shell curl -X POST \ -H "Accept: application/vnd.github+json" \ -H "Authorization: Bearer <TOKEN>" \ https://api.github.com/repos/{owner}/karate-change-impact-test/dispatches \ -d '{ "event_type": "dev_push" }' The automation repository then performs the following steps: Checkout repositoriesStart the Spring Boot impact-analysis serviceWait for API readinessCall the impact-analysis endpointRetrieve impacted Karate tagsExecute only selected testsPublish execution metrics This design helped simulate a lightweight cross-repository CI orchestration model using only GitHub-native capabilities. Implementing Git Diff Analysis Using JGit The core functionality of the project relies on identifying changed files between commits. Instead of using shell-based Git commands inside CI scripts, the implementation uses JGit, a Java library that provides Git functionality directly within Java applications. The service compares the current branch with a target branch reference and extracts modified file paths. Example implementation: Java public Set<String> getImpactedTags(String targetBranch) throws Exception { FileRepositoryBuilder builder = new FileRepositoryBuilder(); Repository repository = builder.readEnvironment() .findGitDir(new File(".")) .build(); try (Git git = new Git(repository)) { AbstractTreeIterator oldTreeParser = prepareTreeParser(repository, targetBranch); List<DiffEntry> diffs = git.diff() .setOldTree(oldTreeParser) .call(); return calculateTags(diffs); } } Once the diff entries are collected, the application extracts changed paths and sends them to the mapping engine. Using JGit provided several advantages: Better portability across environmentsEasier integration with Spring BootReduced dependency on shell scriptingCleaner CI pipeline implementation One challenge encountered during implementation was ensuring full Git history availability inside GitHub Actions runners. Shallow clones occasionally caused incorrect branch comparisons, so complete fetch depth was required for accurate analysis. Rule-Based Impact Mapping After identifying changed files, the framework maps those files to relevant Karate execution tags. Example mapping rules: Changed PathKarate Tag/payments/@payments/transactions/@transactions/transfer/@transfers/auth/@regressionpom.xml@regression Example implementation: Java if (path.contains("/payments/")) { tags.add("@payments"); } if (path.contains("/transactions/")) { tags.add("@transactions"); } The project intentionally uses deterministic rule-based mapping instead of advanced dependency graph analysis or machine learning models. The primary reasons were: SimplicityPredictabilityEasier debuggingFaster implementationLower maintenance overhead Although more sophisticated impact-analysis systems exist, lightweight rule-based mapping was sufficient to demonstrate measurable CI optimization in this project. Dynamic Karate Test Execution Once impacted tags are generated, the automation repository dynamically executes only the relevant Karate tests. Example command: Shell mvn test -Dkarate.options="--tags @payments,@transactions" The Karate framework worked particularly well for this implementation because feature files were already organized by business capability. Example structure: Gherkin features/ ├── payments.feature ├── transactions.feature ├── transfers.feature └── smoke.feature This made selective execution straightforward without requiring custom runners or additional orchestration frameworks. Safe Fallback Mechanism One important concern with selective execution is the possibility of hidden dependencies. A code change may indirectly impact areas not captured by simple path-based rules. To reduce this risk, the framework includes a fallback strategy. If no impacted tags are identified, the pipeline automatically executes baseline smoke tests. Example: Shell @smoke This ensured that critical application validation still occurred even when no direct impact mapping was detected. The fallback mechanism helped balance optimization with CI reliability. GitHub Actions Integration GitHub Actions was used to orchestrate the complete workflow. Example workflow steps: YAML - name: Start Spring Boot Service run: mvn spring-boot:run & - name: Wait for API Readiness run: | curl --retry 10 \ --retry-delay 5 \ http://localhost:8080/actuator/health One practical issue encountered during implementation was synchronization between application startup and API invocation. Without readiness checks, the workflow occasionally attempted to call the API before the Spring Boot application was fully initialized. Adding retry-based health checks improved workflow stability significantly. Execution Metrics To evaluate the effectiveness of the approach, the framework generates execution metrics after each run. Example output: JSON { "impacted_tags": "@payments,@transactions", "scenarios_executed": 6, "scenarios_skipped": 12, "test_reduction_rate": "66%", "timing_metrics": { "total_workflow_seconds": 52, "isolated_test_seconds": 18, "api_overhead_seconds": 6 } } In sample execution scenarios from this project, the framework reduced executed tests by approximately 60–70% depending on the scope of code changes. Localized feature updates benefited the most, while shared dependency changes still triggered broader regression execution. Although these results came from a personal engineering project rather than a production enterprise system, the experiment demonstrated how lightweight impact-aware execution can improve CI efficiency. Limitations and Future Improvements The project also exposed several limitations. Manual Mapping Maintenance Rule-based mappings require periodic updates as repositories evolve. Hidden Dependency Risks Indirect service dependencies may not always be detected through simple path matching. Git Comparison Accuracy Accurate impact analysis depends heavily on proper branch comparison and repository history availability. Future improvements could include: dependency graph analysiscode coverage–based impact detectionhistorical test-failure analysisML-assisted impact predictionmulti-module dependency propagation These enhancements could improve precision while preserving the lightweight nature of the framework. The complete implementation for this project, including the Spring Boot impact-analysis service and GitHub Actions workflow, is available on GitHub. Source Code: Dev repository: https://github.com/raakeshdev20/fintech-impact-servicesAutomation repository:https://github.com/raakeshdev20/karate-change-impact-test Conclusion This project explored how Git diff analysis and selective test execution can help optimize CI/CD pipelines without requiring complex external tooling. By combining the following, the framework demonstrated a practical approach to reducing unnecessary regression execution for localized changes. JGit-based change detectiondeterministic impact mappingdynamic Karate executionGitHub Actions orchestration While the implementation is intentionally lightweight, the experiment highlights how impact-aware testing strategies can improve feedback cycles, reduce redundant execution, and make CI pipelines more efficient as systems continue to scale.

Onboarding a new table into row-level security should be four lines of metadata. Not two new objects, a code review, and a platform-team ticket. This post describes a tag-driven attribute-based access control (ABAC) pattern built on Databricks Unity Catalog primitives that achieves the objective of one UDF per filter shape, one policy per shape, and a single control table that drives all per-group authorization logic. I work as a solutions architect with large enterprises running hundreds of tables across multiple regions, product lines, and source systems, where row-level security follows a pattern: Group A sees records from System X. Group B sees the regions China and India. Group C sees plant key 333. Group D combines two source systems. Group E sees everything except specific values. The domain (finance, healthcare, etc.) doesn't matter. The pattern remains the same. A traditional implementation looks like this: One row-filter UDF per tableOne row-filter policy per (table and group) combinationA SQL query layer that joins every table to an identity mapping table Every new table required writing two new objects (UDF + policy) and updating every existing group definition. Every new group required touching every UDF. Onboarding a new product line meant rebuilding the whole machine. Hundreds of tables × dozens of groups = a maintenance nightmare. Instead, what they needed was a pattern where: A new table joins the RLS scheme with only metadata changes — no new UDFs, no new policiesA new business group is just a data write — no DDL, no code reviewMisconfigured rules fail closed, not open This post describes the four-layer pattern we landed on. It's built entirely on Unity Catalog ABAC primitives (governed tags, row-filter policies, attribute-based binding) and a single control table that drives per-group filter logic. The Four Layers Each layer does exactly one thing and one thing only: Layer 1: Tags are declarative metadata. A table-level tag declares which shape a table is. A column-level tag tells the row filter which physical column corresponds to which logical attribute. All tags do is describe the data. They facilitate the action for the next layers.Layer 2: UDF is the decision logic. Given a row's attribute values, it returns a boolean answer of TRUE or FALSE for current_user(). It doesn't know anything about which table it's filtering; it just answers "is this row visible?"Layer 3: Policy is the binding. It says, "for tables tagged with shape X, call UDF Y with these columns." It uses tag-matching expressions so it auto-attaches to new tables as they're tagged. This is what enables us to avoid per-table DDL. Layer 4: Row-Level Security is what the customer experiences. Their SQL doesn't change; filtered rows just come back. Layer 1 + 2: Tags Describe, UDFs Decide Two governed tags do all the work. A rls_tag on the table says "this is a table_1_filter shape." An rls_attr tag on each column says "this column is the src_sys_cd attribute" or "this column is the region attribute." The column tag is the load-bearing piece — it lets you have a column literally named ws_region_cd and still have the policy treat it as the logical region attribute. Physical naming is decoupled from policy semantics. One UDF per table shape. A "shape" is a set of filterable columns. In this customer's setup, there are two shapes: Table 1 shape: (src_sys_cd, plant_key, region) — three attributes. Maybe it's best to call this shape a combination of the keys. For example, all the tables that have src_sys_cd + plant_key+region fall under this shape.Table 2 shape: (src_sys_cd, order_key) — two attributes Each shape has its own UDF (rf_table_1, rf_table_2). The UDF signature takes one parameter per filterable column. The body joins a user_group_control_table (one row per group rule) to a user_group_membership mapping (or, in production, is_account_group_member()) and applies BOOL_OR across the rules — a union grant. The key property of this UDF design: it doesn't know which table is calling it. It just answers, given attribute values, "does current_user() get this row?" That's what lets the same UDF serve many tables of the same shape. Layer 3: The Policy that ties it all together The policy is the only object in the system that knows about both tags and UDFs. Its four clauses each answer a separate question: clausequestion it answers `ON SCHEMA …` Where does this policy live? (Schema-scoped — broad reach.) `WHEN has_tag_value('rls_tag', '…')` Which tables should it attach to? Anything tagged with the right shape. `MATCH COLUMNS … has_tag_value('rls_attr', '…')` Inside each table, which physical column maps to which logical attribute? `ROW FILTER … USING COLUMNS (…)` Which UDF to call, and in what argument order? The auto-attachment behavior is what makes the pattern scale. The policy doesn't enumerate tables — it matches them by tag. Tag a new table tomorrow, and the policy applies to it on the next query. Zero policy edits. Query Time: What Actually Happens When a user runs SELECT * FROM table_1, here's what Unity Catalog does behind the scenes: The planner sees the table and looks up policies attached to its schema. It finds rls_policy_t1.The policy's `WHEN` clause checks the table tag. Does table_1 have rls_tag=table_1_filter? Yes → the policy attaches. (If no, the policy is skipped for this table.)The policy's `MATCH COLUMNS` resolves attributes. For each logical attribute name, it scans column tags to find the physical column with that role: src_sys_cd → physical column src_sys_cd; plant_key → physical column plant_key; region → physical column region.The query is rewritten to append WHERE rf_table_1(src_sys_cd, plant_key, region) = TRUE. The customer's original SQL is unchanged.The UDF runs per row. It joins the control table to membership for current_user(), evaluates each rule, and BOOL_ORs the results — TRUE if any rule grants the row, FALSE otherwise.The engine emits only the `TRUE` rows. The customer sees only their authorized subset. They never see the UDF call or the policy mechanics. The whole thing is transparent to the application — same SQL, filtered result. The Scale Payoff The reason to build the pattern this way only becomes obvious when you onboard the second, third, and hundredth table. Adding a New Table to an Existing Shape A new dimension table arrives that fits the Table 1 shape. The work to bring it under RLS: Stepeffort shape Create the table (customer's normal DDL) — Tag the table with the shape 1 line of DDL Tag the columns with their logical roles 3 lines of DDL Update the UDF? None Update the policy? None Update the control table? **None** (existing groups apply automatically through their existing rules) Four ALTER lines. That's the whole onboarding cost. Adding a New Business Group A new business group needs access to a specific slice of the data: stepeffort `INSERT` one row into `user_group_control_table` 1 INSERT Add users to the AD group (outside Databricks) — Update the UDF? None Update the policy? None Update any table tags? None One INSERT. Adding a new group is a data write, not DDL — which means operations teams can self-serve through their normal change-management process, without code review or platform-team involvement. GitHub repo: https://github.com/vbablue/databricks-abac-rls-demo/tree/main

My converter has a test suite. It was green. Every example I could think of went in, the right pytest file came out, and I shipped it to PyPI as 1.0. Weeks later it sits there marked Production/Stable. Then I pointed a fuzzer at it, and it stopped being so quiet. The Blind Spot in Example Tests Unit tests check the inputs you thought of. That is their whole nature: you sit down, you imagine how the code gets used, you write an example for each case. The cases you imagine tend to be the cases the code already handles, because you wrote the code and the test in the same afternoon, with the same blind spots. A converter makes that worse. postman2pytest takes a Postman collection, a JSON file, and turns it into a runnable pytest suite. The input is not something I control. It is a file a stranger exports from a tool, edits by hand, truncates over a flaky download, or generates from a script that had a bug. The real test suite for a parser is the set of files you did not write. So I stopped writing examples and let a machine write them for me. Letting Hypothesis Do the Imagining Hypothesis (hypothesis.readthedocs.io) is property-based testing for Python. Instead of one example, you describe the shape of the input and assert a property that must hold for all of them. It then generates hundreds of cases, including the mean ones you would never type out. For a parser, the property is simple to state. Feed it any well-formed JSON, whatever its shape. It should do exactly one of two things: return a list of parsed tests, or raise a clear error that tells the user what is wrong. It must never fall over with a raw stack trace. JSON import json from hypothesis import given, strategies as st json_values = st.recursive( st.none() | st.booleans() | st.integers() | st.text(), lambda children: st.lists(children) | st.dictionaries(st.text(), children), ) @given(value=json_values) def test_parser_is_graceful_on_any_json(value, tmp_path): path = tmp_path / "in.json" path.write_text(json.dumps(value)) try: result = parse_collection(path) except ValueError: return # the one allowed failure: a clear, typed error assert isinstance(result, list) Then I ran it. What It Found The parser handled every real Postman collection the same as before. The fuzzer never touched valid input. It broke my handling of garbage, in five distinct shapes. A collection whose top level was a list instead of an object. A collection whose info block was a string. An item that was a bare number where the code expected a dict. A folder nested deep enough that the recursion bottomed out. And a payload so deeply nested that json.loads itself blew the stack before my code ran at all. In each case the user would have seen an AttributeError, a TypeError, or a RecursionError, with an internal stack trace and no idea what to fix. None of those is a useful message. The tool knew the input was wrong. It just had no manners about saying so. What sold me on the method was the shrinking. When Hypothesis finds a failure, it does not hand you the random 4KB blob that happened to trip the code. It shrinks the case to the smallest input that still fails, so the report is a two-line collection, not a wall of generated noise. That is the difference between a bug you can read and a bug you file under "investigate later". Each of the five failures arrived as a minimal reproducer I could paste straight into a regression test, which is how plugging them stayed a one-afternoon job rather than a week of bisecting. The Fix Is a Contract, Not a Patch Plugging the five holes was the boring part. The contract underneath them was the real work: valid collection in, pytest suite out; anything else, one clear ValueError that names the problem. Degrade, do not crash. That turned into a few isinstance checks at the boundaries, a depth limit on folder recursion, and wrapping the top-level json.loads so a stack-busting document becomes a readable "this file is nested too deeply to parse" instead of a traceback. The same review on my other parser, secure-log2test, which reads Kibana log exports, turned up the same family of gaps in the same afternoon. Same shape of bug, same shape of fix. What I Actually Learned "Production/Stable" is a statement about the API, not about the input. It never promised to survive a corrupt file, and I had let people assume it did. And finding bugs with a fuzzer is the fuzzer doing its job. Point Hypothesis at a parser and it finds nothing? You probably did not let it generate anything nasty enough. The bugs are the evidence the method works. If your tool reads a file someone else made, your unit tests are a comfort blanket. Spend twenty lines on a property test before you trust them. The input you did not write is the one that finds you in production. postman2pytest and secure-log2test are MIT-licensed on PyPI. The fuzz tests are in each repo if you want a template to copy. This article was originally published on dev.to.

Row-level security in PostgreSQL is one of the more useful features for multi-tenant applications. The idea is straightforward: define a policy on a table that tells PostgreSQL which rows a given user is allowed to see or modify, and the database engine enforces it on every query, regardless of which application code issued the request. The trouble comes when your policies form a cycle. This is more common than it sounds, and it produces one of the more confusing failure modes in PostgreSQL: a query that should return data returns nothing, with no error. This article walks through how circular RLS dependencies arise, why they silently eat your data, and how to break the cycle using SECURITY DEFINER functions. How the Circular Dependency Happens Consider a simple multi-tenant schema. You have a properties table and a property_members table that tracks which users have access to which properties: SQL CREATE TABLE public.properties ( id uuid PRIMARY KEY DEFAULT gen_random_uuid(), name text NOT NULL, slug text UNIQUE NOT NULL ); ALTER TABLE public.properties ENABLE ROW LEVEL SECURITY; CREATE TABLE public.property_members ( id uuid PRIMARY KEY DEFAULT gen_random_uuid(), property_id uuid NOT NULL REFERENCES public.properties(id) ON DELETE CASCADE, user_id uuid NOT NULL REFERENCES auth.users(id) ON DELETE CASCADE, role text NOT NULL DEFAULT 'member', accepted_at timestamptz, UNIQUE(property_id, user_id) ); ALTER TABLE public.property_members ENABLE ROW LEVEL SECURITY; Now you write your policies. A user should be able to see a property if they are an accepted member of it: SQL CREATE POLICY "properties_select_members" ON public.properties FOR SELECT TO authenticated USING ( EXISTS ( SELECT 1 FROM property_members WHERE property_id = properties.id AND user_id = auth.uid() AND accepted_at IS NOT NULL ) ); And a user should be able to see other members of a property if they are also a member: SQL CREATE POLICY "property_members_select_comembers" ON public.property_members FOR SELECT TO authenticated USING ( EXISTS ( SELECT 1 FROM property_members pm2 WHERE pm2.property_id = property_members.property_id AND pm2.user_id = auth.uid() AND pm2.accepted_at IS NOT NULL ) ); This looks reasonable. In fact, it compiles without error. Then you run a query, and it returns zero rows. Why This Silently Returns Nothing Here is the execution path PostgreSQL follows when an authenticated user queries properties: Apply properties_select_members. This requires checking property_members. To read property_members, apply property_members_select_comembers. This requires checking property_members again. To check property_members in step 3, apply property_members_select_comembers. This requires checking property_members again. PostgreSQL does not raise an error here. Instead, when it detects the recursive RLS evaluation, it short-circuits and evaluates the recursive reference as returning no rows. The result is that the policy conditions that depend on property_members always see an empty set, every EXISTS(...) check returns false, and no rows are visible. This is consistent with how PostgreSQL handles RLS recursion to prevent infinite loops, but the silent behavior makes it genuinely difficult to diagnose. You add your membership record, you enable RLS, you query your table, and you get nothing. No error message. No warning. Just an empty result. The Fix: SECURITY DEFINER Functions The solution is to introduce a layer of indirection. Instead of having your policies query property_members directly (which triggers RLS on that table), you wrap the membership check in a function that runs with elevated privileges and bypasses RLS entirely. SQL CREATE OR REPLACE FUNCTION public.is_property_member(p_property_id uuid, p_user_id uuid) RETURNS boolean LANGUAGE sql STABLE SECURITY DEFINER SET search_path = public AS $$ SELECT EXISTS ( SELECT 1 FROM property_members WHERE property_id = p_property_id AND user_id = p_user_id AND accepted_at IS NOT NULL ); $$; The SECURITY DEFINER attribute tells PostgreSQL to run the function as the user who defined it (typically a superuser or the role that owns the schema), not as the calling user. Inside the function body, RLS on property_members is not applied, because the function owner has full access. You can add role-specific variants for the same pattern: SQL CREATE OR REPLACE FUNCTION public.is_property_admin(p_property_id uuid, p_user_id uuid) RETURNS boolean LANGUAGE sql STABLE SECURITY DEFINER SET search_path = public AS $$ SELECT EXISTS ( SELECT 1 FROM property_members WHERE property_id = p_property_id AND user_id = p_user_id AND role IN ('owner', 'admin') AND accepted_at IS NOT NULL ); $$; Now rewrite your policies to call the function instead of querying the table directly: SQL DROP POLICY IF EXISTS "properties_select_members" ON public.properties; DROP POLICY IF EXISTS "property_members_select_comembers" ON public.property_members; CREATE POLICY "properties_select_members" ON public.properties FOR SELECT TO authenticated USING ( public.is_property_member(id, auth.uid()) ); CREATE POLICY "property_members_select_comembers" ON public.property_members FOR SELECT TO authenticated USING ( public.is_property_member(property_id, auth.uid()) ); The cycle is broken. properties policies call is_property_member. property_members policies also call is_property_member. But is_property_member is a function that executes with SECURITY DEFINER privileges, so when PostgreSQL evaluates it, it does not apply RLS to the property_members table inside the function body. There is no loop. Applying the Pattern Consistently Once you have your helper functions in place, the pattern composes cleanly across your entire schema. Every table in your multi-tenant application can reference the same small set of helper functions in its policies: SQL -- Bookings: members can read, admins can write CREATE POLICY "bookings_select" ON public.bookings FOR SELECT TO authenticated USING ( public.is_property_member(property_id, auth.uid()) ); CREATE POLICY "bookings_update" ON public.bookings FOR UPDATE TO authenticated USING ( public.is_property_admin(property_id, auth.uid()) ); -- Board posts: members can read and post, owner or author can delete CREATE POLICY "board_posts_delete" ON public.board_posts FOR DELETE TO authenticated USING ( user_id = auth.uid() OR public.is_property_admin(property_id, auth.uid()) ); The policies stay readable and short. The access logic lives in one place. When your membership rules change (say, you add a new role), you update the functions rather than hunting through every policy across every table. A Few Things to Keep in Mind SET search_path = public in the function definition is not optional. Without it, a malicious user could create objects in a schema earlier in the search path and potentially redirect function calls. PostgreSQL's own documentation recommends this for any SECURITY DEFINER function. Marking functions as STABLE (rather than VOLATILE, the default) lets PostgreSQL cache the result within a single query. A single SELECT that reads many rows from properties will call is_property_member once per row, and the STABLE declaration allows the planner to optimize those calls. If your membership table changes mid-transaction, this is worth thinking about, but for most access-control use cases, STABLE is the right choice. Finally, grant EXECUTE on these functions only to the roles that need them. For a Supabase project, that typically means the authenticated role. The function runs as the owner, but you still control who can call it. SQL GRANT EXECUTE ON FUNCTION public.is_property_member(uuid, uuid) TO authenticated; GRANT EXECUTE ON FUNCTION public.is_property_admin(uuid, uuid) TO authenticated; The circular dependency problem is a good example of why it pays to understand what your framework is doing underneath. Supabase makes RLS easy to enable. It does not protect you from cycles in the policies you write. But once you understand the pattern, the fix is clean, and it scales to a large schema without adding complexity.

When an enterprise asks, "Is your agent platform secure?", the question is almost always a bundle of two distinct architectural concerns: Tool layer: Can the agent only call the tools we approved? Are the tool inputs and outputs validated? Are credentials kept out of the LLM's context? Are calls audited?Sandbox layer: When a tool runs code, browses the web, or shells out — is that execution isolated from the host? Can it reach internal networks? Can it write outside its working directory? These look adjacent, but they fail differently. A tool layer fails when an agent calls something it shouldn't have access to — fixable by tightening the tool registry. A sandbox layer fails when an approved tool gets compromised mid-execution (e.g., a Chromium zero-day exploited via a malicious page) — fixable only by reducing what the execution environment can reach. In building helmdeck — an open-source MCP server and pack-based agent infrastructure — our thesis has been that the immediate bottleneck for production-grade agents is the tool layer. We shipped schema-validated Capability Packs, an MCP server that exposes them uniformly, and a vault that injects credentials into outbound HTTP without the agent ever seeing them. But for true enterprise hardening, the tool layer isn't enough. You need a sandbox layer that provides hardware isolation. This is why we designed a composed architecture using NVIDIA OpenShell to handle the execution environment. The Credential Split The most common concern when composing two security layers is a tug-of-war over credentials. If both the agent platform and the sandbox engine handle secrets, who owns what? After mapping the integration between helmdeck and OpenShell, the responsibilities proved entirely non-overlapping: Credential TypeOwnerMechanismInference API keys (Anthropic, OpenAI)Sandbox (OpenShell)Provider-injected environment variables at agent-sandbox startKubernetes service accounts, cloud credentialsSandbox (OpenShell)Provider-injected at sandbox provisioningSaaS PATs (GitHub, Stripe, Notion)Tool Layer (helmdeck)AES-256-GCM vault; ${vault:NAME} placeholder substitution at pack-dispatch timePack output artifact signingTool Layer (helmdeck)Existing artifact store The sandbox layer injects into the process environment. The tool layer injects into the outbound HTTP request body. The layers never collide because they intercept at different points in the request lifecycle. What Changes When You Compose Them Today, agents call helmdeck's 39 packs via MCP. The packs run in Docker containers with seccomp profiles and dropped capabilities. An egress guard rejects outbound URLs against a blocklist. That is solid for most operators. The composed architecture changes one specific thing: helmdeck's SessionRuntime interface — the seam between the pack engine and execution backends — gains a third backend. Instead of shelling out to the Docker SDK, the pack engine calls OpenShell's Gateway API, which provisions the sidecar in a MicroVM with a pack-family-specific OPA policy attached. The pack code doesn't change. The MCP surface doesn't change. The agent doesn't know. But the enterprise reviewing the architecture notices three things: Dedicated kernel isolation: A browser sidecar runs in a dedicated kernel. A zero-day exploit cannot escape to the host because the libkrun MicroVM boundary is a hardware-virtualization line, not a namespace.L7 policy per pack family: A python.run sidecar can be policy-restricted to deny any outbound HTTP — even to internal services — while a browser.screenshot_url sidecar can be allowed to reach exactly the user-supplied target.Landlock filesystem enforcement: Even if the LLM generates code attempting to read /etc/passwd, the kernel returns EACCES before the process can act. Why This Matters to You If you are designing an agentic platform for enterprise deployment, do not attempt to merge the tool layer and the sandbox layer into a single monolithic API. The abstractions will leak. A two-stack story is more honest about what each layer does. An enterprise reviewing a composed architecture can audit each layer independently: they can read the sandbox's policy YAML to verify network isolation, and read the tool layer's pack schemas to verify credential injection. That decoupling is a security property of the architecture, not just an aesthetic preference. If you are an architect reviewing agent infrastructure for production, we are actively prioritizing the next phases of this integration based on community needs. We need to know which pack family worries you most (browser, Python, vision) and what you are isolating against (Chromium zero-days, internal SSRF). You can shape the roadmap by commenting on issue #193, or help us build the deterministic tool layer by contributing SaaS API wrappers following our contribution guide. Note: NVIDIA OpenShell is currently in alpha. The composed architecture described here is our post-v1.0 roadmap for enterprise hardening, ensuring the base tool layer is stable before binding it to an alpha contract.

A few weeks ago, I disabled key authentication on an Azure storage account we used for Terraform state management. It was one of the key security recommendations in Microsoft Defender for Cloud. It made sense to use RBAC-only permissions, enforce PIM approvals for the Infrastructure team, and avoid storing static credentials in config files, where leaks are possible. This is exactly the kind of control you want for state files, which contain the keys to your entire cloud environment. But I missed an important line in the azurerm backend config. If use_azuread_auth = true is not explicitly set, the provider uses key-based authentication by default. Since key authentication had been disabled, terraform init failed and the pipeline broke. The actual fix was easy, but finding what was wrong, not so much. JSON terraform { backend "azurerm" { resource_group_name = "rg-tfstate-prod" storage_account_name = "sttfstateprod001" container_name = "tfstate" key = "platform/prod.tfstate" use_azuread_auth = true } } This is not the kind of detail every engineer should have to remember in every repository. It belongs in the module. That is the gap I am talking about: the security decision was correct, but the delivery path still allowed the wrong configuration. The same pattern shows up elsewhere: storage accounts left open, IAM roles with excessive permissions, credentials committed to repositories, diagnostic settings missed, or Terraform modules that still allow insecure defaults to slip through. Security Enters Too Late, and Everyone Pays For It There is a common pattern: a developer builds a feature, security reviews it and flags something, the developer reworks it, the release gets delayed, and someone gets the blame. The cycle repeats until everyone is frustrated. The cost side of this doesn't get enough attention. Catching a vulnerability while you're still writing the code is a relatively quick fix. Finding the same issue in production is a different situation entirely: incident response kicks in, there may be regulatory questions to answer, and the reputational impact is difficult to measure. The further right security sits in the delivery process, the heavier each failure gets. Most teams are inadvertently set up to find problems at the point where they cost the most. What Shifting Left Actually Looks Like People toss around 'shift left' so much that it’s lost its punch. Here’s what it actually looks like in practice: Plan: Include threat modeling in sprint planning and spend 30 minutes on it rather than managing it in a separate process or document.Code: Use IDE plugins to flag insecure patterns in real time while you code and pre-commit hooks to run secrets detection before committing the code. The developer finds out immediately, not weeks later in a review.Build: Run SAST on every commit to catch injection risks, insecure cryptography, and hardcoded secrets/credentials before code is deployed to a shared environment.Test: Let DAST probe the application in staging as an attacker would. SAST reads code, and DAST attacks the running system. One finds what the other misses.Deploy: Scan your IaC before applying changes, check container images for CVEs, and use OPA policy gates to verify signing, permissions, and network policies before anything reaches production. Running security through each of these stages means issues come up when they are still manageable, rather than after they have already caused damage. Installing Tools Is Not a Program How DevSecOps failures look in practice: Tools like Checkov and Semgrep are configured in the pipeline, and by next month, the developers have written suppression rules for the findings so the feature can be shipped. The tools keep running, but no one is checking their outputs. Three things matter more than which tools you choose: Tuning: SAST generates false positives because it doesn’t know what’s happening at runtime. Run a co-triage session with a developer and a security engineer; work through the first 50 findings; fix the problematic rules; or write a justified suppression. After a couple of sessions, developers start trusting the output because it becomes more accurate and actionable.Signal engineering: Let critical and high CVEs block the pipeline immediately, while medium and low go to a dashboard with remediation SLAs. Developers will find ways to bypass the findings instead of fixing them if you block the commit for every medium, which will end up in a bigger mess than you started with. Ownership: Send findings straight to the person who can fix them, and give them enough info to act. A centralized security queue is where urgency goes to die. The Terraform backend scenario I opened is the exact example. The security decision to use RBAC only and disable key authentication was absolutely the right one. But here’s the catch: use_azuread_auth = true was not enforced during provisioning. If a hardened module had that flag set by default, that misconfiguration simply couldn’t have happened. That’s the real difference between having a security policy and actually building a security platform. The Platform Team Is the Structural Answer Adding more process to a structural problem doesn’t fix it. What’s required is a different model entirely. A real platform team treats the internal platform as a product, with engineers as its customers. Their job is to make secure, compliant delivery the path of least resistance: golden path templates, a shared CI/CD toolchain, secrets management, and self-service provisioning, all built with guardrails from the start. When teams repeatedly provision similar workloads — containerized APIs, data pipelines, Kafka consumers – the same security configuration decisions recur. Golden path templates address this by embedding those decisions up front. Encryption at rest is already configured, IAM permissions are scoped to what the workload actually needs, logging and network policies are in place, and the backend authentication flags in the Terraform modules are set correctly from the start. A developer selects the right template, fills in the required fields, and provisions. The repository they get back already has security gates running in the pipeline. There is no separate step to secure it afterward. Figure 1: A secure golden path platform embeds security controls into the default delivery path. This is what removes the need for individuals to get every detail right under pressure. In my experience, even when you know the correct configuration, you can still miss something in the moment. The platform handles that by making the secure option the default. In many organizations, platform teams work best when they sit within Engineering rather than reporting directly into the CISO function. If they are seen mainly as a compliance function, product teams may treat them as another gate to work around. Security should define the policies and risk boundaries; Engineering should build and operate the platform that makes those policies usable. Where to Start: Sequence the Platform, Don’t Boil the Ocean The most common mistake is trying to implement everything at once. Every scanner, every policy gate, every access control change lands in one big push. It creates noise before it creates trust, and teams lose confidence in the tooling before it has a chance to prove its value. Sequence it instead. Months 0 to 3: secrets scanning as a pre-commit hook, SAST in CI, IaC scanning before Terraform apply, and a security champions program with one dedicated developer per squad. Low friction, immediate signal, nothing that unnecessarily blocks delivery. Months 3 to 6: DAST in staging, container image scanning, OPA policy gates, and SCA on every build. At this point, the platform needs to make a clear distinction: critical and high findings stop the pipeline; everything else goes into a remediation backlog with defined ownership and SLAs. Months 6 to 12 mark the point at which platform security matures into deeper controls: workload identity, privileged access management, zero-trust network policies, and a real-time compliance dashboard. Never trust, always verify, and assume breach stop being principles on a slide and become defaults in the environment. Don't wait for a fully staffed platform team or executive sponsorship. The Terraform backend fix I mentioned earlier eventually became a hardened provisioning module used by the wider infrastructure team, turning a one-off incident into a reusable secure pattern. No one needs to remember the flag because the platform handles it automatically. That's what security as a platform property actually looks like. Not a gate at the end. A system that makes the right thing the easy thing, by default, every time.

A pipeline can finish successfully, schemas can match, and null checks can pass, while the business is still looking at yesterday's truth. Freshness deserves its own quality model. The pipeline succeeded. The schema matched. Required fields were present, ranges were sane, and the dashboard refreshed on schedule. Every quality check was green. The number on the screen was still wrong, because it was built from data that stopped updating two days ago and nobody noticed. This failure is common, and it is quiet. Most data quality programs are built to answer one question: is this data valid? They check for nulls, types, ranges, uniqueness, and referential integrity. Those checks are necessary, and they catch a real class of problems. They also share a blind spot. A record can be perfectly valid and completely stale. Validity is about whether the data is well-formed. Freshness is about whether it is current enough to trust. They are different properties, and a pipeline that measures only the first will keep serving old truth with a green status next to it. Freshness Is Not Correctness Structural quality asks whether a row is shaped correctly. Freshness asks whether the row should still be believed given how much time has passed. A transaction record from Tuesday is structurally identical whether it is read on Wednesday or three weeks later. Its validity never changes. Its usefulness for a decision that assumes current data changes completely. This is why freshness belongs in the quality model rather than in a separate operations dashboard. Most quality dimensions that teams already track, such as completeness, accuracy, consistency, uniqueness, and validity, describe the data as it sits. Freshness describes the data relative to now. Leaving it out of the quality model means the platform can report high quality on data that is too old to act on, which is not a contradiction the business will find reassuring. The concept that ties this together is the freshness gap: the distance between when an event actually happened and when a consumer can first see it. Structural checks never measure this gap, because both a fresh record and a stale one are equally valid. The gap is the part of quality that only time reveals. Why Pipelines Hide Staleness The reason staleness stays hidden is that pipeline success and data freshness measure different things, and teams routinely treat the first as a proxy for the second. A job can complete successfully while delivering nothing new. Common paths to a green pipeline over stale data include: The source sent no new files. The job ran, found the same input as yesterday, processed it correctly, and reported success. Nothing failed. Nothing updated either.Only some partitions arrived. The pipeline loaded the partitions it received and completed. The missing region or date range is not an error to a job that was never told those partitions were mandatory.A late-arriving upstream delayed the real data. The scheduled run fired on time against data that had not landed yet, so it processed an incomplete or old snapshot and finished cleanly.The dashboard cached a stale table. The pipeline updated the table, but the serving layer or BI tool returned a cached result, so the freshest data never reached the screen.A backfill overwrote current data with an older snapshot. A correction job ran a historical range and, through a scope error, replaced newer records with older ones. Every row is valid. The table went backward in time. None of these trip a structural check, because in every case the data that is present is well-formed. The problem is not the shape of what arrived. It is the age of what arrived, and whether anything arrived at all. Freshness Needs Its Own Contract Freshness cannot be governed by a single global rule, because different datasets have different tolerances. A five-minute delay is a crisis for fraud detection and irrelevant for a historical archive. Tying freshness to the pipeline schedule is the common shortcut, and it is wrong, because the schedule describes when the job runs, not when the data is expected to be current for a specific use. The fix is to define freshness expectations per dataset, anchored to the business decision the data supports rather than to the cadence of the job that produces it. DatasetFreshness expectationWhy it mattersFraud eventsUnder 5 minutesDecisions are made in real timeDaily balancesBy 7 AM ETMorning reporting depends on itMonthly finance closeBy business day 3Tied to the reporting cycleHistorical archive24 to 48 hoursLow operational urgency Each expectation is a contract. It states what current means for that dataset, and it gives monitoring something concrete to check against. Without it, freshness is a matter of opinion, and the first time anyone forms an opinion is usually after a stale number has already reached a decision. Measuring Freshness Correctly The technical heart of freshness is that there is no single timestamp called "the time." A record carries several distinct times, and confusing them is how freshness monitoring gives false comfort. Four matter: Figure 1. A record carries four distinct times. Structural checks see only the published value. The freshness gap, which is event time to publish time, is the delay no structural check measures. The relevant times to consider are: Event time: When the thing actually happened in the source system. A purchase was made, a sensor fired, an address changed.Ingestion time: When the record entered the platform. The moment it landed in the queue or the raw zone.Processing time: When the transformation ran over it. The point where it was cleaned, joined, and shaped.Publish time: When it became queryable by a consumer. The moment the serving table or dashboard could return it. The freshness gap that matters to the business is publish time minus event time, because that is the total delay between reality and what a consumer can see. A pipeline that measures only processing time, "the job ran at 06:00," reports a healthy number while the events it processed are hours old, because the delay lived upstream, before ingestion, where the job never looked. Measuring the wrong timestamp is worse than not measuring, because it produces a confident freshness metric that is disconnected from reality. A dataset can show a two-minute processing lag and a six-hour event-to-publish gap at the same time. The first number looks great on a status page. The second is the one the business feels. What Freshness Failures Look Like In practice, freshness failures usually look healthy from the outside. The job finishes, the schema matches, and the records pass validation. The failure lives in the time dimension: no new source data arrived, only some partitions landed, a dashboard served a stale cache, or a backfill moved the table backward. Structural validation sees rows that are well-formed. Freshness monitoring sees that the published dataset no longer reflects the current state of the business. Passing one tells you nothing about the other. A Practical Freshness Pattern Making freshness a first-class quality dimension does not require a new platform. It requires treating the age of data as something the pipeline measures, records, and alerts on, the same way it already treats nulls and types. A workable pattern: Carry timestamps through the pipeline. Preserve event time from the source, and stamp ingestion, processing, and publish times as the record moves. The freshness gap cannot be measured if the timestamps needed to compute it were discarded early.Record freshness in a small audit table. For each dataset and run, store the maximum event time published and the publish time itself. This gives a queryable history of how current each dataset actually was, run over run.Attach a freshness SLA to each dataset. Encode the per-dataset expectation from the contract above as a checked threshold, not a comment in a runbook.Alert on the gap, not on job status. Trigger when publish-time-minus-event-time crosses the dataset's threshold, independent of whether the job reported success. This is the alert that catches the source-sent-nothing case, which job monitoring cannot see.Make freshness visible downstream. Surface the last known freshness next to the data itself, so a consumer can see that a dashboard is running on data from two days ago before they act on it. Track compliance as a percentage over time rather than as a pass or fail on a single run. A dataset that met its freshness SLA 99 percent of the time last month, and is trending down, is a more honest signal than a single green check, and it is the number a business owner can actually reason about. A minimal audit table makes this concrete. One row per dataset per run is enough to compute the gap, compare it against the SLA, and keep a history: ColumnMeaningdataset_nameDataset being monitoredrun_idPipeline run identifiermax_event_timeLatest event included in the published datapublish_timeWhen the dataset became availablefreshness_gap_minutesPublish time minus max event timesla_minutesFreshness threshold for the datasetsla_statusPass or fail for this run The gap column is the one structural checks never produce, and the status column is what the freshness alert reads rather than job success. The Green Check was Measuring the Wrong Thing Validity and freshness are independent. A pipeline can watch one perfectly and never look at the other, which is exactly how a dataset ends up well-formed, internally consistent, and two days out of date with a passing status next to it. The structural checks were doing their job. They were just never the checks that would have caught this. Freshness needs its own contract, its own timestamps, and its own alert that fires on the age of the data rather than the exit code of the job. Decide what current means for each dataset, watch the distance between event time and publish time, and put that distance in front of the people making decisions. A pipeline finishing was never the same claim as the data being fresh, and the sooner a platform stops treating the first as proof of the second, the fewer stale numbers reach a meeting.

The bug report was received as a customer complaint. An AI agent responsible for managing vendor onboarding had sent a rejection email to a supplier the company had been trying to close for three months. Nobody had authorized it. Nobody had configured it to reject vendors in that category. The agent autonomously made the decision after analyzing a compliance document and cross-referencing it with an internal policy database. By the time the complaint arrived, the reasoning chain that produced the decision had been discarded. The agent had no memory of why it did what it did. The logs showed the action but not the thought. That story is fictional in its specifics but accurate in its structure. This phenomenon represents a class of problems that teams deploying AI agents in production are encountering with increasing frequency: the agent performed an action, the output is visible, but the intermediate reasoning, including the sequence of context retrievals, model calls, tool invocations, and decisions that led to the output, is either absent, incomplete, or stored in a format that renders post hoc investigation nearly impossible. Traditional observability was not designed for systems that exhibit cognitive processes. Why Agent Observability Is Structurally Different Conventional service observability is built around a relatively stable model: a request enters a system, passes through a defined set of operations, and produces a response. The execution path may be complex, but it's deterministic and bounded. You can instrument each step, correlate the signals with a trace ID, and reconstruct exactly what happened for any given request. AI agents break this model in at least three ways. First, the execution path is not determined at design time — it emerges from the agent's reasoning. An agent deciding which tools to call, in what order, based on what it reads in a retrieved document, is making structural decisions at runtime that a static trace can't fully capture. The spans exist, but the semantic reason a particular branch was taken lives inside a model call that returned natural language, which most tracing systems treat as an opaque blob. Second, agent systems frequently involve state that persists across requests: memory stores, retrieved context, and conversation history, which means the behavior of the system at time T is partially determined by things that happened at times T-1 through T-n. Debugging a poor decision often requires reconstructing not just the current request but the accumulated state that shaped it. Most observability stacks are not built for these scenarios. Third, multi-agent systems introduce the problem of causal attribution across agent boundaries. When Agent A passes a task to Agent B, which delegates a subtask to Agent C, which calls a tool that returns erroneous data, and that incorrect data propagates back up the chain to produce a wrong output from Agent A, the causal chain is real but fragmented across three separate execution contexts. Without deliberate design, you'll have three separate traces with no shared context that links them. The Minimum Viable Agent Trace The starting point for any serious agent observability implementation is defining what the minimum viable trace looks like for a single agent execution. In practice, this means capturing five things that standard OpenTelemetry spans don't cover by default. The first is the full prompt context, not just the user message but the complete input to each model call, including the system prompt, retrieved documents, tool outputs injected into the context, and the conversation history. The information is costly to store and verbose, but you need it to understand the model's reasoning. Sampling helps here: store full prompt context for a percentage of executions, prioritizing those that result in high-stakes actions or errors. The second is the model's reasoning output before tool calls. If your agent framework supports it, capture chain-of-thought or scratchpad outputs of the model's intermediate reasoning before it decides to call a tool or produce a final answer. This is the closest thing to a stack trace for a reasoning system. Without it, you can see that a tool was called but not why. The third is a tool called "provenance" for each tool invocation, recording not just the inputs and outputs but which part of the reasoning chain triggered it. Fourth is the agent's decision points: moments where the agent chose between multiple possible actions. Fifth is cross-agent delegation context: when one agent hands off to another, the receiving agent's trace must carry a reference to the delegating agent's trace ID. Python # Minimal agent span instrumentation using OpenTelemetry from opentelemetry import trace import json tracer = trace.get_tracer('agent.core') def traced_model_call(agent_id, prompt_context, step_label): with tracer.start_as_current_span(f'agent.model_call.{step_label}') as span: span.set_attribute('agent.id', agent_id) span.set_attribute('agent.step', step_label) # Store truncated prompt for cardinality control span.set_attribute('agent.prompt_hash', hash(str(prompt_context))) span.set_attribute('agent.prompt_len', len(str(prompt_context))) # Full prompt stored separately in blob storage, keyed by trace+span ID store_prompt_context( trace_id=format(span.get_span_context().trace_id, '032x'), span_id =format(span.get_span_context().span_id, '016x'), context =prompt_context ) response = call_model(prompt_context) span.set_attribute('agent.output_len', len(response)) span.set_attribute('agent.tool_calls', extract_tool_calls(response)) return response The pattern above separates high-cardinality content (the full prompt) from the trace span itself, storing it in blob storage keyed by trace and span IDs. This keeps the tracing backend manageable while preserving the ability to retrieve full context for any specific execution. The prompt hash allows you to detect when two executions were given identical contexts, which is useful for identifying cases where the same input produced different outputs, which is a diagnostic signal in itself. Multi-Agent Correlation: The Delegation Chain Problem Here's where things got genuinely complicated in a system I was involved with: we had three agents — a planning agent, a research agent, and a writing agent that collaborated on generating reports. Each was instrumented individually and produced clean traces. But when a report came out wrong, reconstructing which agent's decision caused the problem required manually cross-referencing three separate trace trees, none of which had a shared parent. The fix was implementing what we called a "workflow ID," a UUID generated at the entry point of any multi-agent task and propagated explicitly to every agent that participated in that task, regardless of how many hops away from the origin they were. This workflow ID was added as a span attribute on every agent span and as a field in every log line produced during the task. With it, querying all spans and logs associated with a single end-to-end agent workflow became a single filter, not a manual correlation exercise. Python # Propagating workflow context across agent boundaries from dataclasses import dataclass from opentelemetry import trace, context, propagate @dataclass class AgentWorkflowContext: workflow_id: str # stable across all agents in a task parent_agent: str # which agent delegated this task delegation_depth: int # how many hops from the origin agent def delegate_to_agent(target_agent, task, wf_ctx: AgentWorkflowContext): child_ctx = AgentWorkflowContext( workflow_id = wf_ctx.workflow_id, # same ID propagates parent_agent = wf_ctx.parent_agent, delegation_depth = wf_ctx.delegation_depth + 1 ) span = trace.get_current_span() span.set_attribute('workflow.id', child_ctx.workflow_id) span.set_attribute('workflow.depth', child_ctx.delegation_depth) span.set_attribute('workflow.parent_agent', child_ctx.parent_agent) return target_agent.run(task, child_ctx) The delegation depth attribute turned out to be more useful than expected. In one debugging session, seeing that a particular tool call was happening at delegation depth 4 — four hops from the original request immediately flagged that the agent system had gone significantly deeper into a recursive subtask chain than intended. Without that attribute, the trace looked like any other tool call. Semantic Logging: What Happened vs. Why Standard logging captures what happened. For agent systems, you also need to capture the agent's stated reasoning at key decision points. This doesn't require exotic infrastructure; it requires a logging discipline that treats the model's reasoning output as a first-class log field rather than as data to be discarded after use. In practice, this means that when an agent produces a reasoning step leading to a significant action — such as calling an external tool, delegating to another agent, producing a final output, or deciding to abandon a task — the full reasoning text should be logged alongside the action. Tag it with the workflow ID, the agent ID, and a decision type label. This produces a semantic audit trail that lets you answer the question, "Why did the agent do X?" without having to reconstruct it from indirect evidence. The objection is storage cost, and it's legitimate. Reasoning outputs from LLMs are verbose. Storing them for every execution at scale is expensive. The practical answer is tiered retention: store full reasoning logs for executions that result in errors, high-stakes actions (anything that sends an external communication, modifies a record, or triggers a financial transaction), or random sampling of normal executions for baseline calibration. For the rest, store only the decision label and the action taken. This keeps costs manageable while preserving investigative capability for the cases that matter. What I'd Do Differently In hindsight, the single most important decision to make before deploying an agent in production is defining what a 'high-stakes action' means for that specific agent and ensuring those actions always produce full semantic logs regardless of cost. Initially, we did not define logging requirements; instead, we treated logging as uniform across all action types, which resulted in issues when an agent took an unexpected external action, and we lacked a reasoning log to explain it. I'd also invest earlier in a replay capability: the ability to take a logged prompt context and re-run the agent over it with a modified model or prompt configuration to verify that a fix actually changes the behavior that caused a problem. Without a replay capability, any changes you make are based on hope rather than verification. With it, you can verify that the reasoning path actually differs before deploying. When should you not build this level of observability? If you're prototyping or running an agent in a low-stakes, easily reversible context, the overhead of full semantic logging and workflow ID propagation is probably premature. Build it before you go to production with consequential actions, not after. The cost of retrofitting it once an unexplained agent decision has already caused a real problem is significantly higher than building it in from the start. Key Takeaways Standard distributed tracing captures what happened in agent systems but not why. Semantic logging of reasoning outputs at decision points is the missing layer; treat it as first-class infrastructure, not optional verbosity. Propagate a workflow ID across all agents in a multi-agent task. Without it, correlating signals across agent boundaries requires manual effort that fails under incident pressure. Separate high-cardinality prompt content from trace spans. Store the full prompt context in blob storage keyed by trace and span ID, and reference it from the span. This preserves investigative capability without bloating your tracing backend. Please define high-stakes actions prior to deployment and ensure they consistently generate complete semantic logs. The executions you most need to investigate are exactly the ones where missing reasoning context is most detrimental. Conclusion Observability for AI agents is not a solved problem. The tooling ecosystem is immature, the standards are still forming, and most teams are improvising solutions on top of infrastructure designed for deterministic services. That's not a reason to skip it; it's a reason to be deliberate about what you build, because the defaults will leave you blind at exactly the wrong moment. The deeper challenge is that agent observability isn't just a technical problem. It's also an accountability problem. When an AI agent takes a consequential action, someone needs to be able to answer the question of why, not just for debugging purposes, but for the humans affected by the decision and for the organization responsible for the system. A vendor who received a rejection email deserves a better answer than "the agent decided that." The infrastructure to produce that answer has to be designed in, not bolted on. The open question I keep returning to: as agent systems become more capable and their reasoning chains longer and more complex, at what point does the volume and opacity of their decision-making exceed our practical ability to observe and understand it? We may be building systems that are genuinely difficult to audit, not because of missing tooling but because of fundamental limits on human comprehension of long reasoning chains. What does accountability look like then?

The Hidden Cost of API Versioning Hell Continuous API evolution is non-negotiable in contemporary software development, yet maintaining backward compatibility remains an incredibly expensive and labor-intensive hurdle. Core schema mutations frequently force downstream enterprise clients into disruptive and unplanned refactoring cycles, stalling product velocity. The typical industry fix — maintaining multiple, hard-coded API routes (e.g., /v1, /v2) — inevitably results in severe codebase sprawl, fractured engineering focus, and massive technical debt for the API provider. To break this cycle, this article outlines raqs (Response Agnostic Query System): a novel, dynamic proxy architecture designed to eliminate client-side disruption entirely. By intercepting traffic and executing on-the-fly schema transformations, raqs allows legacy clients to request data against deprecated contracts while the core upstream backend remains free to evolve. The raqs Solution: A Bifurcated Architecture Running complex natural-language processing or machine-learning inference directly within a high-throughput network routing path is typically a recipe for catastrophic latency. To solve this, raqs splits the network and intelligence layers into two distinct operational planes: The Orchestration Plane (Java 21/Spring Boot): Acting as the primary ingress proxy, this layer intercepts requests, manages multi-tier cache retrieval, handles distributed synchronization, and executes structural JSON transformations. The Inference Plane (Python/FastAPI): Operating as a probabilistic fallback mechanism, this agent calculates semantic and structural relationships between schema keys only when a deterministic mapping rule is missing. Core Architectural Decision Matrix ComponentNaive/Standard Approachraqs ImplementationConcurrency ManagementOS Thread Pooling (Tomcat Defaults) Java 21 Virtual Threads (Project Loom) SynchronizationPolling / Thread.sleep() loop Redisson Distributed Locking (Pub/Sub) Caching TierSingle-node In-Memory Cache Multi-tier (Caffeine L1 + Redis L2) Semantic MappingPure Semantic Models (LLM/Dense Vector) Hybrid Ensemble (Vector + Lexical Distance) Scaling Imperatively With Java 21 Virtual Threads The Orchestration Plane must handle thousands of concurrent client requests while checking caches, holding locks, or awaiting responses from the Inference Plane. The traditional platform-thread pooling model introduces massive operating system overhead and memory footprint under heavy I/O saturation. By building on Java 21 virtual threads (Project Loom), raqs assigns a lightweight, user-mode virtual thread to every single request lifecycle. When a thread encounters an L1/L2 cache miss, it is gracefully unmounted from its underlying OS carrier thread. The carrier thread is freed to handle other active network traffic, while the suspended virtual thread waits to resume once the schema mapping becomes available. This allows us to write straightforward, blocking imperative code that scales out with the efficiency of complex reactive systems. Defeating Cache Stampedes: The "Hero Thread" Pattern A major architectural risk for dynamic proxies is the cache stampede (or thundering herd problem). If a rolling backend deployment instantly mutates 50 schema keys, a burst of 1,000 concurrent client requests will simultaneously experience an L1/L2 cache miss. Without intervention, this triggers a massive wave of redundant, CPU-heavy inference calls that can completely crash the system. We mitigate this by implementing the "Hero Thread" pattern utilizing Redisson distributed locks: Java // Conceptual implementation of the Hero Thread pattern in the Orchestration Plane String lockKey = "lock:schema:" + legacyVersion + ":" + upstreamVersion; RLock distributedLock = redissonClient.getLock(lockKey); // Check L1/L2 cache first MappingRule mapping = cacheManager.getMapping(legacyVersion, upstreamVersion); if (mapping == null) { // Attempt to acquire the distributed lock via Redis Pub/Sub mechanisms if (distributedLock.tryLock()) { try { // The "Hero Thread" has the lock and invokes the Inference Plane mapping = inferenceClient.fetchProbabilisticMapping(legacySchema, upstreamSchema); cacheManager.populateCaches(legacyVersion, upstreamVersion, mapping); } finally { distributedLock.unlock(); } } else { // Non-hero threads are suspended by Loom and wait for cache population mapping = waitForCacheOrRetry(legacyVersion, upstreamVersion); } } return transformJsonPayload(rawResponse, mapping); By enforcing this structure, exactly one thread (the "Hero Thread") takes the computational penalty of invoking the ML Inference Plane. The remaining 49 or 999 concurrent threads are cleanly suspended by Loom, waking up via Redis Pub/Sub to read the finalized, cached ruleset. Pragmatic AI: Why "Pure Semantic" Models Fail During initial prototyping, we found that relying solely on dense vector embeddings (like Cosine Similarity) for short JSON dictionary keys yields dangerous false-positive collisions. For instance, a dense vector model will frequently map the legacy key firstName directly to a new key named lastName because they share highly overlapping linguistic contexts within general training data. To prevent silent data corruption, raqs uses a Hybrid Ensemble Scoring Model that evaluates both semantic meaning and lexical structure: Semantic evaluation: Keys are projected into a vector space using the all-MiniLM-L6-v2 transformer model, calculating Cosine Similarity S_semantic. Lexical evaluation: To account for common developer syntax changes (such as camelCase to snake_case), we compute the normalized Levenshtein distance S_lexical. Through empirical calibration, we fixed the hyperparameters at W_semantic = 0.7 and W_lexical = 0.3. If the combined score fails to clear a strict acceptance threshold (e.g., 0.80), the mapping is rejected. Ensemble Scoring Dynamics in Action Legacy KeyNew KeySemantic ScoreLexical ScoreEnsemble ResultfirstNamefirst_name0.950.88 0.929 (Accept)userIdaccount_id0.820.40 0.694 (Reject)firstNamelastName0.880.55 0.781 (Reject)zipCodepostalCode0.890.60 0.803 (Accept) As shown above, a pure semantic evaluation would have mistakenly accepted firstName as lastName due to its high 0.88 similarity vector. The 30% lexical penalty successfully suppresses the final score below the 0.80 threshold, preserving data integrity. Performance Telemetry and Benchmarks To test the efficacy of this architecture, we subjected the raqs proxy to a load test of 1,000 requests with a concurrency cap of 50, simulating a sudden, zero-knowledge v1-to-v2 upstream schema evolution on a standard CPU-bound host machine. The cold start: Upon initialization against an empty cache, the Redisson distributed lock correctly isolated the thundering herd. Exactly one thread executed the Hybrid ML Inference, completing in 504.65 ms. The blocked threads: The remaining 49 concurrent threads were safely unmounted from OS carrier threads by Loom, waiting for lock release via Pub/Sub and completing with an average latency of 554.24 ms. The steady state: Once the rules were promoted to the Caffeine (L1) and Redis (L2) caches, the subsequent 950 requests bypassed the Inference Plane entirely. The Orchestration Plane achieved an outstanding steady-state processing latency of just 10.25 ms ($\sigma = 2.19\text{ ms}$). This performance distribution demonstrates that the computational cost of machine learning inference can be entirely isolated to cold starts, making real-time, dynamic API translation exceptionally practical for enterprise-scale traffic. The Path Forward API evolution shouldn't force a broken trade-off between breaking client applications or drowning in a versioned codebase sprawl. By pairing the non-blocking concurrency of Java 21 with a highly disciplined, multi-tier distributed proxy, we can build data layers that adapt dynamically to contract shifts. Future iterations of this paradigm will expand beyond simple key mutations to incorporate deep structural payload transformations, JSON path awareness, and automatic data type coercion. Key Takeaways Eliminate versioning sprawl: Engineers can reduce the overhead of traditional API versioning by introducing a dynamic proxy that maps evolving schemas to legacy expectations on-the-fly. Scale imperatively via Java 21: Virtual Threads (Project Loom) allow high-throughput routing middleware to scale using a readable thread-per-request model without heavy reactive frameworks. Implement the "Hero Thread" pattern: Utilizing Redisson distributed locking ensures that expensive schema inference tasks are executed exactly once during high-traffic evolution events. Deploy pragmatic hybrid scoring: Combining dense vector embeddings with normalized Levenshtein distance drastically reduces false-positive mapping collisions. Achieve sub-15ms latency: Decoupling high-latency inference from the routing path ensures that 95% of steady-state traffic experiences near-native performance.

I have lost more afternoons than I would like to admit on this exact problem: a seed script that ran cleanly yesterday now crashes on its first INSERT, and the error message tells you something you already knew, namely that you have a chicken-and-egg dependency between two tables. SQL ERROR: insert or update on table "users" violates foreign key constraint "users_organization_id_fkey" DETAIL: Key (organization_id)=(1) is not present in table "organizations". The natural next move is to reorder the inserts, putting organizations first, except that organizations.owner_user_id is NOT NULL REFERENCES users(id), which means you cannot insert an organization without a user that does not exist yet. You are looking at a foreign-key cycle, and no ordering of plain INSERT statements can satisfy every NOT NULL REFERENCES at row-insertion time. The rest of this article walks through three working strategies for seeding a Postgres database that contains FK cycles, plus a decision table for picking the right one. Examples assume Postgres 18, which is the current stable as of mid-2026, but most of the reasoning ports cleanly to earlier versions and to other RDBMSes, with the caveats called out where they matter. Two Flavors of Foreign-Key Cycles The first thing worth understanding is that two distinct cycle shapes show up in real schemas, and the fix for each is slightly different. The friendly kind is the self-referential cycle, which is what hierarchical data tends to produce. The classic example is an employees table with a manager_id REFERENCES employees(id) column: the CEO row has a NULL manager, but every other row points at another row in the same table. Self-references are easy to seed because the root row's reference can almost always be left nullable, and once that's done you insert top-down. The harder kind is the multi-table cycle, where two or more tables point at each other through NOT NULL columns. The canonical case is bidirectional ownership between users and organizations, but real schemas often contain longer cycles that route through join tables, such as users → roles → permissions → resources → users. A four-hop cycle like that one will not yield to "just reorder the inserts" no matter how patient you are. If you want to see exactly which cycles your schema contains, the Postgres catalog will tell you. The following recursive CTE walks pg_constraint, collects every cyclic path, and canonicalizes each cycle to a single representative row so you do not get one duplicate per starting node: SQL WITH RECURSIVE fk_graph AS ( SELECT conrelid::regclass AS from_table, confrelid::regclass AS to_table FROM pg_constraint WHERE contype = 'f' ), walk AS ( SELECT from_table AS start_table, from_table, to_table, ARRAY[from_table, to_table] AS path FROM fk_graph UNION ALL SELECT w.start_table, g.from_table, g.to_table, w.path || g.to_table FROM walk w JOIN fk_graph g ON g.from_table = w.to_table WHERE g.to_table <> ALL(w.path[2:]) ) SELECT path FROM walk WHERE to_table = start_table AND start_table = (SELECT MIN(t) FROM unnest(path) AS t); The MIN predicate at the end picks the rotation that begins at the lexicographically smallest table, which is what produces one row per cycle rather than one row per node-on-cycle. Run this against any schema older than about a year, and there will almost certainly be a cycle you forgot you had introduced. I ran it last quarter against a 30-table schema I thought I knew well, and it returned six. Why Naive INSERT Order Fails The reason hand-written seed scripts feel solvable at first is that most schemas form a directed acyclic graph of foreign keys, and a DAG always admits a topological sort. Countries come before cities, cities before addresses, addresses before users, and as long as you insert in that order, everything resolves on the first pass. The moment a cycle exists, the FK graph stops being a DAG, and no per-table ordering of inserts can satisfy every NOT NULL REFERENCES at row-insertion time. At least one row on the cycle has to reference a row that does not exist yet. This isn't a bug in your seed script; it's a property of the graph, and chasing it with another permutation of the insert order will produce the same error from a different table. You have three working ways out. Strategy A: Two-Pass Insert With Nullable Columns The first strategy is to give up the NOT NULL constraint on at least one edge of the cycle, insert both sides with that edge left NULL, and then close the loop in a second statement. SQL CREATE TABLE users ( id bigserial PRIMARY KEY, email text NOT NULL UNIQUE, organization_id bigint -- nullable on purpose ); CREATE TABLE organizations ( id bigserial PRIMARY KEY, name text NOT NULL, owner_user_id bigint NOT NULL REFERENCES users(id) ); ALTER TABLE users ADD CONSTRAINT users_org_fk FOREIGN KEY (organization_id) REFERENCES organizations(id); With the nullable edge in place, the seed becomes a straightforward two-pass operation inside a transaction: SQL BEGIN; INSERT INTO users (email, organization_id) VALUES ('[email protected]', NULL) RETURNING id; -- -> 1 INSERT INTO organizations (name, owner_user_id) VALUES ('Acme', 1) RETURNING id; -- -> 1 UPDATE users SET organization_id = 1 WHERE id = 1; COMMIT; The technique works on every relational database you are likely to touch, which is its main appeal, but the cost is real and easy to underestimate: you have just modelled a NOT NULL business invariant as nullable at the schema level, and you now have to enforce it somewhere else, whether in application code, in a CHECK constraint flipped on after the seeding is done, or in a deferred trigger that watches commits. Production schemas rarely tolerate that compromise, which is why Strategy A tends to live in ad-hoc dev databases where the cycle is incidental rather than load-bearing. One Postgres 17 quality-of-life improvement worth knowing about is that MERGE ... RETURNING combined with the new merge_action() function makes the two-pass shape less verbose for bulk imports, because you can now route inserts and updates through a single MERGE and capture which path each row took. The underlying two-pass logic is unchanged, but for many real workloads the line count comes down by roughly half. Strategy B: Deferred Constraints Inside a Transaction Postgres offers a more elegant escape: you can declare a foreign key as deferrable, which postpones the constraint check from row-insertion time to transaction-commit time. SQL CREATE TABLE users ( id bigserial PRIMARY KEY, email text NOT NULL UNIQUE, organization_id bigint NOT NULL ); CREATE TABLE organizations ( id bigserial PRIMARY KEY, name text NOT NULL, owner_user_id bigint NOT NULL REFERENCES users(id) DEFERRABLE INITIALLY IMMEDIATE ); ALTER TABLE users ADD CONSTRAINT users_org_fk FOREIGN KEY (organization_id) REFERENCES organizations(id) DEFERRABLE INITIALLY IMMEDIATE; The seed then collapses to a single pass inside a transaction that opts into deferred checking: SQL BEGIN; SET CONSTRAINTS ALL DEFERRED; INSERT INTO organizations (id, name, owner_user_id) VALUES (1, 'Acme', 1); INSERT INTO users (id, email, organization_id) VALUES (1, '[email protected]', 1); COMMIT; -- both rows exist by now, both FK checks pass For day-to-day work, DEFERRABLE INITIALLY IMMEDIATE is the variant you want, because it leaves the constraint behaving exactly like a normal one in every transaction except those that explicitly call SET CONSTRAINTS ALL DEFERRED. The more aggressive INITIALLY DEFERRED defers every transaction by default, which sounds harmless until you realize that errors then surface at commit instead of at the offending statement, making real bugs much harder to chase down. There is a piece of folklore in this area worth dismantling before it bites you: a foreign key declared DEFERRABLE INITIALLY IMMEDIATE does not pay a measurable runtime cost compared to a non-deferrable one. For FK constraints specifically, the check happens at end-of-statement or end-of-transaction in both modes, so there is no fast path you are giving up by marking it deferrable. The reason this confuses people is that UNIQUE and PRIMARY KEY constraints do behave differently when made deferrable, because their underlying index can no longer enforce uniqueness eagerly, so the perf-myth that applies legitimately to UNIQUE indexes gets generalized, incorrectly, to foreign keys. If your team is resisting DEFERRABLE on production tables because of a vague performance worry, the cure is a five-minute benchmark. Two real caveats apply. The smaller one is that only declared-deferrable constraints can be deferred, so if you forgot to write DEFERRABLE in the original migration, you have to alter the constraint after the fact, and that alteration requires SHARE ROW EXCLUSIVE on both tables. It isn't the end of the world for a planned maintenance window, but it isn't a no-op either. The larger and more current caveat is a 2026 footgun that bit teams who adopted the new NOT ENFORCED toggle introduced in Postgres 18. In releases shipped before May 14, 2026, a foreign key declared DEFERRABLE INITIALLY DEFERRED would quietly start behaving as NOT DEFERRABLE after being toggled NOT ENFORCED and then back to ENFORCED. There was no error, no warning, no log line, just a constraint that no longer deferred when you expected it to, which made for some entertaining debugging sessions. The fix shipped in 18.4, 17.10, 16.14, 15.18, and 14.23, and the remediation after upgrading is to toggle any affected constraint NOT ENFORCED and back to ENFORCED one more time. Constraints declared INITIALLY IMMEDIATE, and constraints that have never been routed through NOT ENFORCED, were not affected, which means most production seeds escaped this entirely. Anyone who used NOT ENFORCED for bulk loads in late 2025 or early 2026 should re-verify, however, because the silent nature of the regression makes "we never noticed" the most likely failure mode. The MySQL side of the story is shorter: InnoDB does not support deferred foreign keys at all. Your options there are Strategy A or a scoped SET FOREIGN_KEY_CHECKS = 0 inside a transaction, which is a heavier hammer than most teams are happy to swing in CI. Strategy C: Generator-Driven Cycle Resolution Strategies A and B both work, but they oblige you to hand-write the insert plan for every cycle, and that burden compounds as the schema grows. A team with half a dozen cycles, or with a cycle topology that shifts every few migrations, will eventually find that its seed script has become its own piece of brittle infrastructure that breaks in ways indistinguishable from the original problem, just in different places. Three rough tiers of tooling cover this space, and it is worth knowing where each one sits before reaching for any of them. At the bottom of the budget curve sit column-level generators such as Faker, Mockaroo, and generatedata.com, which produce realistic per-column values like names, emails, and ZIP codes but do not model the foreign-key graph at all. They hand you a CSV or a stream of INSERT statements and leave the cycle resolution to you, which is appropriate for what they do but also means that a Faker-driven pipeline still needs Strategy A or Strategy B underneath it as soon as cycles enter the picture. A newer middle tier of schema-aware data generators treats the FK graph as a first-class input rather than asking you to script around it. Tools in this category read the schema directly from the database, work out a valid load order on their own, and emit a plan that handles the cycle for you, which means you do not have to write Strategy A or B by hand for every cycle in your schema. The category is still small and emerging — Neosync and Seedfast are two examples that take this approach in slightly different ways — and most of these tools originated in regulated-industry workflows where teams could not legally copy production data into development environments and hand-written seeds were not scaling across migrations, which is why they tend to assume that audience by default. The mechanical part of the problem — figuring out a valid load order — turns out to be the easy half. The harder and more consequential half is generating realistic values while keeping every other schema constraint satisfied at the same time, because a type-valid email such as [email protected] passes the column definition but renders a B2B SaaS test dataset useless even when the FK relationships are all intact, and the same trap applies to unique indexes, partial indexes, check constraints, generated columns, RLS policies, and any business invariants encoded in triggers. One tier further up the budget curve, the enterprise test-data-management platforms that show up most often in procurement decks — Tonic.ai, Synthesized, and Delphix — take the inverted approach: they sit in front of a production database, anonymize or otherwise transform real data on the way out, and ship the result into lower environments through pipelines that are typically bought, audited, and operated by a dedicated platform team. The premise is the opposite of the schema-aware generators above, which assume you specifically do not want production data anywhere near dev or staging. Both approaches exist because both audiences exist, and choosing between them is usually a function of your compliance posture, your appetite for managing a TDM platform, and whether your annual tooling budget reaches six figures. Adopting a schema-aware generator is a trade-off, not a default. It pays off when you have a non-trivial number of cycles, when CI rebuilds happen often enough that the seed plan needs to survive without supervision, and when several environments — local, CI, staging, demo — all need data with similar shape. It is overkill when you have a single cycle, a stable schema, and a small team, because in that situation Strategy B in fifteen lines of SQL will still be there working in two years. The enterprise tier becomes the right answer when the compliance question is settled in the opposite direction, namely that production data is the ground truth your downstream environments must look like, anonymization is the only legally defensible path to lower environments, and your organization is willing to absorb the platform-engineering overhead that comes with it. A 2026 Alternative Worth Considering: Clone, Do Not Regenerate Postgres 18 made an older pattern viable in ways it had not been before, and it is worth mentioning here because for CI workloads it may obviate the entire choice between A, B, and C. The relevant pieces are a new server GUC, file_copy_method, and the existing FILE_COPY strategy on CREATE DATABASE. The GUC defaults to COPY, which performs a traditional byte-by-byte copy; setting it to CLONE tells Postgres to use copy_file_range() on Linux and FreeBSD or copyfile on macOS, which lets the kernel share blocks at the filesystem layer instead of physically duplicating them. On a copy-on-write filesystem such as XFS-with-reflinks, ZFS, APFS, or btrfs, the result is a 6 GB template cloned in roughly 212 milliseconds, against about 67 seconds for the default COPY method. On ext4 or any non-CoW filesystem, the kernel cannot honor the clone request, and you fall back to the slow byte-copy regardless, which makes the filesystem choice the load-bearing decision rather than the SQL. If your bottleneck is "how do I get a populated, schema-correct database in front of every test run as fast as possible," then cloning a template you seeded once is often a better answer than regenerating from scratch, regardless of which of the three strategies above you would have used to populate the template. SQL SET file_copy_method = 'clone'; -- session-level; for CI, set in postgresql.conf CREATE DATABASE test_run_42 TEMPLATE seedfast_template STRATEGY = FILE_COPY; The catch is that the template must be idle at clone time, with no other connections, which is awkward inside the same Postgres instance that is running everything else. Most teams who lean on this pattern run a dedicated CI Postgres instance whose only job is hosting the templates, which is more infrastructure but a one-time setup. When to Pick Which SituationStrategyAd-hoc dev DB, one or two cycles you know aboutA: nullable column, two-passPostgres, production-shaped schema with real NOT NULL FKs, frequent rebuildsB: DEFERRABLE + SET CONSTRAINTS ALL DEFERREDMySQL or mixed-RDBMS, no leeway to change the schemaA, or scoped SET FOREIGN_KEY_CHECKS = 0 if you trust the sourceMany cycles, frequent migrations, multiple environments to seedC: schema-aware generatorCI throughput is the bottleneck, schema is stable, CoW filesystem availableTemplate DB + Postgres 18 FILE_COPY clone, on top of any of A/B/COne cycle, stable schema, small teamB if you are on Postgres, A otherwise; resist adding a tool A Note on Cycle Hygiene A surprising number of "cycles" in production schemas turn out, on inspection, to be accidents that crept in across a few migrations rather than load-bearing design choices. Someone added created_by_user_id to an audit table that the users table already referenced, and nobody noticed the loop until a fresh seed run failed two sprints later. If the cycle in question is not actually load-bearing in your business logic, breaking it at the schema level by making one of the FK columns nullable in production is almost always a better long-term move than carrying any of the workarounds above. A seed script that does not have to resolve cycles is faster, simpler, and harder to break than any of the three strategies, and the documentation cost of explaining why the column is nullable is much smaller than the cost of explaining the seeding workaround to every new engineer who joins the team. For the cycles that really are intentional, such as the ownership patterns, hierarchical references, audit chains, and any other places where both sides of the relationship genuinely cannot exist without each other, the right move is to pick the strategy that matches your stack and your appetite for ongoing maintenance, and then to write that choice down somewhere your future colleagues will find it. The undocumented version of "we use deferred constraints because Strategy A broke our integration tests last year" is exactly the kind of folklore that gets reinvented from scratch every eighteen months when the engineer who knew it leaves.