DZone Spotlight

A step-by-step guide to grounding a LangGraph agent in Microsoft Foundry IQ agentic retrieval — without rebuilding your RAG pipeline. Why This Integration Is Worth Doing If you build agents on LangGraph and your enterprise content lives in Azure, you have probably written the same code twice: a chunker, an embedding job, a vector store, a retriever, a reranker, and a permissions filter bolted on at the end. Every new agent gets its own copy. Every copy drifts. Foundry IQ moves that work behind a single endpoint. A knowledge base wraps one or more knowledge sources, and the agentic retrieval engine handles query planning, parallel execution, semantic reranking, and (optionally) answer synthesis. Crucially for anyone outside the Microsoft agent stack: every knowledge base is also a standalone MCP server exposing one tool, knowledge_base_retrieve. Any MCP-compatible client can call it — including LangGraph, via langchain-mcp-adapters. That is the whole integration. The interesting parts are the four places it does not behave like a normal retriever, which this tutorial covers in detail: The MCP tool result has a different shape from the REST/SDK retrieve response.Bearer tokens expire, and a static headers dict will fail an hour into a long-running graph.Per-user permission filtering requires a second token, distinct from your service credential.The knowledge base is itself a planner, so you have two planners per turn and need to decide who does what. By the end, you will have a working LangGraph agent grounded in a Foundry IQ knowledge base, with citations preserved in graph state and a token provider that survives long sessions. What you should already know: LangGraph basics (StateGraph, ToolNode, the ReAct loop), and enough Azure to create a resource and assign a role. Architecture Figure 1 — The knowledge base owns retrieval. LangGraph owns orchestration. MCP is the contract between them. Three things are worth noticing before you write any code. The knowledge base is reusable. It is not scoped to one agent. The same knowledge base can ground a LangGraph agent, a Foundry Agent Service agent, and a Copilot integration simultaneously. That is the point of the abstraction, and it changes how you name things — name knowledge bases after topics (hr-policy-kb, product-docs-kb), not after the agent that happens to consume them first. Sources come in two flavors. Indexed sources (Blob, OneLake, an existing search index) are ingested, chunked, and vectorized into an index on your search service. Federated sources (remote SharePoint, web, MCP servers) are queried live at retrieval time and never ingested. This distinction matters for permissions, which we return to in Step 7. Identity flows in two channels. Your application authenticates to the search service with a service identity. Optionally, you also pass the end user's identity in a separate header so the engine filters documents that user may not see. Conflating these two is the most common source of "why is everyone seeing everything" bugs. Prerequisites An Azure AI Search service with agentic retrieval available in your region. Basic tier or higher if you want managed identity support.A Microsoft Foundry project and resource, with an LLM deployment (e.g. gpt-5-mini) and an embedding model (e.g. text-embedding-3-large).The Search Index Data Reader role assigned to the identity that will query the knowledge base.If your knowledge base specifies an LLM, the search service needs a managed identity with Cognitive Services User on the Foundry resource.Python 3.10+. Install the client libraries: Shell # Preview SDK — required for answer synthesis, configurable reasoning effort, # document-level permissions and multi-turn retrieve. pip install --pre azure-search-documents # Stable SDK is enough if you only need GA features on 2026-04-01: # pip install azure-search-documents pip install azure-identity langchain-mcp-adapters langgraph langchain-openai httpx A Word on API Versions Before You Start This is the decision that will bite you later if you get it wrong, so make it deliberately now. Agentic retrieval is generally available in the 2026-04-01 REST API. The 2026-05-01-preview adds answer synthesis, configurable reasoning effort, the messages input, document-level permissions, and sensitivity-label metadata. Both the Azure portal and the Microsoft Foundry portal expose preview-only behavior regardless of what your code uses, which means the portal is not a reliable preview of what your production code will do. The API version also changes MCP behavior directly. With 2026-05-01-preview, the knowledge base can return synthesized answers when it is configured with an LLM and a compatible reasoning effort. With 2026-04-01, MCP retrieval is always minimal and extractive, and the connection returns grounding data only. Pick 2026-04-01 if you are shipping to production now and can live with extractive grounding. Pick 2026-05-01-preview if you need synthesis or permission filtering, and accept there is no SLA. Step 1: Create the Knowledge Source and Knowledge Base A knowledge source points at your content. A knowledge base wraps one or more sources with retrieval configuration. Create the source first. Python import os from azure.identity import DefaultAzureCredential from azure.search.documents.indexes import SearchIndexClient from azure.search.documents.indexes.models import ( KnowledgeBase, KnowledgeSourceReference, KnowledgeBaseAzureOpenAIModel, AzureOpenAIVectorizerParameters, SearchIndexKnowledgeSource, SearchIndexKnowledgeSourceParameters, ) SEARCH_ENDPOINT = os.environ["AZURE_SEARCH_ENDPOINT"] # https://<svc>.search.windows.net AOAI_ENDPOINT = os.environ["AZURE_OPENAI_ENDPOINT"] CHAT_DEPLOYMENT = "gpt-5-mini" credential = DefaultAzureCredential() index_client = SearchIndexClient(endpoint=SEARCH_ENDPOINT, credential=credential) # A knowledge source over an index you already have. knowledge_source = SearchIndexKnowledgeSource( name="product-docs-ks", description=( "Product documentation, release notes and API reference. " "Use for questions about product behaviour, configuration and limits." ), search_index_parameters=SearchIndexKnowledgeSourceParameters( search_index_name="product-docs-index", source_data_select="id,title,content,url,updated_at", ), ) index_client.create_or_update_knowledge_source(knowledge_source=knowledge_source) The description is not decoration. The retrieval engine uses it when deciding which sources to query for a given subquery, so write it the way you would explain the source to a new colleague: what is in it, and what kinds of questions it answers. Now the knowledge base: Python knowledge_base = KnowledgeBase( name="product-docs-kb", description="Grounding for product support questions.", knowledge_sources=[ KnowledgeSourceReference(name="product-docs-ks"), ], models=[ KnowledgeBaseAzureOpenAIModel( azure_open_ai_parameters=AzureOpenAIVectorizerParameters( resource_url=AOAI_ENDPOINT, deployment_name=CHAT_DEPLOYMENT, model_name=CHAT_DEPLOYMENT, ) ) ], retrieval_instructions=( "Prefer the most recently updated documents when versions conflict. " "For questions about limits or quotas, always consult product-docs-ks." ), ) index_client.create_or_update_knowledge_base(knowledge_base=knowledge_base) print("knowledge base ready") retrieval_instructions steers the planner's source selection. It is the highest-leverage knob in the whole configuration, and the one most people leave empty. Index requirements. If you point at an existing index, it needs a semantic configuration — agentic retrieval uses L2 semantic ranking. If the index has vector fields, it also needs a valid vectorizer so the engine can vectorize subqueries; otherwise vector fields are silently ignored. Step 2: Verify With the Retrieve API Before You Touch MCP Do not debug two systems at once. Confirm retrieval works over the SDK first, where you get the full response envelope including the query plan. Python from azure.search.documents.knowledgebases import KnowledgeBaseRetrievalClient from azure.search.documents.knowledgebases.models import ( KnowledgeBaseMessage, KnowledgeBaseMessageTextContent, KnowledgeBaseRetrievalRequest, ) kb_client = KnowledgeBaseRetrievalClient( endpoint=SEARCH_ENDPOINT, knowledge_base_name="product-docs-kb", credential=credential, ) request = KnowledgeBaseRetrievalRequest( messages=[ KnowledgeBaseMessage( role="user", content=[KnowledgeBaseMessageTextContent( text="What are the rate limits on the ingestion API, and did they change in the last release?" )], ) ], include_activity=True, # gives you the query plan ) result = kb_client.retrieve(request) print(result.response[0].content[0].text[:800]) # grounding data for entry in result.activity: # what the planner actually did print(entry.type, getattr(entry, "elapsed_ms", None)) The activity array is your observability surface. It reports the planner's token usage, the subqueries that were issued to each source, elapsed time per source, and reasoning-token consumption. Read it now, because — as we will see in Step 5 — you do not get it back over MCP. If the response is empty but activity shows matches were found, a document probably exceeded the output budget. Increase max_output_size, or chunk large source documents more aggressively. Step 3: Understand What the MCP Endpoint Gives You (and What It Does Not) Every knowledge base is automatically an MCP server. There is nothing to deploy. The endpoint is: Plain Text https://<your-search-service>.search.windows.net/knowledgebases/<your-knowledge-base>/mcp?api-version=<api-version> It exposes exactly one tool, knowledge_base_retrieve. Clients cannot see index management or source configuration through it — the surface is deliberately narrow. The trap is that the MCP tool result is not the retrieve action's response. It is a plain MCP tool result, which most clients surface under result.content[]: JSON { "result": { "content": [ { "type": "text", "text": "[{\"ref_id\":\"0\",\"title\":\"Ingestion limits\",\"terms\":\"rate limit, throttling\",\"content\":\"<chunk>\"}]" } ] } That text field is a JSON-encoded string, not a JSON object. You have to parse it twice. And the activity and references arrays you relied on in Step 2 are simply absent. Here is the comparison in full — this table is the thing to keep open while you build: AspectRetrieve action (REST / SDK)MCP endpoint (knowledge_base_retrieve)Payload locationresponse[0].content[0].textresult.content[0].textGrounding data formatJSON-encoded stringJSON-encoded string (same inner shape)activity array (query plan, tokens, per-source timings)Returned when includeActivity is setNot returnedreferences array (ref_id → docKey, activitySource)Returned, controllable per sourceNot returnedSensitivity label metadataPer-reference + response-level aggregateSame fields surfaced when configuredAnswer synthesisAvailable on 2026-05-01-previewAvailable on 2026-05-01-preview only; 2026-04-01 is always extractivePer-request tuning (filterAddOn, maxOutputDocuments, failOnError)Full control per knowledge sourceNot exposed — set defaults on the knowledge base insteadAuth mechanismSDK credential objectAuthorization: Bearer header, or api-key headerBest forDeterministic pipelines, evaluation harnesses, observabilityAgent frameworks, tool-calling loops, cross-runtime reuse The practical consequence: use MCP for the agent loop, and keep a direct retrieve client around for evaluation and debugging. They point at the same knowledge base, so there is no duplication of configuration — only of client code, and only where it earns its keep. Step 4: Authenticate, Properly Two options. Only one belongs in production. Admin key (api-key header) grants full read-write access to the search service. Use it for a five-minute spike, never beyond that. Bearer token (Authorization header) is the recommended path. The identity behind the token needs Search Index Data Reader on the search service, and the token must be scoped to https://search.azure.com/.default. The naive version looks like this, and it works — for about an hour: Python from azure.identity import DefaultAzureCredential, get_bearer_token_provider credential = DefaultAzureCredential() search_token_provider = get_bearer_token_provider( credential, "https://search.azure.com/.default" ) MCP_URL = ( f"{SEARCH_ENDPOINT}/knowledgebases/product-docs-kb/mcp" "?api-version=2026-05-01-preview" ) connection = { "foundry_iq": { "transport": "http", # streamable HTTP "url": MCP_URL, "headers": {"Authorization": f"Bearer {search_token_provider()}"}, } } Note the parentheses: search_token_provider() is evaluated once, at construction time, and frozen into a dict. A long-running graph, a checkpointed conversation resumed the next morning, or a service that builds its client at startup will all start returning 401s once that token expires. The Fix: An httpx.Auth That Refreshes langchain-mcp-adapters uses the official MCP SDK underneath, which accepts a custom authentication mechanism implementing the httpx.Auth interface. That is where token refresh belongs. Python import httpx from azure.identity import DefaultAzureCredential, get_bearer_token_provider class EntraBearerAuth(httpx.Auth): """Attaches a fresh Entra ID bearer token to every MCP request. azure-identity caches the token internally and only round-trips to the IdP when it is close to expiry, so calling the provider per request is cheap. """ def __init__(self, credential, scope: str = "https://search.azure.com/.default"): self._provider = get_bearer_token_provider(credential, scope) def auth_flow(self, request: httpx.Request): request.headers["Authorization"] = f"Bearer {self._provider()}" yield request auth = EntraBearerAuth(DefaultAzureCredential()) connection = { "foundry_iq": { "transport": "http", "url": MCP_URL, "auth": auth, # instead of a frozen headers dict } } This single change is the difference between a demo and something you can leave running. Step 5: Load the Tool Into LangGraph With authentication sorted, wiring the tool in is short. Python from langchain_mcp_adapters.client import MultiServerMCPClient client = MultiServerMCPClient(connection) tools = await client.get_tools() print([t.name for t in tools]) # ['knowledge_base_retrieve'] For a quick check, hand the tool straight to a prebuilt agent: Python from langgraph.prebuilt import create_react_agent # (LangChain v1 equivalent: from langchain.agents import create_agent) agent = create_react_agent("azure_openai:gpt-5-mini", tools) response = await agent.ainvoke( {"messages": [{"role": "user", "content": "What changed about ingestion rate limits in the last release?"}]} ) print(response["messages"][-1].content) If that returns a grounded answer, the integration works. Two operational notes before you build the real graph: MultiServerMCPClient is stateless by default — each tool invocation opens a fresh MCP session and tears it down. For a stateful server, you would use client.session(), but for Foundry IQ retrieval, stateless is correct and cheaper.If you register several MCP servers, be aware that a single failing server has historically been able to take down get_tools() for all of them. Register Foundry IQ in its own client if the rest of your tool estate is flaky. Step 6: Build the Graph, and Parse the Response The prebuilt agent hides the thing you most need to control: what happens to the grounding data on its way into state. Here is the explicit version. Figure 2 — Two planners run per turn. Step 2 decides whether to retrieve; step 5 decides how. Python import json from typing import Annotated, TypedDict from langchain_core.messages import ToolMessage from langchain_openai import AzureChatOpenAI from langgraph.graph import StateGraph, START, END from langgraph.graph.message import add_messages from langgraph.prebuilt import ToolNode class AgentState(TypedDict): messages: Annotated[list, add_messages] citations: list # accumulated across the conversation SYSTEM = ( "You answer questions about our product documentation. " "Always call knowledge_base_retrieve before answering a factual question. " "Cite sources using the ref_id values in the grounding data. " "If the grounding data does not contain the answer, say you do not know." ) llm = AzureChatOpenAI(azure_deployment="gpt-5-mini", api_version="2025-04-01-preview") llm_with_tools = llm.bind_tools(tools) async def agent_node(state: AgentState): messages = [{"role": "system", "content": SYSTEM}] + state["messages"] return {"messages": [await llm_with_tools.ainvoke(messages)]} def parse_grounding(state: AgentState): """Pull ref_id/title/url out of the last tool message into state. The MCP tool result is a JSON-encoded string inside a text content block, so it needs a second json.loads(). """ last = state["messages"][-1] if not isinstance(last, ToolMessage): return {} raw = last.content if isinstance(raw, list): # content-block form raw = next((b.get("text", "") for b in raw if isinstance(b, dict) and b.get("type") == "text"), "") try: docs = json.loads(raw) except (json.JSONDecodeError, TypeError): return {} # synthesized answer, not extractive found = [ { "ref_id": d.get("ref_id"), "title": d.get("title"), "url": d.get("url"), } for d in docs if isinstance(d, dict) ] return {"citations": state.get("citations", []) + found} def should_continue(state: AgentState): last = state["messages"][-1] return "tools" if getattr(last, "tool_calls", None) else END builder = StateGraph(AgentState) builder.add_node("agent", agent_node) builder.add_node("tools", ToolNode(tools)) builder.add_node("parse", parse_grounding) builder.add_edge(START, "agent") builder.add_conditional_edges("agent", should_continue, {"tools": "tools", END: END}) builder.add_edge("tools", "parse") builder.add_edge("parse", "agent") graph = builder.compile() The parse node is what keeps citations alive. Without it, the grounding JSON passes through the message history as an opaque blob, the model cites ref_id values, and your UI has nothing to resolve them against. Note the defensive try/except: if you switch the knowledge base to answer-synthesis output mode, the tool returns prose rather than a JSON array, and a parser that assumes JSON will crash on a configuration change made by someone else in the Azure portal. Step 7: Enforce Per-User Permissions Everything so far runs as a single service identity, which means every user sees every document the service can see. For most enterprise deployments, that is unacceptable. Permission enforcement has two halves. At ingestion time, indexed sources need ingestionPermissionOptions set so that ACLs, RBAC scopes, or Purview sensitivity labels are ingested alongside content. If you skip this, results come back unfiltered no matter what you send at query time — and the only fix is to recreate the knowledge source. Federated sources work differently: remote SharePoint queries through the Copilot Retrieval API using the user's own token and never ingests anything, and Fabric and Work IQ sources exchange the user's token for a scoped one. At query time, you pass the end user's access token — scoped to https://search.azure.com/.default, separate from your service credential, and requiring no search-service permissions of its own — in the x-ms-query-source-authorization header. Over the SDK, that is a named parameter: Python result = kb_client.retrieve( retrieval_request=request, x_ms_query_source_authorization=user_token, # the end user, not the service ) Over MCP, it is a per-request header, which means it varies per user while your client is long-lived. Extend the auth class rather than rebuilding the client: Python import contextvars current_user_token = contextvars.ContextVar("current_user_token", default=None) class EntraBearerAuthWithUser(EntraBearerAuth): def auth_flow(self, request: httpx.Request): request.headers["Authorization"] = f"Bearer {self._provider()}" user_token = current_user_token.get() if user_token: request.headers["x-ms-query-source-authorization"] = user_token yield request Set the context variable at the edge of your application — in the FastAPI dependency or middleware that already validates the caller — and every downstream MCP call in that request inherits it, including calls made deep inside a graph. Verify this end to end in your own environment. Microsoft documents the header explicitly for the retrieve action, and notes that MCP clients configure custom headers differently. Test with two users who have genuinely different document access and confirm the result sets differ — do not assume it works because it did not error. Step 8: Decide Who Plans This is the design question the tutorial format tends to bury, so it gets its own step. Look again at Figure 2. Your LangGraph agent node runs an LLM to decide whether to retrieve and how to phrase the query. Then the knowledge base runs another LLM to decompose that query into subqueries and choose sources. Two planners, two model calls, two chances to lose the user's intent. The failure mode is specific: the agent node paraphrases the user's question before handing it over, dropping a constraint ("in the 2026 release", "for part XYZ2B"), and the knowledge base then plans excellent subqueries for the wrong question. Microsoft's own evaluation work identifies exactly this — constraint preservation in the handoff from orchestrator to retriever — as the thing that correlates with retrieval quality. Three rules that follow from it: Pass the question through; do not summarize it. Instruct the agent node to forward the user's wording, including qualifiers, rather than composing a "better" search query. The knowledge base is better at query formulation than your agent node is; that is what you are paying it for.Tune reasoning effort, not prompts. minimal skips LLM planning entirely and runs keyword or hybrid search on the query as given — the right choice for lookups. low and medium add planning; medium adds iterative search, where the engine reviews its own results and issues follow-ups. Answer synthesis requires low or medium. Route cheap questions to a minimal knowledge base and hard ones to a medium one, and you have turned a latency/quality trade-off into a graph edge.Prefer extractive output inside an agent. Answer synthesis produces a finished natural-language answer, which is what you want when retrieval output goes straight to a user. Inside a LangGraph agent, the agent is going to reason over the content anyway — synthesizing first costs tokens and latency, and flattens the structure your parse node wants. Troubleshooting SymptomLikely causeWhat to do403 from Azure AI SearchIdentity lacks Search Index Data Reader on the search serviceAssign the role; confirm you are signed in to the right tenant and subscription401 after roughly an hourBearer token frozen into a static headers dictSwitch to the httpx.Auth provider from Step 4400 Bad RequestA knowledgeSourceName is not attached to the knowledge base, or its kind does not match; or one option requires another that is not enabledRead the top-level error — it names the offending property206 Partial ContentAt least one source failed, none of them marked requiredInspect the activity entries carrying an error; process partial results or mark the critical source failOnError502 Bad GatewayEvery selected source failed, or a source marked failOnError failedDo not assume an outage — read the underlying source failure firstEmpty response, but activity shows matchesMost relevant document exceeded the output budgetRaise maxOutputSize, or chunk large documents at ingestionEvery user sees every documentingestionPermissionOptions was not set when the knowledge source was createdRecreate the knowledge source with the right options; the header alone will not fix itAnswers ignore recent documentsScoring profiles are not applied by agentic retrievalUse freshness-aware retrieval rather than an index scoring profileTool list comes back emptyOne failing server in a multi-server clientGive Foundry IQ its own MultiServerMCPClient Where to Go Next The same knowledge base you just built is reachable from Microsoft Agent Framework, Foundry Agent Service, GitHub Copilot, Claude, and Cursor without any change to its configuration. That is the real payoff of putting retrieval behind MCP rather than inside your agent: when your team standardizes on a different runtime next year, the knowledge layer does not move. Two things worth building next: an evaluation harness that calls the retrieve API directly (so you get the activity array and can measure whether your retrieval_instructions are actually steering source selection), and a second knowledge base at minimal reasoning effort so you can route by question difficulty. References Query a knowledge base using the retrieve action or MCP endpoint — the authoritative reference for the MCP endpoint URL, authentication, response shapes, permission headers, and troubleshooting status codes.Agentic retrieval in Azure AI Search — overviewModel Context Protocol (MCP) — LangChain docs — transports, custom httpx.Auth, session lifecycle.langchain-mcp-adapters on GitHubMultiServerMCPClient API referenceFoundry IQ: build smarter agents faster with unified knowledge and serverless retrieval — Build 2026 announcement; GA scope and the MCP server.Foundry IQ: improve recall by up to 54% with knowledge bases — the constraint-preservation evaluation behind Step 8.

By Jubin Soni, FBCS CORE

The term software factory is getting a lot of attention right now, and for a good reason. AI coding assistants can generate code much faster than before. But faster coding alone does not mean faster, safer delivery. In many teams, it simply moves the bottleneck to review, testing, deployment, and operations. A software factory is a way to organize the entire software development life cycle as one connected, repeatable system. Think of a car manufacturing assembly line. Each station has a clear job, work moves forward in a predictable order, quality checks happen at the right moments, and the finished product is inspected before it leaves the factory. An agentic software factory applies that same idea to software delivery. AI agents do focused work across planning, coding, testing, deployment, monitoring, and feedback. Humans remain in charge of specifications, security, policies, approvals, and the decisions that should never be delegated blindly. Key Takeaways An agentic software factory coordinates AI agents across the complete software delivery lifecycle.Faster AI-assisted coding can create review bottlenecks unless downstream stages also improve.Humans retain ownership by defining guardrails, specifications, approvals, and security requirements.Workflow orchestration connects context, automation, observability, incident handling, and feedback loops. What Is a Software Factory? A software factory is not just a collection of developer tools. It is an operating model where software delivery is designed as a smooth, observable workflow from idea to production and back into improvement. In a car factory, a vehicle moves through assembly, painting, quality inspection, final assembly, and delivery. People are involved at important checkpoints, but the process does not restart from scratch at every station. It is structured, repeatable, and connected. The same model works for software. In an agentic software factory, the flow can look like this: Requirements and design: clarify what needs to be built and collect service context.Planning: turn requirements into a feasible implementation plan.Build: generate or modify code for a feature or bug fix.Testing and continuous integration: validate the change and stop failures early.Human review: approve, reject, or request changes before a risky next step.Deployment: deliver the approved change through continuous delivery.Monitoring and operations: check service health, create incidents, notify teams, and roll back when needed.Feedback loop: feed production signals and outcomes back into future planning. The important point is simple: agents perform work, while humans provide the gates. A software factory is not about removing people from the loop. It is about putting people at the moments where their judgment matters most. How Software Delivery Evolved Into the Software Factory Model The software factory did not appear suddenly. It is the next step in a long evolution toward more reliable software delivery. From Manual Deployment to Automated Delivery In the 1990s, developers often wrote the code, prepared servers, and deployed software manually. A release could take weeks or months. Testing and deployment were labor-intensive, and repeatability depended heavily on individual knowledge. Then continuous integration tools such as Hudson and Jenkins helped teams automate builds and tests. The rise of DevOps brought development and operations closer together, reducing the handoff gap between teams. Continuous delivery, continuous deployment, and infrastructure as code tools such as Terraform pushed automation further. GitOps and platforms such as Docker and Kubernetes added a strong operational model where Git could serve as the source of truth for application and infrastructure changes. Each stage made delivery more repeatable. From Automation to Agentic Engineering After AI agents and coding assistants became practical, teams began using them across more parts of the SDLC. An agent could help gather requirements, propose a plan, write code, prepare tests, review pull requests, monitor a service, or summarize feedback. That is where the software factory becomes agentic. Instead of treating AI as a single chat window or code-completion tool, I treat it as a coordinated group of specialized workers within a governed delivery system. The software factory model gives those agents a place, a sequence, boundaries, and clear outputs. Without that structure, adding more agents can create more confusion rather than more throughput. Why AI Coding Assistants Create New Bottlenecks Before coding assistants, the time required across planning, coding, review, and shipping was comparatively balanced. Writing code often took a large part of the cycle, but every stage had its own workload. Now coding can accelerate dramatically. Tools such as Cursor, GitHub Copilot, Claude Code, and Codex can help teams generate and change code faster. The problem is that the rest of the system does not automatically become faster. When code arrives faster, code review queues can become overloaded. Senior engineers get stuck reviewing a growing number of pull requests. Testing may become backed up. Deployment approvals may take longer. Only a small portion of the increased output may actually reach production. This is exactly why a software factory matters. It looks at the whole system, not only the coding stage. A good software factory improves the flow across the entire lifecycle so that one accelerated step does not jam everything downstream. Preventing Agentic Chaos There is another issue. Developers are already using many tools across the SDLC. Add multiple AI agents without a common operating layer, and it becomes difficult to answer basic questions: Which agent changed this service?What context did the agent use?Which policy or guardrail applied?Who approved the deployment?What should happen if the health check fails? That is agentic chaos: lots of autonomous activity, but little visibility, control, governance, or accountable decision-making. A software factory makes the agent workflow explicit. It creates a visible path for work, controls access to actions, and places checks before high-impact changes. Humans Still Own the System It is tempting to say that an agentic software factory can automate everything end-to-end. Technically, many tasks can be automated. In reality, giving unrestricted authority to agents is risky. A poorly constrained agent can make the wrong decision, trigger the wrong action, or cause damage in production. Humans and developers still own the system. In a well-designed software factory, my role is not to manually do every repeated task. My role is to define the rules of the factory. That includes: Defining product requirements and technical specifications.Setting security checklists and guardrails.Deciding which actions agents may take automatically.Creating human approval gates for important decisions.Reviewing plans, pull requests, release readiness, and incident responses.Maintaining accountability for production systems. This is the right division of responsibility. Agents can gather context, plan work, implement changes, run tests, check health, and notify teams. Developers decide what good looks like, which risks are acceptable, and whether a change should proceed. The Building Blocks of an Agentic Software Factory A practical software factory breaks broad lifecycle phases into smaller, focused responsibilities. Rather than relying on one giant agent to do everything, I can use agents for specific jobs and connect them through workflow orchestration. Plan The planning stage starts with human input and service context. A requirements agent can gather the feature request, identify the affected service, and collect relevant information. A planning agent can then turn that into an implementation plan. A feedback digest or product improvement agent can provide useful context from previous issues and outcomes. Build and Review The build stage can include a feature builder and bug-fixer agent. The review stage can include a pull request reviewer, automated CI checks, and other quality actions. The key is that a failed CI build blocks the workflow. It should not quietly move toward deployment. After CI succeeds, a human review gate can decide whether the change is ready to continue. This is where the software factory protects speed with judgment. Deploy and Operate After approval, a continuous delivery agent can deploy the service or feature. A monitoring agent can then assess the health of the service. If health is degraded or a critical issue appears, the workflow can create an incident, notify the relevant team through Slack, and, where appropriate, perform an automated rollback. The final piece is the feedback loop. Production data should not disappear into dashboards. It should update service context and help inform future planning. That loop is what turns a set of automation steps into an evolving software factory. Building a Software Factory Workflow With Port To put this into practice, I used Port as the context layer for an agentic SDLC. Port brings together workflow orchestration, agent management, service context, and governance so I can automate delivery without losing control. Inside the platform, I can create services, agents, dashboards, self-service actions, and workflows. The workflow is the backbone of the software factory because it makes the entire path visible and enforceable. Here is the workflow I built for a software factory agentic SDLC: Fetch service context: identify what the selected service is, its ownership, and relevant details.Gather requirements: use a requirements agent to understand the feature request.Create a plan: have a planning agent prepare the implementation approach.Build the change: use a coding agent to implement the requested work.Test and run CI: validate the change through testing and continuous integration.Block failures: stop the workflow immediately if CI fails.Request human review: let a developer approve or reject progression to deployment.Deploy through CD: release the approved change with a continuous delivery agent.Monitor health: inspect the health of the deployed service.Respond to degradation: create incidents, notify the right team, and roll back when required.Collect feedback: send outcomes back into the service catalog and planning context.Use a final deploy gate: keep a human decision point before final release or publishing. I can trigger this software factory through self-service by choosing a service and describing a feature, such as adding an API gateway to a fraud detection service. The workflow begins by retrieving context, then moves through requirements, planning, code generation, testing, CI, review, deployment, and monitoring. I can also trigger the workflow through Port AI. For example, I can request an agentic SDLC pipeline for a service and ask to add OpenTelemetry distributed tracing. The system can locate the service, find the appropriate software factory workflow, trigger it, and provide a live path to track the run. That does not mean the workflow is a black box. I can inspect its stages, check the run state, see whether it is currently planning, coding, or testing, and review the workflow configuration. The software factory becomes both automated and observable. Build Your Own Software Factory A software factory can orchestrate AI agents, workflows, service context, and human approval gates across the SDLC. Start Small, Then Expand the Factory You do not need to automate every part of delivery on day one. A software factory can begin with one valuable and repeatable path. For example, start with requirements, coding, CI, and a human review gate. Once that flow is stable, add deployment automation, monitoring, incident creation, rollback rules, and feedback loops. The goal is not automation for its own sake. The goal is a better system for delivering software: faster where tasks are repetitive, safer where risks are high, and clearer at every stage. A mature software factory gives every agent a defined responsibility, every workflow a visible path, and every human a meaningful control point. That is how I can take advantage of agentic engineering without turning the SDLC into chaos.

By Pavan Belagatti CORE

Java Meets the Spreadsheet Apache POI has been the standard Java library for reading and writing Excel files for over twenty years. It handles the majority of everyday spreadsheet tasks well. But a growing category of real-world Excel files now contains formulas that POI's evaluator cannot execute at all. This is one of several situations Java developers hit when working with spreadsheets that are not obvious until you are already in production. Business users produce, share, and reason about data in spreadsheets. Finance teams model in Excel. Operations teams track inventory in Excel. Analysts hand deliverables to engineering as .xlsx files. Java applications end up interacting with all of it: back-office services accept Excel uploads, pricing engines run calculations that were originally authored in a workbook, reporting tools export data in a format the recipient can open in Excel without formatting problems. Despite how common these situations are, "Java + spreadsheets" is not a topic most developers think about until they hit it for the first time. This article provides a practical overview of the category: common scenarios, moving parts, available approaches, and things that tend to catch teams by surprise. Three Common Scenarios Most Java developers who work with spreadsheets fall into one of three cases. It is worth locating yourself in one of them before evaluating tools. File Exchange (Headless Import and Export) The application reads uploaded Excel files and extracts data, or generates Excel files from database contents. There is no spreadsheet UI in the application itself. This is the most common case. Examples include batch data ingestion, report generation, and integration with third-party systems that expect .xlsx. In-App Calculation (Headless Formula Evaluation) The application uses spreadsheet-style formulas as calculation logic. Business users author pricing rules, tax formulas, or allocation logic in Excel; the Java application executes those formulas at runtime, sometimes against data the users never see. This scenario is less common but appears in fintech, insurance, and enterprise resource planning. In-App Editing (Embedded Spreadsheet UI) The application renders an interactive spreadsheet in the browser, similar to Excel Online. Users view, edit, and collaborate on workbooks inside the application. This is common in reporting tools, financial modeling platforms, and any application where end users need the flexibility of a spreadsheet without leaving the application. The three scenarios have very different technical requirements. A library that fits one may be a poor fit for another. What Working With Spreadsheets Actually Involves Developers often assume spreadsheet integration is primarily about reading cell values. In practice, most production issues arise from features beyond raw data: formula evaluation, formatting fidelity, workbook structure, and modern Excel behavior. File formats: The dominant format is .xlsx (Office Open XML). Older files use .xls (binary). Simpler tabular data is often exchanged as .csv, but CSV loses formulas, multiple sheets, formatting, and cell types. Any real spreadsheet integration has to handle .xlsx. Formulas and formula evaluation: Excel files often contain formulas that reference other cells. Reading the file gives you the formula text and the last cached value. Recalculating the formula requires an evaluator that understands Excel's formula language. Libraries vary widely in which functions they implement. Modern Excel behavior: Excel 365 and Excel 2021 introduced dynamic array formulas, spill behavior, and new functions such as UNIQUE, SORT, FILTER, LET, XLOOKUP, and LAMBDA. In a dynamic array formula, a single cell can produce a whole array of values that "spill" into neighboring cells. For example, =UNIQUE(A1:A100) entered in one cell produces the full list of distinct values from that range and fills as many cells as needed. Files created in modern Excel routinely contain these constructs. Older evaluation engines usually cannot execute them. Cell formatting and styling: Number formats, date formats, colors, borders, conditional formatting, merged cells. This matters both for accurate reading (a value formatted as a percentage means something different from a raw decimal) and for export fidelity. Custom number formats such as accounting-style parentheses for negative numbers, and Excel table styles, are among the formats most likely to be lost or changed on round-trip. Charts, images, and other embedded content: Some libraries preserve these on round-trip; others silently drop them. Data validation, filters, tables, and pivot tables: Structural features that users depend on. Coverage varies significantly across libraries. Not every application needs all of this. A batch job that only reads numeric data from a fixed template needs very little. An application that lets users upload arbitrary workbooks and edit them needs almost all of it. Approaches Available in Java There is no single "Java Excel library." The landscape has several categories, each with its own tradeoffs. Apache POI The de facto standard in the Java ecosystem for headless file processing. Open source, mature, widely used. Supports .xlsx and .xls read and write, and includes a formula evaluator. POI's formula evaluator implements around 250 built-in functions; functions outside that list raise NotImplementedException at evaluation time. Dynamic array formulas and spill behavior are not supported. A minimal POI read example: Java try (Workbook wb = WorkbookFactory.create(new File("data.xlsx"))) { Sheet sheet = wb.getSheetAt(0); Cell cell = sheet.getRow(0).getCell(0); System.out.println(cell.getStringCellValue()); } The boundary is the dynamic array family: SEQUENCE, FILTER, SORT, UNIQUE and TEXTSPLIT raise NotImplementedFunctionException at evaluation time, and spilled ranges have no representation in POI's cell model at all. LET is worse still: POI's formula grammar has no notion of variable binding, so a LET formula cannot even be parsed: Java // Cell A1 contains: =LET(total, SUM(B1:B100), total * 1.1) FormulaEvaluator eval = wb.getCreationHelper().createFormulaEvaluator(); Cell cell = sheet.getRow(0).getCell(0); eval.evaluate(cell); // threw: org.apache.poi.ss.formula.FormulaParseException: // Specified named range 'total' does not exist in the current workbook. The file itself opens without error, and reading the cached value works. It is only when the application needs to recalculate that the problem surfaces. (Verified the code above with POI 5.5.1). Commercial Headless Libraries Products such as Aspose.Cells offer broader formula coverage, better format fidelity, and more complete support for advanced features (charts, pivot tables, formatting). They are usually licensed per developer or per deployment. Teams typically choose these when POI's limitations become blockers and rewriting is not an option. Embedded Spreadsheet Components Products such as Keikai (Java) and SpreadJS (JavaScript) render an interactive spreadsheet UI in the browser and coordinate with the backend. They combine file I/O, formula evaluation, and rendering in a single component. Suitable for applications where end users need to view and edit workbooks directly. Cloud Spreadsheet Services Google Sheets API and Microsoft Graph let the application outsource the spreadsheet entirely and integrate over REST. The spreadsheet lives in the cloud service; the Java application reads and writes through the API. This works well when the workbook itself is the artifact users care about, and less well when the spreadsheet needs to be embedded inside a larger application experience. These categories can also be combined. It is common to use POI for backend generation and a separate embedded component for user-facing editing. Choosing an Approach Match the approach to the scenario. For file exchange, start with Apache POI. It is free, well-documented, and adequate for a large percentage of import/export use cases. Move to a commercial headless library if you hit specific limits: modern formula evaluation, complex formatting fidelity, or performance on large workbooks. For in-app calculation, evaluate the formula coverage of your candidate libraries carefully. If the formulas that need to run come from real Excel files authored by real users, they will include functions that not every engine supports. This is where dynamic arrays and modern functions matter most: a formula containing LET or UNIQUE will not evaluate correctly on a library that does not implement them. For in-app editing, POI alone is not enough because it has no UI. You need either an embedded spreadsheet component that runs in the browser, or a cloud spreadsheet service that you integrate with. The choice depends on how tightly the spreadsheet needs to fit into your application experience, and whether user data can leave your infrastructure. The three scenarios can also stack. A single application might use POI for backend batch ingestion, a headless engine for scheduled recalculation of business rules, and an embedded component for the end-user editing screen. Things That Catch Teams By Surprise A few practical issues that tend to appear later in a project than they should. Formula coverage is not uniform. Two libraries may both advertise "Excel formula support," and both fail on different subsets of real workbooks. Modern functions (UNIQUE, SORT, FILTER, LET, XLOOKUP, LAMBDA) are the most common gap. Verify with your actual files, not with synthetic examples. Dynamic array files behave differently on different engines. A file authored in Excel 365 with =UNIQUE(A1:A100) in one cell may open correctly (showing cached values), fail to recalculate, or throw an exception, depending on the library. If your application needs to recalculate uploaded files, this matters. Cached values can mislead you. When a library cannot evaluate a formula, it often falls back to the cached value stored in the file. This masks the problem during development, because everything looks correct. It only fails when the underlying data changes and the formula needs to be re-evaluated, which is often in production, not in testing. Formatting fidelity varies. Custom number formats, conditional formatting rules, and merged cell behavior are not preserved equally across libraries. If your workbook is going back to Excel users, test the round-trip explicitly with the exact templates your business owners use. Memory and performance scale non-linearly. Loading a 100,000-row workbook is a different problem from loading a 1,000-row workbook. Some libraries hold the entire workbook in memory as a rich object model, and applications typically start hitting issues in the range of tens of thousands of rows. Others offer streaming APIs (POI's SXSSF for write, XSSF event model for read) that trade the object model for scalability. If your use case involves large workbooks, benchmark early. Conclusion Spreadsheets remain one of the most widely used data tools in business, and Java applications increasingly need to interact with them. There is no single correct approach — the right one depends on whether you are exchanging files, running calculations, or embedding a spreadsheet UI. The available options have grown in the last few years, especially for teams that need to handle modern Excel behavior such as dynamic arrays and the newer function set. Understanding the scenarios and the moving parts before picking a library, and testing with the workbooks your real users produce, will save meaningful effort later.

By Hawk Chen CORE

Headless CMS architecture solved a real development problem. It separated content from presentation, gave frontend teams control over frameworks and deployment, and made structured content available to websites, apps, and other channels through APIs. The friction often appears later, when content operations become more complex. Routine publishing changes can still depend on engineering, especially when editors need more control over layout, preview, or page composition. That gap is why some teams consider a different architectural pattern: hybrid headless CMS. It keeps the structured, API-based approach of headless while adding a visual authoring layer to assemble approved components. The Authoring Problem Behind Pure Headless In a pure headless setup, the CMS manages structured content while the frontend controls how that content is rendered. For developers, that separation is valuable. Teams can use React, Vue, Svelte, native applications, or another presentation layer without tying the frontend directly to the CMS. The tradeoff becomes more visible when presentation changes frequently. A CMS may contain a hero title, image, CTA, and product description, but the frontend still determines how those elements become a page. Supporting visual preview, flexible layouts, and reusable page composition can therefore require additional engineering around preview APIs, component mapping, draft rendering, routing, deployment, etc. None of this is inherently a weakness in headless architecture. It is implementation work that teams need to account for. For applications with stable layouts and highly structured content, the model can work extremely well. For enterprises running many sites, markets, and campaigns, the amount of presentation-related work can become an operational bottleneck. When Content Work Becomes Engineering Work The clearest signal is the backlog. Consider a marketing team launching ten regional campaign pages. The content already exists, and no new application behavior is required. But several regions need a different component order, one needs an additional promotional block, and another needs a temporary landing page. In a tightly controlled pure headless implementation, those requests may still require developers to modify templates or component configuration. The workflow can become: Content request → development ticket → code change → review → build → deployment → editor validation That process makes sense when the requested change affects application behavior. It becomes expensive when the request is simply to rearrange approved components. Preview creates a similar issue. Headless systems can support preview, but developers often have to connect draft content with the rendering application so editors can see the actual result before publication. The CMS provides structured data. The frontend provides the presentation context. The distinction matters because the application still owns rendering, routing, accessibility, performance, and browser behavior. MDN provides useful background on the separation between server-side systems and client-facing application behavior. What Hybrid Headless Changes Hybrid headless keeps the API-based content model but adds visual composition capabilities for editors. Instead of letting editors create arbitrary frontend code, developers define the available building blocks. A content team can then assemble approved components through the CMS while the frontend remains responsible for how those components render. For example, developers might provide: HeroProduct gridCustomer quotePricing blockCTAFAQ Editors can change the order or selection of those components without changing the underlying application. The key difference is where composition happens. capabilitypure headlesshybrid headless Structured content Yes Yes API delivery Yes Yes Framework freedom Yes Yes Page composition Usually implemented in frontend logic Can be exposed through CMS authoring tools Visual preview Possible, often requires integration Commonly integrated into the authoring workflow Editor-controlled layouts Depends on implementation Typically a core capability Component governance Application specific Central to the model Definitions vary between CMS vendors, so engineering teams should evaluate the architecture rather than the label. A platform described as hybrid should still expose a clean delivery API that applications can consume independently. If the frontend becomes dependent on proprietary page rendering behavior, teams may reintroduce some of the coupling they were trying to remove. Developers Still Own the Architecture Hybrid headless changes who handles routine page composition, but developers still control the technical boundaries. They define components, validation, accessibility, performance, and application behavior. They also own the delivery contract between the CMS and frontend, including the security implications of new integrations and features. For teams adopting AI-powered capabilities, resources with AI security explained in practical terms can help clarify some of those risks. Overall, that means the architecture still depends on disciplined component and API design. Components that are too rigid send editors back to development tickets. Too many overlapping components create governance problems. The goal is simple: editors control approved composition, while developers retain control over how the application works. When Pure Headless Is Still the Better Fit Pure headless remains a strong choice when presentation is primarily application logic. A product dashboard is a good example. Developers may control nearly every screen because layout, state, permissions, and application behavior are closely connected. Pure headless also fits well when content changes are mostly structured data changes rather than page composition. Typical signals include: A small number of highly custom applicationsStable page structuresLimited need for editor-controlled layoutsContent reused heavily across channelsStrong frontend engineering capacityPresentation decisions that should remain in code In these environments, adding visual composition may introduce complexity without solving a real problem. When Hybrid Headless Becomes More Practical Hybrid approaches become more attractive when content operations generate repeated frontend work. Common signals include: Many sites, markets, or brands using the same component libraryFrequent campaign pagesEditors who need reliable visual previewRegular requests to rearrange approved page componentsEngineering queues filled with presentation changes that contain little new logicTeams that need stronger separation between component development and page assembly A useful test is to pull the previous quarter's engineering backlog and count how many tickets were created primarily to move an existing content block, change a layout, build a campaign page from existing components, or make another presentation change that required no new application behavior. Then look at who filed those tickets. If the same content or marketing teams repeatedly depend on developers for short-lived campaign changes, the organization may need more authoring autonomy rather than more frontend capacity. The Tradeoffs Hybrid Headless Does Not Remove Visual composition shifts work rather than eliminating it. Component governance becomes more important because shared components now act as an interface between engineering and content teams. Someone needs to own versioning, accessibility, documentation, budgets, and backward compatibility. Preview also needs production-quality engineering. A visual editor is useful only when what the editor sees accurately reflects what users will receive. Teams also need to decide how much flexibility to expose. Unlimited layout freedom can create inconsistent pages and undermine a design system. Too little flexibility recreates the ticket backlog the architecture was meant to reduce. The goal is controlled composition. Developers create safe building blocks. Editors assemble them within defined constraints. Evaluate the Workflow, Not the Label The architecture decision should start with the actual publishing workflow. Map who creates content, who changes layouts, who builds components, how preview works, what triggers a deployment, and which requests currently require engineering involvement. Then examine the CMS boundary. Can content be consumed independently through APIs? Can developers control component behavior? Can editors perform routine composition without changing application code? Can teams preview changes accurately? Can the architecture support additional channels without rebuilding the content model? Pure headless and hybrid headless preserve the same core idea: separating content from presentation. The practical difference is how much controlled presentation capability the platform gives back to content teams. For developers, the goal is to keep engineering focused on work that actually requires engineering. If developers are building components, integrations, and application behavior, the architecture is doing useful work. If they are repeatedly moving existing blocks around landing pages, the boundary probably needs another look.

By Alex Vakulov CORE

TL; DR: Tokenmaxxing or Reinventing the Wheel Your organization counts AI tokens, seats, and pilots, but can anyone name a single decision those numbers actually changed? Tokenmaxxing is only the symptom; five old Agile Laws explain the cause, and each one comes with a test you can run this week. There is no need to reinvent the wheel with AI transformations and learn the hard way what the veterans of other transformations already figured out. Thesis: Tokenmaxxing is the vanity metric of pushing low-value work through an AI tool solely to inflate usage metrics. Tokenmaxxing emerged in 2026, when large technology companies began ranking employees by token consumption on internal leaderboards. The behavior is rational for the individual but useless for the organization because tokens measure input rather than outcomes. The five Agile Laws in this article explain why organizations keep making this mistake and what to measure instead. Disclaimer: I belong to those who read Charniak/McDermott’s book on “artificial intelligence” decades ago. Of course, I make use of AI for research, translations, proofreading, challenging story arcs and article structures, and summarization. The Folly of Tokenmaxxing In April 2026, Fortune reported, citing The Information, that an employee at Meta had built an internal leaderboard ranking colleagues by how many AI tokens they consumed, drawing on usage from more than 85,000 employees and displaying the top 250. What we now know as “Tokenmaxxing” was born. The highest-ranked user, in Fortune’s wording, “averaged 281 billion tokens” across the 30-day window. The leaderboard handed out titles: “Token Legend” and “Cache Wizard.” Neither Mark Zuckerberg nor CTO Andrew Bosworth made the top 250. At Amazon, the same failure mode surfaced as employees reportedly pushed low-value work through the company’s agentic tool to inflate their usage. As one employee put it: “Some people are just using MeshClaw to maximize their token usage.” You do not need an AI expert to diagnose that “Tokenmaxxing” is a folly. An economist from 1975 will do the trick. Enter Mr. Goodhart. My point is that every new, possibly paradigm-shifting technology or framework arrives with acolytes who relearn the hard way what previous generations already figured out. It is correct that AI lowers the marginal cost of producing work. However, it does not lower the cost of choosing the right work, integrating it, or being accountable for the result. Many of the enterprise AI failure modes currently filling your LinkedIn feed follow from that gap, and five old laws describe them well enough that we should stop acting surprised; every veteran of “Agile” can tell you instantly that tokenmaxxing is a folly. Goodhart Predicted the Tokenmaxxing Leaderboard in 1975 Charles Goodhart, then an adviser to the Bank of England, wrote that “any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes” in a 1975 conference paper on UK monetary management. He offered it as an aside. A correction while I am here. The sentence everyone quotes, “When a measure becomes a target, it ceases to be a good measure,” is not Goodhart’s. Marilyn Strathern, a Cambridge social anthropologist, wrote it in 1997 on page 308 of a paper about audit culture in British universities, offering it as a paraphrase of Goodhart and citing Keith Hoskin. I have repeated the misattribution myself, including in my earlier article on agile laws. My apologies and a hurrah to the capabilities of research agents! What AI changes: The measure is now generated automatically, continuously, and per person, at a granularity no manager could have collected in 1975. Gergely Orosz reported in April 2026, citing engineers at Salesforce, that internal tools displayed minimum spending targets of $100 for Claude Code and $70 for Cursor, with a Mac widget refreshing every 15 minutes. He also reported an internal Microsoft token leaderboard running since January, where one engineer told him: “I am conscious of not wanting to be seen as ‘uses too little AI,’ and I’m not ashamed to say I need to do tokenmaxxing to do this.” What AI does not change: People optimize for what is counted. Jake Paul, a product and innovation analyst at Kyle and Co, described the mechanism to SHRM in one line: “The path from informal leaderboard to team OKR to formal competency on a performance review is short.” Logan Wolfe, a partner at Kyndryl, told CIO what that produces: “When token usage becomes the KPI, you incentivize output volume over outcomes like efficiency, quality, and risk reduction.” Satya Nadella got the relationship right in Microsoft’s FY26 Q4 earnings release on July 29, 2026, when he said the company is “ensuring every customer can turn tokens into business results.” Tokens are the input. Business results are the outcome. Goodhart enters at the point where an organization measures the first and quietly assumes the second. As we know, that assumption has failed in every “Agile transformation” before; sending as many people as possible to Scrum Master training classes does not automatically improve an organization’s bottom line. Your test: For every AI usage number your organization reports upward, ask the person who reports it which decision the number informs. If nobody can name one, the number is AI adoption theater. The Payoff Arrives After the Redesign Robert Solow named the pattern in 1987, reviewing a book for the New York Times: “You can see the computer age everywhere but in the productivity statistics.” Erik Brynjolfsson, Daniel Rock, and Chad Syverson explain the mechanism. General purpose technologies “enable and require significant complementary investments, including co-invention of new processes, products, business models and human capital.” The organization has to invent the new work before the new tool pays for it, and during the invention phase the numbers look worse rather than better. (That is the famous “J curve” during adoption phases.) Microsoft’s own randomized trial of more than 6,000 workers across 56 firms shows the tool’s early task-level effects, before any documented system-level redesign. Regular Copilot users reduced their weekly time reading email by half an hour, an 18% cut, replied nearly 10% faster (46 minutes), and finished documents nearly a full day sooner. They also created about 11% more Word documents and read 14% more of their colleagues’ documents. Meeting time did not fall. The authors state plainly that they “do not observe anything about the content produced by these workers,” so quality is unmeasured. (All four authors work at Microsoft, which is worth knowing when reading the interpretation.) Read those results together rather than as a scorecard. They are consistent with a rebound effect: as producing a document got cheaper, the same workers produced more documents while also consuming more of their colleagues’ work. The study does not prove that every saved minute created fresh demand. It does show why task-level efficiency cannot be treated as system-level capacity without checking. Coming back to the example I mentioned above: An Agile transformation was declared successful when enough people had attended a two-day Scrum Master course. Two days to understand systems, self-management, organizational change, and how to decide what is worth building. The Scrum Master training was real; I put a lot of work into turning it into a worthwhile experience for the participants. However, the organizational redesign that would have made it useful never happened, because nobody asked for it, and, most of the time, nobody lobbied for it or put it at the top of the leadership agenda. (Remember “We Tried Baseball and It Didn’t Work” on Ron Jeffries’ site?) Copilot licenses are the 2026 equivalent of the Scrum Master class purchase orders back in 2017. Deloitte’s EMEA research from October 2025 shows investment running ahead of returns: 85% of surveyed organizations had increased AI investment over the previous 12 months, 91% planned to increase it again, and 6% reported payback in under a year. That does not prove the investments will fail. My own argument above predicts a lag. It shows why adoption statistics cannot stand in for realized value. Conway Explains Your AI Islands Melvin Conway concluded in 1968 that “organizations which design systems (in the broad sense used here) are constrained to produce designs which are copies of the communication structures of these organizations.” What AI changes: The copies now appear in weeks rather than years, because any department can prototype an assistant without asking anyone. What AI does not change: The communication structure being copied. Each department that runs its own AI initiative produces its own assistant, and none of those assistants talk to each other, because the departments do not talk to each other. The greenfield prototype has no contact with operational reality because the team that built it often has no reporting line to the people who own the process. “AI islands” is Conway’s Law with a 2026 vocabulary. Where practitioners get stuck: The organization responds by establishing a central AI platform team, which can become a bottleneck for every department that wants to ship anything. Conway’s Law offers no escape from itself; it only tells you which structure you are about to reproduce. Your test: Draw your current or planned agent estate as a diagram. If it matches your org chart, your architecture may be accidental, and it is worth asking who chose it. Larman Explains Why AI Became a Tool Rollout Craig Larman’s first law states that “[o]rganizations are implicitly optimized to avoid changing the status quo middle- and first-level manager and ‘specialist’ positions & power structures.” The second predicts that any change initiative gets reduced to redefining or overloading the new terminology to mean basically the same as the status quo. What AI changes: Nothing about this. Larman’s laws are about political economy, and the political economy of a license purchase is identical to that of a training budget. What AI does not change: Which changes are permitted. AI adoption gets reduced to tool rollout because tool rollout threatens nobody’s position on the org chart. “AI transformation” most often comes to mean “we bought Copilot seats.” McKinsey’s November 2025 survey found 88% of organizations using AI regularly in at least one function, and 39% able to attribute any EBIT impact to it, most of them below 5%. That gap is consistent with the missing co-invention Brynjolfsson describes: widespread tool adoption typically comes without comparable evidence of workflow and operating-model redesign. The fourth corollary is the one that should make my own profession uneasy: If managers and specialists are still displaced after the change has itself been changed, they become the coaches and trainers for it. When displaced managers and specialists reappear as prompt-engineering trainers, they occupy the same structural position as the certified agile coaches of 2016. Larman’s fifth law is the exit: Culture follows structure. Your test: Name one consequential decision, handoff, or control point that will change within six months, and name who will own the resulting outcome. Write it down with a date. If neither authority nor workflow constraints move, training and licenses are unlikely to change the culture around the work. Little Explains the AI Pilot Inventory John Little’s queuing formula says that in a stable system, average flow time equals average work in progress divided by average throughput. What AI changes: The cost of starting. A prototype that used to require a budget request now requires an afternoon, so far more work enters the system. What AI does not change: The throughput of everything downstream. One platform team, one security review, one procurement path, one operations group willing to own a new dependency. Release more pilots into a system with fixed exit capacity, and the average lead time rises; it is basic arithmetic. One caution against reading the failure statistics too fast, though: S&P Global research, reported by CIO Dive in March 2025, found that the average organization scrapped 46% of AI proofs of concept before they reached production. A high kill rate can be a sign of health. Killing something quickly is what good experimentation looks like, and the number tells you nothing on its own. Your test: Count three things: Active pilots, transitions to production in the past quarter, and median days from start to a scale-or-stop decision. A large, aging inventory with no decisions attached is a common AI adoption failure mode. A fast kill rate, on the other hand, is not. Brooks Moves, He Does Not Leave Fred Brooks wrote in “The Mythical Man-Month” in 1975 that “adding manpower to a late software project makes it later.” His mechanism was ramp-up time plus communication overhead, since the number of pairwise links grows faster than the headcount does. What AI changes: The social half of that cost. Agents remove most of the onboarding delay. They have no careers to build and no territory to defend, so nobody has to be persuaded that an eleventh agent belongs on the project. What AI does not change: The rest of it, which is the expensive part, opportunity costs considered. For example, loading context is fast. Producing context that is correct, current, and sufficiently bounded is not; consider, for example, data quality. Parallel agents still require task decomposition, shared architectural constraints, state management, integration, and testing, and their outputs still embody architectural choices and assumptions. They do not argue with each other unless specifically instructed, which is another can of worms. Their silence sounds like a saving until two of them proceed from incompatible assumptions, and nobody notices until integration. Code generation scales faster than coherent integration. Every consequential output needs a credible verification path. Tests, rules, and automated comparison cover a good deal of it. For ambiguous or high-impact work, that path still ends in qualified human judgment, and buying more compute does not expand that capacity. How far parallel agents scale before integration cost dominates is unsettled, and I have seen no credible data either way. So Brooks is not repealed, but the unit of analysis moves: Adding agents helps when work can be decomposed cleanly and verification costs less than code generation. Where those conditions fail, the result is a larger queue of unverified work. Your test: Measure your verification capacity, in hours per week and in automated checks you actually trust, alongside your generation capacity in tokens. Then check which of the two you have been buying. Conclusion: Productivity and Transformation Are Different Purchases If your organization switched off every AI tool tomorrow, what would change? Most people answer with tasks: writing would take longer, slides would take longer. That is a legitimate answer, as faster writing is a real gain, and the Microsoft trial measured it. However, if that is the whole answer, you bought productivity tooling. Take the gain, but do not call it an “AI transformation.” Transformation starts when workflows, decision rights, service levels, or operating economics change in a way you can measure, and skipping that step does not exempt you from any of the five laws above. All you did was add a new, possibly paradigm-shifting technology to a legacy organization.

By Stefan Wolpers CORE

Open source projects dominated by a single vendor are a hallmark of "open source in name only." Rather than filling the traditional role of open source fostering innovation and decision-making from a diverse community, "open source in name only" projects are often used as marketing tools for proprietary platforms. These projects are also seen as riskier than community-driven projects because a single vendor is more apt to abruptly terminate long-term support, restrict contributions, or switch from an open-source license to a more restrictive one (forcing some previous contributors to pay for the project they helped build). In these projects, critics claim that investments are often lopsided and heavily skewed toward onboarding, marketing, and brand-related support. As a result, technical contributions are frequently less developed, opaque, undocumented, or lacking in real substance, often manifesting merely as a superficial "ease of entry and onboarding." Because of these underlying gaps in documentation and codebase depth, developers are routinely forced to reverse-engineer functionality simply to get the tools to work correctly. An evaluation of three leading open-source observability projects–OpenSearch, Prometheus, and OpenTelemetry (OTel)– by ReveCom was conducted to determine whether they fell under this vendor-dominated category or are truly vibrant community-led projects. According to Gartner research, these three projects are collectively important because together they provide a complete, vendor-neutral observability architecture covering all three fundamental telemetry signals—metrics, logs, and distributed traces — without locking an enterprise into proprietary agent formats or single-vendor cloud platforms. Gartner defines observability as the extent to which internal system states can be inferred from externally emitted data. By pairing OpenTelemetry as a universal collection and routing tier with Prometheus for real-time metric alerting and OpenSearch for high-volume log analytics and trace analysis, organizations gain end-to-end operational visibility, retain full ownership of their telemetry pipelines, and avoid runaway cloud ingestion or lock-in costs. To develop the framework, data from the ReveCom Observability Report 2026 was used, which includes metrics about contribution numbers and quality, including commit frequency, contributor growth, community expansion, and deployment patterns. Based on this data, authentic efforts were separated from perfunctory efforts. "Authentic" contributions were defined as those made to the computing code (i.e., the observability stack for logs, traces, and metrics) and its computational efficiency as measured in latency. The Controversial Fork AWS's controversial decision to monetize and then fork Elasticsearch to create OpenSearch in 2021 (when Elastic made its license more restrictive) is a case study of the risks associated with vendor-dominated projects. It also serves as an example of the issues associated with a vendor forking and heavily promoting a project it contributed minimally to. According to Elastic representatives, although a major beneficiary of Elastic through its managed service, AWS engineers contributed only a "handful" of commits to Elasticsearch from 2020 to 2021, Elasticsearch says. This disparity suggests that the successor project, OpenSearch, was born from a position of minimal technical familiarity with the core codebase. Elastic famously described this as "there is no compression algorithm for experience." For a technical leader, this lack of pre-fork familiarity suggests a significant "experience gap" that can impact the speed and stability of future feature releases. AWS made few fundamental changes to the Elasticsearch codebase it forked to create OpenSearch, largely just rebranding the existing observability tool. Comparing Three Observability Communities In 2024, Amazon donated OpenSearch to the Linux Foundation, bringing it under a governance structure and setting the stage for it to become a more decentralized project. Among other things, once a project is donated to the Linux Foundation, no single company can hold more than 25% of the seats on the technical oversight bodies. Decentralized governance is structured so that substantive, collaborative contributions from several competing observability vendors can better serve the broader community's needs. Amazon's donation set the stage for OpenSearch to become a much more community-driven effort, comparable to the community-led support of the Prometheus and OpenTelemetry projects. Prometheus and OpenTelemetry exemplify healthy, community-led open source standardization. This is how teams should evaluate open source: by the diversity of the entities with "skin in the game." Prometheus emerged from SoundCloud in 2012, where it was designed to track metrics and store them in a time-series database. Around 2014, Grafana and its glassy, visually appealing panels became part of the ecosystem. The combination of Prometheus and Grafana became an integral, de facto standard for monitoring and observability in Kubernetes deployments and infrastructure. Prometheus was donated to the CNCF in 2016 and graduated in 2018. Since then, it has evolved into a very diverse, community-led project, with multiple contributing companies. Grafana Labs remains one of the largest contributors, but the breakdown of substantive commits-excluding documentation-is wide and varied, reflecting the project's broad, collaborative nature. This wider contribution to the project's standardization ensures that engineering talent is portable and the stack remains interoperable. Separating Brand From Backbone A key open source health metric-perhaps the most substantial of all-is ranking substantive engineering contributions, such as code-level commits and pull requests or high-impact technical commits. These are described as commits that can lead to v1.0, v2.0, or v3.0 milestones, signifying production readiness and improvements. The number and frequency of technical contributions, as measured by commits, are markers for a project's community dynamics and value to end users. Looking at OpenSearch, AWS made significant technical contributions in 2025. As the data shows, Amazon contributes the majority of substantive commits (73%) to OpenSearch. Much of this can be attributed to a surge in contributions related to the AI aspects of observability, specifically "search-to-Al infrastructure" commits. IBM and Red Hat have also contributed AI-related work on RAG and vector database optimization. These are solid contributions, and they show that Amazon has moved beyond the early days, when it simply forked Elastic even though it had contributed relatively little to the project. Hopefully, OpenSearch will continue this shift toward increased community participation as new features are added. However, such a dominant share of commits from a single vendor means that one vendor effectively controls the roadmap. In this case, AWS is potentially prioritizing its managed services over users' infrastructure needs. Source: ReveCom Prometheus has a wide range of contributions from vendor organizations. Grafana is the leading technical contributor to Prometheus, largely based on its development of TSDB storage refactoring, Remote Write 2.0, and agent-mode contributions. Red Hat is the second-most frequent technical contributor to Prometheus, a position solidified by its acquisition of CoreOS. As the primary maintainer of the Prometheus Operator-a critical element for monitoring Kubernetes-Red Hat ensures seamless integration between the monitoring stack and the orchestration layer. While Red Hat provides deep engineering support, Prometheus remains a highly collaborative open-source project with contributions from across the industry. Source: ReveCom The OpenTelemetry project, under the leadership of Splunk, Microsoft, Elastic, Grafana Labs, and Google, provides a mature, stable, and innovative framework for the future of observability. By focusing on high-impact technical commits and "good faith" participation, the community helps ensure that observability data remains a standardized utility that empowers developers and platform engineers to navigate the complexities of the modern cloud landscape. Splunk remains the largest contributor of high-impact technical commits to OpenTelemetry. Grafana is a notable contributor at number four by providing Beyla eBPF instrumentation and Prometheus receiver stability improvements. Strategic Recommendations Organizations should adopt an open-source technical strategy that prioritizes authentic engineering and project diversity. The following recommendations are derived from scrutinizing vendor-dominated projects and analyzing high-impact technical commitments. The high-impact focus of companies like Grafana Labs, Splunk, Microsoft, Elastic, and hundreds of other contributor organizations means that OpenTelemetry and Prometheus should remain the foundation of observability for the next several years. When choosing an observability solution, organizations should prioritize vendors that are not only OTel-compliant but also OTel-contributing. This should also apply to Prometheus solutions, especially those for managing Kubernetes environments. ReveCom's findings indicate that the most valuable contributions are those that advance the core "engine" of observability. Procurement decisions should be based on a vendor's ability to demonstrate substantive engineering that solves real-world infrastructure problems rather than relying on superficial marketing claims. Ultimately, none of the three projects covered in this article can be fully characterized as "open source in name only." While OpenSearch arguably fell into that category immediately after it was forked from Elasticsearch, it has evolved since. OpenSearch remains an Amazon-dominated project, but it has seen an upward trend in contributions from the community and from third parties such as Uber, SAP, and Red Hat. For observability community support, as measured by substantive technical contributions that solve infrastructure problems, OpenTelemetry and Prometheus exemplify a healthy balance of governance and code contributions across hundreds of organizations (notably Grafana and Splunk). Led by Grafana and Splunk among the observability providers, these projects fall behind only Kubernetes itself.

By Chris Ward CORE

In this blog, you will take a closer look at the different exchange types that can be used in RabbitMQ. All are demonstrated by means of examples in a Spring Boot application. Enjoy! Introduction In the previous blog, you learned the basic concepts of RabbitMQ and how to use it in a Spring Boot application. However, you only scratched the surface of it, so now it is time to dig a bit deeper into the different exchange types. If you are not yet familiar with the basic concepts, it is advised to read the previous blog. The official RabbitMQ documentation also provides detailed information that is worth reading. Sources used in this blog can be found on GitHub. Prerequisites Prerequisites for reading this blog are: Basic knowledge of Java;Basic knowledge of Spring Boot;Basic knowledge of Docker Compose;Basic knowledge of RabbitMQ. Topics The code can be found in the topics module. In the previous blog, you created two consumers A and B. Consumer A was bound to Queue A with routing key event.general.*. Consumer B was bound to Queue B with routing keys event.general.* and event.specific.*. The asterisk (*) wildcard was used and is a substitute for exactly one word. In the examples, the routing keys event.general.message and event.specific.message were used. You can also use the hash (#) wildcard, and this is a substitute for zero or more words. This is visualized in the figure below. In the RabbitMqConfig, you declare queue C and bind it to the TopicExchange with routing key event.general.#. Java public static final String QUEUE_CONSUMER_C = "consumer-c.queue"; public static final String ROUTING_KEY_NESTED_GENERAL_MESSAGE = "event.general.#"; @Bean Binding bindingConsumerBSpecific(Queue queueConsumerB, TopicExchange exchange) { return BindingBuilder.bind(queueConsumerB).to(exchange).with(ROUTING_KEY_SPECIFIC_MESSAGE); } @Bean public Queue queueConsumerC() { return new Queue(QUEUE_CONSUMER_C, false); } @Bean Binding bindingConsumerCNestedGeneral(Queue queueConsumerC, TopicExchange exchange) { return BindingBuilder.bind(queueConsumerC).to(exchange).with(ROUTING_KEY_NESTED_GENERAL_MESSAGE); } In the MessageController, you create an endpoint for sending a message with routing key event.general.message.nested. This routing key will not match the bindings of consumers A and B. Java @RequestMapping( method = RequestMethod.POST, value = "send-nested-general" ) public ResponseEntity<Void> sendNestedGeneralMessage(@RequestBody String message) { messageService.sendMessage("event.general.message.nested", message); return new ResponseEntity<>(HttpStatus.CREATED); } The ReceiverC listens to messages received in queue C and prints a message. Java @Component public class ReceiverC { @RabbitListener(queues = RabbitMqConfig.QUEUE_CONSUMER_C) public void receiveMessage(String message) { System.out.println("Queue Consumer C received <" + message + ">"); } } Start the application from within the topics module. Shell mvn spring-boot:run First, post a general message; this should be received by all consumers. Shell curl -X POST http://localhost:8080/send-general \ -H "Content-Type: text/plain" \ -d "This is a general message" In the application console log, you notice that all consumers receive the message. Plain Text Queue Consumer B received <This is a general message> Queue Consumer A received <This is a general message> Queue Consumer C received <This is a general message> Now, post a nested general message, which should be received only by consumer C. Shell curl -X POST http://localhost:8080/send-nested-general \ -H "Content-Type: text/plain" \ -d "This is a nested general message" In the application console log, you notice that the message is only received by consumer C. Plain Text Queue Consumer C received <This is a nested general message> Work Queues The code can be found in the work module. With work queues, you can publish a message and dispatch it to a pool of consumers. One of the consumers will pick up the message and start processing it. This is especially useful for dispatching long-running tasks. You use the default direct exchange in this case, and the queue name is used as the routing key. No need to use a custom exchange. This is visualized in the figure below. The RabbitMqConfig is quite small; you only define the queue. Java @Configuration public class RabbitMqConfig { public static final String QUEUE_TASK = "task.queue"; @Bean public Queue queueTask() { return new Queue(QUEUE_TASK, false); } } When sending a message via an endpoint, you use the queue name as the routing key. Java @RequestMapping( method = RequestMethod.POST, value = "send-work" ) public ResponseEntity<Void> sendWorkMessage(@RequestBody String message) { messageService.sendMessage(RabbitMqConfig.QUEUE_TASK, message); return new ResponseEntity<>(HttpStatus.CREATED); } Every consumer listens to the queue. Java @Component public class ReceiverA { @RabbitListener(queues = RabbitMqConfig.QUEUE_TASK) public void receiveMessage(String message) { System.out.println("Task picked up by Consumer A <" + message + ">"); } } @Component public class ReceiverB { @RabbitListener(queues = RabbitMqConfig.QUEUE_TASK) public void receiveMessage(String message) { System.out.println("Task picked up by Consumer B <" + message + ">"); } } @Component public class ReceiverC { @RabbitListener(queues = RabbitMqConfig.QUEUE_TASK) public void receiveMessage(String message) { System.out.println("Task picked up by Consumer C <" + message + ">"); } } Start the application from within the work module. Shell mvn spring-boot:run Send a message to the queue. Shell curl -X POST http://localhost:8080/send-work \ -H "Content-Type: text/plain" \ -d "This is a work message" The message is processed by one consumer. Plain Text Task picked up by Consumer A <This is a work message> Fanout The code can be found in the fanout module. With fanout, you want to broadcast messages to all queues. You send messages to the exchange, but there is no need to specify a routing key. You can also ensure that temporary queues are used. When temporary queues are used, the queue name will be generated. In the RabbitMqConfig, you define a FanoutExchange. The queues are defined as an AnonymousQueue. This creates a non-durable, exclusive, auto-delete queue with a generated name. You bind the queues to the exchange. Java @Configuration public class RabbitMqConfig { public static final String FANOUT_EXCHANGE_NAME = "fanout.exchange"; @Bean FanoutExchange fanoutExchange() { return new FanoutExchange(FANOUT_EXCHANGE_NAME); } @Bean public Queue queueConsumerA() { return new AnonymousQueue(); } @Bean Binding bindingConsumerA(Queue queueConsumerA, FanoutExchange exchange) { return BindingBuilder.bind(queueConsumerA).to(exchange); } @Bean public Queue queueConsumerB() { return new AnonymousQueue(); } @Bean Binding bindingConsumerBGeneral(Queue queueConsumerB, FanoutExchange exchange) { return BindingBuilder.bind(queueConsumerB).to(exchange); } @Bean Binding bindingConsumerBSpecific(Queue queueConsumerB, FanoutExchange exchange) { return BindingBuilder.bind(queueConsumerB).to(exchange); } } In order to send messages, you only need to send them to the exchange. This can be seen in the MessageService. Java public void sendMessage(String message) { rabbitTemplate.convertAndSend(RabbitMqConfig.FANOUT_EXCHANGE_NAME, "", message); } On the receiving side, you listen to the generated queue name (thus not a specific one in this case). Java @Component public class ReceiverA { @RabbitListener(queues = "#{queueConsumerA.name}") public void receiveMessage(String message) { System.out.println("Queue Consumer A received <" + message + ">"); } } @Component public class ReceiverB { @RabbitListener(queues = "#{queueConsumerB.name}") public void receiveMessage(String message) { System.out.println("Queue Consumer B received <" + message + ">"); } } Start the application from within the fanout module. Shell mvn spring-boot:run Send a message to the queue. Shell curl -X POST http://localhost:8080/send-to-all \ -H "Content-Type: text/plain" \ -d "This is a fanout message" In the application console log, you notice that the message is consumed by all queues. Plain Text Queue Consumer B received <This is a fanout message> Queue Consumer A received <This is a fanout message> RPC The code can be found in the RPC module. Remote Procedure Call (RPC) can be used when you need to execute a function on a remote application and wait for the result. The event is sent to the queue and is processed by Consumer A. The result is sent to a queue in the replyTo field of the request. The publisher waits for data to be returned on this callback queue. When the message appears, it checks the correlationId. If it matches the value of the request, the response is returned to the publisher. All of this is done automatically by the RabbitTemplate. In the RabbitMqConfig, a DirectExchange is used. With a DirectExchange, you match exactly on events; you cannot use wildcards here, just like a TopicExchange. Java @Configuration public class RabbitMqConfig { public static final String QUEUE_CONSUMER_A = "consumer-a.queue"; public static final String DIRECT_EXCHANGE_NAME = "events.exchange"; public static final String ROUTING_KEY_RPC_MESSAGE = "event.rpc"; @Bean DirectExchange eventsExchange() { return new DirectExchange(DIRECT_EXCHANGE_NAME); } @Bean public Queue queueConsumerA() { return new Queue(QUEUE_CONSUMER_A, false); } @Bean Binding bindingConsumerA(Queue queueConsumerA, DirectExchange exchange) { return BindingBuilder.bind(queueConsumerA).to(exchange).with(ROUTING_KEY_RPC_MESSAGE); } } The MessageController contains an endpoint for sending the event. Java @RequestMapping( method = RequestMethod.POST, value = "send-rpc" ) public ResponseEntity<Void> sendRpcMessage(@RequestBody String message) { messageService.sendMessage(message); return new ResponseEntity<>(HttpStatus.CREATED); } In the MessageService, you use convertSendAndReceive and process the response. Java public void sendMessage(String message) { Object response = rabbitTemplate.convertSendAndReceive(RabbitMqConfig.DIRECT_EXCHANGE_NAME, ROUTING_KEY_RPC_MESSAGE, message); if (response != null) { System.out.println("Sender received response: " + response); } else { System.out.println("No response received"); } } In the receiver, you receive the message and send a response. Do note that some additional processing is added in order to trigger a timeout. More on that in a moment. Java @Component public class ReceiverA { @RabbitListener(queues = RabbitMqConfig.QUEUE_CONSUMER_A) public String receiveMessage(String message) { System.out.println("Queue Consumer A received <" + message + ">"); if (message.equals("This is an rpc message")) { return "success"; } else if (message.equals("This is a timeout message")) { try { Thread.sleep(10000); } catch (InterruptedException e) { throw new RuntimeException(e); } return "success"; } else { return "failure"; } } } Start the application from within the rpc module. Shell mvn spring-boot:run Send a message to the queue. Shell curl -X POST http://localhost:8080/send-rpc \ -H "Content-Type: text/plain" \ -d "This is an rpc message" In the application console log, you notice that the message is consumed by consumer A, and that a successful response is received by the publisher. Plain Text Queue Consumer A received <This is an rpc message> Sender received response: success But what if it takes too long to process the message? In real life, the remote application can be unreachable for one reason or another. Send a timeout message. Shell curl -X POST http://localhost:8080/send-rpc \ -H "Content-Type: text/plain" \ -d "This is a timeout message" In the MessageService, the response will return null, and a timeout exception is raised. Plain Text Queue Consumer A received <This is a timeout message> No response received 2026-04-25T14:50:16.785+02:00 WARN 482297 --- [MySpringRabbitMqPlanet] [pool-2-thread-8] o.s.amqp.rabbit.core.RabbitTemplate : Reply received after timeout for 2 2026-04-25T14:50:16.785+02:00 WARN 482297 --- [MySpringRabbitMqPlanet] [pool-2-thread-8] s.a.r.l.ConditionalRejectingErrorHandler : Execution of Rabbit message listener failed. org.springframework.amqp.rabbit.support.ListenerExecutionFailedException: Listener threw exception at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.wrapToListenerExecutionFailedExceptionIfNeeded(AbstractMessageListenerContainer.java:1795) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:1687) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.actualInvokeListener(AbstractMessageListenerContainer.java:1612) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:1599) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:1590) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.executeListenerAndHandleException(AbstractMessageListenerContainer.java:1539) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.executeListener(AbstractMessageListenerContainer.java:1520) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.DirectMessageListenerContainer$SimpleConsumer.callExecuteListener(DirectMessageListenerContainer.java:1206) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.DirectMessageListenerContainer$SimpleConsumer.handleDelivery(DirectMessageListenerContainer.java:1163) ~[spring-rabbit-4.0.2.jar:4.0.2] at com.rabbitmq.client.impl.ConsumerDispatcher$5.run(ConsumerDispatcher.java:149) ~[amqp-client-5.27.1.jar:5.27.1] at com.rabbitmq.client.impl.ConsumerWorkService$WorkPoolRunnable.run(ConsumerWorkService.java:111) ~[amqp-client-5.27.1.jar:5.27.1] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090) ~[na:na] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614) ~[na:na] at java.base/java.lang.Thread.run(Thread.java:1474) ~[na:na] Caused by: org.springframework.amqp.AmqpRejectAndDontRequeueException: Reply received after timeout at org.springframework.amqp.rabbit.core.RabbitTemplate.onMessage(RabbitTemplate.java:2721) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.DirectReplyToMessageListenerContainer.lambda$setMessageListener$0(DirectReplyToMessageListenerContainer.java:93) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:1683) ~[spring-rabbit-4.0.2.jar:4.0.2] ... 12 common frames omitted 2026-04-25T14:50:16.790+02:00 ERROR 482297 --- [MySpringRabbitMqPlanet] [pool-2-thread-8] .l.DirectReplyToMessageListenerContainer : Failed to invoke listener org.springframework.amqp.rabbit.support.ListenerExecutionFailedException: Listener threw exception at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.wrapToListenerExecutionFailedExceptionIfNeeded(AbstractMessageListenerContainer.java:1795) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:1687) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.actualInvokeListener(AbstractMessageListenerContainer.java:1612) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:1599) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:1590) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.executeListenerAndHandleException(AbstractMessageListenerContainer.java:1539) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.executeListener(AbstractMessageListenerContainer.java:1520) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.DirectMessageListenerContainer$SimpleConsumer.callExecuteListener(DirectMessageListenerContainer.java:1206) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.DirectMessageListenerContainer$SimpleConsumer.handleDelivery(DirectMessageListenerContainer.java:1163) ~[spring-rabbit-4.0.2.jar:4.0.2] at com.rabbitmq.client.impl.ConsumerDispatcher$5.run(ConsumerDispatcher.java:149) ~[amqp-client-5.27.1.jar:5.27.1] at com.rabbitmq.client.impl.ConsumerWorkService$WorkPoolRunnable.run(ConsumerWorkService.java:111) ~[amqp-client-5.27.1.jar:5.27.1] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090) ~[na:na] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614) ~[na:na] at java.base/java.lang.Thread.run(Thread.java:1474) ~[na:na] Caused by: org.springframework.amqp.AmqpRejectAndDontRequeueException: Reply received after timeout at org.springframework.amqp.rabbit.core.RabbitTemplate.onMessage(RabbitTemplate.java:2721) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.DirectReplyToMessageListenerContainer.lambda$setMessageListener$0(DirectReplyToMessageListenerContainer.java:93) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:1683) ~[spring-rabbit-4.0.2.jar:4.0.2] ... 12 common frames omitted How to solve this? In this case, you are better off using the AsyncRabbitTemplate. This template is not automatically autowired, so you have to define it as a bean. Let's do so in the RabbitMqConfig. Java @Bean public AsyncRabbitTemplate asyncRabbitTemplate(RabbitTemplate rabbitTemplate) { return new AsyncRabbitTemplate(rabbitTemplate); } In the MessageController, you define an endpoint to trigger the async template. Java @RequestMapping( method = RequestMethod.POST, value = "send-async" ) public ResponseEntity<Void> sendAsyncMessage(@RequestBody String message) { messageService.sendAsyncMessage(message); return new ResponseEntity<>(HttpStatus.CREATED); } In the MessageService, you autowire the AsyncRabbitTemplate. And because it is an async call, you catch the response by means of a CompletableFuture. Java public void sendAsyncMessage(String message) { CompletableFuture<Object> future = asyncRabbitTemplate.convertSendAndReceive(RabbitMqConfig.DIRECT_EXCHANGE_NAME, ROUTING_KEY_RPC_MESSAGE, message); future.thenAccept(response -> { if (response != null) { System.out.println("Sender received response: " + response); } else { System.out.println("No response received"); } }); } Start the application from within the rpc module. Shell mvn spring-boot:run Send a message to the queue. Shell curl -X POST http://localhost:8080/send-async \ -H "Content-Type: text/plain" \ -d "This is a timeout message" In the application log, you see the same result: the response is null, but no timeout exception anymore. Conclusion In this post, you learned different exchange types. Each serves its own use case. It is up to you to choose the right pattern for your use case.

By Gunter Rotsaert CORE

Most Docker content targets web developers shipping stateless services. However, data engineers, who represent a huge and growing population of Dockers users, are mostly left to figure things out alone, and it shows. The get pipelines that pass locally, but explode on clusters. They pit notebook-only development against expensive cloud workspaces, and more. This article applies six years of production data platform experience in financial services and healthcare to a question nobody answers well: How to you make a laptop behave like a lakehouse? A Familiar Routine If you build data pipelines for a living, you've lived this story. Your PySpark job runs perfectly in a cloud notebook. You productionize it, push it through CI, deploy it to the cluster, and it fails. A dependency mismatch. A different Spark minor version. A Delta Lake protocol feature your local wheel doesn't know about. A timezone default nobody set. Web developers solved "works on my machine" a decade ago with containers. Data engineers, somehow, are still developing against shared cloud workspaces, paying per-minute cluster costs to debug a GROUP BY, and discovering environment drift in production. This article is the workflow I wish someone had handed me years ago: a fully containerized lakehouse development environment — Spark, Delta Lake, object storage, a catalog, and orchestration — that runs on a laptop, mirrors production closely enough to trust, and plugs into CI without mocks. The Real Problem: Data Pipelines Have Four Environments, Not One A typical stateless web service has one environment to reproduce: the app runtime. A data pipeline has at least four, and they drift independently: The compute runtime — Spark version, Scala version, JVM, Python, native libs (Arrow, Parquet, libhdfs).The table format layer — Delta Lake / Iceberg versions and protocol versions, which are not the same thing.The storage layer — S3/ADLS semantics: multipart uploads, eventual consistency quirks, path-style vs virtual-hosted access.The orchestration layer — the scheduler's Python environment, which is famously not your job's environment. Mocking any one of these in tests means you aren't testing the thing that breaks. The goal of containerizing a lakehouse is to pin all four layers in code and version them together. Step 1: A Reproducible Spark Image You Actually Control Don't develop against latest. Build a base image that pins every layer of the compute runtime and treat it like an artifact: Dockerfile # syntax=docker/dockerfile:1.7 FROM eclipse-temurin:17-jre-jammy AS base ARG SPARK_VERSION=3.5.4 ARG DELTA_VERSION=3.3.0 ARG HADOOP_AWS_VERSION=3.3.6 RUN apt-get update && apt-get install -y --no-install-recommends \ python3.11 python3-pip tini && \ rm -rf /var/lib/apt/lists/* # Pin Spark itself, not just PySpark RUN curl -fsSL https://archive.apache.org/dist/spark/spark-${SPARK_VERSION}/spark-${SPARK_VERSION}-bin-hadoop3.tgz \ | tar -xz -C /opt && mv /opt/spark-${SPARK_VERSION}-bin-hadoop3 /opt/spark ENV SPARK_HOME=/opt/spark PATH=$PATH:/opt/spark/bin PYTHONHASHSEED=0 TZ=UTC # Delta + S3 connectors resolved at build time, never at job submit time RUN /opt/spark/bin/spark-shell --packages \ io.delta:delta-spark_2.12:${DELTA_VERSION},org.apache.hadoop:hadoop-aws:${HADOOP_AWS_VERSION} \ -e "println(\"deps cached\")" && \ cp /root/.ivy2/jars/*.jar /opt/spark/jars/ COPY requirements.lock /tmp/ RUN pip install --no-cache-dir -r /tmp/requirements.lock # Never run Spark as root RUN useradd -m -u 1001 spark USER 1001 ENTRYPOINT ["/usr/bin/tini", "--"] Three details that matter more than they look: --packages at build time, not submit time. Resolving connector JARs at spark-submit is the #1 source of "it worked yesterday" failures — Maven Central is a runtime dependency you didn't mean to have.PYTHONHASHSEED=0 and TZ=UTC kill two classes of "non-deterministic only in prod" bugs.A lockfile, not requirements.txt. Compile with pip-compile or uv pip compile so transitive dependencies (looking at you, pandas/pyarrow) can't drift. Step 2: The Lakehouse-In-A-Box With Docker Compose Here's the part most teams never build: the rest of the lakehouse, locally. MinIO stands in for S3 (it speaks the same API), and a real Spark master/worker pair stands in for the cluster, because local[*] mode hides every serialization and shuffle bug you'll meet in production. Dockerfile # compose.yaml services: spark-master: build: . command: /opt/spark/sbin/start-master.sh environment: [SPARK_NO_DAEMONIZE=true] ports: ["7077:7077", "8080:8080"] spark-worker: build: . command: /opt/spark/sbin/start-worker.sh spark://spark-master:7077 environment: - SPARK_NO_DAEMONIZE=true - SPARK_WORKER_MEMORY=4g - SPARK_WORKER_CORES=2 depends_on: [spark-master] deploy: replicas: 2 # >1 worker = real shuffles, real serialization minio: image: minio/minio:RELEASE.2025-09-07T16-13-09Z command: server /data --console-address ":9001" environment: MINIO_ROOT_USER: localdev MINIO_ROOT_PASSWORD: localdev-secret ports: ["9000:9000", "9001:9001"] volumes: [lake-data:/data] healthcheck: test: ["CMD", "mc", "ready", "local"] interval: 5s mc-init: # create the bronze/silver/gold buckets on boot image: minio/mc:latest depends_on: { minio: { condition: service_healthy } } entrypoint: > /bin/sh -c "mc alias set local http://minio:9000 localdev localdev-secret && mc mb -p local/lakehouse/bronze local/lakehouse/silver local/lakehouse/gold" volumes: lake-data: Point Spark at MinIO with three config lines and your medallion pipeline reads and writes s3a://lakehouse/... paths exactly like production: Python spark = (SparkSession.builder .config("spark.hadoop.fs.s3a.endpoint", "http://minio:9000") .config("spark.hadoop.fs.s3a.path.style.access", "true") .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog") .getOrCreate()) docker compose up and you have bronze → silver → gold on your laptop. Total cloud cost of a debugging session: $0. Step 3: Integration Tests That Run Real Spark — Testcontainers The payoff of all this is CI you can trust. With Testcontainers, your pipeline tests spin up the same images your developers use: Python import pytest from testcontainers.minio import MinioContainer from pyspark.sql import SparkSession @pytest.fixture(scope="session") def lake(request): with MinioContainer("minio/minio:RELEASE.2025-09-07T16-13-09Z") as minio: yield minio def test_silver_dedup_keeps_latest_record(lake, spark): # write duplicate customer events to bronze bronze_path = f"s3a://test/bronze/customers" write_fixture_events(spark, bronze_path, duplicates=True) run_silver_dedup(spark, bronze_path, "s3a://test/silver/customers") result = spark.read.format("delta").load("s3a://test/silver/customers") assert result.count() == EXPECTED_UNIQUE assert latest_record_wins(result) No mocked DataFrames. No unittest.mock.patch("boto3..."). The test exercises Delta's actual transaction log against actual object storage. When this suite is green, deployments stop being scary. A pattern I use in regulated environments: keep a fixtures/ directory of small, synthetic Parquet files that mirror production schemas (never production data), and version them with the code. Schema drift then fails a unit test instead of a 2 a.m. pipeline run. Step 4: One Image From Laptop → CI → Production The final principle: the image you test is the artifact you ship. Multi-stage builds let one Dockerfile serve dev (with Jupyter, debuggers) and prod (minimal, non-root): Dockerfile FROM base AS dev USER root RUN pip install --no-cache-dir jupyterlab pytest debugpy USER 1001 FROM base AS prod COPY --chown=1001:1001 src/ /app/src/ COPY --chown=1001:1001 jobs/ /app/jobs/ # nothing else — no notebooks, no test deps, no shell tools you don't need In CI: build once, tag with the git SHA, run the Testcontainers suite against prod, scan it (Docker Scout, or your registry's scanner), sign it, and promote that exact digest through staging to the scheduler. Whether the scheduler is Airflow's DockerOperator/KubernetesPodExecutor or a managed Spark platform pulling custom containers, the principle holds: environments are immutable, versioned, and identical by construction. Lessons Learned From Production Run ≥2 workers locally.local[*] mode never serializes between JVMs. The day you switch to a real cluster, every closure-capture and UDF-pickling bug appears at once. Two 2-core workers in Compose surfaces them on day one.Pin the table format protocol, not just the library. Delta and Iceberg both evolve table protocol versions. A newer writer can produce tables an older reader can't open. Encode the protocol version in your image build args and test reads with the oldest reader you support.MinIO is a stand-in, not a clone. It won't reproduce S3 request throttling or cross-region latency. Keep a small smoke-test suite that runs against real object storage nightly; do everything else locally.Resource-limit your local Spark. Without SPARK_WORKER_MEMORY caps, a skewed join will cheerfully eat your laptop. Limits also force you to think about partitioning early — which is the point.Treat the orchestrator's image as layer four. Airflow DAG-parse environments drift too. Containerize the scheduler with the same lockfile discipline as the jobs. Production Considerations Before you take this pattern to a real platform team, three things to plan for: secrets (local Compose uses throwaway creds; production should inject via your cloud's secret manager or Docker secrets — never baked into images), image provenance (sign images and generate SBOMs in CI; regulated industries will ask, and in 2026 the tooling is mature enough that "we didn't get to it" no longer flies), and base image hygiene (start from minimal, hardened bases and rebuild on a schedule, not just on code change — CVEs don't wait for your sprint). Conclusion Containers gave application developers reproducibility ten years ago. Data engineering is finally having the same moment — and the teams that containerize their lakehouse development loop ship faster, test honestly, and stop paying cloud bills to find typos. Try it: clone the Compose stack above, point your gnarliest pipeline at it, and see what breaks locally that used to break in prod. Then tell me about it — I'd genuinely like to hear which layer drifted on you. If this was useful, follow me here and on LinkedIn. Next up in this series: load-testing Delta merge performance locally, and contract testing between pipeline stages.

Columnar engines naturally organize computation around vectors to make effective use of single instruction, multiple data (SIMD) instructions. This makes vectors first-class citizens in such engines. The difficult design question appears when an engine's internal application programming interface (API) must be exposed to users: where should programming happen? A native C API is sufficient for embedding, and many engines stop there. Building a complete analytical database, however, requires a full-featured language for programming on top of the engine. There are well-known options: SQL - solves the relational part of this problem, but a database runtime may also expose direct vector calculations, object creation, user-defined lambdas, OS integration, orchestration, control flow, graph algorithms as well as generic-purpose programming, not only analytic queries. Implementing all of these in SQL would produce a standalone dialect besides the relational language.Embed an established scripting language - it immediately solves lambdas and control flow, but it also introduces a second runtime. Thus the language design problem is larger than just query syntax: "How can a columnar engine expose its native data types and operations without restricting users to C, duplicating those values in another runtime or hiding expressions from the engine optimizer"? Rayfall provides a concrete case study. It is an expression language in which native values, vector operations, user-defined lambdas, and relational queries are the natural language components. Its syntax is well-known S-expressions, which bring powerful mechanisms as well as a dramatically simple parser, which is mandatory in the case of real-time requests that require fast responses thus wasting time in parser would be irrational. Requirements RequirementConsequenceNative valuesLanguage values are engine objects rather than wrappers.Array semanticsOperations work on atoms, vectors, and table columns.General programming The language needs user defined lambdas support, control flow, error handling, debug info, stack unwinding.First class queriesQuery expressions must be fully visible to the optimizer.IPC/SerializationThe language should naturally support remote execution as well as local without introducing separate mechanisms or 3rd party protocols. Architecture Rayfall uses S-expressions: Clojure (+ 2 3) (* (+ 2 3) 4) (sum [10 20 30]) The parser does not produce any AST; rather, it produces an evaluation tree immediately where each object is the same struct with a type tag, reference counter, and payload. (...) - is a List[...] - homogeneous vector{...} - dictionary"..." - string literal[0-9]* - number[a-zA-Z]* - symbol(fn [args] (body)) - lambda A dictionary is just a two array of the same length: keys and values. Thus, a table is just a flipped dict where keys are table column symbols, values are lists of column vectors. This is how, for example, Q language acts. Another advantage of an S-expressions-based language is equivalence of code and data: Clojure (+ 1 2) ; an executable list (quote (+ 1 2)) ; the same structure treated as data [AAPL MSFT NVDA] ; a typed symbol vector {from: trades where: (> price 100)} ; a dictionary containing expressions Unification Let's consider multiplication. Clojure (* 12.5 4) ; scalar call (* [12.5 20.0 8.0] 4) ; vector - scalar (select {from: trades notional: (* price qty)}) ; table columns While the expression surface stays the same, the execution strategy changes depending on argument types. This is the core idea behind Rayfall. Vector operations are not a library lying on top of a low-level API, and query expressions are not strings passed to a separate SQL frontend. They are the same language expressions interpreted in the same context. Queries Consider the following simple table: Clojure ‣ (set trades … (table [sym time side price qty] … (list … [AAPL AAPL MSFT MSFT NVDA AAPL] … [09:30:00.000 09:30:30.000 09:31:00.000 … 09:31:30.000 09:32:00.000 09:32:30.000] … [BUY SELL BUY SELL BUY BUY] … [100.0 101.0 400.0 399.0 170.0 102.0] … [150 50 20 30 100 200]))) ┌──────┬──────────────┬──────┬───────┬─────┐ │ sym │ time │ side │ price │ qty │ │ SYM │ TIME │ SYM │ F64 │ I64 │ ├──────┼──────────────┼──────┼───────┼─────┤ │ AAPL │ 09:30:00.000 │ BUY │ 100.0 │ 150 │ │ AAPL │ 09:30:30.000 │ SELL │ 101.0 │ 50 │ │ MSFT │ 09:31:00.000 │ BUY │ 400.0 │ 20 │ │ MSFT │ 09:31:30.000 │ SELL │ 399.0 │ 30 │ │ NVDA │ 09:32:00.000 │ BUY │ 170.0 │ 100 │ │ AAPL │ 09:32:30.000 │ BUY │ 102.0 │ 200 │ ├──────┴──────────────┴──────┴───────┴─────┤ │ 6 rows (6 shown) 5 columns (5 shown) │ └──────────────────────────────────────────┘ Columns are just regular vectors that can be extracted and passed to ordinary functions: Clojure ‣ (set prices (at trades 'price)) [100.0 101.0 400.0 399.0 170.0 102.0] ‣ (set quantities (at trades 'qty)) [150 50 20 30 100 200] ‣ (* prices quantities) [15000.0 5050.0 8000.0 11970.0 17000.0 20400.0] And the result is another typed vector: Clojure [15000.0 5050.0 8000.0 11970.0 17000.0 20400.0] There is no query yet. Multiplication simply lifts its scalar behavior over two equally sized vectors. Broadcasting is straightforward: Clojure (* prices 1.05) Now place the multiplication inside select: Clojure ‣ (select {from: trades sym: sym price: price qty: qty notional: (* price qty)}) ┌──────┬───────┬─────┬────────────────┐ │ sym │ price │ qty │ notional │ │ SYM │ F64 │ I64 │ F64 │ ├──────┼───────┼─────┼────────────────┤ │ AAPL │ 100.0 │ 150 │ 15000.0 │ │ AAPL │ 101.0 │ 50 │ 5050.0 │ │ MSFT │ 400.0 │ 20 │ 8000.0 │ │ MSFT │ 399.0 │ 30 │ 11970.0 │ │ NVDA │ 170.0 │ 100 │ 17000.0 │ │ AAPL │ 102.0 │ 200 │ 20400.0 │ ├──────┴───────┴─────┴────────────────┤ │ 6 rows (6 shown) 4 columns (4 shown)│ └─────────────────────────────────────┘ Here the expression itself has not changed. What changed is name resolution. Inside query, price, and qty are column names. Before any query evaluation, the engine mounts columns to their names as a regular environment frame, naturally making all existing expressions work the same way as they do in a non-query context! Once a query is passed to a select function, it becomes an operation graph and, when its shape is supported, a fused DAG pipeline. Otherwise, it falls back to a regular operators defined as a language primitives. This opens a door to composing vector operations inside a predicate: Clojure ‣ (select {from: trades … where: (and … (> (* price qty) 10000.0) … (in sym [AAPL MSFT])) … sym: sym-name … notional: (* price qty)}) ┌──────┬──────────────────────────────┐ │ sym │ notional │ │ SYM │ F64 │ ├──────┼──────────────────────────────┤ │ AAPL │ 15000.0 │ │ MSFT │ 11970.0 │ │ AAPL │ 20400.0 │ ├──────┴──────────────────────────────┤ │ 3 rows (3 shown) 2 columns (2 shown)│ └─────────────────────────────────────┘ Here is a short breakdown: `(* price qty)` produces a floating-point vector.`>` converts it into a Boolean vector. `(in sym [AAPL MSFT])` produces another Boolean vector.`and` combines the predicates. The entire expression is reused as a projection. At the language level, this follows the same composition rules as: Clojure (and (> (* prices quantities) 10000.0) (in (at trades 'sym) [AAPL MSFT])) But inside the pipeline, the DAG optimizer can fuse or rewrite expressions without changing semantics. From the user's point of view nothing changed. Such homoiconical behavior allows the use of any vector operations existing in the language inside queries. Consider the following example: Clojure ; xbar rounds values down to a fixed bucket. Applied directly to a time vector ‣ (xbar … [09:30:12.000 09:30:48.000 09:31:05.000] … 60000) [09:30:00.000 09:30:00.000 09:31:00.000] ; The same operation can define a group key ‣ (select {from: trades … by: {minute: (xbar time 60000)} … trades: (count qty) … volume: (sum qty) … vwap: (/ (sum (* price qty)) … (sum qty))}) ┌──────────────┬────────┬────────┬────────┐ │ minute │ trades │ volume │ vwap │ │ TIME │ I64 │ I64 │ F64 │ ├──────────────┼────────┼────────┼────────┤ │ 09:30:00.000 │ 2 │ 200 │ 100.25 │ │ 09:31:00.000 │ 2 │ 50 │ 399.4 │ │ 09:32:00.000 │ 2 │ 300 │ 124.67 │ ├──────────────┴────────┴────────┴────────┤ │ 3 rows (3 shown) 4 columns (4 shown) │ └─────────────────────────────────────────┘ This example reveals several execution levels without changing languages: `xbar` transforms a vector into a grouping key.`*` derives a vector consumed by `sum`.`sum` reduces values per group.`-` folds aggregate results into a scalar per group. In a system with separate array and query languages, these often require different syntax, a user-defined function boundary, or intermediate materialized columns. Here they remain one expression tree. Lambdas The integration becomes even more interesting when the expression is named: Clojure ; define user function: ‣ (set trade-value … (fn [price quantity] … (* price quantity))) lambda ; It can be called with ordinary vectors: ‣ (trade-value … (at trades 'price) … (at trades 'qty)) [15000.0 5050.0 8000.0 11970.0 17000.0 20400.0] ; And the same function can be called with query columns: ‣ (select {from: trades … where: (> (trade-value price qty) 10000.0) … sym: sym-name … value: (trade-value price qty)}) ┌──────┬──────────────────────────────┐ │ sym │ value │ │ SYM │ F64 │ ├──────┼──────────────────────────────┤ │ AAPL │ 15000.0 │ │ MSFT │ 11970.0 │ │ NVDA │ 17000.0 │ │ AAPL │ 20400.0 │ ├──────┴──────────────────────────────┤ │ 4 rows (4 shown) 2 columns (2 shown)│ └─────────────────────────────────────┘ For a lowerable single expression lambda, the query compiler reduces the call into the operation graph. Actual arguments are compiled once and referenced by offsets, so using a formal parameter more than once shares the corresponding subexpression rather than rebuilding it. And this is interesting, because Rayfall is not only giving familiar names to built-in query operations. User-defined lambdas can participate in queries as well as any other language expressions. Layout Internally, a Rayfall builtin is a runtime object. Even more, any datatype, including scalar, vector, dict, table, function, or builtin, is the same ray_t struct with type tag, reference counter, and payload union. This allows implementing a simple and efficient buddy allocator that operates on ray_t blocks; even more on-disk data is exactly the same as in-memory, so the runtime doesn't care about actual object allocation and allows lazy mmaping of huge datasets seamlessly. For standalone vectors, the atomic dispatcher can itself build a small operation graph and execute it via the vector engine. For unsupported shapes, it retains a typed per-operation path. And this happens seamlessly, not exposed to a user Compiler Rayfall has two relevant compilation paths. Ordinary user-defined functions compile lazily into bytecode for a stack VM. The bytecode handles local slots, calls, control flow, recursion, traps, and returns. If compilation can not handle a form, execution can fall back into the recursive evaluator and vice versa. Query expressions take another route. The query layer attempts to lower an expression into a typed operation DAG: Literals become constant nodes.Column names become scan nodes.Arithmetic and comparisons become typed operations.Supported lambdas are beta-reduced.Aggregations become reduction nodes.Structural clauses add filters, groups, projections, sorts, and limits. The DAG then passes through type inference, constant folding, predicate and projection pushdown, filter reordering, partition pruning, and dead-code elimination. The Bottom Line Rayfall can be understood as an attempt to close the space between a low-level C API and a high-level query interface. The resulting language is a LISP-like one, with some extensions like first-class homogeneous vectors, dictionaries and tables. The parser, evaluator, and query compiler discussed here are available in the GitHub repo.

Software engineers often feel most comfortable with hard skills, such as writing code, studying frameworks, experimenting with databases, reviewing architectures, or building side projects. As engineers advance into leadership roles, technical depth becomes even more critical. Technical leaders must guide decisions that impact other engineers, teams, and sometimes entire platforms. To do this effectively, strong communication and influence are essential, but they must be grounded in sound technical judgment. Without sufficient technical depth, leadership can steer teams in the wrong direction. Open source is especially valuable in this context. Mature projects expose engineers to challenges rarely found in tutorials or new applications, such as software evolution, legacy modernization, backward compatibility, architectural trade-offs, design decisions, documentation, code reviews, and the internals of widely used frameworks. Open source also offers opportunities to learn from experienced engineers worldwide and to observe how complex technical decisions are made. This article will explore how open source helps Software Engineers develop the hard skills needed to become both better developers and stronger technical leaders. Why Technical Leaders Still Need Hard Skills Leadership in software engineering does not require knowing everything. The technology landscape is vast, systems are complex, and specialization is deep. However, a technical leader must have a solid technical foundation to exercise sound engineering judgment. They should be able to communicate effectively with engineers, understand core software design and architecture concepts, recognize key trade-offs, and know which questions to ask when solutions are unclear. Without this foundation, effective leadership is challenging. Discussions about scalability, consistency, coupling, performance, security, or maintainability can be difficult to follow if the leader does not understand the team's language. The goal is not to be the top specialist in every area, but to have enough context to distinguish meaningful concerns from unnecessary complexity, recognize when further investigation is needed, and help the team progress when discussions stall. Technical Judgment Helps Teams Move Forward Engineering discussions do not always converge naturally. Two experienced engineers may advocate different architectural approaches, each with valid arguments. A migration can involve several strategies. Teams may struggle to decide whether to introduce a new service, adopt a different database, or continue investing in the current system. A technical leader must help navigate these situations. This does not mean making decisions alone. Effective leadership involves asking insightful questions, clarifying assumptions, identifying missing information, running experiments, or helping the team focus on the most important trade-offs. Technical knowledge makes that possible. Without technical knowledge, a leader risks relying on the most confident voice in the room. The Cost of a Bad Decision Grows With Your Scope The higher you progress in a technical or executive career, the larger the potential consequences of your decisions become. A software engineer may make an implementation decision that affects a feature. A staff engineer can influence several teams. A principal engineer may shape a platform used organization-wide. A VP of Engineering or CTO can approve a technical direction that impacts hundreds of engineers and years of investment. As your scope increases, so does the cost of mistakes. Technical knowledge cannot eliminate failure. Architecture involves uncertainty, and even skilled engineers make decisions that later prove incorrect. However, stronger technical judgment reduces the likelihood of avoidable mistakes and helps leaders identify risks earlier. This is why technical depth remains important, even as you write less code. Technical Knowledge Keeps You Connected to the Team Hard skills also help leaders stay connected to the engineers doing the work. A technical leader should be able to join design discussions, understand why the team struggles with integration, follow the impact of legacy constraints, and recognize when tasks that seem simple at the management level are actually complex to implement. That connection matters. When engineers feel a leader understands their work, communication improves. It becomes easier to discuss risks, challenge unrealistic expectations, and translate technical constraints for other parts of the organization. This does not mean micromanaging implementation or overriding specialists. On the contrary, strong technical knowledge helps leaders know when to contribute, when to ask questions, and when to trust the experts. The goal of hard skills in leadership is not technical dominance, but better judgment, communication, and decision-making at scale. How Open Source Builds Technical Leadership Skills Open source is valuable for developing technical skills because it exposes engineers to real systems under real-world conditions, often evolving over many years with decisions made publicly. You see more than just the final code. Many projects allow you to review issues, discussions, rejected alternatives, pull requests, review comments, and the tests that justified changes. For those building technical leadership, this experience develops both implementation skills and sound judgment. Learn Design From the People Who Built the Technology A key advantage of open source is the ability to learn design decisions directly from the project’s creators. Rather than just reading how to use a framework, you can examine why abstractions exist, how APIs evolved, which alternatives were rejected, and what constraints shaped the design. Comparing the public API with its implementation reveals where convenience, performance, compatibility, and maintainability may conflict. This offers a deeper level of learning. You move beyond learning how to use the technology to understanding how its creators approach software design. And eventually, if you contribute long enough, you stop being only an observer and start participating in those decisions yourself. Learn How Software Survives for Decades Many engineers primarily work on relatively new systems. Mature open-source projects introduce a different challenge: evolving software that cannot be easily rewritten. Some open-source technologies have existed longer than many software companies. This involves managing backward compatibility, deprecated APIs, refactoring, migration paths, legacy design decisions, performance requirements, security issues, and users who rely on behaviors that were never intended to be permanent. Here, legacy modernization becomes a tangible challenge. You learn that modernization rarely means replacing everything with the latest architecture. It is typically about advancing a system without disrupting existing users. For technical leaders, this lesson is fundamental. Most architectural work occurs within existing systems, not on a blank slate. Learn Quality Through the Cost of Regression Open-source projects also make the cost of regression very visible. A seemingly minor change can disrupt another operating system, database, integration, or an unforeseen use case. This requires mature projects to be disciplined in testing, compatibility, review, and release processes. Participation in these projects leads to a deeper understanding of software quality. Tests are not there simply to increase coverage. They protect behavior. Code review is not mere bureaucracy. It helps prevent individual misunderstandings from causing widespread issues. Backward compatibility is not resistance to innovation; it is often a contractual obligation to users. These lessons translate directly into technical leadership because leaders are responsible not only for introducing change but also for understanding its consequences. Learn From Some of the Best Engineers in the Industry Open source also removes a significant barrier: organizational boundaries. Within a company, you typically learn from colleagues. In open source, you collaborate with contributors from companies, universities, foundations, and communities worldwide. This allows you to review code from highly experienced engineers, observe their problem-solving approaches, receive feedback, and sometimes discuss technical decisions directly with the creators of widely used technologies. Such access is rarely available elsewhere. Because discussions are often public, you can learn even without direct participation. Carefully reading complex design discussions can provide deeper architectural insights than studying lists of patterns. Understand the Internals of the Tools You Depend On There is a significant difference between knowing how to use a framework and understanding its inner workings. Examining a project's internals reveals the constraints that shape its behavior. Why does this API behave this way? Why is this operation expensive? Why does this abstraction leak under certain conditions? Why was a seemingly obvious feature rejected? This deeper knowledge enhances your ability to debug, design, and make architectural decisions. For technical leaders, this is important because decisions often occur at a level above the application code's abstraction. Understanding underlying mechanisms helps you assess when a framework is suitable, where its limitations lie, and when it is being misapplied. Open-Source Practices Can Scale Beyond Open Source Many organizations now actively seek to replicate these practices internally. There is even a term for it: InnerSource. The goal is to adopt practices from successful open-source communities within the company, such as transparent development, shared ownership, cross-team contributions, visible discussions, reusable components, documented decisions, and review processes that facilitate knowledge sharing. This is important because a major risk in software organizations is knowledge becoming isolated within organizational structures. A team may possess valuable knowledge, but if code, decisions, and practices remain confined, the broader organization cannot benefit. Open-source-style collaboration helps reduce this dependency. In some cases, an engineer contributing a few hours a week to a widely used project can impact more software systems than months of work within a single company. This is not necessarily due to greater skill, but because the work is visible, reusable, reviewable, and accessible to a larger community. This is a key leadership lesson: impact depends not only on the amount of work you do, but on how effectively your knowledge scales beyond yourself and your immediate team. Conclusion Hard skills are essential for technical leadership. As software engineers advance to roles such as staff engineer, principal engineer, software architect, or technology executive, their decisions have a broader organizational impact, making technical judgment critical. Open source offers an ideal setting to develop this judgment by exposing engineers to real-world software evolution, legacy modernization, architecture, design, testing, framework internals, code review, and the insights of experienced engineers worldwide. It teaches not only how to write software, but also how to evolve, protect, and guide it over time. Open source is often described as a philosophy of collaboration, freedom, transparency, and knowledge sharing. While these values are important, open source extends beyond philosophy. Mature open-source ecosystems are among the most advanced software engineering environments, where technologies used by millions are designed, reviewed, tested, maintained, and improved publicly. For software engineers seeking technical leadership, participating in this environment is an excellent way to strengthen hard skills, refine judgment, and learn how impactful software is built and maintained.

By Otavio Santana CORE