Methodologies

"Vibe coding" tweaking a prompt, running it once, and seeing if it looks okay does not scale for enterprise software. Here is how to build a rigorous verification pipeline to audit, bench, and evaluate your Claude agent's behavior over time. If you are building autonomous agents with the Claude API, you have likely experienced the trap of "vibe coding." It usually goes like this: you write a prompt, give Claude access to a tool, run a single test execution in your terminal, and watch it succeed. You think you're ready for production. Then, you deploy. Within hours, a customer inputs an unexpected edge case, Claude gets trapped in an infinite tool-calling loop, consumes 5 million tokens, and fails the task entirely. As the software development lifecycle shifts toward long-running autonomous workflows, engineers must stop evaluating agents like chat logs and start treating them like production software systems. Moving an agentic system from an experimental script to enterprise-grade software requires a deterministic engineering framework: an Automated Evaluation (Evals) Loop. The Core Architecture of an Agentic Eval Loop Unlike traditional software test suites that evaluate a single inputs-to-outputs assertion, agentic evaluations are fundamentally trajectory-based. Your evaluation infrastructure must run the agent through a stateful "agent loop," collect its execution steps, capture its tool requests, and grade the final environmental impact. Step 1: Building a Rigorous Evaluation Dataset An effective eval suite doesn't require thousands of abstract test cases to start. The absolute best way to begin is by curating 20 to 50 complex tasks directly inspired by real-world user failures, support tickets, and edge cases. A production-grade eval dataset item requires three concrete pillars: The User Intent Prompt: An open-ended instruction containing real-world noise or partial context.The Initial System State: A clean configuration file, a localized repository footprint, or a mock database snapshot that resets before every run.The Gold Standard Reference Solution: The unambiguous target state that confirms success. Avoid vague task criteria. Vague metrics generate noisy, inconsistent evaluation data. Vague Task Spec (Prone to Failure) "Look at the customer account records, find the ones with high spending, and generate an alert script." Unambiguous Task Spec (Production-Grade) JSON { "task_id": "mcp_analytics_042", "intent": "Parse the CSV located at /data/q2_raw.csv. Identify all client IDs whose cumulative transaction value exceeds $50,000. Write an executable python script at /scripts/alerts.py that formats these IDs into a clean JSON list.", "environment_setup": "copy_fixture('q2_raw_unfiltered.csv', '/data/q2_raw.csv')", "evaluation_criteria": { "type": "unit_test_and_state_verification", "target_file": "/scripts/alerts.py", "expected_output_contains": ["10425", "10982", "11034"] } } By explicitly stating target file paths, expected data keys, and environment variables, you ensure the agent fails because its reasoning broke, not because the evaluation test harness itself was poorly specified. Step 2: Utilizing a "Reviewer" Claude Agent for Quality Control Not every agentic outcome can be evaluated by a binary file assertion or a hardcoded regex pattern. If your production agent generates human-facing code documentation, structures a complex customer email response, or proposes an architecture blueprint, verifying correctness requires qualitative reasoning. To handle this at scale without manual human review bottlenecks, deploy a separate "Reviewer" Claude Agent to act as a structured quality control judge (often called an LLM-as-a-Judge architecture). Python import anthropic def evaluate_agent_trajectory(task_intent, final_output, execution_log): client = anthropic.Anthropic() # Use a reasoning-optimized model for evaluation, like Claude 3.5 Opus response = client.messages.create( model="claude-3-5-opus", max_tokens=2000, temperature=0.0, # Lock down stochastic variation system="You are an expert Quality Assurance Judge. Your task is to evaluate an agent's trajectory against a true user intent.", messages=[ { "role": "user", "content": f""" ### CRITERIA FOR SUCCESS The agent's final text summary must address the core issue, maintain professional tone guidelines, and explicitly note any API errors encountered. ### ORIGINAL USER INTENT {task_intent} ### AGENT TRAJECTORY (LOGS) {execution_log} ### FINAL OUTPUT GENERATED BY PRODUCTION AGENT {final_output} Analyze the trajectory step-by-step. Output a JSON object containing your 'reasoning' string, an explicit 'score' integer from 1 to 5, and a binary 'pass_verdict' boolean. """ } ] ) return response.content Critical Rules for Model-Based Grading Isolate your models: Never use the exact same agent system prompt or model instance to grade its own output.Enforce zero temperature: Set your grading agent's temperature to 0.0 to maximize consistency across identical test cycles.Provide negative anchor examples: Give your Reviewer Agent concrete examples of what a "Fail" or "Partial Pass" looks like in its system instructions to anchor the scoring boundaries. Step 3: Tracking Production Metrics That Matter To successfully benchmark your system modifications over time, stop relying on subjective impressions and track three critical system performance indicators across every execution run: 1. Task Completion Success Rate (pass@1) The total percentage of test evaluations where the agent successfully reaches the objective on its first complete run. If you run multiple iterations to account for variance, map the divergence carefully. A sharp drop in your pass@1 metrics combined with high variance is a direct indicator of brittle system instructions or ambiguous tool documentation. 2. Tool Execution Accuracy Track how accurately Claude invokes your functions against your schemas. Calculate these two sub-metrics: Tool call precision: The number of valid tool敲 invocations divided by the total tool attempts made by Claude. A lower score indicates Claude is hallucinating parameter properties or passing corrupted syntax values.Redundant loop count: The number of times Claude executes the exact same tool with the exact same inputs consecutively. High redundancy means your system isn't feeding errors back into the context correctly, leaving the agent trapped in a loop. 3. Comprehensive Token Cost Accounting An agent that completes a task successfully but takes 120 sequential steps and handles 4,000,000 raw input tokens might be too slow and financially expensive to deploy to production. Track the full consumption curve across your evaluation runs: Test Run IDModel VersionSuccess RateAvg. Agent Turn StepsTotal Input TokensTotal Output TokensFinancial Cost / Runv1.0-baselineClaude 3.5 Sonnet74%8.2 turns340,00022,000$1.35v1.1-fixed-toolsClaude 3.5 Sonnet92%4.1 turns185,00011,500$0.71v2.0-heavy-reasoningClaude 3.5 Opus96%3.9 turns420,00038,000$3.20 Synthesizing Your Metrics into Actionable Systems Engineering Building an evals loop alters your entire day-to-day workflow. When you update tool definitions, rewrite an orchestration script, or test a brand-new model variation, you no longer guess if the system improved. You simply run your evaluation test runner, observe the changes across your dashboard, and deploy with confidence. Stop vibe coding. Build a robust, data-backed evaluation loop today, and ensure your Claude-powered agentic systems remain stable, efficient, and aligned at enterprise scale.

Large-scale cloud platforms have reached a level of complexity — spanning multi-region Kubernetes clusters, streaming systems like Kafka, and heterogeneous data stores — that often exceeds human cognitive limits. Failures are no longer isolated events; they are emergent behaviors arising from tightly coupled systems where issues propagate across layers such as networking, orchestration, and data pipelines. Even with modern observability stacks, operators must manually correlate signals across dashboards, making incident response slow, inconsistent, and cognitively taxing. Traditional approaches rely heavily on static runbooks and tribal knowledge. These mechanisms do not scale in modern distributed systems. Agentic AI introduces a fundamentally different paradigm. Rather than merely detecting anomalies (as in traditional AIOps), agentic systems use Large Language Models (LLMs) to reason, plan, and act. These systems can iteratively generate hypotheses, validate them using real data, and execute multi-step remediation workflows. The result is not just faster detection, but a closed-loop system capable of autonomous diagnosis and recovery. This article expands on how to architect a production-grade SRE agent that can safely and effectively automate cloud incident response. The system is organized into three layers: Perception (data ingestion), Cognition (multi-agent reasoning), and Action (guarded execution), all operating over a shared knowledge graph. Establish a Cloud Knowledge Graph At the core of any intelligent SRE agent is context. Raw telemetry alone is insufficient; the system must understand how components relate to each other. This is achieved through a domain-specific cloud knowledge graph. The graph models: Nodes: Services, pods, clusters, regions, gateways, Kafka topics, and databasesEdges: Traffic flows, deployment relationships, data lineage, ownership, and failover pathsAttributes: SLOs, capacity limits, configuration history, and prior incidents This structure transforms observability data into a causal reasoning substrate. Instead of treating metrics independently, the agent can traverse dependencies and infer propagation paths. For example, a spike in API latency can be traced through upstream gateways to downstream services and eventually to a throttled database. This graph is not static — it evolves continuously with infrastructure changes and incident learnings. Over time, it becomes a living system model enriched with historical context, enabling better hypothesis generation and faster root-cause analysis. In practice, maintaining graph freshness is critical. You should integrate it with service registries, deployment pipelines, and configuration management systems to ensure it reflects real-time topology. Build the Perception Layer (Observability Pipeline) The Perception Layer acts as the sensory system of the agent, continuously ingesting telemetry across the stack. This includes: Metrics: CPU, memory, I/O, network utilization, Kafka consumer lagLogs: Structured and semi-structured application and infrastructure logsTraces: End-to-end request paths across microservices However, raw ingestion is only the first step. The real value lies in transforming this data into structured, actionable signals. A stream-processing pipeline should: Normalize data across heterogeneous sourcesDetect anomalies using statistical methods and thresholdsEmit structured events tied to entities in the knowledge graph These events act as triggers for the Cognition Layer. Importantly, they should already be enriched with context (e.g., “Service A in region us-east-1 exceeds latency SLO”), reducing the reasoning burden on downstream agents. A critical design consideration is balancing sensitivity and noise. Excessive alerting leads to “signal overload,” a well-known issue where operators — and agents — struggle to prioritize meaningful events . Techniques such as event deduplication, correlation, and temporal aggregation are essential to ensure high-quality inputs. Architect a Multi-Agent Cognition Layer Instead of using a single massive prompt, build a Cognition Layer utilizing a multi-agent LLM architecture (using GPT-5 or Claude-Opus class models) orchestrated by a control plane (e.g., a serverless orchestration layer). Assign specialized roles to different agents: Detector Agent: Monitors the anomaly events and groups related alerts into candidate incidents based on the knowledge graph's dependency structure.Hypothesis Agent: Proposes potential root causes by analyzing the graph and recent telemetry data.Validator Agent: Acts as the investigator by issuing targeted queries back to the observability tools and cloud APIs to confirm or reject the hypotheses based on hard evidence.Planner Agent: Synthesizes an actionable remediation plan. This plan should be an ordered list of operations, complete with preconditions, postconditions, and explicit rollback triggers.Critic (Governance) Agent: Reviews the remediation plan against organizational safety policies before execution, ensuring constraints are not violated. Implement a Guarded Action Layer The Action Layer is what separates an active agent from a passive AIOps recommendation engine. It executes the Planner Agent's steps via the Kubernetes API (scaling, restarting pods) and Cloud Provider APIs (toggling failovers, adjusting traffic weights). Safety is paramount. You must wrap this layer in a strict governance framework: Enforce hard limits on scaling factors and failover scopes.Implement canary rollouts, applying changes to a single zone before expanding.Build auto-rollback mechanisms that trigger immediately if Service Level Objectives (SLOs) deteriorate after an action.Require explicit human-operator approval for high-risk operations like region-wide failovers. Rollout and Optimization Strategies When deploying your SRE agent, start in a "shadow" or assist mode. Allow the agent to observe incidents, propose hypotheses, and draft plans while human operators retain full control and execute the final decisions. As confidence in the system grows, gradually grant it autonomy for low-risk, routine actions. To manage operational costs and latency: Optimize prompts: Externalize static system descriptions into retrieved documents.Caching: Cache intermediate inferences for reuse across similar recurring incidents.Batching: Batch non-urgent tool calls and defer low-impact infrastructure checks to background tasks. Conclusion Agentic AI represents a shift from reactive monitoring to proactive, autonomous operations. By combining a real-time observability pipeline, a continuously evolving knowledge graph, and a multi-agent reasoning system, you can build an SRE agent capable of end-to-end incident management. Using this framework can significantly reduce Mean Time To Recovery, improve root-cause accuracy, and decrease reliance on human escalation — all while maintaining strict safety guarantees. More importantly, these systems create a virtuous cycle: every incident enriches the knowledge graph, improves agent reasoning, and strengthens operational resilience. As cloud systems continue to grow in complexity, agentic SRE architectures will likely become a foundational component of modern reliability engineering.

Switching from one single sign-on (SSO) vendor to another is a complex process that involves more than just changing technologies. This is a high-stakes identity operation that impacts security, user experience, following the rules, accessing applications, and keeping things running smoothly. It's not the same as moving a reporting tool or a collaboration platform because SSO is at the front door of every application in your environment. If you set it up wrong, everything will stop working. But the biggest danger of SSO migrations is not that they won't work. The little things that go wrong are the most annoying Users being locked out of apps that are important to the businessAccounts being left alone that were never deprovisionedMFA enrollments disappearing without a word and Helpdesk queues are getting longer on the morning of cutover because there was no communication about the change. This guide discusses the best ways to move to cloud SSO and the most important things to keep in mind. It discusses everything from getting the identity estate ready for the move of integrations to phased rollout strategies, making the user experience as smooth as possible, and planning for MFA migration. Why Businesses Change SSO Providers Companies don't usually change their SSO platforms on a whim. One of the following things usually makes it happen: Acquisition of a vendor or announcement of the end of a product's life. Cost consolidation or figuring out how to use enterprise licenses. Standardizing platforms under a broader cloud strategy. Requirements for compliance or regulation that the current business can't meet. Issues with scalability, performance, or missing features in the current platform.A merger or acquisition that introduces a second identity domain. Whatever the reason, migration causes compounding risk since SSO is foundational infrastructure, not an individual application. 3 Types of Migration Approaches and Their Differences There are three main ways to move to SSO, and each one has its risks and effects on governance. Federated Protocol Swap Retain the same IdP architecture but replace the vendor platform underneath. For example, moving from PingFederate to Entra ID External Identities. The protocol (SAML, OIDC, SCIM) may remain the same, but attribute mappings, claim transformations, and session behaviors differ in ways that are often not clear until something breaks in production. Full IdP Replacement The old IdP is completely removed, and a new one is put in its place. Need to set up, test, and cut over every connection with a service provider (SP) again. This type has the most risk, and it's also the one that most businesses don't consider. Consolidation Migration A single authoritative platform brings together many IdPs. Such an event can happen when companies merge or acquire another. There are technical and organizational problems, such as different business units having different app owners, SLAs, and levels of tolerance for disruption. Governance alignment needs to happen before any technical work can begin. Migration Process: The 7 Steps Audit and clean upPlan and PrepareMFA MigrationCommunication PlanningPhased RolloutGovernance ConsiderationDecommission and close out Step 1: Audit and Clean up Most organizations rush, ignore, and migrate everything, including unused applications, inactive users, orphaned accounts, and integrations that have remained unused for three years. These don't break, but leave a security risk. Following validations reduces testing and inventory. Create a complete, clean list of applications: Validate against the CMDB or application catalog.Validate apps being used.Validate access logs from SIEM.Validate against IGA platforms.Reduce redundant applications. Create a complete, clean list of valid users: Active users.Exclude accounts with no activity for 90 days. Exclude dormant accounts whose passwords were never changed.Validate against IGA platforms and HR systems. Mark the unused applications for the decommissioning process. Note down the protocols used (SAML, OIDC, WS-Federation, or legacy), application owners, attributes and claims, MFA requirements, CA policies, and session time-out configurations. Step 2: Plan and Prepare Every application that relies on SSO consumes identity attributes passed in SSO protocols. New IdPs rarely use the same attributes and often have case-sensitive and format changes. These mismatches cause silent authentication failures and will be extremely difficult to diagnose during cutover. Application Metadata Prepare the claims transformation registry. Confirm the case and formats.Validate transformation rules. Redirect URLs For each application, configure a transparent redirect from the legacy IdP login URL (or intranet homepage) to the new IdP's login endpoint. The user will not experience major changes. The only change a user would notice would be the new MFA prompt. Rollback Process Identify when you should roll back.Who will be able to make the rollback decision? Rollbacks generally occur in the following use cases: The rate of successful authentications drops below 95%.Validate SSO failures for major applications.More calls to the help desk than usual during the first 2 days of migration. Migration go-live Documentation regarding new login flow end-to-endPlan for extended staff during the migration. Validate helpdesk access to the new platform.Identify and set up escalation contacts for issues that the helpdesk cannot resolve. Step 3: MFA Migration Prepare a complete inventory of existing MFA enrollments that includes How many users have MFA enrolled vs. password only? What factors are in use? Authenticator Apps – Need to re-enrollSMS – Same phone number and email can be used. Hardware token – FIDO2/WebAuthn keys can be reused if the new vendor supports itBiometrics – Need to re-enroll.How many and which users have only a single factor enrolled? Follow the steps for re-enrollment: Open the self-service enrollment portal.Phone numbers and emails can be reused (since they remain the same).Send advance communications at least two weeks out, explaining what will change and why.Track re-enrollment completion rates by department and group.Send follow-up emails, including deadlines.Set up a plan to re-enroll privileged accounts. Step 4: Communication Plan Communication is a major step in the migration process and should be tracked as a separate workstream, treated with its timeline, owners, deadline, and success metrics. There are three different audiences involved in SSO migration. End users who simply need to know what will change and what to do.Helpdesk and IT staff who need operational readiness confirmations.Stakeholders who need status updates and risk visibility. Major email templates include: General UpdatesMFA-Enrollment NoticesCut Over Day notification Step 5: Phased Rollout Never perform a cutover for the entire organization. Instead, choose a phased rollout. This reduces risk, helps validate configurations in production with real users and real traffic, and provides time to identify issues before affecting most of the organization. First Phase—Technology users Internal IT staff.Identity administrator.Helpdesk personnel.power users.Second Phase - High-frequency application users like ERP applications CRM applications Collaboration platform BI toolsThird Phase—General user population Lower-risk departmentsExceptions and low-activity users ContractorsUsers who log in very lessThird-party users Step 6: Governance Considerations To ensure successful migration and validations, consider the following governance aspects: Changes to IGA Solutions JML changes Provisioning accounts in IDP with required attributes for SSO claims.Disabling or deletion of accounts during terminations.User transfers: changes to account attributes and group memberships.Changing birthright roles Update with new SSO groups.Cleanup of legacy vendor applications. Audit Log Monitoring Onboard logs from new vendor to SIEMSet up alerts for notifications, including Authentication failuresCA policy failuresPassword failuresToken expiration Non-Human Identities Create a separate inventory of NHA accounts and migrate their credentials to the new system. These include accounts with no owners. Step 7: Decommission and Close Out The process can move forward once all the checks are done and the MFA enrollments are at acceptable levels. Monitor the new system for 30 days and plan for the decommissioning of the old SSO solution. Conclusion SSO is the authentication layer for all the applications in the organization. Performing migration without a proper plan includes risk. Most companies follow one or a combination of the above-described approaches. Adhering to a proper plan with communication and the right strategies will never make you think about rollback strategies.

Teams often say they are building one app. A lot of the time, that is not true. I saw this while reviewing a telemedicine MVP. At first, the plan sounded simple enough: video visits, messaging, scheduling, and basic records. Then the version-one list kept growing: Patient appprovider dashboardAdmin panelMessagingVideoBillingEHR connectionDevice support later At that point, this was no longer one app. It was several systems being planned as one MVP. A patient-facing productA provider-facing productAn admin productA set of outside-service connections When a team treats all of that like one first release, things get messy before development even starts. The Moment It Stopped Being One App The problem was not the number of screens. The problem was the number of users, roles, and data rules hiding behind those screens. A patient needed intake, booking, reminders, and follow-up. A provider needed schedules, patient context, notes, and quick actions during the day. An admin needed visibility, support tools, and role controls. The outside-services side added video vendors, messaging vendors, EHR work, and, later, device data. That is not one product. That is a group of different systems with different jobs. Once that became obvious, the planning changed. Split the Product by User First Before estimating anything, it helps to split the product by who it is for. For this telemedicine project, the first useful split looked like this: 1. Patient Side This part handled: IntakeBookingRemindersFollow-up messagingJoining a visit The patient's side had to stay simple. It also had to be clear about what the patient could and could not see. 2. Provider Side This part handled: Schedule viewPatient detailsVisit notesQuick responsesRole-based access This was not just a different set of screens. It had different speed needs, different daily habits, and different data access rules. 3. Admin Side This part handled: Role setupSupport actionsVisibility into operationsReportingNon-clinical controls Admin work often looks small during planning. In real projects, it adds a lot of rules and a lot of testing. 4. Outside-Service Work This part handled: Video vendor setupMessaging vendor setupEHR-related workFuture device dataLogging and audit-related movement of data This is where many teams get surprised. Video, messaging, and EHR are not tiny add-ons. Each one brings its own work. Start With Access Rules Before the Feature List In multi-role products, one of the quickest ways to find hidden work is to define access rules early. Before locking the feature list, ask: Who can create this dataWho can read itWho can change itWho can delete itWho can export it For the telemedicine project, this made a big difference. A few features looked simple in the scope doc. Once the team asked who could view or change the related data, the work got much larger. A basic example: Admins can help fix booking problems. That sounds harmless. But then the real questions start: Can admins see messages?Can they see visit notes?Can they see call history?Can they open uploaded files? That one sentence can change a big part of the system. Access rules often show hidden work much faster than a feature list does. Treat Outside Services as Separate Work Another mistake teams make is treating outside services like small items on a checklist. On paper, it can look like this: VideoMessagingEHR later In practice, each one adds its own work: Vendor setupRequest and response formatsError handlingRetry rulesLoggingReplacement cost if the vendor needs to change later That is why these items should be planned separately. For the telemedicine case, once video, messaging, and EHR work were split out from the main product list, the first release became easier to define. Some items that seemed close to launch were clearly not ready for version one. Ship One Complete Path First Once the team stopped calling everything an MVP, the first release got smaller. The version-one path that stayed in looked like this: Patient intakeAppointment bookingSecure video through the chosen vendorFollow-up messagingBasic provider access controls That was enough to test whether the product solved a real problem for a clinic. What moved out of the first release: Deeper EHR workMore reportingDetailed billing flowsDevice supportBroader admin tooling Those things were not bad ideas. They just did not belong in the first build. 4 Simple Documents to Create Before Sprint Planning When a team starts to suspect that one MVP is several systems, four short documents can help a lot. 1. User-to-System Map List each part of the product and the main user for it. 2. Permission Matrix Write down who can create, view, change, delete, and export each type of data. 3. Outside-Service List Separate core product work from vendor work and data that moves in or out of the system. 4. First-Release Path Write the one end-to-end path that version one has to get right. These are short documents, but they make planning much better. Why This Matters Outside Healthcare, Too This lesson is not only for telemedicine. It applies to any multi-role product where the team is building for more than one type of user. That includes: Customer apps with admin panelsSaaS products with back-office toolsPlatforms with provider and client sidesProducts that depend on outside vendors from day one The moment a team has different users with different goals, the work stops being “just one app.” Final Point A lot of MVPs get too big because teams keep calling them one product long after that stops being true. The fix is not always better estimates. Sometimes the fix is much simpler: Split the product by user.Write down the access rules.Separate outside-service work.Ship one complete path first. That makes the first release easier to plan, easier to build, and easier to test.

Editor’s Note: The following is an article written for and published in DZone’s 2026 Trend Report, Platform Engineering and DevOps: How Internal Platforms, Developer Experience, and Modern DevOps Practices Accelerate Software Delivery. The role of the enterprise developer has become more complex over time as organizations adopt new technologies and tools, often without retiring their old ones. Add high staff turnover and increasing time and cost pressure, and developers are confronted with charting their own path through the SDLC. The purpose of internal developer platforms (IDPs) is to create a win-win scenario that benefits developers and their organizations. In this tutorial, you’ll define one golden path for a backend service that covers service setup, deployment, observability, and guardrails end to end. Step 1: Define the Platform Product and First Golden Path Successful IDP efforts focus on end-to-end developer workflows: building a new interface, deploying an updated microservice, running a regression suite, or standing up an environment. Ideally, the whole workflow can be supported directly from your IDP as self-service. Once you have identified the workflow to support, you need to design the “golden path,” which parts you will standardize and what you expose as configuration. It’s important to get that balance right. Components that have to change often, like service accounts, interfaces, and sizing, should be configurable. Creating templates and patterns helps reduce variability between outputs, making it easier to roll out necessary patching and updates. For the first golden path, pick one high-value workflow that is common, repeatable, and easy to measure. We will use the deployment of our backend service to an integration test environment because it touches build, deployment, validation, and evidence capture in one flow. User adoption is the key to success. To measure, it’s important to track both user adoption, such as how often a workflow is triggered, and outcome metrics like the number of compliant application instances, percentage of deployment failures, and average deployment duration. Step 2: Design the Golden Path (Templates and Defaults) Next, we get to design the golden path. An important factor for the developer experience is to provide documentation with contextual guidance. This can be traditional how-to guides or more advanced features such as AI-enabled chatbots. The documentation should explain how testing, application deployments, and other lifecycle activities happen along the golden path, and provide architectural guidance on embedding any newly developed capability in the existing architecture. Standards and governance are other aspects that should be available for self-service, including naming conventions, common libraries, and reusable services. On the technical side, the golden path should cover at least the following: Code repo and standard branching structureSkeleton code based on coding standards (e.g., environment config file, logging framework, data layer)CI/CD pipeline into an ephemeral cloud environment, or pointed at a standard persistent dev environmentSkeleton quality gates in the CI/CD pipeline (e.g., unit test, functional regression, security scan)Access to common utilities; injection of environment values (e.g., URLs, IP addresses, access and secrets management)Ability to spin up the environment (if cloud based) And lastly, the IDP needs to be designed with intuitive naming, a search function, tagging methods, and a hierarchical browsing structure so users can easily find the appropriate golden path. Supporting multiple ways of discovery provides a more resilient interface and eases the adoption of new golden path templates as they become available. For our backend service, choosing the workflow will show a representation of the steps included. Step 3: Wire Self-Service Workflows (Without Tickets) Besides golden path templates, IDPs should aim to be a one-stop shop for developers, so common requests should be available for self-service. Your existing ticket/ITSM systems can be a good source for creating the backlog. Identify the most common requests and start automating them in priority order. In many cases, a ticket continues to be useful even in the self-service model for tracking and approvals, which can be integrated into the automatic workflow. Approvals should be provided automatically based on defined criteria, and only require human approvals when the request is outside of those parameters, such as access to restricted data, use of expensive resources, and non-standard requests. Over time, developers should be able to request new features through a transparent feature backlog and voting mechanism to engage the community. When creating new features, keep things common wherever possible and provide ways for users to tailor their requests. For example, the standard deployment process might define a step for secrets injection, but some teams will tailor the process to skip it as necessary. This approach has two advantages: It creates a common language and process across teams and reduces the work to build and maintain the IDP. Spending a bit more time up front to create customizability pays off over the medium and long term. For our backend service, the first service we define is deployment to the integrated test environment. Step 4: Standardize Delivery With CI/CD + GitOps + IaC in One Flow The principle of the golden path deployment process remains unchanged: You build a software artifact once, and you deploy it multiple times along the environment path. For our backend service, promotion should happen through a versioned change (think GitOps) to the desired environment state, so application version, infrastructure definition, and deployment evidence remain traceable together. In the build stage, code is prepared in any pre-compile steps, then compiled and packaged with all necessary configuration files. In the deployment process, environment variables are injected, and the package is deployed to the target environment, which is scripted as Infrastructure as Code. The validation itself is usually layered: a technical validation to confirm that the deployment was correct, functional regression of core functionality, and testing the new changes. This sequence is based on speed of feedback, which is important in an automated IDP service. When a validation check fails, the golden path needs to have defined failure behavior with clear steps to execute. Pipeline failures like a broken build, failed test, or policy violation will block progression automatically. If the environment is materially impacted, a rollback is automatically initiated. Only in rare cases should a human evaluation be required — for example, when the level of ambiguity is too high and impacts stakeholders who are using the environment. Some policy violations can be treated with time-bound exceptions, such as allowing a new security vulnerability in a non-production environment. This allows functional testing to continue while the team remediates the security vulnerability. Prior to going live, the exception would be removed so the security vulnerability doesn’t progress to production. These types of exceptions should be set to auto-expire to prevent them from being forgotten later. Golden Path Steps and Guardrails stepself-service actionguardrailevidence Build Trigger pipeline via check-in action in source control Code scan and unit test results Build log, composition scan result Promote to non-prod environment Merge to staging branch, promotion request Technical validation, regression test Test results Promote to prod Promotion request Approval and compliance check Approval and audit trail Rollback Automated trigger or manual request Post-rollback validation and regression test Test results Step 5: Bake in Operability for Observability and Day-2 Readiness IDPs reduce cognitive load and toil as solutions to common concerns are built in. This is especially true for the operational concerns. Each workflow and self-service feature creates the log files and traces for auditability. All code and configuration are driven from version control, and the metrics recorded provide insights into the outcomes and performance of the IDP. New operational initiatives, like introducing a software bill of materials, can be rolled out across all technologies that use the IDP. When done correctly, templates can be updated centrally, and the log files provide full auditability to identify where old versions are still in use, reducing the overall security exposure. The IDP governance model needs to define the ownership of templates and any inheritance rules. For instance, some teams will tailor the template by adding additional steps required for their technology. Alongside the IDP instrumentation, standard dashboards and alert definitions ship with the template, pre-wired to the appropriate ownership group. Who responds to what is documented, not assumed. Runbooks and escalation paths are stored in version control alongside the service itself so they evolve with the system rather than rotting in a forgotten wiki page. Our backend service will include the following with the golden path: Logs, metrics, and tracesAlertsRunbook linkOwnership metadata The final piece is the feedback loop. Incidents, near-misses, and recurring friction points are resolved and also used to help continuously improve the platform, first becoming a backlog item. Step 6: Add Guardrails and Governance Without Slowing Delivery The IDP should leverage approved templates where possible and embed basic compliance and policy checks in the workflows. Platform developers will receive immediate feedback on any problems they need to fix. When issue resolution requires a longer time, time-bound exceptions can be allowed. Along the environment path from development to production, the quality gates should become more restrictive as the software quality improves. For our backend service, we define security scanning prior to deployments, and we don’t accept any deviations from the corporate standard for it. We follow a simple block, warn, escalate paradigm. The goal is to address problems that teams can deal with immediately and provide enough time for more complex work. This balance allows work to flow at pace. It is important to version templates and workflows so you can track what is in use. When significant problems are identified with a version, you can use the IDP logs to find any items in use and replace them quickly. Having the right guardrails in place might feel restrictive but in fact reduces the amount of rework over time as there are fewer incidents. Fast feedback reduces the time it takes to resolve problems. Step 7: Measure Adoption, DevEx, and Platform ROI One of the key success factors for IDPs is having the ability to measure adoption (covered earlier), developer experience, and platform ROI (e.g., DORA, SPACE). This allows you to break down and distinguish between adoption measures and outcome metrics. Implementing these criteria in the platform from the beginning captures data systematically. Good adoption measures to start with: number of executed workflows, number and currency of templates, and number of active users. The following outcome metrics can also be used as part of the business case for IDPs: deployment failure rate, MTTR, incident volumes, number of tickets, and security vulnerabilities. The team managing the IDP should actively use the metrics together with captured feedback from the user base (e.g., feature requests) to prioritize the backlog. Executive dashboards should be implemented to provide accountability and increase support across the organization. A Minimal IDP You Can Scale Bringing it together, take the following actions to kick-start your internal developer platform: Choose a common and not too complex workflow for your first golden pathCreate the code repository and CI/CD pipelineDefine a self-service UI for the workflowEmbed quality gates, metrics, and operational tooling into the workflow Start with one workflow for one pilot team, prove the path, then extend to the next workflow or team. Don’t forget to engage with the pilot users to receive feedback and support adoption. If you want to dive deeper, explore the CNCF Platforms for Cloud-Native Computing whitepaper and Platform Engineering Maturity Model. This is an excerpt from DZone’s 2026 Trend Report, Platform Engineering and DevOps: How Internal Platforms, Developer Experience, and Modern DevOps Practices Accelerate Software Delivery.Read the Free Report

By Mirco Hering CORE

Editor’s Note: The following is an article written for and published in DZone’s 2026 Trend Report, Platform Engineering and DevOps: How Internal Platforms, Developer Experience, and Modern DevOps Practices Accelerate Software Delivery. High-performing engineering organizations don’t scale through heroics. They scale through repeatable platform capabilities backed by evidence. This checklist reflects the shift from tool‑centric DevOps to product‑oriented platform engineering, focused on scale, reliability, and developer outcomes. It is intended for platform teams, cloud architects, and engineering leaders building internal developer platforms (IDPs) that deliver consistency, velocity, and control. Architecture and Platform Foundations Establishing standardized, versioned platform foundations makes workloads deployable, observable, and scalable by default while preventing drift and reducing risk. Core platform primitives are standardized: identity, networking, compute, storage, and secretsStandard blueprints exist and are version-controlled for common workloads with clear evolution pathsInfrastructure is provisioned via reusable IaC modules with policy validationEnvironments and clusters follow consistent topology and access modelsNetworking and service communication follow secure, consistent patternsSecrets and configurations are centrally managed and injected securelyArchitectures define scalability mechanisms and fault boundariesResilience is built in through redundancy and failoverShared services are centrally managed with defined ownership and SLAsPlatform capabilities are versioned for backward compatibility Platform Ownership and Operating Model A product‑oriented operating model enables scale without slowing teams. Define clear ownership, interfaces, and governance so the platform evolves without becoming a delivery bottleneck. A dedicated platform team owns roadmap, usability, reliability, and adoptionOwnership boundaries are defined (platform standardizes; app teams own service logic)Platform capabilities are easy to discover and use (e.g., templates, workflows, golden paths)A structured intake and support model exists (e.g., requests, issues, exceptions)Standards are enforced with governed exceptionsPlatform success is measured through adoption and delivery outcomesUsage data and feedback drive continuous improvementCapabilities are versioned and evolved predictably Environments and Golden Paths Translate platform architecture into opinionated, self-service workflows driven by organizational standards that reduce complexity and enforce best practices by default. Golden paths are effective only when they are widely adopted. Environment conventions are standardized across naming, configuration, and accessEnvironment state is enforced through IaC/GitOps to prevent driftGolden paths provide curated, reusable templates for common workloadsSecurity, observability, and policy defaults are built into golden pathsGolden paths balance strong defaults with controlled flexibilitySelf-service workflows enable scaffolding, provisioning, and deploymentEnvironment lifecycle is automated across provisioning, promotion, and teardownDocumentation and onboarding are well integrated into workflowsAdoption is measured through usage and coverageFeedback and production learnings drive continuous evolution Pipelines and Release Reliability Standardize delivery pipelines so every change is validated, traceable, and safely releasable, making delivery more predictable and recoverable, not just faster. Pipelines follow a standardized flow: build, test, package, deploy, and promoteQuality, security, and policy checks are embeddedArtifact promotion across environments is controlled and consistentEach release produces traceable, auditable evidenceRollback and recovery paths are implemented and testedFailures provide fast, actionable diagnosticsReliability metrics are tracked (e.g., success rate, change failure, rollbacks)Release ownership and escalation paths are clearly defined Toolchain and Self-Service Automation Provide consistent self‑service automation through curated tools and embedded guardrails that reduce fragmentation, risk, and operational complexity. A unified developer point of entry exists through an IDP or developer portalStandard workflows exist for deployment, environment setup, and accessReusable modules and templates prevent copy-paste sprawl and reduce cognitive loadProvisioning and deployments are automated with guardrailsRBAC and approvals are embedded into automationHigh-risk actions require audited approvalsWorkflow reliability, usage, and failures are measuredAutomation evolves continuously based on usage and feedback Observability and Operability Embed observability and operational guardrails into self-service automation so systems are consistent, measurable, diagnosable, and operable by default. Logs, metrics, and traces are included by default through templates and golden pathsMinimum observability standards are enforced for promotionDashboards and alerts are preconfigured and actionableTelemetry supports debugging, capacity planning, and optimizationService health targets (e.g., SLOs) guide operationsOperational ownership is defined across on-call, escalation, and boundariesRunbooks guide incident response and recoveryIncident learnings feed platform and template improvements Reliability, Resilience, and Recovery Design for failure up front so systems fail safely, degrade gracefully, and recover predictably, proving resilience through recovery, not uptime alone. Architectures isolate failures to limit blast radiusDependencies are evaluated for availability and fallback strategiesResilience patterns are built in by default (e.g., retries, timeouts, circuit breakers, degradation)Non-critical features degrade without impacting core functionalityRecovery objectives are defined and validatedBackup and recovery mechanisms are implemented and testedRecovery is automated to minimize manual interventionGame days, chaos experiments, or failure drills are conducted to validate system behavior under stressReliability metrics are tracked and optimized (e.g., recovery time, failure rate) Security Guardrails and Governance Enforce security and compliance through codified guardrails embedded in delivery workflows, with continuous monitoring to improve security posture over time. Access follows least-privilege principlesSecrets are centrally managed and securely injectedPolicies are codified and enforced consistently through Policy as CodeSecurity controls are embedded in pipelines, including scanning and config checksHigh-risk actions require controlled approvalsExceptions are time-bound, tracked, and reviewedAll changes are auditable and traceableCompliance requirements map to enforceable controls Developer Experience, Adoption, and ROI Improve DevEx by reducing friction, driving platform adoption, and linking usage to measurable delivery outcomes and business impact. Developer experience is consistent across services and environments Platform abstracts common concerns (e.g., infra, security, observability) through standardized defaultsOnboarding to first deploy is fast and frictionlessDocumentation, examples, and enablement drive consistent adoptionPlatform and golden path adoption are measured through usage, onboarding, and coverageKey DevEx metrics are tracked (e.g., lead time, change failure rate, MTTR, time to first deploy)Workflow usability and reliability are continuously optimizedFeedback and usage data drive platform improvementsROI is measured through delivery outcomes (e.g., reduced toil, incidents, faster releases) Platform Engineering Maturity and Assessment Platform engineering maturity can be assessed across three practical stages that reflect the consistent application, adoption, and improvement of platform capabilities: Foundation focuses on baseline standardization, safety, and operability, with reusable capabilities in place but adoption still uneven.Scale enables reliable self‑service through guardrailed golden paths, improving delivery without increasing operational overhead.Optimize treats platform engineering as a strategic differentiator, using data‑driven decisions to continuously improve resilience, developer experience, cost efficiency, and measurable ROI. Use the Maturity Scoring Matrix to assess maturity across core platform engineering capabilities. Rate each category once, on a scale of 1–5, based on available evidence rather than aspiration. Overall maturity is determined by the dominant scoring pattern across the matrix, with higher maturity requiring consistent strength across Foundation, Scale, and Optimize. The progression bar maps scores from Ad Hoc to Strategic and groups them across the Foundation, Scale, and Optimize stages. Repeat the assessment periodically to identify gaps, track progress, and guide platform roadmap priorities. Conclusion Treat this checklist as a baseline gate and a recurring review mechanism, not a one-time exercise. High-performing platforms evolve through continuous refinement of architecture, automation, governance, and developer experience. Use it to identify gaps, strengthen golden paths, and align platform capabilities with measurable delivery outcomes. This is an excerpt from DZone’s 2026 Trend Report, Platform Engineering and DevOps: How Internal Platforms, Developer Experience, and Modern DevOps Practices Accelerate Software Delivery.Read the Free Report

By Josephine Eskaline Joyce CORE

Feature flags have become standard practice in enterprise applications, enabling teams to release code into production environments without exposing new features to users. As teams leverage feature flags to increase delivery velocity, technical debt accumulates. Left unchecked, this debt will slowly and silently impact application performance, maintainability, and developer productivity. What Is Feature Flag Debt? Feature flag debt occurs when feature flags are left in the codebase after they’ve served their purpose. The most common symptoms of feature flag debt include: Dead code Context switching for developers Feature flag debt can go unnoticed because it typically doesn’t cause broken features. As a result, developers are often reluctant to clean up flags so they can focus on developing new features. Impact on Performance Feature flag debt can have serious consequences for application performance. In front-end applications, this is often overlooked. Once a feature flag has been introduced into a codebase, it incurs a long-term cost every time the application is loaded in the browser. Larger JS bundles: Each feature flag adds logic to the application. When feature flags are not cleaned up, the associated code is typically not removed from the final bundled app. This means more code for users to download and more memory used on the client.Reduced execution speed in client-side rendering: The browser must download, parse, and evaluate the entire bundle, even if certain code paths are never executed. This leads to slower parsing, longer load times, and slower interaction time. Impact on Developer Productivity Feature flag debt also negatively impacts developer productivity. Imagine having to read through an if/else statement that checks a feature flag that will never be true. Developers frequently encounter this scenario when working with feature flags. New engineers, in particular, often struggle to know which feature flags are safe to ignore. Should they be commenting out this code? What if they need it later? Why Aren’t Feature Flags Cleaned Up? It should be standard practice to remove feature flags from the codebase once they’re no longer needed. However, they often become a long-term liability for the application for several reasons: Nobody takes responsibility for cleaning up flags.People are afraid to remove code.There are no tools to help automate the process.There’s always something more pressing to work on. We often don’t see a defined feature flag lifecycle, which leads to indefinite accumulation. Example of Feature Flag Debt For example, let’s take a look at how a feature would typically look when wrapped in a feature flag: JavaScript const isAIAgentsFeatureFlagEnabled = isFeatureEnabled('ai-agents'); if (isAIAgentsFeatureFlagEnabled) { // lines of code // Code to run when the feature flag is enabled } else { // lines of code // Code to run when the feature flag is disabled } When first implemented, this doesn’t look too bad. When this feature is rolled out to production, there’s still the safety net of keeping the original functionality should something go wrong. However, after the feature flag is turned on for everyone and the feature reaches general availability (GA), there is no reason to keep both pathways in the application. The application still ships both pieces of code in the bundle, but only one will ever execute at runtime. The else block now represents dead code that will not get executed, but still takes up space in the bundle and adds to code complexity. Manage and Eliminate Feature Flag Debt Organizations need to take measures to prevent feature flag debt from slowing down their applications. Defining a feature flag life cycle is a great place to start. By enforcing that each feature flag has a description, owner, status, and expiration date, the team can ensure flags aren’t left to become debt. Treat feature flags as temporary and not part of the application's core architecture. When the feature is in GA, remove the flag and delete any code paths that are no longer needed. This results in a cleaner, more maintainable, and performant codebase. JSON [ { "feature_flag_name": "ai-agents", "description": "Feature flag that will allow AI agents to assist users with workflows and provide suggestions", "owner": "architecture crew", "status": "GA", "expiration_date": "2026-12-31" }, { "feature_flag_name": "smart-checkout", "description": "Feature flag that will allow smart checkout features, including dynamic pricing, custom offers", "owner": "architecture crew", "status": "Dev", "expiration_date": "2026-12-31" }, { "feature_flag_name": "ai-agents-eval", "description": "Feature flag to allow the evaluation framework to execute tests against AI agents to determine how accurate they are", "owner": "agent evaluation crew", "status": "QA", "expiration_date": "2026-10-12" }, { "feature_flag_name": "experiment-recommendation-v2", "description": "Feature flag for experimenting v2 recommendation version", "owner": "agent evaluation crew", "status": "GA", "expiration_date": "2026-12-31" } ] Having the feature flags stored in a format similar to the above can help identify who to contact to clean up old flags. Performance Gains From Cleanup Removing unused feature flags reduces bundle size and eliminates unnecessary code execution, resulting in faster load times, improved rendering performance, and a cleaner codebase. Conclusion For most enterprise applications, feature flags aren’t the problem; it’s forgetting to take them down. As the application grows over time, old feature flags accumulate, which will silently bloat the bundle size, degrade performance, and clutter the code.

Partitioning and Z-Ordering have long been fundamental techniques in Delta Lake for optimizing data layout and query performance. However, these methods require significant upfront design and ongoing maintenance and they often struggle to adapt to changing data and query patterns. Databricks Liquid Clustering introduced with Delta Lake 3.0 goes beyond traditional partitioning and Z-Order, offering a self-tuning, flexible approach to organizing data that is especially powerful for Unity Catalog managed tables. In this article, we’ll explore how Liquid Clustering works, how it compares to traditional methods, and how to implement it in Databricks Unity Catalog for improved performance and simpler data management. Recap: Partitioning and Z-Order Limitations Before diving into Liquid Clustering, it’s important to understand the challenges of conventional partitioning and Z-Ordering in large Delta Lake tables: Design Complexity & Rigidity: Choosing an optimal partitioning scheme is difficult and usually fixed. A static Hive-style partition strategy often demands careful upfront planning to avoid data skew and concurrency conflicts and it cannot easily adapt if query patterns change. Changing partition columns later means expensive data rewrites.Partition Explosion & Metadata Overhead: If you partition on high-cardinality columns or many levels, you may end up with too many small partitions. This proliferation of tiny files and directories increases metadata overhead and slows down query planning.Need for Additional Clustering (Z-Order): Z-Ordering is often applied on top of partitions to co-locate related data. While Z-Order can improve data skipping, it is expensive to maintain it requires heavy shuffle and rewrite jobs and does not handle concurrent writes well. In other words, Z-Ordering jobs can be lengthy and costly and must be re-run as new data arrives to maintain clustering.Manual Tuning & Maintenance: Both partitioning and Z-Order require continuous tuning. Data engineers must monitor query patterns and manually decide how to partition or when to re-Zorder. This ongoing maintenance is time-consuming and error-prone. In summary, traditional partitioning/Z-ordering yields performance benefits but at the cost of rigidity and operational overhead. This sets the stage for a more adaptive solution. What Is Liquid Clustering? Liquid Clustering is a new data layout strategy in Databricks Delta Lake designed to replace traditional partitioning and Z-Ordering for Delta tables. The name liquid signifies flexibility data is clustered by one or more columns in a way that can evolve over time without strict, static partitions. Key characteristics of Liquid Clustering include: Dynamic, Self-Tuning Layout: Instead of static partitions, data is dynamically clustered based on specified clustering keys. The table’s storage layout automatically adjusts to changing data and query patterns, incrementally clustering new data as it is written. This means the data layout flows with your workload.Simplicity in Key Selection: You choose a set of clustering columns based on query access patterns, typically the columns most commonly used in WHERE filters or joins. You don’t need to worry about column cardinality, order of keys or file size tuning the platform handles optimal file sizing and clustering internally. Even high-cardinality columns can be used effectively, which would be impractical as partition keys.Flexibility to Change Keys (No Rewrites): Perhaps the most revolutionary aspect is that clustering keys can be redefined without rewriting existing data files. If your query patterns shift, you can alter the clustering columns and the system will gradually reorganize data for the new keys. There’s no massive upfront cost of re-partitioning the entire dataset past data doesn’t need an immediate rewrite.Skew-Resistant & Efficient Storage: Liquid Clustering is designed to maintain balanced file sizes and avoid the pitfalls of skewed partitions. Under the hood, the data engine can combine or split clustering ranges to keep files at an optimal size.Reduced Maintenance Overhead: Because the data layout adapts automatically, the need for manual maintenance is drastically reduced. You no longer have to schedule regular Z-Ordering jobs or hand-tune partition schemes. Liquid Clustering, especially in its automatic mode, offloads these decisions to Databricks. Databricks recommends using Liquid Clustering for most new Delta tables going forward, especially for tables that are large, have high-cardinality filter columns, experience data skew, or have evolving access patterns. It simplifies data engineering by set it and forget it clustering. In fact, thousands of customers have already adopted it as of 2025, over 3,000 monthly customers were writing 200+ PB of data into Liquid Clustered tables. Liquid Clustering vs Traditional Methods Liquid Clustering addresses the limitations of partitions and Z-ordering in several ways: No Rigid Partition Boundaries: Unlike Hive partitions, liquid clustering can store a range of values in each data file. This fluid layout avoids issues like tiny partitions or unbalanced file sizes.Incremental and Low-Shuffle Clustering: New data is clustered as it’s ingested, without requiring a full table rewrite. When you enable clustering on a table, Databricks flags the table to cluster future writes according to the specified keys. Each new INSERT or MERGE automatically writes out files clustered on those keys, and small files are merged as needed. This incremental approach means no huge one-time sort jobs every time you add data. Maintenance operations like OPTIMIZE still play a role but they can operate more efficiently since the incoming data is already sorted/clustered on write. Notably, the OPTIMIZE command for a liquid-clustered table can be more adaptive than traditional OPTIMIZE+ZORDER it only rearranges data that isn’t well clustered yet rather than always rewriting everything.Adapting to Change Without Rewriting Everything: In a partitioned table, if you realize a month later that queries would run faster partitioned by a different column, you’d have to repartition the entire dataset. With Liquid Clustering, you can simply issue an ALTER TABLE to change the clustering column set. The system will use the new keys for all future writes, while existing files remain as they are until an optimization is triggered. You can later run a full optimize to reorganize historical data under the new scheme if needed. This means you can respond to evolving query patterns without incurring an immediate cost for reprocessing the whole table.Better Concurrency and Fewer Conflicts: Because Liquid Clustering avoids overly granular partitions and heavy-duty clustering jobs, it also mitigates concurrency problems. Traditional partitions can suffer write conflicts if too many jobs target the same partition, and Z-order optimize jobs can conflict with concurrent writes. Liquid Clustering’s design results in fewer such bottlenecks.Performance Gains: Ultimately, the goal is faster queries and lower cost. By clustering data on the actual query predicates, Liquid Clustering improves data skipping. This leads to less IO and faster execution. In one benchmark, Databricks observed that a 1 TB warehouse dataset clustered with Liquid Clustering ran 2.5× faster to optimize (cluster) than using Z-Ordering, and yielded significantly better query performance than both partitioning or Z-Order. In real workloads, users have reported dramatic improvements; for example, Healthrise (a Databricks customer) saw some queries run up to 10× faster after enabling Automatic Liquid Clustering on their tables. We’ll discuss Automatic mode shortly. How Liquid Clustering Works (Under the Hood) At a high level, manual Liquid Clustering works by clustering data files on chosen key columns, while automatic Liquid Clustering adds an intelligent layer to choose and adjust those keys for you. Let’s break down the mechanisms: Clustering on Write: When you define clustering keys for a Delta table, the Delta engine ensures that newly written data is organized according to those keys.Maintenance and OPTIMIZE: Over time, as data is appended, you may still accumulate some fragmentation. The OPTIMIZE command can be used on a clustered Delta table to compact small files and sort data more finely according to the clustering columns. Unlike Z-Ordering, an optimize on a liquid-clustered table doesn’t always have to rewrite all files it focuses on incremental clustering, merging files that are sub-optimally placed. You can think of it as tightening the clustering. If you change the clustering columns via ALTER TABLE, you can run OPTIMIZE FULL to recluster all existing records under the new key order. In normal operation, Databricks recommends running periodic OPTIMIZE to keep performance optimal, but these operations are more lightweight than traditional heavy Z-order jobs.Data Skipping with Statistics: Delta Lake maintains statistics that the query engine uses for data skipping. Liquid Clustering maximizes the effectiveness of data skipping by ensuring those min/max ranges align with query filters. Enabling Automatic Clustering To use Automatic Liquid Clustering, you need to have Predictive Optimization enabled for your workspace (this is the feature in Unity Catalog that handles these background optimizations). Many new Databricks accounts have this on by default since late 2024, but it can also be enabled via the account console (under Feature Enablement). Assuming it’s enabled, turning on Automatic clustering for a table is straightforward: SQL: Use the CLUSTER BY AUTO clause when creating or altering a Delta table. For example, to create a new table in Unity Catalog with auto clustering: SQL -- Creating a Unity Catalog managed table with Automatic Liquid Clustering CREATE TABLE main.analytics.user_events ( user_id STRING, event_type STRING, event_date DATE, details STRING ) CLUSTER BY AUTO; -- enables automatic liquid clustering on this table SQL ALTER TABLE main.analytics.user_events CLUSTER BY AUTO; This instructs Databricks to begin monitoring the table’s workload and to auto-select clustering keys for optimal performance. The table does not need to have any manual keys set; the system will determine them. (Under the hood, the first time it chooses keys, it will update the table’s metadata with those columns as clustering keys.) PySpark API: In code, you can also enable auto clustering when writing data. For instance, using the DataFrame Writer API in PySpark: Python # df is a DataFrame we want to save as a Delta table with auto clustering df.write.format("delta") \ .option("clusterByAuto", "true") \ .mode("overwrite") \ .saveAsTable("main.analytics.user_events_auto") The above will create the user_events_auto table as a Unity Catalog managed table with automatic clustering enabled. (If you want to provide an initial hint for clustering columns, you can combine .clusterBy("col1", "col2") with the clusterByAuto=true option, but it’s not required – the system will figure it out if you leave it open.) Once Automatic mode is on, no further action is needed from the user. Databricks will handle running background optimize jobs as needed. It’s worth noting that these maintenance operations run on a serverless compute in the background. The benefit is you no longer need to schedule OPTIMIZE or VACUUM on your own; predictive optimization will run them at optimal times. Using Manual Liquid Clustering (Custom Clustering Keys) In some cases, you may want to manually specify the clustering columns. Unity Catalog supports manual Liquid Clustering on managed tables as well. Here’s how to use it: Table Creation with Cluster Keys: You can define clustering keys in the CREATE TABLE statement via a CLUSTER BY clause. For example: SQL -- Create a Delta table clustered by specific columns (manual clustering) CREATE OR REPLACE TABLE main.analytics.sales_data ( sale_id BIGINT, region STRING, product STRING, sale_date DATE, amount DECIMAL(10,2) ) CLUSTER BY (region, sale_date); In this example, the table’s data will be clustered by region and sale_date. This means each file written will tend to contain a narrow range of region values and sale_date values. This is analogous to creating a partitioned table on multiple keys, but without creating separate directories for each region or date. Altering an Existing Table: If you have an unpartitioned Delta table and want to enable clustering on it, use an ALTER statement. For instance: SQL ALTER TABLE main.analytics.sales_data CLUSTER BY (region, sale_date); This will register region and sale_date as the clustering keys for sales_data. As mentioned, this does not rewrite existing files immediately. It flags the table so that future writes will be clustered by these keys. Any new data you append or merge into sales_data will now be written in clustered order. Data that was already in the table remains in its original layout until you optimize. Reclustering Existing Data: To apply the new clustering to old files, you can run an OPTIMIZE operation. For a large table, you might do this during a maintenance window. For example: Python OPTIMIZE main.analytics.sales_data; The above will compact small files and cluster data incrementally. If you recently changed the clustering keys and want to force a full re-cluster of all data under the new key order, use OPTIMIZE main.analytics.sales_data **FULL**. An OPTIMIZE FULL will read and rewrite all files in the table, arranging them according to the current clustering columns. In most cases, a regular OPTIMIZE will suffice, as it will naturally pick up new keys over time. PySpark Write with Clustering Keys: You can also write data from Spark with clustering, similar to how you’d write partitioned data. For example: Python # Given a Spark DataFrame df, write it to a Delta table with clustering on specified keys df.write.format("delta") \ .mode("append") \ .clusterBy("region", "sale_date") \ .saveAsTable("main.analytics.sales_data"); Here, .clusterBy("region", "sale_date") ensures the data in df gets written out clustered by those columns. If the table sales_data was not already created, this will create it with those cluster keys. Finally, remember that Liquid Clustering is supported only on Delta tables with the latest protocols. Enabling it will bump your table’s Delta protocol version which older clients cannot read. In a Databricks environment this is usually not an issue, but be cautious if you have external readers/writers that might be using older Delta Lake libraries. Conclusion Liquid Clustering represents a major evolution in data layout management for the Lakehouse. By moving beyond the rigidness of partitioning and the heavy operational cost of Z-Ordering, it delivers a simpler and more adaptive way to optimize tables. For Data Engineers, this means less time agonizing over partition strategies and maintenance jobs, and more time focusing on data and insights. With Unity Catalog’s Automatic Liquid Clustering, the process is taken a step further clustering becomes a self-driving process, leveraging query insights to continuously improve performance. In summary, Databricks Liquid Clustering dynamically organizes data based on actual usage, can adjust without expensive rewrites, and has been shown to boost query performance significantly. As you design your next Delta Lake tables in Unity Catalog, consider leveraging Liquid Clustering from the start it can simplify your architecture and ensure your tables automatically stay optimized as your data (and its use cases) grow.

Editor’s Note: The following is an article written for and published in DZone’s 2026 Trend Report, Platform Engineering and DevOps: How Internal Platforms, Developer Experience, and Modern DevOps Practices Accelerate Software Delivery. I am developing a reference guide for platform teams that want continuous optimization embedded directly into their internal developer platforms. In this proposed model, “done” means automated, full-stack tuning recommendations that fit safely and seamlessly into existing engineering workflows. Building golden paths for pre-deployment tasks is relatively straightforward because engineering teams share the primary goal of shipping applications faster. However, after deployment, sustained efficiency frequently becomes a neglected task that is “someone else’s job.” Developers prioritize shipping, SREs protect safety buffers, and FinOps pushes for cost reduction. The reference model proposes a dedicated efficiency layer as a required platform capability designed to reconcile those priorities without requiring a replatform. In this one-layer deep dive, we focus only on the embedded efficiency layer: its interfaces, interaction model, and what it requires to be credible. Project Constraints I anchor my design on the assumption that engineering teams are already managing their production deployments through established IaC and GitOps practices. Unlike pre-deployment pipelines that often enforce strict corporate standards, a post-deployment efficiency optimizer cannot be rigidly opinionated. Every microservice possesses unique architectural characteristics and operational requirements that demand a highly configurable approach to system optimization. I recommend allowing teams to define explicit parameters based on the workload context, dictating whether a particular service requires a specific operational profile. ProfileIntentTradeoff Cost-first Aggressive cloud cost reduction Less headroom, higher reliability risk Performance-first Maximum throughput performance Higher cost (maybe), tighter buffers Reliability-first Expanded reliability buffer for unpredictable traffic spikes Higher baseline spend Architecting the Day-Two Golden Path Effective efficiency optimization requires an architectural deep dive beyond superficial cloud scaling metrics. The framework I recommend orchestrates continuous tuning across the entire technological stack, cascading from the underlying infrastructure nodes down through Kubernetes configurations and directly into the application runtime. Adjusting CPU requests and memory limits at the container level is mathematically insufficient if the underlying Java Virtual Machine or application runtime parameters remain poorly calibrated for those newly allocated resources. Consequently, the guide treats the underlying correlation engine as a mandatory architectural component for producing holistic configuration recommendations. FLOW: infrastructure metrics + Kubernetes signals + app monitoring → correlation engine → recommendations (infra/k8s/runtime) Figure 1: Full-Stack Optimization Layers The Interaction Model The foundational principle governing this architectural layer is an explicit human-in-the-loop (HITL) model. Fully autonomous, black-box changes erode trust when operators can’t see the reasoning behind configuration updates. Instead, the multi-dimensional tuning recommendations surface inside the developer’s GitOps workflow, presenting clear explainability about how a change affects latency, reliability, and cost. HITL ensures engineers retain final approval over critical production changes, but it introduces review latency and requires significantly more comprehensive explainability documentation for every recommendation. Scenario Walkthrough A critical microservice begins experiencing rising cloud costs alongside escalating p95 latency. The embedded optimization engine detects the drift, correlates the cross-stack metrics, and proposes two runtime adjustments via an automated GitOps pull request. The application owner reviews the generated explainability visuals, verifies that the tuning resolves the latency issue without violating any existing rule, and manually merges the request. The platform seamlessly applies the validated configuration and continuously tracks the resulting operational benefits. Figure 2: The Interaction Model That workflow only holds if the following choices are true: Capabilitytradeoffwhat makes it workable Tuning profiles Requires explicit rules definition Profile selection per service or category Full-stack tuning More complexity than infra-only Correlation across infra + app metrics GitOps surfacing Adds workflow touchpoints PR-based delivery in existing process Human in the loop Review PRs and recommendation docs Explainability visuals + approval step Takeaways Based on the framework in this reference guide, here is what I would tell someone building an embedded efficiency layer next, based on their involvement: Designing the interaction model: Prioritize operator trust and mathematical transparency over fully autonomous, unexplainable actions.Defining the technical scope: Ensure your engine tunes the entire stack, from the underlying infrastructure down to the application runtime, rather than settling for superficial cloud resource constraints.Navigating the sociotechnical divide: Treat the optimization layer as a collaborative platform capability that grounds the competing priorities of developers, reliability engineers, and FinOps, not a financial audit mechanism. This is an excerpt from DZone’s 2026 Trend Report, Platform Engineering and DevOps: How Internal Platforms, Developer Experience, and Modern DevOps Practices Accelerate Software Delivery.Read the Free Report

By Graziano Casto CORE

Editor’s Note: The following is an article written for and published in DZone’s 2026 Trend Report, Platform Engineering and DevOps: How Internal Platforms, Developer Experience, and Modern DevOps Practices Accelerate Software Delivery. Recent advances in tooling and automation have moved DevOps beyond a collection of siloed frameworks and tools toward a more unified delivery model. But the sprawl of disconnected tools and the cognitive load of constant context switching have also created analysis paralysis, slowing delivery and shifting attention away from technical progress toward coordination challenges. In response, platform engineering has become the delivery backbone for organizations. In 2026, scaling delivery and adopting AI successfully will require platforms to operate through a product-led model. This article explores how practitioners and leaders can adopt product-led approaches, using real examples and practical best practices to measure the impact of DevOps at scale, where reliability and compliance are both critical. It examines tradeoffs such as speed vs. standardization and autonomy vs. integration. What Breaks as DevOps Scales As DevOps scales across multiple teams and systems, challenges emerge across infrastructure, security, compliance, and observability. These challenges are not only technical or skills-based. A technical solution may work at a smaller scale but will likely fail at a larger one. In a regulated organization, responsibilities such as auditing, logging, data processing, and managing suppliers and contractors are often handled by different teams. This can lead to slower response times and increased errors in deployment and testing. At the same time, the growing number of tools, environments, and versions increases cognitive load and creates tool sprawl, both of which slow delivery. Context switching between disconnected systems adds further friction, reducing velocity and making it harder for teams to work effectively. Over time, these pressures affect delivery outcomes, contribute to burnout, and limit critical thinking. Platform Engineering as the Scaling Mechanism A common misconception is that teams and systems can be optimized individually. While this may be true in smaller organizations, it is not practical at scale. In this context, the platform-led model provides an umbrella under which systems and teams can be optimized as one unified unit, supported by self-service capabilities. If the platform is treated as a product, it comprises all the necessary components, including users, processes, and measurable outcomes. The goal is to simplify and standardize processes so nothing breaks down as DevOps scales. In practice, this creates a shared operating model in which DevOps, SRE, platform engineering, and security teams align around common defaults, guardrails, and delivery expectations. Figure 1 This can be implemented in practice through golden paths. For example, when a new service is requested, a workflow template can be created to add a new repository with all the required steps, including CI/CD pipelines, environment configuration, security, and alerts checks. This path can then be replicated and integrated with other services with minimal deployment effort. At the same time, compliance, resilience, and regulatory steps are implemented automatically. Instead of relying on tickets or legacy knowledge, teams can use these paved paths as self-service workflows with built-in defaults and guardrails. Golden paths reduce error and failure rates because each stage is predefined for release, deployment, and rollback. These pipelines require consistency across tools, environments, and release frameworks. Without it, incidents, cases, and handovers become more difficult to manage. At scale, standardization and integration make these workflows repeatable, reliable, and easier to adopt across teams. The following table compares the two approaches. Old vs. Platform-led DevOps Old DevOps modelwhy it breaks at scaleplatform-led devops Individual teams and pipelines Inconsistency and drift Replicated golden paths Documentation per team/system Outdated knowledge Centralized documentation High autonomy Missing interoperability Consistency is high Low standardization Expensive to maintain High standardization Challenging integration Increased error rates High integration Developer Experience Becomes a First‑Class Delivery Metric Developer experience (DevEx) helps identify friction across tools, teams, and workflows, while also providing a way to measure quantitative and qualitative productivity. This is critical for any platform at scale, where slow onboarding, manual approvals, and persistent development constraints can delay delivery. DevEx measures such as time to first deploy, failure rate, lead time, and MTTR can help uncover bottlenecks in DevOps. Improving them leads to better developer satisfaction, smoother scaling, and clearer platform priorities. Success criteria become even more important at scale, where multiple teams work closely together to produce similar services with similar pipelines under the same or similar compliance conditions. In those environments, friction is reduced, and practitioners benefit directly from a stronger developer experience. Automation and AI: Leverage With Guardrails Automation supports standardization and integration by handling repetitive tasks and default configurations. With the adoption of AI, its value is seen most clearly in assisting rather than replacing decision-making. Combined with automation, AI shortens feedback loops and makes processes easier to audit and monitor, reducing failure rates and improving the developer experience. In practice, platform teams can use AI to intelligently automate triage, reduce alert noise, provide context-aware suggestions, and support guided remediation. However, applying automation and AI requires guardrails so systems and tools operate within clear boundaries, avoid incorrect outputs, and allow immediate rollback where necessary. There is a significant tradeoff between risk and speed, and finding the right balance is one of the first concerns organizations must address when integrating AI. Measuring Platform Value Measuring platform value should be demonstrated through outcomes, with recommendations supporting teams rather than replacing them. Increased platform adoption can act as a leading indicator that teams are choosing to follow golden paths and standardization and integration practices. A low adoption rate, by contrast, may signal growing friction and silos across teams and tools. When done well, the platform’s value becomes apparent in the ability to deliver releases without unnecessary overhead or disruption. The focus should always be on measuring outcomes that reflect integrated and repeatable pipelines, strengthening service continuity, and raising the standard for auditing and compliance. Outcome-based measures validate adoption: reduced operational toil, fewer incidents, faster recovery, and more reliable delivery. These outcomes translate directly into service continuity and audit confidence. However, counting tools or templates say little about impact. Two Failure Modes to Avoid Not all failures are obvious. If teams continue to use old methods and approaches despite the introduction of golden paths, DevEx, automation, and AI, the result can be platform theater, where neither outcomes improve nor value is added. Here, the illusion of productivity is often caused by cultural resistance: Teams adopt new tools but continue using old methods, leading to minimal or no improvement. For example, a team may adopt an internal platform but still rely on tickets, manual approvals, and older team-specific processes to move work forward. Another less visible failure is platform paralysis, where teams are pushed to build pipelines in parallel, leading to slower delivery and more controlled decision-making rather than flexibility, enablement, and repeatability. Here, the loss of velocity is often caused by over-engineering or too many competing solutions, with complex parallel approaches slowing delivery rather than accelerating it. For instance, multiple teams may create overlapping workflows and tooling for the same problem, increasing complexity instead of reducing it. Avoiding these two failure modes requires a clear shift from treating the platform as a project with milestones to treating it as a unified product-led model, with DevEx, automation, and AI focused on improving how work is actually done. What Product-led Delivery Looks Like in 2026 In 2026, delivery is increasingly shaped by standardization, integration, automation, and AI adoption. The goal is to help teams move faster without increasing complexity or raising the risk of bottlenecks and pipeline failure. In platform-led models, golden paths become the norm, allowing teams to follow repeatable processes with a greater degree of confidence in the outcome. Many of the same tools and methods that were introduced to increase speed have also added cognitive strain, fragmentation, and delivery friction. The next step is to reduce that complexity through a platform-led model, where golden paths improve speed and reliability while lowering cognitive load. For organizations looking at the next quarter, two practical priorities are to establish a small number of reusable golden paths and to baseline a focused set of DevEx measures so bottlenecks can be identified and removed earlier. This is an excerpt from DZone’s 2026 Trend Report, Platform Engineering and DevOps: How Internal Platforms, Developer Experience, and Modern DevOps Practices Accelerate Software Delivery.Read the Free Report

By Fawaz Ghali, PhD CORE