IoT Resources

DZone's Featured IoT Resources

Beyond Clicking Buttons: Build a Browser Agent That Verifies Its Results With Playwright MCP

By Akhil Madineni

CORE

Browser agents become useful when they can do more than reach the right page and trigger the right control. A click is only an attempted action; it is not proof that a business operation completed. A form can submit while validation fails, a checkout button can respond while the API returns an error, and a success-looking route can render stale state. Playwright MCP is well suited to closing that gap because it exposes browser interaction through structured accessibility snapshots and adds explicit testing tools for checking visible elements, text, lists, and form values. The result is an agent loop that can treat verification as a first-class phase rather than as an optimistic interpretation of the previous action. An Action Is Not a Result The central design rule is simple: every state-changing action should have a postcondition. In an ordinary browser automation script, success is often inferred from the absence of an exception. That standard is too weak for an autonomous agent. Playwright MCP interaction tools such as browser_click operate on element references taken from accessibility snapshots, and most actions return an updated snapshot after triggered browser work settles. That makes the post-action page state immediately available, but the controller still has to decide what evidence counts as success. A useful contract separates intent, action, and evidence. For a task such as submitting an order, the action is a click on the final submission control. The evidence might be a visible confirmation heading plus an order identifier. The agent should not report completion until those conditions are independently checked. With the testing capability enabled, Playwright MCP provides browser_verify_element_visible, browser_verify_text_visible, browser_verify_list_visible, and browser_verify_value. Verification calls return Done on success and an error on failure, which creates a clean boundary between a browser action and a verified outcome. A minimal controller can therefore make the verification step explicit: TypeScript await client.callTool({ name: "browser_click", arguments: { target: submitRef } }); await client.callTool({ name: "browser_wait_for", arguments: { text: "Order confirmed" } }); const proof = await client.callTool({ name: "browser_verify_element_visible", arguments: { role: "heading", accessibleName: "Order confirmed" } }); if (proof.isError) { throw new Error("Order submission could not be verified"); } The important property is not the specific wrapper around callTool; it is the control flow. Action and verification are different operations, and a failed verifier changes the task status from “completed” to “unconfirmed.” browser_wait_for is appropriate when a specific asynchronous transition must finish, while Playwright MCP already waits for triggered navigation and network activity after most actions. Fixed sleeps should remain a last resort because the server supports waiting for text to appear or disappear directly. Verification Should Match the Business Outcome A reliable verifier checks the state that matters to the task rather than a convenient visual change. A button becoming disabled proves only that the button changed. A toast saying “Saved” is stronger, but still may not prove persistence if the application updates optimistically. Browser-level evidence becomes stronger when multiple independent signals agree: semantic UI state, the resulting page structure, and relevant network activity. Playwright MCP exposes each of these forms of evidence through snapshots, verification tools, and network inspection. Playwright MCP exposes network inspection through browser_network_requests and browser_network_request, allowing an agent to locate a relevant request and inspect its details. Console messages are also available through the core browser_console_messages tool. Those channels are useful when a task appears successful in the DOM while a background request fails or the page emits an uncaught error. Network evidence should still be tied to application semantics; an HTTP response by itself does not establish that the intended record contains the correct data. For a profile update, the strongest browser-side check may be a round trip: submit the change, wait for the completion signal, navigate away or reload, then verify the field value from the newly rendered state. browser_verify_value supports textboxes, checkboxes, radios, comboboxes, and sliders, so persistence checks can stay semantic instead of scraping raw HTML. TypeScript await client.callTool({ name: "browser_verify_value", arguments: { type: "textbox", element: "Display name", target: displayNameRef, value: expectedName } }); This pattern is especially important for agentic workflows because planning logic can be probabilistic while verification can remain deterministic. The model may choose among several valid ways to reach a form, but the acceptance criterion can still be exact: a heading exists, a field equals an expected value, a list contains required entries, or confirmation text is visible. Playwright MCP’s verification tools are designed around those concrete browser states. Accessibility Snapshots Make the Feedback Loop Precise Playwright MCP uses accessibility snapshots as the primary representation for agent interaction. A snapshot contains roles, accessible names, text, and element references used by subsequent tool calls. This is materially different from relying on screenshots as the main control surface. Screenshots remain valuable for visual diagnostics, but the MCP documentation explicitly directs actions toward snapshots rather than screenshot coordinates. That distinction improves verification quality. A semantic check such as “heading named Order confirmed is visible” is less ambiguous than a model deciding whether a collection of pixels resembles a success page. It also aligns the agent’s evidence with the same roles and names used by Playwright locators. When only part of a large page matters, browser_find can search the accessibility snapshot and return matching nodes with local context, reducing the need to repeatedly consume the full tree. Screenshots still have a place when the requirement is inherently visual, such as confirming layout, clipping, or rendering. For correctness of transactional browser work, however, semantic evidence should dominate. Tracing can then provide failure forensics rather than primary success criteria. With the devtools capability, Playwright MCP can record traces containing DOM snapshots, screenshots, network activity, console logs, and timing, making an unverified or failed run reproducible after the fact. Successful Agent Runs Can Become Regression Tests Verification becomes more valuable when it survives beyond a single agent session. Playwright MCP’s testing capability records matching expect(...) code for verification tools, and action responses can include generated Playwright code. The documentation explicitly shows an exploratory sequence being assembled into a conventional Playwright test. That creates a productive path from autonomous exploration to deterministic regression coverage. A verified flow can therefore graduate into a compact test instead of remaining hidden inside an agent transcript: TypeScript test("submits an order", async ({ page }) => { await page.getByRole("button", { name: "Place order" }).click(); await expect( page.getByRole("heading", { name: "Order confirmed" }) ).toBeVisible(); await expect( page.getByText(expectedOrderNumber) ).toBeVisible(); }); The same principle should shape production configuration. Only required capabilities should be exposed, isolated sessions should be preferred for repeatable runs, and origin restrictions can reduce accidental navigation. Playwright MCP supports capability selection, isolated profiles, allowed and blocked origins, secrets redaction, and configurable timeouts. Its documentation also warns that origin controls and secret handling are convenience defenses rather than security boundaries, so client-level permissions remain necessary when an agent can perform consequential actions. Conclusion A browser agent becomes dependable only when completion means more than “the click happened.” Playwright MCP provides the pieces required for a verification-centered design: structured accessibility snapshots for precise targeting, explicit verification tools for semantic postconditions, waiting primitives for asynchronous transitions, network and console evidence for deeper diagnosis, and traces for failed-run analysis. The strongest implementation treats every consequential action as a hypothesis that must be proven by observable browser state. That shift turns browser automation from a sequence of hopeful interactions into a controlled execution loop whose results can be checked, explained, and eventually converted into durable Playwright regression tests. More

Jakarta Batch in Practice: Reliable Chunk-Oriented Processing for Enterprise Workloads

By Otavio Santana

CORE

Batch processing remains vital because many business operations aren't suited to interactive requests. Tasks such as recalculating prices, reconciling transactions, migrating records, generating reports, processing invoices, reclassifying customers, or applying rules across millions of records may require considerable time. Handling these as standard requests leads to fragile systems, increased user wait times, frequent timeouts, challenging retries, and possible data inconsistencies. A batch model handles large workloads predictably, incrementally, and with control over progress and recovery. Rather than processing a massive operation as a single loop, batch processing uses jobs, steps, chunks, checkpoints, filtering, and restartability. This approach separates long-running data tasks from the user experience while delivering a structured execution model. In this article, we will focus on Jakarta Batch and examine its sustained relevance for modern enterprise applications. Why Batch Processing Still Matters in Enterprise Systems Modern applications offer various methods for background processing, such as message queues, event-driven architectures, schedulers, reactive pipelines, and distributed stream-processing platforms. While each addresses specific needs, batch processing is most effective when operations have a defined start and end, involve a known or discoverable dataset, and require controlled execution, progress tracking, restartability, or periodic processing. Batch processing remains essential in enterprise systems. Workloads such as financial reconciliation, billing, payroll, reporting, data migration, regulatory processing, catalog updates, and large-scale reclassification are still prevalent. In these scenarios, the priority is to process large volumes of work safely and predictably, rather than responding to individual events quickly. Batch provides a model specifically designed for these requirements. How Jakarta Batch Works Jakarta Batch organizes background processing into jobs and steps. A job defines the overall batch operation, while each step represents a specific stage. In chunk-oriented processing, a step follows a simple pipeline: read, process, write, and repeat until it processes all input. The Jakarta Batch runtime manages this lifecycle so application code can focus on reading, transforming, and persisting data. A job is the top-level unit of execution and represents a complete business operation, such as importing records, recalculating customer classifications, processing invoices, or reconciling transactions. Jobs can accept parameters at startup, allowing the same batch definition to run with different inputs or business rules. A job consists of one or more steps, each representing a distinct phase of the workload. Simple jobs may have a single step, while complex processes can use multiple steps in sequence, such as importing data, validating it, and generating a final report. Within a chunk-oriented step, the ItemReader supplies data to the runtime one item at a time, from sources such as a database or file. The reader only retrieves the next item and does not need to know how it will be processed or persisted.The ItemProcessor receives each item and applies business rules, such as validation, transformation, classification, enrichment, or filtering. It may return a modified item or null if the item should be excluded from writing.The ItemWriter receives processed items and persists or exports them. Unlike the reader and processor, which handle items individually, the writer typically receives a group of items from the current chunk. This enables more efficient database or bulk operations. Jakarta Batch adds features around this pipeline to support enterprise workloads. The runtime manages chunk boundaries, transactions, checkpoints, execution status, failures, and restart behavior. Chunk size determines how much work is grouped before a write and checkpoint, making it a key parameter for balancing throughput, memory usage, database cost, and recovery. The core model is straightforward: Job → Step → Read → Process → Write → Repeat Jakarta Batch keeps the business pipeline simple while the runtime manages the execution mechanics needed for reliable, long-running data processing. The Sample: Customer Segmentation with Jakarta Batch This example demonstrates the Jakarta Batch model using an e-commerce customer segmentation scenario. Customers are assigned to tiers such as Bronze, Silver, Gold, and Platinum based on configurable spending thresholds. When thresholds change, the application reevaluates the customer base and updates only customers whose classification has changed. The full application includes MongoDB integration, a Jakarta Faces UI, a preview workflow, validation, and supporting services. The complete source code is available at https://github.com/soujava/mongodb-jakarta-batch. This section focuses on the classes directly involved in Jakarta Batch execution. Starting the Batch Job The application initiates the batch process through CustomerSegmentationService. Unlike the reader, processor, and writer, this class is not a batch artifact. Instead, it is an application service that retrieves Jakarta Batch’s JobOperator from BatchRuntime to start and monitor job executions. Java @ApplicationScoped public class CustomerSegmentationService { public static final String JOB_NAME = "customer-segmentation"; private volatile CustomerSegmentationPolicy currentPolicy; // initialization and status methods omitted public long start(CustomerSegmentationPolicy policy) { if (isRunning()) { throw new IllegalStateException( "A customer segmentation batch is already running"); } Properties parameters = new Properties(); parameters.setProperty( CustomerSegmentationPolicy.JOB_PARAMETER, policy.toJson()); long executionId = BatchRuntime.getJobOperator() .start(JOB_NAME, parameters); currentPolicy = policy; return executionId; } public boolean isRunning() { // implementation omitted } } The key API here is JobOperator, which Jakarta Batch provides as the interface for starting, stopping, restarting, and inspecting jobs. In this example, the segmentation policy is serialized into the job parameters to ensure each execution gets the correct business rules. Reading the Input The first batch artifact, CustomerItemReader, extends Jakarta Batch’s AbstractItemReader to implement a chunk-oriented reader. Java @Named("customerItemReader") @Dependent public class CustomerItemReader extends AbstractItemReader { @Inject private CustomerRepository customerRepository; private List<Customer> customers = List.of(); private int nextIndex; @Override public void open(Serializable checkpoint) { try (Stream<Customer> customerStream = customerRepository.findAll()) { customers = customerStream .sorted(Comparator.comparing(Customer::getId)) .toList(); } nextIndex = checkpoint instanceof Integer index ? index : 0; } @Override public Customer readItem() { if (nextIndex >= customers.size()) { return null; } return customers.get(nextIndex++); } @Override public Serializable checkpointInfo() { return nextIndex; } } These methods are part of the Jakarta Batch reader lifecycle defined by AbstractItemReader. open() prepares the reader and accepts a previous checkpoint if available. readItem() provides the next item to the runtime; returning null indicates there is no more input. checkpointInfo() reports the reader’s current position for checkpointing. For simplicity, this sample loads customers into memory. For larger workloads, the implementation might use pagination or a MongoDB cursor without changing the Jakarta Batch model. Processing Each Customer The next artifact implements Jakarta Batch’s ItemProcessor interface. Java @Named("customerTierProcessor") @Dependent public class CustomerTierProcessor implements ItemProcessor { @Inject @BatchProperty( name = CustomerSegmentationPolicy.JOB_PARAMETER) private String thresholdsJson; private CustomerSegmentationPolicy policy; @PostConstruct void initialize() { policy = CustomerSegmentationPolicy.fromJson( thresholdsJson); } @Override public Customer processItem(Object item) { if (!(item instanceof Customer customer)) { throw new IllegalArgumentException( "Expected a Customer item"); } CustomerTier calculatedTier = policy.tierFor(customer.getTotalSpent()); if (calculatedTier == customer.getTier()) { return null; } return Customer.builder() .id(customer.getId()) .name(customer.getName()) .totalSpent(customer.getTotalSpent()) .tier(calculatedTier) .build(); } } Here the Jakarta Batch contract is explicit: ItemProcessor defines processItem(). The runtime calls that method for every item produced by the reader. The processor applies the segmentation rule and either returns the transformed customer or null. Returning null has a specific meaning in Jakarta Batch: the item is filtered and does not continue to the writer. The @BatchProperty is also part of the Batch integration. It receives the thresholds property defined for this job execution, allowing the processor to reconstruct the CustomerSegmentationPolicy before processing begins. Writing the Results The final artifact extends AbstractItemWriter, Jakarta Batch’s base implementation for writing a chunk. Java @Named("customerItemWriter") @Dependent public class CustomerItemWriter extends AbstractItemWriter { @Inject private CustomerRepository customerRepository; @Override public void writeItems(List<Object> items) { List<Customer> customers = items.stream() .map(this::toCustomer) .toList(); customerRepository.saveAll(customers); } private Customer toCustomer(Object item) { if (item instanceof Customer customer) { return customer; } throw new IllegalArgumentException( "Expected a Customer item"); } } writeItems() is defined by the Jakarta Batch writer contract inherited from AbstractItemWriter. Unlike the processor, which receives one item at a time, the writer receives a collection of processed items. In this case, the collection contains only customers whose classification changed, as the processor has already filtered the others. At this point, the Java components of the pipeline are as follows: Plain Text CustomerItemReader extends AbstractItemReader ↓ CustomerTierProcessor implements ItemProcessor ↓ CustomerItemWriter extends AbstractItemWriter These types are what connect the application code to the Jakarta Batch runtime. Connecting the Artifacts With JSL The Java classes define the behavior, but Jakarta Batch requires explicit mapping of the reader, processor, and writer to each job. This orchestration is described in JSL: XML <?xml version="1.0" encoding="UTF-8"?> <job id="customer-segmentation" xmlns="https://jakarta.ee/xml/ns/jakartaee" version="2.0"> <step id="recalculate-customer-tiers"> <chunk item-count="20"> <reader ref="customerItemReader"/> <processor ref="customerTierProcessor"> <properties> <property name="thresholds" value="#{jobParameters['thresholds']}"/> </properties> </processor> <writer ref="customerItemWriter"/> </chunk> </step> </job> The ref values correspond directly to the names declared with @Named in the Java classes: Java @Named("customerItemReader") @Named("customerTierProcessor") @Named("customerItemWriter") The XML therefore tells the Jakarta Batch runtime: for this step, use this reader, then this processor, and finally this writer. It also maps the thresholds job parameter into the processor property. The item-count="20" sets the chunk size for this sample. Jakarta Batch coordinates reading and processing, periodically invoking the writer according to the chunk lifecycle and establishing transaction and checkpoint boundaries. The value 20 is for demonstration; real applications should tune chunk size based on processing cost, database behavior, transaction size, throughput, and recovery requirements. This structure is recommended for the article: present the class declaration first, then describe the lifecycle methods inherited from or required by Jakarta Batch. This approach helps the sample teach the API rather than simply presenting isolated methods. Conclusion Jakarta Batch is valuable because it transforms large-scale data processing into a structured execution model, eliminating the need for custom loops and ad hoc background logic. By separating reading, processing, and writing, and introducing runtime concepts such as jobs, steps, checkpoints, restartability, and chunk-oriented execution, it provides enterprise applications with a predictable approach to handling workloads involving thousands or millions of records. This allows implementations to focus on business logic, while the Batch runtime manages repetitive execution concerns, making the model easier to understand, optimize, and scale as workloads increase. More

Distributing Massive AI Models With Network-Layer Multicast

By Vijayananda jayaraman

Best Practices for Handling Bad Data in Stream Processing Platforms

By Gautam Goswami

CORE

Securing Branch Networks With Firewalls, VPNs, IDS/IPS, and Identity-Based Access

By Kamal chand Narra

Why Do Some Proxies Work Fine for Search But Fail Once You Start Filtering Results?

Automated web scraping, market intelligence data gathering, and large-scale search engine extraction platforms frequently hit an invisible wall. A collection of proxy IPs might execute initial search queries flawlessly, yielding a standard 200 OK status code and complete HTML payloads. However, the exact same backend application might immediately throw 403 Forbidden errors, encounter endless CAPTCHAs, or receive empty JSON responses the millisecond it applies structural filters — such as sorting by price, filtering by date range, or toggling deep category facets. To the application engineer, this behavior feels contradictory. If a network endpoint successfully authenticates, circumvents initial perimeter defenses, and extracts data from a root search page, why does a simple query parameter modifier trigger an immediate failure? Resolving this requires looking past simple HTTP status codes and examining how modern application security layers, distributed databases, and stateful networking layers interact. 1. Asymmetric Security Policies Across Application Layers Modern enterprise web architecture rarely relies on a single monolithic firewall. Instead, engineering teams route inbound traffic through multi-tiered infrastructure, consisting of an Edge Web Application Firewall (WAF), an API gateway, and individual backend microservices. Root search queries are frequently cached aggressively at the edge layer using Content Delivery Networks (CDNs). When a scraper requests the first page of a popular search term, the edge proxy handles the response immediately using cached static assets. Because the request never hits the primary database cluster, security infrastructure keeps the threat verification thresholds intentionally low to maximize throughput. Applying a strict data filter changes the operational footprint: Bypassing the cache engine: Custom parameter combinations (e.g., ?sort=price_asc&min_price=150&date=24h) create unique query strings that miss the CDN cache entirely.Dynamic query compilation: The request must penetrate directly to the core application code and database layer to compile a live dataset.Elevated security sensitivity: Because dynamic database execution consumes massive memory and CPU resources, backend security components apply significantly stricter rate-limiting thresholds and advanced behavioral analysis to filtered endpoints compared to public root URLs. 2. Advanced Fingerprinting and Stateful Behavioral Tracking When an automated script issues a baseline search request, it presents a set of connection attributes. Sophisticated security systems use this initial interaction to establish a baseline state, rather than blocking the IP instantly. Session and IP Footprint Inconsistencies If an application routes requests through standard datacenter proxies, the TCP/IP stack reveals distinct server signatures. Many modern anti-bot frameworks track the progression of a user journey. A legitimate human workflow naturally flows from a broad search query to localized filtering. If a client moves from a highly cached root query directly to resource-intensive processing pages within milliseconds, security engines cross-reference the client's network layer footprint. If the initial request used a rotating proxy line that abruptly shifts TCP sequence numbers, TLS session IDs, or cookies between the search phase and the filtering phase, the security perimeter flags the behavioral state machine as anomalous. High-Volume Query Traversal To extract filtered data systematically, automation loops often iterate through complex arrays of query parameters simultaneously: Python # A typical programmatic loop that exposes a weak proxy infrastructure categories = ["electronics", "apparel", "home"] pricing_structures = ["low", "medium", "high"] for category in categories: for price_tier in pricing_structures: execute_filtered_search(category, price_tier) When an application switches from general queries to rapid, concurrent execution of complex parameter strings, it sets off heuristic anomalies. If the underlying proxy network lacks deep pool diversity or advanced session sticky logic, the target's edge firewall aggregates the client's behavior across those parameters and drops the connections cleanly. To prevent these stateful anomalies from triggering blocks during complex data manipulation steps, network engineers utilize specialized architectures, which support granular switching between high-concurrency dynamic rotation for broad collection phases and static residential ISP connections to sustain long-duration, persistent sessions when deep structural filtering is required. 3. Parameter-Induced Payload Anomalies and TLS Profiling A query parameter change alters the raw HTTP payload string sent across the wire. This modification exposes the HTTP client or request library's default behavioral patterns to deep packet inspection engines. Query String Ordering and JA3 Fingerprints Many automated scrapers built on standard request frameworks (such as Python's requests or Node.js axios) pass query parameters as raw key-value dictionaries. Depending on how the underlying library serializes data into a string, the exact sequence of parameters may not match the explicit structure generated by modern browsers. Anti-bot systems combine this structural layout with a client's JA3 TLS fingerprint. A JA3 fingerprint hashes specific parameters found within the Client Hello packet during the cryptographic handshake, including: TLS versionAcceptable cipher suitesExtension listsElliptic curvesElliptic curve formats If a client sends a standard root search query, a mismatching JA3 profile might only trigger a soft warning score. But when that same client requests a highly specific data filter — a behavioral pattern that consumes higher resource costs — the security system evaluates the warning score against a much tighter tolerance threshold, dropping the connection immediately. 4. Cryptographic Validation and Forced Client Challenges When a user targets deep structural filter routes, advanced application firewalls often issue silent cryptographic challenges, such as Proof-of-Work (PoW) scripts or dynamic JavaScript injection, to verify client authenticity before executing database queries. A basic proxy setup merely passes raw text and network packets back and forth. It has no way to evaluate or solve a JavaScript execution request natively. If your automated pipeline uses a simple HTTP client rather than a fully coordinated headless browser configuration (like Playwright or Puppeteer) capable of solving these dynamic challenges on the fly, the request fails precisely at the filtering step. The root page works because it did not require a challenge, while the complex filter endpoint demands explicit client execution verification. Mitigating Filtering Failures: Engineering Checklists To build resilient data collection pipelines capable of executing complex filtering workflows without triggering continuous network rejections, development teams should implement the following structural optimizations: Decouple the network stack from the automation logic: Ensure your infrastructure abstracts request coordination, allowing headers, cookies, and TLS handshakes to remain completely uniform while parameters rotate.Implement structural header sanitization: Ensure HTTP headers (such as User-Agent, Accept-Language, Sec-Ch-Ua, and Authorization) maintain strict chronological and case-sensitive order across all downstream filtering pipelines.Normalize parameter serialization: Match the exact parameter encoding and serialization sequence used by real browsers. Avoid random dictionary serialization; instead, construct query parameters explicitly using deterministic arrays or ordered maps.Enforce intelligent session persistence: For deep filtering journeys, leverage sticky session proxy lines to maintain a single, unbroken TCP connection and TLS context throughout the entire user funnel, switching back to dynamic rotation only when initiating entirely new search scopes.

By xiyun chen

Architecting Autonomous Network Ecosystems: From Reactive Monitoring to Agentic AI Orchestration

Agentic AI systems represent a paradigm shift in network operations, facilitating the transition from traditional reactive monitoring to fully autonomous management frameworks. For global infrastructure leaders, these specialized AI agents serve as persistent digital engineers, providing round-the-clock expertise across deployment, maintenance, and complex troubleshooting lifecycles. The following blueprint delineates the strategic application of agentic AI within a global enterprise network operations environment. Architectural Blueprint: Multi-Agent Network Engine Rather than deploying a monolithic AI entity, the optimal architecture utilizes a multi-agent system (MAS). In this model, specialized agents collaborate through a centralized Orchestrator to achieve complex operational objectives. Please refer to the System Hierarchy and Flow, as well as the Strategic Vision and Ecosystem diagrams. System Hierarchy and Flow Strategic Vision and Ecosystem Core Use Cases and Agent Workflows Autonomous Provisioning and Zero-Touch Deployment Functional scope: Serves as an automated configuration and systems engineer.Workflow: Human inputs a high-level intent (e.g., "Provision a new Cisco core switch for the Seoul R&D campus").Design Agent reads existing topology diagrams and ipAM (IP Address Management) databases.Configuration Agent generates vendor-specific CLI configs (Cisco IOS/NX-OS).Validation Agent runs the config in a virtual sandbox (e.g., Cisco Modeling Labs) to check for routing loops before pushing to production via Ansible. 24/7 Autonomous Incident Triage and Resolution Functional scope: Functions as the primary intelligence layer for Network Operations Center (NOC) alerts.Workflow: A monitoring tool (e.g., Splunk, SolarWinds) triggers a "BGP neighbor down" alert.Triage Agent intercepts the alert and instantly queries the device via SSH or telemetry API.Diagnostic Agent executes diagnostic scripts (show ip bgp summary, traceroute), analyzes logs, and identifies the root cause (e.g., a flapping fiber link).Resolution Agent opens a Jira ticket, attaches all logs, attempts a safe automated fix (like shifting traffic to a backup path), and pages a human only if the physical hardware needs replacement. Predictive Maintenance and Capacity Optimization Functional scope: Operates as a proactive infrastructure strategist and planner.Workflow: Analysis Agent continuously monitors interface bandwidth, memory usage, and optical transceivers' error rates across global data centers.Forecasting Agent identifies trends (e.g., "Link X between Vietnam and Korea factories will hit 95% capacity in 30 days during peak hours").Planning Agent drafts a change management proposal recommending a bandwidth upgrade or traffic re-routing plan for human review. Automated Compliance and Security Patching Functional scope: Functions as a continuous vulnerability and configuration compliance auditor.Workflow: Cisco releases a critical security advisory for an OS vulnerability.Auditor Agent scans the entire inventory database to find all vulnerable device models.Patching Agent schedules a maintenance window, backs up current configurations, downloads the verified firmware, applies the patch sequentially to avoid downtime, and performs post-flight checks. Implementation Strategy To ensure organizational stability while maximizing technological advancement, a phased implementation strategy is recommended: Phase 1: Read-Only Agents (Information Gathering) Allow agents to access logs, APIs, and read-only commands.Focus on automated documentation, ticket enrichment, and summarizing alerts. Phase 2: Human-in-the-Loop (Co-Pilot) Allow agents to generate fixes and configuration scripts.Require a senior network engineer to review and click "Approve" before any execution. Phase 3: Guardrailed Autonomy (Full Agentic) Give agents autonomous execution rights only for low-risk, repetitive tasks (e.g., port resets, clearing stuck sessions).Enforce hard boundaries using API rate limits and strict verification checklists. To refine the architectural requirements for your specific environment, please provide insights on the following: What monitoring tools (e.g., Datadog, ServiceNow) and automation frameworks (e.g., Ansible, Terraform) does your team currently use?What is the most repetitive issue your 24/7 NOC team spends time fixing manually?Are you looking to build these agents using commercial LLM APIs or host open-source models locally for security reasons? Building an Enterprise-Grade Agentic System Constructing an enterprise-grade agentic system for high-security environments requires the integration of a frontier large language model (LLM), a robust production agent framework, and the Model Context Protocol (MCP) to establish secure interoperability with core network infrastructure. Part 1: Choosing the Right Tool/Solution Do not rely on a simple chatbot web interface. You need an API-driven architecture that combines a foundational model with an agent framework. The Foundational Model (The Brain) Top Choice: Anthropic Claude 3.5 Sonnet / Claude 3 Opus Why: It currently dominates tool-use (function calling), reasons through complex, multi-step engineering logic flawlessly, and natively supports MCP. On-Premises / Air-Gapped Alternative: Llama 3.1 / 3.3 (70B or 405B) or Qwen 2.5 (72B) Why: For internal Samsung networks with strict data privacy laws, you can host these open-source models locally using vLLM or Ollama. They have excellent coding and structured JSON output capabilities. The Agent Framework (The Backbone) Use an open-source framework to manage agent memory, states, and collaboration: LangGraph (by LangChain): Best for networking. It allows you to build cyclical, graph-based agent workflows with strict state control and mandatory human-in-the-loop approval stages.CrewAI: Great for quickly setting up role-based multi-agent teams (e.g., Triage Agent talking to a Scripting Agent). Part 2: Understanding MCP (Model Context Protocol) MCP is an open standard created by Anthropic that acts as a secure plug-and-play adapter between LLMs and local/remote data sources, tools, and systems. Instead of writing custom API integration code for every single Cisco switch, Ansible Tower, or Jira server, you build or use an MCP Server. How it works: Your Agent (MCP Client) connects to an MCP Server. The MCP Server exposes specific tools (e.g., run_cisco_command, query_monitoring_alerts) to the agent dynamically. Security Benefit: The model never gets direct SSH access to your core switches. The model only talks to the MCP Server, which enforces strict argument validation, sanitization, and logging before executing anything on the network. Part 3: Writing agents.md The agents.md file defines the architecture, roles, personas, boundaries, and collaboration patterns of your AI team. Markdown # Network Operations Multi-Agent Architecture This document defines the specialized AI agents operating within the Network Operations Center (NOC). ## 1. System Orchestrator Agent - **Role:** Central Dispatch & Intent Router - **Goal:** Analyze incoming alerts or human engineer requests and delegate tasks to specialized agents. - **Persona:** A highly organized, senior technical project manager. - **Boundaries:** Does not execute network commands directly. Must always route tasks and compile final reports. ## 2. Network Diagnostics Agent (NetDiag-Agent) - **Role:** Incident Triager and Log Analyst - **Goal:** Investigate network anomalies, verify device status, and pinpoint root causes. - **Persona:** An analytical Cisco CCIE-certified troubleshooting expert. - **Tools Allowed:** `ssh_read_only_commands`, `query_splunk_logs`, `ping_traceroute`. - **Boundaries:** Read-only access to infrastructure. Cannot modify configurations. ## 3. Network Automation Agent (NetAuto-Agent) - **Role:** Configuration and Deployment Engineer - **Goal:** Generate valid, syntax-correct network configurations and execute approved automation scripts. - **Persona:** A precise Network Automation Engineer specializing in Ansible, Python (Netmiko/Nornir), and Cisco IOS/NX-OS. - **Tools Allowed:** `generate_ansible_playbook`, `validate_config_syntax`, `execute_stage_change`. - **Boundaries:** Write-access allowed *only* through staging/sandbox tools. Any production push requires an explicit `Human-In-The-Loop` (HITL) approval token. Part 4: Writing skills.md The skills.md file maps out the actual capabilities, API tools, and Python execution blocks that the agents can tap into via MCP. Markdown # Network Agent Skills & Tool Definitions This document details the functional tools exposed to agents via the MCP (Model Context Protocol) layer. ## Category: Network Telemetry & Read Skills ### `query_device_status` - **Description:** Fetches real-time operational data from a specific network device using Netmiko or Cisco DNA Center API. - **Arguments:** - `device_ip` (string, required): The target management IP address. - `command` (string, required): Allowed values: `show ip interface brief`, `show ip bgp summary`, `show environment`. - **Safety:** Input string regex validation enforced to prevent CLI injection attacks. ### `fetch_noc_alerts` - **Description:** Pulls active high-severity network alerts from Splunk/SolarWinds. - **Arguments:** - `severity` (string): Defaults to "CRITICAL". - `lookback_minutes` (integer): Time window to check. ## Category: Network Modification Skills (Write) ### `deploy_ansible_playbook` - **Description:** Triggers an Ansible AWX template to push configuration updates to a device group. - **Arguments:** - `template_id` (string, required): Pre-defined template ID in Ansible. - `extra_vars` (json, required): Key-value pairs for variables (e.g., target VLAN, interface ID). - **Security Policy:** **CRITICAL_APPROVAL_REQUIRED**. This skill automatically pauses agent execution and sends a webhook to the human supervisor's teams/Slack app for a 2FA confirmation click. Next Steps for Implementation To help map out a proof of concept (PoC) for your architecture, tell me: Do your network infrastructure components support REST APIs/gRPC, or do the agents need to interface heavily via legacy SSH/CLI parsing?Do you plan to build this as an internal web-dashboard/chatbot for engineers, or as a background event-driven system triggered by network monitoring systems?What programming language (e.g., Python, TypeScript) is your team's preference for building the MCP servers? Check out more from my series here.

By Daniel Oh

CORE

From Polling to PubSub: Building an Asynchronous OPC UA Stack in Python

Industrial control systems are generating more data than ever before, but the Python tooling used to process this telemetry often encounters severe performance constraints. Traditional OPC UA libraries are built around synchronous, polling-based Client and Server architectures. When industrial networks scale to thousands of sensors broadcasting high-frequency data, these synchronous Python implementations choke. To handle this modern many-to-many topology, developers need a native Publisher and Subscriber solution that does not block the execution thread while waiting for network packets. For Python developers unfamiliar with industrial protocols, OPC UA PubSub (IEC 62541-14) is a standard that decouples data producers from consumers by allowing devices to broadcast telemetry via stateless middleware like UDP Multicast. For industrial engineers new to Python concurrency, asyncio is a standard library that uses an event loop to handle thousands of simultaneous network operations concurrently without the heavy overhead of traditional threading. Bridging these two paradigms requires a completely non-blocking architecture. To address this gap, a complete asyncio driven OPC UA PubSub implementation was architected and integrated into the open source opcua-asyncio library (merged in Commit 2b6f3e5). Implementing this standard from scratch in an asynchronous Python environment presented unique challenges. This article breaks down the engineering decisions and technical design patterns used to build this extension. By contributing this capability to a library that serves thousands of developers in the Python IIoT ecosystem, the goal is to ensure engineers can now build highly scalable publisher and subscriber sensor networks without migrating away from Python. The Shift to Publisher and Subscriber in IIoT In traditional OPC UA, a client polls a server or sets up monitored items. This creates a tightly coupled, connection-oriented topology. The PubSub extension decouples this by allowing publishers to broadcast telemetry data via stateless middleware like UDP Multicast or MQTT, which subscribers can passively ingest. To bring this to the opcua-asyncio ecosystem, the architecture needed to bridge the gap between Python's asynchronous event loop and the highly deterministic, byte-packed UADP (OPC UA Datagram Protocol) structures. The design was broken down into four core pillars. Asynchronous transport layer: Managing non-blocking UDP and IP multicast.UADP binary protocol engine: Bit-level packing and unpacking of network messages.Data abstraction and node mapping: Linking arbitrary network payloads to the OPC UA Address Space.Concurrency and connection management: Orchestrating readers, writers, and tasks via asyncio. Pillar 1: The Asynchronous UDP Transport Layer OPC UA UADP relies on UDP for low-latency transmission. In Python, synchronous socket operations block the main thread, which is fatal to an asyncio application. To solve this, the networking layer was built directly on top of asyncio.DatagramProtocol. The OpcUdp class overrides the standard protocol callbacks to bridge the network socket with the PubSub receiver logic. Here is a look at how the protocol was extended and hooked into the event loop to ensure incoming datagrams never block the main thread. Python class OpcUdp(asyncio.DatagramProtocol): def __init__(self, cfg: UdpSettings, receiver: Optional[PubSubReceiver], publisher_id: Variant) -> None: super().__init__() self.cfg = cfg self.receiver = receiver self.publisher_id = publisher_id.Value def datagram_received(self, data: bytes, source: Tuple[str, int]) -> None: try: buffer = Buffer(data) msg = UadpNetworkMessage.from_binary(buffer) if self.receiver is not None: asyncio.ensure_future(self.receiver.got_uadp(msg)) except Exception: logging.exception("Received Invalid UadpPacket") Socket lifecycle: The UdpSettings class manages socket creation by carefully applying SO_REUSEADDR and handling both IPv4 (AF_INET) and IPv6 (AF_INET6) multicasting.Multicast configuration: Depending on the IP family, IP_ADD_MEMBERSHIP or IPV6_JOIN_GROUP are injected directly into the socket options via the struct module to ensure the application correctly subscribes to IGMP or MLD groups.Non-blocking reception: When a datagram hits the interface, datagram_received immediately passes the raw bytes to the UADP decoding engine and dispatches the resulting parsed message to a background task using asyncio.ensure_future(). This guarantees the networking thread is instantly freed to handle the next packet. Pillar 2: The UADP Binary Protocol Engine The UADP specification defines an extremely dense, highly variable network packet. Headers can dynamically expand or contract based on a series of bit flags. Processing this in Python requires rigorous byte manipulation to maintain both memory efficiency and processing speed. The uadp.py implementation utilizes Python's enum.IntFlag to map the exact bitwise schemas defined in OPC UA Part 14. Python class MessageHeaderFlags(IntFlag): NONE = 0 UADP_VERSION_BIT0 = 0b1 PUBLISHER_ID = 0b00010000 GROUP_HEADER = 0b00100000 PAYLOAD_HEADER = 0b01000000 EXTENDED_FLAGS_1 = 0b10000000 # FlagsExtend1 PUBLISHER_ID_UINT16 = 0b0000000100000000 PUBLISHER_ID_UINT32 = 0b0000001000000000 PUBLISHER_ID_UINT64 = 0b0000011000000000 PUBLISHER_ID_STRING = 0b0000010000000000 Flag-driven serialization: The UadpHeader and UadpDataSetMessageHeader are deeply nested and conditional. For example, the Extended Flags dictate whether a PublisherId is encoded as a Byte, UInt16, UInt32, UInt64, or String.Bitwise extensibility: The implementation cascades flags using EXTENDED_FLAGS_1 and EXTENDED_FLAGS_2 bits. If the integer value of the required flags exceeds 0xFF, the engine dynamically shifts the bytes and appends the extension flags.Binary packing: A standardized Primitives unpacking utility translates the raw buffer directly into strictly typed Python objects like UInt32, Guid, or DateTime. This avoids the overhead of intermediate object instantiation when parsing high-frequency sensor data.Delta Frames vs. raw data: The parser dynamically routes payload deserialization based on MessageDataSetFlags. It distinguishes between Key Frames, Delta Frames, and Raw Data while packing the resulting generic DataValue structs into a unified UadpNetworkMessage. Pillar 3: Data Abstraction and Address Space Integration Receiving data is only half the battle because that data must meaningfully map to the server's Address Space. The architecture introduces PubSubInformationModel to handle this synchronization. Datasets and metadata: A PublishedDataSet defines the structure of the data being transmitted. This includes tracking FieldMetaData, built in types, and value ranks.Dynamic variable substitution: The PubSubDataSourceServer class abstracts the retrieval of data from the server tree. It safely reads attributes and falls back to a SubstituteValue if a node status code is bad. This ensures unbroken telemetric streams.Subscribed mirrors: When an OPC UA client acts as a subscriber, it needs to see the incoming data reflected in its own node tree. The SubscribedDataSetMirror dynamically creates new variable nodes on the fly to match the incoming DataSetMetaData. This dynamic node mapping was engineered by injecting new variables straight into the server tree based on the metadata specification. Python async def _create_and_set_node(self, f: FieldMetaData): if self._node is None: raise RuntimeError("SubscribedDataSetMirror._node is not initialized.") n = await self._node.add_variable( NodeId(NamespaceIndex=Int16(1)), "1:" + str(f.Name), Variant(), datatype=f.DataType ) await n.write_attribute(AttributeIds.Description, f.Description) await n.write_attribute(AttributeIds.ValueRank, f.ValueRank) await n.write_attribute(AttributeIds.ArrayDimensions, f.ArrayDimensions) return n Target variables: Alternatively, SubScribedTargetVariables maps incoming dataset fields directly to existing NodeId references in the server. These references update in real time as UDP packets are decoded. Pillar 4: Concurrency and Connection Management The top-level orchestration is handled by the PubSubConnection and PubSub classes. These act as the asynchronous lifecycle managers. Task gathering: When start() is invoked on a connection, the lifecycle manager utilizes asyncio.gather() to concurrently spin up all associated DataSetReader and DataSetWriter tasks without blocking the main OPC UA server loop. Python async def start(self) -> None: logging.info("Starting Connection %s", await self.get_name()) loop = asyncio.get_event_loop() sock, _, _ = self._network_settings.create_socket() self._transport, self._protocol = await loop.create_datagram_endpoint( lambda: self._network_factory(self._network_settings, self._receiver, self._cfg.PublisherId), sock=sock, ) self._writer_tasks = asyncio.gather(*[writer.run(self._protocol, self._app) for writer in self._writer_groups]) reader_tasks = asyncio.gather(*[reader.start() for reader in self._reader_groups]) await reader_tasks if self._protocol is not None: self._protocol.set_receiver(self._receiver) await self._set_state(PubSubState.Operational) Protocol decoupling: To prevent circular dependencies between the network transport and the information model, strict interfaces defined in protocols.py are used. The UDP protocol layer communicates with the logical layer strictly through these abstract protocols.Wildcard routing and readers: The ReaderGroup acts as an intelligent multiplexer. When a multi-payload UADP packet arrives, it analyzes the GroupHeader and DataSetPayloadHeader. It then routes individual DataSetMessages to the correct DataSetReader instances by matching wildcard filters.Timeouts and state machines: Robust industrial systems must handle connection drops. The DataSetReader wraps its operation in a dedicated timeout task. Using asyncio.wait_for(), it actively monitors for MessageReceiveTimeout events. If a heartbeat or payload is missed, it transitions the internal PubSubState to Error. This allows higher-level application logic to gracefully degrade. Conclusion Building a production-ready OPC UA PubSub stack in Python requires harmonizing the stringent bit-packed demands of the IEC 62541-14 specification with the asynchronous paradigms of asyncio. By leveraging asyncio.DatagramProtocol for deterministic networking, abstracting the UADP bit flags into structured classes, and deeply integrating with the OPC UA Address space via mirrored target variables, this implementation provides a scalable foundation for modern IIoT architectures. Code and Open Source Contributions The architecture and implementation details discussed in this article were merged into the core FreeOpcUa/opcua-asyncio repository. You can explore the complete implementation, including the raw protocol parsing and asyncio abstractions, via the links below. Primary commit: 2b6f3e5 (Initial implementation of OPC UA PubSub UDP and UADP). Key files to explore in the commit: asyncua/pubsub/udp.py: Contains the OpcUdp transport layer and multicast socket configuration.asyncua/pubsub/uadp.py: Houses the flag driven serialization and binary protocol engine.asyncua/pubsub/connection.py: Demonstrates the asyncio task management and lifecycle orchestration.

By Harshith Narasimhan Srivatsa

Solving Data Traffic Jams in Your Network

Stop, start. Stop, start. Nothing brings data flows to a grinding halt (or raises an admin’s blood pressure) quite like network congestion. The unwanted, unexpected extra step in an information request or response operation chain is a telltale sign that something’s changed or isn’t working in your infrastructure. And heavier traffic is more than just an inconvenience – it’s a multifaceted problem with knock-on business effects that falls upon admins to identify and fix. Let’s dig deeper into network traffic jams, their primary causes, and how to resolve and prevent them. Understanding What Causes a Digital Traffic Jam Network congestion occurs when the demand for sending or receiving data exceeds the network’s capacity. In other words, a computer network link can’t handle the volume of data trying to use it. It’s like what happens when a person tries to pour more water through a straw than it can handle at once. At a certain point, there’s simply not enough space, causing a backup in the straw. In computer networks, when data packets exceed the network’s capacity, they’re similarly queued in network devices, leading to increased latency and, in turn, traffic jams. 7 Most Common Causes of Network Congestion Bandwidth bottlenecks: When the capacity of network links (such as cables or wireless connections) is insufficient to handle the amount of data being sent.Network device limitations: Routers, switches, and other devices have limited processing power and memory and can become overwhelmed when handling large volumes of traffic.Broadcast storms: A situation where a network becomes flooded with broadcast or multicast packets, often caused by misconfigured devices or faulty hardware.High-bandwidth applications: Applications that consume a lot of network resources, such as video streaming, large file transfers, and backup operations.DDoS attacks: A distributed denial-of-service (DDoS) attack occurs when a network is intentionally flooded with excessive traffic from multiple sources.Poor network architecture: Inefficient routing or inadequate network capacity planning can lead to congestion hotspots.Insufficient internet speeds: Slow service-provider connections can cause bottlenecks at the edge of the network. Performance and Business Consequences of Network Congestion The consequences of network congestion extend far beyond the digital realm, wreaking havoc on your entire IT infrastructure. As data packets get caught in the congestion chaos, you’ll see increased latency and sluggish application performance. Network devices, overwhelmed by traffic, might then start dropping packets, causing retransmissions that add more load and exacerbate congestion. Worse, applications can start to time out because they can’t handle the lengthy delays in data transmission, further compounding the problem. You're also likely to notice jitter, or uneven packet delays, that affect real-time applications like VoIP and video conferencing. Network throughput suffers too, with the overall amount of data that can be transmitted over the network taking a nosedive. Ultimately, users soon begin to notice this digital snarl-up, with slow network performance leading to a decline in productivity and potentially a negative impact on your bottom line. Quality of Service (QoS) for critical applications can degrade as they struggle to receive the priority they need amid congestion. The overarching message is that network congestion can have serious repercussions on performance, end-user experience, and business operations as a whole. Maintaining healthy network traffic is about speed, sure, but it’s also about supporting day-to-day operations. 10 Proven Solutions for Fixing Bad Network Traffic This doesn’t need to be the network status quo. Here’s how admins can and should take back control: Bandwidth management and QoS: Implement QoS policies to prioritize important traffic, effectively creating an express lane for your VIP data packets. Use traffic shaping to control data flow and prevent one application from hogging all the bandwidth.Network segmentation: Divide your network into smaller subnets to contain congestion and prevent a problem in one area from spreading like wildfire.Upgrade network infrastructure: Sometimes you just need more oomph. Upgrade your network devices, increase link capacities, and consider SDN for greater flexibility in traffic management. Optimize application performance: Collaborate with your development teams to improve network efficiency via data compression and caching.Implement caching and Content Delivery Networks (CDNs): For frequently accessed data or web content, use caching or CDNs to lighten the load on your primary network and improve data transfer speeds.Regular network performance monitoring and analysis: Keep a watchful eye on your network performance to identify congestion points and proactively address network issues before they spiral out of control.Load balancing: Distribute network traffic across multiple paths or servers to prevent any single point from becoming a bottleneck.Traffic prioritization: Prioritize critical unicast and multicast traffic over less important data flows.Optimize routing: Regularly review and optimize routing protocols and configurations to ensure efficient traffic flow.Firewall optimization: Ensure your firewalls are properly configured and can handle the traffic load; poorly configured or underpowered firewalls can become network bottlenecks. Keeping Data Speeds Up and Bottom Line Impacts at Bay Again, this is more than about speed (or lack thereof), but the impact of bad network traffic and how it can become a serious business problem. The good news is that it doesn’t have to be a digital death sentence for your IT infrastructure. With a combination of smart network management strategies and the right monitoring tools, you can effectively tackle network congestion and keep your network in the fast lane.

By Sascha Neumeier

Parallel Kafka Batch Processing With Kotlin Coroutines in Spring Boot

Managing high-volume message traffic in distributed architectures is crucial. Efficient use of database and CPU resources is also very important. There are structures that allow us to receive messages in batches. The default Spring Kafka "BatchMessageListener" structure addresses this need. However, the processing of these messages often goes through a sequential bottleneck. This article will discuss the structure and usage of Kotlin Coroutines in detail. We will examine how to maximize Kafka message processing performance using Structured Concurrency principles and Resource Throttling techniques. Architectural Bottleneck: Sequential I/O Blocking On the current Kafka listener: Database or external service calls made for each message directly increase total processing times. If the processing speed of a message lags behind the message arrival speed and the max-poll-interval-ms time is exceeded, the consumer is removed from the consumer group. Rebalancing is triggered, and the partitions of that consumer are redistributed to other consumers in the group. Kotlin @KafkaListener(topics = ["usage-pool-topic"]) fun usagePoolListener(records: List<ConsumerRecord<String, String>>) { records.forEach { record -> processRecord(record) // Network latency + DB I/O blocking } } Solution 1. Batch-Fetch and In-Memory Map Structure Before any concurrent code is entered, data is retrieved collectively from all necessary entities. Multiple separate queries are converted into a batch query before data processing begins. The N+1 query problem is solved at the application layer. All data is cached once before being broken down into concurrent operations. Having the data cached significantly reduces our reliance on the database. Using the associateBy function, we transform the data into a map structure with X access times. This allows us to read the data safely from the maps instead of reading each concurrent operation from the database. Kotlin val messages = records.map { objectMapper.readValue(it.value(), UsagePoolRecord::class.java) } val usagePoolEntities = usagePoolRepository .findByIds(messages.map { it.usagePoolId.toBigInteger() }) .associateBy { it.usagePoolId } val lockEntities = lockRepository .findByUserIds(messages.map { it.userId }) .associateBy { it.userId } 2. Structured Concurrency Memory Management With Chunking The chunk structure serves two purposes. It prevents the creation of coroutines simultaneously. This prevents unnecessary memory usage. Each chunk writes to the database after all coroutines have completed their operations. Unnecessary connection pool consumption is avoided. Kotlin messages.chunked(150).forEach { chunk -> // Each chunk of 150 records is processed concurrently } Resource Isolation With limitedParallelism Why limitedParallelism? If the database connection pool has, for example, X connections, keeping the parallelism limit below X prevents "Connection Timeout" errors. Kotlin messages.chunked(150).forEach { chunk -> val deferredResults = chunk.map { record -> CoroutineScope(Dispatchers.IO.limitedParallelism(15)).async { try { processRecord(record, usagePoolEntities, lockEntities) } catch (e: Exception) { log.error("Operation error: ${record.key()}", e) buildErrorRecord(record, e) } } } val results = deferredResults.awaitAll() // Structural waiting collectAndAggregate(results) } The Dispatchers.IO.limitedParallelism(X) command limits the number of concurrent coroutines to X, preventing the DB connection pool from being exhausted.Each coroutine returns a result with the async command. The awaitAll() command waits for all coroutines in the chunk to finish before proceeding to the next step. runBlocking This function blocks callers until all concurrent operations are complete. This is the correct approach here because: It ensures that the Kafka consumer remains blocked to maintain its offset commit structure until all records in the batch are processed. We still benefit from concurrent operation parallelism within the runBlocking block. 3. Thread-Safe Result Structure After the awaitAll() operation, all results are collected in thread-safe queues. Then a single batch write operation takes place. Using MutableList structures to combine results returned from parallel processed coroutines can lead to data loss. At this point, lock-free data structures should be preferred. ConcurrentLinkedQueue uses CAS (Compare-And-Swap) algorithms instead of synchronized blocks. This provides superior performance in high-content write operations. Why Shouldn't We Use ConcurrentLinkedQueue? Concurrent operations (concurrent functions) perform simultaneous write operations to a shared collection of results. Using MutableList leads to race conditions. It performs well in secure and concurrent write operations. Kotlin data class AggregatedRecords( val processedSave: ConcurrentLinkedQueue<ProcessedEntity> = ConcurrentLinkedQueue(), val toDelete: ConcurrentLinkedQueue<UsagePoolEntity> = ConcurrentLinkedQueue(), val retryQueue: ConcurrentLinkedQueue<RetryEntity> = ConcurrentLinkedQueue() ) The DataIntegrityViolationException return is important. When two consumer instances are processing the same record, one of them falls into a unique constraint violation. Instead of making the entire batch fail, record-by-record deletion is performed. Kotlin AggregatedRecords.processedSave .chunked(150) .forEach { batch -> try { processedRepository.saveAll(batch) } catch (e: DataIntegrityViolationException) { batch.forEach { record -> try { processedRepository.save(record) } catch (e: DataIntegrityViolationException) {} } } } 4. Error Tolerance in Write Operations Batch write (saveAll) operations are performant. However, a "Unique Constraint" error in a single record can cause the entire batch to fail. The following structure is critical to meet Optimistic Locking or Idempotency requirements. Kotlin aggregatedRecords.processedSave.chunked(150).forEach { batch -> try { processedRepository.saveAll(batch) } catch (e: DataIntegrityViolationException) { // Fallback: Try one by one if batch fails batch.forEach { record -> try { processedRepository.save(record) } catch (innerException: DataIntegrityViolationException) { log.warn("Duplicate record skipped: ${record.id}") } } } } 5. Data Flow Diagram Ingress: The Kafka batch is caught with runBlocking.Preparation: All necessary context data is retrieved bulk from the DB.Execution: Coroutines are started asynchronously in chunks.Synchronization: The completion of all coroutines is awaited as a barrier point with awaitAll().Egress: Collected results are made permanent with saveAll. Performance Analysis and Results Conclusion Processing Kafka messages in Spring Boot with Kotlin Coroutines not only increases speed but also improves code readability and makes resource management deterministic (predictable). The use of runBlocking allows us to build a bridge between the blocking Kafka consumer thread and the suspended world without disrupting Kafka's offset management mechanism. Dependencies XML <dependency> <groupId>org.jetbrains.kotlinx</groupId> <artifactId>kotlinx-coroutines-core</artifactId> <version>1.7.3</version> </dependency> <dependency> <groupId>org.springframework.kafka</groupId> <artifactId>spring-kafka</artifactId> </dependency>

By Erkin Karanlık

Bringing Intelligence Closer to the Source: Why Real-Time Processing is the Heart of Edge AI

Artificial Intelligence is rapidly becoming a part of everyday devices — smartphones, cars, cameras, and even home appliances. Traditionally, these systems rely on cloud servers to send, process, and analyze data before making decisions, which increases latency and delays responses. However, many applications require instant decision-making, where even a slight delay can be critical. In such scenarios, relying on network connectivity is not always practical, and decisions need to be made locally on the device itself. This has led to a growing shift toward running intelligence directly on devices, making real-time local processing more important than ever. In this article, we’ll explore why this shift matters and how it is shaping the future of modern intelligent systems. What is Edge AI? Edge AI refers to running AI models directly on devices such as IoT systems, smartphones, autonomous cars, drones, and sensors — right where the data is generated. With this approach, there is no need to transfer data to cloud servers or centralized systems. Edge AI enables faster, real-time decision-making by processing data locally, without sending it elsewhere. For example, Instead of sending every transaction to a central server for analysis, the system can analyze transaction patterns locally in real time. If any unusual activity is detected — such as an abnormal withdrawal amount, location mismatch, or suspicious behavior — the system can instantly block the transaction or trigger an alert. Why Real-Time Processing Matters? Real-time processing means a system can process data instantly and make decisions without delay. Even small delays in decision-making can create critical situations and lead to serious consequences. For example, an autonomous car must detect obstacles and react within milliseconds. If it relies on the cloud, even a small delay could lead to serious consequences. By processing data locally, Edge AI enables immediate decisions — such as braking or steering — making the system safer and more efficient. Reduce Latency and Faster Decisions Latency is the time it takes for data to travel to the cloud and back. Even a delay of a few milliseconds can be too slow for certain applications. With Edge AI: Data is processed instantly on the device itself.There’s no need to wait for a network response.Performance is faster, more reliable, and less dependent on connectivity. For example, a voice recognition system on a smartphone can respond much faster when speech processing runs locally on the device, rather than relying on cloud or centralized servers. Improved Privacy and Data Security Sending sensitive data to the cloud raises privacy concerns, as it can be exposed during transmission or storage. Edge AI minimizes these risks by processing data directly on the local device instead of sending it to the cloud. This approach enhances data security and helps maintain user privacy, since sensitive information never leaves the device. It also supports compliance with data protection regulations and reduces the chances of unauthorized access or data breaches. For example, a healthcare wearable that monitors heart activity should not transmit sensitive personal health data to external servers. Instead, it can analyze patterns locally on the device and instantly alert the user if any irregularities are detected. This approach not only protects patient privacy but also enables faster, real-time responses in critical situations. Such local processing is especially important in industries like banking, healthcare, finance, and smart homes, where data security and immediate decision-making are essential. Reliability Without Internet Dependency Edge devices can operate even without an internet connection, making them more stable and reliable in remote areas or environments with poor network coverage. This ensures continuous performance without interruptions or delays caused by connectivity issues. As a result, critical applications can function smoothly regardless of network availability. For example, a drone used in disaster rescue operations cannot depend on internet connectivity. It must process images locally and detect survivors in real time, enabling faster and more effective rescue efforts. Lower Bandwidth Usage and Reduce Infrastructure Costs Sending large amounts of data to the cloud consumes significant bandwidth and increases operational costs. Edge AI helps reduce these costs by processing data locally on the device. This minimizes the need for constant data transmission and optimizes network usage. Only relevant or critical information is sent to the cloud, making the system more efficient and cost-effective. For example, a factory machine monitoring system can analyze sensor data locally and send alerts only when an issue is detected, instead of continuously streaming all the data. Scalability and Cost Efficiency Cloud processing for millions of devices can become expensive and resource-intensive. Edge AI addresses this by distributing computations across devices, reducing the load on central servers. This decentralized approach lowers infrastructure costs, improves scalability, and enhances overall system performance. It also reduces latency by minimizing the need for constant communication with the cloud. For example, in a smart city, thousands of cameras can process data locally instead of sending everything to a central cloud system. This not only saves bandwidth and infrastructure costs but also enables faster, real-time insights and responses. Better User Experience Real-time processing significantly improves user experience by making systems feel faster, smoother, and more responsive. Quicker responses lead to higher user satisfaction and a more seamless interaction. With Edge AI, data is processed instantly on the device, eliminating delays and ensuring consistent performance. This is especially important for applications that require immediate feedback. For example, in gaming or augmented reality (AR), local AI can render objects and interactions in real time, creating a smoother, more immersive, and engaging user experience. An edge-based platform helps by enabling data processing and decision-making directly on devices, rather than relying entirely on centralized cloud systems. It supports faster, real-time responses by analyzing data locally, which is essential for applications that require immediate action. This leads to improved performance and reliability, especially in environments with limited or unstable internet connectivity. It also enhances data privacy and security by keeping sensitive information on the device, reducing the need for data transmission. Additionally, it optimizes bandwidth usage and lowers infrastructure costs by sending only meaningful insights or alerts to central systems instead of continuous raw data. Overall, this approach helps build systems that are faster, more efficient, secure, and scalable by bringing intelligence closer to where data is generated. Conclusion Edge AI is transforming modern systems by bringing intelligence closer to where data is created, enabling faster and real-time decision-making. It reduces latency and improves performance by processing data locally instead of relying on the cloud. This approach also enhances privacy and minimizes dependence on constant internet connectivity. Additionally, it helps reduce bandwidth usage and lowers infrastructure costs. From smart cities to healthcare and industrial automation, edge computing is driving a new era of faster, smarter, and more efficient systems. Edge AI brings intelligence closer to where data is created, enabling real-time decisions, faster performance, enhanced privacy, and reliable operation without depending on constant connectivity.

By Jitendra Bafna

Building Enterprise-Grade Real-Time IoT Dashboards with Vue 3, MQTT, and Kafka

The convergence of IoT, real-time data streaming, and modern frontend frameworks is redefining how engineers build enterprise monitoring systems. Over the course of designing and leading the Device IoT Platform — an enterprise-grade solution for real-time monitoring, configuration, and diagnostics of thousands of distributed network devices — I encountered and solved a core architectural challenge: how do you build a frontend dashboard that can handle hundreds of concurrent device telemetry streams without sacrificing performance, maintainability, or user experience? This article shares the architectural patterns, technology decisions, and hard-won lessons from that journey — covering the full stack from MQTT ingestion to Vue 3 reactivity to Kafka-backed event processing. The Core Problem: Real-Time at Scale Most developers are familiar with polling — periodically fetching data from an API endpoint. For IoT, polling is fundamentally broken: Latency: A 5-second polling interval means 5 seconds of stale state.Inefficiency: You're requesting data even when nothing has changed.Scale: 1,000 devices × 1 request/5s = 200 requests/second just to read status — before any user interaction. The solution is event-driven architecture: devices push telemetry when something changes, and the platform reacts. This requires a rethinking of both backend ingestion and frontend state management. Architecture Overview Plain Text [IoT Devices] | MQTT Broker (Mosquitto / AWS IoT Core) | [Node.js Telemetry Microservice] | [Kafka Topic: device.telemetry.raw] | (stream processor) [Kafka Topic: device.telemetry.enriched] | [WebSocket Server (Node.js)] | [Vue 3 Dashboard Frontend] Each layer has a distinct responsibility: MQTT Broker handles lightweight publish/subscribe with devices using minimal overhead.Node.js microservices bridge MQTT → Kafka, performing initial validation and normalization.Kafka provides durable, replayable event streams — critical for audit trails and late-joining consumers.WebSocket server fans out enriched telemetry to connected dashboard clients in real time.Vue 3 handles reactive rendering, ensuring only the affected UI components re-render when new data arrives. Backend: MQTT → Kafka Bridge in Node.js The heart of the ingestion pipeline is a lightweight Node.js service using the mqtt and kafkajs libraries. Plain Text import mqtt from 'mqtt'; import { Kafka } from 'kafkajs'; const mqttClient = mqtt.connect(process.env.MQTT_BROKER_URL!, { clientId: `telemetry-bridge-${process.pid}`, username: process.env.MQTT_USERNAME, password: process.env.MQTT_PASSWORD, clean: true, }); const kafka = new Kafka({ clientId: 'iot-bridge', brokers: [process.env.KAFKA_BROKER!] }); const producer = kafka.producer(); mqttClient.on('connect', async () => { await producer.connect(); mqttClient.subscribe('devices/+/telemetry', { qos: 1 }); console.log('MQTT → Kafka bridge active'); }); mqttClient.on('message', async (topic, payload) => { const deviceId = topic.split('/')[1]; const data = JSON.parse(payload.toString()); await producer.send({ topic: 'device.telemetry.raw', messages: [ { key: deviceId, value: JSON.stringify({ deviceId, timestamp: Date.now(), ...data }), }, ], }); }); Key design decisions here: QoS Level 1 — ensures at-least-once delivery for telemetry messages. For command acknowledgments, we use QoS 2.Device ID as Kafka partition key — guarantees ordering per device while allowing parallel processing across partitions.Separation of raw vs. enriched topics — the device.telemetry.raw topic contains the bare payload; a downstream stream processor enriches it with device metadata, geolocation, and alert thresholds before publishing to device.telemetry.enriched. WebSocket Fan-Out Server The WebSocket layer subscribes to Kafka's enriched topic and pushes updates to connected browser clients. We use Kafka consumer groups to allow horizontal scaling of the WebSocket tier. Plain Text import { WebSocketServer } from 'ws'; import { Kafka } from 'kafkajs'; const wss = new WebSocketServer({ port: 8080 }); const kafka = new Kafka({ clientId: 'ws-fanout', brokers: [process.env.KAFKA_BROKER!] }); const consumer = kafka.consumer({ groupId: 'websocket-fanout-group' }); // Track subscriptions: deviceId → Set<WebSocket> const deviceSubscriptions = new Map<string, Set<WebSocket>>(); wss.on('connection', (ws) => { ws.on('message', (msg) => { const { action, deviceId } = JSON.parse(msg.toString()); if (action === 'subscribe') { if (!deviceSubscriptions.has(deviceId)) { deviceSubscriptions.set(deviceId, new Set()); } deviceSubscriptions.get(deviceId)!.add(ws); } }); ws.on('close', () => { deviceSubscriptions.forEach((clients) => clients.delete(ws)); }); }); async function startKafkaConsumer() { await consumer.connect(); await consumer.subscribe({ topic: 'device.telemetry.enriched' }); await consumer.run({ eachMessage: async ({ message }) => { const payload = JSON.parse(message.value!.toString()); const clients = deviceSubscriptions.get(payload.deviceId); clients?.forEach((client) => { if (client.readyState === WebSocket.OPEN) { client.send(JSON.stringify(payload)); } }); }, }); } startKafkaConsumer(); This design enables selective subscription — a dashboard user viewing 50 devices only receives telemetry for those 50 devices, not the full firehose. This is critical for performance at scale. Frontend: Vue 3 Reactive Architecture The frontend is built with Vue 3 Composition API + Pinia for state management. The goal is to update only the UI components bound to a specific device when its telemetry arrives — not re-render the entire dashboard. WebSocket Composable Plain Text // composables/useDeviceTelemetry.ts import { ref, onMounted, onUnmounted } from 'vue'; import { useDeviceStore } from '@/stores/deviceStore'; export function useDeviceTelemetry(deviceIds: string[]) { const store = useDeviceStore(); let ws: WebSocket | null = null; const connect = () => { ws = new WebSocket(import.meta.env.VITE_WS_URL); ws.onopen = () => { deviceIds.forEach((id) => { ws!.send(JSON.stringify({ action: 'subscribe', deviceId: id })); }); }; ws.onmessage = (event) => { const telemetry = JSON.parse(event.data); store.updateDeviceTelemetry(telemetry.deviceId, telemetry); }; ws.onclose = () => { // Exponential backoff reconnection setTimeout(connect, Math.min(1000 * 2 ** reconnectAttempts++, 30000)); }; }; onMounted(connect); onUnmounted(() => ws?.close()); } Pinia Store with Fine-Grained Reactivity Plain Text // stores/deviceStore.ts import { defineStore } from 'pinia'; import { reactive } from 'vue'; interface DeviceTelemetry { deviceId: string; status: 'online' | 'offline' | 'degraded'; signalStrength: number; latency: number; lastSeen: number; alerts: string[]; } export const useDeviceStore = defineStore('devices', () => { const telemetryMap = reactive<Record<string, DeviceTelemetry>>({}); function updateDeviceTelemetry(deviceId: string, data: Partial<DeviceTelemetry>) { if (!telemetryMap[deviceId]) { telemetryMap[deviceId] = {} as DeviceTelemetry; } Object.assign(telemetryMap[deviceId], data); } return { telemetryMap, updateDeviceTelemetry }; }); Using reactive() with a map structure means Vue's dependency tracking is at the property level — a component subscribed to telemetryMap['device-001'].signalStrength won't re-render when device-002's data changes. This is the key to dashboard scalability. Device Card Component Plain Text  <template> <div class="device-card" :class="statusClass"> <div class="device-header"> <span class="device-id">{{ deviceId }</span> <StatusBadge :status="telemetry?.status" /> </div> <div class="metrics"> <MetricBar label="Signal" :value="telemetry?.signalStrength" unit="dBm" /> <MetricBar label="Latency" :value="telemetry?.latency" unit="ms" /> </div> <AlertList :alerts="telemetry?.alerts ?? []" /> </div> </template> <script setup lang="ts"> import { computed } from 'vue'; import { useDeviceStore } from '@/stores/deviceStore'; const props = defineProps<{ deviceId: string }>(); const store = useDeviceStore(); // Only this device's slice of state — targeted re-renders only const telemetry = computed(() => store.telemetryMap[props.deviceId]); const statusClass = computed(() => ({ 'status-online': telemetry.value?.status === 'online', 'status-offline': telemetry.value?.status === 'offline', 'status-degraded': telemetry.value?.status === 'degraded', })); </script> Performance Optimizations 1. Virtual Scrolling for Large Device Lists When monitoring 500+ devices, rendering all device cards simultaneously tanks performance. We use vue-virtual-scrollerto only render visible cards: Plain Text <RecycleScroller class="device-list" :items="filteredDevices" :item-size="120" key-field="deviceId" v-slot="{ item }" > <DeviceCard :device-id="item.deviceId" /> </RecycleScroller> 2. Debounced Batch Updates Devices can emit bursts of telemetry. Updating the Pinia store on every single message causes excessive re-renders. We batch incoming messages within a 100ms window: Plain Text let pendingUpdates: Record<string, Partial<DeviceTelemetry>> = {}; let batchTimeout: ReturnType<typeof setTimeout> | null = null; function queueUpdate(deviceId: string, data: Partial<DeviceTelemetry>) { pendingUpdates[deviceId] = { ...(pendingUpdates[deviceId] ?? {}), ...data }; if (!batchTimeout) { batchTimeout = setTimeout(() => { Object.entries(pendingUpdates).forEach(([id, update]) => { store.updateDeviceTelemetry(id, update); }); pendingUpdates = {}; batchTimeout = null; }, 100); } } 3. Lazy Loading and Code Splitting Device diagnostic panels (charts, event logs, configuration editors) are loaded on demand: Plain Text const DeviceDiagnostics = defineAsyncComponent( () => import('@/components/DeviceDiagnostics.vue') ); Combined with route-level code splitting, the initial bundle stays under 200KB gzipped. Security Architecture: OAuth 2.0 + RBAC Device management platforms require fine-grained access control. Not every user should be able to issue firmware update commands to production devices. JWT Claims-Based RBAC We encode role information directly in the JWT access token: Plain Text { "sub": "user-123", "roles": ["device:read", "device:configure"], "scope": "region:us-east", "exp": 1699999999 } The frontend reads these claims to conditionally render action buttons, and the backend validates them on every API call: Plain Text // middleware/rbac.ts export function requirePermission(permission: string) { return (req: Request, res: Response, next: NextFunction) => { const token = req.headers.authorization?.split(' ')[1]; const decoded = verifyJWT(token!); if (!decoded.roles.includes(permission)) { return res.status(403).json({ error: 'Insufficient permissions' }); } next(); }; } // Route definition router.post('/devices/:id/firmware', requirePermission('device:firmware'), handleFirmwareUpdate); Deployment: CI/CD on AWS The entire platform is containerized and deployed via a GitLab CI/CD pipeline to AWS ECS with Fargate. Plain Text # .gitlab-ci.yml (excerpt) stages: - test - build - deploy build-and-push: stage: build script: - docker build -t $ECR_REGISTRY/iot-frontend:$CI_COMMIT_SHA . - docker push $ECR_REGISTRY/iot-frontend:$CI_COMMIT_SHA deploy-production: stage: deploy script: - aws ecs update-service --cluster iot-platform --service frontend --force-new-deployment environment: production only: - main Blue-green deployments ensure zero downtime for this 24/7 critical infrastructure platform. Results and Key Metrics After migrating from a polling-based architecture to this event-driven stack: Dashboard latency: reduced from 5–10 seconds (polling) to under 200ms (WebSocket push).Backend API load: reduced by ~78% — telemetry pushes replaced constant polling.Frontend bundle size: kept under 220KB gzipped through lazy loading and tree-shaking.Throughput: validated at 10,000 concurrent telemetry events/second through Kafka partitioning. Conclusion Building a real-time IoT dashboard at enterprise scale requires rethinking the entire data flow — from device communication protocols through streaming pipelines to fine-grained frontend reactivity. The combination of MQTT for lightweight device communication, Kafka for durable event streaming, WebSockets for real-time push to browsers, and Vue 3's targeted reactivity model creates a system that scales gracefully without sacrificing developer ergonomics. The patterns described here — selective WebSocket subscriptions, batched Pinia updates, virtual scrolling, and JWT-based RBAC — have been validated in production on a platform serving critical network infrastructure. They are broadly applicable to any domain requiring real-time monitoring at scale: energy management, fleet tracking, industrial automation, and beyond. Github: Real-Time-IoT-Dashboards-Vue-3-MQTT-Kafka

By Venkata Sandeep Dhullipalla

Scaling Cloud Data Automation: A Practical Guide to Open Table Formats

When we talk about data analytics the way we set up our tables is really important. This is because it can make a difference, in how well our systems work and how fast they can grow. Data analytics and Open Table Formats go hand in hand. Open Table Formats are a part of how we build our data systems today. They make it easy to work with systems. Get more out of our data. In this blog post we will talk about what Open Table Formatsre. We will discuss data analytics and Open Table Formats in detail. We will also look at some examples. Help you figure out which Open Table Format is best for your data analytics needs. We want to help organizations choose the Open Table Format for their data systems because the Open Table Format is very important, for organizations. The Open Table Format is what organizations need to make their data systems work well. What Are Open Table Formats? Open Table Formats are really good at keeping data neat and tidy, in tables. Nobody owns Open Table Formats so they are made to work with lots of tools and systems. This is great because Open Table Formats can be used by people and computers and they all work together. The goal of Open Table Formats is to make it easy for people to share data and use it so everyone can work together smoothly no matter what kind of computer or system they use, with Open Table Formats. Popular Open Table Formats People really, like using Open Table Formats when they are dealing with data. Here are some popular Open Table Formats that people use a lot when they are working with Open Table Formats: Apache Iceberg Apache Iceberg is a way to organize tables. It helps people work with sets of data in an controlled way. Apache Iceberg gives us things like ACID transactions, which's, like a guarantee that Apache Iceberg will handle our data correctly. Apache Iceberg also has isolation so we can look at our data without worrying about people changing Apache Iceberg data at the same time. Apache Iceberg allows for schema evolution, which means we can change the way our Apache Iceberg data is organized without having to start over again with Apache Iceberg. I think Apache Iceberg is really useful for people who deal with datasets in data lakes. Apache Iceberg is very helpful because it makes working with amounts of data a lot easier for people who do this kind of work, with Apache Iceberg. Advantages The main advantages of this system are that it makes sure the data is consistent. It helps with queries. This system also allows the database schema to change and evolve over time without losing any of the data, from the database schema. The system ensures data consistency. It supports queries and it enables the database schema evolution. Use Cases: Ideal for data lakes requiring transactional guarantees and schema flexibility. Delta Lake Delta Lake is a way to store data that's free for anyone to use. It helps make sure the Delta Lake data is reliable. When many people use the Delta Lake data at the time Delta Lake makes sure there are no problems. The Delta Lake also keeps track of a lot of information, about the Delta Lake data. Delta Lake makes it easy to use data that is coming in all the time and old data that is already stored in the Delta Lake. The Delta Lake does all this by using something called ACID transactions to help the Delta Lake work properly. Delta Lake is really great when it comes to dealing with an amount of data. Delta Lake works well with data that is coming in all the time and Delta Lake also works well with data that comes in big groups. This thing has a lot of points. It makes sure the data is good and reliable. You can also go back. Look at old versions of the data. The data works well with the tools that use the data. The tools that process the data, like it when the data is set up this way. Use Cases: Suitable for data lakes requiring reliability, data versioning, and unified data processing. Apache Hudi Apache Hudi is a tool for working with data. It helps you add data to the data you already have. Apache Hudi also makes it easier to build systems that can move data around. This is really helpful when you have a lot of data in a data lake. Anyone can use Apache Hudi because it is source. The best thing about Apache Hudi is that it makes handling data processing and building data pipelines on data lakes simpler. Apache Hudi is very useful, for people who work with data lakes and need to process a lot of data. This system is good because it helps with processing data a little at a time. It also keeps track of versions of the data. The data system makes it easy to get the data in and to ask questions about the data. The data system is really helpful when you want to ask questions, about the data. Use Cases: Ideal for data lakes requiring incremental data processing and data pipeline management. Choosing the Right Open Table Format When you are trying to pick the Open Table Format for the data analytics you need you have to think about a lot of things. You have to think about what you will be using the Open Table Format for. What is your data, like? Will the Open Table Format work with the systems you use? How well does the Open Table Format need to perform for your data analytics? Here are some important things to think about when you're trying to decide on an Open Table Format for your data analytics needs: Use Cases and Workloads When you want to make sure your transactions are safe and your data is consistent you should think about using formats like Apache Iceberg or Delta Lake. These formats give you something called ACID transactions which's, like a promise that everything will work correctly. Apache Iceberg and Delta Lake are options because they help you keep your data safe and make sure everything is consistent. If you are looking for something that will guarantee your data is safe Apache Iceberg and Delta Lake are the way to go because Apache Iceberg and Delta Lake give you this guarantee. When we talk about Incremental Data Processing we need to think about how to handle Incremental Data Processing. For people who work with Incremental Data Processing and manage data pipelines Apache Hudi is an option to consider for their Incremental Data Processing needs. Apache Hudi can really help with tasks related to Incremental Data Processing. Data Characteristics When you are working with data think about how data you will have to deal with. You have to store data. Some ways of storing data are better for sets of data. Data volume is something you should think about because some formats can handle lots of data better, than others. This is really important when you are working with a lot of data. If you are working with data data volume can be a problem if you are not using the format for your data. Data Complexity You have to find out how complicated your data is. This means you need to look at the types of data you have. You should think about if you will need to make changes to how your data's organized. Some data formats, like Apache Iceberg and Delta Lake are very helpful. They are helpful because they let you make changes to your data easily. You can change your data without a lot of trouble when you use Apache Iceberg and Delta Lake. Ecosystem Compatibility When you choose an Open Testing Framework you need to make sure it works well with the data processing tools you already use. For example Delta Lake works with Apache Spark. This is really important because you want your Open Testing Framework to be compatible with your existing data processing frameworks and tools, like your Open Testing Framework and your data processing tools. You want your Open Testing Framework to work smoothly with the tools you have so your Open Testing Framework and your data processing tools work together perfectly. When you think about Cloud Platforms you need to think about how the OTF works with the Cloud Platform you want to use. You have to see if the OTF is compatible with the Cloud Platform you like.. You have to check if it works with the infrastructure you have at home or in your office. This is really important for Cloud Platforms, like the ones you use every day. You need to make sure the OTF and the Cloud Platform work together. The Cloud Platform you choose should be able to work with the OTF. Performance Requirements Let us take a look at the On The Fly system and see how it works when we have to handle queries. The On The Fly system has to be able to handle our queries. We need to check how well the On The Fly system does when it comes to query performance. This is important because we do a lot of work. The On The Fly system has to be good, at handling the kind of work we do. We have to test the On The Fly system to see how it performs with our workloads. The On The Fly system needs to be able to handle these workloads. * We are going to take a look, at how the On The Fly system works when it comes to answering queries. We want to see how the On The Fly system does its job. The On The Fly system is what we are focusing on. * We are going to use this for the work we do when we analyze things for our workloads. This will help us with our workloads. The main thing we want to figure out is how good the On The Fly system is at doing our work. We need to see if the On The Fly system can give us the results we need fast. This will help us decide if the On The Fly system is really good, for the kind of work we do with the On The Fly system. Data Ingestion We need to check how well our Data Ingestion processes are working, especially when we are getting Data Ingestion done on time or really close to time for analytics. This is really important, for Data Ingestion because it helps us understand what is happening now with our Data Ingestion. We need to see how Data Ingestion works with a lot of information. We have to know how fast Data Ingestion can process this information. For Data Ingestion to be really useful it has to be able to handle all this information. Data Ingestion is only good if it can do this. Open Table Formats are really important for working with data these days. They make it easy to work with systems and Open Table Formats can do a lot of things. If you know what makes Open Table Formats like Apache Iceberg, Delta Lake and Apache Hudi special you can pick the Open Table Format that's best, for your company. You need to think about your data. What is your data like? You should figure out what you want to do with your data and what tools you are using with your data. You should also think about what you want your data to be like. Then you can pick the Open Table Format that's best for your data and what you want to do with your data. Open Table Formats are important for your data so choosing the Open Table Format is important, for your data needs.

By Sandeep Batchu

Edge Computing in Utility IoT: Two Architecture Patterns That Actually Work

When centralized control architectures were designed, power flowed from large generation plants down to passive consumers, utilities managed hundreds of large assets, data volumes were modest, and connectivity was reliable at substations. Few of these assumptions hold today. Power flows in both directions as rooftop solar and battery storage inject back into the distribution network. Utilities now coordinate millions of small, variable, distributed assets instead of hundreds of large ones. Data volumes have multiplied by orders of magnitude as smart meters, sensors, and distributed energy resource (DER) controllers generate continuous streams. According to SCE's "Countdown to 2045" analysis, overall electricity demand will nearly double over the next two decades, driven largely by EV adoption, building electrification, and distributed solar. That growth will come from millions of small, distributed resources that centralized systems weren't designed to coordinate. Going forward, the control architecture should match the grid they're actually operating, not the one they planned for decades ago. This gap is exactly what edge computing addresses. Why Edge Architecture Fits Utility Environments Before looking at specific deployment patterns, it helps to understand why edge computing suits utility infrastructure in particular. Three structural characteristics make it the right fit. Utility data has a locality problem. Sensors, meters, and controllers generate data where decisions need to happen — at the substation, the inverter, the distribution feeder. IEEE 2800-2022 specifies that inverter-based resources must achieve step response times within 2.5 grid cycles OSTI — at 60Hz that's roughly 42 milliseconds. A cloud round-trip often takes longer than that. Edge processing keeps logic where the data originates. Utility infrastructure has a connectivity problem. The system needs to keep functioning whether or not it can reach the cloud — despite storms, distance, or unreliable networks. For these environments, autonomous edge operation is a baseline requirement. Utility scale has a bandwidth problem. Modern grid sensors generate continuous, high-resolution streams. Transmitting everything to a central system is economically unfeasible at the deployment scale. Edge filtering means only verified anomalies and events travel upstream, not raw sensor streams. These three characteristics show up in every serious utility IoT deployment. The architecture pattern you choose determines how you address them. Two Patterns, Different Constraints Utility edge deployments fall into two fundamentally different patterns. Which one you're deploying determines your hardware, protocol, and AI strategy. Pattern A: High-Frequency Control Loops This pattern applies when the system needs to detect grid conditions and respond within milliseconds — DER coordination, voltage regulation, frequency response, fault detection. The key difference is that the control decisions are made by the device itself, not a gateway or central system. The Utilidata deployment with Southern California Edison and NVIDIA embedded computing directly into smart meters using NVIDIA's Jetson platform, running Real-Time Optimal Power Flow (RT-OPF) algorithms at the meter. Solar inverters, EV chargers, and battery systems responded to actual grid conditions measured locally, not static dispatch schedules. The published results: 27% reduction in peak demand and 12.5% reduction in electricity costs for the simulated household. This is a meaningful architectural difference that only works with grid-connected hardware with sufficient compute power and no battery constraints. Pattern B: Distributed Sensor Networks This pattern applies when deploying hundreds or thousands of battery-powered sensors across a wide geographic area. The data is being captured periodically, processed locally, and transmitted only when something meaningful is detected. EPCOR's acoustic leak detection deployment across 160 square miles of desert water infrastructure in Arizona demonstrates this. Battery-powered sensors attach directly to water pipes, wake periodically to capture acoustic samples, run local AI inference to detect leak signatures, and transmit only when an anomaly matches trained patterns. The system identified over 250 leaks and helped recover 115 million gallons of water. The results would be economically impossible if sensors were streaming continuous audio to the cloud. Every computation and transmission drains limited energy, yet sensors must run on batteries for years. AI models must be lightweight enough to run within milliwatt budgets while still being accurate enough to distinguish a genuine leak signature from pipe noise. Protocol and Hardware Decisions Follow the Pattern Once you know which pattern applies, hardware and protocol choices follow directly. For control loop deployments (Pattern A), hardware is typically grid-connected — gateways at substations, computing embedded in meters or inverters. Protocol selection centers on what your existing field devices speak: Modbus for legacy equipment, IEC 61850 for modern substations, DNP3 for SCADA-connected devices, MQTT for newer IoT sensors. A single-edge gateway must collect from all of these simultaneously. For sensor network deployments (Pattern B), hardware is battery-constrained, and the protocol choice is driven by range and power requirements. LoRaWAN achieves 15km range with years of battery life at the cost of low data throughput and is the common choice for large geographic areas. NB-IoT provides better penetration in dense or underground environments using cellular infrastructure. LoRaWAN requires a gateway deployment. NB-IoT runs on existing cellular coverage but introduces carrier dependency and ongoing SIM costs. Neither protocol is universally better. The choice depends on service area geography, existing cellular coverage, deployment density, and battery budget. ThingsBoard Edge supports protocol diversity through the IoT Gateway component. It connects to Modbus and OPC-UA out of the box, along with MQTT for modern IoT sensors. For low-power wide-area protocols like LoRaWAN and NB-IoT, ThingsBoard integrations enable connectivity without custom middleware. This allows utilities to deploy either pattern — or both — on unified infrastructure. AI at the Edge: Two Execution Strategies Both patterns differ based on hardware constraints. Pattern A gateways have enough compute to query external AI services when connected — OpenAI, Azure OpenAI, or a self-hosted model via API — and switch to cached local models when the connection drops. This approach keeps models updated centrally without hardware constraints limiting model complexity. For Pattern B deployments, models must run locally within tight power and memory budgets. The EPCOR deployment uses deep learning models trained on extensive plastic pipe acoustic datasets, optimized to run directly on the sensor hardware. Every percentage point of detection accuracy improvement must be weighed against battery life — a more complex model might cut operational lifespan in half. Either way, managing AI models across a distributed fleet requires automation. Utilities can't manually update inference logic on hundreds of field-deployed devices. Modern edge platforms solve this with central model repositories and OTA update pipelines. Engineers train on historical data, check results, and push updates to the whole device fleet during scheduled maintenance — no truck rolls, no downtime. Integration With Existing Infrastructure The most common engineer concern about edge computing isn't the technology — it's disruption to working systems. Replacing functional SCADA infrastructure mid-operation isn't realistic for most utilities, and it isn't necessary. The practical integration approach is additive rather than replacement. Deploy an edge gateway alongside your existing PLC or RTU. It connects to the network, collects data from field devices via the protocols they already speak, and runs additional intelligence — anomaly detection, pattern recognition, predictive alerts — without touching the control loop. Your existing PLC continues executing hard-coded protection logic. The edge layer watches for conditions the PLC wasn't designed to detect. This matters for DER management specifically. Existing SCADA systems weren't architected for thousands of bidirectional resources that both consume and inject power based on real-time conditions. Rather than rebuilding that SCADA layer, an edge platform can sit alongside it, handling the DER coordination layer while SCADA continues managing the assets it was built for. Energenix, a renewable energy SCADA provider operating across South Asia, takes exactly this approach using ThingsBoard Edge. The platform delivers local monitoring and control at solar plant sites, enabling operational staff to respond to events without cloud dependency, while the central ThingsBoard instance handles fleet-wide analytics and long-term storage across their 120+ MW portfolio spanning multiple countries. Managing hundreds of sites requires visibility into the fleet itself, not just the grid assets it monitors: battery levels, communication timestamps, firmware versions, health status. Edge platforms provide centralized dashboards for this fleet-wide visibility while edge nodes maintain autonomous operation. Over-the-air update pipelines push firmware and model updates to device groups from a central interface — no SSH sessions, no truck rolls. Choosing Where to Start For utilities evaluating edge computing deployments, the clearest starting point is identifying which pattern your highest-priority use case falls into, then running a contained pilot before fleet deployment. If your primary concern is DER coordination and real-time grid response, Pattern A applies — start with a single substation or feeder and measure latency improvements against your current SCADA response baseline. If your primary concern is infrastructure monitoring across a wide service area with connectivity constraints, Pattern B applies — start with a single sensor deployment zone, validate detection accuracy, then scale the fleet. The platform infrastructure — protocol integration, rule engine, OTA management, centralized dashboards — should be the same regardless of which pattern you start with. Utilities that deploy both patterns eventually need them to coexist under unified management. Building on a platform that handles both from day one avoids painful integration work later.

By Yevheniia Mala

The Network Attach Problem Nobody Warns You About

We have been here before. When NB-IoT went nationwide at a major U.S. operator in 2018, enterprise teams discovered that activating large device fleets simultaneously did things to the network that nobody in the procurement conversation had mentioned. The spec sheets were silent on it. The vendor demos didn't surface it. It showed up on activation day, at scale, in production. In 2025, RedCap went commercial. AT&T reached nationwide RedCap coverage serving over 200 million POPs in July. Multiple North American operators followed with commercial launches. And the same attach problem is arriving again, with new technology, with a new generation of engineers who haven't seen it before. I've spent several years working on IoT validation and network performance at a Tier-1 U.S. operator serving over 320 million people. The pattern repeats itself with every new cellular IoT technology generation. This article is about what that pattern is, why RedCap doesn't escape it, and what to do before activation day instead of after. The Attach Storm Problem When enterprise IoT fleets activate at scale — tens of thousands of devices registering with the network within a narrow time window — the network sees a concentrated burst of signaling traffic that individual devices were never designed to produce together. Industry analysis has shown that as few as 500 aggressive devices attached in a burst can generate signaling congestion. At enterprise fleet scale, that number is a rounding error. Devices that can't complete attach on the first attempt retry. If firmware back-off timers are set to manufacturer defaults, which they almost always are, because nobody had the explicit conversation about it, devices retry at intervals that compound the congestion rather than relieving it. Activation campaigns that should be completed in two to three hours drag through most of a business day. By the time the problem surfaces, devices are already deployed across hundreds of sites with no clean fix available. This is not a defect in technology. NB-IoT, Cat-M, and now RedCap all have congestion management mechanisms in their specifications. The problem is that those mechanisms work correctly only when device firmware is configured in coordination with operator network policies, a step that routinely falls between the device vendor's responsibility and the operator's - and therefore belongs to nobody until something breaks. Why RedCap Doesn't Escape This The GSA's 2025 RedCap analysis states directly that network policies must be carefully managed to prevent low-complexity RedCap devices from adversely affecting cell-level resource efficiency in high-density scenarios. That is the attach storm problem described in a different language. RedCap introduces a specific wrinkle that NB-IoT and Cat-M don't have in the same form. Because RedCap operates on a 5G Standalone core, the RRC parameter framework is different from anything LTE-based. Extended DRX cycles — central to RedCap's battery life story — need to be explicitly configured at the network layer to align with device behavior. When they're not, devices don't achieve the power savings the spec promises, and the network sees attach retry patterns that are harder to predict than equivalent LTE-M behavior. There is also a coexistence dimension still being characterized in live networks. RedCap devices share spectrum resources with full 5G NR devices in a way that has no direct equivalent in NB-IoT or Cat-M deployments. At low device densities, this is manageable. As enterprise fleets scale into tens of thousands of devices per operator, the interaction between RedCap device density and 5G NR resource efficiency in the same sectors will produce field data that differs from controlled deployments. The module firmware maturity gap compounds this. RedCap firmware stacks are newer than their LTE-M equivalents by several product generations. The back-off timer defaults that caused problems in 2018 NB-IoT deployments are an equivalent risk in RedCap deployments today, compounded by the fact that fewer engineers have production experience with RedCap RRC behavior at scale. What Cat-M Taught Us Cat-M solved the NB-IoT handover problem; devices moving between cells maintain connectivity rather than disconnecting and re-registering. Under mass attach conditions, this matters because a device drifting into a new cell during a retry window doesn't restart the attach sequence from scratch. But Cat-M introduced its own attach-scale behavior: devices competing for LTE scheduling resources alongside consumer traffic. A mass activation storm from a large Cat-M fleet during peak hours produces a measurable effect on LTE performance in that sector. Different problem, same root cause; nobody coordinated firmware configuration with network policy before deployment. RedCap will have its own version of this story. The 5G SA resource sharing model is different again. The lesson from Cat-M isn't that one technology is safer than another. It's that each technology generation surfaces the same configuration gap in a new form, and the teams that get ahead of it are the ones who treated the operator as a design partner before device firmware was finalized. What Actually Prevents It Three parameters matter most for RedCap: access class barring configuration, back-off timer values, and RRC configuration alignment for extended DRX. Your operator's IoT network engineering team has a position on what works in their network. That information is available if you ask for it explicitly during firmware design. It is almost never volunteered proactively. Before finalizing device firmware, engage your operator's IoT team with a specific question: What back-off, access class barring, and DRX parameters should we configure for a fleet of this size activating in these markets? Then, validate those parameters in a controlled activation of a representative sample across multiple sectors and coverage conditions before the full fleet goes live. The DRX alignment step is additional compared to a standard LTE-M validation process. It's where power consumption surprises and attach behavior anomalies tend to appear in early RedCap deployments, and it requires your operator's RAN engineering team — not just the module vendor, the connectivity sales team. Staggered activation scheduling is the other lever, and it's entirely within the enterprise team's control. A fleet of 50,000 devices doesn't need to attach simultaneously. A 30-minute staggered window across deployment sites eliminates the concentrated burst that triggers congestion without any firmware change or carrier conversation. It is the option most consistently overlooked because activation day is treated as a go-live event rather than an operational process with network implications. The Pattern That Keeps Repeating NB-IoT arrived with real deployment momentum. Cat-M followed. Each time, a portion of early enterprise deployments discovered attach-scale problems that pre-deployment testing hadn't surfaced because device counts were too small. RedCap is arriving with the strongest momentum of the three. AT&T has nationwide coverage. Multiple North American operators have commercial launches. Module vendors, including Quectel, Fibocom, and Telit Cinterion, have certified products in the market. Qualcomm's Snapdragon X35 is powering initial commercial devices. The ecosystem is real and moving fast. What is also moving fast is the number of enterprise teams planning their first RedCap deployment based on spec comparisons and lab results, without the production-scale attach behavior validation that would catch the same configuration gap that caught NB-IoT teams in 2018 and Cat-M teams in the years that followed. Technology changes every few years. The pattern doesn't. Getting ahead of it is still the job.

By SESHA KIRAN GONABOYINA

IoT

DZone's Featured IoT Resources

Monthly Top IoT Experts

The Latest IoT Topics