IoT Resources

DZone's Featured IoT Resources

Bringing Intelligence Closer to the Source: Why Real-Time Processing is the Heart of Edge AI

By Jitendra Bafna

Artificial Intelligence is rapidly becoming a part of everyday devices — smartphones, cars, cameras, and even home appliances. Traditionally, these systems rely on cloud servers to send, process, and analyze data before making decisions, which increases latency and delays responses. However, many applications require instant decision-making, where even a slight delay can be critical. In such scenarios, relying on network connectivity is not always practical, and decisions need to be made locally on the device itself. This has led to a growing shift toward running intelligence directly on devices, making real-time local processing more important than ever. In this article, we’ll explore why this shift matters and how it is shaping the future of modern intelligent systems. What is Edge AI? Edge AI refers to running AI models directly on devices such as IoT systems, smartphones, autonomous cars, drones, and sensors — right where the data is generated. With this approach, there is no need to transfer data to cloud servers or centralized systems. Edge AI enables faster, real-time decision-making by processing data locally, without sending it elsewhere. For example, Instead of sending every transaction to a central server for analysis, the system can analyze transaction patterns locally in real time. If any unusual activity is detected — such as an abnormal withdrawal amount, location mismatch, or suspicious behavior — the system can instantly block the transaction or trigger an alert. Why Real-Time Processing Matters? Real-time processing means a system can process data instantly and make decisions without delay. Even small delays in decision-making can create critical situations and lead to serious consequences. For example, an autonomous car must detect obstacles and react within milliseconds. If it relies on the cloud, even a small delay could lead to serious consequences. By processing data locally, Edge AI enables immediate decisions — such as braking or steering — making the system safer and more efficient. Reduce Latency and Faster Decisions Latency is the time it takes for data to travel to the cloud and back. Even a delay of a few milliseconds can be too slow for certain applications. With Edge AI: Data is processed instantly on the device itself.There’s no need to wait for a network response.Performance is faster, more reliable, and less dependent on connectivity. For example, a voice recognition system on a smartphone can respond much faster when speech processing runs locally on the device, rather than relying on cloud or centralized servers. Improved Privacy and Data Security Sending sensitive data to the cloud raises privacy concerns, as it can be exposed during transmission or storage. Edge AI minimizes these risks by processing data directly on the local device instead of sending it to the cloud. This approach enhances data security and helps maintain user privacy, since sensitive information never leaves the device. It also supports compliance with data protection regulations and reduces the chances of unauthorized access or data breaches. For example, a healthcare wearable that monitors heart activity should not transmit sensitive personal health data to external servers. Instead, it can analyze patterns locally on the device and instantly alert the user if any irregularities are detected. This approach not only protects patient privacy but also enables faster, real-time responses in critical situations. Such local processing is especially important in industries like banking, healthcare, finance, and smart homes, where data security and immediate decision-making are essential. Reliability Without Internet Dependency Edge devices can operate even without an internet connection, making them more stable and reliable in remote areas or environments with poor network coverage. This ensures continuous performance without interruptions or delays caused by connectivity issues. As a result, critical applications can function smoothly regardless of network availability. For example, a drone used in disaster rescue operations cannot depend on internet connectivity. It must process images locally and detect survivors in real time, enabling faster and more effective rescue efforts. Lower Bandwidth Usage and Reduce Infrastructure Costs Sending large amounts of data to the cloud consumes significant bandwidth and increases operational costs. Edge AI helps reduce these costs by processing data locally on the device. This minimizes the need for constant data transmission and optimizes network usage. Only relevant or critical information is sent to the cloud, making the system more efficient and cost-effective. For example, a factory machine monitoring system can analyze sensor data locally and send alerts only when an issue is detected, instead of continuously streaming all the data. Scalability and Cost Efficiency Cloud processing for millions of devices can become expensive and resource-intensive. Edge AI addresses this by distributing computations across devices, reducing the load on central servers. This decentralized approach lowers infrastructure costs, improves scalability, and enhances overall system performance. It also reduces latency by minimizing the need for constant communication with the cloud. For example, in a smart city, thousands of cameras can process data locally instead of sending everything to a central cloud system. This not only saves bandwidth and infrastructure costs but also enables faster, real-time insights and responses. Better User Experience Real-time processing significantly improves user experience by making systems feel faster, smoother, and more responsive. Quicker responses lead to higher user satisfaction and a more seamless interaction. With Edge AI, data is processed instantly on the device, eliminating delays and ensuring consistent performance. This is especially important for applications that require immediate feedback. For example, in gaming or augmented reality (AR), local AI can render objects and interactions in real time, creating a smoother, more immersive, and engaging user experience. An edge-based platform helps by enabling data processing and decision-making directly on devices, rather than relying entirely on centralized cloud systems. It supports faster, real-time responses by analyzing data locally, which is essential for applications that require immediate action. This leads to improved performance and reliability, especially in environments with limited or unstable internet connectivity. It also enhances data privacy and security by keeping sensitive information on the device, reducing the need for data transmission. Additionally, it optimizes bandwidth usage and lowers infrastructure costs by sending only meaningful insights or alerts to central systems instead of continuous raw data. Overall, this approach helps build systems that are faster, more efficient, secure, and scalable by bringing intelligence closer to where data is generated. Conclusion Edge AI is transforming modern systems by bringing intelligence closer to where data is created, enabling faster and real-time decision-making. It reduces latency and improves performance by processing data locally instead of relying on the cloud. This approach also enhances privacy and minimizes dependence on constant internet connectivity. Additionally, it helps reduce bandwidth usage and lowers infrastructure costs. From smart cities to healthcare and industrial automation, edge computing is driving a new era of faster, smarter, and more efficient systems. Edge AI brings intelligence closer to where data is created, enabling real-time decisions, faster performance, enhanced privacy, and reliable operation without depending on constant connectivity. More

Building Enterprise-Grade Real-Time IoT Dashboards with Vue 3, MQTT, and Kafka

By Venkata Sandeep Dhullipalla

The convergence of IoT, real-time data streaming, and modern frontend frameworks is redefining how engineers build enterprise monitoring systems. Over the course of designing and leading the Device IoT Platform — an enterprise-grade solution for real-time monitoring, configuration, and diagnostics of thousands of distributed network devices — I encountered and solved a core architectural challenge: how do you build a frontend dashboard that can handle hundreds of concurrent device telemetry streams without sacrificing performance, maintainability, or user experience? This article shares the architectural patterns, technology decisions, and hard-won lessons from that journey — covering the full stack from MQTT ingestion to Vue 3 reactivity to Kafka-backed event processing. The Core Problem: Real-Time at Scale Most developers are familiar with polling — periodically fetching data from an API endpoint. For IoT, polling is fundamentally broken: Latency: A 5-second polling interval means 5 seconds of stale state.Inefficiency: You're requesting data even when nothing has changed.Scale: 1,000 devices × 1 request/5s = 200 requests/second just to read status — before any user interaction. The solution is event-driven architecture: devices push telemetry when something changes, and the platform reacts. This requires a rethinking of both backend ingestion and frontend state management. Architecture Overview Plain Text [IoT Devices] | MQTT Broker (Mosquitto / AWS IoT Core) | [Node.js Telemetry Microservice] | [Kafka Topic: device.telemetry.raw] | (stream processor) [Kafka Topic: device.telemetry.enriched] | [WebSocket Server (Node.js)] | [Vue 3 Dashboard Frontend] Each layer has a distinct responsibility: MQTT Broker handles lightweight publish/subscribe with devices using minimal overhead.Node.js microservices bridge MQTT → Kafka, performing initial validation and normalization.Kafka provides durable, replayable event streams — critical for audit trails and late-joining consumers.WebSocket server fans out enriched telemetry to connected dashboard clients in real time.Vue 3 handles reactive rendering, ensuring only the affected UI components re-render when new data arrives. Backend: MQTT → Kafka Bridge in Node.js The heart of the ingestion pipeline is a lightweight Node.js service using the mqtt and kafkajs libraries. Plain Text import mqtt from 'mqtt'; import { Kafka } from 'kafkajs'; const mqttClient = mqtt.connect(process.env.MQTT_BROKER_URL!, { clientId: `telemetry-bridge-${process.pid}`, username: process.env.MQTT_USERNAME, password: process.env.MQTT_PASSWORD, clean: true, }); const kafka = new Kafka({ clientId: 'iot-bridge', brokers: [process.env.KAFKA_BROKER!] }); const producer = kafka.producer(); mqttClient.on('connect', async () => { await producer.connect(); mqttClient.subscribe('devices/+/telemetry', { qos: 1 }); console.log('MQTT → Kafka bridge active'); }); mqttClient.on('message', async (topic, payload) => { const deviceId = topic.split('/')[1]; const data = JSON.parse(payload.toString()); await producer.send({ topic: 'device.telemetry.raw', messages: [ { key: deviceId, value: JSON.stringify({ deviceId, timestamp: Date.now(), ...data }), }, ], }); }); Key design decisions here: QoS Level 1 — ensures at-least-once delivery for telemetry messages. For command acknowledgments, we use QoS 2.Device ID as Kafka partition key — guarantees ordering per device while allowing parallel processing across partitions.Separation of raw vs. enriched topics — the device.telemetry.raw topic contains the bare payload; a downstream stream processor enriches it with device metadata, geolocation, and alert thresholds before publishing to device.telemetry.enriched. WebSocket Fan-Out Server The WebSocket layer subscribes to Kafka's enriched topic and pushes updates to connected browser clients. We use Kafka consumer groups to allow horizontal scaling of the WebSocket tier. Plain Text import { WebSocketServer } from 'ws'; import { Kafka } from 'kafkajs'; const wss = new WebSocketServer({ port: 8080 }); const kafka = new Kafka({ clientId: 'ws-fanout', brokers: [process.env.KAFKA_BROKER!] }); const consumer = kafka.consumer({ groupId: 'websocket-fanout-group' }); // Track subscriptions: deviceId → Set<WebSocket> const deviceSubscriptions = new Map<string, Set<WebSocket>>(); wss.on('connection', (ws) => { ws.on('message', (msg) => { const { action, deviceId } = JSON.parse(msg.toString()); if (action === 'subscribe') { if (!deviceSubscriptions.has(deviceId)) { deviceSubscriptions.set(deviceId, new Set()); } deviceSubscriptions.get(deviceId)!.add(ws); } }); ws.on('close', () => { deviceSubscriptions.forEach((clients) => clients.delete(ws)); }); }); async function startKafkaConsumer() { await consumer.connect(); await consumer.subscribe({ topic: 'device.telemetry.enriched' }); await consumer.run({ eachMessage: async ({ message }) => { const payload = JSON.parse(message.value!.toString()); const clients = deviceSubscriptions.get(payload.deviceId); clients?.forEach((client) => { if (client.readyState === WebSocket.OPEN) { client.send(JSON.stringify(payload)); } }); }, }); } startKafkaConsumer(); This design enables selective subscription — a dashboard user viewing 50 devices only receives telemetry for those 50 devices, not the full firehose. This is critical for performance at scale. Frontend: Vue 3 Reactive Architecture The frontend is built with Vue 3 Composition API + Pinia for state management. The goal is to update only the UI components bound to a specific device when its telemetry arrives — not re-render the entire dashboard. WebSocket Composable Plain Text // composables/useDeviceTelemetry.ts import { ref, onMounted, onUnmounted } from 'vue'; import { useDeviceStore } from '@/stores/deviceStore'; export function useDeviceTelemetry(deviceIds: string[]) { const store = useDeviceStore(); let ws: WebSocket | null = null; const connect = () => { ws = new WebSocket(import.meta.env.VITE_WS_URL); ws.onopen = () => { deviceIds.forEach((id) => { ws!.send(JSON.stringify({ action: 'subscribe', deviceId: id })); }); }; ws.onmessage = (event) => { const telemetry = JSON.parse(event.data); store.updateDeviceTelemetry(telemetry.deviceId, telemetry); }; ws.onclose = () => { // Exponential backoff reconnection setTimeout(connect, Math.min(1000 * 2 ** reconnectAttempts++, 30000)); }; }; onMounted(connect); onUnmounted(() => ws?.close()); } Pinia Store with Fine-Grained Reactivity Plain Text // stores/deviceStore.ts import { defineStore } from 'pinia'; import { reactive } from 'vue'; interface DeviceTelemetry { deviceId: string; status: 'online' | 'offline' | 'degraded'; signalStrength: number; latency: number; lastSeen: number; alerts: string[]; } export const useDeviceStore = defineStore('devices', () => { const telemetryMap = reactive<Record<string, DeviceTelemetry>>({}); function updateDeviceTelemetry(deviceId: string, data: Partial<DeviceTelemetry>) { if (!telemetryMap[deviceId]) { telemetryMap[deviceId] = {} as DeviceTelemetry; } Object.assign(telemetryMap[deviceId], data); } return { telemetryMap, updateDeviceTelemetry }; }); Using reactive() with a map structure means Vue's dependency tracking is at the property level — a component subscribed to telemetryMap['device-001'].signalStrength won't re-render when device-002's data changes. This is the key to dashboard scalability. Device Card Component Plain Text  <template> <div class="device-card" :class="statusClass"> <div class="device-header"> <span class="device-id">{{ deviceId }</span> <StatusBadge :status="telemetry?.status" /> </div> <div class="metrics"> <MetricBar label="Signal" :value="telemetry?.signalStrength" unit="dBm" /> <MetricBar label="Latency" :value="telemetry?.latency" unit="ms" /> </div> <AlertList :alerts="telemetry?.alerts ?? []" /> </div> </template> <script setup lang="ts"> import { computed } from 'vue'; import { useDeviceStore } from '@/stores/deviceStore'; const props = defineProps<{ deviceId: string }>(); const store = useDeviceStore(); // Only this device's slice of state — targeted re-renders only const telemetry = computed(() => store.telemetryMap[props.deviceId]); const statusClass = computed(() => ({ 'status-online': telemetry.value?.status === 'online', 'status-offline': telemetry.value?.status === 'offline', 'status-degraded': telemetry.value?.status === 'degraded', })); </script> Performance Optimizations 1. Virtual Scrolling for Large Device Lists When monitoring 500+ devices, rendering all device cards simultaneously tanks performance. We use vue-virtual-scrollerto only render visible cards: Plain Text <RecycleScroller class="device-list" :items="filteredDevices" :item-size="120" key-field="deviceId" v-slot="{ item }" > <DeviceCard :device-id="item.deviceId" /> </RecycleScroller> 2. Debounced Batch Updates Devices can emit bursts of telemetry. Updating the Pinia store on every single message causes excessive re-renders. We batch incoming messages within a 100ms window: Plain Text let pendingUpdates: Record<string, Partial<DeviceTelemetry>> = {}; let batchTimeout: ReturnType<typeof setTimeout> | null = null; function queueUpdate(deviceId: string, data: Partial<DeviceTelemetry>) { pendingUpdates[deviceId] = { ...(pendingUpdates[deviceId] ?? {}), ...data }; if (!batchTimeout) { batchTimeout = setTimeout(() => { Object.entries(pendingUpdates).forEach(([id, update]) => { store.updateDeviceTelemetry(id, update); }); pendingUpdates = {}; batchTimeout = null; }, 100); } } 3. Lazy Loading and Code Splitting Device diagnostic panels (charts, event logs, configuration editors) are loaded on demand: Plain Text const DeviceDiagnostics = defineAsyncComponent( () => import('@/components/DeviceDiagnostics.vue') ); Combined with route-level code splitting, the initial bundle stays under 200KB gzipped. Security Architecture: OAuth 2.0 + RBAC Device management platforms require fine-grained access control. Not every user should be able to issue firmware update commands to production devices. JWT Claims-Based RBAC We encode role information directly in the JWT access token: Plain Text { "sub": "user-123", "roles": ["device:read", "device:configure"], "scope": "region:us-east", "exp": 1699999999 } The frontend reads these claims to conditionally render action buttons, and the backend validates them on every API call: Plain Text // middleware/rbac.ts export function requirePermission(permission: string) { return (req: Request, res: Response, next: NextFunction) => { const token = req.headers.authorization?.split(' ')[1]; const decoded = verifyJWT(token!); if (!decoded.roles.includes(permission)) { return res.status(403).json({ error: 'Insufficient permissions' }); } next(); }; } // Route definition router.post('/devices/:id/firmware', requirePermission('device:firmware'), handleFirmwareUpdate); Deployment: CI/CD on AWS The entire platform is containerized and deployed via a GitLab CI/CD pipeline to AWS ECS with Fargate. Plain Text # .gitlab-ci.yml (excerpt) stages: - test - build - deploy build-and-push: stage: build script: - docker build -t $ECR_REGISTRY/iot-frontend:$CI_COMMIT_SHA . - docker push $ECR_REGISTRY/iot-frontend:$CI_COMMIT_SHA deploy-production: stage: deploy script: - aws ecs update-service --cluster iot-platform --service frontend --force-new-deployment environment: production only: - main Blue-green deployments ensure zero downtime for this 24/7 critical infrastructure platform. Results and Key Metrics After migrating from a polling-based architecture to this event-driven stack: Dashboard latency: reduced from 5–10 seconds (polling) to under 200ms (WebSocket push).Backend API load: reduced by ~78% — telemetry pushes replaced constant polling.Frontend bundle size: kept under 220KB gzipped through lazy loading and tree-shaking.Throughput: validated at 10,000 concurrent telemetry events/second through Kafka partitioning. Conclusion Building a real-time IoT dashboard at enterprise scale requires rethinking the entire data flow — from device communication protocols through streaming pipelines to fine-grained frontend reactivity. The combination of MQTT for lightweight device communication, Kafka for durable event streaming, WebSockets for real-time push to browsers, and Vue 3's targeted reactivity model creates a system that scales gracefully without sacrificing developer ergonomics. The patterns described here — selective WebSocket subscriptions, batched Pinia updates, virtual scrolling, and JWT-based RBAC — have been validated in production on a platform serving critical network infrastructure. They are broadly applicable to any domain requiring real-time monitoring at scale: energy management, fleet tracking, industrial automation, and beyond. Github: Real-Time-IoT-Dashboards-Vue-3-MQTT-Kafka More

Scaling Cloud Data Automation: A Practical Guide to Open Table Formats

By Sandeep Batchu

Edge Computing in Utility IoT: Two Architecture Patterns That Actually Work

By Yevheniia Mala

The Network Attach Problem Nobody Warns You About

By SESHA KIRAN GONABOYINA

Ten Years of Beam: From Google's Dataflow Paper to 4 Trillion Events at LinkedIn

In August 2015, a team of engineers at Google published a paper with a title so long it barely fits on a conference slide: "The Dataflow Model: A Practical Approach to Balancing Correctness, Latency, and Cost in Massive-Scale, Unbounded, Out-of-Order Data Processing." The opening line was: We as a field must stop trying to groom unbounded datasets into finite pools of information that eventually become complete. Ten years later, the programming model born from that paper — Apache Beam — processes 4 trillion events daily at LinkedIn alone, powers fraud detection at Transmit Security, runs the cybersecurity backbone at Palo Alto Networks, and handles a large chunk of Google Cloud's data infrastructure through Dataflow. But Beam's story is not a straight line from academic paper to industry dominance. It is better described as a story of ideas that were ahead of their time, engineering trade-offs that still generate debate, and an abstraction layer whose costs and benefits became fully clear years after its inception. The Lineage: MapReduce, FlumeJava, MillWheel Beam did not appear from nothing. It descends from three internal Google systems, each solving a different piece of the data processing puzzle. MapReduce, introduced in a 2004 paper, described the mental model: Split work across machines, process in parallel, and combine the results. Hadoop took that idea open-source and launched a decade of big data infrastructure. But MapReduce was batch-only, so it assumed your data had a beginning and an end. FlumeJava (2010) raised the abstraction. Instead of thinking in terms of maps and reducing steps, engineers described pipelines of transformations on collections. The system handled optimization and parallelization, so engineers had more focus on the domain problem at hand, and thus it made batch pipelines readable and composable. MillWheel (2013) tackled streaming. It processed events one at a time, maintained state, and handled exactly-once semantics at Google's scale, but it was a separate system with a separate programming model. If you wanted to run your pipeline logic in both batch and streaming, you would maintain two codebases. This was a problem: two codebases meant two mental models and, inevitably, two sets of bugs. The 2015 Dataflow paper proposed the fix: Treat batch as a special case of streaming, not the other way around. Bounded data is just unbounded data that happens to end. This sounds obvious in retrospect, but at the time, it was a big shift. The Donation and the Incubator In January 2016, Google and partners — Cloudera contributed a Spark runner, dataArtisans (now Ververica) contributed a Flink runner, and Talend joined the effort — donated the Cloud Dataflow SDKs to the Apache Software Foundation. The project entered the Apache Incubator under the name Beam, a portmanteau of Batch and strEAM. The incubation was fast. By December 2016, Beam graduated to a top-level Apache project. The numbers from the graduation assessment tell a story: out of roughly 22 major modules in the codebase, at least 10 had been developed from scratch by the community with minimal Google contribution. No single organization held more than 50% of unique monthly contributors. A perfect example of open source done right. The first stable release, version 2.0.0, came in May 2017. At that point, Beam was in production use at Google Cloud, PayPal, and Talend. Five runners were officially supported. The programming model had proven itself inside Google for over a decade; now it had the opportunity to prove itself everywhere else. What Beam Got Right Three core design decisions have held up over the past ten years. They are worth examining because they explain why Beam survived in a market crowded with alternatives. Batch Is a Special Case of Streaming The Dataflow paper's central insight was that the same four questions apply to all data processing: What results are being computed? Where in event time are results grouped? When in processing time are results materialized? How do refinements of results relate? This framework — what, where, when, how — turned out to be general enough to express everything from a simple MapReduce job to complex session-windowed streaming aggregations. It meant LinkedIn could write one pipeline and run it in batch mode on Spark for backfills and in streaming mode on Samza for real-time processing. When they did this, their backfill duration dropped from seven hours to 25 minutes, and memory consumption was cut in half. Runner Abstraction Beam pipelines do not execute directly. They compile to a runner — Dataflow, Flink, Spark, Samza, or others — which handles the actual distributed execution. At the time, this was a controversial choice - it meant Beam is always an abstraction over something else, and abstractions have overhead. But in retrospect, the trade-off has aged well. Ricardo, Switzerland's largest online marketplace, built Beam pipelines on a self-managed Flink cluster in their data center. When they migrated to Google Cloud, they switched to the Dataflow runner without rewriting pipeline code. It saved them months of engineering work. Palo Alto Networks runs its cybersecurity platform on Beam with both the Dataflow runner (on GCP) and Flink (on AWS). In their own words: "With the right abstraction we have the flexibility to run workloads where needed. Thanks to Beam, we are not locked to any vendor." Windowing and watermarks as First-Class Concepts Most streaming frameworks bolted on windowing support after the fact. Conveniently fixed windows, sliding windows, session windows, and custom window functions are all part of the Beam core model. Watermarks — heuristic estimates of how far behind your data might be — are a foundational mechanism. In practice, this matters a lot. For example, at LinkedIn, the anti-abuse platform uses Beam's windowing to aggregate user activity signals in real-time, reducing the time to label abusive behavior from days to minutes. At Palo Alto Networks, sub-second windowing over hundreds of billions of security events per day makes the difference between catching an intrusion and missing it. The GCP Angle: Where Beam and Dataflow Reinforce Each Other Beam's relationship with Google Cloud Platform deserves specific examination because it illustrates both the strengths and the tensions of the project. Dataflow is the only fully managed, serverless runner for Beam. With Dataflow, you do not provision clusters, nor do you tune executor memory. You write a Beam pipeline, pass --runner=DataflowRunner in your options, and the service handles autoscaling, fault tolerance, and monitoring. For teams already invested in GCP — using Pub/Sub for messaging, BigQuery for analytics, Cloud Storage for data lakes — the integration is seamless. Google recently introduced Managed I/O for Dataflow, which automatically upgrades your Beam I/O connectors to the latest vetted version during job submission. If a critical bug fix lands in the Beam Kafka connector, Dataflow will pick it up without you changing a line of code, as of writing this blog post no self-managed Flink or Spark cluster can offer this. The pattern I've seen work especially well in my experience: Pub/Sub → Dataflow (Beam) → BigQuery. You can read from BigQuery in batch mode for historical backfills using ReadFromBigQuery with a SQL query, or read from Pub/Sub in streaming mode for real-time ingestion. Google published a codelab in 2025 showing Beam pipelines running Gemini model inference through Dataflow's RunInference API, with results written to BigQuery. The data processing layer and the ML inference layer are the same pipeline. There is, however, tension here: the more you depend on Managed I/O and Dataflow-specific optimizations, the less portable your pipeline becomes in practice. You are using an abstraction layer designed for portability while building on features unique to one runner. This is not necessarily wrong; it might be the right engineering choice for your team, but you should make it with open eyes. What Beam Got Wrong, or at Least Has Not Fixed I believe that honesty about a technology's weaknesses is more useful than cheerleading, and Beam has real gaps. Performance Overhead The runner abstraction adds a translation layer between your code and execution. Benchmarks published by Beside the Park in September 2025 measured Java on Beam's Portable Runner at up to 2x slower than Classic Runners. The Portable Runner enables cross-language pipelines — a Python transform talking to a Java transform in the same pipeline, but if your entire pipeline is Java, you are paying for portability you do not use. Classic Runners (available for JVM languages) perform better, but the gap between Beam-on-Flink and native Flink is still nonzero. Debugging Complexity When a Beam pipeline fails on Dataflow, you are debugging through two layers: Beam's SDK-level logic and the runner's execution translation. When something goes wrong with BigQuery writes, for example, errors surface through Beam's FailedRows side output — a well-designed pattern, but one that adds indirection. When it is 2 AM, and your pipeline is stuck, every layer between you and the root cause adds minutes and is not fun in general. Ecosystem Size Relative to Spark Spark has a vastly larger community, more Stack Overflow answers, more blog posts, more hiring candidates, and more mature notebook-based tooling (Jupyter, Databricks). If you Google a Beam error message, you might find three relevant results. If you Google a Spark error message, you will find thirty. Now, obviously, with the introduction of LLM tools, this is not as pressing a problem as it was in 2016, for example, but this still matters for engineering teams making technology choices. A tool is only as good as the team's ability to debug and maintain it. Beam YAML Is Promising But Unproven for Complex Workloads Beam YAML, the no-code SDK that went stable in version 2.52, lets engineers define pipelines declaratively in YAML configuration files instead of writing SDK code. It just gained Iceberg support in March 2026. The concept is: most production pipelines are not clever, and they do not need 500 lines of Java. But the Beam blog itself acknowledged that YAML "has gained little adoption for complex ML tasks." The Production Evidence at Scale Here is what Beam runs today, based on published case studies: LinkedIn: 4 trillion events daily, 3,000+ pipelines across multiple data centers. Unified streaming and batch processing through Samza and Spark runners. 2x cost optimization with anti-abuse labeling accelerated from days to minutes. Palo Alto Networks: Hundreds of billions of security events per day. 30,000 Dataflow jobs. 15 million events per second. 4 petabytes of daily data volume. Processing costs reduced by more than 60%. Booking.com: 1M+ queries monthly for ad bidding and performance analytics. 2 PB+ of analytical data scanned. 36x processing acceleration. 4x faster time-to-market. Credit Karma: 5-10 TB processed daily at 5K events per second. 20,000+ ML features managed. Pipeline uptime jumped from 80% to 99%. What the Next Decade Needs If Beam is going to remain relevant for the next ten years, there are specific problems the community needs to address. Close the performance gap with native runners. The abstraction tax is real, and in an era where cloud compute bills are under constant scrutiny, a 2x overhead is a hard sell for performance-sensitive workloads. The Portability Framework needs to improve, or the community needs to invest more in engine-specific optimizations within the runner implementations. Make state management competitive with Flink. Flink's built-in state management — with fine-grained checkpointing and queryable state — is ahead of what Beam offers natively. Beam delegates state handling to the runner, which means state behavior varies depending on your execution engine. For stateful streaming applications, this inconsistency is a friction point. Invest in Beam YAML for the 80% use case. Most data pipelines are not LinkedIn-scale streaming systems; they are extract-transform-load jobs that read from one place, apply some business rules, and write to another. If Beam YAML can become the standard way to express those pipelines — with full Managed I/O support on Dataflow and good integration with Iceberg and Kafka — it could expand Beam's reach far beyond the current community of JVM and Python SDK users. Build better tooling for debugging and observability. The gap between Beam's pipeline abstraction and the runner's execution reality is where engineers lose hours. Better error messages, better tracing through the SDK → runner → execution boundary, and better integration with standard observability stacks (OpenTelemetry, Prometheus) would lower the operational cost of running Beam in production. On a more personal note, seeing improvements to the DirectRunner would go a long way. In my experience, the DirectRunner is where most engineers first encounter Beam, and it is also where the gap between "works locally" and "works on Dataflow" is most painful. A DirectRunner that more faithfully simulates distributed execution semantics, even at the cost of being slower, would catch entire categories of bugs before they reach a staging environment. Conclusion Apache Beam is not the right tool for every data pipeline. If your workload is batch-only and your team already knows Spark, switching to Beam for theoretical portability you may never exercise is a bad trade. If you need the absolute lowest latency in a streaming system and you know Flink well, native Flink will outperform Beam-on-Flink. But for a specific and growing set of problems — unified batch and streaming with the same code, genuine multi-runner portability during cloud migrations, serverless execution on GCP via Dataflow, ML inference embedded in data pipelines — Beam is the strongest option available. Ten years ago, a team at Google argued that unbounded data processing needed a new foundation. The model they proposed has survived contact with reality at a scale few other frameworks can claim. Beam Summit 2026 is happening June 22–23 in New York City. If the next decade is anything like the last, the conversations there will shape how we process data for years to come.

By Abgar Simonean

Beyond Caching: Content Delivery Networks

Consider a user in Australia browsing their social media feed to catch up with friends in Europe and America. The media shared by friends takes a considerable time to load despite the user having a reasonably fast internet connection — while the same content loads instantly for those browsing from within Europe. Consider another user in America trying to watch a live concert in Europe on their device. The broadcast is interrupted briefly but frequently. However, for the European audience, the broadcast is seamless. In both cases, users outside the geography faced delays in accessing content over the internet due to increased round-trip time and additional network hops. This happens despite users having reasonably fast internet connections and providers having servers with enough capability to serve traffic and withstand spikes. To provide a fair user experience, content providers need to ensure the geographical disadvantage is blunted by serving content locally. This is the core problem that Content Delivery Networks (CDNs) solve — bringing content closer to the user by caching and serving it from geographically distributed edge servers. Content Delivery Network (CDN) Definition Formally, a Content Delivery Network (CDN) is a geographically distributed network of proxy servers and corresponding data centers. The primary purpose of a CDN is to provide content at high speed. Thus, it shouldn’t be considered as a replacement of a web host but a service to help traditional web host overcome various limitations. Core Components A typical CDN consists of below core components → Origin Server → The primary server where the original, authoritative version of content resides. This server is the web host. Without a CDN, every user request would hit this server directly, regardless of their location.Edge Servers → CDN cache servers deployed at the “edge” of the network, physically closer to end users. They store cached copies of content. When a user requests a resource, the nearest edge server serves it, drastically reducing round-trip time (RTT).Point of Presence (PoP) → A PoP is a physical data center location housing a cluster of edge servers. Major CDN providers operate hundreds of PoPs worldwide. Each PoP serves users in its geographic vicinity — think of them as regional cache of content.Internet Exchange Points (IXPs) → These are physical locations where different networks (ISPs, CDNs, cloud providers) interconnect and exchange traffic. CDNs strategically collocate at IXPs to peer directly with ISPs, minimizing network hops and improving delivery speed. Traffic Management A typical CDN utilizes below to manage traffic → Global Server Load Balancing (GSLB) → A DNS-based mechanism that intelligently routes user requests to the optimal PoP based on factors like geographic proximity, server health, network congestion, and current load.Selector (Request Routing) → The decision logic — often working alongside GSLB — that determines which edge server within a PoP handles a specific request, factoring in content availability, server capacity, and session affinity. Key Concepts Offloading → The percentage of requests served directly by edge servers without going back to the origin. A high cache-hit ratio (e.g., 95%) means significant offloading — reducing origin bandwidth, compute costs, and the risk of origin overload.Footprint → Refers to the CDN’s global reach — the total number and distribution of PoPs, edge servers, and network capacity. A larger footprint means better coverage, lower latency for diverse user bases, and greater resilience against regional failures. In Summary → Users hit a nearby edge server at a PoP (often at an IXP), routed there by GSLB/selectors, offloading traffic from your origin — all enabled by the CDN’s global footprint. Type of CDNs CDNs can be classified based on the networking techniques they use to route and deliver content → Anycast-Based CDN → Uses Border Gateway Protocol Anycast routing, where the same IP address is announced from multiple geographically distributed PoPs. Thus, when a user sends a request, the network’s BGP routing directs the packet to the nearest (in terms of network hops/latency) server advertising that IP. This is simple, fast failover, resilient to DDoS attacks (traffic is naturally distributed) and utilized by Cloudflare, Google Cloud CDN.DNS-Based CDN → Uses DNS resolution to direct users to the optimal edge server. Thus, a user request is resolved to a domain, the CDN’s authoritative DNS server returns the IP of the closest or least-loaded edge server based on the user’s location (via the resolver’s IP or EDNS Client Subnet). This provides fine-grained control over routing decisions (can factor in server load, geography, health). However, it suffers from DNS caching/TTL delays; routing is based on the DNS resolver’s location, not always the end user’s. Utilized by Akamai and Amazon CloudFront.Unicast-Based CDN → Uses a unique IP address for each edge server, and traffic is directed via DNS or application-layer logic. The CDN’s control plane decides which specific server IP to hand back for a given request. Although this provides full control over which server handles which request, it requires more complex routing logic at the application/DNS layer.Multicast-Based CDN → Uses IP Multicast to deliver the same content to multiple recipients simultaneously. A single stream is sent and replicated at network routers to reach all subscribers — avoids sending duplicate copies. This is extremely efficient for live streaming/broadcast scenarios. However, due to limited multicast support across the public internet it is mostly used within managed/private networks (IPTV, enterprise).Peer-to-Peer (P2P) Hybrid CDN → Combines traditional CDN edge servers with P2P networking among end users. Users who have already downloaded content share chunks with nearby peers, reducing load on origin/edge servers. This scales massively for popular content; reduces bandwidth costs. However, it heavily depends on peer availability this latency can vary. Moreover, it has potential security/privacy concerns.Application-Layer (Overlay) CDN → Builds a logical overlay network on top of the existing internet infrastructure, using application-layer routing. Edge servers communicate with each other through an optimized overlay topology (not relying on default BGP paths). Requests are routed through intermediate CDN nodes for optimal performance. It can optimize around congestion, packet loss, and suboptimal BGP routes with added complexity though. This also requires a sophisticated control plane. Benefits & Use cases CDN provides below benefits → Reduced Latency → CDNs cache content on edge servers geographically closer to users, drastically reducing round-trip time.High Availability & Redundancy → Traffic is distributed across multiple servers, so if one node fails, others handle requests seamlessly.Scalability → CDNs absorb traffic spikes (e.g., flash sales, viral content) without overloading the origin server.Bandwidth Cost Savings → Caching reduces the number of requests hitting the origin, lowering bandwidth and infrastructure costs.Security → Many CDNs offer DDoS mitigation, WAF (Web Application Firewall), and TLS termination at the edge.Improved SEO → Faster page loads positively impact search engine rankings. Common Use Cases for CDN are → Static asset delivery → Images, CSS, JavaScript, fonts, and videos (e.g. social media sites).Video/audio streaming → Low-latency media delivery at scale (e.g., Netflix, YouTube).Software distribution → Serving binaries, patches, and updates (e.g., OS updates, game downloads).API acceleration → Caching API responses for read-heavy workloads.E-commerce → Handling global traffic with consistent performance during peak events. In short, CDNs are essential for any application that serves content to a geographically distributed audience and needs fast, reliable delivery. However, its imperative to know When Not to Use a CDN → Highly dynamic/personalized content → User-specific dashboards, real-time data feeds, or authenticated API responses gain minimal caching benefit.Real-time applications → WebSocket connections, live gaming, or chat systems require persistent connections poorly suited to CDN architecture.Geographically concentrated users → If your audience is near the origin server, a CDN adds unnecessary intermediary hops.Sensitive/regulated data → Distributing confidential or compliance-bound content (e.g., healthcare, financial) across third-party edge servers raises security and legal concerns.Small-scale projects → The operational complexity and cost outweigh performance gains for low-traffic applications. Limitations CDNs cache content at edge servers, but cache invalidation is complex — stale content can persist after updates. They add cost overhead (bandwidth fees, per-request charges) that may not justify the benefit for low-traffic sites. CDNs offer limited control over edge server behavior and can introduce debugging complexity when issues arise across distributed nodes. They also have origin dependency — if your origin server fails, the CDN can only serve cached content until it expires. Additionally, latency for cache misses can actually be higher than direct origin requests due to extra routing hops. Conclusion Content Delivery Networks have become a cornerstone of modern web architecture, ensuring that applications deliver fast, reliable, and secure experiences to users regardless of geography. By caching content closer to end users, intelligently routing traffic, and providing resilience against spikes and failures, CDNs address the fundamental challenges of latency and scalability on the internet. While they offer significant benefits — ranging from performance gains to cost savings and security enhancements — CDNs are not a one-size-fits-all solution. Their limitations, such as cache invalidation complexity and added operational overhead, must be carefully weighed against project needs. For software engineers and architects, understanding when and how to leverage CDNs is critical to building systems that balance efficiency, reliability, and cost-effectiveness in a globally connected world. References and Further Reads CDN (Wikipedia)Couldfare — What is CDNAkamai — What is CDNCDN Success StoriesCase Study — Multi CDN

By Ammar Husain

CORE

Data Processing for Real Estate: Enabling Smart Analysis and Decision-Making

Do you think real estate success still depends on gut feelings and market hunches? Those days are over. Data analysis has become the lifeline of modern real estate operations and has changed how property valuations, market trends, and investment decisions work. Administrators in real estate firms now deal with diverse sets of information from various sources, including property records, market transactions, demographic changes, economic indicators, and customer interactions. These datasets remain unused without appropriate data processing techniques. Valuable insights remain concealed under heaps of unstructured data. Market visibility stands as a compelling reason behind this move. Property developers, brokers, and investors who use advanced data analytics get complete views of market conditions, emerging opportunities, and potential risks before their competitors notice them. Data processing help real estate companies spot hidden patterns and connections that humans might miss. Significance of Data Processing for Real Estate Data processing services providers help real estate professionals turn raw property data into practical insights. These services help bridge the gap between large volumes of information and the strategic decisions that modern real estate operations just need. Property data processing products cover everything from collection and organization to cleansing, analysis, and presentation of information. They work as detailed systems built for property market challenges, not just as separate tech tools. Data processing providers act as enablers of change through several methods. They build reliable data systems that bring together scattered information from property records, transaction histories, tenant details, and maintenance logs to create unified knowledge bases.The combined data environment removes the information barriers that impact real estate operations.Processing solution providers bring in standard methods to handle property data. They use consistent classification systems, data validation protocols, and quality control measures to ensure reliable information. These standards are the foundations for meaningful analysis. Companies usually adapt to these changes step by step instead of making sudden changes. Service providers know that changing company culture needs both tech solutions and good change management. They start with projects that show quick results but cause minimal disruption. This builds trust in evidence-based methods. Real estate companies that use external data processing support get expert help without building internal teams. This partnership enables property firms to stick to their main business while making use of information analysis tools. Steps Taken by Data Processing Experts to Enable Analysis and Decision-Making Data processing in real estate follows a clear path that turns scattered information into valuable assets. Real estate enterprises can streamline their operations through professional data processing providers who use a well-laid-out approach. 1. Data Inventory and Source Identification Professional providers start with a full picture of existing information assets throughout the organization. They catalog all data sources from property management systems and transaction databases to CRM platforms and external market feeds. The experts identify information gaps and redundancies. Data maps outline how different sources connect within the real estate operational ecosystem. 2. Data Collection and Extraction Specialized tools extract relevant information from various formats after source identification. Automated data processing techniques handle structured databases and unstructured sources like property descriptions, lease agreements, and maintenance records. Raw data preparation happens through standardized formats that preserve data integrity. 3. Data Centralization and Quality Management Providers build central data repositories where scattered information exists together in unified structures. Through robust quality control mechanisms, experts discover imprecisions and duplications in datasets during this consolidation stage. The cleansing algorithms help data processing experts resolve errors and standardize naming conventions, street addresses, and property classifications. This creates a single source of truth for all real estate data assets. 4. Implementation of Advanced Analytics Techniques Structured datasets transform into valuable insights through analytical solutions. Data processing experts implement advanced analytics algorithms in the real estate digital infrastructure to offer insights into property valuations, maintenance expenses, and market trends. This enables stakeholders to acquire smart forecasting capabilities without developing internal expertise. 5. Visualization and Dashboarding Tailored dashboards showcase key performance indicators, market movement alerts, and predictive forecasts through smart visual components. Real estate professionals can assess the patterns behind recommendations and visualizations through drill-down capabilities. Stakeholders acquire instant access to complex analyses in visual formats. Key Strengths Uncovered by Automated Data Processing for Real Estate Leaders Leveraging smart data processing techniques enable real estate leaders to accelerate digital transformation initiatives and experience various technical strengths. These benefits enable realtors to modernize diverse operational areas and improve market positioning. 1. Continuous Market Monitoring and Forecasting Real estate professionals receive constant property market intelligence by outsourcing data processing. Smart data processing systems enable realtors to monitor pricing variations, inventory changes, and demographic patterns in real time. Stakeholders can discover emerging geographical trends and investment opportunities before their market competitors. Through predictive modeling, realtors can forecast future market movements depending on historical patterns and current indicators. 2. Service Modernization and Cost Reduction By implementing smart data processing solutions, relators can streamline operations that once required human intervention. The data processing automation improves property valuations, maintenance scheduling, and tenant communications. This modernization minimizes operational expenses while improving service delivery speed and precision. 3. Improving Client and Tenant Experiences Data processing experts create individual-specific interactions based on detailed client profiles. Real estate administrators forecast tenant requirements through behavioral analysis and preference tracking. This strategic approach strengthens stronger relationships with customers and improves retention rates through tailored interaction and service offerings. 4. Strengthening Risk Management and Compliance Smart data processing systems help realtors discover potential compliance issues before they transform into legal concerns. Property regulations, tax obligations, and insurance requirements are monitored by data processing tools. Proactive detection and remediation help real estate firms eliminate legal and financial risks. Leveraging Advanced Data Processing Trends for Real Estate Smart real estate companies can utilize advanced data processing techniques to remain ahead of market competition. These enterprises can leverage sophisticated processing capabilities by collaborating with outsourcing service providers under feasible resource investments. 1. Real-Time Streaming and Event Analytics Real estate companies that outsource data processing services get continuous data streams about property interactions. This visibility helps them respond quickly to market changes, tenant activities, and maintenance needs. Environmental data processing mechanisms alert property managers about potential issues before they get pricey. 2. Natural Language Processing and Text Analytics Language processing technologies convert unstructured text from property listings, tenant messages, and market reports into useful data. These systems autonomously pull important details from documents and analyze client feedback sentiment. They also organize property features uniformly across portfolios. 3. Geospatial and Location Intelligence Analytics Location analysis now goes beyond basic mapping. It includes complex modeling of neighborhood changes, traffic patterns, and development opportunities. Companies that outsource data processing can use expert knowledge to combine multiple geographical data layers. This helps them discover hidden factors that drive property values. 4. Automated Data Governance and Quality Assurance Modern data processing systems can monitor themselves to check information accuracy and consistency. The processing systems highlight unusual patterns, observe data sources, and uphold compliance rules. The outcome is a regulated database that remains precise with minimal human intervention. Final Words Data processing has changed the real estate industry from gut-based decisions to informed operations. Real estate professionals now get detailed market visibility and spot hidden patterns to improve their operations. Those who use data analytics gain a major edge in today's complex property world. Professional data processing providers help businesses turn scattered information into valuable assets. They build central data stores and set up quality checks that create a base for advanced analytics and easy-to-use visuals. Real estate enterprises that outsource data processing can access useful insights without building specialized teams.

By Peter Leo

Swift: The Complete Guide to Error Handling in the Network Layer

In my previous article, we explored how to construct a robust, abstract network layer using Clean Architecture. The response was fantastic, but I received a recurring piece of feedback: the error handling was a bit too thin for a real-world production environment. Categorizing HTTP Status Codes To provide a more granular and descriptive way of handling network events, I decided to categorize HTTP status codes into specific enums. This approach ensures that our logic is both type-safe and highly readable. By referencing the MDN Web Docs, I mapped out each response category to its own structure. This categorization allows us to handle informational updates, successful transfers, and various error types with specialized logic rather than a giant, messy switch statement. The Unified Interface: HTTPResponseDescription Before diving into the specific error groups, we need a “blueprint.” The HTTPResponseDescription protocol ensures that every response type in our system, regardless of its origin, exposes two critical pieces of information: the numeric status code and a human-readable description. This is the “secret sauce” that allows our UI layer to display meaningful messages to the user without needing to know the technical details of the error. Swift protocol HTTPResponseDescription { var statusCode: Int { get } var description: String { get } } Handling System-Level Failures: NSURLErrorCode While HTTP status codes (like 404 or 500) tell us what the server thinks, sometimes the request doesn’t even reach the server. This happens when the URL is malformed, the connection times out, or the internet is simply gone. To handle these “pre-response” failures, I created the NSURLErrorCode enum. By conforming it to our HTTPResponseDescription protocol, we can handle these low-level network issues using the exact same pattern as our HTTP responses. Swift enum NSURLErrorCode: Error, HTTPResponseDescription { case unknown case invalidResponse case badURL case timedOut case decodingError case outOfRange(Int) init(code: Int) { switch code { case 0: self = .unknown case 1: self = .invalidResponse case 2: self = .badURL case 3: self = .timedOut default: self = .outOfRange(code) } } var statusCode: Int { switch self { case .unknown: return 0 case .invalidResponse: return 1 case .badURL: return 2 case .timedOut: return 3 case .decodingError: return 4 case .outOfRange(let code): return code } } var description: String { switch self { case .badURL: return "The URL was malformed." case .invalidResponse: return "Invalid response" case .decodingError: return "Failed to decode the response." case .outOfRange(let statusCode): return "The request \(statusCode) was out of range." case .unknown: return "An unknown error occurred." case .timedOut: return "The request timed out." } } } 1xx: Informational Responses The first group represents Informational Responses, which indicate that the request was received and the process is continuing. Swift /// 1..x enum InformationalResponse: Error, HTTPResponseDescription { case continueResponse case switchingProtocols case processingDeprecated case earlyHints case unknown(Int) init(code: Int) { switch code { case 100: self = .continueResponse case 101: self = .switchingProtocols case 102: self = .processingDeprecated case 103: self = .earlyHints default: self = .unknown(code) } } var statusCode: Int { switch self { case .continueResponse: return 100 case .switchingProtocols: return 101 case .processingDeprecated: return 102 case .earlyHints: return 103 case .unknown(let code): return code } } var description: String { switch self { case .continueResponse: return "Continue" case .switchingProtocols: return "Switching Protocols" case .processingDeprecated: return "Processing" case .earlyHints: return "Early Hints" case .unknown(let code): return "Unknown code: \(code)" } } } 2xx: Successful Responses While we often focus on handling errors, understanding the nuances of success is equally important for a high-quality network layer. The 2xx category indicates that the client’s request was successfully received, understood, and accepted. While a simple 200 OK is the most common response, other codes like 201 Created (essential for POST requests) or 204 No Content (common for DELETE operations) provide critical context to your business logic. By explicitly mapping these, we can trigger specific UI updates — like navigating back after a successful creation — with absolute certainty. Swift /// 2xx Success: The action was successfully received, understood, and accepted. enum SuccessfulResponses: Error, Equatable, HTTPResponseDescription { case ok case created case accepted case nonAuthoritativeInformation case noContent case resetContent case partialContent case multiStatus case alreadyReported case imUsed case unknown(Int) init(code: Int) { switch code { case 200: self = .ok case 201: self = .created case 202: self = .accepted case 203: self = .nonAuthoritativeInformation case 204: self = .noContent case 205: self = .resetContent case 206: self = .partialContent case 207: self = .multiStatus case 208: self = .alreadyReported case 226: self = .imUsed default: self = .unknown(code) } } var statusCode: Int { switch self { case .ok: return 200 case .created: return 201 case .accepted: return 202 case .nonAuthoritativeInformation: return 203 case .noContent: return 204 case .resetContent: return 205 case .partialContent: return 206 case .multiStatus: return 207 case .alreadyReported: return 208 case .imUsed: return 226 case .unknown(let code): return code } } var description: String { switch self { case .ok: return "OK" case .created: return "Created" case .accepted: return "Accepted" case .nonAuthoritativeInformation: return "Non-Authoritative Information" case .noContent: return "No Content" case .resetContent: return "Reset Content" case .partialContent: return "Partial Content" case .multiStatus: return "Multi-Status" case .alreadyReported: return "Already Reported" case .imUsed: return "IM Used" case .unknown(let code): return "Unknown Success code: \(code)" } } } 3xx: Redirection Messages The 3xx category of status codes indicates that the client must take additional action to complete the request. In many cases, URLSession handles these redirects automatically under the hood. However, being able to explicitly identify them is vital for advanced scenarios, such as optimizing cache performance with 304 Not Modified or debugging unexpected URL changes. By including redirection messages in our service, we gain full visibility into the “hops” our network requests take before reaching their final destination. This is particularly useful when working with legacy APIs or complex content delivery networks (CDNs). Swift /// 3xx Redirection: Further action needs to be taken by the user agent to fulfill the request. enum RedirectionMessages: Error, HTTPResponseDescription { case useProxy case found case seeOther case notModified case useProxyForAuthentication case temporaryRedirect case permanentRedirect case unknown(Int) init(code: Int) { switch code { case 300: self = .useProxy case 302: self = .found case 303: self = .seeOther case 304: self = .notModified case 305: self = .useProxyForAuthentication case 307: self = .temporaryRedirect case 308: self = .permanentRedirect default: self = .unknown(code) } } var statusCode: Int { switch self { case .useProxy: return 300 case .found: return 302 case .seeOther: return 303 case .notModified: return 304 case .useProxyForAuthentication: return 305 case .temporaryRedirect: return 307 case .permanentRedirect: return 308 case .unknown(let code): return code } } var description: String { switch self { case .useProxy: return "Multiple Choices" case .found: return "Found" case .seeOther: return "See Other" case .notModified: return "Not Modified" case .useProxyForAuthentication: return "Use Proxy" case .temporaryRedirect: return "Temporary Redirect" case .permanentRedirect: return "Permanent Redirect" case .unknown(let code): return "Unknown Redirection code: \(code)" } } } 4xx: Client Error Responses This is where things get interesting — and where your app’s logic needs to be the sharpest. The 4xx category represents errors where the request contains bad syntax or cannot be fulfilled. In short: the client (your app) did something the server didn’t like, or the user needs to provide more information. Properly handling 4xx errors is the difference between an app that just says “Error” and one that intelligently guides the user. For instance, a 401 Unauthorized should trigger a login flow, while a 429 Too Many Requests should tell the user to slow down rather than spamming the retry button. Swift /// 4xx Client Error: The request contains bad syntax or cannot be fulfilled. enum ClientErrorResponses: Error, HTTPResponseDescription { case badRequest case unauthorized case forbidden case notFound case methodNotAllowed case notAcceptable case proxyAuthenticationRequired case requestTimeout case conflict case gone case lengthRequired case preconditionFailed case payloadTooLarge case URITooLong case unsupportedMediaType case rangeNotSatisfiable case expectationFailed case misdirectedRequest case unProcessableEntity case locked case failedDependency case upgradeRequired case preconditionRequired case tooManyRequests case requestHeaderFieldsTooLarge case unavailableForLegalReasons case unknown(Int) init(code: Int) { switch code { case 400: self = .badRequest case 401: self = .unauthorized case 403: self = .forbidden case 404: self = .notFound case 405: self = .methodNotAllowed case 406: self = .notAcceptable case 407: self = .proxyAuthenticationRequired case 408: self = .requestTimeout case 409: self = .conflict case 410: self = .gone case 411: self = .lengthRequired case 412: self = .preconditionFailed case 413: self = .payloadTooLarge case 414: self = .URITooLong case 415: self = .unsupportedMediaType case 416: self = .rangeNotSatisfiable case 417: self = .expectationFailed case 421: self = .misdirectedRequest case 422: self = .unProcessableEntity case 423: self = .locked case 424: self = .failedDependency case 426: self = .upgradeRequired case 428: self = .preconditionRequired case 429: self = .tooManyRequests case 431: self = .requestHeaderFieldsTooLarge case 451: self = .unavailableForLegalReasons default: self = .unknown(code) } } var statusCode: Int { switch self { case .badRequest: return 400 case .unauthorized: return 401 case .forbidden: return 403 case .notFound: return 404 case .methodNotAllowed: return 405 case .notAcceptable: return 406 case .proxyAuthenticationRequired: return 407 case .requestTimeout: return 408 case .conflict: return 409 case .gone: return 410 case .lengthRequired: return 411 case .preconditionFailed: return 412 case .payloadTooLarge: return 413 case .URITooLong: return 414 case .unsupportedMediaType: return 415 case .rangeNotSatisfiable: return 416 case .expectationFailed: return 417 case .misdirectedRequest: return 421 case .unProcessableEntity: return 422 case .locked: return 423 case .failedDependency: return 424 case .upgradeRequired: return 426 case .preconditionRequired: return 428 case .tooManyRequests: return 429 case .requestHeaderFieldsTooLarge: return 431 case .unavailableForLegalReasons: return 451 case .unknown(let code): return code } } var description: String { switch self { case .badRequest: return "Bad Request" case .unauthorized: return "Unauthorized" case .forbidden: return "Forbidden" case .notFound: return "Not Found" case .methodNotAllowed: return "Method Not Allowed" case .notAcceptable: return "Not Acceptable" case .proxyAuthenticationRequired: return "Proxy Authentication Required" case .requestTimeout: return "Request Timeout" case .conflict: return "Conflict" case .gone: return "Gone" case .lengthRequired: return "Length Required" case .preconditionFailed: return "Precondition Failed" case .payloadTooLarge: return "Payload Too Large" case .URITooLong: return "URI Too Long" case .unsupportedMediaType: return "Unsupported Media Type" case .rangeNotSatisfiable: return "Range Not Satisfiable" case .expectationFailed: return "Expectation Failed" case .misdirectedRequest: return "Misdirected Request" case .unProcessableEntity: return "Unprocessable Entity" case .locked: return "Locked" case .failedDependency: return "Failed Dependency" case .upgradeRequired: return "Upgrade Required" case .preconditionRequired: return "Precondition Required" case .tooManyRequests: return "Too Many Requests" case .requestHeaderFieldsTooLarge: return "Request Header Fields Too Large" case .unavailableForLegalReasons: return "Unavailable For Legal Reasons" case .unknown(let code): return "Unknown Client Error code: \(code)" } } } 5xx: Server Error Responses The 5xx category is the server’s way of saying, “It’s not you, it’s me.” These status codes indicate cases where the server is aware that it has encountered an error or is otherwise incapable of performing the request. For an iOS developer, handling 5xx errors correctly is crucial for app stability. While a 4xx error might suggest a bug in your request logic, a 5xx error usually means the backend is having a bad day. Identifying a 503 Service Unavailable versus a 504 Gateway Timeout allows you to decide whether to trigger an immediate retry or to show a "Maintenance" screen to the user. Swift /// 5xx Server Error: The server failed to fulfill an apparently valid request. enum ServerErrorResponses: Error, HTTPResponseDescription { case internalServerError case notImplemented case badGateway case serviceUnavailable case gatewayTimeout case httpVersionNotSupported case variantAlsoNegotiates case insufficientStorage case loopDetected case notExtended case networkAuthenticationRequired case unknown(Int) init(code: Int) { switch code { case 500: self = .internalServerError case 501: self = .notImplemented case 502: self = .badGateway case 503: self = .serviceUnavailable case 504: self = .gatewayTimeout case 505: self = .httpVersionNotSupported case 506: self = .variantAlsoNegotiates case 507: self = .insufficientStorage case 508: self = .loopDetected case 510: self = .notExtended case 511: self = .networkAuthenticationRequired default: self = .unknown(code) } } var statusCode: Int { switch self { case .internalServerError: return 500 case .notImplemented: return 501 case .badGateway: return 502 case .serviceUnavailable: return 503 case .gatewayTimeout: return 504 case .httpVersionNotSupported: return 505 case .variantAlsoNegotiates: return 506 case .insufficientStorage: return 507 case .loopDetected: return 508 case .notExtended: return 510 case .networkAuthenticationRequired: return 511 case .unknown(let code): return code } } var description: String { switch self { case .internalServerError: return "Internal Server Error" case .notImplemented: return "Not Implemented" case .badGateway: return "Bad Gateway" case .serviceUnavailable: return "Service Unavailable" case .gatewayTimeout: return "Gateway Timeout" case .httpVersionNotSupported: return "HTTP Version Not Supported" case .variantAlsoNegotiates: return "Variant Also Negotiates" case .insufficientStorage: return "Insufficient Storage" case .loopDetected: return "Loop Detected" case .notExtended: return "Not Extended" case .networkAuthenticationRequired: return "Network Authentication Required" case .unknown(let code): return "Unknown Server Error code: \(code)" } } } The Orchestrator: Unifying the Network Layer Now that we have defined our granular categories, we need a single source of truth to manage them. This is where the NetworkHTTPResponseService comes in. It acts as a “Master Enum” — an orchestrator that takes a raw HTTPURLResponse and transforms it into a strictly typed, categorized result. By using Associated Values, we can nest our specific enums (like ClientErrorResponses) inside this service. This allows our network layer to remain clean: instead of checking dozens of status codes, it simply checks which "category" the response falls into. Swift // The main orchestrator service that unifies all HTTP response categories. /// It simplifies error handling by wrapping specific groups into associated values. enum NetworkHTTPResponseService: Error, Equatable, HTTPResponseDescription { // MARK: - Equatable Implementation /// Compares two responses based on their numeric status codes. static func == (lhs: NetworkHTTPResponseService, rhs: NetworkHTTPResponseService) -> Bool { return lhs.statusCode == rhs.statusCode } // MARK: - Cases case informationResponse(InformationalResponse) case successfulResponse(SuccessfulResponses) case redirectionMessages(RedirectionMessages) case clientErrorResponses(ClientErrorResponses) case serverErrorResponses(ServerErrorResponses) case unknownError(_ status: Int) case badRequest(codeError: NSURLErrorCode) // Handles system-level URL errors // MARK: - Initializer /// Automatically categorizes the response based on the HTTP status code range. init(urlResponse: HTTPURLResponse) { let statusCode = urlResponse.statusCode switch statusCode { case 100..<199: self = .informationResponse(InformationalResponse(code: statusCode)) case 200..<299: self = .successfulResponse(SuccessfulResponses(code: statusCode)) case 300..<399: self = .redirectionMessages(RedirectionMessages(code: statusCode)) case 400..<499: self = .clientErrorResponses(ClientErrorResponses(code: statusCode)) case 500..<599: self = .serverErrorResponses(ServerErrorResponses(code: statusCode)) default: self = .unknownError(statusCode) } } // MARK: - Convenience Getters /// Safely unwraps the successful status if the response was a success. var successfulStatus: SuccessfulResponses? { if case .successfulResponse(let status) = self { return status } return nil } /// Safely unwraps the client error if the request was malformed or unauthorized. var clientError: ClientErrorResponses? { if case .clientErrorResponses(let status) = self { return status } return nil } // MARK: - HTTPResponseDescription Conformance var statusCode: Int { switch self { case .informationResponse(let code): return code.statusCode case .successfulResponse(let code): return code.statusCode case .redirectionMessages(let code): return code.statusCode case .clientErrorResponses(let code): return code.statusCode case .serverErrorResponses(let code): return code.statusCode case .unknownError(let code): return code case .badRequest(let codeError): return codeError.statusCode } } var description: String { switch self { case .informationResponse(let code): return "Informational: \(code.description)" case .successfulResponse(let code): return "Success: \(code.description)" case .redirectionMessages(let code): return "Redirection: \(code.description)" case .clientErrorResponses(let code): return "Client Error: \(code.description)" case .serverErrorResponses(let code): return "Server Error: \(code.description)" case .unknownError(let code): return "Unknown Status Code: \(code)" case .badRequest(let code): return "Bad System Request: \(code.description)" } } } Putting It All Together: The fetch Implementation This is the final piece of the puzzle. The fetch function is where we apply all the architectural groundwork we've laid. It leverages Swift Concurrency (async/await) and the new Typed Throws feature introduced in Swift 6.0 to provide a compile-time guarantee that this function can only throw a NetworkHTTPResponseService error. Implementation Details The beauty of this method lies in its two-stage validation: Transport level: We catch system-level URLError (like timeouts or lack of connection) and map them to our NSURLErrorCode.Protocol level: Once we have an HTTPURLResponse, we use our orchestrator to decide if the status code represents success or a specific failure. Swift /// Fetches and decodes data from a given URL. /// - Parameter url: The endpoint to request data from. /// - Returns: A decoded object of type T. /// - Throws: A `NetworkHTTPResponseService` error, providing specific details about the failure. func fetch<T>(_ url: URL) async throws(NetworkHTTPResponseService) -> T where T : Decodable { let data: Data let response: URLResponse // Stage 1: Attempt the network transport do { (data, response) = try await urlSession.data(from: url) } catch let error as URLError { // Map low-level system errors to our structured NSURLErrorCode switch error.code { case .badURL: throw NetworkHTTPResponseService.badRequest(codeError: .badURL) case .timedOut: throw NetworkHTTPResponseService.badRequest(codeError: .timedOut) default: throw NetworkHTTPResponseService.badRequest(codeError: .unknown) } } catch { // Fallback for any other non-URLError exceptions throw NetworkHTTPResponseService.badRequest(codeError: .unknown) } // Stage 2: Validate the HTTP protocol response guard let httpResponse = response as? HTTPURLResponse else { throw NetworkHTTPResponseService.badRequest(codeError: .invalidResponse) } // Convert the status code into our categorized enum let responseStatus = NetworkHTTPResponseService(urlResponse: httpResponse) // Stage 3: Handle the categorized result switch responseStatus { case .successfulResponse: do { // Only attempt decoding if the server returned a 2xx status let result = try decoder.decode(T.self, from: data) return result } catch { // Wrap decoding failures as a specific badRequest subtype throw NetworkHTTPResponseService.badRequest(codeError: .decodingError) } default: // Automatically throw 1xx, 3xx, 4xx, or 5xx errors throw responseStatus } } Key Takeaways for Your Network Layer Typed throws (throws(NetworkHTTPResponseService)): By specifying the error type, we eliminate the need for the caller to cast a generic Error to our custom type. The compiler now knows exactly what to expect in the catch block.Decoupled decoding: Decoding only happens inside the .successfulResponse case. This prevents the app from trying to parse a JSON error body into a valid Data Model, which is a common source of "Silent Failures."Readability: The switch responseStatus block is incredibly clean. It clearly separates the "Happy Path" from everything else, making the function easy to scan at a glance. Final Conclusion Building a professional network layer is not just about sending requests; it’s about managing expectations. By categorizing every possible outcome into a strict hierarchy of enums, we’ve transformed a fragile part of our app into a resilient, predictable service. Your UI can now respond with surgical precision to a 401 Unauthorized or a 504 Gateway Timeout, significantly improving the user experience and making your code a joy to maintain. Thank you so much for sticking with me until the very end! I’ve put a lot of thought and effort into this implementation because I believe that clean, predictable code is the foundation of any great app. My goal was to provide you with a “production-ready” pattern that you can literally copy, paste, and adapt into your own projects today. If this guide helped you rethink your error handling or saved you a few hours of debugging, I would truly appreciate your support. Clap for this article to help others find it. Share your thoughts in the comments — I’d love to hear how you handle networking edge cases! Happy coding, and let’s keep building better apps together! Full source code is here.

By Pavel Andreev

Part II: The Network That Doesn't Exist: Zero Trust, Service Meshes, and the Slow Death of Perimeter Security

The conversation that reordered my understanding of enterprise network security happened in a conference room in London in early 2019. The CISO of a mid-size financial services firm — precise, methodical, someone whose threat modeling I trusted — was describing her organization's response to a pen test finding. The testers had gotten onto one internal server through a phishing email. From that single initial access point, within seventy-two hours, they had lateral movement access to fourteen other systems, including two that handled customer account data. The perimeter had been intact throughout. The firewall logs showed nothing anomalous crossing the network boundary. Everything that happened after the initial email was internal traffic, authenticated by the fact that it came from inside the network. There was no enforcement, no verification, nothing that asked whether this particular server had any business talking to those other fourteen. She paused before finishing the thought: "Our security model assumed that if you were inside, you were trustworthy. And for twenty years, that was close enough to true to be acceptable. It is no longer close enough." That was six years ago. The industry has spent those six years building the tooling to replace the assumption with verification. We're far enough along that I can say, with some confidence, that zero trust has crossed from aspiration to implementation for organizations with the resources and operational maturity to do it properly. I can say with equal confidence that the gap between those organizations and the median enterprise remains wide. What "Zero Trust" Actually Means When You Strip the Marketing The term has been applied to so many products and approaches that it has acquired a kind of semantic exhaustion. VPN replacements are marketed as zero trust. Identity providers market their services as zero trust. Network segmentation vendors claim zero trust. The risk is that the label gets applied to any improvement over the worst previous practice, diluting the concept until it means only "better than whatever you had before." The core principle is austere and specific: no network location confers trust. A request originating from inside your data center, from a known server, from an authenticated user, is not trusted until it has been verified at the resource it's trying to access — verified for identity, verified for authorization, and encrypted in transit. The implicit trust granted by network position — "this request comes from inside, so it's probably fine" — is explicitly discarded. In a microservices environment, this plays out at every service-to-service call. When the order service calls the inventory service, the inventory service has no reason, under zero trust principles, to simply accept that call because it comes from an internal IP. It should verify the calling service's cryptographic identity. It should check whether that identity is authorized to call this endpoint. It should require that the connection be mutually authenticated — not just the server presenting its certificate to the client, but both parties verifying each other. This is what mutual TLS, implemented through a service mesh, provides. And this is where implementation gets concrete. The Service Mesh as Zero Trust Infrastructure Istio has become the most widely deployed service mesh for Kubernetes environments — not universally loved, but operationally well-understood and supported by a large enough ecosystem that its patterns have become reference implementations. When Istio's PeerAuthentication resource is set to STRICT mode cluster-wide, no pod-to-pod communication is permitted in plaintext. Every connection requires mutual TLS. Envoy proxies, running as sidecars to each service, handle the certificate management automatically — services don't manage their own certificates, the mesh issues them, rotates them, and verifies them at connection establishment. What this accomplishes in practice is something that traditional network segmentation never cleanly solved: workload identity that's cryptographic rather than positional. The inventory service doesn't trust the order service because it comes from a particular IP range or VLAN. It trusts it because it has presented a valid SPIFFE certificate issued by the cluster's certificate authority to the order service's service account. These are short-lived certificates — typically valid for hours, not years — that are automatically rotated by the mesh. Compromise of a certificate has a strictly bounded impact window. The authorization layer builds on top of this identity foundation. Istio's AuthorizationPolicy lets you express rules like: only the order service's identity may call the inventory service's /reserve endpoint, and only using the POST method. Everything else is denied. This is least-privilege access control at the service level, enforced by the infrastructure rather than by application code — which means it applies even if the application has a bug that would otherwise permit unauthorized access. I want to note something that often gets glossed over in the service mesh literature: this approach requires that you trust the mesh's certificate authority. If Istio's Citadel component is compromised, the trust foundation of your entire zero trust architecture is compromised. This is a concentrated risk that needs to be managed — with proper isolation of the mesh control plane, regular audit of issued certificates, and anomaly detection on connection patterns. Zero trust moves the trust boundary; it doesn't eliminate the need for trust anchors. The Lateral Movement Problem and Why mTLS Solves It Specifically The attack scenario that zero trust architectures are specifically designed to defeat is lateral movement — an attacker who has gained access to one service using that foothold to reach others. The Wiz.io research from late 2024 on cloud security incidents consistently surfaced lateral movement as the mechanism by which initial compromises became material breaches. An attacker gains access to a low-privileged service — perhaps through a vulnerability in a third-party library, or a misconfigured credential — and then uses that service's network position to probe and eventually access higher-value systems. In a traditional flat network, the compromised service can reach anything else on the same VLAN. In an mTLS-enforced mesh with strict authorization policies, it can reach only what its cryptographic identity is explicitly permitted to reach. An engineer at a cloud-native startup in Tel Aviv described a red team exercise to me in December 2025 with a detail I found genuinely striking. Their red team, working with internal access to simulate a compromised service, spent two days attempting lateral movement from an initially compromised low-privilege workload. In their previous architecture — before the Istio migration — the same exercise had taken forty minutes to reach a database containing customer PII. With the mesh in place and authorization policies enforced, the red team concluded after forty-eight hours that lateral movement to any high-value system was not achievable without compromising the mesh control plane itself, which was separately hardened. Forty minutes to forty-eight hours, with no ability to reach the target. That's what enforcement at every hop buys you. The Organizational Friction Nobody Warns You About I've watched a handful of zero trust service mesh deployments go from inception to production, and the consistent surprise — even for organizations that thought they'd planned carefully — is the application portfolio audit. Strict mTLS enforcement breaks any communication that isn't prepared for it. Applications that make direct TCP connections without TLS, services that rely on plaintext HTTP for internal health checks, legacy integrations that predate certificate-based authentication — all of these fail when the mesh enforces mutual TLS. Before you can enforce zero trust, you have to inventory every service-to-service communication in your environment and verify that each one can be migrated. In most organizations of any meaningful age, this inventory doesn't fully exist. The enforcement work reveals the inventory work that should have been done years earlier. This is not a reason to avoid the migration; it's a reason to plan a phased rollout that begins in permissive mode — the mesh observes but doesn't enforce — and uses that observability period to build the communication map before enforcement is enabled. The organizations I've seen do this well ran their mesh in permissive mode for sixty to ninety days, used the resulting telemetry to identify every service-to-service call in the environment, and then worked systematically through the exceptions before flipping the enforcement switch. The organizations I've seen struggle skipped the discovery phase and then spent months firefighting broken integrations after enabling strict mode. A platform architect at a European insurance company who managed their Istio rollout in mid-2025 told me that their ninety-day permissive phase identified forty-three internal services communicating in plaintext that no living engineer knew about. Eleven of them were production services handling policyholder data. They had been invisible to the security team precisely because they predated any network monitoring that would have noticed them. Tokens at the Edge, Certificates Inside The zero trust model splits neatly along a boundary that's worth being explicit about: external traffic and internal traffic require different trust mechanisms, handled at different layers. For traffic entering the cluster from outside — users, partners, external services — the standard is JWT validation at the ingress layer. An OAuth2 token issued by a trusted identity provider, validated by the gateway before any request reaches internal services. The gateway enforces that tokens are present, valid, unexpired, and issued by an authorized identity provider. Claims inside the token can flow inward to services that need to know about the requesting user's identity or permissions. For internal service-to-service traffic, JWT tokens are unnecessary overhead because you already have a better identity mechanism: the SPIFFE certificate issued by the mesh to each workload. The authorization policy can reference these SPIFFE identities directly, with no additional token propagation required. The clean separation matters operationally. Your OAuth2 configuration and your mesh configuration have different lifecycles, different failure modes, and different operational teams. Keeping them conceptually and architecturally distinct prevents a common failure mode where a change to external authentication inadvertently affects internal service authorization, or vice versa. A Note on What Zero Trust Isn't There is a consulting-driven tendency to describe zero trust as a destination — a state you achieve and then maintain. I'd argue this framing creates false confidence and deferred risk. Zero trust is a set of ongoing commitments: to verify every request, to enforce least privilege at every boundary, to audit access patterns continuously, and to update policies as systems and threat landscapes change. A service mesh configured for strict mTLS in January 2025 needs review in January 2026, because new services have been added, old policies may no longer reflect current requirements, and the threat model has evolved. The auditing component — reviewing service-to-service communication logs for unexpected access patterns, tracking certificate issuance, verifying that authorization policies match current architectural intent — is the maintenance work that determines whether zero trust remains zero trust or gradually drifts back into implicit permissiveness through accumulated exceptions and overlooked policy changes. None of this is reason to avoid the architecture. The alternative — flat networks, positional trust, the implicit assumption that inside means safe — has been conclusively demonstrated inadequate. But the work of security isn't a project with a completion date. It's an operational commitment. The mesh enforces the policy you've written. Writing the right policy, keeping it current, and auditing whether it's working as intended — that part is still yours. The author covers cloud security, enterprise infrastructure, and supply chain risk. They have reported on technology organizations across North America, Europe, and the Middle East over fifteen years.

By Igboanugo David Ugochukwu

CORE

Spark on AmpereOne® M Arm Processors Reference Architecture

Introduction Arm technology now powers a broad spectrum of on-premises and cloud server workloads. Building on Ampere Computing's previous reference architecture, which demonstrated that Apache Spark on Ampere Altra – 128C (Ampere Altra 128 Cores) processors delivers superior performance per rack, lower power consumption, and optimized CapEx and OpEx, this paper evaluates and extends that analysis to showcase Spark performance on the latest generation of AmpereOne® M processors. Scope and Audience This document describes the process of setting up, tuning, and evaluating Spark performance using a testbed powered by AmpereOne® M processors. It includes a comparative analysis of the performance benefits of the 12-channel AmpereOne® M processors relative to their predecessors, specifically Ampere Altra – 128C processors. Additionally, the paper examines the Spark performance improvements achieved by using a 64KB page-size kernel over standard 4KB page-size kernels. We outline the installation and tuning procedures for deploying Spark on both single-node and multi-node clusters. These recommendations are intended as general guidelines, and configuration parameters can be further optimized based on specific workloads and use cases. This document is intended for sales engineers, IT and cloud architects, IT and cloud managers, and customers seeking to leverage the performance and power efficiency advantages of Ampere Arm servers across their IT infrastructure. It provides practical guidance and technical insights for professionals interested in deploying and optimizing Arm-based Spark solutions. AmpereOne® M Processors AmpereOne® M is part of the AmpereOne® M family of high-performance server-class processors, designed to deliver exceptional performance for AI Compute and a wide range of mainstream data center workloads. Data-intensive applications such as Hadoop and Apache Spark benefit directly from the 12 DDR5 memory channels, which provide the high memory bandwidth required for large-scale data processing. AmpereOne® M processors introduce a new platform architecture with a higher core count and additional memory channels, differentiating it from earlier Ampere platforms while preserving Ampere’s Cloud Native processing principles. Designed from the ground up for cloud efficiency and predictable scaling, AmpereOne® M employs a one-to-one mapping between vCPUs and physical cores, ensuring consistent performance without resource contention. With up to 192 single-threaded cores and twelve DDR5 channels delivering 5600 MT/s, AmpereOne® M delivers a sustained throughput required for demanding workloads such as Spark, though also including modern AI inference relying on Large Language Models (LLM). AmpereOne® M also emphasizes exceptional performance-per-watt, helping reduce operational costs, energy consumption, and cooling requirements in modern data centers. Apache Spark Apache Spark is a unified data processing and analytics framework used for data engineering, data science, and machine learning workloads. It can operate on a single node or scale across large clusters, making it suitable for processing large and complex datasets. By leveraging distributed computing, Spark efficiently parallelizes data processing tasks across multiple nodes, either independently or in combination with other distributed computing systems. Spark utilizes in-memory caching, which allows for quick access to data and optimized query execution, enabling fast analytic queries on datasets of any size. The framework provides APIs in popular programming languages such as Java, Scala, Python, and R, making it accessible to the broad developer community. Spark supports various workloads, including real-time analytics, batch processing, interactive queries, and machine learning, offering a comprehensive solution for modern data processing needs. Spark supports multiple deployment models. It can run as a standalone cluster or integrate with cluster management and orchestration platforms such as Hadoop YARN, Kubernetes, and Docker. This flexibility allows Spark to adapt to diverse infrastructure environments and workload requirements. Spark Architecture and Components Figure 1 Spark Driver The Spark Driver serves as the central controller of the Spark execution engine and is responsible for managing the overall state of the Spark cluster. It interacts with the cluster manager to acquire the necessary resources, such as virtual CPUs (vCPUs) and memory. Once the resources are obtained, the Driver launches the executors, which are responsible for executing the actual tasks of the Spark application. Additionally, the Spark Driver plays a crucial role in maintaining the state of the application running on the cluster. It keeps track of various important information, such as the execution plan, task scheduling, and the data transformations and actions to be performed. The Driver coordinates the execution of tasks across the available executors, ensuring efficient data processing and computation. Spark Driver, hence, acts as a control unit orchestrating the execution of the Spark application on the cluster and maintaining the necessary states and communication with the cluster manager and executors. Spark Executors Spark Executors are responsible for executing the tasks assigned to them by the Spark Driver. Once the Driver distributes the tasks across the available Executors, each Executor independently processes its assigned tasks. The Executors run these tasks in parallel, leveraging the resources allocated to them, such as CPU and memory. They perform the necessary computations, transformations, and actions specified in the Spark application code. This includes operations like data transformations, filtering, aggregations, and machine learning algorithms, depending on the nature of the tasks. During the execution of the tasks, the Executors communicate with the Driver, providing updates on their progress and reporting the results of each task. Cluster Manager The Cluster Manager is responsible for maintaining the cluster of machines on which the Spark applications run. It handles resource allocation, scheduling, and management of the Spark Driver and Executors, ensuring efficient execution of Spark applications on the available cluster resources. When a Spark application is submitted, the Driver communicates with the Custer Manager to request the necessary resources, such as CPU, memory, and storage, to run the application. It ensures that the resources are distributed effectively to meet the requirements of the Spark application. This includes tasks such as assigning containers or worker nodes to execute the Spark Executors and ensuring that the required dependencies and configurations are in place. Spark RDD Spark uses a concept called Resilient Distributed Dataset (RDD), an abstraction that represents an immutable collection of objects that can be split across a cluster. RDDs can be created from various data sources, including SQL databases and NoSQL stores. Spark Core, which is built upon the RDD model, provides essential functionalities such as mapping and reducing operations. It also offers built-in support for joining data sets, filtering, sampling, and aggregation, making it a powerful tool for data processing. When executing tasks, Spark splits them into smaller subtasks and distributes them across multiple executor processes running on the cluster. This enables the parallel execution of tasks across the available computational resources, resulting in improved performance and scalability. Spark Core Spark Core serves as the underlying execution engine for the Spark platform, forming the basis for all other Spark functionality. It offers powerful capabilities such as in-memory computing and the ability to reference datasets stored on external storage systems. One of the key components of Spark Core is the resilient distributed dataset (RDD), which serves as the primary programming abstraction in Spark. RDDs enable fault-tolerant and distributed data processing across a cluster. Spark Core provides a wide range of APIs for creating, manipulating, and transforming RDDs. These APIs are available in multiple programming languages, including Java, Python, Scala, and R. This flexibility allows developers to work with Spark Core using their preferred language and leverages the rich ecosystem of libraries and tools available in those languages. Spark Scheduler The Spark Scheduler is a vital component responsible for task scheduling and execution. It uses a Directed Acyclic Graph (DAG) and employs a task-oriented approach for scheduling tasks. The Scheduler analyzes the dependencies between different stages and tasks of a Spark application, represented by the DAG. It determines the optimal order in which tasks should be executed to achieve efficient computation and minimize data movement across the cluster. By understanding the dependencies and requirements of each task, the Scheduler assigns resources, such as CPU and memory, to the tasks. It considers factors like data locality, where possible, to reduce network overhead and improve performance. The task-oriented approach of the Spark Scheduler allows it to break down the application into smaller, manageable tasks and distribute them across the available resources. This enables parallel execution and efficient utilization of the cluster's computing power. Spark SQL Spark SQL is a widely used component of Apache Spark that facilitates the creation of applications for processing structured data. It adopts a data frame approach and allows efficient and flexible data manipulation. One of the key features of Spark SQL is its ability to interface with various data storage systems. It provides built-in support for reading and writing data from and to different datastores, including JSON, HDFS, JDBC, and Parquet. This makes it easy to work with structured data residing in different formats and storage systems. Additionally, Spark SQL extends its connectivity beyond the built-in datastores. It offers connectors that enable integration with other popular data stores such as MongoDB, Cassandra, and HBase. These connectors allow users to seamlessly interact with and process data stored in these systems using Spark SQL's powerful querying and processing capabilities. Spark MLlib In addition to its core functionalities, Apache Spark includes bundled libraries for machine learning and graph analysis techniques. One such library is MLlib, which provides a comprehensive framework for developing machine learning pipelines. MLlib simplifies the implementation of machine learning workflows by offering a wide range of tools and algorithms. It simplifies the implementation of feature extraction and transformations on structured datasets and offers a wide range of machine learning algorithms. MLlib empowers developers to build scalable and efficient machine learning workflows, enabling them to leverage the power of Spark for advanced analytics and data-driven applications. Distributed Storage Spark does not provide its own distributed file system. However, it can effectively utilize existing distributed file systems to store and access large datasets across multiple servers. One commonly used distributed file system with Spark is the Hadoop Distributed File System (HDFS). HDFS allows for the distribution of files across a cluster of machines, organizing data into consistent sets of blocks stored on each node. Spark can leverage HDFS to efficiently read and write data during its processing tasks. When Spark processes data, it typically copies the required data from the distributed file system into its memory. By doing so, Spark reduces the need for frequent interactions with the underlying file system, resulting in faster processing compared to traditional Hadoop MapReduce jobs. As the dataset size increases, additional servers with local disks can be added to the distributed file system, allowing for horizontal scalability and improved performance. Spark Jobs, Stages, and Tasks In a Spark application, the execution flow is organized into a hierarchical structure consisting of Jobs, Stages, and Tasks. A Job represents a high-level unit of work within a Spark application. It can be seen as a complete computation that needs to be performed, involving multiple Stages and transformations on the input data. A Stage is a logical division of tasks that share the same shuffle dependencies, meaning they need to exchange data with each other during execution. Stages are created when there is a shuffle operation, such as a groupBy or a join, that requires data to be redistributed across the cluster. Within each Stage, there are multiple Tasks. A Task represents the smallest unit of work in Spark, representing a single operation that can be executed on a partition of the data. Tasks are typically executed in parallel across multiple nodes in the cluster, with each node responsible for processing a subset of the data. Spark intelligently partitions the data and schedules Tasks across the cluster to maximize parallelism and optimize performance. It automatically determines the optimal number of Tasks and assigns them to available resources, considering factors such as data locality to minimize data shuffling between nodes. Spark handles the management and coordination of Tasks within each stage, ensuring that they are executed efficiently and leveraging the parallel processing capabilities of the cluster. Figure 2 Shuffle boundaries introduce a barrier where Stages/Tasks must wait for the previous stage to finish before they fetch map outputs. In the above diagram, Stage 0 and Stage 1 are executed in parallel, while Stage 2 and Stage 3 are executed sequentially. Hence, Stage 2 has to wait until both Stage 0 and Stage 1 are complete. This execution plan is evaluated by Spark. Spark Test Bed The Spark cluster was set up for performance benchmarking. Equipment Under Test Cluster nodes: 3CPU: AmpereOne® MSockets/node: 1Cores/socket: 192Threads/socket: 192CPU speed: 3200 MHzMemory channels: 12Memory/node: 768 GB (12 x 64GB DDR5-5600, 1DPC)Network card/node: 1 x Mellanox ConnectX-6OS storage/node: 1 x Samsung 960GB M.2Data storage/mode: 4 x Micron 7450 Gen 4 NVME, 3.84 TBKernel version: 6.8.0-85Operating system: Ubuntu 24.04.3YARN version: 3.3.6Spark version: 3.5.7JDK version: JDK 17 Spark Installation and Cluster Setup We set up the cluster with an HDFS file system. Hence, we installed Spark as a Hadoop user and configured the disks for HDFS. OS Install The majority of modern open-source and enterprise-supported Linux distributions offer full support for the AArch64 architecture. To install your chosen operating system, use the server Kernel-based Virtual Machine (KVM) console to map or attach the OS installation media, and then follow the standard installation procedure. Networking Setup Set up a public network on one of the available interfaces for client communication. This can be used to log in to any of the servers where client communication is needed. Set up a private network for communication between the cluster nodes. Storage Setup Choose a drive of your choice for the OS to install, clear any old partitions, reformat, and choose the disk to install the OS. Here, a Samsung 960 GB drive (M.2) was chosen for the OS installation on each server. Add additional high-speed NVMe drives to support the HDFS file system. Create Hadoop User Create a user named “hadoop” as part of the OS Install. This user was used for both Hadoop and Spark daemons on the test bed. Post-Install Steps Perform the following post-install steps on all the nodes on OS after the install. yum or apt update on the nodes.Install packages like dstat, net-tools, lm-sensors, linux-tools-generic, python, sysstat for your monitoring needs.Set up ssh trust between all the nodes.Update /etc/sudoers file for nopasswd for hadoop user.Update /etc/security/ limits.conf per Appendix.Update /etc/sysctl.conf per Appendix.Update scaling governor and hugepages per Appendix.If necessary, make changes to /etc/rc.d to keep the above changes permanent after every reboot.Set up NVMe disks as an XFS file system for HDFS. Create a single partition on each of the NVMe disks with fdisk or parted.Create a file system on each of the created partitions using mkfs.xfs -f /dev/nvme[0-n]n1p1.Create directories for mounting as mkdir -p /root/nvme[0-n]1p1. d. Update /etc/fstab with entries and mount the file system. The UUID of each partition in fstab can be extracted from the blkid command.Change ownership of these directories to the ‘hadoop’ user created earlier. Spark Install Download Hadoop 3.3.6 from the Apache website, Spark 3.5.7 from Apache Spark, and JDK11 and JDK17 for Arm64/Aarch64. We will use JDK11 for Hadoop and JDK17 for Spark installs. Extract the tarball files under the Hadoop user home directory. Update Spark and Hadoop configuration files in ~/hadoop/spark/conf and ~/hadoop/etc/hadoop/ and environment parameters in .bashrc per Appendix. Depending on the hardware specifications of cores, memory, and disk capacities, these may have to be altered. Update the Workers’ files to include the set of data nodes. Run the following commands: Shell hdfs namdenode -format scp -r ~/hadoop <datanodes>:~/hadoop ~/hadoop/sbin/start-all.sh ~/spark/sbin/start-all.sh This should start Spark Master, Worker, and other Hadoop daemons. Performance Tuning Spark is a complex system where many components interact across various layers. To achieve optimal performance, several factors must be considered, including BIOS and operating system settings, the network and disk infrastructure, and the specific software stack configuration. Experience with Hadoop and Spark significantly helps in fine-tuning these settings. Keep in mind that performance tuning is an ongoing, iterative process. The parameters in the Appendix are provided as starting reference points, gathered from just a few initial tuning cycles. Linux Occasionally, there can be conflicts between the subcomponents of a Linux system, such as the network and disk, which can impact overall performance. The objective is to optimize the system to achieve optimal disk and network throughput and identify and resolve any bottlenecks that may arise. Network To evaluate the network infrastructure, the iperf utility can be utilized to conduct stress tests. Adjusting the TX/RX ring buffers and the number of interrupt queues to align with the cores on the NUMA node where the NIC is located can help optimize performance. However, if the BIOS setting is already configured as chipset-ANC in a monolithic manner, these modifications may not be necessary. Disks Aligned partitions: Partitions should be aligned with the storage's physical block boundaries to maximize I/O efficiency. Utilities like parted can be used to create aligned partitions.I/O queue settings: Parameters such as the queue depth and nr_requests (number of requests) can be fine-tuned via the /sys/block//queue/ directory paths to control how many I/O operations the kernel schedules for a storage device.Filesystem mount options: Utilizing the noatime option in the /etc/fstab file is critical for Hadoop and HDFS, as it prevents unnecessary disk writes by disabling the recording of file access timestamps. The fio (flexible I/O tester) tool is highly effective for benchmarking and validating the performance of the disk subsystem after these changes are implemented. Spark Configuration Parameters There are several tunables on Spark. Only a few of them are addressed here. Tune your parameters by observing the resource usage from http://:4040. Using Data Frames Over RDD It is preferred to use Datasets or Data Frames over RDD, which include several optimizations to improve the performance of Spark workloads. Spark data frames can handle the data better by storing and managing it efficiently, as they maintain the structure of the data and column types. Using Serialized Data Formats In Spark jobs, a common scenario involves writing data to a file, which is then read by another job and written to another file for subsequent Spark processing. To optimize this data flow, it is recommended to write the intermediate data into a serialized file format such as Parquet. Using Parquet as the intermediate file format can yield improved performance compared to formats like CSV or JSON. Parquet is a columnar file format designed to accelerate query processing. It organizes data in a columnar manner, allowing for more efficient compression and encoding techniques. This columnar storage format enables faster data access and processing, particularly for operations that involve selecting specific columns or performing aggregations. By leveraging Parquet as the intermediate file format, Spark jobs can benefit from faster transformation operations. The columnar storage and optimized encoding techniques offered by Parquet, as well as its compatibility with processing frameworks like Hadoop, contribute to improved query performance and reduced data processing time. Reducing Shuffle Operations Shuffling is a fundamental Spark operation that reorders data among different executors and nodes. This is necessary for distributed tasks such as joins, grouping, and reductions. This data redistribution is expensive in terms of resources, as it requires considerable disk IO, data packaging, and movement across the network. This is crucial to how Spark works, but can severely reduce performance if not understood and tuned properly. The spark.sql.shuffle.partitions configuration parameter is key to managing shuffle behavior. Found in spark-defaults.conf, this setting dictates the number of partitions created during shuffle operations. The optimal value varies significantly, depending on data volume, available CPU cores, and the cluster's memory capacity. Setting too many partitions results in a large number of smaller output files, potentially increasing overhead. Conversely, too few partitions can lead to individual partitions becoming excessively large, risking out-of-memory errors on executors. Optimizing shuffle performance involves an iterative process, carefully adjusting spark.sql.shuffle.partitions to strike the right balance between partition count and size for your specific workload. Spark Executor Cores The number of cores allocated to each Spark Executor is an important consideration for optimal performance. In general, allocating around 5 cores per Executor tends to be a fair allocation when using the Hadoop Distributed File System (HDFS). When running Spark alongside Hadoop daemons, it is vital to reserve a portion of the available cores for these daemons. This ensures that the Hadoop infrastructure functions smoothly alongside Spark. The remaining cores can then be distributed among the Spark Executors for executing data processing tasks. By striking a balance between allocating cores to Hadoop daemons and Spark executors, you can ensure that both systems coexist effectively, enabling efficient and parallel processing of data. It is important to adjust the allocation based on the specific requirements of your cluster and workload to achieve optimal performance. Spark Executor Instances The number of Spark executor instances represents the total count of executor instances that can be spawned across all worker nodes for data processing. To calculate the total number of cores consumed by a Spark application, you can multiply the number of executors by the cores allocated per executor. The Spark UI provides information on the actual utilization of cores during task execution, indicating the extent to which the available cores are being utilized. It is recommended to maximize this utilization based on the availability of system resources. By effectively using the available cores, you can boost your Spark application's processing power and make its overall performance better. It is crucial to look at the resources in your cluster and change the amount of executor instances and cores given to each executor to match. This ensures resources are used effectively and gets the most computational power out of your Spark application. Executor and Driver Memory The memory configuration for Spark's Driver and Executors plays a critical role in determining the available memory for these components. It is important to tune these values based on the memory requirements of your Spark application and the memory availability within your YARN scheduler and NodeManager resource allocation parameters. The Executor's memory refers to the memory allocated for each executor, while the Driver's memory represents the memory allocated for the Spark Driver. These values should be adjusted carefully to ensure optimal performance and avoid memory-related issues. When tuning the memory configuration, it is essential to consider the overall memory availability in your environment and consider any memory constraints imposed by the YARN scheduler and NodeManager settings. By aligning the memory allocation with the available resources, you can optimize the memory utilization and prevent potential out-of-memory errors or performance degradation (swapping or disk spills). It is recommended to monitor the memory usage with Spark UI and adjust the configuration iteratively to achieve the best performance for your Spark workload. Benchmark Tools We used both Intel HiBench and TPC-DS benchmarking tools to measure the performance of the clusters. TeraSort We used the HiBench benchmarking tool to measure the TeraSort performance. HiBench is a popular benchmarking suite specifically designed for evaluating the performance of Big Data frameworks, such as Apache Hadoop and Apache Spark. It consists of a set of workload-specific benchmarks that simulate real-world Big Data processing scenarios. For additional information, you can refer to this link. By running HiBench on the cluster, you can assess and compare its performance in handling various Big Data workloads. The benchmark results can provide insights into factors such as data processing speed, scalability, and resource utilization for each cluster. Update hibench.conf file, like scale, profile, parallelism parameters, and a list of master and slave nodes.Run ~HiBench/bin/workloads/micro/terasort/prepare/prepare.sh.Run ~HiBench/bin/workloads/micro/terasort/spark/run.sh. After executing the above, a file named hibench.report will be generated within the report directory. Additionally, a file named bench.log will contain comprehensive information regarding the execution. The cluster was using a data set of 3 TB. We measured the total power consumed, CPU power, CPU utilization, and other parameters like disk and network utilization using Grafana and IPMI tools. Throughput from the HiBench run was calculated for TeraSort in the following scenarios: Spark running on a single AmpereOne® M node compared with a single node Ampere Altra – 128C (prior generation)Spark running on a single AmpereOne® M node compared with a 3-node AmpereOne® M cluster to measure the scalabilitySpark running on a 3-node AmpereOne® M cluster with 64k page size vs 4k page size TPC-DS TPC-DS is an industry-standard decision-support benchmark that models various aspects of a decision-support system, including data maintenance and query processing. Its purpose is to assist organizations in making informed decisions regarding their technology choices for decision support systems. TPC benchmarks aim to provide objective performance data that is relevant to industry users. For more in-depth information, you can refer to this tpc.org/tpcds/. Similar to TeraSort testing, we conducted TPC-DS benchmark on AmpereOne® M processors using both single-node and 3-node cluster configurations to compare performance with the prior generation Ampere Altra – 128C processors and to assess scalability. Additional performance evaluations on the AmpereOne® M processor compared to Linux kernels configured with 64KB and 4KB page sizes. This test also used a 3 TB dataset across the cluster. To gain deeper insights into system performance, we monitored key performance metrics including total system power consumption, CPU power, CPU utilization, and network utilization. Performance Tests on 3 Node Clusters Figures 3 and 4 We evaluated Spark TeraSort performance using the HiBench tool. The tests were run on one, two, and three nodes with AmpereOne® M processors, and the earlier values obtained on Ampere Altra – 128C were compared. From Figure 3, it is evident that there is a 30% benefit of AmpereOne® M over Ampere Altra – 128C while running Spark TeraSort. This increase in performance can be attributed to a newer microarchitecture design, an increase in core count (from 128 to 192), and the 12-channel DDR5 design on AmpereOne® M (versus 8-channel DDR4 on Ampere Altra – 128C). The output for the 3x nodes configuration, as shown in Figure 4, was found to be close to three times the output of a single node. 64k Page Size Figure 5 We observed a significant performance increase, approximately 40%, with 64k page size on Arm64 architecture while running Spark TeraSort benchmark. Most modern Linux distributions support largemem kernels natively. We have not observed any issues while running Spark TeraSort benchmarks on largemem kernels. Performance Per Watt on AmpereOne® M Figure 6 To evaluate the energy efficiency of the cluster, we computed the Performance-per-Watt (Perf/Watt) ratio. This metric is derived by dividing the cluster's measured throughput (megabytes per second) by its total power consumption (watts) during the benchmarking interval. In these assessments, we observed AmpereOne® M performing 35% better over its predecessor on the Spark TeraSort benchmark. OS Metrics While Running TeraSort Benchmark Figure 7 The above image is a snapshot from the Grafana dashboard captured while running the TeraSort benchmark. During the HiBench test, the systems achieved maximum CPU utilization up to 90% while running the TeraSort benchmark. We observed disk read/write activity of approximately 15 GB/s and network throughput of 20 GB/s. Since both observed I/O and network throughput were significantly below the cluster's scalable limits, the results confirm that the benchmark successfully pushed the CPU to its maximum capacity. We observed from the above graphs that AmpereOne® M not only drove disk and network I/O higher than Ampere Altra – 128C, but it also completed tasks considerably faster. Power Consumption Figure 8 The graph illustrates the power consumption of cluster nodes, the platform, and the CPU. The power was measured using the IPMI tool during the benchmark run. We observe that the AmpereOne® M clusters consumed more power than the Ampere Altra – 128C cluster. This is not surprising in that the latest generation AmpereOne® M systems have 50% more compute cores and support 50% more memory channels. Additionally, as shown earlier, this increased power usage also delivered notably higher TeraSort throughput as well as better power efficiency (perf/watt) on AmpereOne® M (Figure 6). TPC-DS Performance Figures 9 and 10 The TPC-DS benchmarking tool was used to execute the TPC-DS workload on the clusters. The performance evaluation was based on the total time required to execute all 99 SQL queries on the cluster. Queries on AmpereOne® M completed in 50% less time than those run on Ampere Altra – 128C. The TPC-DS scalability improvement observed between 1 and 3 nodes was less compared to the scalability seen with TeraSort. 64k Page Size Figure 11 TPC-DS queries got a 9% boost by moving to a 64k page size kernel. Conclusion This paper presents a reference architecture for deploying Spark on a multi-node cluster powered by AmpereOne® M processors and compares the results with an earlier deployment based on Ampere Altra 128C processors. The latest TeraSort benchmark results reinforce the conclusions of earlier studies, demonstrating that Arm64-based data center processors provide a compelling, high-performance alternative to traditional x86 systems for Big Data workloads. Extending this analysis, the evaluation of the 12‑channel DDR5 AmpereOne® M platform shows measurable improvements in both raw throughput and performance-per-watt compared to previous-generation processors. These gains confirm that the AmpereOne® M is a groundbreaking platform designed for data centers and enterprises that prioritize performance, efficiency, and sustainability. Big Data workloads demand substantial computational resources and persistent storage, and by deploying these applications on Ampere processors, organizations benefit from both scale-up and scale-out architectures, enabling efficient growth while maintaining consistent throughput. For more information, visit our website at https://www.amperecomputing.com. If you’re interested in additional workload performance briefs, tuning guides, and more, please visit our Solutions Center at https://amperecomputing.com/solutions Appendix /etc/sysctl.conf Shell kernel.pid_max = 4194303 fs.aio-max-nr = 1048576 net.ipv4.conf.default.rp_filter=1 net.ipv4.tcp_timestamps=0 net.ipv4.tcp_sack = 1 net.core.netdev_max_backlog = 25000 net.core.rmem_max = 2147483647 net.core.wmem_max = 2147483647 net.core.rmem_default = 33554431 net.core.wmem_default = 33554432 net.core.optmem_max = 40960 net.ipv4.tcp_rmem =8192 33554432 2147483647 net.ipv4.tcp_wmem =8192 33554432 2147483647 net.ipv4.tcp_low_latency=1 net.ipv4.tcp_adv_win_scale=1 net.ipv6.conf.all.disable_ipv6 = 1 net.ipv6.conf.default.disable_ipv6 = 1 net.ipv4.conf.all.arp_filter=1 net.ipv4.tcp_retries2=5 net.ipv6.conf.lo.disable_ipv6 = 1 net.core.somaxconn = 65535 #memory cache settings vm.swappiness=1 vm.overcommit_memory=0 vm.dirty_background_ratio=2 /etc/security/limits.conf Shell * soft nofile 65536 * hard nofile 65536 * soft nproc 65536 * hard nproc 65536 Miscellaneous Kernel changes Shell #Disable Transparent Huge Page defrag echo never> /sys/kernel/mm/transparent_hugepage/defrag echo never > /sys/kernel/mm/transparent_hugepage/enabled #MTU 9000 for 100Gb Private interface and CPU governor on performance mode ifconfig enP6p1s0np0 mtu 9000 up cpupower frequency-set --governor performance .bashrc file Shell export JAVA_HOME=/home/hadoop/jdk export JRE_HOME=$JAVA_HOME/jre export CLASSPATH=.:$JAVA_HOME/lib:$JRE_HOME/lib:$classpath export PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin #HADOOP_HOME export HADOOP_HOME=/home/hadoop/hadoop export SPARK_HOME=/home/hadoop/spark export HADOOP_INSTALL=$HADOOP_HOME export HADOOP_HDFS_HOME=$HADOOP_HOME export YARN_HOME=$HADOOP_HOME export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH core-site.xml XML <configuration> <property> <name>fs.defaultFS</name> <value>hdfs://<server1>:9000</value> </property> <property> <name>hadoop.tmp.dir</name> <value>/data/data1/hadoop, /data/data2/hadoop, /data/data3/hadoop, /data/data4/hadoop </value> </property> <property> <name>io.native.lib.available</name> <value>true</value> </property> <property> <name>io.compression.codecs</name> <value>org.apache.hadoop.io.compress.GzipCodec, org.apache.hadoop.io.compress.DefaultCodec, org.apache.hadoop.io.compress.BZip2Codec, com.hadoop.compression.lzo.LzoCodec, com.hadoop.compression.lzo.LzopCodec, org.apache.hadoop.io.compress.SnappyCodec</value> </property> <property> <name>io.compression.codec.snappy.class</name> <value>org.apache.hadoop.io.compress.SnappyCodec</value> </property> </configuration> hdfs-site.xml XML configuration> <property> <name>dfs.replication</name> <value>1</value> </property> <property> <name>dfs.blocksize</name> <value>536870912</value> </property> <property> <name>dfs.namenode.name.dir</name> <value>file:/home/hadoop/hadoop_store/hdfs/namenode</value> </property> <property> <name>dfs.datanode.data.dir</name> <value>/data/data1/hadoop, /data/data2/hadoop, /data/data3/hadoop, /data/data4/hadoop </value> </property> <property> <name>dfs.client.read.shortcircuit</name> <value>true</value> </property> <property> <name>dfs.domain.socket.path</name> <value>/var/lib/hadoop-hdfs/dn_socket</value> </property> </configuration> yarn-site.xml XML <configuration>  <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> <property> <name>yarn.resourcemanager.hostname</name> <value><server1></value> </property> <property> <name>yarn.scheduler.minimum-allocation-mb</name> <value>1024</value> </property> <property> <name>yarn.scheduler.maximum-allocation-mb</name> <value>81920</value> </property> <property> <name>yarn.scheduler.minimum-allocation-vcores</name> <value>1</value> </property> <property> <name>yarn.scheduler.maximum-allocation-vcores</name> <value>186</value> </property> <property> <name>yarn.nodemanager.vmem-pmem-ratio</name> <value>4</value> </property> <property> <name>yarn.nodemanager.resource.memory-mb</name> <value>737280</value> </property> <property> <name>yarn.nodemanager.resource.cpu-vcores</name> <value>186</value> </property> <property> <name>yarn.log-aggregation-enable</name> <value>true</value> </property> </configuration> mapred-site.xml XML <configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> <property> <name>yarn.app.mapreduce.am.env</name> <value>HADOOP_MAPRED_HOME=$HADOOP_HOME</value> </property> <property> <name>mapreduce.map.env</name> <value>HADOOP_MAPRED_HOME=$HADOOP_HOME, LD_LIBRARY_PATH=$LD_LIBRARY_PATH </value> </property> <property> <name>mapreduce.reduce.env</name> <value>HADOOP_MAPRED_HOME=$HADOOP_HOME</value> </property> <property> <name>mapreduce.application.classpath</name> <value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*, $HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib-examples/*, $HADOOP_MAPRED_HOME/share/hadoop/mapreduce/sources/*, $HADOOP_MAPRED_HOME/share/hadoop/common/*, $HADOOP_MAPRED_HOME/share/hadoop/common/lib/*, $HADOOP_MAPRED_HOME/share/hadoop/yarn/*, $HADOOP_MAPRED_HOME/share/hadoop/yarn/lib/*, $HADOOP_MAPRED_HOME/share/hadoop/hdfs/*, $HADOOP_MAPRED_HOME/share/hadoop/hdfs/lib/*</value> </property> <property> <name>mapreduce.jobhistory.address</name> <value><server1>:10020</value> </property> <property> <name>mapreduce.jobhistory.webapp.address</name> <value><server1>:19888</value> </property> <property> <name>mapreduce.map.memory.mb</name> <value>2048</value> </property> <property> <name>mapreduce.map.cpu.vcore</name> <value>1</value> </property> <property> <name>mapreduce.reduce.memory.mb</name> <value>4096</value> </property> <property> <name>mapreduce.reduce.cpu.vcore</name> <value>1</value> </property> <property> <name>mapreduce.map.java.opts</name> <value> -Djava.net.preferIPv4Stack=true -Xmx2g -XX:+UseParallelGC -XX:ParallelGCThreads=32 -Xlog:gc*:stdout</value> </property> <property> <name>mapreduce.reduce.java.opts</name> <value> -Djava.net.preferIPv4Stack=true -Xmx3g -XX:+UseParallelGC -XX:ParallelGCThreads=32 -Xlog:gc*:stdout</value> </property> <property> <name>mapreduce.task.timeout</name> <value>6000000</value> </property> <property> <name>mapreduce.map.output.compress</name> <value>true</value> </property> <property> <name>mapreduce.map.output.compress.codec</name> <value>org.apache.hadoop.io.compress.SnappyCodec</value> </property> <property> <name>mapreduce.output.fileoutputformat.compress</name> <value>true</value> </property> <property> <name>mapreduce.output.fileoutputformat.compress.type</name> <value>BLOCK</value> </property> <property> <name>mapreduce.output.fileoutputformat.compress.codec</name> <value>org.apache.hadoop.io.compress.SnappyCodec</value> </property> <property> <name>mapreduce.reduce.shuffle.parallelcopies</name> <value>32</value> </property> <property> <name>mapred.reduce.parallel.copies</name> <value>32</value> </property> </configuration> spark-defaults.conf Shell spark.driver.memory 32g # used driver memory as 64g for TPC-DS spark.dynamicAllocation.enabled=false spark.executor.cores 5 spark.executor.extraJavaOptions=-Djava.net.preferIPv4Stack=true -XX:+UseParallelGC -XX:ParallelGCThreads=32 spark.executor.instances 108 spark.executor.memory 18g spark.executorEnv.MKL_NUM_THREADS=1 spark.executorEnv.OPENBLAS_NUM_THREADS=1 spark.files.maxPartitionBytes 128m spark.history.fs.logDirectory hdfs://<Master Server>:9000/logs spark.history.fs.update.interval 10s spark.history.provider org.apache.spark.deploy.history.FsHistoryProvider spark.history.ui.port 18080 spark.io.compression.codec=org.apache.spark.io.SnappyCompressionCodec spark.io.compression.snappy.blockSize=512k spark.kryoserializer.buffer 1024m spark.master yarn spark.master.ui.port 8080 spark.network.crypto.enabled=false spark.shuffle.compress true spark.shuffle.spill.compress true spark.sql.shuffle.partitions 12000 spark.ui.port 8080 spark.worker.ui.port 8081 spark.yarn.archive hdfs://<Master Server>:9000/spark-libs.jar spark.yarn.jars=/home/hadoop/spark/jars/*,/home/hadoop/spark/yarn/* hibench.conf Shell hibench.default.map/shuffle.parallelism 12000 # 3 node cluster hibench.scale.profile bigdata # the bigdata size configured as hibench.terasort.bigdata.datasize 30000000000 in ~/HiBench/conf/workloads/micro/terasort.conf Check out the full Ampere article collection here.

By RamaKrishna Nishtala

Hadoop on AmpereOne Reference Architecture

Ampere processors with Arm architecture deliver superior power efficiency and cost advantages compared to traditional x86 architecture. Hadoop, with its core components and broader ecosystem, is fully compatible with Arm-based platforms. Ampere Computing has previously published a comprehensive reference architecture demonstrating Hadoop deployments on Ampere® Altra® M processors. This paper builds on that foundation and extends the analysis by highlighting Hadoop performance on the next generation of AmpereOne® M processor. Scope and Audience The scope of this document includes setting up, tuning, and evaluating the performance of Hadoop on a testbed with AmpereOne® M processors. This document also compares the performance benefits of the 12-Channel AmpereOne® M processors with the previous generation of Ampere Altra — the 128C (Ampere Altra 128 Cores) processors. In addition, the document evaluates the use of 64k page-size kernels and highlights the resulting performance improvements compared to traditional 4KB page-size kernels. The document provides step-by-step guidance for installing and tuning Hadoop on single- and multi-node clusters. These recommendations serve as general guidelines for cluster configuration, and parameters can be further optimized for particular workloads and use cases. This document is intended for a diverse audience, including sales engineers, IT and cloud architects, IT and cloud managers, and customers seeking to leverage the performance and power efficiency benefits of Ampere Arm servers in their data centers. It aims to provide valuable insights and technical guidance to these professionals who are interested in implementing Arm-based Hadoop solutions and optimizing their infrastructure. AmpereOne® M Processors AmpereOne® M processor is part of the AmpereOne® family of high-performance server-class processors, engineered to deliver exceptional performance for AI Compute and a broad spectrum of mainstream data center workloads. Data-intensive applications such as Hadoop and Apache Spark benefit directly from the processor’s 12 DDR5 memory channels, which provide the bandwidth required for large-scale data processing. AmpereOne® M processors introduce a new platform architecture featuring higher core counts and additional memory channels, distinguishing it from Ampere’s previous platforms while preserving Ampere’s Cloud Native design principles. AmpereOne® M was designed from the ground up for cloud efficiency and predictable scaling. Each vCPU maps one-to-one with a physical core, ensuring consistent performance without resource contention. With up to 192 single-threaded cores and twelve DDR5 channels delivering 5600 MT/s, AmpereOne® M sustains the throughput required for demanding workloads, ranging from large language model (LLM) inference to real-time analytics. In addition, AmpereOne® M delivers exceptional performance-per-watt, reducing operational costs, energy consumption, and cooling requirements, making it well-suited for sustainable, high-density data center deployments. Hadoop on Ampere Processors There has been a significant shift towards the adoption of Arm-based processors in data centers over the past several years. Arm-based processors are increasingly used for distributed computing and offer compelling advantages for Hadoop deployments, a few of which are discussed in this paper. The Hadoop ecosystem is written in Java and runs seamlessly on Arm processors. Most of the Linux distributions, file systems, and open-source tools commonly used with Hadoop provide native Arm support. As a result, migrating existing Hadoop clusters (brownfield deployments) or deploying new clusters (greenfield deployments) on Arm-based infrastructure can be accomplished with little to no disruption. Running Hadoop’s distributed processing framework on energy-efficient Ampere processors represents an important evolution in big data infrastructure. This approach enables more sustainable, power-efficient, and cost-effective Hadoop deployments while maintaining performance and scalability Big Data Architecture The scale, complexity, and unstructured nature of modern data generation exceed the capabilities of traditional software systems. Big data applications are purpose-built to manage and analyze these complex datasets. Big data is defined not only by Volume but also by the Velocity at which the data is generated and processed, the variety of formats it spans (from structured numerical data to unstructured text, images, and video), and its Veracity (the quality and accuracy of the data) and the Value it delivers. Together, these characteristics create both significant challenges and unprecedented opportunities for insight and innovation. Big data includes structured, semi-structured, and unstructured data that is analyzed using advanced analytics. Typical big data deployments operate at a terabyte and petabyte scale, with data continuously created and collected over time. The big data domain includes data ingestion, processing, and analysis of datasets that are too large, fast-moving, or complex for traditional data processing systems. Sources of big data are limitless, and include Internet of Things (IoT) sensors, social media activity, e-commerce transactions, satellite imagery, scientific instruments, web logs, and more. The true power of big data is realized by extracting meaningful insights from this diverse and often unstructured information. By applying advanced analytics like artificial intelligence (AI) and machine learning (ML), organizations can predict trends, gain a deeper understanding of customer behavior and market dynamics, and identify operational inefficiencies at scale. Big data solutions involve the following types of workloads: Batch processing of big data sources at restReal-time processing of big data in motionInteractive exploration of big dataPredictive analytics and machine learning Hadoop Ecosystem The Apache Hadoop software library facilitates scalable, fault-tolerant, distributed computing by providing a framework for processing large volumes of data across commodity hardware clusters. Designed to scale from single-node deployments to thousands of machines, Hadoop distributes both storage through Hadoop Distributed File System (HDFS) and computation via MapReduce and YARN. Hadoop incorporates built-in fault tolerance to handle common node failures in large clusters. Through resilient software techniques such as data replication, the platform maintains high availability and ensures continuous data processing, even during infrastructure failure. By leveraging distributed computing and a resilient data management framework, Hadoop enables efficient processing and analysis of massive datasets. The platform supports a wide spectrum of data-intensive workloads, including data analytics, data mining, and machine learning, providing organizations with the required scalability, reliability, and performance required for complex data processing at scale. The four main elements of the ecosystem are Hadoop Distributed File System (HDFS), MapReduce, Yet Another Resource Negotiator (YARN), and Hadoop Common. Hadoop Distributed File System (HDFS) As the primary storage layer of Hadoop, HDFS manages datasets across distributed nodes. Its architecture ensures high scalability and fault tolerance through data replication and redundancy. HDFS divides data into fixed-size blocks and distributes them across the cluster, optimizing the system for parallel processing and high-throughput data access. MapReduce MapReduce is a programming model and processing framework for distributed data processing within the Hadoop ecosystem. It enables parallel execution by dividing workloads into smaller tasks that are distributed across cluster nodes. The Map phase processes data in parallel, and the Reduce phase aggregates and summarizes the results. MapReduce is commonly used for batch processing and large-scale data analytics workloads. Yet Another Resource Negotiator (YARN) YARN is a cluster of resource management software within the Hadoop ecosystem. It is responsible for resource allocation, scheduling, and workload coordination across the cluster. YARN enables multiple processing frameworks, such as MapReduce, Apache Spark, and Apache Flink, to run concurrently on the same infrastructure, allowing diverse workloads to efficiently share cluster resources. Hadoop Common Hadoop Common is a foundational component of the Hadoop ecosystem, providing shared libraries and utilities for all Hadoop modules to operate. It delivers core services including authentication, security protocols, and file system interfaces, ensuring consistency and interoperability across the ecosystem’s components. Hadoop Common has officially supported ARM-based platforms since version 3.3.0, including native libraries optimized for the Arm architecture. This support enables seamless deployment and operation of Hadoop on modern Arm-based infrastructure. Figure 1 Hadoop Test Bed A 3-node cluster was set up for performance benchmarking. The cluster was set up with AmpereOne® M processors. Equipment Under Test Cluster Nodes: 3CPU: AmpereOne® MSockets/Node: 1Cores/Socket: 192Threads/Socket: 192CPU Speed: 3200 MHzMemory Channels: 12Memory/Node: 768GBNetwork Card/Node: 1 x Mellanox ConnectX-6Storage/Node: 4 x Micron 7450 Gen 4 NVMEKernel Version: 6.8.0-85: Ubuntu 24.04.3Hadoop Version: 3.3.6JDK Version: JDK 11 Hadoop Installation and Cluster Setup OS Install. The majority of modern open-source and enterprise-supported Linux distributions offer full support for the AArch64 architecture. To install your chosen operating system, use the server for a Kernel-based virtual machine (KVM) console to map or attach the OS installation media, and then follow the standard installation procedure. Networking Setup. Set up a public network on one of the available interfaces for client communication. This can be used to log in to any of the servers where client communication is needed. Set up a private network for communication between the cluster nodes. Storage Setup. Choose a drive of your choice for OS installation, clear any old partitions, reformat, and choose the disk to install the OS. A Samsung 960 GB drive (M.2) was chosen for the OS installation in this setup. Add additional high-speed NVMe drives for the HDFS file system. Create Hadoop. User: Create a user named “hadoop” as part of the OS Installation and provide necessary sudo privileges for the user. Post-Install Steps: Perform the following post-installation steps on all the nodes after the OS installation. yum or apt update on the nodes.Install packages like dstat, net-tools, nvme-cli, lm-sensors, linux-tools-generic, python, and sysstat for your monitoring needs.Set up ssh trust between all the nodes.Update /etc/sudoers file for nopasswd for hadoop user.Update /etc/security/limits.conf per Appendix.Update /etc/sysctl.conf per Appendix.Update the scaling governor to performance and disable transparent hugepages per the Appendix.If necessary, make changes to /etc/rc.d to keep the above changes permanent after every reboot.Set up NVMe disks as an XFS file system for HDFS. Zap and format the NVME disks.Create a single partition on each of the nvme disks with fdisk or parted.Create file system on each of the created partitions as mkfs.xfs -f /dev/nvme[0-n]n1p1.Create directories for mounting on root.mkdir -p /root/nvme[0-n]1p1.Update /etc/fstab with entries to mount the file system. The UUID of each partition for update in fstab can be extracted from the blkid command.Change ownership of these directories to the ‘hadoop’ user created earlier. Hadoop Install Download Hadoop 3.3.6 from the Apache website and JDK11 for Arm/Aarch64. Extract the tarballs under the Hadoop home directory. Update the Hadoop configuration files in ~/hadoop/etc/hadoop/ and the environment parameters in .bashrc per the Appendix. Depending on the hardware specifications of cores, memory, and disk capacities, these parameters may have to be altered. Update the workers' file to include the set of data nodes. Run the following commands Shell hdfs namenode -format scp -r ~/hadoop <datanodes>:~/hadoop ~/hadoop/sbin/start-all.sh This should start with the NodeManager, ResourceManager, NameNode, and DataNode processes on the nodes. Please note that NameNode and Resource Managers are started only on the master node. Verification of the setup: Run the jps command on each node to check the status of the Hadoop daemons.Verify that -ls, -put, -du, -mkdir commands can be run on the cluster. Performance Tuning Hadoop is a complex framework where many components interact across multiple systems. Overall performance is influenced by several distinct factors: Platform settings: This includes configurations at the hardware and operating system levels, such as BIOS settings, specific OS parameters, and the performance of network and disk subsystems.Hadoop configuration: The configuration of the Hadoop software stack itself also plays a critical role in efficiency. Optimizing these settings typically requires prior experience with Hadoop. It is important to approach performance tuning as an iterative process. It is important to note that performance tuning is an iterative process, and the parameters provided in the Appendix are merely reference values obtained through a few iterations. Linux: Occasionally, conflicts between different subcomponents of a Linux system, such as the networking and disk subsystems, can arise and negatively impact overall performance. The primary objective is to optimize the entire system to achieve optimal disk and network throughput by identifying and resolving any bottlenecks that may emerge during operation. Network: To evaluate the underlying network infrastructure, the iperf utility can be used to conduct stress tests. Performance optimization involves adjusting specific driver parameters, such as the Transmit (TX) and Receive (RX) ring buffers and the number of interrupt queues, to align them with the CPU cores on the Non-Uniform Memory Access (NUMA) node where the Network Interface Card (NIC) resides. However, if the system's BIOS is already configured in monolithic mode, these specific kernel-level modifications related to NUMA alignment may not be necessary. Disks: When optimizing performance in a Hadoop environment, administrators should focus on specific disk subsystem parameters: Aligned partitions: Partitions should be aligned with the storage's physical block boundaries to maximize I/O efficiency. Utilities like parted can be used to create aligned partitions. I/O queue settings: Parameters such as the queue depth and nr_requests (number of requests) can be fine-tuned via the /sys/block//queue/ directory paths to control how many I/O operations the kernel schedules for a storage device. Filesystem mount options: Utilizing the noatime option in the /etc/fstab file is critical for Hadoop, as it prevents unnecessary disk writes by disabling the recording of file access timestamps. The fio (flexible I/O tester) tool is highly effective for benchmarking and validating the performance of the disk subsystem after these changes are implemented. HDFS, YARN, and MapReduce HDFS In HDFS, the primary parameters to consider for data management and resilience are the block size and replication factor. By default, the HDFS block size is 128 MB. Files are divided into chunks matching this size, which are then distributed across different data nodes. In certain high-performance environments or test beds, a larger block size, such as 512 MB, might be used to optimize throughput for large files. The test bed with the AmpereOne® M processor was also set up with 512MB. The replication factor (defaulting to 3) determines data redundancy. When an application writes data once, HDFS replicates those blocks across the cluster based on this factor, ensuring three identical copies are available for high availability and fault tolerance. Consequently, the total storage space required is directly proportional to the replication factor used (a factor of 3 means you need 3x the raw data size in storage capacity). HDFS 3.x introduced Erasure Coding (EC) as an alternative to traditional replication. EC significantly reduces storage overhead; for example, a 6+3 EC configuration provides data redundancy comparable to a 3x replication factor but uses substantially less physical storage space. It is important to note, however, that while EC saves storage, it introduces additional computational and network load compared to simple replication. In the described test bed environment, a standard replication factor of 1 was employed YARN YARN (Yet Another Resource Negotiator) is the resource management framework within the Hadoop ecosystem. It offers two main scheduler options: the Fair scheduler and the Capacity scheduler. The Fair scheduler (the default configuration) distributes available cluster resources evenly and dynamically among all running applications or jobs over time. The Capacity scheduler allocates a guaranteed, fixed capacity to each queue, user, or job. By default, the behavior of standard configurations is that if a queue does not fully utilize its reserved capacity, that excess may remain unused or might be conditionally shared depending on specific configuration parameters. Key configuration settings for either scheduler involve defining the limits for resource allocation, specifically the minimum allocation, maximum allocation, and incremental "stepping" values for both memory and virtual CPU cores (vcores). We used the default configuration in the testing environment. MapReduce In the MapReduce framework, a job is broken down into numerous smaller tasks, where each task is designed to have a smaller memory footprint and leverage a single or fewer virtual cores (vcores). Resource allocation within YARN is determined by these task requirements, considering the total memory available to the YARN Node Manager and the total number of vcores it manages. These configurations can be directly adjusted within the yarn-site.xml file. Reference parameters used in a specific test bed are often provided in an Appendix for guidance. Benchmark Tools We used the HiBench benchmarking tool. HiBench is a popular benchmarking suite specifically designed for evaluating the performance of Big Data frameworks, such as Apache Hadoop and Apache Spark. It consists of a set of workload-specific benchmarks that simulate real-world Big Data processing scenarios. For additional information, you can refer to this link. By running HiBench on the cluster, you can assess and compare its performance in handling various Big Data workloads. The benchmark results can provide insights into factors such as data processing speed, scalability, and resource utilization for each cluster. Steps to run HiBench on the cluster: Download HiBench software from the link above.Update hibench.conf file, like scale, profile, parallelism parameters, and a list of master and slave nodes.Run ~HiBench/bin/workloads/micro/terasort/prepare/prepare.sh.Run ~HiBench/bin/workloads/micro/terasort/Hadoop/run.sh. The above will generate a hibench.report file under the report directory. Further, a bench.log file provides details of the run. The cluster was using a data set of 3 TB. We measured the total power consumed, CPU power, CPU utilization, and other parameters like disk and network utilization using Grafana and IPMI tools. Throughput from the HiBench run was calculated for TeraSort in the following scenarios: Hadoop running on a single node on AmpereOne® M to compare with the previous generation of Ampere Altra – 128c.Hadoop running on a single node on AmpereOne® M to compare with a 3-node cluster of AmpereOne® M to measure the scalability.Hadoop running on a 3-node cluster with 64k page size on AmpereOne® M to compare it with 4k page size on the same processor. Performance Tests on AmpereOne® M Cluster TeraSort Performance Figures 2 and 3 Using the Hibench tool as mentioned above, we ran Hadoop TeraSort tests on one, two, and three nodes with AmpereOne® M processors and compared the values we got earlier on Ampere Altra – 128C. From Figure 2, it is evident that there is a 40% benefit of AmpereOne® M over Ampere Altra – 128C while running Hadoop TeraSort. This increase in performance can be attributed to a newer microarchitecture design, an increase in core count (from 128 to 192), and the 12-channel DDR5 design on AmpereOne® M. Near-linear scalability was observed when running TeraSort. The output for the 3x nodes configuration was found to be very close to three times the output of a single node. 64k Page Size Figure 4 We observed a significant performance increase, approximately 30%, with 64k page size on the Arm architecture while running the Hadoop TeraSort benchmark. Most modern Linux distributions, support largemem kernels natively. For other systems, building a custom 64k page size kernel is a straightforward procedure that can be implemented with a standard reboot. We have not observed any issues while running Hadoop TeraSort benchmarks on largemem kernels. Performance per Watt on AmpereOne® M Figure 5 To evaluate the energy efficiency of the cluster, we computed the Performance-per-Watt (Perf/Watt) ratio. This metric is derived by dividing the cluster's measured throughput (megabytes per second) by its total power consumption (watts) during the benchmarking interval. In these assessments, we observed AmpereOne® M performing 30% better over its predecessor on the Hadoop TeraSort benchmark. OS Metrics While Running the Benchmark Figure 6 The above image is a snapshot from the Grafana dashboard captured while running the benchmark. The systems achieved maximum CPU utilization while running the TeraSort benchmark using HiBench. We observed disk read/write activity of approximately 10 GB/s and network throughput of 30 GB/s. Since both observed I/O and network throughput were significantly below the cluster's scalable limits, the results confirm that the benchmark successfully pushed the CPUs to their maximum capacity. We observed from the above graphs that AmpereOne® M not only drove disk and network I/O higher than Ampere Altra – 128C, but also completed tasks considerably faster Power Consumption Figure 7 The graph illustrates the power consumption of cluster nodes, the platform, and the CPU. The power was measured by the IPMI tool during the benchmark run. The data reveals that the AmpereOne® M cluster consumed more absolute power than the Ampere Altra – 128C. However, this increased power usage correlated with a higher TeraSort throughput on the AmpereOne® M system. AmpereOne® M cluster delivers a better performance per watt (Figure 5). Conclusions This paper presents a reference architecture for deploying Hadoop on a multi-node cluster powered by AmpereOne® M processors and compares the results against a prior deployment on Ampere Altra – 128C processors. The latest TeraSort benchmark results validate the findings of earlier studies, demonstrating that Arm-based processors provide a compelling, high-performance alternative to traditional x86 systems for big-data workloads. Building on this foundation, the evaluation of the 12‑channel DDR5 AmpereOne® M platform shows measurable improvements not only in raw throughput but also in performance-per-watt compared to previous generation processors. The improvements confirm that the AmpereOne® M is a purpose-built platform designed for modern data centers and enterprises that prioritize both performance and energy efficiency. AmpereOne® M addresses the core requirements of today’s organizations: performance, efficiency, and scalability. Big Data workloads demand significant compute capacity and persistent storage, and by deploying these applications on Ampere processors, organizations benefit from both scale-up and scale-out architectures. This approach enables a higher density per rack, reduces power consumption, and delivers consistent throughput at scale. To learn more about our developer efforts and find best practices, visit Ampere’s Developer Center and join the conversation in the Ampere Developer Community. Appendix /etc/sysctl.conf Shell kernel.pid_max = 4194303 fs.aio-max-nr = 1048576 net.ipv4.conf.default.rp_filter=1 net.ipv4.tcp_timestamps=0 net.ipv4.tcp_sack = 1 net.core.netdev_max_backlog = 25000 net.core.rmem_max = 2147483647 net.core.wmem_max = 2147483647 net.core.rmem_default = 33554431 net.core.wmem_default = 33554432 net.core.optmem_max = 40960 net.ipv4.tcp_rmem =8192 33554432 2147483647 net.ipv4.tcp_wmem =8192 33554432 2147483647 net.ipv4.tcp_low_latency=1 net.ipv4.tcp_adv_win_scale=1 net.ipv6.conf.all.disable_ipv6 = 1 net.ipv6.conf.default.disable_ipv6 = 1 net.ipv4.conf.all.arp_filter=1 net.ipv4.tcp_retries2=5 net.ipv6.conf.lo.disable_ipv6 = 1 net.core.somaxconn = 65535 #memory cache settings vm.swappiness=1 vm.overcommit_memory=0 vm.dirty_background_ratio=2 /etc/security/limits.conf Shell * soft nofile 65536 * hard nofile 65536 * soft nproc 65536 * hard nproc 65536 Miscellaneous Kernel changes Shell #Disable Transparent Huge Page defrag echo never> /sys/kernel/mm/transparent_hugepage/defrag echo never > /sys/kernel/mm/transparent_hugepage/enabled #MTU 9000 for 100Gb Private interface and CPU governor on performance mode ifconfig enP6p1s0np0 mtu 9000 up cpupower frequency-set --governor performance .bashrc file Shell export JAVA_HOME=/home/hadoop/jdk export JRE_HOME=$JAVA_HOME/jre export CLASSPATH=.:$JAVA_HOME/lib:$JRE_HOME/lib:$classpath export PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin #HADOOP_HOME export HADOOP_HOME=/home/hadoop/hadoop export HADOOP_INSTALL=$HADOOP_HOME export HADOOP_MAPRED_HOME=$HADOOP_HOME export HADOOP_COMMON_HOME=$HADOOP_HOME export HADOOP_HDFS_HOME=$HADOOP_HOME export YARN_HOME=$HADOOP_HOME export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH export PATH=$PATH:/home/hadoop/.local/bin core-site.xml XML <configuration> <property> <name>fs.defaultFS</name> <value>hdfs://<server1>:9000</value> </property> <property> <name>hadoop.tmp.dir</name> <value>/data/data1/hadoop, /data/data2/hadoop, /data/data3/hadoop, /data/data4/hadoop </value> </property> <property> <name>io.native.lib.available</name> <value>true</value> </property> <property> <name>io.compression.codecs</name> <value>org.apache.hadoop.io.compress.GzipCodec, org.apache.hadoop.io.compress.DefaultCodec, org.apache.hadoop.io.compress.BZip2Codec, com.hadoop.compression.lzo.LzoCodec, com.hadoop.compression.lzo.LzopCodec, org.apache.hadoop.io.compress.SnappyCodec</value> </property> <property> <name>io.compression.codec.snappy.class</name> <value>org.apache.hadoop.io.compress.SnappyCodec</value> </property> </configuration> hdfs-site.xml XML <configuration> <property> <name>dfs.replication</name> <value>1</value> </property> <property> <name>dfs.blocksize</name> <value>536870912</value> </property> <property> <name>dfs.namenode.name.dir</name> <value>file:/home/hadoop/hadoop_store/hdfs/namenode</value> </property> <property> <name>dfs.datanode.data.dir</name> <value>/data/data1/hadoop, /data/data2/hadoop, /data/data3/hadoop, /data/data4/hadoop </value> </property> <property> <name>dfs.client.read.shortcircuit</name> <value>true</value> </property> <property> <name>dfs.domain.socket.path</name> <value>/var/lib/hadoop-hdfs/dn_socket</value> </property> </configuration> yarn-site.xml XML <configuration>  <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> <property> <name>yarn.resourcemanager.hostname</name> <value><server1></value> </property> <property> <name>yarn.scheduler.minimum-allocation-mb</name> <value>1024</value> </property> <property> <name>yarn.scheduler.maximum-allocation-mb</name> <value>81920</value> </property> <property> <name>yarn.scheduler.minimum-allocation-vcores</name> <value>1</value> </property> <property> <name>yarn.scheduler.maximum-allocation-vcores</name> <value>186</value> </property> <property> <name>yarn.nodemanager.vmem-pmem-ratio</name> <value>4</value> </property> <property> <name>yarn.nodemanager.resource.memory-mb</name> <value>737280</value> </property> <property> <name>yarn.nodemanager.resource.cpu-vcores</name> <value>186</value> </property> <property> <name>yarn.log-aggregation-enable</name> <value>true</value> </property> </configuration> mapred-site.xml XML <configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> <property> <name>yarn.app.mapreduce.am.env</name> <value>HADOOP_MAPRED_HOME=$HADOOP_HOME</value> </property> <property> <name>mapreduce.map.env</name> <value>HADOOP_MAPRED_HOME=$HADOOP_HOME, LD_LIBRARY_PATH=$LD_LIBRARY_PATH </value> </property> <property> <name>mapreduce.reduce.env</name> <value>HADOOP_MAPRED_HOME=$HADOOP_HOME</value> </property> <property> <name>mapreduce.application.classpath</name> <value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*, $HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib-examples/*, $HADOOP_MAPRED_HOME/share/hadoop/mapreduce/sources/*, $HADOOP_MAPRED_HOME/share/hadoop/common/*, $HADOOP_MAPRED_HOME/share/hadoop/common/lib/*, $HADOOP_MAPRED_HOME/share/hadoop/yarn/*, $HADOOP_MAPRED_HOME/share/hadoop/yarn/lib/*, $HADOOP_MAPRED_HOME/share/hadoop/hdfs/*, $HADOOP_MAPRED_HOME/share/hadoop/hdfs/lib/*</value> </property> <property> <name>mapreduce.jobhistory.address</name> <value><server1>:10020</value> </property> <property> <name>mapreduce.jobhistory.webapp.address</name> <value><server1>:19888</value> </property> <property> <name>mapreduce.map.memory.mb</name> <value>2048</value> </property> <property> <name>mapreduce.map.cpu.vcore</name> <value>1</value> </property> <property> <name>mapreduce.reduce.memory.mb</name> <value>4096</value> </property> <property> <name>mapreduce.reduce.cpu.vcore</name> <value>1</value> </property> <property> <name>mapreduce.map.java.opts</name> <value> -Djava.net.preferIPv4Stack=true -Xmx2g -XX:+UseParallelGC -XX:ParallelGCThreads=32 -Xlog:gc*:stdout</value> </property> <property> <name>mapreduce.reduce.java.opts</name> <value> -Djava.net.preferIPv4Stack=true -Xmx3g -XX:+UseParallelGC -XX:ParallelGCThreads=32 -Xlog:gc*:stdout</value> </property> <property> <name>mapreduce.task.timeout</name> <value>6000000</value> </property> <property> <name>mapreduce.map.output.compress</name> <value>true</value> </property> <property> <name>mapreduce.map.output.compress.codec</name> <value>org.apache.hadoop.io.compress.SnappyCodec</value> </property> <property> <name>mapreduce.output.fileoutputformat.compress</name> <value>true</value> </property> <property> <name>mapreduce.output.fileoutputformat.compress.type</name> <value>BLOCK</value> </property> <property> <name>mapreduce.output.fileoutputformat.compress.codec</name> <value>org.apache.hadoop.io.compress.SnappyCodec</value> </property> <property> <name>mapreduce.reduce.shuffle.parallelcopies</name> <value>32</value> </property> <property> <name>mapred.reduce.parallel.copies</name> <value>32</value> </property> </configuration> Check out the full Ampere article collection here.

By RamaKrishna Nishtala

Stop Leap-Second AI Drift in IoT Streams With PySpark

Fintech and Enterprise platforms ingest massive volumes of timestamped data (big data) from IoT devices such as payment terminals, wearables, and mobile apps. Accurate timing is essential for fraud detection, risk scoring, and customer analytics. Yet a subtle irregularity called the leap second can corrupt timestamps and trigger AI drift, gradually degrading model performance in production. In this article, I will attempt to explain clearly what drift types are and how they can be prevented, based on my research paper. Details can be found here. Let's start. What Is AI Drift? AI drift (also known as model drift) occurs when a deployed machine learning model loses accuracy because live data no longer matches the training data distribution. In fintech IoT pipelines, this leads to more false-positive fraud alerts, inaccurate risk scores, and lost revenue. Four key types of drift are relevant: 1. Data Drift (Covariate Shift) The statistical distribution of input features changes while the relationship to the target stays the same. Fintech example: A fraud model trained on average transaction amounts of $50–$200 suddenly sees many $1–$10 micro-payments from new IoT wearables. The feature distribution shifts, causing excessive false positives. 2. Concept Drift The underlying relationship between inputs and the target evolves. Fintech example: Fraudsters switch from large one-time charges to repeated small "card-testing" transactions across IoT devices. The model’s learned fraud patterns become outdated. 3. Label Drift (Prior Probability Shift) The overall proportion of target classes changes. Fintech example: During economic stability, the fraud rate drops from 2% to 0.2%. A model calibrated on the old rate over-predicts fraud and floods teams with alerts. 4. Temporal Drift Timestamp inconsistencies corrupt time-based features (often grouped under data drift). Fintech example: Leap seconds create duplicate timestamps or negative deltas. Features such as "seconds since last transaction" or velocity checks break, distorting every downstream score. These drift types frequently compound. Temporal drift from leap seconds can cascade into data, concept, or label drift if timestamps are not cleaned in real time. Verified Historical Leap-Second Incidents 2012 (June 30): Reddit, LinkedIn, and other major services suffered outages. A Linux kernel timing bug caused 100% CPU spikes and lockups when the extra second was inserted.2015 (June 30): Major exchanges took precautionary measures. The Intercontinental Exchange (ICE), which operates NYSE platforms, paused certain operations for 61 minutes, and other venues shortened after-hours sessions to avoid timestamp-related failures.2017 (January 1): Cloudflare experienced a partial global DNS outage. A negative time delta in their Go-based resolver caused a random-number generator to panic and crash. These documented events show why real-time leap-second handling is essential in financial systems. The Solution: PySpark Structured Streaming Pipeline The framework published in the original research uses Apache Spark Structured Streaming to detect and correct leap-second anomalies in real time, enforce temporal order, and deliver clean monotonic timestamps to AI/ML pipelines. Figure 1. PySpark Leap-Second Data Processing Pipeline Complete PySpark Implementation Python from pyspark.sql import SparkSession from pyspark.sql.functions import * from pyspark.sql.types import StructType, StructField, StringType, DoubleType from pyspark.sql.window import Window import os # Initialize Spark session spark = SparkSession.builder \ .appName("LeapSecondsStreaming") \ .master("local[*]") \ .getOrCreate() spark.sparkContext.setLogLevel("ERROR") # Create input directory for streaming files input_dir = "input_data" os.makedirs(input_dir, exist_ok=True) # Define input schema schema = StructType([ StructField("transaction_id", StringType(), True), StructField("timestamp", StringType(), True), # format: yyyy-MM-dd HH:mm:ss StructField("amount", DoubleType(), True) ]) # Read streaming CSV data (easily replaceable with Kafka) raw_df = spark.readStream \ .schema(schema) \ .option("header", True) \ .option("maxFilesPerTrigger", 1) \ .csv(input_dir) # Leap-Second Cleaning cleaned_df = raw_df.withColumn( "cleaned_ts", regexp_replace(col("timestamp"), r":60$", ":59") ) # Parse timestamps and convert to Unix epoch parsed_df = cleaned_df \ .withColumn("event_time", to_timestamp(col("cleaned_ts"), "yyyy-MM-dd HH:mm:ss")) \ .withColumn("unix_ts", unix_timestamp(col("event_time"))) \ .filter(col("event_time").isNotNull()) # Real-time Temporal Validation in Micro-batches def process_batch(df, epoch_id): print(f"\n=== Processing micro-batch {epoch_id} ===") window_spec = Window.orderBy("event_time") df = df.withColumn("prev_unix_ts", lag("unix_ts").over(window_spec)) \ .withColumn("time_diff_sec", col("unix_ts") - col("prev_unix_ts")) \ .withColumn("anomaly_flag", when((col("time_diff_sec") == 0) | (col("time_diff_sec") > 2) | col("time_diff_sec").isNull(), "LEAP_SECOND_OR_GAP") .otherwise("OK")) df.select("transaction_id", "event_time", "amount", "time_diff_sec", "anomaly_flag") \ .orderBy("event_time") \ .show(truncate=False) # Write cleaned data to Delta Lake / feature store here # Start the streaming query query = parsed_df.writeStream \ .foreachBatch(process_batch) \ .outputMode("append") \ .option("checkpointLocation", "checkpoint_leapsecond") \ .start() query.awaitTermination() PySpark Logic Explanation The PySpark Structured Streaming pipeline processes fintech IoT data in real time by first initializing a SparkSession and reading incoming CSV files (or Kafka topics) as a continuous streaming DataFrame using a predefined schema for transaction_id, timestamp, and amount. The leap-second correction is applied immediately via regexp_replace to convert any invalid :60 second to :59, followed by to_timestamp parsing and conversion to Unix epoch seconds (unix_timestamp) for numerical stability. In every micro-batch processed by foreachBatch, a window function ordered by event_time computes the previous timestamp using lag and derives the time_diff_sec; any zero-difference, null, or excessively large gap is flagged as a leap-second anomaly with an anomaly_flag column. Cleaned, monotonic timestamps and validated time differences are then passed downstream for aggregations and feature engineering, ensuring temporal consistency before data reaches AI/ML pipelines. Why This Matters for All Types of Drift The leap-second cleaning and temporal validation steps directly eliminate temporal drift — the root cause in fintech IoT streams. By making timestamps monotonic and gap-free, the pipeline ensures that all derived features (time deltas, velocity checks, rolling windows, and event ordering) remain accurate. This single fix prevents temporal drift from cascading into the other 3 drift types. This pipeline achieved 100% detection and correction of injected leap-second anomalies in the paper’s controlled experiment (1,000 synthetic transactions with 10 anomalies) at an average batch latency of only 0.8 seconds. How the Pipeline Prevents All Types of Drift Data drift is eliminated by producing consistent Unix epoch timestamps and valid time deltas.Concept drift is avoided because accurate temporal sequences preserve true fraud and risk patterns.Label drift is controlled by reliable time windows that do not artificially inflate or deflate class balances. Figure 2. Types of AI/ML Drift Figure 3. Complete Layered PySpark Architecture – How Leap-Second Anomalies Are Detected and Fixed Conclusion Handling temporal anomalies such as leap seconds is often overlooked in large-scale data systems, yet it plays a critical role in ensuring the reliability of time-sensitive applications, especially in fintech and IoT environments. By leveraging PySpark and designing resilient data pipelines, organizations can proactively mitigate AI drift and maintain the integrity of predictive models operating at scale. As real-world data continues to grow in complexity, engineering systems that are both time-aware and fault-tolerant become essential. The approaches discussed here provide a foundation for building robust, production-grade data processing systems that can handle such edge cases effectively. References [1] Ram Ghadiyaram, Durga Krishnamoorthy, Vamshidhar Morusu, Jaya Eripilla, "Addressing AI Drift in Fintech IoT Data Processing: Handling Leap Seconds with PySpark for Robust Predictive Analytics," International Journal of Computer Trends and Technology, vol. 73, no. 5, 2025. https://doi.org/10.14445/22312803/IJCTT-V73I5P101

By Ram Ghadiyaram

CORE

How Piezoelectric Energy Harvesting Is Solving the Battery Waste Crisis in Industrial IoT

High-temperature energy harvesting exposes the hidden cost of batteries across Industrial Internet of Things (IIoT) deployments, especially in environments where heat and access constraints shorten battery life and raise maintenance risk. Fit-and-forget architectures matter in hazardous and remote locations. Battery replacement introduces downtime and unpredictable operating costs that scale with fleet size, while thermal extremes further reduce cell reliability. Energy harvesting and self-powered sensors emerge as engineering-driven solutions that align with long-term system availability and life-cycle performance. Battery-less IIoT designs become a practical response to operational constraints rather than a sustainability narrative. The Battery Waste Crisis in Industrial IoT Large IIoT sensor fleets rely on millions of batteries, and that dependence scales linearly with network growth across industrial sites. As deployments expand, the global volume of battery materials available for recycling can reach 1.4 million metric tons by 2030. This surge emphasizes the downstream impact of short-lived power sources. Battery replacement cycles introduce maintenance risk and direct labor exposure, particularly in hazardous or hard-to-access environments. Regulatory requirements and environmental scrutiny now add further pressure, pushing industrial operators to reassess battery-heavy architectures as a structural liability rather than a routine operating expense. Why Batteries Are a System Design Bottleneck Batteries face clear reliability limits under vibration, heat, and chemical exposure — all of which are common in industrial operating environments. Lithium-ion cells remain the most common battery type. Yet, less than 1% of lithium is recycled at the end-of-life stage, turning each replacement cycle into a cost and disposal liability. Across multi-year deployments, these constraints drive up the total cost of ownership through safety procedures and ongoing inventory management. Battery dependence also shapes firmware behavior and network design, often forcing aggressive power optimization that reduces data resolution and system responsiveness. In response, many IIoT architects now consider industrial piezoelectric sensors to eliminate batteries and decouple long-term reliability from consumable power sources. High-Temperature Energy Harvesting as a Practical Alternative High-temperature energy harvesting broadens the range of viable power strategies in industrial environments, especially where heat and mechanical stress limit battery performance. Industrial deployments typically evaluate vibration, thermal, and solar energy harvesting sources, each with different power densities and integration requirements. Solar harvesting depends heavily on external conditions and infrastructure, reducing reliability inside enclosed facilities or hazardous zones. Mechanical energy — generated by rotating machinery and structural vibration — offers consistent availability across operating cycles. This characteristic makes it well-suited for long-lived industrial sensor networks. Consistency simplifies power budgeting for embedded systems and reduces reliance on oversized energy storage. In high-temperature settings, mechanically driven harvesting remains stable where thermal gradients or photovoltaic inputs fluctuate. How Piezoelectric Energy Harvesting Works Piezoelectric materials generate an electrical charge when mechanical strain deforms their internal crystal structure. This effect allows vibration and structural motion to convert directly into usable electrical energy. Some high-temperature piezoceramics can withstand temperatures up to 350°C, enabling stable actuation, sensing, and harvesting in environments where conventional materials and batteries fail. The resulting power output is typically low and aligns well with duty-cycled operation and event-driven data transmission in battery-less IIoT systems. In embedded systems design, output characteristics depend on vibration frequency and electrical loading, which influence power conditioning and firmware timing decisions. Designing Self-Powered Industrial Piezoelectric Sensors Effective system design begins with matching harvested energy availability to sensing and transmission loads so that each operation fits within a predictable power budget. Ultra-low-power microcontroller unit selection and firmware strategies become critical, with event-driven execution and adaptive sampling used to operate within tight energy margins. Power management circuits and cold-start behavior must work together to guarantee reliable startup after extended idle periods. These constraints directly shape how industrial piezoelectric sensors integrate into battery-less architectures that favor longevity and maintenance-free operation over continuous data streaming. Integration Challenges Engineers Must Address High-temperature energy harvesting introduces integration challenges driven by wide variability in vibration profiles across equipment types and installation sites. Mechanical coupling and long-term durability become critical design factors, as poor structural integration or excessive wake galloping may lead to instability, overload, and physical damage. These mechanical behaviors directly affect energy availability and harvester lifespan over extended deployments. At the system level, energy constraints impose data reliability and timing trade-offs, forcing careful coordination between sensing, processing, and transmission to maintain predictable operation. Engineers often rely on vibration characterization and conservative mechanical tuning to avoid resonant conditions that accelerate wear. In high-temperature environments, material selection and thermal stability further influence long-term performance and system safety. Industrial Use Cases Where Piezoelectric Harvesting Excels Condition monitoring on rotating and reciprocating machinery provides steady mechanical input, opening new possibilities for powering IoT nodes and sensors without batteries. This capability enables battery-less IIoT deployments in confined or otherwise inaccessible locations where routine maintenance introduces safety and operational risk. Retrofit scenarios benefit in particular, as adding wiring or maintaining battery access often proves impractical or cost-prohibitive. By harvesting energy already present in machine motion, these systems expand monitoring coverage while reducing long-term maintenance exposure across industrial assets. The result is higher sensor density without a corresponding increase in service overhead. For IIoT architects, this shift supports scalable monitoring strategies that remain viable over a multi-year equipment life cycle. When Battery-Less IIoT Architectures Make Sense Replacing batteries with self-powered designs requires clear decision criteria that account for energy availability and acceptable data latency across operating conditions. In many industrial deployments, hybrid approaches that combine energy harvesting with minimal energy storage balance reliability and system flexibility. Long-term scalability and life-cycle planning depend on how effectively these architectures reduce maintenance events as IIoT networks expand. Industrial piezoelectric sensors often sit at the center of this strategy, providing predictable mechanical energy conversion without introducing new service or replacement dependencies. Designing Industrial IoT Systems for Long-Term Resilience Sustainability becomes an operational engineering outcome when high-temperature energy harvesting removes batteries from harsh industrial environments. Self-powered sensors reduce risk, cost, and maintenance exposure by eliminating replacement cycles in hazardous and inaccessible locations. Energy harvesting now stands as a core design pattern for next-generation industrial IoT systems built for longevity and scale.

By Emily Newton

DataFlow — An Open-Source Data Preparation System Accelerating LLM Training

Competition among large language models (LLMs) has intensified significantly over the past two years, with many believing that their core competitiveness lies in algorithms. However, this is not the case. The current open-source ecosystem has made mainstream architectures increasingly transparent — model structures such as Llama, GPT, and Gemma can all be publicly reproduced, and the competitive edge at the algorithmic level is rapidly eroding. The real competitive barrier actually exists at a more fundamental level — data. Data is the sole source of knowledge for LLMs, and data quality determines a model's "emotional intelligence" and "intelligence quotient." This means the development of LLMs has largely relied on large-scale, high-quality training data. However, most mainstream training datasets and their processing workflows remain undisclosed, and the scale and quality of publicly available data resources are still limited. This poses significant challenges for the community in building and optimizing training data for LLMs. Additionally, although there are already a large number of open-source datasets, making them AI-ready remains an obstacle for both the community and industry due to a lack of systematic and efficient tool support. Existing data processing tools, such as Hadoop and Spark, mostly support operators oriented toward traditional methods rather than effectively integrating intelligent operators based on the latest LLMs. Moreover, they provide limited support for constructing training data for advanced large models. How can we address this dilemma? DataFlow: A Data Preparation Engine for LLMs As data preparation becomes the main battlefield of competition, the open-source technology ecosystem is becoming the key to breaking the deadlock. That’s why we created DataFlow, a data-centric AI system that transforms “black-boxed” data preparation engineering capabilities into reusable and scalable open-source AI infrastructure. DataFlow fully supports text-modality data governance and also supports extracting and translating text content from PDFs, web pages, and audio. The processed data can be used for pre-training, supervised fine-tuning (SFT), and reinforcement fine-tuning of LLMs. It can effectively improve the inference and retrieval capabilities of LLMs in both general domains and specific domains such as healthcare, finance, and law. DataFlow Technical Framework When the complexity of LLM data preparation becomes the biggest bottleneck for model evolution, the traditional pattern of “isolated tools + manual orchestration” is clearly not the optimal solution. The technical framework of DataFlow follows a streaming architecture of “input → processing → output,” covering the entire journey from raw data processing to application implementation. Its core is divided into three major layers: Data Input Layer DataFlow supports multimodal machine learning data, such as JSON, PDFs, images, and videos. Key Design: Unified Data Carrier: A pandas DataFrame carries multimodal machine learning data in a structured format.Scalability: A reserved multimodal processing interface (the current version focuses on text; image and video support are under development). Core Processing Layer The core functionality of DataFlow lies in the processing layer, which comprises three modules: Operator, Pipeline, and Agent. DataFlow Operator System An operator is a basic data processing unit that typically executes logic based on rules, deep learning models, or LLMs. Operator TypeUse CasesMultiModal OperatorsPNG → OCR MP4 → automatic speech recognition image → text descriptionGeneral-purpose operatorsData filter/deduplicate/diversity controlDomain-specific operatorsMedical entity identification, financial compliance testingEvaluation operatorsGive a score on Security/Complexity/Inference Difficulty DataFlow Pipeline A pipeline is the logical orchestration of multiple DataFlow operators, designed to complete a full data processing task. DataFlow currently provides eight pipelines as references, and they can also be customized or modified. Preinstalled Pipelines (Out of the Box): Strong Reasoning Synthesis: Generate mathematical or code reasoning chain dataAgentic RAG Optimization: Build a high-quality knowledge base for Retrieval-Augmented GenerationText2SQL: Precise mapping from natural language to SQLKnowledge Base Cleaning: Extract information from PDFs, web pages, and audio, and construct RAG knowledge fragments or question–answer pairs…… Customized Pipelines: Graphical Drag-and-Drop: Connect operators to build a DAG (no code required)YAML Configuration: Supports versioned management and reuse DataFlow Agent The DataFlow Agent is an automated task-processing system based on multi-agent collaboration. It covers the entire workflow of “task decomposition → tool registration → scheduling and execution → result verification → report generation,” and is designed for the intelligent management and execution of complex tasks. Some agent capabilities include: Automatically arranging operators according to user queries to form new pipelinesAutomatically writing new operators based on user queriesAutomatically resolving data analysis tasks Output Layer The generated high-quality data can meet the requirements of LLM training and industry scenarios. Examples include: Multi-dimensional Assessment Reports: Visual displays of quality improvements in cleaned or synthesized dataDownstream Scenario Support, including: Model Training: High-quality data for all stages of pre-training, SFT, and RLHFVector Databases: Output of <Question, Evidence Fragment, Answer> triples adapted for RAGDomain-specific Knowledge Bases: Knowledge Q&A and decision support for the medical and financial industries…… DataFlow Quick Start Guide Next, let’s review best practices for installing and deploying DataFlow. Environment Preparation System Requirements: Operating System: Linux / macOS / Windows (Linux recommended)Python: Version 3.10 or higherConda: For environment isolation and dependency managementIDE: VSCode or PyCharm Recommended Directory Structure: Plain Text workspace/ ├── dataflow_env/ ├── pipelines/ ├── data/ ├── cache_local/ └── logs/ (Note: Simply prepare an empty folder named workspace. The subdirectories, such as pipelines, can be automatically generated by subsequent commands.) Environment Configuration Create a Conda Environment Shell conda create -n dataflow python=3.10 -y conda activate dataflow Tip: The -y parameter automatically confirms installation. Without it, you will be prompted to enter y manually. Install DataFlow Shell pip install open-dataflow # or pip install "open-dataflow[vllm]" We recommend using pip install open-dataflow initially. If you have a GPU, you can later install the version with vllm. Verify Installation Shell dataflow -v If the following message appears, the installation was successful: Plain Text You are using the latest version: x.x.x Note: The message indicating the successful installation is the same for all versions. Project Initialization and Operation Verification Initialize the Project Directory Shell dataflow init After execution, a default pipeline example and configuration file will be generated in the current directory (as shown in the figure below). Run the Example Pipeline Locate the target pipeline file in the working directory.Configure the data source (sample data can be found in the example data directory).Input command: python + target pipeline file pathRun: Shell python example_data/example_pipeline.py The result file will be generated in the cache_local/ directory. Advanced Deployment Practice Build from Source Suitable for developers who need to modify underlying logic or debug the framework. Shell git clone https://github.com/OpenDCAI/DataFlow.git conda create -n dataflow_diy python=3.10 conda activate dataflow_diy cd DataFlow pip install -e . Verify installation: Shell dataflow -v Download a Dataset from Hugging Face Install the package: Shell pip install huggingface_hub Create hf_download.sh: Shell export HF_ENDPOINT=https://hf-mirror.com rep="<huggingface dataset name>" local_dir="./data" huggingface-cli download $rep \ --repo-type dataset \ --local-dir $local_dir \ --force Run the script, and the style of the downloaded dataset is shown in the figure below: Shell bash hf_download.sh Run a Custom Pipeline The steps are similar to those above: Shell python pipelines/custom_pipeline.py --config config/custom.json The input source, operator order, and output path can be flexibly controlled through the configuration file. That concludes the quick start guide for DataFlow. Technical documentation is also available, and the community is welcome to share insights and contribute. Conclusion: A New Paradigm for Data Engineering As the open-source LLM ecosystem continues to grow, one pattern is becoming clear: models evolve quickly, but data challenges remain difficult. DataFlow reframes data as a first-class, evolving system. It introduces operators for each stage of data processing — parsing, generation, filtering, evaluation, and feedback — that can be versioned, debugged, and improved independently, just like model code. For developers building, training, and maintaining open-source LLM systems, this shared structure transforms isolated efforts into collective progress.

By Nan Xiang

IoT

DZone's Featured IoT Resources

Top IoT Experts

The Latest IoT Topics